Distributed Deep Learning In Open Collaborations
Authors: Michael Diskin
Alexey Bukhtiyarov
Max Ryabinin
Lucile Saulnier
Quentin Lhoest
Anton Sinitsin
Dmitry Popov
Dmitriy Pyrkin
Maxim Kashirin
Alexander Borzunov
Albert Villanova del Moral
Denis Mazur
Ilia Kobelev
Yacine Jernite
Thomas Wolf
Gennady Pekhimenko
Venue: NeurIPS 2021 Poster (2021)
Keywords: distributed training
volunteer computing
heterogeneous hardware
self-supervised pretraining
transfer learning

Abstract: Modern deep learning applications require increasingly more compute to train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High-Performance Computing clusters, whose construction and maintenance are both environmentally costly and well beyond the budget of most organizations. As a result, some research directions become the exclusive domain of a few large industrial and even fewer academic actors. To alleviate this disparity, smaller groups may pool their computational resources and run collaborative experiments that benefit all participants. This paradigm, known as grid- or volunteer computing, has seen successful applications in numerous scientific areas. However, using this approach for machine learning is difficult due to high latency, asymmetric bandwidth, and several challenges unique to volunteer computing. In this work, we carefully analyze these constraints and propose a novel algorithmic framework designed specifically for collaborative training. We demonstrate the effectiveness of our approach for SwAV and ALBERT pretraining in realistic conditions and achieve performance comparable to traditional setups at a fraction of the cost. Finally, we provide a detailed report of successful collaborative language model pretraining with nearly 50 participants.

Distributed Deep Learning In Open Collaborations
Michael Diskin V+
Alexey Bukhtiyarov
Max Ryabinin to
Lucile Saulnier
Quentin Lhoest
Anton Sinitsin
Dmitry Popor
Dmitry Pyrkin
Maxim Kashirin
Alexander Borzunov
Albert Villanova del Moral
Denis Mazur
Ilia Kobelevi
Yacine Jernite
Thomas Wolf
Gennady Pekhimenko
Yandex, Russia
Hugging Face, USA
V HSE University, Russia
Moscow Institute of Physics and Technology, Russia
+ University of Toronto, Canada Vector Institute, Canada
Abstract
Modern deep learning applications require increasingly more compute train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High-Performance Computing clusters, whose construction and maintenance are both environmentally costly and well beyond the budget of most organizations. As a result, some research directions become the exclusive domain of a few large industrial and even fewer academic actors. To alleviate this disparity, smaller groups may pool their computational resources and run collaborative experiments that benefit all participants. This paradigm, known as grid- or volunteer computing, has seen successful applications numerous scientific areas. However, using this approach for machine learning is difficult due high latency, asymmetric bandwidth, and several challenges unique to volunteer computing. In this work, carefully analyze these constraints and propose novel algorithmic framework designed specifically for collaborative training. We demonstrate the effectiveness our approach for SwAV and ALBERT pretraining in realistic conditions and achieve performance comparable to traditional setups at a fraction of the cost. Finally, we provide a detailed report of successful collaborative language model pretraining with 40 participants.
Introduction
The deep learning community is becoming increasingly more reliant on transfer learning. computer vision, pretraining convolutional networks large image collections such ImageNet [1] the de facto standard for a wide range of applications ranging from object detection [2] and semantic segmentation [3] image classification [4] and even learning perceptual similarity [5]. A growing number of natural language processing systems capitalize on language models with billions of
"Equal contribution. Correspondence mryabininO@gmail.. com Detailed author contributions are listed at the end of the work.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

parameters [6. 8, 9, 10, 11] trained on vast unlabeled corpora. Similar trends have emerged in areas such as speech processing [12], reinforcement learning [13], and computational biology [14, 15].
Training these models is a notoriously time-consuming and challenging task: it often requires hundreds of high-end GPU servers [10, 16] and would take multiple years on single device [17].
Most academic and independent researchers simply cannot afford to train state-of-ther thr- models from scratch, which slows down scientific progress and practical adoption of deep learning.
Historically, the deep learning community has addressed this problem via "model hubs" or "model zoos"- public repositories for pretrained model checkpoints [18, 19, 20, 21]. These repositories have played a significant role in the democratization of deep learning, allowing everyone to reap the benefits of large-scall scale training runs conducted by corporations and universities with sufficient resources. However, model hubs are limited to a narrow subset of datasets and tasks that match the interests of model creators. For instance, in natural language processing. it is often difficult to find up-to-date models for more than a handful of languages [22]. In turn, computer vision hubs rarely feature models trained on drawings, satellite images, 3D renders, microscopy, or any other data that does not resemble ImageNet. As result, many researchers in these areas can only work on problems for which there are available pretrained models rather than the problems that most need solving.
However, there might be an alternative way to obtain pretrained models: to train these models collaboratively. This approach, known as volunteer (or grid) computing, allows many independent parties to combine their computational resources and collectively perform large-scale experiments [23, 24, 25]. The compute performance of such collaborations often exceeds that of the fastest supercomputers [26]; however, fully utilizing it can be challenging due to several reasons. First, devices that contribute to collaborative experiments can range from GPU servers and high-end workstations to consumer-grade computers and even smartphones [27]. Second, most of these devices use household internet connection with limited bandwidth and low reliability. Third, participants in such projects often donate their hardware part-time, joining and leaving the experiment at will.
While it is theoretically possible to train neural networks this kind of infrastructure, modern distributed training strategies are only efficient in a narrow range of conditions. For instance, training with Ring All-Reduce [28] works well for identical servers but suffers significant performance penalties from network latency or bandwidth variation [29]. Another technique known as Parameter Server can handle heterogeneous devices at the cost of being less scalable [30]. Applying any of these strategies outside their preferred conditions may significantly reduce the training throughput [31], which makes them difficult to apply in the volatile infrastructure of volunteer computing. This issue is further complicated by the unique limitations of volunteer devices, such as network address translation (NAT), regional access restrictions, or variations in performance.
In this study, we carefully analyze the above challenges and come up with a practical solution for Distributed Deep Learning in Open Collaborations (DeDLOC). DeDLOC is based on novel algo- rithm that adapts to the available hardware in order to maximize the training throughput. Depending on the infrastructure, DcDLOC can recover parameter servers [30], All All-Reduce SGD [32], decen- tralized SGD [33], BytePS [34], or an intermediate strategy that combines all of them. Using this algorithm, propose system for collaborative training designed accommodate large number of heterogeneous devices with uneven compute, bandwidth, reliability, and network capabilities.
The contributions of our work can be summarized as follows:
We analyze the unique challenges of distributed training in open collaborations and propose a practical recipe for training in these conditions.
We formulate a novel distributed training algorithm that interpolates between traditional strategies to directly maximize the training performance for the available hardware.
We verify the effectiveness of the proposed algorithm and system design for unsupervised pretrain- ing of ALBERT-Large and SwAV under realistic conditions.
We run collaborative training with actual volunteers, achieving competitive results to models trained on hundreds of data center GPUs We also report insights on the collaborator activity and share the codebase for running similar experiments in the future?
"ode and training configurations are available at gi com./yandex-eesoarcch//oooo

---

2 Related work
2.1 Distributed training
In this work, we focus on distributed data-paraltel training, where each device runs forward and backward pass of the entire model on subset of training examples. While there are many alternative techniques [35, 36, 37], data-parallel is still the most popular strategy. Even the model-parallel
approaches for extremely large models rely on data parallelism the top level [37, 16, 38].
Training on multiple nodes was first implemented with parameter server (PS) [30]. This training strategy relies dedicated node that stores model parameters and executes optimization steps using the gradients sent by workers. In turn, worker nodes iteratively download the latest version of model parameters from the server, compute gradients and submit them back to the PS. This strategy is easy to implement and use, but it has an unavoidable bottleneck: the entire system performance is limited by the network throughput of single server. Since then, the scientific community proposed numerous extensions to PS that alleviate the bottleneck by reducing the communication load [39, 40, 41, 42, 43), introducing asynchronous updates [44, 45] or training with multiple servers [46, 34].
The issue of uneven communication load has also inspired the development and widespread adoption of another group of methods that rely on All-Reduce for gradient averaging [47, 48, 49], All-Reduce is a family of collective operations that allow nodes to efficiently aggregate (e.g. average) their local vectors and distribute the result across all devices [28, 50, 51]. Unlike parameter servers, All-Reduce assigns equal roles to all devices, making it easier to scale to a large number of homogeneous workers.
The popularity of AR-SGD sparked many practical applications for different scenarios. One par- ticularly relevant application elastic training [52, 53), which allows the user to add or remove workers at any point without interrupting the training run. While this bears a lot of similarity with collaborative training, we have found that elastic training systems are designed around global state synchronization, which makes them highly dependent on the homogeneity of the workers and their network connectivity. The overall efficiency is bounded by the performance of the lowest-performing node; as result, introducing even single wowbbanwwidth participant such systems reduces the training speed by orders of magnitude.
Seeking to avoid the need for synchronization and centralized orchestration, the research community has developed decentralized training algorithms. These algorithms can be broadly divided into two categories: directly passing updates between peers [54, 55] or running All-Reduce in small alternating groups [56, 29]. Compared to PS and All-Reduce, both categories provide a greater degree of fault tolerance but often require more steps to converge due to delayed updates [33, 29].
Most practical use cases of the above techniques take place in HPC or cloud conditions, but there is one notable exception. In Federated Learning, multiple parties train a shared model on decentralized privacy-sensitee sensitive data that cannot be shared between devices [57]. For that reason, federated learning algorithms prioritize data privacy over training efficiency, often leaving most of the compute resources unused [58, 59]. For more detailed overview of Federated Learning, refer to Appendix A.
2.2 Volunteer Computing
Volunteer computing (VC) is a paradigm of distributed computing where people donate the idle time of their desktops, smartphones, and other personal devices solve computationally hard problem collectively. This approach has seen successful applications in bioinformatics, physics and other scientific areas [60, 61, 62, 23, 63, 64, 65].
In all these applications, volunteer computing allows researchers to access vast computational re- sources. In Folding @ home, over 700,000 volunteers have collectively contributed 2.43 exaFLOPs of compute to COVID-19 19 research in April of 2020 [26]. Another project named BOINC (Berkeley Open Infrastructure for Network Computing) brings together 41.548 LOPs from over 790,000 active computers as of 17 March 2020 [25]. Volunteer computing systems were also the first "super- computers" to reach petaFLOP and 1 exaFLOP barriers [26, 66). These results became possible due to the contributions of a broad range of devices from high-end workstations to smartphones and even gaming consoles [67].
Unfortunately, this compute diversity is also the main limitation of VC. Any volunteer computing system should be able to run wide range of available hardware and maintain integrity even if
3

---

some participants disconnect. Furthermore, the resources available to project can vary over time, as most volunteers are only sharing their hardware when it unused. Finally, volunteer devices are interconnected with shared high latency network at typical home internet connection speeds.
As result, there only few successful attempts apply volunteer computing to machine learning workloads. One such project is MLC@ Home [68], which relies on volunteers to train many small independent models. This specific problem can be solved with no direct communication between participants. By contrast, distributed training of a single model requires significantly more communication and does not allow a natural way to "restart" failed jobs. When it comes to distributed training of neural networks. most volunteer computing projects rely on parameter server architectures [69, 70, 71]. As a result, these systems are bounded by the throughput of parameter servers and the memory available on the weakest GPU. The only notable exception home
Learning @ home [72], which uses expert parallelism to train larger models spanning multiple
computers; however, this approach has only been tested in simulated conditions.
Distributed Deep Learning in Open Collaborations
There are two unsolved challenges that stand in the way of practical collaborative training. The first challenge is algorithmic: how to maintain optimal training performance with dynamically changing hardware and network conditions? Another major challenge ensuring consistent training outcomes with inconsistent composition of participants. Thus, we organize this section around these two issues:
Section 3.1 provides general overview of DeDLOC and explains how maintains consistency in a dynamic environment.
In Section 3.2, describe the generalized communication strategy that maximizes training throughput by adapting to the currently available devices.
In Section 3.3, we address system design challenges, such as circumventing NAT and firewalls, training on large datasets and managing collaborator access.
3.1 Ensuring training consistency
Many state-of-the-at models, notably GANs [73] and Transformers [74], require a strict training regimen. Deviating from the recommended batch size introducing stale gradients may significantly affect the training outcome [75, 76, 77]. Since in collaborative setting one has little control over the devices that participate in the experiment, it is almost guaranteed that the specific hardware setup will vary between runs and even during a single run. Without special precautions, these runs may result in models with vastly different final accuracy.
To avoid this pitfall, DeDLOC follows synchronous data-paralle training with fixed hyperparameters regardless of the number of collaborators. In order to compensate for relatively slow communi- cation, we adopt training with extremely large batches [78, 49, 75, 79, 10], which allows peers to communicate less frequently. This strategy also provides a natural way to deal with heterogeneous hardware [80]: each device accumulates gradients its own pace until the collaboration reaches the target batch size. Once ready, the collaborators exchange their gradients and perform one optimizer step. Using synchronous updates makes DeDLOC mathematically equivalent to large-batch training on a regular HPC cluster; see Appendix G for more detailed explanation. Figure gives high-level visual explanation of this algorithm.
Time
Peer
Microbatch
Stute averaging
P3
Peer failure
P4
P5
Figure 1: Two DeDLOC training iterations with example collaborator dynamics.

---

3.2 Adaptive averaging algorithm
As we discussed Section 2.1, each distributed training algorithm has narrow range of conditions where it can reach optimal performance. For instance, Ring All-Reduce works best on homogeneous hardware with low-latency communication, while Parameter Server strategy requires dedicated high-bandwidth devices that communicate with large number of "workers". Since all devices are provided by volunteers, our training infrastructure is in a constant state of flux.
For instance, a collaboration can start with several homogeneous nodes that could be trained optimally with All-Reduce. If new participants bring devices with less bandwidth, it may be more efficient to use the original nodes as parameter servers. As more peers join, these servers will eventually become unable to handle the network load and the collaboration will need to switch a different strategy.
Running efficient training on this kind of infrastructure requires a protocol that can dynamically assign roles to every peer given their hardware and network capabilities:
Compute performance: Each peer i E .... compute gradients over 8; samples per second. A peer that is unable to compute gradients (i.e. that has no GPU) will have s;=0.
Bandwidth: Peers communicate with limited throughput: d; for download and u; for upload. Geographical limitations: In addition to individual bandwidth, the communication throughput between two peers i is also restricted by ty and ty: in each direction.
Given these constraints, our objective throughput, that is, the one that makes the most SGD steps with target batch size B per unit of time. In turn, the training throughput of collaboration depends on how we split the load among the participants. Each peer can be assigned to compute gradients over subset of training examples, aggregate part of those gradients from all peers, or both.
find a communication strategy that has the highest training
For simplicity and efficiency, we use delayed parameter updates (DPU) [81] - a technique that allows gradient computation and communication to run in parallel, at the cost of exactly one round of staleness. This strategy can improve time to convergence for wide range of models, including Transformers [81, 82]. That said, our approach can be easily adapted to non-concurrent updates.
With DPU, the frequency of training updates is determined by either the time to compute gradients or the time to aggregate them, whichever takes longer. In total, a collaboration processes Σ 8 cs samples per second, where G is the binary indicator denoting whether i-th peer is assigned to contribute gradients. Assuming the target batch size B, the frequency of the computation phase can be expressed as Feompute Em= 8; Ca ///BB
During the communication phase, each peer is first assigned to accumulate gradients over a fraction of model parameters. After that, everyone partitions their local gradients and sends each partition to the corresponding peer. On the other end, receiver nodes accumulate the gradients from all senders and return the average. In modern distributed training systems, this procedure is highly parallelized [34, 83]: reducer can aggregate one chunk of gradients while downloading the next chunk and distributing the previous one back to the same senders.
In order to properly optimize the training throughput, we must account for this parallelism. As such, we explicitly define the speed diy at which peer sends gradients to peer j for aggregation. turn, j-th peer aggregates gradients from all peers at the rate of the slowest sender aj min Gij. The senders can then get the aggregated results from the j-th reducer at gjz S aj. Finally, the total a. and gi, for each peer cannot exceed their maximum download/upload speed. The only exception is that transfer within one node (aat, 9a)) does not count towards network throughput.
The frequency of the gradient aggregation phase is simply the rate at which the slowest peer can aggregate the full gradient vector: Fage min, Σ, 9yr / P. where P is the number of model parameters. The final optimization problem can be formulated follows:
max 4.9.0
min
8. B
min: E, 9n P
s.t.
91
millkee==1 aki
Vi,j Vi
(1)
/41 (azi + 9js < d;
Σ jzi (aij + Dij) < u;
Vi
Oij + Dij S tij
Vi.j
5

---

Equal bandwidth
All-Reduce
One fäst peer
Parameter Server
Heterogeneous
Hybrid
Figure 2: Example collaboration setups and corresponding strategies for optimal averaging. Each square represents one of the peers, line thickness denotes pairwise connection speed.
This problem must be solved regularly as participants joining and leaving. Thus, must ensure that the benefits of the optimal strategy outweigh the overhead of computing it. For that reason, we formulate optimal strategy search as linear program that can be solved efficiently'. more formal definition of problem (1) with detailed LP reduction can be found in Appendix B.
After this problem is solved, assign each peer aggregate fraction of gradients proportional to minj 9ji- Peers with 4=1 are also tasked with computing the gradients, while peers with c== remain idle and only participate in communication. This results in natural division of labor. In the presence of many compute-heavy peers, some participants without accelerators will dedicate all their bandwidth to gradient aggregation instead of sending their local gradients.
Node failures. The resulting procedure can find the optimal communication strategy for averaging gradients across all participants. However, the number of participants grows, it might be impractical to compute the global average due to node failures. Based on our experiments with several hundred active volunteers, most training iterations will have least participant with network issues. This implies that without necessary precautions, the entire averaging round will fail more often than it will succeed. To combat this issue, we techniques [56, 29] that replace global averaging with several consecutive iterations in alternating groups of size m. The groups are chosen in such a way that the collaboration can obtain the exact average in logm. steps. Furthermore, if any single participant fails, will only affect his immediate group rather than the entire collaboration.
We adaptively choose the optimal group size m based on the number of peers and their failure rates. This optimization problem is independent Equation (1) and aims to maximize the rate at which collaborators compute the global average. We elaborate this procedure in Appendix C.
Comparison with existing techniques. Our method was designed as a generalization of existing data-pant parallel strategies that recovers them in special cases. To illustrate this idea, we provide example configurations for which DeDLOC recovers specific well-known strategies:
1. AR-SGD: homogeneous collaboration with reliable peers will use Butterfly All-Reduce [84];
Parameter Server: adding a single participant with a very high bandwidth and low compute
performance will turn the previous collaboration into parameter server [30];
3. BytePS: participants with the same bandwidth as AR-SGD nodes, but without compute accelera- tors, will behave as auxiliary summation services from BytePS [34];
4. Decentralized SGD: any collaboration with sufficienlly high failure rate will converge to m-22.
In this mode, all communication is performed between pairs of nodes, similarly to D-PSGD [33].
However, when training with actual volunteer devices, DeDLOC typically follows hybrid communi- cation scheme that differs from each the above options. We display several examples schemes that can arise solution for the optimal strategy search problem in Figure 2.
3.3 System design
Training with volunteer hardware requires specialized system architecture that can dynamically scale with collaboration size and recover from node failures. DeDLOC achieves these properties by operating as a swarm, similarly in spirit to BitTorrent [85] and I2P [86]. Individual peers coordinate by forming a Distributed Hash Table a fully decentralized fault-tolerant key-value storage [87, 88). Collaborators use this shared "dictionary" to count the number of accumulated gradients, find groups for averaging and keep track of the training progress.
In experiments, the LP solver consistently converged in 50ms and called z 2 times per minute.

---

DeDLOC ensures that all peers use up-to-date parameters by tracking the number of global steps of each peer. If a peer skips a step, it will observe that others made more steps and download the latest parameters and optimizer statistics from one of the up-to-date peers before resuming training.
In order to ensure the integrity of DHT throughout the training run, DeDLOC requires few peers with stable internet access. These "backbone" peers are responsible for welcoming new collaborators and performing auxiliary functions, such as storing checkpoints and tracking learning curves. The only requirement for those peers is that at least one of them is available at all times. As such, the backbone peers can be hosted on inexpensive servers without GPU (see Appendix F for cost analysis).
All other devices are treated as regular collaborators. Depending on their hardware and network bandwidth, these devices can be assigned to (i) compute gradients, (ii) aggregate gradients computed by other peers or (iii) do both, according to the adaptive averaging algorithm. However, performing these steps with actual volunteer devices requires solving another set of challenges described below.
Training under NAT and firewalls. In addition having uneven compute and network capabil- ities, volunteer devices also deviate from traditional servers in network configuration. One major difference is the use of Network Address Translation (NAT) [89] - the technology that allows multi- ple devices to share the same IP address. In practice, the majority of household and organizational computers around the world use one or multiple layers of NAT (see Appendix D for more details). Unfortunately for distributed training, NAT makes it harder to establish peer-to-peer connections [90].
When operating under NAT, DeDLOC participants use one of the following techniques:
1. Hole punching: use third peer to temporarily open access to both devices. Once both peers are accessible, they can establish direct connection and transfer data as usual [90];
2. Circuit relays: both devices connect to a relay (another peer that is mutually accessible), then forward all communication through that relay [91];
3. Client mode: if everything else fails, peer can still send gradients to others without the need for incoming connections. This imposes an additional constraint a; for Equation (I).
similar set strategies can be found in a wide range of distributed systems that rely on peer-to-peet communication, such as WebRTC, VolP (IP telephony), and BitTorrent. Most of these systems rely on dedicated servers to establish connections between peers. However, in our case it is more appealing to use fully decentralized NAT traversal where the regular peers perform hole punching and relaying by themselves. We describe this approach in more detail in Appendix E.
Training on large datasets. Many prospective applications of DeDLOC require training on large datasets that can take multiple hours to download. We circumvent this problem by allowing par- ticipants to download the data progressively during training. To support this behavior, we split the dataset into shards; upon joining the collaboration, peer begins downloading examples shard by shard a streaming fashion. Once the first several examples are obtained, a collaborator can begin training right away while downloading the rest of data in background.
To ensure that the training examples are independent and identically distributed, each participant loads shards in different random order and uses buffer to shuffle the data within each shard. Each participant loads the first S = 10, 000 examples into a buffer, then randomly picks a training batch from this buffer and replaces the chosen examples with newly downloaded ones. In experiments, we stream the training data from a dedicated storage service. However, this service can be replaced with peer-to-peet data sharing protocol akin to BitTorrent; see Appendix H for details.
Collaborator authentication. Many prospective applications of DeDLOC need way to keep track of individual peer contributions and protect against malicious peers. In our experiments, we achieve this using an allowlist authentication system that we describe in Appendix I.5.
4 Experiments
In this section, we evaluate the performance of DeDLOC in realistic collaborative training conditions. Our primary focus is on training models that are useful for wide range of downstream tasks and thus would attract a large number of collaborators. One area that fits this description self-supervised learning, i.e., learning reusable feature representations on large unlabeled datasets. First, we conduct controlled experiments on two popular self superviedd learning tasks Sections 4. I and 4.2. Then, we set up real-world collaborative training run with volunteers and report our findings in Section 4.3.
7

---

4.1 Self-supervised learning of visual representations
Our first set of experiments uses SWAV [92] - a self-supervised learning technique that learns image representations by contrasting cluster assignments. Similarly to the original paper, we train the ResNet-50 [93] model on the ImageNet dataset [1] without labels. Our experiments follow the recommended training configuration 192, 94]: 2+6 random crops, early prototype freezing and queue with 3,840 samples for each worker, LARS [78] optimizer, and 32,768 samples per batch across all workers. In this and further experiments, for decentralized averaging. We train with three hardware setups: SERVER, WORKSTATION and HYBRID. The SERVER setup contains 8 workers, each with single V100 GPU and 1 Gb/s symmetric bandwidth. In turn, the WORKSTATION setup consists of 16 nodes with 1080 Ti and 200 Mb/s bandwidth per worker. Finally, the HYBRID setup combines both previous configurations for total of 24 nodes. Unlike servers, workstation GPUs train in full precision because they do not support accelerated f computations [96].
use Hivemind [95] implement the infrastructure
We report learning curves for each hardware configuration in Figure 3. As expected, the HYBRID setup converges the fastest, beating SERVER and WORKSTATION setups by 40% and 52% accordingly. When used in a supervised setting (Section 4.1 from the original paper), the model learned in this setup achieves comparable accuracy of 72.2%. Another important observation is that the workstation-only experiment achieves reasonable training throughput despite using dated hardware. To provide more insight into the performance of DeDLOC, different configurations. We report the mean over 100 averaging rounds; the standard deviation was below 1% in all setups. As demonstrated in Table 1, adaptive averaging does not affect the performance for homogeneous setups while running 1.9 times faster on the hybrid infrastructure.
also measure the time it takes to averaging in
SERVER
Algorithm
Setup
WORKSTATION
AR
PS
Ours
HYBRID
Loss
A: 8x1Gb/s
1.19
4.73
1.20
B: 16x0.22Gbs
5,3
39.6
5.3
5-
C:A+B +
5.69
14.1
2.96
D: B + 1x2.5Gb/s
5.3
3.22
3.18
0h
24h
48h
72h
96h
120h
Time elapsed
Figure 3: SwAV pretraining performance.
Table 1: ResNet-50 averaging performance.
4.2
Self-supervised pretraining for language understanding
Next, we investigate how collaborative training performs for more complex models. In this experiment, we pretrain the ALBERT-large [7] masked language model the WikiText-1033 103 dataset [97]. We chose this setup for two reasons: first, ALBERT is very sensitive to the choice of hyperparameters, and specifically batch size, even more than regular Transformers [75]. This makes it easier to verify that DeDLOC can reproduce the traiming conditions of regular data-parallel training. Second, because of weight sharing, training ALBERT is relatively more compute- and less cmmmunication-intensse than regular BERT [6], which makes possible to train with lower bandwidth.
As before, we follow the exact training configuration from the original paper, but use GPUs instead of TPUs. We use the implementation of ALBERT from the transformers library [99]. We run all experiments on cloud instances with Tesla T4 GPUs and report the training loss a function of time, similarly to [87, In order to evaluate how DeDLOC performs with different network speeds, we consider the following setups the same platform with controlled conditions:
High-bandwidth: 16 workers, each with Tesla T4 and 25 Gb/s symmetric bandwidth;
Heterogeneous: same, but with 4x 200 Mb/s, 8x 100 Mb/s and 4x 50 Mb/s bandwidths;
Heterogeneous load balancing: like Heterogeneous,, but with adaptive averaging (Section 3.2);
Auxiliary peers: the previous setup with 4 additional CPU-only peers Gb/s bandwidth.
Time-varying: same as previous, but with 8 additional peers at 100 Mb/s. The extra peers are training part-time, jointly alternating between hours of training and 8 hours of downtime.
8

---

High-bandwidth
Heterogeneous
-00
Scconds por step
8
load balancing
+5
+ 4x CPU-only
6-
10
time-varying
4-
2
16
0h
24h
72h
96h
120h
144h
168h
Number of of
Time elapsed
Figure 5: Scalability measure-
Figure ALBERT pretraining performance.
ments for ALBERT pretraining.
As one can see in Figure 4, naive training with low-bandwidth peers results in an 2 2.5x slowdown compared high-bandwidth ones. Enabling load balancing accelerates that setup by 47%. This effect grows to over 60% when adding 4 auxiliary peers. Finally, adding part-time peers allows the collaboration train at 74% the speed of the high-bandwidth setup without sacrificing the training stability. This turns the latter setup into a viable alternative to traditional distributed training without the need for expensive infrastructure (see the cost analysis in Appendix F), In addition, we demonstrate the high scalability of DeDLOC in Figure 5, which was obtained by running the same experiment with a varying number of nodes and measuring the time between gradient descent steps.
4.3 Real-wortd world collaborative training
For our final evaluation, we organized an actual collaborative training run with volunteer participants, who were asked to pretrain Transformer masked language model for the Bengali language. This task was chosen deliberately to show the benefits of collaborative training: Bengali has over 230M native speakers who can benefit from recent advances in NLP, but there few pretrained models available for this language. We recruited 30 participants received instructions training on local computers. To avoid bias, we did not encourage any specific form of participation: volunteers were free to choose what hardware they contributed and for how long.
volunteers and 10 outside collaborators. All contributing with free cloud platforms and access to code for
Specifically, we trained the ALBERT- CAR [100] multilingual corpus. The model was named sahajBERT after conducting a poll among the participants. We adapted our preprocessing by following the best practices for the Bengali language described in Appendix I.3. stream from mix of Wikipedia and OSCAR, the training process iteratively sampled examples from one or the other dataset, as described in Section 3.3. We accounted for uneven size and quality of data by oversampling Wikipedia by factor of 2, which resulted in mixing probabilities of 0.23 for Wikipedia and 0.77 for OSCAR. Other hyperparameters were set to same values as in Section 4.2. Also, in Appendix 1.7 we report the results of sahajBFRT-XX.. four times larger model with specialized architecture that used both GPU and TPU resources.
model on Wikipedia and the Bengali part of the OS-
In total, the 40 volunteers contributed compute time from 91 unique devices, most of which were running episodically. Figure 6b shows that although the median GPU time contributed by volunteers across all devices was 1.5 days, some participants ran the training script on several devices, attaining more than 200 hours over the duration of the experiment. With the exception of the start and the end of the collaborative run, the number of simultaneously active devices mostly varied between 15 and 35 depending on the local time. There was less activity in the last 3 days, likely because the volunteers could see that the model has converged on public Weights Biases [101] dashboard.
60
45%
45%
Active poers
45
30%
30%
30
15%
15%
15-
0%
0%
0
60
120
180
240
300
K80 RTX 2060 P100 V100
Time elagsed, days
Time,
GTX 1060
T4 RTX 4000
(a) Collaboration activity.
(b) Participation time histogram.
(c) Summary volunteer hard- ware with example GPUs.
Figure 6: Collaborative experiment summary.

---

As depicted in Figure 6c, individual device performance varied significantly among the collaborators. Along with the resources provided by participants, we also used 16 preemptible single-GPU cloud T4 instances for training. We have estimated that the average volunteer device consumed 6.95 GB of network traffic per hour of training. While this bandwidth usage by no means insignificant, is comparable with cloud gaming [102] or high-quality video streaming [103].
The model converged after 8 days of training, which is 1.8x fast as regular distributed training with 8 V100 GPUs that we ran as a baseline; Figure 7 displays the convergence plots for both setups. At the same time, the stepwise learning curves of the two runs were virtually identical (see Appendix 1.6), which supports our hypothesis that training with DeDLOC is equivalent to a regular large-batch SGD.
Collahomation
Model
Wikiann F1
NCC Accuracy
10
Baseline (8xV100)
bnRoBERTa
82.32 + 0.67
80.94 0.45
Training
IndicBERT
92.52 + 0.45
74.46 1.91
XLM-R
96.48 + 0.22
90.05 + 0.38
sahajBERT
95.45 + 0.53
91.97 = 0.47
sahajBERT-XL 96.59 + 0.26 92.91 + 0.43
Time elapsed, days
Figure 7: Training progress of sahajBERT.
Table 2: Downstream evaluation results.
Finally, we compared the Bengali language representations of sahajBERT with those of other pretrained models on several downstream applications. The first model is XLM-R Large [9]- cross-lingual Transformer-baal based masked language model that was pretrained on 100 languages and remains a strong baseline for multilingual representation learning. Similarly to sahajBERT, the second model, IndicBERT [104], is also based on the ALBERT architecture; however, it was pretrained on 12 languages, including Bengali and Indian English. The third model, bnRoBERTa [105], is a RoBERTa architecture trained on a monolingual Bengali corpus. We evaluate the model quality on two tasks: WikiANN [106] named entity recognition dataset and Soham News Category Classification benchmark from IndieGLUE [104]. For detailed description of the setup, refer to Appendix L88.
As shown in Table 2, BERT performs comparably to three strong baselines despite being pretrained in a heterogeneous and highly unstable setting. Notably, our collaboratively trained model outperforms two specialized monolingual baselines and demonstrates competitive results to XIMMR Large, even though the latter has significantly more parameters (560 million instead of 17 million) and was trained on five hundred high- nigh-performance data center GPUs instead of tens of low-cost or even free-tier accelerators. This result confirms previous findings on the benefits of parameter sharing that were made by authors of ALBERT. Also, it highlights one additional advantage of such architectures: specifically, one can train high-quality representation model in communication-constainne setting (for instance, over the Internet) without facing noticeable data transfer bottlenecks.
5 Conclusion
In this work, we proposed DeDLOC- a collaborative deep learning approach that enables large- scale collective distributed training on whichever computers available to participants, regardless of hardware and network limitations. We demonstrated with several experiments that this is viable approach that maintains its efficiency in a broad range of conditions. Finally, we report the first real collaborative training run of such a scale and share findings on volunteer activity pave the road for similar experiments in the future.
An essential property of collaborative training is its environmental impact. While all distributed training experiments have a negative impact due to carbon emissions [107], DeDLOC has one unique advantage. Due to the ability to utilize heterogeneous low-end devices, it can prolong the effective lifespan of existing computers. We discuss other aspects of environmental impact in Appendix J.
One issue that needs to be addressed before starting collaborative experiments is the need to gather a community of volunteers. Although our proposed authentication mechanism (see Appendix 1.5) allows acknowledging participants for their contributions (briefly discussed in Appendix 1.2), the best approach to recruit volunteers an open question: one needs to take into account both the resources of community members and their motivation for training a specific model.
10

---

Acknowledgements
We thank Stas Bekman, Dmitry Abulkhanov, Roman Zhytar, Alexander Ploshkin, Vsevolod Plokhot- nyuk and Roman Kail for their invaluable help with building the training infrastructure. Also, we thank Abhishek Thakur for helping with downstream evaluation and Tanmoy Sarkar with Omar Sanseviero, who helped us organize the collaborative experiment and gave regular status updates to the participants over the course of the training run. Finally, we thank the anonymous reviewers for their feedback on the content and the presentation of our paper.
In addition, authors would like to thank the students of Yandex School of Data Analysis who volunteered to participate in preliminary experiments.
We kindly thank all participants of the Neuropark community* who contributed sahajBERT training. Below, we list the community members who agreed to provide their name for this paper: Aakash Gupta, Aninda Goswamy, Anjali Prasad, Anurag Singh, Arijit Sarkar, Chirranjit Ghosh, Debajit Mallick, Ibraheem Muhammad Moosa, Ishan Bagchi, Khalid Saifullah, Laxya Agarwal, Manan Dey, Mir Ali. Mrinal Mathur, Nilavya Das, Preetha Suri, Priyadarshan Sarkar, Sagnik Roy, Sahil Saha, Sanjeev Kumnrr, Sanskar Upadhyay, Shyam Sunder Kumar, Soumi Kaibartya, Subhranil Sarkar, Sujit Pal, Syed Modassir Ali, Tanmoy Sarkar, and Vaishali Pal.
Training sahajBERT-XX and hybrid GPU-TPU experiments were made possible by John Kintree, Debajit Mallick, Avijit Saha, Ishan Bagchi, Nilavya Das, Priyadarshan Sarkar, Sagnik Roy, Eduard Pokonechnyy. Arina Ruck. Finally, we would like to acknowledge Tanmoy Sarkar for setting up the backbone peer for sahajBERT-XL on his server and contributing the evaluation codebase.
The computational resources for internal experiments on cloud instances were provided by the Amazon Research Awards program.
References
[1] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 06 2009.
[2] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 580-587, 2014.
[3] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431-3444 2015.
[4] J. Donahue, Y. Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. DeCAF: A deep convolutional activation feature for generic visual recognition. In Proceedings of the st International Conference International Conference on Machine Learning, pages 647-655, 2014.
[5] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, volume 9906, pages 694-711, 10 2016.
[6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186, 06 2019.
[7] Zhen-Zhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. International Conference on Learning Representations, 2020.
[8] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1692, 2019.
"huggingfacee.c//euuuppa

---

[9] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer. and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 8440-8451, 07 2020.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosber Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
[11] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. LLe Training multi-billion parameter language models using GPU model parallelism. arXiv preprint arXiv: (00.::/99000000555 0.0000,0....... 2019.
[12] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, volume 33, pages 12440, 2020.
[13] Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning:
A survey. arXiv preprint arXiv:2009. 200.00000.. 2020.
[14] Amy X. Lu, Haoran Zhang, Marzyeh Ghassemi, and Alan Moses. Self-supervised contrastive
learning of protein representations by mutual information maximization. bioRxiv, 2020.
[15] Shion Honda, Shoi Shi, and H. Ueda. SMILES transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint /91 2019.
[16] Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on GPU clusters using Megatron-LM. arXiv preprinl 1xXiv:210400447:: 2021.
[17] Jiahuang Lin, Xin Li, and Gennady Pekhimenko. Multi-node BERT-pretraining: Cost-efficient approach. arXiv preprint 2xX11:20088000007 2020.
[18] TensorFlow Hub. https://sww.eensortooooooo///// Accessed: 2021-05-20.
[19]
PyTorch Hub. https://pytoorccoorr///bb/ Accessed 2021-05-20.
[20] Hugging Face Hub. httpss//hhuggiinffccc..../oo
Accessed: 2021-05-20.
[21] Ryan Chard, Zhuozhao Li, Kyle Chard, Logan Ward, Yadu Babuji, Anna Woodard, Steven Tuecke, Ben Blaiszik, Michael J. Franklin, and Ian Foster. DLHub: Model and data serving for science. In 2019 JEEE International Parallel and Distributed Processing Symposium (IPDPS). pages 283-292, 05 2019.
[22] Pratik Joshi. Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
07 2020.
[23] David Anderson, Jeff Cobb, Eric Korpela, Matt Lebofsky, and Dan Werthimer. SETI SETI@home: An experiment in public-resource computing. Commun. ACM, 45:56-61, 61, 11 2002.
[24] L. Beberg, D. Ensign, G. Jayachandran, Khaliq, and V. Pande. Folding @ home: Lessons from eight years of volunteer distributed computing. 2009 IEEE International Symposium on Parallel & Distributed Processing, pages 1-8, 2009.
[25] David P Anderson. BOINC: system for public-resoure computing and storage. In Fifth IEEE/ACM international workshop on grid computing, pages 4-10. IEEE, 2004.
[26] @olding@ @ home gets 1.5+ exaflops to fight covid- 19. https://blon niiiia. wwidia.com/bbog.. com/blog/ http://olddiaateoeeeeeeeeeeeeeee
Accessed: 2021-05-20.
[27] C. Tapparello, Colin Funai, Shurouq Hijazi, Abner Aquino, Bora Karaoglu, H. Ba, J. Shi, and W. Heinzelman. Volunteer computing on mobile devices: State of the art and future research directions. In Enabling Real-Time Mobile Cloud Computing through Emerging Technologies,
pages 153-181. 2016.
12

---

[28] Pitch Patarasuk and Xin Yuan. Bandwidth optimal all-reduce algorithms for clusters of workstations. Journal of Parallel and Distributed Computing, 69:117-124, 02 2009.
[29] Shigang Li, Tal Ben-Nun, Giorgi Nadiradze, Salvatore Digirolamo, Nikoli Dryden, Dan Alistarh, and Torsten Hoefler. Breaking (global) barriers in parallel stochastic optimization with wait-avoiding group averaging. IEEE Transactions on Parallel and Distributed Systems, page 1-1, 2020.
[30] Mu Li, D. Andersen, J. Park, Alex Smola, Amr Ahmed, V. Josifovski, J. Long, E. Shekita, and Bor- Yiing Su. Scaling distributed machine learning with the parameter server. In Proceedings of the 2014 International Conference on Big Data Science and Computing, 2014.
[31] Shaohuai Shi, Qiang Wang, and Xiaowen Chu. Performance modeling and evaluation of distributed deep learning frameworks on GPUs. In IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing. 16th Intl Conf on Pervasive Intelligence and Computing. 4th Inil Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress, pages 949-957, 2018.
[32] Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint 1802.005999 2018
[33] Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentral- ized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. In Advances in Neural Information Processing Systems, volume 30, 2017.
[34] Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. A unified architecture for accelerating distributed DNN training in heterogeneous GPU/CPU clusters. In 14th USENIX Symposium Operating Systems Design and Implementation (OSDI 20), pages 463-479, 11 2020.
[35] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. GPipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems, pages 2019..
[36] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 170100:370..0633, 2017.
[37] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimization towards training trillion parameter models. In SC, 2020.
[38] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101 0096/1 2021.
[39] Yujun Lin, Song Han, Huizi Mao, Yu Wang. and William J Dally. Deep Gradient Compres- sion: Reducing the communication bandwidth for distributed training. In The International Conference on Learning Representations, 2018
[40] Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradient descent, In Advances Neural Information Processing Systems, volume 23, pages 2595-2603, 2010.
[41] Sebastian Urban Stich. Local SGD converges fast and communicates little. In International Conference on Learning Representations, 2019.
[42] Anastasia Koloskova, Tao Lin, Sebastian U Stich, and Martin Jaggi. Decentralized deep learning with arbitrary communication compression. In International Conference Learning Representations, 2020.
[43] Zhize Li, Dmitry Kovalev, Xun Qian, and Peter Richtarik. Acceleration for compressed gradi- ent descent in distributed and federated optimization. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5895-5904, 07 2020.
[44] Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. Advances in neural information processing systems, pages 693-701, 2011.
13

---

[45] Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 571 571-582, Broomfield, CO, October 2014, USENIX Association.
[46] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Mare' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang. Quoc Le, and Andrew Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, volume 25,
pages 1223-1231, 2012.
[47] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv preprint arXiv: 02677, 2017.
[48] Hiroaki Mikami, Hisahiro Suganuma, Pongsakorn U-chupala, Yoshiki Tanaka, and Yuichi Kageyama. Massively distributed SGD: ImageNe//ReNeet500 training in flash. arXiv preprint arXiv:1811 1XX0::811.00233, 2019.
[49] Yang You. Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations, 2020.
[50] Rajeev Thakur, Rolf Rabenseifneer and William Gropp. Optimization of collective com- munication operations in MPICH. Int. J. High Perform. Comput. Appl., 19(1):49-66, 02 2005.
[51] Paul Sack and William Gropp. Collective algorithms for multiported torus networks. ACM Trans. Parallel Comput., 1(2), February 2015,
[52] PyTorrch Elastic. httpss///yyoorch.org/eassssi Accessed: 2021-05-20.
[53] Elastic Horovod, http:://pooova.tttaaaaaaattttttt cessed: 2021-05-20.
Ac-
[54] Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 344-353, 06 2019.
[55] Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. SlowMo: Improving communication-efficien distributed SGD with slow momentum. In International Conference on Learning Representations, 2020.
[56] Max Ryabinin, Eduard Gorbunov, Vsevolod Plokhotnyuk, and Gennady Pekhimenko. Moshpit SGD: Communication-efficceell decentralized training on heterogeneous unreliable devices. arXiv preprint arXiv:2 03239, 2021.
[57] Brendan McMahan, Eider Moore, Daniel Ramage. Seth Hampson, and Blaise Aguera y Arcas. Communication--fficii learning of deep networks from decentralized data, In Artificial Intelligence and Stutistics, pages 1273-1282, 2017.
[58] K. A. Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé M Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning scale: System design. In Proceedings of Machine Learning and Systems (MI.Sys), 2019.
[59] Thorsten Wittkopp and Alexander Acker. Decentralized federated learning preserves model and data privacy. In International Conference on Service- Computing, pages 176-187. Springer, 2020.
[60] Stefan M. Larson, Christopher D. Snow, Michael Shirts, and Vijay S. Pande. Folding @ home and Genome home: Using distributed computing to tackle previously intractable problems in computational biology. arXiv preprint rxXiv00901.0860 2009.
[61] Boldinghhom update on SARS-CoV. (10 mar 2020). foldingatiooe.oor/2000///0
covid19-update, Accessed: 2021-05-20.
[62] Javier Barranco, Yunhi Cai, David Cameron, Matthew Crouch, Riccardo De Maria, Lau- rence Field, M. Giovannozzi, Pascal Hermes, Nils Høimyr, Dobrin Kaltchev, Nikos Karas- tathis, Cinzia Luzzi, Ewen Maclean, Eric Mcintosh, Alessio Mereghetti, James Molson, Yuri
14

---

Nosochkov, Tatiana Pieloni, Ivan Reid, and Igor Zacharov. LHC& Home: a BOINC-based volunteer computing infrastructure for physics studies CERN. Open Engineering, 7, 12 2017.
[63] Jeongním Kim, Andrew D) Baczewski, Todd D Beaudet, Anouar Benali, M Chandler Bennett, Mark A Berrill, Nick S Blunt, Edgar Josué Landinez. Borda, Michele Casula, David M Ceperley, Simone Chiesa, Bryan K Clark, Raymond C Clay, Kris T Delaney, Mark Dewing, Kenneth P Esler, Hongxia Hao, Olle Heinonen, Paul R C Kent, Jaron Krogel, Ilkka Kylänpää, Ying Wai Li, M Graham Lopez, Ye Luo, Fionn D Malone, Richard M Martin, Amrita Mathuriya, Jeremy McMinis, Cody A Melton, Lubos Mitas, Miguel A Morales, Eric Neuscamman, William D Parker, Sergio D Pineda Flores, Nichols A Romero, Brenda M Rubenstein, Jacqueline A R Shea, Hyeondeok Shin, Luke Shulenburger, Andreas F Tillack, Joshua P Townsend, Norm M Tubman, Brett Van Der Goetz, Jordan E. Vincent, D ChangMo Yang, Yubo Yang. Shuai Zhang, and Luning Zhao. QMCPACK: an open sourceab initioquantum monte carlo package for the electronic structure of atoms, molecules and solids. Jourmal of Physics: Condensed Malter, 30(19):195901, 04 2018.
[64] Folding @ home project timeline. cessed: 2021-05-20.
org wttpt//www.rrp/pojjjjcoooo// peojject-tmmelinn Ac-
[65] Steltner, M. A. Papa, H. B. Eggenstein, B. Allen, V. Dergachev, Prix, B. Machenschalk, S. Walsh, S. J. Zhu, and S. Kwang. Einstein @ Home all-sky search for continuous gravitational
waves in LIGO 02 public data. The Astrophysical Journal, 909(1):79, 03 2021.
[66] Michael Gross. Folding research recruits unconventional help. Current biology: CB, 22:R3-58.. 01 2012.
[67] Tetsu Narumi, Shun Kameoka, Makoto Taiji, and Kenji Yasuoka. Accelerating molecular dynamics simulations on playstation 3 platform using virtual-grape programming model. SIAM
J. Scientific Computing, 00:3108-3125, 01 2008.
[68] John Clemens. MLDS: A dataset for weight-space analysis of neural networks. arXiv preprint arXiv:2104. 10555, 2021.
[69] Gian-Carlo Pascutto and Gary Linscott. Leela chess zero. lczero.org, 2019. Accessed: 2021-05-20.
[70] Ekasit Kijsipongse, Apivadee Piyatumrong, and Suriya U-ruckolan. A hybrid gpu cluster and volunteer computing platform for scalable deep learning. The Journal of Supercomputing, 04 2018.
[71] Medha Atre, Birendra Jha, and Ashwini Rao. Distributed deep learning using volunteer computing-like paradigm. arXiv preprint 103.v22103.08894, 2021.
[72] Max Ryabinin and Anton Gusev. Towards crowdsourced training of large neural networks using decentralized mixture-of-experts. Advances Neural Information Processing Systems, volume 33, pages (679- 2020.
[73] Ian J. Goodfellow, Jean Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems- Volume 2, page 2672-2680, 2014.
Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil
[74] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008 2017.
[75] Martin Popel and Ondřej Bojar. Training tips for the transformer model. The Prague Bulletin of Mathematical Linguistics, 1 110 03 2018
[76] Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the difficulty of training transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747-5763, 11 2020.
[77] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. PipeDream: Generalized pipeline parallelism for DNN training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, page 1-15,0199
15

---

[78] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks arXiv preprint arXiv:1708.03330 03888, 2017.
[79] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0e. 1.0361, 2020.
[80] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine transla- tion. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages
1-9, 10 2018.
[81] Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. ZeRO-offload: Democratizing billion-scale model training. arXiv preprint vrXiv:2101.6644 2021.
[82] Alham Fikri Aji and Kenneth Heafield. Making asynchronous stochastic gradient descent work for transformers. Proceedings of the 3rd Workshop on Neural Generation and Translation, 2019.
[83] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training. Proc. VLDB Endow.. 13(12):3005-300 August 2020.
[84] Zhenyu Li, James Davis, and Stephen Jarvis. An efficient task-based all-reduce for ma- chine learning applications. In Proceedings of the Machine Learning on HPC Environments, MLHPCC 17, New York, NY, USA, 2017. Association for Computing Machinery.
[85] Bram Cohen. The BitTorren Protocol Specification. http::/www.bbttorrren..rr///// hep_0003.html. 2008.
[86] jrandom (Pseudonym). Invisible internet project (i2p) project overview. geti2p.net/
staticcppf/ /ttp:pdfff2pp/// http:/ippppppppp httppphhiloppppppp -df, August 2003. Accessed: 2021-05-20.
[87] Petar Maymounkov and David Mazieres. Kademlia: A peer-to-pees peer information system based on the XOR metric. In International Workshop on Peer-to-Peer Systems, pages 53-65. Springer, 2002.
[88] M Frans Kaashoek and David R Karger. Koorde: simple degree-optimal distributed hash table. In International Workshop on Peer-to-Peer Systems, pages 98-107. Springer, 2003.
[89] Andrew Biggadike, Daniel Ferullo, Geoffrey Wilson, and Adrian Perrig. NATBLASTER: Establishing TCP connections between hosts behind NATs. In Proceedings of ACM SIGCOMM ASIA Workshop. 2005.
[90] Bryan Ford, Pyda Srisuresh, and Dan Kegel. Peer-to-peer communication across network address translators. Proceedings of the Annual Conference on USENIX Annual Technical Conference, ATEC '05, page 13, USA, 2005, USENIX Association.
[91] T. Reddy, A. Johnston, P. Matthews, and J. Rosenberg. Traversal using relays around NAT (TURN); Relay extensions to session traversal utilities for NAT (STUN). RFC 8656, 02 2020.
[92] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pages 9912 99244, 2020.
[93] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2015.
[94] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl. https:
2021.
[95]
Learning@home team. Hivemind: Library for Decentralized Deep Learning. https:
/ //gi
htwpiiing-athhee//eeee hiemindd, 2020.
[96] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018.
16

---

[97]
Stephen Merity, Caiming Xiong, James Bradbury, and R. Socher. Pointer sentinel mixture models. arXiv preprint (rXiv::609.078433 2017.
[98]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024-8035, 2019.
[99] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, 10 2020.
[100] Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, 07 2020.
[101] Lukas Biewald. Experiment tracking with Weights and Biases, 2020. Software available from wandb.com.
[102] Google Stadia data usage. http://ssaapppprroooo//////////a cessed: 2021-05-20
Ac-
[103] Netflix data usage.
Accessed: 2021-05-20.
[104] Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1948-49612 11 2020.
[105] Kushal Jain, Adwait Deshpande, Kumar Shridhar, Felix Laumann, and Ayushman Dash. Indic- transformers: An analysis of transformer language models for Indian languages. arXiv preprint 2020.
[106] Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1946-1958, 07 2017.
[107] Lasse F. Wolff Anthony, Benjamin Kanding, and Raghavendra Selvan. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. ICML Workshop Challenges in Deploying and monitoring Machine Learning Systems, 07 2020.
[108] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for hivacc-preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference Computer and Communications Security, pages 1175-1191 191, 2017.
[109] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q. S. Quek, and H. Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. JEEE Transactions on Information Forensics and Security, 15:3454- 3469. 2020.
[110] Seymour Kaplan. Application of programs with maximin objective functions to problems of optimal resource allocation. Operations Research, 22(4):802-807, 1974.
[111] Erling D. Andersen and Knud D. Andersen. The MOSEK interior point optimizer for linear programming: An implementation of the homogeneous algorithm. In Applied Optimization, pages 197-232. Springer US, 2000.
[112] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, J. Vanderplas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Daniel Henriksen, E. A. Quintero, Charles R. Harris, A. M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa,
17

---

Paul van Mulbregt, Aditya Alessandro Pietro Alex Andreas Andreas Anthony Ant Vijaykumar Bardelli Rothberg Hilboll Kloeckner Sco, Aditya Vijaykumar, Alessandro Pietro Bardelli, Alex Rothberg, Andreas Hilboll, Andre Kloeckner, Anthony M. Scopatz, Antony Lee, Ariel S. Rokem, C. Nathan Woods, Chad Fulton, Charles Masson, Christian Häggström, Clark Fizzger- ald, David A. Nicholson, David R. Hagen, Dmitrii V. Pasechnik, Emanuele Olivetti, Eric A. Martin, E. Wieser, Fabrice Silva, Felix Lenders, Florian Wilhelm, Gert Young, Gavin A. Price, Gert-Ludwig Ingold, Gregory E. Allen, Gregory R. Lee, Hervé Audren, Irvin Probst, Jorg P. Dietrich, Jacob Silterra, James Webber, Janko Slavic, Joel Nothman, Johannes Buchner, Johannes Kulick, Johannes L. Schönberger, José Vinícius de Miranda Cardoso, Joscha Reimer, Joseph E. Harrington, Juan Luis Cano Rodríguez, Juan Nunez-Iglt Iglesias, Justin Kuczynski, Kevin Lee Tritz. Martin Dr Thoma, Matt Newville, Matthias Kümmerer, Max- imilian Bolingbroke, Michael Tartre, Mikhail Pak, Nathaniel J. Smith, Nikolai Nowaczyk, Nikolay Shebanov, Oleksandr Pavlyk, Per Andreas Brodtkorh, Perry Lee, Robert T. McCibbon, Roman Feldbauer, Sam Lewis, Sam Tygier, Scott Sievert, Sebastiano Vigna, Stefan Peterson, Surhud More, Tadeusz Pudlik, Taku Oshima, Thomas J. Pingel, Thomas P. Robitaille, Thomas Spura, Thouis Raymond Jones, Tim Cera, Tim Leslie, Tiziano Zito, Tom Krauss, U. Upad- hyay, Yaroslav O. Halchenko, and Y. Vázquez-Baeza, Scipy 1.0: fundamental algorithms for scientific computing in python. Nature Methods, 17:261 - 272, 2020.
[113] J. Rosenberg, J. Weinberger, C. Huitema, and R. Mahy. STUN simple traversal of user datagram protocol (UDP) through network address translators (NATs). RFC 3489, 03 2003.
[114] libp2p. https:
Accessed: 2021-05-20.
[115]
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfel- low, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensor- Flow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04445 2016
[116] Sebastian U. Stich. Unified optimal analysis of the (stochastic) gradient method, 2019.
[117] Ahmed Khaled, Othmane Sebbouh, Nicolas Loizou, Robert M. Gower, and Peter Richtárik. Unified analysis of stochastic gradient methods for composite convex and smooth optimization, 2020.
[118] Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 66 (6675,222 2018.
[119] Anthony MOI, Pierric Cistac, Nicolas Patry, Evan P. Walsh, Funtowiez Morgan, Sebastian Pütz. Thomas Wolf. Sylvain Gugger, Clément Delangue, Julien Chaumond, Lysandre Debut, and Patrick von Platen. Hugging face tokenizers library. tokenizers, 2019.
[120] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander M. Rush, and Thomas Wolf. Datasets: A community library for natural language processing. arXiv preprint 10x.:2200.002466, 2021.
[121] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint (ee::1607.064500 2016.
[122] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016.
[123] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Lan- guage models are unsupervised multitask learners. 2019.
18

---

[124]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 2020.
[125] Noam Shazeer. GLU variants improve transformer. arXiv preprint arXiv:2002.05202. 2020.
[126] Sharan Narang, Hyung Won Chung, Yi Tay. William Fedus, Thibault Févry. Michael Matena, Karishma Malkan, Noah Fiedel, Noam M. Shazeer, Zhenzhong Lan, Yanqi Zhou, Wen hong Li, Nan Ding. Jake Marcus, Adam Roberts, and Colin Raffel. Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv:2102.11199000 2021.
[127] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0940
2021.
[128] Stella Biderman, Sid Black, Charles Foster, Leo Gao, Eric Hallahan, Horace He, Ben Wang, and Phil Wang. Rotary embeddings: A relative revolution.
wwweuther.airrttteeee ai/rotary-
embeddings, 2021. Accessed: 2021-05-20.
[129] Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2Net: Accelerating learning via knowledge transfer. arXiv preprint arXiv: 15 1.00664,, 11 2015.
[130] Afshin Rahimi, Yuan Li, and Trevor Cohn. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 151-164, 07 2019.
[131] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[132] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:/711005100 2017.
[133] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 3645-3650, 2019.
[134] Roy Schwartz, Jesse Dodge, Noah Smith, and Oren Etzioni. Green Al. Communications of the ACM, 63:54-63, 2020.
[135] Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(248):1-43, 2020.
[136] Donald Kline, Nikolas Parshook, Xiaoyu Ge, E. Brunvand, R. Melhem, Panos K. Chrysanthis, and A. Jones. Holistically evaluating the environmental impacts modern computing systems. 2016 Seventh International Green and Sustainable Computing Conference (IGSC), pages 1 1-8, 2016.
[137] Rabih Bashroush. A comprehensive reasoning framework for hardware refresh in data centers.
IEEE Transactions Sustainable Computing, 3:209-220, 2018.
[138] Xinchi Qiu, Titouan Parcollet, Daniel J. Beutel, Taner Topal, Akhil Mathur, and Nicholas D. Lane. Can federated learning save the planet? arXiv preprint arXiv:2010.06537, 2020.
19


Reviews:
[{"review_id": "uDAPxYb49Lg", "reviewer": "Reviewer_HY7V", "summary": "The paper discusses a distributed training setup where multiple small institutes/groups pool computational resources together for training ML models collaboratively. The paper focuses on two key problems: (1) maintaining consistent training outcomes under dynamic composition of participants. (2) determining the communication strategy adaptively based on dynamic participants & network conditions.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Quality and Clarity**: Poor.\n\nIt appears that the paper tries to touch too many problems (optimization, communication, system design, self-supervised learning) at once while did not discuss any topic in depth. It is unclear what its core technical contributions are. For example, the paper discussed training consistency but only adopted synchronous data-parallel training and extremely large batches without showing how the consistency problem is resolved from optimization perspective. The paper also discussed adaptive communication strategies: all reduce, parameter servers, and hybrid while in fact these are not mutually exclusive (one can implement all reduce with parameter servers).\n\n**Weaknesses**\n\n1. Re section \"3.1 ensuring training consistency\", the authors argue that multiple runs under dynamic participants can lead to different results and claim synchronous data-parallel training with fixed hyperparameters and large batches can resolve this issue. However this is little analysis supporting this claim. Even the authors state in L164 that \"synchronous updates makes DeDLOC mathematically equivalent to large-batch training on a regular cluster\", it does not resolve the issue that the training would converge to different solutions with dynamic participants. \n\n2. The authors did not discuss how data are distributed on different participants. The system heterogeneity issue are different from the data heterogeneity issue. It looks like this paper focuses on the former by discussing hardware/network/bandwidth differences. This leads to two concerns: (1) there is little insights presented in this paper as the staleness, stragglers, unstable hardwares are well studied for distributed training in computer clusters. And in fact the authors did not offer any discussion or comparison on this. (2) without discussing the data heterogeneity, the notion of \"collaborative learning\" is overly broad and therefore misleading.\n\n3. The experiment setup is flawed. In L297, the authors say that their primary focus is \"training models that are useful for a wide range of downstream tasks\" and then show results from two self-supervised learning experiments. The authors should have compare the proposed method with alternatives algorithms or systems instead.\n\n4. The paper's title is \"Distributed Deep Learning in Open Collaboration\". It's not clear which part of the paper is unique to deep learning.\n\n**Originality** and **Significance**: limited due to the concerns above.\n\nThe paper could be stronger if the authors focus on one core technical challenges, analysis it in depth, and compare the proposed method with alternative methods to show its superiority.\n\n=====\n\n**Post rebuttal comments**\n\nThanks to the authors for the thorough response. I have read the response and all other reviews too. I agree the paper addresses an interesting problem in heterogeneous training and provides a linear programming based adaptive averaging algorithm for computing optimal strategies. Despite the good empirical results and thorough presentation on system design, a main shortcoming of the paper is the lack of rigorous analysis from the optimization perspective. With training consistency being one of the main challenges, the technical proposal presented in section 3.1 and 3.3 as well as in the rebuttal are rather handwavy -- highly empirical and procedural in system design. Overall, I am raising my rating to 5: Marginally below the acceptance threshold.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "o9-uvIdhjZ", "reviewer": "Reviewer_Z6aL", "summary": "The authors studied an important topic for many potential future applications in this paper.\n\nCollaborative training means different users in different physical locations share their computing resources to train a deep learning model together.\n\nOtherwise, a single user does not have enough computing resources to finish the training.\n\nOverall, this paper is well written. This could be an important solution for the unbalanced geographic computing resources distribution.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors studied an important topic for many potential future applications in this paper.\n\nCollaborative training means different users in different physical locations share their computing resources to train a deep learning model together.\n\nOtherwise, a single user does not have enough computing resources to finish the training.\n\nThe focus of this paper is to analyze these constraints and propose a novel algorithmic framework designed specifically for collaborative training.\n\nThe authors provided some strong empirical results based on realistic workloads such as SwAV and ALBERT pretraining to support the proposed method.\n\nOverall, this paper is well written. This could be an important solution for the unbalanced geographic computing resources distribution.\n\n\nWe know the size of the communicated gradients is equal to the size of the model.\n\nTherefore, communication overhead could become a big concern to super-large models like GPT-3 (1750 billion parameters) in future real-world deep learning applications.\n\nOne feature of DeDLOC is to reduce the communication volume by large-batch optimization to speed up the training process, which seems to be very practical.\n\nAnother feature is the adaptive averaging algorithm, which essentially makes the users can get the optimal solution in different situations. Figure 1 and Figure 2 provided some good illustrations.\n\n\nAnother contribution for this work is that their efforts for implementation. I believe the engineering part of this work is non-trivial.\n\n\nSome of my questions for future work:\n\nIs it a better way of collaboration for different people to share their quota on cloud and create a large-scale stable computing pool?\n\nHow can people quantize their contributions and benefits in open computing collaboration? Some users may contribute less computing resources, but their computing power is very high, which may be more useful.\n\nDid the authors tried conducting a convergence analysis for the proposed method?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "fG-zMRAYZ0N", "reviewer": "Reviewer_d2iV", "summary": "The paper proposes *DeDLOC* framework to enable distributed Deep Learning in open collaboration by leveraging the computing resources in a volunteering user setup. The framework adapts to the available hardware and communication network to maximize training throughput and accordingly behaves as PS  [33], AR-SGD [34], decentralized SGD [35], BytePS [36], or a combination of them in hybrid mode. The experiments are done with real training collaboration across 49 volunteers for Bengali language modeling using ALBERT-large. This is a first-of-its-kind volunteering effort for collaborating training. Additionally, the proposed framework is also applied to self-supervised tasks under 3 compute setups (WORKSTATION: homogeneous low bandwidth, SERVER: homogeneous high bandwidth, and HYBRID: heterogeneous) ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:**  \n\nThe work is nicely motivated and provides a practical solution validated with real experiments. The problem setup is novel and clearly addresses key challenges in open collaboration environments (Sec 3.1 to Sec 3.3). The paper clearly distinguishes itself from related work and provides an adaptive averaging framework that can be modeled as one of the various popular data-parallel distributed deep learning strategies (PS, AR-SGD, etc) based on collaboration setups to compute the optimal solution.\n\n\n**Quality:**  \n\nThe paper provides a linear programming optimization problem to model optimal communication strategy with the highest training throughput while the participants are joining and leaving. The work is complete in its dealing with the challenges posed and presenting a practical solution to democratize deep learning and provide opportunities for unconventional datasets.\n\n\n**Clarity:**  \n\nThe paper is very well-written and provides a great narrative for the reader to think about the current challenges in real-world deployment and applications of distributed deep learning. The work is a bold attempt to inspire future research in open collaborative systems. \n\nFew typos\n- Line 98:  ... them ~are~ highly...\n- Line 132: Space between period (.) and *\"This specific problem...\"*\n\n\n**Significance:**  \n\nThis is important work and the authors formulate the problem carefully to motivate the community to focus on large-scale collaborative distributed training strategies. The experiments are the highlight of the work and the results are a stepping stone to incorporate volunteer setups.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed it sufficiently", "review_text": "", "extra_scores": {"ethical_concerns": "The authors have discussed this", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Jg5L2I_KCzB", "reviewer": "Reviewer_vAEP", "summary": "This paper attempts to deploy distributed training of deep neural networks over an open collaborative distributed environment, where participators can dynamically join the computational task. \nThe approach is mainly based on modification of data parallel training to overcome the technique challenges introduced by the collaborative environment. \nBoth standard and real-world pre-train benchmarks are included to evaluate the proposed system. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper talks about a very interesting problem, where fragmented computational resource would be able to more efficiently leveraged for jointly training deep neural networks. As such models can only be trained inside data-center clusters currently, the approach proposed in this paper would be broadly beneficial to a very large research area. \n\nHowever, the methodology proposed in this paper is relatively limited:\n+ First, the method limited itself to data parallel training, however, this would introduce inconsistency with the motivation of the work. In a dynamic environment, it is unwanted to assume the whole model can be hold in a single participator's RAM, which is the fundamental constraint for data parallel training. \n+ Further, even limiting the design to the data parallel approach, the approach solves the problem by simply extending very large batch training at the algorithmic level. However, very large batch training would compromise the generalization performance in some cases. On the other hand, there could be lots of other design choices neglected by the paper, for example, asynchronous SGD and local SGD are proposed to replace large batch training. At least some discussion to justify the proposed design choice would be necessary.\n\nThe paper is not well polished for its writing. \n+ There are lots of typos throughout the paper: e.g., Line 204 \"peer i peer\"; Line 319 \"Figure 1\" (it should be Table 1 I conjecture). \n+ Figure 2 is confusing, what do the rectangles and lines represent?\n\nThere are some flaws in the empirical study:\n+ Since the benchmarks are based on some pre-training and fine-tuning tasks, it looks a little unconfident to just report the training loss. At least some results about the generalization performance should be reported. For example, for the SwAV task, at least some linear model should be used to justify the proposed approach can reach a similar level of accuracy reported in the original paper.\n\n+ There is a missing of scalability benchmarks, for example, when adding more instances, how would the system scale out. It is at least reasonable to report  the training sample throughput when including 4, 8, 16, 32, etc instances for one particular setting. \n\n+ The experimental section is not well presented, there is a missing of clear hypothesis of the experiments. \n\nPost rebuttal update:\n\nI really appreciate the great effort the author has made to address my concerns. Based on these feedbacks, I have some follow-up comments:\n+ It seems that the proposed method can achieve an almost linear speedup w.r.t training sample throughput, which is awesome. On the other hand, I think there could be some analysis about why the proposed approach can reach this desired performance, i.e., given the model size and the network bandwidth, how the communication is hidden inside the computation slot (considering the FLOP in the forward and backward pass and FLOPs of the GPU devices.)\n+ I believe the paper should clearly state the category of the training tasks the proposed work can be applied, e.g., fine-tuning tasks where large batch training can be adopted. \n+ The writing can be further polished according to the helpful feedbacks from all the reviewers.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No applicable. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although there was a spread of opinions from the reviewers, they did all agree that the problem area addressed by the paper is both interesting and important.  Indeed, finding ways to create more opportunities for those outside of select industry or academic groups to participate in massive scale model training is important from both a research perspective and from a community perspective that takes into account the overall health of the field into account.

Additionally, while the reviewers initially raised some important concerns, these have been well addressed by the author responses.  In particular, the scaling results and the convergence analysis are both very helpful, and should be included in the final version in some way. 

One thing to note is that the spirit of reviewer HY7V's comment that the paper is trying to address many (important) issues is indeed a factor here, as evidenced by the repeated references to the (extensive) appendix.  I recognize that the page limits of NeurIPS are a significant constraint, and that the "best" form of this paper is likely a journal article that incorporates the appendix information more fully into the main text and narrative.  That said, taking all of the reviews into account and the ensuing discussion, I do think that this conference-paper version has significant merit and will spark interest and discussion within the field. 
Decision: Accept (Poster)