Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors studied an important topic for many potential future applications in this paper.

Collaborative training means different users in different physical locations share their computing resources to train a deep learning model together.

Otherwise, a single user does not have enough computing resources to finish the training.

Overall, this paper is well written. This could be an important solution for the unbalanced geographic computing resources distribution.
main_review: The authors studied an important topic for many potential future applications in this paper.

Collaborative training means different users in different physical locations share their computing resources to train a deep learning model together.

Otherwise, a single user does not have enough computing resources to finish the training.

The focus of this paper is to analyze these constraints and propose a novel algorithmic framework designed specifically for collaborative training.

The authors provided some strong empirical results based on realistic workloads such as SwAV and ALBERT pretraining to support the proposed method.

Overall, this paper is well written. This could be an important solution for the unbalanced geographic computing resources distribution.


We know the size of the communicated gradients is equal to the size of the model.

Therefore, communication overhead could become a big concern to super-large models like GPT-3 (1750 billion parameters) in future real-world deep learning applications.

One feature of DeDLOC is to reduce the communication volume by large-batch optimization to speed up the training process, which seems to be very practical.

Another feature is the adaptive averaging algorithm, which essentially makes the users can get the optimal solution in different situations. Figure 1 and Figure 2 provided some good illustrations.


Another contribution for this work is that their efforts for implementation. I believe the engineering part of this work is non-trivial.


Some of my questions for future work:

Is it a better way of collaboration for different people to share their quota on cloud and create a large-scale stable computing pool?

How can people quantize their contributions and benefits in open computing collaboration? Some users may contribute less computing resources, but their computing power is very high, which may be more useful.

Did the authors tried conducting a convergence analysis for the proposed method?
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes *DeDLOC* framework to enable distributed Deep Learning in open collaboration by leveraging the computing resources in a volunteering user setup. The framework adapts to the available hardware and communication network to maximize training throughput and accordingly behaves as PS  [33], AR-SGD [34], decentralized SGD [35], BytePS [36], or a combination of them in hybrid mode. The experiments are done with real training collaboration across 49 volunteers for Bengali language modeling using ALBERT-large. This is a first-of-its-kind volunteering effort for collaborating training. Additionally, the proposed framework is also applied to self-supervised tasks under 3 compute setups (WORKSTATION: homogeneous low bandwidth, SERVER: homogeneous high bandwidth, and HYBRID: heterogeneous) 
main_review: **Originality:**  

The work is nicely motivated and provides a practical solution validated with real experiments. The problem setup is novel and clearly addresses key challenges in open collaboration environments (Sec 3.1 to Sec 3.3). The paper clearly distinguishes itself from related work and provides an adaptive averaging framework that can be modeled as one of the various popular data-parallel distributed deep learning strategies (PS, AR-SGD, etc) based on collaboration setups to compute the optimal solution.


**Quality:**  

The paper provides a linear programming optimization problem to model optimal communication strategy with the highest training throughput while the participants are joining and leaving. The work is complete in its dealing with the challenges posed and presenting a practical solution to democratize deep learning and provide opportunities for unconventional datasets.


**Clarity:**  

The paper is very well-written and provides a great narrative for the reader to think about the current challenges in real-world deployment and applications of distributed deep learning. The work is a bold attempt to inspire future research in open collaborative systems. 

Few typos
- Line 98:  ... them ~are~ highly...
- Line 132: Space between period (.) and *"This specific problem..."*


**Significance:**  

This is important work and the authors formulate the problem carefully to motivate the community to focus on large-scale collaborative distributed training strategies. The experiments are the highlight of the work and the results are a stepping stone to incorporate volunteer setups.
limitations_and_societal_impact: The authors have discussed it sufficiently
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper discusses a distributed training setup where multiple small institutes/groups pool computational resources together for training ML models collaboratively. The paper focuses on two key problems: (1) maintaining consistent training outcomes under dynamic composition of participants. (2) determining the communication strategy adaptively based on dynamic participants & network conditions.
main_review: **Quality and Clarity**: Poor.

It appears that the paper tries to touch too many problems (optimization, communication, system design, self-supervised learning) at once while did not discuss any topic in depth. It is unclear what its core technical contributions are. For example, the paper discussed training consistency but only adopted synchronous data-parallel training and extremely large batches without showing how the consistency problem is resolved from optimization perspective. The paper also discussed adaptive communication strategies: all reduce, parameter servers, and hybrid while in fact these are not mutually exclusive (one can implement all reduce with parameter servers).

**Weaknesses**

1. Re section "3.1 ensuring training consistency", the authors argue that multiple runs under dynamic participants can lead to different results and claim synchronous data-parallel training with fixed hyperparameters and large batches can resolve this issue. However this is little analysis supporting this claim. Even the authors state in L164 that "synchronous updates makes DeDLOC mathematically equivalent to large-batch training on a regular cluster", it does not resolve the issue that the training would converge to different solutions with dynamic participants. 

2. The authors did not discuss how data are distributed on different participants. The system heterogeneity issue are different from the data heterogeneity issue. It looks like this paper focuses on the former by discussing hardware/network/bandwidth differences. This leads to two concerns: (1) there is little insights presented in this paper as the staleness, stragglers, unstable hardwares are well studied for distributed training in computer clusters. And in fact the authors did not offer any discussion or comparison on this. (2) without discussing the data heterogeneity, the notion of "collaborative learning" is overly broad and therefore misleading.

3. The experiment setup is flawed. In L297, the authors say that their primary focus is "training models that are useful for a wide range of downstream tasks" and then show results from two self-supervised learning experiments. The authors should have compare the proposed method with alternatives algorithms or systems instead.

4. The paper's title is "Distributed Deep Learning in Open Collaboration". It's not clear which part of the paper is unique to deep learning.

**Originality** and **Significance**: limited due to the concerns above.

The paper could be stronger if the authors focus on one core technical challenges, analysis it in depth, and compare the proposed method with alternative methods to show its superiority.

=====

**Post rebuttal comments**

Thanks to the authors for the thorough response. I have read the response and all other reviews too. I agree the paper addresses an interesting problem in heterogeneous training and provides a linear programming based adaptive averaging algorithm for computing optimal strategies. Despite the good empirical results and thorough presentation on system design, a main shortcoming of the paper is the lack of rigorous analysis from the optimization perspective. With training consistency being one of the main challenges, the technical proposal presented in section 3.1 and 3.3 as well as in the rebuttal are rather handwavy -- highly empirical and procedural in system design. Overall, I am raising my rating to 5: Marginally below the acceptance threshold.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper attempts to deploy distributed training of deep neural networks over an open collaborative distributed environment, where participators can dynamically join the computational task. 
The approach is mainly based on modification of data parallel training to overcome the technique challenges introduced by the collaborative environment. 
Both standard and real-world pre-train benchmarks are included to evaluate the proposed system. 
main_review: This paper talks about a very interesting problem, where fragmented computational resource would be able to more efficiently leveraged for jointly training deep neural networks. As such models can only be trained inside data-center clusters currently, the approach proposed in this paper would be broadly beneficial to a very large research area. 

However, the methodology proposed in this paper is relatively limited:
+ First, the method limited itself to data parallel training, however, this would introduce inconsistency with the motivation of the work. In a dynamic environment, it is unwanted to assume the whole model can be hold in a single participator's RAM, which is the fundamental constraint for data parallel training. 
+ Further, even limiting the design to the data parallel approach, the approach solves the problem by simply extending very large batch training at the algorithmic level. However, very large batch training would compromise the generalization performance in some cases. On the other hand, there could be lots of other design choices neglected by the paper, for example, asynchronous SGD and local SGD are proposed to replace large batch training. At least some discussion to justify the proposed design choice would be necessary.

The paper is not well polished for its writing. 
+ There are lots of typos throughout the paper: e.g., Line 204 "peer i peer"; Line 319 "Figure 1" (it should be Table 1 I conjecture). 
+ Figure 2 is confusing, what do the rectangles and lines represent?

There are some flaws in the empirical study:
+ Since the benchmarks are based on some pre-training and fine-tuning tasks, it looks a little unconfident to just report the training loss. At least some results about the generalization performance should be reported. For example, for the SwAV task, at least some linear model should be used to justify the proposed approach can reach a similar level of accuracy reported in the original paper.

+ There is a missing of scalability benchmarks, for example, when adding more instances, how would the system scale out. It is at least reasonable to report  the training sample throughput when including 4, 8, 16, 32, etc instances for one particular setting. 

+ The experimental section is not well presented, there is a missing of clear hypothesis of the experiments. 

Post rebuttal update:

I really appreciate the great effort the author has made to address my concerns. Based on these feedbacks, I have some follow-up comments:
+ It seems that the proposed method can achieve an almost linear speedup w.r.t training sample throughput, which is awesome. On the other hand, I think there could be some analysis about why the proposed approach can reach this desired performance, i.e., given the model size and the network bandwidth, how the communication is hidden inside the computation slot (considering the FLOP in the forward and backward pass and FLOPs of the GPU devices.)
+ I believe the paper should clearly state the category of the training tasks the proposed work can be applied, e.g., fine-tuning tasks where large batch training can be adopted. 
+ The writing can be further polished according to the helpful feedbacks from all the reviewers.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: Although there was a spread of opinions from the reviewers, they did all agree that the problem area addressed by the paper is both interesting and important.  Indeed, finding ways to create more opportunities for those outside of select industry or academic groups to participate in massive scale model training is important from both a research perspective and from a community perspective that takes into account the overall health of the field into account.

Additionally, while the reviewers initially raised some important concerns, these have been well addressed by the author responses.  In particular, the scaling results and the convergence analysis are both very helpful, and should be included in the final version in some way. 

One thing to note is that the spirit of reviewer HY7V's comment that the paper is trying to address many (important) issues is indeed a factor here, as evidenced by the repeated references to the (extensive) appendix.  I recognize that the page limits of NeurIPS are a significant constraint, and that the "best" form of this paper is likely a journal article that incorporates the appendix information more fully into the main text and narrative.  That said, taking all of the reviews into account and the ensuing discussion, I do think that this conference-paper version has significant merit and will spark interest and discussion within the field. 
Decision: Accept (Poster)