Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work explores different local training schemes and demonstrates a better compute efficiency of local training schemes over the traditional synchronous backprop. The paper is well written and clearly articulates a contribution to the literature. It is intuitive and clear that local training could have advantages over backprop as it fully parallelizes the blocks of layers in DNNs. The experimental evidence is provided for both language and image classification tasks using an analytical model and real hardware. Most of the related works are cited.

Concerns: 
1) Pareto optimal curves: The authors use Pareto optimal curves to show the tradeoff between total computational cost and walltime (number of sequential FLOPs) using different training schemes. Although this simplistic model provides a sense of how fast a training method could be and is agnostic to underlined hardware, the accuracy of this model is questionable as it does not consider communication and memory overhead. The roofline model used in Google TPU paper [1], which considers the memory throughput, might be a better way to estimate the walltime. Also, I think it is crucial to justify the Pareto optimal curves with real hardware experiments under a similar setting. 

2) Experiments on real hardware: The current evaluation on real hardware seems to be weak as the experiments are with different local batch size and backdrop batch size. For example, I would expect local training of a deeper network to have a higher speedup than backprop. However, as the batch size is chosen differently, it is hard to compare the results and obtain insights from the results. I would appreciate a more comprehensive evaluation on real hardware.

3) Novelty: Although this paper provides an analytical model and real hardware evaluation for local training, the novelty of the paper is moderate as it does not offer new solutions or improvements over the existing local training method. As the paper mainly demonstrates the effectiveness of local training from a practical point of view, providing new insights or methodologies for implementing the local training could increase the strength of the paper.

Minor:
1) What is the number of blocks used in the experiments? There would also be a tradeoff between the number of blocks, the walltime, and accuracy. Besides, how are the layers been grouped into blocks? Does each block contain the same number of layers or a similar amount of FLOPs?
2) In Table 1, could you please provide the actual walltime (in ms) for each experiment? Without knowing the actual walltime, it is hard to know how realistic this evaluation is.
3) How does local training compared to async backprop?

[1] https://dl.acm.org/doi/10.1145/3140659.3080246
review_title: The work explores and compares different local training schemes and demonstrates the benefits of local training over synchronous backprop  on real hardware.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper explores and compares several methods for parallel training of deep nets. It presents the results on multiple datasets for image classification and language modelling.

# Quality

This work provides many experiments with neat visualizations. 

Pros:
- The paper provides many rigorous experiments for 7 different methods of parallel training
- The paper conducts experiments on various datasets, including image classification on CIFAR-10, language modelling on LM1B, image classification on ImageNet
- The paper conducts experiments analyzing the properties of the various methods of parallel training

Cons:
- Not many architectures explored: only Resnet for ImageNet, only Transformer for LM1B
- The paper doesn't touch on the recurrent architectures
- It is always great to see even more experiments on more datasets and tasks

# Clarity

The paper is well written and easy to follow. Although I am not an expert in the parallel training, it was easy to understand.

Pros:

- The literature is reviewed well
- The figures are compact, dense and informative

Cons:

- Sometimes, the figures are hard to read in print
- The conclusion is not very informative, I expected to see clear "do"s and "don't"s of parallel training

# Originality

The research presented in the paper is original.

# Significance

The work presented is clearly of great significance to the community.

# Conclusion

This is a thorough exploration of several methods for parallel training. The work provides a multitude of experiments comparing the given architectures. While the work doesn't introduce any groundbreaking ideas, it conducts rigorous experiments and presents them well.

Suggestions for improvements:
-  I expected to see a list of "do"s and "don't"s in the conclusion
- It is hard to read the figures when printed

# EDIT: Update

Thank you for the rebuttal.

Despite the skepticism of the other reviewers, I still think that this is a valuable, thorough paper of high quality. 
review_title: Good work, accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: As someone not exposed to local training methods I found that the survey was thorough, well written and easy to understand. However, I might be missing something but it seems that the "chunked" strategy is just the same as Greedy with J divided by 2. This should probably be clarified in the manuscript.

I found section 5 particularly interesting and happened to answer most of the question I would be asking myself if I was trying to understand the behavior of these optimization methods.

My main criticism lies in the performance evaluation. Any running time analysis is highly dependent on the hardware used. First, the hardware used for this evaluation is clearly niche. It is almost impossible to get a sense of what the running times mean here. Before making any conclusion about performance I think one should compare on multiple kind of hardware or at least the most popular one. Secondly, As the authors empathize it, the drawback of training large networks is communication between machines. The fact that this setting has been overlooked in the performance evaluation is disappointing.

As the paper rightfully note, some methods might lead to poor generalization. It might be also better to use a validation accuracy cutoff instead. I also find the plots for high cutoff values uninteresting. I don't think most people care about how long it takes/how much resources are needed to train a model to a fraction of its potential.


review_title: Interesting survey but poor performance evaluation

final_score: rating: 3: Clear rejection
review_content: It is a very poorly written paper. Basic idea of finding a way to not have to wait for full forward pass is not new. Multiple research papers have been published from the extreme of using stale weight to some form of sub-network backdrop as a proxy for the full network. This paper proposed no new idea for local update. Prior work have all suffered with one or both of these two limitations: a) poor experimental framework, or b) not being able to meet the accuracy bar set by backprop. This work suffers from both.  Very poorly described experimental basis - and failing to come even close to the backprop accuracy target with any decent speedup claim. Former is my biggest concern. Section 6 starts with 'Here we show that performance gains of local parallelism can be realized on real hardware' - with near-zero description of any 'real' hardware, except a footnote on '1000 IPUs on a chip'. 

Reference: https://deepai.org/publication/loco-local-contrastive-representation-learning - also cited in this details the same idea of local learning (see, Fig 1 - similar to overlapping in this work). And they do offer same benchmark Resnet-50 with ImageNet - meeting accuracy and memory saving.  Other cited work: arXiv:1901.08164 has shown the same local learning concept deliver accuracy for Resnet-152. Therefore, I was hoping this submission would go significantly beyond these, which it does not. It does add one language model benchmark, however, the model is very small (6M parameters). 

Most concerning part though is its experimental framework - details are almost completely missing. No standard CPU-GPU details are mentioned where the experiments where conducted.  Only mention of experimental platform is in first part of Sec 6: "We implement the models in TensorFlow (Abadi et al., 2016) and train them across 4 or 8 IPUs" - then there is a footnote on that page describing hardware with 3-4 additional lines: "Intelligence Processing Units (IPUs) are massively parallel machine learning hardware accelerators. Each chip incorporates > 1000 processor cores" - This gives me no reason to believe that the authors experimented on any 'real hardware'. Hence, my overall low rating for this work.  
review_title: Submission aims to demonstrate how local parallelism can help improve training performance. Local parallelism refers to not having to the traditional backprop where one waits for full forward pass triggered weight update before next iteration can begin.

Meta-review: After reading the paper, reviews and authors’ feedback. The meta-reviewer agrees with the reviewers that the paper touches an important topic(scale up training). However, as some of the reviewers pointed out, the paper could be further improved by clarifying the novelty and more thorough evaluation justification of the metric being used. Therefore this paper is rejected.

Thank you for submitting the paper to ICLR. 

Decision: Reject