Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Accelerating DNN Training through Selective Localized Learning

In this paper, the authors proposed a new approach by the name of LoCal + SGD (Localized Updates) to replace the traditional Backpropagation method. The key idea is to selectively update some layers’ weights using localized learning rules. For these layers, the computation cost is reduced from two matrix multiply operations to one matrix multiply operation. The authors also proposed the Learning Mode Selection Algorithm to maintain the accuracy and convergence.

The authors provided some experimental results on common deep learning benchmarks such as ImageNet/ResNet and CIFAR/VGG. Overall, the authors reported that this approach can achieve around 1.36x speedup for a 0.4% loss in accuracy for ResNet-50. The authors also reported that they can achieve a higher speed than recent methods such as Structured Pruning and stochastic depth.

This work is a trade-off between computation and accuracy (details in Figure 5). I have some questions for the authors:

(1) How stable is the proposed method (LoCal + SGD)? Does it still work for large-batch optimization and asynchronous training?

(2) What is the overhead of hyper-parameter tuning?

(3) Did the authors use the same number of epochs as the baseline to finish the training?

(4) What is the absolute speed (e.g. in GFlops or TFlops)? Can the proposed method beat a well-optimized NVIDIA implementation?

(5) What is the limit of the proposed method?

(6) Can the Learning Mode Selection Algorithm work with other methods?

Since this work fundamentally changes the way of learning, it is probably necessary to do a convergence analysis for the proposed method.
review_title: interesting work but needs more details

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a combination of SGD with selective application of a non-backprop learning rule (Hebbian). The two learning rules are not applied together, but rather a boundary is determined where layers prior use SGD, and the ones after use the Hebbian approach. A selection algorithm dynamically adjusts the boundary over training. For accuracy reasons, they include weak supervision by using the overall classification loss to control the sign of the update. 

From computational efficiency perspective, the contributions reduce the need for a backprop calculation, and also leads to a smaller memory footprint, since the activation values need not be stored. On ImageNet benchmark models, they show <0.5% Top1 drop in exchange for ~1.3x runtime speed compared to vanilla SGD.

Strengths
+ Focus on run-time improvements brings practical significant to their proposed method
+ Algorithm is relatively simple to implement
+ Convergence results only show a small degradation from SOTA
+ The tuning results for alpha (Figure 5) are useful for practitioners that need to balance accuracy and compute

Weaknesses
- The 'meta' boundary selection and weak supervision approaches add additional hyperparameter complexity to the tuning process. These are also empirically but not theoretically motivated, and unclear if they generalize to other domains. I understand that non-classification models are out of scope for this paper, but his paper's impact would be improved by some comment on transferability. For example, U-net models have long range skip connections that span the model. 
- While the focus on runtime is welcome, what is relevant to the practitioner is time-to-train to a particular accuracy target, similar to the metric adopted by MLPerf. Since this method does introduce an accuracy degradation (e.g. for RN-34, 27.04% versus 26.6%; Table 1), the more fair comparison would account for the fewer epochs the baseline SGD needs to hit 26.6%. To make a more convincing argument to practitioners, I would compare either the wall-clock time, or the total FLOPS needed, to hit the target accuracy.
- Several techniques are introduced without ablations to measure their effectiveness and justify the added complexity. This is particularly important as these techniques add additional burden on the practitioner in terms of tuning the new hyperparameters. 

The work combines existing learning rules (SGD, Hebbian) with some novelty in how they are employed, and with a weak supervisory signal, to achieve reasonable results. These contributions were not foundational improvements, so the paper's main merit is in the potential practical impacts of this method. The significance to practitioners however, is greatly reduced by the weaknesses described above.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work demonstrates that localized learning can improve DNN training efficiency by reducing computation and memory requirements. The effectiveness of this approach is shown by the experimental results that report a nice trade-off of minimal accuracy loss and good throughput improvement compared to baseline SGD and competing efficient training techniques. 
While the experimental results are indeed appealing, a major flaw is that the paper does not sufficiently explain why the underlying techniques (learning mode selection and weak supervision) work. Since these techniques are configured with a number of hyper-parameters, and I could not gain an intuition of how/why/whether they work in general. For example, $t\_{shift}$ is set to recurring block size of residual nets, but what is not given is the justification for this choice, or how to set for non-residual nets. In other words, the hyperparameter settings for these techniques appear ad-hoc (I suspect that it is not), and so it is not clear to me how much exploration is required. What could have helped is to include a study of the incremental benefits of these techniques in the experiment section. In summary, while the results are good, the writing and presentation could be greatly improved to help readers learn and use the proposal.
`
**Pros**
1. Tackles an important problem of reducing time and resource requirements of DNN training.
2. The general approach of computing layers differently over the course of training is quite intuitive. 
3. Presents two techniques that appear to make localized learning practical and effective for DNN training.  

**Cons**
1. The proposed techniques are parameterized (e.g., $t\_{shift}$, $\alpha$, etc.), but how, and effort required to configure them is not clear. 
2.  The proposed techniques are not sufficiently explained to help build intuition. For example, the weak supervision suggests that reversing the weight update direction can effectively reverse increase in classification loss (i.e., divergence), but it is not clear why this is the case or where this observation applies to SGD and other optimizers.  
3. Incremental benefits of the techniques is not provided in evaluation. 
review_title: Needs more work

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper try to leverage the benefit of Hebb learning to reduce CNN training time cost. In order to achieve this,  a learning mode selection algorithm is proposed to progressively increase  number of layers using Hebb learning. The writing of this paper is good and the idea is also interesting, however, the experimental part should be improved:

1. The criterion used in learning mode selection algorithm is the model-update norm of current epoch. If the norm is small enough, the transition layer index will be increased. A small model-update norm also means that current layer is nearly convergent. Could you just fix these layers to accelerate training? Yes, freezing layer exps are tried but the comparison is not fair in my opinion. When Hebb-Learning-Layers are frozen, the final accuracy drops, but the training speedup is improved. So if you freeze less layers to make training speedups of freezing-layer-training and Hebb-Learning same, what will the accuracy relationship be? Does the proposed method still outperforms freezing strategy?

2. A weak supervision scheme is proposed in this paper, but I did not find any experiments to evaluate its effect, could you add this part? 
review_title: A possible useful training practice, but some important experiments are missing

final_score: rating: 7: Good paper, accept
review_content: EDIT: 2020-11-27: Updated my score. Further explanations at the end of this post.

Summary: 
The authors propose an algorithm that separates a network into two distinct regions where one is trained with SGD while the other is trained with a local Hebbian learning rule. A weak supervision rule is proposed that improves localized learning. The authors demonstrate close to baseline performance on CIFAR-100 and ImageNet for diverse networks while speeding up training.

Strong points:
- Very creative use of multiple learning rules during training.
- People could never get localized learning to work well enough to compete with SGD. This is the first work that uses some localized learning and manages to come close. This is a big success.
- These findings have broad applicability, from brain-like learning algorithms, efficient training on one GPU, and efficient learning algorithms for model parallelism for massive networks where communication is the bottleneck.

Weak points:
- Missing ablation for the weak supervision algorithm.
- This work is very impactful in many different ways, but it only mentions the computational efficiency perspective. The related work needs to be expanded to make readers aware of the impact of this work.
- The heuristic for learning mode selection is complicated. Initial experiments suggested that a simple learning-rate-schedule-like way to select the learning mode would be possible.
- Some algorithmic details in the weak supervision algorithm not clear.

Recommendation (short):
This is very important work with results that will significantly impact many fields (efficient training, parallelization, brain-like algorithms). It is a creative solution to a significant problem in localized learning. I strongly recommend accepting this work. If this work is rejected, I will no longer review for future ICLR conferences. I recommend this work to be accepted as an oral presentation. Selecting this work for a best paper award would be appropriate.

Recommendation (long):
This work is impactful for multiple reasons:
- Localized learning is difficult. There has not been a work that uses localized learning and makes it work close to SGD performance on large datasets/models.
- The brain does not use SGD, and it is difficult to think about algorithms that work in the brain and yield good performance. It might be that initial learning in the brain is done differently until local learning rules are used. This view is mostly ignored, but this paper yields evidence that such learning might be possible and efficient.
- Layers updated with localized rules can be updated independently of other layers (if the weakly supervised rule is not used). This enables fully asynchronous training for early layers. There will be a synchronization point at the SGD layers, but through pipeline parallelism, the communication overhead can be hidden in Hebbian layers. This enables the training of massive neural networks with trillions of parameters. With current parallelism tools becoming more and more limited, this is a crucial innovation since previous asynchronous parallelism procedures always decrease predictive performance. This is the first work that shows a way to do asynchronous parallel training without performance degradation.

Beyond this, the paper also yields some speedups for tasks while decreasing predictive performance only slightly. This is also an impressive feat, but the overall broad insights this paper yields make it much more impactful than this result. As such, I do not view the experimental results as the main contribution, but overall, the papers' main contribution is that it shows a way to include (gradual) localized learning in a neural network while not impacting performance.


Comments for authors:
This is excellent work — well done! I think the main weakness is currently a missing ablation on the effect of the weak supervision rule. You note that improves performance but by how much would be an important detail. If you do not include these ablations, I would still accept the paper, but I might rescind my oral presentation recommendation.

Another issue is that your work is relevant in many different domains, but you keep it confined to the idea that your method is only useful for faster training. I think making the reader aware that local training has many advantages across many domains could be very valuable. You do not need to elaborate on this, but I would like to see some of these connections in the paper because not everyone has the background to see these connections. I think you can do this mostly by mentioning it in the conclusion since you already mention a little bit of work in parallelism, and you mention previous results about local learning that failed to obtain good performance. Another line of work that I would mention in the related work section is that of neuroscientific faithful learning rules. The most relevant line of research is the work on various forms of feedback alignment and other algorithms. For a summary of past research and results on large datasets, see Bartunov et al., 2018[1]. Beyond this, you might want to add "sparse training" to the related work on efficient deep learning. Sparse training differentiates from pruning during training by initializing the neural network sparsely during initialization (not densely and then prune to sparse). See work on a mixture of experts[2,3] and sparse dynamic training[4,5,6]. I do not require you to include these references, but they might improve the related work section. 

On the algorithmic and experimental side, it seems that a simple learning-rule-like schedule might be sufficient for selecting the learning mode. While I do not require you to add these experiments, it would make the algorithm simpler and more appealing if you can show that a simple learning rule works a la "warmup with SGD for 5 epochs, then shift by 1 layer (block) every 5 epochs" etc.

[1] Sartunov et al., 2018. Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
[2] Shazeer et al., 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
[3] Lepikhin et al., 2020. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
[4] Mostafa & Wang et al., 2019. Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization.
[5] Dettmers & Zettlemoyer et al., 2019. Sparse Networks from Scratch: Faster Training without Losing Performance.
[6] Evci et al., 2019. Rigging the Lottery: Making All Tickets Winners.


Update:
==================================

Comments for Area Chair and Reviewers: 

If I view this work merely by the story conveyed in the paper, my assessment would be more in line with the other reviewers. I am not quite sure if this is the right way to evaluate this paper since I view it as having a broader impact that goes beyond the story in the paper, but other reviewers disagree with my view on its broader impacts. I see this as a sign that the paper is currently not in a good enough state to really convey its potential impact.

Comments for authors:

I believe you still did good work here on the merits of the "speedup training" story that you convey in your paper. I believe that you have much more than this in your hands though. I think you could go two ways from here: (1) get this paper accepted in this form and work closely on the other angles that this work offers in a new paper, e.g. learning which is in line with biological or efficient parallelization of large networks. The second way (2) would be to rewrite this paper more in line with that view and resubmit. I think (1) might be better for you. I do not think many reviewers would understand a paper that comes from the process in (2). Good luck!
review_title: First competitive results for a (partial) localized learning procedure. Has wide applications across many fields.

Meta-review: In this paper, the authors proposed a new approach by the name of LoCal + SGD (Localized Updates) to replace the traditional Backpropagation method. The key idea is to selectively update some layers’ weights using localized learning rules, so as to reduce the computational complexity of training these layers so as to achieve a better tradeoff between overall speed and accuracy.
The paper received quite mixed reviewers. Some reviewers criticized the incremental nature of the proposed technology, while some other reviewers thought that this is one of the very early papers that demonstrates the practical effectiveness of localized learning. 

The reviewers have made several rounds of discussions, and as a result of that, we think while this direction (localized learning) is very important and promising, this particular paper might not have provided a sufficiently novel and good solution to it.  Specifically, in terms of localized learning, this paper has not proposed brand new concepts or methodologies, instead it adopts existing methods in selective layers. In this sense, it does not really resolve the accuracy issue of localized learning, rather, it achieves the tradeoff by only applying localized learning in some layers. In other words, the current results still heavily rely on BP and has not brought a real breakthrough to localized learning.

Decision: Reject