Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper describes stochastic back propagation for image classification and object detection. They dropout certain feature maps during back propagation which helps in saving memory and speeding up training.
strengths_and_weaknesses: It is good that you have showed the SBP for a more general task of image classification and object detection, building upon the previous work on video models. I like that you have verified its application for a transformer and ConvNeXt. The analysis is exhaustive and the insights are useful. There are minor typos and grammatical mistakes. Another round of proofreading would be useful.
questions: None
limitations: Yes
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper evaluates the concept of Stochastic Backpropagation (SBP) on object detection and image classification using a traditional convolutional neural network backbone (ConvNeXt) and a transformer backbone (ViT). This expands on the work of [4] that introduced the concept of SBP exclusively for video inputs. The paper provides a formalization of the topic, evaluates efficient implementations, and examines design strategies in determining how to best apply SBP. Quantitative performance is evaluated on the widely used ImageNet and COCO datasets.
strengths_and_weaknesses: __Technical Contribution__

This paper demonstrates the widespread applicability to vision tasks of a reasonably straightforward concept: improved memory efficiency during network training by stochastically choosing some subset of layers to compute gradients for at each backpropagation pass. This work builds on the work of [4], providing a theoretical and empirical groundwork for broader usages than the video prediction use case of [4].  Although this work did not introduce the concept of SBP, it provides a very good argument that SBP should see more prominent usage.

The efficient implementation proposed in Algorithm 1 is clearly demonstrated, and it is explained that it does not require any recomputation of values required for backpropagation.  The design analysis in Section 3 is useful context for understanding the impact that design choices will have (e.g. Figures 1-4). However, the combinatorially explosive number of ways SBP could be applied to a network raises questions about how generalizable these insights are. Nevertheless, specific outcomes like grid sampling outperforming random sampling when choosing gradients to keep, as shown in Figures 3 and 4, are insightful.


__Presentation__

This paper sets up the problem really well in L25-36. It makes very clear how [4] provides a limited scope of evaluation and then indicates exactly how it plans to expand on the scope of [4]. Without even leaving the first page, it is clear to the reader what this paper is about.

One odd convention that stood out was the inclusion of Related Work as Section 5, all the way on the last page of the paper. This paper does an effective job setting the board context for SBP in the introduction (L13-24), but having Related Work as the second section is usually helpful for a reader to compare and contrast the introduced topic to prior work. This contextualizes the information they are about to consume. The authors might consider shifting Section 5 as is up to follow the introduction directly.

There are a couple grammar and spelling errors throughout the work (e.g. L212 "loss" should be "lose"), but nothing too glaring. It would be worth reviewing the paper for such errors before any final submission.

__Summary__

This paper demonstrates impressive results using SBP for vision tasks. The quantitative performance are enormous (Tables 1 and 2, 25-40% memory reduction with less than 1% accuracy loss) on two commonly used backbones for typical vision tasks. The work also demonstrates a simple implementation, which suggests that the concept can become a effective widespread idea. The in depth analysis of design choices also provides future researchers with a strong starting point for further study. While I am admittedly not well read in this line of research, this paper does not appear to have many flaws, generally. It is well-structured and easy to understand, and it provides strong empirical support for the benefits of SBP. Moreover, it asks the right questions about SBP and provides solid initial answers. I think this work is a welcomed contribution to NeurIPS.
questions: * A major difficulty in analyzing any training algorithm that differs from layer to layer is that there too many different ways to structure the algorithm to exhaustively examine every one. The design experiments in this paper only look at a 4-layer network (L204). How well might these insights generalize to deeper networks, especially those with skip connections?

* When using SBP, are precautions necessary to prevent vanishing gradients? It seems that any time gradient information is discarded, the network runs the risk of catastrophically interrupting the flow of information.
limitations: The paper addresses a small set of limitations (decrease in accuracy) in Section 6.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Stochastic Backpropagation (SBP) is a memory efficient training method for training video models. In this work authors presented the in-depth study of stochastic back propagation in generalized way and use this strategy for training deep neural network models for image classification and object detection tasks.

Unlike earlier work of SBP on video models, where frames were during backpropagation however keeping all the frames for forward propagation. The authors in this paper, drop feature maps in earlier layers during backpropagation, keeping all the feature maps in forward propagation phase. This is somewhat conceptually like drop out, however, limited to the backpropagation phase. 

The SBP is studied for point-wise convolution layers, convolution layers and transformer layers. SBP approximation behaves differently for different operators. For point-wise convolution SBP approximation of gradients result in 0 gradients for dropped indices for kept indices the gradients are exact in comparison to SGD. This contrasts with convolution layers with kernel size greater than 1, SBP approximation to gradient is inexact and not zero irrespective of the indices dropped or kept except some special scenarios.

In addition to the study of SBP, authors experimented with various design choices example, number of layers on which SBP needs to be applied, kept ratio, dropping strategy.  Based on the experiments, was concluded to apply SBP to 2/3 layers and have 0.5 keep ratio using grid wise sampling. This design strategy gave sufficient GPU memory saving without large drop in accuracy, in addition, the gradients gave higher correlation to SGD gradients without SBP in terms of cosine similarity.

The generalizability of SBP is evaluated on two computer vision benchmarks: ImageNet and object detection COCO. Applying SBP reduced 40% GPU memory with less than 1% drop in accuracy

strengths_and_weaknesses: SBP is memory efficient training method. It seems  to generalize well across tasks like image classification and object detection and is very simple and efficient to implement.
SBP provides 40% GPU saving with  ~1.1 x training speed up with less than 1% accuracy drop.
Unlike other training methods like gradient accumulation, SBP is not exact and result in loss of training accuracy.
The paper is very promising , since it gives a training approach which is memory efficient and comparatively faster with equivalent performance.




questions: Q1) Even though the accuracy drop is less than 1% there are few aspects of SBP that raise question if SBP can be thought as efficient training methodology or altogether different training strategy. 
1) Fixed grid sampling for backpropagation is clearly making few features more important than the others.
2) Cosine similarity though gives a promise, i.e., e SBP approximated gradients are correlated with SGD gradients and eventually the trained model might end up in similar final weights. However, how well it extends to the benchmarking tasks of ImageNet and classification is not clear.
May be one way to show SBP as an approximation could be to show if with or without SBP the correct examples are similar. 
or if the weights with or without SBP keeping all training parameters constant are close.
Can you please throw light on this ?

limitations: SBP is memory efficient training method. However, SBP is not exact and result in loss of training accuracy.
SBP specifically when applied to convolution layers  with kernel size greater than 1 and also in query of transformers are neither zero nor exact. The approximations even though result in very minimal loss in accuracy, however, the effect of this approximation on the model in the perspective of of explainibility and reliability requires separate study.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: A study of SBP, applied to image classification and object detection, is provided. A significant amount of memory is saved during training while the degradation of accuracy is relatively small. Derivations and experimental results are comprehensive and they validate the author's main points. 
strengths_and_weaknesses: The main strength of the proposed method is its practicality. It saves memory with almost no harm to the final models of interest. There is no reason someone faced with memory limitation not try this simple approach. Derivations related to stochastic backprop. are comprehensive and most of them look sound but I am not fully confident. 

I have checked the details of experimental results and ablation studies, and as far as I understand they are convincing. For example, Figure 3 (grid mask vs random mask) experiment matches the theory from the derivation. 

Three main weaknesses are novelty, technical contribution, and writing. First, the concept of stochastic backprop. is introduced in the original CVPR paper and the proposed method in the paper is incremental to the original paper. Second, it is not easy to find a non-trivial technical challenge introduced, tackled, and solved by the paper. Third, writing is difficult to follow. I have listed some of the parts that can be improved: 
L77 fully connection layer -> fully-connected layer
L84 caching of feature maps dominants the memory usage. -> did not understand 
L311 slightly loss -> slight loss  
questions: As I pointed out in the weaknesses part, I want to know more about novelty (compared to CVPR `22 paper) and technical contribution (what was the most difficult technical problem?). 
limitations: Limitations are adequately addressed.
review_title: null

Meta-review: metareview: The contribution is novel and well received by the reviewers and shows that SBP should indeed be considered more often. 
confidence: Certain
Decision: Accept