An In-depth Study of Stochastic Backpropagation
Authors: Jun Fang
Mingze Xu
Hao Chen
Bing Shuai
Zhuowen Tu
Joseph Tighe
Venue: NeurIPS 2022 Accept (2022)
Keywords: Memory efficient training method
stochastic backpropagation
image classification
object detection

Abstract: In this paper, we provide an in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks. During backward propagation, SBP calculates gradients by using only a subset of feature maps to save GPU memory and computational cost. We interpret SBP as an efficient way to implement stochastic gradient decent by performing backpropagation dropout, which leads to significant memory saving and training run-time reduction, with a minimal impact on the overall model accuracy. We offer best practices to apply SBP for training image recognition models, which can be adopted in learning a wide range of deep neural networks. Experiments on image classification and object detection show that SBP can save up to 40% of GPU memory with less than 1% accuracy degradation. Code is available at: https://github.com/amazon-research/stochastic-backpropagation

An In-depth Study of Stochastic Backpropagation
Jun Fang Mingze Xu' Hao Chen
Bing Shuai
Zhuowen Tu Joseph Tighe
AWS AI Labs
(junfa, xumingze, hxen, bshuai, ztu, (egheoo@ammzonncco
Abstract
In this paper, we provide in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks. During backward propagation, SBP calculates gradients by using only subset of feature maps save GPU memory and computational cost. We interpret SBP as an efficient way to implement stochastic gradient decent by performing backpropagation dropout, which leads to significant memory saving and training run-time reduction, with minimal impact on the overall model accuracy. We offer best practices to apply SBP for training image recognition models, which can be adopted in learning a wide range of deep neural networks. Experiments on image classification and object detection show that SBP can save up to 40% GPU memory with less than 1% accuracy degradation. Code is available at: com awwaaaaaannnaaaa arrazoonrreeearcc/sttcccateeeese http:oocccttttbbbcccppppgggiaooo
backpropagation
Introduction
A common practice improve accuracy when training deep neural networks is to increase the input resolution [19]. model depth 77 and model width [21]. However, this can significantly increase the GPU memory usage, making training devices with limited resources difficult. For example, training an object detector using ConvNeXt- -ase [14] as the backbone with batch size 2 requires over 17 GB of memory, which cannot fit on many modern GPUs.
number prior works have explored memory-efficient training designs, especially for large model training on high resolution images or videos. Mixed-preciiing training [17] uses lower precision to represent the network weights and activations for certain layers the network. Gradient checkpointing (2) restores intermediate feature maps by recomputing nodes in the computation graph during the backward pass, which requires extra computations. Gradient accumulation [12] splits a mini-batch into smaller chunks to calculate gradients iteratively, which slows down training process.
recent method, Stochastic Backpropagation (SBP) [3]. offers new perspective for memory- efficient model training. It performs partial execution of gradients during backpropagation on randomly selected frames when training video models [13 24 23 22]. It achieves considerable memory saving by only caching a subset of video frame feature maps without significant accuracy loss. However, SBP B only applied to video-based models using two architectures and its design space has not be thoroughly explored on wider tasks. The explanation provided in K for the effectiveness of SBP is simple but narrow attributes the validity of the method to the high redundancy of video frames.
This leaves number of questions around SBP unanswered: (1) Is SBP 's effectiveness limited the temporal redundancy of video or can it effectively leverage spatial redundancy? (2) If so, can SBP be generalized to image-based based tasks, with more limited redundancy? (3) SBP generalizable
Cooresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

to different networks (e.g., MLP, CNN, Transformer) and operators? (4) What are the key design choices when implementing SBP?
To answer these questions, we first formalize the SBP process and analyze the effect of SBP on the calculation of gradients. With this formulation, show how generalize the idea of SBPto image models and present an in-depth study of the design choices when using SBP. During backward propagation, SBP calculates gradients from subset of spatial feature maps to save memory and computation cost. We observe that the calculated gradients by SBP are highly correlated with the standard SGD gradients, indicating that SBP is reasonable approximation. We also provide new, straightforward implementation of the idea that requires only few lines of code. We explore several important design strategies, which have noticeable impact on the performance. We validate the generalizability of SBP on two common tasks (image classification and object detection) with two popular network architectures (ViT 6 and ConvNeXt [44 ). We show that SBP can effectively train image models with less than 1% accuracy loss in a memory-efficient fashion.
2 Related Work
Since deep neural networks have expanded in both depth 7 and breadth 21 in recent years, the area of designing memory-effiient training methods has attracted fair amount of attention. By saving and recomputing nodes in the computation graph during backpropagation, gradient checkpointing 12] can save a large amount of memory. Gradient accumulation [12) splits the large sample batch into several mini batches and runs them iteratively without updating model parameters. This saves the GPU memory proportionally without affecting the accuracy, because accumulating the gradients of these sub-iterations and then updating the model parameters are identical to directly optimizing the model with a global batch size. However, these two methods slow down the training process. Multigrid [20] proposes to use different
shapes to speedup the model training. Sparse network [S] particularly targets the recognition tasks but can only save the memory theoretically. Sideways [15] and its follow-up [16] reduce the memory cost by overwriting activations whenever new ones become available, but they are limited to causal models. SBP B reduces a large portion of GPU memory by only backpropagating gradients from incomplete execution for video models. In this paper, we generalize SBP to more standard computer vision tasks including image recognition.
Understanding Stochastic Backpropagation (SBP)
In this section, we first formalize the process of Stochastic Backpropagation (SBP) (sec. 3.1 and show how to compute the gradient for two common operations (sec. 3.2). With these formulations established, we analyze how to perform the chain rule of gradient caleulations for SBP (sec. 3.3
Finally, we present simple implementation of SBP in practice with only few lines of code (sec. 3.4
3.1 Formulation of SBP
We first present a general formulation of SBP 3] We denote loss function L(O) to learn a model on a dataset D- = {d, J where θ is the collection (vector, matrix, or tensor) of all parameters of the model, N is the total number of training samples, d, E xxxxxxXxx is an input data sample, where H and W are spatial resolution height and width respectively, C' is the input channel size, T= 1 represents an image and T > represents a video.
In each training iteration, the model with SBP processes mini-batch of data samples S R xxxxxxxW>x xc C D to compute the loss after the forward pass, where B is the batch size. During the backward pass, different from the traditional backpropagation that relies the full feature maps X; E R XxxTxxH..We W. XC.(C (i is the index of the layers), an incomplete execution for backprop- agation (i.e., SBP) only utilizes subset of the feature maps X,ub E 12xI; "h xH."">W." xC, to approximate the gradient calculation. Specifically, it updates the model weights as
Quer11 Oster -" "B
1
Σ L
VeL(x Qater)
where 7) the learning rate, Ve the collection of model weight gradients, Oster and Oster+1 are model weights at the current and next iteration, respectively.

---

SBP computes all forward paths, but only caches subset of feature maps in the backward pass. Note that the method in [3] is a special case of 1 as it operates on the temporal dimension by using subset of video frames X E ER "H,, xXC.
3.2 Backward Gradient Calculation
Next, we derive equations to compute the gradient during the backward phase of SBP. We denote that at layer i of the model, operator fi with learnable weights WG, to processes the feature maps X; E XBBTTx,,,,,,........
wheere C refers to the input channel size.
3.2.1
Calculation of Linear Layers
We start by exploring f. as point-wise convolutional (PW-Conv) layer. PW-Conv is widely used in the state-of-the-art computer vision models such as ViT [6] and ConvNeXt (14). It is equivalent to the linear layers in Transformer MLP blocks [14] and is also known as channel-wise fully connected layer. It can be either implemented through the linear layer, or the 1 x convolutional layer. It is mainly used to enrich the channel information with weights W E pe!""x"" and the analysis on PW-Conv can be easily extended to other layers.
The forward pass of this layer is (the bias term is omitted for simplicity):
X = f((x.) = X,W,,
(2)
where X4+1 E (BxT, (XXW)) (X; and X++1 are reshaped as matrix format). During the traditional backward pass with full backpropagation. the gradients of both activations and weights are calculated by:
dX441 2Loss/8X.14
dW,. @Loss/OW, = XTT dX
dx, aLoss/0X, = dd
It requires storing the weights Wr, E xC,"" and full feature maps X; E RRBBXTxx to compute the gradients. In many cases, B T₁ x H; x W; >> Cruut i.e., caching of feature maps dominates the memory usage. In order to effectively save GPU memory during training, SBP only caches subset of the feature maps for backpropagation.
In the backward pass of SBP process, we split the full feature maps X, into two subsets Xheep and X.rrop, which denote the corresponding indices of the sampled feature maps that are kepl and dropped, respectively. An example of X, X_Kepp can be Xi(:, even index :, :] for video input, or [g :, even index on H, odd index on W,:] for image input (e.g., Fig. left), and x-rop is the complementary part of the feature maps. We have X-50ep n X-drop = and x-Keep U X = X,- We denote keep-ratio as the number of gradient kept indices j E Zaeepp over the number of all indices.
One advantage of simplifying the SBP problem by using PW-Conv operator is that the nodes in spatial or temporal dimensions can be calculated independently through the forward and backward passes. Therefore, we can update the forward pass separately on the kept and dropped indices:
X [X x [xtee,, x.roo|W,,, ie.. Xkeep Xeee x XdropW
as well as the backward pass:
dX [dxtero dxdrrap)
dx keep = dx.ersy.. dx.rrp = dx
(5)
dW, =
(X
dx dxdren
x axteep drrop" ax
SBP drops all the gradients calculation with superscripts drop and keeps all the gradients calculation with superscripts keep, thus saves memory and computation on the dependencies of dropped indices.
In other words, the memory usage and the backward computation cost are proportional to the keep-
ratio of the original case. Mathematically, it is equivalent to set d'x.drop 0 (thus d.x.
0) and
3

---

uses d = X kroop dx,+1 as a stochastic approximation of the original weight gradients dW,, to update the model weights.
3.2.2 Calculation of Convolutional Layers
We then derive the calculation of a general convolutional layer (ConvLayer) as ft. Unlike the PW- Cony, nodes of general Layer with kernel size > spatial or temporal dimensions are no longer independent in forward and backward calculations. Assume that we apply a convolutiona kemel k x k with stride s. The regular backpropagation of the ConvLayer is calculated as follows:
dX;- zeropod (dX++1) Wa
(6)
JW,, X; dX,+1
where is the convolution operation, zeropad is to pad the edge the matrix with zeros, and W', represents the 180 rotation of W
From Eq. (6) in the gradient calculation of ConvLayer at given location j on the input feature map as xy E xx;, all the elements from dX;+1 inside the region of the kernel size will contribute to the gradient calculation of dæ: Therefore, in SBP of ConvLayer, when set dx 0 for certain areas, in general, two things will happen: (a) even if j E Zdrop and dr = 0, dx will not be 0 due to the non-zero gradient contribution of the neighbors of on dxx++1; (b) dx; will be an approximated gradient unless all dx with in the neighbor areas of j in terms the kernel size (i.e., indices p within the convolutional kernel of index f) are kept and there is chain rule effect on dx 1- However, there is a special scenario: when $ 2 k. the neighbor effects disappear due to the fact that d'i++1 needs to be interweaved in gradient calculation. As result, nodes in spatial or temporal dimensions are patch-wise independent, which is similar to the case of PW-Conv.
3.2.3 Discussion
From the above derivation, we see that the approximation effect of SBP performs differently when the operator changes. For layers, such as linear layer or PW-Conv, the activation gradients (i.e, dx;;) are either 0 or exacf" While for convolutional layers with kernel size larger than 1, most of the gradients on the activations are neither nor exact. Due to space limitation, we only derived SBP on two most popular operators. For derivation of other operators, please refer to the supplementary material.
3.3 Chain Rule Effect
We have derived the calculation of SBP single layer in the previous section. In this section, derive the calculation of SBP on a stack of multiple layers. For simplicity, we take two layers as an example, which can be casily generalized to more layers.
3.3.1 Two PW-Conv Layers
We first consider scenario with two PW-Conv layers. We assume a layer before fi, i.e., fi-1 1. is another PW-Conv layer with weights W f--1. The forward pass of this layer is (the bias term is also omitted):
X, fi 1X X.-
Then we can calculate the gradients of weights and activations in layer fi-1 similar to Eq.
dX,-1 dx,WT_ d..+IW.WW....
(8)
dW, j = x ,dx; dX.
We first consider the case when there is no additional gradient dropout at layer ft-1- Since dX has non-zero gradients at kepi indices (i.e., dx and zero gradients at dropped indices (i.e.. dx.rop), Eq. (8) can be updated to:
dx,-1 [dxter dx
dx.eep d axterpWTWW WT dx = axxdr WT.. = 0,
(9)
dW, = x ax
"By "exact" mean the exact calculation of the gradient for the given batch, though it is still an estimation due SGD.

---

From the equation, we can see that if there is no additional gradient dropout applied on layer fi-1. through the chain rule, the keep and drop set will transfer from X to X; identically.
we apply gradient dropout on layer fi-1 as well, L.e, on layer fi-1, we split dX, into dx, and dxprmr Assume that the keep and drop indices set at layer + is #441 and 2441 and at layer i is 2 Zheep and T, respectively. Then through Eq. 8 and Eq. (9) the keep and drop indices on layer i will be:
2 knep = Zhoep n 1
2 2/41 U U.
(10)
Eq. 9 becomes:
dx,-1 = [dx dx
dx = dx W7, WT 11
dx 0,
(11)
dW f-- xteep" ax W
where keep' keepp and drop' = 22-11
3.3.2 PW-Conv + ConvLayers
Now let's look the chain-rule effect of the case when f is ConvLayer with kernel size k >> 1 and stride s. Using the chain rule, we can update the Eq. (6) as follows:
dX dx zeropad(dX.) W J:-1 2x W n-11
(12)
dw, /- X dX, X;-1 dX,+117.
We can see from above equation that each dx:- receives the backward gradients from the neighbors of dæy, which might be partially dropped out, thus the gradient dx is longer exact.
From Eq. 12). the dropout at layer makes the matrix dX;; sparse (only indices at 2keep are non-zero). If conduct dropout the current layer fi-1 (i.e., have zKeep and Z Z the matrix dX;-1 --1 will become even more sparse unless the dropout happening on the same location Z:41 Z Zdrop
If we stack PW-Conv ConvLayer further and further in interweaving way (i.e., Zdrap # Z Z the chain rule effect may lead to a very sparse output derivative matrix (e.g., dxX;-1). However, stacking ConvLayers (due to the space limitation, we skip the math derivation) will not have such trend, as the gradient dropout in ConvLayer only creates approximated gradient but not many zeros, because of the neighbor effect of gradient calculation.
3.3.3 Discussion
Overall, in terms of the chain rule effect, we can see from Eq. 11 that stacking PW-Conv layers, especially in interweaving way. may lead to sparse keep set Z/ee and vanishing gradient of dW,,,. f--.. Thus, in practice, we recommend to drop gradients at the same positions across layers to avoid the gradient vanishing and maintain high accuracy. Stacking PW-Conv ConvLayers (in Eq. (22) also tends to make the keep set sparse and thus should be avoided. While stacking ConvLayers doesn't 't have such effect, it stacks the non-exact gradient almost every position but tends to make the gradients gradually drift from the dense gradients.
3.4 Efficient Implementation
With a deeper understanding of how SBP works, we provide a simple and efficient implementation of the SBP technique in Alg. [ We point out that this is more efficient than the prior work (3) as there is (1) no need do the re-compute the forward during the backward pass, thus is computationally more efficient, and (2) no need to manually cache the intermediate feature maps and fill zero gradients in the backward pass, which is much simpler. We emphasize that it is very simple and easy implement, it only needs 3 lines of code to do the forward, and the backward (no need to implement manually) can seamlessly handled by the ease of "autograd engine" the deep learning frameworks. supports point-wise convolutional layers, the dot-product attention layers, and multilayer perception
(MIP) in popular networks including ViT 6 and variants 6 I1 13 ConyNeXt [4), and hybrid architectures 4
5

---

Algorithm Pytorch-like pseudocode of SBP for an arbitrary operation f.
It f: an arbitrary operation
grad_keep_idde sampled indices where gradients are kept
# grad_drop_idx: sampled indices where gradients are dropped
def sbp_f(f, imputs, grad_ keep__eep_idx, grad_drop_idx):
initiate outputs
outputs torch (outuut shape, device-inpute.deve devicee # forward with
calculation, gradients will be calculated with torch.autore
with wraich.enable_gree
outputs (grad_keep kerp__xeee_id (inputs [grad_ forward without gradient calculation
with corch.no_grad(. outputs ggradddrop_idee f (inputs ggrad_drop (trop_idx))
outputs
How set S0P keep-ratio
61-
Lnefom soup
Dec macweiing keep catles trcresiin eeppralis
Percarage 18 Eapas 11g D0P
30P koct 1ob0
SSP-0.5
58P-6.25
Illustration of the effect on applying SBP on different network layers and keep-ratios. Left: accuracy curve of applying SBP different percentage of layers, the keep ratio of SBP is 0.5. Middle: trade-off of keep-ratio versus accuracy, every SBP layer has the same keep-ratio. Right: comparison of using uniform, decreasing and increasing keep-ratios for the SBP layers.
Figure I:
Uniform keep-ratic
Decreasing keep ratios
Increasing keep-aatio
520
1816
1500
2005
500
1000
1503
2000
500
1000
3500
2000
k0
Lyprtt
1.3
500
1036
1500
2000
3.0
500
3300
1500
2000
Layer
500
1000
3500
2010
520
300
Laperac
500
batches
batches
batches
Figure 2: Cosine similarity of gradient weights with and without applying SBP ViT-Tiny with uniform, decreasing, and increasing keep-ratios. Results on Layer 20 (MHSA), 40 (MLP), 60 (LayerNorm), 80 (MLP) displayed. Results other layers have similar behavior.
4 Design Strategies
In this section, we investigate the effectiveness of different design strategies of SBP image tasks. We use ViT-Tiny [6] model to evaluate the top-1 accuracy on ImageNet [18] validation dataset. The ViT-Tiny is smaller version of the ViT [6] variants, it uses patch size 16 with 12 transformer blocks (more than 80 layers), the embedding dimension 192 and the number of heads is 3, i.e., each head has dimension 64. The experiments trained for 100 epochs the ImageNet dataset. The conclusions remain consistent for larger models and longer training schedules.
How many layers should be applied with SBP? To answer this question, we progressively apply SBP to transformer layers from the last layers to early layers and present their results in Fig. Left. As shown, we observe a sweet spot when applying SBP to 8 out of 12 block layers, which reduces GPU memory consumption by 33% with an accuracy drop within 1.5%. Although applying SBP on all the transformer layers can save the more GPU memory, it causes large accuracy drop. Therefore, unless stated otherwise, apply SBP 2/3 of all layers of VIT models for the rest of this work.

---

1960 1752
2010
180 1560 1750
Figure 3: Left two: a particular example of gradient keep mask for grid-wise sampling and random sampling. Right two: cosine similarity of weights gradient between with and without applying SBP on ViT-Tiny. The SBP are applied with keep-ratio of 0.5 on these two mask sampling methods.
How much gradients should be kept in SBP? this section, we discuss the trade-off between the gradient keep-ratio and accuracy in SBP. As reminder, keep-ratio r is the number of gradient kept indices over the number of all indices, and it represents the percentage of spatial feature map values used to calculate the gradients. In general, the higher the keep-ratio is, the more gradient information is preserved, and the higher the accuracy is, the less memory is saved. A natural question arises: what is good keep-ratio to balance the accuracy drop and memory saving?
In Fig. Middle, we show the trade-off between model accuracy and the keep-ratio of gradients. In this case, we apply the same keep-ratio to all the SBP layers at the same gradient kept indices. We name this strategy "uniform keep-ratio". We observe that setting the keep-ratio to be 0.5 achieves a reasonable trade off while having a keep-ratio of 0.25 is too aggressive, which results in dramatic accuracy drop (3.04%) of the model. We also explore different keep-ratio methods by linearly increasing or decreasing the keep-ratios from early to late layers while keeping their average keep- ratio to be 0.5. Specifically, we use keep-ratios of [0.25, 0.32, 0,39, 0.46, 0.53, 0.60, 0.68, 0.75] (increasing) its inverse (decreasing) on 8 transformer blocks. As shown in Fig. Right, both models perform significantly worse.
We simulate the gradient calculation on ViT-Tiny with 2000 data batches randomly sampled from the ImageNet training dataset. We apply different keep-ratio methods on the same model with the same fixed weights, and calculate the correlation (measured by cosine similarity) between the weight gradients of applying SBP and the original exact weights gradients, ie., Cosine(dW, sgP; dWW.. ". see)- From Fig. 2 we observe that the weight gradients have a stronger correlation when using uniform keep-ratios compared to the case of increasing or decreasing keep-ratios. We believe that the fact that uniform keep-ratios keep the location of the dropped gradients consistency between different layers helps preserve gradient information at these spatial locations. However, non-uniform keep-ratios will drop at different locations between different layers, which adds noise to produce less accurate estimation of the original gradients, and hence lowers accuracy.
How to sample the gradient keep mask? SBP calculates the gradients by only using a subset of feature maps. A key question is how to sample the subset of feature maps to mask the kept gradients. If we fix the spatial locations of the gradient kept indices for every training step, the backward only propagates the valid gradients (non-zeros) on these fixed locations, which forces the model to only learn from fixed partial spatial information and therefore severely hurts the model performance. A better strategy is to randomly sample the gradient kept indices in each training step, so that it can statistically visit every spatial location with equal importance.
We examine two mask sampling strategies: grid-wise mask sampling and random mask sampling. See Fig. left as a particular example of keep-ratio 0.5. We report the accuracy in Fig. Overall, grid-wise mask sampling achieves 0.5% ~ 1.7% higher accuracy over random mask sampling. In Fig. 3 stronger correlation is also observed by using the grid-wise sampling. Grid-wise sampling more structured sampling approach seems to provides more accurate estimation of gradients. Unless stated otherwise, we use grid-wise sampling in this paper.
How to apply SBP on MHSA? In general, the transformer block consists of two sub-blocks: Multi- Head Self-Autention (MHSA) and Multi-Layer Perception (MLP). Theoretically, the memory usage of applying SBP on MLP (two linear layers) is proportional to the keep-ratio However, the memory saving of applying SBP on MHSA depends on the sampling method on the query, key, value (QKV), V), and the ratio of head dimension d over the number of tokens n. From (3), activation maps needed
7

---

Effect of gradients keep mask
Effect drop methods on MHSA
05
max
61-
Top 1ocourac
63-
Top-
56
59
Figure 4: Left: effect of different sampling method for gradient keep mask. Right: effect of different SBP dropping methods on Multi-Head Self-Attention (MHSA).
Drop OKV
Drop Query only
1.3
1.0
500
0.5
6.5
3.5
0..8
6.8
5.8
5to
1000
1200
2002
500
1000
1502
2000
500
1000
2010
00
520
1000
1500
2000
500
1000
1503
2000
500
1000
3503
2000
* of data b batches
# of data batches
data batches
Figure 5: Cosine similarity of weights gradient for different method on applying SBP on ViT-Tiny MHSA layers with a keep-ratio of 0.5. Left: Drop QKV, middle: Drop Query only, right: Drop Head.
for gradients calculation in MHSA include the inputs and outputs (2hda), QKV vectors (each with hdn). the attention weight maps (2hnn before and after the softmax), thus requiring the memory of 3hdn + 2hnn in total, where h is the number of heads. Dropping gradients query only (Drop Query only) or on all QKV (Drop QKV) enjoys the following memory usage ratio:
Drop Query only:
2hdn+ Sointhhnr--0 Shdu than
" +2
+2))+
(13)
Drop QKV:
3hutor 42hamr2
3+
3ludee 2han
3+2
In video transformers, the number of tokens is much larger than the head dimension d (i.e., in video Swin Transformer [13) n 392 and d = 32, = 0.082) and thus the attention weight maps 2hmn comprise most of the memory. Therefore, dropping gradients on query only in the video model is good enough to save memory that its memory is linearly proportional the keep ratio [3].
However, in image task models such as VIT with a typical input size 224 and patch size 16, 14 14 196 and d 64, 78 0.327. With keep-ratio 0.5 (or r 0.25), dropping gradients on query alone uses 0.61 x (or 0.42x) memory while dropping all QKV reduces its memory usage ratio to 0.46x (or 0.22x). From Fig. 4 dropping gradients all QKV also gains better accuracy. We also report the performance of dropping gradients along the attention heads, but we did not observe accuracy improvement. Hence, we apply SBP on all QKV for image-based transformers.
We plot the cosine similarity of the weights gradients of applying SBP and the original exact gradients in Fig. 5 One interesting observation is that Drop QKV has slightly better correlation Drop Query only and a much higher correlation to Drop Head. We believe an accurate estimation of the gradients might be an important factor to the model performance trained with SBP, which provides insight for further investigation.
5 Generalizability of SBP
We evaluate the generalizability of our proposed SBP on two computer vision benchmarks: image
classification on ImageNet [18) and object detection COCO [0]] In order to do a fair comparison of the same network, we keep the training hyper- paraamparameters of optimizer, augmentation, regularization, batch size and learning rate the same for given model, and only adopt different stochastic depth

---

Table I: Accuracy and memory results of applying SBP for ViT and ConvNeXt on ImageNet.
Network
Keep-ratio
Batch size
Memory (MB / GPU)
Top-1 accuracy (%)
Vin-Tiny ViT-Tiny
no SBP
256
8248
73.68
0.5
256
5587 (0.68x)
73.09 (-0.59)
ViT-Base
no SBP
64
10083
81.22
ViT-Base
0.5
64
7436 (0.74x)
80.62 (-0.60)
ConvNeXt- SBP 0.5
128
12134
82.1
Non-NeXt-Tiny
128
7059 (0.58x)
81.61 (-0.49)
Next-aaee SBP
64
14130
83.8
ConvNeXt-Ban 0.5
64
8758 (0.62x)
83.27 (-0.53)
augmentation [2] for different model sizes. The Mixed Precision Training [17] method is enabled for faster training. All experiments are conducted machines with 8x Tesla 16GB V100. Each accuracy is reported with an average result of three runs with different random seeds. For more details of experimental settings, please refer to the supplementary material.
5.1 Classification on ImageNet
We report the memory and accuracy trade-off in Tab. based VIT [6] and convolutional based ConvNeXt 14]. We train 300 epochs for both these two networks by following the training recipe of ConvNeXt [4].
[ab.[for for two state-of-the-arr networks: transformer-
For VIT models, we apply SBP on the last 8 transformer blocks including MHSA and MLP layers with the keep-ratio of 0.5, reducing GPU memory usage by approximately 30%. Note that the memory saving ratio of the network not proportional the the drop-ratio (1- keep-ratio) because we only apply SBP on of all the layers. With keep ratio of 0.5, the SBP technique can still effectively learn with an accuracy drop of only 0.59% and 0.60% for ViT-Tiny and ViT-Base, respectively.
For ConvNeXt models. it builds the basic block with depth-wise convolutional (DW-Conv) layer with kernel size 7 x followed by two point-wise convolutional (PW-Conv) layer and the DW-Conv layer only consumes a small portion of GPU memory (less than 10%). It has multi-scale stages where early stages have higher redundancy and occupy most of the GPU memory. Therefore, we apply SBP on the PW-Conv and down- sampling layers in the first three stages, and do not apply it on the last stage. With keep ratio of 0.5, ConyNe model with GPU memory saving of above 40% and accuracy drop within 0.53%.
and ConvNeXt-Bar Base can efficiently learn the
5.2 Object Detection and Segmentation on COCO
We fine-tune the vanilla Mask R-CNN g and Cascade Mask R-CNN [1 the COCO dataset IO with ConvNeXt-Timm and ConvNeXt-Bi backbones pretrained on ImageNet-I K, respectively. Following the same hyper-parameteer settings of [14], we apply SBP on the ConvNeXt backbones to save training memory. Results in Tab, show that SBP with keep-ratio of 0.5 can still learn the detection task reasonably well with 0.2% and 0.7% loss on the box and mask average precision (AP) for ConvNeXt-Bar and ConvNeXt-Tiny backbones, respectively. However, only consumes about 0.7x of the GPU memory.
Note that training detection models on high resolution images (up 800 1333) is memory intensive even for a very small batch size. For example, with backbone of ConvNeXt -t--BBeee and batch size 2 2, the model training requires 17.4GB of GPU memory which may lead to out of memory error for common devices such as Tesla 16GB V100. While with the help of SBP, the model can still be effectively trained on GPUs with 12.5GB memory, which is much more feasible for many research groups. Here apply SBP only in the backbone, we believe it has potential to apply SBP more other layers in the detection model to achieve more promising results in terms of memory efficiency.
'The two PW-Cony layers equivalent MLP block in transformers.

---

Table 2: COCO object detection and segmentation results using Mask-RCNN with backbone with backbone ConvNeXt-B.
ConvNeXt-T and Cascade
Backbone
Keep-
Batch
Memory (GB (GB/GPU)
Aplox
AP7
Apmuis
AP
AP7
ratio
size
ConvNeet-TT no SBP
8.6 5.9 x0.69x)
46.2
67.9
41.7
65.0
44.9
ConvNeXt-T
0.5
45.5
67.4
50.1
41.1
64.4
44.1
Conv-Nett-B no SBP
52.7
71.3
57.2
45.6
68.9
49.5
ConvNeXt-B
0.5
12.5 (0.72x)
52.5
71.3
57.2
45.4
68.7
49.2
6
Discussion
In this work, we present comprehensive study of the Stochastic Backpropagation (SBP) mechanism with generalized implementation. We analyze the effect different design strategies to optimize the trade-off between accuracy and memory. We show that our approach can reduce up 40% of the GPU memory when training image recognition models under various deep learning backbones.
In general, SBP is memory efficient training method, but it still causes slight loss of accuracy. In our current results, SBP produces only a small amount of training speedup(~> 1.1x). How to further speed up the training process is part of our future work.
Limitations:
Societal impact:
Our work studies the stochastic backpropagation mechanism, which be used for training general deep learning models. Misuse can potentially cause societal harm. However, we believe work is general approach and is consist with the common practice in machine learning.
7 Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their helpful suggestions. This work was funded by Amazon.
References
[1] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018.
[2] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv:1604.061 1604.061444444 2016.
[3] Feng Cheng, Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Li, and Wei Xia. Stochastic backpropagation: A memory efficient strategy for training video models. CVPR, 2022.
[4] Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurlPS, 2021.
[5] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv:1907.04840, 2019.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv:2010.11929, 2020.
[7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[8] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[9] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinherger. Densely connected convolutional networks. In CVPR, 2017.
10

---

[10] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
[11] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[12] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Official Implementation of Swin transformer. htttp:://githbb... wttp///...ooom/miooo/oo/oooo Swin-Transf ormer) 2021. [Online; accessed 12-Sep-2022).
[13] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv:2106. 13230, 2021.
[14] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. CVPR, 2022.
[15] Mateusz Malinowski, Grzegorz Swirszez, Joao Carreira, and Viorica Patraucean. Sideways: Depth-parallel training of video models. In CVPR, 2020.
[16] Mateusz Malinowski, Dimitrios Vytiniotis, Grzegorz Swirszez, Viorica Patraucean, and Joao Carreira. Gradient proppadproopgttionn for scage temporal video modelling. CVPR, 2021.
[17] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training. 17Xiv:1710.037400 2017.
[18] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015.
[19] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In JCML, 2019.
[20] Chao- Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. multigrid method for efficiently training video models. In CVPR, 2020.
[21] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
[22] Mengmeng Xu, Chen Zhao, David Rojas, Ali Thabet, and Bernard Ghanem. G-tad: Sub-graph localization for temporal action detection. In CVPR, 2020.
[23] Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S Davis, and David J Crandall. Temporal recurrent networks for online action detection. ICCV, 2019.
[24] Mingze Xu, Yuanjun Xiong. Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, and Stefano Soatto. Long short-term transformer for online action detection. NeurIPS, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof's of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [Yes] See Section 5 first paragraph and Section 8.3 in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section S first paragraph and Section 5.T]s first paragraph. (c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Yes] Each accuracy report is average result of running experiments three times with different random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5 first paragraph and Section 8.3 in the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We cited the all datasets we used, such as ImageNet (18) ad COCO [TO].
(b) Did you mention the license of the assets? [No] ImageNet is under BSD 3-Clause License and COCO is under Creative Commons Attribution 4.0 License.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identi- fiable information offensive content? [No] We do not have sufficient resource to exhaustively screen offensive content from the datasets we used. We expect, because the datasets only contain images of people with daily activities and common objects, they are unlikely to contain offensive content. They are both very widely used.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "ohe-fZZHnJu", "reviewer": "Reviewer_yhnq", "summary": "Stochastic Backpropagation (SBP) is a memory efficient training method for training video models. In this work authors presented the in-depth study of stochastic back propagation in generalized way and use this strategy for training deep neural network models for image classification and object detection tasks.\n\nUnlike earlier work of SBP on video models, where frames were during backpropagation however keeping all the frames for forward propagation. The authors in this paper, drop feature maps in earlier layers during backpropagation, keeping all the feature maps in forward propagation phase. This is somewhat conceptually like drop out, however, limited to the backpropagation phase. \n\nThe SBP is studied for point-wise convolution layers, convolution layers and transformer layers. SBP approximation behaves differently for different operators. For point-wise convolution SBP approximation of gradients result in 0 gradients for dropped indices for kept indices the gradients are exact in comparison to SGD. This contrasts with convolution layers with kernel size greater than 1, SBP approximation to gradient is inexact and not zero irrespective of the indices dropped or kept except some special scenarios.\n\nIn addition to the study of SBP, authors experimented with various design choices example, number of layers on which SBP needs to be applied, kept ratio, dropping strategy.  Based on the experiments, was concluded to apply SBP to 2/3 layers and have 0.5 keep ratio using grid wise sampling. This design strategy gave sufficient GPU memory saving without large drop in accuracy, in addition, the gradients gave higher correlation to SGD gradients without SBP in terms of cosine similarity.\n\nThe generalizability of SBP is evaluated on two computer vision benchmarks: ImageNet and object detection COCO. Applying SBP reduced 40% GPU memory with less than 1% drop in accuracy\n", "questions": "Q1) Even though the accuracy drop is less than 1% there are few aspects of SBP that raise question if SBP can be thought as efficient training methodology or altogether different training strategy. \n1) Fixed grid sampling for backpropagation is clearly making few features more important than the others.\n2) Cosine similarity though gives a promise, i.e., e SBP approximated gradients are correlated with SGD gradients and eventually the trained model might end up in similar final weights. However, how well it extends to the benchmarking tasks of ImageNet and classification is not clear.\nMay be one way to show SBP as an approximation could be to show if with or without SBP the correct examples are similar. \nor if the weights with or without SBP keeping all training parameters constant are close.\nCan you please throw light on this ?\n", "limitations": "SBP is memory efficient training method. However, SBP is not exact and result in loss of training accuracy.\nSBP specifically when applied to convolution layers  with kernel size greater than 1 and also in query of transformers are neither zero nor exact. The approximations even though result in very minimal loss in accuracy, however, the effect of this approximation on the model in the perspective of of explainibility and reliability requires separate study.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "SBP is memory efficient training method. It seems  to generalize well across tasks like image classification and object detection and is very simple and efficient to implement.\nSBP provides 40% GPU saving with  ~1.1 x training speed up with less than 1% accuracy drop.\nUnlike other training methods like gradient accumulation, SBP is not exact and result in loss of training accuracy.\nThe paper is very promising , since it gives a training approach which is memory efficient and comparatively faster with equivalent performance.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "_RGMLqSQe-V", "reviewer": "Reviewer_WMoQ", "summary": "The paper describes stochastic back propagation for image classification and object detection. They dropout certain feature maps during back propagation which helps in saving memory and speeding up training.", "questions": "None", "limitations": "Yes", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is good that you have showed the SBP for a more general task of image classification and object detection, building upon the previous work on video models. I like that you have verified its application for a transformer and ConvNeXt. The analysis is exhaustive and the insights are useful. There are minor typos and grammatical mistakes. Another round of proofreading would be useful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CUwBVHKzE5P", "reviewer": "Reviewer_Jp2J", "summary": "This paper evaluates the concept of Stochastic Backpropagation (SBP) on object detection and image classification using a traditional convolutional neural network backbone (ConvNeXt) and a transformer backbone (ViT). This expands on the work of [4] that introduced the concept of SBP exclusively for video inputs. The paper provides a formalization of the topic, evaluates efficient implementations, and examines design strategies in determining how to best apply SBP. Quantitative performance is evaluated on the widely used ImageNet and COCO datasets.", "questions": "* A major difficulty in analyzing any training algorithm that differs from layer to layer is that there too many different ways to structure the algorithm to exhaustively examine every one. The design experiments in this paper only look at a 4-layer network (L204). How well might these insights generalize to deeper networks, especially those with skip connections?\n\n* When using SBP, are precautions necessary to prevent vanishing gradients? It seems that any time gradient information is discarded, the network runs the risk of catastrophically interrupting the flow of information.", "limitations": "The paper addresses a small set of limitations (decrease in accuracy) in Section 6.", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Technical Contribution__\n\nThis paper demonstrates the widespread applicability to vision tasks of a reasonably straightforward concept: improved memory efficiency during network training by stochastically choosing some subset of layers to compute gradients for at each backpropagation pass. This work builds on the work of [4], providing a theoretical and empirical groundwork for broader usages than the video prediction use case of [4].  Although this work did not introduce the concept of SBP, it provides a very good argument that SBP should see more prominent usage.\n\nThe efficient implementation proposed in Algorithm 1 is clearly demonstrated, and it is explained that it does not require any recomputation of values required for backpropagation.  The design analysis in Section 3 is useful context for understanding the impact that design choices will have (e.g. Figures 1-4). However, the combinatorially explosive number of ways SBP could be applied to a network raises questions about how generalizable these insights are. Nevertheless, specific outcomes like grid sampling outperforming random sampling when choosing gradients to keep, as shown in Figures 3 and 4, are insightful.\n\n\n__Presentation__\n\nThis paper sets up the problem really well in L25-36. It makes very clear how [4] provides a limited scope of evaluation and then indicates exactly how it plans to expand on the scope of [4]. Without even leaving the first page, it is clear to the reader what this paper is about.\n\nOne odd convention that stood out was the inclusion of Related Work as Section 5, all the way on the last page of the paper. This paper does an effective job setting the board context for SBP in the introduction (L13-24), but having Related Work as the second section is usually helpful for a reader to compare and contrast the introduced topic to prior work. This contextualizes the information they are about to consume. The authors might consider shifting Section 5 as is up to follow the introduction directly.\n\nThere are a couple grammar and spelling errors throughout the work (e.g. L212 \"loss\" should be \"lose\"), but nothing too glaring. It would be worth reviewing the paper for such errors before any final submission.\n\n__Summary__\n\nThis paper demonstrates impressive results using SBP for vision tasks. The quantitative performance are enormous (Tables 1 and 2, 25-40% memory reduction with less than 1% accuracy loss) on two commonly used backbones for typical vision tasks. The work also demonstrates a simple implementation, which suggests that the concept can become a effective widespread idea. The in depth analysis of design choices also provides future researchers with a strong starting point for further study. While I am admittedly not well read in this line of research, this paper does not appear to have many flaws, generally. It is well-structured and easy to understand, and it provides strong empirical support for the benefits of SBP. Moreover, it asks the right questions about SBP and provides solid initial answers. I think this work is a welcomed contribution to NeurIPS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4Db1TXjbOyH", "reviewer": "Reviewer_ySTN", "summary": "A study of SBP, applied to image classification and object detection, is provided. A significant amount of memory is saved during training while the degradation of accuracy is relatively small. Derivations and experimental results are comprehensive and they validate the author's main points. ", "questions": "As I pointed out in the weaknesses part, I want to know more about novelty (compared to CVPR `22 paper) and technical contribution (what was the most difficult technical problem?). ", "limitations": "Limitations are adequately addressed. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of the proposed method is its practicality. It saves memory with almost no harm to the final models of interest. There is no reason someone faced with memory limitation not try this simple approach. Derivations related to stochastic backprop. are comprehensive and most of them look sound but I am not fully confident. \n\nI have checked the details of experimental results and ablation studies, and as far as I understand they are convincing. For example, Figure 3 (grid mask vs random mask) experiment matches the theory from the derivation. \n\nThree main weaknesses are novelty, technical contribution, and writing. First, the concept of stochastic backprop. is introduced in the original CVPR paper and the proposed method in the paper is incremental to the original paper. Second, it is not easy to find a non-trivial technical challenge introduced, tackled, and solved by the paper. Third, writing is difficult to follow. I have listed some of the parts that can be improved: \nL77 fully connection layer -> fully-connected layer\nL84 caching of feature maps dominants the memory usage. -> did not understand \nL311 slightly loss -> slight loss  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept