Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: * This paper considers the scenario that one want to not only deploy pre-trained models but also continue train/finetune these models on local devices. It proposes a novel method called Black Razor that reduces the training/finetuning memory of neural networks, by pruning/compressing the activation during back-propagation. BlackRazor is able to achieve 96% sparsity, saving 9.2x memory without losing accuracy. The paper also conducts a theoretical analysis of BlackRazor's convergence. 
strengths_and_weaknesses: Strength:
* The proposed method seems to have great results: successfully reducing much training memory without much loss of accuracy.
* The proposed method seems to be simple and generalizable. 

Weakness:
* One weakness of this paper is the novelty of the proposed method. The method is to conduct magnitude pruning on activations for back-propagation.  This seems to be more like an engineering trick rather than a science / learning paper.
* This paper will be more solid if it’s validated on more downstream tasks. 
* The theoretical results are a bit confusing: for example, in algorithm 1, it prunes activations during forward pass, yet in Lemma 1, it states that “so pruning the activation during backward does not change its gradient”.

questions: * Typo in Figure 2: “Storeage”
* What is the memory difference between activations in forward pass and backward pass? How can pruning activations in backpropagation reduce memory by more than 2x?
* Have you tried on language transformers?
limitations: N'/'A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a simple and memory-efficient method called Back Razor that prunes activations only for back-propagation, presents a theoretical analysis that the convergence rate of Back Razor can be similar to that of SGD, and shows the effectiveness of Back Razor on CNN and Vision Transformer.
strengths_and_weaknesses: Strengths

(1) The paper is well structured and the main idea is clearly explained.

(2) The extensive experiments demonstrate the efficacy of Back Razor to some extent.

Weaknesses

(1) Proof-reading is required due to some errata and awkward expressions.

(2) More experiments on large models such as ViT-L/16 seem to be needed to verify whether Back Razor is really working well on large pre-trained models.
questions: In Figure 3, it would be better if the memory footprint of training as well as accuracy is plotted when comparing pruning both forward and backward with pruning only backward.
limitations: As mentioned in the paper, the authors plan to apply Back Razor to several downstream tasks.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: To reduce memory use of training a transfer learning model, the paper proposes to prune activations stored for back-propagation, instead of traditionally pruning activations stored for both forward-propagation and back-propagation. Experiments on CNN and ViT show that the proposed method achieves comparable accuracy with more aggressive memory reduction compared to competitive methods TinyTL, Bitfit, and Mesa.
strengths_and_weaknesses: Strengths.
1) In Section 1 and Section 3.1, the paper provides clear explanations on why the paper focuses on compressing activations stored for back-propagation. 
2) In Section 3.2, the proposed techniques are clearly written and reasonable. Theoretical analysis on convergence is also provided in Section 3.3.
3) Experiments on CNN and ViT are convincing. For ViT, the paper provides comparison between theoretical memory use and practical memory use in Table 2 and Table 3, respectively. 

Weaknesses.
1) I think the authors should provide peak memory use in the experiments. Since the method prunes activations stored for back-propagation, I agree with the authors that the theoretical memory use and averaged memory use are both decreased. However, as the activations are not sparsified during forward-propagation, I am not sure whether peak memory use is decreased compared to previous methods. If the peak memory use is not decreased, the proposed method actually requires computing devices with the same memory capacity as previous methods.
2) A potential problem of the method is discrepancy between theoretical memory use and practical memory use, as shown in Table 2 and Table 3. The problem is incurred by the adopted unstructural sparsification strategy, pruning the smallest magnitude activations, as introduced in Section 3.2. According to memory working mechanism, memory is accessed in column-wise or row-wise manner. A column/row is accessed even if there is only one non-zero element. Therefore, unstructural sparsification cannot effectively practical memory use.
3) I think the traditional method gradient checkpointing is very related to this paper. Is it possible to qualitatively or quantitatively compared with this method in the experiments?
questions: 1) Is it possible to provide peak memory use during training?
2) Have you tried structural sparsification strategies?
3) Is it possible to qualitatively or quantitatively compared with the traditional method gradient checkpointing in the experiments?

limitations: As I mentioned above, a major limitation is the adopted unstructural sparsification strategy which incurs discrepancy between theoretical memory use and practical memory use.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Back Razor is a memory-efficient algorithm for transfer learning on edge devices, which works by using sparse activations during the backward pass of neural network training. The paper supports its claims with experimental validation on four image classification datasets and two architectures (CNN and ViT), as well as some theoretical convergence results.

strengths_and_weaknesses: ## Strengths

The proposed method is elegant, simple to understand, and easy to implement. If the early experimental results hold up, this method can substantially help with neural network training on edge devices. Experimental evaluation looks quite promising.

The placement of the algorithm in the context of transfer learning on edge devices is appropriate and compelling. Overall, the paper flows well.

## Weaknesses

### [major issue] Experiments [This was addressed to my satisfaction during the discussion period]
The authors only show experiments for vision tasks using two architectures (a CNN optimized for mobile performance and ViT). Besides the basic baselines of full finetuning / FC finetuning / FC+Norm, the paper only benchmarks on one other method in the case of CNNs and two methods in the case of ViTs; it is not clear from the paper that these methods are SOTA for each of the tasks considered. In almost all cases, the experiments are presumably run only once, as generally no confidence intervals were given.

The biggest concern is that the experimental coverage is insufficient to draw strong conclusions. In particular, TinyTL, which is used as a benchmark, provides results for nine datasets of which BackRazor is only tested on four for CNN, and three for ViT. There is also no investigation of how well the method performs with different batch sizes and optimizers, or for other categories of tasks, such as segmentation or language understanding. In the case of CNNs, it would also be useful to see a comparison to other methods that use reduced precision (i.e. quantization); for ViT this was provided.

Finally, it would be useful to see the actual on-device memory usage in the case of the CNN architecture.

### [minor issue] Theory [This was addressed to my satisfaction during the discussion period]
The theory gives a convergence result for a linear network, and then some argumentation is made to extend this result to CNNs. However, Theorem 1 states that the algorithm converges to a neighborhood of a stationary point, and the argument that this neighborhood can be reduced by increasing the batch size does not work in this memory-constrained setting. Thus, it is not clear how well the theory applies, even though it is also experimentally verified that the convergence rates are similar, hence this is overall a minor issue.

### [moderate issue] Writing [This was addressed to my satisfaction during the discussion period]
While the flow of the paper is good, important details are often not made as clear as they could be, or are hard to find. For example, the TopK operator used to prune the activations should be clearly indicated in the algorithm. The theoretical assumptions, and the grounds for them are not explained, leaving the reader to consult another paper for context. Finally, the transition in line 129 of moving from an arbitrary activation $z_i$ to the final activation $z_l$ is somewhat confusing, since none of the rest of the paper focuses on the last layer specifically.

Some sentences make overly broad claims, for instance the very first statement of the abstract calls transfer learning the unqualified _de facto_ choice, which is not generally true. Likewise, in line 101, the claim “transfer learning always happens on personal computing devices” is unsubstantiated.

The proof in the appendix contains numerous typos in the math notation, for instance:
In line 4, $ \tilde{z}’_{k} = \text{diag}(m)(\tilde{z}$ should have been  $ \tilde{z}’_{k} = \text{diag}(m)(\tilde{z}_{k}$ 	 
In line 6, $\tilde{g}_k$ should have been $\tilde{g}_k$
In the last formula of line 5, $\beta_i$ should have been the boldface $\beta$. Also, some of the gradients in that equation are incorrectly marked as the full gradient $g$ rather than $\tilde{g}$.
The $\kappa$s in line 13 should have been $K_a$.	

Other typos are likewise numerous, although they generally do not adversely affect understanding:

Line 40 of -> or
Line 99 - missing “the” before “de facto”
Line 102 constrain -> constraints
Line 124 by -> of
Line 130 FLOPs should be capitalized
Line 157 actions -> activations
Line 166 value -> values
Line 179 weight -> weights
Line 201 Assumption -> assumptions
Line 231 the word “based” shouldn’t be there
Line 267 Missing verb after “even”
Line 288 employ -> employs
Line 313 illustrate -> illustrates
Line 329 backpropogation -> backpropagation

questions: The main suggestion to the authors is to conduct a substantially more thorough experimental validation of the method.
limitations: The authors correctly pointed out that the paper only looks at image classification tasks, and also addressed the limitations of their theory’s application to ViTs.
review_title: null

Meta-review: metareview: This paper focuses on pruning the backpropogation activation to reduce the memory footprint in transfer learning. The paper is well structured and the method is simple to understand. All the reviewers acknowledge that the experimental results are convincing. Overall, the meta-reviewer recommends acceptance of the paper.
confidence: Certain
Decision: Accept