Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies activation compression to reduce the memory consumption when training models with stochastic gradient descent. The goal is to establish an activation compression framework with convergence guarantees, while eliminate the tuning overhead from conventional search based method for compression precision selection. The contributions of this paper come in three folds:

1. The authors first established a connection between the activation compression error and the convergence bound (i.e. the upper bound on training loss).

2. Built on the above theory, the authors formulate the activation compression precision selection problem as a constrained optimization problem, which can accommodate typical DNN layers such as linear layers, convolution layers and normalization layers.

3. Empirically, the authors combine this optimization-based activation precision selection method with various compression methods (with bounded errors at each compression precision). They show that it could attain matching accuracy (minimal acc degradation compared to no activation compression) compared to conventional grid search method for activation precision without the grid search (i.e. hyperparameter tuning) overhead.
main_review: Originality:

Previous activation compression works typically use time-consuming hyperparameter tuning methods to select compression precision. This paper studies the connection between activation error and convergence, and explores a novel approach to automatically select activation compression precision. I think the theoretical analysis, the methods brings new thoughts and methodology to the area.



Quality:

My major concern/question on the paper is about the claim that the proposed activation precision selection approach *does not need any hyperparameter tuning* to achieve minimal accuracy loss.  The basis of claiming no tuning is to uniformly use a fixed e^2 value (e^2 = 1 is used in the paper) across models and datasets. Theoretically, the limit of the loss bound in equation (13) can increase by 50% if e^2 = 1 is introduced due to activation compression, which could trigger significant loss / accuracy degradation; such a case can be instantiated using a quadratic function.  Totally understand e^2 = 1 could potentially be a good hands down option for a broad range of SOTA models. But to validate this, it needs more model/datasets (such as transformer based models for vision and NLP) using e^2=1 for activation precision while the current experiments only cover convolutional models.

Besides the major above question, it would be great if the authors can resolve the following questions in the rebuttal and paper draft:

1. Theorem 1 (line 141) assumes the expectation of the gradient error norm is zero. Does this means the theorem assumes there is no gradient error at all (because the assumption is only satisfied when the activation compression does not trigger any gradient change)? If this is the case, the assumption is too unrealistic. Also from the proof in the appendix A, it is unclear how this assumption is leveraged in the proof.

2. For the constrained problem in equation (7), the presentation needs to be improved to clearly define the problem. From the proofs in Appendix B, I believe equation (7) is actually referring to \Delta X* = \min_{b} B(\Delta X), s.t. D(\Delta X) < e^2V^2 / 2. The current formulation is not rigorously reflecting the optimization problem.

3.  (Not a major concern) In table 3, it seems the grid searched QuantZ is giving 0.7% higher accuracy on ImageNet than the no compression baseline. Is it because of random seed variation? Or is it because the no compression baseline is not well tuned to optimize validation accuracy (The typical ResNet 50 ImageNet validation top 1 accuracy is around or around 75% to 80% [1] while the baseline results in the paper is giving 72-73% top 1 accuracy)?


Clarity: I think the paper explain intuitions and general ideas very well. But technical clarity (such as terminology definition and etc.) should be improved in several places.

1. In equation 3, is \Delta x_{nchw} a function of compression precision b. If this is the case, it is unclear how this function (defined on discretized supports) is used in the continuous constrained problem in equation (7).

2. In line 161, the range of \Delta X is not properly defined if \Delta X is a multi dimensional vector. My guess is you mean a ball centered at 0 with radius \Delta X*.

3. In line 252, there is a claim on "all networks examined have loss changes below 2%". It needs a bit clarification in the text because the loss change at -log|e| is definitely larger than 2% given loss changes from ~0.015 to ~0.025.


Significance: If the proposed methods can show strong accuracy in broader models / tasks (such as NLP or vision transformers) still with a uniform value for hyperparameter e^2, the impact can be significant to eliminate / dramatically reduce the tuning overhead on compression precision selection.


I am willing to raise the score if the above concerns are resolved.


Reference

[1] Deep Residual Learning for Image Recognition. He et. al.

limitations_and_societal_impact: I think the attached checklist answers the questions adequately. The overhead / limitation of the method is discussed in Section 6.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a theoretical convergence upper bound for DNN training with compressed activation. Using this bound, the compression rate is controlled over training epochs by computing the quantization error. Moreover, to reduce the complexity of computing the activation’s quantization error, the paper uses statistics of this error such as mean and max instead of computing the complex norm. The efficiency of this new approach (AC-GC) is evaluated in various benchmarks including the ImageNet dataset and compared with previous approaches such as WAGE, GIST, and JEPG-ACT. The result showed that the AC-GC compressed the activation more than other approaches while maintaining accuracy compared to FP32. 

main_review: Strength:

1- The idea behind the constraint (activation precision) convergence rate under the Polyak-Łojasiewicz Condition is novel and interesting. The author uses a common Lagrange multiplier approach to optimize the constraint.

2- The paper is well organized. The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.

3- The proposed approach is evaluated in large and variant benchmarks, which mean the proposed approach can be deployed for different models 

Weakness:

1-  The author did not compare the new approach with previous papers such as [1] and [2], where the theatrical convergence rate is proposed for low-precision DNN parameters and activations. It would be suggested the comparison with these works is discussed in the paper.

2-  The author needs to add more clarity on the motivation behind storing the activation ( not cited two frameworks ) and provide the assumptions to satisfy the Polyak-Łojasiewicz Condition in the paper. 
3- Is this the convergence rate depends on the dimension of activation? How does the dimension of activation affect the compression rate? 

4- The last sentence of the abstract is a bold statement, and I would suggest changing it. The AC-GC has shown valuable results for the experiments presented in the paper, and it is hard to conclude that the approach works for any model. 

5-  It would be suggested to compute and report the approximation error using the max and mean of activation error instead of an accurate norm.

References: 

[1] Li, Z., & De Sa, C. (2018). Dimension-free bounds for low-precision training (https://papers.nips.cc/paper/2019/hash/d4cd91e80f36f8f3103617ded9128560-Abstract.html)

[2] Li, Hao, et al. "Training quantized nets: A deeper understanding." Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017.

Post Rebuttal ======
Thanks for the clarification. I think the response resolved my comments on the clarity and quality part. I raised my score from 6 to 7. I encourage authors to add their responses to my comments to the various parts of the papers to improve the quality of the paper. I also suggest that emphasize the weak assumption of convexity in the abstract or introduction.


limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors bound the increase in loss in the case that activations used for backpropagation have errors. By doing so, they can set a fixed acceptable level of increase in the loss, and solve for the error level that can be tolerated for each activation used in backpropagation (and therefore deduce a level of compression). This analysis is relatively agnostic to the precise implementation of the compression. They evaluate their approach on a variety of datasets, including relatively large datasets such as ImageNet, demonstrating that their method can achieve high compression levels with relatively little tuning.
main_review: ### Detailed Feedback

Abstract: I think the authors did a good job here. Motivated well, and clear description.

Intro: The motivation is good, and well explained (as with the abstract). I think that fig1 is also nicely presented.

Preliminaries:
- Throughout the paper, you seem to use $\phi$ for weights, which is a little unusual. Might be better to use $\theta$ to be more consistent with other works?
- A purely stylistic observation: It might be good to move the related work here, and motivate why what you're doing is different to that. This is my preference (but I am not going to consider this with regard to scoring)
- L91: There's a comment here about compression not slowing down optimization too much; I note that this claim is made later too. But what evidence is there for this assertion?

S3:
- This entire section is written fairly well, with nice figures and useful intuition provided. But the one thing I would note is that I am _not_ an expert on this area, and as such I find it difficult to provide much technical criticism -- it all seems reasonable to me. Nothing stands out to me as being obviously questionable.

S4:
- This section was more difficult to follow than the last one -- it is very dense. If it's possible to provide any figures it would be very appreciated by readers.
- It might be a good idea to discuss information theory and its relation to compression, along with lagrange multipliers in the preliminaries?
- L199: the discussion of max pooling has been forgotten (although it is very simple).

S5:
- L210: could expand on how you collected this data?

S6
- The caption for fig5 is not useful, and you have to search for the interpretation in the text. You may want to push the explanation to the caption.
- The results in table 2 are great. It would be appreciated if you could add some results for training time though :-)
- The results for table 3 are less impressive, unfortunately (although they have been achieved in far less time -- you undersell that in the table). Also, what about AutoCuSZ and AutoJPEG? The results don't seem particularly impressive when compared with [1]
- (I don't expect this for rebuttal) It'd be really great if the authors could look at providing some experiments using a transformer. This might be a fair bit of work -- such as deriving bounds for layer norm -- but would make this work more complete.
- L287: can you provide numbers for how long compression takes?

S7:
- It would be a good idea to cite ActNN [1] that was recently published at ICML 21.

### Overall Thoughts

I quite like this paper overall, as there is a (seemingly) sensible theoretical analysis, and it is generic to the particular choice of compression. The problem area is extremely important, and this work will be of wide interest. I think the only real downside is that the results obtained are not favorable when compared to [1] (at least on ImageNet), although it is not possible to make a direct comparison. The authors may be able to deal with this by swapping in a different compression backend on the ImageNet experiments. I would ask the authors to address some of the concerns above -- especially the ones regarding overheads. I am sympathetic that the implementation is likely to be far from optimal, but it is important information to provide.

==== Post rebuttal

I'm happy with the author's response, and I have increased my score accordingly :-)

[1] ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
limitations_and_societal_impact: I think it is fine as it is.
review_title: null

Meta-review: While there was a discrepancy between the overall scores initially, the authors' rebuttal letter clarified several concerns of the reviewers, which resulted in all the scores being positive. Hence, I recommend an acceptance. Please implement all the changes suggested by the reviewers in the camera-ready version.
Decision: Accept (Poster)