Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Training deep learning models is becoming increasingly challenging: there is simply not enough memory to store the activations generated by the forward pass until the corresponding gradients are computed. The paper proposes to split each activations in two tensors, $o$ and $a$. The $o$ tensor can be discarded quickly, while the $a$ tensor needs to be preserved in memory to compute the gradients.  The paper develops a theoretical criteria, the learning indicator, under which this partitioning scheme is guaranteed to succeed (subject to some reasonable assumptions). Based on this criteria, the authors design two schemes to partition the activations, auxiliary residual activation (ARA) and auxiliary sign activations (ASA), and empirically demonstrate their effectiveness on a comprehensive set of models.

strength_and_weaknesses: Strengths:
* The approach is compelling and completely novel as far as I know. It is effective on its own and can be combined with other existing approaches to save more memory.
* The learning indicator criteria is sound under reasonable assumptions, and used effectively to develop two schemes, ARA and ASA.
* The empirical evaluation of the scheme is comprehensive and demonstrates that the approach is applicable to a wide range of DNN architecture.

Weaknesses:
* There is a minor error in Theorem 1: $\eta$ must be strictly positive for the loss function to converge
* The convexity assumption under which the learning indicator criteria was developed often does not hold in many cases. That said, most of our theoretical understanding of deep learning suffers from the same limitation, and empirical proof is used to prove that this approach works in practice.
* Table 1 doesn't list the training speed and memory reduction like the other tables do. It would have been great to be able to compare ASA and ARA side by side. I am especially curious to see whether there is a correlation between training speed and mean value of the learning indicator.
* In table 2, the speed and accuracy of ARA is compared against that of GCP and ActNN. The comparison would be more fair if there was a way to decrease the compression rate achieved by GCP and ActNN to be in line with these of ARA. The same applies to table 3.
* Table 4 and 5 propose to use the memory savings to increase the batch size, but this isn't very effective with less than 5% savings in the best case. I wonder if the authors could instead try to increase the width/depth the the neural networks to improve the accuracy of the predictions instead, which I suspect is what most people will use the memory savings for.
clarity,_quality,_novelty_and_reproducibility: The paper is well written, and easy to follow. It proposes a completely novel approach that makes it possible to compress the activations by 1.1x to 1.3x depending on the scenario with no noticeable impact on accuracy and negligible impact on training time, The authors have released the code needed to reproduce their results, though I haven't had a chance to use it. 

summary_of_the_review:  The memory bottleneck is an acute problem that impacts most of the deep learning community This paper provides a novel solution that alleviates this problem. Furthermore, this new approach can be effectively combined with existing solutions such as rematerialization to save more memory.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a training method that saves the memory that the activations take. Instead of using the originally dense activation, the authors propose to use the auxiliary activations during the backward process. During the forward pass, the originally dense activation will still be used so the precision is still high. The authors discuss two ways to construct the auxiliary activation: the residual and the sign. Experiments on image classification and NLP tasks show that the training time and the memory can be saved, while the performance on downstream task is not sacrificed too much. 
strength_and_weaknesses: Strengths:
- Efficient training is an important topic to study, and experiments in this paper have shown the proposed method can indeed achieve practical efficiency on training.
- The presentation of this paper is overall clear and it is easy to follow and understand. 
- The proposed method is straight-forward and does not bring much overhead. 

Weakness:
- The proposed technique is not essentially novel. As authors have already discussed, there are already many works trying to approximate the activations during training. Activation quantization and pruning, although argued by the authors to be slower which I do not believe, is a very simple way to achieve memory reduction. 
- (Minor) The baseline on ImageNet looks weaker compared to the commonly used one, which is around 76% (1% lower approximately). It might not be negligible at the scale of ImageNet.  
- (Minor) $f$ is never assumed to be non-negative (although in common practice we use a non-negative one). It is only twice differentiable and convex.  
clarity,_quality,_novelty_and_reproducibility: The overall presentation and quality is good. The method should be easy to reproduce since it is simple. Novelty somewhat limited, since some similar techniques can be founded. 
summary_of_the_review: This work proposes an effective way to efficiently train models with less memory cost, although the novelty is not outstanding in its current shape.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel memory-efficient training method Auxiliary Activation Learning, which reduces the amount of data to be stored in memory without additional computations. The experimental results illustrate that Auxiliary Activation Learning can significantly reduce the memory cost on computer vision and NLP tasks without training speed reduction.   
strength_and_weaknesses: Strength: 
The paper proposes a novel algorithm to achieve memory-efficient training of neural networks. 
The experiments on ResNet, ViT, MLP-Mixer illustrates the efficiency of the proposed method Auxiliary Activation Learning, which can significantly reduce the memory cost without sacrificing training speed. 

Weakness: 
Some memory-efficient algorithms are proposed in recent years, such as SM3. Some people also try to propose an efficient architecture to achieve memory-efficient training [1]. Maybe you can provide some introduction and experimental results about that.  
NLP models are usually very large and memory-efficient training is also very important for them. I find you have reported the results about BERT-Large. It will be great if you can provide more results about that. 

[1] Yi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler. Efficient Transformers: A Survey. 
clarity,_quality,_novelty_and_reproducibility: NA
summary_of_the_review: NA
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new learning rule, named Auxiliary Activation Learning to reduce memory requirements of deep neural networks
strength_and_weaknesses: Strengths:
- The proposed strategy seems to reduce memory space (11.8x) for ResNet, while not having a big influence in training time.
- Extensive evaluation on training times and memory savings for Vision (ResNet) and NLP architectures (BERT, Transformers).

Weaknesses:
- “For mathematical analysis, we consider a loss function f(x) which is CONVEX, differentiable, and Lipschitz continuous.” Good luck with that. This is a very strong constraint and an unfeasible one when talking about Neural Networks.
- It only compares with GCP (Chen et al., 2016) and ActNN (Chen et al. 2021), but it failed to compare to recent state-of-the-art methods, most notably Momentum ResNets (Sander et al., 2021). And looking at the Momentum ResNets paper it would seem to me that Momentum ResNets provide much less memory requirements and they provide higher accuracy than what is reported in table 1.
- Memory savings are usually reported in parameter requirements, here it just says "reduce memory space (11.8x) for ResNet" and tables 2 and 3 show minimal memory reduction when compared to backpropagation (1.1x to 1.3x), such memory space is when compared to ActNN (much higher overhead).
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and there is a link to code in Github.
Quality got me puzzled when considering the cost function as convex. Is Bishop wrong?
summary_of_the_review: The paper seems to be well written an experimentation is exhaustive. The mathematical proof seems to be flawed as the convexity constraint would practically not apply to any deep neural network. Or maybe I misunderstood the message authors try to convey here.
Experimental evaluation even is extensive misses the comparison with recent state-of-the-art work which seem to be better at reducing memory requirements and are better in accuracy.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an alternative to backpropagation that enables avoiding caching the forward-pass activations for reuse during the backwards pass. In particular, the method introduces an auxiliary activation during the forward pass that is added to each layer's input and is used alone during the backwards pass. The auxiliary activation can be shared across layers and/or can be small relative to the activations that would be stored during backprop. The approach provides some modest improvements in terms of memory savings without sacrificing performance or speed, and can be applied on top of existing methods for saving memory. The paper includes extensive experiments in a variety of settings. Reviewers felt the paper was clear and that the method was novel and well-justified. All reviewers recommended acceptance.
justification_for_why_not_higher_score: There seems to be a good amount of novelty here, and the proposed method does seem effective, but the gains are somewhat minimal. A good paper for exploring new and unusual methods, but probably will not see wide adoption.
justification_for_why_not_lower_score: All reviewers recommended acceptance.
Decision: Accept: poster