Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes three simple yet effective optimizations to speedup the pre-training of large language models by reducing the memory footprint. The resulting approach is called Tempo. Tempo is able to reduce the memory requirement for the pre-training of language models such as BERT, RoBERTa, and GPT-2 significantly. In addition, the authors demonstrate that memory savings can be translated into increased throughput.
strengths_and_weaknesses: Strengths: The three proposed optimizations for reducing the memory footprint are simple and straightforward. I am convinced that the proposed ideas can help reducing the memory requirement for the pre-training of language models significantly. The paper is well written and easy to understand. The experimental evaluation is solid, as the authors demonstrate not only the results of reduced memory footprint, but also the improvement in throughput.

Weaknesses: My main concern with this paper is whether the proposed optimizations are considered as new contributions. In particular, the first two optimizations propose to turn the activation and normalization functions into in place functions, which helps to reduce the required memory for saving the intermediate tensor for backward propagation. The idea of using in-place function to reduce memory footprint is not new. Although not directly applicable to the Transformer architecture, some previous work and in-place ABNs have proposed to perform activation and batch normalization in-place to reduce memory footprint. In addition, attempts have also been made to add support for in situ GELU in pytorch (https://github.com/pytorch/pytorch/pull/74629).
questions: 1. Could you elaborate on how in-place GELU and layer normalization are different from in-place ReLU and batch normalization? Are there any new treatments involved in converting these two operations in-place?
2. Have you evaluated the pre-training performance on other libraries with complier optimization enabled (e.g., Pytorch with JIT or Tensorflow with XLA enabled)? Is it possible that some compiler optimizations, such as operation/kernel fusion, have helped to reduce memory footprint and improve training throughput? If so, are the proposed optimizations complementary to the existing compiler optimizations? For example, Pytorch performance optimization guide [1] uses GELU as an example for kernel fusion as follows:
```
@torch.jit.script
def fused_gelu(x):
    return x * 0.5 * (1.0 + torch.erf(x / 1.41421))
```

[1] https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a solution named Tempo that includes three core techniques, in-place GELU, in-place LayerNorm and sub-layer dropout recomputation, to reduce the memory footprint of training transformer-based models. Given the reduction of memory usage, the same GPU is able to conduct gradient-based optimization with larger mini-batches; therefore, the training throughput is improved. 
strengths_and_weaknesses: Strengths:

- The out of memory issue is a serious problem when training transformer-based models, especially giant foundation models. 

- The analysis of the memory consumption of transformer models during the training procedure is informative. 

- The performance boost over the state-of-the-art baseline and checkpoint approach is significant.  

Weaknesses:

- The novelty inherited from the proposed methods is relatively limited. These optimizations seem straightforward from the point of view of implementation.   

- There is a lack of theoretical discussion about the lossy compression in the Tempo system. 

questions: It seems the experiments are conducted on multi-GPU machines, Is there any parallel schema applied in these benchmarks?

limitations: Not applicable.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an approach to reduce the memory footprint of transformer-based models for faster training without sacrificing accuracy. The proposed approach Tempo uses In-Place GELU, In-Place LayerNorm, Sub-Layer Dropout Re-computation, and softmax engineering optimization. The main advantage is to compute the backward pass as normal while requiring less storage. Evaluation results show 2x improvement in batch size for BERTLARGE pre-training at a sequence length of 512 while increasing the training throughput by up to 16%.
strengths_and_weaknesses: Strengths:
+ Clever techniques to enable recomputing input from output in GELU and LayerNorm with minimal overhead
+ Good observation on dropout that avoids recomputing of unnecessary input
+ Well organized paper and clear writing. Supplemental materials are helpful

Weaknesses:
1. Tempo requires more memory than checkpointing (although expected, this is still a fact/weakness)
2. Optimizations in Tempo are limited to transformer-based models while some other alternatives are not (e.g., [48][28][26])
3. Evaluation-related:
- In Section 2.1, a sequence length of 512 is used for showing that feature maps account for 56% of the encoder layer activation memory, but a sequence length of 128 is used for showing that the memory function takes 17%. It would be more convincing if both results were given for the same sequence length. 
- In ablation studies, it would be great to show the impact on accuracy and memory footprint when increasing the number of hidden layers. 

Minor issues and suggestions:
- Figures 3 and 6 are missing the main caption.
- GPU names can be added as a row in Table 2.

questions: If there are any data for the above issue 3, please briefly summarize.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes optimizations that reduces memory usage when training transformer models. Reducing memory usage for training models allows using larger batch size which improves training throughput.
Previous methods that reduce memory footprint in training are: checkpointing, offloading and compression. 
The changes presented in the paper are: in-place GELU layer, in-place Layer Normalization and Dropout.
The proposed method was evaluated on BERT, RoBERTa, GPT2 model. They show improved training throughput and ability to use larger batch sizes when training BERT with their method.



strengths_and_weaknesses: Pros:
The paper demonstrates techniques to improve training of relevant transformer models using in-place layer compute. The paper tackles an important problem

Overall the paper is clear. But improvements on diagrams could improve clarity of the paper

Cons:
It is dubious that this is the first paper to analyze transformer training memory footprint

in-place compute is fairly known optimization method for training thus the method lacks originality 

More experiments and comparison could be added to strengthen the paper



questions: Some suggestions and questions that could improve the paper are:

Adding a comparison to check checkpoint+tempo to demonstrate that tempo can be used with other techniques 

Adding a comparison with compression techniques to demonstrate the trade-off of accuracy loss with more efficient training from tempo 

Try on more models that uses GELU to demonstrate that this method isn't only suited for transformers

Having a list of layers from various models (CNN, RNN, GANs, etc) that could be done in-place (that arent done yet) to improve memory footprint in training would be a value study

Table2 should also list and highlight throughput achieved and have total memory usage 

Figure3 caption format seems confusing



limitations: The paper doesn't discuss limitations or future work.
Discussion how improving training on transformer models can lead to lower power consumption in servers systems.
And add future work on what other layers could be done in place.
review_title: null

Meta-review: metareview: The paper is well organized, the writing is clear, the proposed method shows good performance and the empirical results are convincing. The main concerns that came up in the reviews concerned the novelty of the approach because similar memory-footprint optimization techniques have been used in other contexts. The authors provided valuable clarifications in the response regarding the challenges that are particular to GELU and explained what puts their approach apart from existing uses of in-place operations. 

Overall, the rebuttal clarified most of the concerns, the sentiment among the reviewers is positive and I think the contribution is sufficiently novel and targets a very important use case (transformer models). Thus, I recommend acceptance, but I would recommend the authors to be more explicit about the challenges that are particular to their method. Furthermore, for the camera ready the authors should use the additional page to carefully take into account the feedback of the reviewers, make the diagrams more readable, and add clarifications where needed to anticipate the questions that came up. 
confidence: Certain
Decision: Accept