Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes efficient packing methods for training sequences of BERT, such that the 50% of the padding tokens in the Wikipedia dataset is avoided to speed up the training. These methods include shortest-pack-first histogram-packing (SPFHP) and non-negative least-squares histogram-packing (NNLSHP) algorithms, which are shown to be straightforward to implement and have little impact on the performance. Empirical studies show that a near 2x speedup over the vanilla BERT training is achieved by the proposed methods. 
main_review: Strength:

The idea is simple and easy to understand.


Weakness

1. The authors revealed their identity (affiliation) in the code snippets of appendix. It might violate the double-blind rule of ICLR.

2. The novelty of this paper is limited. Packing is not a new idea and it has been widely used in the official tensor2tensor library and achieved good results: https://github.com/tensorflow/tensor2tensor/blob/3f12173b19c1bad2a7c37eb390f3ad46baee0c19/tensor2tensor/data_generators/ops/pack_sequences_ops.cc. So this can be a useful trick but the contribution might be not significant enough to publish at ICLR.

3. The scenario discussed in this paper is too restricted. For example, 1) it only discuss the wikipedia dataset (that’s how the number 50% comes), but there are a lot more datasets; 2) it only works for BERT training, but there are quite a few other important tasks, such as language modeling (GPT-3). All the numbers reported in this paper are based on this setting, making its generalization capability questionable. 

4. The experiment section only shows the training loss of pretraining, but never talks about the downstream fine-tuning. Then how do you conclude that the performance is little affected? After all, the accuracies of downstream applications is the final metric.

5. The paper is a bit hard to read for the following reasons: 1) The contents are not self-contained in the main text — quite a few important contents are deferred to appendix that one cannot easily follow the ideas in the main text; 2) The paragraphs are usually lengthy and verbose —  they can be as long as 30 lines! 3) There are quite a few typos, e.g. “For achieve this”, “¡CLS¿” etc.
summary_of_the_review: The paper violates the double-blind rule (in the appendix). It lacks novelty, only works on a restricted setting and is a bit hard to read.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two packing algorithms for bert pretraining, shortest-pack-first histogram-packing (SPFHP) and non-negative least-squares histogram-packing (NNLSHP). 2* speed was achieved under datasets such as Wikipedia for bert-large training.These packing algorithms packed Wikipedia’s 16M sequences in 0.02s (SPFHP). Packing depths of from 1 to 16 were testified. This paper also has 20 pages near appendix telling about packing algorithms, packedBERT of model changes and hypermeter adjusting, and detailed experiments such as bin-packing algorithm comparison, scaling analysis and technical background on packing with some core codes attached as well.
main_review: Generally, simple packing ideas are frequently used such as sorting sequences by their lengths and then try to “cluster” near length sequences into one mini-batch. This paper further goes deeper into two interesting packing algorithms and experimented on IPU hardware with Wikipedia style datasets and BERT-large models.

Detailed questions and comments:
1.	Possibly trying larger models such as GPT3 and later will be an important directly to testify your ideas: bert-large is not that big enough and saving 50% training time will be more meaningful for super-large models such as GPT3.
2.	Wikipedia is not generally used for domain specific bert such as healthcare and finance domains, in which pubmed and financial news are frequently used. So, the findings of specific datasets are quite limited and hard to say it is a general finding.
3.	Not find any baseline packing algorithms here. Any simple heuristics used in the baselines? Such as sorting by length and then perform the training with as fewer packing symbols as possible in minibatch.
4.	From reading the paper, I think the two algorithms are not limited to BERT (its MLM and NSP pretraining tasks) and really willing to see other types of pretraining tasks, such as T5, GPT style’s results. These will make an evaluation of these two algorithms’ novelty and generalization ability to be with richer evidences. 
5.	Also wonder if the packing algorithms can be applied to ASR tasks such as packing audios as well. It looks that not only applicable to textual sequences.
6.	IPU is less available to most readers yet, so possibly also mentions details in GPU/TPU are preferred as well.

summary_of_the_review: 
Strong:
1.	Two packing algorithms for bert-large and related pretraining models;
2.	2* speeding up without losing accuracy for bert-large pretraining under datasets such as Wikipedia;
3.	Rich analysis of technical backgrounds, experiments, and code details.
Weak:
1.	Limited to Wikipedia and bert-large is a small scope considering that there are rich pretraining tasks, rich domain and rich datasets of different languages;
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper profiles the training data of BERT and finds out opportunities to reduce padding thus saving computation. By packing multiple sequences into one fixed length sequence (two packing algorithms were used), the author shows that a BERT training speed-up of 2x can be achieved without loss of quality. The author also studies how to correspondingly change the modeling configuration and optimizer configurations to compensate the effective batch size increase due to the packing.
main_review: The motivation makes great sense and the overall structure of the paper is very clear (though I did not read anything in the appendix, i.e., from page 13-37).

Usually dynamic sequence length is not supported in GPU or TPU training libraries. The most common way is to pad a fixed sequence length (e.g., 512) and provide a mask in the input so the padded data can be masked out during computation. The advantage of this approach is that this is straightforward to implement.  However, it loses efficiency by padding all the data into the max sequence length.

The proposed work is to reduce padding via packing multiple sequences into one fixed length. I am not familiar with publications on this front, but I am aware of an existing implementation of similar methodology in `tensor2tensor` library, there is a `pack_dataset` method (https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/generator_utils.py#L671), which is designed following a very similar idea. This implementation in tensor2tensor is not profiling the dataset on sequence length histogram because it is receiving the data in a streaming fashion. So I agree that this paper is definitely doing something related but differently by considering the entire problem as bin-packing so as to look for approximate optimal global solution.

General Significance: The main significance is the two packing algorithms being proposed and experimented in BERT pre-training. Multiple sequences can be packed into one fixed length. However, I see the impact is limited because:
1) this strategy is data dependent. Though there could be many datasets that have this property, there are also other datasets that could follow a different sequence length distribution in which case the proposed methodology won't achieve similar amount of speedup.
2) because the packing algorithm enforces a specific order (by selecting which sequence to add to which pack), this affects the i.i.d assumption. Also this assumption is actually important in many data preprocessing pipelines we use for BERT encoder. If we have to use one model (packed) for training pre-trained BERT model, then using another model (non-packed dataset and modeling code) for training downstream models this is really not ideal in development. 
3) this workaround (to reduce padding) creates complexity in the user code, both in the input pipelines and the modeling changes. And more importantly, as the author already pointed out, this workaround also increases the effective batch size which might need a complete re-tuning of hyper-parameters of the model.

Novelty: A baseline implementation of multi-sequence-packing already exists in a public library (tensor2tensor). The author might want to clarify what are the extensions on top of tensor2tensor. As I pointed above, tensor2tensor is consuming data in a streaming fashion, and  this paper is considering the entire problem as bin-packing so as to look for approximate optimal global solution. Could you quantify the difference between using tensor2tensor library vs your proposed two bin-packing algorithms regarding speedup vs quality?

Technical quality: The overall technical quality is limited because of the limitation mentioned in "significance" and "novelty".

Clarity: The paper is well-structured, with clear relationship between each section. The ideas are clearly conveyed with a good writeup.

Minor issue(s):
* Formatting problem in section 2 before and after CLS and SEP token. (¡CLS¿, ¡SEP¿ , which I guess they should be <.>) Hopefully this is not a problem from my browser viewing pdf file.
summary_of_the_review: marginally below the acceptance threshold
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a simple way to pack sentences together for efficient training. In order to recover the accuracy/performance, several model arch and optimizer hyper-parameter changes are introduced.
main_review: Strongness:
(1) The method is easy to understand and easy to incorporate into the existing training pipeline. 
(2) The performance gain looks great. 
(3) Have the ablation study to show the effectiveness of the model arch changes.
(4) Easy to understand 

Weakness:
(1) Although pre-training is an important part of BERT, did you test the different pre-trained models on downstream tasks?
(2) I did not find the GitHub link in Appendix. The text code is hard to do the reproduction.

Questions:
(1) In section 3.3, the authors mentioned the learning rate is not adjusted since the gradient is accumulated by averaging. However, as far as I know, LAMB also uses the averaging of the gradient but it still needs to scale the learning rate. Can you explain more?
(2) Do you have a comparison of the two proposed packing algorithms?
summary_of_the_review: Please see above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposed to pack sequences instead of padding to reach the max sequence length for each sample for BERT pretraining. To achieve similar optimization results with the original training process, the proposed method modified positional embedding, additional attention masks and other optimization hyper-parameters. On the Wikipedia dataset, the proposed method can achieve 2x speedup while achieving similar training loss. 

main_review: Most part of the paper is clearly presented. The proposed packing strategy should be novel, easy to implement and also effective. However, I have the following concerns and comments. 

1. The paper claimed the packing algorithm doesn't loss accuracy compared with the original algorithm by showing the training loss and training accuracy. However, since the sequences are not i.i.d. in the packing algorithm, I am wondering if the generalization performance on downstream tasks (such as GLUE) will be affected. 
2. As mentioned in the paper, BERT is pre-trained in two phases, where the first phase uses sequence length 128 for 900K steps and the
second phase uses sequence length 512 for 100K steps. Therefore, if compared with this approach, the speedup from the proposed method is probably only 1.2x instead of 2x. 
3. Why do we need a limit for pack depth? 
4. Other than the original training approach, the paper only compares with the un-packing methods in multi-accelerator case. It lacks comparison with baselines such as grouping samples by size before batching as mentioned in Page 2. 

Minor:
1. `s_m` in Sec. 3 is not clearly defined. 
2. The abstract mentioned 0.02 seconds overhead time for SPFHP and 28.4 seconds for NNLSHP. In the main text, I only see 0.03 seconds for SPFHP and no mentioning about NNLSHP. 
summary_of_the_review: Overall, the paper proposed a simple and effective method for speeding up BERT training especially on the datasets with a large variance of sequence length. However, due to lack of comparison with other similar baseline methods, I think the current version is slightly below the acceptance threshold.
review_title: null

Meta-review: This paper proposes to re-organize the training data in such a way that padding can be avoided. The novelty is somewhat limited and the results are what one would expect - a nice speed-up of 2x but nothing really game-changing. While the reviewer scores straddle the decision boundary, nobody is very strongly supportive of acceptance and the positive reviews actually have lower confidence.
Decision: Reject