Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper examine optimization stabilities encountered during training of GPT-2 models. It demonstrates that as model size, learning rate, and batch sizes grow training dynamics of the model become unstable. To alleviate these issues, the authors propose a curriculum learning approach based on the length of the sequence. They show that this approach stabilizes the learning dynamics and drastically improves the training efficiency. 
main_review: Training massive transformer models in a stable manner has emerged as a challenge for practitioners trying to take advantage of the benefits of model scaling. As such, providing new tools to stabilize the training dynamics is of significant interest to the community. The proposed paper indeed demonstrates significant improvements in training stability and speed. 

This being said, there are a number of issues that the paper does not properly address:

1- How general are the results of the paper? Do we observe similar instability issues for other causal transformers beyond GPT-2? If so, is the proposed curriculum learning approach effective there?

2- The experimental design is not clear: The paper compares only two different hyper-parameter choices. These two hyper parameter choices differ in every aspect (learning rate, batch size, #steps). As such, it is unclear where the instability issue is coming from. Moreover, from the paper, it is unclear to me if these instability issues represent fundamental training limitations that require an involved solution as presented in the paper. It might be the case that better tuning of the hyper-parameters, alongside with more careful clipping of the gradients and activations is just enough to stabilize training. 

summary_of_the_review: While the paper describes an intriguing phenomenon, the training setups the paper uses as baselines are exceedingly weak and underdeveloped. As such, it is not clear to me that the paper is presenting a real improvement to our training setup.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed a length-based curriculum to pretrain GPT models, that linearly increases the input lengths until a maximum value over the first 20-100K steps. Such curriculum are often proposed and evaluated for different NLP tasks (e.g. Platanios et al for MT), however, they seem to have not been evaluated for GPT models. Despite its simplicity, the approach achieves more stable training, faster convergence to a target perplexity (⪆50% time reduction), and better generalization as measured by validation perplexity or downstream accuracy. The method has three additional hyperparams (initial/final sequence lengths and the curriculum duration in updates), which are relatively easy to set by observing initial model behaviour on the validation set.


main_review: Strengths:
- simple but effective curriculum method
- large savings in time and improvements in quality

Weakness:
- Because of the surprising method simplicity, more analysis would be interesting to add that could shad light on the nuances of interplay between gradients and the CL, and why it helps. Some suggestions:
  * Does CL lead to fewer instance of gradient clipping compared to the baseline?
  * "The largest variance on certain dimensions" is mentioned as a problem in the intro and in the last sentences of sec. 5.3, but no experiment measures it w/ and w/o CL.
  * Gradient similarity between neighbouring batches w/ and w/o CL
  * A simulated experiment: while doing CL, insert a sequence of a few baseline (full-length) batches, -- will you see the "instability spikes" like in baseline learning curves?
- Given that Platanios et al reported good results both with length- and word rarity-based curricula, it would be nice to run a few experiments with the word-rarity difficulty definition.
- Regarding tuning the T hyperparameter: since the validation set exhibits fluctuation for some T, I wonder if it's simply because, for larger T, the curriculum hasn't yet ramped up to the actual lengths of validation data? Esp. since the fluctuation seem to fade away closer to the 10K cutoff. If that is true, could you simply set T to a function of some length statistic of the validation set?

Clarification questions/remarks:
- When trimming the batches to the current length, what happens to the shorter sentences? If they stay in the batch, than, effectively it's sampling from all lengths below the current seqlen_t?
- The two-stage curriculum has actually two spikes (@20K in Fig. 3(f) and later @30K), what is the reason for the 2nd spike if the transition to full-length has already happened?
- Figure 4: if CL60K is preferred here, does it mean it overtakes the validation curves for CL100K and CL80K? When does that happen?
- Some of the prior work is mistakenly described as using fine-tuning (e.g. most references in the paragraph starting with "In the natural language processing area,..."), while they are actually using one-stage training, without fine-tuning, as is standard in machine translation.

Minor remarks:
- "extremely huge" -> "extremely large" or simply "huge"
- "the gradient variance norm" -> "the norm of this variance" (to be more specific)
- "Inspired of the highly organized curriculum": what does the definite article refer to? probably, can omit it
- "human and animal," -> "humans and animals,"
- could you add a citation to the sentence ending with ", and model divergence." in sec. 2?
- something is wrong with grammar in the sentence "To quantitatively measure the token...", sec 5.1
- Table 2: I'd suggest using more intuitive '+' and '-' instead of arrows-up and arrows-down
- Tables 1 and 2 are hard to parse because they contain different types of information: consider splitting into 2 half-tables horizontally in the middle (i.e. on table for target ppl and the rest in the other)
- a coma missing after "Because of 1)"
- please use more specific wording instead of "we find that this is not ideal" (both in sec. 5 and A.2). What do you mean - not ideal for 117M or in general? What is "ideal" here?
- Figure 2. "the first 60K" -> "the first T=60K"
- please explain the term "token reduction" in sec 5.4
summary_of_the_review: A simple approach that nevertheless fixes GPT's stability issues and improves empirical performance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a curriculum learning approach that gradually increases the input sequence length for training large-scale transformer models. The study various aspects of how the approach influences the training of GPT-2, for example showing that curriculum learning can lead to reduced wall-clock time and number of tokens.
main_review: Strengths:
+ The paper is generally well written and easy to follow.
+ The authors provide suficient technical details to reproduce results.
+ The experimental results show the utility of curriculum learning.

Weaknesses:
- The paper applies and existing curriculum learning approach on GPT-2. Hence, the paper lacks novelty. The study could have been performed by other researchers without too much effort. While the work might be suitable perhaps as a workshop, the level of novelty for ICLR is definitely not met.

- It is not clear that truncating text data leads to easier examples. The truncated part might contain relevant features and by removing them, it would be impossible for the model (and for a human) to make a prediction. Therefore, proposed method might also be regarded as anti-curriculum approach. Is there any linguistic motivation that could support truncating the examples?

- The curriculum baselines chosen in Sec. 5.4 are very weak. There are plenty of curriculum learning methods that could have been considered as baselines.

Minor language corrections:
"which helps improves" => "which helps to improve" or "which improves";
summary_of_the_review: In my opinion, the weaknesses outweigh the strengths of this paper. In particular, the lack of novelty is a major issue that is enough to justify rejecting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the pretraining of large language models. They address the issue of instability by proposing curriculum learning. They say that curriculum learning enables training with 8x larger batch size and 4x larger learning rate. This reduces number of tokens and wall clock time by the order of magnitude of 2x.
main_review: Strengths
- Method is simple and easy to understand
- Topic of efficient pretraining is important

Weaknesses (reasons to reject)
- In terms of new ideas, the paper is not particularly ambitious or novel. I am not an expert, but given the cited related work, the results of this paper are not surprising or thought-provoking to me. Curriculum learning has been shown to improve training speed in a variety of scenarios, so I am not at all surprised that if one tries hard, they could make it work with GPT pretraining. Sorry if this is not actionable.
- The paper claims that the new lesson learned is that stability is improved with curriculum learning. If a convincing result for this was shown, I have missed it. I think part of it is that it is still unclear to me what the particular operationalization of "stability" is that is used in the paper. The fact it is really hard to match the colors to lines in Figure 3 doesn't help, either. For instance, where is the training instability in Figure 2? Did the GPT-2 authors originally struggle a lot to do "stable" training? If there is prior work on operationalizing stability, then it should probably be explained further in the paper, given that stability is the novel contribution. 
- To make the above bullet point even worse, the use of the word "regularization" further convolutes what the main learning is. My understanding of regularization is that it prevents overfitting. Where's the overfitting in these charts?

Weaknesses (not reasons to reject)
- The paper would be more attractive in my opinion if they really made it more about curriculum learning than it currently is. What I mean by this is that the current paper seems more like "BERT trains fast and with more stability if you gradually increase the sequence length of examples." It would be a lot more interesting, for example, if it were shown that the author's proposed method worked for other operationalizations of curriculum learning in addition to sequence length (e.g., age of acquisition, word frequency, etc). 
- Related to the above point, there is very little motivation for why sequence length is a correct operationalization. 
- In fact, I do not think it is even shown that curriculum learning is the reason for the improvement. For example, the author's experiments do not distinguish curriculum learning from the simple hypothesis "putting similar examples into batches" improves training, which is consistent with smaller gradient variances. One way to show that curriculum learning is actually at play here would be to do an "anti-curriculum" that starts with long sequence and ends with short sequences. 
- It seems to be that it will be hard for curriculum learning to gain traction for pretraining. The cost of pretraining is high, and so introducing additional complexity will probably be done with caution, and if curriculum learning requires additional hyperparameters, people may hestitate to use it in case they do not choose correct curriclum learning hyperparameters.

Minor comments
- Cite the transformer paper in sentence 1
- It is probably not best practice to cite the GPT-3 training time if you aren't doing GPT-3. Just cite the GPT-2 training time instead.
- The top margin seems too small.
summary_of_the_review: This paper studies a straightforward pretraining strategy. It does not propose a novel method, and is not well-motivated enough as an empirical evaluation, in my opinion. Given prior work, results were not unexpected. At this point, I cannot clearly understand the paper's claims regarding "stability" and "regularization", and hence cannot recommend acceptance to ICLR (though I think it's fine as a workshop paper).
review_title: null

Meta-review: This submission proposes a simple way to improve the stability of training GPT-2: Increase the sequence length of examples over the course of training. It is shown that this simple heuristic can result in using larger learning rates, therefore significantly speeding up convergence. Reviewers agreed that this was a simple and effective approach, but shared various concerns about the paper:
- The paper focuses on GPT-2, while stability issues can arise in a much wider range of models. Additional experiments with other models (and ideally other codebases/training setups) would help verify that the proposed method is broadly applicable.
- Better analysis of why using the sequence length as the difficulty metric would be helpful. What other criteria would be possible? Why is sequence length the best?

I would suggest that the authors significantly expand the submission based on the above suggestions and resubmit.
Decision: Reject