Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors study the performance of transformer models on downstream tasks as the total computational budget is decreased.  This process, known as cramming in the paper, turns the problem of training these enormous language models in a new direction from the typical scenario used in industrial labs that train models on a seemingly endless supply of resources. The author's place and exception small limit on the total computation that is allowed to train a transformer model from scratch to the total FLOPs available on a single GPU in 24 hours. By considering the scaling laws of large model transformers the authors mainly investigate training setups that keep the total number of parameters in the model constant but reduce the cost of performing a gradient update. By enumerating a small number of interesting features of the transformer training design space the authors demonstrate that cramming can achieve interesting and sometimes comparable results with larger models using more computation in particular settings and for particular datasets.
strength_and_weaknesses: Strengths:
- The motivation for the study proposed in the paper is interesting for a number of reasons. The volume of computation required by many modern transformer models has been prohibitively expensive and therefore out of reach for most researchers for quite a while. By studying the implications of constraining the computational resources on the ability of the model to perform well on certain tasks the authors could provide a way for researchers with limited budgets to participate and utilize these models in fundamentally new ways.
- The trend in the paper to consider modifications that mainly reduce the gradient update cost without significantly impacting the total number of parameters in the model, based on the scaling laws, provides an interesting and unifying theme throughout. The persistence of the scaling laws to influence the performance of the model on tasks is reinforced through empirical evidence throughout and yields interesting insights.
- Performance evaluation on a shoe-string budget of FLOPs compare to other prominent models is impressive.

Weaknesses:
- Similar studies were conducted on a single node with 8 GPUs as noted by the authors. Though that setup had considerably more computational resources the total volume of computation was still a fraction of the amount used by many large research institutions. In light of that work, the scenario presented in this paper may seem somewhat derivative and only marginally interesting.
- It is not clear if or how the observations made in the cramming regime may be used to make more informed decisions regarding the training process in the normal training setting.
clarity,_quality,_novelty_and_reproducibility: The writing is clear and the presentation of issues motivating the current work is adequately articulated in the text. While I am not an expert in the transformer field I feel the authors did a good job explaining the connection between the scaling laws and the downstream performance of the models under consideration. The novelty of the work pertains to the training strategies used to reduce computational costs without removing the total number of model parameters. Although previous works looked at training with limited resources the author's study and extreme training scenario that is likely to be more pertinent and representative of the resources available to typical, non-institutional, researchers.
summary_of_the_review: Overall I find the motivation for the work and claims made by the authors to be an interesting departure from the traditional language training papers that use exorbitant computational resources. It seems more practical to answer questions about how researchers can do more with less when it comes to allocating resources for training transformer models.

My remarks should be taken with a grain of salt as I am not an expert in this particular area but I would feel more inclined to experiment with transformer models if I felt I could train them to a reasonable level of ability on my modest desktop setup. I believe this sentiment represents the spirit of the paper and the results should be of interest to other members of the research community that are hesitant to participate in this research area because of the perceived computational overheads.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigated pretraining a mask language model in a resource-constrained setting, i.e. a single GPU for one day. The authors empirically tested various architectural and data changes in order to maximize performance. There are some interesting findings, such as per-gradient efficiency only depends on model size, several strategies to filter and sort the training data brought improvement, etc. As a results, the authors were able to push the performance close to BERT base if excluding some outlier tasks.
strength_and_weaknesses: Strengths:
- This paper adds more insights on scaling-down, which is less understood as most concurrent efforts are around scaling-up. 
- The experiments have a good coverage in terms of testing various architectural changes from recent literature.
- The final performance on downstream tasks (GLUE) are impressive given the limited compute budget. 
  
Weaknesses:
- The biggest missing piece is an ablation study on the improvement from various architecture changes. In the current version, the authors provided a comprehensive list of things they tried, what helped and what didn't. But it's unknown which change(s) brought the bigger improvement.  
- Related to this is the poor performance on CoLA. Although the authors provided several hypotheses, some of them should be tested to verify whether they're actually related to any of the architecture or data changes, or mostly due to reduced model size. For example, one possible cause provided by the authors is that reasonable performance CoLA would need more training data. But are models in these experiment trained on less data compared to BERT-base? Is it possible to see whether the performance gap can actually be closed if the models were trained longer than one day (i.e. seeing more data)?
clarity,_quality,_novelty_and_reproducibility: Clarity:
- For all the architectural modifications in Sec 4.2, does "no improvement" refer to pretraining loss or downstream tasks?
- In Sec 4.3 batch size schedule, you found optimal performance from different batch size for pretraining (1536) and downstream tasks (4032). Why do you think pretraining loss benefit from smaller batch size? Similarly, could any of the architectural changes in Sec 4.2 have different effects on pretraining vs. downstream?
- For all the changes in Sec 4.2 and 4.3, when you test a specific modification, what were used for the rest of the architecture and training setup? How were they chosen?

Quality:
- The 128 sequence length differs drastically from common choice in language models pretraining (e.g. 1024, or at least 512). To make sure conclusions from this work would apply, some additional experiments with longer sequence length would be helpful.

summary_of_the_review: Overall, this paper tackles an important problem, the experiments design are sound and the empirical findings are informative for language models pretraining. If the authors could add more clarity to generalizability and robustness of the findings, e.g. experiments with sequence lengths longer than 128, further ablate on the causes of drop in CoLA, etc. then the results would be more valuable to language models pretraining.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the paper investigate language model pipeline to see which modifications improve performance in the scaled-down scenario ( a single GPU for 24 hours).  
strength_and_weaknesses: Strength:
1. In this paper, several modifications (architecture, training setup and datasets) are explored to check whether there is any improvement. All of these aspects are important and interesting. These can give good insights for the community. 

2. Some interesting conclusion are got, for example, training recipe and data setup lead to decent downstream performance on GLUE.

Weaknesses:


1. The investigation of modifications lack convincing experiments. For example, only one task performance (MNLI) is reported for when studying the impact of training hyper-parameters. Other tasks can have a different trend.  And when exploring the effect of the architecture, only MLM loss is report. The performance of downstream tasks can be also important. 

2. The technical novelty of this paper is a little limit. The total contributions are also limit. 
clarity,_quality,_novelty_and_reproducibility: Clarity is clear, however, novelty is limit. 
summary_of_the_review: This paper does a lot of Interesting investigations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: While research on enormous language models have dominated empirical NLP lately, most researchers and practitioners do not have the resources and access to work with these models. This work seeks to answer the empirical question: what is the best performance one can achieve on the GLUE benchmark by training a BERT-like model from scratch on a consumer GPU for one day? The numerous Transformer variants proposed in recent years present another challenge to answering this question – which of these variants are beneficial when one’s compute is extremely constrained?

The authors investigate a wide range of design choices in terms of model architecture, training recipe, and data curation. They also note that the final MLM loss is mostly correlated with the FLOPs spent, not the particular Transformer type and size. This motivates them to choose architectures that parallelize well on GPUs. The final result demonstrates that some combination of the variants proposed in the three years after BERT yields a model that is almost as performant as the original BERT base while using 1/50 of the FLOPs.
strength_and_weaknesses: This work presents a thorough empirical investigation on a topic that is of interest to many researchers who do not have access to large compute clusters. The overall methodology appears to be sound and the final result promising.

However, I am not sure if ICLR is the best venue for this work. There are no new theoretical or algorithmic contributions nor any new insight into representation learning. This does not change my belief that this is an informative paper to many in the community, but it might find a more suitable audience if submitted to venues such as EMNLP. (Disclaimer: I do not regularly publish in this area. I am happy to defer to more experienced reviewers and the AC on this point.)

An additional concern is that it is not clear if the MLM loss is the best metric to track when comparing different model/training design choices. This paper (https://openreview.net/pdf?id=F5uYcwABMu), for example, clearly demonstrates that models with near-identical pretraining loss can perform very differently on downstream tasks due to implicit biases. This should be taken into consideration since many conclusions in this work are based on the MLM loss alone. I encourage the authors to also observe downstream performance and note if they agree or disagree with the MLM loss.

The connection between “cramming” and scaling laws can be clarified. The scaling laws mentioned in this work are empirical observations that a model’s performance strongly correlates its size but not necessarily its shape. The empirical results from this work show that this holds for the low-compute regime, which is somewhat surprising. However, these “laws” are merely empirical observations. It is not clear what the authors mean by “we discuss scaling laws in the low compute regime and find that cramming is hard because the conclusions of Kaplan et al. (2020) so easy to reproduce” (emphasis mine) in the conclusion. It would be better to simply state that the empirical observation from Kaplan et al. holds in the setups investigated, which motivated using architectures that parallelize well. A related concern is that since the dominating factor for performance is the number of FLOPs we can squeeze out of a GPU within a given timeframe, this makes the conclusions of this work somewhat hardware-specific, e.g., they might not hold on TPUs or newer/older GPUs.

clarity,_quality,_novelty_and_reproducibility: The paper is mostly well-written with just a few typos. I am not aware of similar investigations in the low-compute regime and believe that many in the community might find this work informative.
summary_of_the_review: Neat empirical investigation with conclusions that might interest many. However, it is not clear if ICLR is the best venue. Moreover, one might argue that MLM loss is not the best criterion to study model and training design choices.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper focuses on pre-training the best-possible (masked) language model on a modest budget (a single day on a single consumer-grade GPU). They consider various modifications to the standard training techniques, including changing the architecture, training setup (e.g. loss function, learning rate schedule, etc.), and data. Congruent with past work, they find that architectural variants make a small difference, whereas changing the training setup and data can be beneficial. Based on these findings, they propose a recipe for producing a BERT model that attains reasonable accuracy on GLUE. While all reviewers appreciated the paper's focus (training a decent MLM on a tiny budget), there were various concerns that could not be fully address in the rebuttal. Specifically, reviewers were interested in a more comprehensive ablation of the different changes introduced in the modified recipe, less of a focus on MLM loss as a target metric, lack of novelty in findings (e.g. it was already known which of the studied components could improve performance), and lack of clarity in terms of what the final recipe was. The consensus is therefore towards rejection.
justification_for_why_not_higher_score: See concerns in metareview.
Decision: Reject