Emergent Inabilities? Inverse Scaling Over the Course of Pretraining
Authors: James Michaelov
Ben Bergen
Venue: EMNLP 2023 Findings (2023)
Keywords: language models
inverse scaling
transformers
training dynamics
TL;DR: Language models can show inverse scaling not only as a function of number of model parameters, but also of model training data.

Abstract: Does inverse scaling only occur as a function of model size, or can it also occur over the course of training? We carry out an exploratory study investigating whether the performance of language models on specific tasks can decrease (while general performance remains high) during training on the language modeling task. We find 8 tasks on which Pythia 12B (Biderman et al., 2023) shows decreased performance over the course of training. Five of these tasks (TruthfulQA-MC1, TruthfulQA-MC2, Hindsight Neglect, Memo Trap, and Pattern Match Suppression) additionally show a consistent relationship whereby larger language models show a greater decrease in performance the more they are trained, despite showing standard (positive) scaling overall. This highlights the importance of testing performance at all relevant benchmarks any time models are trained on additional data, even if their overall performance improves.

Emergent Inabilities? Inverse Scaling Over the Course of Pretraining
James A. Michaelov and Benjamin K. Bergen Department of Cognitive Science, University of California San Diego (j (ilmichae, bebergen)@ucsd. edu
Abstract
is the TruthfulQA benchmark, where larger lan- guage models are more likely to predict popular misconceptions over statements of fact (Lin et al., 2022). More recently, additional tasks that report- edly show such an effect have been identified as part of the Inverse Scaling Prize (McKenzie et al., 2023b), as well as by other researchers (Jang et al.. 2023; Michaelov and Bergen, 2023).
Does inverse scaling only occur a function of model size, or can it also occur over the course of training? We carry out an exploratory study investigating whether the performance of lan- guage models on specific tasks can decrease (while general performance remains high) dur- ing training on the language modeling task. We find tasks on which Pythia 12B (Bi- derman al., 2023) shows decreased perfor- mance over the course of training. Five of these tasks (TRUTHFULOQ QA A-MC1, TRUTHFULQA A- MC2, HINDSIGHT NEGLECT, MEMO TRAP, and PATTERN MATCH SUPPRESSION) addi- tionally show consistent relationship whereby larger language models show a greater decrease performance the more they are trained, de- spite showing standard (positive) scaling all. This highlights the importance of testing performance at all relevant benchmarks any time models are trained on additional data, even if their overall performance improves.
Inverse scaling is a serious concern for several reasons. At high level, inverse scaling may in- dicate 'outer misalignment" (Perez et al., 2022) between the model training approach and the pur- poses to which they are applied. The lack of robust- ness observed in inverse scaling phenomena may thus indicate that the apparent successes of specific language models at wide range of benchmarks (e.g.. Hendrycks al., 2021; Srivastava et al., 2022) do not necessarily entail that they have the capa- bility ostensibly being tested (Bowman and Dahl, 2021; Raji et al., 2021).
The existence of inverse scaling is also concern- ing because of the possibility of other as yet uniden- tified tasks where performance similarly scales inversely with model size. Models that perform well on variety of tasks may well present dete- riorating performance in unseen tasks with scale, even as performance at established benchmarks increases. This is of particular concern if better performance at established benchmarks and more natural-seeming output leads users to place more trust in such models as general- purpose natural language understanding systems (see, e.g., Bender et al., 2021, for general discussion of such risks). Finally, inverse scaling is also of concern be- cause it is often unpredictable. In the same way that certain capabilities appear to emerge at scale (Wei et al., 2022), inverse scaling also appears or accelerates at given scales. For example, as McKenzie et al. (2022b) show, the performance of Gopher (Rae et al., 2022) and Plain LM (Ganguli et al., 2022) at the Inverse Scaling Prize's negated
1 Introduction
For language models, bigger is usually better. Re- cent research has found that both increased number of model parameters and increased size of the train- ing dataset positively influence model performance (Brown et al., 2020; Kaplan et al., 2020; Chowdh- ery et al., 2022; Clark et al., 2022; Du et al., 2022; Rae et al., 2022; Hoffmann et al., 2022; Thoppilan et al., 2022; Wei et al., 2022; Taylor et al., 2022; Srivastava al., 2022; Touvron al., 2023a). One particularly striking pattern that has been reported is emergence, a nonlinearity in these relationships, where at particular scale, language models im- prove rapidly at given task (Wei et al., 2022).
However, while increased scale usually leads to improved performance, on certain tasks it corre- lates with decreased performance. This is known inverse scaling (Lin et al., 2022). An exam- ple of task on which inverse scaling is observed

---

question-answering task (NEQA) appears be stable or even increasing as model size increases, only dropping as model size increases to around 7 billion parameters and beyond (McKenzie et al., 2022b). Thus, inverse scaling may occur not just for unidentified tasks, but also for well-established tasks: model architecture that performs well at a benchmark at small scale may suddenly per- form surprisingly worse scale increases-it is not safe to assume that performance will continue to improve or even remain stable.
conceptions or statements of fact more predictable in a given context. In this version, there is one truc option and the remainder are false. For example, for the prompt what should you take for a viral infection?, the true answer is you should take a rest and drink fluids, and other answers include you should take a full course of antibiotics. The language model is considered to respond conrectly if the probability of the correct answer is higher than any other.
TRUTHFUL QA-MC2 is an alternative multiple- choice variant of TRUTHFULQA A in which there are often multiple correct answers. Here, the model is considered to respond correctly if the total prob- ability of correct responses is greater than that of incorrect responses.
While previous work has focused on inverse scal- ing based on the number of model parameters (Lin et al., 2022; McKenzie et al.. b022a,b, 2023a; Jang et al., 2023; Michaelov and Bergen, 2023); as dis- cussed, scaling effects more generally occur not just in relation to model size but also as funetion of training data quantity. Recent work has shown that this latter effect has been substantially under- estimated (Hoffmann et al., 2022; Touvron et al., 2023a). Thus, in the present study, we consider the possibility of inverse scaling over the course of pretraining; that is, the hypothesized situation where performance on some tasks decreases as the total quantity of training data increases. If inverse scaling of this type exists, it may be more prob- lematic than inverse scaling as function of model size, because if performance at a given task can degrade over the course of pretraining even as the performance at the majority of tasks improves, a specific model that has previously performed well at a given task may unexpectedly show decreased performance later in training.
SIG FIGS, like all the Inverse Scaling Prize tasks, is multiple-choice task with one correct answer. It also has two incorrect answers, and the model is considered to respond correctly if the cor- rect answer has the highest probability. The task itself tests whether, when prompted to round a num- ber to a number of significant figures, the language model finds the number rounded correctly to be more probable or the number rounded to the same number of decimal places.
NEQA is zero-shot task with negated ques- tions such as As the barometer reading goes lower there is not a greater chance of A. sunshine B. get- ting wet.
REDEFINE is a zero-shot task where expressions are redefined in a range of ways, and then ques- tions are asked are asked about these redefined expressions e.g,, a prompt may ask for the first digit of 5 15, where 5 + 15 is first redefined as text string rather than an equation. The task tests whether the language model does indeed treat the expression in the redefined way rather than its usual interpretation.
We test how well the 8 models of the Pythia (Biderman et al., 2023) suite perform at 12 tasks over the course of training. These tasks are the 10 multiple-choice tasks that won the Inverse Scaling Challenge (McKenzie et al., 2022b), plus the two multiple-choice versions of the TruthfulQA bench- mark, on which language models have also been found to show inverse scaling (Lin et al., 2022). The aim of this study is therefore exploratory analysis see whether tasks that show inverse seal- ing as model parameter size increases also show inverse scaling over the course of training.
MEMO TRAP is task where a language model is instructed to write famous quote with specific last word, c.g., write quote that ends in the word "heavy": Absence makes the heart grow. In this case, the correct answer would be heavy and not the expected
HINDSIGHT NEGLECT is a few-shot multiple- choice task where the input contains information about a bet and its outcome and the task is to cor- rectly determine whether or not the bet should have been taken. In the task a number of examples are provided where the expected value aligns with the
2 Method
2.1 Tasks
TRUTHFULQA-MCE is the first multiple-choice variant of TRUTHFULQQ (Lin et al., 2022), which tests whether language models find common mis-

---

result (if the task has a positive expected value, the individual taking the bet wins, and if has neg- ative one, the individual taking the bet loses). For the final question (the one that is answered for the task), the value won or lost does not align (the in- dividual either wins a bet with negative expected value or loses one with a positive expected value).
task on The Pile (Gao al., 2020), an 800GB text datasct comprising 300 billion tokens. All models were trained on this dataset, with checkpoints re- leased at every 2 billion tokens of training. Given that scaling is often considered on a logarithmic scale, we tested each model's performance at 8 checkpoints based on powers of 2: checkpoint 2 (4 billion tokens), checkpoint 4 (8B tokens), check- point 8 (16B), checkpoint 16 (32B), checkpoint 32 (64B), checkpoint 64 (128B), checkpoint 128 (256B), and checkpoint 143 (300B tokens, i.e., fully trained).
INTO THE UNKNOWN is a task that involves description of setting and question, with the twist that the task identify which of two pieces of information would help to answer the question. One option (the correct answer) contains new in- formation and the other repeats information from the original description.
We run our analyses of model performance using the Language Model Evaluation Harness (Gao et al., 2021). All code, data, and statistical analyses are provided at https://gis thub. com/ inabiitttiaaiiiiiis..
MODUS TOLLENS tests whether language mod- els can make predictions in line with the modus tol- lens form of deductive inference, i.e., '[i]f p. then qi not q; therefore, not p' (McKenzie et al., 2023b). The task involves an example of such an inference, and then a question of whether the conclusion is valid or not.
3 Results
Model performance at each task is shown in Fig- ure 1. In order to quantify the patterns observed, we also fit a least-squares linear regression for each dataset, with the logarithm (base 10) of model pa- rameters, the logarithm (base 10) of training tokens, and the interaction between them predictors of task accuracy. All variables were z-scored. The results these tests are shown in Table 1.
PATTERN MATCH SUPPRESSION tests whether language models can violate a repeated pattern. For example, one prompt is to generate a sequence of 6 symbols alternating between two symbols (A B) but ending unexpectedly. A, B, A, B, A, with possible answers or B.
RESISITING CORRECTION is a few-shot task, with the instruction to repeat a text without chang- ing it and two examples. In the final example, the sentence to be repeated includes an atypicality, e.g.. spelling mistake or switched word of a famous quote. The task tests whether the model follows the instruction and replicates the atypical, whether it 'corrects' it.
The clearest inverse scaling effects can be seen with TRUTHFULQA-MMC--AAaN
models per-
form worse, performance overall decreases with number of training tokens, and the rate at which performance deteriorates with training tokens in- creases with model size. Inferential statistics show negative effect of number of parameters, num- ber of training tokens, and their interaction. other words, the regression predicts that model performance decreases with number of parame- ters and training tokens; and in addition, that the larger model is, the more there is a decrease in performance as the model continues to train. Whether this pattern of statistical results is spe- cific to the tasks used in the present work or all tasks that show inverse scaling question for future work. However, does also appear to be present for most of the other tasks clearly displaying inverse scaling, namely, HINDSIGHT NEGLECT, MEMO TRAP, PATTERN MATCH SUP- PRESSION, and TRUTHFULQA-MC!
REPETITIVE ALGEBRA is few-shot task based on simple algebra questions. Until the penul- timate question, all questions have the same answer (provided in the prompt), and the penultimate ques- tion has an answer that differs (also provided in the prompt). For the final question that needs to be answered, the answer is the same as the initial answers. The task tests which of the two answers (initial or penulatimate) the model predicts to be more likely.
2.2 Models
We use the 70 million parameter (70M), 160M, 410M, IB, 1.4B, 2.8B, 6.9B, and 12B Pythia mod- els (Biderman et al.. 2023). The models were trained on the autoregressive language modeling
Some of the remaining tasks, namely INTO THE UNKNOWN, MODUS TOLLENS, NEQA, and SIG FIGS display no clear pattern across models. But

---

TruthfulQ4: MC1
TruthiuQQ:: MC2
ISP: Hiidssight Neglect
ISP:Ints Into the Unknown
0.50
0.51
0.26
0.45
0.24
0.22
0.35
0.20
Mem: Trpp
ISP: Nodus Tollens
ISP: NeQA
ISP: Pattern Match S suppression
milon
0.9
0.48
160 milon
Accuracy
410 milon
0.8
0.4
0.50
1 billenn
0.25
0.2
0.7
28 billon
0.00
0.45
0.0
6.9 bilion
Redefin
Repesiive Algebra
billen
0.39
0.55
0.8
0.36
0.55
0.50
0.25
0:33
10th
10935
10
10115
10
10l45
10
10
10t
10
10
10345
10.0
10l45
10
1011
Training Tokens
Figure I: Performance of the 8 Pythia (Biderman al., 2023) models 8 stages the course of training the two multiple-choice variants of TRUTHFUL QA (Lin et al., 2022) and the 10 multiple-choice winners of the Inverse Scaling Prize (McKenzie et al., 2023b).
Task
Parameters
Tokens
Interaction
Hindsight Neglect
((60))=4.22,
p<0.001
1(60))-4.69, p<0.001
f(60))-2.88, p=0.012
Into the Unknown
060))= p=0.824
(60))=2.04. p=0.079
(00)=00.22.p00.90
Memo Trap
1660)=-10.05, p=0.001
1000==11.34,p<0.00
(60))=9.71,,
Modus Tollens
(6009-7.16..=-09277
(60))=2.133 p=0.071
(600=-1150, p=0.208
NeQA
((60)=0.79, p
1(60)=-0.08, p=0.963 3
1000=2445,p-000000
Pattern Match Supp.
1(60)=33.00,--00
1000==99.8,,00..0
t(60)=-6.37, p<0.001
Redefine
((60)=0660, p=0.645
(60))=0.79,pp00..
t(60)=-1.53, p=0.205
Repetitive Algebra
((60)=-0.49, p=0.706
(60)=-2.11,
1(60)=-1.00, p=0.443
Resisting Correction
((60)=5.13, p<0.001
(600) p<0.000 p<0.001
((60)=-1.89 p=0.104
Sig Figs
-6.5) p=0.645
(60))=0.74, p=0.574
(660)=-1.46, p=0.215
TruthfulQA-MCI
1(60)=10090, p<0.001
(660)=11.45, p<0.001
(660)=2..7, p=0.010
TruthfulQA-MC2
1(60)=-4.77,
p<0.001 100.00000..p
((60)=-3.89,, p<0.001
t(60)=-12.02, p<0.001
Table Statistical tests carried out on the performance of the Pythia models, testing the effect of (og-traanffrmme) number of parameters, (log-transformed) number training tokens, and their interaction. positive t-value indicates that the variable is significantly correlated with higher accuracy. All p-values are corrected for multiple comparisons based on false discovery rate (Benjamini and Hochberg, 1995).
when focusing on just the two largest models, RE- DEFINE appears to show inverse scaling the course of training, and the largest (12 billion pa- rameter) model shows inverse scaling during train- ing on REPETITIVE ALGEBRA and RESISTING CORRECTION. These may be a case of emergent inverse scaling (i.e., nonlinearities that cannot be accounted for using linear statistical models), espe-
cially in the case of RESISTING CORRECTION, but models with larger number of parameters would be needed to verify this.
Discussion
We find clear evidence of inverse scaling over the course of training on TRUTHFUL.A--MII.. TRUTHFULQA-MCC. HINDSIGHT NEGLECT,

---

MEMO TRAP, and PATTERN MATCH SUPPRES- SION, as well as possible evidence of the same phe- nomenon on REDEFINE, REPETITIVE ALGEBRA, RESISTING CORRECTION for the largest model or models. In addition, RESISTING CORRECTION ap- pears to present an example of emergence in inverse scaling over the course of training- performanc only decreases with training on the largest model.
tern of inverse scaling that we idmmtify-namely greater decrease in performance during training in the largest models-is general feature of in- verse scaling, or only due to the fact that we use tasks already known to show inverse scaling as models increase in number of parameters. Future work should establish what kinds of relationships (if any) hold between inverse scaling as func- tion of model parameters and inverse scaling as function of training data.
At the time of initial writing, this study was the first have identified an example of inverse seal- ing over the course of pretraining. Since then, an official Inverse Scaling Prize paper has been re- leased (McKenzie et al., 2023b). In addition to exploring scaling in terms of the number of float- ing point operations (FLOPs) needed to train each model, McKenzie et al. (2023b) also analyze the performance of different sizes of the Anthropic LM model (2M, 13M, 42M, 197M, 805M, 3B, 13B, 52B) over the course of training on 400B tokens, providing a valuable point of comparison. On the whole, their results are similar to ours over the same scales. At the larger scale, they find that the 13B and 52B models begin to show inverse scaling on NEQA, SIG FIGS, and INTO THE UNKNOWN. Conversely, only the 52B model begins to show inverse scaling on RESISTING CORRELATION. McKenzie al. (2023b) also classify the tasks into different types.' These classes do not clearly delincate between ones that show inverse scaling and ones that do not based on either our analy- ses or their analyses. Nonetheless, they provide a valuable starting point for considering the kinds of features of tasks that may lead to different scaling patterns.
Perhaps the main takeaway of the present study is that of instability in model performance. As we see with Pythia 12B on the RESISTING CORREC- TION task, a model that was previously among the best at given task can relatively suddenly experi- ence decreased performance as it continues to train. Good performance a task at one stage doesn' 't guarantee continued good performance, even in cases where the model only continues to be trained On text data. This highlights the importance of reg- ular and rigorous evaluation. For this reason, users of models subject to updates would be well advised to verify continuing performance regularly, and it is incumbent on parties who provide such models for use in applications to notify users of updates.
5 Conclusions
In this study, we set out to investigate whether inverse scaling can occur not only as a function of number of model parameters, but also number of training tokens. We find clear evidence that it does occur with the Pythia (Biderman al., 2023) suite of models five of the twelve tasks analyzed, and additional evidence that may occur on up to eight.
Indeed, the question of whether there are con- sistent scaling patterns based on task features re- mains an open one. We find several clear cases of inverse scaling that share the pattern model per- formance decreasing more rapidly over the course of training as the number of model parameters in- creases. In several cases there is only a decrease in performance in the largest models. These are not necessarily different phenomena; it may be that the threshold of number of parameters and to- simply
Limitations
The main limitations of this study relate to the mod- els used and tasks evaluated. With respect to the former, our analysis is limited to 8 models at vari- ous stages in their training. While this means that we can make the inference that the performance of a specific model can deteriorate over the course of training, it also means that it is possible that some of the models have idiosyncratic features that would not generalize to other models of the same size or with the same amount of training data. Ad- ditionally, these models cover only part of the pos- sible range of scales for language models there are contemporary models with many more parame- ters (e.g., 540 billion parameters in the case of the
kens for tasks like BRUTHFUI.QA-ME lower than for tasks like RESISTING CORRECTION. Additionally, it is not clear whether the main pat-
SStoong Prior (R (RESITING CORRECTION, MEMO TRAP, REDEFINE), Unwanted Imitation (MODUS TOLLENS, TRUTH- FULQA), Distractor Task (PATTERN MATCH SUPPRESSION, NEQA, SIG FIGS, INTO THE UNKNOWN), and Spurious Few- Shot (HINDSIGHT NEGLECT, REPETITIVE ALGEBRA).

---

largest PaLM; Chowdhery et al., 2022) and trained on more data (c.g,, 2 trillion tokens in the case of LLaMA 2: Touvron et al., 2023b).
Usvsn Sai Prashanth, Edward Raff, Aviya Skowron. Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A Suite for Analyzing Large Language Mod- els Across Training and Scaling, In Proceedings of the 40th International Conference on Machine Learn- ing, pages 2397-2430. PMLR.
Similarly, our analysis is limited to the two multiple-choice versions of TRUTHFUT.Q2 QQ and the ten multiple-choice Inverse Scaling Prize tasks. As noted in Section 4, these are all tasks that have been found to exhibit inverse scaling number of parameters increases. A question for future re- search is whether the patterns of inverse scaling that we find in the present study occur in all cases of inverse scaling, or whether it is possible to have inverse scaling over the course of training that is not impacted by the number of model parameters.
Samuel R. Bowman and George Dahl. 2021. What Will Take to Fix Benchmarking in Natural Language Un- derstanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4843 Online. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei, 2020. Language Models are Few-Shot Learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Ethics Statement
Our work complies with the ACL Ethics Policy. As discussed in the paper, we believe that studies asking questions such as those addressed in the present study are vital for reducing possible harms from language models. We did not train any models for this study, and so the energy consumption is limited to evaluation only: all analyses were run on an NVIDIA RTX A6000 GPU, taking just under 42 hours.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai. Thanumalayan Sankaranarayana Pil- lai. Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Tek, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling Language Modeling with Pathways.
Acknowledgements
We would like to thank EleutherAl for making the Pythia suite of language models and the Language Model Evaluation Harness available, as well as all those involved with the Inverse Scaling Prize for creating and releasing the tasks. Models were evaluated using hardware provided by the NVIDIA Corporation as part of an NVIDIA Academic Hard- ware Grant.
References
Emily M. Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Mod- els Be Too Big? s. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trans- parency, FAceT '21, pages (100-62,, New York, NY, USA. Association for Computing Machinery.
Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bog- dan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George Bm Van Den Driessche, Eliza Rutherford, Tom Hennigan, Matthew J. Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya. David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Mare' Aurelio Ranzato, Jack Rac, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. 2022. Unified Scaling Laws for Routed Language
Yoav Benjamini and Yosef Hochberg. 1995. Control- ling the False Discovery Rate: A Practical and Pow- erful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodollogcaal) 57(1):289-300
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' Brienn, Eric Hal- lahan, Mohammad Atlah Khan, Shivanshu Purohit,

---

Models. In Proceedings of the 39th International Conference on Machine Learning, pages 4057-4086. PMLR.
Joel Jang, Seonghyeon Ye, and Minjoon Sco. 2023. Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts. In Proceedings of The 1st Transfer Learning for Natural Language Processing Workshop, pages 52-62. PMLR.
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin. Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang. Emmma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier- Hellsteml Toju Duke, Lucas Dixon, Kun Zhang, Quoe Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2022. GLaM: Efficient Scaling of Language Models with Hixture-of-Experte In Proceedings of the 39th In- ternationnl Conference on Machine Learning, pages 5547-5569. PMLR.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Mcasuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32 14-3252, Dublin, Ireland. Association for Computational Linguisties.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell. Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez. Nicholas Schiefer, Kamal Ndousse. Andy Jones, Sam Bowman, Anna Chen, Tom Con- erly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk. Stanislav Fort, Zac Hatfield-Dodds Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer. Eli Johnsoo Dario Amodei. Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned.
Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller. Najoung Kim, Sam Bowman, and Ethan Perez. 2022a. The inverse scal- ing prize.
Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bowman, and Ethan Perez. 2022b. Inverse scaling prize: First round winners.
Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim. Sam Bowman, and Ethan Perez. 2023a. Inverse scaling prize: Second round winners.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe. Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Model- ing.
lan R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McL.aan Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauff- man, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang. Zhengping Zhou, Najoung Kim, Samuel R. Bowman, and Ethan Perez. 2023b. Inverse Scaling: When Bigger Isn't 't Better.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Fric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. Zenodo.
James Michaelov and Benjamin Bergen. 2023. Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifiers. In Findings of the Association for Computational Linguistics: ACL 2023, pages 14162-14174, Toronto. Canada. Association for Computational Linguistics.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika. Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Un- derstanding. In International Conference on Learn- ing Representations.
Ethan Perez, Ian McKenzie, and Sam Bowman. 2022. Announcing the Inverse Scaling Prize ($250k Prize Pool).
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bog- dan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. 2022, An empirical analysis of compute- optimall large language model training. In Advances in Neural Information Processing Sys- lems.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song. John Aslanides. Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George van den Driessche. Lisa Aune Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbi, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen,

---

Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan. Michela Pa- ganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Bar Baptiste Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi. Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. 2022. Seal- ing Language Models: Methods, Analysis & Insights from Training Gopher.
rina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, EI- lie Pavlick, Emanuele Rodola, Emma Lam, Fric Chu, Eric Tang. Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice En- gefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez- Happé, Francois Chollet, Frieda Rong. Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Ger- mán Kruszewski, Giambattista Parascandolo, Gior- gio Mariani, Gloria Wang, Gonzalo Jaimovitch- López, Gregor Betz, Guy Gur-Ari, Hana Galijase- vic, Hannah Kim, Hannah Rashkin, Hannaneh Ha- jishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kermion, Jacob Hilton, Jae- hoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Ko- coú, Jana Thompson. Jared Kaplan, Jarema Radom, Jascha Sohl-Diekstein, Jason Phang, Jason Wei, Ja- son Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse En- gel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song. Jil- lian Tang. Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gim- pel, Kevin Omondi, Kory Mathewson, Kristen Chi- afullo. Ksenia Shkaruta, Kumar Shridhar, Kyle Mc- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin. Lidia Contreras- Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam. Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturn, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana. Marie Tolkiehn. Mario Giulianelli, Martha Lewis, Martin Potthast. Matthew L. Leavitt, Matthias Hagen, Mátýás Schu- bert, Mcdina Orduna Baitemirova, Melody Amaud, Melvin McElrath, Michael A. Yee, Michael Co- hen, Michael Gu, Michael Ivanitskiy, Michael Star- ritt, Michael Strube, Michat Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mo- hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Ni- tish Shirish Keskar, Niveditha S. Iyer, Noah Con- stant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans. Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormo- labashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Facersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti
Francesca
Deborah Raji, Emily Denton, Emily M. Bender, Alex Hanna, and Amandalynne Paullada. 2021. Al and the Everything in the Whole Wide World Benchmark. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Par- rish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane. Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, An- drew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajan, Asher Mul- lokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts. Bao Sheng Loe, Barret Zoph. Bartiomiej Bojanowski, Batuhan Öxyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ek- mekei, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Ar- gueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf. Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burde
Joseph Boudeman.
Chris Waites, Christian Voigt. Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera. Clemencia Siro, Colin Raf- fel, Courtney Asheraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Ju- rgens, Debajyoti Datta, Deep Ganguli. Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dicuwke Hupkes, Diganta Misra, Dilyar Buzan, Dim- itri Coelho Mollo, Diyi Yang, Dong- Ho Lee, Ekate-

---

Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le- Bras, Rosamne Liu, Rowan Jacobs, Rui Zhang, Rus- lan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Sto- vall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebas- tian Gehrmann, Sebastian Schuster. Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi. Shikhar Singh, Shima Asaadi, Shixi- ang Shane Gu, Shubh Pachchigar, Shubham Tosh- niwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee. Spencer Torene, Sriharsha Hatwar, Stanislas De- haene, Stefan Divic, Stefano Ermon, Stella Bider- man, Stephanie Lin, Stephen Prasad, Steven T. Pi- antadosi, Stwart M. Shieber. Summer Misherghi, Svet- lana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster. Tao Li, Tao Yu, Tariq Ali. Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Sha- ham, Vedant Misra, Vera Demberg. Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fe- dus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong. Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yanggiu Song. Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. 2022. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.
Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Edd Chi, and Quoe Le. 2022. LaMDA: Language Models for Dialog Applications.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. LLaMA: Open and Efficient Foundation Language Models.
Hugo Touvron, Louis Martin, Kevin Stone, Peter AL- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang. Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan. Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open Foundation and Fine-Tuned Chat Models.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus, 2022. Emer- gent Abilities of Large Language Models. Transac- tions on Machine Learning Research.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshom, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A Large Language Model for Science
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin. Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man. Kathleen Melste-ell Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran,


Reviews:
[{"review_id": "bOqAajjjc8", "reviewer": "Reviewer_K1CC", "summary": "", "questions": "I found the linear regression experiments very confusing. What is \"the interaction between log parameters and log training tokens\"? What does t(52) represent (I suspect 52 is the degree of freedom, but where does it come from)? As it is, I was not able to understand the takeaways from this section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Responses to specific comments are given below.\n\n*“With the limited experimental setup, it is hard to draw any reliable takeaways.*\n* *The authors only study 4 of 11 Inverse Scaling tasks, and do not include other datasets that have demonstrated inverse scaling (e.g., TruthfulQA). With only 4 datasets, it is hard to draw conclusions about training dynamics more broadly, compared to task-specific phenomena.*\n* *The largest model studied has 6.9B parameters, while popular LMs these days have reached a scale two orders of magnitude larger. For instance, for some of the Inverse Scaling tasks (like Sig Figs and Into the Unknown), performance does not begin deteriorating until models hit 7B parameters. While such large models are usually not publicly available (and would require significant compute to train from scratch), it does limit the relevance of this paper's study into today's LMs.”*\n\nWe thank the reviewer for these comments. We agree that the limited model sizes and number of datasets in the original submission limit the inferences that can be drawn from the results. We are addressing this in two ways. First, have carried out our analyses on the largest Pythia model, which has 12 billion parameters, and thus is closer in size to many of the commonly-used open models today). Second, we have followed the reviewer’s suggestion to include a wider range of tasks. \n\nWe have carried out a new analysis along these lines, looking at the 70m, 2.8b and 12b Pythia models at three steps along their learning trajectory: 4000, 64000, and 143000 (final model), and including all 10 multiple-choice Inverse Scaling tasks and both multiple-choice versions of TruthfulQA. Results for the 12-billion-parameter model are clearest—they show inverse scaling over training on the Resisting Correction, Repetitive Algebra, Redefine, Pattern Match Suppression, Memo Trap, and Hindsight Neglect tasks of the Inverse Scaling Prize. We also see inverse scaling over training for both versions of TruthfulQA. We believe that this substantially improves the work: adding a larger model makes the work more relevant to current models; adding more tasks demonstrates the extent of the phenomena and as shown below points to explanations for why inverse scaling over training occurs.\n\n* *\"The 4 datasets do not demonstrate consistent trends over training, and the authors do not offer discussion of why two tasks in particular show inverse scaling, while the other two do not.”*\n* *\"The authors leave many questions open which would make the paper a more complete investigation. For instance, how do trends over training relate to trends over parameter count? There is speculation that different inverse scaling behavior has different underlying causes (e.g, strong priors, or misleading few-shot examples); would these reflect differently in training dynamics?”*\n\nWe hope that our extended analyses can help to shed light on whether there are differences in which kinds of tasks show inverse scaling over parameter count vs. training data. We thank the reviewer for suggesting considering the different types of underlying cause—indeed, our new results described above are striking in that we see inverse scaling for all the tasks we tested that are labeled as falling under the “Strong Prior” and “Spurious Few-Shot” type; but not the “Unwanted Imitation” task (Modus Tollens), and only one of the “Distractor Task” tasks (Pattern Match Suppression). This may be an example of a difference between inverse scaling over parameter count or training tokens. Or it may suggest that there is an as-yet-unidentified feature of the Pattern Match Suppression task leading to inverse scaling. These possibilities will be discussed in the revised version of the paper.\n\n*\"I found the linear regression experiments very confusing. What is \"the interaction between log parameters and log training tokens\"? What does t(52) represent (I suspect 52 is the degree of freedom, but where does it come from)? As it is, I was not able to understand the takeaways from this section.”*\n\nWe thank the reviewer for their comment. By the “interaction between log parameters and log training tokens\", we mean that in addition to fitting parameters reflecting the relationship between the number of parameters (log-transformed) and accuracy on the one hand and the number of training tokens (log-transformed) and accuracy on the other, the linear regression also fit an additional parameter reflecting the combination of the two, i.e., whether there is a difference in the relationship between the number of training tokens and accuracy depending on the number of parameters.\n\n“t(52)” indicates there are 52 degrees of freedom for the purposes of carrying out the t-test. Degrees of freedom for t-tests are calculated as the number of observations (7 models * 8 checkpoints = 56) - number of predictors (number of parameters + number of training tokens + interaction between the two = 3) - 1 = 52 (see, e.g., https://www3.nd.edu/~rwilliam/stats2/l02.pdf).\n\n*\"The original Inverse Scaling paper (McKenzie et al., 2013) actually reports model performance over FLOPs, which is a function of the number of training tokens. While the arxiv paper is contemporary, this paper would benefit from discussion of those findings as it is heavily built off this work.”*\n\nWe thank the reviewer for noting this. The McKenzie et al. (2023) paper appears to have been put on arXiv a day before the abstract deadline (and therefore just over a week before the paper deadline), and so we were not aware of this at the time of submission. Our revised draft will engage directly with their findings, which we hope will provide a useful comparison and a starting point for our analysis of model performance performance at tasks they do not investigate (e.g. TruthfulQA).\n\n* *\"Line 221: I think this sentence should be the start of the next paragraph.”*\n\nWe thank the reviewer for this suggestion and will edit the manuscript accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper tackles a timely, interesting research question and will spark more interest in the phenomenon of inverse scaling. It may also encourage model developers to publicly release checkpoints (which is not common practice) for the study of training dynamics.", "reasons_to_reject": "With the limited experimental setup, it is hard to draw any reliable takeaways.\n- The authors only study 4 of 11 Inverse Scaling tasks, and do not include other datasets that have demonstrated inverse scaling (e.g., TruthfulQA). With only 4 datasets, it is hard to draw conclusions about training dynamics more broadly, compared to task-specific phenomena.\n- The 4 datasets do not demonstrate consistent trends over training, and the authors do not offer discussion of why two tasks in particular show inverse scaling, while the other two do not.\n- The largest model studied has 6.9B parameters, while popular LMs these days have reached a scale two orders of magnitude larger. For instance, for some of the Inverse Scaling tasks (like Sig Figs and Into the Unknown), performance does not begin deteriorating until models hit 7B parameters. While such large models are usually not publicly available (and would require significant compute to train from scratch), it does limit the relevance of this paper's study into today's LMs.\n- The authors leave many questions open which would make the paper a more complete investigation. For instance, how do trends over training relate to trends over parameter count? There is speculation that different inverse scaling behavior has different underlying causes (e.g, strong priors, or misleading few-shot examples); would these reflect differently in training dynamics?", "questions_to_authors": "I found the linear regression experiments very confusing. What is \"the interaction between log parameters and log training tokens\"? What does t(52) represent (I suspect 52 is the degree of freedom, but where does it come from)? As it is, I was not able to understand the takeaways from this section.", "paper_topic_and_main_contributions": "There are some tasks known to the NLP community as demonstrating inverse scaling, meaning that performance decreases with greater model scale. These tasks include TruthfulQA and the 11 tasks collected in the Inverse Scaling Prize. However, the community has mainly focused on how performance on these tasks change with model size. This paper instead investigates how performance on 4 of the Inverse Scaling tasks change *over the course of training*. They find that 2 of the 4 tasks demonstrate \"inverse scaling.\"", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 221: I think this sentence should be the start of the next paragraph."}, "extra_text": {"Missing_References": "The original Inverse Scaling paper (McKenzie et al., 2013) actually reports model performance over FLOPs, which is a function of the number of training tokens. While the arxiv paper is contemporary, this paper would benefit from discussion of those findings as it is heavily built off this work."}}, {"review_id": "LZ3QWMHGuY", "reviewer": "Reviewer_qpm9", "summary": "", "questions": "A) Why not look at tasks which don't display inverse scaling with respect to model size? Is it possible that these could show inverse scaling with respect to training data size?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Responses to specific comments are given below.\n\n*“Although this is a short paper, the analysis is a bit limited. The authors only present performance on four tasks, and only two of these tasks were found to show inverse scaling with respect to the amount of training data. While this is a promising start, the paper could have provided a broader analysis of the phenomenon that would allow for stronger conclusions. For example, the authors could have experimented on more tasks, particularly emergent tasks (e.g. Wei et al., 2022), to determine the extent of data-related inverse scaling on tasks that LLMs are popularly benchmarked on.”*\n\nWe agree with the reviewer on this point. The revised version of the paper will include analysis of a wider range of tasks. As noted in the response to anonymous reviewer K1CC, we ran a new analysis on all 10 of the multiple-choice Inverse Scaling tasks and both versions of the multiple-choice TruthfulQA task on a subset of the models and checkpoints (models: 70m, 2.8b and 12b; checkpoints: 4000, 64000, and 143000 (final)) . We found evidence of inverse scaling on the Resisting Correction, Repetitive Algebra, Redefine, Pattern Match Suppression, Memo Trap, and Hindsight Neglect tasks of the Inverse Scaling Prize, and both TruthfulQA tasks. This directly informs the extent of inverse scaling that occurs over training in popular tasks. \n\n*“Additionally, it's unclear from the paper whether model size-related inverse scaling occurs independently of data size-related inverse scaling. The paper only focuses on tasks that already display model size-related inverse scaling. But since the data size aspect has not been extensively studied, the paper should either justify why only those tasks were examined, or for the sake of completeness, include tasks that have not yet shown model size-related inverse scaling.”*\n\n*“A) Why not look at tasks which don't display inverse scaling with respect to model size? Is it possible that these could show inverse scaling with respect to training data size?”*\n\nAt the time of writing, the results of the Inverse Scaling Prize challenge only showed evidence of inverse scaling as a function of model parameters. The aim of this study was to test whether there was any evidence at all of inverse scaling over the course of training, and so we selected tasks on which language models had been observed to show inverse scaling of any form as a starting point. In the revised version of the paper, we will discuss the results on the remaining Inverse Scaling tasks and Truthful QA, where, as might be expected given how the tasks were selected, we see the strongest inverse scaling over training for the largest models. We agree with the reviewer that investigating whether the two types of inverse scaling—inverse scaling as a function of number of patterns and inverse scaling as a function of number of training tokens—occur independently would indeed be a valuable direction for future research, and that the emergent tasks reported by Wei et al. (2022) would be a good place to start.\n\n* *'Line 33: Duplicate \"where\"'*\n* *'Line 58: \"specter\" should be \"spectrum\"'*\n* *'Line 282: \"section 1\" should be capitalized'*\n* *'Line 337: \"section 4\" should be capitalized'*\n\nWe thank their reviewer for noting these.\n\n*“Presentation suggestion: It could be beneficial to do a brief qualitative analysis of the model outputs. This is particularly important for larger models (> 1B), since these can produce acceptable and correct answers from a human perspective, but be automatically evaluated as wrong because they do not adhere to the specific output format of the task. This analysis could have been done for the model checkpoints that showed a significant sudden performance drop, to make sure that the larger models aren't actually doing better than the numerical scores make it seem.”*\n\nWe agree that this is a valuable way to holistically evaluate the output of language models, and indeed may reduce under-estimation of their capabilities. However, the tasks investigated in the present study are multiple-choice tasks, and thus the way in which the models are evaluated is not through free generation, but by comparing the relative probabilities of pre-defined continuations or answers. This means that only the probabilities of the correct vs. incorrect generations are taken into account. However, we thank the reviewer for noting this, as it is something important to consider for other forms of evaluation task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic of scaling and whether a solved task \"stays solved\" during the course of pretraining is important, particularly since the current trend is to train models on increasingly large amounts of data. The experiments cover an extensive range of LLM sizes (8 models ranging from 70M to 12B parameters), using an architecture similar to GPT-3, which make this work a valuable foundation for future research. The authors make several important observations, namely 1) some tasks do indeed show inverse scaling as training data size increases, and 2) smaller models have less inverse scaling behavior than larger models. These observations should be investigated further in future works, as they might lead to more insights into LLMs. In particular, it may be the case that some models are \"overtrained\" and that less pretraining data is in fact more optimal for certain tasks.", "reasons_to_reject": "Although this is a short paper, the analysis is a bit limited. The authors only present performance on four tasks, and only two of these tasks were found to show inverse scaling with respect to the amount of training data. While this is a promising start, the paper could have provided a broader analysis of the phenomenon that would allow for stronger conclusions. For example, the authors could have experimented on more tasks, particularly emergent tasks (e.g. Wei et al., 2022), to determine the extent of data-related inverse scaling on tasks that LLMs are popularly benchmarked on.\n\nAdditionally, it's unclear from the paper whether model size-related inverse scaling occurs independently of data size-related inverse scaling. The paper only focuses on tasks that already display model size-related inverse scaling. But since the data size aspect has not been extensively studied, the paper should either justify why only those tasks were examined, or for the sake of completeness, include tasks that have not yet shown model size-related inverse scaling.", "questions_to_authors": "A) Why not look at tasks which don't display inverse scaling with respect to model size? Is it possible that these could show inverse scaling with respect to training data size?", "paper_topic_and_main_contributions": "This paper examines the phenomenon of inverse scaling, which is when an LLM's performance on certain tasks decreases as model size grows. This challenges the current observation trend that LLM performance generally increases as model and data size increase.\n\nWhile previous works have focused on inverse scaling with respect to model size, this paper investigates whether inverse scaling also occurs as training data size increases (i.e. during the course of pretraining itself, when model size is fixed). The authors experiment with the models of the Pythia suite, a collection of 8 LLMs of various sizes with a similar architecture to GPT-3. Each model size of the Pythia suite comes in various checkpoints, each representing a cutoff point after pretraining on a certain amount of tokens. The authors test the models on a set of 4 tasks, which have previously displayed inverse scaling with respect to model size, to investigate whether these tasks also display inverse scaling with respect to data size. They find that 2 of the 4 tasks show inverse scaling with respect to training data size.\n\nThe paper contributes with the following findings:\n\n1) Inverse scaling with respect to data size can occur, at least for tasks that already display model size-related inverse scaling.\n2) Smaller models display less of this phenomenon than larger models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n- Line 33: Duplicate \"where\"\n- Line 58: \"specter\" should be \"spectrum\"\n- Line 282: \"section 1\" should be capitalized\n- Line 337: \"section 4\" should be capitalized\n\nPresentation suggestion:\nIt could be beneficial to do a brief qualitative analysis of the model outputs. This is particularly important for larger models (> 1B), since these can produce acceptable and correct answers from a human perspective, but be automatically evaluated as wrong because they do not adhere to the specific output format of the task. This analysis could have been done for the model checkpoints that showed a significant sudden performance drop, to make sure that the larger models aren't actually doing better than the numerical scores make it seem."}}, {"review_id": "vcUiuqII62", "reviewer": "Reviewer_7HVU", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. Responses to specific comments are given below.\n\n*“The contribution is too limited. The paper only took a pre-trained model family and evaluated them on 4 existing datasets.”*\n\nWe appreciate that the reviewer found the finding interesting and novel and agree that the scope of the experiments limits its potential impact. For this reason, we have carried out the same analyses on the 6 remaining multiple-choice Inverse Scaling and TruthfulQA tasks. For the 12-billion-parameter model, we see inverse scaling for both multiple-choice TruthfulQA tasks, as well as the Resisting Correction, Repetitive Algebra, Redefine, Pattern Match Suppression, Memo Trap, and Hindsight Neglect tasks of the Inverse Scaling Prize\n\nAs for the concern about using pre-trained models, we find models like these to be frequently used in impactful work to address research questions. In this case, these particular pre-trained models effectively serve the purpose we require of them, which we believe would make training models of a similar scale an unnecessary use of resources. \n\n*“No in-depth analysis. The authors found inverse scaling happens over compute, but why? It would make the paper much more solid if the authors can provide some analysis explaining such training dynamics.”*\n\nWe agree that this is indeed a limitation of the original submission and thank the reviewer for pointing it out. As part of the process of remedying this, we have incorporated the feedback of all three reviewers to carry out the same analysis on wider range of tasks, including the remaining 6 multiple-choice Inverse Scaling tasks, TruthfulQA, and other tasks for which inverse scaling has not yet been reported. Based on our previously-mentioned new results, we do see a relatively clear pattern that, of the tasks tested, all the tasks that McKenzie et al. (2023) consider to fall under the “Strong Prior” and “Spurious Few-Shot” type exhibit inverse scaling, but the same is not true of the one “Unwanted Imitation” task (Modus Tollens), and is only true of one of the “Distractor Task” tasks (Pattern Match Suppression). This will be discussed in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The finding is interesting and novel. This paper provides another perspective of inverse scaling: performance decreases with training compute, besides previous findings about inverse scaling with model sizes. \n\n- The paper is generally well-written and easy-to-understand.", "reasons_to_reject": "- The contribution is too limited. The paper only took a pre-trained model family and evaluated them on 4 existing datasets. \n\n- No in-depth analysis. The authors found inverse scaling happens over compute, but why? It would make the paper much more solid if the authors can provide some analysis explaining such training dynamics.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Language model performance ($y$) typically improves with increased model size ($m$) and training token amount ($t$) --- a concept known as normal scaling. Recent studies reveal that sometimes, $y$ declines with $m$, particularly in tasks such as negation QA, which is referred to as inverse scaling. This paper reveals $y$ also declines with $t$ dimension, by evaluating the Pythia model family across four tasks that are previously demonstrated inverse scaling over $m$ dimension.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies inverse scaling behavior over the course of pre-training using a series of Pythia models. They find that for a set of tasks that demonstrate inverse scaling laws (performance decreases as model size increases), models exhibit a decreasing performance over of the course of training. Reviewers have requested more extensive experiments to support the claims made in the paper and authors provide more comprehensive results and discussions in the rebuttal. Thus I recommend acceptance of this paper.
Decision: Accept-Findings