Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws
Authors: Zeyuan Allen-Zhu
Yuanzhi Li
Venue: ICLR 2025 Spotlight (2025)
Keywords: scaling laws
knowledge capacity
language models
Primary area: foundation or frontier models, including LLMs

Abstract: Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate information-theoretically the number of knowledge \emph{bits} a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store \emph{2 bits of knowledge per parameter, even when quantized to int8}, and such knowledge can be flexibly extracted for downstream applications. 

More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model's knowledge storage capacity.

Published as a conference paper at ICLR 2025
PHYSICS OF LANGUAGE MODELS: PART 3.3, KNOWLEDGE CAPACITY SCALING LAWS
[EXTENDED ABSTRACT]*
Zeyuan Allen-Zhu
FAIR at Meta teyuanalllezhuu@met... com
Yuanzhi Li
Mohamed bin Zayed University of AI Yuanzhi Li@mbzua ac ae
ABSTRACT
Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate model's capability via loss or benchmarks, we estimate anformatiinntthooetiical bits model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to in18, and such knowledge can be flexibly extracted for downstream applications. More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model's knowledge storage capacity.
the number of knowledge
INTRODUCTION
The scaling laws of large language models remain a pivotal area of research, enabling predictions about the performance of extremely large models through experiments with smaller ones. On the training time aspect, established scaling laws (Hoffmann et al. 2022; Kaplan et al. 2020 Hernandez eet al. 2021 Alabdulmohsin et al., 2022; Henighan al. 2020) discuss the optimal training flops versus model size. However, recent studies (Muennighoff et al., 2023 Gunasekar et al., 2023; Li al. 2023) challenge these laws, demonstrating that training smaller models with significantly more flops can yield superior results. While these laws talk about how much time/data is needed to train a model of a certain size, another fundamental question is: what the ultimate performance model can achieve, assuming sufficient training? Despite the known emergent behaviors in large models (Bubeck et al. 20233 Yu al. 2023), or even qualitative arguments that modern large models have reached 1.2 or L3vle intelligence (Allen-Zhu & Xu, 2025), there is a lack of principled, quantitative analysis on how model size impacts its capacity when adequately trained."
Traditional theory overparameterrzation suggests that scaling up model size in sufficiently trained models can enhance memorization of training data (AllenZZhu et al. 2019b), improve generalization
"This paper is part of the Physics of Language Models series, one of the first six papers presented as a Full and future editions of Part 3.3,
two-hour tutorial at ICML 2024 in Austria (youtu. including additional experiments and potential code releases, are available at pphysics allenn enuzhu..... com and ssrn. / abstract=52505s
'There is a rich literature comparing how pretrained models perform on benchmark tasks. Most comparisons for different model families trained over different data: if LLaMA-70B better than Mistral-7B, does the gain come from its choice pretrain data, or the architecture difference, or really the size the model? Some comparisons are among the same architecture, such as LLaMA- 70B scores 63.6% on the world knowledge benchmark while LLaMA-7B scores only 48.9% (Touvron et al. 2023b); does this mean increasing model size by 10x increases its capacity only to 130% = 63.6/48.9? Thus, is highly important to use more principled framework to study scaling laws in controlled setting.

---

Published as a conference paper at ICLR 2025
error (Hestness al., 2017: Rosenfeld 2021;
Rosenfeld al. 2019), and better fit complex target functions (Li & Liang, 2018 Allen- et al. 2019a). However, these results often overlook large constant polynomial factors, leading to significant discrepancy from practical outcomes.
In this paper, we introduce a principled framework to examine highly accurate scaling laws concern- ing model size versus its knowledge storage capacity. It is intuitive that larger language models can store more knowledge, but does the total knowledge scale linearly with the model's I's size? What is the exact constant of this scaling? Understanding this constant is erucial for assessing the efficiency of transformer models in knowledge storage and how various factors (e.g., architecture, quantization, training duration. etc.) influence this capacity.
Knowledge is a, if not the, pivotal component of human intelligence, accumulated over our extensive history. Large language models like GPT-4 are celebrated not just for their sophisticated logic but also for their superior knowledge base Despite rumors of GPT-4 having over IT parameters, is necessary to store all human knowledge? Could 10B model, if trained sufficiently with high- quality data, match GPT-4's knowledge capacity? Our paper seeks to address these questions.
Knowledge Pieces. Defining "one piece human knowledge" precisely is challenging. This paper aims to make progress by focusing on restricted, yet sufficiently interesting domain. We define piece of knowledge as (name, attribute, value) tuple, e.g.. (Anya Forger, birthday, 10/2/1996); and many data in world knowledge benchmarks can be broken down into pieces like this?"
We generate synthetic knowledge-oni datasets by uniformly random generating (name, attribute, value) tuples from knowledge base and converting them into English descriptions. We pretrain language models (e.g., GPT-2, LLaMA, Mistral) on these texts using a standard auto-regressive objective from random initialization, and "estimate" the learned knowledge. By varying the number of knowledge pieces and model sizes, we outline a knowledge capacity scaling law.
Our idealized setting, free from irrelevant data, allows for more accurate scaling law computations we also discuss how "junk" data affects capacity. In contrast, it is difficult to quantify real- life knowledge; for instance, if LLaMA-70B outperforms LLaMA-7B by 30%
benchmark, it doesn't 't necessarily mean a tenfold model scaling only boosts capacity by 30% (see Footnote 1). The synthetic setting also lets adjust various hyperparameters, like name/value lengths and vocabulary size, to study their effects on knowledge capacity scaling laws.
Most of the paper shall focus on setting with synthetically-generaae human biographies as data, either using predefined sentence templates or LLaMA2-generated biographies for realism.
Bit Complexity and Capacity Ratio. For N knowledge pieces (i.e., N tuples), define the bit complexity as the minimum bits required to encode these tuples. For any language model trained on this data, calculate its "bit complexity lower bound" (see Theorem 3),,, describing the minimum number of bits needed for the model to store the knowledge at its given accuracy. This formula is nearly as precise as the upper bound, within 1 o(1) factor.
We train language models of varying sizes on knowledge data with different N values. By compar- ing the models' trainable parameters to the bit complexity lower bounds, we evaluate their knowl- edge storage efficiency. A model with 100M parameters storing 220M bits of knowledge has capacity ratio of 2.2 bits per parameter.
Our results.
Our findings are summarized as follows:
RESULTS 1-3: BASE SCALING LAW FOR GPT2. 3
RESULT 1+2+3 GPT2, trained with standard AdamW, consistently achieves 2bit/param capacity ratio across all data settings after sufficient training. This includes various model sizes, depths, widths, data sizes, types name/value length, attribute number, value diversity).
hetpeeeiiissmmmsyytttttteeeeee and hyperparameters (c.g.,
Examples include (Africa. largest country, Sudan) and (It Happened One Night, director, Frank Capra) in TriviaQA (Joshi et al. 2017), (Teton Dam, collapse date, 06/05/1976) and (USA, Capital, Washington D.C.) Natural Questions (K wiatkowwkii et al. 2019).
In this paper, GPT2 refers to that the GPT2 model with rotary emhedding instead of positional embedding and without dropout.
2

---

Published as a conference paper at ICLR 2025
20
10
18B
18: parmm
0.S.1 12 222E
05 BJ parm
learned knowledge (bits)
0.25 BT. parm))
learmed knowteddee (bib))
0.5 parmn
DNTERICNE
Am20032200
1-11003300
W-1000000
hw2293322
km11203026
Av-330255
10
199322
Bu-Ca000
model (#parama)
(a) (N) data (1000 exposures), peak R(F) 2 2 (b) (N) data (100 exposures), peak R(F) 2
Figure 1: Scaling laws for GPT2 pretrained on bioS(N) data using fp16 (mixed-precissoon) for 1000/100
exposures.
Conclusion. The peak capacity ratios consistently exceed R(F): 2 (resp. 2 1) for 1000 exposures (resp. 100 exposures) of pretraining on each knowledge piece, regardless of model depth/size.
Remarks. Each dot é-h represents GPT2 with l layers, h heads. and 64d dimensions. The learned knowledge is calculated by the bit-complexity lower bound Theorem 3.1 The full paper also includes: similar results for bioSsimple (N) and (io)(N) data, the same holds for quantization using int8, and confirming full extractability of all learned knowledge
Larger models? Training GPT2-20-16 on bioS(10M)) for 1000 exposures costs 8.5 days with 64 A 100s, while GPT2-12-32 on bioS(20M)) for 100 exposures took 2.4 days. In synthetic setting, we see no need scale up further. Instead, we prefer to allocate GPUs to explore other aspects covered in this paper.
Remark 1.1. This predicts a sufficiently trained 7B language model can store 14B bits of knowledge, surpassing the knowledge of English Wikipedia and textbooks by our estimation 4 Remark 1.2.
When we say the model stores knowledge, it isn't word-by-word memorization. Instead, the knowledge is flexibly extractable (e.g., via QAs like "What is Anya Forger's birth- day") (Allen-Zhu & Li. (024) and applicable downstream tasks (e.g., comparing birthdays) via fine-tune (Allen-Zhu & Li 2025).
RESULT 4: HOW TRAINING TIME AFFECTS MODEL CAPACITY.
Achieving 2bit/param capacity requires each knowledge piece be visited 1000 times during training, termed 1000-exposure to differentiate from traditional "1000-pass" terminology, as a single data pass can expose knowledge piece 1000 times." 6
RESULT 4. With 100 exposures, an undertrained GPT2's capacity ratio falls to 1bit/param. (See Figure 1)
Remark 1.3. Another perspective on Result 4 that rare knowledge, encountered only 100 times during training, is stored at bit/param ratio.
RESULTS 5-7: How MODEL ARCHITECTURE AFFECTS MODEL CAPACITY.
We tested LLaMA, Mistral, and GPT2 architectures with reduced or even no MLP layers.
*As of February 1, 2024, English Wikipedia contains a total of 4.5 billion words, see https: 7/en. www.iediaaiia..mmmmmmmmm org/ w wikii/i / wttp//iww/....ee....... wii.iiaii:a::s. Size of_ Wikiiiiieaa FrikleedattSzee sataatttii____ehheeeeeee_eeeeeee the of_the__ng__________e___________ Eng-ist. Mikipeaia dat abase accessed March 2024. We estimate that the non-overlapping contents of English textbooks have fewer than billion words in total, see Remark P.I. This amounts to 20.5 billion words, and knowledge.
we helieve they contain fewer than 14 hillion
"A distinction exists between memorizable knowledge (c.g., text memorized during pretraining) and knowl- edge flexibly extractable via instruction fine-tuning (Allen-Zhu & Li 2024); results in this paper apply both.
Foor example, is plausible that one pass through Wiki data might present the knowledge piece (US, capital, Washington D.C.) 1000 times, and pass through the Common Crawl might present million times.
3

---

Published as a conference paper at ICLR 2025
RESULT 5: In the 1000-exposure setting, 2bit/param capacity ratio appears to be univer- sal rule: all models, even without MLP layers, closely achieve this ratio.
RESULT 6) With 100 exposures, some archs show limitations; notably, LLaMA/Mistrals
capacity ratio is 1.3x lower than GPT2's, even after best-tuned learning rates.
RESULT 7: Further controlled experiments indicate that "gated MLP" usage leads to LLaMA/Mistral architecture's 's underperformance in knowledge storage.
Remark 1.4. Our framework offers principled playground to compare models. This con- trasts with traditional comparisons based on loss/perplexity, which can produce debatable con- clusions. Controlled data also reveal more significant differences between models"
RESULT 8: How QUANTIZATION AFFECTS MODEL CAPACITY.
We applied GPTQ (Frantar et al., 2022) to quantize models from the base scaling laws to int8 or int4. Surprisingly,
RESULT 8 Quantizing to int8 does compromise model capacity (even for models on the boundary of 2bit/param)) however, quantizing to int4 reduces capacity to 0. 0.7bit/param.
Remark 1.5. Since int8 is 8bit, LLMs exceed 1/4 of the theoretical limit for storing knowl- edge; thus knowledge must be very compactly stored inside the model across all layers.
Remark 1.6. Since 2bit/param is obtained after sufficient training, training longer may not further improve model capacity, but quantization
While covered in this paper, our framework also provides principled playground to compare different quantization methods.
RESULT 9: How SPARSITY (MoE) AFFECTS MODEL CAPACITY.
Mixture-of-experts (MoE) models offer faster inference than dense models but often underper- form dense models with the same total parameter count (not effective parameters). We show that this performance drop is likely not due to a lack of knowledge storage capability.
RESULT 9, MoE models, even with 32 experts, only reduce 1.3x in capacity compared to the base scaling laws, despite using just 8.8% of the total parameters during inference.
RESULTS 10-12 How JUNK KNOWLEDGE AFFECTS MODEL CAPACITY.
Not all pretrain data are equally useful. Much of the internet data lacks valuable knowledge for training language models (Li et al., 2023), while knowledge-rich sources like Wikipedia represent only a small fraction of the training tokens. We explore the impact on model capacity by conducting a controlled experiment with both useful and "junk" data.
RESULT 10 11; Junk data significantly reduces model capacity. As an example, with a 1:7 ratio of "useful junk" training tokens, capacity for useful knowledge loses by factor of 20x, even when useful knowledge is exposed 100 times? 9
- -REUULT 12 An effective mitigation is to prepend a special token to all useful knowledge. This is akin to adding domain name like wikipedia.on org the start of every Wikipedia paragraph; the model autonomously identifies high-quality data without prior knowledge of valuable domains. In the example above, the loss factor improves from 20x 2x.
Conclusion. Overall, our approach to studying knowledge capacity scaling laws offers a flexible and more accurate playground compared to traditional methods that evaluate language models trained on internet data against real-world benchmarks. This accuracy is partly due to the synthetic nature of our dataset, which eliminates concerns such data contamination that could compromise the validity of real-world benchmark results. In this paper, we've 've conducted thorough comparison across different model architectures and types of knowledge. While we haven't 't explored various quantization methods, this represents promising direction for future research. We' ve also inves- tigated the impact of junk data and proposed mitigation strategies. We believe the insights gained from this principled exploration can assist practitioners in making informed decisions about model selection, training data preparation, and further theoretical research into LLMs.
A model might achieve better perplexity by performing much better on simpler data but poorer on complex data, or by excelling reasoning but not in knowledge. Our results offer a more nuanced view: GatedML.P doesn't 't affect frequent knowledge but does impact moderately rare knowledge (c.g., with 100 exposures). 1% accuracy boost benchmark tasks; findings of 1.3x difference translates for instance to accuracies 90% vs. 70%.
$ For example, Shazeer (2020) found GatedMI.P offers
9The loss factor improves to 3x/1_5x/1.x 1xx with 300/600/100 100 exposures of useful knowledge, compared to Result 4 which involves training without junk for only 100 exposures.
4

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES
In this paper, a piece of knowledge is a tuple of three strings: (name, attribute, value) = (n, a, v). For instance, n "Anya", a "birthday", "Oct 2, 1996".
2.1 KNOWLEDGE (THEORETICAL SETTING)
The complexity of knowledge set is determined not only by the number of knowledge pieces but also by the length of the value string v, the diversity of the vocabulary, and other factors. For instance, if the attribute a "passport number," then the value v contains more bits of knowledge compared with = "genderr," because the former has significantly higher diversity. If the attribute "birth date," then the value could consist of chunks: (10, 1996).
Considering these examples, we propose a set of hyperparameters that may influence the complexity of knowledge:
1. N the number of (distinct) names denoted by M.
2. K the number of attributes a, with A representing the set of attributes. For simplicity, we assume |A K is fixed.
3. T the number of tokens T. where every character in v belongs to 'T for some |T| = T.
For example,
think of I'' as "vocab size" in tokenizer.
4. C' and I, the number of chunks and the length of each chunk for the value: each value u E (TL)C can be expressed
(V1, U2, ... Uc), where i; E T 5. the diversity of chunks: for each piece of knowledge (n, a, v) and E [C]. the chunk 1; belongs to D. CTL, for some set with cardinality D |Dal < T
Remark 2.1. For notation simplicity, we have assumed that all chunks within an attribute a E A more
share the same diversity set Da, and all chunks are of equal length, etc. This enables easily demonstrate the influence each hyperparameter on a model's capacity. In practice, different attributes may have different diversity or value lengths e.g., Passsort could be much larger than D gender. Our theoretical results do apply to these settings, albeit with more complex notation.
In our theoretical result, we introduce dataset bioD K, C, D, T) defined as follows:
Definition 2.2 (bioD data generation). Consider a fixed set of K attributes, such a set A = }, and a fixed set No of candidate names (with No def (Nol > N).
[ID .... "ID "ID K"
Generate N names uniformly at random (without replucement) from Ng to form N.
2. For each attribute E A. generate D distinct strings Wi,as' WD.a E T uniformly at random (without replacement) to form the diversity set Da-
3. For each name N and attribute A, generate value 8² (n, a) (V1, "*. sampling each U/
... ,...,,,,, ;t)) by
E Da uniformly at random.
Let Z del {(n, a, o² a)} aEN be the knowledge set.
Proposition 2.3 (trivial, bit complexity upper bound). Given No and and T, to describe a knowl- edge set generated in Definition 2.2 one needs at most the following number of bits:
log2 (loo)) + NKCloo, D K loga (5") = N
Wal N + NNC'llg D + KD KDlog T
D
(The approximation is valid when [ N and T >> D.) We will present bit complexity lower bound in Section 3.
2.2 KNOWLEDGE (EMPIRICAL SETTING)
We utilize both the synthetic bioD dataset, generated as per Definition 2.2, and several human biog- raphy datasets to evaluate language model scaling laws.
Allen-Zhu & Li (2024) introduced a synthetic biography dataset comprising N randomly-generian (fake) individuals, each characterized by six attributes: birth date, birth city, university, major, em-
5

---

Published as a conference paper at ICLR 2025
ployer, and working city 10 To translate these tuples into natural language, in their bioS dataset, each individual is described by six randomly selected English sentence templates corresponding to their attributes. We direct readers to their paper for more details but provide an illustration below:
Anya Briar Forger was born on October 2, 1996. She spent her early years in Princeton, NJ. She received mentorship and guidance from faculty members at Massachuserts Institute of Tochnology, completed education with focus on Communications. She had a professional role at Meta Platforms, She was employed in Mealo Park, CA.
(2.1)
In this paper, we explore three variations of such datasets:
(i) represents online dataset for N individuals, where each biography is generated with new randomness for the selection and ordering of six sentence templates on-the-ffy.
bioSsimple (N) denotes a similar dataset, but here, each biography is generated once with fixed random selection and ordering of the sentence templates.
(i) refers to the same dataset, but with each biography written 40 times by
LLaMA2 (Tovrron et al., 2023b) to increase realism and diversity.
These datasets correspond to the bioS permute, bioS pingle+ and bioR multi data types discussed in (Allen-Zhu & Li. 2024), albeit with minor differences. While their study focused on N = 100K, we expand our scope for bioS to consider N up to 20M; for bioR, we limit to 1M, which already yields dataset size of 22GB.
As introduced in Section if each knowledge piece is seen 1000 times during training, we call this 1000 exposures. For (IV), 1000 exposures will unlikely include identical biography data because there are 50 sentence templates for each attribute and a total of 50 x 6! possible biogra- phies per person. For siisee (N), 1000 exposures mean 1000 passes of the data. For (No)(N), 1000/100 exposures mean only 25/2.5 passes of the training data.
For the bioD dataset, we define No to be identical to bioS, with [Nol = 400 x 400 x 1000. We encapsulate person's 's attributes within single paragraph, employing random sentence orderings and a consistent sentence template. For example:
Anya Briar Forger's ID 7 is 17,1 07,1:...,,7.c v7,C. Her ID 2 is 02.1 ......... 22 [..]] Her ID 5 is vs.1. 05,0-
In this paper, we primarily utilize bioS. To illustrate broader applicability and to better connect to theoretical bounds, we also present results for bios simple bioR, and bioD.
2.3 MODELS AND TRAINING
GPT2 was introduced in (Radford et al., 2019). Due to its limitations from the absolute positional embedding (Allen-Zhu & Li. 2023), we adopt its rotary positional embedding variant (Su et al.) 2021 Black et al., 2022), which we still refer to as GPT2 for convenience. Additionally, we disable dropout, which has been shown improve performance in language models (Touvron et al. 2023b). We explore a wide range of model sizes while using a fixed dimension-per-heet of 64. The notation GPT2-G-h represents & layers, h heads, and 64h dimensions; for example, GPT2-small corresponds to GPT2-12-12. 12. The default GPT2Tokenizer is used, converting people's names and most attributes into tokens of variable lengths. In examining the impact of model architectures on scaling laws, we will also use LLaMA/Mistral architectures (Touvron et al., 2023a: Jiang et al. 2023).
We train language models from scratch (L.e,, random initialization) using the specified datasets. Knowledge paragraphs about individuals are randomly concatenated, separated by <EOS> tokens, and then randomly segmented into 512-token windows. The standard autoregressive loss is employed for training. Unless specified otherwise, training utilizes the default AdamW optimizer and mixed-precision fp16. Learning rates and weight decays are moderately tuned (see full paper).
Training.
10 All attributes, except for the working city (determined by the employer's headquarters), are chosen uni- formly and independently at random. There are No = 400 x 400 x 1000 possible person names, 12 x 28 200 birth dates, 200 birth cities, 300 universities, 100 majors, and 263 employers. Additionally, random pronoun with 2 possibilities is chosen for each person.

---

Published as a conference paper at ICLR 2025
3 BIT COMPLEXITY LOWER BOUND
When assessing the knowledge stored in a model, we cannot simply rely on the average, word-by- word cross-entropy loss. For example. the phrase "received mentorship and guidance from faculty members" in (2.1) ( does not constitute useful knowledge. We should instead focus on the sum of the loss for exactly the knowledge tokens.
Consider a model F with weight parameters W W. Assume F is trained on (ioD(N, K,C. C, D, L. T) dataset as defined in Definition 2.2) using any optimizer; this process is represented as W (2) (the model's weight is trained as function of the training dataset 2). During the evaluation phase, we express F' through two functions: F (W, R), which generates names, and FW a, Rt,, which generates values given (n, a), where R denotes the randomness used generation. Let (W n, a, It) represent the first chunk of F F- (2), n, a, R).. We evaluate F by calculating the following three cross-entropy losses!
losSname (2)
E nEW
log Pr R
F (2)
lossealuell (2)
E aEA
Pr log
FT (W n, R) un a)]
R
losSvalue (3)
del
E nEN aEE a
log Pr [ (W n, (2) R) 8" (n, a)]
ft.
We shall explain in the full paper that these quantities are easy to be derived from the auto-regressive entropy-loss using examples, and below we quickly state our bit-complexity lower bound theorem: Theorem 3.1 (bit complexity lower bound). Suppose N 2 Q(DloggN) We have
No loss
DC
loga [ 2
N Vloga
+ NK logg
2
Joss.atue (2)
D
KDlogg Dell+o losstoal)ls natuel (2)
KD log2
(KD)
No - N
DC
Nlog
NN log2
&B lossnume (2)
LE loss (2)
TL-D
KD Dolgg
o (KD)
De(1 + of Ez lass (2)
The goal of the paper is study how the number of model parameters competes with this bound. We defer the proof to the full paper. and shall explain over there why proving such bound is non- trivial.
4 CAPACITY RATIO
Motivated by Theorem 3.1. ignoring lower order terms, we define the empirical capacity ratio as
Definition 4.1. Given a model F with P parameters trained over bioD , K, C, D, I, T) dataset Z. suppose it gives pı lossnome (Z), p2 loss (2), P3 loss (Z), define its capacity ratio and max capacity ratio
N logg EPI
No NK log,
D ePI
KD KD log2 DePS
T
R(F)
P
Rmaa (F) N Vlogz
No N + NKClog, KD T D
P
Remark 4.2. One must have R(F) < (F), and equality is obtained if the model is perfect.
For a fixed dataset, further increases in model size do not yield additional knowledge, thus Rmax (F) approaches
the model size P increases. On the other hand, Theorem 3.1 implies, ignoring lower-order terms, that if the model parameters are 8-hit (such as int8), then R (F) S 8.
We use F., En denote uniform random selection of
A.

---

Published as a conference paper at ICLR 2025
For our bio data, define slightly reduced capacity ratio by omitting the diversity term
Definition 4.3. Given model F with P parameters trained over the bioS( N) dataset Z, suppose gives p1 = -Ossnamml (2) ( and p2 = losS (2), its capacity ratio
R(F) N 1ogg No +N TOR2 2 and Temax (F) Nog, +N log, Sp
for No- 400 400 1000 and So 2 x (12- 20-000) 200 300 x 100 x 263 (c.f. Footnote 10).
Remark 4.4. Ignoring names, each person contains (Sg) = 47.6 bits of knowledge.
5 MAIN BODY OF THIS PAPER
This paper contains too many technical results (12 in total), so many that it was initially rejected by NeurlPS 2024 since the paper "is very valuable to the developers and researchers of GPT-like LLMs" but "deserves a full hour to present than a focused conference paper (which only has a few minutes to present)", quoting the original words from the NeurIPS area chair (AC).
For this reason, we omit all technical details in this ICLR 2025 camera-ready version and encourage readers to refer to our full paper at ssrn. com/ hte//abstao abstract 5250617, to ICML 2024 We remark that the full paper underwent the ICLR 2025
tutorial at youtuu.uuuu be/ wtt.bbeeBBBBB J0kL7 review process, but elected to present this camera-ready version as extended abstract, aligning with the tradition in the theory community.
CONCLUSION
We investigated the scaling laws language models, specifically the relationship between model size and the total bits of knowledge stored. Our findings reveal precise, universal scaling law: a sufficiently-trained transformer (i.e., one whose training loss has plateau-ed) can store 2 bits of knowledge per parameter, even when quantized to int8, which only 1/4 away from the information-theoos
maximum. We also examined how these scaling laws are influenced by various hyperparameters, including training duration, model architectures, floating point precision, sparsity constraints like MoF, and data signal-noise ratios.
In terms of knowledge capacity, our methodology provides a more accurate and principled play- ground for comparing model architectures, training techniques, and data quality. We believe this playground can assist practitioners in making informed decisions about model selection, training data preparation, and further theoretical research into LLMs. Finally, our research represents an initial step towards addressing a fundamental question: how large does a language model need to be? We hope our findings will inspire further research in this area. Ultimately, aim provide a principled answer to the question, "Are language models with 1T parameters sufficient to achieve AGI?" in the future.
Finally, Part 3 of this work series focuses on how language models store, extract and manipulate knowledge (including Part 3.1 and 3.2 (Allen-Zhu & Li, 2024 2025)). We also cover grade-school math and reasoning in Part 2 (Ye et al. 2025a b), hierarchial language structure learning in Part 1 (Allen-Zhu & Li. 2023), and architecture design in Part 4 (Allen-Zhu, 2025).
REFERENCES
Ibrahim M Alabdulmobsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-2231 2022.
can A version of Theorem 3.1 can be proven for this dataset with simpler proof, excludes the diversity set. This could also mean the model full prior knowledge of the diversity set (c.g., assuming a fixed of 300 university names) without counting this knowledge towards its learned bits.
13 Here, one can let K. {birth date, birth city, university, major, employer, gender) and accordingly define
losSegluel (2) def FREN Enek - log Præ (W (W), m. R) x² (1,,a)]
8

---

Published as a conference paper at ICLR 2025
Zeyuan Aluen-Zhu. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. SSRN Electronic Journal, May 2025. doi: (0.2139/ssm.5540330.. https:
//ssrn. com/ abstractt
Zeyuan Allen- and Yuanzhi Li. Physics of Language Models: Part 1, Learning Hierarchical Lan- guage Structures. SSRN Electronic Journal, May 2023. https:/ ssrn. com/ abstract=
5250639
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. In Proceedings of the s/ International Conference on Machine Learning,
ICML 2024, 2024. Full version available at Pbttps://saeee
com/ /atmm/abtttttttt33333
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.2, Knowledge Manip- ulation. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025. Full version available at httptttss:/sss....tt.t /ww.. com// absstaactt55000
Zeyuan All and Xiaoli Xu. DOGE: Reforming Al Conferences and Towards a Future Civ- ilization of Fairness and Justice. SSRN Electronic Journal, February 2025, doi: 10.2 1399smmm 5127931. https::/ssttttee
com/ aataabatttttttt23330
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers. Advances neural information processing systems, 32, 2019a.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterizzatoon In International conference on machine learning, pp. 142-252. PMLR, 2019b.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT- NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Work- shop on Challenges & Perspectives in Creating Large Language Models, 2022. URL https: //arxiv.erg/aaaaa...... httpe//ab//22004 06745
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 1rXiv:2303.12712, 2023.
Hal Daumé III. Frustratingly easy domain adaptation. arXiv preprint arXiv:0907. 1815, 2009.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):
5232-5270, 2022.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17333, 2022
Olga Golovneva, Zeyuan Allen- Jason Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403.1377
2024.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorn, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprini arXiv:2306. 1644, 2023.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Danny Hermandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.0m 022.2.0...... 2021.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad,
Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint 171.0047,, 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.1 15556, 2022.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2021.

---

Published as a conference paper at ICLR 2025
Changho Hwang. Wei Cui, Yifan Xiong. Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yonggiang Xiong.
Tutel: Adaptive cfxpere- at scale. CoRR, abs/2206.03382, June 2022. URLhttps
pdf /2006 03382. pdff
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.
Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoye.. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.033: 03551.05,,,,. 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint S.KV::2001.0836... 2020.
Dmitrii Krasheninnikov, Egor Krasheninnikov, Bruno Mlodozeniec, and David Krueger. Meta- of-context) learning in neural networks. arXiv preprint Sttt::2311.1504F 2023,
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-465 463,466,222 2019.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent structured data. In Advances in Neural Information Processing Systems, 2018.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.0544 2023.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Noua- mane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv preprint arXiv:2305. 1626, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Jonathan Rosenfeld. Scaling laws for deep learning. arXiv preprint arXiv:2 07686, 2021.
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint arXiv: 12099,1267,,.. 2019.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.055: 200.0:00202..2002, 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2016.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer
with rotary position embedding, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 232,1 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2300.0994 092888, 2023b.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physies of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025a. Full version available at https: 7/ssrn. com
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physies of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems. In Proceedings of the 13th International Conference on Learning Representations, ICLR 2025, 2025b. Full version available at atttpps:/ //rn com/ abst ract==50633
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: A flexible and expandable family of evaluations for ai models. arXiv preprint arXiv:2310. 17567, 2023.
10


Reviews:
[{"review_id": "sjf86z98gr", "reviewer": "Reviewer_Z8yV", "summary": "The paper investigates the relationship between language model size and its capacity to store factual knowledge, quantified in bits per parameter. The authors introduce a framework that measures a model’s knowledge based on tuple-based information (e.g., (Entity, Relation, Attribute)) and propose that language models, after sufficient training, achieve an approximate capacity of 2 bits per parameter. The study extends this analysis across various factors such as model architecture, quantization levels, sparsity, and training data quality. Using synthetic and controlled datasets, the authors examine how these elements influence the knowledge storage capacity of language models.\n\nThe technical claims are generally supported by experiments; however, there are concerns about the robustness of the findings. The reliance on synthetic data raises questions about the applicability of the results to real-world scenarios. Additionally, the paper does not thoroughly explore the impact of quantization during training.\n\nThe paper has several formatting issues that hinder its readability. Notably, it lacks a conclusion section. Some figures are not clear. The organization of the paper could be improved drastically.", "questions": "1.\tHow would the proposed 2-bit/parameter scaling law hold up when applied to language models trained on real-world, diverse datasets?\n\n2.\tCould incorporating quantization into the training process mitigate the reduction in capacity observed with int4 quantization?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "•\tOriginality: Introducing a framework to measure language model capacity in bits per parameter is a novel approach that adds a quantitative dimension to model evaluation.\n\n•\tMethodology: The use of controlled synthetic datasets allows for the isolation of specific variables, providing clarity in the analysis of different factors affecting knowledge capacity.", "weaknesses": "•\tFormatting Issues: The absence of a conclusion section and unclear figures detract from the overall quality of the paper and impede the reader’s understanding.\n\n•\tGeneralization to Real-world Data: The heavy reliance on synthetic data limits the applicability of the findings to natural language processing tasks involving complex and diverse datasets.\n\n•\tIncomplete Exploration of Quantization: The paper does not investigate quantization during training, which could provide insights into mitigating the observed decrease in capacity with int4 quantization.\n\n•\tLimited Architectural Diversity: The study does not explore a wide range of model architectures, such as encoder-only or decoder-only models, which could affect the generalizability of the proposed scaling law.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kT32TK1TdD", "reviewer": "Reviewer_ZiZe", "summary": "The authors investigate how the size of language models influences their ability to store factual knowledge. Unlike previous studies that assess model capabilities through loss metrics or benchmarks, this research estimates the amount of knowledge from an information theory perspective. The authors find that an LLM can store about 2 bits of factual knowledge per parameter, meaning a 7B parameter model can hold approximately 14B bits of knowledge, covering Wikipedia-level information.\nOther influencing factors of the amount of stored information include: extended training, specific architectures (e.g., GPT-2 with rotary embeddings), and higher data quality. Precision reduction (e.g., int8) minimally impacts storage, while domain annotations (like labeling Wikipedia data) significantly boost capacity.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose a novel method to measure the knowledge stored in an LLM by a bit complexity lower bound, and show that the amount of information stored in a single parameter is approximately 2 bits.\n\n2. The authors investigate various influencing factors to the knowledge storage capacity of LLMs, which provide many practical insights to LLM training.", "weaknesses": "1. What do the long plateaus in the stored information mean? Does the maximum information reached by the plateaus equal to the bit complexity upper bound? The main scaling law only describes the linear increasing part, not the plateaus part.\n\n2. The scaling law seems to be incomplete. i.e. the paper describes the influencing factors to the stored knowledge separately. Is there an empirical law that can describe and summarize all the results?\n\n3. What does it mean if we say that the LLM stores N bits of knowledge? Does it mean this amount of knowledge is ready for extraction or just memorized in a fixed form? There are some memorization v.s. extraction accuracy plots in the appendix, which seem also show a first linearly increase and then plateau trend, but these results are not well associated with the bit information results. e.g. Would the bits of knowledge stored linearly be associated with the knowledge memorization/extraction accuracy? If the meaning of this information theoretical definition of storage knowledge is not well understood, it is hard to interpret the insights obtained from it.\n\n4. The derivation of the bit complexity lower bound feels a bit ad hoc. It seems that the authors just tried to build a lower bound in a similar form to the upper bound involving the training losses. Some steps in the derivation do not feel natural to me, especially lemma 1.6. What is the meaning of the RHS of equation I.1? How tight would this lower bound be? \n\n5. The training data generation process is basically uniformly random. There are also similar uniformly random assumptions used in the proof of the bit information lower bound. I wonder if the derived lower bound can be applied to real-world data, which is not uniform, but usually naturally exhibits a long-tailed distribution. I understand that certain assumptions are needed for synthetic experiments, but it is also important to understand how much the obtained conclusions can be applied to real-world applications.\n\n6. The writing can be improved.\n\nOverall, I appreciate the information theory approach adopted and the empirical results seem to be quite inspiring. \nI'm willing to improve my score if my concerns are properly addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z3BXU9oUmO", "reviewer": "Reviewer_e6BX", "summary": "This paper examines scaling laws concerning model size versus its knowledge storage capacity. The paper addresses the following research questions: \n- Does the knowledge scale linearly with  model size, and what is the exact constant of this scaling?\n- How does training affect model capacity? \n- How does  model architecture relate to model capacity?\n- How do quantization and model sparsity affect model capacity?\n- How does irrelevant/noisy data affect model capacity?\n\nUnlike prior studies that evaluate a model’s capability via loss\nor benchmarks, this paper  estimates  the number of knowledge bits a model store, focusing  on factual knowledge represented as tuples, such\nas (USA, capital, Washington D.C.) from a Wikipedia page. Experimental results across model architectures establish that language models can only store 2\nbits of knowledge per parameter. Detailed analyses further illustrated that a sufficiently trained 7B language model can store 14B bits of knowledge. Achieving 2bits per parameter capacity requires each knowledge piece to be revisited 1000 times. Quantizing to int8 does not compromise model capacity,  however, quantizing to int4 reduces capacity to 0.7bit/param.\nMixture-of-experts (MoE)  only reduce 1.3x in capacity,  despite using just 8.8% of the total parameters during inference.\nFinally, noisy data significantly reduces model capacity but an  effective mitigation is to prepend a special token to all useful knowledge.\nThe model autonomously identifies high-quality data without prior knowledge of\nvaluable domains.", "questions": "- Please explain why you selected GPT2, Llama, and Mistral model families for your experiments. Although I understand that these represent the most popular model families, it would have been interesting to see how other, less similar architectures fare like S4. \n\n- I would have been good to mention something about the computational requirements for your experiments. \n\n- Aside from knowledge-base tuples, are there any other areas you think your framework might be applied to? \n\n- I would have liked to see explicit suggestions to researchers pertaining or using LLMs. It seems that you are saying that the\nquality of the data and the times of exposure matter a lot more than differences in architecture.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Scaling laws are of interest to the community as they allow us to analyze and understand the capacity of large language models. They also enable the design of new models and pertaining experiments. The authors claim they are the first to propose a scaling law for the knowledge capacity of LLMs. \n\n- The paper contains a very thorough and exhaustive list of experiments, answering several questions. \n\n- There are practical recommendations about LLM model builders, such as domain tagging for pertaining data. \n\n- the paper provides an explanation as to why Quantized models perform on part to their non-quantized variants in terms of knowledge storage, and similarly why  mixture of experts models perform decently despite being sparse. \n\n- Finally, it reveals specific architectural choices in models like Llama and Mistral which might lead to slightly inferior performance (e.g., weight tying, MLP layers). \n\n- The experimental framework is reproducible, and although the authors focus on binary knowledge base tuples, it could be extended to of the knowledge-based facts or event language structures.", "weaknesses": "- the paper is very dense, the appendix is very many pages long.  As a result, it is not easy to absorb all important details. \n\n- the graphs are in very small scale and are not explained appropriately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LnMaifsvja", "reviewer": "Reviewer_aRy5", "summary": "This paper studies knowledge capacity scaling laws for language models. The authors design a set of novel and solid evaluation methods to assess the knowledge capacity ratio of language models. The evaluation covers different model architectures from dense to sparse LLms as well as different settings such as quantization and data noise, making it a complete and solid paper.", "questions": "Current I do not have questions on this paper.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed knowledge capacity evaluation is novel and useful in practical applications\n2. The experiments are solid and comprehensive", "weaknesses": "One limitation could be the readability of this paper. Some figures are not visible such as result 8. The fonts in some figures such as Figure 1 and 3 are also too small to read. I would suggest the authors remove some sections such as result 8 to appendix, therefore more space could be left for more detailed explanation on other sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)