Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zeyuan Allen-Zhu
authors: Yuanzhi Li
reviews: 
Comments: This paper studies knowledge capacity scaling laws for language models. The authors design a set of novel and solid evaluation methods to assess the knowledge capacity ratio of language models. The evaluation covers different model architectures from dense to sparse LLms as well as different settings such as quantization and data noise, making it a complete and solid paper.
Strengths: 1. The proposed knowledge capacity evaluation is novel and useful in practical applications
2. The experiments are solid and comprehensive
Weaknesses: One limitation could be the readability of this paper. Some figures are not visible such as result 8. The fonts in some figures such as Figure 1 and 3 are also too small to read. I would suggest the authors remove some sections such as result 8 to appendix, therefore more space could be left for more detailed explanation on other sections.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: This paper examines scaling laws concerning model size versus its knowledge storage capacity. The paper addresses the following research questions: 
- Does the knowledge scale linearly with  model size, and what is the exact constant of this scaling?
- How does training affect model capacity? 
- How does  model architecture relate to model capacity?
- How do quantization and model sparsity affect model capacity?
- How does irrelevant/noisy data affect model capacity?

Unlike prior studies that evaluate a model’s capability via loss
or benchmarks, this paper  estimates  the number of knowledge bits a model store, focusing  on factual knowledge represented as tuples, such
as (USA, capital, Washington D.C.) from a Wikipedia page. Experimental results across model architectures establish that language models can only store 2
bits of knowledge per parameter. Detailed analyses further illustrated that a sufficiently trained 7B language model can store 14B bits of knowledge. Achieving 2bits per parameter capacity requires each knowledge piece to be revisited 1000 times. Quantizing to int8 does not compromise model capacity,  however, quantizing to int4 reduces capacity to 0.7bit/param.
Mixture-of-experts (MoE)  only reduce 1.3x in capacity,  despite using just 8.8% of the total parameters during inference.
Finally, noisy data significantly reduces model capacity but an  effective mitigation is to prepend a special token to all useful knowledge.
The model autonomously identifies high-quality data without prior knowledge of
valuable domains.
Strengths: - Scaling laws are of interest to the community as they allow us to analyze and understand the capacity of large language models. They also enable the design of new models and pertaining experiments. The authors claim they are the first to propose a scaling law for the knowledge capacity of LLMs. 

- The paper contains a very thorough and exhaustive list of experiments, answering several questions. 

- There are practical recommendations about LLM model builders, such as domain tagging for pertaining data. 

- the paper provides an explanation as to why Quantized models perform on part to their non-quantized variants in terms of knowledge storage, and similarly why  mixture of experts models perform decently despite being sparse. 

- Finally, it reveals specific architectural choices in models like Llama and Mistral which might lead to slightly inferior performance (e.g., weight tying, MLP layers). 

- The experimental framework is reproducible, and although the authors focus on binary knowledge base tuples, it could be extended to of the knowledge-based facts or event language structures.
Weaknesses: - the paper is very dense, the appendix is very many pages long.  As a result, it is not easy to absorb all important details. 

- the graphs are in very small scale and are not explained appropriately.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The paper investigates the relationship between language model size and its capacity to store factual knowledge, quantified in bits per parameter. The authors introduce a framework that measures a model’s knowledge based on tuple-based information (e.g., (Entity, Relation, Attribute)) and propose that language models, after sufficient training, achieve an approximate capacity of 2 bits per parameter. The study extends this analysis across various factors such as model architecture, quantization levels, sparsity, and training data quality. Using synthetic and controlled datasets, the authors examine how these elements influence the knowledge storage capacity of language models.

The technical claims are generally supported by experiments; however, there are concerns about the robustness of the findings. The reliance on synthetic data raises questions about the applicability of the results to real-world scenarios. Additionally, the paper does not thoroughly explore the impact of quantization during training.

The paper has several formatting issues that hinder its readability. Notably, it lacks a conclusion section. Some figures are not clear. The organization of the paper could be improved drastically.
Strengths: •	Originality: Introducing a framework to measure language model capacity in bits per parameter is a novel approach that adds a quantitative dimension to model evaluation.

•	Methodology: The use of controlled synthetic datasets allows for the isolation of specific variables, providing clarity in the analysis of different factors affecting knowledge capacity.
Weaknesses: •	Formatting Issues: The absence of a conclusion section and unclear figures detract from the overall quality of the paper and impede the reader’s understanding.

•	Generalization to Real-world Data: The heavy reliance on synthetic data limits the applicability of the findings to natural language processing tasks involving complex and diverse datasets.

•	Incomplete Exploration of Quantization: The paper does not investigate quantization during training, which could provide insights into mitigating the observed decrease in capacity with int4 quantization.

•	Limited Architectural Diversity: The study does not explore a wide range of model architectures, such as encoder-only or decoder-only models, which could affect the generalizability of the proposed scaling law.
Recommendation: 5
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 2