Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process

Authors: Tian Ye, Zicheng Xu, Yuanzhi Li, Zeyuan Allen-Zhu
Primary area: interpretability and explainable AI
Keywords: linear probing, language model, grade math problems, logic following, reasoning

Abstract:
Recent advances in language models have demonstrated their capability to solve mathematical reasoning problems, achieving near-perfect accuracy on grade-school level math benchmarks like GSM8K. In this paper, we formally study how language models solve these problems. We design a series of controlled experiments to address several fundamental questions: (1) Can language models truly develop reasoning skills, or do they simply memorize templates? (2) What is the model's hidden (mental) reasoning process? (3) Do models solve math questions using skills similar to or different from humans? (4) Do models trained on GSM8K-like datasets develop reasoning skills beyond those necessary for solving GSM8K problems? (5) What mental process causes models to make reasoning mistakes? (6) How large or deep must a model be to effectively solve GSM8K-level math questions?

Our study uncovers many hidden mechanisms by which language models solve mathematical questions, providing insights that extend beyond current understandings of LLMs.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730020874292

thread_type:
official_review

<thread_object>
creator: reviewer_35ln

<content>
summary:
This work focuses on evaluating the grade-school level mathematical reasoning of neural language models (GPT models).
The authors set out to explore several critical questions surrounding the actual abilities of LMs to reason mathematically.
To make clear and controllable experiments, the authors introduces a synthetic dataset iGSM.
Using iGSM to train and test GPT-2, the authors argue that the model can truly understand and reason through problems rather than just memorizing instances.
The examination of the causes behind reasoning mistakes made by LMs is another highlight, shedding light on the limitations and potential areas for improvement.

strengths:
1. Overall, the exploration of the model's hidden reasoning process provides valuable insights into the black-box nature of neural language models.
2. This work introduces a new synthetic dataset iGSM which could facilitate the training and evaluation of contemporary large language models.
3. This work provides many interesting findings. It provides a valuable addition to better understanding the reasoning capabilities emerged in large language models.

weaknesses:
My major concern is about the generalizability of the conclusions drawn in this work.
It is unclear whether the experimental findings on formal language reasoning can be generalized to natural language reasoning.
The formal language patterns in iGSM are too much limited and inflexible compared with natural language expressions, although it may contain a large number of templates if omitting the primitives in the expressions.

For instance, for the math operation *multiplication*, there are many different ways to express using natural language, e.g., "there are three children in the house and each child eats two eggs for breakfast".
But in iGSM, it seems that *multiplication* is always mapped from "times" in the input expression.
I think for grade-school level mathematical reasoning, one major capability is to map the flexible natural expressions to calculations.
Therefore, I think reasoning on iGSM could be much easier than reasoning on natural language questions.

questions:
1. Can you provide some evidence to support the generalizability of your conclusions, even some of them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625638878

thread_type:
official_review

<thread_object>
creator: reviewer_arcv

<content>
summary:
The paper investigates the capability of language models (the authors used the GPT-2 model for their experiments), to solve grade-school math problems to explore whether models genuinely develop reasoning skills beyond template memorization. They address several key questions: Can LMs generalize reasoning skills? Do they replicate human-like problem-solving methods? Are their errors predictable, and does model architecture (depth versus width) influence reasoning abilities?

It is important to control the dataset to explore these questions. For this, the authors generate a synthetic dataset with high template diversity, reducing contamination and ensuring complex, logical dependencies in problems. Results indicate that the model can indeed generalize reasoning processes and even learns dependencies and plans solutions. The paper explores the models internal states through probing to understand how the model is solving the problem. 

Overall, the paper is highly experimental with the goal to identify the factors that enable a model to develop reasoning skills through systematically controlled experiments.

strengths:
The biggest strength of the paper is the controlled experiments that the authors conducted. Training a GPT-2 model from scratch with their own synthetic data can give them full control over the experiments. Also, generating their own data can prevent contamination of the training test data, which might be one of the problems in the other benchmark data. 

The authors also introduce the probing tasks to explore the internal states of the model, which gives some idea of the reasoning process of the model. In particular, the V-probing approach is interesting because it can tell whether pre-training or fine-tuning was responsible for the probing signal. 

Finally, the paper presents many interesting results, including the depth vs. width of the models, the generalizability of the model to unknown tasks of greater depth, models learning backward reasoning processes like humans, and so on. These results (if they can be scaled) may allow better design of language models in the future and may help to understand the role of synthetic data in learning a concept.

weaknesses:
The weakness of the paper also lies in the main choice of experiments: the creation of the dataset and the choice of the model. 
- Regarding the dataset, the use of synthetic data may not fully capture the complexity and nuances of real-world mathematical problems. This limitation could affect the generalizability of the results to practical applications. It would be important to mix and match the training dataset with real-world data or test the methodology on some real-world data such as GSM8K.
- Regarding the models, although it is difficult to test the methodology on multiple models, the model performance scales well with depth is kind of a known fact and with more layers, the performance improves creating a question that if scaling leads to memorization?

The use of math word problems is a good choice to control the experiments, but the question is if the results are limited to a structured reasoning problem like math word problems? This is important to know in order to understand the limitations of the approach. 

The analysis of the model errors was omitted from the paper, saying that the authors are writing another paper because it is not good for the current paper. It would be good to see the error analysis. At least a summarized version needs to be presented. 

Finally, since the training of the model was done by the authors, it would be interesting to see in which training iterations the model learned certain skills. This can easily be discussed in the paper, but I found it missing. 

[MINOR] The use of words like AGI while experimenting with synthetic datasets and the GPT-2 model is not correct and can be avoided.

questions:
It would be interesting to see some results on real-world datasets like GSM8K to see how much the models can generalize to real-world problems when trained on synthetic datasets.

Is it possible to test the models on similar synthetic datasets like mathworld (https://arxiv.org/abs/2306.04347)?

Model errors and how LLMs can fix them (Result 6) needs to be discussed in the paper and not left saying its there in another paper.

The results of the backward thinking process are missing from the paper and saying that the models learned it without any discussion is not correct. It needs to be added to the paper. 

Finally, it would be interesting to see an analysis of when a model learned a particular reasoning skill during training. And can it be learned in a continuous training fashion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711266631

thread_type:
official_review

<thread_object>
creator: reviewer_gm4v

<content>
summary:
This work conducts controlled experiments to probe several fundamental questions about math reasoning capability of language models, the experimental results show that language models can solve math problems like humans and conduct backward thinking process.

strengths:
- This work introduces a framework to generate a set of diverse GSM problems that focus on “logical reasoning” aspect.
- This work finds that “the model has learned to plan ahead, identifying necessary
parameters before starting to generate the solution”.

weaknesses:
- The small model size of GPT2-small (100M-200M) may effect the generalization of the experimental conclusion. It would be appreciated if conducted experiments on LLMs with billion parameters.
- The conclusions drawn from experiments on synthetic data may not be applicable to real-world scenarios with large data volumes.

questions:
Since IGSM only has four categorizations, which may result in poor diversity in the description of quantitative relationships. Could this affect the generalization of the experimental conclusions of this paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730778066254

thread_type:
official_review

<thread_object>
creator: reviewer_ub35

<content>
summary:
This paper dives deep into how language models solve grade-school math problems. The authors first design an elaborated data generation pipeline to synthesize a large amount of grade-school math problems with a specific form of Chain-of-Thoughts. Then the authors design a probing method to analyse how the language models solve the problems, and acquire some interesting conclusions.

strengths:
* This paper carries out a substantial experiment (including large-scale data synthesis, fine-tuning, and probing) to systematically research the reasoning ability of LLMs in grade-school math problems.
* To understand how the metal process of language models, the paper proposes a few **probing tasks** which align with human problem-solving strategies. Such probing tasks might be insightful to other works.
* Multiple interesting conclusions are found in this paper.

weaknesses:
* **Limitation of specific CoT form**: This paper trains the GPT-2 model using synthetic data with a specific form of CoT. The fune-tuning and probing experimental conclusions are based on such CoT. However, the CoT that is now used to train the LLMs on grade school math problems deviates considerably from this form. This limits the generalizability of the paper's conclusions.
* **Lack of analysis on data quantity**: Although this paper synthesizes a large amount of data to train a language model, some studies have shown that more training data is not always better for language models. I expect to see a curve or table of [model performance - the quantity of training data].

questions:
1. **Lack of analysis on data quantity**: See the second point in Weaknesses.
2. **About the backward thinking process**: In line 447-453 of this paper, the authors define the “backward thinking process” as “because I want to compute X, but X depends on Y and Y depends on Z, so let me compute Z first”. Moreover, they conclude that this backward thinking process can be autonomously learned through language modeling with abundant data. **I can't understand how the author came to this conclusion**. I cannot understand how the authors arrive at this conclusion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731052308708

thread_type:
official_review

<thread_object>
creator: reviewer_yuz7

<content>
summary:
This paper attempts to measure the reasoning ability of models by training on a synthetic dataset based on GSM8k. Among other things, they show that training from scratch on this synthetic dataset of grade school math has some signs of generalization to harder problems.



Edited: score to 6 after reviewer rebuttal.

strengths:
Measuring reasoning via training an LLM from scratch is an interesting direction. There are lots of interesting results and experiments in the paper.

weaknesses:
I think the writing can be improved somewhat. Additionally, I think there are some questions about whether the findings are broadly applicable given a single synthetic eval on GSM8k. In some sense, LLMs are useful because they generalize widely beyond a small set of very specific problems.

questions:
Q: My big question is whether this is a proper use of generalization, especially with LLMs, which are useful primarily because they generalize to very different kinds of data. "Generalization" from a synthetic dataset to a similar synthetic dataset is not nearly as ideal as generalization to a very different kind of dataset.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary of the paper: This paper investigates the mathematical reasoning capabilities of LMs, particularly focusing on grade-school math problems. The paper develops a synthetic dataset called iGSM, designed to enhance template diversity and reduce contamination, enabling controlled experiments. Through elaborate data generation and probing methods, they explore several critical questions lie in LMs: whether models can generalize reasoning skills, replicate human-like problem-solving methods, and how model architecture influences reasoning abilities. Results indicate that LMs, specifically GPT-2, can indeed generalize reasoning processes and learn complex dependencies. The paper finds that these models can solve math problems similarly to humans, engaging in backward thinking processes rather than mere memorization. The study also examines the nature of reasoning mistakes, providing insights into the models' limitations and areas for potential improvement. 

Strengths of the paper:
- Innovative Methodology: This paper presents a novel approach to understanding the reasoning capabilities of LMs. Its originality and scientific rigor address an important problem in LLM research. This method advances our understanding of how these systems process and reason about information.
- Systematic Experimentation: By implementing an extensive framework of data synthesis, fine-tuning, and probing techniques focused on grade-school math problems, the study demonstrates a rigorous experimental design. Notably, the decision to train a GPT-2 model from scratch using synthetic data enhances experimental validity and effectively avoids contamination issues that often affect other benchmarks in the field. 
- Valuable Insights: The investigation into the LMs' hidden reasoning processes yields important insights into the often opaque workings of neural LMs. The paper presents valuable findings regarding the relationship between model depth and width, the model's generalizability to unknown tasks, and its ability to learn backward reasoning processes. These contributions are important for the future design of language models.

Weaknesses of the paper: I do not have significant concerns about this paper. While many reviewers have raised issues regarding the generalizability of the proposed method—specifically concerning the training and test datasets (iGSM), the size of the LM (GPT2), and the format of the CoT, etc., I believe the authors provided satisfactory clarifications during the rebuttal process. After reading the paper, I consider it a solid starting point for understanding how LLMs function. It is important to begin with simpler approaches to build a foundational understanding.

Reasons for the decision: After carefully considering all the comments, I believe that most concerns have been adequately addressed by the authors. The issues raised primarily revolve around conceptual questions rather than the need for additional empirical experiments. It appears that many reviewers are at least partially convinced by the authors' explanations. All reviewers agree that this paper should be accepted at ICLR. Personally, I find the paper very appealing, particularly the concept of "control experiments" and the overall study framework. This work offers a solid foundation for understanding the reasoning process of LMs in a systematic way. It has the potential to engage the broader community and make a significant impact.