Not All LLM Reasoners Are Created Equal

Authors: arian  hosseini
alessandro  sordoni
daniel  toyama
google  deepmind
aaron  courville
mila  rishabh
agarwal  mila
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-26 04:43:00

Abstract: We study the depth of problem-solving capabilities of LLMs, and to what extent they perform mathematical reasoning in a compositional manner. To this end, we create a new benchmark by composing pairs of existing math word problems together so that the answer to the second problem depends on correctly answering the first problem. We measure the difference between the performance of solving each question independently and solving the compositional pairs as the reasoning gap of a model. Our findings reveal a significant reasoning gap in most frontier LLMs. This gap is more pronounced in smaller and more cost-efficient models. The objective of this study is not to introduce yet another benchmark, but rather to provide a case study aimed at gaining deeper insights into current models' reasoning abilities, and to reassess existing established training methods and benchmarks.

Main paper: 
 introduction: 
 The strong performance of large language models (LLMs) on high-school and college-level math reasoning benchmarks (Dubey et al., 2024; Google, 2024; OpenAI, 2023b) , has led to the common belief that LLMs have "mastered" grade-school math, particularly as measured by the GSM8K benchmark (Cobbe et al., 2021) . This apparent mastery of grade-school math problems raises a deeper question: do LLMs truly grasp the underlying concepts or do they mostly rely on dataset contamination or memorization (Srivastava et al., 2024) ? For example, a recent examination on private "held-out" grade-school problems (Zhang et al., 2024) reveals that while frontier closed-source LLMs show minimal signs of overfitting, some open-weights models show systematic overfitting, possibly due to test data contamination.In this work, we perform a case study to evaluate how well LLMs can combine learned concepts to solve unseen problems, to probe the brittleness of their reasoning abilities. To do so, we introduce Compositional GSM, a two-hop version of GSM8K with higher difficulty, where each problem chains two test questions together such that the answer to the first question is used as a variable in the second question (Figure 2 ). As LLMs can easily solve grade-school math problems, they should also be capable of solving combinations of those problems. As such, we measure the gap between their performance on solving the questions individually and on Compositional GSM. Specifically, we benchmark frontier open-weights and closed LLMs, including Gemini (Google, 2023 , 2024 ), Gemma2 (Gemma Team et al., 2024) , Llama-3 (AI@Meta, 2024), GPT (OpenAI, 2023a), Phi (Abdin et al., 2024) , Qwen (Yang et al., 2024) and Mistral families (Jiang et al., 2024) .Here are our key findings:• Most models exhibit a gap between their performance on GSM8K test set and Compositional GSM (Figure 1 ). • This reasoning gap is larger in small and more cost-efficient models (Figure 5 and Figure 3 ). • Instruction-following tuning of LLMs heavily favours the original GSM8K split (Figure 4 ). • Finetuning with human data and synthetic data results in a similar reasoning gap trend (Figure 7 ).• Smaller models benefit more from generating code rather than natural language Chain-of-Thought (CoT) to solve Compositional GSM problems (Figure 6 ). 
compositional grade-school math (gsm): 
 Each question in compositional GSM consists of two questions, Question-1 and Question-2, from a subset of 1200 examples of the original GSM8K test set. The final answer of Question-1 is refereed to as X which is a variable in Question-2 (Figure 2 ). The final answer of Question-2 is obtained by substituting X and solving it. The choice of Question-1 and the number to modify and replace with X in Question-2 was made in a a way such that the new final answer of Question-2 is different from its old final answer, and is a positive integer not too far from the old final answer. Reasoning Gap Question-1 and Question-2 in our compositional queries are from the original test split D original , and the modified test split D modified respectively. Assuming that a model has an accuracy of S 1 on D original and S 2 on D modified , it is expected for it to have an accuracy of S 1 × S 2 on the compositional split D comp . We report the following as the compositional reasoning gap score, Compositional reasoning gap :EQUATIONwhere S c is the performance of the model on D comp . 
experiments & results: 
 The distance to the trend-line in Figure 1 shows the reasoning gap of models. The x-axis corresponds to √ S 1 × S 2 , which is the geometric mean of the accuracies on the set of Q 1 and Q 2 independently. We find that most models fall below expectation on Compositional GSM. Specifically, it is evident that cost-efficient models have a larger gap than more expensive models. More analysis is provided in the Appendix.The reasoning abilities of cost-efficient LMs has been rapidly improving over time, as evaluated using standard benchmarks (Bansal et al., 2024) . For example, GPT-4o mini and Gemini 1.5 Flash both achieve above 90% accuracy on GSM, while costing 25 -35× cheaper than GPT-4o and Gemini 1.5 Pro respectively. This progress could be attributed to several factors, such as better pretraining data (AI@Meta, 2024), and knowledge distillation (Agarwal et al., 2024; Team et al., 2024) . To this end, we investigate whether these reasoning gains on GSM8K still persist on Compositional GSM.We study four family of models, each comprising both a high-cost and low-cost option, where cost is measured via parameter count or API pricing. Figure 3 shows the original GSM8K test split performance and Compositional GSM performance for all models. The numbers above the bars represents the reasoning gap defined in Eq 1. While cheaper models perform comparably on the original GSM8K test, they exhibit a notable drop in performance on the Compositional GSM test set. These results suggest that critical flaws of cost-efficient LLMs in their reasoning may be obscured by high scores on standard benchmarks. This underscores the need to rethink current strategies for developing cost-efficient language models.We compare pretrained and instruction-following tuned versions of models in three families of Mistral, LLAMA3 and Gemma2. Figure 4 Figure 4 : Impact of Instruction-Tuning on Reasoning Gap: comparing pretrained and instruction-following tuned variant of models from Mistral, LLAMA3 and Gemma2 families. Numbers above bars represent improvements from instruction-tuning on each set. For smaller models (top), we observe that instruction-tuning is highly optimized for GSM8K questions, which results in a greater improvement on the original GSM8K test set compared to the Compositional GSM test. However, this pattern does not hold for larger models (bottom).instruction-tuning, displayed above bars for each test set. On small models (top row), this comparison shows that current instruction-tuning is heavily optimized for GSM8K questions. Instruction-tuning leads to a significantly larger improvement on the original GSM8K test set than the Compositional GSM test across model families. However, this trend does not apply to larger models (bottom row), where the improvements are inconsistent. 
discussion and conclusion: 
 We designed the Compositional GSM benchmark, which requires solving dependent pairs of math word problems. These problems are from the original GSM8K test split. We investigate the "System 2" mathematical reasoning capabilities of LLMs by comparing their performance on the original GSM8K test split and our Compositional GSM test set. Our analysis reveals a notable reasoning gap in most models. Many leading LLMs exhibit a substantial difference in performance when solving questions independently versus as part of the compositional pair. Our study indicates that smaller and more cost efficient models exhibit a larger reasoning gap. Models frequently struggle with pairs of questions and get distracted likely because they are tuned to handle one question at a time. They often answer the first question correctly, but lose attention to details and make subtle errors in answering the second question. We also noticed that learning from human data and self-generated data results in similar behaviour. In both settings, as training progresses, the model's performance on the original test split improves. However, beyond a certain point, performance on the Compositional GSM test begins to decline.We emphasize that this benchmark should not be viewed as an endpoint or merely as a tool for generating additional training data, but as a catalyst to gain insights about current models and to reevaluate and improve existing benchmarks. Our findings are intended to stimulate further exploration and provide new perspectives. Future research could build on this setup by incorporating more challenging questions, such as those from the MATH dataset, or by extending the framework to multi-modal problems to gain deeper insights into the reasoning capabilities of LLMs.  
: 
 Figure 1 : Reasoning Gap: Pairs of GSM8K test questions are chained together so that the answer of the first question (Q1) is a variable in the second one (Q2). The model is required to correctly answer both questions to solve the problem. If a model has an accuracy of S1 on the Q1 set, and S2 on Q2 set, then the expected Compositional GSM accuracy is S1 × S2. The x-axis corresponds to the geometric mean √ S1 × S2, labeled GSM8K accuracy for simplicity. The trend-line y = x 2 is the expected Compositional GSM accuracy.Let X be the answer to the Q1: Q1: There are 27 unicorns left in the world. One third of them are in the Scottish Highlands. Two thirds of the Scottish unicorns are female. How many female Scottish unicorns are there? Solve it and use the value of X to solve Q2. Explain your answer step by step.Q2: Zack's locker is half as big as Timothy's locker. Peter's locker is 1/4 as big as Zack's locker. If Peter's locker is X cubic inches, how big is Timothy's locker in cubic inches?Figure 2 : Example Problem from the Compositional GSM benchmark. The answer of Question-1 (Q1) is a variable X in . Therefore, the model has to be able to solve the first question correctly in order to solve the second question. The new final answer of Question-2 is calculated by modifying its code-form solution and executing it. Question-1 and the number to modify in Question-2 are chosen to have a new final answer which is a positive integer not too far from the old answer of Question-2. 
 
 Appendix: To obtain the new final answer of Question-2 automatically, we replace a number in the code-form solution of Question-2. We used a slightly modified version of the code-form solutions from Gao et al. (2023) . The new final answer is the result of executing the code with the new number. We put significant efforts into ensuring that the modification to Question-2 is sensible. Figure 10 shows the distribution of final answers (magnitude) of the original test set of GSM8K and compositional GSM.Both test sets have a similar distribution of final answers.Quality Checks To make sure that the modified questions are sensible and logical, we generated 16 candidate solutions per modified question from GPT-4o and Gemini 1.5 Pro. We filtered those questions for which less than 4 (out of 16) agree with the expected final answer from code execution. We checked these questions manually and modified them if needed so that they are logical (about 25% of questions).Setup We evaluate each model on three test sets: 1) the original GSM8K test split, 2) the modified GSM8K test split which are the questions with X being substituted, and 3) the compositional GSM test set. Each test set has 1200 examples. Following Zhang et al. (2024) , we evaluate all models with an 8-shot prompt (Appendix I) for the original and modified GSM8K test splits. We also created a similar 8-shot prompt (Appendix J) for the compositional GSM questions. We evaluate GPT-4o, GPT-4o mini, LLAMA3-70B and 8B (PT and IT), Phi 2, Phi-3-mini-instruct, Gemini 1.0, 1.5 Flash and 1.5 Pro, Gemma2 9B and 27B (PT and IT), Mistral-7B (PT and IT), Mixtral-8x7B (PT and IT) and Mathstral-7B. All models are sampled with temperature 0, and pass@1 (Chen et al., 2021) is used to measure the performance on each test split. Some of the models require a preamble prefixed to the 8-shot prompt in order to output in a consistent format (Appendix F). We test both cases and report the best performance for each model. Smaller models benefit more from generating code rather than natural language CoT to solve Compositional GSM questions, further highlighting that smaller models demonstrate systematic differences in reasoning capabilities.C Thinking in Natural Language v.s. CodeBreaking down the natural language problem into executable code steps has been shown to improve models' reasoning and generalization abilities (Gao et al., 2023; Gou et al., 2023) . To this end, we evaluate whether the compositional problem-solving ability of LLMs improves when generating natural langauge CoT rationales compared to generating executable Python code. For code generation, we utilize a compositional 8-shot prompt(Appendix K), where the answers are written as two functions, one which solves the first question solve_q1(), and solution() which solves the second question with a X = solve_q1() line at the beginning.Our results are shown in Figure 6 for three families of open-weight instruction-tuned models: LLAMA3-8B and 70B, Gemma2-9B and 27B, and Mistral 7B and (Mixtral) 8 × 7B. Notably, generating code generally improves performance on Compositional GSM problems, albeit not uniformly. Specifically, comparing relative improvements, smaller models benefit significantly more from generating code solutions compared to generating natural language CoTs. This ruther underscores the systematic differences in reasoning behaviors of smaller models.Finetuning models on task specific problems is a common strategy to improve reasoning performance.In this section, we explore how it impacts the performance on Compositional GSM. We investigate the performance of Gemma2 27B PT as we finetune it on the original GSM8K training data, and self-generated rationales (aka synthetic data) to identify any difference in the characteristics of these two sources. We collect self-generated rationales which results in correct final answers for all GSM8K training queries.See Appendix H for details of data generation and training for this set of experiments. We evaluated intermediate checkpoints (at 50, 100 and 400 training steps) from both settings on GSM8K original test split and Compositional GSM split (Figure 7 ). We observer a similar pattern for both settings.The Compositional GSM performance increases with some training (up to 100 steps), but drops with more training steps while GSM8K test performance keeps increasing, which suggests overfitting.Our results show that training on synthetic data generally leads to a higher Compositional GSM performance. We did not observe further improvements on either test splits after 400 training steps. E Failure Modes of LLMs on Compositional Questions Does Solving Question-1 Guarantee Solving Question-2? Correctly solving Question-1 is a prerequisite to solve Question-2 in the compositional format. In Figure 8 , we look at how often models are able to solve a question independently versus how often can they solve it given they have correctly solved the previous question in the compositional format. What remains for the model to do here is to substitute X and solve Q 2 . The deviation from the diagonal line indicates that certain models may have become too specialized in handling GSM8K-style questions, and are unable to answer a second question having generated the solution to the first question. Our qualitative analysis shows that when given two questions, the model might answer the first one correctly, but often makes subtle errors and overlooks details, leading to inaccurate reasoning and solution for the second question. 2 Solved | Q 1 = correct Gemini 1.5 Pro Gemini 1.0 Pro Gemma2-27B-IT Gemma2-9B-IT Gemma2-27B-PT Gemma2-9B-PT GPT-4o GPT-4o mini LLAMA3-70B-IT LLAMA3-8B-IT Phi-2 Phi-3-mini-4k-IT Mixtral-8x7B-IT Mistral-7B-IT Mistral-7B-PT Mathstral-7B Gemini 1.5 Flash LLAMA3-8B-PT LLAMA3-70B-PT Mixtral-8x7B-PT NuminaMath-7B-CoTQwen2.5-MATH-7B-IT Qwen2.5-MATH-72B-IT Figure 8 : Can models answer the second question if they have correctly answered the first one? Here, we compare how often models are able to solve a question independently to how often they are able to solve them in the compositional format given that the first question is solved correctly. This is an alternate measurement of the compositional reasoning gap. If a model can solve a question independently, it should be able to solve it in a compositional setting given that the prerequisites are met. The gap from the diagonal line suggests that some models have overfit to the format of GSM8K type questions. While models may correctly answer the first question, they frequently makes subtle errors and miss key details when solving the second question. Gemma2-27B-IT Gemma2-9B-IT Gemma2-27B-PT Gemma2-9B-PT GPT-4o GPT-4o mini LLAMA3-70B-IT LLAMA3-8B-IT Phi-2 Phi-3-mini-4k-IT Mixtral-8x7B-IT Mistral-7B-IT Mistral-7B-PT Mathstral-7B Gemini 1.5 Flash LLAMA3-8B-PT LLAMA3-70B-PT Mixtral-8x7B-PT NuminaMath-7B-CoT Qwen2.5-MATH-7B-IT Qwen2.5-MATH-72B-ITSynthetic data was generated by prompting Gemma2 27B PT model with the 8-shot prompt in Appendix I to solve GSM8K training questions. We generated 10 solutions for each question in the original GSM8K training data, and only kept those solutions with a correct final answer. These model generation solutions were used to train the model.Let X be the answer to Q1: Q1: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today? solve it and use the value of X to solve Q2. Explain your answer step by step. Q2: There are X students in Marissa's class. Each student started the year with 10 pencils. After two months, 1/5 of the total pencils in class were used. At the end of the year, only 1/3 of the remaining pencils were left. How many pencils were left?A: There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 -15 = 6. The Q1 answer is 6. Therefore X=6. So there were 6 x 10 = 60 pencils in the class at the start of the year. After two months, 60 x 1/5 = 12 pencils were used. Thus, 60 -12 = 48 pencils were left unused after two months. Therefore, 48 x 1/3 = 16 pencils were left at the end of the year. The final answer is 16.Let X be the answer to Q1: Q1: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot? solve it and use the value of X to solve Q2. Explain your answer step by step. Q2: Ingrid drinks X cups of water every day. If there are 16 cups in a gallon, how many gallons of water does she drink in 32 days? A: There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5. The Q1 answer is 5. Therefore X=5. So Ingrid drinks 5 cups of water a day so after 32 days she drinks 5 * 32 = 160 cups of water. There are 16 cups in 1 gallon so she drinks 160 / 16 = 10 gallons of water in 30 days. The final answer is 10.. . .Let X be the answer to Q1:Q1: {QUESTION_1} solve it and use the value of X to solve Q2. Explain your answer step by step.A:Some examples in the prompt are omitted due to space constraints. The remaining question-andanswer pairs follow the same format. 



Reviews: review: title: title: A compositional version of GSM8K that challenges LLMs
main_review: main_review: **Summary**

The paper presents a new version of the GSM8K benchmark, which is created by composing two questions together. The final answer for the first question is substituted as a variable into the second question, and the LLM is asked to solve the second question using the first answer substituted in for the variable.

**Strengths**
* The paper draws interesting insights about the difference between the performance and behavior of small vs. large models, base vs. instruction-tuned models, etc.
* The paper includes a detailed appendix with all prompting information, failure modes, and additional experiments that make it easy to conduct future work.
* The method proposed in the paper can be applied to any mathematical benchmark that relies on numerical calculations.
* The average accuracy of most models is considerably below what the expected average accuracy would be, showing an inherently more challenging task.

**Weaknesses**
* This is more of a suggestion, but it would be interesting to compare the performance of models on this benchmark to their performance on a 2-prompt setup of the benchmark (i.e. asking the first question, letting the model calculate the answer, and then asking the second question as “Let X be the answer to the previous question… [insert second question here]”)
title: title: Intriguing case-study
main_review: main_review: **Summary** 

This paper introduces a 2-hop reasoning version of GSM8k, *Compositional GSM*, consisting of pairs (Q1,Q2) of modified problems drawn from (a subset of) GSM8k where the final answer to Q1 is referred to as X which is used as a variable in Q2. They measure the "reasoning gap" as the difference in accuracy on (the subset of) GSM8k vs. the compositional version. The main result is that smaller models have a much greater reasoning gap than larger models. 

**Comments** 

References (\ref) from the main text to the relevant appendix sections should be added. I also think descriptions of figures should occur in the captions of the figures themselves, but perhaps this is just my opinion :) 

What is instruction-tuning?

65: "We study four family of models"

What are the odds that the reasoning gap boils down to simple context-length? (I'm not an LLM expert -- perhaps smaller models simply don't have enough attention for a two-paragraph question?)

**Pros**

* Clearly written
* The reasoning gap has a simple and natural definition
* Shows a clear trend that model size is crucial for "reasoning capacity" even on tasks thought to be easy for small models
* Inspires further research into "composed questions" or some underlying concept of sequential reasoning
* I think the fact that larger models have a smaller reasoning gap is intriguing and deserves further investigation. Even if not, circumventing this gap with smallest models is surely of interest. 

**Cons**
* Minimal investigation into why the reasoning gap occurs (trends in incorrect responses, etc.) 
* "They often answer the first question correctly, but lose attention to details and make subtle errors in answering the second question." This should be justified by examples, if not a full experimental analysis.
* Details important for reproducibility are lacking (e.g., exactly how is the dataset constructed (all-pairs (Q1,Q2) from GSM8k)? Pretraining and instruction tuning details? Code?). 
* Related work?
* Should the checklist not be attached to the appendices?
title: title: Review of "Not All LLM Reasoners Are Created Equal"
main_review: main_review: This paper presents a study on compositional GSM8K where two GSM8K examples are chained together. This paper presents a thorough study on one of the key questions for reasoning with LLMs, i.e., compositional generalization and multiple interesting observations. The appendix presents comprehensive experimental details and futher discussions such as thinking with natural language vs code.