AutoCode4Math: Learning Autonomous Code Integration for Math LLMs

Authors: Haozhe Wang, Long Li, Chao Qu, Weidi Xu, Fengming ZHU, Yi Xin, Wei Chu, Fangzhen Lin
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models

Abstract:
Recent research on tool integration for math Large Language Models (LLMs)  aims to combine complementary strengths of chain-of-thought (CoT) reasoning and code execution. However, we discover a critical limitation: current tool-integrated math LLMs rely on externally dictated instructions to decide whether to use CoT or code, lacking the autonomy to choose the most appropriate method independently. This prompts us to study \emph{Autonomous Code integration } (AutoCode) for math LLMs, which enables models to \emph{independently} develop their own methodology-selection strategy in the absence of reliable supervision. To address this challenge, we propose an innovative Expectation-Maximization (EM) formulation that refines the model's decision-making through the exploration of its capabilities. This framework alternates between (a) computing a reference strategy that improves the model's belief over its capabilities through self-exploration, and (b) updating the model based on the refined belief. We further enhance this framework with an efficient implementation, incorporating a novel data synthesis strategy and off-policy reinforcement learning. Extensive experiments demonstrate that our approach, using only a public query set, significantly boosts the performance of existing math LLMs, raising accuracy by nearly 20% to 65.28%  on the challenging MATH benchmark, while reducing code executions by up to 65% . 

Decision: reject
Reviewer scores: 8, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730350470314

thread_type:
official_review

<thread_object>
creator: reviewer_nghb

<content>
summary:
The paper, titled "AutoCode4Math: Learning Autonomous Code Integration for Math LLMs," addresses the limited ability of math-focused language models to make independent decisions on whether to use chain-of-thought (CoT) reasoning or code execution for problem-solving.
Generally, such models follow externally given instructions for choosing amongst them, narrowing their ability to operate autonomously while the integration of the best option will be done for answering the query on mathematics. To tackle it, the authors build an Expectation-Maximization framework that offers the ability for the LLM to learn autonomous code integration by furthering decision-making capacity through self-exploration. In this regard, the model alternates between self-teaching phases, E-step, after which an update derived from refined beliefs is set, M-step, to form an optimized methodology selection strategy. Experiments show that such an approach results in a significant accuracy gain when tested with lower code-execution requirements: an improved performance of nearly 20% on the MATH benchmark, with virtually a 65% reduction in code executions.

strengths:
The study makes a significant contribution by addressing how models can autonomously choose between Proof-of-Thought (PoT) and Chain-of-Thought (CoT) approaches during mathematical problem-solving. This research identifies an important gap in current language models' ability to self-direct their reasoning strategies. Through their experiments, the authors demonstrate that incorporating autonomous decision-making in training leads to measurable performance improvements. The results validate that self-directed methodology selection, driven by expectation maximization, enhances mathematical problem-solving capabilities.

weaknesses:
# Motivation for Using Expectation-Maximization (EM)

The paper introduces Expectation-Maximization (EM) to autonomously select between Chain of Thought (CoT) or code methodologies for solving math queries without external supervision. However, the motivation for this choice of EM remains unclear in several respects:
- Lack of Comparative Analysis: While the paper suggests EM is superior to standard reinforcement learning (RL), it does not provide a rigorous comparative analysis or an explanation of its advantages over other optimization techniques. The narrative suggests that EM holds unique benefits, but it is unclear why it is preferred over RL or simpler alternatives like supervised fine-tuning with reinforcement. Additional justification is needed to establish EM’s unique suitability for method selection in this context.
- Efficiency Assumptions: The use of EM is partly based on the assumption that it narrows the policy search space, thereby enhancing efficiency. However, there is no empirical evidence provided to demonstrate that EM’s efficiency leads to actual performance gains in selecting between CoT and code for complex queries. Without comparative data, it is difficult to assess whether this efficiency is observed in practice or remains theoretical in this case.
# Insufficient Comparison with Baselines for Model Selection
The effectiveness of AutoCode4Math’s approach to method selection would benefit from more detailed comparisons with existing baselines:
- Baseline Limitations: While current baseline models (e.g., Qwen2Math, GPT-4o, and DeepseekMath) primarily employ CoT over code for math queries, the lack of detailed performance metrics for these baselines limits the reader’s ability to assess AutoCode4Math’s advantages. Controlled comparisons with these models would help substantiate claims about AutoCode’s effectiveness in methodology selection.
- Effectiveness of Autonomous Decision-Making: The paper highlights AutoCode4Math’s advantage in reducing code executions and improving accuracy, but lacks direct performance comparisons with established models like GPT-4 and Mammoth. Quantitative evidence of accuracy improvements tied to better method selection would strengthen the claims of AutoCode4Math’s benefit.
- Comparisons with Standard RL Techniques: Although the paper briefly mentions an offline RL baseline, it does not detail the data, algorithm specifics, or other critical aspects of that experiment. A comparison with rejection-sampling and preference-learning-based approaches would further contextualize AutoCode4Math’s methodology and showcase its relative effectiveness.

questions:
### Motivation for Using EM
1. **Why is EM chosen over alternative methods like standard reinforcement learning (RL) or supervised fine-tuning?**  
   - What specific advantages does EM bring to methodology selection that simpler or more commonly used techniques cannot provide?

2. **How does EM uniquely contribute to the narrowing of the policy search space?**  
   - Could you provide empirical evidence or a theoretical basis for how EM improves efficiency in selecting between Chain of Thought (CoT) and code-based approaches?

3. **What is the theoretical or experimental basis for assuming that EM will reduce computational load or enhance efficiency in methodology selection?**  
   - How does EM’s efficiency in narrowing the search space compare to RL or other optimization methods in this particular task?

### Comparison with Baselines for Model Selection
4. **How does AutoCode4Math perform compared to current baselines, specifically in terms of method selection effectiveness?**  
   - Could you provide a detailed, controlled comparison with baseline models like Qwen2Math, GPT-4o, and DeepseekMath, showing how frequently and accurately AutoCode4Math selects CoT versus code?

5. **What improvements in accuracy or reduction in code execution does AutoCode4Math achieve over established models such as GPT-4 and Mammoth?**  
   - Are there quantitative metrics available that directly correlate the model’s method selection accuracy with overall performance improvements?

6. **Could you expand on the offline RL baseline and provide details on the data, algorithm, and setup of this experiment?**  
   - What comparisons are made with other RL methods or preference-learning-based approaches? How does AutoCode4Math fare against these methods, particularly in terms of accuracy and computational efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666532099

thread_type:
official_review

<thread_object>
creator: reviewer_3njd

<content>
summary:
The authors identify a major challenge in current training regimes where models are typically told explicitly whether to execute code. Addressing this, the paper introduces an innovative EM algorithm that allows models to choose their execution paths autonomously, either using CoT reasoning or code execution. The results are quite impressive, showing a 65.28% accuracy in mathematics-related tasks, alongside a 65% reduction in code execution. This advancement not only improves efficiency but also enhances the practical applicability of the model.

strengths:
1. The performance of the proposed method is commendable. It even surpasses the SFT approach on the same set of math related queries, which indicates its effectiveness and potential for broader application.
2. The area of LLM reasoning with tool usage is an important topic. 
3. This paper involves many in-depth analysis on the proposed method and different experiment settings.

weaknesses:
The organization of the second section of the paper appears somewhat confusing. It could benefit from a restructuring to enhance readability and coherence. Specifically, introducing an overarching pipeline at the beginning of the section could provide a clearer framework. And the relationship between sections 2.2 and 2.3 with section 2.1 needs to be more explicitly defined to better understand the flow and integration of ideas.

questions:
1. Given that the proposed method primarily requires math-related queries, has there been any exploration into having the model generate its queries autonomously? This could potentially lead to self-improvement paradigm, further enhancing its effectiveness and applicability.
2. Will the authors consider open-sourcing the code? 
3. Have the authors tried larger size models? (e.g. 13B)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709898151

thread_type:
official_review

<thread_object>
creator: reviewer_pg3w

<content>
summary:
The paper introduces a method to train an LLM to select between use code-as-a-tool or chain-of-thought-reasoning actions, allowing the model to do some degree of inference-time search between the two actions. The training process is instantiated as an off-policy RL problem without labels for intermediate steps, by maximizing the joint probability of picking the correctness-maximizing action and correct response, decomposed into an expectation step to estimate state-action-values (via random rollouts) and a maximization step to train on the reward-weighted actions. 7B models trained using this method show improvement over just 2 iterations on the GSM8k and MATH datasets.

strengths:
Significance: The topic addressed in this paper is exciting, with the release of o1 the community is interested in methods that can improve inference time decision making for math, especially on smaller models.
1. The results on GSM8k and MATH are interesting, over just 2 training iterations AutoCode improves performance in a stable way over baseline RL (Figure 4). 
2. The method only relies on input and outputs, not using human generated labels is useful for data bottlenecked researchers. Framing the RL actions as CoT vs. code generation addresses the gap between self-learning the optimal policy instead of behaviorally cloning (SFT) a given policy as with prior models.

weaknesses:
Performance:
1. Assuming I followed the table in Figure 3 correctly, it seems this method does not outperform the Qwen2-Math 7B-instruct model on any math dataset, which uses a simpler training method. The authors can try to see if their method improves Qwen2-Math 7B-instruct further. The method also doesn't give significant performance gains outside of MATH and GSM8k datasets over open-source baselines.

Presentation: 
1. Section 2 is difficult to follow, it should be organized into clearer components by aligning better to Figure 2, highlighting (1) how the data is generated (expectation) and (2) how the model is trained on that data (maximization). I summarized the paper above based on my understanding, please clarify if there is misalignment.
2. The main results, Figure 3, looks like a spreadsheet screenshot, please re-organize it. I am not sure I followed the results of this figure properly (see weaknesses in performance section). Figure 4 needs to be larger.
3. Several typos, Figure 1: "sythesis" and Appendix B "referece"

questions:
1. If you extend the AutoCode training to more iterations (Figure 4), does the performance on GSM8k and MATH continue to improve? It looks like the accuracy is still going up.
2. Why is the initial accuracy for Qwen-2-Math on GSM8k and MATH very different between Figure 3 and Figure 4? 
3. I can't view the anonymized repo, can you double check it works? I would like to see the code and try the final model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717282779

thread_type:
official_review

<thread_object>
creator: reviewer_uehe

<content>
summary:
In this paper, the authors address the problem of letting LLMs autonomously decide which planning strategy to use, between plain CoT and interaction with a python interpreter, in order to reliably solve math and reasoning problems. The authors correctly observe that such autonomic decision making cannot be induced via simple SFT on expert-annotated data, and therefore propose an RL strategy based on the EM algorithm, as a way to allow the model to self-explore and train itself on its own generated trajectories.

strengths:
- The authors observation that autonomous code integration cannot be taught via simple SFT is definitely correct.
- The authors consider also multi-turn interactions featuring an interleaving of CoT and Python code execution, which is not found very often in the CoT and code execution literature.

weaknesses:
The main weakness I can see is that it is not quite clear why the authors' EM approach is necessary. While I do agree that SFT is not sufficient for the task at hand, there are plenty of much simpler alternatives to the authors' approach, which would equally allow self-exploration and on-policy self improvement. Given how complex the authors' framework is, I feel that it is even more necessary to outline its advantage and convince readers that it is indeed worthwhile and preferable to simpler methods.

More points I wish to make:

- The author's outline of the method is extremely long and technical, and difficult to digest for any reader not already familiar with EM. Focusing more on its high-level idea and providing at least a few examples would go a long way.
- The main result table in figure 3 is honestly difficult to parse. Some lines are color-coded in blue and red, but the text and caption do not explain what this means. It's not clearly stated which lines correspond to the authors' proposed approach and which correspond to baselines. I personally would focus on comparing the author's proposal with whatever method happens to be the SOTA for the benchmarks the authors consider, in a much smaller table.
- Some typos can be found here and here, including in the abstract.

questions:
- The authors' EM approach appears very complicated and requires a lot of moving parts, and is intended as a replacement for simple SFT. There are much simpler and proven ways of replacing SFT, such as Expert Iteration and RLHF, which can provide similar advantages in terms of self-exploration. Could the authors more clearly outline why their EM approach is preferable to these alternatives, and which advantage exactly is provided by each of its components?
- In section 2.3, the authors outline a framework for data synthesis. Fine-tuning LLMs on synthetically generated data is a proven technique for enhancing their abilities. What benefit do the other component of the EM framework bring?
- What exactly is the "standard RL" baseline mentioned in section 3.2? The authors should be a bit more specific.
- Again in section 3.2, the authors state "We conjecture that the proposed EM framework outperforms standard RL because: the
reference methodology-selection strategy helps narrow down the policy search space during training". Can the authors provide any experimental evidence for this statement?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732445597253

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Rebuttal by Authors

comment:
Dear Reviewers and Area Chairs,

We sincerely thank all the reviewers for their constructive and thoughtful feedback. ***We are particularly encouraged by the following positive remarks:***

**Significance of the Autonomous Code Integration:**

- ﻿Reviewer **#uEHE**: "The authors observation that autonomous code integration cannot be taught via simple SFT is deìnitely correct"

- Reviewer **#Pg3w**: "﻿The topic addressed in this paper is exciting."

- Reviewer **#3NjD**: "﻿The area of LLM reasoning with tool usage is an important topic."

- Reviewer **#nGHB**: "﻿This research identiìes an important gap in current language models' ability to self-direct their reasoning strategies"

**Contribution and Novelty of the Proposed Method:**

- Reviewer **#nGHB**: "﻿The study makes a signiìcant contribution by addressing how models can autonomously choose between .."

- Reviewer **#Pg3w**: "﻿The method only relies on input and outputs, not using human generated labels is useful for data bottlenecked researchers."

- Reviewer ﻿**#uEHE**: "﻿The authors consider also .., which is not found very often in the CoT and code execution literature"

**Effectiveness and Soundness of the Proposed Method:**

- Reviewer **#nGHB**: "﻿Through their experiments, the authors demonstrate that .. leads to measurable performance improvements".

- Reviewer **#3NjD**: "﻿The performance of the proposed method is commendable."

- Reviewer **#Pg3w**: "﻿The results .. over just 2 training iterations AutoCode improves performance in a stable way"

﻿We deeply appreciate these positive comments and are grateful for the reviewers' recognition of our work.

Besides, we extend our sincere thanks for the valuable suggestions on improving the presentation of the manuscript. Based on this feedback, ***we have uploaded a revised version of the manuscript. We marked in blue the revisions in the uploaded pdf and summarized the revisions in the above post (https://openreview.net/forum?id=QhjosARfay&noteId=nQdxk1NfaQ)***.

In addition, ***we have carefully addressed each reviewer's questions individually***, including the following key points:

- Clarified the motivation and unique benefits of the proposed methods (per #uEHE, #nGHB's feedback).

- Conducted detailed comparisons with relevant alternatives (per #uEHE, #nGHB's feedback)

- Performed additional experiments to further analyze the performance convergence of the proposed approach (per #Pg3w's feedback)

- Provided an in-depth analysis to justify how the proposed method helps narrows the policy search (per #uEHE, #nGHB's feedback)

- Provided an in-depth analysis of methodology selection accuracy and its connection to final accuracy (per #nGHB's feedback)

***Reviewer #uEHE and Reviewer #Pg3w have acknowledged that their concerns have been addressed and recognized the improvements made in the paper.*** They have updated their scores to eight and six, respectively. We are currently awaiting further feedback from Reviewer #nGHB and Reviewer #3NjD.

We sincerely thank all the reviewers and the area chairs for their time, effort, and thoughtful insights throughout the review process. Your feedback has been invaluable in refining our work, and we deeply appreciate your contributions.

Best regards,

The authors of Submission 5899
</content>
</thread_object>

</thread>
<thread>
creation date:
1732445731696

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Paper Revision

comment:
In response to the reviewers' suggestions for improving the presentation of the manuscript, we have made the following refinements:

**Restructuring Section 2 for Improved Readability and Coherence:**\
Following reviewer **#3NjD**'s feedback, we have reorganized Section 2 into three subsections:

1. Section 2.1 formally introduces the problem statement.

2. Section 2.2 presents the EM framework for solving this problem.

3. Section 2.3 provides a streamlined and efficient implementation of the EM framework.

**Refining Technical Derivations of the EM Framework:**\
Incorporating reviewer **#uEHE**'s advice to use examples and **#3NjD**'s suggestion of an overarching outline, we have made several enhancements:

-   At the start of Section 2.1, we clearly articulate the problem challenges (line 145) and use an analogy based on human cognitive processes to motivate the discussion (line 152, as advised by #uEHE).
-   We introduce an overarching pipeline (line 160, per #3NjD's suggestion) before diving into technical derivations.
-   The derivations are now more concise and focus on high-level ideas, addressing reviewer #uEHE's feedback.

**Adding Comparative Analysis with Relevant Alternatives:**\
Based on feedback from reviewers **#uEHE** and **#nGHB**, we have included a new paragraph discussing the advantages of our proposed approach compared to SFT and standard RL (line 226).

**Aligning Section 2 with the Figures:**\
In response to reviewer **#Pg3w**'s advice to better align section 2 with Figure 2, we do the following:
- Before detailing the EM framework, we summarize the high-level idea of the proposed method as self-exploration in E-step and self-refinement in M-step (line 180), which now aligns with the left side of the figure.
- In discussing the EM framework, we explain how the E-step identifies the reference strategy (line 212) and the M-step improves policy (line 223), which now aligns with the middle of the figure.
- In discussing the practical implementation of the EM framework (Section 2.3), we describe data curation and efficient joint training which now corresponds to the right side of the figure.

**Improving Figures 2, 3, and 4:**\
Following reviewer **#Pg3w**'s suggestions, we have refined the figures as follows:

-   Figure 2: Enhanced to better align with the revised manuscript's method overview.
-   Figure 3: Replaced with a LaTeX table for clarity, presenting baseline results and highlighting improvements achieved by our approach.
-   Figure 4: Adjusted for size and revised to remove occluded numbers for improved readability.

**Adding Experiments**

Based on the feedback of reviewer  **#uEHE** and **#nGHB**, we put the comparison on autonomous code integration with baselines in the Appendix Section C.

**Misc**

We highlight in the abstract that code and data is released via an annonymous repo, as we notice some reviewers do not find the repo we previously note in line 373.
 
Thank again for the reviewers' valuable suggestions, which greatly help us improve the manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, the authors address the problem of letting LLMs autonomously decide which planning strategy to use, between NL CoT and code generation / execution, in order to reliably solve math and reasoning problems. The author observe that such autonomic decision making cannot be induced via simple SFT on expert-annotated data, and therefore propose an RL strategy based on the EM algorithm, as a way to allow the model to self-explore and train itself on its own generated trajectories. They show improvements on GSM8k and MATH.

Reviewers generally liked the strengths of the reported results. All reviewers did not like how the method was presented in the paper and most mentioned not understanding the details in the EM and the ELBO bound. This paper treats using NL vs. code as the latent variable and then applied EM. While EM is a 60 year old method, it is not commonly used in LLM-training where neither steps are solvable. Reviewers nGHB and Pg3w questioned how this method should be compared and baselined against RL and if they were correctly compared with EM. After thinking about this, this is quite important since it is unclear why EM is more suitable at all since p(reward | latent c) can be estimated by rollouts. EM is needed when the latent variables cannot be observed, which does not seem to be the case here.

So this paper took a very simple concept (model picks which strategy to use), and results that seem to be strong (good improvements over the original baseline and over a RL baseline on 1+ round), but wrote it so that reviewers and I cannot figure out what's actually the main method and how can it be possibly be better than baseline (within a reasonable amount of time). Here are some reasons why I think it's not just due to us not understanding the authors properly:

* The authors included unnecessary details of how EM works like the ELBO bounds. The key to adding EM is just what is the latent variable and what are the parameters. They should highlight how EM can possibly be better.

* nGHB: The authors did not clearly identify what is their RL baseline or even what is the supervised baseline, since their EM also needs to do the rollouts and collect some sufficient statistics p(r | c, x). They could have clearly explained which part of the EM is not handled well by RL, but they did not. This is a very important point, since there are many possible RL baselines, some of which can easily model p(c | x) using Bayes rule, and p(r | c) with a reward model, bring it to feature parity with the EM method. There were no mentions or discussion of this nor is it included in the main result table.

* They do not control for or mention inference time compute so it is hard to interpret the results. For this paper, I think the empirical strength matter a lot. For instance, Deepseek actually gets 60%+ by aggregating over 64 samples. Using the tool integrated setting, Deepseek gets 86.7% on GSM8k and 58.8% on MATH. Their paper seems to fall under this tool-integrated setting. While under the RL COT, which is also similar to their setting, reporting 88.2% 51.7%. 

Voting reject since the reviews were mixed and we should discourage papers that make a simple concept hard to understand and not presenting results against reasonable baselines. The authors are encouraged to simplify and highlight the key improvement if they believe in their method, instead of framing it as EM vs RL or SFT.

Reviewer uEHE voted accept after author rebuttal but does not explain why, so their review still reads like a reject. I took their opinion into account but cannot see how they advocate for acceptance.