Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling
Authors: Yitian Chen
Jingfan Xia
Siyu Shao
Dongdong Ge
Yinyu Ye
Venue: NeurIPS 2025 poster ()
Keywords: Optimization
Optimization Problem Formulation
Problem Definition
Foundation Model
Primary area: optimization
TL;DR: We propoed the first application of RLVR to directly enhance LLMs' proficiency in optimization modeling: SI-RL, achieves state-of-the-art  on diverse public benchmarks

Abstract: Optimization modeling is fundamental to decision-making in fields such as supply chain management, logistics, and financial engineering, but its complexity presents a major barrier to adoption. Automating model creation from natural language is key to improving efficiency and access. However, while Large Language Models (LLMs) are a promising tool for this, they often produce flawed or infeasible results due to errors and hallucinations.
   To address this issue, we propose Solver-Informed Reinforcement Learning (SIRL), a framework that uses Reinforcement Learning with Verifiable Reward to improve LLMs’ ability to generate accurate and executable optimization models. Specifically, SIRL automatically assesses the executable code and the instance-level mathematical model represented by the associated .lp files. This process yields precise feedback on syntactic validity, feasibility, and solution quality, which serves as a direct reward signal to guide the reinforcement learning process. Furthermore, this verification mechanism also supports our instance-enhanced self-consistency method for creating high-quality training data.
    Extensive experiments on diverse public benchmarks demonstrate that models trained with our SIRL framework achieve state-of-the-art performance, substantially outperforming existing methods in generating accurate and executable optimization models. Specifically, our SIRL-32B model surpasses DeepSeek-V3 and OpenAI-o3 on the majority of these benchmarks.
    Our code is publicly available at https://github.com/Cardinal-Operations/SIRL.

Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling
Yitian Chen';
Jingfan Xia?.;
Siyu Shao!
Dongdong Ge Yinyu Ye4s
Cardinal Operations, China
Shanghai University of Finance and Economics
3 The University of Hong Kong
Antai School of Economics and Management, Shanghai Jiao Tong University
Department of Management Science and Engineering, Stanford University
chenyitian@shanshie ai, xf.x110163. sufe. cn,
siyu shw.cconnccccc..c...o... h.u. hk, ddge@sjtu edu. yyye@stanford.com edu
Abstract
Optimization modeling is fundamental to decision-making in fields such as sup- ply chain management, logistics, and financial engineering, but its complexity presents major barrier to adoption. Automating model creation from natural language is key to improving efficiency and access. However, while Large Lan- guage Models (LLMs) are promising tool for this, they often produce flawed or infeasible results due to eπrоrs and ballucinations. To address this issue, we pro- pose Solver-Informed Reinforcement Learning (SIRL), framework that uses Re- inforcement Learning with Verifiable Reward to improve LLMs ability to generate accurate and executable optimization models. Specifically, SIRL automatically as- sesses the executable code and the instance-level mathematical model represented by the associated .1p files. This process yields precise feedback on syntactic validity, feasibility, and solution quality, which serves direct reward signal to guide the reinforcement learning process. Furthermore, this verification mech- anism also supports our instance-enhanced self-consistency method for creating high-quality training data. Extensive experiments on diverse public benchmarks demonstrate that models trained with our SIRL framework achieve tfte-of-theeart performance, substantially outperforming existing methods in generating accurate and executable optimization models. Specifically, our SIRL-32B model surpasses DeepSeek- V3 and OpenAl-o3 on the majority of these benchmarks. Our code is publicly available at https://github.cc wwt.ggit......narteeeeeeeoo com/Cardinal-Ceee comm/Cardiall-ppeaationnS///
1 Introduction
Optimization modeling provides a powerful framework for decision-making across diverse fields, from logistics and finance to engineering and machine learning [II D, E. [J]. The process involves two primary steps: first, converting natural language problem description into a formal mathemat- ical model, and second, employing a solver to solve an optimization problem to get the optimal solution. Despite the maturity and powerful capabilities of modern optimization solvers, such as Gurobi [o]. COPT [ and CPLEX [8], formulating these intricate real-world problems into precise mathematical models and executable codes remains a significant bottleneck, often requiring substan- tial domain expertise and considerable manual effort [2], D
Botth authors contribute equally to this research.
Conresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To bridge this gap, modeling languages such as MiniZine [I0]] and Essence [|]]]] emerged, offering a high-level declarative syntax abstract away low-level solver details, particularly for constrained programming. However, even with these languages, the crucial first step, transforming a natural lan- guage problem description into a correct and efficient model, remains formidable manual task [[2]]. The advent of Large Language Models (LLMss, (e.g., GPTss [13]] Gemini [TT] [S], Deepseek [[]6]), offers a promising avenue to automate or assist in this mathematical modeling and code generation process, potentially democratizing access to optimization solvers. However, ensuring the correct- ness, feasibility, and solver-compatibiiiyy of --gnerrrte optimization models presents a signifi- cant and increasingly active research challenge. In general, existing approaches that leverage LLMs for optimization modeling can fall broadly into two categories. Firstly, prompt- boaptt-baedd based or agent-based approaches utilize the frozen capabilities of powerful foundation LLMs [TE3, IT. IEL [ guide the models to extract relevant problem information, and generate the corresponding mathematical mod- els and executable codes for optimization solvers [I17, 18, 199 20J. Although accessible, these meth- ods do not adapt the underlying model parameters and can be sensitive to prompt design and the chosen foundation LLMss.
distinct line of work addresses training open-source LLMs [2I, 22] to enhance their capabilities for optimization modeling using offline learning approaches. This encompasses techniques such as Supervised Fine- Tine-Tunig (SFT) on demonstrations data, and alignment techniques (e.g., DPO [2S], KTO [ on preference data. The success of existing offline methods [25, 26, [x] hinges the availability of carefully curated datasets. These datasets, comprising problem descriptions paired with either detailed mathematical models and code demonstrations or comparative human prefer- ence labels, are typically generated through human annotation [27]] or synthesis [25,, [6]]- Train- ing on these curated datasets enables offiin learning approaches to capture the data's stylistic and structural patterns, such as mathematical formulation and solver code, and achieve good perfor- mance. However, since their training objective focuses on mimicking demonstrations aligning preferences, these methods still struggle to inherently guarantee functional correctness or solution feasibility, which are essential for reliable solver execution.
Recently, the development of powerful Large Reasoning Models (LRMs) marks a major lcap in the reasoning capabilities of LLMs [28, 29 BOJ. The process begins with
(CoTss prompting [TI]. which unlocked the ability for LLMs to generate explicit, multi-step reasoning paths. Building on this, Reinforcement Learning with Verifiable Rewards (RLVR) [29, (2) provides the essential feedback mechanism to ensure the correctness of these paths. RLVR directly optimize
LLMs' policies using objective feedback from verified outcomes, such as the validation of mathe-
matical solutions or the execution of generated code against test cases [22 23,
Н. GR [4]. This
powerful combination has led to LRMs [28, 23, [i] that excel at complex reasoning, demonstrated by high performance on challenging mathematical competitions like the AIME and the International Mathematical Olympiad (IMO)
For optimization tasks, solving a real-world problem involves a multistep process encompassing problem analysis and reasoning, mathematical modeling, and code implementation [[7] I8. 19, 20. 25. 26, [2]], resembling a CoT reasoning process. The outputs of this process-the the mathematical model and the solver code-can be verifiable using external optimization solvers. Verification in- volves steps such as syntax checks, feasibility assessment through model solving, and comparison of the objective value against known optimal value. These verifiable checks yield objective and richness reward signals, enabling RLVR [G5), 29 62, G6] to directly optimize the LLM generation towards producing correct, feasible, and high-quality outputs for complex optimization problems.
To the best of our knowledge, this is the first application of RLVR to directly enhance LLMs' profi- ciency in optimization modeling. The rewards, including fcasibility status, objective function value, and mathematical model statistics from the .1p file, are obtained by executing the generated code. Specifically, the .1p file, which is standard format for optimization solvers, provides an explicit instance-level representation of the generated mathematical model, implicitly reflecting the LLM's reasoning process. This allows to move beyond simple, results-based rewards and design more refined, process-aware reward [37]. By combining common solver outputs with detailed statistics from the .1p file, we ingeniously blend the correctness of the outcome-based reward with the fine-grained nature of the process-based reward. This enables a solver-informed evaluation that en- sures accurate assessment of the LLM's performance and validity both mathematical modeling and outcome correctness.
2

---

Our main contributions are fourfold: (1) We introduce simple yet effective instance-enhangn self- consistency method for synthesizing high-quality training data for optimization tasks. (2) We intro- duce SIRI., in optimization modeling with a novel surrogate function. By enabling a balance between diverse reasoning exploration and the requirements for accuracy and validity in mathematical models and code, this function leads to a significant improvement the authenticity and reliability of the gen- erated optimization solutions. (3) We demonstrate how classical optimization solvers can serve as effective and powerful tools for both enhancing the data synthesis process and providing rich reward signals for the proposed SIRL framework.
reinforcement learning, an automated RIVR framework for LLMSs
(4) Through extensive experiments diverse public benchmarks, our SIRL-trained 7B model achieves state-of-the-an performance, significantly outper- forming existing offline learning and agent- based methods in generating correct and reliable models. Moreover, our larger 32B model establishes new benchmarks by surpassing powerful baselines like DeepSeek-V3 and OpenAT-o3 on the majority of these benchmarks.
2
Related work
Our work builds upon and contributes to several research areas, primarily LLMs for optimiza- tion, synthetic data generation for LLM training, the paradigm of reinforcement learning with tool- verified feedback mechanisms.
LLMs for optimization modeling. The application of LLMs for optimization modeling has emerged as a prominent research direction. Early work mainly relied on prompt engincering tech- niques, including agent-based prompting [ПЯ]] and multi-agent reasoning [T& 20J. but was limited by careful prompt design and the capability of the foundation LLMs. More recent work focuses on offline learning approaches to adapt LLM parameters using specialized mathematical modeling datasets. For instance, ORLM [25]] and OptMATH [2ū]] employed Supervised Fine- Fine-Tuning (SFT) with datasets constructed via semi-automated synthetic data generation workflows, LLMOPT [27]] introduced an alignment learning framework with multi-instruction tuning and a self-correction mechanism.
LLM-Based data synthesis. Fine-tuning LLMs on specialized tasks necessitates high-quality datasets, a resource-intensive requirement often necessitating domain expertise. Data synthe- sis offers a scalable solution. Examples of general-purpose synthesis techniques include self- instruction [38]] and the WizardInstruct series [39, 20. [Ill Specifically, to improve reasoning ca- pabilities, recent work focuses on synthesizing reasoning trajectory data, exemplified by the work [_ aiming to enhance complex reasoning ability in tasks such as math and code. Within the domain of optimization modeling, several approaches have been explored: ORLM [25] introduced semi-automated method for synthesizing operations research datasets based on self-instruct frameworks [G8]; ReSocratic [45] employs reverse data synthesis via back-translation; and OptMATH [26] generates controllable complexity data with verification.
Star [22], rstar [+50] and Phi
Ensuring correctness and formal adherence is a central challenge when synthesizing high-quality data for tasks requiring rigorous output. Techniques addressing this include LLM-as-a-judge [[6], which leverages foundation models for evaluation, rejection sampling [17 26]. which provides au- tomated verification, and self-consistency [38], which relies on mamli voting for robustness.
Reinforcement learning with verifiable reward. Reinforcement Learning from Human Feedback (RLHF) [48] marked a significant step in aligning LLMs, typically using reward models trained on human preferences [19 550]. However, for tasks where the desired output has objectively verifiable properties, such as mathematical reasoning [511, 50 [52) and code generation [533) 54]] relying solely on subjective human preference is suboptimal and can struggle with objective correctness and poten- tial reward hacking [155, 56, 57] For example, model might generate code with correct syntax but a flawed function. RLVR has demonstrated significant success in enhancing LLMs' performance across these domains, evidenced by strong results on benchmarks like GSMSK [50]] MATH [[:3], AIMO [52]. HumanEval [533], CodeForce [544], and has been a key technique in developing highly capable large reasoning models (e.g.. OpenAl-O1 Dee).eee R1 [29]] Kimi-k1.5 (GIT) and state- of-the-art art coding models (c.g.. Tulu-3 [32]).
External tools as verifier. Leveraging external tools or formal verification mechanisms to validate I. L.e--geeee structured outputs, particularly in domains that require high fidelity and correctness, is becoming increasingly critical area of research [58, 36 59). In mathematical theorem proving,
3

---

the Lean proof assistant [60]] acts as verifier for LLM-generated proofs [59, 58. 611 52].. Googles AlphaGeometry series 663. [ combines reasoning LLMs with symbolic engines DDAR, achieving breakthroughs in the IMO-level mathematical tasks. For formal domains like mathematics and pro- gramming that demand high output fidelity, integrating code interpreters is commonly employed to improve I.LMs' performance and ensure correctness. The basic approach involves generating com- plete solutions and validating them post-hoe via external execution using interpreters or compilers [5.3, 655]. Feedback from these validation checks provides a signal for model refinement. More ad- vanced methods either integrate tool interactions directly within the LLMs' reasoning process (e.g.. ToRA [660), MARIO [6ZI]) or focus enabling the LLMs to learn effective strategies for using tools autonomously (e.g.. START [33], ReTool
[64]]].
Our work proposes a direct analogy: just as Lean verifies mathematical proofs and code compilers verify code for general math and coding problems, classical optimization solvers [R IZ, & 688 serve as the natural, powerful, and domain-speciff objective oracles, which can both enhance the synthe- sis of high-quality optimization task data and provide rich reward signals for the proposed RLVR framework.
3 Method
Let the training dataset be D ((x;, o])) where x; represents the natural language description of the i-th optimization problem and yf is the corresponding ground-truth optimal objective function value. We model the problem solver as an LLM policy To parameterized by Given an input problem description I, the policy Ta generates response containing sequences of reasoning process leading to an objective function value y, derived via a mapping function g(x; z).
To guide the learning process, we introduce the SIRL framework, which incorporates a verifiable reward function r(x, z. v²). This function quantifies the quality of the derived objective value y for the problem x, using the ground truth y" as a reference. Our goal is to optimize the policy parameter 0 to maximize the expected reward:
max 8
y²)]
In the following subsection, we outline the key components of our framework: the data synthesis pipeline used to construct the training dataset, the solver-informed reinforcement learning method with its surrogate function design tailored for optimization tasks and the two-stage training curricu- lum including reward design.
3.1 Data synthesis framework
Overall framework. The data synthesis process starts from curated seed data and integrates LLM generation with subsequent steps including self-instruction, augmentation, evaluation, con- sistency assessment, and filtering. While our work primarily follows the OR-Instruct pipeline in ORLM [2S]. this approach contrasts with previous work synthesizing complete (question, model, code) sequences [25 20, (59] by focusing on generating high-quality (question, answer) pairs, where the answer is the optimal value by executing an optimization solver.
As illustrated Figure we firstly sample questions from the curated seed data, combine them con- textually with scenarios sampled from predefined list, and generate new problems that remain struc- turally similar to the original. Subsequently, we implement an augmentation phase to increase the challenge related to semantic interpretation, mathematical modeling complexity, and solution com- plexity, obtaining a larger corpus of extended questions. Then, the LLM-as-a-judge approach [26] validates the generated problems for practical relevance and semantic consistency. Following the validation the problem, the LLM is employed to generate mathematical models and correspond- ing executable code. This code is then executed to produce the objective value, feasibility status and the instance-level mathematical models represented by .1p files.
To ensure high correctness of the final answer, we assign multiple LLM roles (10 roles) per prob- lem [ZO, [Z]] and apply a novel instance-enhanced self-consistency framework when generating an- swers. Furthermore, an iterative reflection and refinement process [[22] is employed to address exe- cution issues, regenerating or refining code upon errors, and regenerating the model and code when infeasible solutions are encountered.

---

Reject
Training data
Scenario list
Agumentation
LLM as judge
Questions
Extended Questions
(Question, answer)
Seed dltta
Multiple LLM roles to obtain the answers
Refine and regenerate
Filtering
Infeasible
min cxx
Feasible
Self col-oossistnncy
Mathematical model
Code
Solver
Lp
Error
Refine and regenerate
Figure 1: An overview of the data synthesis process.
Finally, guided by the principle "Less is More" Samples are excluded a baseline Qwen-32B-Inseee model [22]] achieves an 80% pass rate (8/10 attempts across different roles) in generating executable code matching the optimal value, these instances are considered too elementary. The retained pairs are incorporated into the final training
74
filter the (question, answer) pairs.
dataset.
In the next part, we detail our novel, simple yet effective instance-cnhanced celf-consistency method used within data synthesis framework.
Relying solely on majority voting of final results for self- consistency in optimization modeling can be limiting, potentially ignoring embedded model in- formation. We enhance this by integrating structural data extracted from the instance's .1p file. The 1p files are chosen as they formally encode key model properties (e.g., variable types, objec- tive direction), providing a formalized, implementation-agnooii
Instance-enhanced self-consistency.
representation of the instance-level mathematical model. Specifically, after executing the generated code associated with role and obtaining the corresponding 1p file, we extract the following features:
O,: The final objective function value.
D, E (max, min]: The optimization direction (maximization or minimization).
: The count of binary (0-1) variables.
Nint,:: The count of general integer variables (distinct from binary variables).
These statistics provide detailed, model-level insights that supplement the final numerical outcome O,.- Let R be the set of roles that generated responses for given question. We assign a score S(r) the response of each role r E R using a weighted voting mechanism that measures consensus with other roles. We define a consensus function (X,,) for feature X E {0, D, Noi, Nint as the count of the roles r' E Rt whose corresponding feature value X,, identical X,::
((,)) = Itr R X = X,H|.
(2)
The final score S(r) for the response from the role is calculated weighted sum reflecting consensus across the extracted features:
S(r) = w1 (O,) + w2 (D,) + w3 (NNei...)) + w4 / (V(
The weights (Wi, W2, u3, W4) determine the relative contribution of the individual consensus com- ponents to the final result. In our current implementation, all weights are set to 1, giving equal importance to each consensus component.
5

---

Trander teu Ciinci C1866- 3309 Trandr far Cilid cT
Cude
Traste tea Clised Clack 88. Traals bon Clac7-3853 Trander Yeu Biee Cloac Weelbarneeeeee can
Qlestie:: tragtic soah cardinaiig
Oijectts faaste valte e00
the difican ciilie aa papuae be ipoen
latgie vartalles eiRect
acotian vocc of supiles spur rupirsment to permii anfornation
LLMb
Asmer: Metenss nnd 6811
Coei wcllk :000
eeppicc
dlick
iepormile
Code
Salver
Opinnal Casl 9015
Clae kx 1:23
Oijeetto feestii vake p10,11 Blaa raralles al wit
LPttc
Faalsor
Figure 2: An overview of the data synthesis process.
Figure D provides an illustrative example of this enhanced celficonsi method. For given question, multiple LLM roles generate "Math model + Code" trajectories. Each code is executed by solver, producing a solution and the corresponding .1p file. From the .1p file, we extract structural features (objective value consensus (O,),, optimization direction (b(D)), variable counts v (NBin.)) ()). The final score calculated according to Equation Finally, the objective
value O,- from the response achieving the highest score, where r = arg max S(r), is selected as the definitive answer to the question.
3.2 SIRL: Solver-Informed Reinforcement Learning
RLVR for LLMs in optimization modeling. In optimization tasks, to obtain the optimal value from problem description æ, a complex reasoning process is involved with distinct stages: first, analyzing the problem description to identify the key information, such optimization problem type and its core components; second, constructing the mathematical formulation which typically involves the parameter set, objective functions, and constraints; and finally, generating the corre- sponding executable code, which then executed by the optimization solver to produce the objec- tive value and other relevant output (e.g., decision variable values and solution status). To address this complex reasoning process that integrates both mathematical modeling and code implementa- tion, utilize the Chain of Thought (CoT) [GI]] method, where the LLM policy ma generates sequence of intermediate thoughts z = (z ",,...,""" initial problem and the final result y.-
2 that serve as a crucial bridge between the
Specifically, a well-designed system prompt structures the sequence of thoughts z into segments reflecting the defined reasoning, modeling, and code generation stages: (z² ,,,...,,""" z²²)) contains the analysis and reasoning process (e.g., the identification of optimization problem type, algorithmic choices, reasoning steps towards the final model structure); zm contains the concise mathemat- ical modeling; and z contains the derived executable code. The final value y is obtained deter- ministically by extracting and executing code in zm formally represented y g(x, z), where g denotes the deterministic code execution function.
Update
<think>
Solver
Decision variables
Policy
<model>
Execution
Questions
Objective value
I.I.M
<code>
Reward
Advantage
Binary/ Integer
Rollout
Answers
.p file
Big M
Nonlinea
Figure 3: An overview of the SIRL training framework.

---

At the token level, each thought z² is realized as sequence of tokens z² (z,,...,,27,) The token of within this sequence is sampled autoregressively from the model's policy To(-1x, z² ...'.""'11 -1:- .....: z(-1), conditioned on the initial input all previously completed thoughts (z' ...2!=1), and all tokens generated far in the current thought.
Surrogate function design: Partial KL. To maximize the expected verifiable reward objective defined in Equation (I), we employ REINFORCE+ [[/5], robust policy gradient algorithm that incorporates key techniques from Proximal Policy Optimization [766].
In each training iteration, a batch of data {=;, V/ } is sampled from the training dataset. Then, for each 24, the policy яe is used to sample a set of K complete response trajectories { where each 2 is sequence of tokens generated autoregressively and composed of a sequence of thoughts. For simplicity, we denote the collected batch as B, where each tuple (zz, z, yy) ) E B consists of æ, the input problem description, 2, the generated response trajectory, and y", the ground-truth objective value.
The algorithm updates the policy parameters 0 by maximizing the following clipped surrogate ob- jective:
T3 Σ
プラ=1 T min (p/A (p,,1-c,1+ 44)
J RReinforee++ (0)
B
(Z,
where is the clipping hyperparameter; pf [a xxx)))) is the probability ratio of generating
new <,,
token 2, under the new policy versus the reference policy mows and A denotes the token-level advantage computed for token a!-
Building on this algorithmic structure, the per-timestep reward signal f and its corresponding ad- vantage A/ that is normalized across the mini-batch are defined as follows:
P/ I[= [EOS))r(,,, (2,,"""" 3KL(j, (),
A -P/ (x://
where I[(s/ [EOS]) is an indicator function that assigns the reward r(x, z. v.y") only when of is the end-of-sequence token. A token-level KI. penalty component, KI.(j, t), is included penalize policy deviations from the reference policy Told-
To reconcile the tension between exploratory reasoning trajectories diversity (which may deviate significantly from the reference model distribution) and strict adherence to mathematical formula- tion/solver syntax requirements in optimization tasks, we propose Partial KL. This novel design selectively applies the KL penalty to the mathematical formulation zn and solver code z" seg- ments. The value for the KL term, KL(j, t), within these segments is computed using the unbiased estimator described in [|Z]]
KL(J,)) - ( - pf -1 j e {m 1, m]) otherwise.
10
(5)
The detailed training process is illustrated in Figure B.
The rationale for the Partial KL design that utilizes selective KL regularization is twofold:
1. Exploration in reasoning: For reasoning steps (z) .z....,,,. zm 2-2), 2), the KL penalty is omit- ted. This fosters exploration, enabling the policy to better understand the problem back- ground and identify diverse reasoning paths and implicit constraints [78, [79].
2. Stability in modeling and code generation: For the critical mathematical formulation z" -=-1 and solver code zd segments, the KL penalty ensures the generated output remains well-structured and adheres to expected formats, preventing policy collapse while still al- lowing gradual improvement guided by the reward.
Our SIRI. framework, which incorporates Partial KL, enables the policy to achieve effective ex- ploration and understanding of the problem while improving code execution accuracy, and yields high-quality outputs for optimization tasks.
7

---

3.3 Reward design and training scheme
The success of our SIRL framework hinges on its verifiable reward function, implemented as a staged, rule-based system [29] B2]. Integrated with the optimization solver, this system provides objective verification signals used within a two-stage curriculum [80 8 to progressively train the model and enhance its optimization modeling capabilities.
Given a question x, generated trajectories z, ground-truth answer y", the two-stage reward function r(I, z, v²) is defined as follows:
{
Riforma (2) + Rexec (.) + Raccur (2. 2, y²)
Stage-1, Stage-2.
r( r(z, (1,,,9))-
(6)
Riormat (2) + Rexec (z) Raccur [2, 2, y²) Roonus (x, &, 1/"
In stage-1, we focus on building the model's fundamental capabilities in formulating and solving standard optimization problems. The reward function comprises three key components: format, execution, and accuracy. Emphasis is placed on the execution component via additional incentives for correctly executed code. This ensures that the generated models both mathematically sound and executable.
Building fundamental capabilities, Stage 2 aims to tackle more complex problems through the bonus reward Riomus. which is based on the generated mathematical model associated with the LP file and designed to incentivize advanced modeling techniques (e.g., Big-M, nonlinear formulations) which are crucial for complex and challenging problems. This bonus is granted only when two conditions met: (1) the generated solution is correct, and (2) it incorporates advanced modeling strategies. The complete reward function formulation is detailed in the Appendix.
4 Experiments
4.1 Main results
Table 1: Performance comparison of models on benchmarks.
Acc (pass@1) 1)
Types
Models
Macro AVG
NL4OPT MAMO
MAMO
IndustryOR OptМATH
Easy
Complex
Baseline
GPT-4
89.0%
87.3%
49.3%* 63.5%
33.0%
16.6%
55.0%*
DeepSeck-V3.1
84.8%
88.9%
44.0%
43.9%
65.0%
DeepSeck-RI
82.4%
87.2%
67.9%
45.0%
40.4%
64.6%
LRMs
OpenAl-o3
69.4%
77.1%
51.2%
44.0%
44.0%
57.1%
Agent-based
OptiMUS
78.8%
77.2%
43.6%
31.0%*
20.2%*
49.4%
ORIM-1 aMA-3-88
85.7%% *
82.3%*
37.4%
24.0%*
2,6%
46.4%
Offlin-lleamngg
LLMOpt- LLMMOpt--wen2.5-11 Owen2.5-14B
80.3%
+
44.1% 1%6
29.0%*
12.5%
51.1%
OptMATH--weenl.o
94.7%* *
86.5%
51.2%
20.0%
24.4%
55.4%
OptMATH- -00pAHHH--wn22.-32
95.9%
89.9%
54. 1%
31.0%
34.7%
61.1%
Online-RL
SIRL-OWen2.5-7BB SIRL-Qwen2.5-32B
96.3%
91.7%
51.7%
33.0% 42.0%
30.5% 45.8%
60.6%
98.0%
94.6%
61.1%
68.3%
Values marked with *
from original or repraduced papers with the criterion: relative
10-6
To evaluate the effectiveness of the proposed SIRI. framework, we developed two models at differ- Both were initialized from their respective and Qwen2.5- 1ene225-32B-Inttreee [22]] without any prior su- pervised fine-tuning. Performance was evaluated on four benchmarks- NL4OPT [82]] MAMO [X]] IndustryOR [2>] and OptMATH [ -using pass @ accuracy. Consistent with the evaluation proto-
ent scales, SJRL-Qwen2.5-7B and SIRL-Qwen2.5-322
base instruction models, -wwe22.57BBIInse
col proposed by OptMATH [26], solution is considered valid if the relative error is less than 1e-6. A detailed description of these datasets and evaluation criterion is provided in Appendix H.. Table []] presents the main results.
Performance of SIRL-QQwe2.5-7B. Our SIRL-QWee2.5-70 model consistently outperforms all existing 7B and 14B models trained with other offline learning methods [25, 12, 26],, as well as the agent-based approach OptiMUS [[T]]
8

---

Performance of SIRL-QQwe2.5-32BB The SIRL-Qwen2.5-322 model demonstrated superior per- formance across all evaluated benchmarks. Despite its significantly smaller size, our 32B-333BBaatt -araameee model achieved higher Macro Average than much larger models, including the 671 B--ppppp 1--parameter foundation model Deepseek-V3. 13.1 and powerful reasoning models like DeepSeek-RI and OpenAl- o3.
These finds highlight the efficiency of our proposed SIRL mechanism in enhancing LLMs' abili- ties for optimization formulation and code solving, demonstrating its ability of tackling complex optimization modeling challenges. Details regarding our experimental setup, which include prompt templates, training hyperparameters, and decoding strategies, are provided in the Appendix for a complete overview.
4.2 Instance-enhanced self-consistency
Table 2: Performance of value-based and instance-enhanced self-consistency on Qwen2.5 Models.
NL4OPT
MAMOEasy
MAMOComplex
IndustryOR
OpIMATH
Average
Metric
7B
32B
7B
32B
7B
32B
32B
7B
32B
32B
pass @
65.7%
67.8%
81.9%
83.7%
17.1%
26.5%
19.0%
23.0%
4.1%
16.6%
37.6%
43.5%
val_sc@5
69.8%
70.2%
85.3%
86.0%
25.1%
35.1%
25.0%
32.0%
5.7%
22.3%
42.2%
49.1%
inst sc@5
70.6%
71.0%
85.4%
86.3%
29.9%
38.4%
26.0%
33.0%
13.0%
27.5%
45.0%
51.2%
(inst-val)
1.1%
1.1 %
0.1%
0.3%
19.1%
9.4%
4.0%
3.1%
128.1%
23.3%
30.5%
7.5%
vaL_sc@ 10
68.6%
73.5%
85.6%
85. 9%
29.9%
38.4%
28, 0%
34.0%
9.8%
27.5%
44.4%
51.9%
inst_ sc@ 10
69.0%
72.2%
85.7%
86.0%
32.2%
39.3%
30.0%
36.0%
16.6%
34.2%
46.3%
53.5%
Diff (inst-val) @ 10
0.6%
-1.8%
0.1%
0.1%
7.7%
2.3%
7.1%
5.9%
69.4%
24.4%
17.0%
6.2%
We evaluated different self-consistency approaches on the Qwen2.5 models [22]] (7B-Instruct and 32B-Instruct to assess the effect of leveraging instance-level information. The value-based self- consistency method (val_sc) is a direct adaptation of the standard self-consiseency approach where the final score of different roles depends only on the final objective function value. The instance- enhanced coniisensst method (inst_st (onst_sc also includes structural information within the generated optimization models, augmenting the consensus mechanism. The consensus function of the final objective function value, optimization direction, the count of binary variables, and general integer variables are given the same weight in Equation EL
Table 0 indicates that self-consistency through majority voting outperforms the baseline single-pass generation (pass@) @ I). Both val_sc sc and inst_se methods demonstrate consistently higher accuracy than the pass@1 baseline. Furthermore, a comparative analysis between the two self-consistency variants suggests that incorporating instance-level information (optimization direction, variable counts) into the voting mechanism provides more robust measure of consensus, leading to im- proved selection of correct solutions compared to relying solely on the final objective value.
4.3 Ablation Study
In this section, we present a series of ablation studies examine the impact of the surrogate function design based on the Partial KL strategy and the proposed two-stage reward mechanism.
4.3.1
Ablation study on different surrogate function designs.
Table 3: Ablation study different surrogate function designs.
MAMO Complex
IndustryOR
OptMATH
Type
Acc(pass I)
ER
Acc(pass@1)
ER
Acc(pass@@ I)
ER
Partial KL
51.7%
98.1%
33.0%
96.0%
30.5%
92.2%
Full KL
48.3% 48.3%)
98. (1.45)
30%0()
95.0% 15.0%(11.00% 1.0%)
28. 28.3%(11..666
93.4%((1.29%)
Without KI.
47.3%(([4.4%)
(12.5)))
29%0[4))%))
87.0%(19.0%)
29.5%((1..9%)
80. 10.12 16)
We evaluated three distinet surrogate function designs: (i) Full KL: the standard approach apply- ing full KL-divergence regularization against the reference policy; (ii) Without KL: an approach

---

omitting KL-divergence regularization. which is popular in RLVR training for mathematical prob- lems [778. 7 such as AIME [52]]: (iii) Partial KL: our novel design that applies the KL penalty selectively to the mathematical formulation and code segments.
reppotss both the pass@1 @ accuracy and execution rate (ER), which measures the percentage of generated solutions that successfully compile and return a valid result, across three more challenging datasets. The results show that the proposed Partial KL approach achieves the best performance across all benchmarks. In contrast, the Without KL design exhibits a dramatically lower execution rate than the other two strategies. This lower rate stems from removing KL divergence: while pro- moting more diverse exploration, it can lead to introducing irrelevant constraints from the problem background, increasing invalid generations. Partial KL resolves this issue by applying KL se- lectively, improving the execution rate while preserving reasoning diversity. The full comparison on all benchmarks and a detailed qualitative analysis, including case study, are presented in the Appendix.
4.3.2
Ablation study on reward design.
Table 4: Performance results of the ablation study on reward design.
Ace (pass @1)
Reward Type
NLAOPT
MAMO
MAMO Complex
IndustryOR
OptMATH
Easy
Two-stage rewards
96.3%
91.7%
51.7%
33.0%
30.5%
Stago-1 reward only 96.7% (10.4%)
88.8% (2.9%)
46.8%
(4)
27.0% (16.0%)
28.9% (L1.6%)
Stage-2 reward only 92.2% (14.1%)
89.6% (12. 1%)
49.3% (12.4%)
28. 28.0% (15.0%)
33.1% (12.6%)
Ablation study on reward design. We compared the performance of the proposed two-stage reward mechanism (6) against models trained using only the stage-1 I reward and using only the stage-2 re- ward. As shown in Table H. using only the stage-l reward yielded comparatively strong results on simple tasks such as NL4OPT. This indicates that this reward enables the model to learn stable foundational skills for optimization tasks. Meanwhile, employing only the stage-2 reward, which in- cludes a bonus component incentivizing the model to learn advanced strategies, achieves the best per- formance on the most challenging OptMATH dataset. However, this led to diminished performance on simpler tasks such as NL4OPT. Overall, the integrated two-stage reward mechanism successfully balanced the objectives of the individual stages, resolving the trade-offs observed with single-stage rewards, thereby achieving superior performance in most benchmark tasks, with the exception of the OptMATH dataset.
5 Conclusion
In this paper, we present SIRL, novel RLVR framework addressing the challenge of generating authentic mathematical optimization models with LLMs. The core contributions of this work are its unique surrogate function design, Partial KL, which selectively applies KI. divergence to math- ematical formulation and code segments, and its two-stage reward system, which leverages opti- mization solvers for automated verification. The comprehensive signals derived from this verifica- tion were valuable for both RL training and enhancing our data synthesis. Extensive experiments showed that the SIRL-trained model achieved superior performance in generating accurate, well- formed optimization models compared to existing methods. More broadly, the proposed techniques applicable to tasks requiring LLMs to balance exploring diverse reasoning with ensuring solution validity, particularly in augleneeeeteed tasks. However, challenges such as reward hacking persist, and performance difficult benchmarks like IndustryOR and OptMATH remains limited even for advanced models. Our future work will therefore target these gaps through systematic error analysis and targeted improvements.
Acknowledgments and Disclosure of Funding
This research is partially supported by the National Natural Science Foundation of China (NSFC) [Grant NSFC-72225009, 72394360, 72394365].
10

---

References
[1] Ajay Singh. An overview of the optimization modelling applications. Journal of Hydrology. 466:167-182, 2012.
[2] Andreas Antoniou Wu-Sheng Lu. Practical optimization algorithms and engineering applica- tions, 2007.
[3] Mazura Mokhtar, A Shuib, and D Mohamad. Mathematical programming models for portfolio optimization problem: A review. International journal of social, management, economics and business engineering. 8(2):443-45) 2014.
[4] Gerard Cornuejols and Reha Tütüncü. Optimization methods in finance. Camegie Mellon University, Pittsburgh, 2005.
[5] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[6] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2024.
[7] Dongdong Ge, Qi Huangfu, Zizhuo Wang, Jian Wu, and Yinyu Ye. Cardinal optimizer (copt) user guide. arXiv preprint arXiv:2208. 14314, 2022.
[8] IBM Ilog. User's manual for cplex. http:///ww. ilog, com/, 2010.
[9] David G Luenberger, Yinyu Ye, et al. Linear and nonlinear programming, volume 2. Springer, 1984.
[10] Nicholas Nethercote, Peter Stuckey, Ralph Becket, Sebastian Brand, Gregory J Duck, and Guido Tack. Minizinc: Towards standard cp modelling language. In International conference on principles and practice of constraint programming, pages 529-543. Springer, 2007.
[11] Alan M Frisch, Warwick Harvey, Chris Jefferson, Bernadette Herefnez-Hernnndezz, and Ian Miguel. Essence: A constraint language for specifying combinatorial problems. Constraints, 13(3):268-306, 2008
[12] Dimos Tsouros. Hélène Verhaeghe, Serdar Kadıoğlu, and Tias Guns. Holy grail 2.0: From natural language to constraint models. arXiv preprint arXiv:2308 0855999,, 2023.
[13] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 0887 2023.
[14] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Sapttsap Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: a family of highly capable multimodal models. arXiv preprint 2312..:2312.110000
2023.
[15) Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
[16] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[17] Rindranirina Ramamonjison, Haley Li. Timothy T Yu, Shiqi He, Vishnu Rengan, Amin Banitalebi-Dehkordii, Zirui Zhou, and Yong Zhang. Augmenting operations research with auto-formulation of optimization models from problem descriptions. arXiv preprint 2209. 15565, 2022.
[18] Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, et al. Chain-of-experts: When llms meet complex operations research problems. In The Twelfth International Conference on Learning Representations, 2023.

---

[19] Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. Optimus: Scalable optimization modeling with (MI)LP solvers and large language models, arXiv preprint arXiv:2402.1017. 2024.
[20] Nicolás Astorga, Tennison Liu, Yuanzhang Xiao, and Mihaela van der Schaar. Autoformula- tion of mathematical optimization models using Ilms. arXiv preprint urXiv2441.006679 2024.
[21] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
[22] An Yang. Baosong Yang. Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
[23] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[24] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiw:2402.01366, 2024.
[25] Chenyu Huang, Zhengyang Tang, Shixi Hu, Ruoqing Jiang, Xin Zheng, Dongdong Ge, Benyou Wang, and Zizhuo Wang. Orlm: A customizable framework in training large models for auto- mated optimization modeling. Operations Research, 2025.
[26] Hongliang Lu, Zhonglin Xie, Yaoyu Wu, Can Ren, Yuxuan Chen, and Zaiwen Wen. Optmath: scalable bidirectional data synthesis framework for optimization modeling. arXiv preprint 12Xi::2502.1102 102, 2025.
[27] Caigao Jiang, Xiang Shu, Hong Qian, Xingyu Lu, Jun Zhou, Aimin Zhou, and Yang Yu. LI- mopt: Learning to define and solve general optimization problems from scratch, arXiv preprint arXiv:2410. 13213, 2024.
[28] OpenAl.
Learning to reason with 1lms.
https://opena..com//
learning to--t-o-rroooo with with-1lms/ Accessed: 2024-09-12.
[29] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprini arXiv:2501. 2025.
[30] Kimi Team, Angang Du Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12500 2025.
[31] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 11.24824-24881 2022.
[32] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T ulu Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 2024.
[33] Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang. Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. Start: Self-taught reasoner with tools. arXiv preprini arXiv:2503. 046625 2025.
[34] Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.1 1536, 2025.
[35] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 202::24402.03000 2024.
12

---

[36] Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrieval- augmented language models. Advances in Neural Information Processing Systems, 36:21573- 21612, 2023.
[37] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[38] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions, arXiv preprint arXiv:22 22. 10560, 2022.
[39] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models follow complex instructions. arXiv preprint AXX::23304112444 2023.
[40] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08555 2023.
[41] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering math- ematical reasoning for large language models via reinforced evol-instruct. arXiv preprint 2XXY:2308.09583, 2023.
[42] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
[43] Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Iims can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.044 1955999 2025.
[44] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint 24Xi:24422008005 2024.
[45] Zhicheng Yang. Yinya Huang, Wei Shi, Liang Feng, Linqi Song, Yiwei Wang. Xiaodan Liang, and Jing Tang. Benchmarking iims for optimization modeling and enhancing reasoning via reverse socratic synthesis. arXiv e-prints, pages arXiv-2407, 2024.
[46] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
[47] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint ar3iv: 01825, 2023.
[48] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[49] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, An- tonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. arXiv preprint arXiv:2211.14275 2022.
[50] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. arXiy preprint 1rXi:2110.11466. 2021.
13

---

[51] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprini arXiv:2103.03874, 2021.
[52]
AI-MO Team. AI-MO Validation and AIME Test Set Dataset. Hugging Face Datasets, 2024.
[53] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint urXiv:2107.03374 2021.
[54] Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, al. Codeelo: Benchmarking competition-level code generation of Iims with human commancco elo ratings. arXiv preprint arXiv:2501.01257, 2025.
[55] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and charac- terizing reward gaming. Advances in Neural Information Processing Systems, 35:9460 94771, 2022.
[56] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: 2016.
[57] Lilian Weng. Reward hacking in reinforcement learning. lilianweng, github. io, 2024.
[58] Huajian Xin, ZZ Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, et al. Deepseck-prover-v// 5: Harnessing proof as- sistant feedback for reinforcement learning and monte-carlo tree search. arXiv preprint arXiv:2408.08/ 08152 2024.
[59] Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem prov- ing. arXiv preprint MrXiv:2009,0339; 2020.
[60] Leonardo de Moura and Sehastian Ullrich. The lean theorem prover and programming lan- guage. In Automated Deduction-CADD 28: 28th International Conference Automated Deduction, Virtual Event, July 12-15, 2021, Proceedings 28, pages 625-635, Springer, 2021.
[61] Huaiyuan Ying, Zijian Wu, Yihan Geng, Jiayu Wang. Dahua Lin, and Kai Chen. Lean work- book: A large-scale lean problem set formalized from natural language math problems. arXiv preprint arXiv:2406.. 03847, 2024.
[62] Bohan Li, Jiaxuan Wang, Cong Fang, Fan Yang, Jing Xiong, Yingkang Xu, Shiyang Li, Xiang Li, Songfang Huang, and Jingren Zhou. DeepSeek-Prover- Advancing Formal Mathe- matical Reasoning via Reinforcement Learning for Subgoal Decomposition. arXiv preprint arXiv:2504.2180) 2025.
[63] Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-482. 2024.
[64] Yuri Chervonyi, Trieu H Trinh, Miroslav Olšák, Xiaomeng Yang, Hoang Nguyen, Marcelo Menegali, Junchyuk Jung, Vikas Verma, Quoe V Lc, and Thang Luong. Gold-medalist perfor- mance in solving olympiad geometry with alphageometry2. arXiv preprint arXivv2502.033.4 2025.
[65] Chengpeng Li, Guanting Dong, Mingfeng Xue, Ru Peng, Xiang Wang, and Dayiheng Liu. Dotamath: Decomposition of thought with code assistance and self-correction for mathemati- cal reasoning. arXiv preprint arXiv:2407.04078, 2024.
[66] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309. 11199//7455 /7452, 2023.
[67] Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output-a reproducible pipeline. arXiv preprint arXiv:2401.08190 2024.
14

---

[68] MOSEK ApS. Mosek optimization suite. 2019.
[69] Zhicheng Yang, Yiwei Wang, Yinya Huang, Zhijiang Guo, Wei Shi, Xiongwei Han, Liang Feng, Linqi Song, Xiaodan Liang, and Jing Tang. Optibench meets resocratic: Measure and improve lims for optimization modeling. arXiv preprint arXiv:2407.09888 2024.
[70] Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint wrXiv:2406.2009/ 2024.
[71] Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, et al. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:2407.21009, 2024.
[72] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634 8652, 2023.
[73] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma. Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-55021, 2023.
[74] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXIv:2502.03387,220 2025.
[75] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprini arXiv:2501.03262. 2025.
[76] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 10X.v:/707.0644 2017.
[77] John Schulman. Approximating kl divergence. John Schulmans Homepage, 2020.
[78] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system scale. arXiv preprint arxiv:2503. 14476, 2025.
[79] Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xi- aoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork
open reasoner series.
attrae//capriccioue-lrrrrgegg notion.oton....m site/
2025.
Notion Blog.
[80] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
[81] Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Au- tomated curriculum learning for neural networks. In international conference machine learning, pages 131 3-1-03 Pmlr, 2017.
[82] Rindranirina Ramamonjison, Timothy Yu, Raymond Li, Haley Li, Giuseppe Carenini, Bis- Ghaddar, Shiqi He, Mahdi Mostajabdaveh, Amin Banitalebi-Dehkonn
Zirui Zhou, et al. NI. competition: Formulating optimization problems based on their natural language de- scriptions. In NeurlPS 2022 Competition Track, pages 189-203. PMLR, 2023.
[83] Xuhan Huang, Qingning Shen, Yan Hu, Anningzhe Gao, and Benyou Wang. MAMO: A mathematical modeling benchmark with solvers. arXiv preprint, 2024.
[84] Mokhtar Bazaraa, John Jarvis, and Hanif Sherali. Linear programming and network flows. John Wiley & Sons, 2011.
[85] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rihf framework. arXiv preprint arXiv:2409, 240..2499,. 2024.
15

---

[86] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 10ii:1904.09751,2000 2019.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See the conclusion
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This is an application work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See the appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We provide the checkpoint and the test code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public//uides/Codeer SttiedeeeeoodeSSSssssiiiiioo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips.cc/ppblli/uuiddde CowelleeeooeeSSeeeeeeeeeo wttp:/aaeeeeeeeeeaaaaaaaaaaaa.aa ssionnolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specifyy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See the experiment part. More details can be found in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
19

---

is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://arrp..ccpaalaa/laacacaaaa
Answer: [Yes]
Justification: NA.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitiga- tion strategies (e-g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
allow for controlled use of the model, for example by re-
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: NA.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
21

---

Answer: [Yes]
Justification: We will upload the checkpoint.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
22

---

Justification: NA
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard componetss.
Please refer to our LLM policy httess//Deeerrriccccc/oorreeeeee for what should or should not be described.
23

---

A Technical Background for Automated Optimization Modeling
The automated conversion of natural language problem into optimization model follows multi- step pipeline, as illustrated in our work. This process transforms an unstructured text description into verifiable mathematical artifacts and then into solved optimization problem. To fully appreciate the mechanisms behind our data synthesis and RL framework, we overview the core components below.
A.1 Core Components
The pipeline relies on the interplay of several key components:
Natural Language Problem A natural language problem is an unstructured textual description of a decision-making scenario, often presented as word problem real-world query (e.g., How should a factory allocate resources to maximize profit given limited labor and materials?). These problems are intuitive for humans but ambiguous and non-executable for computers. It's necessary to translate into formal mathematical representations.
Mathematical Model A mathematical model formalizes a real-world decision problem into structured optimization problem. consists of three primary elements: (1) decision variables, which represent the unknowns be determined (e.g.. quantities produce); (2) objective function, mathematical expression to maximize or minimize (e.g.. profit or cost); and (3) constraints, which enforce limitations or requirements (e.g., resource availability or logical conditions).
Solvers are the computational engines the heart of optimization. They are highly specialized algorithms designed to find optimal solutions to formally defined mathematical problems. Their input is a precise mathematical model (variables, constraints, objective), and their output is solution (if one exists).
Optimization Solvers
Modeling Libraries Modeling libraries are Application Programming Interfaces that provide bridge between general-puryy
programming language (like Python) and the low-level solver. They allow developers to define variables, constraints, and objectives using familiar programming constructs, abstracting away the complexities of direct solver interaction. The code generated by the L.M in our work utilizes these libraries.
Optimal Solution and Optimal Value The optimal solution refers to the set of values for the decision variables that satisfy all constraints and achieve the best possible value of the objective function (e.g., maximum profit or minimum cost). The optimal value is the corresponding objective function value this solution point. For infeasible or unbounded problems, no optimal solution exists, and the solver indicates the status accordingly.
The .1p File Format The .pp file is standardized, human-readable text format for representing linear and mixed-integer programming problems. It serves as a universal blueprint the mathe- matical model, independent of the programming language library used to create it. This solver- agnostic nature makes it an ideal artifact for formally verifying the structure of a generated model. typical file includes sections for the objective function, constraints, variable bounds, and variable types:
Objective Function: Begins with Maximize or Minimize, followed by the objective func- tion expression.
Constraints (Subject To):
This section lists all the linear constraints of the model. Each constraint an equation or inequality involving the decision variables.
Bounds:
This section specifies the lower and upper bounds for each variable (e.g., 0). If omitted, default bounds are often assumed by the solver.
Variable Types (General, Binary): This section declares which variables must be integer (General or Integers) or binary (Binary). Variables not declared here are assumed to be continuous.
End: A final End statement marks the end of the file.
24

---

A.2 A Concrete Example
To illustrate the entire process, consider the following simple production-ne problem. This example shows the complete chain: from natural language formal model, then Python code using modeling library, which in turn uses solver to produce both a final solution and a verifiable .1p file. Both the numerical solution (e.g., objective value) and the structured .p artifact crucial for providing the rich feedback signals used in our framework.
Natural Language Problem:
Natural Language Problem
A factory produces products, tables and chairs. Each table yields profit of $16 and requires 2 hours of labor and 4 units of wood. Each chair yields profit of $10 and requires hour of labor and units of wood. The factory has 80 hours of labor and 150 units of wood available. How many tables and chairs should be produced maximize profit?
2. Mathematical Model:
Mathematical Formulation
Let Tiables and Tchuirs be the number of tables and chairs to produce. mathematical model is:
The formal
maximize
16 + 10.
(Total Profit)
subject to
2nubles + Fehuins
S 80
(Labor Constraint)
4.tubles 2.Echairrs
< 150
(Wood Constraint)
Euables Tehairs
2 0,
(Integer Variables)
3. Python Code: Here is the Python code that uses the GurobiPy library to implement the mathematical model.
25

---

Python Code(using the GurobiPy Modeling Library)
import gurobipy as from gurobipy import GRB
Create a new model
Sp.Model ("poedlcppouction"
# Create variables
tables m. addVar (vtype INTEGER, name= tabless)
chairs m. addVar (vtype=GRB INTEGER, name= chairs")
# Set objective
m. www.sbeeeettiee666.... tables
chairs, GRB. MAXIMIZE)
Add constraints addConstr (2 tables 1 chairs <= 80, "labor")
m. addConstr tables 2 chairs 150, "wood")
Write the LP file for inspection write peproductiio_mmdee__1pp
# Optimize model m. optimizemie
# Print solution print (fojeccv value: {m om.ol""))
4. Generated .1p File:
production model.lp
Model production
\ LP format for model browsing.
Maximize
16 tables + 10 chairs Subject To
labor: 2 tables + chairs <= 80
wood: tables 2 chairs <= 150
Bounds
Generals
tables
chairs
End
5. Solver Output and Solution:
The 'mootiiiiee() command then invokes the Gurobi solver, which finds the optimal pro- duction plan. The typical output includes:
Status: Optimal solution found.
Objective Value: 750.0 (Maximum Profit)
Decision Variables:
tables 0
chairs 75
B Benchmark dataset
We evaluated the performance our trained models on four key optimization modeling datasets:
NL4OPT [2]] MAMO [3]] IndustryOR [25]] and OptMATH [6]]. As noted in prior work [2A, 27].
26

---

errors exist within the test sets of these benchmarks. To ensure the integrity of our evaluation, we rigorously reviewed and corrected the test sets, updating both the questions and corresponding solutions, with a specific focus on the NL.4PT and IndustryOR datasets. The sample statistics, before and after this revision process, are summarized in Table 5
Table 5: Summary statistics for the optimization problem datasets
Dataset
Number Before Correction
Number After Correction
NL4OPT
245
245
Mamo Easy
652
642
Mamo Complex
211
203
IndustryOR OptMATH
100
100
166
166
Originating from the NLAOPT competition, this benchmark assesses the conver- sion of natural language problem statements into executable optimization models. It pre- dominantly features linear programming (LP) tasks drawn from diverse domains, although the complexity of the underlying mathematical structures is relatively consistent. A final set comprising 245 high-quality instances was obtained by filtering low-quality instances [26]]
NLAOPT:
MAMO consists of subsets: MAMO EasyLP and MAMO ComplexLP. MAMO EasyLP contains 652 instances, primarily high-school level mixeddinteger linear program- ming (MILP) problems designed for fundamental learning; MAMO ComplexLP includes 211 more challenging undergraduate--ever problems, integrating both LP and MILP con- cepts suitable for assessing advanced modeling skills. Note that to guarantee the quality of this benchmark for our experiments, we performed a thorough data curation process. We manually verified and corrected the MAMO EasyLP and MAMO ComplexLP dataset.
IndustryOR: Introduced the first benchmark specifically targeting industrial applica- tions, IndustryOR comprises 100 real-world optimization scenarios sourced from various sectors. It distinguishes itself by incorporating wider range of problem types, including linear programming (LP), integer programming (IP), mixed-integer programming (MIP), nonlinear programming (NLP), and other specialized formulations, categorized across three difficulty levels.
The benchmark was constructed to address the limitations of existing datasets, particularly their frequent focus linear problems and the potentially insufficient chal- lenge level. OptMATH features a curated selection of complex mathematical optimization problems designed to effectively differentiate advanced modeling capabilities. Its scope is intentionally broad, encompassing LP, MILP, IP, NLP, Second-Order Cone Programming (SOCP), and other relevant optimization paradigms. Two variants of the dataset exist, with 166 and 193 instances, respectively. We present the results from the larger 193-instance variant in the main results part to ensure a thorough evaluation.
OptMATH:
Note that to ensure the validity of our findings and contribute more robust benchmark for the community, we performed a manual verification and correction of the datasets used our evaluation. This rigorous process involved identifying and rectifying range of issues within the original data, including problems with contradictory or incomplete conditions. flawed logi- cal premises, and incorrect ground-truth answers. We believe these curated versions enhance the reliability of benchmarking for future research. For the benefit of future research, com- prehensive description detailing our correction methodology. the rationale for each modification. and the revised datasets are publicly available at the Github repository https://github.com http://tthub.cmm/aaaiiiiii /aapatoonn/SIRL//o/////aaaa
data/README.nd
We evaluate performance across these benchmarks using a strict accuracy metric: a generated solu- tion is considered correct if the related difference between its predicted objective value Upred and the ground truth value Ytohel satisfies:
Upred
Mabel
< 10 1
Vlabett
27

---

Similar to [26]. the enror about the type of the variables are ignored. We thoroughly review problems exhibiting such errors. In these cases, the decision variables typically represent time or similar quantities, which can reasonably be treated as either integer or continuous variables.
C
Seed data
The seed data consist of 686 real-world industry cases collected from operations research textbooks and list of 100 industry scenarios [23]] Here is example:
Seed data example
Problem: ** Restaurant Seating Allocation Problem**
**Problem Background:*
The tourist restaurant seating allocation problem involves arranging the distribution of 100 seats in the restaurant at two different periods to improve customer satisfaction, optimize ser- vice efficiency, and maximize restaurant revenue. This common management challenge in the catering industry, especially during peak dining periods.
* #Ottimization Goals:**
Maximize customer satisfaction: Improve customer dining experience through reasonable seat allocation.
Maximize restaurant revenue: Maximize restaurant revenue by optimizing seat allocation. Optimize service efficiency: Ensure that the restaurant can operate efficiently at different times and reduce waiting time.
** **Numerical Problem Constraints:*#
1. **Total Seats Limit:
At any given period, the total number of seats allocated must not exceed the restaurant's total capacity of 100 seats.
**Fluctuating Demand Across Periods:**
In Period 1, the seat allocation must be between the minimum demand of 20 seats and the maximum demand of 60 seats.
In Period 2, the seat allocation must be between the minimum demand of 30 seats and the maximum demand of 70 seats.
** **Ojeettive Function:**
The objective is to maximize customer satisfaction and restaurant revenue, where the com- bined weight for customer satisfaction and revenue per allocated seat is 1.0 for each.
##Forullation:**
Let 'x[i. jl' denote the allocation of seat 'j' in period 'i' (where i 1, 2 for two periods and j= 1, 1, 100' for total seats).
** **Ojjective Function:**
Maximize:
M Isum \sum_[i=1] (2) \sum_[j=1] (100) Meft(( 1.0 1.0 \right) x(i.
** *#Conttriitss**
Total seat allocation in each period should not exceed the total available seats:
M[ sum sum_(j=1) [100] x[i. jl Meq 100, 'quad \forall \in 1, 21) \]
2. Seat allocation per period must satisfy minimum and maximum demand:
N 20 Ale ssum[jj=]] [ 100) x[1. j] Mlg 60 \J \[ 30 Veg (am__(j=]]([] x(2. jl Veg 70 V] ** **Exeetted Output:**
The optimal seating allocation for each period ('xfi, j] values) that maximizes customer satisfaction and revenue while adhering all constraints.
Answer: 260.0
28

---

D Prompt templates
In this section, we summarize the prompts used in the framework.
D.1 Ten roles for self-consistency
In the self-consistency mechanism, we generate multiple candidate solutions for each problem by prompting the LLM with distinct roles. Each role represents specific expert persona, designed to elicit varied approaches to optimization modeling by emphasizing different combinations of ex- pertise in operations research, Python development, and Gurobi solver application. The ten roles are defined as:
Ten roles
Role 1:"A highly skilled Python engineer and optimization specialist with deep expertise operations research and the Gurobi solver."
Role 2:" An optimization expert and Python engineer specializing in operations search and the Gurobi solver."
Role 3:"A A Python engineer and optimization specialist with a strong background in operations research and the Gurobi solver."
Role 4:"A AA skilled Python engineer and optimization specialist proficient in opera- tions research and the Gurobi solver."
Role 5:"A results-driven Python engineer and optimization expert with strong foundation in operations research and the Gurobi solver."
Role 6: "A seasoned operations research scientist and Python developer, leveraging advanced optimization techniques and the Gurobi solver to tackle complex business challenges."
Role "An innovative optimization modeler and Python programmer, specializ- ing in the development and implementation of high-performance solutions using operations research methodologies and the Gurobi optimization suite." "
Role "A pragmatic problem-solver with expertise in operations research, pro- ficient in Python and the Gurobi API, focused on translating real-world scenarios into efficient and scalable optimization models."
Role 9: "A meticulous optimization analyst and Python coder, deeply familiar with the theoretical underpinnings of operations research and the practical application of the Gurobi solver for achieving optimal outcomes."
Role 10: "A strategic optimization architect and Python implementation specialist, with a proven track record of designing and deploying robust operations research solutions powered by the Gurobi optimization engine."
D.2 I.L.-as-a-judge for generated problem evaluation
We employ the L-a---s-a methodology [Н6] to validate the generated problems for practical relevance and semantic consistency. The prompt utilized for this validation process is detailed below.
29

---

Problem evaluation prompt
You are an expert in operations research. You'll II receive an operations research problem. You will analyze it and determine whether the problem is valid one considering the following aspects:
Determine the problem's 's language and structure are consistent with typical operations research problem formulations.
Assess whether the problem scenario has real-world applicability or practical significance.
3. Identify any semantic inconsistencies, contradictions, ambiguities within the problem statement.
Below is the operations research problem:
[Question]}
Please provide your step-by-step analysis and your final judgment for each of these points.
## Analysis Process:
(Your detailed step-by-step reasoning for each point above]
## Final Judgment:
[Yes No]
D.3 Refine and regenerate for the error case
Upon encountering code execution errors, we leverage the LLM for refinement and code regenera- tion [2]. The prompt used for this error correction mechanism as follows.
Error regenerate prompt
You experienced operations research algorithm engineer. You presented with an operations research problem and previous attempt to model and code a solution. That attempt resulted in error.
Problem Description:
[Questioon
Previous Code Solution Attempt:
(Previous c
After running the provided code from the previous attempt, the following error occurred: {[Erro output after executing the code))
Your task:
Based the information above, please perform the following:
1. Analyze Root Cause & Identify Pitfalls
Thoroughly analyze the root cause of the error.
Summarize potential pitfalls or common mistakes related to this type of code error.
Provide Corrected Gurobi Code:
Write the complete and corrected Python code using the 'gurobipy' library to accurately solve the problem.
Please structure your response strictly as follows:
## Cause the Error and Potential Pitfalls: [Your detailed analysis of the error's cause and summary of potential pitfalls.]
## Corrected Gurobi Code:
[Your complete and corrected Gurobi Python code.]
D.4 Refine and regenerate for the infeasible case
If an infeasible solution is obtained after executing the code using the solver, we leverage the L.M to refine the entire result by regenerating both the mathematical model and corresponding code [72]]. The prompt used for this infcasibility resolution mechanism is follows.
30

---

Infeasible regenerate prompt
You are experienced operations research algorithm engineer. You are presented with an operations research problem and previous attempt model and code solution. That attempt resulted in an infeasible solution.
Problem Description:
[ }
Previous Model and Code Solution Attempt:
[Previous model]]
{ (Previous code]
After running the provided code from the previous attempt, the answer could not provide feasible solution.
Your task:
Based on the information above, please perform the following:
1. Analyze Root Cause & Identify Pitfalls
Thoroughly analyze the root cause of the infeasibility.
Summarize potential pitfalls or common mistakes related to this type of infeasibility.
2. Provide an Improved Mathematical Model: Develop a mathematical model for correctly modeling this OR problem. This should address the flaws in the previous attempt.
3. Provide Corrected Gurobi Code:
Write the complete and corrected Python code associated with the mathematical model using the 'gurobipy" library to accurately solve the problem.
Please structure your response strictly as follows:
## Cause of the Infeasibility and Potential Pitfalls:
(Your detailed analysis of the infeasibility's 's cause and summary of potential pitfalls.] ## Corrected Mathematical Model:
[Your improved mathematical model.]
## Corresponding Gurobi Code:
[Your complete and corrected Gurobi Python code associated with the mathematical model.]
D.5 System prompt and the LLM responses
We follow the approach of DeepSeek-RR -Zeo [2 by providing a specific output format within the system prompt. This format guides the LLMs through sequential process: it first asks the LLM to analyze the problem, then formulate a mathematical model, and finally output the correspond- ing solver code. To ensure clear delineation and parsability of these steps, we prompt the LLM to explicitly separate each component using distinct tags: <think><//tinkk. <model></model>> <python></python>> An example output response guided by the system prompt is provided below. The corresponding problem was sourced from the NL4OPT dataset.
System Prompt
SYSTEM:
You are helpful Assistant with expertise in operations research and the Gurobi solver. When the User provides OR question, you will analyze it, build detailed mathe- matical model, and provide the Gurobi code to solve it. Your response should follow these steps:
<think> Carefully analyze the problem to identify decision variables, objective, and con- straints<</think
<model> Develop a complete mathematical model, explicitly defining: * Sets Parame- ters Decision Variables (and their types) Objective Function Constraints </model> <python> Provide the corresponding Gurobi Python code to implement the model. </python>
The output must be in Markdown format, with each step enclosed in the specified tags.
USER: Answer the following mathematical modeling question: [Question]}
Please think step by step.
31

---

Problems
An industrial tire company delivers large tires for equipment to remote engineer- ing sites either by cargo planes or ultrawide trucks. Each cargo plane can transport 10 tires per trip and costs $1000. Each ultrawide truck can transport tires per trip and The company needs to transport at least 200 tires and has available $22000. Because most re- mote sites don't 't have proper airports, the number of plane trips cannot exceed the number of ultrawide truck trips. How many trips of each should be done to minimize the total number of trips?
Problem:
$700.
32

---

Response
<think> To solve this problem, we need to identify the decision variables, the objective func-
tion, and the constraints.
1. **Decision Variables:***
Let æ be the number of plane trips.
Let be the number of truck trips.
**Objective Function:**
We need minimize the total number of trips, which is y-
**Constraints:** **
The total number of tires transported must be at least 200: 10r 6y 200. The total cost must not exceed S22000: 1000x + 700y s 22000.
The number of plane trips cannot exceed the number of truck trips: S y-
Non-negativity constraints: x 2 0, y 2 0.
Since these linear constraints, we use linear programming solve this problem. </think>
<model>
### Sets
There are no sets in this problem as it is simple linear programming problem. ### Parameters
The number of tires each plane can transport: 10.
The cost per plane trip: $1000.
The number tires each truck can transport: 6.
The cost truck trip: S700.
### Decision Variables
-r: Number of plane trips.
y: Number of truck trips.
### Objective Function
Minimize the total number of trips: x +
### Constraints
10.r + 6y 2 200
1000x + 700y 22000
3. x S y
4. 0, 2
</model> <python>
import gurobipy as gp from gurobipy import GRB
# Create new model
Aop.MMode("ttrr transport")
# Decision variables
x v..addVarnnnmm=">> vtype=GRB. INN.GGER)) Number of plane trips
y m.add Var(name="""" viyp=GGBBINTTGER # Number of truck trips
Objective function
settbjectiveex + GRB.MINIMIZE)
Constraints
m.addConstr(11 10 6 * >= 200, ftire_oosttrait""
addConstr( 1000 * + 700 * <= 22000, "cost_constraint")
m m.addConstr(x y 0, "trip_ sosppoonttrint"
# Optimize the model
m m.optimize()
# Print the solution
if m.status == GRB. OORB.OPTIAAL:
print("'Optimal solution found:")
print(f"Plane trips: x.x")
print(f"Truck trips: y.x")
Total trips: X.X y.x")
(f cost: Sm.obj
else:
print( No optimal solution found.")
</python>

---

E
Reward function design
The overall two-stage reward function r(z, 2, y²) defined as follows:
Reormat z) + Rexoe ()) + Racour (2, 3²)
Stage- 1,
r(z, (,,,,,0))
Rionmal (2) + Rexer (2) Raseur (2. 2, y² + Ravmus (2, 2, y")
Stage-2.
The reward function comprises the following components:
Format reward (z): the format reward guides the LLM policy to produce response with a specific, parsable structure trajectory defined by the system prompt. This structure segregates the solution trajectory via tags like <think></thinis optimization model, and Syython></pythen for executable code. Riomar (z) a binary reward (1 or 0) awarded only if z strictly includes all required tags in their correct order.
for reasoning steps, <model></modes
for the
Let T {<think>, ... </think>, <model>. </model>, <python>... </pp:/hoon//// be the set of required tag pairs. The reward is:
0.5 if contains all tags in T according to system prompt
Riformat 22)
0
otherwise.
This reward is also foundational for enabling the extraction and evaluation of the generated model and code.
Execution reward (z)) assigns a reward of if the optimization code within response z is executable, and 0 otherwise.
1 the code is executable, Rexec (z)
otherwise.
Accuracy reward Raccur (I, z, y the accuracy reward evaluates the correctness of the final answer y = g(a, z) obtained by executing the code in z. The answer is considered correct if matches the ground truth y" within a tolerance s²1 < 0.01 In the first stage, the reward is defined as
2
if the answer right,
Raceur (r, z,y²)
0 otherwise.
Bonus accuracy reward Rooous (x,, 2, real-world optimization problems frequently involve nonlinear relationships or discrete variables, to encourage our model to tackle more complex op- timization problems requiring techniques beyond standard Linear Programming (LP), introduce a bonus reward. By analyzing the .p file generated by the solver code, we can verify whether these advanced techniques (Big-M methods [X2], binary variables,
nonlinear formulations) is used. The binary bonus Roveus (z) is awarded for output z if, and only if, both the correct answer derived from is correct and the generated model utilizes advanced modeling techniques detectable through instance analysis.
if advanced modeling techniques used, otherwise.
Bmonus (2) =)=00
F Details of experiments
Training setup. All experiments for the 7B model were conducted on single compute node equipped with eight 80GB NVIDIA H100 GPUs. The two-stage training process required approxi- mately 24 hours of wall-clock time per stage, for total computational cost of 384 GPU hours.
Starting from the synthetic dataset, we applied filtering strategy guided by the principle "Less is More" [73, /4]] Specifically, we excluded (question, answer) pairs if the baseline Qwen-32B- Instruct model [22]] achieved an 80% success rate (8/10 attempts across different prompting roles) in generating executable code matching the ground-truth optimal value, as such samples were deemed too trivial. This process yielded approximately 70,000 samples. From this set, we then randomly sampled 10,000 instances form our training data.
34

---

We used Qwen2.5-7B-Instmen [22]] the base model and adapted the Verl framework [85] for reinforcement learning training, modifying its implementation incorporate our novel surrogate function design with the Partial KL strategy and two-stage reward mechanism.
The key hyperparameters for SIRL training detailed in Table ft
Table 6: Training Parameters
Type Algorithm
Parameter
Value
Advantage Estimator
reinforce palu__ppuuu
Data
Batch size
128
Learning rate
le-6
Max prompt length
2048
Max response length
8192
Truncation
left
Actor/Rollout
KL loss type
low var
KL loss coefficient
0.005
Rollout number
8
PPO mini batch size
8
PPO miero batch Size per GPU
4
Clip ratio low
0.20
Clip ratio high
0.28
Decoding strategy. We employ the top-P (nucleus) decoding strategy [86] for the training and in-
ference phases. The exact sampling hyperparameters used to generate our main results are specified Table B
Table 7: Sampling parameters used for text generation.
Parameter
Value
Temperature
0.5
Top p
0.9
Max tokens
8192
Repetition penalty
1.02
G Further Comparative Analysis
Sampling-Based Performance Metries To further assess the robustness of our method under stochastic generation, we conducted experiments using top-p (nucleus) sampling with temperature 0.5 and top-p 0.95. The mean accuracy and standard deviation across multiple reported in Table
Table Mean pass@1 accuracy (%) standard deviation under top-p sampling (temperature 0.5,
top-p = 0.95).
Metric
NL4OPT
MAMO Easy
MAMO Complex
IndustryOR
OptMATH
mean + std 96.2 + 0.32
90.1 + 0.14
52.8 + 1.04
31.7 + 1.5
31.4 + 1.29
These results confirm the stability and high efficacy of our SIRL approach, even under stochastic sampling conditions, with low variance indicating consistent performance.
Performance without External Solver Assistance We first evaluate the performance of models
trained solely via textual step-by-step reasoning, without leveraging external solvers for verification or execution. This setup relies entirely on the L.MM's internal reasoning capabilities to generate solutions. The results, reported pass @ accuracy, summarized in Table !!
35

---

Table @ accuracy (%) on optimization benchmarks for models trained via textual reasoning
without solvers.
Models
NL4OPT
MAMO Easy
MAMO Complex
IndustryOR
OptMATH
Qwen2.5-7B-Inss Instruct
24.5
14.7
1.4
7.0
5.7
Math4Opt-RL
15.5
42.2
3.3
13.0
13.5
As the results in Table @ demonstrate, this approach yields extremely low performance across most benchmarks, highlighting the limitations of LLM-only reasoning for optimization tasks. While large reasoning models (e-g.. DeepSeek-R1 excel at sequential reasoning in tasks like AIME mathemat- ics, optimization modeling introduces unique challenges. It demands a global understanding of highly interconnected objective functions, constraints, and decision variables, coupled with the ex- ecution of complex computational steps. This computational burden is difficult to manage through purely textual, step-by-step reasoning alone. In contrast, the optimization community relies on ma- ture solvers to provide the necessary global perspective and computational efficiency. Our approach bridges this gap by using LLMs for accurate problem comprehension and initial mathematical formu- lation, while delegating computationally intensive aspects specialized solvers through generated code blocks. This division labor central to achieving high performance optimization tasks and motivates the integration of external tools in framework.
Comparison with SFT Building the limitations of pure textual reasoning, next compare our online RI, approach against strong SFT baseline, where solver-assisted code generation is incorporated. The pass@ @1 results across models presented in Table III.
Table 10: Pass@ accuracy (%) comparison: Base model, SFT, and RL.
Models
NL4OPT
MAMO Easy
MAMO Complex
IndustryOR
OptMATH
BaseModel
75.1
81.3
22.7
13.0
4.1
SFT
83.3
86.4
38.0
24.0
20.8
RL
96.3
90.0
51.7
33.0
30.5
As evidenced in Table II0 our RL method consistently outperforms both the base model and SFT across all benchmarks. The gains are especially notable on challenging datasets like MAMO Com- plex (24.1% % over SFT) and OptMATH (8.2% over SFT), emphasizing RL's role in enhancing com- plex problem-solving through iterative feedback and solver verification. This progression from pure reasoning to hybrid RL-solver paradigms illustrates a scalable path toward robust optimization mod- cling with LLMs.
Error Type Analysis To dissect the sources of failure and provide deeper insights into the im- provements, categorize errors into types such code extraction failures, timcouts, execution errors, and wrong answers (with the remainder being correct solutions). Detailed breakdowns are provided in Tables I and [12] for the base 2.een-25--B-Insteeee model and our SIRL--Qwe2.5577B variant, respectively.
Table 11: Error type distribution (%) for Qwwen-2.577-IInstruct
Error Type
NL4OPT
MAMO Easy
MAMO Complex
IndustryOR
OptMATH
Code Extraction Failed
0
0.4
2.1
Timeout
0
0
0
0.5
Execution Error
6.1
3.2
33.6
73.1
Wrong Answer
18.0
15.5
43.2
67
20.2
Correct
75.1
81.3
22.7
4.1
Comparing the two tables, our SIRL method substantially reduces execution emror and wrong an- swers, particularly simpler benchmarks like NL4OPT and MAMO Easy, where correct rates
36

---

Table 12: Error type distribution (%) for SIRL-QWen2.5-7B.
Error Type
NL4OPT
MAMO Easy
MAMO Complex
IndustryOR
OptMATH
Code Extraction Failed
0
1.6
Timeout
0
0
0
0
Execution Error
0.4
1.9
11.9
Wrong Answer
3.3
10.3
36.5
61
56.0
Correct
96.3
89.7
63.6
30.6
exceed 89%. On more complex datasets (e.g.. MAMO Complex and OptMATH-193), execution er- rors persist but are markedly lower than in the base model, underscoring the value of reinforcement learning in refining code generation and solver integration.
H In-depth analysis of the Partial KL strategy
Full ablation study on different surrogate function designs. Here, we present detailed analysis of the Partial KL divergence. Table II3 shows the results of an ablation study all benchmarks, which, due to page limitations, was not included the main paper. The results are consistent with those reported in the original manuscript, the surrogate function design employing Without KL strategy demonstrates a significantly reduced execution rate compared to the other two designs.
Table 13: Ablation study
Partial KL
NL4OPT
MAMOEasy
MAMOCemplex
IndustryOR
OрtМATH
Type
Acctpass
Acc(pased))
Apcipass 1)
Acctpass
Acc
Partial KL
96.3%
100.0%
91.7%
100.0%
51.7%
98.1%
33.0%
96.0%
30.5%
92.2%
Full KI.
95. 19.19 (22)
99. 2%
89.9 19.9%6110.
(0.0100 196
99.7%
48 3..33%
3.4%)
97.6%
95 95.0%
28. 39. 2%)
93.4%
Without
92.7%))
1..33)))333)
(7.39))
293((4409))
29.5%((1.0%)
Case study on different surrogate function designs. In this part, two case studies are presented to demonstrate how different surrogate function configurations affect LLM-generated responses. The first case study demonstrates the limitations of Without KL strategies, showcasing execution er- rors in an elementary mathematical modeling question; The second examines how the Partial KL strategy, by selectively removing the KI. divergence term, enhances reasoning capabilities to better understand questions, especially those involving logic constraints.
The first example chosen from the IndustryOR dataset. this case, we compare the generated code snippets to present the execution error arising from the model trained with the Without KL strategy. We observe that omitting the KL divergence in the code component leads to execution errors. Specifically, as marked in red, the response from the Without KL model attempts to enforce that decision variables are greater than zero, but the implemented code non-functional since '>' not supported between instances of 'Var' and 'int'.
Case Study 1:
Case study The second case study involves a problem from the IndustryOR dataset, character- ized by more complex logical constraints. In this case, an incorrect reasoning step (highlighted in red) leads to a flawed problem formulation. This category of problems demands deep understand- ing of the underlying structure and inherent logical dependencies. We observe that omitting the KI. divergence penalty encourages the model to enhance its reasoning capabilities. This fosters more effective exploration of the solution space.
Case Study 1: Question instance from the IndustryOR dataset
A product be processed any of the four devices: A, B, D. The preparation completion costs when each device is enabled, the unit production cost for the product, and maximum process- ing capacity of each device are shown in Table 5-7. 2000 units of the product need to he produced, how can the total cost be minimized?? Try to establish mathematical model. Table 5-7 Device, Completion Cost (Yuan), Unit Production Cost (Yuan/Unit), ing Capacity (Units) A. 1.1000 2009000,900 B,920.24, 1000 C.800, 1200 D.700,28.1600
Mitan//nit)MMximmun Maximum Process-
37

---

Case Study 1: Generated response under the Full KL strategy
<python>
import gurobipy gp
from gurobipy import GRB
# Create new model
m prapuee("ppoduut_pppea
# Decision variables
X_A Vn.addVar(wtyye=GRBBB CONTINUOUS, name="x B")
CONTINUOUS, name= AA")
X C T...dd/Varvvyyc=GRBB.OOTTT CONTINUOUS, name= C")
X x_D Sww.ll//wwwypccGRBBB..TTOTOOOOOO name= D_D
y A m.adddVaar/yyyy--GRBBBBBBBM BINARYY, name="y_AT)
y B m...dVaa(1vyy==ORBBBBBNAA. BBINRY. name="y_Bee y__B")
y Tn..dd/Yarvtyye=GRBBBB BINARY. name="y "yte="""""")
y D = TIL.ddVartvyp==GRBBBBBM..
name="y y
# Objective function
nn.se0bbectivec110 (1000 *y_A+ 920 y_B 800 * *__C+ +0 x_C
28 D_D, GRIMMIN
# Constraints
m.addConstr(x_A-41 + B
x__ + == 2000, "total produlppoductio")
m.addConstr(x A <= 900 $ A, "capacity_Ar
m.addConstr(x__
1000 y_B, "capacity__ _B) B))
m.dddConstr(x, C 1200 y_C, "capacity_ C*)
m.dddConttr(x D <= 1600 y_D, "capacity_Di D
# Optimize the model
m.optimize()
# Print the solution
statuus GRB. OPTIMAL:
Total cost: (mlobjVal]") "))
print(("Units processed on A: [x Ax
print(("Units processed B: {x B.)]")
print(f"Unnis processed C: (x_C.x)")
print(f" processed on D: (X_Dx)"")
print(f"Device A enabled: (y_A.x]")
print(("Device B enabled: [y B.x)")
privt(("Devicee C enabled: (y Cx)")
print(("Device enabled: [xx]""
else:
print("No optimal solution found.")
</python>
38

---

Case Study 1: Generated response under the Without KL strategy
<python>
from gurobipy import
# Define the model
model Model( (rodel("Productt Optimization")
# Parameters
hupp_completio_coom !"A": 1000, "B": 920, "C": 800, "D": 700) unit_production_com costooooooootoooooooooooo ("A": 20, "B": 24, "C": 16, 28)
max_processing_.com capacity ("A": 900, "B": 1000, "C"; 1200, "D": 1600) # Decision variables
x x=
for in trep_completion ceaippeettoocostttkeee cost.keys()
x[i] = modell..ddVar((nameettt lb=0, ub=max processingceggggeeooo processingccapacittjj. Sype=GRB.CONTINUOUU
Objective function
BadelessOOjeciiiequuusssnieee completion cos{i]] (x[i] > 0) + unit production_ x[i] for
in coopplccion_costteesssss) GRB.MINIMIZE)
# Constraints
CanelaaaddCoossrrquiccuuum/o for in prap_completion_cost.eyyy 2000, "Total_Producee" ")
# Optimize the model
model.optimize()
# Print the solution
if model. status (innt"lpiimal Solution:")
StatuStats..C OPTIMAL:
for in wwwplltioo_cosseeeetteeoo
privt(""Deviee (i]: [xfi].x)")
print(f"Total Cost: (model.objVall]"
else:
print(" No optimal solution found.")
</python>
39

---

Case Study 1: Generated response under the Partial KL strategy
<python>
from gurobipy import
# Define the parameters
prep_compplee completeolnnnnnccc cost ('A': 1000, 920, 'C': 800, 'D': 700) unit_ _uiit_poduction.tcmm
ionduction_costt.oo cost I'A': 'B': 24, 'C': 16. 'D': 28}
max_prrcessing_cappe capacctypccccccccc = ('A": 900, 'B': 1000, 'C"; 1200, 'D': 1600] # Create model
model Model( ProductProduction")
# Define decision variables
x=
(=
for device in processing processingg_cappctty.kekk. sapacity.keysff
x[device] Moodel..dVaarttyyeGGRB... BINARY. name=f"x_ [device]")
y [deevcee] (etr:aaddWaaraamme""""eedeeeiiie
# Set objective
x/device] yfdevicel for device in max_proeesing__eppcityyyyyy
unit_ condduuttoo_css((ddeee
GRB.MINIMIZE)
# Add constraints
for device in max processing capacity.keysO) = 2000)
for device in max_ pooccpprocssngggeeoee
processingg_cppccty..ee..e capacity.keys():
addeel.ddCoonsrr [evice] max wwwppeesinng.pppccitlleeeeeeeeee x]device])
# Optimize the model
model.optimizel
# Print the optimal solution if model. status
OPTIIAL::
print(" Opptimal solution found:")
for device in max processing capacity.keys(:
print(("Devic) [device]: [x]device]xx). (y[device]xx)"
print((" Minimum cost: (model.objVal])) clse:
print("N No optimal solution found.")
</python>
Case Study 2: Question instance from the IndustryOR dataset
manufacturing company needs transport 1800 units of product from the warehouse to three different sales points. The company has four transportation options to choose from: truck, van, motorcycle, and electric vehicle. Since the van and electric vehicle both consume a lot of energy, the company wants to choose only of these two options. Each trip with a truck generates 100 units of pollution, a van generates 50 units of pollution, motorcycle generates 10 units of pollution, and electric vehicle generates units of pollution. The total pollution generated from all trips cannot exceed 2000 units. At least 10 trips must use a truck. Trucks, vans, motorcycles, and electric vehicles can transport 100 units, 80 units, 40 units, and 60 units of product per trip, respectively. The company needs to ensure that the total amount of transported product is at least 1800 units.
40

---

Case Study 2: Generated response under the Full KL strategy
<think>
To solve this problem, we need to identify the decision variables, the objective function, and the con- straints.
**=Decision Variables
-Let V. T be the number of trips using trucks.
-Let V. V \) be the number of trips using vans.
Lett V M \) be the number of trips using motorcycles.
V. !) be the number of trips using electric vehicles.
**Objective Function**; **;
The objective is to minimize the total pollution, which can be expressed as the equation V, 100T + 50V
10M + OF \).
3. **Coostraints***
The total product transported must at least 1800 units: 100T 400 60E lgeq 1800
The total pollution generated cannot exceed 2000 units: V 100T + 50V + 10M Veq 2000 \).
At lcast 10 trips must use trucks: V T gggq 10 V). Non- nennneiativity constraints: V. T, V, M, E 0g)
4. * "BBnnary Variable for Van Electric Vchicle**:
Since the company wants to choose only one of the van or electric vehicle, we can use binary variable V( B \) where V( = 1) a van is used and V( B = () if an electric vehicle is used. This can be handled by the constraints V. V Vle B \ and V( E Meg 1- V). Let's now develop the mathematical model and provide the Gurobi Python code.
</think>
<model>
### Sets
No sets needed for this problem.
### Parameters
Pollution per trip for trucks: 100
Pollution
per trip for vans: 50
Pollution per trip for motorcycles: 10
Pollution per trip for electric vehicles:
### Decision Variables
-( T V) Number trips using trucks
V( V \): Number of trips using vans
X( M )):
Number of trips using motorcycles
A( E \): Number of trips using electric vehicles
B \):
Binary variable (1 if van used, electric vehicle used)
### Objective Function
Minimize total pollution: M( MextMinimize 10UT 50V 10M 1)
### Constraints
1. Total product transported: V. 100T 80V 40M 60E Igeq 1800 ))
Total pollution: V. 100T 50V 10M Veqq 2000 \)
3. At least 10 trips using trucks: Y( ' \geq 10
4. Binary variable constraint for van or electric vehicle: 1( V Veg B and V E Veg 1 Non-negativity constraints: T, M. E, lgeg
</model>
41

---

Case Study 2: Generated response under the Without KL strategy
<think>
To solve this problem, we need to identify the decision variables, the objective function, and the con-
straints.
**#DDeision Variables:
-Let V. T be the number of trips made by trucks.
-Let V. V \) be the number of trips made by vans.
Let V M \) be the number of trips made by motoreycles.
V \) be the number of trips made by electric vehicles.
**Objective Function:
The objective is to minimize the total pollution generated, which can be expressed as:
M MiextMinimize 100T 50V 10M \]
**Constraints:**
The total amount of product transported must at least 1800 units: M 100T
Igeq
1800 J
The total pollution generated cannot exceed 2000 units: Vf 100T + 50V + 10M Vlq 2000 \]
At least 10 trips must use truck: M T lee 10
Non-negativity constraints: V V. M, lgeq 'qquad Mext(integer values) \]
Since the problem involves integer values for the decision variables, we will use integer linear program- ming (ILP) to solve it.
</think>
<model>
Sets
There are no sets in this problem as it is a simple linear programming problem.
### Parameters
The product be transported: 1800 units.
Pollution generated by each type of vehicle: truck (100), (50), motorcycle (10), electric vehicle
(0).
Product transported by each type of vehicle: truck (100), van (80), motorcycle (40), electric vehicle
(60).
Maximum pollution: 2000 units.
Minimum truck trips: 10.
### Decision Variables
M(T \): Number of trips made by trucks.
N( V \): Number of trips made by vans.
A( M Number of trips made by motorcycles.
E V): Number trips made by electric vehicles.
### Objective Function
M \textMinimize 100T 50V 10M V]
### Constraints
Product transportation constraint: VI 100T 80V + 40M 60F \geq 1800 Pollution constraint: \ 100T + 50V + 10M llq 2000 \]
3. Minimum truck trips: T 'geq 10 \]
4. Non-negativity constraints: \[ T, V, M. E lgeq lquad Mext(integer values)
</model>
42

---

Case Study 2: Generated response under the Partial KL strategy
<think>
To solve this problem, need to define the decision variables, objective function, and constraints
based on the given transportation options and pollution limits. Here's step-by-step approach:
:*#Decision V
-Let V. x_ 1) be the number of trips with trucks.
'Let V, x_2 \) be the number of trips with vans.
-Let V, x_3 be the number of trips with motorcycles.
V. x_4
number of trips with electric vehicles.
-eet Y binary variable where V. y = 1) van is chosen, and y = V) if an electric vehicle
is chosen.
2. **Objective Function:
minimize the total pollution generated. The pollution generated by trucks is 100 units per trip. by vans units per trip. by motorcycles is units per trip, and by electric vehicles is 0 units per trip.
The objective
Therefore, the objective function is: V \textminimize 100x_1 50x_2+ 2 + 10x_3 0x_4N]
**Constraints:**
'Tee total product transported must at least 1800 units. The total pollution generated must be at most 2000 units.
At least 10 trips must use truck.
Only one of the </think>
electric vehicle be chosen.
<model>
### Sets
-,, 2, 41 where V( ()) is the set of transportation options.
### Parameters
-( -(Pii)) is pollution generated by transportation option 1).
M( C (CCi is the capacity of transportation option V( 1).
### Decision Variables
-A( () the number of trips with trucks.
x__ \) the number of trips with vans.
x_3 !) the number trips with motorcycles.
M( x__4 !) is the number of trips with electric vehicles.
y \) is binary variable where (( = 1 \) if van is chosen, and y 0 () if an electric vehicle is chosen.
### Objective Function
M \textMinimize 100x_ + 50x 2 + 10x 3 0x_4
### Constraints
M 100x_ + 80x 2 40x 60x \geq 1800 'quaad \text(Total product constraint)
M 100x 50x_ 2 10x_3 0x Vlqq 2000 lquad (oett pollution constraint)
M x_] lgeq 10 'quaad (text(AA least trips with trucks) \]
M y lin 10, 1\ 'quad Mext(Binary variable for van or electric vehicle) \]
[ 2 lle 100y kquad Meet(( Van constraint) !
[[ Aleq 100(1 y) kquad (Eeecteic vehicle constraint) \]
</model>
43


Reviews:
[{"review_id": "qmgZQkkZAu", "reviewer": "Reviewer_4Zmo", "summary": "The topic of this paper is to use LLMs to assist in modeling combinatorial optimization problems to then be fed into optimization solvers (like Gurobi, Cplex etc.). \n\nThis is an emerging and relevant area (that said, the paper misses some important reference listed below). \n\nThe novelty of the paper is the introduction of Solver-Informed Reinforcement Learning (SIRL) to steer the LLM to generate correct code and models. \n\nThe main idea is that; given that these optimization solvers exists, use them for verification in an RL loop, essentially using solver feedback as a source for RL reward for syntax check and solution quality. This feedback is used to train an LLM. This idea is not new but an application of RL with Verifiable Rewards (RLVR) to optimization modelling seems novel. \n\nPartial KL is used as a regularization strategy in RL training. IF I understand fully, it applies to code and the model but omits other steps, hence allows exploration while keeping the model/code correct (?). Interestingly, it adds bonus for the use of advanced modelling techniques (see comments about this below). \n\nExperiments across benchmarks compares SIRL trained Qwen2 model with agent-based and trained LLM approaches and larger LLMs with promising results.", "questions": "Q: For the comparisons, how did you obtain their results (copy from their papers or re-built their method, or re-run them or else?). This is regarding the agent-based and off-line learning methods in Table 2. \n\nQ: Why are there no results for CoE in Table 2 on several columns? \n\nQ: How would this handle ambiguous, noisy, or incomplete natural language inputs, which are common in practice? \n\nQ: What's the main value proposition (see detailed comments above)", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \nThe application of RL with Verifiable Rewards (RLVR) to optimization modeling seems new. \nUsing external solvers as reward oracle and automated feedback is nice (but not without its problems, see below)\nCompetitive with Deepseek (671B) using only 7B params\nAblation studies are nice, and interestingly partial KL seems to pay off compared to no KL or full KL. \nUsing structural features of the features in the data synthesis part is nice. \n\nWeaknesses: \nIt is not obvious while Advanced Modelling is rewarded. Is this an artifact of the benchmarks? Would we not want simpler model than more complicated models especially if we want to assist human modelers who needs to understand these auto-generated code. \n\nWhat are the limitations of using \"solvers\" as verifiers? It should be note that running a full-blown optimization solver is different than compiling/running code with, say, Python. How expensive is this step and/or cannot this become prohibitive. How many times solvers \"stuck\"? This needs to be explained more in the paper. How would this even work for large-scale/complex problems? \n\nSomewhat related, how expensive is the training of the base LLM? Some commentary on this would be nice. The value proposition is not clear; does the paper suggest that the Online RL trained Qwen now becomes the LLM Model Assistant .. OR.. the methodology enable others to train their own models for LLM Modelling. IF the latter, we need more explanation on cost/feasibility of the Online RL to generalize for others. Again, IF the latter, than the design stage introduces many moving parts; there is are two versions of intstance selection and two step rewards and several weights. It is not obvious how these would generalize. I believe it would require careful tuning for different datasets.\n\nThe paper confuses between Linear Programming (LP), the LP file format, and Mixed-Integer Programming (MIPs). This needs to revised throughout the paper (and should be doable). My understanding is that this method is NOT specific to LPs, but in parts the paper comes across as applied to LPs only. I believe their benchmarks also include MIPs (and the method works for MIP also) but the confusion comes from the \".lp\" file extension as the output formats. Please clarify if your method is solely for LPs? \n\nMissing references to include: \nNer4Opt: named entity recognition for optimization modelling from natural language, Constraints 2024  (Consider comparing/positioning your results with what's published already here) \nEnhancing decision making through the integration of large language models and operations research optimization, S Wasserkrug, AAAI 2025 (common reference for LLM Co-Pilots)\nHoly grail 2.0: From natural language to constraint models, D Tsouros, (common reference for LLM Co-Pilots)\nText2Zinc: A Cross-Domain Dataset for Modeling Optimization and Satisfaction Problems in MiniZinc, Singirikonda (Consider using this dataset to expand your tests) \nConstraint modelling with LLMs using in-context learning, K Michailidis (I don't think this is a direct comparison for this method but must be mentioned as complementary for satisfaction/logic problems)", "was_revised": true, "final_justification": "Thank you for the thoughtful discussions, I have commented and revised accordingly.", "rebuttal": "**Weakness-1**: Unclear Reward for Advanced Modeling\n\n**Response:**  \n\nThank you for highlighting this point. We fully agree that, in general, *simpler models are preferable*.  \nOur advanced modeling bonus, **R_bonus**, is *not* meant to encourage unnecessary complexity. Rather, it is designed to reward the use of *essential reformulation techniques* from Operations Research—methods that are often required to make a problem tractable for high-performance MILP solvers.\n\nFor instance, problems involving logical \"if-then\" conditions or fixed charges often lead to non-linear or logically invalid initial formulations. While LLMs may generate simple, approximate solutions for tasks like the 0-1 knapsack or certain TSP instances, *consistently obtaining optimal solutions requires these advanced methods*. A typical example is the Big-M formulation, which linearizes such conditions by introducing binary variables, making the problem solvable as a MILP. Without these reformulations, the resulting model would be either incorrect or unsolvable by standard solvers.\n\nTherefore, the bonus is awarded *only when these techniques are indispensable* for producing a correct and solvable model. Our model automates this critical (and often non-trivial) reformulation step, providing human modelers with a mathematically rigorous model that is ready for high-performance solvers.\n\n---\n\n**Weakness-2:** Limitations of using \"solvers\" as verifiers.\n\n**Response:**  \n\nThank you for your question about the limitations and computational cost of using solvers as verifiers. This is indeed a crucial aspect.\n\nTo illustrate the challenges throughout the process, we present the error types for the Qwen-2.5-7B-Instruct model and SIRL-Qwen2.5-7B, with a maximum execution time of 100 seconds.\n\n*Error Types (Qwen-2.5-7B-Instruct)*\n\n| Error Type              | NL4OPT | MAMO Easy | MAMO Complex | IndustryOR | OptMATH-193 |\n|-------------------------|--------|-----------|--------------|------------|-------------|\n| Code Extraction Failed  | 0      | 0         | 0.4%         | 0          | 2.1%        |\n| Timeout                 | 0      | 0         | 0            | 0          | 0.5%        |\n| Execution Error         | 6.1%   | 3.2%      | 33.6%        | 30%        | 73.1%       |\n| Wrong Answer          |  18.0%          |  15.5% |  43.2% |  67%     |  20.2% |  \n| Correct               |  75.1%         |  81.3% |  22.7% |  13%     |  4.1%   |  \n\n*Error Types (SIRL-Qwen2.5-7B)*\n\n| Error Type              | NL4OPT | MAMO Easy | MAMO Complex | IndustryOR | OptMATH-193 |\n|-------------------------|--------|-----------|--------------|------------|-------------|\n| Code Extraction Failed  | 0      | 0         | 0            | 0          | 1.6%        |\n| Timeout                 | 0      | 0         | 0            | 0          | 0           |\n| Execution Error         | 0.4%   | 0         | 1.9%         | 4%         | 11.9%       |\n| Wrong Answer            | 3.3%   | 10.3%     | 36.5%        | 61%        | 56.0%       |\n| Correct                 | 96.3%  | 89.7%     | 63.6%        | 35%        | 30.6%       |\n\nAs shown above, the proportion of *\"Timeout\"* errors—where the solver becomes stuck—is very low, occurring only for the most challenging OptMATH-193 dataset (0.5%) in the Instruct model. This indicates that, for these textbook-level optimization problems, solver execution is generally *not* a computational bottleneck. The main error types are *\"Execution Error\"* (where the LLM generates unexecutable code) and *\"Wrong Answer\"* (where the model is logically incorrect).\n\nWhile solver execution is not currently a primary bottleneck for our tested problems, we acknowledge that for truly large-scale, real-world optimization challenges, the computational cost of solver verification could indeed become a limiting factor.\n\n\n---\n\n**Weakness-3**  The cost of training and value position.\n\n**Response:**  \n\n*First question:*  \nOur training process consists of two distinct stages, both using the *same* training dataset.  \n- In *Stage 1*, we use a specific \"stage-1 reward\" signal to guide training.  \n- In *Stage 2*, training resumes from the checkpoint obtained at the end of Stage 1.  \nEach stage required approximately 24 × 8 H100 GPU hours, totally 48×8 H100 GPU hours.\n\n*Second question:*  \n- We have open-sourced and released checkpoints that achieve state-of-the-art performance among models with 7B–14B parameters,. Any member of the community can download these checkpoints to build their own LLM model assistants. We also plan to release more checkpoints.\n- Additionally, we introduce a new **RLVR** framework for optimization modeling and will continue to release corresponding codebase and a portion of training dataset , empowering the community to build their own  LLM assistants for diverse optimization problems.\n\n---\n\n**Weakness-4**: The paper confuses Linear Programming (LP), the LP file format, and Mixed-Integer Programming (MIP), needing clarification on whether the method is LP-specific or also applies to MIPs.\n\n**Response:**  \n  \nThank you for your insightful feedback, which has greatly improved the clarity of our manuscript. We apologize for any confusion caused by our earlier writing and appreciate the opportunity to clarify.\n\nOur method is *not limited to Linear Programming (LP)*; it supports a broad range of optimization problems, including *Quadratic Programming (QP)*, *Mixed-Integer Programming (MIP)*, and *Second-Order Cone Programming (SOCP)*. During data synthesis and the reinforcement learning stage, we utilize the `.lp` file format, which is a standard for solvers to represent various optimization models. This format is not exclusive to LP but is widely compatible with models such as QP and MIP. Our strong performance on the OptMATH test set, which includes MIP and SOCP instances, further demonstrates that our method generalizes effectively beyond LP.\n\nIn the revised manuscript, we will clarify that the `.lp` file format supports multiple optimization models, not just LP, and we will revise relevant sections to emphasize the method’s applicability across diverse problem types.\n\nWe sincerely appreciate your thorough review and welcome further suggestions to improve our work.\n\n---\n\n**Weakness-5** Missing References: The paper omits key references like Ner4Opt, Enhancing Decision Making (AAAI 2025), Holy Grail 2.0, Text2Zinc, and Constraint Modeling with LLMs, which are relevant for comparison, benchmarking, or complementary approaches.\n\n**Response:**  \n\nThank you for these highly valuable references. We confirm that we will revise our Related Work section to address all of your suggestions.\n\n---\n\n**Reply to Questions**\n\n*Q-1: For the comparisons, how did you obtain their results (copied from their papers, re-implemented their methods, re-ran them, or otherwise)? This refers to the agent-based and offline learning methods in Table 2.*\n\n**Response:**  \n\nThe results in Table 2 are from the paper *\"OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling.\"* We used their figures directly because of their rigorous evaluation protocol, which validates solutions only if the relative error is less than 1e-6. Further details on their evaluation methodology and results are publicly available on the paper's GitHub repository.\n\n---\n\n*Q-2: Why are there no results for Chain-of-Experts (CoE) on these benchmarks?*\n\n**Response:**  \n\nThank you for pointing this out. The absence of results for *Chain-of-Experts (CoE)* on these benchmarks is due to a fundamental mismatch in the required input data format, which poses significant challenges for direct and fair comparison.\n\nOur evaluation benchmarks, consistent with most recent work in this area (e.g., ORLM, OptMATH), consist of problems presented as *natural language descriptions* paired with a ground-truth optimal value.\n\nIn contrast, the CoE framework, as presented in its original paper, operates on *pre-processed, structured data inputs* rather than raw natural language. The proprietary data-processing pipeline used to convert natural language problems into this required structured format is difficult to adapt to other benchmark datasets. Reimplementing this pipeline for each of our diverse benchmarks would require significant and non-trivial effort, with no guarantee of faithfully replicating their methodology. Furthermore, to the best of our knowledge, we are not aware of any subsequent work that has benchmarked the CoE framework on these standard datasets.\n\n---\n\n*Q-4: How does your method handle ambiguous or incomplete inputs?*\n\n**Response:**  \n\nOur current research primarily focuses on *well-defined, textbook-level optimization problems*, which allowed us to rigorously validate the core methodology of training LLMs for optimization modeling.\n\nFor ambiguous or incomplete inputs, one promising direction is to develop *interactive clarification modules*, where the LLM can query the user for necessary missing details, potentially through multi-turn conversational frameworks. An even more advanced approach could involve a *multi-agent system* framework. In this setup, an *\"analyzer agent\"* would first attempt to autonomously infer and supplement the content of an incomplete natural language input before passing it to the core modeling agent. To effectively judge and improve the quality of its inferred content, this analyzer agent would require a robust reward system, potentially combining our RLVR methodology with *Reinforcement Learning from Human Feedback (RLHF)*.\n\nThis will require concerted efforts from the entire research community."}, {"review_id": "r8Q784iL9R", "reviewer": "Reviewer_17gf", "summary": "Inspired by the recent success of RLVR, this paper introduces SIRL, a framework that uses external optimization solvers to produce the verified rewards. SIRL breaks down the reasoning process for solving optimization problems using the Chain of Thought approach, generating a sequence of intermediate thoughts. The final generated thought is executable code, which an optimization solver executes to produce the objective value. A partial KL surrogate function is designed to foster exploration of early thinking steps, and a verifiable reward function is designed to ensure soundness of the generated mathematical models and incentivize advanced modeling techniques for challenging problems. The experiments demonstrate that the SIRL-Qwen2.5-7B model achieves performance gains over small models and achieves comparable performance to Deepseek-V3 across the benchmarks.", "questions": "- Section E in the appendix states that top-P (nucleus) decoding was used for training and inference. Why weren’t there multiple runs for the experiments? No standard deviations were reported.\n- Do the authors have a possible explanation for why using only stage-2 reward boosts performance on OptMATH but diminishes performance on NL4OPT? Some analysis on the differences in performance across different datasets would have been useful.\n- The proposed technique relies on the generation of an optimization model from an LLM at inference time, that is then executed by a solver. How could this approach be modified to only use solvers in the data-generation process, so that the LLM acts as the “solver” during inference time?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The approach takes many steps to ensure the quality of the synthesized dataset, including using an LLM-as-a-judge to validate the generated problems as removing samples for which a baseline model achieves at least an 80% pass rate.\n- The proposed instance-enhanced self-consistency approach is constructed to reward not only correct final values, but also the other LP features, which improves the quality of the synthesized data.\n- Several ablation experiments were performed to compare different design choices, i.e., Partial/Full/Without KL divergence experiments justify the use of partial KL for the surrogate function.\n- The experiments show SIRL performs better than similar sized models across the benchmarks, and has performance comparable to much larger models. \n\nWeaknesses:\n\n- I would have liked to see more description of the experimental setup. The Appendix says that top-P decoding was employed, so I had expected to see several runs with some standard deviations reported. But I couldn’t find any in the paper.\n- SIRL helps LLMs output more accurate optimization model code, rather than the final answer. It would have been interesting to see a reward design that does not make use of a solver's answer (letting the LLM perform all reasoning steps). Although I do agree that the most promising approach to ensure interpretability and high accuracy would be to use an actual optimization solver as the final step.\n- The checklist was not filled in adequately. I would have liked to see more explanations/specific sections that addressed each point, rather than just saying “Appendix.” Some parts of the are not explained, e.g., Claims section.\n- Minor comments:\n    - Define LP\n    - Line 33: “information, generate” -> “information, and generate”\n    - Line 103: “rstar [43],Phi [44]” -> “rstar [43], and Phi [44]”\n    - Line 117: Missing period before “For example”\n    - Line 340: “simpltasks” -> “simple tasks”", "was_revised": true, "final_justification": "My knowledge of this area is limited, but RLVR with solvers for optimization modeling is a novel idea, and the experimental results do demonstrate its usefulness. I would have liked to see more information about the statistical significance of results, and other items in the checklist like claims and limitations in this submission.", "rebuttal": "**Weakness 1 and Question 1: Experimental Setup and Reporting of Standard Deviations**\n\n**Response:**  \nThank you for your insightful question regarding the experimental setup and reporting of standard deviations. We apologize for omitting this crucial information in our initial submission.\n\nWe have now conducted additional experiments and report the following performance metrics using top-p (nucleus) sampling (temperature = 0.5, top-p = 0.95):\n\n| Metric         | NL4OPT           | MAMO Easy        | MAMO Complex     | IndustryOR      | OptMATH-193      |\n|:-------------- |:----------------|:-----------------|:-----------------|:----------------|:-----------------|\n| mean ± std     | 96.2% ± 0.32%   | 90.1% ± 0.14%    | 61.3% ± 0.68%    | 33.7% ± 1.5%    | 28.3% ± 0.66%    |\n\nAs shown, the results are highly stable and consistent with our initial findings.\n\n---\n\n**Weakness 2 and Question-3 : Reward Design With/Without Solver Answers**\n\n**Response:**  \nWe agree that using actual optimization solvers for the final step is highly effective for interpretability and accuracy. We have investigated the scenario where solvers are used exclusively during data generation, and the LLM is required to perform all reasoning and solution steps at inference—acting as the \"solver.\" To assess this, we trained models to reason step-by-step through text, without solver assistance at inference.\n\nOptimization modeling differs fundamentally from tasks like AIME mathematics, as it requires a global understanding of interconnected objectives, constraints, and variables, as well as complex computational steps (e.g., simplex method for linear programming, branch-and-bound for mixed-integer problems). These are challenging to resolve through purely textual, step-by-step reasoning.\n\nTo illustrate, we present early results from models trained solely via textual reasoning, without solver assistance:\n\n| Model                  | NL4OPT | MAMO Easy | MAMO Complex | IndustryOR | OptMATH-193 |\n|------------------------|--------|-----------|--------------|------------|-------------|\n| Qwen2.5-7B-Instruct    | 24.5%  | 14.7%     | 1.4%         | 7.0%       | 5.7%        |\n| Math4Opt-RL            | 15.5%  | 42.2%     | 3.3%         | 13.0%      | 13.5%       |\n\n(Math4Opt-RL is trained via RLVR without solver verification.)\n\nThese results demonstrate that relying solely on LLM sequential reasoning leads to very low performance. The optimization community uses mature solvers because they provide the global view and computational power needed for these tasks. Moreover, deriving precise decision variable values for optimal solutions remains extremely difficult for LLMs using only step-by-step textual reasoning.\nIn summary, although it is possible to rely on the LLM as the \"solver\" at inference, this approach currently results in substantially lower accuracy and reliability.\n\n---\n\n**Weakness 3: Insufficient Checklist Completion**\n\n**Response:**  \nThank you for highlighting the need for more detailed explanations and explicit references in our checklist. We acknowledge that our initial submission lacked clarity in this regard. In the revised version, we confirm that we will:\n\n- Carefully review each checklist item.\n- Provide direct references to the relevant sections or appendices for every point.\n- Ensure that all previously unexplained parts (e.g., the Claims section) are now explicitly addressed.\n\n---\n\n**Weakness 4: Minor Corrections and Clarifications**\n\n**Response:**  \nWe appreciate your detailed suggestions, which have improved the clarity and accuracy of our manuscript. The following changes will be made:\n\n- **Definition of LP:**  \n  We now define Linear Programming (LP) as:  \n  “Linear Programming (LP) is a mathematical method for optimizing a linear objective function subject to linear equality and inequality constraints.”  \n  We also clarify that the .lp file format, used in data synthesis, is a standard format for solvers to write and process models.\n\n- **Line 33:**  \n  Corrected “information, generate” to “information, and generate”.\n\n- **Line 103:**  \n  Added the conjunction “and” in “rstar [43],Phi [44]” → “rstar [43], and Phi [44]”.\n\n- **Line 117:**  \n  Inserted a missing period before “For example”.\n\n- **Line 340:**  \n  Corrected the typo “simpltasks” to “simple tasks”.\n\n---\n\n**Question 2: why using only stage-2 reward boosts performance on OptMATH but diminishes performance on NL4OPT**\n\n**Response:**  \nThe stage-2 reward, with its $R_{bonus}$, \n encourage advanced optimization techniques, including reformulation methods that are native to and leveraged by mature solvers for efficient processing of complex SDP, MILP, and SOCP problems, \nwhich is critical for solving  challenging complex problems in hard benchmark dataset such as OptMATH.\n\n When the model is primarily incentivized for these advanced optimization techniques, it may tend to over-explore or  involve redundant exploratory steps, \nleading to diminished performance for tasks where a simpler approach would have been more accurate and efficient."}, {"review_id": "Q43ovVJm6v", "reviewer": "Reviewer_9M7X", "summary": "Solver-Informed Reinforcement Learning (SIRL) trains large language models to write correct and executable optimization models by treating classical solvers as automatic verifiers whose syntax, feasibility and objective-value checks are fed back as reinforcement-learning rewards, directly “grounding” the model in solver reality.\nSIRL couples a data-synthesis pipeline with instance-enhanced self-consistency, a Partial-KL surrogate that regularizes only the mathematical-model and code tokens, and a two-stage verifiable reward curriculum, together striking a balance between diverse reasoning exploration and strict solver-format fidelity.\nA 7 B-parameter Qwen2.5 model trained with SIRL reaches state-of-the-art pass\\@1 accuracy on NL4OPT, MAMO, IndustryOR and OptMATH (e.g., 62.1 % on the hardest MAMO-Complex set), outperforming all prior 7 B/14 B offline or agent baselines and rivalling the DeepSeek-V3.", "questions": "* What is the curated seed data used for training data synthesis? Is it exactly the same as the seed data used in ORLM and OR-Instruct (e.g., 686 cases in their study)? If yes, will it be biased or overfitting issue when evaluate on the IndustryOR benchmark? Despite no evidence shown all experiment results (table 1,2,3). I did see the example in appendix but wanna make sure.\n* Do you see the reliance on external optimization solvers as a potential limitation, given that verifiable reward mechanisms are only applicable to a narrow set of problem domains? I’m curious about your thoughts on how this approach might extend to tasks without access to classical solvers, or whether there are alternative strategies you envision to overcome this constraint.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "# Strength\n\n* SIRL lifts pass@1 score with 7B Qwen2.5 model across four benchmarks, beating all 7–14 B baselines and matching the 671 B Deepseek-V3.\n* First RL-with-verifiable-reward framework that lets classical solvers act as objective oracles; introduces instance-enhanced self-consistency, Partial-KL selective regularization, and a two-stage reward curriculum to balance exploration with solver fidelity.\n* The paper presents thorough experiments with comprehensive ablations and complete baseline comparisons, clearly demonstrating gains in accuracy and execution rate while isolating the sources of improvement.\n\n# Weaknesses\n\n* Compute efficiency is not reported, including GPU hours, solver runtimes, and comparisons between online RL and prior methods such as SFT.\n* The curated seed data used for training data synthesis is somewhat unclear but appears crucial to the performance evaluation; improving clarity in this section would be beneficial.", "was_revised": true, "final_justification": "The rebuttal clarified compute costs, seed data usage, and solver reliance, addressing my concerns. I will remain my positive score.", "rebuttal": "**Weakness 1: Compute Efficiency Reporting**\n\n**Response:**  \n\n- **GPU Running Time:**  \n  Our model was trained from the Qwen2.5-7B-Instruct base, following a two-stage process using the same training dataset throughout.  \n  - **Stage 1:** Training is guided by a specific 'stage-1 reward' signal.  \n  - **Stage 2:** Training resumes from the checkpoint at the end of Stage 1.  \n  - **Total Compute:** Each stage required approximately 24 × 8 H100 GPU hours, totally 48×8 H100 GPU hours.\n\n- **Solver Runtimes:**  \n  For each problem, we set a 100-second timeout for the solver during both training and inference. This duration was found sufficient for the textbook-level problems in our benchmark. Preliminary experiments with longer timeouts (200–300 seconds) showed diminishing returns, supporting our choice of 100 seconds.\n\n- **Comparison: Base, SFT, and RL Approaches**  \n  To highlight the advantage of our online RL approach over a strong supervised fine-tuning (SFT) baseline, we present the following pass@1 results:\n\n  | Models         | NL4OPT | MAMO Easy | MAMO Complex | IndustryOR | OptMATH-193 |\n  |---------------|--------|-----------|--------------|------------|-------------|\n  | BaseModel     | 75.1%  | 81.3%     | 22.7%        | 13.0%      | 4.1%        |\n  | SFT           | 83.3%  | 86.4%     | 38.0%        | 24.0%      | 20.8%       |\n  | RL        | 96.3%  | 90.0%     | 62.1%        | 33.0%      | 29.0%       |\n\n  As shown, our method consistently outperforms SFT across all benchmarks. The performance gain is especially pronounced on more challenging datasets, underscoring the necessity of RL for complex problem solving.\n\n---\n\n**Weakness 2: Clarity of Curated Seed Data for Training Data Synthesis**\n\n**Response:**  \n\n  Thank you for highlighting the importance of seed data clarity. The quality and composition of the seed data are indeed crucial for our data synthesis pipeline and final model performance.\n\n  To clarify, the seed data used in our work is identical to the dataset established and utilized in the foundational ORLM work. For brevity and due to space constraints, we did not repeat the detailed curation process in our manuscript, as it is thoroughly documented in the original ORLM paper. Based on your feedback, we have revised Section 3.1 (\"Data synthesis framework\") to explicitly state that our process builds upon the ORLM seed data, and we have added a clear citation to the original paper for readers seeking a comprehensive description. We believe this revision improves the clarity and self-containedness of our methodology.\n\n---\n\n**Q1: What is the curated seed data used for training data synthesis? Is it exactly the same as the seed data used in ORLM and OR-Instruct (e.g., 686 cases in their study)? If yes, is there a risk of bias or overfitting when evaluating on the IndustryOR benchmark?**\n\n**Response:**  \n\nAs stated in the appendix, the seed data is identical to that used in ORLM (686 cases). While there is a potential risk of overfitting on IndustryOR, our model demonstrates robust generalization across other test datasets. Furthermore, the data synthesis process significantly alters the data distribution. This is evidenced by the fact that SFT alone could not replicate our results, highlighting the effectiveness of our approach.\n\n---\n\n**Q2: Is reliance on external optimization solvers a limitation, given that verifiable reward mechanisms are only applicable to a narrow set of problem domains? How might this approach extend to tasks without access to classical solvers, or are there alternative strategies to overcome this constraint?**\n\n**Response:**  \n\n  We appreciate this insightful question regarding the role of external solvers. Rather than a limitation, we view this as a core design principle.\n\n  The results of training via textual step-by-step reasoning without external solver assistance are shown below:\n\n  | Models               | NL4OPT | MAMO Easy | MAMO Complex | IndustryOR | OptMATH-193 | \n  |----------------------|--------|-----------|--------------|------------|-------------|\n  | Qwen2.5-7B-Instruct  | 24.5%  | 14.7%     | 1.4%         | 7.0%       | 5.7%        | \n  | Math4Opt-RL          | 15.5%  | 42.2%     | 3.3%         | 13.0%      | 13.5%       | \n\n  Here, Math4Opt-RL is trained via the RLVR approach without using solvers as verifiers. As these results demonstrate, relying solely on an LLM's sequential reasoning leads to extremely low performance across most optimization benchmarks.\n\n  While large reasoning models (e.g., DeepSeek-R1) excel at sequential reasoning for tasks like AIME mathematics, optimization modeling presents unique challenges. It requires a global understanding of highly interconnected objective functions, constraints, and decision variables, as well as the execution of complex computational steps (e.g., the simplex method for linear programming, branch-and-bound for mixed integer programming). This computational burden is difficult to handle through purely textual, step-by-step reasoning.\n\n  The optimization community relies on mature solvers because they provide the global perspective and computational power needed to resolve these intricate interdependencies. By leveraging LLMs for accurate problem understanding and initial mathematical formulation, and delegating the computationally intensive aspects to specialized solvers via generated code blocks, we achieve high performance in optimization tasks. This division of labor is fundamental to our approach."}, {"review_id": "OqnUq7eUGY", "reviewer": "Reviewer_fAF4", "summary": "This paper presents SIRL, a learning framework that finetunes LLMs for optimization modeling using reinforcement learning. Using optimization solver as reward verifiers and with a partial KL surrogate reward design, the proposed method achieves performance improvement on many optimization modeling benchmarks.", "questions": "See Weaknesses", "limitations": "While the authors claim in the checklist they discuss limitations in the conclusion section, I do not see much discussion on the limitation in the conclusion. Hence, I do not believe the authors adequately addressed the limitations of the work. The authors should provide a more open discussion in terms of the limitations of the work.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "[Strengths]\n\n1. The paper is clear and easy-to-follow. \n2. I find the partial KL design interesting and makes sense in the RL training context. The reward bonus design in sec 3.3 also makes sense.\n3. There seems to be reasonable performance improvement comparing with the baseline, if we look at the numbers in the result table. (However, I have doubts in the comparsion, so please see my comments in weakness 4 for details.) \n\n[Weakness]\n\n1. While the authors mention they will provide the checkpoint and test code in the appendix, it is unclear if they will open source the training data and the training code. Hence, I feel it’s questionable regarding the commitment of the open source. I wonder if the authors can comment on their plan to open source the training data and the full code (including training)?\n2. Limited contribution of the training data: many previous works propose to synthesize training data (e.g. ORLM, LLMOPT, OptMath). I’m a bit confused why the authors in this paper proposes another data synthesis framework. What’s the difference between the proposed data synthesis framework and those in the previous works? Why not just use the previous papers’ synthesized datasets? The contribution in terms of the training dataset seems limited given its very similar to those proposed in previous works.\n3. Novelty may be a little limited in terms of RL training: RL has been widely used in post-training LLMs for many domains such as math / competitive programming. So, one can argue that the contribution of RL training proposed in this work is limited.\n4. Questionable baseline comparisons: \n     - I find the comparison with LLMOpt and OptMath a bit questionable. I believe the LLMOpt paper reports a higher performance via self-correction and OptMath has better results with their 14b and 32b models. The authors should report the best of those baselines instead of the weakest when comparing the results. \n    - Furthermore, the authors should also report the performance of the SOTA reasoning models (E.g. GPT-o3, Deepseek-r1, instead of only reporting the non-reasoning models). \n    - Lastly, the authors should also compare the RL model with (1) pure supervised-finetuning on their synthesized training data and (2) combining supervised-finetuning with RL to see the benefit of RL training (if any).", "was_revised": true, "final_justification": "My main concern for this paper is that the test benchmarks evaluated by the methods are often times faulty. In my experience, there could be up to 20-30% of the data in the test benchmarks that are wrong (e.g. all TSP in MAMO complex has wrong \"ground truth\" answers, and the IndustryOR benchmark has gone through multiple iterations of ground truth value correction up till last month, and many instances in Optmath are ambiguous). Given that the methodology proposed in this paper is purely empirical, I feel like a thorough cleaning of the test benchmarks is necessary to justify the performance improvement from the proposed method. Another concern is that, I understand this paper is the first paper that uses RL to train a LLM for optimization formulation, but given the abundance of paper that uses RL to train LLMs for math / coding / planning tasks, I do not feel like the paper is novel enough.\n\nNotes regarding empirical rigor: if the AC reads the authors' response carefully, we can see a significant performance drop of their method after they clean up the wrong TSP labels in MAMO complex, while the performance of GPT-o3 and Deepseek-r1 significantly improved -- in my experience, there are many other wrong instances in MAMO complex beyond TSPs, and I'm worried that their model's performance will further drop  after those wrong labels are fixed. Furthermore, I feel like the authors should have taken the initiative to clean up the test benchmarks before the paper submission, instead of relying on the reviewer to tell them where in the benchmark is wrong. I personally feel like the way that the authors ask the reviewer for the cleaned test dataset feels super unjustified given they should be responsible for their results.", "rebuttal": "**Weakness 1: Open Source Commitment**\n\n**Response:**  \n\nWe appreciate the reviewer’s question regarding our open-source commitment. We  confirm our plan to open-source the full training codes and a significant portion of training data.\n\n   * To demonstrate this commitment, we have already released two checkpoints (SIRL-Qwen2.5-7B-Gurobi and SIRL-Qwen2.5-7B-COPT) for immediate community use, and we plan to release more. \n   * In addition, we are actively preparing to release the complete training codebase and a significant portion of our training dataset. Our goal is to empower the community to build their own frameworks for diverse optimization problems or different optimization tools based on our work.\n\n---\n\n**Weakness 2: Contribution of the Training Data**\n\n**Response:**  \n\nThank you for this insightful question. We acknowledge that our data synthesis pipeline builds upon foundational works like ORLM. Our primary motivation stems from the distinct data requirements of our Reinforcement Learning (RL) approach compared to traditional Supervised Fine-Tuning (SFT). While SFT relies on (question, CoTs, code) style solution trajectories, our RL framework requires high-quality (question, answer) pairs to generate a reliable reward signal.\n\nTherefore, our contribution is not a completely new synthesis framework, but a crucial enhancement focused on maximizing the correctness of the final answer. We achieve this with new emerging techniques for the LLM community(LLMs as a judge, reflection) and also our novel instance-enhanced self-consistency (\"inst_sc\") method. The instance-enhanced self-consistency goes beyond traditional majority voting on the final value by leveraging the solver's LP file to enforce consensus on the underlying mathematical structure (e.g., variable types). As shown in our experiments, this provides more robust verification, ensuring the high-quality data essential for our RL training paradigm.\n\n---\n\n**Weakness 3: Novelty of RL Training**\nNovelty may be a little limited in terms of RL training: RL has been widely used in post-training LLMs for many domains such as math / competitive programming. So, one can argue that the contribution of RL training proposed in this work is limited.\n\n**Response:**  \n\nWe appreciate the reviewer’s comment regarding the novelty of RL training within our framework, especially given its broader application in LLM post-training. We acknowledge that RL has indeed been widely adopted for enhancing LLMs in various domains, such as mathematics and competitive programming.\n\nHowever, optimization modeling presents unique challenges. While large reasoning models (e.g., DeepSeek-R1), are great at step-by-step thinking for tasks like AIME math problems, optimization modeling requires a global understanding of highly interconnected objective functions, constraints, and decision variables, followed by computationally intensive steps (e.g., simplex method for linear programming, branch-and-bound for mixed integer programming). These complex computational burdens are inherently difficult for purely textual, step-by-step reasoning.\n\nBelow are results for models trained via step-by-step reasoning, without external solver assistance:\n\n| Model                | NL4OPT | MAMO Easy | MAMO Complex | IndustryOR | OptMATH-193 |\n|----------------------|--------|-----------|--------------|------------|-------------|\n| Qwen2.5-7B-Instruct  | 24.5%  | 14.7%     | 1.4%         | 7.0%       | 5.7%        |\n| Math4Opt-RL          | 15.5%  | 42.2%     | 3.3%         | 13.0%      | 13.5%       |\n\nAs these results demonstrate, relying solely on an LLM's sequential reasoning leads to extremely low performance across most optimization benchmarks.\n\nThe established practice within the optimization community utilizes mature solvers because they provide the global view and computational rigor necessary to resolve these intricate interdependencies. Our novelty stems from effectively integrating RL to teach LLMs to accurately understand problems and formulate them into solver-executable code, thereby delegating the heavy computational lifting to these specialized tools.\n\n---\n\n**Weakness 4: Baseline Comparisons**\n1. Baseline comparisons lack fairness; LLMOpt (self-correction) and OptMath (14B/32B) results omitted.\n\n2. Compare against top reasoning models (e.g., GPT-o3, Deepseek-r1).\n\n3.  Include ablation results for RL training, supervised fine-tuning, and their combination.\n\n**Response:**  \n\n**A-1: Baseline Evaluation Protocol**  \nWe appreciate the insightful question about our baseline comparisons, specifically regarding LLMOpt and the role of self-correction. We acknowledge that self-correction is a powerful strategy that can boost the performance of LLMs in various domains. However, our primary goal was to provide a rigorous and consistent comparison of the core generation capabilities of different models and training methodologies.\n\nTherefore, our main results are based on the OptMath paper's rigorous and well-defined evaluation protocol, which uses Pass@1 accuracy, where a solution is considered valid only if the relative error is less than 1e-6. We believe this strict, single-pass evaluation is crucial for the optimization community as it provides a clear comparison of how well a model can generate a correct and solvable optimization formulation on its first attempt, without iterative refinement or external strategic guidance during inference.\n\n**A-2: Comparison with Stronger Baselines and SOTA Reasoning Models**  \nWe are grateful to the reviewer for the excellent suggestion to include comparisons with stronger OptMath baselines  and leading reasoning models like DeepSeek-R1 and OpenAI-O3. We also include results from our 32B SIRL-model, ensuring equivalent parameter size across the evaluated models .\nThis is a valuable addition that clarifies the landscape. We have now run these experiments and present the updated Pass@1 comparison in the table below:\n\n| Model         | NL4OPT | MAMO Easy | MAMO Complex | IndustryOR | OptMATH-193 |\n|---------------|--------|-----------|--------------|------------|-------------|\n| SIRL-7B       | 96.3%  | 90.0%     | 62.1%        | 33.0%      | 29.0%       |\n| SIRL-32B      | 97.1%  | 88.8%     | 65.9%        | 39.0%      | 46.6%       |\n| optMath-7B  | 94.7%|  86.5% | 51.2%   | 20.0% |   24.4%      |\n| optMath-32B   | 95.9%  | 89.9%     | 54.1%        | 31.0%      | 34.7%       |\n| DeepSeek-R1   | 82.4%  | 77.8%     | 49.3%        | 45.0%      | 50.3%       |\n| OpenAI-O3     | 69.4%  | 70.1%     | 38.8%        | 44.0%      | 39.9%       |\n\nIt is interesting to note that while large reasoning models (LRMs) exhibit surprisingly strong performance on some of the harder, more complex benchmarks such as IndustryOR and OptMATH-193, they generally perform poorly on simpler questions.\n\nOur initial analysis shows that for \"simpler\" optimization questions (where the problem structure might be straightforward but requires precise, direct solver invocation), these reasoning models tend to \"overthink.\" They tend to generate unnecessary verbose reasoning steps or fail to precisely generate the correct solver code, as they are not explicitly optimized for concise, error-free tool-use.\n\n**A-3: RL vs. SFT and SFT+RL**  \nWe appreciate the reviewer’s crucial question regarding the comparison of our RL model with supervised fine-tuning (SFT) baselines. To thoroughly address this, we have conducted the requested experiments and present the Pass@1 comparison in the table below, clearly demonstrating the benefit of our online RL training:\n\n| Model      | NL4OPT | MAMO Easy | MAMO Complex | IndustryOR | OptMATH-193 |\n|------------|--------|-----------|--------------|------------|-------------|\n| BaseModel  | 75.1%  | 81.3%     | 22.7%        | 13.0%      | 4.1%        |\n| SFT        | 83.3%  | 86.4%     | 38.0%        | 24.0%      | 20.8%       |\n| RL     | 96.3%  | 90.0%     | 62.1%        | 33.0%      | 29.0%       |\n\nAs shown, our method consistently outperforms SFT across all benchmarks. Notably, the performance gain is most significant on more challenging datasets like MAMO Complex (+24.1%) and IndustryOR (+9.0%), demonstrating the necessity of RL for solving complex problems.\n\n---\n\n**Limitation**\n\nThe conclusion section lacks a thorough discussion of the work's limitations. The authors should more openly and explicitly address the limitations of their approach.\n\n**Response:**  \n\nThank you for pointing out the insufficient discussion of limitations. In our understanding, the limitations an further work can be broadly categorized as:\n\n***Robustness of Reward Mechanism with Solver Feedback:**\n\nWhile our framework benefits from utilizing rich feedback signals directly from classical solvers (e.g., LP files and optimal values), extracting more signals and precisely translating this information into a perfectly robust reward mechanism for LLMs remains a significant challenge, there is still scope for enhancing performance and mitigating 'reward hacking.\n\n**Depth of Error Analysis:**\nOur current evaluation primarily relies on quantitative metrics, and our error analysis is largely qualitative, focusing on observed failure cases. We also recognize that even the  LRMs (e.g., Deepseek-R1)  continue to face significant challenges in achieving high scores on difficult benchmarks such as IndustryOR and OptMATH—an open problem that warrants further investigation and the effort of all optimization community. In future work, we plan to conduct a more systematic and comprehensive analysis of error patterns, and to implement targeted model improvements aimed at addressing the specific weaknesses identified."}, {"review_id": "gcyiecqFat", "reviewer": "Reviewer_DwV1", "summary": "The paper proposes a reinforcement learning with verifiable rewards training pipeline to improve LLMs' capabilities to interface with external optimization solvers. Given verbal descriptions of optimization problems, the language model needs to write down the corresponding mathematical formula, variables and constraints, as well as code that can be executed by the solver to find the optimal solution. Specifically, the work proposes a synthetic training data generation pipeline to derive ground truth answers, using a per-feature self-consistency score, and designs a reward function for this task, balancing answer format, code executability, answer correctness and the use of advanced modelling techniques. In RL training, the paper proposes to maintain the KL penalty in Reinforce++ for the mathematical and code answers but remove it for the verbal reasoning parts. The proposed modifications improve performance on several evaluation benchmarks compared to ablations and prior baselines.", "questions": "- Did you consider other ways of encouraging exploration in verbal reasoning other than removing the KL penalty, such as using a higher temperature at training time? Or other ways of maintaining the correct format for math or code, such as structured output / limiting the token vocabulary when appropriate?\n\n- Why does inst_sc outperform val_sc? What kind of omissions or mistakes do the highest-performing solution attempts chosen by val_sc make? How did you identify inst_sc was an appropriate strategy for those issues?\n\n- How is the two-stage RL training implemented: when are stage 1 rewards replaced by stage 2 rewards?", "limitations": "Limitations of the work are not discussed, despite claiming so in the NeurIPS checklist.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n- As a fully simulated environment, defining specifications for optimization problem solvers based on the achieved objective value is a scalable source of ground truth rewards for RL training, and therefore a good application domain for reasoning model training strategies.\n- Empirical results are encouraging: fine-tuned Qwen2.5-7B outperforms advanced models GPT-4 and DeepSeek-V3 as well as 5 previous LLM-for-optimization methods.\n- Ablations validate the contribution of each proposed component.\n\n**Weaknesses**:\n- The main weakness is the work’s limited novelty. The novel parts, namely the selective application of the KL-penalty to certain CoT segments, and computing self-consistency on a per-dimension basis, are rather minor changes to existing RLVR approaches, especially given that the KL-penalty is already commonly omitted in RLVR training for mathematical problems in previous works.\n- The presentation and writing could be improved. As it stands, the main contributions come across as empirical and out of the blue, without much general motivation presented. A thorough motivating analysis for why and how they were derived and why they are a sensible solution to specific observed issues would help in understanding the generalizability of the findings. Even if the implementation changes themselves are small, it could make for a stronger paper if they were well motivated.\n- The reported results do not include error bars, despite claiming so in the NeurIPS checklist, so the statistical significance of the results is unclear.\n- The problem description in the abstract and introduction is very generic (e.g. optimization \"across diverse domains\"): a motivating example of the problem to be solved would improve readability for a non-OR audience. Some of the description at the beginning of Section 3.2 could be mentioned earlier.\n- Some experimental details are not clarified, such as the meaning and implementation of the 10 LLM roles. The relevant section in the main text should refer to the corresponding Appendix section. It is also not explained when stage 1 rewards are replaced with stage 2 rewards.", "was_revised": true, "final_justification": "The reviewers have adequately addressed all of the questions and the weaknesses 3-5 that I raised.\n\nThe answers to question 2 and weakness 3, in particular, improve the paper, and clarify the value and motivation of instance-enhanced self-consistency.\n\nHowever, I will not increase my score to Accept (5) due to modest conceptual novelty, as highlighted in weaknesses 1 and 2. Still, the contributions, though small, seem valid and empirically justified, which is why I increased to Borderline Accept (4).", "rebuttal": "**Question 1: Other ways of encouraging exploration and approaches to maintain the correct format for math or code**\n\n**A-1: Other ways of encouraging exploration**   \n\nThank you for this insightful question. Indeed, we did experiments with several alternative sampling strategies.\nwe tested various temperature settings (e.g., T=0.8, 1.0, 1.2) with top-p sampling and also explored more recent methods like min-P[1].\nEmpirically, the default configuration of top-p sampling(T=1.0, top-p=0.95) paired with our method yielded consistently good performance.\n\nReference list:\n\n   [1]. Nguyen, M. N., Baker, A., Neo, C., Roush, A., Kirsch, A., & Shwartz-Ziv, R. (2024). Turning up the heat: Min-p sampling for creative and coherent llm outputs\n\n\n**A-2: maintenance of correct format for math or code**   \n\n Compared to fixed structured output or limiting token vocabulary, our primary mechanism is to integrate system prompt template and the precisely aligned reward mechanism.\n\nInspired by DeepSeek-R1's practices, we use the system prompt o guide the overall output format. \nCorrespondingly, we explicitly define a ``format reward'' derived from successfully parsing specific delimiters such as <model>...</model> and <python>...</python>, alongside an ```execution reward'' for runnable code.\n This design directly encourages the policy to maintain and improve the correct format through active learning and feedback.\n\nThis reward-based approach, combined with our selective KL penalty for policy control, provides a more general and adaptable solution for LLMs in optimization modeling, accommodating the diverse programming language styles of various solvers, e.g., Gurobi, Cplex, COPT.\n\n---\n\n**Question 2: Effectiveness of Val_sc**\n\n**Response:**  \n\n Thank you for this insightful question. The key reason \"inst_sc\" outperforms \"val_sc\" is that it moves beyond comparing only the final objective value. \"Val_sc\" is a \"black-box\" method that can be misled when multiple, structurally different (and often incorrect) model formulations happen to produce the same numerical answer by coincidence. \"Inst_sc\" provides a more discerning evaluation by enforcing consensus on the underlying mathematical structure itself, using features extracted from the solver's LP file.\n\nA primary mistake made by top-scoring solutions chosen by \"val_sc\" is incorrect variable types. LLMs often struggle with the semantic nuance between a binary choice (e.g., \"to build a facility or not,\" requiring a binary variable) and a general discrete quantity (requiring an integer variable). A solution might incorrectly use an integer variable, but if the optimal value happens to be 0 or 1, \"val_sc\" would incorrectly validate it based on the objective value. Other errors include using the wrong optimization direction (e.g., maximization instead of minimization).\n\nWe identified \"inst_sc\" as the appropriate strategy after observing these specific failure modes. Since the LP file provides objective, structured statistics of the model, we hypothesized that incorporating its structural features (like binary/integer variable counts and optimization direction) into the consensus check would directly target these common LLM errors. Our experiments confirmed this, showing \"inst_sc\" effectively filters out these structurally flawed models and improves accuracy.\n\n---\n\n**Question 3: Two-stage RL Training Implementation**\n\n**Response:**  \n\nThe training process involves two distinct stages, both utilizing the exact same training dataset.\n   * In stage 1, we utilize the specific 'stage-1 reward' signal to guide the training. \n   * For stage 2, training commences from the checkpoint obtained at the conclusion of Stage 1.\n   * Both stages  required approximately 24×8 H100 GPU hours for completion, totally 48×8 H100 hours.\n\n---\n\n**Weakness1 and Weakness2:  Motivation and Limited Novelty**  \n\n**Response:**  \n\nThank you for your constructive and essential feedback. We appreciate your perspective on the novelty and motivation of our work. We want to clarify that our core contribution lies not in proposing entirely new RLVR mechanisms in isolation, but in critically adapting and refining existing RLVR techniques to overcome fundamental limitations of LLMs when tackling complex optimization problems that necessitate external solver interaction.\n\nWhile LRMs (e.g., DeepSeek-R1) excel at sequential reasoning for tasks like AIME mathematics, optimization modeling presents a unique challenge. It inherently demands a global understanding of highly interconnected objective functions, constraints, and decision variables. Furthermore, it requires complex computational steps (e.g., executing the simplex method for linear programming, branch-and-bound for mixed-integer problems), which can be difficult to handle through purely textual, step-by-step reasoning.\n\nBelow are early results using purely textual step-by-step reasoning, without external solver assistance:\n\n| Models                | NL4OPT | MAMO Easy | MAMO Complex | IndustryOR | OptMATH-193 |\n|-----------------------|--------|-----------|--------------|------------|-------------|\n| Qwen2.5-7B-Instruct   | 24.5%  | 14.7%     | 1.4%         | 7.0%       | 5.7%        |\n| Math4Opt-RL           | 15.5%  | 42.2%     | 3.3%         | 13.0%      | 13.5%       |\n\nAs these results demonstrate, relying solely on an LLM's sequential reasoning leads to extremely low performance across all benchmarks. The established practice within the optimization community is to utilize mature solvers precisely because they offer the global view and computational power necessary to resolve these intricate interdependencies.\n\nBy leveraging LLMs for accurate problem understanding and initial mathematical formulation, we can delegate the computationally intensive aspects of optimization to robust, specialized solvers through the generation of solver code blocks. This crucial division of labor is fundamental to achieving high performance in optimization tasks.\n\nBuilding on these findings, one of the main challenges  is to achieve both diverse reasoning exploration (which LLMs are good at) and strict accuracy and validity (which solvers demand).\nThe selective KL-penalty with the tailed solver-based reward design, offers a robust and generalizable solution, makes it highly adaptable for extending LLM assistance across a variety of optimization solvers.\n\n---\n\n**Weakness3: Error Bars of the Results** \n\n**Response:** \n\n We sincerely apologize for the oversight in not including this crucial information in our initial submission. We have now conducted ten runs using top-p (Nucleus) sampling (temperature=0.5, top-p=0.95). Here are the mean and standard deviation results:\n\n| Metric         | NL4OPT           | MAMO Easy        | MAMO Complex     | IndustryOR      | OptMATH-193      |\n|:-------------- |:---------------- |:---------------- |:---------------- |:--------------- |:---------------- |\n| mean ± std     | 96.2% ± 0.32%    | 90.1% ± 0.14%    | 61.3% ± 0.68%    | 33.7% ± 1.5%    | 28.3% ± 0.66%    |\n\nAs the table shows, our results are stable and consistent with those presented in the initial submission.\n\n---\n\n**Weakness4: Writing of the Problem Description** \n\n**Response:**  \n\nThank you for this very constructive feedback. We fully agree that a concrete, motivating example would significantly improve the manuscript's readability and accessibility, especially for readers without a background in Operations Research. Your suggestions are extremely helpful for strengthening the introduction.\n\nIn our revision, we will restructure the introduction to better frame the core challenge. We will incorporate a classic motivating example—such as a logistics or resource allocation problem—to illustrate the practical journey from a real-world business scenario to a formal mathematical model. This will help ground the generic phrase \"optimization across diverse domains\" by providing a tangible application.\n\nFurthermore, as you astutely pointed out, the description of the modeling process at the beginning of Section 3.2 is better suited for an earlier section. We will move this high-level overview—which describes the stages of problem analysis, mathematical formulation, and code implementation—into the introduction. This will provide readers with a clear, upfront understanding of the complex task our work aims to automate.\n\nWe believe these changes will make the problem statement clearer from the outset and the motivation for our SIRL framework more compelling to a broader audience. Thank you again for your valuable guidance.\n\n---\n\n**Weakness5: Experimental Details Clarification** \n\n**Response:** \n\nWe acknowledge that the role and implementation of the \"10 LLM roles\" were not sufficiently detailed in the main text, and we appreciate the opportunity to clarify.\n\nThe primary rationale for using 10 distinct LLM roles is to enhance the diversity of the generated outputs. By prompting the model with different expert personas (e.g., an \"operations research scientist\" versus a \"Python engineer\"), we encourage it to explore varied reasoning paths and formulation strategies for the same problem. This diversity is crucial for the robustness of any self-consistency method; it mitigates the risk of all generation attempts converging on the same plausible but incorrect solution due to a shared systemic bias in the model. A more diverse pool of candidate solutions increases the probability that the correct formulation is present and can be identified by our consensus mechanism.\n\nDue to page limitations, we placed the specific prompts that define each of these roles in the Appendix. We recognize our oversight in not explicitly linking to this from the main text. In our revised manuscript, we will amend Section 3.1 (\"Instance-enhanced self-consistency\") to briefly explain the purpose of these roles and add a direct reference to Appendix C.1 for the detailed prompts.\n\n---"}]

Author rebuttal:

Meta-review: The paper proposes Solver-Informed Reinforcement Learning (SIRL), a framework for grounding large language models in authentic optimization modeling by using classical optimization solvers as verifiable reward providers. Solvers serve as automatic oracles that check syntax, feasibility, and solution quality, producing reward signals that guide RL training. The approach incorporates a two-stage reward curriculum, a Partial-KL surrogate that selectively regularizes code and model tokens, and an instance-enhanced self-consistency method for robust data synthesis.

The strengths of the paper lie in its novel application of RL with verifiable rewards to optimization modeling, strong experimental results with ablations justifying design choices, and clear improvements over both smaller and larger baselines. The rebuttal effectively clarified several reviewer concerns: it explained that the advanced modeling reward encourages only necessary reformulations, demonstrated that solver bottlenecks are minor for benchmark problems, provided concrete training cost and compute details, clarified that the method generalizes beyond LP to MIP and SOCP, and confirmed plans to open-source code, checkpoints, and data. Reviewers remained somewhat concerned about novelty relative to prior RL applications in LLMs, reliance on solvers as verifiers, possible overfitting or data bias, benchmark cleanliness, and incomplete open-sourcing at submission time. Post-rebuttal comments, however, acknowledged that the authors’ clarifications strengthened the paper, with some reviewers moving to acceptance while others stayed borderline due to empirical rigor and novelty concerns.

Overall, the paper presents a strong and well-validated contribution, and despite some concerns about benchmark quality and incremental novelty, the clear improvements, thoughtful design, and solid rebuttal justify acceptance.
Decision: Accept (poster)