StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models
Authors: Chenyu Zhou
Tianyi Xu
Jianghao Lin
Dongdong Ge
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Operations Research
Self-Evolving Framework
Generative Process Supervision
Primary area: optimization

Abstract: Large Language Models (LLMs) have shown promising capabilities for solving Operations Research (OR) problems. 
While reinforcement learning serves as a powerful paradigm for LLM training on OR problems, existing works generally face two key limitations. First, outcome reward suffers from the $\textit{credit assignment problem}$, where correct final answers can reinforce flawed reasoning.
Second, conventional discriminative process supervision is $\textit{myopic}$, failing to evaluate the interdependent steps of OR modeling holistically. 
To this end, we introduce $\textbf{\texttt{StepORLM}}$, a novel self-evolving framework with generative process supervision. 
At its core, $\texttt{StepORLM}$ features a co-evolutionary loop where a policy model and a generative process reward model (GenPRM) iteratively improve on each other. 
This loop is driven by a dual-feedback mechanism: definitive, outcome-based verification from an external solver, and nuanced, holistic process evaluation from the GenPRM. 
The combined signal is used to align the policy via Weighted Direct Preference Optimization (W-DPO) and simultaneously refine the GenPRM. 
Our resulting 8B-parameter $\texttt{StepORLM}$ establishes a new state-of-the-art across six benchmarks, significantly outperforming vastly larger generalist models, agentic methods, and specialized baselines. 
Moreover, the co-evolved GenPRM is able to act as a powerful and universally applicable process verifier, substantially boosting the inference scaling performance of both our own model and other existing LLMs. 
We release our models and code to facilitate future research (https://github.com/0xzhouchenyu/StepORLM).

Published as a conference paper at ICLR 2026
STEPORLM: A SELF-EVOLVING FRAMEWORK WITH GENERATIVE PROCESS SUPERVISION FOR OPERATIONS RESEARCH LANGUAGE MODELS
Chenyu Zhou, Tianyi Xu, Jianghao Lin; Dongdong Ge
Antai College of Economics and Management, Shanghai Jiao Tong University
Shanghai, China 1 {chenyuzhou, crimsonflag, linjianghao, ddge)@sjtu.. edu.
ABSTRACT
Large Language Models (LLMs) have shown promising capabilities for solving Operations Research (OR) problems. While reinforcement learning serves as a powerful paradigm for LLM training on OR problems, existing works generally face two key limitations. First, outcome reward suffers from the credit assignment problem, where correct final answers can reinforce flawed reasoning. Second, conventional discriminative process supervision is myopic, failing evaluate the interdependent steps of OR modeling holistically. To this end, we introduce StepORLM, novel self-evolving framework with generative process supervision. At its core, S StepORLM features a co-evolutionary loop where policy model and generative process reward model (GenPRM) iteratively improve each other. This loop is driven by dual-feedback mechanism: definitive, outcome-basse verification from an external solver, and nuanced, holistic process evaluation from the GenPRM. The combined signal is used to align the policy via Weighted Direct Preference Optimization (W-DPO) and simultaneously refine the GenPRM. Our resulting 8B- 8B-parameter StepORLM establishes new state-of-the-arr across six benchmarks, significantly outperforming vastly larger generalist models, agentic methods, and specialized baselines. Moreover, the co-evolved GenPRM is able to act as a powerful and universally applicable process verifier, substantially boosting the inference scaling performance of both own model and other existing LLMs. We release our models and code to facilitate future researct
INTRODUCTION
Large language models (LLMs) have shown remarkable capabilities for complex reasoning, en- abling new frontiers in specialized domains like Operations Research (OR) (Xiao et al. (025). Research on applying L.LMs to OR has largely advanced along two fronts: 1) developing agentic Teshnizi systems TAhmadFTeshnizi et al. [2024a] Xiao al., (024) and improving LLMs through specialized
training (Huang et 2025). Within the latter, reinforcement learning (RL) has become a dominant
paradigm through outcome reward (Chen al. 2025b) or process supervision (Astorga et al. 2024
Chen et al. 2025a).
However, these established RI paradigms exhibit critical flaws when applied to operations research, which requires long-houd long-horizon, interdependent reasoning. Outcome-basee supervision, which rewards model based solely on the final solution's 's correctness, suffers from the credit assignment problem. As illustrated in Figure aa), optimal outcome can emerge from flawed reasoning process, inadvertently reinforcing incorrect intermediate steps. While process supervision aims to resolve this by rewarding intermediate steps, standard discriminative step-wise PRMs often suffer from myopic view, evaluating steps in isolation. This is particularly detrimental in OR, where the validity
"Corresponding author. htttlp////ggttuuucc///////iieeee

---

Published as a conference paper at ICLR 2026
Whole Trajectory
Failure Case
Failure Cose B
Correct Case
Variables
Discrininative PRM
Objective
Variables
Constriants
Stop expanding Revise before generate next step!
Code
Objective
Same Trajectory
Constraints
Generative PRM
Variables
+1
Solver
Optimal Value
Objective
+1
O 0
Constriants
- The whole trajectory done! Now I know that
Pass Foil
Code
+1 the variable step is correct!
(a) Incorrect steps within passed cases
(b) The advantage of Generative PRM in OR problems
Figure 1: The illustration (a) the credit assignment problem of outcome reward and (b) the myopic issue of discriminative process supervision, which motivates the application of generative process supervision for operations research language models. of constraint scoring unreliable.
heavily on variable definitions introduced many steps earlier, making isolated
As shown in Figure [b), the correctness of an early step, like a variable definition, may only become apparent once subsequent constraints are defined, causing discriminative PRMs to provide unreliable or inconsistent rewards.
Therefore, we argue that the contextual and interdependent nature of OR modeling demands paradigm shift from myopic, step-wise evaluation holistic, trajectory-level process supervision. The critic should assess the entire reasoning path retrospectively and understand the intricate de- pendencies between steps before assigning credit. As illustrated FigureT[),, this motivates the application of generative PRMs that can reason about the completed trajectory and generate holistic critique, rather than a discriminative one that provides disconnected, localized scores.
To this end, we propose novel Self-Evolving Framework with Generative Process Supervision for Operations Research Language Models (StepORLMM. At the heart of our framework a co- evolutionary loop where policy model and novel generative process reward model (GenPRM) iteratively improve each other. At each iteration, the policy generates reasoning trajectories that are evaluated from complementary perspectives: definitive outcome verification from external solver and nuanced, holistic process feedback from GenPRM. This dual-feedback signal is distilled into preference pairs align the policy via Weighted Direct Preference Optimization (W-DPO) objective and 2) filtered trajectory rewarding data to further fine-tune GenPRM. In this way, both the policy model and GenPRM would iteratively improved through the self-evolving loop, ultimately leading to process-sound and outcome-comect reasoning capabilities to solve complex OR problems.
Our contributions are summarized follows:
To the best of our knowledge, we are the first to deploy generative process supervision in the solver-based OR domain. Unlike general reasoning tasks, we ground the supervision strict solver feasibility and numerical optimality, mitigating the credit assignment problem and myopia issue inherent to existing methods.
We propose StepORLM, novel self-evolving framework centered on the co-evolution of a policy model and a generative process reward model (GenPRM). It integrates a scalable data synthe- sis pipeline with a dual-feedback loop that grounds the models in both process-level logical consistency and final-outcome correctness.
Extensive experiments on six benchmarks show that StepORLM achieves the SOTA performance against both vastly larger generalist/agentic models (e.g., GPT-4o) and specialized baselines. We also demonstrate that the co-evolved GenPRM acts as a universal inference-time verifier, substantially boosting the performance of other OR language models.
2

---

Published as a conference paper at ICLR 2026
We release code, the StepORLM model weights, and the GenPRM verifier weights to the community, providing a strong foundation for future research in LLM-based OR problem solving,
2
RELATED WORK
Large Language Models for Optimization Modeling. Early works explore using LLMs to parse natural language descriptions into mathematical models. For example, the NL4Opt competition (Ramamonjison et al. 2023) demonstrates that ChatGPT and other LLMs can identify problem
entities and generate LP formulations from word problems. Pipeline frameworks (Huang et al.
2025) combine LLM components with traditional solvers: AutoFormulation (Huang et al. [222 Jiang et al. 2025 Lu et al. 2025] Zhou et al., (2225) uses LLMs to tag variables and constrants in Text descriptions, and systems like OptiMUS(A OptiMUS(Ahmadi Ahmadi Teshnizi et al. 2024b) pipeline textual inputs AamaAdTTeheTTe through I.LMMto-code translators before invoking MII. solvers. These methods improve accessibility,
but often produce hallucinated constraints or missing logic, as the LLMs have no built-in notion of
optimization semantics, Agentic methods Zhang et al. ; Yang et al. (2025b) are also proposed, which assign specialized LLM agents under a conductor that orchestrates and reflects on partial solutions (Xiao et al. [024] Deng et al.) 2024 Ahmadii I Teshnizi et al. 2024a)) While these approaches outperform simpler pipelines on benchmarks, they still rely on repeated LLM exchanges without explicit verification of each reasoning step. In practice, even advanced models exhibit unsound reasoning: for instance, ]Tiag et al. (2025) note that alignment and "self-correetion" mechanisms needed to prevent LLM ballucinations in optimization contexts. Concurrent works also explore iterative data synthesis to improve OR reasoning. For instance, Step-Opt (Wu et al. (2025) and Lima et al. (2025) propose pipelines generate verifiable synthetic data using rule-based checkers or symbolic representations. While these methods focus on enhancing data quality on the data side, StepORLM focuses on the model side by establishing co-evolutionary loop between the policy and a generative process reward model.
Advanced Training and Refinement Paradigms. To enhance the reliability of LLM reason- ing, recent work moves beyond simple pipelines incorporate advanced training and refinement paradigms Fang et al. (2025); ; Xi et al. (2025). One major direction is reinforcement learning, which applied with varying feedback granularity. Outcome-driven methods like Solver-Informed RL (SIRL) (Chen et al. 2025b) use solver's 's final verdict for precise rewards but suffer from the credit assignment problem. In contrast, standard discriminative Process Reward Models (PRMs) provide denser feedback but can be myopic, rewarding locally correct steps that are globally inconsistent (Dai Zheng et al. 2025a).
ct al.
2025
Very recently. generative process supervision has emerged to address these limitations. For instance, GenPRM (Zhao et al. (225) and R-PRM (She et al. 2025) utilize generative reasoning to produce holistic critiques rather than isolated scores. Similarly, frameworks like rStar-Math (Guan et al. 2025) demonstrate the power of self-evolution strategies for policy improvement, while others explore inference-tine -iee scaling for generalist reward models (Liu al., 2025). However, these approaches typically rely on answer-string supervision. StepORLM adapts these concepts to the solver- based OR domain, introducing dual-feedhack W-DPO objective and explicitly co-evolving the policy and verifier grounded in numerical feasibility. Complementing these training methods are iterative refinement and search strategies. These techniques embed LLMs into structured search frameworks like beam search (Wang et al. (002)) and Monte Carlo Tree Search (Zheng et al.) 2025b) Liu et al. 2024), or employ evolutionary loops with self-refiection iteratively improve generated solutions (Ye et al. 2024). While these approaches make final outputs more reliable by exploring wider solution space, they often operate at the output level rather than being deeply integrated into the model's core training process. Thus, a trade-off remains: outcome-basee RI lacks intermediate supervision, stepwise PRMs lack global coherence, and search methods are often applied post-hoc. To bridge this gap. discriminative methods based on both final outcome correctness and stepwise feedback can penalize incorrect reasoning steps while maintaining global coherence, effectively combining the strengths of both process- and outcome- based approaches (Chen et al. 2025a).
The research community actively builds end-to-ent OR modeling systems that merge symbolic rigor with learned reasoning. This effort is supported by growing number of benchmarks (Wang et al.) 2024; Huang et al., [2024] 2025; Yang et al., 2024 Xiao et al., (025). StepORLM is designed to address these gaps by embedding process- level supervision and iterative refinement, ensuring each
3

---

Published as a conference paper at ICLR 2026
Teacher LUMs
Step-Level Reasoning
Problem Analysis
<step> </Step>
model
Generation
Questions
Enhancement
Expanded
Generation
<step> </step>>
Questions
Muthematical Model
Scenario
<step> </sttop>
refine question if answer is infeasible
<slep? </step>
refine until executable
CDPT solver Code
Training Dataset
<step> </stepp>
(Q, for initial policy model
Reasoning
Step Labeling
Majority
Trajectories per
(Q, A] for evolving training
Answer
Solver
Figure 2:
The illustration data synthesis pipeline for the warm-up stage. A teacher LLM generates diverse OR problems and their step-hy-step reasoning solutions. Each solution is rigorously validated and filtered by an external solver, with an automated refinement loop to ensure the quality. This verified corpus is then used to train the initial policy model via SFT.
modeling step is verified. Its co-evolutionary framework combines generative PRM for nuanced step-level feedback with final-solver rewards, mitigating the credit assignment problem via weighted DPO objective. By having its policy and generative PRM co-evolve, StepORLM learns self-verify internally, leading to more robust and error-resistant modeling behavior that advances the goal of reliable and scalable OR reasoning with LLMs.
METHODOLOGY
OVERVIEW OF STEPORLM
We train StepORLM using two-stage, self-evolving framework designed instill robust, process- sound reasoning for operations research problems. As illustrated in Figure 2 and Figure 3 the framework is composed two main stages:
bootstrap the training process by creating a high-quality, solver-verified dataset, which consists of OR problems and step-by-step solutions. This corpus used train an initial policy mo via Supervised Fine-Tuning (SFT). In practice, this warm-up corpus contains 50K verified training instances that cover both synthesized OR modeling tasks and instruction-style OR queries.
Stage 1: Warm-Up for Initial Policy. As shown in Figure
Stage Iterative Co-Evolution for Policy and GenPRM. As shown in Figure starting with the initial policy and base LLM as the critic pa., we commence a self-evolving training loop. The policy model and the generative process reward model (GenPRM) co-evolve, with the policy generating increasingly better solutions and the GenPRM becoming more discerning critic. Concretely, we construct a hard subset Dhard of 13K problems from the 50K warm-up pool, consisting of instances where the warm-up policy with 5-way majority voting fails to reach unanimous correctness. All self-evolving iterations Stage 2 operate on this hard subset.
3.2 WARM-UP FOR INITIAL POLICY
The goal of the warm-up stage is to produce strong initial policy model capable of generating structured, multi-step solutions to OR problems. We achieve this through a scalable data synthesis pipeline, shown in Figure 2
We begin with a large pool of base OR problem templates and diverse industry scenarios. A powerful teacher LLM (GPT-40 in this paper) first pairs compatible templates and scenarios to draft natural- language question Q. To enrich the data, each question is augmented through paraphrasing, unit conversion, and controlled parameter scaling. The teacher model then generates complete, multi-
4

---

Published as a conference paper at ICLR 2026
step reasoning trajectory for each question Q, mirroring the workflow of a human expert from problem analysis to executable solver code. See Appendix E for more details.
To ensure data quality, every generated solution undergoes deterministic validation process: exccute the code, inspect the solver status, and verify the objective value. Trajectories that produce enrors would trigger automatic self-refinement loop, prompting the teacher LLM revise the formulation or code until validation succeeds or the retry budget is exhausted. For questions with multiple valid solutions, yields a high-quality dataset of verified (Q, Rt) pairs, i.e., Question (Q) and Reasoning (R). We use the constructed dataset to train the initial policy model Ty via Supervised Fine- Tine-Tuning (SFT).
select the one achieving the best objective value. This rigorous process
3.3 ITERATIVE Co- EVOLUTIUUO FOR POLICY AND GENPRM
Following the warm-up stage, the iterative co-evolution stage begins. As illustrated in Figure 3 this self-evolving loop simultaneously enhances the model (GenPRM) Ps. Each iteration involves three key steps: 1) trajectory collection and evaluation, 2) policy alignment, and 3) GenPRM refinement.
policy model Ty and the generative process reward
3.3.1 TRAJECTORY COLLECTION AND DUAL-SOURGH EVALUATION
In the self-evolving stage, we utilize the hard subset Dhard (defined Section 3.1 as the prompt source. Crucially, unlike the static training in the warm-up stage, the training data for W-DPO is dynamically generated. At each iteration, the current policy To generates new set of k candidate solution trajectories for each question in Dhard. These trajectories are then put through dual-source evaluation:
Outcome Verification. The final code of each trajectory executed, and an external solver provides a definitive, ground-truth label of success failure.
Process Evaluation. The GenPRM holistically assesses each complete trajectory and provides step-by-step correctness scores, capturing the quality of the reasoning process. Note that, different from conventional discriminative process supervision, our GenPRM is required to first perform chain-of-thon thothh reasoning over the OR solution and then generate holistic, post-hoc evaluation of the entire reasoning trajectory. This enables GenPRM to capture long-term dependencies that step-wise verifiers often miss (see Appendix C.2 for case study on TSP illustrating how this global view prevents reward hacking).
After assessing the generated trajectories, we use the dual-feedback signals to further refine both the policy and GenPRM. The improved models would then start another round of self-evolving iteration. Next, we introduce the policy alignment and GenPRM refinement at each iteration.
3.3.2 POLICY ALIGNMENT VIA W-DPO
The dual-source feedback is first used to align the policy model. We distill the trajectory evaluation results into preference pairs (Tw, 7, w). where Tw the winning trajectory and 7 is the losing one. The selection logic, as detailed in Algorithm[] prioritizes solver-verified solutions. If solver outcomes identical for pair of trajectories (e.g., both pass or both fail), then the trajectory with higher ratio of correct steps, as judged by the GenPRM, is preferred.
Next, we design a Weighted Direct Preference Optimization (W-DPO) objective for policy training. The weight w is defined to quantify the quality gap between two trajectories: solver- solver-distingun distinguished pairs receive fixed, high weight (i.e., w: = 1.0), while others are weighted proportionally to the difference in their process quality scores r(r), i.e., w = |r(Tpux) r(Trg)).. Unlike unstable step DPO objectives, this weightee formulation stabilizes training by aggregating fine-grained process feedback into robust trajectory-leved signal. The final objective focuses the model training on the most informative pairs weighted by w::
(W-DPO(f) = - -E (T...)) log of log To (T [x) - log n =)))]
(1)
where ((T is the weight for trajectory pair (Turs 71). ( [ x)) is the log likelihooo of generating trajectory given the input question v, and (3 is scaling factor.
5

---

Published as a conference paper at ICLR 2026
Self-evolving Training with GenPRM
Question
polkyy model xe
Base Model
Rollouts
Step
preference learming
Model-ito
GenPRM-itO
[a>1], [a>3], [R>4] [3+1], [3>4], [4>1]
W-DPO
SFT
Step
Model-it1
GenPRM-it1
Step
[1, A]
W-DPO
SFT
Model-it2
GenPRM-it2
Step
W-DPO
GenPRM Pe
StepORLM
Answer
Figure 3: The co-evolutionary loop of StepÖRLM. At each iteration, the policy model generates multiple trajectories. The feedback from both the external solver (outcome) and the GenPRM Po (process) is used to create training data that simultancously refines the policy via W-DPO and improves the GenPRM via SFT, fostering reciprocal improvement.
Algorithm Preference Pair Construction
1: Input: Two trajectories Ta: Tb-
2: Output: A tuple (Twi TI, w).
3: // Rule 1: A solver-verified trajectory always preferred.
4: if Ta has correct result and т does not then
5: return (Ta, Tb, 1.0)
6: else if' 76 has correct result and Ta does not then
7: return (To, Ta, 1.0)
8: end if
9: // Rule 2: Otherwise, prefer the trajectory with higher ratio of correct steps. Let diff +- ConrectSteRRatio(..)) Correc(SteRRati((T)
11: if diff >> 0 then
return (Te, Th, diff)
13: else if diff < () then
14: return (fb, Tas diff)
15: else
16:
return None
end if
3.3.3 GENPRM REFINEMENT VIA SET
The same trajectory evaluation data from the current iteration is then used refine the critic, ie.. GenPRM. Specifically, to ensure high-quality supervision, we select only solver-consistent trajectories, i.e., instances where GenPRM's verdict aligns with the external solver's outcome. For these high-confidence samples, we treat GenPRM's own generated critique and step-wise judgments as targets form new SFT dataset. We provide the exact prompt-completion format and training data schema in Appendix F This dataset is used continue the supervised fine-tuning of GenPRM Pe with process supervision signals.
We can see that such process creates positive feedback loop: the policy improves, it generates more diverse and accurate reasoning paths, providing higher-quality training data for the GenPRM. In turn more capable GenPRM provides more precise and reliable reward signal, further accelerating the policy alignment. This symbiotic refinement is crucial for surpassing the performance of the initial SFT model, which is thus referred to the self-evolving framework.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Benchmarks. We conduct our evaluation on a diverse set of six public benchmarks widely used in the OR community: NL40pt (Ramamonjison al. 2023). MAMO (split into EasyLP and ComplexLP) (Huang et al., 2024), NLP4LP AhmadiTeshhip et al., 2024b), ComplexOR (Xiao et al.
2024), IndustryOR (Huang et al. 2025) and ReSocratic Yang 2025c More details about the
SIX benchmarks are provided in appendix B
Baselines. To ensure a thorough comparison, we benchmark against a comprehensive set of baselines in three categories:
Zero-shot Generalist LLMs: Powerful, large-scale models without specific fine-tuning, in- cluding GPT-4o (OpenAl, 2024) OpenAI (3penAT 2025), Gemini-2.. 5irii-22opproo
jet al. 2025), Kimi K2 Team et al. 2025). DeepSeek- R1 (Guo al. 2025), DeepSeek- V3
(DeepSeek-Al AI et al. 2025). Qwen3-32B 32B (Yang et al. 2025a), Qwen3
(wee3-8BB 8B Yang et al. 2025a).
and wwene. 5-72--IInstttee (Qwen al. (022)).
Specialized Fine-tuned LLMs: Models specifically trained on OR solving tasks, including ORLM
(8B) (Huang et al.] 2025), LLMOPT (14B) (Jiang et al. (2025) and OptMATH (32B) (Lu et al. 2025). We try our best to re-evaluate all publicly accessible models. For those not open-soureed, we report their original performance according to corresponding published papers. Note that due to our use of cleaned dataset (Xiao et al. 2025). these original scores serve as a reference and may not be directly comparable.
Multi-step reasoning frameworks that orchestrate L.MMs solve problems, including OptiMUS-VO. (Ahmadi et al.) (024a), Shain-of-Thouge (CoT) (Wei et al. 2022), Chain-of-Exppe -f-perrrsss (CoE) (Xiao al. 2023) and CAFA (Deng al. 2024 To ensure fair and powerful comparison, all agentic methods are implemented using GPT 40 as their backbone model.
Agentic Methods:
Implementation Details We develop our StepORLM models by fine-tuning the publicly available Qwen3-8B (Yang et al. 2025a). Training and inference are conducted on single node with 8 x NVIDIA H100 80 GB GPUs We used synthesized dataset of 50K questions for subsequent training. The model was trained using the AdamW optimizer with learning rate of 7e-6 and weight decay of 0.1. The maximum sequence length was to 8192, and we introduced special tokens <step> and </step> to enhance reasoning. A batch size of 64 was used, and training proceeded for 3 epochs. During the self-evolving sampling phase, we generated trajectories for each response to be used for subsequent preference learning. For W-DPO, we set the hyperparameter 3 to 0.1. To ensure optimal performance and a fair comparison across different frameworks, we selected solvers best suited to each category of model. For general- parrpo--puppose LLMs and all agentic methods, its broad compatibility. For the fine-tuned models, we used their respectively adapted solvers: both our StepORLM and ORLM employed the COPT solver, whereas LLMOPT is evaluated using Pyomo.
used Gurobi due to
4.2 MAIN RESULTS
We report the main results in Table Our StepORLM establishes a new state-of-the-art in OR problem-solving tasks, significantly outperforming both vastly larger generalist/agentic models and specialized LLMs. As an SB-parameter model, StepORLM outperforms much larger, state-of-the-an generalist models, such as DeepSeek-V3 (671B) and Qwen2. (rrenn.55728--10 5-72B-Instruct (72B), and even those agentic methods based on GPT-4o. This result underscores the profound effectiveness of specialized, process-supervised training, enabling compact model to achieve more reliable and accurate OR modeling capabilities than its far larger counterparts.
Moreover, leveraging the co-evolved GenPRM as an inference-time process verifier (i.e., StepORLM+GEnPRM) further pushes the performance to an average Pass@1 accuracy of 85.6%. Such configuration excels especially the most challenging benchmarks, with remarkable accuracy gains of 9.9% on ComplexOR and 9.5% on Industry OR compared with the basic StepORLM. This highlights the dual benefit of co-evolutionary framework: the policy model learns to generate
7

---

Published as a conference paper at ICLR 2026
Table The overall performance of StepORLM and baselines with Pass@ I aceuracy (%) on six OR benchmarks. All agentic method baselines utilize GPT-4o as the base model. StepORLM+GenPRY indicates using the GenPRM as process verifier to enable the inference sealing of StepORLM. Scores cited from original publications are marked with the symbol (*), while missing entries are denoted with (-). Best results are highlighted in bold and the aecond-highest values are underlined.
Model
Params NL4OPT
MAMO
NLP4LP CompOR IndOR ReSocratic Avg.
La ComplexLP
Zero-shot LLMs
OpenAI 03
Closed
78,4
93.9
63.1
93.8
72.2
76.2
84.4
80.3
Semini-2.5-Pre
Closed
82.6
87.9
52.3
94.9
66.7
78.6
89.6
78.9
GPT-4o
Closed
61.2
70.3
57.7
73.6
42.9
38.1
48.4
56.0
Kimi-K2
IT
77.9
93.4
55.9
61.1
59.5
81.9
73.4
DeepSeek-R1
671B
77.5
90.3
59.5
87.6
66.7
59.5
83.9
75.0
DeepSeek-V3
671B
79.8
95.2
53.2
92.1
55.6
66.7
85.1
75.4
Qwen2.5 5-2..---78-mmn Inst
72B
78.9
95.8
44.1
88.2
50.0
57.1
81.1
70.7
Qwen3-32B
32B
77.5
92.3
46.9
93.8
50.0
61.9
85.1
72.5
Qwen3-8B
8B
63.8
73.6
45.0
58.4
27.8
52.4
61.0
54.6
Fine-tuned LLMs
ORLM
8B
73.8
90.4
59.5
76.4
50.0
42.9
61.8
65.0
LLMOPT
14B
75.1
83.5
67.6
86.0
22.2
52.4
73.2
65.7
OMTMA (origin)
32B
95.9*
89.9
54.1*
StepORLM
8B
97.7
97.2
79.3
97.8
55.6
59.5
82.6
81.4
Agentic Methods
OptiMUS-VO.3
Closed
78.0
46.8
88.8
46.8
45.2
67.1
CoT
Closed
62.2
49.5
42.3
74.7
39.2
40.5
43.6
50.3
CoE
Closed
66.7
94.4
50.6
87.4
57.1
31.2
71.2
65.5
CAFA
Closed
68.1
71.2
44.5
50.0
46.4
41.1
40.1
51.6
StepORLM+GenPRM
8B+8B
97.2
97.8
87.4
98.9
61.1
61.9
94.6
85.6
Abbreviations: CompOR: ComplexOR, IndOR: IndustryOR, Avg: Macro-Average, Qwen2.5-72B-Inst Qwenn25-77BBInsttma
NI.4PT
MAMO
NLP4LP
100
142,87u
+3.156
+1.055
90
-5.4%%
+0,55
162.885
+0.0%
12.4%
10.%%
Accutcc (%)
+55.2%
50
30
60
Niri
Wiem- SPT
Brd
Werm SF1
Marl
Warms Up
ComplexOR
IndustryOR
ReSocratic
Ascuracy (%6)
30-
00
59,
3%
10.0%
112.0%
11.22%
0.0%
431
39%
11.0%
40
30
60
20
Qendlst 10
Stame SP
Nay
celec-10
stame NPT
Nart
siane
Figure 4: The analysis on the self-evolvvng process by tracking the performance (Pass accuracy) each training iteration. The relative improvement of current iteration over the previous one is demonstrated on the corresponding bar.
high-quality solutions, while the GenPRM leamss to effectively identify and select the best one. Furthermore, StepORLM excels at avoiding reward hacking, point we illustrate with a case study in Appendix C.T
8

---

Published as a conference paper at ICLR 2026
Table Performance comparison of StepORLM under different inference scaling strategies.
Model
NL4OPT
MAMO
NLP4LP CompOR
IndOR
ReSocratic
Avg.
EasyLP ComplexI.P
StepORLM
Policy Model
StepORLM
97.7
97.2
79.3
97.8
55.6
59.5
82.6
81.4
Major Vote
97.2
97.6
81.1
96.6
61.1
83.5
+Solver Exec
97.7
98.4
81.1
96.1
61.1
66.7
90.3
84.5
+ +Discriminative PRM
97.2
97.2
81.1
97.2
55.6
59.5
87.8
82.2
GenPRRM (initial)
97.8
97.6
82.8
97.2
55.6
58.5
93.1
83.2
+GenPRM (final)
97.2
97.8
87.4
98.9
61.1
61.9
94.6
85.6
ORLM
as Policy Model
ORLM
73.8
90.4
59.5
76.4
50,0
42.9
61.8
65.0
Major Vote
78.7
88.4
50.5
78.7
44.4
47.6
73.0
65.9
Solver Exec
82.2
88.6
63.1
79.8
52.4
78.9
69.9
Discriminative PRM
75.1
91.7
63.1
GenPRRM (initial)
87.3
90.6
55.0
90.4
44.4
47.6
65.5
68.7
GenPRMM (final)
91.5
91.0
64.9
91.0
50.0
57.1
79.4
75.0
4.3
ANALYSIS OF SELF-EVOLVING PROCESS
We conduct in-depth analysis on the self-evolving process by tracking the performance at each training iteration, including the warm-up and co-evolving stages. The results are reported in Figure 4 from which we can obtain the following observations:
The warm-up supervised fine-tuning (SFT) provides a massive foundational performance lift across all benchmarks (e.g., 62.5% accuracy improvement on NLP4LP).
The subsequent self-evolution iterations consistently build upon the prior models, delivering incremental but crucial accuracy improvements. This confirms that the model progressively refines its capabilities through self-evolving loop, rather than stagnating after initial fine-tuning.
The performance trend is notably non-monotonic on the IndustryOR dataset, where the perfor- mance first declines and finally largely increases the third iteration. We attribute this to the small size of testing set and the challenging questions in IndustryOR. Our case-level inspection reveals a potential progression. Early iterations primarily rectify structural modeling errors flagged by the PRM, whereas later iterations concentrate on code-level issues (e.g., mndex-out-of-bounee errors), which finally leads to accuracy improvements.
4.4 INFERENCE SCALING WITH GENPRM
In this section, investigate the generalization capability of our co-evolved GenPRM process verifier for inference scaling. To evaluate this, we employ Best-of-k strategy with a fixed sampling budget of 4. Specifically, sequentially sample candidate trajectories and select the first one that GenPRM verifies as process-correct; if no trajectory passes verification within the budget, the final sample selected. We apply this method our final StepORLM and another open-sourced ORLM bascline, comparing its effectiveness against other inference scaling strategies including majority voting, solver execution, and discriminative PRM.
As shown in Table 2 our self-evolved GenPRM is generally the most effective inference scaling strategy, achieving the highest accuracy on average for both StepORLM and ORLM. When paired with ORLM, our GenPRM boosts its average performance by 10%, demonstrating its powerful generalization. learns fundamental, model-agnostic principles of valid OR reasoning, allowing it to function as universal verifier that significantly improves the performance of other models, not just its co-evolved policy.
4.5 ABLATION STUDY
To study the impact of each component in proposed self-evolving framework, conduct experiments based on the following variants: (1) w/o SFT: skipping the warm-up supervised fine- tnnng stage; (2) w/o self-evolution: directly sampling single reasoning trajectory after SFT and

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study of StepORLM on Pass accuracy (%). The best results are given bold, and the second-best values are underlined.
Model
NL4OPT
MAMO
NLP4LP CompOR
IndOR
ReSocratic Avg.
EasyLP
ComplexLP
StepORLM
97.7
97.2
79.3
97.8
55.6
59.5
82.6
81.4
w/o SFT
71.4
88.3
55.9
72.5
38.9
50.0
75.2
64.6
w/o Self-evolution
96.7
97.2
67.6
94.9
44.4
52.4
81.4
76.4
w/o GenPRM Evolution
94.4
96.3
68.5
97.8
50.0
54.8
82.4
77.7
w/o W-DPO
97.2
95.8
73.9
96.1
50.0
52.4
80.6
78.0
applying DPO without iterative refinement; (3) w/o GenPRM Evolution: using the initial GenPRM with frozen parameters for all subsequent iterations; (4) w/o W-DPO: replacing the W-DPO with standard DPO during the self-evolving training phase. The results are shown in Table
Removing the warm-up SFT (w/o SFT) causes the largest performance drop, confirming the necessity of a strong initial policy. Disabling the iterative loop (w/o Self-evolution) prevents the model from refining its initial capabilities, leading to lower scores challenging benchmarks like ComplexOR. Freezing the GenPRM after initialization (w/o GenPRM Evolution) also degrades performance, validating the importance of the co-evolutionary dynamic between the policy and the critie. Finally, replacing our W-DPO objective with standard DPO (w/o W-DPO) results a noticeable performance decrease, indicating that our trajectory-level quality weighting provides more effective learning signal between different trajectory pairs. Together, these results demonstrate that the synergy among all components drives the success of the StepORLM framework.
To disentangle data quality from data scale, we additionally run controlled SFT study on LLaMA-3-BR 3-----------ttu where both our warm-up corpus and ORLM's In-Irstr dataset are sub- sampled to 3K instances. Under identical training settings, the model trained OR-Instruct reaches an average Pass@1 of 60.9%, whereas training on our 3K warm-up subset yields 66.8% (+5.9 points), indicating that the gains from Stage mainly stem from the quality of our curated data rather than simply using more examples (see Appendix D.I for the full breakdown).
We also instantiate the full StepORLM pipeline on by ORLM. Starting from a weak base model with only 16.1% average Pass 1, warm-up SFT on our 50K corpus lifts performance to 69.4%, three self-evolving iterations further improve it 82.2%. and GenPRM- based inference scaling pushes to 85.0%. On this shared backbone, StepORLM therefore surpasses the released ORLM checkpoint (65.0% average Pass @ 1) by than 17 points, demonstrating that our co-evolving framework is effective and compatible across different policy backbones (see Appendix D.2 for the per-benchmark breakdown).
1AAA--3-8-Innttrrn the same backbone used
CONCLUSION
We design StepORLM, a novel self-evolving framework to overcome critical limitations in training large language models for operations research. Our approach addresses the credit assignment problem of outcome-based rewards and the myopia of conventional process supervision through co-evolutionary loop between policy model and generative process reward model (GenPRM). StepORLM integrates definitive solver-based outcome verification with holistic, generative process feedback. Such a dual-feedback mechanism ensures the process-sound and outcome-correct reasoning capabilities of the policy model to solve complex OR problems. Our 8B- 3B-parameter StepORLM achieves the SOTA performance across six benchmarks, significantly outperforming much larger generalist models, agentic systems, and specialized baselines. Furthermore, the co-evolved GenPRM is able to act as a powerful, universal inference-time process verifier, substantially enhancing the performance of our model and other LLMss for OR problem-solving tasks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGGENTTS
Lin's research is supported by the National Natural Science Foundation China (NSFC) [Grant NSFC-624B2096, 72542012, 72595872]. Ge's research is partially supported by the National Natural Science Foundation of China (NSFC) [Grant NSFCC72225009, 12399466 72394360, 72394365).
REFERENCES
Ali Ahmadi Teshnizi, Wenzhi Gao, Herman Brunborg, Shayan Talaei, and Madeleine Udell. Optimus- 0.3: Using large language models model and solve optimization problems scale, 2024a. URL
https://arxiv org.org///a....o http//b//200 240. 2t 19633
Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024b.
Nicolás Astorga, Tennison Liu, Yuanzhang Xiao, and Mihaela van der Schaar. Autoformulation of Mathematical Optimization Models Using L.LMs. arXiv preprint orXiv:241..0667) 2024.
Peter Chen, Xiaopeng Li, Ziniu Li, Xi ChenD, and Tianyi Lin. Stepwise guided policy optimization: Coloring your incorrect reasoning in grpo. arXiv preprini arXiv, 2505, 2025a.
Yitian Chen, Jingfan Xia, Siyu Shao, Dongdong Ge, and Yinyu Ye. Solver-informed rl: Grounding large language models for authentic optimization modeling, 2025b. URL https://////iivv.oo
abs/2505. 11792
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626,, 2025.
Ning Dai, Zheng Wu, Renjie Zheng. Ziyun Wei, Wenlei Shi, Xing Jin, Guanlin Liu, Chen Dun, Liang Huang, and Lin Yan. Process wwwpeervvviiinguuuddee policy optimization for code generation, 2025.
URL https:
17621
DeepSeek-A1, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjic Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song. Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun. W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun. Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song. Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin,
Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma. Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren. Zhangli Sha. Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu
11

---

Published as a conference paper at ICLR 2026
Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Ziyi Gao, and Zizheng Pan. Deepseek-v3 v3 technical report, 2025. URL httths:s//aaattae http////www...r//////./111...../ 19437
Haoxuan Deng, Bohao Zheng, Yirui Jiang, and Trung Hieu Tran. Cafa: Coding auto-formulation can boost large language models in solving linear programming problem. In The 4th Workshop on Mathematical Reasoning and Al at NeurlPS 24, 2024.
Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, et al. A comprehensive survey of self-evolving ai agents: A new paradigm bridging foundation models and lifelong agentic systems. arXiv preprint
000:22508.07007,, 2025.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Iims can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.0453 045500 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Chenyu Huang, Zhengyang Tang, Shixi Hu, Ruoging Jiang, Xin Zheng, Dongdong Ge, Benyou Wang, and Zizhuo Wang. Orlm: A customizable framework in training large models for automated optimization modeling. Operations Research, 73(6):2986-309, 2025. doi: 10.124./0pr..2224. 1233.
Xuhan Huang, Qingning Shen, Yan Hu, Anningzhe Gao, and Benyou Wang. MAMO: A mathematical modeling benchmark with solvers. arXiv preprint, 2024.
Caigao Jiang, Xiang Shu, Hong Qian, Xingyu Lu, Jun Zhou, Aimin Zhou, and Yang Yu. LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch. In International Conference Learning Representations (ICLR), 2025.
Vinicius Lima, Dzung T. Phan, Jayant Kalagnanam, Dhaval Patel, and Nianjun Zhou. Toward a trustworthy optimization modeling agent via verifiable synthetic data generation, 2025. URL
03117
Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Models. In International Conference Machine Learning (ICML), 2024.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-ting timee scaling for generalist reward modeling. arXiv preprint 2rX4v:2504.02495, 2025.
Hongliang Lu, Zhonglin Xie, Yaoyu Wu, Can Ren, Yuxuan Chen, and Zaiwen Wen. Optmath: A scalable bidirectional data synthesis framework for optimization modeling, 2025. URL https:
OpenAl. Gpt-4o system card.
2024.
OpenAl. Openai o3 and o4-mini system card. Technical report, OpenAI, 2025. URL https
/ooenaii
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL 2412
Rindranirina Ramamonjison, Haley Li, Timothy T Yu, Shiqi He, Vishnu Rengan, Amin Banitalebi- Dehkordi, Zirui Zhou, and Yong Zhang. Augmenting operations research with formulationn of optimization models from problem descriptions. arXiv preprint arXiv:2209. 15565, 2022.
12

---

Published as a conference paper at ICLR 2026
Rindranirina Ramamonjison, Timothy T. Yu, Raymond Li, Haley Li, Giuseppe Carenini, Bissan Ghaddar, Shiqi He, Mahdi Mostajabdaveh, Amin Banitalebi Dehkkrdd, Zirui Zhou, and Yong Zhang. NL40pt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions. arXiv preprint arXiv:2303.0823: 2023.
Shuaijie She, Junxiao Liu, Yifeng Liu, Jiajun Chen, Xin Huang, and Shujian Huang. R-prm: Reasoning-driven process reward modeling. arXiv preprint arXiv:2503.21225, 2025.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen. Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534. 2025.
Teng Wang, Wing- Yin Yu, Zhenqi He, Zehua Liu, Xiongwei Han, Hailei Gong, Han Wu, Wei Shi, Ruifeng She, Fangzhou Zhu, and Tao Zhong. BPP-Search: Enhancing Treu-of-Thought Reasoning for Mathematical Modeling Problem Solving. arXiv preprint arXiv:2411.17400 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pp. Associates, Inc., 2022.
Curran
Yang Wu, Yifan Zhang, Yurong Wu, Yuran Wang, Junkai Zhang, and Jian Cheng. Step-opt: Boosting optimization modeling in Ilms through iterative data synthesis and structured validation, 2025. URL https: .rw.lvo.rr///eeoo wow. arr//bb//200... 17637
Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. A survey of llm-based deep search agents: Paradigm, optimization, evaluation, and challenges. arXiv preprint arXiv:2508. 05668, 2025.
Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, et al. -hain-of-experts: When Ilms meet complex operations research problems. In The Twelfth International Conference on Learning Representations, 2023.
Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan J. Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng. Mingli Song, and Gang Chen. Chain-of-Experts When LLMs Meet Complex Operations Research Problems. In International Conference on Learning Representations (ICLR), 2024.
Ziyang Xiao, Jingrong Xie, Lilin Xu, Shisi Guan, Jingyan Zhu, Xiongwei Han, Xiaojin Fu, Wing Yu, Han Wu, Wei Shi, et al. A survey of optimization modeling meets llms: Progress and future directions. arXiv preprint arXiv:2508. 10047, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan I. Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL https://aaxxi orggabs/////oooooooo..... abs/2505 09388
Yingxuan Yang, Huacan Chai, Yuanyi Song. Siyuan Qi, Muning Wen, Ning Li, Junwei Liao, Haoyi Hu, Jianghao Lin, Gaowei Chang, et al. A survey of ai agent protocols. arXiv preprint arXiv:2504. 16736, 2025b.
Zhicheng Yang, Yiwei Wang, Yinya Huang, Zhijiang Guo, Wei Shi, Xiongwei Han, Liang Feng, Linqi Song, Xiaodan Liang, and Jing Tang. Optibench meets resocratic: Mcasure and improve Ilms for optimization modeling, 2024. URL httLitttss://rxi.orrrg///// wttp.//////bb/////////////////// .088877
Zhicheng Yang, Yiwei Wang, Yinya Huang, Zhijiang Guo, Wei Shi, Xiongwei Han, Liang Feng, Linqi Song, Xiaodan Liang, and Jing Tang. Optibench meets resocratic: Measure and improve lims for optimization modeling, 2025c. httpss/sss:://xxvv...m org/
http://aaxxiv..oo//a/////////// abs//40/ 09887
13

---

Published as a conference paper at ICLR 2026
Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. ReEvo: Large Language Models as Heuristics with Reflective Evolution. In Advances Neural Information Processing Systems (NeurIPS), 2024.
Weinan Zhang, Junwei Liao, Ning Li, Kounianhua Du, and Jianghao Lin. Agentic information retrieval. arXiv preprint urXiv:2410.09713, 2024.
Jian Zhao, Runze Liu, Kaiyan Zhang, Zhimu Zhou, Junqi Gao, Dong Li, Jiafei Lyu, Zhouyi Qian, Biqing Qi, Xiu Li, al. Genprm: Scaling test-time compute of process reward models via generative reasoning. arXiv preprint arXiv:2504.0000 2504000944 2025.
Congming Zheng. Jiachen Zhu, Zhuoying Ou, Yuxiang Chen, Kangning Zhang, Rong Shan, Zeyu Zheng. Mengyue Yang, Jianghao Lin, Yong Yu, and Weinan Zhang. A survey of process reward models: From outcome signals to process supervisions for large language models, 2025a. URL
https://arxiv.org//bs///5 08049
Zhi Zheng, Zhuoliang Xie, Zhenkun Wang, and Bryan Hooi. Monte Carlo Tree Search for Compre- hensive Exploration in LLM-Based Automatic Heuristic Design. arXiv preprint arXiv:2501.088 008603, 2025b.
Chenyu Zhou, Jingyuan Yang. Linwei Xin, Yitian Chen, Ziyan He, and Dongdong Ge. Auto- formulating dynamic programming problems with large language models, 2025, URL https //arxiv. segxiv.orr/abs/2507.173
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE DECLARATION
In the preparation of this manuscript, a Large Language Model (LLM) was utilized exclusively for the purpose language editing and refinement. The LLM's role was limited improving grammar, syntax, and clarity. All scientific content, including the conceptual framework, data analysis, and interpretation of results, is the original work of the authors.
B BENCHMARKS
mass of datasets and methodologies for dataset construction have been proposed improve the capacity of LLM to address OR problems, with these datasets becoming increasingly intricate and et al., 2024b Yang al., 2024). A recent survey by (Xiao et al. 2025) provides a comprehensive and timely review of benchmark datasets for optimization problems described in natural language. The study identified prevalent issues within the datasets, including logical fallacies, ambiguous problem definitions and incorrect ground-truth answers, and also corrected a portion of the errors identified. Based on that, our work builds upon their initial efforts by conducting further manual verification and cleaning of these widely-used benchmarks. Subsequently, the data is processed into a unified format, thereby establishing a more robust and standardized foundation for subsequent research and model assessment.
realistic( Yang et al. (AhmaddA
B.1
NL40PT
The NL4Opt dataset was introduced by Ramamonjison et al. as part of the NeurIPS 2022 NL40pt competition (Ramamonjison et al. 2022 2023). It consists of approximately 1,100 annotated word problems describing linear programming (LP) scenarios, with the primary purpose of bridging natural language descriptions and formal optimization models. In survey, Xiao et al. reported that NL40pt has a complexity score of 5.59 and an original error rate of at least 26.4%. Each problem presents a self-contained, real-world scenario (e.g., resource allocation, scheduling) with annotations identifying key optimization elements. The official shared task for the dataset is divided into two subtasks: (1) recognizing and labeling problem entities and (2) generating a corresponding mathematical model. The original dataset provides 713 training. 99 validation, and 289 test instances. For our evaluation, we selected 213 validated problems from this benchmark.
B.2 MAMO
The MAMO dataset was introduced by Huang et al. (2024) as a benchmark for evaluating large language models complex mathematical modeling tasks, with specific focus on the formulation process itself rather than just solution accuracy (Huang et al. (024).. The dataset concentrates on linear programming (LP) and mixed-integer linear programming (MILP) problems, excluding nonlinear or differential equation modeling. MAMO is divided into two main components; Easy LP (652 instances) and Complex LP (211 instances). According to an analysis by Xiao al.) EasyLP has a complexity of 7.12 with an error rate of at least 8. 8,13,, while ComplexLP is significantly more difficult, with a complexity of 13.35 and an error rate at least 23.7%. By delegating the solving step to an OR solver, MAMO focuses specifically on whether an LLM can accurately translate detailed natural language description into a correct mathematical formulation. From this benchmark, selected 545 easy and 111 complex instances for study.
B.3 NLP4LP
The NLP4LP benchmark, introduced by AhmadiTeshnizi et al. (2023) in their work on the OptiMUS system (Ahmadi et al., 2024b a). evaluates LLM-based agents on translating natural language operations research problems into solver-ready code and mathematical models. The survey by Xiao et al. assigned it complexity score of 5.58 and identified an error rate of at least 21.7% in the original dataset. It consists of 269 human auman-authored LP and MILP problems, where each instance presents an optimization scenario from classical OR domains such as scheduling, knapsack allocation, and production planning. The benchmark provides a fine-grained testbed for assessing formulation accuracy. In our work, we chose to evaluate 178 verified instances from M.P4I.P
15

---

Published as a conference paper at ICLR 2026
B.4 COMPLEXOR
The ComplexOR dataset was introduced by Xiao et al. as part of their Chain-of-Experts framework stress-test the reasoning and modeling capabilities of LLMs on scenarios those earlier datasets (Xiao et al. 2024). The analysis by Xiao et al. noted a complexity score of 5.98 and a significant eπrог rate least 24.3%. ComplexOR was curated from advanced OR case studies and classic difficullt optimization problem classes, such as multi-step lot-sizing problems with setup costs, intricate scheduling tasks, and supply chain design problems. The problems are primarily large-scale or conceptually complex mixed-integer linear programs that require multi-step reasoning and often involve hierarchical or time-indexed structures. The dataset's ss intended purpose to facilitate the development and evaluation of advanced reasoning strategies for OR problem solving. Following our verification process, we selected 18 reliable problems from this dataset for evaluation.
challenging than
B.5 INDUSTRYOR
The IndustryOR dataset, introduced by Huang al. (2024), is the first industrial operations research benchmark for large language models (Huang et al. 2025). comprises 100 real-world optimization scenarios from sectors like manufacturing. logistics, finance, and energy. The study by Xiao al. highlighted this dataset's high difficulty, reporting a complexity score of 14.06 and substantial error rate of at least 54.0%. Designed to evaluate LLMs on practical, domain-specific tasks, IndustryOR covers five OR categories- linear, integer, mixed mixed-integer. nonlinear, and others- and is labeled by difficulty (Easy, Medium, Hard). For our experiments, we selected 42 validated problems from this benchmark.
B.6 RESOCRATIC
The ReSocratic dataset introduced by (Yang al. 2025c), the name which actually data synthesis method proposed by this work as well. This method synthesize the formatted optimization demonstration in a reverse manner first, and then intermediate reasoning steps, ReSocratic has higher quality than previous methods. In this paper, we selected 403 validated problems from the original ReSocratic- dataset.
into a question. Through these
C CASE STUDIES
C. VISUALIZING THE ALIGNMENT PROBLEM TSP
The following presentation will outline a comprehensive prototype of the failure case B, as referenced Figure [ The sequence of events the present case study is as follows: firstly, sketch map of the problem presented; secondly, detailed problem description is given; and thirdly, the responses given by Qwen3 and our model StepORLM are outlined. As previously stated, the ensuing discourse will focus the results obtained from the two models. The graph illustrates the problem and demonstrates that an eтror in the constraints step will not affect the optimal value.
A tour guide is planning a bus tour across five cities. They must start from one city, visit each city exactly once, and return the starting city.
Objective: The total cost
->> Correct solution
-> : Wrong salution without constraint for subtour eliminating
E
D
But, total cost of the two solutions are the same!
Figure 5: A simplified illustration of problem 74 in ComplexLP
This is problem with the identifier 74 from the ComplexI. dataset:
16

---

Published as a conference paper at ICLR 2026
Problem Description
Consider scenario where a tour guide is planning bus tour across five cities, named F, G, H, and I. The tour must start and end in the same city, and each city should be visited exactly once. The objective is to minimize the total cost of the tour, which could be influenced by factors such as distance, tolls, and fuel expenses.
Here the travel costs between the cities:
From City E, it costs 37 units travel to 72 units G, 66 units to H, and 33 units to Travelling from City F, the costs are 37 units to E, 26 units to G. 56 units to H, and a whopping 100 units I. If the journey starts from City it takes 72 units to reach E, 26 units to F, 49 units to H, and 32 units to I. From City H. it costs 66 units to get to E, 56 units F, 49 units to G, and mere 13 units I. Lastly, from City I, the travel costs are 33 units to E, 100 units to F, 32 units to G, and 13 units to H.
Keeping these costs in mind, what is the minimum total travel cost for the tour guide to take the each city exactly once and return to the starting city?
And the following is our generate prompt for both Qwen3 and StepORLM.
Generate Prompt
You helpful assistant with expertise in mathematical modeling. Python code and the COPT solver. When the User provides an optimization question, you will analyze it, build detailed mathematical model, and provide the COPT code to solve it.
Your response should follow these steps:
1. Problem Description
Sets and Parameters
3. Decision Variables Objective Function
5. Constraints
Mathematical Model
7. Nonlinear Relationships
Final Model
9. Python Code Using coptpy
please enclose the content within <step>...</ste
Below an optimization modeling question. Build mathematical model and corresponding python code using coptpy that appropriately addresses the question:
PROBLEM DESCRIPTION HERE
Keeping these costs in mind, what is the minimum total travel cost for the tour guide to take the bus to each city exactly once and return to the starting city?
*
Make sure to import necessary packages, such as 'import coptpy as cp" and 'from coptpy import COPT*.
# When you create a model make sure to use 'env "env= EnvEn) and 'model = env.createMModel COPT."
* When you add variable,
Do not name variables and constraints
* Use "ddCConstr' or ".addConstrs' add constraints. If you want set 'lb' or 'ub' as infin- ity, please use 'I=CCOT.INFIITT" or 'ub-COPTINFINITY' instead of INF.NIYY. 8 When you set objective, you should use the 'modellsetOhjeectvee method and use COPT or COPTMAXIMIZE:
17

---

Published as a conference paper at ICLR 2026
* Do 'model.optimizee)..
Make sure to use 'ooeelsolveO"" to solve the question.
* The code output statement is:
if model.status = COPTOРTIMALL solution var. var.getName() var.X for var in model.getVars() print(' Just print the best obj:', Oboee.Objjii Val) else: print( No Solution')
Think step by step.
C.1.1
STEPORLM's RESPONSE
Inference
step >
Problem Description:
The tour guide needs to plan a bus tour that visits each of the five cities exactly once and returns the starting city, minimizing the total travel cost. This problem a classic example of the Traveling Salesman Problem (TSP), which is a well-known combinatorial optimization problem.
< </slep>
< step >
Sets and Parameters:
Cities: E, F. G. H
Cost Matrix: A matrix where each element Cı represents the cost of traveling from city to city j.
sstepp
< step >
Decision Variables:
Ziy: A binary variable that is if the tour includes direct path from city i to city j. and otherwise.
< </step>
step >
Objective Function:
Minimize the total travel cost:
Minimize
Σ
Σ
Ciy Fij
< </step < step >
Constraints:
Each city is visited exactly once:
Σ Dij 1 Vi
Each city left exactly once:
Σ Fiz =1 Vj
3. Subtour elimination constraints:
"j + Tij S n 1
Vi,j > 1, ビ
where is an auxiliary variable representing the position of city in the tour, and is the number of cities.
<ssepp step >
18

---

Published as a conference paper at ICLR 2026
Mathematical Model:
min
Cjj Jij
Σ Tij =1 Vi
s.t.
Σ Tij =1 Vj
"; + n Taj S n-1 1
Tij E {0,1} Vi,j
< <stepp>
< step
Nonlinear Relationships:
No nonlinear relationships present in this model. This step is unnecessary. < /step
< step >
Final Model:
The final linear programming model is defined by the objective function and constraints above.
< </step>
Python Code Using COPT
import coptpy as cp from coptpy import COPT
4 Define the cost matrix 5 ccostmmatri =
[0, 37, 72, 66, 33],
[37, , 0, 26, 56, 108], [72, 26, 0, 49, 32],
[66, 56, 49, 0, 13], 10 [33, 100, 32, 13, 0] ]
Number cities 18 n (en(cos_mmatrix
15
# Create COPT environment and model
Envr
model env. env. enw.creeteeodell(Bur ("BusllllBBBusuuuua ("Poooopoiooooiiiiiaaaaiatoo"
19
# Create binary decision variables (}
for i range
23 for j in range
if i 1= j:
x[i, jl = model. addVar (vtype=COPT. name=f"x_(i)_(0))
4 Create auxiliary variables 28 u (
for i in range n): u[i] model. addVar ad (vtype=COPT. CONTINUOUS, name=f"u_(i""
Set objective function
obj = cp. quicksumm x[i, j]
cost_matrix(i][j]
for i in range (n) for j range (n)
19

---

Published as a conference paper at ICLR 2026
if ]
)
model... setobjective sense=CoPT MINIMIZE)
# Add constraints: each city visited once
for i range (n)
43 model. addConstr (ep. quicksum (xli, jl for j in range(n) if i != j) =
1)
Add constraints: each city left once
for j range (n)
47 model. addConstr (cp. quw.kuuckkummmmm (x[i, jl for in range(n) if i != j) =
1)
Add subtour elimination constraints 50 for i in range n): for j in (1,ge(1, n): if i != j:
model. addConstr (u[i] u[j] x[i, 1)
55 # Solve the model model. solee.
C.1.2 QWEN3'S RESPONSE
Python Code Using COPT
1 import coptpy as cp
2 from coptpy import COPT
env En.r
5 model env. createModel ()
6
# Define cost matrix
cost =
[0, 37, 72, 66, 33],
10 [37. 0, 26, 56, 100].
[72, 26, 0, 49, 32],
[66, 56, 49, 0, 13], [33, 100, 32, 13, 0] ]
15
# Create variables [[None for in range for in range (5)
for i range (5):
19 for j in range
20 if i j:
x[i][i] = adeeladdVaas (vtype=COPT.C) BINARY)
Objective function 24 obj_expr = for i in range for j in range (5) if i !=
obj_ cost[i][i]] x[i][i] m wwwbbjtetttieeebbeeee.ooo.
model.
COPT. MINIMIZE)
# Constraints
32 for i range (5)
20

---

Published as a conference paper at ICLR 2026
model addConstr (ssmmssssxxxx))) (x[i][j] for in range (5) if ] 1) 1) for i range (5)
35 model... addConstr (sum (x [[]][i] for j in range (5) if j != 1) = 1)
# Solve
model... solve ()
C.1.3 DISCUSSIONS ON TWO MODELS' RESULTS
This example clearly shows why rewards should be based on step-by-step reasoning rather than just the optimal value. During the reasoning process, StepORLM correctly recognises the importance of subtour elimination constraints, whereas Qwen3 initially identifies them but then overlooks them due circular self-debate. In this case, the presence absence of these constraints does affect the final result. Therefore, both codes can obtain the correct optimal solution.
C.2 GENPRM VS. DISCRIMINATIVE VERIFICATION IN TSP
In this section, we provide concrete case study on the Traveling Salesman Problem (TSP) illustrate how GenPRM avoids the "reward hacking" (or myopia) issues common in standard discriminative PRMs.
C.2.1 THE CHALLENGE: STRATEGIC CONSISTENCY vs. LOCAL CORRECTNESS
In optimization modeling, locally valid mathematical decision sometimes lead to implementation failure steps later. A classic example is the choice of subtour elimination formulation in TSP:
DFJ Formulation: Uses only binary variables zij but requires exponential subset constraints (E zzj S ||| - This is mathematically elegant but impossible to implement in a static script without complex callbacks
generation.
MTZ Formulation: Uses binary variables Tıj plus auxiliary continuous variables u, and polynomial constraints. This is less intuitive but easily implementable in standard solvers like COPT or Gurobi.
A myopic discriminative PRM evaluating steps in isolation tends to reward the definition of only variables (Step 3) as CORRECT, because it is mathematically valid. However, this leads to a dead end the coding step (Step 9).
In contrast, as shown in the comparison below, GenPRM evaluates the entire trajectory. detects that the code failure in Step 9 stems from the insufficient variable definition in Step Consequently, it retroactively marks Step 3 as INCORRECT, forcing the policy to adopt the robust MTZ formulation.
C.2.2 COMPARATIVE TRAJECTORIES
Below we display the process supervision results for two attempts. Note how GenPRM identifies the "strategic error" in Attempt 1.
Attempt 1: The Myopic Trap (DFJ Formulation)
The model chooses mathematically valid but practically unimplementable strategy.
Student Trajectory (Excerpt):
Step 3 [Decision Variables]:
Binary variable, equals if the path from city i to city j is included.
(Note: Missing auxiliary variables u.)
Step 5 [Constraints]:
1. Assignment constraints..
2. Subtour elimination (DFJ): Enyes Fig S 1
AA C N
21

---

Published as a conference paper at ICLR 2026
Step 9 [Python Code]:
Note: Since listing all subsets is computationally expensive... # I will omit the specific subset constraints for this simple pass. # This might result in subtours in the solution. model. model.solved solve()
GenPRM Verification Output:
STEP 1 CORRECT STEP_2: CORRECT
STEP 3: INCORRECT
EXPLANATION.3: While defining only binary variables Hş is theoretically sufficient for the DFJ formulation, it is strategically incorrect in this context. revealed Step 9, model fails to implement necessary subset constraints because it lacks row-generation mechanism. To ensure a solvable and implementbble static model, the decision variables should have included auxiliary continuous variables for the MTZ formulation. STEP 4: INCORRECT (Cascading error)
STEP_5: INCORRECT (Relies incomplete variable set)
STEP_9: INCORRECT
Attempt 2: The Globally Consistent Solution (MTZ Formulation)
Guided by GenPRM, the model adopts strategy that ensures code executability.
Student Trajectory (Excerpt):
Step 3 [Decision Variables]:
Zig: Binary variable...
24 Continuous variable, representing the visit order city
Step [Constraints]:
2. Subtour elimination (MTZ formulation):
Uy + | -ag & [|]]
Step 9 [Python Code]:
for in range(1, n):
for in range(1, n):
if i != j: model. (u[i] u[j] n x[i, j] <= n model.solved
GenPRM Verification Output:
STEP 1: CORRECT
STEP 2: CORRECT
STEP.3 CORRECT
EXPLANATION_3: The decision variables are correctly defined. Including the auxiliary variables us along with the binary variables æú is the correct strategy for implementing the MTZ subtour elimination formulation, which allows for direct static implementation in the code.
STEP CORRECT
EXPLANATION 9: The Python code perfectly implements the MTZ formulation derived in the previous steps.
22

---

Published as a conference paper at ICLR 2026
This TSP case study concretely illustrates the "myopic PRM" failure mode discussed in the main text: a purely step-wise discriminative verifier can reward locally valid but globally unusable modeling choices (DFJ-style subtour climination), whereas solver-aware GenPRM jointly reasoning over the whole trajectory steers the policy toward the implementable MTZ. formulation. In our full experiments, combining GenPRM with strict solver feedback in the proposed dual-feedback framework prevents such reward hacking from becoming systematic or degrading overall performance.
D ADDITIONAL ABLATION STUDIES
D.1 IMPACT OF DATA QUALITY (CONTROLLED SFT)
To disentangle the impact of data quality from data scale, compare warm-up SFT data against the OR-Instruct dataset (Huang et al. 2025). We sample 3K instances from our warm-up data to match the size of the released OR-Instruct set and train LLaMA-3-88 --B--rrrtt backbone under identical settings. As shown in Table our data yields a +5.9% average improvement, highlighting the superior quality of our synthesized corpus.
Table Controlled SFT performance comparison (3K training samples) on LLaMA-3-8B-Instte
Model
N1.40pt
EasyLPP
Compl.P
NLP4LP
CompOR
IndOR
ReSoe
Avg
OR Instrust (3K)
80.8
85.5
50.5
80.9
27.8
31.0
69.7
60.9
Warm-up (3K)
89.2
90.0
55.0
87.1
33.3
40.5
72.2
66.8
D.2 STEPORLM PIPELINE ON LLAMA-3-8B BACKBONE
To demonstrate the backbone compatibility of our framework, we instantiate the full StepORLM pipeline LLaMA-3-8B- Table details the performance evolution across the warm-up, self-evolution, and inference scaling stages. The consistent gains confirm that our framework is robust across different model architectures.
Table 5: Performance evolution of StepORLM using LLaMA-3-8B-Instnun the backbone.
Training Phase
NL4Opt
EasyLP
CompLP
NLP4LP
CompOR
IndOR
ReSoc
Avg
Base (MA--88)88
19.2
18.9
14.4
18.5
11.1
7.1
23.8
16.1
+
Warm-up SFT
88.3
86.2
69.4
87.6
27.8
47.6
79.2
69.4
Sell-evol. (StepORLM)
97.2
97.8
84.7
93.8
55.6
61.9
84.6
82.2
+ Inference Scaling
97.7
98.0
87.4
96.6
61.1
66.7
87.3
85.0
E STEP-LEVEL REASONING TRAJECTORY TEMPLATE
Each reasoning step is enclosed within <step>...<<step) </step> tags. While this structure is fixed during the initial warm-up stage, it becomes flexible in subsequent self-evolving iterations. This is because we do not constrain the model's output format or the number steps during this phase, allowing the I.LM to learn complex and implicit reasoning patterns
23

---

Published as a conference paper at ICLR 2026
1. Problem Deseription:
Clearly identify the problem, restating unclear expressions and defining the optimization objective.
2. Sets and Parameters: Define all relevant sets (c.g., products, time periods, loca- tions) and parameters (e.g., costs, capacities, demands).
3. Decision Variables: Introduce appropriate decision variables, specifying their domains and interpretations.
4. Objective Function:
Formulate the mathematical expression for the objective be minimized or maximized.
5. Constraints: Develop all necessary constraints, explaining the reasoning behind each.
6. Mathematical Model Summary: Present concise mathematical formulation of the complete model.
7. Nonlinear Relationships applicable): Handle any nonlinear relationships through transformation or approximation techniques.
8. Final Model and Implementation Considerations: Articulate the detailed model with any implementation-speciff adaptations.
F TRAINING DETAILS AND EFFICIENCY
F.1 GENPRM TRAINING DATA FORMAT
The GenPRM is trained via Supervised Fine-Tuning (SFT) using standard (prompt, completion) pairs, Each training example consists of:
1. A natural language OR problem instance.
2. A full step-by-step modeling trajectory generated by the policy model (typically containing
9 steps: problem description, sets, variables, objective, constraints, full model, and code).
3. Task instructions requiring GenPRM judge each step and provide concise explanation.
Prompt Format.
The input prompt aggregates the problem context and the student's 's trajectory into single long-context message. The exact template used in experiments is provided below:
Prompt
You are a committee of world-class Operations Research (OR) experts. You will review student's 's step-by-step modeling process and provide, for each step, (1) correctness verdict and (2) a short explanation.
You must follow this output format exactly:
For each step i = 1, .... 9, output:
STEP (CORRECT/NCOORECT]
EXPLANATION.I: [1-3 sentences explaining your decision]
# CONTEXT
## Problem Description
<problem>
[Full natural-language description of the OR instance]
(e.g., production planning, knapsack, scheduling, network design, ...)
</problem>>
## Student's 's step-by-step trajectory
<step> 1: [Problem restatement and informal formulation] </step>
<step>>STEP.1. [Sets and parameters] </step>
<teep>STEP.33 [Decision variables] </step>
<step>>STEP.9 [Final mathematical model solver code] </step>>
24

---

Published as a conference paper at ICLR 2026
# TASK
Now, based on the above problem and trajectory, provide:
For each step I....., verdict "CORRECT" "INCORRECT";
For each step, short natural-languas explanation.
Remember follow the exact output format specified above.
Illustrative Example. To make the above schema more concrete, we provide a shortened example of GenPRM training instance for simple knapsack problem.
Example (prompt excerpt)
System Instruction:
You are a committee of world-class Operations Research (OR) experts. You will review student's step-by-step modeling process and provide, for cach step, (1) a correctness verdict and (2) short explanation. You must follow this output format exactly: for each step i, output STEP_i: CORRECT/INCORRECT and EXPLANATION.i: [1--3 sentences].
## Problem Description
<problem>>
factory produces two products, A and B. Each unit of A consumes 2 hours of machine time and 1 unit of material, while each unit of B consumes I hour of machine time and 2 units of material. The factory has most hours of machine time and 8 units of material per day. Profit is $3 for each unit of and $4 for each unit of Formulate linear program that maximizes daily profit. </problem3>
## Student's seep-by-step trajectory <step>
STEP_I: 11: Restate the problem and clarify that the goal is to maximize total daily profit subject to resource constraints machine time and material.
</step>>
<step>>
STEP 2: Sets and parameters: let the planning horizon be one day; define available machine time H = 8 hours and material M = units.
</step>
<step>
STEP_3: Decision variables: æ4 units of product to produce; =e units of product produce.
</step>>
[... remaining steps omi tted for brevity ...]
Example (completion excerpt)
STEP.1: CORRECT
EXPLANATION_1: The student correctly restates the objective (maximizing daily profit) and notes that the problem is constrained by machine time and material, which aligns with the original description.
CORRECT
EXPLANATION 2: The parameters and are properly defined and match the available resources in the problem statement.
STEP 3: CORRECT
EXPLANATION_3: The decision variables æA and an clearly defined daily production quantities, which is appropriate this formulation.
C... subsequent steps and their explanations omitted ...J
25

---

Published as a conference paper at ICLR 2026
Completion Format. The target completion follows the fixed schema required by the system
instruction. During the self-evolving loop. we use the GenPRM itself to generate these completions.
Format
STEP.I: CORRECT INCORRECT EXPLANATION.I: [1-3 sentences explaining the verdict]
STEP 2: CORRECT/ INCORRECT EXPLANATION 2: [...]
STEP-9: CORRECT INCERRECT EXPLANATION.I: [...]
Solver-Consistent Filtering. To ensure the quality of these training targets without human annota- tion, we employ solver-consistent filtering strategy. We only retain training samples where the GenPRM's 's overall verdict aligns with the external solver's ground truth. Specifically:
If the solver returns valid optimal solution, we keep trajectories where GenPRM marks all steps CORRECT.
If the solver indicates failure (infcasibility syntax eπror), keep trajectories where GenPRM correctly identifies at least one step as INCORRECT.
This mechanism ensures that the SFT process is anchored by objective solver verification.
F.2 CONSTRUCTION OF THE SELF-EVOLVING DATASET
The dataset used for the co-evolution loop is derived from the warm-up phase. We generate large pool of synthetic OR problems and solve each multiple times using the initial policy mo with majority voting.
Easy Problems:
Instances where majority voting consistently yields the correct answer are considered "solved" and are excluded from the active training pool improve efficiency,
Challenging Pool: We identify approximately 13,000 challenging instances where the model either fails completely or shows inconsistency (mixed success rates). These form the shared 13k 13k-problem pool used for the iterative co-evolution loop (Figure
F.3 ITERATION SCHEDULE AND STOPPING CRITERIA
In each self-evolving iteration, the current policy generates new trajectories on the 13k-problem pool. These evaluated to construct preference pairs for W-DPO (Policy update) and solver-consistent samples for SFT (GenPRM update).
We perform total of three self-evolving iterations after the warm-up stage. As illustrated in results (Figure the main text), the average Pass accuracy improves consistently over the first three iterations. We observed that the performance gain between the third and potential fourth iteration drops below I %,, indicating saturation. Consequently. we terminate the process and select the checkpoint at the end of the third iteration the final StepORLM.
F.4 COMPUTATIONAL COST AND TRAINING EFFICIENCY
We conduct our training on a single node with x NVIDIA H100 80GB GPUs. The computational cost for each stage is detailed follows:
Warm-up SFT: The initial supervised fine-tuning on the 50K curated dataset (2 epochs) requires approximately 64 GPU-hours.
Each iteration involves trajectory collection, W-DPO policy updates, and GenPRM refinement. The three iterations incur cumulative cost comparable to the warm-up stage. In total, the full StepORLM pipeline (Policy + GenPRM) requires approximately 102 GPU-hours.
Self-Evolution Loop:
26

---

Published as a conference paper at ICLR 2026
This computational footprint is significantly smaller than pre-training or fine-tuning larger generalist models (e.g., 70B parameters), making our framework highly efficient.
27


Reviews:
[{"review_id": "YwpV8DT2UC", "reviewer": "Reviewer_Yvzp", "summary": "This paper introduces StepORLM, a novel framework for training large language models to solve operations research (OR) problems more reliably. The key innovation is generative process supervision through a Generative Process Reward Model (GenPRM) that holistically evaluates full reasoning trajectories, rather than step-by-step discriminative scoring. StepORLM optimizes a policy model through a dual-feedback loop: (1) outcome verification from an external solver ensures correctness of final results, and (2) process evaluation from GenPRM assesses reasoning quality. Training proceeds via Weighted Direct Preference Optimization (W-DPO) to align the policy with high-quality reasoning. Meanwhile, it also optimizes GenPRM during the process. Experiments on six benchmarks show StepORLM (8B parameters) outperforming larger models like GPT-4o, Qwen3-32B, and specialized ORLM baselines, achieving state-of-the-art Pass@1 accuracy. The paper also shows GenPRM’s transferability as a universal process verifier for inference-time scaling.", "questions": "1. Can this method be used in other domains outside OR?\n2. Do you observe any reward hacking using PRM? Is Generative PRM always reliable?\n3. How good is the generalization of this algorithms across dataset and domain?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Introduces generative process supervision, a step beyond discriminative PRMs, enabling trajectory-level reasoning assessment and solving the credit assignment problem.\n2. The co-evolution between policy and GenPRM provides a mutually reinforcing refinement process that avoids overfitting.\n3. Empirical results are strong with demonstrated consistent SOTA performance across six diverse OR benchmarks, outperforming vastly larger generalist and agentic systems.", "weaknesses": "1. Most benchmarks are still synthetic or adapted from prior academic datasets; stronger demonstration on industrial-scale, more real-world tasks would enhance credibility.\n2. The framework assumes deterministic, verifiable outcomes from OR solvers; applicability to open-ended or ill-posed reasoning tasks remains uncertain.\n3. The dual-model co-evolution loop may be expensive to scale, and the paper provides limited discussion of training efficiency.\n4. While numerical gains are clear, qualitative examples of how GenPRM critiques or refines reasoning are sparse. For example, any reward hacking examples from the PRM would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "16CXGrE4HH", "reviewer": "Reviewer_hi6J", "summary": "StepORLM introduces a self-evolving 8B-parameter framework that couples a policy model with a generative process reward model (GenPRM) to train LLMs for operations-research tasks. By iteratively distilling dual feedback—solver-verified outcomes and holistic trajectory-level critiques—into weighted DPO updates, the system attains new state-of-the-art accuracy on six benchmarks, while the co-evolved GenPRM further boosts inference-time scaling for both its own policy and external OR models.", "questions": "Why is there such a large performance gap between LLMOPT (origin) and LLMOPT (reproduce)?\n\nWhat model is the CoT on line 344 based on?\n\nThe StepORLM performance in Tables 2 and 3 is inconsistent with that in Table 1.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed method is simple yet effective, achieving performance improvements.", "weaknesses": "GenPRM and Self-Improve are not novel concepts; they have been explored in prior research, eg. [1][2][3][4].\n\n[1] R-PRM: Reasoning-Driven Process Reward Modeling\n\n[2] GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning\n\n[3] rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking\n\n[4] Inference-Time Scaling for Generalist Reward Modeling", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iq2BZfxW39", "reviewer": "Reviewer_vMGd", "summary": "The paper presents StepORLM, a novel self-evolving framework for LLM-based Operation Research (OR), in which a policy model and a generative process reward model (GenPRM) iteratively improve each other. The framework employs a dual feedback mechanism, combining outcome-based verification from an external solver with process-level evaluation from the GenPRM. The policy model is trained using this combined signal via weighted DPO, while the GenPRM is subsequently refined using new training data generated by the latest policy model. Experimental results demonstrate that the resulting 8B StepORLM model outperforms existing state-of-the-art models across six OR benchmarks, showing the effectiveness of the proposed approach.", "questions": "While the proposed method demonstrates promising performance on OR tasks, the paper in its current form suffers from several major weaknesses concerning its core motivation, methodological clarity, and experimental rigor. The overall presentation is not up to the ICLR standard. In order to raise my rating, I would like the authors to address the different points in the major concerns.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The related work section is clear and reasonable.\n\n2. The proposed self-evolving framework is reasonable and novel to me.\n\n3. The experimental results are strong, demonstrating the effectiveness of the proposed method.", "weaknesses": "## Major Concerns\n\n1. Unsupported Motivation (Myopic PRM). While the credit assignment problem is intuitive and makes sense, the myopia PRM problem appears weak to me since many existing PRMs generate process-level supervision signals based on the holistic model output. The authors need to provide stronger evidence or clearer citations to support their claim and better differentiate their work from existing PRMs. \n\n2. There is an apparent disconnect between the paper's claim of using \"generative process supervision\" and the described implementation. Specifically, according to Algorithm 1 and Equation (1), the detailed, step-by-step process-level correctness scores are seemingly collapsed into a single, global weight $w_{\\tau_{w},\\tau_{l}}$ for the weighted DPO. This suggests that the fine-grained, process-level feedback is lost and not directly utilized to supervise the model's optimization, which contradicts the paper's central claims.\n\n3. Lack of Clarity on GenPRM Refinement: The methodology for refining the GenPRM is not described in sufficient detail. Key information about the training process is missing, particularly the exact format of the training data.\n\n4. The experimental comparisons in Table 1 are incomplete, making it difficult to fully assess the performance of StepORLM. The missing models include the base model Qwen3-8B, SOTA commercial reasoning models (e.g., OpenAI o-series, Gemini 2.5) and SOTA open-source reasoning models (e.g., DeepSeek-R1, Kimi-K2).\n\n5. Important implementation details are missing, including the number of self-evolving iteration, the prompts for process evaluation and inference scaling experiments, etc.\n\n## Minor Issues\n\n1. L98, “Extensive experiments on seven benchmarks” -> “Extensive experiments on six benchmarks”\n\n2. L298, “ComplexORXiao et al. (2024)”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fYqMjftouG", "reviewer": "Reviewer_Bj5d", "summary": "This paper proposes StepORLM, a self-evolving framework for modeling OR. It aims to address the credit assignment problem, where flawed reasoning might lead to a correct outcome , and the limitations of myopic process supervision that overlooks step interdependencies. The framework's core is a co-evolutionary loop between a policy model and a GenPRM. This loop uses dual feedback from an external solver for outcome verification and the GenPRM for holistic process evaluation. The paper reports that its resulting model achieves state-of-the-art results across six benchmarks and that the GenPRM can be used as a process verifier to improve the inference performance of other LLMs.", "questions": "1. The Data Pipeline: The paper must clarify whether the DPO phase re-uses the static 50K SFT instances or dynamically regenerates new ones. The SFT-only model's results must also be provided to quantify the actual gain from the co-evolutionary method.\n\n2. Omitted Work: The authors must discuss the highly relevant concurrent works by Wu et al. (2025) and Lima et al. (2025)  and situate their novelty accordingly.\n\n3. Backbone Choice: The SOTA claim is confounded by the Qwen3-8B backbone. The authors need to provide experimental results on a common backbone (e.g., LLaMA-3-8B) to enable a fair comparison against baselines like ORLM.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Significant Adaptation of Process Supervision:** The paper’s primary strength is its effective adaptation of process-based supervision to the OR domain. By proposing a GenPRM to evaluate entire reasoning trajectories holistically , it aims to mitigate the known limitations of simple outcome-based rewards (the credit assignment problem) and myopic step-wise checks.\n\n**Well-Motivated Co-evolutionary Design:** The paper proposes simultaneous refinement of policy and GenPRM, which is conceptually sound and avoids a static critic. Table 3 shows that freezing GenPRM reduces performance by 3.7% on average, validating the co-evolution mechanism.\n\n**Strong Empirical Results: ** The framework's quality is supported by SOTA performance across six benchmarks. The 8B-parameter StepORLM is shown to outperform vastly larger generalist models and other specialized baselines.", "weaknesses": "**Unclear Attribution Due to Ambiguities in the Data Pipeline:** The paper's SOTA claims are complicated by ambiguities in its data strategy. The 50K-sample SFT dataset is significantly larger than baselines like ORLM (30K) , and Figure 4 shows this SFT-only model is already exceptionally strong . This makes it difficult to attribute the final SOTA performance to the co-evolutionary method (Stage 2) rather than just the high-quality initial data (Stage 1). This ambiguity extends to the DPO phase. Section 3.3.1 is unclear whether the DPO phase re-uses these static 50K instances or regenerates new ones , hindering reproducibility.\n\n\n **Omission of Relevant Concurrent Work:** The literature review overlooks highly relevant concurrent work that addresses the same problem of intermediate-step validation. Specifically, Step-Opt[1] is directly relevant as it proposes a \"Stepwise Validation Mechanism\" with explicit checkers for variables, constraints, and programs . This structured, explicit validation is a conceptually similar (though different in approach) solution to the same problem that StepORLM's learned GenPRM aims to solve. Additionally, Lima et al.[2] on verifiable synthetic data is also relevant and should be discussed .\n\n**Backbone Choice Obscures Method Effects:** The paper's SOTA claims are ambiguous. StepORLM is built exclusively on Qwen3-8B , while principal baselines (like the 8B ORLM) rely on different 8B architectures (LLaMA-3-8B). Without cross-backbone controls (e.g., training StepORLM on the baseline's backbone), the comparison fails to isolate the method’s contribution from the backbone’s.\n\n[1] Wu Y, Zhang Y, Wu Y, et al. Step-Opt: Boosting Optimization Modeling in LLMs through Iterative Data Synthesis and Structured Validation[J]. arXiv preprint arXiv:2506.17637, 2025.\n\n[2] Lima V, Phan D T, Kalagnanam J, et al. Toward a trustworthy optimization modeling agent via verifiable synthetic data generation[J]. arXiv preprint arXiv:2508.03117, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)