StepORLM: A Self-Evolving Framework With Generative Process Supervision For Operations Research Language Models

## Abstract

Abstract Large Language Models (LLMs) have shown promising capabilities for solving Operations Research (OR) problems.
While reinforcement learning serves as a powerful paradigm for LLM training on OR problems, existing works generally face two key limitations. First, outcome reward suffers from the credit assignment problem , where correct final answers can reinforce flawed reasoning.
Second, conventional discriminative process supervision is myopic , failing to evaluate the interdependent steps of OR modeling holistically.
To this end, we introduce StepORLM , a novel self-evolving framework with generative process supervision.
At its core, StepORLM features a co-evolutionary loop where a policy model and a generative process reward model (GenPRM) iteratively improve on each other.
This loop is driven by a dual-feedback mechanism: definitive, outcome-based verification from an external solver, and nuanced, holistic process evaluation from the GenPRM.
The combined signal is used to align the policy via Weighted Direct Preference Optimization (W-DPO) and simultaneously refine the GenPRM.
Our resulting 8B-parameter StepORLM establishes a new state-of-the-art across six benchmarks, significantly outperforming vastly larger generalist models, agentic methods, and specialized baselines.
Moreover, the co-evolved GenPRM is able to act as a powerful and universally applicable process verifier, substantially boosting the inference scaling performance of both our own model and other existing LLMs.

## 1 Introduction

Large language models (LLMs) have shown remarkable capabilities for complex reasoning, enabling new frontiers in specialized domains like Operations Research (OR) (Xiao et al., 2025). Research on applying LLMs to OR has largely advanced along two fronts: 1) developing agentic systems (AhmadiTeshnizi et al., 2024a; Xiao et al., 2024) and 2) improving LLMs through specialized training (Huang et al.,). Within the latter, reinforcement learning (RL) has become a dominant paradigm through outcome reward (Chen et al., 2025b) or process supervision (Astorga et al., 2024; Chen et al., 2025a).

However, these established RL paradigms exhibit critical flaws when applied to operations research, which requires long-horizon, interdependent reasoning.
Outcome-based supervision, which rewards a model based solely on the final solution’s correctness, suffers from the credit assignment problem. As illustrated in Figure [1](https://arxiv.org/html/2509.22558v2#S1.F1)(a), an optimal outcome can emerge from a flawed reasoning process, inadvertently reinforcing incorrect intermediate steps.
While process supervision aims to resolve this by rewarding intermediate steps, conventional discriminative process reward models (PRMs) are often myopic and overlook the critical interdependencies among sequential steps.
They evaluate each step in isolation and thereby fail to capture the holistic context of an OR solution.
As shown in Figure [1](https://arxiv.org/html/2509.22558v2#S1.F1)(b), the correctness of an early step, like a variable definition, may only become apparent once subsequent constraints are defined, causing discriminative PRMs to provide unreliable or inconsistent rewards.

Figure: Figure 1: The illustration of (a) the credit assignment problem of outcome reward and (b) the myopic issue of discriminative process supervision, which motivates the application of generative process supervision for operations research language models.
Refer to caption: figs/temp0924open008_black.png

Therefore, we argue that the contextual and interdependent nature of OR modeling demands a paradigm shift from myopic, step-wise evaluation to holistic, trajectory-level process supervision.
The critic should assess the entire reasoning path retrospectively and understand the intricate dependencies between steps before assigning credit.
As illustrated in Figure [1](https://arxiv.org/html/2509.22558v2#S1.F1)(b), this motivates the application of *generative* PRMs that can reason about the completed trajectory and generate a holistic critique, rather than a discriminative one that provides disconnected, localized scores.

To this end, we propose a novel Self-Evolving Framework with Generative Process Supervision for Operations Research Language Models (StepORLM).
At the heart of our framework is a co-evolutionary loop where a policy model and a novel generative process reward model (GenPRM) iteratively improve each other.
At each iteration, the policy generates reasoning trajectories that are evaluated from two complementary perspectives: definitive outcome verification from an external solver and nuanced, holistic process feedback from GenPRM.
This dual-feedback signal is distilled into 1) preference pairs to align the policy via a Weighted Direct Preference Optimization (W-DPO) objective and 2) filtered trajectory rewarding data to further fine-tune GenPRM.
In this way, both the policy model and GenPRM would be iteratively improved through the self-evolving loop, ultimately leading to process-sound and outcome-correct reasoning capabilities to solve complex OR problems.

Our contributions are summarized as follows:

- •
To the best of our knowledge, we are the first to introduce generative process supervision to holistically evaluate the entire reasoning trajectories on OR problems, mitigating the credit assignment problem and myopia issue of existing RL-based works.
- •
We propose StepORLM, a novel self-evolving framework centered on the co-evolution of a policy model and a generative process reward model (GenPRM).
It integrates a scalable data synthesis pipeline with a dual-feedback loop that grounds the models in both process-level logical consistency and final-outcome correctness.
- •
Extensive experiments on seven benchmarks show that StepORLM achieves the SOTA performance against both vastly larger generalist/agentic models (e.g., GPT-4o) and specialized baselines.
We also demonstrate that the co-evolved GenPRM acts as a universal inference-time verifier, substantially boosting the performance of other OR language models.
- •
We release our code, the StepORLM model weights, and the GenPRM verifier weights to the community, providing a strong foundation for future research in LLM-based OR problem solving.

## 2 Related Work

##### Large Language Models for Optimization Modeling.

Early works explore using LLMs to parse natural language descriptions into mathematical models. For example, the NL4Opt competition (Ramamonjison et al., 2023) demonstrates that ChatGPT and other LLMs can identify problem entities and generate LP formulations from word problems. Pipeline frameworks (Huang et al.,) combine LLM components with traditional solvers: AutoFormulation (Huang et al.,; Jiang et al., 2025; Lu et al., 2025; Zhou et al., 2025) uses LLMs to tag variables and constraints in text descriptions, and systems like OptiMUS(AhmadiTeshnizi et al., 2024b) pipeline textual inputs through LLM-to-code translators before invoking MILP solvers. These methods improve accessibility, but often produce hallucinated constraints or missing logic, as the LLMs have no built-in notion of optimization semantics. Agentic methods Zhang et al. (2024); Yang et al. (2025b) are also proposed, which assign specialized LLM agents under a conductor that orchestrates and reflects on partial solutions (Xiao et al., 2024; Deng et al., 2024; AhmadiTeshnizi et al., 2024a). While these approaches outperform simpler pipelines on benchmarks, they still rely on repeated LLM exchanges without explicit verification of each reasoning step. In practice, even advanced models exhibit unsound reasoning: for instance, Jiang et al. (2025) note that alignment and “self-correction” mechanisms are needed to prevent LLM hallucinations in optimization contexts.

##### Advanced Training and Refinement Paradigms.

To enhance the reliability of LLM reasoning, recent work moves beyond simple pipelines to incorporate advanced training and refinement paradigms Fang et al. (2025); Xi et al. (2025). One major direction is reinforcement learning, which is applied with varying feedback granularity. Outcome-driven methods like Solver-Informed RL (SIRL) (Chen et al., 2025b) use a solver’s final verdict for precise rewards but suffer from a severe credit assignment problem, as a correct final model might be reached via flawed intermediate steps (Chen et al., 2025b). In contrast, process-driven methods using Process Reward Models (PRMs) provide denser, step-by-step feedback but can be myopic, rewarding locally correct steps that are globally inconsistent (Dai et al., 2025; Zhu et al., 2025). To implement such reward schemes, preference-based training objectives like Direct Preference Optimization (DPO) emerge as a stable and effective alternative to complex RLHF pipelines (Zheng et al., 2025a). Complementing these training methods are iterative refinement and search strategies. These techniques embed LLMs into structured search frameworks like beam search (Wang et al., 2024) and Monte Carlo Tree Search (Zheng et al., 2025b; Liu et al., 2024), or employ evolutionary loops with self-reflection to iteratively improve generated solutions (Ye et al., 2024). While these approaches make final outputs more reliable by exploring a wider solution space, they often operate at the output level rather than being deeply integrated into the model’s core training process. Thus, a trade-off remains: outcome-based RL lacks intermediate supervision, stepwise PRMs lack global coherence, and search methods are often applied post-hoc. To bridge this gap, discriminative methods based on both final outcome correctness and stepwise feedback can penalize incorrect reasoning steps while maintaining global coherence, effectively combining the strengths of both process- and outcome-based approaches (Chen et al., 2025a).

The research community actively builds end-to-end OR modeling systems that merge symbolic rigor with learned reasoning. This effort is supported by a growing number of benchmarks (Wang et al., 2024; Huang et al., 2024; ; Yang et al., 2025c; Xiao et al., 2025). StepORLM is designed to address these gaps by embedding process-level supervision and iterative refinement, ensuring each modeling step is verified. Its co-evolutionary framework combines a generative PRM for nuanced step-level feedback with final-solver rewards, mitigating the credit assignment problem via a weighted DPO objective. By having its policy and generative PRM co-evolve, StepORLM learns to self-verify internally, leading to more robust and error-resistant modeling behavior that advances the goal of reliable and scalable OR reasoning with LLMs.

## 3 Methodology

### 3.1 Overview of StepORLM

We train StepORLM using a two-stage, self-evolving framework designed to instill robust, process-sound reasoning for operations research problems.
As illustrated in Figure [2](https://arxiv.org/html/2509.22558v2#S3.F2) and Figure [3](https://arxiv.org/html/2509.22558v2#S3.F3), the framework is composed of two main stages:

- •
Stage 1: Warm-Up for Initial Policy. As shown in Figure [2](https://arxiv.org/html/2509.22558v2#S3.F2), we bootstrap the training process by creating a high-quality, solver-verified dataset, which consists of OR problems and step-by-step solutions.
This corpus is used to train an initial policy $\pi_{0}$ via Supervised Fine-Tuning (SFT).
- •
Stage 2: Iterative Co-Evolution for Policy and GenPRM. As shown in Figure [3](https://arxiv.org/html/2509.22558v2#S3.F3), starting with the initial policy $\pi_{0}$ and a base LLM as the critic $\rho_{\theta_{0}}$, we commence a self-evolving training loop. The policy model and the generative process reward model (GenPRM) co-evolve, with the policy generating increasingly better solutions and the GenPRM becoming a more discerning critic.

Figure: Figure 2: The illustration of data synthesis pipeline for the warm-up stage. A teacher LLM generates diverse OR problems and their step-by-step reasoning solutions. Each solution is rigorously validated and filtered by an external solver, with an automated refinement loop to ensure the quality. This verified corpus is then used to train the initial policy model via SFT.
Refer to caption: figs/data_synthesis_framework.png

### 3.2 Warm-Up for Initial Policy

The goal of the warm-up stage is to produce a strong initial policy model capable of generating structured, multi-step solutions to OR problems. We achieve this through a scalable data synthesis pipeline, shown in Figure [2](https://arxiv.org/html/2509.22558v2#S3.F2).

We begin with a large pool of base OR problem templates and diverse industry scenarios. A powerful teacher LLM (GPT-4o in this paper) first pairs compatible templates and scenarios to draft a natural-language question $Q$. To enrich the data, each question is augmented through paraphrasing, unit conversion, and controlled parameter scaling. The teacher model then generates a complete, multi-step reasoning trajectory for each question $Q$, mirroring the workflow of a human expert from problem analysis to executable solver code. See Appendix [C](https://arxiv.org/html/2509.22558v2#A3) for more details.

To ensure data quality, every generated solution undergoes a deterministic validation process: we execute the code, inspect the solver status, and verify the objective value.
Trajectories that produce errors would trigger an automatic self-refinement loop, prompting the teacher LLM to revise the formulation or code until validation succeeds or the retry budget is exhausted. For questions with multiple valid solutions, we select the one achieving the best objective value. This rigorous process yields a high-quality dataset of verified $(Q,R)$ pairs, i.e., Question (Q) and Reasoning (R). We use the constructed dataset to train the initial policy model $\pi_{0}$ via Supervised Fine-Tuning (SFT).

### 3.3 Iterative Co-Evolution for Policy and GenPRM

Following the warm-up stage, the iterative co-evolution stage begins. As illustrated in Figure [3](https://arxiv.org/html/2509.22558v2#S3.F3), this self-evolving loop simultaneously enhances the policy model $\pi_{\theta}$ and the generative process reward model (GenPRM) $\rho_{\phi}$. Each iteration involves three key steps: 1) trajectory collection and evaluation, 2) policy alignment, and 3) GenPRM refinement.

#### 3.3.1 Trajectory Collection and Dual-Source Evaluation

Our training data is collected in the warm-up stage.
For each problem in the training set, the current policy $\pi_{\theta}$ generates a set of $k$ candidate solution trajectories. These trajectories are then put through a dual-source evaluation:

- •
Outcome Verification. The final code of each trajectory is executed, and an external solver provides a definitive, ground-truth label of success or failure.
- •
Process Evaluation. The GenPRM $\rho_{\theta}$ holistically assesses each complete trajectory and provides step-by-step correctness scores, capturing the quality of the reasoning process.
Note that, different from conventional discriminative process supervision, our GenPRM is required to first perform chain-of-thought reasoning over the OR solution and then generate a holistic, post-hoc evaluation of the entire reasoning trajectory.

After assessing the generated trajectories, we use the dual-feedback signals to further refine both the policy and GenPRM. The improved models would then start another round of self-evolving iteration.
Next, we introduce the policy alignment and GenPRM refinement at each iteration.

Figure: Figure 3: The co-evolutionary loop of StepORLM. At each iteration, the policy model $\pi_{\theta}$ generates multiple trajectories. The feedback from both the external solver (outcome) and the GenPRM $\rho_{\theta}$ (process) is used to create training data that simultaneously refines the policy via W-DPO and improves the GenPRM via SFT, fostering reciprocal improvement.
Refer to caption: figs/self_evolving_training.png

#### 3.3.2 Policy Alignment via W-DPO

The dual-source feedback is first used to align the policy model. We distill the trajectory evaluation results into preference pairs $(\tau_{w},\tau_{l},w)$, where $\tau_{w}$ is the winning trajectory and $\tau_{l}$ is the losing one.
The selection logic, as detailed in Algorithm [1](https://arxiv.org/html/2509.22558v2#alg1), prioritizes solver-verified solutions.
If solver outcomes are identical for a pair of trajectories (e.g., both pass or both fail), then the trajectory with a higher ratio of correct steps, as judged by the GenPRM, is preferred.

Next, we design a Weighted Direct Preference Optimization (W-DPO) objective for policy training. The weight $w$ is defined to quantify the quality gap between two trajectories: solver-distinguished pairs receive a fixed, high weight (i.e., $w=1.0$), while others are weighted proportionally to the difference in their process quality scores (i.e., ratio of correct steps).
The final objective focuses the model training on the most informative pairs weighted by $w$:

$$ $\mathcal{L}_{\mathrm{W\text{-}DPO}}(\theta)\;=\;-\,\mathbb{E}_{(x,\tau_{w},\tau_{l})}\Big[w_{(\tau_{w},\tau_{l})}\cdot\log\sigma\!\big(\beta\big(\log\pi_{\theta}(\tau_{w}\mid x)-\log\pi_{\theta}(\tau_{l}\mid x)\big)\big)\Big],$ (1) $$

where $w_{(\tau_{w},\tau_{l})}$ is the weight for trajectory pair $(\tau_{w},\tau_{l})$, $\pi_{\theta}(\tau\mid x)$ is the log-likelihood of generating trajectory $\tau$ given the input question $x$, and $\beta$ is a scaling factor.

Figure: Algorithm 1 Preference Pair Construction

#### 3.3.3 GenPRM Refinement via SFT

The same trajectory evaluation data from the current iteration is then used to refine the critic, i.e., GenPRM.
Specifically, we extract all step-level correctness labels derived from the solver-verified trajectories to form a new, high-quality dataset.
This dataset is used to continue the supervised fine-tuning of GenPRM $\rho_{\theta}$ with process supervision signals.

We can see that such a process creates a positive feedback loop: as the policy improves, it generates more diverse and accurate reasoning paths, providing higher-quality training data for the GenPRM. In turn, a more capable GenPRM provides a more precise and reliable reward signal, further accelerating the policy alignment. This symbiotic refinement is crucial for surpassing the performance of the initial SFT model, which is thus referred to as the self-evolving framework.

## 4 Experiments

### 4.1 Experiment Setup

##### Benchmarks.

We conduct our evaluation on a diverse set of six public benchmarks widely used in the OR community: NL4Opt (Ramamonjison et al., 2023), MAMO (split into EasyLP and ComplexLP)(Huang et al., 2024), NLP4LP(AhmadiTeshnizi et al., 2024b), ComplexORXiao et al. (2024), IndustryOR(Huang et al.,) and ReSocratic(Yang et al., 2025c). More details about the six benchmarks are provided in appendix [A](https://arxiv.org/html/2509.22558v2#A1).

##### Baselines.

To ensure a thorough comparison, we benchmark against a comprehensive set of baselines in three categories:

- •
Zero-shot Generalist LLMs: Powerful, large-scale models without specific fine-tuning, including GPT-4o (OpenAI, 2024), DeepSeek-V3 (DeepSeek-AI et al., 2025), Qwen3-32B (Yang et al., 2025a), and Qwen2.5-72B-Instruct (Qwen et al., 2025).
- •
Specialized Fine-tuned LLMs: Models specifically trained on OR solving tasks, including ORLM (8B) (Huang et al.,), LLMOPT (14B) (Jiang et al., 2025) and OptMATH (32B) (Lu et al., 2025). We try our best to re-evaluate all publicly accessible models.
For those not open-sourced, we report their original performance according to corresponding published papers.
Note that due to our use of a cleaned dataset (Xiao et al., 2025), these original scores serve as a reference and may not be directly comparable.
- •
Agentic Methods: Multi-step reasoning frameworks that orchestrate LLMs to solve problems, including OptiMUS-v0.3 (AhmadiTeshnizi et al., 2024a), Chain-of-Thought (CoT) (Wei et al., 2022), Chain-of-Experts (CoE) (Xiao et al., 2024), and CAFA (Deng et al., 2024). To ensure a fair and powerful comparison, all agentic methods are implemented using GPT-4o as their backbone model.

##### Implementation Details

We develop our StepORLM models by fine-tuning the publicly available Qwen3-8B (Yang et al., 2025a).
Training and inference are conducted on a single node with 8 × NVIDIA H100 80 GB GPUs. We used a synthesized dataset of 50K questions for subsequent training. The model was trained using the AdamW optimizer with a learning rate of 7e-6 and weight decay of 0.1. The maximum sequence length was set to 8192, and we introduced special tokens <step> and </step> to enhance reasoning. A batch size of 64 was used, and training proceeded for 3 epochs. During the self-evolving sampling phase, we generated 4 trajectories for each response to be used for subsequent preference learning. For W-DPO, we set the hyperparameter $\beta$ to 0.1. To ensure optimal performance and a fair comparison across different frameworks, we selected solvers best suited to each category of model. For general-purpose LLMs and all agentic methods, we used Gurobi due to its broad compatibility. For the fine-tuned models, we used their respectively adapted solvers: both our StepORLM and ORLM employed the COPT solver, whereas LLMOPT is evaluated using Pyomo.

Table: Table 1: The overall performance of StepORLM and baselines with Pass@1 accuracy (%) on six OR benchmarks. StepORLM+GenPRM indicates using the GenPRM as process verifier to enable the inference scaling of StepORLM. Scores cited from original publications are marked with the symbol (*), while missing entries are denoted with (-). Best results are highlighted in bold and the second-highest values are underlined.

### 4.2 Main Results

We report the main results in Table [1](https://arxiv.org/html/2509.22558v2#S4.T1). Our StepORLM establishes a new state-of-the-art in OR problem-solving tasks, significantly outperforming both vastly larger generalist/agentic models and specialized LLMs.
As an 8B-parameter model, StepORLM outperforms much larger, state-of-the-art generalist models, such as DeepSeek-V3 (671B) and Qwen2.5-72B-Instruct (72B), and even those agentic methods based on GPT-4o.
This result underscores the profound effectiveness of specialized, process-supervised training, enabling a compact model to achieve more reliable and accurate OR modeling capabilities than its far larger counterparts.

Moreover, leveraging the co-evolved GenPRM as an inference-time process verifier (i.e., StepORLM+GenPRM) further pushes the performance to an average Pass@1 accuracy of 85.6%.
Such a configuration excels especially on the most challenging benchmarks, with remarkable accuracy gains of 9.9% on ComplexOR and 9.5% on IndustryOR compared with the basic StepORLM.
This highlights the dual benefit of our co-evolutionary framework: the policy model learns to generate high-quality solutions, while the GenPRM learns to effectively identify and select the best one. Furthermore, StepORLM excels at avoiding reward hacking, a point we illustrate with a case study in Appendix [B](https://arxiv.org/html/2509.22558v2#A2).

Figure: Figure 4: The analysis on the self-evolving process by tracking the performance (Pass@1 accuracy) at each training iteration. The relative improvement of current iteration over the previous one is demonstrated on the corresponding bar.
Refer to caption: figs/Iter_compare.png

### 4.3 Analysis of Self-Evolving Process

We conduct in-depth analysis on the self-evolving process by tracking the performance at each training iteration, including the warm-up and co-evolving stages.
The results are reported in Figure [4](https://arxiv.org/html/2509.22558v2#S4.F4), from which we can obtain the following observations:

- •
The warm-up supervised fine-tuning (SFT) provides a massive foundational performance lift across all benchmarks (e.g., 62.5% accuracy improvement on NLP4LP).
- •
The subsequent self-evolution iterations consistently build upon the prior models, delivering incremental but crucial accuracy improvements. This confirms that the model progressively refines its capabilities through our self-evolving loop, rather than stagnating after initial fine-tuning.
- •
The performance trend is notably non-monotonic on the IndustryOR dataset, where the performance first declines and finally largely increases at the third iteration. We attribute this to the small size of testing set and the challenging questions in IndustryOR.
Our case-level inspection reveals a potential progression. Early iterations primarily rectify structural modeling errors flagged by the PRM, whereas later iterations concentrate on code-level issues (e.g., index-out-of-bounds errors), which finally leads to accuracy improvements.

### 4.4 Inference Scaling with GenPRM

In this section, we investigate the generalization capability of our co-evolved GenPRM as the process verifier for inference scaling.
We apply it to our final StepORLM and another open-sourced ORLM baseline, and compare GenPRM’s effectiveness against other inference scaling strategies including majority voting, solver execution, and discriminative PRM.

As shown in Table [2](https://arxiv.org/html/2509.22558v2#S4.T2), our self-evolved GenPRM is generally the most effective inference scaling strategy, achieving the highest accuracy on average for both StepORLM and ORLM.
When paired with ORLM, our GenPRM boosts its average performance by 10%, demonstrating its powerful generalization.
It learns fundamental, model-agnostic principles of valid OR reasoning, allowing it to function as a universal verifier that significantly improves the performance of other models, not just its co-evolved policy.

Table: Table 2: Performance comparison of StepORLM under different inference scaling strategies.

Table: Table 3: Ablation study of StepORLM on Pass@1 accuracy (%). The best results are given in bold, and the second-best values are underlined.

### 4.5 Ablation Study

To study the impact of each component in our proposed self-evolving framework, we conduct experiments based on the following variants:
(1) w/o SFT: skipping the warm-up supervised fine-tuning stage; (2) w/o self-evolution: directly sampling a single reasoning trajectory after SFT and applying DPO without iterative refinement; (3) w/o GenPRM Evolution: using the initial GenPRM with frozen parameters for all subsequent iterations; (4) w/o W-DPO: replacing the W-DPO with standard DPO during the self-evolving training phase.
The results are shown in Table [3](https://arxiv.org/html/2509.22558v2#S4.T3).

Removing the warm-up SFT (w/o SFT) causes the largest performance drop, confirming the necessity of a strong initial policy.
Disabling the iterative loop (w/o Self-evolution) prevents the model from refining its initial capabilities, leading to lower scores on challenging benchmarks like ComplexOR.
Freezing the GenPRM after initialization (w/o GenPRM Evolution) also degrades performance, validating the importance of the co-evolutionary dynamic between the policy and the critic.
Finally, replacing our W-DPO objective with standard DPO (w/o W-DPO) results in a noticeable performance decrease, indicating that our trajectory-level quality weighting provides a more effective learning signal between different trajectory pairs. Together, these results demonstrate that the synergy among all components drives the success of the StepORLM framework.

## 5 Conclusion

In this paper, we design StepORLM, a novel self-evolving framework to overcome critical limitations in training large language models for operations research. Our approach addresses the credit assignment problem of outcome-based rewards and the myopia of conventional process supervision through a co-evolutionary loop between a policy model and a generative process reward model (GenPRM).
StepORLM integrates definitive solver-based outcome verification with holistic, generative process feedback.
Such a dual-feedback mechanism ensures the process-sound and outcome-correct reasoning capabilities of the policy model to solve complex OR problems.
Our 8B-parameter StepORLM achieves the SOTA performance across six benchmarks, significantly outperforming much larger generalist models, agentic systems, and specialized baselines.
Furthermore, the co-evolved GenPRM is able to act as a powerful, universal inference-time process verifier, substantially enhancing the performance of our model and other LLMs for OR problem-solving tasks.

## References

- AhmadiTeshnizi et al. (2024a)
Ali AhmadiTeshnizi, Wenzhi Gao, Herman Brunborg, Shayan Talaei, and Madeleine Udell.
Optimus-0.3: Using large language models to model and solve optimization problems at scale, 2024a.
URL [https://arxiv.org/abs/2407.19633](https://arxiv.org/abs/2407.19633).
- AhmadiTeshnizi et al. (2024b)
Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell.
OptiMUS: Scalable Optimization Modeling with (MI)LP Solvers and Large Language Models.
In *Proceedings of the 41st International Conference on Machine Learning (ICML)*, 2024b.
- Astorga et al. (2024)
Nicolás Astorga, Tennison Liu, Yuanzhang Xiao, and Mihaela van der Schaar.
Autoformulation of Mathematical Optimization Models Using LLMs.
*arXiv preprint arXiv:2411.01679*, 2024.
- Chen et al. (2025a)
Peter Chen, Xiaopeng Li, Ziniu Li, Xi Chen, and Tianyi Lin.
Stepwise guided policy optimization: Coloring your incorrect reasoning in grpo, 2025a.
URL [https://arxiv.org/abs/2505.11595](https://arxiv.org/abs/2505.11595).
- Chen et al. (2025b)
Yitian Chen, Jingfan Xia, Siyu Shao, Dongdong Ge, and Yinyu Ye.
Solver-informed rl: Grounding large language models for authentic optimization modeling, 2025b.
URL [https://arxiv.org/abs/2505.11792](https://arxiv.org/abs/2505.11792).
- Dai et al. (2025)
Ning Dai, Zheng Wu, Renjie Zheng, Ziyun Wei, Wenlei Shi, Xing Jin, Guanlin Liu, Chen Dun, Liang Huang, and Lin Yan.
Process supervision-guided policy optimization for code generation, 2025.
URL [https://arxiv.org/abs/2410.17621](https://arxiv.org/abs/2410.17621).
- DeepSeek-AI et al. (2025)
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang
Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang
Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan.
Deepseek-v3 technical report, 2025.
URL [https://arxiv.org/abs/2412.19437](https://arxiv.org/abs/2412.19437).
- Deng et al. (2024)
Haoxuan Deng, Bohao Zheng, Yirui Jiang, and Trung Hieu Tran.
Cafa: Coding as auto-formulation can boost large language models in solving linear programming problem.
In *The 4th Workshop on Mathematical Reasoning and AI at NeurIPS ’24*, 2024.
- Fang et al. (2025)
Jinyuan Fang, Yanwen Peng, Xi Zhang, Yingxu Wang, Xinhao Yi, Guibin Zhang, Yi Xu, Bin Wu, Siwei Liu, Zihao Li, et al.
A comprehensive survey of self-evolving ai agents: A new paradigm bridging foundation models and lifelong agentic systems.
*arXiv preprint arXiv:2508.07407*, 2025.
- Huang et al. (0)
Chenyu Huang, Zhengyang Tang, Shixi Hu, Ruoqing Jiang, Xin Zheng, Dongdong Ge, Benyou Wang, and Zizhuo Wang.
Orlm: A customizable framework in training large models for automated optimization modeling.
*Operations Research*, 0(0):null, 0.
doi: 10.1287/opre.2024.1233.
URL [https://doi.org/10.1287/opre.2024.1233](https://doi.org/10.1287/opre.2024.1233).
- Huang et al. (2024)
Xuhan Huang, Qingning Shen, Yan Hu, Anningzhe Gao, and Benyou Wang.
MAMO: A mathematical modeling benchmark with solvers.
*arXiv preprint*, 2024.
- Jiang et al. (2025)
Caigao Jiang, Xiang Shu, Hong Qian, Xingyu Lu, Jun Zhou, Aimin Zhou, and Yang Yu.
LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch.
In *International Conference on Learning Representations (ICLR)*, 2025.
- Liu et al. (2024)
Fei Liu, Xialiang Tong, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang.
Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Models.
In *International Conference on Machine Learning (ICML)*, 2024.
- Lu et al. (2025)
Hongliang Lu, Zhonglin Xie, Yaoyu Wu, Can Ren, Yuxuan Chen, and Zaiwen Wen.
Optmath: A scalable bidirectional data synthesis framework for optimization modeling, 2025.
URL [https://arxiv.org/abs/2502.11102](https://arxiv.org/abs/2502.11102).
- OpenAI (2024)
OpenAI.
Gpt-4o system card.
[https://openai.com/research/gpt-4o](https://openai.com/research/gpt-4o), 2024.
- Qwen et al. (2025)
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025.
URL [https://arxiv.org/abs/2412.15115](https://arxiv.org/abs/2412.15115).
- Ramamonjison et al. (2022)
Rindranirina Ramamonjison, Haley Li, Timothy T Yu, Shiqi He, Vishnu Rengan, Amin Banitalebi-Dehkordi, Zirui Zhou, and Yong Zhang.
Augmenting operations research with auto-formulation of optimization models from problem descriptions.
*arXiv preprint arXiv:2209.15565*, 2022.
- Ramamonjison et al. (2023)
Rindranirina Ramamonjison, Timothy T. Yu, Raymond Li, Haley Li, Giuseppe Carenini, Bissan Ghaddar, Shiqi He, Mahdi Mostajabdaveh, Amin Banitalebi-Dehkordi, Zirui Zhou, and Yong Zhang.
NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptions.
*arXiv preprint arXiv:2303.08233*, 2023.
- Wang et al. (2024)
Teng Wang, Wing-Yin Yu, Zhenqi He, Zehua Liu, Xiongwei Han, Hailei Gong, Han Wu, Wei Shi, Ruifeng She, Fangzhou Zhu, and Tao Zhong.
BPP-Search: Enhancing Tree-of-Thought Reasoning for Mathematical Modeling Problem Solving.
*arXiv preprint arXiv:2411.17404*, 2024.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language models.
In *Advances in Neural Information Processing Systems*, volume 35, pp. 24824–24837. Curran Associates, Inc., 2022.
- Xi et al. (2025)
Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang.
A survey of llm-based deep search agents: Paradigm, optimization, evaluation, and challenges.
*arXiv preprint arXiv:2508.05668*, 2025.
- Xiao et al. (2024)
Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan J. Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, and Gang Chen.
Chain-of-Experts: When LLMs Meet Complex Operations Research Problems.
In *International Conference on Learning Representations (ICLR)*, 2024.
- Xiao et al. (2025)
Ziyang Xiao, Jingrong Xie, Lilin Xu, Shisi Guan, Jingyan Zhu, Xiongwei Han, Xiaojin Fu, WingYin Yu, Han Wu, Wei Shi, et al.
A survey of optimization modeling meets llms: Progress and future directions.
*arXiv preprint arXiv:2508.10047*, 2025.
- Yang et al. (2025a)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report, 2025a.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Yang et al. (2025b)
Yingxuan Yang, Huacan Chai, Yuanyi Song, Siyuan Qi, Muning Wen, Ning Li, Junwei Liao, Haoyi Hu, Jianghao Lin, Gaowei Chang, et al.
A survey of ai agent protocols.
*arXiv preprint arXiv:2504.16736*, 2025b.
- Yang et al. (2025c)
Zhicheng Yang, Yiwei Wang, Yinya Huang, Zhijiang Guo, Wei Shi, Xiongwei Han, Liang Feng, Linqi Song, Xiaodan Liang, and Jing Tang.
Optibench meets resocratic: Measure and improve llms for optimization modeling, 2025c.
URL [https://arxiv.org/abs/2407.09887](https://arxiv.org/abs/2407.09887).
- Ye et al. (2024)
Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song.
ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution.
In *Advances in Neural Information Processing Systems (NeurIPS)*, 2024.
- Zhang et al. (2024)
Weinan Zhang, Junwei Liao, Ning Li, Kounianhua Du, and Jianghao Lin.
Agentic information retrieval.
*arXiv preprint arXiv:2410.09713*, 2024.
- Zheng et al. (2025a)
Congmin Zheng, Jiachen Zhu, Jianghao Lin, Xinyi Dai, Yong Yu, Weinan Zhang, and Mengyue Yang.
Cold: Counterfactually-guided length debiasing for process reward models.
*arXiv preprint arXiv:2507.15698*, 2025a.
- Zheng et al. (2025b)
Zhi Zheng, Zhuoliang Xie, Zhenkun Wang, and Bryan Hooi.
Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Design.
*arXiv preprint arXiv:2501.08603*, 2025b.
- Zhou et al. (2025)
Chenyu Zhou, Jingyuan Yang, Linwei Xin, Yitian Chen, Ziyan He, and Dongdong Ge.
Auto-formulating dynamic programming problems with large language models, 2025.
URL [https://arxiv.org/abs/2507.11737](https://arxiv.org/abs/2507.11737).
- Zhu et al. (2025)
Jiachen Zhu, Congmin Zheng, Jianghao Lin, Kounianhua Du, Ying Wen, Yong Yu, Jun Wang, and Weinan Zhang.
Retrieval-augmented process reward model for generalizable mathematical reasoning.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Findings of the Association for Computational Linguistics: ACL 2025*, pp. 8453–8468, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-256-5.
doi: 10.18653/v1/2025.findings-acl.444.
URL [https://aclanthology.org/2025.findings-acl.444/](https://aclanthology.org/2025.findings-acl.444/).

## Appendix A Benchmarks

A mass of datasets and methodologies for dataset construction have been proposed to improve the capacity of LLM to address OR problems, with these datasets becoming increasingly intricate and realistic(AhmadiTeshnizi et al., 2024b; Yang et al., 2025c). A recent survey by (Xiao et al., 2025) provides a comprehensive and timely review of benchmark datasets for optimization problems described in natural language. The study identified prevalent issues within the datasets, including logical fallacies, ambiguous problem definitions and incorrect ground-truth answers, and also corrected a portion of the errors identified. Based on that, our work builds upon their initial efforts by conducting further manual verification and cleaning of these widely-used benchmarks. Subsequently, the data is processed into a unified format, thereby establishing a more robust and standardized foundation for subsequent research and model assessment.

### A.1 NL4Opt

The NL4Opt dataset was introduced by Ramamonjison et al. as part of the NeurIPS 2022 NL4Opt competition (Ramamonjison et al., 2022; 2023). It consists of approximately 1,100 annotated word problems describing linear programming (LP) scenarios, with the primary purpose of bridging natural language descriptions and formal optimization models. In a survey, Xiao et al. reported that NL4Opt has a complexity score of 5.59 and an original error rate of at least 26.4%. Each problem presents a self-contained, real-world scenario (e.g., resource allocation, scheduling) with annotations identifying key optimization elements. The official shared task for the dataset is divided into two subtasks: (1) recognizing and labeling problem entities and (2) generating a corresponding mathematical model. The original dataset provides 713 training, 99 validation, and 289 test instances. For our evaluation, we selected 213 validated problems from this benchmark.

### A.2 MAMO

The MAMO dataset was introduced by Huang et al. (2024) as a benchmark for evaluating large language models on complex mathematical modeling tasks, with a specific focus on the formulation process itself rather than just solution accuracy (Huang et al., 2024). The dataset concentrates on linear programming (LP) and mixed-integer linear programming (MILP) problems, excluding nonlinear or differential equation modeling. MAMO is divided into two main components: Easy LP (652 instances) and Complex LP (211 instances). According to an analysis by Xiao et al., EasyLP has a complexity of 7.12 with an error rate of at least 8.13%, while ComplexLP is significantly more difficult, with a complexity of 13.35 and an error rate of at least 23.7%. By delegating the solving step to an OR solver, MAMO focuses specifically on whether an LLM can accurately translate a detailed natural language description into a correct mathematical formulation. From this benchmark, we selected 545 easy and 111 complex instances for our study.

### A.3 NLP4LP

The NLP4LP benchmark, introduced by AhmadiTeshnizi et al. (2023) in their work on the OptiMUS system (AhmadiTeshnizi et al., 2024b; a), evaluates LLM-based agents on translating natural language operations research problems into solver-ready code and mathematical models. The survey by Xiao et al. assigned it a complexity score of 5.58 and identified an error rate of at least 21.7% in the original dataset. It consists of 269 human-authored LP and MILP problems, where each instance presents an optimization scenario from classical OR domains such as scheduling, knapsack allocation, and production planning. The benchmark provides a fine-grained testbed for assessing formulation accuracy. In our work, we chose to evaluate 178 verified instances from NLP4LP.

### A.4 ComplexOR

The ComplexOR dataset was introduced by Xiao et al. as part of their Chain-of-Experts framework to stress-test the reasoning and modeling capabilities of LLMs on scenarios more challenging than those in earlier datasets (Xiao et al., 2024). The analysis by Xiao et al. noted a complexity score of 5.98 and a significant error rate of at least 24.3%. ComplexOR was curated from advanced OR case studies and classic difficult optimization problem classes, such as multi-step lot-sizing problems with setup costs, intricate scheduling tasks, and supply chain design problems. The problems are primarily large-scale or conceptually complex mixed-integer linear programs that require multi-step reasoning and often involve hierarchical or time-indexed structures. The dataset’s intended purpose is to facilitate the development and evaluation of advanced reasoning strategies for OR problem solving. Following our verification process, we selected 18 reliable problems from this dataset for evaluation.

### A.5 IndustryOR

The IndustryOR dataset, introduced by Huang et al. (2024), is the first industrial operations research benchmark for large language models (Huang et al.,). It comprises 100 real-world optimization scenarios from sectors like manufacturing, logistics, finance, and energy. The study by Xiao et al. highlighted this dataset’s high difficulty, reporting a complexity score of 14.06 and a substantial error rate of at least 54.0%. Designed to evaluate LLMs on practical, domain-specific tasks, IndustryOR covers five OR categories—linear, integer, mixed-integer, nonlinear, and others—and is labeled by difficulty (Easy, Medium, Hard). For our experiments, we selected 42 validated problems from this benchmark.

### A.6 ReSocratic

The ReSocratic dataset is introduced by (Yang et al., 2025c), the name of which is actually a data synthesis method proposed by this work as well. This method synthesize the formatted optimization demonstration in a reverse manner first, and then back-translates it into a question. Through these intermediate reasoning steps, ReSocratic has a higher quality than previous methods. In this paper, we selected 403 validated problems from the original ReSocratic-29K dataset.

## Appendix B Case Study

The following presentation will outline a comprehensive prototype of the failure case B, as referenced in Figure  [1](https://arxiv.org/html/2509.22558v2#S1.F1). The sequence of events in the present case study is as follows: firstly, a sketch map of the problem is presented; secondly, a detailed problem description is given; and thirdly, the responses given by Qwen3 and our model StepORLM are outlined. As previously stated, the ensuing discourse will focus on the results obtained from the two models. The graph illustrates the problem and demonstrates that an error in the constraints step will not affect the optimal value.

Figure: Figure 5: A simplified illustration of problem 74 in ComplexLP
Refer to caption: figs/case_guide_tour.png

This is a problem with the identifier 74 from the ComplexLP dataset:

And the following is our generate prompt for both Qwen3 and StepORLM.

### B.1 StepORLM’s response

### B.2 Qwen3’s response

### B.3 Discussions on two models’ results

This example clearly shows why rewards should be based on step-by-step reasoning rather than just the optimal value. During the reasoning process, StepORLM correctly recognises the importance of subtour elimination constraints, whereas Qwen3 initially identifies them but then overlooks them due to circular self-debate. In this case, the presence or absence of these constraints does not affect the final result. Therefore, both codes can obtain the correct optimal solution.

## Appendix C Step-level Reasoning Trajectory Template

Each reasoning step is enclosed within <step>...</step> tags. While this structure is fixed during the initial warm-up stage, it becomes more flexible in subsequent self-evolving iterations. This is because we do not constrain the model’s output format or the number of steps during this phase, allowing the LLM to learn more complex and implicit reasoning patterns

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Can this method be used in other domains outside OR?
2. Do you observe any reward hacking using PRM? Is Generative PRM always reliable?
3. How good is the generalization of this algorithms across dataset and domain?
soundness: 2
strengths: 1. Introduces generative process supervision, a step beyond discriminative PRMs, enabling trajectory-level reasoning assessment and solving the credit assignment problem.
2. The co-evolution between policy and GenPRM provides a mutually reinforcing refinement process that avoids overfitting.
3. Empirical results are strong with demonstrated consistent SOTA performance across six diverse OR benchmarks, outperforming vastly larger generalist and agentic systems.
summary: This paper introduces StepORLM, a novel framework for training large language models to solve operations research (OR) problems more reliably. The key innovation is generative process supervision through a Generative Process Reward Model (GenPRM) that holistically evaluates full reasoning trajectories, rather than step-by-step discriminative scoring. StepORLM optimizes a policy model through a dual-feedback loop: (1) outcome verification from an external solver ensures correctness of final results, and (2) process evaluation from GenPRM assesses reasoning quality. Training proceeds via Weighted Direct Preference Optimization (W-DPO) to align the policy with high-quality reasoning. Meanwhile, it also optimizes GenPRM during the process. Experiments on six benchmarks show StepORLM (8B parameters) outperforming larger models like GPT-4o, Qwen3-32B, and specialized ORLM baselines, achieving state-of-the-art Pass@1 accuracy. The paper also shows GenPRM’s transferability as a universal process verifier for inference-time scaling.
weaknesses: 1. Most benchmarks are still synthetic or adapted from prior academic datasets; stronger demonstration on industrial-scale, more real-world tasks would enhance credibility.
2. The framework assumes deterministic, verifiable outcomes from OR solvers; applicability to open-ended or ill-posed reasoning tasks remains uncertain.
3. The dual-model co-evolution loop may be expensive to scale, and the paper provides limited discussion of training efficiency.
4. While numerical gains are clear, qualitative examples of how GenPRM critiques or refines reasoning are sparse. For example, any reward hacking examples from the PRM would be helpful.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: Why is there such a large performance gap between LLMOPT (origin) and LLMOPT (reproduce)?

What model is the CoT on line 344 based on?

The StepORLM performance in Tables 2 and 3 is inconsistent with that in Table 1.
soundness: 3
strengths: The proposed method is simple yet effective, achieving performance improvements.
summary: StepORLM introduces a self-evolving 8B-parameter framework that couples a policy model with a generative process reward model (GenPRM) to train LLMs for operations-research tasks. By iteratively distilling dual feedback—solver-verified outcomes and holistic trajectory-level critiques—into weighted DPO updates, the system attains new state-of-the-art accuracy on six benchmarks, while the co-evolved GenPRM further boosts inference-time scaling for both its own policy and external OR models.
weaknesses: GenPRM and Self-Improve are not novel concepts; they have been explored in prior research, eg. [1][2][3][4].

[1] R-PRM: Reasoning-Driven Process Reward Modeling

[2] GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoning

[3] rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

[4] Inference-Time Scaling for Generalist Reward Modeling

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 1
questions: While the proposed method demonstrates promising performance on OR tasks, the paper in its current form suffers from several major weaknesses concerning its core motivation, methodological clarity, and experimental rigor. The overall presentation is not up to the ICLR standard. In order to raise my rating, I would like the authors to address the different points in the major concerns.
soundness: 2
strengths: 1. The related work section is clear and reasonable.

2. The proposed self-evolving framework is reasonable and novel to me.

3. The experimental results are strong, demonstrating the effectiveness of the proposed method.
summary: The paper presents StepORLM, a novel self-evolving framework for LLM-based Operation Research (OR), in which a policy model and a generative process reward model (GenPRM) iteratively improve each other. The framework employs a dual feedback mechanism, combining outcome-based verification from an external solver with process-level evaluation from the GenPRM. The policy model is trained using this combined signal via weighted DPO, while the GenPRM is subsequently refined using new training data generated by the latest policy model. Experimental results demonstrate that the resulting 8B StepORLM model outperforms existing state-of-the-art models across six OR benchmarks, showing the effectiveness of the proposed approach.
weaknesses: ## Major Concerns

1. Unsupported Motivation (Myopic PRM). While the credit assignment problem is intuitive and makes sense, the myopia PRM problem appears weak to me since many existing PRMs generate process-level supervision signals based on the holistic model output. The authors need to provide stronger evidence or clearer citations to support their claim and better differentiate their work from existing PRMs. 

2. There is an apparent disconnect between the paper's claim of using "generative process supervision" and the described implementation. Specifically, according to Algorithm 1 and Equation (1), the detailed, step-by-step process-level correctness scores are seemingly collapsed into a single, global weight $w_{\tau_{w},\tau_{l}}$ for the weighted DPO. This suggests that the fine-grained, process-level feedback is lost and not directly utilized to supervise the model's optimization, which contradicts the paper's central claims.

3. Lack of Clarity on GenPRM Refinement: The methodology for refining the GenPRM is not described in sufficient detail. Key information about the training process is missing, particularly the exact format of the training data.

4. The experimental comparisons in Table 1 are incomplete, making it difficult to fully assess the performance of StepORLM. The missing models include the base model Qwen3-8B, SOTA commercial reasoning models (e.g., OpenAI o-series, Gemini 2.5) and SOTA open-source reasoning models (e.g., DeepSeek-R1, Kimi-K2).

5. Important implementation details are missing, including the number of self-evolving iteration, the prompts for process evaluation and inference scaling experiments, etc.

## Minor Issues

1. L98, “Extensive experiments on seven benchmarks” -> “Extensive experiments on six benchmarks”

2. L298, “ComplexORXiao et al. (2024)”

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The Data Pipeline: The paper must clarify whether the DPO phase re-uses the static 50K SFT instances or dynamically regenerates new ones. The SFT-only model's results must also be provided to quantify the actual gain from the co-evolutionary method.

2. Omitted Work: The authors must discuss the highly relevant concurrent works by Wu et al. (2025) and Lima et al. (2025)  and situate their novelty accordingly.

3. Backbone Choice: The SOTA claim is confounded by the Qwen3-8B backbone. The authors need to provide experimental results on a common backbone (e.g., LLaMA-3-8B) to enable a fair comparison against baselines like ORLM.
soundness: 3
strengths: **Significant Adaptation of Process Supervision:** The paper’s primary strength is its effective adaptation of process-based supervision to the OR domain. By proposing a GenPRM to evaluate entire reasoning trajectories holistically , it aims to mitigate the known limitations of simple outcome-based rewards (the credit assignment problem) and myopic step-wise checks.

**Well-Motivated Co-evolutionary Design:** The paper proposes simultaneous refinement of policy and GenPRM, which is conceptually sound and avoids a static critic. Table 3 shows that freezing GenPRM reduces performance by 3.7% on average, validating the co-evolution mechanism.

**Strong Empirical Results: ** The framework's quality is supported by SOTA performance across six benchmarks. The 8B-parameter StepORLM is shown to outperform vastly larger generalist models and other specialized baselines.
summary: This paper proposes StepORLM, a self-evolving framework for modeling OR. It aims to address the credit assignment problem, where flawed reasoning might lead to a correct outcome , and the limitations of myopic process supervision that overlooks step interdependencies. The framework's core is a co-evolutionary loop between a policy model and a GenPRM. This loop uses dual feedback from an external solver for outcome verification and the GenPRM for holistic process evaluation. The paper reports that its resulting model achieves state-of-the-art results across six benchmarks and that the GenPRM can be used as a process verifier to improve the inference performance of other LLMs.
weaknesses: **Unclear Attribution Due to Ambiguities in the Data Pipeline:** The paper's SOTA claims are complicated by ambiguities in its data strategy. The 50K-sample SFT dataset is significantly larger than baselines like ORLM (30K) , and Figure 4 shows this SFT-only model is already exceptionally strong . This makes it difficult to attribute the final SOTA performance to the co-evolutionary method (Stage 2) rather than just the high-quality initial data (Stage 1). This ambiguity extends to the DPO phase. Section 3.3.1 is unclear whether the DPO phase re-uses these static 50K instances or regenerates new ones , hindering reproducibility.


 **Omission of Relevant Concurrent Work:** The literature review overlooks highly relevant concurrent work that addresses the same problem of intermediate-step validation. Specifically, Step-Opt[1] is directly relevant as it proposes a "Stepwise Validation Mechanism" with explicit checkers for variables, constraints, and programs . This structured, explicit validation is a conceptually similar (though different in approach) solution to the same problem that StepORLM's learned GenPRM aims to solve. Additionally, Lima et al.[2] on verifiable synthetic data is also relevant and should be discussed .

**Backbone Choice Obscures Method Effects:** The paper's SOTA claims are ambiguous. StepORLM is built exclusively on Qwen3-8B , while principal baselines (like the 8B ORLM) rely on different 8B architectures (LLaMA-3-8B). Without cross-backbone controls (e.g., training StepORLM on the baseline's backbone), the comparison fails to isolate the method’s contribution from the backbone’s.

[1] Wu Y, Zhang Y, Wu Y, et al. Step-Opt: Boosting Optimization Modeling in LLMs through Iterative Data Synthesis and Structured Validation[J]. arXiv preprint arXiv:2506.17637, 2025.

[2] Lima V, Phan D T, Kalagnanam J, et al. Toward a trustworthy optimization modeling agent via verifiable synthetic data generation[J]. arXiv preprint arXiv:2508.03117, 2025.


Decision: Accept (Poster)