OR-PRM: A Process Reward Model for Algorithmic Problem in Operations Research
Authors: Yilin Wang
Heng Zhou
Dongxing Mao
Linjie Li
Jingru Tan
Haochen Han
Zhengyuan Yang
Alex Jinpeng Wang
Min Li
Venue: ICLR 2026 Poster (2026)
Keywords: Operations Research
Process Reward Model
Large Lanugage Model
Primary area: optimization
TL;DR: A Process Reward Model (PRM) boosts LLM reasoning in operations research by reinforcing logically correct steps—enabled by the first step-by-step supervised OR dataset.

Abstract: Large language models (LLMs) with Process Reward Models (PRMs) have shown strong reasoning ability, yet their potential in Operations Research (OR) remains unexplored. We present the first PRM tailored for OR, but find that directly training on mainstream datasets yields surprisingly weak performance.
To understand this gap, we conduct a systematic analysis and identify the primary bottleneck: the datasets themselves, where over 30\% of annotations are severely flawed.
To overcome these limitations, we first collect all existing synthetic datasets and apply a carefully designed filtering pipeline to construct a high-quality seed dataset. 
Building upon this seed, we then build OR-ProcessQA, the first large-scale dataset for OR with step-by-step supervision, where diverse solution pathways are generated via Monte Carlo Tree Search (MCTS) and each step is validated for logical consistency by GPT-4o.
Building on this foundation, we train OR-PRM, the first Process Reward Model in the OR domain, designed to evaluate and guide reasoning at every step rather than only the final outcome.
Together, these advances enable OR-PRM to substantially improve LLMs’ reasoning capability, achieving a maximum absolute improvement of 12.5\% over the base model in Best-of-N settings, and highlighting the power of process-oriented supervision for reliable problem solving in operations research.

OR-PRM: A PROCESS REWARD MODEL FOR ALGO- RITHMIC PROBLEM IN OPERATIONS RESEARCH
Yilin Wang ' Heng Zhou Dongxing Mao?, Linjie Li Jingru Tan?, Haochen Htn Zhengyuan Yang', Alex Jinpeng Wang" Min Li
(yilin. wang, hengzhouj @zju.edu.cn m962479949 1962479944@2mmiil.com
[tanjingru, jinpengwang liminmin] @esu.edu.cn. (lindsey.li, zhengyang) @microsoft.com hhc2077 @otloook.com
Zhejiang University, CCentral South University, 3 Miicoosoft, "Pengcheng Laboratory
"Equal Contribution,
Cooresponding Authors
ABSTRACT
Large language models (LLMs) with Process Reward Models (PRMs) have shown strong reasoning ability, yet their potential in Operations Research (OR) remains unexplored. We present the first PRM tailored for OR, but find that directly train- ing on mainstream datasets yields surprisingly weak performance. To understand this gap, we conduct systematic analysis and identify the primary bottleneck: the datasets themselves, where over 30% of annotations are severely flawed. To overcome these limitations, we first collect all existing synthetic datasets and ap- ply carefully designed filtering pipeline to construct high-quality seed dataset. Building upon this seed, we then build OR- PRRProcessQA, the first large-scale dataset for OR with step-by-step supervision, where diverse solution pathways are gener- ated via Monte Carlo Tree Search (MCTS) and each step validated for logical consistency by GPT-4o. Building Process Reward Model in the OR domain, designed to evaluate and guide rea- soning every step rather than only the final outcome. Together, these advances enable OR-PRM to substantially improve LLMs reasoning capability, achieving a maximum absolute improvement of 12.5% over the base model in Best-of-N settings, and highlighting the power of process-oriented supervision for reliable problem solving in operations research.
this foundation,
train OR-PRM, the first
INTRODUCTION
Large Language Models (LLMs) DeepSeek-Al (2024); Yang al. (2025a) have recently demon- strated strong reasoning ability, largely attributed to post-training methods such as reinforcement learning and Process Reward Models (PRMs). Their rapid progress is evident across challenging domainsfor instance, GPT-5 has already surpassed all human competitors in the 2025 ICPC World Finals OpenAl (202).. notoriously difficult zero-shot programming contest. These advances sug- gest that I.MMs are no longer merely fluent generators, but are evolving into powerful engines for rigorous problem solving.
Operations Research (OR) provides especially compelling testbed for such reasoning, it in- volves modeling and solving complex real-world decision-making problems using mathematical optimization, simulation, and analytical methods to efficiently allocate scarce resources and maxi- mize performance within constrained systems. Solving OR problems demands not only correctness in the final answer, but also step-by-step logical consistencya natural match for PRMs, which are designed to explicitly evaluate the correctness of intermediate steps. At first glance, it seems natural expect PRMs to excel in OR just as they do in mathematics programming.

---

Problems with existing datasets
Seed Dataset
Set Routes: R 1A, C). Rasources: [Trucks, Driees)...
Question: Accoppaty comp
1. IInfeasible Problem
Parameters: Costhue 1900, 700) Costoeer [300, 350, 400)
Hin priving of Firoo Books Book Blokk
eg. The adivar returs inkoa be
All Verisbles: Number of tnicks on muses A, B. C: F
Objective: Min Cost 1900, 100%s
Modeling Error
Constsinta: 5, it
-%, Xg
Modeling:
og model incerpeeen in ieparnanttee
Al Code: from pyomo envron import "trin...
Inear oggann agaamiing recoll to musambse the risiiiingg mode
OR-ProcesSQA
famuldeed follws:
3. C.do Error
[Parameter Definitons] The decision variebrre in osrnet. in [Objectives Constraints) The objecive and budger constraints are correct but cxclusivity constraitts are improporty formualedd in [Generated Code] Function 'Rul' expects two arguments ( prop3 prope ( but only rece one props cousing Typafiomr in [Execution Output] Since the code falls, is no valid output The message indicates a falure constructing the exclustvit constraint. [Correct step] Code implementatio: from pyomo envzon import...
undetiedd tadiee maza li vasss logicall lagg.
Inpan: COPT Create COPT
seup cocklt
Differeent Solvers vp.
intrommen
of raage derasel...
Figure 1: Noisy Data (left) vs. Our Data (right). The left panel illustrates common issues in existing datasets, such as infeasible problems, modeling errors, and coding defects. The right panel showcases well-structured seed data, which serves the foundation for our OR- OR-ProcessQA dataset, characterized by step-by-step solutions with explicit correctness labels and ground-truth corrections.
Yet this expectation does not hold. When we developed the first PRM tailored for OR, its perfor- mance was far weaker than anticipated, even with state-of-the-art LLM backbones. Our analysis shows that the main obstacle is data quality, since existing OR datasets alarmingly unreliable. In the Industry OR dataset, even more than 30% of the samples contain serious errors in the final answer, and as with other datasets, many include incomplete or noisy reasoning steps (Figure D one dataset, even, More than 30% of the samples contain serious errors in the final answer, and many include incomplete or noisy reasoning steps (Figure [T. This noise makes it extremely difficult for PRMs to learn faithful reasoning, leading to solutions that look plausible but often violate hidden constraints or break logical consistency.
To overcome these challenges, we first curated a high-quality seed dataset through a rigorous three- stage filtering pipeline. Building on this foundation, we combined MCTS for solution exploration with GPT-4o for fine-grained step-wise annotation, generating hundreds of thousands of problem- solution trajectories. After strict consistency checks, this process yielded OR- RRoPrrcessQA, the first large-scale OR dataset with reliable step-level supervision for training PRM.
Leveraging this resource, we developed OR-PRM, the first Process Reward Model tailored for Op- erations Research. Unlike conventional PRMs that collapse reasoning quality into single scalar score, OR-PRM delivers structured feedback by categorizing errors and offering targeted correc- tions. This design enables it to evaluate not only the correctness of final answers but also the validity of every intermediate step. By distinguishing between correct code, incorrect yet runnable code, and non-runnable code, OR-PRM provides actionable guidance for refinement. Our experiments demon- strate that such feedback substantially improves the logical consistency and rule-following behavior of LLMss marking an important step toward trustworthy cpeision-making in OR applications.
Overall, our contributions are three-fold: ① We introduce OR-PRM, the first Process Reward Model tailored for Operations Research, trained to evaluate and guide reasoning at every step rather than relying solely on final answers. @ We curate high-quality seed dataset by filtering existing syn- thetic OR data, and further expand it with MCTS exploration and GPT-4o annotations into OR- ProcessQA, the first OR dataset with reliable step-level correctness labels for training PRM. 3 We empirically demonstrate that process-oriented supervision with OR-PRM substantially improves the logical reliability and correctness of LLMs in OR tasks (e.g., achieving average 12.5% accuracy gain on six benchmarks), paving the way toward trustworthy decision-maaing in real-world applications.
2 RELATED WORK
LLMs for Operations Research The remarkable capabilities of LLMs in natural language derstanding and complex reasoning have propelled their application in operations research recently. A core challenge lies in effectively translating these naturally described optimization problems into
2

---

precise mathematical models that solvers can process. Current academic exploration primarily fol- lows two technical paths Xiao et al. (2025): One path involves reasoning-en enhannee methods, which guide general-purose LLMs in modeling through carefully designed prompts. Examples include X- of-Though approaches (e.g.. the tree-search reasoning employed by Autoformulation Astorga et al. (2025) ) and Multi-Expert system (e.g., Chain-of-Expern Xiao et al. (2024) and OptiMUS Ahma- diTeshnizi et al. (2024)). The second path focuses on domain -speciiic fine-tuning, where models are fine-tuned on specialized datasets enhance their professional capabilities. Studies such as ORLM Huang et al. (2025a) and LLaMoCo Ma et al. (2024) have demonstrated that fine-tuned models can outperform ceneral-purpose LLMs like GPT-4. Building on this, the LLMOPT Jiang et al. (2025) further advances this direction by introducing the five-element formulation as universal problem definition paradigm and employing Sahneman-Tverske Optimization (KTO) for model alignment,
improving the model's generalization ability.
Data Synthesis for Operations Research However, both technical paths above are highly depen- dent high-quality datasets. Consequently, researchers have begun exploring data synthesis tech- niques, broadly categorized into problem-centric and model-centric approaches Xiao et al. (2025). The former, exemplified by OR-Instruct Huang al.] (2025a), augments data by modifying exist- ing problems. The latter prioritizes generating models first and then inversely constructing problem descriptions, thereby offering better control over difficulty and correctness. For instance, the Re- Socratic Yang et al. (2025b) method generates problems inversely from formalized proof', while OptiMath Lu et al. (2025) and MILP-Evolve Li et al. (2055) generate directly from model code or types. Concurrently, the academic community has released several evaluation benchmarks, includ- ing NL4Opt, MAMO, and IndustryOR. Yet, recent studies have uncovered a surprisingly high crror rate in these widely used benchmarks (with some datasets exhibiting error rates exceeding 50%) Xiao et al. (2025), , severely compromising the reliability of evaluations. Addressing this bottleneck of data quality, this study innovatively clean and construct batch of high-quality optimization mod- eling data, laying a solid foundation for training and evaluating more reliable optimization models.
Process Reward Models Process Reward Models Cobbe et al. (2021): He et al. (2024); Zhang
et al. (2025b a) provide process-level supervision by scoring intermediate reasoning steps, guiding models to reason step-by-step with improved logical consistency and accuracy. Building on this capability, PRMs have been successfully applied to Best-of-N sampling Wang et al. (025) and offline data selection Xie et al. (2023), significantly improving reasoning quality and model opti- mization. Representative works such as Skywork-PRM]
al. (2024) and Qwen2.5-Math MaM Zhang et al. (2025b) combine human annotations with synthetic rewards to evaluate performance across mathematics, science, and programming domains. They often fail out-of-distribution rea- soning. Zhu et al. (2025) address this with RetrievalPR.M, a process reward model using question- and step-level retrieval improve generalization. Beyond general domains, PRMs are also being extended to vertical domains; for instance, Fin-PRM Zhou et al. (2025) adapts PRMs to finance with trajectory-awwe awarr, Nomain-sppciallin domain-specific knowledge; therefore, we synthesized dataset and conducted training tailored to the characteristics of the Operation Research.
reward modeling. Applying PRM to vertical domains requires
METHODOLOGY
Our method tackles the core challenges of applying LLMs to Operations Research through a three- stage pipeline, as shown in Figure 2] We begin by establishing a robust data foundation. Firstly, we construct a high-quality seed dataset in Section 3.1.1 to mitigate data noise and inconsisten- cies. Next, we build the OR-ProcessQA dataset in Section 3.1.2] which provides the first process- supervised data in the OR domain with fine-grained, step-level annotations. Finally, we develop the Process Reward Model for OR domain (OR-PRM) in Section 3.3 This specialized PRM offers natural language critiques and corrections beyond scalar scores for OR reasoning steps. Our ap- proach significantly enhances the reliability and performance of LLMs in OR by providing detailed, interpretable feedback throughout the solution process.
3

---

((1) OR-ProcesS Dataset Construction
Modeling Process
Data selection and synthesis
K-greedy
RAW
Solver Code
Selection
Selected
Policy Model
Synthesis
OR-ProceeSDA
b. Synthesis
Low-to-High Selection
Seed Data
Paramelars Variables
Objective Constraints
Solver Code
Mome Carlo
Tree Search
Stage
Stagre
Stage 1
Modeling
Satisfed Constraints
Executable Code
(2)
OR-PRM Model Training
Step
Error Analysis
Data Diversilication
Gvi pposeet: JSspt Stapi constaaee vrvorg sieaate ahe ewtin pedepiipilyy silick
Rollout
Stepp,
Fine-tuning
Rollout
St Step3...
??
Rollout
St
Data Processing
OR-ProcessOA
Post
Figure 2: Overview of our automated framework. We first construct OR-ProcessQA through three-stage filtering pipeline and MCTS-based trajectory generation with step-level verification. Built on this dataset, OR-PRM is trained to provide structured, stepwise feedback.
3.1 DATASET CONSTRUCTION
high-quality dataset is essential to ensure the effectiveness of PRM supervision. We propose a stricter way build the dataset. Specifically, we first create cleaner seed dataset by careful filtering and many rounds of checking in Section Then, we utilize this curated seed dataset to generate diverse and accurate process- annotated data in Section 3.1.2
3.1.1 SEED DATA CONSTRUCTION
In this section, we first standardize the problem representation for consistent generation. We then employ existing strong OR model, LLMOPT Jiang (2025), for solver code generation. Finally, we adopt a multi-stage procedure to filter out high-quality data.
Problem representation. We adopt LLMOPT a generative policy that first produces each prob- lem in the canonical five-element tuple form (S, 0 æ. f((x), (xx) I c), ensuring compatibility with downstream validation and modeling stages. This policy-based generation ensures a mathematically well-formed and solver-agnostic structure from the start.
To enable consistent modeling and automated validation, represent each problem p via a compact five-element tuple:
(S, O, x, f(x), g(x) S c).
where (index sets), (parameters), x (variables), f(x) (objective), and g(x) < (constraints) collectively define the optimization task in canonical form min f(x) s.t. g(x) S This schema ensures solver-agnostic structure, enabling deterministic code-output validation against declared constraints and objectives, which is critical for scalable, error-free seed dataset construction.
Solver Generation. We directly use ILMOPT to auto-generate solver code tailored for each prob- lem tuple, linking the mathematical formulation directly to executable implementation.
Multi-Stage Validation. Each generated sample is then subjected to a three-stage validation pipeline ensure high-quality reasoning. Samples were evaluated along three axes: code execution, constraint satisfaction, and modeling accuracy, and were discarded if they failed any stage.
Code Execution: We execute the provided code and verify that it runs without error and produces the expected output. This validates the code's executability and establishes the
4

---

resulting numerical solution È the ground truth for subsequent constraint satisfaction checks.
2. Constraint Satisfaction: We employ Qwen3-8B Yang et al. (2025a) as reasoning verifier: given the constraint expressions g(x) I c from the five-element tuple and the numeri- cal solution & produced by the solver code, it performs symbolic numeric substitution to verify whether constraints are satisfied. This enables automated, groundeg feasibility checking without requiring additional code generation.
3. Modeling Accuracy: Finally, we use GPT-4o to validate whether the mathematical formula- tion accurately reflects the original problem statement. This ensures the five-element tuple (S, 0, x. f.g) faithfully captures the problem semantics.
A sample is retained if and only if passes all three validation stages: successful code execution, constraint satisfaction, and modeling accuracy. This integrated, generative process gave us a clean, reliable seed dataset.
3.1.2 STEP-WISE ANNOTATION GENERATION
Seed data can only support SFT but not PRM training, so we further expand into step-wise trajecto- ries and annotate them, obtaining a high-quality dataset suitable for PRM supervision. Specifically, this process consists of three parts: (I) automated step generation via MCTS based on the seed problems; (2) structured evaluation of each step using GPT-4o to identify potential errors; and (3) consistency filtering between MCTS and GPT-4o outputs to retain only logically sound trajectories.
Automated Annotation via MCTS. Following OmegaPRM Luo et al. (2024), apply MCTS to problems from our seed dataset to sample solution trajectories. Correct steps are labeled 1.0, while the first error in any failed path is labeled 0.0. This process yields dataset of over 550K annotated steps.
Structured Error Analysis with GPT-40. To enhance reliability, we employ GPT-4o to systemati- cally re-evaluate every candidate reasoning step. The model inspects each component in predefined sequence: (1) parameter definitions, (2) objectives and constraints, (3) generated code, and (4) code execution output. Upon detecting the first error, it halts further analysis and outputs four structured fields:
Issue: A natural language description of the error:
Judgement: A binary label Correct or Incorrect;
Corrected Version: The fixed content of the erroneous component;
Corrected Step: The complete, revised reasoning step incorporating the fix.
This structured analysis ensures consistent, interpretable, and actionable feedback for training and refinement.
Consensus-based Filtering. We employ a dual-validation mechanism to curate the final training set. sample is retained only if Labelıcts(s) = LabelGFT4a(s) where Label denotes the binary validity label (correct or incorrect) and s is the reasoning step.
Through this pipeline, we obtaín high-confidence annotated samples, which constitute our final dataset: OR- RR-PPrcess-QA. This dataset strikes balance between scale and precision, effectively supporting OR-PRMs fine-grained reward modeling and step-wise error correction capabilities.
3.2 GENERATIVE PRM FOR OR PROBLEM
Traditional PRMs often output scalar score to represent the judgment. They employ a step-wise evaluation method. First, scalar score is assigned to each reasoning step in response. These scores are then aggregated, through methods like a weighted sum or by taking the minimum value, to caleulate the final reward. However, traditional PRMs typically assign only scalar value per step. This is not enough for complex tasks like operations research.
Such tasks require detailed analysis of variable relationships (e.g.. over S), constraint satisfaction (g(x) < c), and logical structure of the objective fxx). Furthermore, while finding problems like syntax emors in code generation depends the generation abilities of large language models,
5

---

simple score is not enough properly catch these potential issues especially when the code must align with the canonical form ming f(x) s.t. g(z) S c.
Generative PRM replaces binary labels such as correct or incorrect with natural language judg- ments. During inference, the model generates a textual critique and judgment for cach reasoning step, enabling interpretable and step-by-step evaluation. Inspired by GM-PRM (ra.Zaanng et al. (2025a), we adopt generative process reward modeling approach tailored for operations research tasks. In- stead of assigning scalar scores reasoning steps, our model generates natural language critiques and judgments for each component of the solution. This enables fine-grained, interpretable evalua- tion grounded in domain comain-specific logic.
Concretely, given an optimization problem = (S, x, f(x), S c) and its step-by-ster solution, the model analyzes four key components in sequence: (1) variable definitions (æ over S, parameterized by 0), (2) objective f(x) and constraints g(x) I c, (3) code implementation (if present), and (4) final output. For each, produces brief intent statement, focused analysis of critical issues, and binary judgment correct incorrect. If any component is judged incorrect, the model outputs a corrected version of the first flawed section only.
3.3 TRAINING OBJECTIVE
Our training process structured in two main stages, to leverage our OR-ProcesSAA dataset ef- fectively. We first use Supervised Finetuning (SFT) to teach the model the fundamental format of generating critiques, followed by an Aligmment phase with Direct Preference Optimization (DPO) to refine its logical judgment.
3.3. SUPERVISED FINETUNING
The first stage, SFT, adapts base model to the generative PRM task. The primary goal of SFT to teach the model the correct format, style, and step-by-step reasoning process required for OR problem-solving.
Specifically, the model trained on our high-quality annotated samples using standard auto- regressive next-token prediction objective. The input consists of problem description and candi- date solution, while the target the complete, structured critique generated during our data annota- tion pipeline (Section 3.3.2). The SFT loss function, Lser, is defined
LSFT (0) = (Z(
Σ log Po
where y represents the target sequence containing the four structured fields: Issue, Judgement, Corrected Version, and Corrected Step. This process teaches the model to perform the fine- grained, step-wise error analysis and correction that defines our generative PRM.
3.3.2 ALIGNMENT
Supervised fine-tuning results in correctly formatted steps but lacks logical reliability. This is be- cause the model simply imitates examples without deeper understanding. To address this, we use an alignment phase. This phase employs DPO to promote true logical reasoning.
Direct Preference Optimization We leverage our OR-ProcesSQA dataset in conjunction with outputs from the SFT model: we re-run inference using the SFT model, identify failure cases (i.e., where the model produces incorrect inferior reasoning), and construct preference pairs (æ, Uwr v1) accordingly. For each prompt x, Yw is the correct or superior reasoning step, while y is the flawed step generated by the SFT model.
DPO directly optimizes the language model policy, To. to increase the likelihood of the preferred responses over the dispreferred ones, relative reference policy, meet- The DPO loss function is given by:
To ( - 3 log (Blog
ne (3112²) ( x))
CDPO(TT: )eet) = Be wwD log
(2)
T Tver(Ven (x)

---

where 3 is temperature parameter controlling the strength of the preference, and a(-) is the logistic function. This loss aligns the model with correct reasoning without requiring a separate reward model.
4 EXPERIMENTS AND ANALYSIS
In this section, we introduce our experimental setup for OR-PRM in Section 4.1 We then assess its performance in two distinet settings, discussed in Section 4.2 Finally, we present ablation studies in Section 4.3
4.1 EXPERIMENTAL SETUP
Model. We evaluated the performance of OR-PRM when applied to several leading language mod- els, including the Qwen2.5 series (7B, 14B, and 32B) and I.LMOPTjiang et al. (2025), a specialized model tailored for Operations Research. We chose Qwen2.5 because it offers complete range of model sizes, enabling us to study scaling effects, and because it has demonstrated strong reasoning capabilities and wide adoption in recent LLM research.
Benchmark.
We evaluated the model performance on a set of optimization benchmarks. However, prrors/Xao et al. (2025); Jijng et al. (2025) even benchmarks in Operation Research contain serious errors Xiao
To provide fair evaluation and preventing misleading answer, we utilized cleaned benchmarks from
Xiao et al. (2025) to ensure the reliability of our results including Industry OR Huang et al. (2025a).
Ea Huang et al. (2025b), Complex-LP Huang et al. (2025b), NL4LP Ahmadi Teshnizi et al.
2024
NL4OPT Ramamonjison et al. (2022).
Owen2.5-7B-Codle base model. The training process was conducted in two stages on eight Nvidia A100 GPUs using DeepSpeed ZeRO-2 and 6 precision. First, we perform supervised finetuning with learning rate of 2e-5. Following this, the model undergoes Direct Preference Optimization (DPO) with a learning rate of 4e-5 and beta of 0.2. A per-device batch size of 2 is applied in both training stages.
Training Details To train OR-PRM,
Inference Details. We evaluate OR-PRM under two complementary inference settings. The first focuses on selection, where multiple candidate reasoning paths are generated and OR- PRM identifies the most reliable one (Best-of-N sampling). The second emphasizes refinement, where OR-PRM critiques intermediate steps and guides the model toward improved solutions Conoeeiingrittiqueeeennn
pipeline). For evaluation, correctness is verified numerically, and because many problems admit multiple solution paths, we compare only the final optimal value when reporting performance.
BEST-OF-N SAMPLING. By default, we set N=8. The model generates distinct Chain-of- Thought (CoT) Wei et al. (2022) reasoning paths with temperature 1.0. OR-PRM evaluates each reasoning step in every path as correct or incorrect, and selects the path containing the highest number of correct steps, favoring the most coherent and accurate reasoning trajectory.
MODELING, CRITIQUE, AND CODE GENERATION PIPELINE. In this setting, the base language model follows a structured three-stage workflow, guided by OR-PRM. First, the model constructs formal problem modeling with step-by-step reasoning. Next, OR-PRM critiques each reasoning step by identifying potential errors or inconsistencies. Finally, the original modeling and its cri- tique are concatenated and fed back into the model guide the generation of executable Python code that satisfies predefined input-output specifications. This process enforces self-correcting, implementation-an aware reasoning trajectory through iterative feedback.
To thoroughly assess the efficacy of our proposed pipeline, employed two primary evaluation metrics: @ 1, which measures the first-attempt correctness and reflects the model's immedi- ate problem-solving capability: and pass@8, which evaluates the upper-bound potential when the model is allowed up to eight attempts, thereby revealing its capacity for self correcore and iterative refinement within given search space.

---

Model
IndustryOR Easy-LP Complex-LP NLALP NL4OPT ReSocratic Overall
Proprietary Models
GPT-4o
40.5
69.5
35.1
56.2
53.1
47.9
50.4
Deepseek-v3
66.7
91.9
39.6
92.7
76.5
73.9
73.6
Open Open-source source Models
Owen-2.5-7B
19.0
49.7
12.6
50.0
41.3
36.7
34.9
+PRM
23.8
61.8
16.2
56.7
52.1
46.7
42.9
+4.8
+12.1
+3.6
+6.7
+10.8
+10.0
+8.0
Qwen-2.5-14B
35.7
66.2
3.6
75.8
61.0
50.4
48.8
+PRM
45.2
89.4
12.6
86.5
67.6
66.7
61.3
+9.5
+23.2
+9.0
+10.7
+6.6
+16.3
+12.5
Qwen-2.5-32B
47.6
80.0
8.2
87.1
68.5
66.3
59.6
+PRM
57.1
96.0
32.4
89.3
74.2
72.7
70.3
49.5
+16.0
+24.2
+2.2
+5.7
+6,4
+10.7
LLM-OPT
52.4
96.0
48.6
90.4
81.7
72.2
73.6
+PRM
59.5
97.8
67.6
93.8
85.0
79.2
80.5
+19.0
+3.4
+7.0
+6.9
Table Results on Six Reasoning Benchmarks. Experimental results demonstrate that using OR- PRM the critic model significantly enhances reasoning performance under the Best-of-8 evalua- tion strategy. The line in blue indicates performance improvement.
4.2
MAIN RESULTS
Best-of-N Sampling. As shown in Table OR-PRM consistently and significantly enhances rea- soning performance across different scales of the Qwen model family. It achieves uniform gains on the Qwen2.5 Yang et al. (2024) series (7B32B) and the specialized model LLMOPT Jiang el al. 2025 demonstrating its effectiveness and strong scalability with respect to model size. No- tably, on the 14B model, OR- achieves the highest average improvement of nearly 12.5%.
Moreover, the performance gains introduced by OR-PRM are consistently evident across tasks of varying difficulty levels. On the most challenging Complex-LP benchmark, Qwen2.5-32D attains an impressive absolute improvement of 24.2% For relatively easier benchmarks such as Easy-LP, the 14B model achieves substantial gains of 23.2%. Even for LLMOPT, model already extensively op- timized for reasoning and exhibiting strong performance on difficult tasks, OR-PRM contributes an additional 19.0% improvement on Complex-LP. These results further subslantiate the effectiveness of OR- PRM in accurately identifying and prioritizzin high-qualing reasoning steps under demanding conditions.
Results of demonstrates remarkable performance enhancements across both the prominent open-source model Qwen-2.5-145 14B and the advanced closed-soures model GPT-4o.
Pipeline. As shown in Figure 3 OR-PRM consistently
The most substantial improvements are particularly evident on the challenging Complex-LP bench- mark, underscoring potent ability of OR-PRM to tackle intricate problems. The pass @ accuracy for Qwen2.5- 14B surged by an impressive 23.4%, while even the state-of-the-an art GPT-4o achieved notable increase of 8.1 %. The gains in pass 8 are also notable: Qwen2.5-144 14B witnessed signif- icant rise 36.16 1%, and GPT-4o improved by 6.3%.
These gains underscore ability of OR-PRM to raise the reasoning ceiling by effectively recovering correct solutions from initial failures. Even when the first attempt falters, OR-PRM enables iterative correction, enhancing robustness under uncertainty and complexity. On the simpler Easy-LP bench- mark, still yields consistent 24% improvements, demonstrating reliability across task difficulty.
At the heart of OR-PRM is its critic componentan intelligent feedback loop that evaluates each reasoning step. It reinforces correct steps and precisely diagnoses errors, offering targeted guid- ance rather than binary judgments. This fine-grained feedback helps the model iteratively refine its reasoning, much like a human learner, leading to notable accuracy gains. Such interactive error correction is key to broad effectiveness of OR-PRM across models and tasks.
8

---

100%
Qwen2.5-148 (w/o critic)
100%
54.7%
37.0%
Qwen2 n225-118 (w/o critic)
Qwen2.5-144
critic) GPI-4o (wio critic) GPT-4o (W critic)
GPT-40 (w/o critic)
80%
GPT-40 (w critic)
80%
70.0%
72.0%
Accuracy Score (5)
Accuracy (%0
59.3% 69.4%
60%
43.2%
35.1%
23.4%
3.6%
Easy-LP
Complex-LP
Easy-LP
Complex-LP
(a) pass Accuracy across models.
pass@8: Accuracy across models.
Figure 3: OR-PRM enhances optimization ability across models. It consistently improves per- formance on both open-source (Qwen2.5-14B) and closed-source (GPT-40) models, and enables solving problems that remain unsolved even with 8 samples.
4.3 ABLATION STUDIES
In this section, we analyze the effectiveness of model alignment via DPO and examine performance trends across task difficulty levels. The results are presented in Table
Method
Easy-LP
Complex-LP
Average
P
94.7%
23.4%
59.1%
self-consistency
50.8%
3.6%
27.2%
OR-PRM (Ours)
89.4%
12.6%
51.0% %
OR-PRM (SFT)
79.6%
6.3%
43.0%
Qwen2.5 (Zero shot)
72. 12.1%
9.9%
41.0%
self-consistency (filtered null)
88.3%
9.9%
49.6%
Table Ablation results. Results on Qwen2.5-14B.
Effectiveness of Model Alignment Our ablation study confirms the effectiveness of Direct Pref- erence Optimization (DPO) within the OR-PRM model training. As shown in Table the full model incorporating DPO top of SFT achieves an average accuracy of 51.0%. This represents an 8.0% absolute improvement over the SFT-only baseline (43.0%), demonstrating DPO's crucial role in im- proving model. Other baselines include the Qwen2.5 (Zero shot) model, which represents the raw base model performance without any SFT DPO training, and the self-consistency (filtered null) approach, which performs majority voting on the N = 8 paths after filtering out those that fail to produce valid numerical objective value.
Performance Across Task Difficulty Levels shown Table OR-PRM consistently outper- forms the Major Voting baseline across both easy and challenging benchmarks. This performance demonstrates that OR-PRM has the ability to detect a significant majority errors within reasoning paths across both easy and challenging benchmarks.
4.4 DISCUSSION
We further discuss the limitations in current training data and fine-grained discrimination capability, with future directions outlined below.
Our OR-PRM performs well the OR- OR-ProcessQA dataset. However, it is hard to provide a comparison, as existing datasets cannot be used for PRM training. Furthermore, our Best-of-N performance is strong, but it still falls short of the theoretical upper bound. This performance gap is

---

mainly attributed to the current size of our dataset and model. Therefore, will expand the training data in the future, to make the model better at detecting subtle reasoning errors.
CONCLUSION AND LIMITATION
In this work, we introduce OR-PRM, the first Process Reward Model (PRM) tailored for Operations Research (OR), designed to address the core challenge of reliable LLM reasoning in this domain. Our investigation revealed that the primary obstacle to developing such a model was the pervasive unreliability of existing OR datasets, which prevents PRMs from learning to accurately distinguish between valid and invalid reasoning steps. To overcome this fundamental data bottleneck, we first curated high-quality seed dataset and expanded into OR-ProcesSQA, the first OR dataset with reliable, step-level correctness annotations. This provided the essential foundation for our model. Building on this unique resource, OR-PRM delivers structured, step-level feedback rather than single scalar score. Experiments demonstrate that our approach is highly effective. OR-PRM sub- stantially improves L.LM performance, yielding an average 12.5% gain in the Best-of-N setting and notable robustness when serving as critic during inference. These results underscore the value of process-oriented supervision for LLM reasoning in OR, suggesting a promising direction for de- veloping more trustworthy AI in other domains that require verifiable, step-by-step logic. Indeed, these successful results affirm the foundational value of our dataset. However, we also acknowledge current limitation: the lack of datasets compare. Therefore, to enhance the credibility of research findings and support broader applications, we plan to further expand and refine our dataset, including by increasing the diversity of problem types and solver environments.
ACKNOWLEDGMENT
This work supported in part by the National Natural Science Foundation of China (Grant No. 6250074347), and the Major Key Project of PCL. under Grants PCI.2025AS10 and PCL,2024006.
10

---

ETHICS STATEMENT
This work focuses improving the reliability of large language models (LLMs) in Operations Re- search (OR) through process-oriented supervision. No human subjects were directly involved in data collection. Our dataset, OR-ProcessQA, is derived entirely from synthetic sources and existing pub- lic benchmarks, followed by automated filtering and GPT-4o verification. All data are anonymized, contain no personal sensitive information, and comply with open licensing terms of the source datasets.
Potential risks include the possibility of misuse of OR-capable LLMs in high-stakes decision making (e.g., logistics, finance, or defense). To mitigate such risks, our method emphasizes correctness, transparency, and logical consistency, making model outputs more interpretable and auditable. We also release detailed dataset construction protocols to encourage responsible use.
We declare that there are no conflicts of interest or external sponsorship that might unduly influence the presented results. This research adheres to the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure reproducibility.
Dataset:
The construction pipeline for the high-quality seed dataset and OR-ProcessQA is fully described in Section 3.2, with additional filtering rules and statisties detailed in the Appendix.
Models:
The architecture and training procedure of OR-PRM are explained in Section 3.3, with hyperparameters, optimization details, and ablation results provided in the supplemen- tary materials.
We will release anonymized source code, dataset filtering scripts, and training configurations supplementary material.
Code & Resources:
Evaluation: All metrics, baselines, and Best-of-N setups are documented in Section 4 and Appendix.
These resources, combined with detailed documentation, ensure that independent researchers can reproduce the reported results.
REFERENCES
Ali AhmadiTeshnize Wenzhi Gao, and Madeleine Udell. OptiMUS: Scalable optimization model- ing with (MI)PP solvers and large language models. In Forty-first International Conference Machine Learning. 2024. URL (tth:t::: httttt:://ppeeeeeeeeeeee opwpreveeeeeeeeeepp http://www.ee/oouummmeemm htr////ormmmm -
Nicolás Astorga, Tennison Liu, Yuanzhang Xiao, and Mihaela van der Schaar. Autoformulation of mathematical optimization models using LLMs. In Forty- International Conference on Machine Learning. 2025. /
www.review..ecp net.ieewwwe.eeeeee..eee..ee.eeee net http://wwweee//oomumld333333/
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10,166, 2021.
DeepSeck-AL. Deepscck-v3 technical report, 2024. URL http:::///arvvv.og//////// 19437
Jujie He, Tianwen Wei, Rui Yan, Jiacai Liu, Chaojie Wang, Yimeng Gan, Shiwen Tu, Chris Yuhao Liu, Liang Zeng, Xiaokun Wang, Boyang Wang, Yongcong Li, Fuxiang Zhang, Jiacheng Xu, Bo An, Yang Liu, and Yahui Zhou. Skywork-ol open series, November 2024. URL https:
7/doi. 5281 028.///od..2 zenodo. ctt.ood....998..3.. 16998085

---

Chenyu Huang. Zhengyang Tang, Shixi Hu, Ruoqing Jiang. Xin Zheng, Dongdong Ge, Benyou Wang, and Zizhuo Wang. ORLM: customizable framework in training large models for au- tomated optimization modeling. Operations Research, May 2025a. ISSN 1526-5463. doi: 10. 20.1287/0pr..202..11 1233. htt.tttpp///xddo//t http//xx.ii.oooooooooooooo.ooo.. http://.....i..r//1.././ 11.00.1188 1287 www.///p...200000 2024. 1233
Xuhan Huang. Qingning Shen, Yan Hu, Anningzhe Gao, and Benyou Wang. Llms for mathematical modeling: Towards bridging the gap between natural and mathematical languages, 2025b. URL
https: org/ abs 2405 13144
Caigao Jiang, Xiang Shu, Hong Qian, Xingyu Lu, JUN ZHOU, Aimin Zhou, and Yang Yu. LL- MОPT: Learning to define and solve general optimization problems from scratch. In The Thir- teenth International Conference on Learning Representations, 2025. URL httpthttps//peeeee
view. nww. www.ooooommddd----eeeee
Sirui Li, Janardhan Kulkarni, Ishai Menache, Cathy Wu, and Beibin Li. Towards foundation models for mixed integer linear programming. The Thirteenth International Conference on Learning Representations, 2025.
Hongliang Lu, Zhonglin Xie, Yaoyu Wu, Can Ren, Yuxuan Chen, and Zaiwen Wen. OpIMATH: A scalable bidirectional data synthesis framework for optimization modeling. In Forty-second International Conference on Machine Learning, 2025. URL httpss://peerevvieee net/f
orumPid=9P5e61E44
Liangehen Luo. Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, and Abhinav Rastogi. Improve mathematical reasoning in language models by automated process supervision, 2024. //tl:tttts://arii..ooo
s/2406. 07446660592. 06592
Zeyuan Ma. Hongshu Guo, Jiacheng Chen, Guojun Peng. Zhiguang Cao, Yining Ma, and Yue-Jiao Gong. LLaMoCo: Instruction tuming of large language models for optimization code generation, 2024. URL BRLL:httts///axiiii wttp:/iiirrg///.....a 01031
01131
OpenAl. OpenAl outperforms humans and google at the world's top collegiate programming con- test, 2025, URL https::/ttttte decole-dcceer-ee com/ otteeffoo humans nd add-googls at the wor lds top www.ecollleeoooo wwwgppooooi.......... contest/ Accessed: 2025-09-17.
Rindranirina Ramamonjison, Timothy Yu, Raymond Li, Haley Li, Giuseppe Carenini, Bissan Ghad- dar, Shiqi He, Mahdi Mostajabdaveh, Amin Banitalebi-Dehheee
Zirui Zhou, and Yong Zhang. Formulating optimization problems based on their natural language descrip- tions. Marco Ciccone, Gustavo Stolovitzky, and Jacob Albrecht (eds.), Proceedings of the NeurlPS 2022 Competitions Track, volume 220 of Proceedings of Machine Learning Research, pp. 183.203. PMLR, 28 Nov-09 Dec 2022. URL htthtt:::// htttps::/prooeediiii 9220/ jisnoonniiin2222 html
N14opt competition:
mlr press/
Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint (tti::2503.1099// 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. ofain-of-thoughee prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 248377, 2022.
Yang Wu, Yifan Zhang, Yurong Wu, Yuran Wang, Junkai Zhang, and Jian Cheng. Evo-step: Evolutionary generation and stepwise validation for optimizing LLMs OR, 2025. URL https::/cpenrevi//
net http://neet/oe////// adwpmmuuuuaaaaa0app
Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang. Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, and Gang Chen. Chain-of-Expprts: When LLMs meet complex operations research problems. In The Twelfth International Conference on Learning Representations, 2024. URL https:
12

---

Ziyang Xiao, Jingrong Xie, Lilin Xu, Shisi Guan, Jingyan Zhu, Xiongwei Han, Xiaojin Fu, Wing- Yin Yu, Han Wu, Wei Shi, Qingcan Kang, Jiahui Duan, Tao Zhong, Mingxuan Yuan, Jiahang Zeng, Yuan Wang, Gang Chen, and Dongxiang Zhang. A survey of optimization modeling meets Ilms: Progress and future directions. In Proceedings of the Shirty-Fourdd International Joint Conference on Artificial Intelligence, 2025. URL httpt::t
semant icscoo ar.
g/CorpusID: ID: 280649838
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36:69798- 69818, 2023.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.55115. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang. Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.0m 09388, 2025a.
Zhicheng Yang, Yiwei Wang, Yinya Huang, Zhijiang Guo, Wei Shi, Xiongwei Han, Liang Feng, Linqi Song, Xiaodan Liang, and Jing Tang. OptiBench Meets ReSocratic: Measure and improve LLMs for optimization modeling. In The Thirteenth International Conference on Learning Rep- resentations, 2025b. URL https: http:://////peeeiiie..... net wwtp://eneeetototdete.e.e. (ttp://oormmmdddd055555555553
Jianghangfan Zhang, Yibo Yan, Kening Zheng, Xin Zou, Song Dai, and Xuming Hu. Gm-prm: A generative multimodal process reward model for multimodal mathematical reasoning, 2025a.
www.:bs.g//b/boo.oo 2508 04088
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang. Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint 250 2025b.
Yuanchen Zhou, Shuo Jiang, Jie Zhu, Junhui Li, Lifan Guo, Feng Chen, and Chi Zhang. Fin-PRM: A domain -peeiiispe process reward model for financial reasoning in large language models. arXiv preprint arXiv:2508. 15202, 2025.
Jiachen Zhu Congmin Zheng, Jianghao Lin, Kounianhua Du, Ying Wen, Yong Yu, Jun Wang, and Weinan Zhang. Retrieval-angmeentdd process reward model for generalizable mathematical reasoning, 2025, URL hhttp:: //arxiv..////....../../vv 2200 14361
13

---

A THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large Language Models were employed as general-purose assistive tools throughout the research process. Specifically, LLMs were used to aid and polish the writing of this manuscript, including refining grammar, improving clarity, and restructuring sentences for better readability.
In this work, LLMs were utilized for data processing. Specifically, GPT-4o was used assess the modeling accuracy of the initial data and to perform step-by-step error analysis and annotation the process. Meanwhile, Qwen3-8B served as reasoning verifier, automatically checking constraint satisfaction via numeric substitution for feasibility validation. All LL L.M--eenrrtee content under- went cross-validation manual spot-checking to ensure the models functioned strictly assistive tools.
All outputs generated by LLMS were critically evaluated and edited by the authors, and content was used without verification. The use of LLMs did not replace human intellectual contributions but served to accelerate and enhance various stages of the research workflow.
B BENCHMARKS AND EVALUATION
We conduct experiments the following real-world optimization task datasets.
IndustryOR Huang et al. (2025a) is the first caluuttrallggade dataset specifically designed for optimization modeling. It integrates real-world operations research (OR) problems from eight different industries, covering five types of optimization problemslinear programming, integer programming, mixed-intege programming, nonlinear programming, and other spe- cial problem typesacross three difficulty levels. The training set contains 3,000 instances without optimal solutions, while the test set includes 100 instances with optimal solutions, aiming to comprehensively evaluate a model's ability to solve optimization problems in real-world industrial scenarios.
MAMO Li et al. (025) offers a novel optimization dataset for evaluating the mathe- matical modeling capabilities of large language models. The dataset is divided into two parts: Easy LP, which contains 652 high school-level Mixed-Integer Linear Program- ming (MILP) problems for foundational learning, and Complex IP,, which provides 211 undergraduate- challenges that blend complex applications of linear and linear programming. Notably, this dataset does not include any Nonlinear Programming (NLP) problems.
NL.4LP Ahmadi et al. (2024) dataset features 65 curated cases from optimization textbooks and lecture notes. These cases cover various application areas, including facility location, network flow, scheduling, and portfolio management. Each instance includes a detailed problem description, parameter data file, and the optimal value derived from textbook solutions or manual solving, offering a range of complex optimization challenges of varying difficulty.
NL4OPT Ramamonjison et al. (2022) is curated dataset developed from the compe- tition the same name, which tocuses on converting natural language descriptions of optimization problems into solver-ready code. The dataset primarily addresses Linear Pro- gramming (LP) problems across different scenarios but lacks more complex Mixed-Integer Programming and Scheduling (MIPS) problems. In experiments, filtered test set of 213 high-quality instances was used.
ReSocratic Yang et al. (2025b) is an innovative reverse data synthesis method that gen- erates high-quality operations research optimization problems by following a unique from answer to question path. Starting with 27 well-designed seed demonstrations, this method uses the DeepSeck-Vi -2 model to progressively generate new structured cases, ensuring quality through a dual-filter mechanism. Finally, it reverse-translates these formatted cases into natural language problems and corresponding executable code, ultimately creating the RESOCRATIC-29E dataset.
We use the clean version from Xiao et al. (2025) accurate subset of the benchmark. Specifically, we employ Qwen2.5- 1-55--1----nssuu to extract the corresponding optimal values and then compare them with the ground truth.
14

---

C SEED DATASET
C.1 SAMPLING STATISTICS OF THE EXISTING DATASET
Dataset
Sampling Size
Error Rate
Opt-Math-train
500
16%
IndustryOR-trant
500
2 31 %
Resocratic-tras
500
2 30%
Evo-step
500
25%
Table 3: Sample data from different synthetic datasets.
Table shoows the eтrоr rates across several datasets. We also performed an error attribution analysis on Industry-OR and found that approximately 84% of errors were modeling errors (e.g., missing constraints, incorrect objective functions, unit mismatches), 11 % were code implementation er- rors (c.g.. variable definition or logic mistakes), and only about 4% were result inconsistencies (i.c., output solutions violating constraints not matching computed values).
C.2 DETAILS OF BUILD SEED DATASET
Code Execution We perform a straightforward execution of the generated code and then evaluate two criteria: (I) whether the execution completes successfully without errors, and (2) whether the output matches the ground truth.
Constraint Satisfaction In this stage, we use an Qwen3-8B verifier to confirm the feasibility of the solver's numerical solution. The verifier is given the mathematical constraints and the solution, and it performs symbolic or numeric substitution to automatically check if all conditions are met, as demonstrated in the manufacturing example (Figure
Modeling Accuracy This final and most critical stage employs powerful LLM to evaluate if the mathematical formulation faithfully captures the intent of the original problem statement. It iden- tifies crucial semantic flaws, such as a misaligned objective function (e.g., maximizing total parts instead of complete sets). This check ensures the model is not just feasible but also semantically correct, as illustrated in the factory production example (Figure
C.3 FINAL SEED DATASET
Dataset
Size
Full Size
Opt-Math-train
3282
210000
IndustryOR --triin
1375
3000
Resocratic-trane
4036
29000
Evo-step 3351
4464
Table 4: Sample data from different Synthetic.
We sampled data from four sources: Opt-Math Lu et al. (2025). IndustryOR Huang al. (2025a) Resocratic Yang et al. (0025)), and Evo-step Wu et al. (2025). For the Opt-Math and Resocratic datasets, first applied k- E-gredyy filtering to the iniiil data. Following three-stage filtering pro- cess and deduplication, we obtained final dataset of 8,656 instances. We manually checked 100 samples from the final data, and the accuracy is approximately 96%.
15

---

Example: Verifying Constraint Satisfaction
Question:
manufacturing company produces five electronic devices: Smartphones,
Tablets, Laptops, Smartwatches, and Cameras. The profit per unit and labor hours required
are given in the table below:
Device
Profit (S)
Labor Hours
Smartphones
100
5
Tablets
150
8
Laptops
200
10
Smartwatches
50
3
Cameras
300
12
The objective is to maximize total profit.
Solution The optimization solver returns the candidate solution:
f 11 23. as) = (0, 500, 200, 133, 300),
Feasibility Verification by Qwen3-8B
corresponding to (Smartphones, Tablets, Laptops, Smartwatches, Cameras). Qwen3-8B substitutes & into each constraint expression to verify feasibility:
Labor hours: 5(0) + 8(500) + 10(200) + 3(133) + 12(300) 9999 - 10000
Smartphones + Tablets: + 500 = 500 < 500
Laptops: 200 < 200
Smartwatches: 133 100
Cameras: 300 300
All constraints are satisfied, confirming that é is a feasible solution.
Figure Constraint Satisfaction Example
Problem: A company has three factories (X, Y, Z) that produce three different components (1, 2, 3) required for product. Each factory has limited number of production hours, and their productivity (units/hour) for each component varies. The objective is to allocate production hours across the factories to maximize the number of complete sets of components that can be assembled.
Factory
Product
Product
Check modeling
14
16
Z 100
12
15
8
#iSett::
Modeling Process
Factories P- (X, Y.
Components C= 2,3
Verdict: Incorrect
##Parameters:
Productivity: p. p
Max hours: H. X-120, H. Y-140, 11.40,112200 Z-10
Analysis:The formulation maximizes the total number units produced, but the problem description requires maximizing the number of completed sets components. This implies need for balance among the components produced, which is not addressed in the current objective function. The formulation objective that
##Variables:
X. X2. X3: Hours allocated Factory X
or compoments 2....
##Objective Function:
Maximize (14% 11"x X2 9*, X3 16tx Y1 +..))
(Maximizing the quantity of individual componentt)
lacks constraints of complete sets, such as additional constraints to balance production different components.
the production
##Cosstraints:
x_XI xxXxX (Production limit Factory AI..lll All variables (Non-negativity)
Figure 5: Modeling Accuracy Example: LLM checks whether the modeling aligns with the intended meaning of the question.
16

---

D OR-PROCESSQA CONSTRUCTION
D.1 DETAILS OF MONTE CARLO TREE SEARCH
Monte Carlo Tree Search is heuristic search algorithm for sequential decision- makinn large state spaces. In our OR-PRM, we employ it as the first stage of our automated data synthesis pipeline efficiently generate large volume of candidate reasoning steps along with their preliminary correctness labels. MCTS iteratively constructs search tree T = (V, E), where each node v represents a partial solution (i.e., a reasoning prefix), and each edge (v,a) reasoning step generated by the policy model.
represents
Selection Starting from the root node (i.e., the original problem), the algorithm recursively selects child nodes to balance exploitation and exploration. adopt the following Upper Confidence Bound applied to Trees formula.
In N(v) V N(v.a)
a" arg max aEA(v)
Q(u,a)+
Here, Q(v, a) is the average probability of reaching the correct final answer after taking action from node u; N(v) and N(u,a) a) are the visit counts for node v and edge (u, a), respectively; e is constant controlling the strength of exploration.
Expansion When the search reaches leaf node t that still has unexplored actions, the algorithm invokes the policy model to generate a new reasoning step a based on the current state vy, thereby creating new node Unew and adding it to the tree.
Simulation From the newly expanded node Unews the algorithm performs one or more rollout simulations by prompting the policy model autoregressively generate a complete reasoning path to final answer. The simulation outcome is a binary reward: = = 1 if the final answer is correct, otherwise O.
The simulation result is propagated back up the search path, updating the statistics for traversed nodes:
Backpropagation
N(v) + N(v) +1 Q(v,a)+ + Q(. 2 =- Q(v,a)
(4)
(5)
N(v,a)
In the OR-PRM data synthesis pipeline, the core value of MCTS lies its automation. We config- ured key hyperparameters to balance exploration diversity and efficiency: sampling temperature T 1.0, Top-k sampling k = 50, nucleus sampling (TTp--) 0.9. and exploration coefficient = 1.0. Through this structured search, the algorithm efficiently generates over 550,000 candidate reasoning steps with preliminary labels from our carefully curated set of 8,000 seed problems. This provides ample material for the subsequent stages: structured error analysis and consensus-based filter- ing performed by GPT-4o. The preliminary hard labels (0 or 1) generated by MCTS, combined with the detailed natural language critiques from GPT-4o, ultimately produce the high-quality, high- reliability OR OR-ProcesSQA dataset, forming a solid foundation for training the powerful generative process reward model, OR-PRM.
D.2 STRUCTURED ERROR ANALYSIS WITH GPT-40
To generate the rich, step-level supervision required for generative PRM, employ GPT-4o as an expert annotator.
We initially sampled approximately 40,000 data points to form the starting dataset. This final consensus- based filtering step proved to be highly selective, discarding approximately 50% of the candidate annotations and resulting in high-fidelity dataset of roughly 20,000 samples.
17

---

We inspected sample of 100 data points and found that the critics judgments correct in 95% of the cases.
E INFERENCE WITH OR-PRM
OR-PRM can identify different scenarios, such as correct cases, modeling errors, code errors, etc. Three examples as follow:
18

---

MODELING
Problem Statement:
The Li family plans invest their retirement fund in commercial real estate. Property has annual income of $12,500, Property 2 has annual income of $35,000, Property 3 has an annual income $23,000, and Property has an annual income of $100,000. The decision to be made is whether to buy not buy each property, not the quantity, there is only onc property per property. Help them decide which properties to purchase to maximize their annual income. Property costs $1.5 million, Property 2 costs $2.1 million, Property 3 costs $2.3 million, and Property costs $4.2 million. The Li family's 's budget is $7 million. If they purchase Property 4, then they cannot purchase Property 3.
MODELING
SETS
VARIABLES
P= (1,2,,33,4
2, E (0,1), iEP
PARAMETERS
OBJECTIVE
12500
if =
35000
if 2
CaT
:: - R. ==
23000
if = 3
100000
if 4
CONSTRAINTS
150000
if i 1
Dez: < B
210000
if i 2
1=1
Ps: P::Pa P R, ps
230000
i = 3
33 + 24 S
420000
if i 4
(I), ITA
R B
7000000
PYOMO IMPLEMENTATION
from Pyono. envizen inport
Tnit ialization Concrel MoModel ()
Set's (P) and Paraneters (c, D. B) are defined here based on the model.
Key Model Logic
Define Vabial
preperty is porchased, otherwieee model.x Var (model .P, dona in-Binary)
i8 binary variable,
2. Definw Ob jeccive Function
purchased properties,
model. income Objessive iow!
expr=suninodee ( model.xie] for in model..1, senseenaxini
3. Betine Conetraints
The total cost of purchazed progerties cannet exceed the hudget 3, mudel, bundgel por sepr=sum.inddel..
Constraind
nodel.xii]
It property is purchased, progerty connet be. Constraint xx[4)
mosel redclleexcuusio
expremodel
Solver is colled and solution is processed here.
ANSWER
Objective 135000.0
Optimal Solution: 2 = 0.0 22 1r0 0.. 1.0
Figure 6: Correct Case: Modeling Process.
19

---

STRUCTURED ERROR ANALYSIS
1. Variable Definition Analysis
Intent:
Define decision variables. The decision variable 2; is defined correctly as binary, indicating whether to purchase each property.
Analysis:
Judgement: Correct
2. Objective Constraint Analysis
Intent: Formulate model. Objective:
Correct. The objective maximizes total annual income from selected properties. Budget constraint (Ei P:z; S B) correct. Constraint on purchasing
Constraints:
properties 3 and 4 (I3 + 1) is correct.
Judgement: Correct
3. Code Analysis
Intent: Implement model in Pyomo/Python.
Sets, parameters, and variables defined correctly in the code. The objec- tive function and constraints match those specified in the problem statement. The output matches the expected result based the problem deseription. Judgement: Correct
Analysis:
4. Final Answer Analysis
Intent:
Report solution or error.
Analysis: The reported solution shows that properties 2 and 4 should be purchased, yielding an optimal annual income of $135,000. This aligns with the problem requirements and constraints.
Judgement: Correct
No corrections needed; all parts of the solution accurate.
Figure 7: Correct Case: Structured Error Analysis.
20

---

MODELING
SOLUTION
SETS
VARIABLES
A represents the set properties, with (l. 2, 3.344
2; E {0, 1), i e P. where x; is purchased, n; = 0 otherwise.
if property
PARAMETERS
OBJECTIVE
P: is the annual income of property i, P,
Piita
12500, P2 35000. ps 23000, P4 100000
(0.1) HEP
c; the cost property i, E P. C1 1500000,
CONSTRAINTS Ene Gat S b. (0, 13
C2 = 2100000, c3 2300000, Cı 4200000
is the budget, b = 7000000
PYOMO IMPLEMENTATION
from syono.envizeo inport
nodel concrazedodda ()
Sals and Pasaneteza
defined here based the model.
Kay Modeling Logic
Define Variables
x is binary variable, preperky is puzchaxed, otherwise. nodei.x Vas (nodel.P, donain-Binary)
Define Maximize ze the total annual incone from purchased properties, model vobject fow Ob Objecttive ivee
Funcl iun
expresun [nodel.p(i] nodel.x[:] for ln mudel.P1,
3. Define Constraints
purchazed progerties exceed budget.
mudel, budgel coralssitt Constraink expr=sun expr=sum/nodellell [vj(f] sellf.nddelleee for in se fi <- self mode bodiget
mnodel.
INCORRECT CONSTRATNT 3333
The problen states: "EE property 4 la purchased, then property beards be. logic
x3 de 1.
incorrectly inplements
model, www.deeeoontttraill.......o expr=modeLxx(s] <= model.x131
reint
... Solver is colles and solution is processed here.
ANSWER
Objective value 12500.00
Optimal 2 mg
Figure 8: Incorrect Answer Case: Modeling Process.
21

---

STRUCTURED ERROR ANALYSIS
1. Variable Definition Analysis
Intent:
Define decision variables. Analysis:
The decision variable 2; is defined correctly as binary, indicating whether each property should be purchased.
Judgement: Correct
2. Objective Constraint Analysis
Intent: Formulate model. Objective: Correct. The objective maximizes the total annual income from selected proper- ties.
Constraints: Budget constraint is correct; ensures that the sum of costs does not exceed the budget. Order constraint is incorrect. It states 244 æ3, meaning if property 4 is purchased, then property 3 must also be purchased. However, the original problem specifies that if property 4 is purchased, property cannot be purchased. Judgement:
Incorrect due to the constraint error.
3. Code Analysis
Intent: Implement model in Pyomo/Python.
Analysis: Binary decision variables correctly implemented. The budget constraint is implemented correctly. There is an implementation error in the order constraint, Instead of x4 S 33, should be 24 + T3 S Judgement: Incorrect
4. Final Answer Analysis
Intent:
Report solution or error.
Analysis: The reported solution suggests purchasing properties and yielding total annual income of 123,000 dollars. This contradicts the problem statement since it implies purchasing both properties and 4 simultaneously, violating the logical condition specified. Judgement: Incorrect
5. Corrected Step: Code Implementation
(imports and mosel sctup are identical to the incorvect version!
Drcenn_conattonnneecoo. (DOLE)
t The budget constraint remeine the came.
self mol.f.ndde... wtwdddeee_oossaaaaaeeee Constraint
self.modbl.xil
in seli.mods1.PI
selt.mosei.
CORSECTED LOSIC
exclusion constraint:
This nои corsectly implements
propezty purchased,
preperty cannot bet as xi x3 <" self wnl.enode..co
www.ddel..dddd.....m www.ooddddroto condtroonnttaatt Constraint(
sclf.nodel.x(3
(objective deficition and soiver calls ramain the sanc)
This corrected version properly reflects the logical constraint that if property 4 purchased, property 3 cannot be purchased.
Figure 9: Incorrect Answer Case: Structured Error Analysis.
22

---

MODELING
SOLUTION
SETS
OBJECTIVE
Properties {1,2,3,4]
max Purchase
Income (1) Purchase
PePrepectiss
PARAMETERS
CONSTRAINTS
Income Properties) = (12500, 35000, 23000,
100000).
Σ
Cost(i) Purchase S 7000000
Cost(Proeettie)
(1500000,
2100000,
iEPropentiss
2300000, 4200000)
Purchase(3) < Purchase(4)
VARIABLES
if Property i is purchased, otherwise
Purchase Purchase(Properties
PYOMO IMPLEMENTATION
ervno.envizen inport
Hodel Init initialisatin
model ()
Sets, Farenetors, Variables
ob jeclive iin-ion is elsu defined.
Modsling (Constraints)
Correctly ixplamenced budget coastraint def hudget conat rule (nadel):
nuntandel. Cost[i] model
medell Purchasw(1) for Properttts))) 7000003
model. BoooaaBBddettccatteaaa Constraint
INCORRECT
IMPLENENTATION
wingle, scalar corauzaint.. It should not be indexed ovez the set of all propercioa. The code below incerrect defires
const Purchaxe
Purchass(()
indexed constraint, leacing paraneter prepá, propf): indices (pcop3, propd) Pyono only provides one. If prop3 en 3 and propd
funet signature
This rule expecta
cwbarn model. (3)) <= Parchaw[:
else:
return Conetrwint.Skk
Line
indexed rule's sigrature does
Pyono mudel. www.eelwwiiyyooaaaaacc...o Constsaint (nodel, Propecties, culemxxclaiivil www.://wwwttttttoteooooooooom comstaainn__eelle
Solver call leads TypeRrrar.
ANSWER
Figure 10: Code Execution Error Case: Modeling Process.
23

---

STRUCTURED ERROR ANALYSIS
1. Variable Definition Analysis
Intent:
Define decision variables. Analysis:
The variable 'Purchase' is defined correctly as binary, indicating whether each property should be purchased or not.
Judgement: Correct
2. Objective & Constraint Analysis Intent: Formulate model.
Objective: Correct. The objective maximizes total income based on selected properties.
Constraints: Budget constraint is correct; it ensures that the total cost does not exceed $7 million. Exclusivity constraint is intended to prevent purchasing both properties 3 and 4 simultaneously. However, the implementation has an issue due to the way constraints are indexed and formulated.
Judgement: Incorrect (due to implementation issue in exclusivity constraint)
3. Code Analysis
Intent: Implement model in Pyomo/Python.
Analysis: There is critical error in defining the exclusivity constraint. The function exclu- sivity constraint rule expects two arguments (prop3, prop4) but only receives one (prop3). This mismatch causes a TypeError. Additionally, the constraint logic needs adjustment to properly enforce mutual exclusion between properties 3 and 4. Judgement: Incorrect
4. Final Answer Analysis
Intent: Report solution or error.
Since the code fails at the implementation level, there is no valid output or solution reported. The error message indicates fallure in constructing the exclusivity constraint. Judgement:
Analysis:
Incorrect
5. Corrected Step: Code Implementation
(imports and modal sctup aze identical to the previcus versior.)
MuCinoo.onatrainaa (solf):
t The budget constraint is scalar constraint and remsins the sane. welf.modell budgee conaa twint. Conai-aannl
expr=sun (aell wow.e.....odl..mmmmm. e(f]) self meif.nedsl.x11
in mold.nodel.P1
selff .udget
CORRECTED TNP INFRRINTATION
peevious version socorsellly tried Lo define this an indexed constraiat over a set, The correct approach is co define it as single, scalar constraint, condel order posstessoosstttt
dows not depend Index. expe=self x046 wtl...odol.xx33 xx3]
inbjective defirition
This corrected version defines the exclusivity constraint more clearly and appropriately, ensuring that properties and cannot both purchased.
Figure 11: Code Execution Error Case: Structured Error Analysis.
24

---

F PROMPTS USED IN EXPERIMENTS
CRITIC PROMPT
\label (critic)
You are an expert in Operations Research (OR).
You will be given optimization problem and (optionally) step- by-step solution, which may or may include code.
Task: Review the solution, Analyze each applicable part in order. He concise only highlight critical errors or omissions. Skip any section if the input doesn't contain (e.g., code skip Code Analysis).
Evaluate in this order:
Variable Definitions
2. Objective Function and Constraints
3. Code implementation (if provided)
4. Pinal Answer Ouzput.
Question:
[Question]
Solution Steps:
(Solution)
Cutput Fornat (be brief and precise)
Variable Definition Analysis
Intent: [e.g., Define decision variables]
Analysis: [Only note missing, redundant, or misdefined variables]
Judgement:
[Cozroct/Incorreett
Objective and Constraint Aralysis
Intent: [e.g., Formulate model]
Objective: [Correct? Brief reason is wrong]
Constraints:
Miising//nnorreett List only key issues [Cozrect/Incorrett]
Sudgement:
3. Code Analysis (Skip if no code) Intent:
Implement model in Fyomo/Python
Analysis: [Only flag mismatches: missing vars/constraints, wrong indexing, type errors] Judgement:
[Correct/Incorrect or Skipped]
4. Final Answez Analysis
Intent: [e.g., Report solution error]
Analysis: [Must show valid optimal solution AND objective value. If output contains ANY error/traceback (e.g., Syntaxsrror, Attributekrror) Incorrect. [Plausible? Error meaningful? Root cause if wrongl]
Judgement:
[Cozrect/Ineorrett]
Corrected Step (Only if any part above is Incorrect) [Rewrite only the first incorrect section e.g., fix constraints or variables in full, clearly labeled.]
25

---

QUESTION TO MODELING PROMPT
You are expert in Operations Research (OR).
The following is optimization problem. You need write the corresponcing Рyоmo code based on the problem description and information provided.
The problem description is follows.
..
(ques)
The following is the five-element model of optimization problem: (five)
Please write the corresponding Pyomo code. Please add 'from pyemo. at the beginning of your code (You can add
environ import other 'import' as well). Please print the optimal solution and the value of the objective functien. Please de not output the running log. You need to write it in the form of class and add main function:
''yython
[write your code here]
MODELING TO CODE PROMPT
You are an expert in Operations on Research (OR).
The five-e" ement model is the abstraction of optimi zatioonn problem, which transforms speci fic problem scenarios into formal mathematical problems. You need to write the corresponding Pyomo code based on the five-element model provided.
The following is the five-element model of an optimization problem: (five)
Please write the corresponding Pyomo code. Please add 'frmm pyomo. environ import x' at the beginning of your code (You can add other 'import' as well). Please print the optimal solution and the value of the objective functien. Please not output the running log, You need to write it in the form of class and add a main function:
" 'python
[write your code here]
26

---

EXTRACT ANSWER PROMPT
You are an expert in Operations Research (OR),
Your task is to precisely extract and return exactly one line from the multi-line text provided belcw. This line must state the final optimization value (e.g., maximum profit, minimum cost, or total objective value).
# Core Instructions
**Exact Extraction**:
The returned content must complete original text.
unmodified line as appears
++Single OutputA4: Your response must contain only the
racted 1inc. Do not add any prefixes, suffixes, explanations introductery phrases, extra formatting.
**Keyword Recognitione:: Prioritize identifying and racting the line that contains cemmon optimization keywords such as:
'cost
'profit'
'objective"
'value'
'revenue'
'optinal value*
'Total'
Text to analyze:
[text]
27


Reviews:
[{"review_id": "uTY4ed8lpe", "reviewer": "Reviewer_smtZ", "summary": "This paper introduces the first Process Reward Model tailored for Operations Research (OR-PRM), aiming to explore the potential of Large Language Models (LLMs) in this complex reasoning domain. The authors surprisingly find that direct training of a PRM on existing mainstream OR datasets yields weak performance. Through systematic analysis, they identify the primary bottleneck as data quality, noting that over 30% of existing annotations are severely flawed. To address this, the authors collect all existing synthetic datasets and employ a carefully designed filtering pipeline to construct a high-quality seed dataset, which is then used for model training and evaluation. This work not only lays the foundation for applying LLMs in the OR field but also provides a crucial warning and proposed improvement strategy for the quality of current OR algorithm datasets.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is the first application of the Process Reward Model paradigm to algorithmic problems in the field of OR. OR is a domain highly dependent on structured reasoning and complex algorithms, making the integration of LLMs highly valuable for research and application.\n\n2. The authors conduct a systematic analysis that clearly identifies a severe annotation quality issue (over 30% defects) in mainstream OR datasets. This is a significant contribution and warning to the wider community, as identifying data bottlenecks is a crucial step for field advancement.\n\n3. The authors do not avoid the data quality issue but instead proactively address it by collecting existing synthetic datasets and applying a \"carefully designed filtering pipeline\" to build a high-quality seed dataset. This strategy of solving the problem at the data source is commendable.\n\n4. If OR-PRM proves effective, it will greatly simplify the modeling and solving process for OR problems, providing new avenues for automated algorithm discovery and solution.", "weaknesses": "No serious weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oFY8SEsRer", "reviewer": "Reviewer_gX4v", "summary": "The authors develop the first Process Reward Model (PRM), which is specifically intended for Operations Research. Based on prior research that existing synthetic datasets in this area are significantly flawed, the manuscript proposes a filtering pipeline to initially construct a \"seed\" dataset of problems and their description in a formalized way as well as a solution. Each problem is validated in multiple ways by ensuring that the reference code runs and yields the correct result, that constraints are valid as well as that the mathematical model to solve represents the problem faithfully. This dataset is then used to construct several reasoning trajectories of multiple reasoning steps based on Monte Carlo Tree Search (MCTS), where each step is validated, to generate a dataset for training (called OR-ProcessQA). A process reward model, OR-PRM, is then trained on this dataset to provide dense rewards for guiding the reasoning effectively. The reward is not a single score, but instead provides feedback in natural and corrections in natural language, so is more like a critic model.", "questions": "* 3.1.1: What does $\\hat x$ represent? The new solution? But if the expected output is already known, why not use that as ground truth?\n* 3.1.2: What is $\\mathcal{L}$? I suppose it is not the loss function?\n* 3.2: How are Generative PRMs different from a general critic model employed for each reasoning step?\n* Table 1: What would have been the improvements, if the best solution would have been selected based on the ground truth, i.e. what would have been the upper bound of improvement when generating additional trajectories?\n  * This would give context how well OR-PRM selects reasoning chains or whether improvements \"just\" stem from the additional generation, i.e. more token use.\n  * this sentence in 4.4 seems to suggest that such data exists: \"our Best-of-N performance is strong, but it still falls short of the theoretical upper bound\" (line 466)\n  * Why was OR-PRM not used in this setting for the proprietary models\n* Table 2:\n  * What does Qwen2.5 (Zero Shot) do? Why are the results different than in Figure 3?\n    * same for OR-PRM (Ours)\n  * How does majority voting (filtered null) work?\n  * Which scenario is used for the ablation study? probably the second one\n* Did you try any out of distribution benchmarks?\n* 4.4/conclusion: How would expanding the dataset help to create credible baselines?\n* Appendix F, Critic Prompt: seems only for the full solution\n  * Can we actually call this a Process Reward Model, if it does not look at individual reasoning steps?\n  * What is difference between the two prompts on page 24? Where are they applied?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* new dataset seems to be addressing a need in this field, if existing datasets are so severely lacking\n* writing seems to be mostly clear, especially the description of the dataset construction", "weaknesses": "* field of Operations Research is never properly introduced\n* novelty is a bit diminished by over-claiming: abstract/introduction/conclusion give the impression that the analysis of the existing datasets is done by the authors of this manuscript, but judging from the related work section it was done by someone else\n  * this directly hurts the motivation for the new dataset, since the problems in existing datasets are not discussed in detail\n* dataset statistics are mostly missing, for example:\n  * number of samples in the (seed) dataset\n    * What is a sample?\n  * average length of trajectories, i.e. number of steps\n  * number of trajectories per sample\n  * how many failed/successful trajectories\n  * dataset composition from the original datasets (partially in Appendix C.2)\n  * little empirical analysis of the dataset creation pipeline\n* limited evaluation:\n  * generation (not the primary focus of the manuscript): while different model sizes are used, they are only from one model family, leaving the question whether the approach generalizes to other model families\n  * only a single model, small, trained as a reward model, so generalization is a big question\n  * analysis focuses too much on the employed generative models and less on the capabilities of the reward model\n  * no baselines for the first scenario (Best-of-N) besides the original model\n  * unclear baselines for second scenario:\n    * How does pass@1 with critic work? It considers the proposed corrected step/solution?\n    * How does pass@8 without critic work?\n  * potential baseline: Use of a general purpose model with the same prompt to act as an critic?\n  * missing cost analysis\n\nminor issues:\n* it seems that the wrong cite command is used in general, so brackets around the references are missing\n* related work:\n  * title - seems to be not all caps like the other section titles\n  * line 147: for some reason \"offline\" is incorrectly hyphenated\n* Figure 2:\n  * you might want to make the bubble (bottom row, roughly the middle) a bit bigger, so that \"Data Diversification\" does not touch the outline\n  * same for the bubble inside \"OR-PRM Model Training\"\n* titles of 3.1 and 3.1.1: You might want to properly capitalize the titles.\n* 4.1:\n  * line 327: \"Specifically, Industry OR...\" - not a proper sentence\n  * line 331: \"OR-PRM, We\" - \"we\" should not be capitalized\n  * line 346: reference for CoT?\n* references:\n  * cited differently than other arXiv references (can only be surmised from the URL):\n    * [DeepSeek-AI 2024]\n    * [Luo et al. 2024]\n    * [Ma et al. 2024]\n    * [Zhang et al. 2025a]\n  * additionally consider capitalizing the titles to be consistent, especially abbreviations and proper names (which is sometimes done):\n    * [Huang et al. 2025a/b]\n    * [Ma et al. 2024]\n    * [OpenAI 2025]\n    * [Wang et al. 2025]\n    * [Wu et al. 2025]\n    * [Xiao et al. 2024]\n    * [Xiao et al. 2025]\n    * [Xie et al. 2023]\n    * [Yang et al. 2025b]\n    * [Zhang et al. 2025a]\n    * [Zhou et al. 2025]\n  * missing place of publication: [Wu et al. 2025]\n  * [Xiao et al. 2025] was published IJCAI '25\n* Appendix B:\n  * NL4OPT: The number of samples (245) seems to differ from Table 3.\n  * line 711: Wrong table reference (Table 3 instead of 4)?\n* Figure 5, caption: \"Example:LLM\" - missing whitespace\n* Appendix E, line 858: \"etc.Three\" - missing whitespace; \"Three examples as follow:\" - not a proper English sentence\n* Appendix F: title sounds a bit strange", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Egkf7aJmHI", "reviewer": "Reviewer_xTZk", "summary": "The paper introduces OR-PRM, a domain-specialized Process Reward Model for Operations Research (OR). The authors (1) diagnose high noise in existing OR datasets; (2) curate a cleaner seed set via a three-stage pipeline; (3) generate step-wise trajectories with MCTS; (4) have GPT-4o perform structured, step-level judgments and corrections; (5) train a generative PRM that outputs critiques rather than scalar scores; and (6) show gains under Best-of-N selection and a Modeling→Critique→Code pipeline, reporting up to +12.5 pp average improvement across benchmarks and model sizes (Qwen2.5 series, LLMOPT).", "questions": "1. How often does GPT-4o disagree with the MCTS label? Show confusion matrices and human audits on a random 500-step sample. How many “corrections” by GPT-4o were later proven wrong?\n\n2. If you allow the critic to emit corrections (you already do), can you implement Refined-BoN like GM-PRM and report deltas? (https://arxiv.org/abs/2508.04088)\n\n3. Evaluate with new solver backends (e.g., Pyomo→PuLP / OR-Tools), new template families, and noisy text to test OOD per RAPRM concerns. (https://arxiv.org/abs/2502.14361)\n\n4. Ablate the “generative” aspect: Replace the critic with (i) scalar PRM, (ii) critique-only (no correction), (iii) correction-only (no explanation). Which component drives Complex-LP gains?\n\n5. Add self-consistency, vote-with-length-normalization, and retrieval-augmented PRM. Current “Major Voting” is too weak to claim SOTA.(https://arxiv.org/abs/2502.14361)\n\n6. End-to-end token/cost for data curation + training + inference (critic calls per step)? Compare with OmegaPRM’s cost per step label. (https://arxiv.org/abs/2406.06592)\n\n7. Failure modes: Provide qualitative cases where OR-PRM gives confident but wrong constraints/objectives (classic OR pitfall), and whether the system catches feasible but semantically wrong models.\n\n8. For 100 randomly sampled instances, have expert OR graders rate the usefulness and correctness of critiques (Likert & error-type taxonomy). Your current evidence relies mostly on automatic verifiers.\n\n9.  As far as I know, the algorithm complexity of MCTS is relatively high, and I am interested in the impact of introducing MCTS on computational efficiency.\n\n10. If we don't use GPT-4o, which is a relatively good model, but some relatively poor models, can your method still demonstrate robustness?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. PRMs are a natural fit because OR solutions require step-wise logical validity (not only final objective values). The paper targets this gap explicitly.\n\n2. Three-stage filtering for a seed set, MCTS exploration, and GPT-4o step audits with issue/judgement/correction fields—this is more structured than typical PRM pipelines in math reasoning.\n\n3. Returning natural-language critiques and a corrected first error goes beyond scalar PRMs and mirrors trends in “corrective” PRMs in broader reasoning literature.\n\n4. Best-of-N and model-agnostic critic usage show uniform gains; the Complex-LP results are especially notable (e.g., +24.2 pp on Qwen2.5-32B).", "weaknesses": "1. The paper should deeply contrast with OmegaPRM (automated process supervision via MCTS) which established scalable step-labeling at large scale in math reasoning (1.5M annotations) (https://arxiv.org/abs/2406.06592), and with recent generative/critic-style PRMs designed to explain and correct steps such as GM-PRM and VisualPRM/Athena-PRM (multimodal, but methodologically very close in “PRM generates critiques + refines BoN”) (https://arxiv.org/abs/2508.04088). The proposed “generative PRM” feels incremental w.r.t. these trends.\n\n2. The paper claims >30% severe flaws in mainstream OR data and a rigorous three-stage filter, but I don’t see inter-rater reliability, error taxonomy distributions, or random spot-check protocols beyond illustrative examples. Given PRM sensitivity to label noise (documented in PRM surveys/lessons) (https://arxiv.org/abs/2501.07301) , stronger auditing is necessary.\n\n3. Step labeling and final verification heavily rely on strong LLMs (GPT-4o/Qwen verifiers) inside the pipeline that also guide critique at inference. Without cross-model, cross-vendor checks and held-out validators, there is potential circularity (the critic agrees with the validator it was trained/selected with).\n\n4. Since BoN/selection tends to give large lifts, the paper should compare against strong non-PRM baselines (self-consistency; majority vote with route-length filters; retrieval-augmented PRM or OOD-robust PRMs) that recent works show to be competitive, e.g., Retrieval-Augmented PRM (R-PRM / RAPRM) and R-PRM variants focusing on OOD and data bootstrapping. (https://arxiv.org/abs/2502.14361) Current ablations (e.g., “Major Voting”) look weak and not representative of the SOTA toolkit.\n\n5. OR is a heterogeneous space (LP/MILP/NLP/CP-SAT with industry quirks). I don’t see OOD splits (new templates, new constraint families, solver switches) or robustness to noisy/problematic instances (e.g., infeasible but realistic specs). OOD weaknesses are a known pain point for PRMs. (https://arxiv.org/abs/2502.14361) \n\n6. The Modeling→Critique→Code pipeline may conflate several effects: (i) data curation, (ii) structured prompts, (iii) critic guidance. A factorized ablation (swap in a scalar PRM; blind the critic to code; remove “Corrected Step”) would clarify whether “generative PRM” itself is the key.\n\n7.Important but under-specified: exact prompts, MCTS hyper-parameters per task, GPT-4o temperature and refusal handling, and the policy/critic decoupling at test time (who conditions on whom).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GcWpmNyUAY", "reviewer": "Reviewer_mUPb", "summary": "This paper introduces OR-PRM, the first Process Reward Model (PRM) tailored for Operations Research (OR) reasoning tasks. The work targets a fundamental challenge: large language models (LLMs) often fail to produce reliable, logically consistent reasoning in OR problems involving optimization modeling, constraints, and solver code generation.\n\nTo address this, the authors propose a three-stage pipeline:\n\nData construction – Build a high-quality, process-annotated dataset (OR-ProcessQA) through careful seed curation, constraint validation, and semantic verification.\n\nMonte Carlo Tree Search (MCTS) – Generate diverse reasoning trajectories and preliminary correctness labels automatically.\n\nProcess Reward Model (OR-PRM) – Train a generative PRM using Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO), enabling fine-grained, interpretable step-level feedback for OR reasoning.\n\nExtensive experiments on multiple open-source (Qwen2.5, LLMOPT) and closed-source (GPT-4o) models show that OR-PRM substantially improves reasoning accuracy, robustness, and interpretability.", "questions": "The pipeline heavily depends on LLMOPT. It would be nice to discuss if a different model were used to generate the data, how much impact that would have on the results.\n\nIt seems that the OR problems have data that appeared in the problem description in scalar form. What if the data is stored in CSV files? Would the pipeline still work?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Well-motivated and novel contribution: Addresses a clear gap between generic reasoning PRMs and domain-specific reasoning in mathematical optimization. OR-PRM is the first reward model explicitly designed to evaluate reasoning steps in mathematical optimization, going beyond scalar scoring to produce structured, natural-language critiques and corrections.\n\nMethodologically rigorous: Multi-stage pipeline (SFT + DPO) is logically constructed, with explicit checks (execution, constraints, semantics) to ensure data correctness.\n\nExcellent technical clarity: The paper defines each component precisely (seed data, MCTS, PRM training), making it reproducible and interpretable.\n\nComprehensive evaluation: Benchmarks include both open- and closed-source models, with ablation studies isolating the effect of DPO alignment.\n\nStrong empirical results and interpretability: The improvement is not just numerical but also interpretable, showing how OR-PRM critiques and corrects reasoning steps.", "weaknesses": "It is not clear to me whether the evaluation pipeline can serve as the ground truth. Since LLMOPT and GPT-4o may make mistakes. It is unclear how far the evaluation is from having an OR expert evaluate the reasoning steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)