Title: MURKA: Multi-Reward Reinforcement Learning with Knowledge Alignment for Optimization Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Reinforcement Learning

Optimization Modeling

Multi-Agent Systems

Knowledge Distillation
Abstract: Optimization plays a central role in Operations Research (OR) and numerous industrial applications, yet automating the end-to-end process of translating natural language descriptions into executable optimization programs remains a formidable challenge. While recent efforts have applied Large Language Models (LLMs) to this task, existing approaches are hindered by high inference costs, limited robustness across domains, and weak verification mechanisms. In this work, we propose MURKA, a reinforcement learning and knowledge distillation-based framework that enhances LLM-driven optimization modeling via collaborative agent alignment. MURKA orchestrates three specialized agents---Extractor, Solver, and Checker---to achieve accurate problem understanding, robust formulation, and verifiable execution. The Extractor is trained using group relative policy optimization with a composite reward function that incorporates semantic correctness and execution fidelity. The Solver benefits from knowledge distillation from a powerful teacher model, yielding structurally valid and executable formulations in AMPL. The Checker iteratively verifies solution correctness via solver feedback. 
We validate MURKA's generalizability through extensive experiments across diverse OR benchmarks, demonstrating its robustness and scalability.
Experimental results on eight diverse OR benchmarks, including NLP4LP, ComplexOR, and NL4Opt, demonstrate that MURKA, built on the LLaMa3-8B backbone, achieves a 5.9\% absolute improvement in solution accuracy and a 5.1\% increase in execution success rate compared to leading baselines. These results establish MURKA as an effective and scalable paradigm for LLM-driven optimization, with strong potential for deployment in real-world OR applications.


Peer reviews:
comments: SUMMARY:
The paper introduces MURKA, a reinforcement learning and knowledge distillation-based multi-agent framework for translating natural language descriptions of optimization problems into verifiable, executable solutions. MURKA leverages three specialized agents: Extractor, Solver and Checker. It is trained via a composite reward function and guided by distilled knowledge from a high-capacity teacher model. It achieves strong results on eight OR benchmarks, outperforming several state-of-the-art methods in both solution accuracy and execution robustness, despite using a relatively small 8B-parameter LLM backbone.

STRENGTHS AND WEAKNESSES:
Strengths:
* Novel architecture.
* Effective training.
* Competitive results.
* I like that it achieves high performance using a smaller LLaMa3-8B model, something that shows cost-efficiency.
* Thorough evaluation results and ablation study

Weaknesses:
* It relies on reliance on synthetic data, something that I don't like.
* Sometimes it slightly underperforms in certain domain-specific cases, for instance, in IndustryOR.
* Paper difficult to read, but the overall idea is good.

QUESTIONS:
* How does the composite reward function balance semantic accuracy vs syntactic format?
* What is the role of each reward component in the final performance gains?
* How sensitive is performance to the weights of the reward terms?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MURKA, a framework designed to automate the process of solving Operations Research problems from natural language descriptions. The core of MURKA is a multi-agent system composed of three specialized agents: an Extractor, a Solver, and a Checker. The Extractor, trained using reinforcement learning with a composite reward function, parses the natural language problem to identify key components. The Solver, enhanced through knowledge distillation from a powerful teacher model, then formulates this structured information into executable AMPL code. Finally, the Checker verifies the solution's correctness using feedback from an optimization engine like Gurobi. The authors demonstrate through extensive experiments on 2,224 OR problems that MURKA, using a LLaMa3-8B backbone, outperforms existing methods in solution accuracy and execution success rate.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper tackles the challenging and highly relevant problem of automating OR modeling from natural language. The proposed MURKA framework is original in its combination of a multi-agent architecture, reinforcement learning with a multi-faceted reward function (GRPO), and knowledge distillation for this specific task. This approach presents a significant step forward from simpler prompt-based or supervised fine-tuning methods.\n2. The design of the MURKA framework is well-structured. Decomposing the complex problem into specialized agent tasks (Extract, Solve, Check) is a logical approach that enhances modularity and interpretability. \n3.  The paper is very well-written, clear, and easy to follow. The framework is explained logically, and Figure 2 provides an excellent visual overview of the different components (Extractor Alignment, Solver Alignment). The extensive appendices provide commendable detail on the experimental setup, hyperparameters, and prompts, ensuring a high degree of reproducibility.\n\nWeaknesses:\n1. The framework's effectiveness, particularly the Solver agent, is heavily dependent on knowledge distillation from a high-capacity teacher model like DeepSeek-R1. This creates a potential bottleneck, as the student model's quality is capped by the teacher's performance, and it requires access to powerful, often proprietary models for training. This reliance is particularly critical because, as the paper's own results show, even the best-performing LLMs struggle with the most complex OR problems; solution accuracy on the Mamo Complex and Specific benchmarks remains low, around 51-55% and 37-45% respectively. This highlights a more fundamental issue: while the framework excels at teaching an LLM to structure explicitly stated information into a correct format (like AMPL code), its ability to infer complex, unstated constraints or formulate entirely new mathematical relationships seems limited. The framework serves more as a powerful method for optimizing an LLM's performance on known problem patterns rather than a system that fundamentally expands its core logical and mathematical reasoning abilities for the OR domain.\n\nQUESTIONS:\nQuestions\n1. The composite reward function includes four weighted components.  How sensitive is the Extractor's performance to these weights, and how well would this reward function generalize to a new domain or problem type not represented in the training benchmarks (e.g., Second-Order Cone Programming)? Would introducing a new problem type require a full re-design and re-tuning of the reward function?\n2. The shampoo example in Figure 1 illustrates the extraction of explicitly stated numerical constraints.  How does MURKA handle problems where constraints are implicit or require deeper semantic understanding and logical deduction to formulate correctly? For instance, a problem might describe relationships that need to be translated into a system of equations not explicitly provided in the text. How does the framework push the LLM to perform this kind of multi-step reasoning, rather than just information extraction?\n3. What is AMPL? (A Mathematical Programming Language)?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations in Section 5.  They correctly identify the reliance on structured problem descriptions, the potential inflexibility of predefined reward templates, and the risk of inheriting biases from the teacher model during knowledge distillation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents MURKA, an end-to-end framework that automates converting natural-language optimization problems into executable models and verified solutions via a three-agent pipeline including Extractor, Solver and Checker.​\n\nKey Contributions​\n* Modular design separates extraction, generation, and verification, boosting transparency and reducing latency.​\n* Multi-dimensional RL reward offers fine-grained feedback on extracted specs' syntax and semantics.​\n* Cross-domain knowledge distillation lets the 8B-parameter model match or outperform larger LLMs in optimization tasks at lower compute costs.​\n* Validated on 8 benchmarks (e.g., NL4Opt, Mamo), with 5.9% higher solution accuracy and 5.1% higher execution success rate than prior methods. Ablation studies confirm Extractor and Solver value.​\n\nMURKA is an effective, scalable alternative to prompt engineering or monolithic fine-tuning for LLM-driven optimization.\n\nSTRENGTHS AND WEAKNESSES:\n​One of the key aspects of MURKA is its modular design. By clearly separating the tasks of information extraction, model generation, and verification, the framework enhances transparency. This modularity also has the added benefit of reducing latency, as each component can be optimized independently. The ability to understand and modify individual parts of the system without affecting the others makes MURKA more accessible and easier to maintain.​\n\nThe multi-dimensional RL reward system employed in MURKA provides fine-grained feedback on both the syntactic and semantic aspects of the extracted specifications. This detailed feedback mechanism is essential for improving the quality of the information processed by the framework. It allows for more precise adjustments during the learning process, ensuring that the output models are both accurate and meaningful.​\n\nAnother significant feature of MURKA is its use of cross-domain knowledge distillation. This technique enables an 8B model to match or even outperform larger models in optimization tasks while keeping compute costs low. By leveraging knowledge distillation, MURKA can achieve high-performance results with a relatively smaller model, making it more resource-efficient and scalable.​\n\nHowever, one notable concern regarding MURKA is its performance in comparison to DeepSeek R1. Although MURKA shows strong results on some tasks, there are instances where its performance falls short of that of DeepSeek R1. It is possible that long-CoT models with enhanced reasoning capabilities, such as DeepSeek R1, have an inherent advantage in NL-to-optimization tasks. The performance gap might indicate that for achieving optimal results, alternative models like R1-distilled Qwen 7B, which could potentially combine the benefits of knowledge distillation with more advanced reasoning mechanisms, might be more appropriate than the LLaMA3 8B model used in MURKA.\n\nQUESTIONS:\n1. Could you please provide more analysis on the reason why the performance of the proposed method is worse than DeepSeek R1? Is it because long-CoT models with enhanced reasoning capabilities are especially useful in this task? Is it indicate that R1-distilled qwen 7B should be used in this task rather than llama3 8B?\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MURKA, a framework that uses reinforcement learning and knowledge distillation to convert natural language descriptions into optimization programs. The system has three agents: an Extractor that parses problem descriptions, a Solver that generates AMPL code, and a Checker that verifies solutions. The Extractor is trained with a modified GRPO algorithm using multi-dimensional rewards. The Solver learns from a teacher model through knowledge distillation. Tests on eight benchmarks show MURKA achieves 5.9% better solution accuracy and 5.1% better execution success compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n1.nThe paper presents a well-structured pipeline with three specialized agents. The division of labor between extraction, solving, and checking makes sense.\n2. The authors test on eight different benchmarks covering various optimization problems. Results show consistent improvements across most metrics.\n3.The authors show that both the Extractor and Solver components contribute to performance, with clear evidence in Figure 4.\n\n4.The core ideas (multi-agent systems, RL for LLMs, knowledge distillation) are not new. The contribution is mainly in combining these existing techniques.\n5. While the paper claims to address scalability (C1), there's no analysis of training time, inference latency, or computational requirements compared to baselines.\n\nQUESTIONS:\nCan you provide concrete numbers on training time, inference latency, and memory usage? How does MURKA compare to baselines in terms of computational cost?\n\nLIMITATIONS:\nThe authors partially address limitations in Section 5. They mention assumptions about structured input, template dependency, and potential bias from teacher models. However, they don't discuss computational costs, and all that ., so I would recommend the authors to improve this section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}