Autoformulation of Mathematical Optimization Models Using LLMs
Authors: Nicolás Astorga
Tennison Liu
Yuanzhang Xiao
Mihaela van der Schaar
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
optimization modeling
autoformulation
Primary area: applications
TL;DR: Automation mathematical optimization modeling using LLMs

Abstract: Mathematical optimization is fundamental to decision-making across diverse domains, from operations research to healthcare. Yet, translating real-world problems into optimization models remains a difficult task, often demanding specialized expertise. This paper approaches the problem of $\textit{autoformulation}$: the automated creation of solver-ready optimization models from natural language problem descriptions.
We identify three core challenges of autoformulation: $\textit{(1)}$ the vast, problem-dependent hypothesis space, $\textit{(2)}$ efficient and diverse exploration of this space under uncertainty, and $\textit{(3)}$ evaluation of formulation correctness against problem description.
To address these challenges, we present a novel method leveraging $\textit{Large Language Models}$ (LLMs) with $\textit{Monte-Carlo Tree Search}$, exploiting the hierarchical nature of optimization modeling to generate and systematically explore possible formulations. To enhance search efficiency, we introduce symbolic pruning to eliminate trivially equivalent search paths (branches), and employ LLM-based evaluation of partial formulations to guide search.
Empirical analysis on linear and mixed-integer programming benchmarks demonstrates our method's effectiveness, with significant performance gains from both LLM-based value estimation and symbolic pruning techniques.

Autoformulation of Mathematical Optimization Models Using LLMs
Nicolás Astorga
Tennison Liu
Yuanzhang Xiao Mihaela van der Schaar'
Abstract
lems are characterized by maximizing an objective function subject to constraints (Williams, 2013). Traditionally, opti- mization modeling follows a three-step process: gathering problem requirements, typically expressed in unstructured formats and domain terminology: + formulating these re- quirements into formal mathematical model, including variables, constraints, and objective functions; implement- ing the model computationally using specialized modeling language for solution using commercial solvers.
Mathematical optimization is fundamental to decision-making across diverse domains, from operations research to healthcare. Yet, translat- ing real-world problems into optimization models remains difficult task, often demanding special- ized expertise. This paper approaches the prob- lem of autoformulation: the automated creation of solver-ready optimization models from natu- ral language problem descriptions. We identify three core challenges of autoformulation: (I) the vast, problem-dependent hypothesis space, (2) ef- ficient and diverse exploration of this space un- der uncertainty, and (3) evaluation of formulation correctness against problem description. To ad- dress these challenges, we present a novel method leveraging Large Language Models (LLMs) with Monte-Carte Tree Search, exploiting the hierar- chical nature of optimization modeling to gener- ate and systematically explore possible formula- tions. To enhance search efficiency, we introduce symbolic pruning to climinate trivially equiva- lent search paths (branches), and employ LLM- based evaluation of partial formulations to guide search. Empirical analysis on linear and mixed- integer programming benchmarks demonstrates our method's 's effectiveness, with significant per- formance gains from both LLM-based value esti- mation and symbolic pruning techniques.
Autoformulation. Despite major advances in solving algo- rithms over the past decades, the process of formulating optimization models still relies largely on human expertise to understand problem requirements and translate them into mathematical programs that software can efficiently solve to find optimal decision values. Autoformulation aims to address this bottleneck by automating the formulation pro- cess, with the potential to significantly improve the time- and cost- cost-efficiency of the modeling process. For modelers, autoformulation assists with rapid prototyping and itera- tion of different formulations, reducing development time and costs while minimizing implementation errors. For do- main experts, makes optimization tools accessible without requiring deep optimization expertise, allowing domain ex- perts to focus critical business aspects like requirements gathering, use-case development, and communication.
At its core, we conceptualize autoformulation as a search for an optimal formulation within a vast hypothesis space. This search faces several key challenges. First, the hypoth- esis space is large and problem-dependent encompassing diverse variable definitions, constraint structures, or objec- tive function forms, with complex dependencies between these modeling decisions. Second, efficiently navigating this space requires balancing exploitation and exploration, particularly given the uncertainty in correct formulations and redundancy in the hypothesis space. Finally, like any search process, autoformulation requires reliable evalua- tion mechanism for candidate solutions. While solvers can assess optimality and computational efficiency, determin- ing whether formulation accurately captures the intended real-world problem remains particularly challenging.
1. Introduction
Mathematical optimization has long been a cornerstone of decision-making processes across various domains, from supply chain management (Bramel & Simchi-Levi. 1997)
and healthcare resource allocation (Delgado et al., 2022) to portfolio optimization (Mokhtar et al., 2014). These prob-
'Equal contribution 'DAMTP, University of Cambridge, Cambridge, UK 'ECE, University of Hawaii at Manoa, Hon- olulu, USA. Correspondence to: Nicolás Astorga, Tennison Liu <(nja46,1522)) @ca...c.uk>.
Recent works (Ramamonjison et al., 2023; Xiao et al., 2023; AhmadiTeshnizii al.. 2024) have demonstrated the promis- ing potential of Large Language Models (LLMs) in autofor-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Autoformulation of Mathematical Optimization Models Using LLMs
Problem (d E D)
[C1] Problem-dependent hypothesis space
M
A jarmer has 10 acres of land and $100 brudgst to plant 520Vacre with $2/acre cost, and ylelds $30facre with $3lacre cost. The farmer seeks to maximize returns.
corn. Wheat yields
[C2] Efficient search
H(dd
Ps
symbolic prunning
Program (c E C)
Solver
Ps (.)
self-evaluation
Optinal objeciiv:: 300
[C3] Model evaluation
(m* c²) arggaxm-pp..- [Q c; d)]
Figure 1. Autoformulation and its challenges. Autoformulation translates problem description (d D) into mathematical (m E .M) and computational (e E C) models. The challenges include [C1] vast, problem-dependent hypothesis space, [C2] efficient search under formulation uncertainty and redundancy, and [C3] evaluating formulation correctness against problem requirements.
mulation, laying important groundwork in this field. LLMs contribute several crucial capabílities to this process: nat- ural language understanding of problem descriptions, vast domain knowledge incorporate relevant modeling tech- niques, and in-context learning and reasoning capabilities (Brown, 2020; Chowdhery et al., 2023). Building upon these contributions, our work focuses on developing techniques for efficient, systematic exploration and introducing robust mechanisms for evaluating formulation correctness.
esis generators and correctness evaluators alongside sym- bolic pruning techniques. 3 We demonstrate our method's superior performance across two real-world benchmarks containing linear and mixed-integer programming problems, observing significant performance gains from both search pruning and -LM--ased formulation evaluation.
2. Autoformulation: Towards Automated Optimization Modeling
Key considerations. By conceptualizing autoformulation as search problem, exploit the inherent hierarchical struc- ture of optimization modeling to efficiently search through hypothesis space, guided by feed- back on formulation correctness. Our first innovation is to (1) decompose optimization modeling into hierarchical com- ponents and develop a Monte-Carlo Tree Search (MCTS) method to incrementally explore each component's formu- lation space (Coulom, 2006). This exploration is powered by LLMs serving as conditional hypothesis generators, cre- ating diverse component formulations at each level of the search. To improve search performance, we introduce two additional innovations: (2) a pruning technique using Satisfi- ability Modulo Theories solvers (Barrett & Tinelli, 2018), to eliminate redundant hypotheses (i.e., syntactically different yet functionally equivalent); and (3) LLM-based evaluators of formulation correctness, combined with solver feedback, to obtain reward signal to guide efficient search.
Optimization modeling seeks to minimize an objective func- tion subject to specific constraints on decision variables (Dantzig, 1990). The mathematical model can be expressed in a general form:
the
Minimize
f(x)
subject to
9:(x) S 0,
i= 1......, I hx(x) = 0, -==,....."
(1)
Here x E X' represents the vector of decision variables, and X C Re x z is the domain for which the objective and constraints functions are all defined. Furthermore, f X' + R is the objective function to be minimized, 9 X - R are inequality constraints, hy X' + R are equality constraints, and I and J are the numbers of inequality and equality constraints respectively. The feasible region is the set of all possible points that satisfy the problem constraints: (x X' gi (x) < 0, Vi E (x) 0, Vj E [J]].
Contributions, Our main contributions are: 1 We formal- ize autoformulation of mathematical optimization models as a search problem and identify its core challenges. ② We develop a novel approach combining LLMs, symbolic tools with MCTS to enable efficient and systematic exploration of the optimization model space, using LLMs as both hypoth-
Convex problems. An optimization problem is convex if I and gi Vi E [7] are convex, and h2 V j E [J] are affine. Convexity is significant as any local optimum of a convex problem is globally optimal, and specialized solvers can efficiently solve convex problems to global optimality using advanced algorithms (e.g., Gurobi (Gurobi Optimization,
2

---

Autoformulation of Mathematical Optimization Models Using LLMs
LLC, 2024), CVXPY (Diamond & Boyd, 2016)). Before utilizing these solvers, the mathematical models are first represented in code as computational models, which are then passed to the solvers for optimization.
achieve zero optimality gap), and computational effi- ciency (solution time and resource requirements, which can vary significantly between equivalent formulations). "
"Following convention (Sumers al., 2024), we define the weights and procedural prompts as the parameters (ó. 1/) of an LLMeb autoformulator.
2.1. Problem Definition
Boyd & Vandenberghe (2004) aptly recognized that "the challenge, and art, in using convex optimization is in recog- nizing and formulating the problem. Once this formulation is done, solving the problem is ... (almost) technology". While solver technology has significantly matured, the pro- cess of formulating optimization models remains largely human expertise driven. Responding to this challenge, auto- formulation is the automated process of transforming natural language descriptions of real-world problems into formal optimization models, thus automating the "challenge and art" of problem formulation.
"In Appendix E, we discuss and empirically analyze the effects of problem formulation and solver configuration optimality and computational efficiency.
A few observations. While autoformulation involves two transformations, the mathematical formulation step (p&) generally presents significantly greater challenges than creat- ing computational models (p+). This has also been observed empirically in recent studies, where formulating mathemat- ical models was the primary source of errors (Xiao et al., 2023; AhmadiTeshnizi et al., 2024). Indeed, this step re- quires deep domain understanding, abstraction of real-world complexities into mathematical constructs, and certain ativity in effective reformulations. While the translation to computational models often follows a more standardized pat- tern, with some automation already available through com- mercial packages (Fourer et al., 1990). Thus, our subsequent analysis focuses primarily on the mathematical formulation step. However, we note that the second transformation also presents unique challenges, most notably through the choice of solving algorithm and its hyperparameter configuration.
Autoformulation: Formal Definition
Let D, M, and C represent the spaces of natural language problem descriptions, mathematical formulations, and computational models respectively, with d E D, m E M, and c E C as their elements. Autoformulation involves two transformations:
1. Mathematical formulation P DD + P(M): Trans- forming problem description into a mathematical formu- lation. Here, P(.) represents the space of probability distributions.
2.2. Challenges
2. Computational representation Ps M +
P(C): Converting the mathematical formulation into computational formats suitable for solvers, which in- cludes representing the model in programming frame- work, and specifying a solving algorithm.
Our conceptualization of autoformulation search prob- lem reveals a few key challenges:
[C1] Problem-dependent hypothesis space: For each problem d, there exists a vast and problem-dependent hy- pothesis space H(d). encompassing various variable def- initions, constraint structures, objective functions, and their interdependencies. This interdependence and domain- specificity makes it infeasible to manually enumerate or construct the search space (required in traditional search problems), requiring automated methods to generate valid and interdependent modeling components.
Autoformulator. Here, Pe and Pe are models of each transformation, with d, v their respective parameters." " The complete autoformulation process can thus be repre- sented as inferring the joint distribution Po. (m, d) Pv (cl m)) P P d). We refer to any algorithm designed for autoformulation as an autoformulator.
Objective. For a given problem d, the autoformulator aims to find optimal mathematical and computational
[C2] Efficient search: Efficiently searching the hypoth- esis space is challenging, as 'good' formulations can be sparse. There are two key uncertainties complicating this search: uncertainty in formulation choices and un- certainty due to ambiguous requirements (e.g.. implicit or common-sense constraints, including non-negativity of re- source constraints). Additionally, trivial model equiva- lence syntactically different but functionally identical for- mulations (e.g., 2x + 3y versus 3y + 2xr) can lead to inef- ficient exploration of superficial variations at the expense of discovering more diverse and valuable formulations. Here, 'trivial' refers to syntactic variations, distinct from mathe- matical reformulations that change the underlying structure
formulations that maximize an evaluation measure Q(-):
(m",c")) c²) E arg max Q(m, c; d) pomps
(2)
Evaluation criteria. Here, Q assesses the quality of (m, c) relative to d. There are many possible instantia- tions of Q, primary example is formulation correct- ness ness-accuracy of the formulation in reflecting problem requirements. Given that a formulation is correct, other measures could consider optimality gap (distance from optimal value, where certain convex formulation can
3

---

Autoformulation of Mathematical Optimization Models Using LLMs
converting non-convex to convex constraints).
tion process into four distinct stages, each represented by m4. The complete mathematical formulation is defined as = ofm, where + denotes the composition of model components: mh parameters and decision variables, m2 objective function, ma- euu-equaltty constraints, and ma inequality constraints. Given a problem d, the joint distribution Po.v (c, m d) is decomposed hierarchically:
[C3] Model evaluation: While solvers can assess compu- tational aspects like efficiency and solvability, evaluating formulation correctness, whether model faithfully cap- tures the intended problem requirements, remains core challenge. This absence of a correctness signal complicates the search process, as an efficient and optimal solution to an incorrectly formulated problem is ultimately invalid.
(o m |d) = pe m) II Po(mm
| [mai, d)
(3)
Here, note that the complexity of autoformulation also varies significantly with problem characteristies of d. par- ticularly convexity properties-whit some problems allow direct solution for global optimality, others require the aut- oformulator to identify convex reformulations or develop relaxation strategies balancing optimality and computational efficiency (please see Appendix for detailed discussion).
Here, (o mmci d) represents the sequential nature of mathematical formulation. where each component m, de- pends the partial formulation m-ii o) (with mo 0) and the problem description d.
3.2. MCTS-Based Autoformulator
3. LLLM-Enhanced MCTS Search for Autoformulation
Having established structured decomposition of the aut- oformulation process, we now address the challenge of ef- ficiently navigating this hierarchical space. We employ an MCCTS-based algorithm, which is particularly well-suited for exploring complex, hierarchical search spaces (Coulom, 2006). Our MCTS constructs a search tree of depth 4 to explore possible formulations, where each of the four levels corresponds to component in our structured decomposition (m1 to m4). Nodes in this tree contain component formula- tions, and a complete formulation is represented by path from the root to a terminal node.
Overview. Recent developments have shown the promis- ing potential of using Large Language Models (LLM) for autoformulation, leveraging their ability to generate formu- lations dynamically and bypassing the need of manually constructing hypothesis spaces ([C1), Xiao et al. (2023); et al. (2024)). Our approach builds on these works, differing in three key ways. First, we decom- pose the search space using optimization modeling's s hier- archical structure, enabling systematic exploration through Monte-Carie Tree Search (MCTS) rather than generating complete formulations at once [C2]. Second, we enhance ef- ficiency by combining LLM-based evaluation of partial and complete formulations with symbolic pruning of equivalent branches, reducing redundant exploration while improv- ing search guidance [C3]. Third, we employ a determinis- tic parser automatically transform mathematical models into solver-ready computational code. While this success- fully handled all problems in our experiments, eliminating a source of error where LLM-based translation (as used in existing works) proved unnecessary, we acknowledge that more complex transformations may require sophisticated approaches future work. In what follows, we discuss each aspect of our method in turn.
The MCTS algorithm iteratively builds the search tree through four key steps: > selection, expansion. > eval- uation, and A backpropagation. For notational clarity, we denote tree node n and any of its child nodes as nichild E Child(n),, where Child(n) is the set of all child nodes of n. We use 7ī to represent the partial formulation by concatenating the path from root to node n. For instance, ri for a node of depth 2 is the partial formulation containing the parameters, decision variables, and the objective func- tion. Terminal nodes are denoted as ni-- In the interest of space, we present detailed information about all prompts used in the algorithm in Appendix B, providing only high- level details in the following subsections.
1.2.1. EXPANSION
3.1. Hierarchical Decomposition
Upon reaching an unexpanded node n, we generate its child nodes Child(n) through expansion. Unlike traditional MCTS, which expands all actions from a predefined space, our expansion explores an undefined space of possible com- ponent formulations. We leverage LLMs to generate these potential formulations, using the partial formulation con- structed so far as context to ensure coherent expansions. Our process involves: (1) generating diverse candidate for- mulations through -asedd exploration, and (2) pruning trivially equivalent candidates to maintain a manageable yet
Optimization modeling inherently complex, involving multiple interconnected components. To manage this com- plexity and improve search efficiency, we propose decom- position of the formulation process. This approach allows us to sequentially explore each model component rather than searching for entire formulations at once, potentially leading to more efficient search.
Specifically, we structurally decompose the autoformula-
4

---

Autoformulation of Mathematical Optimization Models Using LLMs
Generate of candidate fommulations
equivalence by showing x exists where the systems dif- fer, while satisfiability indicates distinct formulations. We apply this check pairwise across candidates in ())ld(n), pruning trivially equivalent ones. Detailed SMT formulae are provided in Appendix B.
Cuncnt pattial solution
(2) Prune trivially equivalent candidates
While SMT solvers effectively detect equivalent formula- tions, their decidability varies across problem types (Monni- aux, 2016). Linear arithmetic over real and integer domains is generally decidable, but mixed-integer or non-linear func- tions may be undecidable depending on problem properties. When solver cannot determine equivalence, we conser- vatively treat formulations distinct, potentially explor- ing some redundant paths but avoiding premature pruning. While this is heuristic approach that has scope for future improvements, our empirical analysis indicates that it yields significant efficiency gains through pruning (Section 5.3). Additionally, since SMT solvers require consistent variable domains X, we apply them only to levels m2-M4 where nodes share decision variables. For level m1, which defines decision variables, query an LLM for pruning.
equivalent
(3) Estimate value function for each solution
R Ranking solutiooss
Figure2 2. Expansion and evaluation. Expansion involves gener- ating candidate formulations, which are then pruned to remove trivial equivalences. Remaining candidates are assigned a normal- ized rank score as value initialization.
diverse search space. The combined expansion and evalua- tion process is illustrated in Figure 2.
Generating candidate formulation. At node n, the LLM generates potential child nodes (containing formulations of the next component formulations) by conditioning on the partial formulation and problem description, which we denote as [LMastecaa 17, d). The LLM is queried through a structured prompt with three elements: > prob- lem description: the original natural language problem description d; partial formulation: the current partial formulation rī in JSON format; level-specific instruc- tions: guidelines for the current modeling stage, includ- ing output format and relevant considerations. We repre- sent formulations and request formulations using JSON format, where keys are descriptive labels and values are mathematical expressions. For example, when generat- ing possible inequality constraints, the LLM might return the formulation: ("material balance": 31 + 22 S 100, 0.8% + 0.622 2 75]. For each node expansion, we sample H E N hypotheses from the LLM's distribution: Child(n) [Mchild (h()) L. d), Vhe E [M],, where i chitt represents the h-th candidate component formulation.
3.2.2. EVALUATION
After expansion, each newly created child nodes undergoes an initial evaluation to estimate its value, guiding subsequent exploration. While it is possible to use uniform priors, we employ LLMs evaluate child nodes to provide informed value estimates, helping guide search toward promising for- mulations earlier. Specifically. the LLM is provided with all partial formulations of newly expanded child nodes, namely [Nchald Rehild E Child(n)). and instructed to assign a numerical rank (from to (Child(n))) based on its evalu- ation of formulation correctness, constraint feasibility and alignment with the original problem description). These ranks are then genter-normalizer dle rank centered at 0.5. We denote this normalized score s (Tehaaa) which is used to initialize the child node's value V Ypror Ppiio(Tehsstee (Nehsta) + s(nanaa). Subsequently, we retain the top I E N candidates, based on their normalized rank scores.
[o. 1]. with the mid-
Search pruning, After generating candidate formulations, we prune the search space to ensure diversity and effi- ciency. Specifically, we eliminate trivially equivalent for- mallations-eepresssiinn that differ only in syntax while remaining functionally identical (e.g.. 2x + 3y versus 3y + 2r). This pruning operation can be expressed as Child(n) prunirg(Chiddn)) We employ Satisfia- bility Modulo Theories (SMT) solvers to detect equivalent formulations (Barrett & Tinelli, 2018). For components m2- (objective functions, equality, and inequality constraints respectively), we represent each candidate formulation as a system of equations or inequalities. To compare two sys- tems S1 and Sy over domain X, we check the satisfiability of -VVx: (x) = 52(x))). Unsatisfiability proves
3.2.3. DUAL REWARDS AND BACKPROPAGATION
Terminal rewards. We continue expanding until a termi- nal node n, is reached, where ni, represents complete formulation (from root to terminal node). We evaluate the complete formulation using dual approach, combining assessments of both mathematical correctness and computa- tional model's performance to obtain reward r(ū)):
r(m,) = I (Es (perseee = =1) ET (ū, d) (4)
where I is the indicator function and c = parser( is our custom parser that converts each mathematical formu- lation into a computational representation. (M, d) is the LLM'' evaluation of the mathematical formulation's
5

---

Autoformulation of Mathematical Optimization Models Using LLMs
correctness, assessing how well it captures the problem re- quirements and constraints in d. Figovver ( (parser(f) is the solver's s binary feedback on whether the program was solved optimally. We note that this is an imperfect signal, as an incorrectly formulated model could be solved to optimal- ity despite not faithfully representing the original problem, highlighting the importance of dual evaluation.
Summary. Our MCTS-based algorithm iterates through the aforementioned steps, progressively constructing and refin- ing tree of possible formulations. We execute this process for T E N iterations, thoroughly exploring the space of potential models and identifying promising formulations.
The final output is a set of M E N, M < 'T' functionally distinct optimization models (achieved through search prun- ing), where each model defined by unique path through the tree. Formally, we express the overall algorithm as: (m(() ((0), 3.M = McTS) The superscript i in- dexes the functionally distinct formulation, and r(()) is the estimated value/reward of the corresponding terminal node.
To evaluate formulation correctness for complete models, we employ comparative evaluation approach rather than independent scoring. While LLMs could directly assign scores to each formulation, our empirical analysis showed mixed results with this approach-liket due to scoring inconsistencies when evaluating solutions in isolation. In comparison, relative comparisons yield more robust and consistent evaluations. However, the approach described in Section 3.2.2 is no longer practical (as each new solution would require re-ranking and re-computing reward for all previous formulations). Instead, we introduce compara- tive method where each formulation is evaluated against a consistent set of baseline models my. The LLM outputs a score in [0, 1). where values above 0.5 indicate preference for the candidate formulation over the baseline. Formally, we express this as E (i7;; d) LLM( Me ma; d). This approach ensures comparable rewards across all terminal nodes by maintaining a consistent reference point.
4. Related Work
Advances in LLMs. Recent works have demonstrated the substantial potential of LLMs in solving complex reason- ing tasks, including language understanding (Hendrycks et al., 2021), commonsense reasoning (Brown, 2020), log- ical reasoning (Wei et al., 2022: Yao et al., 2024), math- ematical problem-solving (Lewkowycz et al., 2022), and coding tasks (Chen et al., 2021). Of particular relevance are studies employing LLMs in optimization and search tasks, such as Bayesian Optimization (I.iu al., 2024), prompt optimization (Guo et al., 2023), evolutionary optimization (Yang et al., 2024; Liu et al., 2025), and symbolic program refinement (Madaan et al., 2024). In contrast, our focus is on leveraging LLMS to bridge the gap between natural language description and formal optimization models.
Backpropagation. Following the reward calculation, we backpropagate this value to update the statistics of all nodes along the trajectory. For each node in this path from root to terminal node, nhy we apply the following updates: (n) +
Autoformulation. Early work by Ramamonjison et al. (2023) introduced the first autoformulation competition. The competition focused on linear programming problems, but required predicting formulations in specific formats (i.c., entity problem tagging), using Pre-LLMM era NLP models with limited generalization beyond given formats. Recent advances by Xiao et al. (2023) and AhmadiTeshnizi et al. (2024) employed multi-agent LLM frameworks, where mul- tiple agents (e.g., coding and formulation agents) collabo- rate to generate and iteratively refine complete formulations. Our approach differs by decomposing the formulation into components, using MCTS for systematic exploration, and incorporating symbolic pruning and composite rewards to improve search efficiency. In parallel, Tang et al. (2024) de- veloped the first LLM specifically finetuned for optimization modeling using mixture of real and synthetic data.
(()))))(())+(())) ((n) + ((n) + N(x)+1
1, Vn E n.. Here, we increment the visit count (n) by 1 and update the value (n) with a weighted aver- age of its previous value and the new reward r(mt). This backpropagation process ensures that the tree gradually ac- cumulates more accurate estimates of node values. These updated statisties then inform the selection strategy in sub- sequent iterations. The node value used in selection is then ((n)= > (n) + (1 (n).
3.2.4. SELECTION
The selection step guides the search towards promis- ing regions of the tree. Starting from the root, the algorithm recursively selects child nodes using the Upper Confidence Bound for Trees (UCT): nehald In arg max maaaeeahhhh (((eetata)) + / N(nentta) (Kocsis& & Szepesvári, 2006). This process continues until reaching an unexpanded node. Here, nenaa is the selected child node, V ((neaaa)) is its estimated value, (n)) and N (Nohsta) are visit counts for the parent and child nodes respectively and w is an exploration constant. This formula balances exploitation (first term, favoring high-value nodes) with exploration (second term, favoring bess-visited nodes).
Planning. Recent research have also explored the inte- gration of LLMs with planning algorithms (Huang et al., 2024a), the most pertinent of which consider approaches that generate and select from multiple plans (Wei et al., 2022; Wang et al., 2023). Such approaches are particularly effective for complex tasks, where a single plan generated by I.M is likely to be suboptimal, thus requiring exploration. Yao et al. (2023) employed an LLM to generate multiple rea-
6

---

Autoformulation of Mathematical Optimization Models Using LLMs
Table Benchmark comparison. Formulation correctness results on four benchmarks containing LPs/MILPs.
Method
NL4OPT IndustryOR MAMO (ComplexLP)
ComplexOR
Finetuned methods
0am--88
85.7%
38.0%
39.3%
44.4%
Methods based on G2 T4
Standard
47.3%
28.0%
24.6%
9.5%
Reflexion
53.0%
36.0%
19.1%
Chain-of-Experts
64.2%
40.2%
38.1%
OptiMUs
78.8%
66.7%
Autoformulator (N=1)
85.24%
35.0%
43.8%
66.7%
Autofozmulator (N=3)
92.21%
42.0%
61.4%
72.2%
Autoformulator (All) 92.62%
48.0%
62.3%
72.2%
soning paths and self-evaluating choices to decide the next action. Hao et al. (2023); Zhao et al. (2024) employ LLMs as policy functions in MCTS framework, where potential ac- tions are generated through LLM calls. Plans are evaluated either through grounded feedback from the environment LLM self-evaluation, including the probability of 'good' actions (Hao et al., 2023), or a continuous score (Yuan et al., 2024). Compared with these search-guide methods, our work differs in hierarchical search, SM'T-based pruning, and comparative/ranking based evaluation correctness, inno- vations specifically tailored to autoformulation.
5.1. Benchmark Comparisons
Baselines. We compare against several methods: zero- shot prompting (Standard), the Ref lexion (Shinn et al., 2023), and three specialized aut- Ma-in-of-Exper
oformulators:
(Xiao et al., 2023) and Opt iMUS (AhmadiTeshnizi et al., 2024), both based multi-agent frameworks, and ORLM (Tang et al., 2024), Llama3-based model finetuned on mix of real and syn- thetic optimization datasets.
Analysis. We configure our method with = 10 candidate formulations,
children retained after pruning and scoring, and T 16 total rollouts. Unlike prior approaches that return a single model, our MCTS-bass based search gener- ates up to T distinct formulations. Accordingly, we report P Pass @ N metrics to capture performance across multiple candidates. Results in Table show that our method matches baseline performance with just one rollout, illustrating the efficiency hierarchical search decomposition. With three rollouts, we surpass all baselines, including the finetuned ORLM model. While additional rollouts further improve accuracy, gains taper off due to diminishing returns and increased computational cost.
5. Experiments
We present our experimental evaluation across three key areas. First, we benchmark our autoformulator against base- line approaches on real-world problems (Section 5.1). We then analyze two critical components: our ranking and com- parative evaluation methods for assessing formulation cor- rectness (Section 5.2), and search space pruning tech- niques for improved efficiency (Section 5.3). Section 5.4 concludes with insights on exploration diversity, perfor- mance across problem types, and failure modes.
Benchmarks. We evaluate our methods on four world benchmarks: NLP4OPT (Ramamonjison et al., 2023), a curated set of 244 linear programming problems (based on (Tang et al., 2024)); IndustryOR (Tang et al., 2024), consisting of 100 problems spanning linear, integer, and mixed-intege programming at various difficulty levels; ComplexOR (Xiao et al., 2023), with 37 real-world opera- tions research problems from diverse domains; and MAMO (Huang et al., 2024b), using the more advanced ComplexLP subset, which includes 211 problems.
Additional results are provided in Appendix A, including comparisons with two ablated variants our method that underscore the importance of structured tree search. We also present Best-of-N comparisons against ORLM (our closest competitor), where we select the candidate with the highest estimated value, highlighting the benefits of exploration in our framework.
5.2. Formulation Correctness Evaluation
Next, examine our formulation evaluation methods and their effects on search performance. Specifically, we analyze the estimated reward of complete formulations, and the estimated value of partial formulations.
Evaluations. Following (Tang et al., 2024; AhmadiTesh- nizi al., 2024), we report accuracy the proportion of problems where the discovered formulation yielded opti- mal objective values. All baselines and experiments use GPT4-0613 as the underlying LLM.
Complete formulation reward. To analyze our ap- proach to evaluate complete formulations, we considered

---

Autoformulation of Mathematical Optimization Models Using LLMS
Cumulative Soletions Found Rollouts
Groody path
0.5
Ramdom path
Percentags (96
04
of Soletion
42
0.
40
180
Cumulative # Solutions Found Solutions
Problems with at lesst (2) # unique formilations
Number of Rollouts
Figure Evaluation of value initialization. Comparison of node selection based on initial value estimates vs. random selection.
Figure 5. Rate of formulation discovery. Our method continues to discover unique and correct formulations during scarch.
A Nodes Afeer Priming and Seleetson
5.3. Gains in Search Efficiency
The goal of this experiment is to analyze the gains in search efficiency. Recall that, in each formulation stage, candidate child nodes are (1) pruned using SMT to eliminate redun- dant formulations, and (2) selective retention the top- candidates based on normalized rank scores, In Figure 4, we visualized the number of retained solutions of each filtering stage. We note that, average, around 20% of formulations are retained after the symbolic pruning stage, with further reduction after the selection stage. To quantify the efficiency gains, we compared our approach against non-hierarchical search method. The analysis reveals that a non non- non-hicrarchical approach would require 1000x more formulation genera- tions to produce the same number of unique formulations, demonstrating substantial savings in search budgets.
Phoceutinn (80)
dusision wutalles
dojective
equality coust.
Insquasit
Formmlaioon Step
Film
(1) Caafilao Iesedali
( SVT meaiin
Xeraalan look okistios
Eatindted ollicincc in sccumllltel pansation mve Srmmiaaik slags
deikan muilke
Ineqcalty
SMI
Namuiad
Figure 4. Improvements in search efficiency. Pruning and selec- tion by normalized rank score significantly reduces search space.
5.4. Performance Analysis
We conclude our evaluation by analyzing our method's per- formance through three lenses: (1) the rate of correct model discovery, (2) performance across problem categories, and (3) the underlying sources of error in formulated models.
all problems in IndustryOR where correct solution found, and collected the scores assigned to correct and in- correct formulations. We found that correct solutions were evaluated with higher scores than incorrect formulations, ob- taining a biserial correlation coefficient of 0.48 (p-value of 2.068le-3). 3). We compare this with a direct scoring method (Zhang al., 2024) that independently scores each formula- tion from 1-100. This yielded correlation of 0.23 (p-value of 1. 1.1185e-11 1), underscoring the effectiveness of our com- parative evaluation approach.
Formulation discovery rate. Figure describes the number of correct formulations discovered on IndustryOR as a func- tion of MCTS rollouts (where 48 total correct formulations were found). We observed that while additional rollouts consistently yielded more correct solutions, the search pro- duces diminishing returns: discovering the final 4 correct solutions required 10 additional rollouts. This illustrates that while our method continues to explore useful candi- dates, the marginal gain per rollout decreases, highlighting trade-off between coverage and computational budget.
(2) Partial formulation scores. To evaluate estimated prior scores, we first obtain a fully expanded tree using Depth- First Search, where each node can have up to three children. Then we used our comparative evaluation to obtain node scores. For evaluation, we compared the correctness of greedy formulation obtained by greedily selecting the highest scoring node in each level of the search tree and randomly obtained solution. Figure reveals that greedy solutions obtained from prior scores were significantly more accurate, with the gap increasing as the number of unique formulations contained in the tree increases.
Finegrained performance. We further examined accuracy across different problem categories in Table 2. Our method performs consistently well across categories, with no signifi- cant drop in performance for any specific type. Interestingly, we observed that categories with lower accuracy tended to exhibit greater average tree entropy, a measure of diversity in the generated search trees. This suggests that tree entropy
8

---

Autoformulation of Mathematical Optimization Models Using LLMS
may be useful indicator of uncertainty and potential predictor of formulation success.
further enhances efficiency by eliminating redundant formu- lations. Empirical evaluations across real-world benchmarks demonstrate our method's s's superior performance in formu- lating correct models, with notable efficiency gains from pruning and L.LM-based correctness evaluation.
Table 2. Finegrained results. By problem type and difficulty.
Accuracy
Entropy
Problem Difficulty
Future Work. Looking ahead, we see autoformulation as a promising domain where LLMs can meaningfully augment human expertise. Future research directions include devel- oping collaborative frameworks that integrate humans in- the-loop with autoformulator capabilities, potentially lever- aging active acquisition techniques (Astorga et al., 2024; Kobalczyk et al., 2025). Additionally, exploring advanced LLM- -aaddaaeddd-- methods, such as retrieval- augmented genera- tion (Lewis et al., 2020), can further enhance formulation processes. Our current method can be viewed as a form of test-time search, and its core principles, hierarchical decom- position, LLM-based evaluation of formulation quality, and search space pruning, can be naturally extended to inform LLM finetuning via outcome- or process-level supervision (Lightman et al., 2023; Wan et al., 2024). Advancing this field will also require the creation of large-scale, diverse benchmarks spanning a wider range of problem types and complexities. In particular, future benchmarks should go beyond integer and mixed-integer programming and include more challenging problems that demand advanced or cre- ative reformulations.
Easy
0.68
1.96
Medium
0.29
3.04
Hard
0.50
2.73
Problem Type
IP
0.55
1.65
0.42
2.17
MIP
0.52
3.32
Sources of error. To understand where our method fails, we conducted targeted expert evaluation on 18 autofor- mulated problems from the ComplexOR benchmark. An optimization expert manually reviewed each model and as- sessed the correctness of four key components: decision variables, objective function, equality constraints, and in- equality constraints. These assessments were also compared against our objective-value--bsse proxy for correctness.
Table 3. Sources of error in incorrect formulations.
Component
Dec var
Obj fun
Eq const
Ineq const
Agree % 82%
Error rate
23%
15%
54%
54%
The expert analysis revealed that constraint modeling, espe- cially inequality constraints, was the most frequent source of error. Issues included incorrect formulations, omissions, or misclassifications (c.g., treating an inequality as an equal- ity), with constraint-related errors present in over 50% of incorrect models. Notably, there was an 82% agreement be- tween the expert's judgments and our objective-valus proxy. In cases, the expert assessed the model be incorrect despite matching the correct objective value; in two others, models assessed to be correct by the expert produced slightly incorrect objectives. These findings suggest that while accu- racy based on comparing returned objective values serve as a strong and scalable proxy for formulation correctness, it does not always capture semantic correctness, highlighting the need for caution when interpreting matching objective values evidence of fully correct formulations.
6. Discussions
In summary, this work formally defines autoformulation for mathematical optimization models, establishing objectives, evaluation metrics, and identifying key challenges. We in- troduced a novel approach that frames autoformulation search problem, effectively leveraging the hierarchical struc- ture of optimization modeling. Our method integrates L.MMs as conditional hypothesis generators and evaluators of for- mulation correctness within an MCTS framework, system- atically exploring the hypothesis space of possible formula- tions. The introduction of search pruning using SMT solvers

---

Autoformulation of Mathematical Optimization Models Using LLMS
Impact Statement
neural estimation. In International conference on ma- chine learning, pp- 531-540. PMLR, 2018.
While automated optimization model formulation through LLMs offers promising efficiency gains, raises concerns about model reliability and verification challenges, as LLM- generated formulations may contain subtle errors that could lead to incorrect solutions in critical applications. As for- mulations grow more complex, verifying their correctness becomes increasingly challenging. Prior to deployment in critical applications, practitioners must have a clear under- standing of system capabilities and limitations, alongside robust institutional frameworks that ensure human oversight and expert validation.
Boyd, S. and Vandenberghe, L. Convex Optimization. Cam- bridge University Press, 2004.
Bramel, and Simchi-Levi, D. The Logic of Logisties: Theory, Algorithms, and Applications for Logistics Man- agement. Springer, 1997.
Brown, T. B. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Chen, M., Tworek, Jun, H., Yuan, Q.. Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
Reproducibility
We provide details on implementing our methods and repro- ducing results in Section 5 and Appendix B. We provide the code to reproduce our results at https:/ hpps://githbb. ator.!
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
com/ http://ommmnittooo/tooooooooooo
Acknowledgements
We thank the anonymous ICML reviewers, members of the van der Schaar lab, and Andrew Rashbass for many insightful comments and suggestions. Tennison Liu would like to thank AstraZeneca for their sponsorship and sup- port. Nicolás Astorga thanks W.D. Armstrong Trust for their support. Yuanzhang Xiao was supported by the National Science Foundation under Grant NRT-AI 2244574. This work was supported by Microsoft's Accelerate Foundation Models Academic Research initiative.
Coulom, R. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.
Dantzig, G. B. Origins of the simplex method. In A history of scientific computing, pp. 141-151. 1990.
Delgado, E. J., Cabezas, X., Martin-Barreine Barreiro, C., Leiva, V., and Rojas, An equity-based optimization model to solve the location problem for healthcare centers applied to hospital beds and covid-19 19 vaccination. Mathematics, 10(1 (M(11):180) 1825, 2022.
References
AhmadiTeshhizi, A., Gao, W., and Udell, M. Optimus: Scalable optimization modeling with (mi) lp solvers and large language models. arXiv preprint 10X112240..00172 2024.
Diamond, S. and Boyd, S. CVXPY: A Python-embeddeed modeling language for convex optimization. Journal of Machine Learning Research, 17(83)11-5, 2016
Fourer, R., Gay, D. M., and Kernighan, B. W. A modeling language for mathematical programming. Management Science, (645)::19-5555 1990.
Alizadeh, F. and Goldfarb, D. Second-order cone program- ming. Mathematical programming, 95(1):3-5,, 2003.
Astorga, N., Liu, T., Seedat, N., and van der Schaar, M. Active learning with llms for partially observed and cost- aware scenarios. Advances in Neural Information Pro- cessing Systems, 19-2079, 2024. URL https: //openreview
Fuchs, B.
Application of convex relaxation array syn- thesis problems. IEEE Transactions Antennas and Propagation, 02(2):634-64M) 2013.
Goemans, M. X. and Williamson, D. P. Improved approx- imation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), (2(5)) 1995.
Barrett, C. and Tinelli, C. Satisfiability modulo theories. Handbook of model checking, pp. 305-343, 2018.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Ben- gio, Y., Courville, A., and Hjelm, D. Mutual information
Goldsmith, A. J. and Varaiya, P. P. Capacity, mutual in- formation, and coding for finite-state markov channels. IEEE transactions on Information Theory, 42(3))868- 886, 1996.
Also available the wider lab repository https:// gi thub. www.///wwwsschaeeeeeeeeeo htt.eercccaaaarbbb/ttoooooaaaao.
10

---

Autoformulation of Mathematical Optimization Models Using LLMs
Guo, Q., Wang, R., Guo, J., Li, B., Song. K., Tan, X., Liu, G., Bian, J., and Yang. Y. Connecting large language mod- els with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.000
Representations, 2024. URL https: openrevie..ee..eeee net/ (irrrrummdd=00xttttmmmm
Liu, T., Huynh, N., and van der Schaar, M. Decision tree in- duction through LLMs via semantically-awar evolution. In The Thirteenth International Conference Learning Representations, 2025. URL https://////.. http:::///ppeeeveee.. net/ / wwwffoorrroooooommoo dintmntttdUUuuubhuoo.
2023.
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2024. URL https:ttp..... : www. , gurobi ccm.
Hao, S., Gu, Y., Ma. H., Hong, J. J., Wang. Z., Wang. D. Z., and Hu, Z. Reasoning with language model is planning with world model. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffee S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. International Conference on Learning Representations, 2021.
Mokhtar, M., Shuib, A., and Mohamad, D. Mathematical programming models for portfolio optimization problem: A review. International Journal of Mathematical and Computational Sciences, 8(2):428-433 2014.
Huang, X., Liu, W., Chen, X., Wang. X., Wang, H., Lian, D., Wang, Y., Tang, R., and Chen, E. Understanding the planning of Ilm agents: A survey. arXiv preprint arXiv:2402.025 02716, 2024a.
Monniaux, D. A survey of satisfiability modulo theory. In Computer Algebra in Scientific Computing: 18th Inter- national Workshop, CASC 2016, Bucharest, Romania, September 19-23, 2016, Proceedings 18, pp. 401-425. Springer, 2016.
Huang, X., Shen, Q. Hu, Y. Gao, A., and Wang, B. Mamo: a mathematical modeling benchmark with solvers. arXiv preprint ar 2ix::22405. 13144, 2024b.
Ramamonjison, R., Yu, T., Li, R., Li, Carenini, G., Ghad- dar, B., He, S., Mostajabdaveh, M., menitalebi-Dehkorte A., Zhou, Z., et al. Nl4opt competition: Formulating optimization problems based on their natural language descriptions. In NeurIPS 2022 Competition Track, pp. 189-203. PMLR, 2023.
Kobalczyk, K., Astorga, N., Liu, T., and van der Schaar, M. Active task disambiguation with LLMs. In The Thirteenth International Conference on Learning Representations, 2025, URL https::/ http///wpppeeee////////p/o wwtpriiwwwweeeeeeeeeeemmmeeeeeee n forum? id= JAMXRSXLFZ..
Kocsis, L. and Szepesvári, C. Bandit based monte-carlo planning. In European conference on machine learning, pp. 282-293. Springer, 2006.
Shinn, N., Labash, B., and Gopinath, A. Reflexion: an au- tonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.1 11366, 2023.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retreeval-augmented generation for knowledge- intensive nlp tasks. Advances in Neural Information Pro- cessing Systems, 33:9459 9474, 2020.
Sumers, T., Yao, S., Narasimhan, K., and Griffiths, T.
Cognitive architectures for language agents. Transac- tions on Machine Learning Research, 2024. ISSN 2835- 8856. URL https: httttp:://ppeeeee/eeat /opp/reeieeeeeeeeeeeeeeeeee net www//eeuuunneeeee forum? id= id=1i62cvf10J, Survey Certification.
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Soll T., et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Tang, Z., Huang, C., Zheng, X., Hu, S., Wang, Z., Ge, D., and Wang, B. Orlm: Training large language models for optimization modeling, arXiv preprint arXiv:2405.11743, 2024.
Wan, Z., Feng, X., Wen, M.. McAleer, S. M., Wen, Y., Zhang, W., and Wang, J. Alphazero like tree-search can guide large language model decoding and training. In Forty-first International Conference on Machine Learn- ing, 2024.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint 23X.:22305.20050 2023.
Wang. X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Confer-
Liu, T., Astorga, N., Seedat, N., and van der Schaar, M. Large language models to enhance bayesian optimization. In The Twelfth International Conference on Learning

---

Autoformulation of Mathematical Optimization Models Using LLMs
ence on Learning Reprexentations, 2023. URL https:
Wei, J., Wang, X., Schuurmans, D.. Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., al. Chain-of-thougee prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Williams, P. Model building mathematical program- ming. John Wiley & Sons, 2013.
Xiao, Z., Zhang, D., Wu, Y., Xu, L., Wang, Y. J., Han, X., Fu, X., Zhong, T., Zeng, J., Song, M., et al. Chain-of- experts: When ilms meet complex operations research problems. In The Twelfth International Conference on Learning Representations, 2023.
Yang. C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models as optimizers. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https: httpss:/ppenreviee.... Eorum? dieeendddBBbbbeee (--BB44GOOWLLL
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, L., Cao, Y., and Narasimhan, R. Tree of thoughts: Deliberate problem solving with large language models. In Thirty- seventh Conference Neural Information Processing Systems, 2023. URL
net/
forum (rrumiidd5Xclex0010
Yao, S., Yu, D.. Zhao, J., Shafran, I.. Griffiths, Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yates, R. D. A framework for uplink power control in cellular radio systems. IEEE Journal on selected areas in communications, 13(7):1341-1347, 1995.
Yuan, W., Pang, R. Y., Cho, K., Li,, X., Sukhbaatar, S., Xu, and Weston, J. E. Self-rewarding language mod- els. In Forty-first International Conference on Machine Learning, 2024. URL https: wpwpreview.e.e.cccc/cp n forum? 1d
Zhang, D., Huang, X., Zhou, D., Li, Y., and Ouyang, W. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b, 2024.
Zhao, Z., Lee, W. S., and Hsu, D. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Autoformulation of Mathematical Optimization Models Using LLMs
A. Additional Results
A.I. Pass@N vs. Best-of-N Results
We conduct two additional analyses: (1) evaluating the effectiveness of selecting the best formulation from our method using its estimated reward, and (2) comparing our method's performance against ORIM under the Pass@ N metric. For fair comparisons, we generate N independent samples from ORLM, matching our rollout count. We focus on ORLM for comparison, since our method already outperforms other baselines at N 1. These comparisons aim to highlight key distinction: method performs structured exploration with redundancy pruning, encouraging diversity and functional distinctness among formulations. In contrast, naive sampling (as with ORLM) often produces redundant or similar outputs due to the lack of guided search.
Table 4 reports the Best-of-N results, where we select the top-ranked formulation by estimated score (i.e., arg max over the N outputs). Our method selects the best formulation on over 90% of problems. consistently outperforming baselines and supporting the value of our evaluation mechanism. In Table 5, we compare Pass N performance with ORIM. Our method maintains a clear advantage, further demonstrating its strength in structured, Ceedback-driven search over functionally diverse solution candidates, an ability ORLM does not inherently possess.
Table 4. Pass vs. Best-of-N.
Method
NL4OPT IndustryOR MAMO (ComplexLP) ComplexOR
Pass N results
MCTS (N=1)
85,245
35.0%
43.8%
66.7%
MCTS (N=3)
92.21%
42.0%
61.4%
72.2%
Best-of-N results (selected using formulation reward)
MCTS (N=3)
88.11%
37.00%
53.3%
72.2%
Table 5. Pass@ N comparison to ORLM.
Method
NL4OPT
IndustryOR
MAMO (ComplexLPP) ComplexOR
ORLM (N=1)
85.7%
38.0%
39.3%
44.4%
ORLM (N=3)
90.2%
42.0%
56.3%
61.1%
MCTS (N=l)
85.24%
35.0%
43.8%
66.7%
MCTS (N=3)
92.21%
42.0%
61.4%
72.2%
A.2. Comparisons Against Additional Baselines
In this section of the Appendix, we present additional ablation studies to isolate the key contributors to the performance of our MCTS-based method. We compare against two additional baselines:
1. A Tree-of-Thought (implemented using Depth-First Search) baseline using the same hierarchical structure but without uncertainty guidance or search feedback;
2. A naive sequential sampling baseline with the same hierarchy but no structured search or pruning: each component is sampled sequentially, conditioned only on the partial formulation.
Results in Table 6 show that our method consistently outperforms both baselines across all benchmarks. This highlights the importance of structured decomposition, feedback-driven search, and redundancy pruning. Compared to Tree-of- Thoughh,, MCTS offers more effective exploration by using uncertainty and cumulative feedback to avoid suboptimal branches and refine search paths. The comparison with the naive baseline shows that decomposition alone is insufficient: without guided search or pruning, performance degrades, and manual inspection reveals higher incidence of invalid or redundant formulations.
Table 6. Benchmark comparisons against additional baselines. Method
NL4OPT IndustryOR
Additional baselines
Tree-of-Thought (N=3)
66.53%
28.0%
Sequent (N=3)
60.82%
28.0%
Autoformulatoz (N=1)
85. 85.24%
35.0%
outoformulate (N=3)
92.21%
42.0%
Autoformulate (All)
92.62%
48.0%
13

---

Autoformulation of Mathematical Optimization Models Using LLMs
B. Additional Details on Method
B.1. Formulation Equivalence Checks
SMT solvers offer a powerful approach for verifying equivalence between various components of optimization models (Barrett Tinelli, 2018). These tools can rigorously check if different formulations of objective functions, sets of equality constraints, or of inequality constraints are logically equivalent. By encoding the components as logical formulas within appropriate theories (such as línear arithmetic), SMT solvers determine if the formulations same conditions. For objective functions, the solver can check if the difference between two functions is always zero across the feasible region. This is formally described in Equation (5), For constraint sets, it can verify if they define identical feasible regions by checking that each constraint in one set is implied by the other set and vice versa, formally described in Equations (6) and (7). This approach not only ensures the correctness of model transformations or reformulations but also aids identifying redundant constraints and simplifying complex models. However, effectiveness of SMT solvers in this context depends on the nature of the optimization problem, as nonlinear highly complex formulations may pose challenges for current solvers.
satisfiable under the
1. For objective functions f(i) and f6);
Equivalent ((5) = Vx (x)
(5)
For sets of equality constraints g = f9h (i) and g00) {::
Equivalent( (g) , g(0)) = Vx X, (A
9b (x) =0) = (x) =0)
(1)
(6)
3. For sets of inequality constraints (100 -1ib (6) 3 and 100 (1((0) 31:
Aquivalent(h99 (h(0,,1))) = Vx E X. (A (x) = 1A (x) 50)
77)
B.2. Prompt Design
Template instruction
have problem in operational research:
#HPROBBEM DESCRIPTION###
have the following formalization:
formalization.die. ["parameters": O, rdecision.vrrablees:: O, "objective"; f
"epuallity.onsstraanttt"" () "iregalliy..onsrrinnss"" (}}
Template for generating parameters (depth=0)
You are an optimization modeling ing expert. Complete formal wwwlllizttionniicc based on the problem description, you should complete the "parametera" field, which
consists of assigning constants to descriptive variable names.
Only complete "parameters" and nothing else. Follow these guidelines:
1. Your primary responsibility is to define all the parameters from the problem description that will later be used to define decision variables, the objective, and constraints (both equality and inequality).
2. You may include additional parameters in a format suitable for facilitating the subsequent tasks of defining decision variables, the objective function, and constraints.
3. For parameters that involve multiple indices (e.g., x[i] or xii, use the most appropriate data structure, such lists, dictionaries, dictionaries with tuple keys, to represent them.
14

---

Autoformulation of Mathematical Optimization Models Using LLMs
4. For each parameter, include a clear, descriptive comment explaining its meaning.
5. Ensure that the paraneter names (keys) are descriptive and intuitive. Return only the python dictionary update (i.e.,
formalization.dict ("parameters"] ... following the described requirements.
Template for generating decision variables (depth 1)
You are an optimization modeling expert. Complete only the "decisiio..vrrialles"" field within the "formali iiationiaiion dict." based the provided problem description.
Ensure the decision.variable accurately model the optimizate problem. Each key-value pair in the dictionary must adhere to the following structure:
comprehensively cover all essential elements to
<key>: {
"description": <description), "type": <type>, "iteration_space": <space>
The structure should meet these requi remuntsmtns::
1. Each <key> represents decision variable that will later be used to implement the objective, equality, and inequality constraints in a Python program.
2. Replace <key> with symbolic name representing the decision variable. Ensure that each <key> represents a distinct decision variable with unique symbolic name.
Replace <description> with detailed explanation of the role of the decision variable in the optimization model.
3.
4.
Replace <type> with string representing the Gurobi variable type (e.g.,
GRB. INTEGE)), as this will be used to create the variable via Gurobi's addVar function.
5. If the decision variable is indexed, replace <space> with a string representing Python for-loop using list comprehension syntax to represent the index space. For this, assume direct access to these parameter variables (i.e., avoid using parameters [variables] syntax).
6. If the variable is not indexed, set <space> to None.
7. If the variable is indexed, do not write the index in the symbol (do not put the index when writing <key>).
8. You are encouraged to create decision variables that are general. two
decision variables represent the same concept write them as one key, creating appropriate iteration space.
Return only the Python dictionary update (i.e.,
innaaiizatio.dict " = ...) following the described requirements.
Template for generating objective functions (depth = 2)
You are an optimization modeling expert. Complete only the "objective" field within the itormalizaation.dieee based on the provided problem description. Do not complete any other fields. Follow these requirements:
1. Write the objective function mathematically using decision variables.
2. Preface the key-value pair with Python comment explaining the rationale
15

---

Autoformulation of Mathematical Optimization Models Using LLMs
behind the objective. DO NOT make commentary inside the mathematical
description.
3. Use parameter-defined variables instead of hard-coded values. Assume direct access to these parameter variables (1.e., avoid using parameters [variables] syntax).
4. The dictionary key must be 'min' or 'max', reflecting the nature the objective (minimization
maximization).
5. The dictionary value must be a string representation of the objective function based on the problem description, written in valid Python syntax. ["objective")
Return only the Python dicticnary update (i.e., ("objective") = "min": ...) following the
or described requirements.
"max":
Template for generating equality constraints (depth 3)
You optimization modeling expert. Complete the formalization.dict by filling in the aqualityy.cosstrannts field based on the problem description and the decision variables provided.
These constraints include border constraints, initialization, and equality constraints derived from the problem description. Do not complete the "inequal ity constraints" field. Follow these requitements:
1. Descriptive constraints: Each key the dictionary should represent a unique, clearly named constraint, with the value being string that describes the corresponding mathematical equality using "=="
2. Parameter Variables: Use parameter-defined variables instead of hard-coded values. Assume direct access to these parameter variables (i.e., avoid using parameters [variables] syntax).
Indexed Variables: For indexed decision variables, indicate the index within brackets (e.g., x(i]).
3.
4.
Handling Multiple Constraints: For similar constraints that repeat across indices variables, efficient representation.
Python for loops and list comprehensions for
5. String mathematical description: Note, the value (mathematical description) should be a single string. DO NOT use (join() or anything else. Even if it represents multiple constraints using a for loop.
6. No Inequality Constraints: Only define equality constraints. Inequality constraints will be handled separately by subsequent expert.
7. Comments: Include Python comment before each key-value pair, explaining the rationale behind the constraint.
Return only the Python dictienary update (i.e., formali i wiaiiiionnniieoo wwwoonndiceeeeuulllllaaaaaaeaaae
raints"] = ...) following these requi rements. Important: If the problem contains only inequality constraints and no equality constraints, return: formalization dict oy This will signal the need to focus on inequality constraints in subsequent modeling steps.
(None: None).
Template for generating inequality constraints (depth 4)
You are an optimization modeling expert. Complete the formalization.diict by adding the inequality.constra.ii field based on the problem description.
Follow these requirements:
1. Descriptive constraints: Each key the dictionary should represent a unique, clearly named constraint, with the value being string that describes the corresponding mathematical inequality.
16

---

Autoformulation of Mathematical Optimization Models Using LLMs
Parameter Variables: Use parameter-defined variables instead of hard-coded values. Assume direct access to these parameter variables (i.e., avoid using parameters [variables] syntax).
2.
Indexed Variables: For indexed decision variables, indicate the index within brackets (e.g., x(il).
3.
4. Handling Multiple Constraints:
similar constraints that repeat across Python for loops and list comprehensioos for
indices variables, efficient representation.
5. String mathematical. description: Note, the value (mathematical description) should be single string without using join or anything else. Even if it represents multiple constraints using a for 1oop.
6. Inequality Constraints Only: Include only inequality constraints. Exclude any constraints already covered under equality constraints.
7. Comments: Include Python comment before each key-value pair, explaining the rationale behind the constraint.
Return only the Python dictionary update (i.e.,
permalizationddict (oottt"innngg ""inqqualiyy constraints") [ = ...) following these requirements.
Important: Think carefully of inequality constraints that are not explicit in the problem description that should be considered.
after thinking you conclude the problem contains only equality constraints and no inequality constraints, return: formalization diet ("inequality constraints"] {None: None).
Template for pruning decision variables
Objective:
As an expert in aptimization modeling, your role is to evaluate multiple sets of decision variables provided for an operations research problem. You are responsible for determining if two or more sets of decision variables should be grouped together based on their equivalency from an optimization perspective. Task Breakdown:
Your grouping decision is critical for assisting subsequent optimization expert, who will define the objective function, equality constraints, and inequality constraints for each group. To facilitate this process, follow these precise guidelines:
Equivalent vecyy Criteria:
Same Objectives and Constraints: Two sets of decision variables should be grouped together if they result in the definition of the same objective function, equality constraints, and inequality constraints, even if the variable names differ.
1.
Conceptual Equivalency: Variable sets should be grouped together if, despite having different variable names, they define the same underlying concepts that ultimately lead to identical objectives and constraints (both equality and inequality).
2.
3. Non-Equivalencyy Conditions: Two sets of decision variables should not be grouped together if they lead to differences in any the following: Objective function, Equality constraints, Inequality constraints.
4. Naming Convention Irrelevance: The names of the decision variables are irrelevant for grouping purposes. Only the functional impact of the variables on the objective function and constraints should be considered. If two sets of variables lead to the same results, group them together, even if the names differ.
17

---

Autoformulation of Mathematical Optimization Models Using LLMs
By following these guidelines, you will help ensure that decision variable sets are clearly classified for the next expert in the process.
Please list your clusters as follows:
###
groups {
1: group_1,
n: group_n]
###
Where group_i is python list containing the names (string) of all the set of decision variables that are equivalent. One set of decision variables can only belong to one group. The list should consider at least one element. Important: Think carefully STEP BY STEP about your grouping decision, then conclude your assessment using the structured format provided above. Here are the current solutions:
solutions
()
Template for ranking expanded children nodes
You are an expert in optimi matiziiionn modeling. Using the formal formalization www.liiittiinniittctc as your current progress, you are tasked with selecting the optimal #VARTABLEH from the provided options.
Please follow these steps:
1. Carefully evaluate each potential #VARIABLE#.
2. Rank the variables from best to worst based on their suitability.
Present your rarkings in the following format:
###
rank {
1: solution_l,
n: solution_n)
###
where:
solution.1 represents the best #VARIABLE#.
solution.n represents the least suitable HVARIABLE..
Important: Think carefully STEP BY STEP about your ranking decision. Then conclude by listing the solutions in string format as structured above. Here are the possible solutions:
solutions ()
B.3. Benchmarks
NL4OPT: A widely adopted benchmark for Operations Research originating from a NeurIPS competition (Ramamonjison et al., 2023). Since the original NL4OPT provides only mathematical formulations, we utilize the labelled problem set prepared by (Tang et al., 2024). This dataset consists 289 linear programming problems for which optimal solutions were generated using GPT-4 with the assistance of experts, facilitating evaluation based on execution accuracy.
MAMO: benchmark specifically designed to assess mathematical modeling capabilities of Large Language Models. comprises two subsets: 652 easy and 21 complex linear programming problems, each accompanied by optimal solutions. Our experiments focus exclusively on the complex subset due the casier is relatively saturated comparison.
IndustryOR: Introduced in (Tang et al., 2024), this benchmark focuses on industrial applications of Operations Research. It includes 100 problems from 13 distinct industries, primarily covering linear programming (L.P), integer
18

---

Autoformulation of Mathematical Optimization Models Using LLMs
programming (IP), and mixed-integer linear programming (MILP), with the addition of one non-linear programming instance. The dataset categorizes problems into three difficulty levels.
This dataset encompasses 37 complex Operations Research problems across various application domains, including 12 MILP problems. For our evaluation, we utilize all 18 publicly available problems from this dataset".
ComplexOR:
B.4. Metrics
Execution Accuracy: The primary evaluation metric, defined by the correctness of executable code generated by a model. A response is deemed correct if its computed optimal value aligns closely with the ground truth solutions provided, allowing a margin of error within 5% following the evaluation protocol used in (Tang et al., 2024) (see official implementation).
@ass@k: This metric assesses inference quality by generating & candidate solutions for each problem. The model is considered successful at least one of these & candidates achieves correctness based on the execution accuracy criterion.
Best of N: This is inference strategy where the model generates N candidate solutions. A selection mechanism (which could be another model, a heuristic, a verification process) then chooses the single "best" solution out of the N candidates. In our case, the selection mechanism computes the average V values along the path from the root node to the final node, which is the complete formulation, selecting the candidate with the highest average.
C. Categorization of Autoformulation Challenges by Optimization Problem Structure
The exact challenges faced by an autoformulator depends on the nature of the problem d. Here, we provide categorization of optimization problems and their characteristics. To help elucidate different types of problems, we introduce two concepts. First, we define the set of correct formulations for a problem M(d) C .M as the set of all equivalent formulations that correctly model a problem d. Second, we introduce the set of original forms Mo(d) S. M(d)) the set containing the natural representations of the problem, typically the initial models an optimization expert would create. This is set, as can contain trivially equivalent formulations. Finally, we partition the set M into the set of convex problems Meony and the set of non-convex problems Mnone-
1. Type problems. These are problems where the original form is inherently convex, namely M. C MMeeme Examples include certain resource allocation problems that can be naturally formulated as linear programs. The challenge of solving Type I problems is to ensure that the problem is correctly represented (formulation correctness, i.e. H(d)n.M.(d) # 0), which would entail that it can be efficiently solved to global optimality.
2. Type II problems. These are problems where the original form is non-convex, but can be reformulated into an equivalent convex problem, namely Mo(d) C Muone but M(d) n MM * 0. In addition to formulation correctness, another challenge of solving Type II problems is to ensure the autoformulator can identify and apply appropriate reformulation strategies (e.g. change of variables) transform the non-convex into an equivalent convex form, namely H(d) n (M(d) n Meaw)) ) # 0. For such problems, evaluation extends beyond correctness include the ability to achieve global optimality through reformulation.
Type III problems. These are problems where the original form is non-convex and cannot be reformulated into a convex problem, namely M(d) S -Monne- In such cases, there are two general options: a) solve the non-convex problem using general-purpose algorithms (e.g. gradient descent), b) relax into convex problem that approximates, but is not equivalent to, the original problem (e.g. semidefinite relaxation of a Max-Cut problem (Goemans & Williamson, 1995)).
A crucial nuance here is that mathematically equivalent models, even when both are convex, can exhibit vastly different computational complexities. An example of this is quadratic programming and second-orde cone programming (SOCP) reformulations of the same problems (Alizadeh & Goldfarb, 2003). Although mathematically equivalent, SOCP formulations often allow for more efficient solution methods. Therefore, computational efficiency is an important evaluation metric across all three problem types, significantly impacting practical utility of model formulations. Appendix D, we provide concrete examples illustrate each type of optimization problems.
Phttps: //tp::://///t thub. com http:///w....pmttts..s//caaaaaaa Experts/treee//eeeeteete http::///hhhhcooooooooereetee//e
19

---

Autoformulation of Mathematical Optimization Models Using LLMs
D. Illustrative Examples of Problem Categorization
In this section, we provide examples canonical problems engineering and machine learning that belong to each the identified problem types. Specifically:
Type I: Problems that have precise mathematical model, which is convex in its original form. Examples are provided in Appendix D.1.
Type II Problems that have precise mathematical model, which is non-convex in its original form but can be reformulated as convex problem (sometimes additional assumptions are needed). Examples are provided in Appendix D.2.
Type III: Problems that have precise mathematical model, which is non-convex in its original form but can be relaxed to convex problem (sometimes additional assumptions needed). The difference from Type II is that the convex relaxation is not equivalent to the original problem. Examples are provided in Appendix D.3.
D.1. Examples of Type-I Problems
Overview, Maximizing mutual information a wireless channel.
Mutual information a quantity that measures the divergence between two random variables, with applications in wireless communications (Goldsmith Varaiya, 1996) and in data science (Belghazi al., 2018). Here, we describe it in the context of maximizing Shannon capacity in wireless communications.
We consider discrete memoryless channel with an input random variable X E f1....,f} f) an output random variable Y E [1,...,)] y). and a channel transition matrix P E Ryxe with the element on the j-th row and the i-th column being Pyt = prob (Y = x X= =).
Input X
Transition Probability P
Output Y
Our goal is to choose the optimal probability distribution of input denoted x e RR with x; = prob (X i), in order maximize the mutual information between input X and input Y
y
I(x;Y)
J.pr log,
Σk
j=
The optimal value of the problem is called Shannon capacity.
This problem is convex in its original form:
Maximize
Σ
Σ
Pji loga Pfi
Σ
Pjtts
loga
Σ
1=1
1=1
1=1
i=
i=
subject to
2 0,
i= 1,
i
(8)
Σ 24
1
Reformulation strategies: None.
Difficulty in reformulation: Not applicable.
Difficulty in solving the reformulated/originall problem: Easy.
D.2. Example of Type-II Problem
Overview. Power control to satisfy SINR requirements with minimum power usage (PC-MinPower)
We consider the problem of determine the transmit power of pairs of transceivers. They operate in the same frequency at the same time, hence causing interference to each other. The problem data is channel gain matrix E R(xE where gij
20

---

Autoformulation of Mathematical Optimization Models Using LLMS
the channel gain from transmitter j to receiver the noise power vector R with o2 the noise power receiver i, and the minimum SINR requirement vector R with " the minimum SINR required by the transceiver i.
Our goal is to choose the transmit power, denoted z E R4, with being the power of transmitter i, in order to minimize the total transmit power while satisfying the SINR requirements of each transceiver (Yates, 1995).
This problem non-convex in its original form:
Minimize
(9)
subject to
2 Mi
i= 1, I....,
juh Diy3t + d;
But it is not hard to observe that the constraints of SINR requirements can be reformulated as linear constraints, resulting in a LP:
Minimize
(10)
Σ Egojr + oi
subject to
Ia- 2 7s
i i= 1, I....,
Reformulation strategies:
Transformation of function.
Difficulty in reformulation: Easy (straightforward observation).
Difficulty in solving the reformulated/original problem: Easy (the reformulated problem is LP),
D.3. Examples Tyle-IIl Problems
consider the same setting as Beamform Seaaffor--Minniiin MinSidden fiiniideoobe, But here, our goal is to maximize the gain at the target direction Par. while limiting the ripple effect directions f...... 6n outside the target area.
This problem is non-convex: (Fuchs, 2013)
Maximize
[G(w; g
(1I)
subject 1/8 < [G(w; S i- i,,,..,,...
non-convex because we maximize convex function (i.e., norm) and have constraints on convex functions greater than or equal to a constant.
In this case, the standard semidefinite relaxation technique be used, which "lifts" the problem to higher dimensions. Specifically, we define a rank-1 semidefinite matrix W ww Then the gain at direction 0 satisfies
GG(;; 0) trace (E(0.) W),
where E(0,) = e(f:)" e(f,)T E Cuxn with
Ma, cos 0 sin 0) 1
e(0;)
(((z cos 01 sin
With the new matrix variable W, we have the following equivalent problem:
Maximize
trace (E(Otar) W)
subject
(1/8)² trace (E(0.) W) 82, i= 1,...,
(12)
W > 0,
rank (W) 1.
21

---

Autoformulation of Mathematical Optimization Models Using LLMs
Here, the only nonconvexity comes from the rank constraint, By removing it, get the following convex relaxation:
Maximize
trace (E(r))) W)
subject to
(1/6)² < trace (E(0;) W) S 8, i ,,..., W ž 0.
(13)
In general, we need to recover an approximate solution vector w from the solution matrix W. Under certain conditions (e.g., uniform linear arrays), we can guarantee to recover the exact solution vector.
Relaxation strategies: Semidefinite relaxation (SDR).
Difficulty in reformulation: Easy (standard SDR techniques were used).
Difficulty in solving the reformulated/orrggnnl problem: Medium (the relaxed convex problem is SDP, and recovery methods are needed).
E. Effect of Formulation on Optimality, Solution Time
In this section, present simulation results comparing the performance of various optimization solvers the PC.MinPower problem, both in its original non-convex form and in reformulated convex form (see Appendix D.2). We consider problem instances with é = 10 and é = 100 users (i.e., l optimization variables). For each instance, we evaluate the solvers in terms of success rate, optimality gap, and average solve time over 100 random samples.
E.1. Experimental Setup
The PC_Min Minnooowrr problem aims to minimize the total power consumption in system while satisfying certain constraints. The original formulation of this problem is non-convex, which can pose challenges for optimization algorithms. However, by reformulating the problem, it can be converted into an equivalent convex problem, which is generally easier to solve efficiently.
We evaluated the following solvers:
General-Purpose Solvers:
TRCA: Trust-Region Constrained Algorithm
- SLSQP: Sequential Least Squares Programming.
COBYLA: Constrained Optimization BY Linear Approximations.
COBYQA: Constrained Optimization BY Quadratic Approximations.
Convex Program Solvers:
-LLARABE:: A conic optimization solver.
ECOS: Embedded Conic Solver.
-CCS: Splitting Conic Solver.
OSQP: Operator Splitting Quadratic Program Solver.
For each solver and problem instance, we recorded:
Success Rate: The percentage of runs where the solver successfully found a feasible solution.
Optimality Gap: The difference between the objective value obtained by the solver and the known optimal value.
Average Solve Time: The average computation time (in seconds) required by the solver.
E.2. Results
Tables 7 and 8 present the performance of the solvers for problem sizes 10 and 100, respectively.
E.3. Discussion
The results demonstrate several key observations:
22

---

Autoformulation of Mathematical Optimization Models Using LLMs
Table
Solver Performance for
10
Original Nonconvex Problem
Reformulated Convex Problem
Solver
Type
Success
Optimality Cap
Time (s)
Success
Optimality Gap
Time (s)
General-Purpose Solvers
TRCA
General-Purpon
100%
7.31 x 10
0.0399
100%
1.25 x 10
0.0420
SLSQP
Gereral-Purpose
100%
6.47 x 10-7
0.0019
100%
6.48 x 10-7
0.0009
COBYLA
Purpoee
67%
2.94 x 10-6
0.0073
100%
7.15 x 10-6
0.0039
COBYQA
General-Purposs
0%
6%
9.80
14.4067
Convex Program Solvers
CLARABEL
Convex Solver
100%
6.32 10
0.0002
ECOS
Convex Solver
100%
6.16 10
0.0001
SCS
Convex Solver
100%
4.45 X 10-7
0.0002
OSQP
Convex Solver
100%
6.48 10-7
0.0003
Table 8. Solver Performance for = 100 Users
Original Nonconvex Problem
Reformulated Convex Problem
Solver
Type
Success Optimality Gap Time (s)
Success
Optimality Gap
Time (s)
General-Purporr Solvers
TRCA
General-Purpose
100%
7.75 10-2
0.6628
100%
1.28
10
0.6856
SLSQP
Gerpral-Purpose
100%
1.04 10-6
0.0750
100%
1.04
10
0.0298
COBYLA
Purprel
0%
6.0764
100%
2.86
10-5
9.9629
COBYQA
General-Purpose
0%
0%
Convex Program Solvers
CLARABEL
Convex Solver
100%
6.22
10-1
0.0121
ECOS
Convex Solver
100%
9.91
10
0.0097
SCS
Convex Solver
100%
1.05
10 10-6 6
0.0055
OSQP
Convex Solver
100%
1.04
10-6
0.0110
Importance of Problem Reformulation: For general general-purpose solvers, reformulating the original nonconvex problem into an equivalent convex problem significantly improves solution quality. This improvement is more pronounced for larger problem sizes (f 100). For instance, COBYLA's success increased from 0% 100% when the problem was reformulated.
Solver Selection Matters: Different general purprose solvers exhibit varying performance levels. SLSQP consistently achieves near-zero optimality gaps and high success rates with relatively low solve times across both problem formulations and sizes. In contrast, COBYQA fails find feasible solutions in most cases, highlighting the necessity of careful solver selection.
Performance of Convex Program Solvers: For the reformulated convex problem, convex program solvers (CLARABEL. ECOS, SCS, OSQP) show excellent and consistent performance. They all achieve 100% success rates, negligible optimality gaps, and minimal solve times. The differences among these solvers are minimal, suggesting that any of them would be suitable for solving the convex formulation efficiently.
These findings underscore the importance of problem reformulation and solver selection optimization tasks. Reformulating non-convex problem into convex one can significantly enhance the performance of general purppse solvers. Additionally, selecting the appropriate solver crucial, as it can greatly impact the success rate and computational efficiency.
23


Reviews:
[{"review_id": "cTbXurS66w", "reviewer": "Reviewer_fWQR", "summary": "This paper introduces autoformulation, the automated translation of natural language problem descriptions into solver-ready mathematical optimization models, addressing the reliance on human expertise in traditional modeling. The proposed method integrates LLMs with MCTS to hierarchically decompose and systematically explore the model space, enhanced by symbolic pruning to eliminate redundant formulations and LLM-based evaluation for guided search. Empirical results on linear and mixed-integer programming benchmarks demonstrate significant performance improvements over existing approaches, showcasing the effectiveness of combining hierarchical exploration, pruning, and dual reward mechanisms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We appreciate the reviewer’s detailed and thoughtful evaluation and positive feedback.*\n\n---\n\n### [P1] MCTS motivation\n\nThank you for the thoughtful question. We interpret this as raising two related concerns: **(1)** why use a tree-based search framework at all, and **(2)** why choose MCTS specifically.\n\n**(1) Tree-based search rationale.** Although the search depth is modest, optimization models naturally decompose into hierarchical components—decision variables, objective function, and constraints—making tree-based search well-suited. This structure enables:\n* Clearer credit assignment, isolating which components contribute to success;\n* Efficient feedback sharing across subtrees;\n* Tractable redundancy pruning via SMT, which operates more effectively at the component level.\n\nThese factors contribute to focused exploration, efficient reuse of components, and better search guidance. These benefits are lost when searching directly over complete formulations, resulting in a flat, entangled space.\n\n**(2) Why MCTS.** We choose MCTS over alternative tree search methods (e.g., DFS, BFS, A*) for two reasons:\n* Exploration under uncertainty. MCTS balances exploration and exploitation using value estimates and UCB, making it better-suited for the inherent ambiguity in autoformulation, as it progressively focuses on promising branches. This stands in contrast to deterministic tree-based algorithms.\n* Feedback-driven search. MCTS uses backpropagation to update its search policy, whereas other methods follow fixed traversal policies (without adjusting the search from observed outcomes).\n\n**Empirical evidence.** We point to three existing lines of evidence supporting the efficacy of our MCTS framework:\n* Sustained exploration: our method continues discovering novel, functionally distinct formulations over iterations (`Fig 5`);\n* Search efficiency: SMT-based pruning reduces redundancy by two orders of magnitude, improving efficiency (`Fig 4`);\n* Improved search guidance: our comparative scoring method provides more robust feedback, enhancing search guidance (`S5.2`).\n\n**Additional results.** Following your feedback, we further isolate our MCTS method's contributions through two baselines:\n* A Tree-of-Thought (DFS) baseline with the same hierarchy but no uncertainty guidance or search feedback;\n* A naive sequential sampling baseline (same hierarchy) without structured search or pruning (i.e., each component is sampled sequentially, conditioned on the partial formulation).\n\nResults from these additional comparisons are provided in **[P2]** and are referenced for further analysis.\n\n\n---\n\n### [P2] Tree-of-Thought\n\nThank you for this comment. Please find a conceptual comparison below:\n\n| **Dimension** | **Autoformulator** | **Tree-of-Thought (ToT)** |\n|---|---|---|\n| **Tree structure** | Structured, domain-specific: variables → objective → constraints | Free-form, unstructured intermediate thoughts |\n| **Node expansion** | LLM-generated components + SMT-based pruning | LLM-generated thoughts; no symbolic pruning |\n| **Node evaluation** | LLM-based comparative scoring of nodes and full models | LLM self-evaluation of nodes in isolation |\n| **Search strategy** | MCTS with UCT (uncertainty-guided, feedback-driven) | Greedy, DFS, or BFS; no feedback or uncertainty modeling |\n\nCompared to ToT, our approach incorporates (1) structured, domain-specific decomposition, (2) uncertainty-aware, feedback-guided exploration, and (3) symbolic pruning to improve efficiency and reduce redundancy.\n\n**Empirical comparison.**  We include both Tree-of-Thought and the sequential sampling baseline in our [updated results](https://imgur.com/a/2NjUrAA). Our method consistently outperforms both across all benchmarks, underscoring the importance of structured decomposition, feedback-driven search, and redundancy pruning. Compared to ToT, MCTS enables more effective exploration by leveraging uncertainty and cumulative feedback to avoid suboptimal branches and refine search trajectories. The comparison with the naive baseline highlights the limits of decomposition alone: without guided search or pruning, performance deteriorates, and manual inspection reveals more invalid or redundant formulations.\n\n**Actions taken.** We have integrated these additional baseline results into the updated manuscript in `Table 1`.\n\n---\n\n### [P3] Other comments\n\n* **Figure captions:** Thank you for this suggestion; we have revised the captions to include more detailed and self-contained explanations.\n\n---\n\n*Thanks again; we hope our responses addressed your concerns and would appreciate your consideration in updating the score. We welcome further discussions.*", "claims_and_evidence": "see the section Strengths And Weaknesses", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Reasonable experimental designs and analyses", "relation_to_broader_scientific_literature": "This paper bridges advancements in LLMs and MCTS by introducing a hierarchical, pruning-enhanced framework for automated optimization modeling, extending their applications to mathematical formulation challenges while addressing efficiency and correctness evaluation gaps in prior autoformulation research.", "essential_references_not_discussed": "Essential References are well-discussed.", "other_strengths_and_weaknesses": "### **Strengths**\n\n1. The paper is well-organized and easy to follow, with a clear structure that enhances readability.\n2. It gives a deep analysis of the current challenges in this direction\n3. Comprehensive ablation experiments are conducted to evaluate each component of the proposed pipeline.\n\n### **Weaknesses**\n\nI am not an expert in this field, so my current stance is a tentative weak acceptance. My recommendation could change following insights from other reviewers’ comments or if the authors provide more clarifications during the rebuttal period.\n\nThe authors propose a pipeline that leverages LLMs integrated within an MCTS framework for autoformulation tasks. However, I have several concerns that merit further explanation. First, the tree search employed in the proposed method is notably shallow (consisting of only four layers). I am curious about how the use of MCTS in this context yields substantial benefits over more straightforward or naive search algorithms.\n\nFurthermore, the pipeline bears some resemblance to the tree-of-thought framework, albeit with what appears to be more advanced search strategies tailored specifically for autoformulation. I would appreciate a detailed comparison between the proposed method and the tree-of-thought approach. Including the tree-of-thought method as one of the baseline comparisons in the experimental evaluation would provide a more comprehensive context for assessing the merits of the proposed approach.\n\nAdditionally, I suggest that the authors enhance the figure captions with more detailed explanations. As it stands, I found myself repeatedly referring back to the main text to fully understand the figures.", "other_comments_or_suggestions": "No.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I34JakW4Kw", "reviewer": "Reviewer_Pqeu", "summary": "This work studies the autoformulation for mathematical optimization models, or the task of building an optimization model from natural language prompts describing the problem. The approach begins by defining the construction of an optimization model as a hierarchical task, with steps of selecting parameters and decision variables, an objective function, equality constraints, and inequality constraints. These tasks are treated hierarchically using Monte Carlo tree search, where LLMs are used to generate candidate nodes. The authors provide some creative ways to prune trivially equivalent nodes, as well as to score the candidate nodes.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We appreciate the reviewer’s detailed and thoughtful evaluation and positive feedback.*\n\n---\n\n### [P1] Additional analysis\n\nThank you for this thoughtful suggestion. While objective-value correctness is a standard metric in prior work, we agree that it is an imperfect proxy. To address this, we conducted a targeted expert evaluation on 18 autoformulated problems from the ComplexOR benchmark.\n\nAn optimization expert manually reviewed each generated model and assessed the correctness of its major components—**(1)** decision variables, **(2)** objective function, **(3)** equality constraints, and **(4)** inequality constraints—along with an overall correctness label. This allowed us to identify sources of modeling errors and compare agreements between expert assessments and our objective-value-based proxy.\n\n\n| Component | Dec var | Obj fun | Eq const | Ineq const | Agreement % |\n|---|---|---|---|---|---|\n| Error rate among incorrect models | 23% | 15% | 54% | 54% | 82% |\n\n**Analysis.** The expert's analysis indicates that the most common sources of error were related to constraint modeling---particularly inequality constraints (e.g., misclassification as equality constraints, incorrect formulation, or omission)---with both equality/inequality constraint errors accounting for over half of all incorrect formulations. Additionally, we observed 82% agreement between the expert’s correctness judgments and the objective-value proxy. In two cases, the expert flagged structural errors despite a correct objective value; in two others, the expert deemed the model correct despite a mismatch in optimal objective values. This suggests that objective-value correctness is a useful but imperfect proxy for full structural accuracy.\n\n**Actions taken.** We have incorporated these findings into the manuscript and now highlight expert evaluation as a valuable future direction for benchmark development and model assessment.\n\n---\n\n### [P2] Other problem types\n\nThank you for this observation. Existing benchmarks in this space—NLP4OPT, IndustryOR, MAMO, and ComplexOR—currently consist exclusively of LP and MILP problems, which shaped the scope of our evaluation. Expanding to nonlinear or more complex problems is an important direction for future work, though it first requires the development of suitable benchmarks.\n\nWe note that LPs and MILPs remain highly relevant in practice: according to [R1], 61% of real-world OR problems are MILPs and 41% are LPs. These classes also present the same meaningful modeling challenges **[C1-C3]** for autoformulation, making them a useful starting point.\n\n**Additional results.** That said, we agree that broader evaluations are valuable. Based on your comment and related feedback, we extended our experiments to additional datasets to further strengthen our evaluations. While these new benchmarks, ComplexOR and the ComplexLP subset of MAMO, still consist of LPs and MILPs, they feature more complex problems and serve as a stronger assessment of generality. Our method outperforms all baselines on both, as shown in our updated [Table 1](https://imgur.com/a/Rl5AGhv).\n\n[R1] Gurobi State of Mathematical Optimization 2023 Report https://stage.gurobi.com/resources/report-state-of-mathematical-optimization-2023/\n\n---\n\n*Thanks again; we hope the reviewer’s concerns are addressed and welcome further discussions.*", "claims_and_evidence": "The claims that the tree search, pruning steps, and scoring steps are all effective are well evidenced by comparison against LLM benchmarks and ablation studies in Sections 5.1-5.4.", "methods_and_evaluation_criteria": "The proposed method is tested on two problem sets, NLP4OPT and IndustryOR, and compared against several recent methods for autoformulation. The main evaluation criterion used is the proportion of generated formulations returning the correct optimal objective value. While this is an easy proxy for model correctness, it would be more interesting to have even a limited number of instance that are checked by a human optimization expert (i.e., is this the model an expert would have built?). Moreover, the majority of models tested are linear programs; it would be more interesting to see how the proposed approach generalizes (or requires improvements) in nonlinear or discrete settings.", "theoretical_claims": "The only theoretical claim made appears to be that the proposed SMT-solver can correctly prune trivially equivalent models, which is described in the Appendix.", "experimental_designs_or_analyses": "The experiments are designed well, and several ablation experiments are performed. It would be interesting to see more evaluation criteria in addition to just having the correct objective value. For example, how many of the models correctly model, overapproximate, or underapproximate the feasible region? How many of the models have variables and parameters correctly defined?", "relation_to_broader_scientific_literature": "This paper follows on a recent line of work on autoformulation of optimization models.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kuwuNFL8R0", "reviewer": "Reviewer_14k8", "summary": "This paper proposes a search-based autoformulation of mathematical optimization problems. The authors provide a formal definition of autoformulation and use MCTS to construct the formulation. Experiments demonstrate the method can outperform the baselines.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We appreciate the reviewer’s detailed and thoughtful evaluation and positive feedback.*\n\n---\n\n### [P1] Addressing challenges\n\nThank you for raising this point. However, we believe this may stem from a misunderstanding. Please allow us to clarify how our method addresses all three challenges:\n\n| Challenge | Description | Our approach | Location in paper |\n|----------|-------------|--------------|-------------------|\n| **[C1] Problem-dependent hypothesis space** | The formulation space is vast and problem-specific, making manual definition infeasible. | LLMs serve as problem-conditioned hypothesis generators that implicitly define the hypothesis space; candidate components are expanded hierarchically.| `L238–L259 L` |\n| **[C2] Efficient search** | Explore the space efficiently under uncertainty, avoiding redundant or unpromising paths. | MCTS over a hierarchical decomposition balances exploration-exploitation; symbolic pruning eliminates trivial equivalences. | `L182–L208 R`; `L260 L–L240 R` |\n| **[C3] Model evaluation** | Evaluate whether generated formulations faithfully represent the original problem. | Partial evaluation via node-level ranking during expansion; comparative scoring of complete formulations. | `S3.2.2`; `S3.2.3` |\n\n---\n\n### [P2] Evaluations\n\nThank you for pointing this out—we agree with your concern. Our method was originally evaluated using a pass@N metric (i.e., success if any of N iterations is correct), while baseline comparisons were reported under pass@1.\n\n**Actions taken.** We have re-evaluated ORLM under pass@N by generating N independent samples to match the iteration count (Note: our method still outperforms non-ORLM baselines at N=1). We report in [this table](https://imgur.com/a/1DUogjH) this fairer comparison (now added to the appendix), and clarified evaluation metrics in the main text. We observe that our approach maintains a clear advantage even under pass@N. This reflects a core strength of our approach: it performs structured exploration over functionally diverse formulations, which ORLM does not inherently support. We appreciate your feedback, it helped improve the rigor of our evaluation.\n\n---\n\n### [P3] Selection\n\nIn our method, each MCTS iteration produces a complete formulation along with a score from our evaluation mechanism, allowing us to select the highest-scoring one if desired. **Actions taken.** We now report best-of-N accuracy based on score-ranked outputs in [this table](https://imgur.com/a/0haw2V6) (i.e., selecting best-of-N formulations using greedy selection). Our selected formulations continue to outperform baselines (specifically, our mechanism selects the best formulation on >90% of problems), supporting the effectiveness of our scoring mechanism.\n\n---\n\n### [P4] MCTS\n\nThank you for raising this point. We refer the reviewer to our response to Reviewer `fWQR` (**[P1]**) for a detailed discussion of why optimization modeling benefits from hierarchical decomposition and MCTS-based structured exploration. **Empirical evidence.** We include two additional baselines: (1) a Tree-of-Thought using a different tree search strategy, and (2) a naive sequential sampling baseline without tree search. Our method consistently outperforms both, indicating that even at shallow depths, our MCTS framework yields meaningful gains through principled exploration and pruning.\n\n---\n\n### [P5] Additional benchmarks\n\nWe agree that broader benchmarking is important. In response, we added experiments on both MAMO (focusing on the more challenging ComplexLP subset) and ComplexOR. Due to time constraints, we prioritized ComplexLP over EasyLP, which is relatively saturated. On both benchmarks, our method outperforms all baselines, further validating its effectiveness. Results have been added to the updated [Table 1](https://imgur.com/a/Rl5AGhv).\n\n---\n\n### [P6] Other comments\n\n* **Baseline results on IndustryOR.** We attempted to run Chain-of-Experts and Optimus on IndustryOR using their released code but encountered compatibility issues. As noted by the IndustryOR authors, these methods require non-trivial adaptation. Due to time constraints, we were unable to complete this during the rebuttal, but we plan to include these results in a future version (if accepted).\n* **App D.** App D is not meant to evaluate our method directly, but to explore how solver and formulation choices impact downstream performance. While not central, we included it for context and will trim it in a future version to improve clarity.\n\n---\n\n\n*Thanks again; we hope our responses addressed your concerns and would appreciate your consideration in updating the score. We welcome further discussions.*", "claims_and_evidence": "The paper proposes three challenges of autoformulation. \"(1) the vast, problem-dependent hypothesis space, (2) efficient and diverse exploration of this space under uncertainty, and (3) evaluation of formulation correctness against problem description.\" However, the paper seems to address only the second challenge.", "methods_and_evaluation_criteria": "1. My biggest concern is the unfair experiment settings. The MCTS can have many rollouts. If I understand correctly, the authors think it is successful in solving the problem if any one of the rollouts is correct. This is unfair since the baselines have only one \"rollout\" during evaluation. As shown in Figure 5, if we set the rollout number to be 1, the accuracy in Figure 5 is under 36%, which is inferior to the 38% of ORLM. \n2. The author may want to clarify how to choose the best answer in the rollouts. In my understanding, the method should provide a final answer among the answers given in different rollouts, instead of providing all the answers as final outputs.\n3. In this paper, the depth of the MCTS is under 5. The complex MCTS seems not necessary for the shallow search depths.", "theoretical_claims": "This paper does not contain any proof for theoretical claims.", "experimental_designs_or_analyses": "1. The dataset is limited. The paper only conducts experiments on two datasets in the main paper and more datasets are needed, such as the MAMO dataset and ComplexOR dataset.\n2. Some experimental results are missing. I wonder whether the results of Reflextion/Chain-of-Experts/Optimus are missing on the IndustryOR dataset.", "relation_to_broader_scientific_literature": "This paper is related to autoformulation for operations research problems.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "Please see the comments above. I suggest the authors refine the experiments part of this paper.", "questions_for_authors": "Please see the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKpEFh5FJ2", "reviewer": "Reviewer_nBsi", "summary": "This paper proposes an approach for autoformulation—the automated creation of solver-ready mathematical optimization models from natural language problem descriptions. The authors frame autoformulation as a search problem and leverage Large Language Models (LLMs) with Monte-Carlo Tree Search (MCTS) to systematically generate, explore, and evaluate candidate optimization formulations.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We appreciate the reviewer’s detailed and thoughtful evaluation.*\n\n---\n\n### [P1] Engineering vs theory\n\nThank you for the thoughtful comment. Our framing of optimization modeling as a three-step process (requirements gathering $\\rightarrow$ mathematical model $\\rightarrow$ computational model) is intended to reflect the **engineering** side of the modeling pipeline. We agree that in many academic OR contexts, the emphasis lies in **theoretical** justifications—such as proving optimality bounds, analyzing relaxations, or exploring structural properties—which are essential to advancing the science of optimization.\n\nWhile theoretical work is crucial, we also believe our focus remains valuable (further discussed in **[P2]**). For domain scientists who are not optimization experts, this technology can help democratize modeling by lowering the barrier to entry. For experts, the autoformulator can accelerate prototyping and iteration. When theoretical validation is ultimately required, our system can automate time-consuming or error-prone steps, surface promising candidate formulations, and free up modeling experts to focus on deeper theoretical analysis.\n\n**Actions taken.** We have updated the manuscript to acknowledge this theoretical dimension more clearly. Specifically, we revised lines `L10–L16 R` (Introduction) to highlight the role of theoretical modeling and added a discussion in `L405–L420 R` (Discussion) on future directions that connect autoformulation with theoretical reasoning.\n\n---\n\n### [P2] Real-world utility\n\n**Practical utility.** We believe that in many domains—such as logistics, healthcare, energy, and manufacturing—there exists a real and growing need to bridge the gap between domain expertise and optimization modeling expertise. Gurobi’s \"State of Optimization Report 2023\" [R1–R2], based on a survey of 394 commercial users, identified optimization modeling as a rapidly growing area of demand, while also highlighting a significant skills gap. Just as coding LLMs have empowered non-expert programmers to build software, autoformulators can lower the barrier to modeling, reduce translation bottlenecks, and empower domain experts to experiment with formal optimization tasks. For experienced modelers, these systems can accelerate iteration, support exploratory modeling, and free cognitive effort for higher-level design and analysis.\n\n**Emerging research & industry focus.** Autoformulation is gaining traction in both research and industry. On the research side, recent systems such as Optimus and Chain-of-Experts (both appeared at ICLR 2024) highlight growing interest in the scientific and technical challenges of this space. We note that the benchmarks, tasks, and metrics we used are consistent with those employed in prior works. Industry interest further reinforces the relevance of this problem: (1) Gurobi’s modeling assistant [R3], (2) AWS’s optimization AI tools [R4], and (3) IBM’s research on AI-assisted modeling [R5] all address related aspects of the autoformulation pipeline. We view these developments as strong signals of real-world utility, and our system contributes foundational capabilities to this emerging and impactful area.\n\n**Actions taken.** Thank you for raising this important point. We have added a new appendix section outlining practical and research use cases of our system.\n\n[R1, R2] stage.gurobi.com/resources/report-state-of-mathematical-optimization-2023/, stage.gurobi.com/resources/report-state-of-mathematical-optimization-in-data-science-2023/\n\n[R3] gurobi-ai-modeling.readthedocs.io\n\n[R4] aws.amazon.com/cn/blogs/industries/optimization-in-the-era-of-generative-ai/\n\n[R5] research.ibm.com/publications/enhancing-decision-making-through-the-integration-of-large-language-models-and-operations-research-optimization-bridge-talk\n\n---\n\n### [P3] Highlighting novelty\n\nWe introduced a novel MCTS-based technique tailored to the structure of optimization modeling tasks, grounded in the core challenges **[C1-C3]**, including (1) domain-specific MCTS strategy to enhance hierarchical exploration; (2) SMT-based pruning of trivial equivalences, improving search efficiency by two orders of magnitude; and (3) comparative scoring of candidate formulations, enhancing feedback for search guidance. Empirically, we show that generic, off-the-shelf approaches fail to address key challenges in evaluating formulation quality (`S5.2`) and eliminating trivial redundancies (`Fig. 4`). In contrast, our approach outperforms prior state-of-the-art methods—including LLMs specifically finetuned for autoformulation—achieving new best results on formulation correctness across multiple challenging benchmarks.\n\n\n---\n\n*Thanks again; we hope our responses addressed your concerns and would appreciate your consideration in updating the score. We welcome further discussions.*", "claims_and_evidence": "- I personally disagree with the claim that \"optimization modeling follows a three-step process.\" Recent papers in operations research primarily focus on proving why their (human-derived) formulations are theoretically correct. While understanding the problem description in text and generating execution code can be useful, the core aspect of optimization modeling—the theoretical justification of the formulation—is entirely missing from this perspective.  \n\n- A key limitation of this work is that the task itself may not reflect a real-world need—one that domain scientists would actually use. Can you justify the practical use case of such a system and provide an estimate of who, and how many, would realistically adopt it? Otherwise, creating such a task solely for benchmarking against potentially irrelevant metrics raises concerns about its practical impact.", "methods_and_evaluation_criteria": "yes.", "theoretical_claims": "No theory is provided in this paper.", "experimental_designs_or_analyses": "The authors empirically evaluate the proposed method and demonstrate that it achieves better results.", "relation_to_broader_scientific_literature": "The developed software can be beneficial to the operations research community.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths:**  \n- The experimental evaluation is solid.  \n\n**Weaknesses:**  \n- The paper is purely empirically driven. Given its focus, it may be more suitable for an EMNLP conference.", "other_comments_or_suggestions": "NA", "questions_for_authors": "There is already extensive research on using LLMs with MCTS across various tasks aimed at improving automation, so the main algorithm in this work is not novel. \n\nAdditionally, a key limitation of this work is that the task itself may not be a *real* task—one that domain scientists would actually use. Why create such a task solely for benchmarking against metrics that may lack practical relevance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}]

Author rebuttal:

Meta-review: **(a) Summary**
This paper introduces an approach for the autoformulation of mathematical optimization problems: translating natural language problem descriptions into solver-ready optimization models.
It proposes a novel framework combining Large Language Models (LLMs) with Monte-Carlo Tree Search (MCTS), guided by symbolic pruning and comparative scoring. The method addresses the key challenges in autoformulation: vast hypothesis space, efficient search under uncertainty, and evaluation of correctness. Empirical results across multiple benchmarks (e.g., NLP4OPT, IndustryOR, MAMO, ComplexOR) demonstrate state-of-the-art performance, and the paper further validates results through ablations and expert evaluations.

**(b) Strengths**
* Clear and well-structured presentation (Reviewer fWQR): The paper is easy to follow and offers an in-depth discussion of core challenges in autoformulation.
* Strong empirical results (Reviewers Pqeu, fWQR, nBsi): The method consistently outperforms baselines across standard and complex benchmarks.
* Methodological novelty (Reviewers fWQR, Pqeu): The use of hierarchical MCTS, symbolic pruning, and comparative scoring is well-motivated and effective.
* Broader relevance (Reviewer nBsi): The paper contributes to the growing interest in combining LLMs with symbolic reasoning for automation in domain-specific tasks.

**(c) Weaknesses**
* Concerns on fairness in evaluation (Reviewer 14k8): Initially, the comparison with baselines was unfair due to differing rollout strategies. The authors addressed this by harmonizing evaluation metrics (pass@N) and adding best-of-N selection.
* Limited scope of datasets (Reviewer 14k8): The original version focused mainly on LPs and MILPs. This was addressed with new results on ComplexLP and ComplexOR.
* Lack of theoretical foundation (Reviewers nBsi, 14k8): The paper is largely empirical and lacks formal theoretical contributions.
* Practical utility questioned (Reviewer nBsi): One reviewer raised concerns about the real-world demand for such autoformulation systems. The authors responded with industrial surveys (e.g., Gurobi 2023) and comparisons to similar efforts at AWS and IBM to support relevance.
* Redundancy of complex search (Reviewer fWQR, 14k8): Concerns were raised about the necessity of MCTS given shallow search depth. The authors defended the choice through empirical comparisons and structured design benefits.

**(d) Decision**
Overall, the reviewers found the paper to be a strong empirical contribution with a clear methodology and promising results. The integration of LLMs, symbolic pruning, and search strategies is well executed and demonstrates performance improvements across diverse benchmarks. Despite lacking theoretical depth and some early concerns about fairness in evaluation and generalizability, the authors have responded thoroughly during the rebuttal period. Given its novel framing, practical impact, and strong empirical support, I recommend acceptance.

**Additional Comments on Reviewer Discussion:**
* fWQR: Appreciated the authors’ clarifications and chose to maintain their weak accept (score 3) after rebuttal.
* Pqeu: Acknowledged the expert evaluation and extended benchmarks, maintaining a score of 4 (accept).
* 14k8: Despite detailed responses and additional experiments by the authors, did not update their weak reject (score 2).
* nBsi: After a nuanced discussion, agreed the work may be better suited to OR venues but acknowledged the application potential. Maintained weak reject (score 2).

Given the split in reviewer scores (4, 3, 2, 2) but a consistently strong author rebuttal and the innovative nature of the work, this paper represents a valuable contribution to ML-driven automation and deserves a place at the conference.
Decision: Accept (poster)