OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling
Authors: Zhicheng Yang
Yiwei Wang
Yinya Huang
Zhijiang Guo
Wei Shi
Xiongwei Han
Liang Feng
Linqi Song
Xiaodan Liang
Jing Tang
Venue: ICLR 2025 Poster (2025)
Keywords: large language models; optimization problem; data synthesis
Primary area: datasets and benchmarks
TL;DR: This paper proposed a comprehensive benchmark for optimization problem solving along with a novel data synthesis method.

Abstract: Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning. Solving realistic optimization (OPT) problems in application scenarios requires advanced and applied mathematics ability. However, current OPT benchmarks that merely solve linear programming are far from complex realistic situations. In this work, we propose **OptiBench**, a benchmark for End-to-end optimization problem-solving with human-readable inputs and outputs. **OptiBench** contains rich optimization problems, including linear and nonlinear programming with or without tabular data, which can comprehensively evaluate LLMs' solving ability. In our benchmark, LLMs are required to call a code solver to provide precise numerical answers.
Furthermore, to alleviate the data scarcity for optimization problems, and to bridge the gap between open-source LLMs on a small scale (e.g., Llama-3-8b) and closed-source LLMs (e.g., GPT-4), we further propose a data synthesis method namely ***ReSocratic***. Unlike general data synthesis methods that proceed from questions to answers, \ReSocratic first incrementally synthesizes formatted optimization demonstration with mathematical formulations step by step and then back-translates the generated demonstrations into questions. Based on this, we synthesize the ***ReSocratic-29k*** dataset. We further conduct supervised fine-tuning with ***ReSocratic-29k*** on multiple open-source models. Experimental results show that ***ReSocratic-29k*** significantly improves the performance of open-source models.

Published as a conference paper at ICLR 2025
OPTIBENCH MEETS RESOCRATIC: MEASURE AND IMPROVE LLMs FOR OPTIMIZATION MODELING
Zhicheng Yang' Yiwei Wang' Yinya Huang* Zhijiang Guo' Wei Shi Xiongwei Han Liang Feng" Lingi Song Xiaodan Liang 7.8 Jing Tangl
The Hong Kong University of Science and Technology (Guangzhou)
Thhe Hong Kong University of Science and Technology
*University of California, Merced
*ETH Zurich CCiy University of Hong Kong Huawei Noah's Ark Lab
"un Yat-sen University *MBZUAI "Chongqing University
{yangzhch6, wangyw.evar xd1 1alg3a8] gma com, yinya. hww..hhotnnghtaaal com, zhi sriangguuuhkuse -22 edu. jingtangBust.
ABSTRACT
Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning. Solving realistic optimization (OPT) problems in ap- plication scenarios requires advanced and applied mathematics ability. However, current OPT benchmarks that merely solve linear programming are far from com- plex realistic situations. In this work, propose OPTIBENCH, benchmark for end-to-end optimization problem-solving with human-readable inputs and out- puts. OPTIBENCH contains rich optimization problems, including linear and non- linear programming with or without tabular data, which can comprehensively eval- uate LLMs' solving ability. In our benchmark, L.MMs are required call a code solver to provide precise numerical answers. Furthermore, to alleviate the data scarcity for optimization problems, and bridge the gap between open-source LLMs small scale (e.g.. Llama-3-8b) and closed-source LLMs (e.g.. GPT-4), we further propose data synthesis method namely ReSocratic. Unlike general data synthesis methods that proceed from questions answers, ReSocratic first incrementally synthesizes formatted optimization demonstrations with mathemat- ical formulations step by step and then back-translates the generated demonstra- tions into questions. Based on this, we synthesize the DESOCRATIC-229 dataset. We further conduct supervised fine-tuning with RESOCRATIC-29K on multiple open-source models. Experimental results show that RESOCRATIC-29K signifi- cantly improves the performance of open-source models.
INTRODUCTION
Large language models (LL.Ms), such GPT-3 (Brown al. 2020), GPT-4 (Achiam al. 2023
and Llama (Touvron et al.) 2023a b), have demonstrated their superior capability in logical reason-
ing (Suzgun al. [2023] Huang al. 2023) and mathematical reasoning (Ling al. 2017 Patel
cet al. 2222 Yang et al. 2022 2023), such as solving elementary (Cobbe et al. 2021) to high-school
level (Hendrycks et al., 2021) math problems. Yet follow-up curiosity is what extent LLMs
apply their mathematical intelligence to practical scenarios. Optimization problem solving (Rama- monjjson et al. 2022b Xiao al., 2023) AbmadiTeshniz! et al. 2024, HHang et al. (0024)) is field of applied mathematics that has been proven beneficial in many applications such as supply chain management, power energy scheduling, marketing. and quantitative trading. Optimization problem-solving comprehensive task that evaluates the mathematical and coding capabilities of LLMs. To provide the optimal solution to an optimization problem, LLMs are not only required to understand and construct the mathematical formulation according to the given problem but also to call an optimization solver to get the final answers.
"Corresponding Author: Jing Tang (jintaan@@ustht

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of optimization problem solving benchmarks. "End2End" indicates whether the benchmark requires the model to solve for the optimal values of the variables and the optimiza- tion objective. "ImplicitExplliii" refers whether the numeric values in the question are displayed.
Question
Linear
Nonlinear
Benchmark
Size
End2End
Form
w/ table
w/o table
w/ table
w/o table
ComplexOR Xiao et al.
2023
Implicit
37
x
x
NLP4LP AhmadiTeshhizi et al.
2024
Implicit
57
X
V V
NL4OPT Ramamonjison et al.
2022b
Explicit
289
X
OPTIBENCH (Ours)
Explicit
605
V
V
V
Previous studies [Ramamonjison et al. 2022a Xiao et al.) 2023 A et al.. (024) have
explored using LLMs to solve operations research problems. However, these studies have not yet extended to more generalized scenarios regarding practical optimization problems. Specifically, NL4OPT [Ramamoniionn et al. 2022a b) uses named entity recognition to extract entity and nu- merical values in the given question text, and then formulate it into mathematical models. They only measure the model's ability to correctly construct mathematical formulations, without considering solving the mathematical formulations being constructed. To further evaluate models providing the final optimal solution, i.e., the numerical values of the variables and the optimization objective, ComplexOR (Xiao et al.) (023) and NLP4LP (AhmadiTeshnizi al. 2024) benchmark the mod- els to solve a problem with an optimization solver in the setting without explicit input numbers. However, due the difficulty of collecting such data, these benchmarks are still on small scale. Moreover, the recent MAMO (Huang et al. (024a) proposes to further benchmark optimization problem solving with code solver. Nevertheless, one common pitfall of all the aforementioned works is that they merely focus on linear programming, whereas nonlinear optimization problems and practical tabular format are not included. Table Tablleprrvicce provides comparison of the aforementioned benchmarks. We also discuss the differences between current benchmarks in detail in Appendix A
To bridge this gap, we propose OPTIBENCIL a new benchmark with high-quality data to evaluate LLMs' end-to-end solving ability in optimization tasks. We carefully select 605 questions and con- duct careful manual verification to form the dataset. OPTIBENCH contains linear and nonlinear pro- gramming with both integer and mixed integer variables in the programming problems. OPTIBENCH also includes tabular data, which fills the gap in current optimization benchmarks. Figure I demon- strates OPTIBENCH examples in the four problem types. A model solves OPTIBENCH problem by reading the natural language input and then generating Python code that solves the problem, where the code will be processed to acquire the numerical value of the variables and the objective function.
Additionally, the data scarcity issue in optimization tasks (Xiao al. 2023 Teshhizii al. 2024) cannot be ignored. Annotators are required to possess good professional knowledge, making the process not only expensive but also tone-consuming and labor-intensive. In addition, there is a significant performance gap between small open-source models (e.g.. 1lama---7B, Llama-3-8B) and large models (.. GPT-44) many complex reasoning tasks (Ling et al. [017]] Patel et al. [02T]]
Suzgun et al., 2023 Yang et al. 2022 2023 Huang et al. 2023). To this end, we propose ReSo- cratic, a novel method for synthesizing diverse and reliable data for optimization problems. Unlike previous methods that synthesize questions first and then answers, ReSocratic incrementally syn- thesizes the formatted optimization demonstration in a reverse manner, and finally back back-translates into a question. Benefiting from intermediate reasoning steps, the quality of ReSocratic's 's synthetic data is higher than that of previous methods. We collect 29k samples with ReSocratic, resulting in the RESOCRATIC-29K dataset. In summary, our contributions
follows:
We introduce a high-quality benchmark OPTIBENCH for optimization problems with com- plex instances in multiple forms. As far as we know, this is the first large-scale benchmark including nonlinear and tabular data to measure LLMs' end-to-end problem solving abili- ties. We conducted an in-depth evaluation of a range of LLMs under various settings.
We propose ReSocratic, novel method for generating diverse and reliable data for op- timization problems. In particular, ReSocratic synthesizes complex reasoning data from scratch in a reverse manner.
2

---

Published as a conference paper at ICLR 2025
We synthesize the RESOCRATIC- 29k dataset with 29k samples by using ReSocratic. Experimental results show that conducting supervised fine-tuning with RESOCRATIC- 29k significantly improves the performance of open-souree models OPTIBENCH (e.g., Hlama-2-7B-CMhe from 0.0% to 30.6%; 1ama---8B-Instteeee from 13.6% to 51.7%), which further demonstrates the validity of our synthetic data.
linear problem without table
nonlinear problem without table
Textual Question:
Textual Question:
There are ways to extract metal from mined ozes. The first way is to use pricess J and the second is process P. Process] can extract 5 units metal using units of water produces units of pollution. Process P can extmact units of metal using 6 units of and produces units pollution. There be most 1500 units of water 1350 units of pollution. How many of each type of processes should performed to maximize the amount, oxtracted?
A piece of carthoard is I meter by 1/2 meter: A square is to be cut from each comer the sides folded up to make an open-top box. What are the dimensions of the box with macimum possible volume?
Optimization Model:
Optimization Model:
Define W.Las width length the cartboarrd respectivaly, x the length of each corner be out make the box.
Define as the process number in the type of J. the process number in type P.
Optimization Target:
Constraints:
Optimization Target:
Constraints:
max J.PEN
5j + 9p
8.f.
8j 6p < 1500
max
(W 2z)(L
2x)
s.t.
2z S W
3j + 5p S 1300
linear problem with table
nonlinear problem with table
Textual Question:
Textual Question:
There are six cities (citiss 1-6) in Kilroy County. The county must determnne where to build stations. The county wants to build the minimum number of fire stations needed to ensure that least. one fire station is within 15 minutes (driving time) of each city. times (in mizutes) required drive between the cities in Kilmoy County are shown in the following T'able.
company is planning to optimize its production of five different products (Product A, Product B, Product C, Product and Product E) to maximize profit while considering the environmental impact of production. The profit per unit and the envirunmental impoct per for each product giver. the following Table.
Product
Unit
Environmental Impact
Kilroy how many stations should be built and where they should be located.
$50
10unis
Table (Time Required Travel betwven Clides in Kilmmy):
$70
units
From/
City 1
City 2
City
City
City 5
City
560
units
To
$80
20 usits
City
20
The compony bas total production capacity of 1500 units across pruducts. company must produce least 200 units Product A and 300 units of Product В fulfill contractual abligations. The total environmental impact should not exoeed 20,000 units. The company wants maximize the Profis- ratio, where the Profit- Impact ratio is defined as the total profit divided total environmental impact.
City 3
20
25
16
30
20
City
15
Ctty
Ghy6
20
10
20
14
0
Optimization Model:
Optimization Model:
Defice x; 1,2.... as whether the station should be build in
Define the number of products A-E that the company should produce.
City i.
Optimization Target:
Constraints:
Optimization Target:
Constraints:
00ke 705 60e 90c
min
Σ 2
Rg
2
12c 20d
2g
it
a.4. a n-
did + 5000
33
33
2
200
$22300 300
10ko
822
26
Figure 1: Our OPTIBENCH contains various types of data (linear, nonlinear, table). To enhance readability, we present the table in an Excel format and include diagram illustrate the nonlinear example without table.
2 RELATED WORK
Benchmarks for Optimization Modeling, More closely related to our approach, the NL4OPT benchmark |Ramamonjison et al., 2022a investigates controlled generation techniques to obtain automatic suggestion of formulations. They first use named entity recognition methods to tract a set of entity-typer declarations, then they transform it into linear program models. As one see, NL.OPT only evaluates an AT model's ability to establish mathematical models, while
3

---

Published as a conference paper at ICLR 2025
contribute end-to end-to-end end framework in this work. Optimus (Ahmadi Teshnizi et al. 2024) and Com- plexOR (Xiao et al.] 2023) also make significant research in the field of operations research with LLMs. However, they provide minimal test set, containing less than 70 test samples. Recently, MAMO (Huang et al. 2024a) is proposed to benchmark mathematical modeling with code solvers. However, all these works merely focus on linear programming. ignoring the nonlinear problems that exist widely in practical applications. In addition, these benchmarks are simple in form, ignoring the tabular data that often occurs in industrial scenarios. Additionally, we notice a work work[Tang et al. (2024) explores synthesizing problems via a semi-automated process. This work is based on forward synthesis. Moreover, they did not focus on tabular data and nonlinear problems in real scenarios. In contrast, in this paper, we aim to benchmark practical optimization modeling with a high-quality manually checked test-bed OPTIBENCH and also automatically synthesize more comprehensive op- timization data including tabular data and code solutions resulting in RESOCR ATIC-2KK,, In this work, we contribute OPTIBENCH, which is an end-to-end benchmark containing 605 multi-type data samples. OPTIBENCH is comprehensive benchmark that involves linear, non-linear, and tab- ular data, and the types of variables involved in the problems include continuous, integers (IP), and mixed integers (MIP).
Data Synthesis for Mathematical Reasoning. Improving the performance of language models in mathematical reasoning tasks significantly depends on increasing the quantity of fine-tuning data for LLMs. A substantial body of work has been dedicated to these areas (Yu et al. 2023] Liu &
Yao 2024 Li et al.] [023] Yuan al. 2023 Lu et al. 2024b; Yue et al.) (023). One notable approach is Rejection Sampling Fine-Tuning (RFT: Yuan et al. 2023), which employs supervised models to generate and collect correct reasoning paths, creating augmented fine-tuning datasets.
MAmmo'TH Yue et al. 2023) utilizes RFT with GPT-4 to gather both Chain-of- Thouhhtttt solutions in natural language and Program-of- Trogram-of-Thought
Thought solutions in formal language. Similarly, MetaMath (Yu et al., 2023) focuses data augmentation for both question and answer texts. MathGenie (Lu et al., 2024b) collects a vast amount of data through open-source language models. While there has been significant progress in synthesizing data for informal mathematical reasoning, efforts have also been made to address formal reasoning through the use of formal languages and compilers (Xiong et al.) 2023 Huang et al. (2024b] Lu al. 2024a). However, major challenge remains: there is a scarcity of high-quality data for optimization problems. This lack of data limits the direct application of these prior approaches optimization contexts.
Socratic Method.
The Socratic method (Gose 2009; Scholle 2020) is a critical thinking method
with dialogic disassembled multi-step sub-questions and answers cultivating in answering a com- plex question. This method has been applied by current language model techniques for advanced
reasoning tasks, such as prompting step-wise reasoning (Qi et al.) 2023 Chang 2023 Shridhar iet al. 2022), multi-agent interaction (Zeng et al.] 2023). and discovering math knowledge (Dong al. 2023). For example, Qi et al. (2023) proposes a divide-and-conque style algorithm that mimics recursive thinking by asking Socratic questions, it thus relieves the reliance on the initial decision as chain-of-thought (CoT) and achieves performance improvements on several complex reasoning tasks. Chen & Lampouras (2023); Xie et al. (2023) utilize back-translation as core mod- ules in their data generation frameworks, but our ReSocratic places more emphasis on step-by-step reverse construction of the chain-of oh-thhught from scratch, with back- translation being one tiny step in our framework. Another line of work (Ang et al. 2023 Cobbe et al. (02T) applics the Socratic method for fine-grained dataset construction. GSMSK Socratie datase (Cobbe et al. 2021) is the most related work to our paper. They inject automatically generated "Socratic sub-questions" before each step, resulting in fine-grained math data. To construct a step-by-step benchmark for optimiza- tion problem solving with intermediate solutions, in this work, we explore the Socratic method to synthesize optimization problems. Unlike the previous study, we propose a reverse Socratic ap- proach (ReSocratic) that generates optimization problems from the answer back to question, and we demonstrate its superiority to traditional forward Socratic synthesis.
3 OPTIBENCH: BENCHMARK FOR OPTIMIZATION MODELING
The benchmark OPTIBENCH is to evaluate the capability of large language models to solve end- to-end optimization problems. Table compares OPTIBENCH and related optimization coimizzaionnproblemm
httpa: htt. https///ww........ooooooeeeeeeer wow.oootttttttttooooooooooo.
4

---

Published as a conference paper at ICLR 2025
OptiBench Sample
Solving Optimization Problem
Textual Question:
Code Generation:
A manufacturnng company produces three types of products: ProductA, ProductB, and ProductC. The company needs to decide the number of units to produce for each product in the month
fron pyscipopt inport Model
Define variables
Solver
UnitsA nodel. (ddra 1b-0)
invest automation for each product to incrase prodaction efficiency. The relationship between incrusse profit and automation investment, profit per unit, and production cost for each product is given the following Table.
Define objective function obj = model. eodel.eddVar( addvar("ce]" sebessssettjet "naximize") model. addens ProfitaiProfiteeProttt))
Product
Inital Profit
Production Cost Unit
Increase Profit per Unit $100 Autumation investment
Print the results
Product.A A
$50
$10
$5
Produc: 0
$70
$15
$7
Execute
Product C
$60
$12
$6
maximize product. The company bas total budgut of $50,000 for production and automation investments. The production capacity for the next month is limited to 1,000 units in total. Due to market demand, the company must produce at least units of ProductA and 150 units of ProductB.
company
profit,
Execution Output:
Answer Extraction Prompt
UnitsA 1100
A Accoding the output, please give your final answer for the following query. Number of Units of ProductA: Number of Units of ProdactB:
Investment a 900
Ground Truth Answer:
Number of Units of ProductA: 180.0
Answer Extractor
Number of Units of Product8: 960.0
Number of Units ProductC: 8.0
Investment in Automation for ProductA: 0.8
Number Units of ProductA: 100.0 Number Units of Product8: 906.0
Investment Automation for Product8: 35508.0
Investment Automation for ProductC:
Evaluate
Profit: 2384580.8
Figure 2: Evaluation procedure of OPTIBENCH example. This example is about mixed integer nonlinear optimization problem. The LLM is first required to write code to solve the question. Then, the LLM is required extract the exact numbers according to the code execution output.
benchmarks. OPTIBENCH covers a substantial number of optimization problems with wider range of problem types. Specifically, OPTIBENCH features linear programming (linear), non-linear opti- mization problems (non-linear), and table content as industrial use (Table), resulting in a com- prehensive and versatile benchmark for LLM optimization problem-solving. OPTIBENCH is an end-to-end benchmark, that takes natural language as input and numerical values of variables and objective as output. We show the four types of questions in Figure !
Data Collection and Annotation. In the data annotation stage, we assign workers to collect ques- tions from textbooks (Bertsimas & Tsitsiklis 1997, Conforti et al.) 2014] Wolsey] 2020), and university's course assignments and examinations. We require our workers to write Python code, call the pyscipopf solver to solve each problem, and ask them to output the values of the variables and optimization targets at the end of the code. Figure Souree]ssoww shows a mixed integer nonlinear program- ming sample of OPTIBENCH. For each sample, we provide the "Question" and "Results". More details of data collection and annotation are shown in Appendix
Data Statisties. In Figure 4 show the statistical results of four data formats (linear w/ table, linear w/o table, nonlinear w/ table, and nonlinear w/o table). Overall, our OPTIBENCH exhibits good diversity in terms of question types, number of variables, and text length.
Evaluation. Unlike NL4OPT (Ramamonjison et al. 2022b), which only measures the mathematical modeling ability of the language model, we also measure the solving ability of the language model to call code solver. In this paper, the evaluation approach we adopt is an end-to-end process where natural language text is the input and numerical form answers are the output. Given an optimization problem p. the LLM is required to generate a solution including code e = LLM(p). Next, a Python interpreter is used execute the code and obtain the code output o Py)hon(c). Then, we request the LLM to give the numerical form answer the problem, where r; is the natural language description of "Ground Truth Answer" Figure ?
LLM((o, r:]) for each variable and objective in
httpa: www.:tttt..cm...eeeeooommotttt http:l/pptppptttteeeeeetee /3ySCT
5

---

Published as a conference paper at ICLR 2025
esample questions
syzthhetic
Formatted Demonstration:
LLM
quission
LLM
Define Variables:
groundskeepar Birdie Valley oold club. There are four fectilizees ( 1-4) availlable
chip Green
(a) vanilla forward synthesis method
narket, Chip world to mix then together ubtain a nixture. chip needs to determine the optirol proportion of each rertilizer in the ntxture.
formatted demonstrations
1/ ('propartion of Pertilizer 17; txt", Trange":
"écodied"" "typet: "continucus")
// ("proportion of Fertillzon 2nd "x2", "range":
"ecoacei",, "type"; "continuous")
// ("proportion or Fertiltzer 511 "xa", "range":
cule Millrr
LLM
(/ ("proportion of Fertilizon 4%; "xe", "range":
"ecodded", "type"; "continucus")
"ecodesi", "type": "continucus")
// the sue of the proportions should be 1: xt x2 24
Step 1
ax Define objective Function:
Step 2
Fertslizerr 1-4 is $23.75, $22
$19.58 109 respectively-
Chip wants to minteize the rost the misture per 100 pounds.
Step 3
// Mininize: 21,75'x1 23.75 22.087a 19.58"
Step
Generate Constratnt-I:
Nitrooenn percentage of Fertillier 1-4 is 128, and 185; Phospherus parcentage of Fertilize 1-4 is 55, 118, 7%, and 181; the Potash percentage of Fertilizer 1-4 is 12, 15%, 128, and 188. chip knows that content should be 10-8-12 (188 sitrogen, as phaspharus, and 125 potash), but no mone than 9.5% above then. 5o the nitrogen level should be between 185 and 18. 18.58; the phosphorus level should be between and 8.58; the potash level should be between 12% and 12.5%.
try exacute
discard
simllarty fittr
code
best propartion
chesical
synthetic demonstrations
question
18 191xi 87x2 + 32th3 189x <o 20.5
ReSooralio229..
8Tx1 + 11"x2 71y3 187x4 <- 8.5
12 121x1 15 a 12"x3 10h24 de 12.5
Slep.
accepted step
transtranslate
ax Generate Constrotnt-2:
The mixturw should contain at least 20% Fertllizer I/ 8.2
rejected step by
question- translate
step
(b) pipeline overview of our ReSocratic
(c) code & question translation for a 4-step demonstration
Figure 3: (a) The forward data synthesis method to synthesize the question first, and then let the
LLM generate the answer to the synthetic question. (b) In contrast, ReSocratic first synthesizes
carefully designed formatted demonstration and then transforms into code and questions. (c) An example of formatted demonstration.
Finally, we compare the numerical form answer a, with the ground truth answer of to calculate the accuracy. A problem is considered solved if and only if all the variables and objectives are correctly matched.
4 RESOCRATIC: DATA SYNTHESIS WITH REVERSE SOCRATIC
In this section, we introduce ReSocratic, a novel data synthesis method for eliciting diverse and reliable data. The ReSocratic framework is shown in Figure 3(a). Former methods (forward syn- thesis) skipped the intermediate reasoning steps and directly generated the question, relying more on the intuition of LLMs. Whereas, the main idea of ReSocratic is to incrementally synthesize an optimization problem with step-by-step generation via the Socratic method in reverse manner from our elaborately formatted seed demonstrations to questions. Our ReSocratic consists of three steps: 1) Seed Demonstration Formalization, 2) New Demonstration Synthesis, and 3) Question and Code Translation.
1) Seed Demonstration Formalization. The seed demonstrations are rigorously selected by hu- mans and each seed data is of diverse operations research scenarios. Figure 3)) shows an example of seed demonstration. It is formatted step by step, where each step clearly delineated and builds upon the previous one. Each step consists of three parts:1) A header with "##" that introduces the specific aspect of the demonstration being addressed, such as "Defining Variables", "Objective Func- tion". or "Constraints". 2) A narrative description (colored in blue) in natural language that provides context and details about the element being introduced. This helps to understand the rationale and the requirements of that particular part of the optimization problem. 3) Mathematical formalization following "//" that translates the natural language description into a precise mathematical expression constraint.

---

Published as a conference paper at ICLR 2025
2) New Demonstration Synthesis. The ReSocratic method prompts an LLM to generate new demonstrations based on the seeds. We sample seeds each time from the pool to form the syn- thesis prompt shown in Appendix E.2 The LLM will follow the given prompt to generate new demonstrations step by step. The generated data is rigorously selected via (1) each intermediate step should follow the format (as shown in the first step) by rule filter and (2) the overall demonstration does not overlap with generated ones by a similarity filter. For all generated demonstrations, we set similarity filter, which converts all texts into TF-IDF vectors and filters out demonstrations with cosine similarity higher than threshold. The procedure is shown in Figure ().
3) Question and Code Translation. We construct code generation prompt to solve the mathe- matical formulations in the synthetic demonstrations and output the optimal solution results. If the code runs incorrectly, we delete this demonstration. Next, to acquire the questions in plain text for- mat and the questions in table format, we construct two back-translation prompts. All the prompts are shown in Appendix E Then, for each generated demonstration starting from the third step, we translate it into question-code pair, as shown in Figure 3b). Each generated code will be executed automatically to ensure there are no bugs.
5 EXPERIMENTS AND ANALYSIS
5.1 BASELINES AND SETTINGS
Evaluation Setting. The evaluation metric of our OPTIBENCH is the answer accuracy, as detailed in Section 3 We show the solving accuracy of the four data types along with the code pass rate. We evaluate LLMs under three settings: Zero-shot, Few-shot, and Supervised Fine-Tuning (SFT) setting. We provide the zero-shot prompt and the few-shot prompt to solve the problem in Ap-
pendix E
Baselines. We select GPT-family (Brown et al. 2020 Achiam al.) 2023), Llama-family (Tou-
vron et al. 2023b Team 2024), Qwen2 Yang et al. (2024), Mistral-vO.. Jiang et al. (2023). and feapSeek-family DDepSSekk-A] Al 2024) the baselines in zero-shot and few-shot settings. For the
SFT setting, we use Llama-2-7B-Chai and texib/Llama--3- 3-8B-Instruct.
Setting of Data Synthesize. We use DeepSeek-V2 (DeeSSek-AA] 2024) apply ReSocratic. As an open-source large language model, DeepSeek-V2 (DeepSeek-Al 2024 stands out due to its com- petitive performance GPT-4, while concurrently offering more cost-effective alternative. Fur- thermore, it exhibits a superior througbput, approximately 6 times greater, when contrasted against the existing 70b open-source model (DeepSeek-Al (024). Utilizing the advanced capabilities of DeepSeek-V 22, we contribute 29k synthetic data. This results in the RESOCRATIC-29K dataset. The threshold of the aforementioned similarity filter set at 0.7, we also set the temperature as 0.7, and sample 50 responses for each query.
Fine-tuning Setting. For a given language model, we utilize our contributed RESOCRATIC-29K to conduct supervised fine-tuning. Based on this, we conduct fine-tuning experiments on two A800 GPUs, the epoch is set as 3, the learning rate is 2e and the batch size is 128.
5.2 DATA STATISTICS AND VISUALIZATION
For both OPTIBENCH and DESOCRATTIC-29K we show the statistical results of data distribution in question type, variable numbers, and question length. The question length refers to the number of characters in the question text. The results are shown in the following Figure 4 The distribution of variable numbers in both OPTIBENCH and RESOCRATIC-29K generally conforms to the long-tail distribution. In addition. the distribution of question length in OPTIBENCH also conforms to the tail distribution, while RESOCRATIC- 29K is more balanced.
Furthermore, we show the visualization results of OPTIBENCH and RESOCRATIC-22 -2K using the SNE algorithm based on question semantic embedding. The visualization results of each type (linear w/o table, linear w/ table, nonlinear w/o table, nonlinear w/ table) shown in Figure
In general, from the statistical results and visualization results, our RESOCRATIC-299 has a good diversity in the question types, variable numbers, question length, and text semanties.
7

---

Published as a conference paper at ICLR 2025
CptiBecch
OydReech
96.7%
25k
ReSoomatte22
ReSocnatic
22.005
30.1%
13.2%
8.3%
11.0%
9.15
13.59
7.45
5.1%
10.2%
17, 17.5%)
12.3%
15.8%
2.6% 2.5
601-900
901-1200
1201-1500
tifle
(a) Data type distribution
(b) Variable numbers distribution
(c) Question length distribution
Figure 4: Statistical results OPTIBENCH and RESOCRATIC-299
lincarwoo table
linear w/ table
nonlincar table
nonlincarw table
Figure 5: t-SNE visualization results of OPTIBENCH and RESOCRATIC-29K. ('A' indicates the data point of OPTIBENCH and indicates the data point of RESOCRATIC-29K)
5.3 MAIN RESULTS
As shown in Table 2 GPT-4 has the strongest overall performance and achieves state-of-the-an performance on almost all kinds data formats. The performance of the two open-source models, llama3-70b and deepseek-v2, is close that of GPT-4 in the few-shot setup. In addition, open source small models perform extremely poorly on OPTIBENCH, with Llama-2-7B-Chr not getting a single question correctly solved and 1lama-3-8E -----uutttuct getting only 13.6% accuracy on the few- shot setting. From the perspective of data type, the nonlinear data of our OPTIBENCH is more challenging than the linear data, and the data with table (w/ table) is more challenging than without table (w/o table). Then, to show the validity of ReSocratic, we SFT 8B-Instruct with our synthetic data RESOCRATTE ATIC-29K. We improved the performance the Llama- from 0.0% to 30.6%, and the Llama-3-88 InIstrutt from 13.6% to 51.1% (+37.5%), which is very close to the GPT-3.5-Turbo.
and Llama-3-
addition, Llama-3-8B 3-8B-Instttct even exceeds GPT-4 the data type of linear w/table, reaching ttate-of-the-art performance. We present more detailed dataset performance analysis in Figure 6
5.4 PERFORMANCE ANALYSIS ON DATA SPLIT
We show the detailed evaluation results under the few-shot setting for GPT-4 and GPT-3.5- Turbo F The performance on OPTIBENCH is split by data type, variable numbers, and question length. According to the results, we can find that:
1) Tabular questions are harder. is more difficult solve table questions than no-table questions, and nonlinear questions are more difficult than linear questions.
2) Nonlinear problem is harder. According Table 2 for all language models, the performance of nonlinear questions is significantly lower than that of linear questions. In Figure [ (a), we show the average performance of linear problems and nonlinear problems for GPT-4 and GPT-3.5-Turbo. The performance is 66.9% for linear problems and 30.8% for nonlinear problems.
3) In general, questions with more variables and longer text lengths are more difficult. We show the performance of GPT-4 and GPT-3.5-Turbo Figure(((bb. From the linear trend line we provided, we can see that model performance decreases as the number of variables increases. As can be from (c), GPT-3.5-Turbe has significant performance degradation on long text questions, while GPT-4 has a more balanced performance on different text lengths.
the data with different numbers of variables in
8

---

Published as a conference paper at ICLR 2025
Table 2: Main results OPTIBENCH. "Code Pass" refers the success rate of code execution. Bold indicates the sota in the current setting, underline indicates the sota in the overall setting.
Linear
Nonlinear
Model
Code Pass
w/o Table
Table
wo Table
Table
Zero-shot Prompt
Mama--3-8-Inntruue
0.0%
0.29%
0.0%
0.0%
0.17%
8.8%
-lama-3-70B- 3-70BBInnstruc
76.9%
50.0%
30.8%
32.0%
59.5%
86.8%
Mistral-7B-Instrueet-
0.0%
0.0%
0.3%
Owen2-7b-Instrrun
3.5%
3.0%
0.0%
2.6%
19.2%
DeepSeek-V2
40.4%
27.5%
29.3%
18.0%
34.4%
74.0%
DeepSeek-V2.55
78.4%
67.5%
33.1%
24.0%
62.5%
92.7%
GPT-3.5-Turo
68.1%
37.5%
19.5%
16.0%
49.1%
85.0%
GPT-4
75.4%
62.5%
42.1%
32.0%
62.8%
88.8%
GPT-40-mini
76.0%
48.8%
35.3%
34.0%
60.0%
84.8% 8%
GPT-4o
78.1% %
65.0%
45.9%
40.0%
66.1
90.1 %
Few-shot Prompt
Llama-3-8B-1 3-8t-urttucttt
17.8%
2.5%
11.3%
8.0%
13.6%
26.9%
Llama-3-70B 7--tt-nnttt
79.2%
57.5%
33.8%
32.0%
62.5%
91.2%
Dissral-7B--nsttuee--00
40.0%
23.8%
13.5%
18.0%
27.9%
83.8%
Qwen2-7b-
1wen2-7b-Instruet
65.5%
27.5%
18.8%
14.0%
46.0%
87.6%
DeepSeck-V2
79.5%
56.3%
27.1%
32.0%
61.0%
85.5%
DeepSeek-V2.5
79.5%
71.3%
40.6%
48.0%
67.3%
91.2%
GPT-3.5-Turbo
75.4%
40.0%
28.6%
26.0%
56.4%
93.2%
GPT4
80.7%
71.3%
34.6%
34.0%
65.5%
88.3%
GPT-4o-mini
74.6%
52.5%
14.3%
34.0%
55.0%
74.4%
GPT-4o
81.0%
63.8%
50.4%
50.0%
69.4%
91.7%
SFT with Synthetic Data
1lama-2-BB-hha
40.6%
11.3%
15.8%
32.0%
30.6%
93.7%
Ilama- 3-8---ntrttt-
63.5
32.5%
33.0%
44.0%
51.1 %
96.3%
GPT-35-Tubo
71.8%
7L1%
6L.%%
GPT-35 Tubto
177.0%%
GPT-3 S-Tubo
GPP-4
00.9%
100005
51.156
50.0%
linex
41.0%
30.77%
28 EX
26.456
ZELORS
8.6%
-0.%
Accuracy
type
(b) Accurary on variable numbers
(r) Accuracy on question length
Figure 6: Performance analysis of GPT-4 and GPT-3.5-Turbo.
5.5 ABLATION STUDY ON RESOCRATIC
In Table B we present the performance outcomes of the models with and without the application of filters, specifically, the rule filter and similarity filter, which illustrated in Figure 3(b). Ad- ditionally, we compare the performance when incorporating step questions versus when they are excluded. We conduct an ablation study on Llama-2-7B-O imental results show the validity of our filters, this conclusion is similar to RFT (Yuan et al. 2023) that redundant data can negatively affect language models. Moreover, the step questions generated in ReSocratic can bring improvement to the model's 's solving ability.
and Llama-3-8B-nng In-struttt The exper-

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study synthetic data.
Linear
Nonlinear
Model
SFT Data
All
w/o Table
Table
w/o Table
w/ Table
ReSocratic w/o step questions
38.3%
10.0%
15.0%
32.0%
28.9%
Llama-2-7B-Char
ReSoorati filters
40.1%
11.3%
14.3%
30.0%
29.6%
RESOCRATIC-29K
40.6%
11.3%
15.8
32.0%
30.6 %
ReSocratic ie w/o step questions
62.9%
32.5%
31.6%
42.0%
50.2%
Llama-3-SB- 3-S-intett
ReSocraLic Le w/o filters
62.3%
31.3%
32.3%
36.0%
49.4%
RESOCRATIC-29K
63.5
32.5%
33.0%
44.0%
51.1% %
5.6 COMPARISON BETWEEN REVERSE SYNTHESIS AND FORWARD SYNTHESIS
Furthermore, we compared the forward data synthesis approach with the reverse data synthesis approach (our ReSocratic method). WizardLM(XU al. 2023 a typical forward data synthesis method, which first prompts the language model to generate questions similar to the seed data and then answer them. Using the same seed data, we sample 1000 responses from DeepSeek-v2 with The experimental
WizardLM and ReSocratic respectively, and then fine-tune Llama-2-7B-CHS results are shown in Table 4 The experimental results show that our ReSocratic synthesis method is superior the forward synthesis method. Moreover, sample 30 pieces of data generated by ReSocratic and WizardLM respectively, and manually identify the data accuracy, which also shows that the data generated by ReSocratic is more accurate.
Table 4: Comparison between ReSocratic and other forward methods (Evol-Instruct and Self- Instruct).
Data
Linear
Nonlinear Table
Model
All
Method
Acc
w/o Table
w/ Table
Table
Self-Instruct (Ik responses)
80.0%
16.1%
5.0%
3.0%
4.0%
10.7%
Llama-2-7B-Chas Evol-Instruct (1k responses)
76.7%
15.5%
7.5 %
3.8%
6.0%
11.1%
ReSocratic (lk responses)
86.7 %
21.6%
6.3%
5.3%
6.0%
14.4%
6 CONCLUSION
In this paper, we propose the OPTIBENCH benchmark, which includes various types of data, to evaluate the ability of language models solve mathematical optimization problems end-to-end. Furthermore, in order to alleviate the issue of data sparsity and mitigate the performance gap be- tween large models and small open-source models, introduce the ReSocratic method, reverse data synthesis approach. The experimental results show that our ReSocratic method outperforms the forward data synthesis method. After fine-tuning with our synthetic data, RESOCRATIC-29%. the performance of Llama-2, 2-7---77BB-- Bhat and Bamma---B-IInstruee has been significantly improved, demon- strating the effecttivnees of our synthesis method. In the future, we plan to extend ReSocratic to other complex reasoning tasks such math word problem-solving and evaluate more large language models on our proposed OPTIBENCH benchmark.
ACKNOWLEDGMENTS
Jing Tang's work is partially supported by National Key R&D Program of China under Grant
No. 2023 2023YFF0725100 and No. 2024YFA1012701, by the National Natural Science Foundation of
China (NSFC) under Grant No. 62402410 and No. U22B2060, by Guangdong Provincial Project
(No. 2023QN10X025), by Guangdong Basic and Applied Basic Research Foundation under Grant
No. 102311 by Guangzhou Municipal Science and Technology Bureau under Grant
No. 202360310667 and No. 2024A04J4454, by Guangzhou Municipal Education Bureau (No.
2024312263), and by Guangzhou Municipality Big Data Intelligence Key Lab (No. 2023A03J0012), Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628) and Guangzhou Municipal Key Laboratory Financial Technology Cutting-Edge Research (No.
2024A03J0630).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087774 2023.
Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. Optimus: Scalable optimization modeling with (mi) Ip solvers and large language models. arXiv preprint arXiv:2402. 10/72, 2024.
Beng Heng Ang, Sujatha Das Gollapalli, and See-Kiong Ng. Socratic question generation: A novel dataset, models, and evaluation. In Andreas Vlachos and Isabelle Augenstein (cds.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pp. 147-165. Association for Computational Linguistics, 2023. doi: 10. 100.1863/V 18653/v1/2022 eacl main.
htttt:://///o// org/
Dimitris Bertsimas and John N Tsitsiklis. Introduction to linear optimization, volume 6. Athena Scientific Belmont, MA, 1997.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Edward Y. Chang. Prompting large language models with the socratic method. In 13th IEEE Annual Computing and Communication Workshop and Conference, CCWC 2023, Las Vegas, NV, USA, March 8-11, 2023, pp. 351-360. IEEE, 2023. doi: 10.1 N0.119//CCCC534.2223.10 10999991. URL
https: //ooi doi htt.:/oi..ogggo// /t..r/g/o..11 1109/ CCWCS 7344. 2023. 1.0..10099177
Pinzhen Chen and Gerasimos Lampouras. Exploring data augmentation for code generation tasks.
arXiv preprint arXiv:2302.03499, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Michele Conforti, Gérard Cormujjls, Giacomo Zambelli, Michele Conforti, Gérard Cornuéjols, and Giacomo Zambelli. Integer programming models. Springer, 2014.
DeepSeek-AL. Deepseck-v2 v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
Qingxiu Dong, Li Dong, Ke Xu, Guangyan Zhou, Yaru Hao, Zhifang Sui, and Furu Wei. Large language model for science: A study on P vs. NP. CORR, http//2299.066999 2023. doi: 10.48550/ ARXIV.2309.05689, https: wttp::e..rr11.............
Michael Gose. When socratic dialogue is flagging: Questions and strategies for engaging students. College Teaching, 57(1):45-50, 2009.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Xuhan Huang, Qingning Shen, Yan Hu, Anningzhe Gao. and Benyou Wang. Mamo: mathematical modeling benchmark with solvers. arXiv preprint arXiv:2405. 13144, 2024a.
Yinya Huang, Ruixin Hong, Hongming Zhang, Wei Shao, Zhicheng Yang, Dong Yu, Changshui Zhang, Xiaodan Liang, and Linqi Song. Clomo: Counterfactual logical modification with large language models. arXiv preprint arXiv:2311.17435 2023.
Yinya Huang, Xiaohan Lin, Zhengying Liu, Qingxing Cao, Huajian Xin, Haiming Wang. Zhenguo Li, Linqi Song, and Xiaodan Liang. MUSTARD: mastering uniform synthesis of theorem and proof data. CoRR, abs/2402.008577. 2024b. doi: 1044555/AARXI..442..8852 URL https:
48550/ rxvv. 2402. 08957

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Anee Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
06825
Chengpeng Li, Zheng Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang. and Chang Zhou. Query and response augmentation cannot help out-of-domain math reasoning generalization. arXiv preprint urXiv:2310.055000 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gen- eration: Learning solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158-167, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v 1/10655//11177-0115 URL /tttttpss/aclllathooooooooo 1015
Haoxiong Liu and Andrew Chi- Chi-Chih Yao. Augmenting math word problems via iterative question composing, arXiv preprint S.II::2401.00000 2024.
Jianqiao Lu, Zhengying Liu, Yingjia Wan, Yinya Huang, Haiming Wang, Zhicheng Yang. Jing Tang, and Zhijiang Guo. Process-driven autoformalization in lean CoRR. abs/2406.01190, 2024a. doi: 10.4550/ARXIV.244.00029
URL htttt:t:://// //doi. wwt////.../////////////////// 2406.
01 940
Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question trak-translttion for enhanc- ing mathematical reasoning of lms. arXiv preprint arXiv:2402. 16352, 2024b.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080- 2094, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/222 naacl-main.16f 168. URL Mittts::/aalaanthlogg/ org. wwtp///..2220 http:/22221nnaaac.aa mww.. 168
Jingyuan Qi, Zhiyang Xu, Ying Shen, Minqian Liu, Di Jin, Qifan Wang, and Lifu Huang. The art of SOCRATIC QUESTIONING: Recursive thinking with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing, pp. 4177-4199, Singapore, December 2023. hetp:////22222../mmmmmm
Association for Computational Linguistics. doi: otr.:/oooooo...roo2....m..eomoo. in. i...n.55. 255
URL
Rindra Ramamonjison, Haley Li, Timothy T. L. Yu, Shiqi He, Vishnu Rengan, Amin Banitalebi- Dehkordi, Zirui Zhou, and Yong Zhang. Augmenting operations research with auto-formul foro-foomulation of optimization models from problem deseriptions. In Yunyao Li and Angeliki Lazaridou (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: EMNLP 2022 Industry Track, Abu Dhabi, UAE, December 7 11, 2022, pp. 29-62. Association for Computational Linguistics, 2022a. doi: 10.1166///12002.EMM..PPPP https:://dbi.ozg/1... 111.11...163./8 20222 www..pmm.ins.s....emmmmmm
URL
Rindranirina Ramamonjison, Timothy Yu, Raymond Li, Haley Li, Giuseppe Carenini, Bissan Ghad- dar, Shiqi He, Mahdi Mostajabdaveh, Amin Banitalebi-Dehkordi, Zirui Zhou, and Yong Zhang. N14opt competition:
Formulating optimization problems based on their natural language descrip- tions. In Marco Ciccone, Gustavo Stolovitzky, and Jacob Albrecht (eds.), Proceedings of the NewrIPS 2022 Competitions Track, volume 220 of Proceedings of Machine Learning Research, pp. 189-203. PMLR, 28 Nov-09 Dec 2022b. URL httthh:::///oceeeeee http://amaaoooiiioooooo 3a. 3a. html
.m1 mress/
Charles Scholle. Understanding the socratic method of teaching. Abraham Lincoln University (blog). https://www. alu. eduualuhoogundderstannng www.://iinnnieettoooocccceeettee -ffaethhnnd..-cthing.. 2020.
Kumar Shridhar, Jakub Macina, Mennatallah El-Assady, Tanmay Sinha, Manu Kapur, and Mrin- maya Sachan. Automatic generation of socratic subquestions for teaching math word problems.
12

---

Published as a conference paper at ICLR 2025
In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing. EMNLP 2022, Abu Dhabi, United Arab Emirates. December 7-11, 2022, pp. 4136-41499 Association for Computational Linguisties, 2022. doi: 10.
hrtp//000.orr0//..../ URL. / 18653/
v1/2022. emn amm.:22an.mmmmmm....2
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le. Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG- bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1:000-130511 Toronto, Canada, July 2023, Associ- ation for Computational Linguistics. doi: 10.18653/v 102002//////223333 findings--cl.... acl.224. URL https: www.aatthhooooy.o.2222222 //Acleecchooory..rr//222.. http:///2222...innnnngeecee.
824
Zhengyang Tang, Chenyu Huang, Xin Zheng, Shixi Hu, Zizhuo Wang, Dongdong Ge, and Benyou Wang. Orlm: Training large language models for optimization modeling. arXiv preprint
arXiv:2405. 2024.
Llama Team. The Ilama 3 herd models, 2024. URL httpps//arravv.orr////////
21783
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2xXiv22307.09288 2023b.
Laurence A Wolsey. Integer programming. John Wiley & Sons, 2020.
Ziyang Xiao, Dongxiang Zhang, Yangjun Wu, Lilin Xu, Yuan Jessica Wang, Xiongwei Han, Xiaojin Fu, Tao Zhong, Jia Zeng, Mingli Song, et al. Chain-of-experts: When 1lms meet complex opera- tions research problems. In The Twelfth International Conference on Learning Representations, 2023.
Yiqing Xie, Atharva Naik, Daniel Fried, and Carolyn Rose. Data augmentation for code translation with comparable corpora and multiple references. arXiv preprint wxXiv:231..0011 2023.
Jing Xiong, Jianhao Shen, Ye Yuan, Haiming Wang, Yichun Yin, Zhengying Liu, Lin Li, Zhi- jiang Guo, Qingxing Cao, Yinya Huang, Chuanyang Zheng, Xiaodan Liang, Ming Zhang, and Qun Liu. TRIGO: benchmarking formal mathematical proof reduction for generative language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, De- cember 6-10, 2023, pp. 11594-1 11632. Association for Computational Linguistics, 2023. doi: http:5//111222..MMLL....PP
emm p- i in. 711
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.122:4 2023.
An Yang, Baosong Yang. Binyuan Hui. Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jianxin Yang. Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu. Yang Fan. Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https://aras org/ 2407 10671
13

---

Published as a conference paper at ICLR 2025
Zhicheng Yang, Jinghui Qin, Jiaqi Chen, Liang Lin, and Xiaodan Liang. Logicsolver: Towards interpretable math word problem solving with logical prompt-enhanced learning. arXiv preprint TrXii:2205.08222. 2022.
Zhicheng Yang. Yiwei Wang, Yinya Huang, Jing Xiong, Xiaodan Liang, and Jing Tang. Speak like native: Prompting large language models in a native style. arXiv preprint arXiv:2311.13538, 2023.
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scal- ing relationship learning mathematical reasoning with large language models. arXiv preprint arXiv:2308. 01825, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05663, 2023.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S. Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratie models: Composing zero-shot multimodal reasoning with language.
The Eleventh International Conference on Learning Represen- tations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: //openreview. net wtt.::/..ee.ffffttt..mmmm mtt2
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A MORE COMPARISONS WITH OTHER BENCHMARKS
NL4OPT is too easy for current LLMs. As we have mentioned in our paper, the NL4OPT bench- mark is not an End-to-End evaluation benchmark. conducts two-stage process to evaluate the ability construct mathematical models. In this work, transform NL4OPT into our OPTIBENCH data format and contribute NL4OPT-E. We then evaluate a few LLMs, the results are shown in the following Table 5 All the experiments conducted under the zero-shot setting. It is observed that LLMs solve NL4OPT almost completely even using the simplest prompt, but our OPTIBENCH still poses a strong challenge.
Table 5: Evaluation results comparison of NL4OPT-E and OPTIBENCH.
Models
NL44PT-E
OPTIBENCH
Deepseek-V2
89.3%
34.4%
GPT-3.5-Turbo
83.0%
49.1%
GPT-4
93.1%
62.8%
'Implicit' and 'Explicit' data. Problems studied by Chain-of-Expertss (Xiao et al. 2023) and Opti- MUS Teshhize al. (2044) are orthogonal ours. These related works (Xiao et al [203] AhmadiTeshnizz et al. 2024) examine the abstract modeling capabilities of LLMs for optimization problems. Specifically, the problems they focus on not include explicit numerical values, pri- marily investigating the abstract modeling capabilities of LLMs in domain-specific scenarios. We denote this form of the problem as 'Implicit'. In contrast, the problems we study include explicit numbers, and researching the conerete problem-solving capabilities of LLMs. We denote this form of the problem as 'Explici'. The form of the problems we study is closer to practical applications. We present an 'Impiici'' sample and 'Explicit' sample in Figure 7
Implicit Sample
Explicit Sample
A fishery wants to transport their catch. They can either use local sled dogs or trucks. Local sled dogs and trucks can take different amount of fish per trip. Also, the cost per trip for sled dogs and truck is also differs. You should note that the budget has an upper limit and the number of sled dog trips must be less than the number of truck trips. Formulate an LP to maximize the number of fish that can be transported.
A fishery wants to transport their catch, They can either use local sled dogs or trucks. Local sled dogs can take 100 fish per trip while trucks can take 300 fish per trip. The cost per trip for sled dogs is $50 while the cost per trip for truck is $100. The budget is at most $1000 and the number of sled dog trips must be less than the number of truck trips.
Formulate an LP to maximize the number of fish that can be transported.
Figure 7: Examples of 'Implicit' and 'Explicit' data. The 'Implicit' sample is constructed by Xiao et al. (203) according to the original 'Explicit" example of NL4OPT.
In this work, we mainly focus on the 'Explicit' problems. We consider such a form to be more related to real work scenarios, such the query question containing numerical table. Therefore, Chain-of-F Obain-of-Experts (Xiao et al., 2023) and OptiMUS (AhmadiTeshin Teshnizt et al. 2024) are orthogonal to work.
15

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTAL RESULTS
B. FINE-GRAINED ERROR ANALYSIS
We provide the statistical results on the Code Pass rate under various data categories. The results of Code Pass rate are shown follows ('Code Pass" refers to the success rate of code execution).
Table 6: Code Pass rate under various data categories.
Models
Prompt
Linear w/o Table
Lincur w/ Table
Nonlineur Table
Nonlincar Table
Overall Pass
zero-shot
6.3
0.1
6,9
Qwen2-7b-Instruet
zero-shot
21.3
0.1
21.8
19.2
8pt-4o
zero-shot
93.6
93.8
83.5
78.0
90.1
Mistral-7B 7B Instract-vO.3
few shot
87.7
85.0
75.9
84.0
83.8
Qwen2-7b-Instruee
fow-shot
85.0
gpt-4o
few-shor
94.2
80.0
85.7
86.0
91.7
From the above statistical results, observe that the Code Pass for nonlinear problems is lower than that for linear problems, indicating that solving nonlinear problems poses certain chal- lenges terms of coding. Furthermore, we provide case studies show the errors in code gener- ation by language models, shown in the following.
## Error Case
import pyscipopt
Create model
model pyscipopt Model
Define variebles
acai
addvar ("acai_smm) smmocthies" vtype="1", vtypos"I*, 1b=0)
somana_smoothies model. addVar
Lefine objective function
Minimize the total amount of water used
addel.a ('obj")
model. setbssertejjeeecccce.... (cbj, "minimize") madel. addcons tobj
scai_
banana_emocthies)
Add constraints
constraint for acal berries model. addCona 17 x scal_smoothies <- 3500)
constreint for banana chocolaze madel. addCons banama_smoothion
3200)
Constraint for more banana smoothies than acsi emoothies __moothiee)
addCons (banana aaanana_amocthees
Constraint
smoothies being acai berzy
madel. addcons jacai_ mccai_mmothhiss >= 0.35 jacsi_smoothies. banona__ncothhiee
Solve the problem
model.optimiee
## Error Analysis 1: PYSCIPOpt does not support the symbols <,>; it only supports S, 2.
## Eror Case 2:
Import math
import pyscipopt
Create madal
model pyscipopt Model!!
vefine variehles for the rescue team's location
x aodel. ('x', 1b=01 x must he non-negative
y model.addVas (fy')
16

---

Published as a conference paper at ICLR 2025
man_ mon_y
Define the objective Funetien: minimize the distance to the distance addvar distance') madel. set (diatance, "minimize") # minimiza the distence
distance constroint
model addCons (df atannee =* math.aqrt man_x)++>> man_yl++2])
Add the constraint for the swamp: x"2
model. addCons ly >= x*+
Solve the problem
madel.opt omtiiiipttittoo
## Error Analysis 2: In PySCIPOpt, you cannot use math.sqrt() directly.
Moreover, measure the accuracy of the mathematical formulations using the LLM-as-a-judge approach and demonstrate the accuracy of transferring the mathematical formulations to code, We consider this analysis can provide more comprehensive understanding of the models' performance. In this experiment, we use deepseck- v2.5 to judge the results.
Table 7: Accuracy of the Mathematical Formulations.
Models
Prompt
Linear wlo Table
Linear Table
Nonlinear Table
Nonlinear Table
Overall Pass
Mistral 7B Instract v0.3
zero shot
56.4
25.0
23.3
24.0
42.3
Mistral- 7B Instruct-v0.3
Qwen2-7b-Instruct
zero-shot
71.1
40.0
39.1
26.0
56.2
Qwen2-7b-Instriic
few-shot
78.7
43.8
41.4
46.0
63.1
deepseck-v2.5
aero-shot
91.8
77.5
63.2
56.0
80.7
deepseek-v2.5
few-shot
93.6
81.3
70.7
68.0
84.8
gpt-4o-mini
zero-shol
72.5
69.2
60.0
apt-4o-mini
few-shor
90.1
67.5
72.9
70.0
81.7
gpe-4o
zero-shot
gpt-4o
few-shor
92.9
73.8
82.0
70.0
86.1
As shown in the table, small open-source models like Bitrral-7B-Instuuctt-d.. and Qwen2-7b- Instruct tend to make more mistakes in mathematical formulations than large models like deepseek- v2.5 and gpt-4o. Moreover, the few-shot setting can significantly decrease the errors of mathematical formulations for all models than the zero-shot setting. These phenomena are also observed in bench- marks like GSM8K and MATH, leading us conclude that they stem from the model's capacity for mathematical reasoning.
Table 8: Accuracy of Transferring the Mathematical Formulations to Code.
Models
Prompt
Linear w/o Table
Linear w/ Table
Nonlinear w/o Table
Nonlinear w/ Table
Overall Pass
Mistral-7B. 78-nnmttttt--
zero-shot
0.0
0.0
0.0
0.7
Mistral-7B. 1aaaa-77--nntmet--....
few-shot
50.8
61.3
41.9
39.1
49.9
Qwen2-7h Tb-Instruct
shot-shot
0.0
7.7
0.0
4.7
Qwen2-7b-- Innttrrr-
few-shot
62.9
45.5
30.4
72.8
deqpseck-v2.5
zero-shot
85.4
87.1
52.4
42.9
77.5
deepscck-v2.5
few-shot
87.7
70.6
gpt 4o mini
zero shot
85.0
67.2
51.1
56.7
74.4
gpt mini
few-shot
go
sero-shol
86.1
78.8
56.0
57.0
76.9
gpt-4o
few-shot
86.4
71.4
For models gpt-4o, gpt-4o-mini, and deepseek-v2.5, their errors in mathematical formulations are relatively few. gpt-4o achieves the sota performance both two processes. Morcover, be seen from Table [8] the code transfer accuracy of Mistral-7B I--nsttttttttttttt and Qwen2-7b-Instrun is significantly lower in the zero-shot setting compared to the few-shot setting, which echoes
17

---

Published as a conference paper at ICLR 2025
previous observation (models with weaker coding abilities unable correctly write code relying on their capabilinies in a zero-shot setting but can solve problems correctly when provided with few-shot examples).
In addition, we have conducted buman-LLNy consistency statistics. We randomly select 20 samples and ask 2 master's degree students in computer science label them, comparing the labels those of LL--as-a-judge. The final results show that 19 out of 20 samples were consistent, This indicates the effectiveness of LLM-as-a-judge fine-grained error analysis. The prompt of LLM-as-a-judge is shown in Appendix E.1.5
B.2 IMPACT OF DIFFERENT PROMPTING STRATEGIES
We investigate whether step-by-step reasoning then generates the code can help improve model performance, We construct new prompt (first step reason then write code) named 'few-shot (first reason)". The experimental results are shown in the following:
Table 9: Performance of Other Prompting Strategy.
Models
Prompt
Overall Ace
GPT-3.5-Turbo
few-shot (ori)
56.4
GPT-3.5-Turbo
few-shot (first reason)
55.5 (-0.9)
GPT-4
few-shot (ori)
65.5
GPT-4
few-shot (first reason)
63.8 (-1.7)
GPT-4o-mini
few-shot (ori)
55.0
GPT-4o-mini
few-shot (first reason)
54.9 (-0.1)
GPT-4o
few-shot (ori)
69.4
GPT-4o
few-shot (first reason)
67.1 (-2.3)
'few-shot (ori)' is the original few-shot prompt in out paper, and "few-shot (first reason)' is provided in the Appendix E.1.3 It can be seen from the results that "first understanding the reasoning and then writing code does not significantly improve performance.
B.3 P PERFORMANCE
We provide the Pass @k results follows (we set the temperature 0.7) as following. The perfor- mance of the model improves with the increase in the number of generation attempts.
Table 10: Pass @k Performance.
Models
P
Pass 10
Pass@1 15
Pass@20
Pass 25
Pass 30
Llama-3-8P --struutt
40.5%
55.5%
59.5%
61.5%
62.5%
63.3%
Mistral-7B. In-trrructt--.3
Mistral-78-Instmarr-003...
47.1%
57.0%
61.3%
63.8%
64.7%
66. 1%
Qwen2-70 102-7b-IInstee
59.5%
63.5%
65.8%
67.3%
68.3%
68.6%
B.4 SOLVING EFFICIENCY OF THE GENERATED CODES
To fairly compare with code written by humans, for each tested LLM, only compare the runtime of the LLM's code with that of Human--ritten code on the samples where the LLM can provide correct answers. The statistical results follows (in seconds):
The aforementioned outcomes indicate that the efficiency of LLM written code is akin to that of human beings. The principal challenges for LLMs in solving optimization problems pertain to the accurate formulation of mathematical formulations and the generation code that devoid of emors.
18

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of Solving Efficiency
Models
Prompt
LLM code runtime
Human code runtime
zero-shot
0.160
0.154
Distral-7B-nnstruct--00
fow-shot
0.147
0.151
Qwen2-7b-Instruce
zero-shot
0,403
0.157
Qwen2-7b-Instne
few-shot
0.212
0.143
deepseek-v2.5
zero-shot
0.259
0.157
deepseek-v2.5
few-shot
0,146
0,135
gpt-4o-mini
zero-shot
0.283
0.159
gpt-4o-mini
few-shot
0,114
0.145
gpt-4o
zero-shot
0,199
0.157
gpt-4o
few-shot
0.152
0.158
B.5 POTENTIAL OF LLMS TO EXTEND PROBLEM COMPLEXITY
We conducte the following attempts: We provid sample DeepSeck- V2.5 and prompt the LLM to expand the number variables and constraints the sample. One of the expansion results is shown in the following:
Original Sample (6 variables, 3 constraints):
Define Variabies:
Gandhi Company is capable of manufacturing three types clething: shirts, shorte, pants. The manufacture of each type of clathing requi res Gandhi to rent the appropriate
sype of machinery. The company needs to determine the optinal aumber each type of
clathing manufacture, and tha rumber of each type of machinery
11 (*number of shirts to manufacture*; "Shirt", "range": "Shirt >- 0", "type"; "integer") manufacture"; "Shorts", "range";
("number of
"Lype"; "integez")
1/ (*aumber of pants to man:facture": "sants", "range": "Pants >= 0", "type": "integer")
1/ (*number of shirt machinery to rent": "Shirt_Xachiners" "range": "Shirt_ Machine_ >= D",
type*: "integer")
1/ (*numbez of shorts machinery to zent": MaoortsMachhnerr"" "range"; "Shorts_Machinery >-
"sype": "integer"|
11 (*numbe: of pants nachinezy to zeat"; "Panta_ Machineach,,, "range"; "Panis Machinery >- 0", type": "integez")
#t Define Objective Function:
The revenue shirt is $12, the revenue ahorts is $8, and the revenue pants is $15. The cost per shirt $6, the cost pez shorte is $1, and the cost per pan:s $8. machirery per
per shirt machinery per week is $150, and the reatal cost per pants machinery per week is $100. company wents maximize the weekly profit.
$200,
cos: per
// hatal_Revenus, = 12+shirt + 8+Shorts 15+Fants
// Total_Cost 6+Shirt 4+Shorts B<Fanta 200+3hirt_Mann MMachrrr_Maceerryy AodtShorta_Machinnee MMcchaets
// Objective Punctien: Maximize: @etel_Revene Cost
st Generste Comstreint-1:
hours required per shorts 2, labor hours required per pants is 6. Each week, 150 hours of labor are available.
hours required per shirt 3,
// 3+Shirt 2+Shorts 6*Pants 150
st Generate Constreint-2:
The clath raquirad shirt is aq the cloth required per shorts is yd, and the
cloth zequired per pants is sq yd. Each week, 160 yd of cloth are availabie.
!/ 4+Shirt 3+Shorts 4+Pants
44 Generate Constreint 3:
The company needs to
least machinery for each type of clothing. // Machi_nay >= 1, Machinyyy >= 1, Pants Manhs__nchhneey >e
Expanded Samples (10 variables, 8 constraints):
#4 Define Variables:
Gandhi cloth Company is capable of manufscturing five types of clothing: shirts, sherts, pasts jackets, and skirts. The menufacture of ezch type of clothing requires Bandhi to rent the appropriate type machinery. compeny needs determine the optimel each zype of clothing to manufacture, and the number of each type of machinery to rent. manufacture"; "Shirt", *range":
!! ("uumber of
D", "typa": "'nteger")
19

---

Published as a conference paper at ICLR 2025
(aaumbez of shorts manufacture"; "Shorts", "zange"; "Shorts >e 6", "type"; "integez")
17 (*number of pants to mansfacture": "Fants", "range": "Ponts >e d", "type": "integer"]
17 ("number of Jacketa to manufacture"; "Jackets", "range"; "Jackets 2e 0", "type"; "Integer")
17 (*aumber of skirts manufacture"; "Skirts", "range"; "Skirus se 0*, "type"; "integes";
// ("numbez of shirt machinery to reat"; "shirt_Xachinery", "range"; "Shirt_Ahchineey > 0", type*: *inzeger")
11 (*number of shorts machinery to rent"; "Shorta Machinerr, "ranga";
Machinery >
Q", "=ype": "integer"!
1/ (*number of pants machinery to rent*: Sants_Kachfnery", *range": MPants_Mach'mae nery >a D",
-ype"; "integer";
1/ (*aumber of jacket machinery to rent": "accket_ "range": "Jacket_lachinery >
0", "=ype": "inzeger"!
1/ (*number of skirt machinery rent": "Skirt_ Machtn__c,,,,r "range": rtkcht_ nery sype"; *integer")
34 Define iveective Punction:
The revenue per shirt is $12, the zevenue per shorts is 98, the zevenue pez pants is 815, the per skirt is $10.
jacket is 320, and the
The cost per shirt is $6, the cost pez shorts is $4, the cost per panta is se, the cost per $5. The rental cost per shirt machinery per week is $200, the rental cos. per shorts machinery per cest pez pants machinery pez week 8100, jacke: nachinery week ls $250, and the rental coat akizt machinery week is $120.
Jacket $10,
per
week $150,
cest per
the compeny
maximize the weekly profit.
// Total _evenneeeeeeeee 12+shirt + 8+Shorts 15-Fanta 20-Jackets
10-5kirts
// Total_ 6*8hirt
8-Fants 104tacketa SeSkitts S00-8hirt_Machhnee Soo-Shoott_Machineee i _ 250wJacket
// objective Punctien: Kaximize: ietal_Kevenue Tosal_
att Generate Constrain:-l:
The labor hours required per shirt is 3, the labar hours required par shorta is the labor hours required red per pants is 6, the labor hours requi red per jacket 1s 8, and the labor week, hours of laboz available. // 3+Shirt + 2+Shorts + 6+Pants 8+Jackets 4+Skirts <= 1500
requised per skirt is
34+ Generate Constraint-2::
The cloth required per shirt is sq yd, the cloth required per sheres is oq yd, the clath required per panta is yd, the cloth requi per jacket is 5 yd, and the clath required per skirt 1s 3 aq yd. Each week, 1600 aq yd of cloth are available. S+Jackets 3+Shirts
// 4+Shirt 3+Shorts
Cenerate Conatraint-3:
The compeny needs to
least machinery for each type of clothing. // Shirt_Kachineryy >= 1, Machiayyy >= 1, Machs__a >e 1, Mackee_Machhnery >e 1,
akirt_Machinery >=
Additional Constraints:
to ensure the problem remains feasible and interesting, can add additional constraints based the production capacity each machinery nery type.
#t## Constroint-4;
Bach machinery can produce up shirts per
// Shirt <- 20 Machineyy
lt+ constraint-S:
Tach shorts nachinery can produce up to 30 ahorta per week.
// Shorts
Shorta_ltchireryy
44 Constraint-6:
Each pants zachinery can produce to 15 pants per week.
// Fants 15 #enta_Machinery
344 Constraint-7:
Each jacket machinery produce up to 10 jackets week.
// Jacketa <= 10 Backet_Machines
*t* Constraint-8:
Each skirt machinery can produce up skirts por weak.
// skirts <= 25 Macht
Furthermore, we had 3 master's degree students in computer science to examine 10 expanded sam- ples. Specifically, the following points were checked:
Variables: Does the expanded variable relate to the optimization objective? Constraints: Are the expanded constraints reasonable? Formulation: Whether the formulation of the expanded mathe- matical model is correct. The verification results indicate that 9 out of the 10 expanded samples are correct.
20

---

Published as a conference paper at ICLR 2025
We consider this to be the foundation for the model to self-improve its complex problem-solving abilities. In fact, this is also one the future work plans we have.
C MORE DETAIL OF RESOCRATIC
We have already shown the process of our synthesis method in our paper. This section adds more detail our ReSocratic.
C.1 SEED DEMONSTRATION ATIONS
We collect 27 elaborate formatted demonstrations (13 linear scenarios and 14 nonlinear scenarios) in the seed pool. An example is shown in the following bellow.
Variables:
Chip Green la the head groundakeeper at Birdie vailey Golf club. There are four fert isera
Pertilizer 1 4) available the macket, chip would like to mix them Logether to obtain mixture. Chip aeeds determine the optimel proporticn of each festilizer in the mixture.
!! prroportion Ferti Hizrr in the compost": "x1", *range": "O
1%, "typa":
continuous"]
// ("pooportion Fertilizer continuous")
the compast": "x2", "range":
15, "typo":
// ("preportion of Fertilizer in the compost"; "x3", "range"; x3 14, "Lype":
continuous")
1/ ("proportion af Fertilizer in the compost"; "x4", "range": "0 x4 17, "type": continueus"
// The sum af the proportions should be 1: xl x2 x3 x4 =
94 Define Ob jeetive Function:
The price of Pestilizer 1-4 is $21.75, $23.75, $22.00, and $19.50 per 100 pounds, respectively
chip wanta to minimize the cost of the mixture per 100 pounda.
// Minimize: 21.75 23.75ex2 22,00xx3 19.50+x4
44 Generate Constraint-1:
10%, 89, 123, and 108; the Phosphorus percentage of bertilizer 1-4 is 88, 113, 76, and 104;
The Nitrogen percentage of Pertilizer
the Potash pereentage of Fortilizor 1-4 is 12, 154, 124, and 104.
Chip knows that the best proportion of the chemical content should be 10-8-12 (108 nitrogen, more
phosphorus, should be between 104 and 10.5t; the phosphorus level should be between 88 and 8.58; the potash level should be between 123 and 12.5%.
polast),
nitrogen level
// 10 <= 10xx1 Bix2 12-x3 10-x4 <= 10.5
// 6exi 11xx2 7xx3 + 10xx4 8.5
!! <= 12xx1 15+x2 12+x3 10+x4 12.5
Constraint the mixture should contain at least 208 Festilizer 1.
#4
// x1 5- 0,2
We show some statistical results of our formatted demonstration pool in Figure g
48.1%
29.0%
25.9%
14.8%
14.8%
0.0%
linear
nonlinse
<
4
5
6
7-777
0-600
601-1200 1201- 1800 1801-2400 --9001
(a) mount of type
Distribution of number variables
(c) Distribution scenario length
Figure 8: Statistical results of our demonstration pool.
We have shown the distribution of the formatted demonstrations of synthetic data in Figure The data distribution our synthetic dataset RESOCRATIC-29K similar to the data distribution in our formatted demonstration pool.
21

---

Published as a conference paper at ICLR 2025
C.2 TABULAR DATA SYNTHESIZE
To facilitate the synthesis of different types of data (linear and nonlinear), we produce different prompts, which are shown in Section E.2.1 and Section E.2.2 In addition, in the back-translate stage, to get the question text with table and the question text without a table, we construct two different
prompts, as shown in Section E.2.3
C.3 TACK-TRANSLATIO
We show example of trenssaanlltioon in the following:
Demonstration:
#4 Define Variables:
A cereal comрany makes nutritional. cereal, kida' cereal, and sugary cereal. The company needs
desermine the optimal aumbes boxes produce for each type cereal.
1/ (*number of nutritional cereal boxes"; "x", "range"; "x >- 0*, "type": "integer")
boxes"; "y", "sange"; "y o", "Lype": "integer") // (*aumber of sugary cereal boxes": "2", "range": "2 0f, "type": "integer")
// (*aumber kids'
#4 befine Object we Functions
The revenue per box of nutritional ceseal is $1, the revenue per kida' cereal is $1.50, and the
per sugary cereal is $2, Hlww many each should they make meximize revenue?
//
1.5y
Generate constraint-1:
Each box of nuteitional cereal requires 3 unita of eat, each kids' cereal secuices 1.5 units of os2, and each sugazy cereal zequires units of oat, The company has available 500 units of
// 3x
1.5y 22 <- 500
at Generate Constraint-2:
Tach box of nutrit ticnal ceresl requi unit sugar, each cereal requi 1.5 units cereal requizes units of The company has available 700
of sugar, and each
units of sugar.
//
4z 700
Back- tack-translated question:
cereal comрany makes nutritional. cereal, kida' cereal, and sugary cereal. The company needs desermine the optimal aumber boxes produce for each type cereal. Each box nutricional cereal requires unita of cat, each kida' cereal requizes 1.5 unita of ost, each sugary cereal requires eat. Eech box of nutritionsl cereal requires unit of sugar, each kids' ceresi requires sugar, and each sugary cereal requires units augar. available 700 units of sugar, The revenue box of nutritional. cereal is $1, the zevenue per kids' cereal is $1,50, and the zevenue per sugary cereal $2, Hlo many of each should they to maximize revenue?
company
available
company
As shown in this example, back-translation is almost equivalent to extracting the natural language part the demonstration, which is a very simple task.
We asked 3 master's ss degree students in computer science to examine 20 determine whether the generated questions in this step are correct. Specifically, the following points were checked:
samples to
Whether the generated question text includes all the information from the demonstration.
Whether the generated question is consistent with the question in the demonstration.
The test results show that all samples are correct.
C.4 TRAINING SAMPLE FORMAT
Specifically, we construct the training sample as follows:
"systea": "Please use python pyscipopt given optimization question." "user": "oom
22

---

Published as a conference paper at ICLR 2025
"assistant" (Codee)"
We replace "[Question)" " and "[Code}" with the synthetic question and the verified code in our RESOCRATIC-29K to form the training sample.
D BENCHMARK AND DATASET
D.1 MORE DETAILS OF DATA COLLECTION AND ANNOTATION
Scale of Labeling Team: Our labeling team consists of professional annotators with master's degrees. Four of them assigned labeling tasks, while the other two annotators are responsible for correcting inconsistent labeling samples.
Labeler Qualifications:
All team annotators have successfully passed the course Operations Research and have good theoretical foundation of mathematical modeling and optimization.
All team annotators are proficient in Python programming language and the PySCIPOpt library.
Source Data Details:
Original data sources: We assign annotators collect questions from textbooks (Bert- simas & Tsitsiklis, 1997 Conforti et al. 2014 Wolsey 2020), and a university's course assignments and examinations.
Data curation and annotation details:
1.
First, the annotators are required to write the questions markdown format. Simulta- neously, record the question type (linear, nonlinear, with table, without table).
Each question annotated by 4 annotators, and each annotator performs the annota- tion independently. Specifically, the annotators are required to write the mathematical models in markdown format for those collected questions. If there are standard an- swers during the data collection process, we will instruct the annotators to directly record the mathematical model the form of formula the markdown file. Other- wise, the annotators will be asked to write down the mathematical model themselves.
3. After this step, the annotators the problem and to record the values of the variables and the objective function.
required to write code using PySCIPOpt solve
For each question, we compare the code execution results the assigned annotators. If the results the same, the question is deemed correctly labeled; otherwise, will re-label and determine its final result.
D.2 DATA FORMAT
Data Format OPTIBENCH. We store E-OPT data in the form of JSON files. sample of our OPTIBENCH benchmark is shown below:
questtian"
*A rectangular gardan is to be constructed vaing a rock wall an one side of garden wize rencing determine the dimensions that would creete garden of maxinum ares. What is the maximum area?", resulta":
aldes. Given 103ft
rencing,
"The length of the garden": "50.0",
"The width the garden": *25.0", "The max! mom area of the garden": "5250.0"
type*: ncctable,, index"
We construct samples in dictionary format, and all the data stored as a list a JSON file. Each sample has the following fields:
23

---

Published as a conference paper at ICLR 2025
"question": The question text, presented in natural language, contains the background as well as the optimization objective and associated constraints. In order to solve the question, it is necessary to first find out the variables that can be optimized, then build mathematical model, and then call a code solver to get the optimal numerical results the variables and objective.
"results": This field presented in the form of dictionary, where the key is the natu- ral language description of the variables and objectives, followed by their optimal values. During the annotation process, if the taggers cannot confirm that there only one opti- mal solution to the problem, the results only contain the description of the optimization objective and its optimal value.
"type": This field records the type of the current sample, and there are four types: linear- table, linear-notable, non-linear-table and nonlinear-notable
"index": The index of the sample.
Data Format of RESOCRATIC-29K. We show sample of our RESOCRATIC-29K in the follow- ing bellow.
"question": logistics company operates different routes delivering packages. They need to determine the number of trucks to allecate to each route to optimize their operatiens, Each route has different cost and reveaue structure. On route 1, each truck incurs route 2, each truck thours cest of $120 per day and generatea revenue of $180 per day, coute 3, eaca Lzuck
$100 per day and generates
$150 par day,
per day generales zevenue of #210 per day. dn route 4, each truck incurs a cost of $150 per day and generates revenue $240 per day. The company aims maximize
cost
total daily profit across all rostes. The company has total of 50 trucks avai lable. Please he the company to determine the optimal allccation of trucks to each reute.",
"onde_ solution"; *import mazaknimport pyscipept (xtpeel Model() AnAn4 Define variables\nti model addVaa number of trucks 1b=0) f number of trucks routs 2\n13 = addel "INTEGERY 1133
model model'smoool pyscipopt name- 1b- "ame=
1'a22 model. (etea
1b=0; number trucks an route 3ATT4 medel. (
name=\*re
(r, Ib-0] number of trucks on route 4'ming Define objective
150 ri 100 Tl'nProfit_roue pouttel 180 r2 123 12 12AnProfit__ouure 210 r3
140 T3AmProfit routed 240 т4
T44nt the object function 18:
Maximize (Profit_routel Profit_route? Profit_reutes
\nob;" model \mmxximzze"" inmodel. (ob;
!'ddVar'' Ammodel. setoeeetieeeeeeeeeeeeee..oeee
Profit_ + srefit_ coute2 Profit_rcute: #rafit_ \ninf Add constraints'n iaddons Anka4 Frint the optimal solution (value
t The corрany has total 50 trucks avai vaaalable. Anmoda addCons +
T3
50) An'nt Solve the of the variables & the abjestivel Anprint (f-1 /nif getil.getttouua) == \"optimall print "Numbbr Trucka Route 1: model. geee...etaallo (T
print "Kumbr print (V "Numbes of Trucks on Route 3 (Y"umbee
of Trucks on Roate 2: \*, print 13 NNmmerr Trucks Boute d: \", model. : M,, (o3dl
An print (3 "Maximized zed Tatal Daily Prefit: \4, model. godel..gttbjvall. ()) Anclsesin print (V"The problem could not be solved to oatimality..")
We construct samples in dictionary format, and all the data stored list in JSON file. Each sample has the following fields:
"question": The question text, presented in natural language, contains the background as well as the optimization objective and associated constraints. In order to solve the question, it is necessary to first find out the variables that can be optimized, then build mathematical model, and then call code solver to get the optimal numerical results of the variables and objective.
"code solution": The corresponding python code solve the question.
E ALL PROMPTS
We show all the prompts we used in this section.
24

---

Published as a conference paper at ICLR 2025
E.1 PROMPTS FOR EVALUATION OF OPTIBENCH
E.1.1 ZERO-SHOT PROMPT
"system":
use python
given question.
"user":
[Code :emplate]:
"pythhn
import math
import popstpopt
Create new model
model pyscipost.Mtdel1)
Define variebles
Define jectiive Cunction
e4 set objective as variable (pyseipopt does not support zon-linear objective)
obg model aaddlladdvar//bb..
m setoeject ive ")..") $ "maxinize" or "ninimize"
model. addCona tob)
...) ob] Zunotion ion as constraint
Add conatraints
problem
model
Print the optimal solution (value the variables the object!
print (1-1+10)
if model. (edel.ggtStatus:) "sptimal":
elaes
print ("The problem could not be solved to optimality."!
[Follow the code Lemolate to solve the given question, your code should be enclosed in python'n( [M]
rqquestion
reating question here ...>
E.1. FEW-SHOT PROMPT
"system":
Please follow the given examplea and python code to solve the given question.
"user":
[Examole-1];
"iuusstion
bakery apecializes in producing two types of cakea: chocolate and vanilla. The bakery needs decide many
profit while considering the availsbility of ingredients and the minimu daily production requirement. profit from airs so maximize its daily profit from cake sales. Rach chocolate cake requirea eggs, and each vanilla cake requires egg, The bakery has daily supply of 130 eggs. Please help the bakery determine the optimal number of chocolate and vanilla cakes to preduce daily.
produce daily chocolate cake 95,
type
each vanilla cake $4.
bakery
"pythoo
inport math
import pyscipopt
Create a new modal
model pyscipept.Mech Modeel
variables
25

---

the production is production Published as a conference paper at ICLR 2025
number each type
produce daily
Choc model. zddVar (veype= veypP==INTTGER", INTEGER"" name="Choc", 1b=0) $ numbar of chocolate cakes
Van nodel.addVar ivt
name="Van", _bw0) & number or vanilla cakea
Lefine objective function
et objective
variable
ob) model aadvarr ('oj')
madel. setobject (cbj, "maximize")
model. addCons looj == S+Choc + 4+Van)
Add constraints
ct Each chocolate cake requires eggs, and ezch venills cake requires egg. The bakery has daily aupply of 100
model.addCoom 12+choc
Van <= 103)
Solve the problem
model. imiilze
Print the optimal solution (valuc of the variahles the objactival print (10)
if gelél..e -- "optimal";
print ("Number chocolate
", model.getVal (Choel)
print ("Number of vanilla cakes: mode:..geVal (Van)
print ("Xaximized Daily Prefit: ", medel.getOh
else:
print ("The problem could not be solved to opt imylimy.y"
[Example 2]:
"question
company producas three types of widgets: and The cempany neads determine hew many units of each widget to produce in next week.
Widget hours.
selling price is 105,
widget
selling price 159, materiel cost 75,
hours.
for widget 2, the selling price is 205, the meteriel cost is 95, and the production time is hours.
The company has 5500 available for material costs next week. The company wants to produce at units each widget next week. company wants to spend production next week. the company has only one preduction line and can only produce one widgos time. (which is defined the the selling profit divided by the of the production simes:.
hours
halp company to maximize the rate
earns profits
"pythhr import math
import pysc! popt
Create a new madel
pyscipopt Moo.pp....Moel!]
Lefine variebles
#t The company wants to produce at least 15 unita of each widgez nexs weak,
model.addVa vutypp=*TNTSSS
name="x*, 1h-10) number of unita of widget
addel (vtype=* INTSGER", name="y", lb=10) number of unita of widget y
model.addVas (vtype=* (vt.pp=""NTRGFR* TNTRGER", name="**, 1b-10) number of units of widget 7.
Define objective function
objective
(pyseipopt not support zon-linear objective)
obj aadel.addvar "maximize")
sedel..setoojjotlleeeeo
Prefit_X (10
X
Profit_ Y 015 7) Y
Profit_2
PreductionTime x
objective function Maximize (Profit_X Profit_y Profit_a) ProductionTimo #4 convert the at vilioon to multipl loation
model. tobj Production
ProfiL_X Frofit_y Procit_2)
Add constraints
#4 company
available naterial <= 500)
15 + #t company want.s to apend nost 200 hours production next week.
model. addCona
Y 4 2 200)
solve the problem
model. opllize
26

---

Published as a conference paper at ICLR 2025
optimal solution (value
objective)
print (f-f
ir model.gettttatus 1) "optima]";
print ("Number of Kidget X: model.getVal. (X1
print ("Number of kidget Y: ", gedei.getVall (X))
print ("Number of Kidgat
godel.
print ("Maximizen zed Profit Rate: ", mocel.get0oaee. 1
print ("The problem could not be sol ved to opt optimality.")
lFollow the examples volve the question):
"nuquustto ion
testing question
E.1.3 FEW-SHOT (FIRST ReASon) PROMPT
"system":
follow given exomples use python code given question.
"user":
Given question, you sould first cstablish the #fformulation step by step, then generate the ""yython" code to salve the problem vaing the "pysct Hibrary.
[Example-1]:
input:
"quesstonn
bakery specializes in producing two types of zakes: cherolats and vanilla. The bekery needs dacide many of each type cake produce daily maximize profit whi considering the availability of ingredients and the minimun daily production requirement. $5,
profit ains zo maximize its daily profi: from cake sales. Each chocolate cake requires 2 eggs, and each vanilla
chocolate
each vanilla
bakery
requirea egg. bakery has daily supply 130 eggs. Please help the bakery determire the optimal number of chocolate and vanilla cakes to produce daily.
Output:
"*formulation
variables
number of chocolate cakes
numbar of venills cakes
e4 Define ab jecttiiee function
maximize daily profit
sales:
constraints
Each chocolate cake requices 2 and each vanilla cake requires The bakery has daily supply of 100 eggs. 12+Choo Van <a 1001
"pyther
import math
inport pyscipopt
model
pyscipopt MssdipoptMMdeel1 Modelll
Chce model. eddvar (VEEA,,,, INTEGER", name="Choc", 1b-0) names"Van", 1b=0)
ob) model. o'b
model. setCbjective "maximize")
model. addCona tob] -- 5xChoe + 4*Van)
model.addcom 12+thoc Van <e 102)
podel.optiim.e.
print (riin1'==
if modal. get.stttttttttt ()
"optimal":
print ("Number chocolate ", model.getVal (Choet)
print ("Numbor of vanilla cakes: ", getva. (Van))
ptint ("Maxim nai
model..eeth.ppe (2)
27

---

Published as a conference paper at ICLR 2025
print ("The prohlem could not be solved to optimolsty..)
[Example22):
Inppt::
"question
company produces three types of widgets: y, and 7, The company needs to determine how many units of each widget produce in next week.
For kidget the selling price is 108, the material cost is 58, and the production time is hours.
Widget the selling price is 155, material
and the production time ia selling price 205, moterial coat is 96, the production time
hours.
widget
hours.
The company has 8500 available for material costs next week. The cempany wants produce at least 10 units each widget production next week, The company has only one production Line and con only produce one widger
week. the company
spend
200 hours
the company maximize the rate which (which is defined as the aum of the selling profit divided by the sun of the production times:
Flease
profits
Your Output:
ufforulation
24 Define variables
number of units of widget x
number
of widget
number of units of widget &
et Define objective function.
Maximize (Profit_X Profit_Y Profit_7) / Froductions i where,
Prefit_X (10 5 x
Prefit_y
Prefit_2 (20
ProductionTime
3 =
#4 Add constrainta
costs ihe company wants to spend at most 200 hours on produetion next week. (2 x + 200)
company
availsble
The company >= 10)
produce least 10 units of each widget week. >= 10, 2 10,
"pyther
import math
import popt popt
madel
ppsscipppt Model
x
model.addVaz (ytype-" INTEGER", (VIYyE="INIGGRR*
name="X*, 1b-10) * number of units of widget x name="y", 1b=10) number of units of widget
model.addVaz (vtype-" INTSGER".
model.adéwar (vtype=* INTSCER".
name="**, 1b=10) numbor of unita of widget a
addel_.ddvar'''bpj"
model. setcajecttttttteeeeoeooeoaaae.es leercaasettveeeeoo [ob), *maxi ml ze")
Profit_X
Prefit_Y Y [15 7) Y
Prefit_
Productioon:: me x +
model.addCom. to5) ProductionTinn Protit_X Profit_Y Procit_2i
modeon 15 mddonn
+ 9 500)
200)
madel.optimizel
print r.10:
if soddel.getttatus// ()
"optimsi":
print ("Number of Widget X: ", model.getVal. (X) print ("Number Widget
modei.getVal
print ("xumber of Kidget 2: ", model.getval (21)
print ("Xaximi Profit Rate: madel.getoajjea.
elaes
print ("The problem cculd not be solved to optimality..!
28

---

Published as a conference paper at ICLR 2025
[Fellow examples the given question:;
E.1.4 RESULTS EXTRACTION PROMPT
"'python
K... solution code generated by the LLM ....
coode output
s... code execution result
Accoding to the code outputr please give your final answer for the following query. (The boxed (\bboxed] only in Abbxed(3.22000))) and (axxed((0.2333))
answer
decimal places, such as NNboxed/27.000000
query for the variebles and objective ...>
E.1.5 LLM AS A JUDGE PROMPT
"system":
You are a math Leacher. You are assessing whether a student's answer is correct. Specifically, you will be provided with question, the ground truth mathematical formulation, and the student's a aoda. What you need to do is datermine whether the mathemstical formulation in mathematical formulation You need to first extract the nathenalical formulation from the student code and then ground truth. should express your judgment as \ooxed[ corract) or \boxed[Wrong,
student's
compare it
"user":
et Question:
iquestion (on
#4 Ground Truth Mathemat loal Pormolatho fom on:
#t Student Sode:
[student_code]
E.2 PROMPTS OF RESOCRATIC
E.2.1 LINEAR DEMONSTRATION GENERATION
"system":
please follow the scenario examples generate [Now Scanario) with new background. The acenario should be real-world linear opt imization problen. Make aure that the nathematical logie in [New Scenario] is correct.
"user":
[Scenario Format]
#t Define Variables:
natural languzge description.
// formal definition of variables (integer, real, binary, stc.) and sheir doma
#t Define Objective Function:
natural language description.
definition be one objective function.
objective function, maximize minimize something. only
94 Generate Constraint-1;
natural language description.
// formal definition of constraint-1
Generate Constraint-n:
29

---

Published as a conference paper at ICLR 2025
natural Language description.
// formal dafinition of constraint-n
K... Sample scenerios in the example pool ...?
[New Scenaric]:
E.2.2 NONLINEAR DEMONSTRATION GENERATION
"system":
Please follow the scenario examples to generate a [New Scenario] with new background. The acenario should real-world # opt zation probl em. Make aure that the mathematical logic in [New Scenario] is correct,
"user":
[Scenario Format]
#4 Define Varlables:
language description.
// fornal definition of variables (integer, real, binary, etc.) and cheir domains.
et Define Objective Function:
natural languzge description.
!/ formal dafinition of innoni objectiva function, maximi minimi iin.
There can only be one objective.
st Generate Constreint-l:
natural language description,
// formal definition of constraint-1
Generate Constreint-n:
natural languzge description,
//
of constraint-n
Sample acenarios
example pool
[New :cerario]:
E.2.3 QUESTION GENERATION
"system":
are mathematics assi stant. Now, you provi ded iptimization scenario. Please Iollow the example to convert the given acenario to question.
Generating questions without table.
"user":
ITask Description]:
involves optimizetion problem. scenazio organized into few sections start with "td*. taxt
given
scenario. tha mathomatics:. formal solution of the scenario is provided in the commenta starting with "//".
saction contains
Your job is to convert the scenaclo into question without missing any information. The question should be clear and concise, and do expose the mathematicel formal solution of the scenario.
[Example convarting Scenario to Question]
"Ssoenario
Define Variables:
company produces five types of widgets:
and v. The company needs determine how many units each widget to produce in next weak.
(*aumber units widget x"; "x", "range": "x
"type": "intege=")
// (*aumber of units of widget y"; "y", "range": "y >= 6", "type": "intege=")
// ("umberr of anita widget "z", "range":
o", "type": *intege"*)
30

---

Published as a conference paper at ICLR 2025
(Maumbes of units widget w": "w", "range": "w 0", "type": "intege=") // (*number of unita of widget v*: "v", "range": "y >e 0", "type": "intege=*)
44 Define ive Funetion:
For widget X, the selling price is $15, the meteriel cost is 95, and the production time is hours.
For Kidget %, the selling price is $15, the material cost 18 57, and the production time 1a
For Widget the sel price is $29, the material cost is 89, and the product on ti 1a
hours.
For widget the selling price is $25, the material cost is $11, and the production time material
hours.
Widget the selling price is
and the production time
houra,
company only one production
can only produce one widget time. The company alms to maximize the rate at which it earns protits (waich is detined the sum of the selling prefit divided by the sur: of the production times).
// Selling profit of X: Profit_X (13 5)
// Selling profit of Y: Profit_ Y (15 - 7) +
// Selling profit of Profit_2 123
1/ Selling profit of K: Prořit_K - (25 111 +
Selling profit of V: Profit_V (30
1/ So, the obctetive Lunction ls: Maxinize (Profit_X Profit_Y Profit_3
Profit_N
Profi=_V) / (2
4 1
3t Genarate Constraint-1:
company
available material 11 13 900
// 5 1
44 Generate Constraint 2:
The company wents to produce at least 10 units of each vidget next week.
// 10; >= 10; >= 10; >= 10; >= 10
44 Constraint
The company wents to spend at most 200 hours on production zext week.
//
st Gemerate Constreint-4;
The company
that the total production of Widget does exceed the comaired production of Widgets X, Y, and 2.
// XIII2
. "'question
company produzes five types of widgets: x, Y, 2, W, and v. The company needs to determine how many units each widget to produce
For Widget X, the selling price is $10, the material cost is $5, and the production Lime is hours.
for widget the selling price is $15, the matericl cost is $7, and the production time is
hours.
widget the selling price is $23, the materiel is 89, and the production time is hours.
Widget
selling price is $25, material
the production time production time to spend
hours.
widget selling price is $30, materiel cost $13, hours.
company has 8900 available material costa next week. The company wants prodice least 10 units each widget production next week, The company wants to ensure that the total production of Widget k exceed
waek. the company
200 hours
production Nidgets X, Y, production line and can oaly produce ene widget at time. help company maximize the rate the sum of the sel selling profit vidid the sum of the production times).
company only
earns profits (which defined as
[Follow Example Convert following Scenario Ques:ion]
Generating questions with table.
"user":
[Task Description]:
You will be given scenario that involves optimization problem. The scenazio is organised into few sections start with "Hd
Bach section contains faw lines of text that deseribe the scenario. the mathematics:. formal
solution of the scenario is provided in the commenta starting with "//". job to convert
missing any information. question should be clear and concise, and do not expose the mathematical formal solution the acanario.
scenaric into question
31

---

widget next Published as a conference paper at ICLR 2025
[Example of converting Scenario to Question with table]:
"'scenario
#4 Define Variables:
company producas five types of widgets:
v. The company needs determine
how many units of each widget to produce in next week.
1/ ("number of units widget x": "x", "range"; "x 0", "type": "intege=")
1/ (*number of unita of widget y": "y", "range": "Y >u 0", "type": *intege"")
1/ ("aumber of units of widget g"; "3", "range"; "5 0", "Lype": *intege:")
// (*aumber of units of widget W": "w", "range": "a >= 0", "type": "intege=")
// ("number of units of widget v": "y", "range"; "y >= 0", "type": "intege=*)
44 Define jeettivee Lons
Widget x, selling price $10, moterial coat is 95, the production time hours.
For Widget V, the selling price is $15, the material cost is 87, and the production time is hours.
For widget 2, the colling price is $20, the matericl cest is 89, and the production time in Tor Widget 8, the selling price is $25, the material cost is $11, and the production Lime de hours.
For Widget V, the selling price is $30, the material cost is $13, and the production time de hours.
The compeny has only production line and only preduce widget time. The company aims to maximize the rate at which it earns profita (which is defined as the sum production times).
the selling profit divided by
1/ Selling profit of X: Profit_K (10
5)
X
// Selling profit of Y: profit_y (15 7)
1/ Selling profit of 2: Profit_2 (23 9)
// Selling profit of #: Prošit_K (25
111 *
// Selling profit of V: Profit_y 130 131
// 5o, the abject ive function la: Maxinize (Frof't Profit_Y Profit_a Profit_N
Profi__V)
Generate Constreint-l:
The compeny has 8900 available for material costs week. 11
//
<= 900
#4 Generate Constraint-2:
company wents produce at 10 units // x >m 10; >m 10; >m 10; w dm 10; V >m 10
#t Generate Constreint-3:
The company wents to apend at most 200 hours on production next wock.
// 2
200
# Constraint-4:
The company wents to ensure that the total production of widget w does not exceed the combined production widgets x, Y, +
//
""question
company produces five types widgets: how many units of each widget to produce in next weak, The selling price, matoricl cost, and production time for each widget are we the fowinging Table.
V. company determine
kldget
Sel 1in Price Material Coat
Production Tima
X
105
5$
2 hours
203
95
hours
255
115
hours
30$
138
hours
company
available material least 10 units of each widget next week. The company wants to spend at most 200 hours production next week. does not exosed the combined prodsction of Widgeta X, Y, and 7, The company has only one production line and can oaly produce one widget et Lime.
company
produce
company wants ensure that total production widget
Plesse help the company maximize the rate which it earns profizs (which is defined as the sum of the aelling profit divided by the sum of the production times).
Folllw the Example to Convert the following Scenario to Ques:ion with cable]:
32

---

height cylindrical jaz Published as a conference paper at ICLR 2025
E.2.4 CODE GENERATION
"system":
mathematical assistant, you provided with intimization scenario with its corcespending question. Please follow the examples to solve the opt sptinization scenaric using python code with pyscipopt. tlips: objective non-linear objective. 2. To expedita compitation, convert division to miltpplicttion.)
avoid
"user":
[Example-]]=
s'sccnario
et Define Variables:
Now we need to creste a cylindrical metal for with a metal shell.
!/ variables: ("radiua of andrrdiaall jart: ty", "height the iydincccal jar"; "h"),
where 2, h >-
44 Define jective Functions
The cost of the netal $10 per square meter. Find the dimensions that will minimize the cost the metel manufacture the jar.
!! The surface area of the cylindrical jar is the sum of the area of the two ciroular ends and of circular metal is 10 (2\pive"2 2ipisrh).
surface surface ares is ZApi+zh.
\pi and lateral
So, the surface area the cylindzical jar 2)pixr'2 2ipivrh,
// So, the objestive function is: Minimize 10 (2Apiv="2 2\piarh)
3t Generate constraint-1: jaz
least 1003 cubic centimetezs.
// Api-r 2h >- 1000
"pyythen
import math
inport pyacipopt
create new model
pyscipopt. MMdell!)
Leffnee varicbles
et radius height cylindrical jar
model. (VLYI==" name="r", 1b-0, ub-100) radiua of the oy cylindrical Jac name="h", 1b-0, ub-100)
model.addVaz
Refine cojectivv function
94 set ivve
variable (pyacipopt does not support non -ineer objective)
ohj model.addVars ('ojj')
model. setcojet
(ob), "minimise")
44 the ob jectivve function is: Minlmize 10 x (2NplAr"2 2Apisch:
model. addCons tobj
(2smsca.piezve2 2-math.pivvsa))
coratraints
44 The volune of the jaz must be least 1000 cubic imetera.
model. addCons (math.pf pre be 1000)
Solve the problem model.aptimizel)
Print the optinal solution (value the variables the objective!
print (1=r00;
it model.getštatus. =- "optimai":
print ("Radius
cylindrical jar: ", gedel
print ("Height of the cylindrical jar: ", model. model.getVall(h)
print ("Xinimized cost: ", model. getoblget (I)
aet
print ("The problem could not be solved to optimality.."
[Example=2]:
sgcenario
44 Define Variables:
You aze designing sectangular postez by cutting from rectangular piece of paper.
// variables: ("width the poster": *w", "height of the poster": *h";, where h >=
#t Define Objective Function:
The top and bottom margins are 2 inches, and the side margins are inch. what dimensions of the area paper used?
the poster should you use
33

---

papez 2+2. Published as a conference paper at ICLR 2025
/7
used paper // Therefore, the objactive function i5: Minimiza (M 2) th 4)
height
44 Generate Constraint 1:
the poster must have an area of 100 squaze inches.
// The of the
given by the product of the width and the haight, and it is gaven that the area is 103, Therefore, the constraint 1s w h 103
...
""pythoo
import math
import pyscipopt
Create new model
pyscipopt. MMdell
Define variebles
st The width and height of the poster
v modcl.addVar (vtype=" "CNTIII="N,, nome=""", 1b=0, ub=100) width of the poster
aadear "ONTTNONOO,,,,, name="h", Ib-0, ub-100) height poster
Define dbjective function
94 set jeetivve
variable (pyscipopt does not support non linear ob jective)))
obj addea..ddVa
model. setubjective "minimize")
#t the objective function is: Minimize (w + 2) th 4))
model. addCona loo)
#4 The poster must have
of 100 aquare inches. model.addComd iw 1001
solve the probl em
model. (m)
print optimal solution (value variables objactivs)
(" (1=)
if model. godel..etstatus
"optimsl":
print ("Width of poster: ", modal. getval (H))
print ("fleight of the poster: ", model geeVVal
print ("Minimized
Paper Used: ", model. (V1
else:
print ("The problem could
solved optimality."!
[Convert the following Scenario to code]:
Fut your synthetic scenario here ...>
34


Reviews:
[{"review_id": "SjN0F0tXWh", "reviewer": "Reviewer_hRyJ", "summary": "This paper proposes OPTIBENCH, an end-to-end benchmark for evaluating large language models (LLMs) on optimization modeling. OPTIBENCH covers various problem types, including linear and nonlinear programming with/without tabular data. To further alleviate data scarcity, the authors introduce ReSocratic, a `reverse` data synthesis method that generates the RESOCRATIC-29K dataset. The authors demonstrated that fine-tuning open-source LLMs (Llama2 7b and Llama3 8B) on RESOCRATIC-29K significantly improves their performance on OPTIBENCH. The paper contributes a comprehensive benchmark, a reverse data synthesis method, and a synthetic dataset to advance evaluations on optimization modeling with LLMs.", "questions": "How does the current perform if using the pass@k metric, which measures the performance of the generated code after k attempts? \n\nThe experimental results show that DeepSeek-V2 severely underperforms in the zero-shot setting compared to its few-shot version, while other models do not exhibit such a drastic difference. Is there any reason behind this?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "OPTIBENCH is a comprehensive benchmark that effectively evaluates the optimization problem-solving abilities. It inlcudes nonlinear programming problems, along with tabular data, reflecting realistic scenarios. It also makes the benchmark challenging enough.\nBy requiring LLMs to understand the problem, perform sound reasoning, and generate code to invoke a solver, OPTIBENCH provides a holistic assessment of LLMs' reasoning and coding skills. \nThis general evaluation approach is valuable for measuring the models' ability to apply their knowledge to practical scenarios and can guide future research on LLMs' capabilities in reasoning tasks. Moreover, the data annotation with strong human guidance looks reliable.\n\nThe benchmark evaluates the correctness of LLMs' solutions by comparing the execution results with ground truth answers. This objective evaluation method is more reliable than pure text-based benchmarks in my perspective.\n\nThe authors conduct an ablation study that provides valuable insights into the effectiveness of their benchmark and proposed data synthesis method. This includes examining the impact of removing reasoning steps and filters, as well as comparing forward and reverse data synthesis approaches, along with the visualization of the dataset.\n\nThis work fills a gap in existing benchmarks by providing a diverse and challenging dataset for end-to-end optimization problem-solving. Moreover, the proposed ReSocratic method and the resulting RESOCRATIC-29K dataset offer a promising approach to address the data scarcity issue.", "weaknesses": "The paper lacks a fine-grained error analysis, which could provide valuable insights into the specific challenges faced by LLMs in optimization tasks. A breakdown of error types, such as errors in understanding the problem, formulating the optimization model, or transferring the mathematical model to code, or summarizing the execution into required formatted outputs, would be helpful to guide future improvements in LLM design and training.\n\nThe authors didn't provide an analysis (maybe just a few case studies) of false positives and false negatives in the benchmark results. As with any benchmark, it is expected that both types of errors can exist, and understanding their characteristics could contribute to the benchmark's reliability.\n\nThe data collection and annotation process lacks details on deduplication and quality control measures. This helps to ensure the collected optimization problems were diverse enough to measure LLM's modeling capability. \n\nThe paper didn't explore the impact of different prompting strategies during inference. For example, investigating whether step-by-step reasoning then generates the code can help improve model performance.\n\nThe range of models tested is relatively limited. As a benchmark paper, one would expect a wider variety of LLMs to be included, such as code-specialized models like CodeLLaMA or the Mistral line of models. \n\nThe reliability of the back-translation step in the ReSocratic data synthesis is not thoroughly investigated. This helps measure the quality and consistency of the questions generated through back-translation, as well as the potential impact of any errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uFjflE5Id4", "reviewer": "Reviewer_GR55", "summary": "This paper proposes a benchmark for end-to-end optimization problem-solving and a data synthesis method. The benchmark contains various optimization problems, including linear and non-linear programming with or without tabular data. With the constructed benchmark, the fine-tuned LLMs exhibit a stronger performance.", "questions": "- Could you please provide some results on LLMs handling the large-size problem instances?\n- Could you please verify the accuracy of the formulation given by the LLMs?\n- Would it be possible to compare the solving efficiency of the codes produced by the LLMs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a benchmark containing various problems, including linear and non-linear programming with or without tabular data, which can better evaluate the ability of LLMs.\n- The reverse data synthesis approach is novel and reasonable.\n- The experimental results show that ReSocratic outperforms the forward data synthesis method, and the fine-tuning results are promising.", "weaknesses": "- The authors may want to generate instances with more constraints and variables, as few instances in the paper have more than 7 variables. Thus, this raises my concern about LLMs' ability to model problems with large instance sizes.\n- Given that a single optimization problem can have multiple valid formulations, it would be beneficial for the authors to verify the accuracy and equivalence of these formulations with ground-truth ones.\n- There are questions regarding the solving efficiency of the generated codes. It would be valuable to assess whether the code produced by LLMs can outperform human-designed formulations and codes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9k0yMYCgPS", "reviewer": "Reviewer_PMkM", "summary": "This paper proposes a study on the challenge of evaluating and improving large language models with respect to solving optimization problems. While the capabilities of large language models have recently improved with respect to mathematical and logical reasoning, the few existing benchmarks are limited to very simple tasks in linear programming and include little complexity representative of real-world applications. The authors present OPTIBENCH-a benchmark that attempts to evaluate the end-to-end ability of LLMs to solve optimization problems. It includes 605 diverse problems in the areas of linear and nonlinear programming, both tabular and nontabular; models are expected to generate code to execute and produce numerical answers.\n\nMotivation: The authors mention that all existing optimization benchmarks are small-scaled and overly simplistic, where most of their problems are linear. Besides, open-sourced LLMs are substantially outperformed by more complex closed-source models like GPT-4. This demonstrates that a more challenging benchmark is needed along with efficient data augmentation methods to make smaller, open-source models perform better.\nMain Contributions: OPTIBENCH Benchmark: The authors propose a new benchmark that includes various optimization problems, intended to test the LLM's capabilities in handling optimization tasks of varying complexity. The benchmark problems come in natural language formulation, which requires the generation of Python code that can be executed.\nReSocratic Data Synthesis Method: Based on this, the paper proposes ReSocratic-an approach for attempting to synthesize synthetic data in optimization. Unlike other methods, which first generate questions, the proposed ReSocratic works backward, constructing incremental step-by-step optimization demonstrations, and back-translates those into questions afterwards. This process aims to produce varied and consistent data. In this way, a dataset of questions, called RESOCRATIC-29K, is created.\nPerformance Improvement: This work presents data suggesting that fine-tuning open-source models over RESOCRATIC-29K might help narrow some of the gap with proprietary models. It reports accuracy improvements from 0.0% to 30.6% for Llama-2-7B-Chat and from 13.6% to 51.1% for Llama-3-8B-Instruct.\nEvaluation and Analysis: The authors carried out extensive experiments in which the various models compared against each other in zero-shot, few-shot, and supervised fine-tuning settings. They mentioned nonlinear data and tabular data to be really challenging; increasing problem difficulty considerably raises the challenge, and their experiments indicate the ReSocratic method performs better than the straightforward forward synthesis techniques. Ablation studies further support the influence of key factors such as mechanisms for filtering and step-by-step synthesis.\nThe paper presents arguments for the potential utility of OPTIBENCH, and examines the effects of ReSocratic on open-source LLMs' optimization capabilities.", "questions": "My major questions are around the data curation process.\n# Data Labeling Process Questions\n\n1. Scale of Labeling Team\n- Total number of data labelers involved\n\n2. Labeler Qualifications\n- Required experience\n- Necessary skills\n- Background requirements\n\n3. Source Data Details\n- Data collection methods\n- Criteria for benchmark curation\n- Original data sources", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: This work introduces the OPTIBENCH benchmark, a comprehensive tool designed to rigorously evaluate large language models (LLMs) on complex optimization problems, significantly advancing beyond existing benchmarks like MAMO and NLP4LP. Unlike prior efforts that primarily focus on simplistic, linear problems or abstract formulations, OPTIBENCH incorporates nonlinear and tabular data, simulating real-world scenarios more accurately. Additionally, the introduction of the ReSocratic data synthesis method is an inventive departure from traditional forward synthesis techniques, employing a reverse step-wise approach that yields a higher quality and diversity of synthetic optimization problems. While similar methods have been applied in other contexts, the adaptation and customization for optimization modeling distinguish this work meaningfully.\n\nQuality: The paper is grounded in rigorous experimental design and analysis. The authors perform extensive evaluations using a variety of open-source and proprietary models, including detailed comparisons in zero-shot, few-shot, and fine-tuning settings. The results are compelling, showcasing substantial performance improvements, particularly when using ReSocratic-generated data, as demonstrated by a 30.6% accuracy improvement for Llama-2-7B-Chat and a 37.5% boost for Llama-3-8B-Instruct. The ablation studies and the analysis of data distribution and variable types further bolster the credibility of the experimental findings, highlighting the robustness and effectiveness of the proposed benchmark and synthesis method.\n\nClarity: The paper is well-structured and communicates its contributions clearly. The presentation of the OPTIBENCH benchmark, complete with illustrative examples of linear and nonlinear problems, aids in understanding its scope and novelty. The ReSocratic method is also carefully detailed, with step-by-step descriptions and visual aids that effectively convey the synthesis process. Moreover, the inclusion of a thorough comparison with existing benchmarks and methods provides a clear context for the advancements made.\n\nSignificance: The significance of this work lies in its practical applicability and potential impact on the field of optimization problem-solving with LLMs. By addressing the limitations of previous benchmarks and providing a more realistic and challenging testbed, OPTIBENCH has the potential to become a standard for evaluating and advancing LLMs in applied mathematics and operations research. Furthermore, the ReSocratic approach offers a scalable method to generate high-quality data, which could influence future research in data synthesis for complex reasoning tasks. The reported performance improvements also suggest practical benefits for enhancing open-source models, bridging the performance gap with proprietary models like GPT-4.", "weaknesses": "## Benchmark Contribution:\nThe present paper introduces OPTIBENCH for the evaluation of large language models in respect to solving optimization problems. However, such benchmarks already exist, for example MAMO, ComplexOR, and NLP4LP, which test LLMs concerning their ability to interpret natural language descriptions of optimization problems and create corresponding mathematical models. The extension in OPTIBENCH relates to nonlinear elements and tabular data. This feels like an incremental extension rather than a transformative one. The paper should better highlight the aspect of how OPTIBENCH goes significantly beyond the established benchmarks.\n## Data Generation Process: \nIt proposes the data synthesis methodology, namely ReSocratic, and claims novelty in generating high-quality synthetic data. Earlier research has addressed reverse synthesis methodologies or back-translational techniques, especially for data augmentation and machine translation. Following are some related works which prove similarity:\n\"Data Augmentation for Code Generation Tasks\" shows back-translation-style data augmentation that aims at improving code generation, just like ReSocratic. These techniques create synthetic variations from existing data. There are a number of similarities with this paper which raises the suspicion that the contribution of ReSocratic is more of an extension of the currently known methods and not genuinely new.\nMathGenie: Synthetic Data Generation through Question Back-translation to Improve LLMs' Mathematics Reasoning, 2024: MathGenie synthesizes questions from mathematical solutions through a backward process, just like ReSocratic. Be that as it may, both methods are distinguished by their enhancing of reasoning through backward data synthesis, an issue that raises a question as to the novelty of ReSocratic in the given context.\n\"Data Augmentation for Code Translation with Comparable Corpora and Back-Translation\" (2023): This work describes the generation of diverse code examples for translation tasks using back-translation, which aligns with ReSocratic's structured data synthesis approach. In fact, this would mean there is some correlation in the methodological procedure between the current work and existing strategies.\nThese related works demonstrate that such techniques as back-translation and reverse synthesis have already been successfully applied for code generation and mathematical reasoning. This paper could better explain how ReSocratic is different or improves these established methods. Moreover, discussing the possible biases of the reverse synthesis process would help to enhance the reliability and applicability of the generated data.\n## Conclusion:\nIn general, this paper contains some interesting ideas; however, the contributions of both OPTIBENCH and the ReSocratic data generation method appear relatively modest with respect to prior work. It also shares similarities with approaches that have previously been proposed, indicating potential for better establishing the unique footprint and benefits introduced by the methods. An extended comparative study and some empirical evidence could be more enriching in showing points of strength for the proposed contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)