OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling

## Abstract

Abstract Large language models (LLMs) have exhibited their problem-solving abilities in mathematical reasoning.
Solving realistic optimization (OPT) problems in application scenarios requires advanced and applied mathematics ability.
However, current OPT benchmarks that merely solve linear programming are far from complex realistic situations.
In this work, we propose OptiBench , a benchmark for e nd-to-end opt imization problem-solving with human-readable inputs and outputs. OptiBench contains rich optimization problems, including linear and nonlinear programming with or without tabular data, which can comprehensively evaluate LLMs’ solving ability.
In our benchmark, LLMs are required to call a code solver to provide precise numerical answers.
Furthermore, to alleviate the data scarcity for optimization problems, and to bridge the gap between open-source LLMs on a small scale (e.g., Llama-3-8b) and closed-source LLMs (e.g., GPT-4), we further propose a data synthesis method namely ReSocratic .
Unlike general data synthesis methods that proceed from questions to answers, ReSocratic first incrementally synthesizes formatted optimization demonstrations with mathematical formulations step by step and then back-translates the generated demonstrations into questions. Based on this, we synthesize the ReSocratic-29k dataset. We further conduct supervised fine-tuning with ReSocratic-29k on multiple open-source models.
Experimental results show that ReSocratic-29k significantly improves the performance of open-source models.
The code and data can be found at https://github.com/yangzhch6/ReSocratic .

## 1 Introduction

Large language models (LLMs), such as GPT-3 (Brown et al., ), GPT-4 (Achiam et al., ), and Llama (Touvron et al., ; ), have demonstrated their superior capability in logical reasoning (Suzgun et al., ; Huang et al., ) and mathematical reasoning (Ling et al., ; Patel et al., ; Yang et al., ; ), such as solving elementary (Cobbe et al., ) to high-school level (Hendrycks et al., ) math problems. Yet a follow-up curiosity is to what extent LLMs apply their mathematical intelligence to practical scenarios.
Optimization problem solving (Ramamonjison et al., ; Xiao et al., ; AhmadiTeshnizi et al., ; Huang et al., ) is a field of applied mathematics that has been proven beneficial in many applications such as supply chain management, power energy scheduling, marketing, and quantitative trading. Optimization problem-solving is a comprehensive task that evaluates the mathematical and coding capabilities of LLMs. To provide the optimal solution to an optimization problem, LLMs are not only required to understand and construct the mathematical formulation according to the given problem but also to call an optimization solver to get the final answers.

Previous studies (Ramamonjison et al., ; Xiao et al., ; AhmadiTeshnizi et al., ) have explored using LLMs to solve operations research problems.
However, these studies have not yet extended to more generalized scenarios regarding practical optimization problems. Specifically, NL4OPT (Ramamonjison et al., ; ) uses named entity recognition to extract entity and numerical values in the given question text, and then formulate it into mathematical models. They only measure the model’s ability to correctly construct mathematical formulations, without considering solving the mathematical formulations being constructed.
To further evaluate models providing the final optimal solution, i.e., the numerical values of the variables and the optimization objective, ComplexOR (Xiao et al., ) and NLP4LP (AhmadiTeshnizi et al., ) benchmark the models to solve a problem with an optimization solver in the setting without explicit input numbers. However, due to the difficulty of collecting such data, these benchmarks are still on a small scale.
Moreover, the recent MAMO (Huang et al., ) proposes to further benchmark optimization problem solving with a code solver.
Nevertheless, one common pitfall of all the aforementioned works is that they merely focus on linear programming, whereas nonlinear optimization problems and practical tabular format are not included. Table 1 provides a comparison of the aforementioned benchmarks.
We also discuss the differences between current benchmarks in detail in Appendix A.

**Table 1: Comparison of optimization problem solving benchmarks. “End2End” indicates whether the benchmark requires the model to solve for the optimal values of the variables and the optimization objective. “Implicit/Explicit” refers to whether the numeric values in the question are displayed.**
| Benchmark | Question | Size | End2End | Linear | Nonlinear |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Form | w/ table | w/o table | w/ table | w/o table |  |  |  |
| ComplexOR (Xiao et al., ) | Implicit | 37 | $\surd$ | $\times$ | $\surd$ | $\times$ | $\times$ |
| NLP4LP (AhmadiTeshnizi et al., ) | Implicit | 57 | $\surd$ | $\times$ | $\surd$ | $\times$ | $\times$ |
| NL4OPT (Ramamonjison et al., ) | Explicit | 289 | $\times$ | $\times$ | $\surd$ | $\times$ | $\times$ |
| OptiBench (Ours) | Explicit | 605 | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ |

To bridge this gap, we propose OptiBench, a new benchmark with high-quality data to evaluate LLMs’ end-to-end solving ability in optimization tasks. We carefully select 605 questions and conduct careful manual verification to form the dataset.
OptiBench contains linear and nonlinear programming with both integer and mixed integer variables in the programming problems.
OptiBench also includes tabular data, which fills the gap in current optimization benchmarks.
Figure 1 demonstrates OptiBench examples in the four problem types.
A model solves an OptiBench problem by reading the natural language input and then generating Python code that solves the problem, where the code will be processed to acquire the numerical value of the variables and the objective function.

Figure: Figure 1: Our OptiBench contains various types of data (linear, nonlinear, table). To enhance readability, we present the table in an Excel format and include a diagram to illustrate the nonlinear example without a table.
Refer to caption: x1.png

Additionally, the data scarcity issue in optimization tasks (Xiao et al., ; AhmadiTeshnizi et al., ) cannot be ignored. Annotators are required to possess good professional knowledge, making the process not only expensive but also time-consuming and labor-intensive. In addition, there is a significant performance gap between small open-source models (e.g., Llama-2-7B, Llama-3-8B) and large models (e.g., GPT-4) in many complex reasoning tasks (Ling et al., ; Patel et al., ; Suzgun et al., ; Yang et al., ; ; Huang et al., ).
To this end, we propose ReSocratic, a novel method for synthesizing diverse and reliable data for optimization problems. Unlike previous methods that synthesize questions first and then answers, ReSocratic incrementally synthesizes the formatted optimization demonstration in a reverse manner, and finally back-translates it into a question.
Benefiting from intermediate reasoning steps, the quality of ReSocratic’s synthetic data is higher than that of previous methods.
We collect 29k samples with ReSocratic, resulting in the ReSocratic-29k dataset. In summary, our contributions are as follows:

- •
We introduce a high-quality benchmark OptiBench for optimization problems with complex instances in multiple forms. As far as we know, this is the first large-scale benchmark including nonlinear and tabular data to measure LLMs’ end-to-end problem solving abilities. We conducted an in-depth evaluation of a range of LLMs under various settings.
- •
We propose ReSocratic, a novel method for generating diverse and reliable data for optimization problems. In particular, ReSocratic synthesizes complex reasoning data from scratch in a reverse manner.
- •
We synthesize the ReSocratic-29k dataset with 29k samples by using our ReSocratic.
Experimental results show that conducting supervised fine-tuning with ReSocratic-29k significantly improves the performance of open-source models on OptiBench (e.g., Llama-2-7B-Chat from 0.0% to 30.6%; Llama-3-8B-Instruct from 13.6% to 51.7%), which further demonstrates the validity of our synthetic data.

## 2 Related Work

Benchmarks for Optimization Modeling.
More closely related to our approach, the NL4OPT benchmark (Ramamonjison et al., ; ) investigates controlled generation techniques to obtain an automatic suggestion of formulations. They first use named entity recognition methods to extract a set of entity-typed declarations, then they transform it into linear program models. As one can see, NL4OPT only evaluates an AI model’s ability to establish mathematical models, while we contribute an end-to-end framework in this work. Optimus (AhmadiTeshnizi et al., ) and ComplexOR (Xiao et al., ) also make significant research in the field of operations research with LLMs. However, they provide a minimal test set, containing less than 70 test samples. Recently, MAMO (Huang et al., ) is proposed to benchmark mathematical modeling with code solvers. However, all these works merely focus on linear programming, ignoring the nonlinear problems that exist widely in practical applications. In addition, these benchmarks are simple in form, ignoring the tabular data that often occurs in industrial scenarios.
Additionally, we notice a work Tang et al. () explores synthesizing problems via a semi-automated process. This work is based on forward synthesis. Moreover, they did not focus on tabular data and nonlinear problems in real scenarios. In contrast, in this paper, we aim to benchmark practical optimization modeling with a high-quality manually checked test-bed OptiBench and also automatically synthesize more comprehensive optimization data including tabular data and code solutions resulting in ReSocratic-29k.
In this work, we contribute OptiBench, which is an end-to-end benchmark containing 605 multi-type data samples. OptiBench is a comprehensive benchmark that involves linear, non-linear, and tabular data, and the types of variables involved in the problems include continuous, integers (IP), and mixed integers (MIP).

Data Synthesis for Mathematical Reasoning.
Improving the performance of language models in mathematical reasoning tasks significantly depends on increasing the quantity of fine-tuning data for LLMs. A substantial body of work has been dedicated to these areas (Yu et al., ; Li et al., ; Yuan et al., ; Yue et al., ; Zhao et al., ; Lu et al., ; Liu & Yao, ). One notable approach is Rejection Sampling Fine-Tuning (RFT; Yuan et al. ), which employs supervised models to generate and collect correct reasoning paths, creating augmented fine-tuning datasets. MAmmoTH (Yue et al., ) utilizes RFT with GPT-4 to gather both Chain-of-Thought solutions in natural language and Program-of-Thought solutions in formal language. Similarly, MetaMath (Yu et al., ) focuses on data augmentation for both question and answer texts. MathGenie (Lu et al., ) collects a vast amount of data through open-source language models. While there has been significant progress in synthesizing data for informal mathematical reasoning, efforts have also been made to address formal reasoning through the use of formal languages and compilers (Xiong et al., ; Huang et al., ; Lu et al., ). However, a major challenge remains: there is a scarcity of high-quality data for optimization problems. This lack of data limits the direct application of these prior approaches to optimization contexts.

Socratic Method.
The Socratic method(^1^11[https://en.wikipedia.org/wiki/Socratic_method](https://en.wikipedia.org/wiki/Socratic_method)) is a critical thinking method with dialogic disassembled multi-step sub-questions and answers cultivating in answering a complex question.
This method has been applied by current language model techniques for advanced reasoning tasks, such as prompting step-wise reasoning (Qi et al., ; Chang, ; Shridhar et al., ),
multi-agent interaction (Zeng et al., ),
and discovering math knowledge (Dong et al., ).
For example, Qi et al. () proposes a divide-and-conquer style algorithm that mimics recursive thinking by asking Socratic questions, it thus relieves the reliance on the initial decision as chain-of-thought (CoT) and achieves performance improvements on several complex reasoning tasks.
Another line of work (Ang et al., ; Cobbe et al., ) applies the Socratic method for fine-grained dataset construction.
GSM8K Socratic dataset(^2^22[https://github.com/openai/grade-school-math?tab=readme-ov-file#socratic-dataset](https://github.com/openai/grade-school-math?tab=readme-ov-file#socratic-dataset)) (Cobbe et al., ) is the most related work to our paper.
They inject automatically generated “Socratic sub-questions” before each step, resulting in fine-grained math data.
To construct a step-by-step benchmark for optimization problem solving with intermediate solutions, in this work, we explore the Socratic method to synthesize optimization problems.
Unlike the previous study, we propose a reverse Socratic approach (ReSocratic) that generates optimization problems from the answer back to a question, and we demonstrate its superiority to traditional forward Socratic synthesis.

## 3 OptiBench : Benchmark for Optimization Modeling

The benchmark OptiBench is to evaluate the capability of large language models to solve end-to-end optimization problems.
Table 1 compares OptiBench and related optimization-problem benchmarks.
OptiBench covers a substantial number of optimization problems with a wider range of problem types.
Specifically, OptiBench features linear programming (linear), non-linear optimization problems (non-linear), and table content as in industrial use (Table),
resulting in a comprehensive and versatile benchmark for LLM optimization problem-solving.
OptiBench is an end-to-end benchmark, that takes natural language as input and numerical values of variables and objective as output.
We show the four types of questions in Figure 1.

Figure: Figure 2: Evaluation procedure of an OptiBench example. This example is about a mixed integer nonlinear optimization problem. The LLM is first required to write code to solve the question. Then, the LLM is required to extract the exact numbers according to the code execution output.
Refer to caption: x2.png

Data Collection and Annotation.
In the data annotation stage, we assign workers to collect questions from textbooks (Bertsimas & Tsitsiklis, ; Conforti et al., ; Wolsey, ), and a university’s course assignments and examinations. The workers are required to collect the questions first and save the text of the question in markdown format. If the question contains a table, convert the table to markdown format as well. If there is a standard answer to the question, it is also saved; if not, we let the worker complete the answer in natural language form.
We require our workers to write Python code, call the pyscipopt(^3^33[https://github.com/scipopt/PySCIPOpt](https://github.com/scipopt/PySCIPOpt)) solver to solve each problem, and ask them to output the values of the variables and optimization targets at the end of the code. These numerical solutions are recorded in the dataset as ground truth answers.
Additionally, for each collected sample, we require two additional annotators to participate in determining whether the annotation is correct and to correct any samples with incorrect annotations.

Figure 2 shows a mixed integer nonlinear programming sample of OptiBench. For each sample, we provide the “Question” and “Results”.
For every problem, we ask the workers to judge the uniqueness of the optimal solution. If the workers determine that the solution is unique, the “Results” will contain the description (colored in orange in Figure 2) and optimal value (colored in green in Figure 2) of all the variables and optimization objective; otherwise “Results” contains only the optimal value of the optimization objective.

AI4MATH Contest. A preliminary version of OptiBench is released in April 2024 as a contest track(^4^44[https://www.codabench.org/competitions/2438/](https://www.codabench.org/competitions/2438/)) of ICML 2024 Challenge on Automated Math Reasoning(^5^55[https://sites.google.com/view/ai4mathworkshopicml2024/challenges](https://sites.google.com/view/ai4mathworkshopicml2024/challenges)).

Data Statistics.
In Figure 4, we show the statistical results of four data formats (linear w/ table, linear w/o table, nonlinear w/ table, and nonlinear w/o table).
Overall, our OptiBench exhibits good diversity in terms of question types, number of variables, and text length.

Evaluation.
Unlike NL4OPT (Ramamonjison et al., ), which only measures the mathematical modeling ability of the language model, we also measure the solving ability of the language model to call code solver. In this paper, the evaluation approach we adopt is an end-to-end process where natural language text is the input and numerical form answers are the output.
Given an optimization problem $p$, the LLM is required to generate a solution including code $c=\rm{LLM}(p)$. Next, a Python interpreter is used to execute the code and obtain the code output $o=\rm{Python}(c)$. Then, we request the LLM to give the numerical form answer $a_{i}=\rm{LLM}([o,r_{i}])$ for each variable and objective in the problem, where $r_{i}$ is the natural language description of “Ground Truth Answer” in Figure 2. Finally, we compare the numerical form answer $a_{i}$ with the ground truth answer $a_{i}^{*}$ to calculate the accuracy.
A problem is considered solved if and only if all the variables and objectives are correctly matched.

## 4 ReSocratic: Data Synthesis with Reverse Socratic

In this section, we introduce ReSocratic, a novel data synthesis method for eliciting diverse and reliable data. The ReSocratic framework is shown in Figure 3(a). Former methods (forward synthesis) skipped the intermediate reasoning steps and directly generated the question, relying more on the intuition of LLMs. Whereas, the main idea of ReSocratic is to incrementally synthesize an optimization problem with step-by-step generation via the Socratic method in a reverse manner from our elaborately formatted seed demonstrations to questions.
Our ReSocratic consists of three steps: 1) Seed Demonstration Formalization, 2) New Demonstration Synthesis, and 3) Question and Code Translation.

Figure: Figure 3: (a) The forward data synthesis method is to synthesize the question first, and then let the LLM generate the answer to the synthetic question. (b) In contrast, ReSocratic first synthesizes carefully designed formatted demonstration and then transforms it into code and questions. (c) An example of a formatted demonstration.
Refer to caption: x3.png

1) Seed Demonstration Formalization. The seed demonstrations are rigorously selected by humans and each seed data is of diverse operations research scenarios. Figure 3(b) shows an example of seed demonstration.
It is formatted step by step, where each step is clearly delineated and builds upon the previous one. Each step consists of three parts:1) A header with “##” that introduces the specific aspect of the demonstration being addressed, such as “Defining Variables”, “Objective Function”, or “Constraints”. 2) A narrative description (colored in blue) in natural language that provides context and details about the element being introduced. This helps to understand the rationale and the requirements of that particular part of the optimization problem. 3) Mathematical formalization following “//” that translates the natural language description into a precise mathematical expression or constraint.

2) New Demonstration Synthesis.
The ReSocratic method prompts an LLM to generate new demonstrations based on the seeds.
We sample 2 seeds each time from the pool to form the synthesis prompt as shown in Appendix D.2. The LLM will follow the given prompt to generate new demonstrations step by step.
The generated data is rigorously selected via (1) each intermediate step should follow the format (as shown in the first step) by a rule filter and (2) the overall demonstration does not overlap with generated ones by a similarity filter. For all generated demonstrations, we set a similarity filter, which converts all texts into TF-IDF vectors and filters out demonstrations with cosine similarity higher than a threshold.
The procedure is shown in Figure 3(b).

3) Question and Code Translation.
We construct a code generation prompt to solve the mathematical formulations in the synthetic demonstrations and output the optimal solution results. If the code runs incorrectly, we delete this demonstration. Next, to acquire the questions in plain text format and the questions in table format, we construct two back-translation prompts. All the prompts are shown in Appendix D.
Then, for each generated demonstration starting from the third step, we translate it into a question-code pair, as shown in Figure 3(b). Each generated code will be executed automatically to ensure there are no bugs.

## 5 Experiments and Analysis

### 5.1 Baselines and Settings

Evaluation Setting.
The evaluation metric of our OptiBench is the answer accuracy, as detailed in Section 2. We show the solving accuracy of the four data types along with the code pass rate. We evaluate LLMs under three settings: Zero-shot, Few-shot, and Supervised Fine-Tuning (SFT) setting. We provide the zero-shot prompt and the few-shot prompt to solve the problem in Appendix D.1.

Baselines.
We select GPT-3.5-Turbo (Brown et al., ), GPT-4 (Achiam et al., ), Llama-2-7B-Chat (Touvron et al., ), Llama-3-8B-Instruct (Team, ) and Llama-3-70B-Instruct (Team, ) as the baselines of zero-shot and few-shot settings. For the SFT setting, we use Llama-2-7B-Chat and textbfLlama-3-8B-Instruct.

Setting of Data Synthesize. We use DeepSeek-V2 (DeepSeek-AI, ) to apply ReSocratic.
As an open-source large language model, DeepSeek-V2 (DeepSeek-AI, ) stands out due to its competitive performance to GPT-4, while concurrently offering a more cost-effective alternative. Furthermore, it exhibits a superior throughput, approximately 6 times greater, when contrasted against the existing 70b open-source model (DeepSeek-AI, ).
Utilizing the advanced capabilities of DeepSeek-V2, we contribute 29k synthetic data. This results in the ReSocratic-29k dataset. The threshold of the aforementioned similarity filter is set at 0.7,
we also set the temperature as 0.7, and sample 50 responses for each query.

Fine-tuning Setting.
For a given language model, we utilize our contributed ReSocratic-29k to conduct supervised fine-tuning. Specifically, we construct the training sample as follows:

We replace “{Question}” and “{Code}” with the synthetic question and the verified code in our ReSocratic-29k to form the training sample.
Based on this, we conduct fine-tuning experiments on two A800 GPUs, the epoch is set as 3, the learning rate is $2e^{-5}$, and the batch size is 128.

### 5.2 Data Statistics and Visualization

For both OptiBench and ReSocratic-29k, we show the statistical results of data distribution in question type, variable numbers, and question length. The question length refers to the number of characters in the question text. The results are shown in the following Figure 4. The distribution of variable numbers in both OptiBench and ReSocratic-29k generally conforms to the long-tail distribution. In addition, the distribution of question length in OptiBench also conforms to the long-tail distribution, while ReSocratic-29k is more balanced.

Figure: Figure 4: Statistical results of OptiBench and ReSocratic-29k
Refer to caption: x4.png

Furthermore, we show the visualization results of OptiBench and ReSocratic-29k using the t-SNE algorithm based on question semantic embedding. The visualization results of each type (linear w/o table, linear w/ table, nonlinear w/o table, nonlinear w/ table) are shown in Figure 5.

In general, from the statistical results and visualization results, our ReSocratic-29k has a good diversity in the question types, variable numbers, question length, and text semantics.

Figure: Figure 5: t-SNE visualization results of OptiBench and ReSocratic-29k. (‘$\triangle$’ indicates the data point of OptiBench and ‘$\bullet$’ indicates the data point of ReSocratic-29k)
Refer to caption: x5.png

### 5.3 Main Results

As shown in Table 2, GPT-4 has the strongest overall performance and achieves state-of-the-art performance on almost all kinds of data formats. The performance of the two open-source models, llama3-70b and deepseek-v2, is close to that of GPT-4 in the few-shot setup.
In addition, open source small models perform extremely poorly on OptiBench, with Llama-2-7B-Chat not getting a single question correctly solved and Llama-3-8B-Instruct getting only 13.6% accuracy on the few-shot setting.
From the perspective of data type, the nonlinear data of our OptiBench is more challenging than the linear data, and the data with table (w/ table) is more challenging than without table (w/o table).
Then, to show the validity of ReSocratic, we SFT Llama-2-7B-Chat, and Llama-3-8B-Instruct with our synthetic data ReSocratic-29k. We improved the performance of the Llama-2-7B-Chat from 0.0% to 30.6%, and the Llama-3-8B-Instruct from 13.6% to 51.1% (+37.5%), which is very close to the GPT-3.5-Turbo. In addition, Llama-3-8B-Instruct even exceeds GPT-4 in the data type of linear w/table, reaching state-of-the-art performance. We present a more detailed dataset performance analysis in Figure 6.

**Table 2: Main results on OptiBench. “Code Pass” refers to the success rate of code execution. Bold indicates the sota in the current setting, underline indicates the sota in the overall setting.**
| Model | Linear | Nonlinear | All | Code Pass |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| w/o Table | w/ Table | w/o Table | w/ Table |  |  |  |
| Zero-shot Prompt |  |  |  |  |  |  |
| Llama-3-8B-Instruct | 0.0% | 0.29% | 0.0% | 0.0% | 0.17% | 8.8% |
| Llama-3-70B-Instruct | 76.9% | 50.0% | 30.8% | 32.0% | 59.5% | 86.8% |
| DeepSeek-V2 | 40.4% | 27.5% | 29.3% | 18.0% | 34.4% | 74.0% |
| GPT-3.5-Turbo | 68.1% | 37.5% | 19.5% | 16.0% | 49.1% | 85.0% |
| GPT-4 | 75.4% | 62.5% | 42.1% | 32.0% | 62.8% | 88.8% |
| Few-shot Prompt |  |  |  |  |  |  |
| Llama-3-8B-Instruct | 17.8% | 2.5% | 11.3% | 8.0% | 13.6% | 26.9% |
| Llama-3-70B-Instruct | 79.2% | 57.5% | 33.8% | 32.0% | 62.5% | 91.2% |
| DeepSeek-V2 | 79.5% | 56.3% | 27.1% | 32.0% | 61.0% | 85.5% |
| GPT-3.5-Turbo | 75.4% | 40.0% | 28.6% | 26.0% | 56.4% | 93.2% |
| GPT-4 | 80.7% | 71.3% | 34.6% | 34.0% | 65.5% | 88.3% |
| SFT with Synthetic Data |  |  |  |  |  |  |
| Llama-2-7B-Chat | 40.6% | 11.3% | 15.8% | 32.0% | 30.6% | 93.7% |
| Llama-3-8B-Instruct | 63.5% | 32.5% | 33.0% | 44.0% | 51.1% | 96.3% |

### 5.4 Performance Analysis on Data Split

We show the detailed evaluation results under the few-shot setting for GPT-4 and GPT-3.5-Turbo in Figure 6. The performance on OptiBench is split by data type, variable numbers, and question length. According to the results, we can find that:

1) Tabular questions are harder. We find from the experimental results, that it is more difficult to solve table questions than no-table questions, and nonlinear questions are more difficult than linear questions.

2) Nonlinear problem is harder than linear problem. According to Table 2, for all language models, the performance of nonlinear questions is significantly lower than that of linear questions. In Figure 6 (a), we show the average performance of linear problems and nonlinear problems for GPT-4 and GPT-3.5-Turbo. The performance is 66.9% for linear problems and 30.8% for nonlinear problems.

3) In general, questions with more variables and longer text lengths are more difficult. We show the performance of GPT-4 and GPT-3.5-Turbo on the data with different numbers of variables in Figure 6 (b). From the linear trend line we provided, we can see that model performance decreases as the number of variables increases. As can be seen from Figure 6 (c), GPT-3.5-Turbo has a significant performance degradation on long text questions, while GPT-4 has a more balanced performance on different text lengths.

Figure: Figure 6: Performance analysis of GPT-4 and GPT-3.5-Turbo.
Refer to caption: x6.png

### 5.5 Ablation Study on ReSocratic

In Table 3, we present the performance outcomes of the models with and without the application of filters, specifically, the rule filter and similarity filter, which are illustrated in Figure 3(b). Additionally, we compare the performance when incorporating step questions versus when they are excluded.
We conduct an ablation study on Llama-2-7B-Chat and Llama-3-8B-Instruct. The experimental results show the validity of our filters, this conclusion is similar to RFT (Yuan et al., ) that redundant data can negatively affect language models. Moreover, the step questions generated in ReSocratic can bring improvement to the model’s solving ability.

**Table 3: Ablation study on synthetic data.**
| Model | SFT Data | Linear | Nonlinear | All |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| w/o Table | w/ Table | w/o Table | w/ Table |  |  |  |
| Llama-2-7B-Chat | ReSocratic w/o step questions | 38.3% | 10.0% | 15.0% | 32.0% | 28.9% |
| ReSocratic w/o filters | 40.1% | 11.3% | 14.3% | 30.0% | 29.6% |  |
| ReSocratic-29k | 40.6% | 11.3% | 15.8% | 32.0% | 30.6% |  |
| Llama-3-8B-Instruct | ReSocratic w/o step questions | 62.9% | 32.5% | 31.6% | 42.0% | 50.2% |
| ReSocratic w/o filters | 62.3% | 31.3% | 32.3% | 36.0% | 49.4% |  |
| ReSocratic-29k | 63.5% | 32.5% | 33.0% | 44.0% | 51.1% |  |

### 5.6 Comparison between Reverse Synthesis and Forward Synthesis

Furthermore, we compared the forward data synthesis approach with the reverse data synthesis approach (our ReSocratic method). WizardLM(Xu et al., ) is a typical forward data synthesis method, which first prompts the language model to generate questions similar to the seed data and then answer them. Using the same seed data, we sample 1000 responses from DeepSeek-v2 with wizardLM and ReSocratic respectively, and then fine-tune Llama-2-7B-Chat. The experimental results are shown in Table 4. The experimental results show that our ReSocratic synthesis method is superior to the forward synthesis method. Moreover, we sample 30 pieces of data generated by ReSocratic and WizardLM respectively, and manually identify the data accuracy, which also shows that the data generated by ReSocratic is more accurate.

**Table 4: Comparison between ReSocratic and other forward methods (Evol-Instruct and Self-Instruct).**
| Model | SFT Data | Linear | Nonlinear | All |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Data Acc | w/o Table | w/ Table | w/o Table | w/ Table |  |  |
| Llama-2-7B-Chat | Self-Instruct (1k responses) | 80.0% | 16.1% | 5.0% | 3.0% | 4.0% | 10.7% |
| Evol-Instruct (1k responses) | 76.7% | 15.5% | 7.5% | 3.8% | 6.0% | 11.1% |  |
| ReSocratic (1k responses) | 86.7% | 21.6% | 6.3% | r5.3% | 6.0% | 14.4% |  |

## 6 Conclusion

In this paper, we propose the OptiBench benchmark, which includes various types of data, to evaluate the ability of language models to solve mathematical optimization problems end-to-end. Furthermore, in order to alleviate the issue of data sparsity and mitigate the performance gap between large models and small open-source models, we introduce the ReSocratic method, a reverse data synthesis approach. The experimental results show that our ReSocratic method outperforms the forward data synthesis method. After fine-tuning with our synthetic data, ReSocratic-29k, the performance of Llama-2-7B-Chat and Llama-3-8B-Instruct has been significantly improved, demonstrating the effectiveness of our synthesis method. In the future, we plan to extend ReSocratic to other complex reasoning tasks such as math word problem-solving and evaluate more large language models on our proposed OptiBench benchmark.

## Appendix

## Appendix A More Comparisons with Other Benchmarks

NL4OPT is too easy for current LLMs. As we have mentioned in our paper, the NL4OPT benchmark is not an End-to-End evaluation benchmark. It conducts a two-stage process to evaluate the ability to construct mathematical models. In this work, we transform NL4OPT into our OptiBench data format and contribute NL4OPT-E. We then evaluate a few LLMs, the results are shown in the following Table 5. All the experiments are conducted under the zero-shot setting.
It is observed that LLMs solve NL4OPT almost completely even using the simplest prompt, but our OptiBench still poses a strong challenge.

**Table 5: Evaluation results comparison of NL4OPT-E and OptiBench.**
| Models | NL4OPT-E | OptiBench |
| --- | --- | --- |
| Deepseek-V2 | 89.3% | 34.4% |
| GPT-3.5-Turbo | 83.0% | 49.1% |
| GPT-4 | 93.1% | 62.8% |

‘Implicit’ and ‘Explicit’ data. Problems studied by Chain-of-Experts (Xiao et al., ) and OptiMUS (AhmadiTeshnizi et al., ) are orthogonal to ours. These related works (Xiao et al., ; AhmadiTeshnizi et al., ) examine the abstract modeling capabilities of LLMs for optimization problems. Specifically, the problems they focus on do not include explicit numerical values, primarily investigating the abstract modeling capabilities of LLMs in domain-specific scenarios. We denote this form of the problem as ‘Implicit’. In contrast, the problems we study include explicit numbers, and researching the concrete problem-solving capabilities of LLMs. We denote this form of the problem as ‘Explicit’. The form of the problems we study is closer to practical applications.
We present an ’Implicit’ sample and an ’Explicit’ sample in Figure 7.

Figure: Figure 7: Examples of ‘Implicit’ and ‘Explicit’ data. The ’Implicit’ sample is constructed by Xiao et al. () according to the original ‘Explicit’ example of NL4OPT.
Refer to caption: x7.png

In this work, we mainly focus on the ‘Explicit’ problems. We consider such a form to be more related to real work scenarios, such as the query question containing a numerical table. Therefore, Chain-of-Experts (Xiao et al., ) and OptiMUS (AhmadiTeshnizi et al., ) are orthogonal to our work.

## Appendix B More Detail of ReSocratic

We have already shown the process of our synthesis method in our paper. This section adds more detail to our ReSocratic.

### B.1 Seed Demonstrations

We collect 27 elaborate formatted demonstrations (13 linear scenarios and 14 nonlinear scenarios) in the seed pool. An example is shown in the following bellow.

We show some statistical results of our formatted demonstration pool in Figure 8.

Figure: Figure 8: Statistical results of our demonstration pool.
Refer to caption: x8.png

We have shown the distribution of the formatted demonstrations of synthetic data in Figure 4. The data distribution of our synthetic dataset ReSocratic-29k is similar to the data distribution in our formatted demonstration pool.

### B.2 Tabular Data Synthesize

To facilitate the synthesis of different types of data (linear and nonlinear), we produce different prompts, which are shown in Section D.2.1 and Section D.2.2.
In addition, in the back-translate stage, to get the question text with a table and the question text without a table, we construct two different back-translation prompts, as shown in Section D.2.3.

## Appendix C Benchmark and Dataset

### C.1 Data Format

##### Data Format of OptiBench .

We store E-OPT data in the form of JSON files. A sample of our OptiBench benchmark is shown below:

We construct samples in dictionary format, and all the data is stored as a list in a JSON file. Each sample has the following fields:

- •
“question”: The question text, presented in natural language, contains the background as well as the optimization objective and associated constraints. In order to solve the question, it is necessary to first find out the variables that can be optimized, then build a mathematical model, and then call a code solver to get the optimal numerical results of the variables and objective.
- •
“results”: This field is presented in the form of a dictionary, where the key is the natural language description of the variables and objectives, followed by their optimal values. During the annotation process, if the taggers cannot confirm that there is only one optimal solution to the problem, the results only contain the description of the optimization objective and its optimal value.
- •
“type”: This field records the type of the current sample, and there are four types: linear-table, linear-notable, non-linear-table, and nonlinear-notable.
- •
“index”: The index of the sample.

##### Data Format of ReSocratic-29k .

We show a sample of our ReSocratic-29k in the following bellow.

We construct samples in dictionary format, and all the data is stored as a list in a JSON file. Each sample has the following fields:

- •
“question”: The question text, presented in natural language, contains the background as well as the optimization objective and associated constraints. In order to solve the question, it is necessary to first find out the variables that can be optimized, then build a mathematical model, and then call code solver to get the optimal numerical results of the variables and objective.
- •
“code_solution”: The corresponding python code to solve the question.

## Appendix D All Prompts

We show all the prompts we used in this section.

### D.1 Prompts for Evaluation of OptiBench

#### D.1.1 Zero-Shot Prompt

“system”:

“user”:

#### D.1.2 Few-Shot Prompt

“system”:

“user”:

#### D.1.3 Results Extraction Prompt

### D.2 Prompts of ReSocratic

#### D.2.1 Linear Demonstration Generation

“system”:

“user”:

#### D.2.2 Nonlinear Demonstration Generation

“system”:

“user”:

#### D.2.3 Question Generation

“system”:

Generating questions without table.

“user”:

Generating questions with table.

“user”:

#### D.2.4 Code Generation

“system”:

“user”:

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: How does the current perform if using the pass@k metric, which measures the performance of the generated code after k attempts? 

The experimental results show that DeepSeek-V2 severely underperforms in the zero-shot setting compared to its few-shot version, while other models do not exhibit such a drastic difference. Is there any reason behind this?
soundness: 2
strengths: OPTIBENCH is a comprehensive benchmark that effectively evaluates the optimization problem-solving abilities. It inlcudes nonlinear programming problems, along with tabular data, reflecting realistic scenarios. It also makes the benchmark challenging enough.
By requiring LLMs to understand the problem, perform sound reasoning, and generate code to invoke a solver, OPTIBENCH provides a holistic assessment of LLMs' reasoning and coding skills. 
This general evaluation approach is valuable for measuring the models' ability to apply their knowledge to practical scenarios and can guide future research on LLMs' capabilities in reasoning tasks. Moreover, the data annotation with strong human guidance looks reliable.

The benchmark evaluates the correctness of LLMs' solutions by comparing the execution results with ground truth answers. This objective evaluation method is more reliable than pure text-based benchmarks in my perspective.

The authors conduct an ablation study that provides valuable insights into the effectiveness of their benchmark and proposed data synthesis method. This includes examining the impact of removing reasoning steps and filters, as well as comparing forward and reverse data synthesis approaches, along with the visualization of the dataset.

This work fills a gap in existing benchmarks by providing a diverse and challenging dataset for end-to-end optimization problem-solving. Moreover, the proposed ReSocratic method and the resulting RESOCRATIC-29K dataset offer a promising approach to address the data scarcity issue.
summary: This paper proposes OPTIBENCH, an end-to-end benchmark for evaluating large language models (LLMs) on optimization modeling. OPTIBENCH covers various problem types, including linear and nonlinear programming with/without tabular data. To further alleviate data scarcity, the authors introduce ReSocratic, a `reverse` data synthesis method that generates the RESOCRATIC-29K dataset. The authors demonstrated that fine-tuning open-source LLMs (Llama2 7b and Llama3 8B) on RESOCRATIC-29K significantly improves their performance on OPTIBENCH. The paper contributes a comprehensive benchmark, a reverse data synthesis method, and a synthetic dataset to advance evaluations on optimization modeling with LLMs.
weaknesses: The paper lacks a fine-grained error analysis, which could provide valuable insights into the specific challenges faced by LLMs in optimization tasks. A breakdown of error types, such as errors in understanding the problem, formulating the optimization model, or transferring the mathematical model to code, or summarizing the execution into required formatted outputs, would be helpful to guide future improvements in LLM design and training.

The authors didn't provide an analysis (maybe just a few case studies) of false positives and false negatives in the benchmark results. As with any benchmark, it is expected that both types of errors can exist, and understanding their characteristics could contribute to the benchmark's reliability.

The data collection and annotation process lacks details on deduplication and quality control measures. This helps to ensure the collected optimization problems were diverse enough to measure LLM's modeling capability. 

The paper didn't explore the impact of different prompting strategies during inference. For example, investigating whether step-by-step reasoning then generates the code can help improve model performance.

The range of models tested is relatively limited. As a benchmark paper, one would expect a wider variety of LLMs to be included, such as code-specialized models like CodeLLaMA or the Mistral line of models. 

The reliability of the back-translation step in the ReSocratic data synthesis is not thoroughly investigated. This helps measure the quality and consistency of the questions generated through back-translation, as well as the potential impact of any errors.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Could you please provide some results on LLMs handling the large-size problem instances?
- Could you please verify the accuracy of the formulation given by the LLMs?
- Would it be possible to compare the solving efficiency of the codes produced by the LLMs?
soundness: 3
strengths: - This paper proposes a benchmark containing various problems, including linear and non-linear programming with or without tabular data, which can better evaluate the ability of LLMs.
- The reverse data synthesis approach is novel and reasonable.
- The experimental results show that ReSocratic outperforms the forward data synthesis method, and the fine-tuning results are promising.
summary: This paper proposes a benchmark for end-to-end optimization problem-solving and a data synthesis method. The benchmark contains various optimization problems, including linear and non-linear programming with or without tabular data. With the constructed benchmark, the fine-tuned LLMs exhibit a stronger performance.
weaknesses: - The authors may want to generate instances with more constraints and variables, as few instances in the paper have more than 7 variables. Thus, this raises my concern about LLMs' ability to model problems with large instance sizes.
- Given that a single optimization problem can have multiple valid formulations, it would be beneficial for the authors to verify the accuracy and equivalence of these formulations with ground-truth ones.
- There are questions regarding the solving efficiency of the generated codes. It would be valuable to assess whether the code produced by LLMs can outperform human-designed formulations and codes.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: My major questions are around the data curation process.
# Data Labeling Process Questions

1. Scale of Labeling Team
- Total number of data labelers involved

2. Labeler Qualifications
- Required experience
- Necessary skills
- Background requirements

3. Source Data Details
- Data collection methods
- Criteria for benchmark curation
- Original data sources
soundness: 3
strengths: Originality: This work introduces the OPTIBENCH benchmark, a comprehensive tool designed to rigorously evaluate large language models (LLMs) on complex optimization problems, significantly advancing beyond existing benchmarks like MAMO and NLP4LP. Unlike prior efforts that primarily focus on simplistic, linear problems or abstract formulations, OPTIBENCH incorporates nonlinear and tabular data, simulating real-world scenarios more accurately. Additionally, the introduction of the ReSocratic data synthesis method is an inventive departure from traditional forward synthesis techniques, employing a reverse step-wise approach that yields a higher quality and diversity of synthetic optimization problems. While similar methods have been applied in other contexts, the adaptation and customization for optimization modeling distinguish this work meaningfully.

Quality: The paper is grounded in rigorous experimental design and analysis. The authors perform extensive evaluations using a variety of open-source and proprietary models, including detailed comparisons in zero-shot, few-shot, and fine-tuning settings. The results are compelling, showcasing substantial performance improvements, particularly when using ReSocratic-generated data, as demonstrated by a 30.6% accuracy improvement for Llama-2-7B-Chat and a 37.5% boost for Llama-3-8B-Instruct. The ablation studies and the analysis of data distribution and variable types further bolster the credibility of the experimental findings, highlighting the robustness and effectiveness of the proposed benchmark and synthesis method.

Clarity: The paper is well-structured and communicates its contributions clearly. The presentation of the OPTIBENCH benchmark, complete with illustrative examples of linear and nonlinear problems, aids in understanding its scope and novelty. The ReSocratic method is also carefully detailed, with step-by-step descriptions and visual aids that effectively convey the synthesis process. Moreover, the inclusion of a thorough comparison with existing benchmarks and methods provides a clear context for the advancements made.

Significance: The significance of this work lies in its practical applicability and potential impact on the field of optimization problem-solving with LLMs. By addressing the limitations of previous benchmarks and providing a more realistic and challenging testbed, OPTIBENCH has the potential to become a standard for evaluating and advancing LLMs in applied mathematics and operations research. Furthermore, the ReSocratic approach offers a scalable method to generate high-quality data, which could influence future research in data synthesis for complex reasoning tasks. The reported performance improvements also suggest practical benefits for enhancing open-source models, bridging the performance gap with proprietary models like GPT-4.
summary: This paper proposes a study on the challenge of evaluating and improving large language models with respect to solving optimization problems. While the capabilities of large language models have recently improved with respect to mathematical and logical reasoning, the few existing benchmarks are limited to very simple tasks in linear programming and include little complexity representative of real-world applications. The authors present OPTIBENCH-a benchmark that attempts to evaluate the end-to-end ability of LLMs to solve optimization problems. It includes 605 diverse problems in the areas of linear and nonlinear programming, both tabular and nontabular; models are expected to generate code to execute and produce numerical answers.

Motivation: The authors mention that all existing optimization benchmarks are small-scaled and overly simplistic, where most of their problems are linear. Besides, open-sourced LLMs are substantially outperformed by more complex closed-source models like GPT-4. This demonstrates that a more challenging benchmark is needed along with efficient data augmentation methods to make smaller, open-source models perform better.
Main Contributions: OPTIBENCH Benchmark: The authors propose a new benchmark that includes various optimization problems, intended to test the LLM's capabilities in handling optimization tasks of varying complexity. The benchmark problems come in natural language formulation, which requires the generation of Python code that can be executed.
ReSocratic Data Synthesis Method: Based on this, the paper proposes ReSocratic-an approach for attempting to synthesize synthetic data in optimization. Unlike other methods, which first generate questions, the proposed ReSocratic works backward, constructing incremental step-by-step optimization demonstrations, and back-translates those into questions afterwards. This process aims to produce varied and consistent data. In this way, a dataset of questions, called RESOCRATIC-29K, is created.
Performance Improvement: This work presents data suggesting that fine-tuning open-source models over RESOCRATIC-29K might help narrow some of the gap with proprietary models. It reports accuracy improvements from 0.0% to 30.6% for Llama-2-7B-Chat and from 13.6% to 51.1% for Llama-3-8B-Instruct.
Evaluation and Analysis: The authors carried out extensive experiments in which the various models compared against each other in zero-shot, few-shot, and supervised fine-tuning settings. They mentioned nonlinear data and tabular data to be really challenging; increasing problem difficulty considerably raises the challenge, and their experiments indicate the ReSocratic method performs better than the straightforward forward synthesis techniques. Ablation studies further support the influence of key factors such as mechanisms for filtering and step-by-step synthesis.
The paper presents arguments for the potential utility of OPTIBENCH, and examines the effects of ReSocratic on open-source LLMs' optimization capabilities.
weaknesses: ## Benchmark Contribution:
The present paper introduces OPTIBENCH for the evaluation of large language models in respect to solving optimization problems. However, such benchmarks already exist, for example MAMO, ComplexOR, and NLP4LP, which test LLMs concerning their ability to interpret natural language descriptions of optimization problems and create corresponding mathematical models. The extension in OPTIBENCH relates to nonlinear elements and tabular data. This feels like an incremental extension rather than a transformative one. The paper should better highlight the aspect of how OPTIBENCH goes significantly beyond the established benchmarks.
## Data Generation Process: 
It proposes the data synthesis methodology, namely ReSocratic, and claims novelty in generating high-quality synthetic data. Earlier research has addressed reverse synthesis methodologies or back-translational techniques, especially for data augmentation and machine translation. Following are some related works which prove similarity:
"Data Augmentation for Code Generation Tasks" shows back-translation-style data augmentation that aims at improving code generation, just like ReSocratic. These techniques create synthetic variations from existing data. There are a number of similarities with this paper which raises the suspicion that the contribution of ReSocratic is more of an extension of the currently known methods and not genuinely new.
MathGenie: Synthetic Data Generation through Question Back-translation to Improve LLMs' Mathematics Reasoning, 2024: MathGenie synthesizes questions from mathematical solutions through a backward process, just like ReSocratic. Be that as it may, both methods are distinguished by their enhancing of reasoning through backward data synthesis, an issue that raises a question as to the novelty of ReSocratic in the given context.
"Data Augmentation for Code Translation with Comparable Corpora and Back-Translation" (2023): This work describes the generation of diverse code examples for translation tasks using back-translation, which aligns with ReSocratic's structured data synthesis approach. In fact, this would mean there is some correlation in the methodological procedure between the current work and existing strategies.
These related works demonstrate that such techniques as back-translation and reverse synthesis have already been successfully applied for code generation and mathematical reasoning. This paper could better explain how ReSocratic is different or improves these established methods. Moreover, discussing the possible biases of the reverse synthesis process would help to enhance the reliability and applicability of the generated data.
## Conclusion:
In general, this paper contains some interesting ideas; however, the contributions of both OPTIBENCH and the ReSocratic data generation method appear relatively modest with respect to prior work. It also shares similarities with approaches that have previously been proposed, indicating potential for better establishing the unique footprint and benefits introduced by the methods. An extended comparative study and some empirical evidence could be more enriching in showing points of strength for the proposed contributions.

Meta-review: The paper presents a benchmark for evaluating LLMs on optimization problems, along with ReSocratic, a data synthesis method. The work includes a comprehensive benchmark dataset and demonstrates improvements in open-source LLM performance through fine-tuning. Overall, the work represents a solid contribution to the field, with clear practical value and well-documented methodology, despite some limitations in scope and novelty.

Decision: Accept (Poster)