Technical Report for ICML 2024 Automated Math Reasoning Challenge: Solving Optimization Problems with Open Source Large Language Model

Authors: duc m nguyen
sungahn  ko
l  kaiser
j w kim
c  kim
a  krueger
m  lin
s  lin
p  lue
d  m'ely
o ' keefe
Venue: ICML_cc_2024_Workshop_AI4MATH
Published: 2024-06-01 13:12:19

Abstract: This technical report presents an approach utilizing open-source Large Language Models for Automated Optimization Problem-solving With Code Challenge at the ICML 2024 AI4Math Workshop. This challenge emphasizes the ability of Large Language Models (LLMs) to handle complex mathematical reasoning from formulating to solving the problem at hand. By exploring different prompting techniques, such as few-shot, self-consistency, chain-of-thought, and tree-of-thought, we aim to explore the current state-of-the-art LLMs' mathematical reasoning abilities.

Main paper: 
 introduction: 
 Recent research has highlighted the remarkable potential of state-of-the-art Large Language Models like GPT-4 (Achiam et al., 2023) showcasing their promising abilities in reasoning across diverse fields, encompassing tasks such as solving mathematical word problems and proving theorems (Huang et al., 2024) . Automated mathematical reasoning, which requires sophisticated multi-step planning and reasoning, has attracted active research to evaluate and develop intelligent agents capable of obtaining advanced forms of human intelligence such as mathematical reasoning.In this technical report, we investigate the ability to formulate and solve optimization problems, which is critical across various domains, ranging from operations research and engineering to finance and machine learning, by Open Source Large Language Models. Traditionally, solving optimization problems has required human expertise in mathematical modeling and algorithm design. However, the rise of LLMs presents an opportunity to automate this process, enabling machines to understand, interpret, and solve optimization problems expressed in natural language. To tackle this problem, we investigate current Large Language Models' capability to solve optimization problems by augmenting them with various prompt engineering frameworks. Our contributions can be summarized as follows:• We formulate the automated optimization problemsolving as a prompt engineering problem, leveraging the capabilities of LLMs to understand and model optimization problems expressed in natural language.• We propose a symbolic approach to repair incorrect problem formulation automatically, improving the accuracy and reliability of the models.• We provide a comparative analysis of state-of-theart prompting techniques such as few-shot, selfconsistency, chain-of-thought, and tree-of-thought, offering insights into their strengths and limitations in automated optimization problem-solving. 
large language models: 
 In the past few years, the boom of deep learning in Natural Language Processing and the birth of the Transformers (Vaswani et al., 2017) architecture dictates the trend in how language models are designed. Bigger and deeper models are preferred since they are capable of producing human-like text and achieving state-of-the-art performance on most of the benchmarks. In 2023, Achiam et al. (2023) transformed the world of NLP research with the release of GPT-4 with unprecedented state-of-the-art overperformance on most benchmarks. Following their success, Anthropic AI released a competing series of models, namely Claude 1, 2, and 3, with similar performance (Anthropic, 2023) . However, they were closed-source and people had to pay for API access.To provide an open-source alternative, numerous researchers have attempted to train their own LLM and published their findings. One of the earliest open-source LLM, LLaMA (Touvron et al., 2023a) , was released by Meta. It inspired others to develop better models based on the LLaMA architecture. BigScience Workshop (2022) introduced BLOOM, an open-source LLM with similar sizes and performance to GPT-3. Shliazhko et al. (2022) reproduced the GPT-3 architecture using GPT-2 sources and introduced mGPT with multilingual capability. Touvron et al. (2023b) improved the previous LLaMA by introducing LLaMA 2 with a focus on instruction fine-tuning. The model approached GPT-3's performance but ten times smaller and faster. Deepseek AI introduced DeepSeek-LLM (Bi et al., 2024) , achieving superior performance compared to GPT-3.5, and its programming variant DeepSeek-Coder (Guo et al., 2024) , n a are the length of the system, user, assistant respectively, and s i , u i , a i ∈ V. The historical conversation is a list of these sequences, denoted as P = S, U (1) , A (1) , U (2) , A (2) , U (3) , . . . , where P 1 is the system prompt and P i will alternate between U and A for i > 1.Definition 3.5. Reasoning Path. Let R = (r 1 , r 2 , . . . , r nr ) be an ordered set representing the sequence of intermediate reasoning steps, where n r is the length of the reasoning path and r i ∈ V.Definition 3.6. Reasoning Path Evaluation. Let's define a function E : V nr×1 → R to be an evaluator of a reasoning path R that gives a score representing how good the reasoning is.Code data have been demonstrated to significantly enhance the reasoning capabilities of large language models (LLMs) by providing structured and logical examples that improve their problem-solving skills (Ma et al., 2024) . Given this advantage, we restrict the output sequence in Definition 3.2 to be Python source code in this technical report. This restriction allows us to leverage the benefits of code data, ensuring that the generated solutions are executable and verifiable. By focusing on Python, we can take advantage of its simplicity and the extensive range of libraries available for numerical optimization and algorithm development, facilitating efficient implementation and testing of our automated reasoning framework. Examples of the prompt template can be found in Appendix A. 
automated optimization problem solving: 
 In this section, we discuss the main approaches to tackle this problem.Few-shot Prompting Few-shot prompting involves providing the model with a small number of examples within the prompt to guide it toward the correct solution. Formally, let D train = {(P i , C i )} k i=1 be a set of k training examples, where each P i is an input sequence representing an optimization problem statement, and C i is the corresponding output sequence representing the solution.Given a new optimization problem P new and a System instruction, we construct the prompt P based on Definition 3.4 as follows:EQUATIONThe model M then generates the solution Ĉnew for the new problem P new based on the constructed prompt P:Ĉnew = M(P)Incorporating few-shot prompting allows the model to leverage examples to infer patterns and structures that are useful for solving the new problem, effectively reducing the need for extensive fine-tuning and enabling the model to generalize from minimal context.Self-consistency Prompting Self-consistency prompting builds on the concept of few-shot prompting by introducing multiple reasoning paths and selecting the most consistent one to enhance the reliability of the model's outputs. Formally, given the prompt P constructed as in Equation 1, the model generates multiple candidate solutions { Ĉ(j) new } M j=1 , where M is the number of generated solutions.new is evaluated for consistency. The final solution Ĉfinal new is selected based on a majority voting mechanism:Ĉfinal new = mode { Ĉ(j) new } M j=1This approach improves the robustness of the generated solutions by considering multiple possibilities and converging on the most consistent answer, thereby enhancing the accuracy and reliability of the model in solving optimization problems. EQUATIONThe model M then generates the intermediate steps Rnew and the final solution Ĉnew as follows:ÂCoTnew = ( Rnew , Ĉnew ) = M(P CoT )We can combine COT with self-consistency by letting the model M generate multiple candidates { Â(j) CoTnew } M j=1 , then selecting the final solution ÂfinalCoTnew with majority votingÂfinal CoTnew = mode { Â(j) CoTnew } M j=1By including these intermediate reasoning steps, CoT prompting helps the model to decompose the problem into smaller, more manageable parts, thereby improving its ability to generate accurate and coherent solutions. This method not only enhances the model's performance on complex tasks but also provides transparency into the reasoning process, making it easier to understand and verify the generated solutions.Tree-of-thought Prompting Tree-of-thought (ToT) prompting extends the concept of chain-of-thought (CoT) prompting by exploring multiple reasoning paths in a treelike structure. This approach allows the model to consider various possible solutions and converge on the most optimal one, further enhancing its problem-solving capabilities. The original ToT algorithm integrates Depth-First-Search (DFS), and Breadth-First-Search (BFS) into the framework to search for the best reasoning path. Given an optimization problem P new , we construct the tree-of-thought prompt P ToT similarly to Equation 2 as followsEQUATIONIn this technical report, we investigate the Beam Search strategy to enhance the ToT framework. Beam Search is a heuristic search algorithm that explores a graph by expanding the most promising nodes in a limited set, known as the beam width, at each level. To obtain promising candidates, we utilize a neural evaluator E defined in Definition 3.6.The evaluator E is composed of the same LLM used for generating solutions, and a detailed instruction to instruct the model to grade the candidate solutions. This strategy balances the exploration and exploitation of the search space, maintaining a fixed number of best candidates while pruning less promising paths (Lowerre, 1976) . The algorithm is summarized in Algorithm 1.Let b 1 and b 2 be the beam width, representing the number of top candidates retained at each level of the tree, and the number of samples to generate for each retained candidate respectively. The Beam Search of Thought (BSOT) is as follows.1. Start with the root node representing the prompt "Let's think step by step" to initiate a chain-of-thought answer. This prompt is a common practice prior works use to initiate the chain-of-thought mechanism (Kojima et al., 2022) .2. The model M then generates b 2 intermediate reasoning steps {R i,1 , R i,2 , . . . , R i,b2 } using the model M for each candidate at the current level.3. For each generated reasoning step, concatenate with the previous path A ′ ToT and obtain evaluations from the evaluator E. TOTnew is selected based on a majority voting mechanism:Âfinal TOTnew = mode { Â(j) TOTnew } b1b2 j=1By incorporating Beam Search into the ToT framework, we aim to improve the efficiency and effectiveness of the model's reasoning process. This method allows the model to navigate the solution space more intelligently, retaining the most promising paths while discarding less relevant ones.Ensemble Strategy It has been shown that combining the outputs of multiple models can significantly improve performance and robustness. This approach, known as the ensemble strategy, leverages the strengths of different models to produce more accurate and reliable results (Sagi & Rokach, 2018) . Formally, let { Fi } N i=1 be a set of N diverse estimator of the function F, each trained or configured differently. Given an optimization problem P new , each model Generate candidate reasoning steps Ri,j Ri,j = M(T ), 1 ≤ j ≤ b2for each candidate Ri,j do 10:Append Ri,j to path T to form new path T ′ 11:Add T ′ to Bi+1 Ĉ(i) new = Fi (P new )The ensemble strategy combines these candidate solutions to produce the final solution Ĉfinal new . One common method is to use a voting mechanism where the majority vote among the candidate solutions determines the final solution:Ĉfinal new = mode { Ĉ(i) new } N i=1In the context of automated optimization problem-solving, the ensemble strategy can be particularly effective when combined with other prompting techniques discussed earlier, such as few-shot prompting, self-consistency prompting, chain-of-thought prompting, and tree-of-thought prompting. By integrating the outputs from multiple models employing these techniques, we can achieve better performance and reliability, further advancing the capabilities of large language models in solving complex problems.In many cases, the initial output generated by the model may contain errors or suboptimal solutions. To address this, we employ two symbolic approaches using SymPy (Meurer et al., 2017) to repair and refine the generated solutions, ensuring better accuracy. Fixing Incorrect Conditions In optimization problems, modeling constraints with mathematical inequalities is crucial. However, many Linear Programming Solvers only support weak inequalities (i.e., ≤ or ≥). This leads to issues when the problem statements include terms such as "less than" or "more than", causing the model to produce strict inequalities (i.e., < or >), resulting in Runtime Error exception. To mitigate this problem, we symbolically transform strict inequalities of two types E 1 > E 2 and E 1 < E 2 into the weak version E 1 ≥ E 2 + 1 and E 1 ≤ E 2 -1, assuming that both E 1 and E 2 are all integers.Fixing Incorrect Division Expressions Optimization problems often require modeling ratios involving decision variables, making constraints or objective functions nonlinear. Non-linear expressions can pose challenges, as many optimization solvers, particularly linear programming solvers, cannot handle them directly. To correct the issue, we symbolically remove all division operators involving decision variables by deriving equivalent transformations.• Fixing objective function: If it contains division between a decision variable and a constant, i.e., v c , where v is the variable and c is the constant, we transform it into 1 c v.• Fixing constraint inequality: If an inequality contains a division operator, we transform the inequality using Algorithm 2 to remove all divisions. The algorithm extracts two sides of the inequalities, finds the common multiple D, and multiplies D on both sides. This approach removes all divisions on both sides of the inequality while keeping it equivalent to the original one.{ "id": <unique integer>, "question": "<Problem Statement>", "code": "<source code>" "results": { "<Output 1>": "<empty or number>", "<Output 2>": "<empty or number>", "<Output 3>": "<empty or number>", "<Output 4>": "<empty or number>", ... } }, To evaluate our approaches, we use the newly published data in the Automated Optimization Problem-solving With Code Challenge at ICML 2024 AI4Math Workshop (Huang et al.) . The data set consists of two partitions: a Train set and a Test set. The training set and the testing set contain 1025 and 421 Linear Programming problems respectively with various difficulties. Some of the data points in the training set were synthesized by GPT-3.5 and GPT-4, which would introduce some degree of noise to the training process. However, the organizer assures that all data points in the testing set are valid.The training data is stored in JSON format with the fields: id, question, code, and results representing the id, optimization problem statement, the sample solution, and the required outputs. The testing set is also stored in JSON format with the fields: id, question, results represent the id, optimization problem statement, and the required outputs respectively. The data format can be found in Figure 1 . The task was to fill in the required blanks in the results field of the testing data, and the model's performance was evaluated with answer accuracy. 
implementation details: 
 To create few-shot examples, we sample the optimization problems and their corresponding solution randomly and uniformly from the training data set. To create intermediate reasoning steps, we instruct Llama-3-8B-Instruct to produce reasoning steps before generating the final Python code. The final intermediate reasoning steps were manually analyzed and selected after some iterations. All of the experiments were implemented in Python using the VLLM framework (Kwon et al., 2023) . The models' inferences were able to run on a single RTX 3090 GPU, apart from TECHNIQUE ACCURACYLLaMA-3-8B-Instruct + 5-SHOT 57.48% LLaMA-3-8B-Instruct + SELF-CONSISTENCY k = 15) 60.81% LLaMA-3-8B-Instruct + CHAIN OF THOUGHTS k = 15) 78.15% LLaMA-3-8B-Instruct + TREE OF THOUGHTS b1 = 3, b2 = 5) 62.00% DeepSeekCoder-7B-Instruct + CHAIN OF THOUGHTS k = 15) 71.50% DeepSeekCoder-7B-Instruct + TREE OF THOUGHTS b1 = 3, b2 = 5) 61.52% DeepSeekCoder-33B-Instruct + TREE OF THOUGHTS (5-SHOT, b1 = 1, b2 = 5) 62.95% ENSEMBLE STRATEGY 80.52% DeepSeekCoder-33B-Instruct, which we need 8 GPUs to run inference on. The full implementation can be found at https://github.com/kurone02/AutoLP.We have conducted experiments demonstrating the performance of different prompting techniques using open-source large language models. The accuracy of each technique is presented in Table 4 .1. The baseline performance using 5shot prompting with LLaMA-3-8B-Instruct achieves an accuracy of 57.48%, showcasing that even with minimal context, a generic LLM like LLaMA-3 is able to perform complex mathematical reasoning. Introducing the self-consistency mechanism shows a modest increase in accuracy to 60.81%. Incorporating a Chain of Thoughts with self-consistency significantly boosts the model's accuracy by nearly 18%, proving the importance of intermediate reasoning steps. Surprisingly, the Tree of Thoughts method cannot outperform COT, falling short by a wide margin. Our hypothesis for the underperformance of TOT is that the current state of Large Language Models could not self-verify such complex reasoning path (Kambhampati et al., 2024) , thus, the evaluator E is not a reliable heuristic.We also experimented with another LLM that is specifically fine-tuned for code-related tasks, the DeepSeekCoder series (Guo et al., 2024) . Due to a smaller context size, the 7B parameters variant can only process 2 examples. Nevertheless, the model achieves good performance at 71.5% with the COT technique and comparable results to LLaMA-3 in using TOT. The performance in TOT can be explained by better mathematical ability from DeepSeekCoder, making it comparable to LLaMA-3 even when provided with fewer examples. We also conducted experiment with DeepSeekCoder-33B-Instruct, however, due to limited resources, we cannot fully test other prompting techniques other than TOT, which boosts the performance of TOT by a small margin.Finally, we assemble the results obtained from various methods and models to produce the final answers, achieving 81% accuracy, ranked 10 th place on the private leaderboard. 
future work: 
 Even though open-source LLMs demonstrate good experimental results in the challenge, there are important areas needing further investigation.• Improvement of Evaluation Mechanisms. The underperformance of ToT prompting suggests a need for better evaluation mechanisms within LLMs. Future work can focus on improving the self-evaluation capability of LLMs by training with synthetic data similar to Chen et al. (2024) or utilizing RAG-based mechanism (Wei et al., 2022b) .• Fine-tuning on the training data set. The performance can be further enhanced by fine-tuning an LLM or a small model on the competition training data set, which we have omitted in this technical report due to the lack of resources.• Incorporate more difficult questions in the data set.Most of the questions in the data set are Linear Programming problems, lacking diversity in difficulties. Future work could focus on building a more diverse data set consisting of other types of problems in optimization such as Convex Optimization, Dynamic Programming, or Stochastic Optimal Control. 
conclusion: 
 This technical report has investigated the capabilities of open-source Large Language Models (LLMs) in formulating and solving optimization problems through various prompting techniques. The exploration of methods such as few-shot prompting, self-consistency prompting, chain-ofthought (CoT) prompting, and tree-of-thought (ToT) prompting has provided valuable insights into how LLMs can be effectively utilized for complex mathematical reasoning tasks.Requirements:• The number of LED bulbs produced on machine 1: ?• The number of incandescent bulbs produced on machine 2: ?• The number of LED bulbs produced on machine 3: ?• The total cost of production: ?Assistant: Let's think step by step.This problem can be formulated as a linear programming problem to minimize the total cost of producing 5000 LED bulbs and 6000 incandescent bulbs per week. Subject to the following constraints:1. The total number of LED bulbs produced must be at least 5000:num_led_on_machine1 + num_led_on_machine3 >= 50002. The total number of incandescent bulbs produced must be at least 6000: num_inc_on_machine2 >= 6000 # Output the answer 27 print("## start solving") 28 print("The number of LED bulbs produced on machine 1:", value(num_led_on_machine1)) 29 print("The number of incandescent bulbs produced on machine 2:", value( num_inc_on_machine2)) 30 print("The number of LED bulbs produced on machine 3:", value(num_led_on_machine3)) 31 print("The total cost of production:", value(objective)) 32 print("## end solving") A food truck wants to make bean burritos and beef burritos using 5000 grams of lettuce. The profit per bean burrito is $6.5 and the profit per beef burrito is $9. The bean burrito contains 25 grams of lettuce and the beef burrito contains 18 grams of lettuce. The beef burrito is much more popular and at least four times the amount of beef burritos need to be made than the bean burritos. However, a minimum of 5 bean burritos need to be made. How many of each burrito should be made in order to maximize profit? Requirements:• The number of bean burritos: ?• The number of beef burritos?: ?• Total profit: ? We want to maximize the total profit, which is the sum of the profit from bean and beef burritos: Maximize: 6 * num_bean_burritos + 9 * num_beef_burritos 
 
 Appendix:  


Decision: **Paper Summary:**

The paper explores the use of two pre-trained large language models (LLaMA-3-8B-Instruct and DeepSeekCoder-7B-Instruct) to solve linear programs by leveraging several advanced prompt augmentation techniques, including few-shot, self-consistency, chain-of-thought (CoT), and tree-of-thought (ToT). The authors provide a comprehensive comparison of these techniques in terms of their accuracy and performance.



**Review Summary:**

The paper presents a valuable contribution to the field of automated mathematical reasoning by demonstrating the application of LLMs and advanced prompt engineering techniques to solve linear programs. Despite the need for minor revisions and deeper analysis in certain areas, the paper's strengths in clarity, organization, and methodological rigor make it a worthy addition to the literature. With the suggested improvements, the paper could significantly impact the understanding and application of prompt engineering in mathematical problem-solving.

**Reasons to Accept:**

1. The paper is clear, well-written, and well-organized, making it accessible and easy to follow. The methodologies and results are presented in a structured manner, contributing to the overall readability of the manuscript.
2. The authors provide a thorough qualitative review of past work, clearly defining and explaining the different prompting methods considered. They also detail the modifications and experiments conducted, enhancing the reader's understanding of the study.
3. Sections 3.2 and 3.3 are particularly strong. Section 3.2 provides an in-depth analysis of CoT and the ensemble strategy, while Section 3.3 is well-motivated with a detailed examination of how to handle inequalities and divisions.
4. Missing code is effectively replaced with algorithmic explanations, such as in Algorithm 1, providing clarity and maintaining the technical rigor of the paper.

**Reasons to Reject:**

1.  A few minor changes and corrections could enhance the manuscript's clarity and readability. These include outlining the authors' contributions more explicitly in the introduction, correcting typographical errors, and providing more concrete examples and explanations for certain concepts.
2. The definition and implementation of the neural evaluator ($\varepsilon$) are not entirely clear. Further details on whether a pretrained model or auto-evaluation was used would be beneficial.
3. The paper lacks depth in explaining why certain prompting techniques outperform others. A deeper analysis of the mechanisms and failure cases would provide valuable insights and improve the overall contribution to the field.

Reviews: review: paper_summary: paper_summary: The paper investigates the ability of open-source LLMs such as Llama3 and DeepSeekCoder to solve easy optimization problems from automatically generating Python codes using few-shot learning. The paper aims at comparing different prompting techniques such as few-shot, self-consistency, chain-of-thought, and tree-of-thought in few-shot learning. The paper also proposes an evaluation module using LLMs to evaluate the quality of a reasoning path. However, the evaluation performance is not as good as expected. The results show that chain-of-thought achieves the best performance among the proposed prompting techniques.
strength_weakness: strength_weakness: 1. The paper doesn't show the content/template of each prompt used in different prompting techniques.
questions: questions: 1. Please list the specific prompts used in chain-of-thought prompting method.
2. Do you also use definition 3.6 to give a score of different models in the ensemble method?
paper_summary: paper_summary: The authors use two pre-trained large language models (LLaMA-3-8B-Instruct and DeepSeekCoder-7B-Instruct) to find solutions to linear programs by leveraging several prompt augmentation techniques (few-shot, self-consistency, chain-of-thought and tree-of-thought) and comparing their accuracy.
strength_weakness: strength_weakness: Overall, I believe a few (minor) changes and corrections would improve the clarity and readability of the manuscript:
 - 1
	 - The authors could outline their contribution more clearly at the end of the introduction (namely, listing the prompt engineering frameworks that are considered and mentioning that it is tried on LPs). 
- 2.1
	- Calude -> Claude
	- "The model approached GPT-3’s performance but ten times smaller and faster" is missing a "was"
- 3.1
	- Definition 3.4: $u_s$ -> $n_u$ in $U$
	- Definition 3.4: It seems that $U_1$, $A_1$ etc correspond to different ordered sets (and not indexing into the sets), while $\mathcal{P}_i$ appears to be an indexing into $\mathcal{P}$? If so, having the same notation is a bit confusing
	- If space allows, a concise and concrete example of the different definitions would help with understanding. Since the authors test the methods on LPs, a generic LP could be a possible example.
- 3.2
	- An example of "System instruction" would be appreciated
	- A quick explanation for what "mode" / "majority voting mechanism" are would be appreciated. Is this conducted through another neural network model? 
	- "we construct the tree-of-thought prompt PToT is similar to Equation 2": is similarly -> similarly
	- Eq 3: last $P_k$ -> $P_\text{new}$
	- Ensemble Strategy - "It has been known" should be "shown"?
- 3.3
	- In "Fixing Incorrect Conditions", $E2$ -> $E_2$
	- "we transform the inequality using Algorithm 2 to remove all divisions" -> The authors could add a one-sentence description of what Algorithm 2 does.  
	- Are these two repairing approaches (including the algorithm) common in literature? If so, a citation would be appreciated. If this is the authors' contribution, this could be emphasized in the introduction.
- 4.1
	- "with the fields ... represent ..." -> representing
	- The task was to fill in the required blanks in the results field of the testing data -> Could the authors add an example sample LP from the dataset, to see what the "blanks" look like and what an expected Python solution looks like? 
	- I am unclear on whether the LLMs try to find the solution set of the LP, or something else? Is the solution always unique here?
- 4.2
	- "we needs"
- 4.3
	- A brief discussion of how the authors' results compare to what they have observed in the literature on similar tasks would be appreciated, either in the results section or in the introduction.
questions: questions: n/a (addressed above)
paper_summary: paper_summary: The paper shows how to leverage open LLMs and well studies techniques, such as few-shots or CoT, to translate the problem from natural language to the full solution.
strength_weakness: strength_weakness: W1. Although you mentioned a lack of resources, it would be great to see smaller models finetuning. It's possible to finetune even with 1 GPU for less than 10 hours (i.e. 1.5B models after a LoRA finetuning). 

W2. The definition of the neural evaluator $\varepsilon$ is not clear. Did you pretrain a model? Did you use a autoeval?
questions: questions: Q1. Doing everything in one shot is probably hard. It would be extremely interesting to see if it's easier to use few-shots / CoT to translate the problem from words to formal language and use again few-shots / CoT to translate from formal language to the final solution. With this breakdown, authors can determine if the bottleneck is in the Python translation or the optimization problem formulation.

Q2. It's not super clear why ToT doesn't improve compared to CoT. It would be great to see a depth-in analysis, or at least a bunch of examples where CoT works/doesn't work and ToT doesn't work/works, respectively.

Q3. In section 3.2 you always denote the final guess as mode(...). For easy and solved problems, this is fine and probably encouraged. It would be interesting to see if you consider all possible paths and validate the solution when one path (for example one of the multiple CoT) is correct. 

Q4. Is there a particular reason why you use the "Let's think step by step" prompt?
paper_summary: paper_summary: The paper explores the capabilities of open-source Large Language Models (LLMs) to solve optimization problems using different prompting techniques. It contributes to the field of automated mathematical reasoning, specifically focusing on optimization problems.
strength_weakness: strength_weakness: 1. The paper is mainly about prompt engineering with existing methods. There's no innovation.
2. The paper could delve deeper into the mechanisms behind why certain techniques outperform others, which could provide valuable insights into how to optimize prompt engineering for future research.
3. The paper could benefit from more analysis of the results, particularly regarding the failure cases of certain techniques.
questions: questions: 1. Can you discuss some of the limitations or failure cases encountered during the experimental phase that did not perform as expected?
2. Since the methods used are all existing, how do you envision this work impacting the field of automated mathematical reasoning?