Chain-of-Experts: When LLMs Meet Complex Operations Research Problems
Authors: Ziyang Xiao
Dongxiang Zhang
Yangjun Wu
Lilin Xu
Yuan Jessica Wang
Xiongwei Han
Xiaojin Fu
Tao Zhong
Jia Zeng
Mingli Song
Gang Chen
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Model
Operations Research
Primary area: general machine learning (i.e., none of the above)
TL;DR: This paper introduces Chain-of-Experts (CoE), a multi-agent LLM framework that boosts reasoning in complex operation research problems by integrating domain-specific agents under a conductor's guidance and reflection mechanism.

Abstract: Large language models (LLMs) have emerged as powerful techniques for various NLP tasks, such as mathematical reasoning and plan generation. In this paper, we study automatic modeling and programming for complex operation research (OR) problems, so as to alleviate the heavy dependence on domain experts and benefit a spectrum of industry sectors. We present the first LLM-based solution, namely Chain-of-Experts (CoE), a novel multi-agent cooperative framework to enhance reasoning capabilities. Specifically, each agent is assigned a specific role and endowed with domain knowledge related to OR. We also introduce a conductor to orchestrate these agents via forward thought construction and backward reflection mechanism. Furthermore, we release a benchmark dataset (ComplexOR) of complex OR problems to facilitate OR research and community development. Experimental results show that CoE significantly outperforms the state-of-the-art LLM-based approaches both on LPWP and ComplexOR.

Published as a conference paper at ICLR 2024
CHAIN-OF-EXPERTS: WHEN LLMs MEET COMPLEX OPERATIONS RESEARCH PROBLEMS
Ziyang Xiao', Dongxiang Zhang'; Yangjun Wu', Lilin Xu', Yuan Wang', Xiongwei Han", Xiaojin Fu", Tao Zhong', Jia Zeng', Mingli Song', Gang Chen'
Zhejiang University 2 Huawei Noah's s Ark Lab
School of Business, Singapore University of Social Sciences
{xiaoz yang, zhangdongxiang, brooksong, eg) ug)ezju.edu. hau.eddu.cm
yangjjn, yangjun. wu@connect. polyu. .k, (hanxicngwei, Zeng. Beng.Jit Jia]
xulilin201e126 com, Jessi dessicavangg$eemal..cc fuxiaojin328hotmaill com, Shongtaco@huaweii.
ABSTRACT
Large language models (LLMs) have emerged as powerful techniques for various NLP tasks, such as mathematical reasoning and plan generation. In this paper, we study automatic modeling and programming for complex operations research (OR) problems, as to alleviate the heavy dependence benefit spectrum of industry sectors. We present the first LLM-based solution, namely Chain-of-Expert. (CoE), novel multi-agent cooperative framework enhance reasoning capabilities. Specifically, each agent is assigned a specific role and endowed with domain knowledge related OR. We also introduce ductor to orchestrate these agents via forward thought construction and backward reflection mechanism. Furthermore, of complex OR problems to facilitate OR research and community development. Experimental results show that CoE significantly outperforms the ttate-of-the-art LLM- L-M-ased approaches both on LPWP and ComplexOR.
domain experts and
build a benchmark dataset (ComplexOR)
INTRODUCTION
Operations research (OR) aims mathematically model complex decision-making problems that arise from a wide spectrum of industry sectors. To automate the procedure and reduce the depen- dence domain-specifie modeling experts, NI.4Ott (Natural Language for Optimization) (Rama- monjison et al., 2022a) has recently emerged as an attractive but challenging NLP task. Its objective translate the text description of an OR problem into math formulations for optimization solvers. To facilitate understanding the task, an example from the current NL4Opt benchmark dataset is de- picted in Figure 1. The prevailing NL4Opt models adopt two-stage framework. Initially, they perform NER to identify variables, parameters, and constraints from the input text, which are subse- quently converted into a mathematical optimization model. Despite their efficacy in elementary OR problems, these approaches fail in tackling complex real-world challenges.
In this paper, we study the automatic modeling and programming of complex OR problems derived from real-world industrial demands. As shown in Figure 1, their text descriptions often contain implicit constraints, posing a substantial interpretation challenge for existing NL40pt solvers. For instance, the phrase "zero lead times", highlighted in green, conveys the absence of any time lag between production orders. Additionally, it is imperative to possess -omain--pecific knowledge to understand terminologies such as "backlogging"" "carryover", and "lot-sizing". Finally, in contrast to the explicit input numbers the simple example, complex OR problems exhibit an abundance of implicit variables that require specification from domain modeling experts. The magnitude of variables and constraints in these complex problems introduces formidable hurdles and results in longer reasoning chain.
"Corresponding author.

---

Published as a conference paper at ICLR 2024
An example of current NL.4Opt dataset
Modeling result
A theme park transports its visitors around the park either by scooter or rickshaw. A scooter can carry 2 people while a rickshaw can carry 3 people. To avoid excessive pollution, at most 40% of the vehicles used can be rickshaws. If the park needs to transport at least 300 visitors, minimize the total number of scooters used.
Variables: s,T Constraints:
S 0.4(r +) 3r + 2s 2 300
Objective: minimize
An example of our dataset
In the context of manufacturing planning, we tackle the Capacitated Multi-level Lot Sizing Problem with Backlogging. We make the following assumptions in defining and formulating this problem. First, we assume that setup times and costs are won--onneeeeee sequence dependent, setup carryover between periods is not permitted, and all initial inventories are zero. Second, all production costs are assumed to be linear in production output and do not vary over time; hence, they can be dropped from the model for simplicity. Setup and holding costs also are assumed not vary over time. Furthermore, end items assumed have successors, and only end items have external demands and backlogging costs. Finally, we assume zero lead times ho lost sales. important note that all these assumptions (except setup carryover) are made for ease exposition only and without loss of generality, i.e., the theoretical results remain valid even when they are removed. See Ozturk and Ornk (2010) for the lot-sizing problem with setup carryover as well as with external demands for component items.
Figure Comparison between elementary and complex NI.4ptt problems. In the complex OR example, phrases in green indicate implicit constraints, and the Domain-specifte terminologies are highlighted in yellow. The model output is presented in the Appendix A.I.
To resolve the above issues, we leverage the power of LLMs and present the first LLM-based solu- tion. We propose a multi-agent reasoning framework, namely Chain-of-Experr (CoE), to orches- trate multiple LLM agents for complex OR problem solving. At the helm of the collaborative en- deavor. there presides a central entity, designated as the "Conductor", responsible for orchestrating the sequence of interactions among the agents. Each agent assigned a specific role and is equipped with domain-spectie expertise. We implement diversified agents with different skills, including but not limited terminology interpreter, construction of mathematical models and programming. Fur- thermore, we incorporate a backward reflection mechanism. Through a systematic analysis of the output, the framework has the capacity to detect potential errors the problem-solving process.
Comparison with other LLM-based reasoning. In recent years, extensive research efforts have been devoted to enhancing the reasoning capabilities of Large Language Models (LL.Ms). Notable examples in this domain include Chain-of-Thoughh (Wei et al., 2022), Self-consistency (Wang al., 2023a), Tree of Thoughts (Yao et al., 2023a), Graph of Thoughts (Besta et al., 2023), Progressive- Hint Prompting (Zheng et al., 2023), ReAct (Yao et al., 2023b). These works have formulated distinct prompting schemes and approaches to thought transformation. Further elaboration on these methodologies presented in the subsequent section. Unfortunately, these single-agent LLMs as well as multi-agent schemes like Solo-Performance Prompting (Wang et al., 2023b) exhibit conspic- uous limitations when confronted with complex OR problems because they cannot simultaneously tackle the challenges of implicit constraints, external knowledge prerequisites and long reasoning chain. In our CoE, we address these challenges via multi-expert collaboration and experimental results indicate that CoE can significantly outperform the L.-M-based approaches.
Contributions. (1) We study NL40pt the more challenging level, which requires the model to have implicit constraint discovery, domain-specific knowledge, and complex reasoning capabil- ity. (2) This the first LLM-based solution to complex OR problems. (3) We propose novel multi-agent framework called Chain-of- Experts (CoE), enabling collaborative problem-solving and iterative modeling optimization based on the forward thought construction and backward reflection mechanism. (4) We also build new dataset (ComplexOR) and the experimental results it affirm the superior performance of CoE over 8 other L.LM-based reasoning baselines.
2 RELATED WORK
NL40pt Problems. NL40pt aims to translate the descriptions of OR problems into mathematical formulations. A benchmark dataset' was curated by Ramamonjison et al. (2022a). To bridge the gap
hhttps://gh thg.t com/n/ 4opt/n /n1 4cpt- compet ti
2

---

Published as a conference paper at ICLR 2024
between the natural language input p and context-free formulation r, they proposed two-stage map- ping p + r + f that first adopted the BART-base model (Lewis et al., 2020) with copy mechanism to generate intermediate representation r, which was then parsed into canonical formulation. Edit-based models (Malmi et al., 2022) can be applied as a post-processing step for error correction. The two-stage framework was followed by subsequent studies. He et al. (2022) introduced semble text generator leveraging multitask learning techniques to enhance the quality of generated formulations, In a similar vein, Ning al. (2023) proposed a prompt-guided generation framework, complemented by rule-based propprocssingg and post-pon processsssingg techniques, to enhance accuracy. In related research endeavor, Prasath Karande (2023) investigated the synthesis of mathematical programs. GPT-3 with back translation was utilized to synthesize the canonical forms as well as Python code.
LLMs-based Reasoning. Language models have shown substantial potential in solving complex reasoning tasks within specific domains, such as TSP(Zhang et al., 2023), databases(Xuanhe Zhou, 2023) and knowledge systems(Zhu al., 2023). The Chain-of-Thought (CoT) (Wei al., 2022) broke a complex reasoning task into a series of intermediate reasoning steps. Self-consistency (Wang et al., 2023a) replaced the greedy decoding CoT by sampling a diverse set of reason- ing paths and selecting the most consistent answer. Tree of Thoughts (ToT) (Yao et al., 2023a) and Graph of Thoughts (GoT) (Besta al., 2023) further enhanced the reasoning capability by allowing LLMs to explore and combine thoughts in structured manner. Progressive- Prompting (PHP) (Zheng et al., 2023) progressively refined the answers by leverageing previously generated answers as hints. Subsequent works, such ReAct (Yao et al., 2023b) and Reflexion (Shinn et al., 2023), allowed LLMs to interface with additional information or feedback from external sources. Recently, cooperation among multiple agents has also been explored. CAMEL (Li et al., 2023) introduced a novel communicative agent framework for autonomous cooperation. Solo Performance Prompt- ing (SPP) (Wang al., 2023b) transformed a single L.M into cognitive synergist by simulating multiple personas and demonstrated the potential problem-solving abilities for multi-agent systems.
3 PROPOSED METHOD
3.1 EXPERT DESIGN
In our reasoning framework, an "expert" refers to specialized agent based on Large Language Model (LLM) augmented with domain-specific knowledge and reasoning skills. Each expert is assigned a specific role and undergoes four steps:
Step 1: In-context Learning. Each agent is allowed to access external knowledge base and perform top-k retrieval against the knowledge base. The retrieved information is then provided to the LLM to facilitate in-context learning. For example, expert responsible for generating Gurobi programs can access the Gurobi official API documentation. This step is optional, depending on the availability of the knowledge base.
Step 2: Reasoning. Thought self-consistency to perform reasoning task according their specific role. Our reason- ing procedure consists of forward thinking and reflection modes, whose details will be presented in the subsequent section.
-LaMe expert utilizes existing prompting techniques, such as Chain-of-
Summarize. Due to the token limit constraint in a single interaction with LLM,, an expert can choose summarize their reasoning output. Since this step may result in significant information loss, it is optional for certain experts (e.g., modeling experts).
Step 3:
Step 4: Comment. This step is inspired by Solo Performance Prompting (Wang et al., 2023b), in which the participants are allowed to give critical comments and detailed suggestions. The objective make the communication between agents more constructive.
3.2 THE WORKFLOW OF CHAIN-OF-EXPERTS
The framework of our proposed Chain-of-Expert (CoE) is depicted in Figure 2. We initialize collection of 11 experts such as terminology interpreter, modeling experts, programming experts, and code reviewing expert. Their detailed design specifications available in Appendix 1.2.1.
3

---

Published as a conference paper at ICLR 2024
Problem Input: the conrest of monsficcturng plotning, we tockle the Multi-tevel Los Stzing Probiem with Backlogging, We asssume that...
Conductor
Terminology Interpreter: "sackiogging" refers to costoneer
Modeling Expert:
Programmer:
Variobles ff.
Import gurabapy from gurobige imporr
start
Constratitte...
orders comnot be net nime....
Object Mininioo
model op Model"
Terminology Interpoeter
Modeling Expert:
Fapolingor that Thave
Pse resiened the codfe, and confiru that uccuranely reflects the modeling...
Programmer:
4
Evaluntor's 's Feedback: 41: Variable "O" defined...
WY Modklling Expert
resiewed modeling process, (here vaas di errer
+ (
Programmet
Programmer:
Evalustor's s Feedback:
forward pass
Hlore's 's the corrected Pyshon based
Final answer
backward pass
nuccessfully!
forward pass
Evaluator
An example to illustrate the workflow of Chain-of-Bxperts. this example, the Conductor receives the input problem and starts coordinating the experts. The exemplar workflow consists of @: terminology interpretation for the input problem; @: problem modeling; @: program generation; @ evaluation of correctness and identify issue; @: reflection of program, confirming correctness; @ reflection modeling, find a mistake; (.: proceed with program generation again; @: final evaluation, confirming correctness.
Figure 2:
There is Conductor to effectively coordinate these experts. It iteratively and dynamically selects an expert to construct forward thought chain. The candidate answer generated by the forward reasoning chain will be passed to an external program execution environment, whose feedback signal triggers a backward reflection procedure. The details of these two key components are elaborated in the following:
Forward Thought Construction. During the forward-thought construction phase, the experts sequentially selected by the Conductor. We can formulate forward thought construction as a sequen- tial decision-making process, where view the set of experts the action space. Let's define the input problem description as P, and a set of pre-defined experts as = {E,,, Esb,,... E J. where is the total number of experts and Φb; represents the configuration of the i-th expert. Each expert is associated with an optional knowledge base and a prompt template. We denote the set of comments the f-th reasoning step C, and define the state in Equation 1.
S, = (P, C),
(I)
Unlike traditional reinforcement learning which requires a large amount of training data, we utilize a training- free approach by leveraging the prompt technique of large language models. This allows us to achieve the same functionality as a decision-makinn agent without any training. Consequently, we model the policy of Conductor in Equations 2, where the Conductor acts as a policy function to select the experts, FF represents the large language model, 0' represents the parameters of L.M,, and PT, denotes the prompt template at the I-th step.
Conductor ye' (FT.) (cls) = P, = e|S, =8]
Based on the above formulation, the expert selection policy can be translated into the design of prompt template ITT,, which requires prompt engineering to achieve an optimal policy. The detailed design of the prompt template presented in Appendix A.2.2. After designing Conductor, we can update the comment set in each reasoning step follows.
Es,, Conductor
(3)
Es,, E= (P.C.)
(4)
C++1 = (c)
(5)
where Es,, represents the selected i,-th expert at step t and c denotes the comment of the selected expert. We concatenate the previous comments Ct and to obtain C++1 as the new state. After fixed number of steps T. the forward process in the Cf framework is terminated. At this point, all the comments are summarized to form the final answer A.
Backward Reflection. The backward reflection mechanism in the ofiinerf-Experts enables the system to leverage external feedback and adjust the collaboration among experts based the
4

---

Published as a conference paper at ICLR 2024
Algorithm Chain-of-Experts
Input: problem description p
Paramefers: forward steps maximum forward- backward trials 7''
I: Initialize set of comments C' +- 0
Initialize a stack experts t
3: for t 1.... do
4: for 1iri=,,,,, N do
5: expert, Conductor (p. C))
6:
comment experts(p, C) C- CU {comement) E.push(erperl.)
8:
9:
end for
10: answer + Reducer(p, C)
feedback, passed + (oaluator(onsser)
12:
if passed then
13:
return answer 14: end if
15: stop backward +- false
while not stop backward and not E.emply() do
17: expert +- E.pop()
18: feedback, stop.backward expert.refledtpp C, feedbock)
19:
C + CU {feedbock)
20:
end while
21:
end for
22:
return arsswer
evaluation of problem-solving results. Let's define the trajectory of experts selected in order as {Eg., E Es., }. where represents the index of the expert selected at step The back- ward reflection process starts with external feedback Traw, which typically provided by a program execution environment. This process can be denoted as Traw + ezecution(A). Then, the initial signals are derived from the evaluation of the raw external feedback: (ro. srp) + evaluate(rreed where to is boolean value indicating whether the backward process needs to continue and sro represents the natural language summary of the feedback, which is used to locate errors during the backward reflection process. If the answer A is deemed correct, is set to false and the whole reasoning procedure terminates. Otherwise, the Chain-of-Experts initiates a backward self-reftection process to update the answer. The process begins with the last expert E and backpropagates in reverse order to iteratively update the feedback signal. At the I-th backward step, the update of the state described by Equation and 7, where reflect represents one of the reasoning abilities in expert design. The ref reflee flect function also produces tuple of r, and sT which aligns with the output of the evaluate function. In this case, T; is set to true when the expert confirms the correctness of its previous comment.
(ri, + reflect(E P,C,,
C
The backward reflection process continues until the feedback signal indicates that the expert Eøsr the one who made the mistake or or until all experts have been reflected upon. Subse-
quently, a forward process will performed again.
It is worth noting that reflection method differs from Reflexion (Shinn et al., 2023), where reflection is performed at the system level with interaction among multiple experts, and the feedback recursively backpropagated. In contrast, Reflexion just involves single LLM.
3.3 IMPLEMENTATION DETAILS
Algorithm 1 provides the implementation pseudo-code of the Chain-of-Expert framework, which consists of four main stages:
Initialization (lines 1 2): The process begins by initializing the set of comments C. Additionally, a stack is used to store the selected experts, ensuring a fnst-in-last-or order for forward thought construction and backward reflection.
5

---

Published as a conference paper at ICLR 2024
Forward Thought Construction (lines 4- 9): Experts selected sequentially by the Conductor, with each expert contributing their comments to the global comment set C. Forward construction process continues for fixed number of steps N. As depicted in line 10, once the forward construction is completed, a Reducer is employed to summarize all the comments and generate a final answer. Since the comment set contains numerous comments after the forward construction, the Reducer plays crucial role in summarizing and reconciling these comments. For more detailed prompt template design of the Reducer, please refer to Appendix A.2.3.
Backward Reflection (lines 11 20): In line 11, once a solution is obtained, an Evaluator gathers feedback and converts it into natural language feedback to assess its correctness. If the solution is deemed incorrect, the system enters reflection phase. In this phase, experts are consulted iteratively in reverse order by removing them from the stack. They are prompted reflect on their solution and provide additional comments if necessary. As indicated in line 16, the backward process continues until a mistake is found by self-reftection or the first expert is reached.
Iterative Improvement (loop in line 3): The steps of forward thought construction and backward reflection are repeated iteratively until satisfactory solution is achieved or maximum number of trials T is reached.
4 EXPERIMENTS
4.1 DATASETS
LPWP. The LPWP dataset (Ramamonjison et al., 2022h) is collected from the N1.4pt competition in NuerIPS 2022. comprises 1101 clementary-leve linear programming (LP) problems. Each problem consists of a text description with IR annotations including parameters, variables, linear constraints and the objective function. The dataset is partitioned into 713 training samples, 99 validation samples, and 289 test samples for performance evaluation.
ComplexOR. With the assistance from three specialists with expertise in operations research, we constructed and released the first dataset for complex OR problems. We selected 37 problems from diversifed sources, including academic papers, textbooks, and real-world industry scenarios. These problems cover a wide range of subjects, spanning from supply chain optimization and scheduling problems to warchousing logistics. It took the experts nearly a month to annotate each problem with model formulation, and minimum of five test cases to verify the correctness of generated code.
4.2 MODEL SETUP AND PERFORMANCE METRICS
In our experimental setup, we use the GPT-3.5-turbo as the default large language model. We set the parameter temperature to a value of 0.7 and conduct five runs to average the metrics. The number of iterations is set to 3, with each iteration consisting of forward steps by default.
Since infeasible for domain experts to manually evaluate the output from the LLM-based so- lutions, we employed automated code evaluation process. Specifically, require each solution to generate the programming code for each OR problem. the code can pass the associated test cases annotated by the OR specialists, we consider the problem is successfully solved. We use Ac- curacy to indicate the success rate. Besides this rigorous metric, we also adopt compile error rate (CE rate) to capture the percentage of generated programs that fail to compile, possibly caused by issues the automatic modeling process; Alternatively, runtime erro rate (RE rate) measures the percentage of generated programs that encounter errors during execution, which caused by internal logic errors such as unsolvable models or non-linear constraints. The experimental code is http://ghhuubb.ooo/myymmsmxxxyy/ wwwpu//swwecoccaaaaaaoooeeeeess.
4.3 BASELINES
We compare CoE with baselines. As to traditional approaches for NL4Opt, we we consider tag- tag-BART (Neeraj Gangwar, 2022) as a SOTA model, which won 1st place in the NeurlPS competition (Ra- mamonjison al.. 2022b). We also compare CoE with prevailing LLM-based methods. including Chain-of- Chain-of-Thought. Progressive Hint, Tree-hf-Thought, Eraph-of-Thoughh ReAct, Reflexion and Solo Performance Prompting. The default GPT without any optimization the reasoning

---

CE rate Published as a conference paper at ICLR 2024
Table Comparison with baselines on LPWP and ComplexOR
LPWP
ComplexOR
Method
Accuracyt
RE rate
Accuracyt
CE ratel
RE rate
tag-BART
47.9%
0%
Standard
42.4%
18.1%
13.2%
0.5%
36.8%
8.6%
Thaoughf-
Thought
45.8%
20.5%
9.4%
0.5%
35.3%
8.6%
Progressive Hint
42.1%
19.4%
10.3%
2.2%
35.1%
13.5%
Tree-of-Thoughe
47.3%
17.4%
9.7%
4.9%
31.4%
7.6%
Graph-of-Thoughh
48.0%
16.9%
9.1%
4.3%
32.4%
8.1%
ReAct
48.5%
15.5%
11.2%
14.6%
31.9%
10.8%
Reflexion
50.7%
7.3%
9.0%
13.5%
12.9%
10.1%
Solo Performance
46.8%
17.9%
13.6%
7.0%
46.5%
13.5%
CoE without expert
55.1%
4.0%
11.9%
18.8%
7.9%
15.0%
Chain-of-Expen offextstts
58.9%
3.8%
7.7%
25.9%
7.6%
6.4%
chain is named Standard, which is expected to achieve inferior performance. We also implement a baseline that uses the same model, which uses a uniform system prompt. "You are a helpful as- sistant," across all roles, without any additional knowledge bases. The detailed implementation of these algorithms described in Appendix A.3.
4.4 OVERALL PERFORMANCE ON LPWP AND COMPLEX OR
The results in terms of accuracy, CE rate and RE rate in the two benchmark datasets are reported in Table 1. Since the traditional method fag-BART' is not capable of generating code, measure its accuracy by requiring its constraints and objective in the math model to be correct. Note that generating valid linear programming model is prerequisite step for correct code generation. Even under such a loose evaluation metric, Tag-BART is still inferior to certain LLM-based baselines, verifying the feasibility of applying LLM to solve OR problems. We also observe that tag-BART fails in all problem instances of ComplexOR.
Among the I.L.-based baselines, Reflexion stands out as the most promising OR problem solver. Due to its self-reflection mechanism, it achieves the lowest CE rate and RE rate in both datasets. When confronted with complex OR problems, its overall accuracy slightly inferior to ReAct. The reason is that in complex OR problems, the ability access external knowledge bases becomes more crucial, which is a strength ReAct. Even though Solo Performance also adopts a multi- agent reasoning framework, its performance is not satisfactory. Unlike collaborative reasoning framework, agents are simply initialized by leader LLM and lack effective cooperation to solve the challenging OR problems.
Our proposed CoE established clear superiority among all performance metrics in both datasets. In LPWP, the accuracy is 58.9%, surpassing the state-of-the-art agent Reflexion by 8.2%. In its best performance, CoE also manages to solve 10 out of 37 complex problem instances in the Com- plexOR. The outstanding performance owes the effective design of the Chain-of-Experts reason- ing framework, including the expert design methodology, the roles of the Conductor and Reducer, and the reflection mechanism. We also find that the removal of experts' features leads to de- crease in accuracy, which suggests that the CoE benefits from using specialized experts. In the next experiment, we will investigate the effect of these ingredients through an ablation study.
4.5 ABLATION STUDY
Regarding single expert design, Table 2 highlights the positive roles played by both the knowledge base and reasoning ability. Removing these components results in a slight drop in accuracy. Inter- estingly, we found that summarization is the most crucial design aspect. Reasons are two-fold. First, the length of comments may exceed the token limit of GPT-turbo-3.5 and the overflowed tokens will be discarded. Second, compact and meaningful summary is more friendly for decision making in the downstream experts.
7

---

Published as a conference paper at ICLR 2024
Table 2: Ablation study of Chain-of-Experts
LPWP
ComplexOR
Method
Accuracy
CE rate
RE rate
Accuracy
CE rate
RE rate
CoE (Full)
58.9%
3.8%
7.7%
25.9%
7.6%
6.4%
w/o knowledge base
58.0%
4.0%
8.5%
23.3%
8.4%
7.9%
inner-
w/o CoTT reasoning
58.2%
3.7%
7.9%
24.3%
8.1%
6.4%
agent
w/o summarize
56.3%
3.8%
9.4%
20%
7.6%
10.3%
w/ Reflection
55.6%
4.2%
12.2%
22.7%
7.8%
10.6%
inter-
w/o Conductor
54.2%
6.5%
8.2%
21.1%
8.1%
8.6%
agent
w/o Reducer
56.5%
5.5%
8.8%
23.0%
9.2%
8.1%
For inter-expert collaboration, evaluate the effect of backward reflection, forward thought struction in Conductor, and reducer. As shown in Table 2, after removing the component of back- ward reflection, the accuracy drops significantly from 58.9% 55.6%, and the RE rate increases noticeably from 7.7% to 12.2%. These results imply that without the reflection mechanism, the system prone to mistakes in logical reasoning and lacks the ability for self correction. To eval- uate the effect of Conductor, we replace with random selection of subsequent experts during the construction of the forward chain of thought. The performance also degrades significantly because the experts are no longer well coordinated and the random selection of experts may even be detri- mental the reasoning process. It's surprising to find that the Reducer component also contributes remarkably. This module summarizes the collective insights from multiple preceding experts. If we remove it, the answer will be extracted from the concatenation of the experts' raw comments, which may lead to incorrect conclusions, as the comments can be fragmented and even conflicting with each other.
4.6 PARAMETER SENSITIVITY ANALYSIS
We evaluate critical parameters related reasoning capabilities, including the number of steps in the forward thought construction and the temperature in the Large Language Model. From the results shown in Figure 3a, we can draw conclusions. Firstly, a lower value for the temperature parameter tends to lead to better performance. This suggests that, for the experts benefit from providing more deterministic and consistent thoughts, rather than creative or diverse ones. Secondly, longer reasoning chain that involves more experts in the forward thought construction generally improves accuracy. However, occurs at the cost of higher reasoning time cost and more API requests. That's why select temperature 0 and forward steps 5 as the default parameter configuration.
problems,
$5.3%
0.58
F0
Frequacy (%)
60-
Accuracy
0.56
46.0%%
47.3%
38.5%
0.54
temperature=0.0 0.0
10.2%
0.52
0emperature-0.3
6.0%
lemperature-O.5 0.6
Vesiele Detraction
OSfeet Extrctia
Modeing bege
Modollag
LP Conerttion
Rualbillyy Chock
Poggnmiin
Cade Resiever
temperature-0.9
0.50
Enumpl
Teminalyy
3
Forward Steps
(a) CoE performance on different parameter settings.
(b) Selection firequeccy of individual expert.
Figure 3: Parameter sensitive analysis and selection frequency analysis LPWP
8

---

Published as a conference paper at ICLR 2024
Table Robustness Chain-of-Experts under different large language models
GPT-3.5-turbo
GPT-4
Claude2
Method
LPWP
ComplexOR
LPWP
ComplexOR
LPWP
ComplexOR
Standard
42.4%
0.5%
47.3%
4.9%
44.9%
0.0%
Reflexion
50.7%
13.5%
53.0%
16.8%
51.4%
12.4%
-hxpersf-Experts
58.9%
25.9%
64.2%
31.4%
62.0%
27.0%
4.7
OTHER LLMS AS BASE REASONING MODEL.
We also conduct an investigation into the impact of using different LLMs within the Chain-of- Experts. We consider GPT-4 and Claude2 as two alternative LLMMs and select Standard and Reflexion as two baselines. As shown in Table 3, all methods benefit from the upgrade of more advanced LLMs. However, our Chain-of-Experts approach exhibited the most substantial improve- ment. For instance, when GPT-4 is used, we observed accuracy boost of 8.3% in LPWP and 5.5% in ComplexOR, the highest among the three methods.
4.8 EXPERIMENTAL ANALYSIS OF EXPERT SELECTION FREQUENCY
In the final experiment, we aim gain deeper understanding of the Conductor's behavior and examine the rationality behind its selection of experts.
First, conduct experiments the LPWP dataset and analyze the selection frequency of each expert. In Figure 3b, we observe that the Programming Expert and Modeling Expert are the two most frequently selected experts. This finding is consistent with the expectation that modeling and programming are crucial to solving OR problems. Additionally, we notice that the extraction of parameters, variables, constraints, and objective functions is rarely selected. This can be attributed to advancements in language comprehension capabilities of LLMs, which now can understand problem statements directly, without the need for the step-by-step NER traditional methods.
Moreover, we study the most frequently sampled collaboration paths involving multiple experts. Each problem-solving process provides path that represents the order of experts involved. In Table 4, observe that when the parameter forward steps is set to 2, where only two experts collaborate to solve a problem, the most frequent path from the Modeling to the Programming Expert. This finding aligns with the importance of these roles in problem-solving. Additionally, when the steps is set to 6, the collaboration path becomes more complex and resembles real-world workflows.
Table 4: The most frequently collaboration paths for experts on different forward steps settings.
Forward steps
Most frequent path
2
Modeling + Programming
Knowledge >> Modeling Programming
4
Terminology Interpreter Knowledge Modeling Programming
5
Terminology Interpreter + Modeling + LP file Generator - Programming
Code Reviewer
6
TI + Modeling -
LP file Generator Programming Example Provider +
Programming + Code Reviewer
CONCLUSION
In this paper, we presented the first LLM-based solution to complex OR problems. To enhance reasoning capabilities, we devised Chain-of-Expeen (CoE), novel multi-agent cooperative frame- work. The core of CoE was a conductor orchestrating a group LLM- experts via forward thought construction and backward reflection mechanism. We built facilitate OR research and community development. Experimental results indicated that our CoE significantly outperformed the ofate-of-the-an reasoning methods both on LPWP and ComplexOR.
dataset, ComplexOR to

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The work is supported by the National Key Research and Development Project of China (202YFF0902000).
REFERENCES
Maciej Besta, Nils Blach. Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoe- fler. Graph of thoughts: Solving elaborate problems with large language models. CoRR, abs/2308. 09687, 2023. doi: 10.48550Vax21..2338.88885 (8550/arXis 2tt.0///xxiv........ 2308. 09687.
URL https::/dol org/10.
JoangLong He, Mamatha N, Shiv Vignesh, Deepak Kumar, and Akshay Uppal. Linear program- ming word problems formulation using ensemblecrf ner labeler and t5 text generator with data augmentations, 2022.
Dave Hulbert. Tree of knowledge: Tok aka of knowledge dataset for large language models Ilm. https: //gttttt tiitthu....
1010/ http///teeeeoooouttttt--------
ing, 2023.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequennceto--eqquence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line. July 2020. Association for Computational Linguistics. doi: (0.18653/v1/2022 URL https::tp//..m.. accl/aacccca// anllannnollloo ogy- -egyyooooooo org/ 2020 acl -ma 703.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: communicative agents for "mind" exploration large scale language model society. CoRR, abs/2303.17760, 2023. doi: 10.48550/arXi..2333..777 10. 405448550// arxiv. 2303. 17760.
URL atttps:/// //doi.. org/
Eric Malmi, Yue Dong, Jonathan Mallinson, Aleksandr Chuklin, Jakub Adámek, Daniil Mirylenka, Felix Stahlberg, Sebastian Krause, Shankar Kumar, and Aliaksei Severyn. Text generation with text-editing models. CoRR, abs/2206.07045 2022. doi: 10485500aaXi..226..0033. URL arxiv 220. 07043.
atttps///oli http:://dooooogg/1/// /10 http//1..48550000
48550/
Nickvash Kani Neeraj Gangwar. Tagged input and decode all-at-once strategy. https:// github. wwwp:////gwoo//////.oooo
generation, 2022.
Yuting Ning, Jiayu Liu, Longhu Qin, Tong Xiao, Shangzi Xue, Zhenya Huang, Qi Liu, Enhong Chen, and Jinze Wu. A novel approach for auto-formulation of optimization problems, 2023.
Ganesh Prasath and Shirish Karande. Synthesis of mathematical programs from natural language specifications. DL4C Workshop, ICLR, 2023.
Rindra Ramamonjison, Haley Li. Timothy Yu, Shiqi He, Vishnu Rengan, Amin Bunitalebi-dehkordh Zirui Zhou, and Yong Zhang. Augmenting operations research with nnto-formulation of optimiza- tion models from problem descriptions.
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 29-62. Abu Dhabi, UAE, Decem- ber 2022a. Association for Computational Linguistics. doi: 10. htt.8655/1112222mmmmll.... URL httph:tt///.mm. http://aaaaaannnnaaaccooco org/ 222220222222 emnlp incustry.4.
Rindra Ramamonjison, Haley Li, Timothy T. L. Yu, Shiqi He, Vishnu Rengan, Amin Banitalebi- Dehkordi, Zirui Zhou, and Yong Zhang. Augmenting operations research with auto-formulation of optimization models from problem descriptions. In Yunyao Li and Angeliki Lazaridou (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing: EMNLP 2022 Industry Track, Abu Dhabi, UAE, December 7 11, 2022, pp. 29-62. Association for Computational Linguistics, 2022b. doi: 11.8633/v1/0222emmmlpp / //doi. org/orgg/// 1 10. v /20/2.02222 emn -immmll--innrrree industry.444444
URL https:
10

---

Published as a conference paper at ICLR 2024
Noah Shimn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic
memory and self-reflection. CoRR, abs/2303. 1366, 2023. doi: 10.48550/amXii2220..1111
URL /tttps:///oliiiii wttg/o.......ooooooooooo... 401.00.5550//raaa 2303.113631
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc V. Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL / pdfFid=1PLINIM0t..
http:////peeeeeeeeeeeeeee net/
Zhenhailong Wang. Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through
self- collaboration. CoRR, abs/2307 05300, 2023b. doi: 10448550/a2xw..337..666 2307005300. URL https:
2307.05300.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, 2022. URL http:: mips. wwtppper_ciiiccpccccc fils /paper/ 10722/hash/9d56000 hhthhhh/2555000113444444ff111111 html.
Guoliang Li Xuanhe Zhou, Zhaoyan Sun. Db-gpt: Large language model meets database, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. CoRR, abs/2305. 10601, 2023a. doi: 10.48550//2Xiv.2355.00 48550/arxiv 2305.10601
URL https: //doi org/10.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representationss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023b. URL httts:://ppeeeeeee http:///pppeeeieeeeeeeeee net hwtl/pdfffu pdffid=WE vlwYUL-X.....
Dongxiang Zhang, Ziyang Xiao, Yuan Wang, Mingli Song, and Gang Chen. Neural tsp solver with progressive distillation. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artifi- cial Intelligence and Thirteenth Symposium on Educational Advances Artificial Intelligence, AAAI23/AAAT233AAAA AAAI Press, 2023. ISBN 978-1-57735-880-4 doi: 10.1609/aaai. V37210.26432. URL https: / doi. wtt.:o..o./...////...o.o aaai w37.10..26333
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hin prompting improves reasoning in large language models. CoRR, abs/2304.097977, 2023. doi: 10.48550/ arXiv.2304.097977 URL https: /ow.///oii//////////////.... org/10.../..... /1/ (118850/aaxx.a www.:xx........99.
Di Zhu, Hailian Yin, Yidan Xu, Jiaqi Wu, Bowen Zhang. Yaqi Cheng. Zhanzuo Yin, Ziqiang Yu, Hao Wen, and Bohan Li. A survey of advanced information fusion system: from model- driven to knowledge-enabled. Data Science and Engineering, 8:1-13, 04 2023. doi: 10.1007. 141019-223-0220998
A APPENDIX
A. AN EXAMPLE OF COMPLEXOR DATASET
The example shown in Figure shows a typical instance of a complex operations research problem. Specifically, it illustrates a Capacitated Lot Sizing Problem, which is much more challenging. This problem involves a wide range of constraints, such as summations, equations, and inequalities. Un- like simpler problems, the objective function in this is not straightforward linear expression but rather a sunmation across multiple set variables. These combined characteristics categorize this problem complex optimization challenge.
11

---

Published as a conference paper at ICLR 2024
An example of our datasct
the manufacturing planning, we tackle Capacitated Multi-level Sizing Problem with Backlogging make the following assumptions in defining and formulating this problem. First, we assume that setup times costs are non-sequence dependent, setup carryover between periods not permitted, and initial inventories are zero. Second, all production costs are assumed to be linear in production output and do not vary over time; hence, they can be dropped from the model for simplicity, Setup and holding costs also are assumed not to vary over time. Furthermore, end items are assumed to have no successors, and only end items have external demands and backlogging costs, Finally, we assume zero lead times and no lost sales. It important to note that all these assumptions (except setup carryover) are made for ease of exposition only and without loss of generality, le,, the theoretical results remain valid even when they removed. See Ozturk and Ornek (2010) for the lot-sizing problem with setup carryover well with external demands for component items.
Modeling
Scts: Perios, M, end, eta
Parameters: bc, he, st, gd, Mn,r.C Variables: x,s, y,
Constraints:
invBalance Xxz St.c1 bu: bi gdz Sty end,t Periods for E end,t Perlods invBalance2: X2 + St,- gdd, + Ejecta, T2 *ja +
capConstraints: Σie a.m Xie Eng st.m Vie <= Cmz for m M, E Periods setupConstraints Xi Mn Via for E Periods
initStalus: Sr, lür E
endStatus: bir
Objective: minimize EieIseperioa, Vı, + he: 9.2) + LieendtePerions bc: byc
Figure 4: An example of ComplexOR dataset.
A.2 MORE IMPLEMENTATION DETAILS OF CHAIN-OF-EXPERTS
A.2.1 EXPERTS DESIGN
In this section, we provide in-depth overview of the individual experts participating in the Chain- of-Experts framework. Table offers a comprehensive list these experts, each assigned a specific role and domain knowledge relevant to OR problem-solving.
Table 5: All experts involved in Chain-of-Fxperts
Expert name
Knowledge base
Terminology Interpreter
Supply Chain Optimization & Scenario Modeling
Parameter Extraction Expert
Variable Extraction Expert
Constraint Extraction Expert
Objective Function Extraction Expert
Modeling Knowledge Supplement Expert
CAMS-Cutting Edge Modeling
Modeling Expert
LP File Generation Expert
LP format Documentation
Constraint Feasibility Check Expert
Programming Example Provider
Gurobi Example Tour Gurobi Reference Manual
Programming Expert
Code Reviewer
Below, we present the detailed descriptions and prompt template implementations for each expert. Please note that text enclosed within curly braces signifies placeholders that will be dynamically populated during runtime based on the specific problem description, comments provided by experts, and retrieved knowledge.
Terminology Interpreter:
12

---

Published as a conference paper at ICLR 2024
Role description: Provides additional domain-specing knowledge to enhance problem understand- ing and formulation.
Prompt template: As domain knowledge terminology interpreter, your role is to provide addi- tional information and insights related to the problem domain. Here are some relevant background knowledge about this problem: (knowleddee). You contribute by sharing your expertise, plaining relevant concepts, and offering suggestions to improve the problem understanding and for- mulation. Please provide your input based on the given problem description: {problem).
Parameter Extraction Expert:
Role description: Provides additional domain-specine knowledge to enhance problem understand- ing and formulation.
Prompt template: As a variable extraction expert, your role is to identify and extract the relevant variables from the problem statement. Variables represent the unknowns or decision variables in the optimization problem. Your expertise in the problem domain will help in accurately identifying and describing these variables. Please review the problem description and provide the extracted variables along with their definitions: (problem).
Variable Extraction Expert:
Role descríption: Proficient in identifying and extracting relevant variables from the problem state- ment.
As a variable extraction expert, your role to identify and extract the relevant variables from the problem statement. Variables represent the unknowns decision variables in the optimization problem. Your expertise the problem domain will help in accurately identifying and describing these variables. Please review the problem description and provide the extracted variables along with their definitions: (problem).
Prompt template:
Constraint Extraction Expert:
Role description: Skilled in extracting constraints from the problem description.
Prompt template: As a constraint extraction expert, your role is to identify and extract the con- straints from the problem description. Constraints represent the limitations or conditions that need to be satisfied in the optimization problem. Your expertise in the problem domain will help in ac- curately identifying and formulating these constraints. Please review the problem description and provide the extracted constraints: {problem]: }. The comments given by your colleagues are as fol- lows: {comments). please refer to them carefully.
Objective Function Extraction Expert:
Role description: statement.
Capable of identifying and extracting the objective function from the problem
You are expert specialized in Operations Research and Optimization and re- sponsible for objective function extraction. Your role is to identify and extract the objective function from the problem statement. The objective function represents the goal of the optimization problem. Now, the problem description as following: (problem).
Prompt template:
Modeling Knowledge Supplement Expert:
Role description: Offers supplementary knowledge related to modeling techniques and best prac- tices.
As modeling knowledge supplement expert, your role is provide additional knowledge and insights related to modeling techniques and best practices in the field of Operations Research and Optimization. Here some relevant background knowledge about modeling tech- nique: [knowlegge]. You can contribute by explaining different modeling approaches, suggesting improvements, or sharing relevant tips and tricks. Please provide your input based the given problem description and the modeling efforts so far: {problem).
Prompt template:
13

---

Published as a conference paper at ICLR 2024
Modeling Expert:
Role description: Proficient in constructing mathematical optimization models based on the ex- tracted information.
Prompt template: You are a modeling expert specialized in the field Operations Research and Optimization. Your expertise lies in Mixed-Integer Programming (MIP) models, and you possess in-depth understanding various modeling techniques within the realm of operations research. At present, you given Operations Research problem, alongside additional insights provided by other experts. The goal is to holistically incorporate these inputs and devise a comprehensive model that addresses the given production challenge. Now the origin problem is follow: (problem). And the modeling is as follow: {comments) Give your model of this problem.
LP File Generation Expert:
Role description: Expertise in generating LP (Linear Programming) files that can be used by opti- mization solvers.
As an LP file generation expert, your role to generate LP (Linear Program- ming) files based on the formulated optimization problem. LP files are commonly used by opti- mization solvers to find the optimal solution. Here is the important part source from LP file format document: (knowledge). Your expertise in generating these files will help ensure compatibility and efficiency. Please review the problem description and the extracted information and provide the gen- erated LP file: (problem) J. The comments given by your colleagues are as follows: {commments). please refer to them carefully.
Prompt template:
Programming Example Provider:
Role description: Provides programming examples and templates to assist in implementing the optimization solution.
Prompt template: As programming expert in the field of operations research and optimiza- tion, you offer programming examples and templates according to the background knowledge: (knowledge). Now, according to problem description: {problem). Could you please comprehend the extract code snippets in background knowledge and understand the their function, then give your code example assist with addressing this problem. The comments given by your colleagues are as follows: hwtp:::(coommerr {comments), please refer to them carefully.
Programming Expert:
Role description: Skilled in programming and coding, capable of implementing the optimization
solution in programming language.
You are Python programmer in the field of operations research and optimiza- tion. Your proficiency in utilizing third-party libraries such as Gurobi is essential. In addition to your expertise in Gurobi, would be great if you could also provide some background in related libraries tools, like NumPy, SciPy, or PuLP. You are given specific problem and comments by other experts. You aim to develop an efficient Python program that addresses the given problem. Now the origin problem is as follow: {problem] } And the experts along with there comment are as follow: (comments) Give your Python code directly.
Prompt template:
Code Reviewer:
Conducts thorough reviews of the implemented code to identify any errors, inef- ficiencies, or areas for improvement.
Role description:
Prompt template: As Code Reviewer, your responsibility is conduct thorough reviews of implemented code related to optimization problems. You will identify possible errors, inefficiencies, or areas for improvement in the code, ensuring that it adheres to best practices and delivers optimal results. Now, here is the problem: {problem). You are supposed to refer to the comments given by your colleagues from other aspects: (comments)
14

---

Published as a conference paper at ICLR 2024
A.2.2 THE CONDUCTOR
The role of conductor is highly specialized and significant, which necessitates more intricate prompt design compared to other experts. The following is a prompt template for conductor:
You are a leader of an expert team in the field of operations research. Now, You need to coordinate all the experts you manage so that they can work together solve a problem.
Next, you will be given a specific OR problem, and your goal is to select the expert you think is the most suitable to ask for insights and suggestions.
Generally speaking, the solution of complex OR problem requires analysis, in- formation extraction, modeling and programming to solve the problem. The de- scription of problem is presented as follows: {problem}
Remember, based on the capabilities of different experts and the current status the problem-solving process, you need to decide which expert to consult next. The experts' capabilities are described as follows: (erperts fn Experts that have already been commented include: [commented experts] REMEMBER, THE EXPERT MUST CHOOSE FROM THE EXISTING LIST ABOVE
Note that you need to complete the entire workflow within the remaining fremaining [tttiiiinnsssss steps.
Now, think carefully about your choice and give your reasons.
A.2.3 THE REDUCER
The Reducer's role is serve summarizer for all comments provided by the selected experts. They must meticulously analyze the comments and generate the final answer, which can take various forms, such as modeling program. The Reducer's prompt template may vary based on the specific type of final answer required. Here's 's an example of the Reducer's 's prompt template when the goal is to obtain program:
You have been assigned the critical task of generating a program to solve the complex operations research problem presented. This program should incorporate the insights and suggestions provided by the selected experts. Your role is to synthesize the information effectively to create a functional program. The program is described as follows: {problem}
The comments from other experts are as follows: {comments}
Could you please write Python GUROBI code according to the comments.
A.3 BASELINES' IMPLEMENTATION
To ensure a fair comparison, we have implemented the bascline algorithms following their original
papers' guidelines.
The traditional model, tag-BART, typically requires a training process. If we were to directly use pretrained tag- BART model from the LPWP dataset to test the ComplexOR dataset, there would likely be a domain shift. To mitigate this issue, we can adopt a two-step approach. First, pretrained the tag-BART model on the LPWP dataset. This initial pretraining enables the model to acquire basic NER abilities. Next, we fine-tune the pretrained model on an additional set of 30 problems that similar the ComplexOR problems. These problems have the format as the LPWP dataset. By fine-tuning the model on this specific set of problems, we aim to maximize the performance and adapt the model to the requirements of the ComplexOR domain.
annotated
For the Standard prompting technique, we leverage the in-context learning ability of the language model. Following the recommended approach outlined in the OpenAl official documentation, we design the following prompt template:
You are a Python programmer in the field of operations research and optimization. Your proficiency in utilizing third-party libraries such Gurobi is essential. In
15

---

Published as a conference paper at ICLR 2024
addition to your expertise in Gurobi, it would be great if you could also provide some background in related libraries or tools, like NumPy, SciPy, or PuLP. You are given specific problem. You aim to develop an efficient Python program that addresses the given problem. Now the origin problem is as follow: follow:prrollem] (problem).
Give your Python code directly.
technique similar to standard prompting but with some additional steps. begins with the sentence "Let's 's think step by step" to guide the model's s thought process. After that, a further summarization step added because the output generated by Chain-of-Thoughh can be lengthy and fragmented.
Chain-of-Thoughh
For Freeof--Thoughts and Graph-of-Thoughee we set the parameters based on the experiments conducted in the respective papers. The width of exploration is set to 3, and the maximum explo- ration step is set to 5. We adopt the prompt paradigm proposed the work by Tree-of-Thoughts Prompting (Hulbert, 2023). The prompt is designed follows, where (ezploration ste-ppropptppppppp represents the original prompt used in each exploration step:
Imagine three different experts in the field of operations research and optimization are modeling and writing programmer for hard problem.
All experts will write down 1 step of their thinking, then share it with the group. Then all experts will go on to the next step, etc. If any expert realises they're re wrong any point then they leave. The problem description is: {problem} cexploration_stepp-pee
For Progressive-Hint Prompting, the original implementation may not be suitable for complex OR problems. In the original paper, the answer an immediate number, which makes easy to determine consistency between multiple interactions. However, in complex OR problems, the answer can a model or a program, both of which are not directly comparable. To address this, we follow the underlying idea of Progressive- Hint Prompting and make some modifications. We generate an initial answer and then use an additional interaction with the language model ask whether the current answer is the same the previous one. In this way, the PHP algorithm is implemented in a more appropriate manner.
In the ReAct approach, there are two main steps: reasoning and acting. In the reasoning step, use the same prompt in CoT to guide the model's thought process. In the acting step, we limit the actions to retrieving knowledge from a knowledge base. This because, in complex OR problems, accessing and utilizing external knowledge is crucial for making informed decisions. To ensure a fair comparison, we allow the ReAct agent to access all the knowledge bases mentioned in Chain-of-Experts.
The design of Reflexion aligns with the backward reflection process described in Chain-of-Exxe -f-Exxsss. In Reflexion, feedback is obtained from the compilation and runtime of the modeling program, allowing for iterative refinement of the previous steps until the agent is satisfied with the answer. It's worth noting that in our experiment setting, we do not generate test units.
A.4 MORE EXPERIMENTAL RESULTS
A.4.1 DETAILED EXPERIMENT RESULT ON COMPLEXOR
Table presents detailed overview of the performance of baseline algorithms and the Chain-of- Experts approach applied to the ComplexOR dataset. In the interest of brevity, we employ abbre- viations, referring to the main content of the traditional algorithm as "BART". Additionally, we use shorthand labels for various algorithms. The results highlight the challenges faced by both traditional algorithms, such as BART, and prompting techniques like CoT. as they were unable to successfully address all the problems in the dataset. Notably, GoT achieved success in solving the relatively straightforward "Blending" problem. Among the methods employing Large Language Models agents, which include ReAct and Reflexion, several less complex problems were solvable,
16

---

Published as a conference paper at ICLR 2024
Table Detailed experiment results for different methods on ComplexOR
problem
BART
Standard
CoT
PHP
ToT
GoT
ReAct
Reflexion
SPP
CoE
Blending
X
X
X
Car Selection
X
X
X
X
Capacitated Warehouse Location
X
X
X
X
X
X
X
Employee Assignment
X
X
X
X
X
Aircraft Landing
X
X
X
X
X
X
X
X
X
X
VRPTW Routing
X
X
X
X
X
J
Flowshop
X
X
X
X
X
X
X
X
X
X
Distribution Center Allocation
X
X
X
X
X
X
Aircraft Assignment
X
X
X
X
X
X
X
X
Traffic Equilibrium
X
X
X
X
Robot Arm
X
X X X
X
X
X
X
Largest Small Polygon
X
X
X
X
X
CFLP
X
X
X X
X
Cut Problem
X
X
X
X
X X X
Diet Problem
X
X
X
X X
Dietu Problem
X
X
X
X
Knapsack
X
X
X
X X
X
Multi-commodity Transportation
X
X
X
X
X
X
PROD
X
X
X
X
Single Level Big Bucket
X
X
X
X
X
X
X
X
X
V
Overall
0/20
0/20
0/20
0/20
0/20
1/20
3/20
2/20
0/20
7/20
overall performance remained suboptimal. The Chain-of- Experts (CoH) approach demonstrated the highest success rate, solving out of 20 problems, making it the most effective algorithm.
We also found that certain algorithms, particularly PHP and SPP, struggled with token limitations when confronted with complex problems. This token limitation issue not only hindered their perfor- mance but also incurred increased computational costs and inefficiencies in their execution. con- trast, the Chain-of-Experrt approach incorporates three key strategies to mitigate token limitation- related errors. Firstly, the utilization of expert summaries significantly reduces memory stress by nearly 50%. Secondly, the adoption of a Conductor, opposed to round-robin approach, further reduces the maximum context, effectively halving the token usage. Lastly, the incorporation of a visible map led to substantial reduction of approximately 70% in token consumption.
A.4.2 ABLATION EXPERIMENT ON INDIVIDUAL EXPERT
6.90
3.58
6551
0.553
0159
Areray
0555 IL300 0350
$.250
0,250
0.355
0.133
0,200
0.200
0.50
Eeepete
Baazar Bxtaoo
Vanall tedaccoeoooo Iocaacio
Oqeedive Faacsio Iecacco
LP Fiee Goarrti
Connrsee trakiity Check
Trigiening Prightuatie
Temibiigg Irqpere
Westh Demeeeio
Medkine Karosted teteee Sumplenee CT
coerraie sastbii Chec
Comprrmnn th Provide
Prigeneiing
Codi Rovve
Oqeedive innciin Ra
Modting Kaooteege Saap
Constakke
(a) Expert ablation experiment LPWP dataset
(b) Expert ablation experiment dataset
ComplexOR
Figure 5: Impact accuracy when removing individual experts from CoE
Figure 5 presents the results of the ablation experiment conducted each expert in the Chain-of- Experts framework. The x-axis labels represent the removal of specific experts from CoE, while the y-axis represents the accuracy achieved after removing each expert. The blue horizontal line
17

---

Published as a conference paper at ICLR 2024
represents the accuracy when all experts integrated. Based on the results, we observe the following insights regarding the importance of each expert for both datasets.
In the subfigure 5a, which corresponds the LPWP dataset consisting of easy problems. removing a single expert does not lead to a significant performance drop. However, the most crucial expert is the Programming Expert. This finding aligns with the nature of the LPWP dataset, where the final evaluation is based on the correctness of the program. Therefore, having an expert who can provide insights into programming essential. The second important expert is the Modeling Expert, as mathematical modeling plays a crucial role in problem-solving.
Example Problem
context of manufacturing planning, we tackle Capacitated Multi-level Sizing Problem Backlogging, We make the following assumptions in defining and formulating this problem. First, we assume that setup times and costs are non sequence dependent, setup carryover between periods not permitted, and zero. Second, production costs assumed to linear production output and not vary over time; hence, they can be dropged from the model for simplicity. Setup and holding costs also are assumed not to vary over time, Furthermore, end items are assumed have no succesors, and only end items have external demands and backlogging costs. Finally, to note that all these assumptions (except setup carryover) are made for ease of exposition only and without loss of generality, le., theoretical results remain even when they removed. See Ozturk Ornek (2010) for lot-sizing problem with setup carryover well with external demands for component
initial inventories
lead times
lost sales. important
Standard Prompting Result
Sets: Parins, M, L,ed,, eta
Parameters: br, lic, a, M
Variables: x, y.y
X
Missing variable
Constraints:
invBalance Xi. Siz-1 big - biges gdu +xy for i end,t Periods
invBalance2: xig siz-1 = gdd. Lijeen Tu -1, siz for
end, Periods
capCetstraings Yier Rim -Kie Eig. Yor <= Cayt for E M, t Periods Missing constrains
sctupConstrainh Xiz <- Mn Yir E 1,t Pertods
Objoctive: minimize Eiencereriaas (5G. Vic hG,
X
Missing item
SSP Result
Personas Manufaactn Planner. Demand Planner. Data Analysts, Communication Specialists
Manufacturing Planner: address this problem effectively. enscial to carefully consider production
quantitics and schodules while adhering the defined assumptions... as lead times and no lost sales.
Demand Planner: In this context, the Demand Planner critical forecasting external...
Data Analysts: My recommandations intrumental helping the Manufacturing Planner make informed...
Communication Specialists: Edfective communication is key to solving Capacitated Multi-level Sizing Problem successfully, Commonication specialists facilitate collaboration...
Result: Variables: s, b. Constrains: ... Stp 0, Dur Objective: minimize [se Yiz hi. See
Conductor
CoE Result
Terminology Interpreter: In this context, I. No Setup Carryover: Setup carryvver would mean that
2. Initial Inventorics Zero means ....: 3. Zero Lead Times and No Lost Sales: Lead time is the delay...
Variable Extractor: Sets:Perios, Meneeeett Purameters:ac.be. he, st, a ...: Variables: x,s, x,s, b,
Modeling Expert: Variables: x, b, bo
Constrains: ... $t,0
bir
Objective: minimize
E(sc Yu Si
Programmer: LP file this modeling: Minimize: obj...: Subject to: inv in-Balancel(id): x(ia] + sfi + b[u] bbjid-1|= gdlial + sli for all end, Periods
Run the file and get modeling unsolvable crror. Start backward reflection to adjust modeling...
Variables: x.s, b,
Constrains: ... Stg byr
Objective: minimize
E(sq Yur + Rqj 912) + Ziemit bqj big
Figure 6: Case study
18

---

Published as a conference paper at ICLR 2024
Subfigure 5b shows that individual experts have much more significant impact problems. Apart from the Programming Expert and Modeling Expert, the removal of the Terminol- ogy Interpreter leads to significant drop of approximately 20% in accuracy. This result highlights the knowledge-intensivee nature of the ComplexOR dataset, which heavily relies on the supplementa- tion of external knowledge. Interestingly, the LP File Generator Expert also proves to be important. This finding suggests that for harder problems, utilizing LP files as an efficient intermediate struc- tural file for modeling is good approach, it yields better results compared writing Python Gurobi program files.
challenging
A.4.3 CASE STUDY
In this experiment, we conducted a detailed case study to gain insights into the effectiveness of approach, depicted in Figure 6. To reduce the uncertainty of sampling results, methods five times and based our findings on the majority response. In this result, the standard prompting approach failed correctly identify the variable be as backlogging cost, due insuf- ficient knowledge about the Multi-level Lot Sizing Problem. And there was lack of constraints regarding initial and end statuses, which are essential in the context of real manufacturing process constraints. Moreover, the objective function was deemed unsolvable due to missing critical items. The SSP provided some basic background knowledge through a Manufacturing Planner created by leader persona. However, this method had limitations: three out of four personas offered negligible assistance in solving the problem, which a critical issue in domain-specifte problems like OR. In CoE, the Conductor effectively selected appropriate experts for different stages of the problem- solving process. Initially, a Terminology Interpreter was chosen to provide essential background knowledge. Although the Modeling Expert initially repeated the same mistake regarding the objec- tive function, this error was rectified in the backward reflection process.
each
19


Reviews:
[{"review_id": "P1AxOxL82C", "reviewer": "Reviewer_CHCN", "summary": "This paper focuses on utilizing large language models (LLMs) to address operations research problems. It employs an approach where LLMs role-play as agents in the problem-solving pipeline, collaboratively breaking down and resolving problems. The paper also incorporates external feedback for the backpropagation reflections in the problem-solving pipeline, allowing the LLMs within the pipeline to self-improve. Moreover, the research introduces a new operations research dataset, which appears to be more intricate compared to existing ones. The proposed approach is tested on the newly-created dataset as well as another benchmark, and results indicate that it outperforms used baseline prompting methods.", "questions": "Why can't the method proposed in this paper be represented through Solo Performance Prompting, and where exactly does it differ from Solo Performance Prompting? From the description, it seems that the approach is entirely representable under the Solo Performance Prompting framework.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Overall, the methodology presented in this paper is straightforward, easy to implement, and demonstrates strong empirical results across two benchmarks.\n2. The paper offers a new operations research dataset that, based on experimental outcomes, is more challenging than existing ones.\n3. I find the mechanism of propagating feedback from external sources to enhance the performance of language models both innovative and interesting. The results suggest that this mechanism also boosts model performance.", "weaknesses": "1. While the paper focuses on tackling complex operations research problems, it doesn't seem to introduce any techniques specifically tailored for operations research challenges.\n2. I believe the novelty of this work is somewhat limited, as several studies have already explored the \"planning with feedback\" approach with LLMs. Please refer to \"A Survey on Large Language Model based Autonomous Agents (https://arxiv.org/pdf/2308.11432.pdf)\" for more details. I think the authors should offer a more in-depth comparison with these existing works. Moreover, though the methodology is described as a multi-expert framework, it essentially relies on deploying various prompts to the same LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WVN4KRyJVv", "reviewer": "Reviewer_B8wx", "summary": "This paper presents a multi-agent reasoning method for operations research problems solving. In particular, all the expert agents called in a sequence by another conductor agent, and all the agents are based on LLMs, acting different roles. The approach (named Chain-of-Experts) achieves better results compared with other SOTA models on the LPWP dataset and they also release a new dataset on complex OR problems.", "questions": "* If we use other less competent LLMs, like smaller models or open sourced models, how much the performance will be affected?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Propose a multi-agent method for OR problem solving with one conductor and multiple experts; and achieves better empirical results\n* Release a dataset on complex OR for the community", "weaknesses": "* Lack of evaluation on individual expert agents, as well as the conductor\n* The comparison with other models might not be fair, since they call the LLM differently. Maybe add some measurements of how different methods use the LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4NE0dF4ZQ5", "reviewer": "Reviewer_ocDx", "summary": "This paper proposes chain of experts (CoE), a framework that uses multiple LLM agents to solve operations research (OR) problems. Most complex OR problem requires coordination among multiple experts, each solving a subproblem. In CoE, these experts are implemented using specialized LLMs augmented with, e.g., knowledge bases or reasoning skills targeting the subproblems they are designed to solve. A separate conductor model orchestrated this coordination process. This framework is further augmented by a backward reflection process, that, conditioning on the feedback provided by the program execution environment, recursively runs backward to identify potential errors in the chain. CoE does not require updating the parameters of the LLM agents, and thus is applicable to both proprietary and open-source models.\n\nCoE is evaluated on LPWP (elementary linear programming problems), and complexOR (a newly created dataset by the paper, containing 20 expert-annodated OR problems). Experiments with GPT-3.5, GPT-4, and Claude-2 suggest that CoE outperforms baselines. An ablation analysis quantifies the contribution of each design choice in CoE.", "questions": "- ComplexOR is very small. Can the authors provide more details on the consistency of the results across multiple runs?\n- It would be interesting to compare to a baseline that applies CoE, and uses the same model to play all the different roles.\n- Eq. 3 reads like applying the LLM to the prompt template outputs a new set of parameters, which does not align with what happens with prompting. At a higher level, do we really need the $\\theta$ notations in Eqs. 2 and 3?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- CoE is an interesting and novel framework for solving complex problems with multiagent collaboration.\n- CoE’s design is grounded in real-world applications and proves effective.\n- Requiring no training, CoE is applicable to both open-source and proprietary models.\n- The presentation is reasonably clear.", "weaknesses": "- The paper would be more interesting to the LM community and have a larger impact if it could test out CoE on some of the well-established benchmarks\n- ComplexOR is very small; I wonder how significant the results are\n- The paper does not provide enough details on how the experts are specialized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yenXIqqOr0", "reviewer": "Reviewer_suKy", "summary": "This work investigates solving complex operations research problems via the cooperation of multiple LLM-based agents.\nThe authors suggest Chain-of-Experts, which is a multi-agent framework comprised of 11 experts, for different aspects, and a conductor to coordinate these experts.\nThe experts are powered by common techniques such as In-context Learning and Reasoning based on LLMs.\nThe CoE framework sovle OR problems in an iterative way, where failed answers will get feedback via the reflection step.\nThis workflow will stop when the answer passes the evaluator or the iteration exceeds the given number.\nA new benchmark, ComplexOR, is contributed to evaluate on 20 complex OR problems.\nExperiments on LPWP and ComplexOR demonstrates that the proposed CoE outperforms previous LLM-agent methods.", "questions": "1. Is the CoE suitable for other reasoning tasks? What is the difference if applied to other tasks?\n2. I suggest the paper give more attention to the CoE framework.\n\nThe answers have already addressed our questions.  Thanks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the CoE framework.\n2. A combination of existing techniques to solve OR problems.\n3. A new small-scale real-world dataset", "weaknesses": "1. The results on ComplexOR seem not sense. A too small dataset.\n2. The description of  CoE is not clear. It should be well-moviated and started with several backgrounds.\n\nThrough the response,  indeed the construction of ComplexOR is very difficult, and the authors acknowledge that the dataset will continue to be updated, which could be a potential contribution to the field and answer our questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)