AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversations
Authors: Qingyun Wu
Gagan Bansal
Jieyu Zhang
Yiran Wu
Beibin Li
Erkang Zhu
Li Jiang
Xiaoyun Zhang
Shaokun Zhang
Jiale Liu
Ahmed Hassan Awadallah
Ryen W White
Doug Burger
Chi Wang
Venue: COLM (2024)
Keywords: Large Language Model, Multi-Agent Conversation Framework, Open-Source Library
TL;DR: AutoGen is an open-source framework that allows developers to build LLM applications using multiple conversable agents and conversation programming. Experiments show benefits on diverse domains.

Abstract: We present AutoGen, an open-source framework that allows developers to build LLM applications by composing multiple agents to converse with each other to accomplish tasks. AutoGen agents are customizable, conversable, and can operate in various modes that employ combinations of LLMs, human inputs, and tools. It also enables developers to create flexible agent behaviors and conversation patterns for different applications using both natural language and code. AutoGen serves as a generic infrastructure and is widely used by AI practitioners and researchers to build diverse applications of various complexities and LLM capacities. We demonstrate the framework’s effectiveness with several pilot applications, with domains ranging from mathematics and coding to question-answering, supply-chain optimization, online decision-making, and entertainment.

Published as a conference paper at COLM 2024
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversations
Qingyun Wu
Gagan Bansal Microsoft Research
Jieyu Zhang
Penn State University qingyun@autogen. team
University of Washington jieyuz2@cs washington. edu
gaganbansallmmicoosoft... com
Yiran Wu
Beibin Li
Erkang Zhu Microsoft Research erkang wtw.eeiiirroot..om com
Penn State University yiran. wuêpsu. edu
Microsoft Research
beibin. cemmirrooott.com com
Li Jiang Microsoft lijiang 10mgg1@micmm wem.crosoft.cmm
Xiaoyun Zhang Microsoft xi xiaoyun. www.anemicoooottttoooooooooo
Shaokun Zhang
Penn State University shaokun. zhwng@psu..n..omo edu
Jiale Liu
Ahmed Awadallah
Ryen W. White
Penn State University 1117199@psu.eet edu
Microsoft Research
Microsoft Research
hassanam@mi crosoft.com
ryenwemi crosoft.com
Doug Burger Microsoft Research dburgerrmicrosofft.com
Chi Wang
Microsoft Research chi@autogen. team
Abstract
We present AutoGen, 1 an open-source framework that allows developers build LLM applications by composing multiple agents to converse with each other to accomplish tasks. AutoGen agents are customizable, conversable, and can operate in various modes that employ combinations of LLMs, human inputs, and tools. also enables developers to create flexible agent behaviors and conversation patterns for different applications using both natural language and code. AutoGen serves as a generic infrastructure and is widely used by AT practitioners and researchers to build diverse applications of various complexities and LLM capacities. We demonstrate the framework's effectiveness with several pilot applications, on domains ranging from mathematics and coding to question-ansswriigg, supply-chain optimization, online decision-making, and entertainment.
Introduction
Large language models (LLMs) are becoming crucial building block in developing power- ful agents that utilize LLMs for reasoning, tool usage, and adapting to new observations (Yao et al., 2022; et al., 2023; Wang al., 2023). As the scope and complexity of tasks suitable for LLMs increase, natural strategy for enhancing agent capabilities is to employ multiple cooperating agents. Prior work suggests that multiple agents can help encourage divergent thinking (Liang et al., 2023), improve factuality and reasoning (Du et al., 2023; Naik et al., 2023), and provide guardrails (Wu et al., 2023). Given the early promising evidence, an intriguing question is: How can facilitate the development of LLM applications that span a broad spectrum of domains and complexities using multi-agent approach? Our insight
"Equal contribution
'https://githuu http://gghhuu.ocoomtrrroo/aaaaaa

---

Published as a conference paper at COLM 2024
Pltt META
Output
and IESLA steck change TD.
the
following sodk...
Agent Customization
Multi-Agent Conversations
Error package yfrcncc
changel
Here is reviedd code
Please and than the code
Outputt
Instuling.
Sequential Chat
Nested Chat
Group Chat
Hierarchical Chat
Morch
Flexible Conversation Patterns
Example Agent Chat
Figure 1: AutoGen enables diverse LLM-based applications using multi-agent conversations. (Top-left) AutoGen agents are conversable, customizable, and can be based on LLMs, tools, humans, or even a combination of them. (Top-middle) Agents can converse to solve tasks. (Right) They can form chat, potentially with humans in the loop. (Bottom-left) The framework supports flexible conversation patterns.
is to use multi-agent conversations. There are at least three reasons confirming its general feasibility and utility, thanks to recent advances in LLMs: First, chat-optimized LLMs, such as GPT-4, demonstrate ability to incorporate feedback. LLM agents can cooperate through conversations with each other or humans, for example, in a dialogue where agents provide and seek reasoning, observations, critiques, and validation. Second, because single LLM can exhibit broad range of capabilities, conversations between differently configured agents can help combine these broad LLM capabilities in a modular and complementary manner. Third, LLMs have demonstrated the ability to solve complex tasks when broken into simpler sub-tasks. Multi-agent conversations can intuitively facilitate this partitioning and integration.
We desire multi-agent conversation framework with generic abstraction and effective implementation that has the flexibility to satisfy different application needs. Achieving this requires addressing two critical questions: (1) How to design individual agents that are capable, reusable, customizable, and effective in multi-agent collaboration? (2) How can we develop a straightforward, unified interface that accommodates a wide range of agent conversation patterns? In practice, applications of varying complexities may need distinct sets of agents with specific capabilities and may require different conversation patterns, such single- multi-turn dialogues, different human involvement modes, and static vs. dynamic conversations. Moreover, developers may prefer the flexibility to program agent interactions in natural language code. We present AutoGen, generalized multi-agent conversation framework (Figure 1), based on the following new concepts:
1 Customizable and conversable agents. AutoGen uses a generic agent design that can leverage LLMs, human inputs, tools, or a combination thereof. Developers can conve- niently create agents with different roles or responsibilities by selecting and configuring subset of built-in capabilities or defining new capabilities. To make these agents suitable for multi-agent conversation, every agent is made conversable they can receive, react, and respond to messages. When configured properly, an agent can hold multiple turns of conversations with other agents autonomously conversable agent design leverages the strong capability of the most advanced LLMs in taking feedback and making progress via conversation, and also allows combining capabilities of LLMs modular fashion. (Section 2.1)
with humans in the loop. The
2 Conversation programming. A fundamental insight of AutoGen is to simplify and unify complex LLM applications as multi-agent conversations. Thus, AutoGen adopts a pro- gramming paradigm centered around these inter-agent conversations. We refer to this paradigm as conversation programming, which streamlines the development of intricate applications via two primary steps: (1) defining set of conversable agents with specific capabilities and roles; (2) programming the interaction behavior between agents via conversation-centric compulation and control. Both steps can be achieved via a fusion of
2

---

Published as a conference paper at COLM 2024
natural and programming languages. AutoGen provides ready-to-use implementations and also allows easy extension and experimentation for both steps. (Section 2.2)
We offer a suite of multi-agent applications realized with AutoGen, showcasing the frame- work's ability support applications of varied complexities. With these applications, we demonstrate AutoGen's potential to significantly enhance task completion performance and innovate LLM usage while minimizing development effort. Beyond the demonstrated applications, AutoGen has also seen widespread adoption in the wild, fostering a vibrant and active community.
Related Work. Several contemporaneous explorations of multi-agent approaches exist, including Generative Agents (Park et al., 2023), multi-agent debate (Liang et al., 2023; Du et al., 2023), CAMEL (Li et al., 2023b), BabyAGI (BabyAGI, 2023), MetaGPT (Hong et al., 2023), ChatDev (Qian et al., 2023), AgentVerse (Chen et al., 2023b), AutoAgents (Chen et al., 2023a). These systems are designed for specific types of scenarios or problem-solving paradigms, which limits their flexibility and generalizability as comprehensive frameworks. For instance, MetaGPT and ChatDev prioritize software engineering tasks and only sup- port certain multi-agent structures, such as chains or Standardized Operating Procedures. AgentVerse primarily simulates the problem-solving processes of a human group following sequence of pre-defined stages. CAMEL supports multi-agent systems with two or three agents following fixed workflow pattern. One notable difference of AutoGen that supports diverse workflows because of its composable conversation patterns and does not explicitly restrict the number of agents. We include an expanded discussion of this related work and single-agent systems hatps//faaewoossss in Appendix B.
2 The AutoGen Framework
To reduce the effort required for developers to create complex LLM applications across various domains, core design principle of AutoGen is to streamline them using multi- agent conversations. This approach also aims to maximize the reusability of implemented agents. This section introduces the two key concepts of AutoGen: conversable agents and conversation programming.
2.1 Conversable Agents
In AutoGen, a conversable agent is an entity with specific role that can send message to and receive message from the other conversable agents, e.g., start or continue conversation. It maintains its internal context based on sent and received messages and can be configured to possess set capabilities, e.g., enabled by LLMs, tools, human input, etc. The agents can act according to the programmed behavior patterns described next.
Agent capabilities powered by LLMs, humans, and tools. AutoGen allows flexibility to equip its agents with various capabilities, which directly affect how it processes and responds to messages. The built-in composable agent capabilities include: 1) LLMs. LLM- backed agents utilize advanced capabilities such role-playing, implicit state inference, making progress based on conversation history, and coding. These capabilities can be combined and enhanced in different ways via novel prompting techniques?, AutoGen also offers enhanced LLM inference features such result caching, error handling, message templating, etc., via an enhanced LLM inference layer. 2) Humans. Human involvement is desired or even essential in many LLM applications. AutoGen lets a human participate in agent conversation via human-backer agents, which could solicit human inputs at certain rounds of conversation depending on the agent configuration. The default user proxy agent allows configurable human involvement levels and patterns, e.g., frequency and conditions for requesting human input including the option for humans to skip providing input. 3) Tools. Took-backed agents have the capability to execute tools via code execution function
Appendix D presents an example of such novel prompting techniques which empowers the default ILM backed assistant agent in AutoGen converse with other agents in multi-step problem-solving.
3

---

Published as a conference paper at COLM 2024
SrrversableAgent
Agent Customization:
Unified Conversation
input "EVER* code_ wwwcutien.ccoeeeeee config False DEFAULT MESSAGE *You are helpful At assistant. In the following cases, Suggest python code."
generateereply
indut__ aroup
AutoGen Agents
hundn, Input,
Assistantágent
IkerProxyAgent
GroupChatMonager
1.2 Register a Custom Reply Func:
1.1 Define Agents:
This func will invoked in gererata_ _riply
A.register-nng ster-reply(i, roply. runc A2B) def reply_ FunccAAR((rss))
fune stered, list of defauls reply functions will be used.
Developer Code
input: hunan()
Proxy
Assistant B
iF not Cupat:
includes code: output executa(rse)
Initiate Conversations: Annitiote_chat(P1 chart of NETA and TESLA stock price change YID.",
The Resulting Automated Agent Chat:
Conversation-Driven Control Flow
Plot chart META and
receive
generate_reply
TESLA stock price change YTD.
receive
Execute the following code...
send
Program Execution
generate_repp
Error: package yfinance installed
generate_ reply
send
Conversation-Centiic Computation
Sorryl Please pip install yfinance and then execute
Figure 2: Illustration of how to use AutoGen to program multi-agent conversation. The top sub-figure illustrates the built-in conversable agents provided by AutoGen. The middle sub-figure shows an example of using AutoGen to develop a two-agent system with custom reply function. The bottom sub-figure illustrates the resulting automated agent chat from the two-agent system during program execution.
execution. For example, the default user proxy agent in AutoGen is able to execute code suggested by LLMs, or make LLM-suggested function calls.
Agent customization. Based application-speeific needs, each agent can be configured to have mix of basic back-end types to exhibit complex behavior in multi-agent conversations. AutoGen allows easy creation of agents with specialized capabilities and roles by reusing or extending the built-in agents. The yellow-shaded area of Figure 2 provides a sketch of the built-in agents in AutoGen. The ConversableAgent class the most basic agent abstraction and, by default, can use LLMs, humans, and tools. The AssistantAgent and IlserProyyAgent are two pre-configured ConversableAgent subclasses, each representing a common usage mode, i.e, acting as an Al assistant (backed by LLMs) and acting as human proxy to solicit human input or execute code/ function calls (backed by humans and and/or tools). In the example on the right-hand side of Figure 1, an L.M--backed assistant agent and a tool- and human--backe user proxy agent are deployed together to tackle a task. Here, the assistant agent generates solution with the help of LLMs and passes the solution to the user proxy agent. Then, the user proxy agent solicits human inputs or executes the assistant's code and passes the results feedback back to the assistant. One compose complex agent using nested chat (introduced in the next subsection) among simpler agents and increae the complexity recursively.

---

Published as a conference paper at COLM 2024
2.2 Conversation Programming
To develop applications where agents make meaningful progress on tasks, developers also need to be able to specify and properly control these multi-agent conversations. To this end, AutoGen utilizes conversation programmingg paradigm that concerns two concepts: the first is computation the actions agents take to compute their response in multi-agent conversation. And the second is control flow the order and conditions under which individual computations in the conversation are executed or evaluated. As we will show in the applications section, the ability to program these helps implement many flexible multi- agent conversation patterns. AutoGen, agent computations are conversation-centric. An agent takes actions based on conversations it is involved in, and these actions further lead to message passing for subsequent conversations. Similarly, control flow is conversation- driven the participating agents' decisions which agents to send messages to and the procedure of computation are functions of the inter-agent conversation. This paradigm facilitates intuitive reasoning about complex workflows through actions of agents and message-passing between agents.
Figure 2 provides a simple illustration. The middle sub-figure shows how each individual agent performs its role-specific, conversation-centri computations to generate responses (e.g,, via LLM inference calls and code execution). The bottom sub-figure demonstrates conversation-basel control flow. When the assistant receives message, the user proxy agent generates a reply based on code execution or solicits human inputs. The task progresses through conversations displayed in the dialog box. AutoGen features the following design patterns to facilitate conversation programming.
Unified interfaces and auto-reply mechanisms for automated agent chat. Agents in AutoGen have unified conversation interfaces for performing the corresponding conversation- centric computation. Those low-level interfaces include:
send/receive for cendinng/recivvinn messages; and
generate_reple for taking actions and generating a response based on the received message;
register_reply for registering custom reply function.
AutoGen also introduces and by default adopts an agent auto-reply mechanism realize conversation-driven control: Once an agent receives a message from another agent, it automatically invokes generate_reply and sends the reply back to the sender unless termination condition is satisfied. AutoGen provides built-in reply functions based on LLM inference, code or function execution, or human input. One can also register custom reply functions (via the register reply interface) to customize the behavior pattern of agent, e.g., to chat with another agent before replying to the sender agent realizing the nested chat conversation pattern. Under this mechanism, once the reply functions are registered, and the conversation is initialized, the conversation flow is naturally induced, and thus the agent conversation proceeds naturally without any extra control plane, i.e., a special module that controls the conversation flow. For example, with the developer code in the blue-shaded area (marked "Developer Code") of Figure 2, one can readily trigger the conversation among the agents, and the conversation would proceed automatically, as shown in the dialog box in the grey shaded area (marked "Program Execution") of Figure 2. The auto-reply mechanism provides decentralized, modular, and unified way to define the workflow.
Control by fusion of programming and natural language. AutoGen allows the usage of programming and natural language in various control flow management patterns:
Natural-language control via LLMs: One can control the conversation flow by prompting LLM-backed agents with natural language. For instance, the default system message of the built-in assistantAgent uses natural language to instruct agents to write code and debug when needed. It also guides the agent confine LLM outputs, making it easier for other agents to consume. More examples of such controls can be found in Appendix D.
saggrammiin-laanggaage control: In AutoGen, Python code can used to specify the termination condition, human input mode, and tool execution logic, e.g., the max number
5

---

Published as a conference paper at COLM 2024
of auto replies. One also register programmed auto-reply functions to control the con- versation flow with Python code, as shown in the code block identified as "Conversation- Driven Control Flow" in Figure 2.
Control transition between natural and programming language: AutoGen also supports flexible control transition between natural and programming language. One can achieve transition from code to natural-language control by invoking an LLM inference containing certain control logic in customized reply function; or transition from natural language to code control via LLM-proposed function calls (Eleti et al., 2023).
Composable conversation patterns. The conversation programming paradigm enables the composition of multi-agent conversations with diverse patterns, both statically and dynami- cally. For enhanced usability, provide interfaces for constructing several commonly used conversation patterns, including two-agent chat, sequential chats, nested chat and group chat. We provide the detailed interfaces for specifying these patterns in Appendix D. Beyond these built-in patterns, one can employ these higher-level interfaces and the low-level interfaces such pefiisterrrplly necessary recursively to compose more complex and creative patterns, e-g., nested chat with group chat nested within, allowing one agent to create its inner monologue, realizing the Society of Mind idea from Minsky (1988). The composed conversation workflow can be static or dynamic. AutoGen provides few general ways to achieve dynamic conversation flows: 1) custom reply functions and triggers. Nested chat and group chat are examples of conversation patterns using built-in custom reply functions. In nested chat, one agent can hold the current conversation while invoking conversations with other agents depending chat, one can define the speaker transition conditions based on the current conversation status. 2) LLM-driven function calls, in which language model decides whether or not to call a particular function depending on the conversation status.
the content of the current message and context. In group
3 Applications of AutoGen
We demonstrate six applications using AutoGen (see Figure 3). These application are selected based on their real-world relevance (A1, A2, A4, A5, A6), problem difficulty (A1, A2, A3, A4), and innovative potential (A5, A6). Together, these criteria showcase AutoGen's 's role in advancing the LLM application landscape. All the applications and their evaluations presented in this section are based on code version 0.1.1 of the library, with code and instructions provided in the Reproducibility Statement section at the end of this paper.
Student Assistant
Assistant
Retrieval-augmented
Retrieval-augmentee
User Proxy
Assistant
Ask expert
Execulor
Expert
Assistant
Grounding Agent
A1. Math Problem Solving
A2. augmentd Q&A
Decision Making in Embodied Agents
Commander
Chess Board
Manager
Broadcast
Speak
Human/Al Chess Player
HumenvAl Chess Player
Writer
Safeguard
A4. Supply-Chain Optimization
Dynamic Task Solving Group Chat
A6. Conversational Chess
Figure 3: Six examples of applications built using AutoGen. These applications demonstrates AutoGen's flexibility in supporting diverse applications with flexible conversation patterns.

---

Published as a conference paper at COLM 2024
120 Level-5 problems
RAGEbat (ute Autslie))
Whole Dataset
RAGChot
66. 80%
Success Rakio (%6
50.50%
32.59
42
10.0%
25,880% 22. 79.79%
10-
(+b Auto0en
ChORGHT
IMuin Methods
Debote
NAC
Recall
Metrics
(a) A1: Performance on MATH (w/ GPT-4).
(b) A2: Q&A tasks (w/ GPT--355.
Average
Sinle-GPPa
Best
98.00%
M.NN-GFTR
SingleSPPT
69%
63%
68%
Per Percencage (%)
83.00%
2.0%%
54%
54%
Success
48 02%0
22.00%
28-
3. agggnt chot. (via Auto
agent chat (vis AutoGon) Methods
RoAct
F1
Recall
Metrics
(c) A3: Performance ALFWorld
(d) A4: Performance OptiGuide.
Figure 4: Performance on four applications A1-A4. (a) shows that a simple two-agent system with built-in agents from AutoGen can be used out of the box to achieve the most competitive performance on math problem solving tasks; (b) shows that AutoGen can be used to realize effective retrieval augmentation and realize novel interactive retrieval feature to boost performance on Q&A tasks; (c) shows that AutoGen can be used to introduce three-agent system with a grounding agent improve performance on ALFWorld; (d) shows that multi-agent design is helpful in boosting performance in coding tasks that need safeguards.
A1: Math Problem Solving
Mathematics foundational discipline and the promise of leveraging LLMs assist with math problem solving opens up new plethora of applications and avenues for exploration, including personalized AI tutoring, Al research assistance, etc. This section demonstrates how AutoGen can help develop LLM applications for math problem solving, showcasing strong performance and flexibility in supporting various problem-solving paradigms.
(Scenario 1) We are able to build a system for autonomous math problem solving by directly reusing two built-in agents from AutoGen. We evaluate our system and several alternative approaches, including open-source methods such as Multi-Agent Debate (Liang et al., 2023), LangChain ReAct (LangChain, 2023), vanilla GPT-4, and commercial products ChatGPT + Code Interpreter, and ChatGPT Plugin (Wolfram Alpha), the MATH (Hendrycks et al., 2021) dataset and summarize the results in Figure 4a. We perform evaluation tions over 120 randomly selected level-5 problems and on the entire test dataset from MATH. The results show that the built-in agents from AutoGen already yield better performance out of the box compared to the alternative approaches, even including the commercial ones. (Scenario 2) We also showcase human-in-the-loog problem-solv incorporate human feedback with AutoGen, one only needs to set human input mode= 'AWWYS' in the UserProxyAgent of the system in scenario 1. We demonstrate that this system can
process with the help of AutoGen. To
We did not evaluate ChatGPT on the whole dataset since it requires substantial manual effort and is restricted by its hourly message-number limitation. Multi-agent debate and LangChain ReAct were also evaluated since they underperformed vanilla GPT-4 on the smaller test set.
7

---

Published as a conference paper at COLM 2024
effectively incorporate human inputs to solve challenging problems that cannot be solved without humans. (Scenario 3) We further demonstrate a novel scenario where multiple human users can participate in the conversations during the problem-solving process. Our experiments and case studies for these scenarios show that AutoGen enables better performance new experience compared to other solutions experimented with. Details of the evaluation, including case studies in three scenarios are in Appendix E.
A2: Retrieval-Augmente Q&A
Retrieval augmentation generation (RAG) has emerged practical and effective approach for mitigating the intrinsic limitations of LLMs by incorporating external documents. We employ AutoGen to build RAG system (Lewis et al., 2020; Parvez et al., 2021) named Retrieval-augmented Chat. The system consists of two agents: a Retrieval- aegreeealaugmmented User Proxy agent and a Retrieval-augmented Assistant agent, both of which are extended from built-in agents from AutoGen. The Retrieval-augmented User Proxy includes a vector database (Chroma, 2023) as the context retriever. A detailed workflow description of the Chat (RAGChat) is provided in Appendix E. We evaluate Retrieval- augmented Chat in both question- -eeeeiion-nnwwriing and oddegeneration scenarios. (Scenario 1) We first perform an evaluation regarding natural question answering on the Natural Questions dataset (Kwiatkowski et al., 2019) and report results in Figure 4b, In this evaluation, we compare our system with DPR (Dense Passage Retrieval) following
detrieval-augment
existing evaluation practice (Adlakha et al., 2023). Leveraging the conversational design and natural-language control, AutoGen introduces novel interactive retrieval feature in this application: whenever the retrieved context does not contain the information, instead of terminating, the LLM- based assistant would reply "Sorry, cannot find any information about... UPDATE CONTEXT." which will invoke more retrieval attempts. We conduct an ablation study in which we prompt the assistant agent to say "I idon't know" instead of "UPDATE CONTEXT." in cases where relevant information is not found, and report results in Figure 4b. The results show that the interactive retrieval mechanism indeed plays a non-trivial role in the process. We give a concrete example and results using this appealing feature in Appendix E. (Scenario 2) We further demonstrate how netrieval-augmenter Chat aids in generating code based on given codebase that contains code not included in GPT-4's training data. Evaluation and demonstration details for both scenarios are included in Appendix E.
A3: Decision Making in Embodied Agents
In this subsection, we demonstrate how AutoGen can be used to develop effective applica- tions that involve interactive or online decision-making, We perform the study using the ALFWorld (Shridhare et al., 2021), diverse collection of synthetic language-based interactive decision-making tasks household environments. With AutoGen, we implemented two- agent system for ALFWorld. It consists of an LLM-backed assistant agent for suggesting plans to conduct a task and an executor agent for executing actions. This system integrates the ReAct prompting (Yao et al., 2022) and could achieve similar performance. A common challenge encountered in both ReAct and the AutoGen-basse based two-agent system is their occa- sional inability to follow basic commonsense knowledge about the physical world, therefore getting stuck with repetitive errors. Fortunately, the modular design of AutoGen allows us to address this issue effectively: With AutoGen, we introduce a grounding agent, which supplies crucial commonsense knowledge-such as "You must find and take the object before you can examine it. You must go to where the target object is before you can use it." "wwhneverr the system exhibits early signs of recurring errors. It significantly enhances the system's 's ability to avoid getting entangled in error loops. We compare the task-solving performance of the two variants of our system with GPT-3.5-turbo and ReAct on the 134 unseen tasks from ALFWorld and report results in Figure 4c. The results show that introducing grounding agent leads to 15% performance gain on average. Upon examining the systems' outputs, we observe that the grounding agent, by delivering commonsense knowledge at the right junctures, significantly mitigated the tendency of the system to persist with flawed plan, thereby avoiding the creation of error loops. For an example trajectory comparing the systems see Appendix F, Figure 10.
8

---

Published as a conference paper at COLM 2024
A4: Supply-Chain Optimization
In this subsection, we use AutoGen to build a multi-agent coding system based on OptiGu- ide (Li et al., 2023a), a system that excels at writing code to interpret optimization solutions and answer user questions, such as exploring the implications of changing a supply-chain decision or understanding why the optimizer made a particular choice. The second sub- figure of Figure 3 shows the AutoGen-based implementation. The workflow is as follows: the end user sends questions, such as "What if we prohibit shipping from supplier 1 to roastery 2?" to the Commander agent. The Commander coordinates with two assistant agents, including the Writer and the Safeguard, to answer the question. The Writer is responsible for crafting code and the Safeguard is responsible for checking the code safety. With AutoGen the core workflow code for OptiGuide was reduced from over 430 lines to 100 lines, leading to significant productivity improvement. We provide detailed comparison of user experience with ChatGPT+ + Code Interpreter and AutoGen-basen OptiGuide in Appendix E, where we show that AutoGen-basee based OptiGuide could save around 3x of user's 's time and reduce user interactions by 3 5 times on average. We also conduct an ablation showing that multi-agent abstraction is necessary. Specifically, we construct single-agent approach where a single agent conducts both the code-writing and safeguard processes. We tested the single- and multi-agent approaches on dataset of 100 coding tasks, which is crafted to include equal numbers of safe and unsafe tasks. Evaluation results as reported in Figure 4d show that the multi-agent design boosts the F-1 score in identifying unsafe code by 8% (with GPT-4) and 35% (with GPT-3.5-turbo), and the recall by 20% (with GPT-4) and 40% (with GPT-3.5-turbo).
A5: Dynamic Task Solving with Group Chat
There are many real-world scenarios that require dynamic task solving, meaning that the exact workflow cannot be predetermined due to the complex nature of the tasks. For example, tasks involving coding and web scraping. The group chat conversation pattern in AutoGen is desirable pattern in such scenarios. In a group chat, the participating agents share the same context and converse with the others in a dynamic manner instead of following a pre-defined order. Dynamic group chat relies on ongoing conversations to guide the flow of interaction among agents. These make dynamic group chat ideal for situations where collaboration without strict communication order is beneficial. AutoGen, class serves as the conductor of conversation among agents and
the GroupChatManage repeats the following three steps: dynamically selecting a speaker, collecting responses from the selected speaker, and broadcasting the message (Figure 3-A5). For the dynamic speaker-selection component, we use a role-play style prompt. Through a pilot study on 12 manually crafted complex tasks, we observed that compared based on the task, utilizing a role-play prompt often leads to more effective consideration of both conversation context and role alignment during the problem-solving and speaker- selection process. Consequently, this leads to higher success rate and fewer LLM calls. We include detailed results in Appendix E.
prompt that is purely
A6:: Conversational Chess
Using AutoGen, we developed Conversational Chess, natural language interface game shown in the last sub-figure of Figure 3. It features built-in agents for players, which can be human or LLM, and a third-party board agent to provide information and validate moves based on standard rules. With AutoGen, we enabled two essential features: (1) Natural, flexible, and engaging game dynamics, enabled by the customizable agent design in AutoGen. Conversational Chess supports range of game-play patterns, including AI-AI, AT-human, and numan-human, with seamless switching between these modes during single game. An illustrative example of these entertaining game dynamics can be found in Figure 15, Appendix E. (2) Grounding, which is a crucial aspect to maintain game integrity. During gameplay, the board agent checks each proposed move for legality; if a move is invalid, the agent responds with an error, prompting the player agent to re-propose a legal move before continuing. This process ensures that only valid moves are played and helps maintain a consistent gaming experience. As an ablation study, we removed the board agent and instead only relied relevant prompt "you should make sure both you and the opponent
9

---

Published as a conference paper at COLM 2024
making legal moves" to ground their move. The results highlighted that without the board agent, illegitimate moves caused game disruptions. The modular design offered flexibility, allowing swift adjustments to the board agent in response to evolving game rules varying chess rule variants. A comprehensive demonstration of this ablation study is in Appendix E.
Discussion
We introduced an open-source library, AutoGen, that incorporates paradigms of conversable agents and conversation programming. AutoGen also provides various additional supports, including multimodality, asynchronous operations, and enhanced LLM inference. Further- more, AutoGen seamlessly interoperates with numerous single-agent systems, LLM tools, and libraries, such as OpenAl Assistant and MemGPT (Packer et al., 2023). Although still in an early stage, AutoGen is already benefiting a wide range of vertical industries and empowering researchers to build multi-agent AT systems for various scientific studies. For example, AutoGen has been used to realize a multi-agent system for accessing task utility in LLM-powered applications (Arabzadeh et al., 2024). AutoGen has been used in studying behaviors of embodied agents in organized teams (Guo et al., 2024). AutoGen is used for producing synthetic dataset for language model fine tuning (Mitra et al., 2024), or in RL environments to train LLMs for agents (Zhou et al., 2024). AutoGen is also used in diverse science and engineering domains such as mechanics (Ni & Buehler, 2023), protein discovery (Ghafarollahi & Buehler, 2024a), and material design (Ghafarollahi & Buehler, 2024b).
AutoGen also paves the way for numerous future directions and research opportunities. For instance, is worth investigating which strategies, such as agent topology and conversation patterns, lead to the most effective multi-agent conversations while optimizing the overall efficiency, among other factors. While increasing the number of agents and other degrees of freedom presents opportunities for tackling more complex problems, it may also introduce new safety challenges that require additional studies and careful consideration. We consider it important future work to explore those safety implications. We provide expanded discussion in Appendix C, including guidelines for using AutoGen and future work. We welcome contributions from the broader community.
Acknowledgments
The work presented was made possible through discussions with and feedback from a large number of colleagues, including Peter Lee, Johannes Gehrke, Eric Horvitz, Steven Lucco, Umesh Madan, Robin Moeur, Piali Choudhury, Saleema Amershi, Adam Fourney, Victor Dibia, Guoqing Zheng, Corby Rosset, Ricky Loynd, Ece Kamar, Rafah Hosn, John Langford, Ida Momennejad, Brian Krabach, Taylor Webb, Shanka Subhra Mondal, Wei-ge Chen, Robert Gruen, Yinan Li, Yue Wang. Suman Nath, Tanakorn Leesatapornwongsa, Xin Wang, Shishir Tianjun Zhang, Saehan Jo, Ishai Menache, Kontantina Mellou, Runlong Zhou, Feiran Jia, Hamed Khanpour, Hamid Palangi, Srinagesh Sharma, Julio Albinati Cortez, Amin Saied, Yuzhe Ma, Dujian Ding, Linyong Nan, Prateek Yadav, Shannon Shen, Ankur Mallick, Mark Encarnación, Lars Liden, Tianwei Yue, Julia Kiseleva, Anastasia Razdaibiedina, and Luciano Del Corro. Li Jiang would like to thank Jeff Zheng and DJ Lan for their support during this work. Authors would like to especially thank many colleagues from Microsoft Research AI Frontiers who contributed substantially to AutoGen's 's growth, key new features, and reliability, especially Jack Gerrits, Adam Fourney, Victor Dibia, Olga Vrousgou, Rajan Chari, Cheng Tan, and Ricky Loynd. Last but not least, we would like to thank individuals from the open-source community, including but not limited to Davor Runje, Hrushikesh Dokala, Mark Sze, Wael Karkoub, Yulong Zhai, Joshua Kim, Olaoluwa Ademola Salami, Rajan Chari, Maxim Saplin, Ikko Eltociear Ashimine, Aaron Ward, Linxin Song, David Luong, Zvi Baratz, Yixuan Zhai, among others for contributing to and helping maintain the open-source project.
10

---

Published as a conference paper at COLM 2024
References
Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy, Evaluating correctness and faithfulness of instluctoonfoolooinng models for question answering. arXiv preprint ar 2tti:2307..6887,2022
Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi lqbal, Paul N Bennett, Kori Inkpen, et al. Guidelines for human-ai interaction. In Proceedings of the 2019 chi conference on human factors in computing systems, 2019.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016.
Negar Arabzadeh, Julia Kiseleva, Qingyun Wu, Chi Wang, Ahmed Awadallah, Victor Dibia, Adam Fourney, and Charles Clarke. Towards better human-agent alignment: Assessing task utility in ilm-powered applications. ar Xiv preprint ar Xtt:24022000015 2024.
AutoGPT. Documentation auto-gpt. https:///bcc.aagtt.cmm 2023.
BabyAGI. Github- - babyagi. http::///tthh.cccomm/////iaa/aaa 20233
2023.
Carrie J. Cai, Samantha Winter, David F. Steiner, Lauren Wilcox, and Michael Terry. "hello ai": Uncovering the onboarding needs of medical practitioners for human-ai collaborative decision-making. Proceedings of the ACM on Human-Compinn Interaction, 2019.
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. arv preprint arXiv:230..17126, 2023.
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. arXiv preprint 2tt.:230977288 2023a.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent col- laboration and exploring emergent behaviors in agents. arXiv preprint wrXim:2308.10848 2023b.
Weize Chen, Ziming You, Ran Li, Yitong Guan, Chen Qian, Chenyang Zhao, Cheng Yang, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. Internet of agents: Weaving a web of heterogeneous agents for collaborative intelligence. arXiv preprint ar Xiiv:2407.07061 2024.
Chroma. Chromadb. https: //gi thub. www./coooaaaoooooooocccco.cc.aaa wochrrmaacrreecrrooooooo 2023.
Victor Dibia. LIDA: A tool for automatic generation of grammar-agnosta visualizations and infographics using large language models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Toronto, Canada, July 2023. Association for Computational Linguistics.
Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. arXiu preprint ar Xiv:2304.007900 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arviv preprint arXiv:2305.14325 2023.
Atty Eleti, Jeff Harris, and Logan Kilpatrick. Function calling and other api updates. http://nnnttooccacaaaaannnnnonaa apipurrraapppddees,.. 2023.
https://openai.
Alireza Ghafarollahi and Markus J. Buehler. Protagents: protein discovery via large language model multi-agent collaborations combining physics and machine learning. Digital Discovery, 1-1389-1409, 1409,202444 2024a.
Alireza Ghafarollahi and Markus J. Buehler. Atomagents: Alloy design and discovery through physics-aware multi-modal multi-agent artificial intelligence, 2024b.
11

---

Published as a conference paper at COLM 2024
Guidance. Guidance. https://giis /ttp://////uuaaaaeaaa/aagaaanna 2023.
Xudong Guo, Kaixuan Huang, Jiale Liu, Wenhui Fan, Natalia Vélez, Qingyun Wu, Huazheng Wang, Thomas L Griffiths, and Mengdi Wang. Embodied Ilm agents learn to cooperate in organized teams. Xiv preprint ar Xri::2203..244 124822 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ar Xiv preprint ar arXiv:2103.03872 2021.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352, 2023.
Eric Horvitz. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI
conference Human Factors in Computing Systems, 1999.
HuggingFace. Transformers agent. http:://wuuginffccc...oooottttta
transformers.agents. 2023.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Xiv preprint arXiv:2303.1799, 2023.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwe- barena: Evaluating multimodal agents realistic visual web tasks. arXiv preprint (rXiv:2401.13649, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 2019.
LangChain. Introduction - langchain. https://ppttooninagggaainn.oooo html, 2023.
Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end-to-end learning for negotiation dialogues. arXiv preprint ar arXio:1706.00125 2017.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 2020.
Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache. Large language models for supply chain optimization. arXiv preprint arXiv:2307.00 003875, 2023a.
Guohao Li, Hasan Abed AI Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large scale language model society, 2023b.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-agent debate, 2023.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Rein- forcement learning on web interfaces using workflow-guided exploration. arviv preprint arXie:1802.08802. 2018
Jerry Liu. Llamalndex, November 2022. URL httpps://github.com/jerrr/
index.
Marvin L. Minsky. The Suciety of Mind. Simon & Schuster, New York, 1988. ISBN 978-0-671- 65713-0.
12

---

Published as a conference paper at COLM 2024
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of sims in grade school math. arxiv preprint arXio:2402.14830, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. ar Xrv preprint ar dxXt::312..600,, 2013.
Ranjita Naik, Varun Chandrasekaran, Mert Yuksekgonul, Hamid Palangi, and Besmira Nushi. Diversity of thought improves reasoning abilities of large language models. arXiv preprint ar xrXi:23100000088 2023.
Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: Origins, inventory and discussion. ACM Journal of Data and Information Quality, 2023.
Bo Ni and Markus J. Buehler. Mechagents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledge, 2023.
OpenAL. ChatGPT plugins. httpss//openna http::///ppeaaaccoooooooo/////// haapll/g/chaappppluuuiii 2023.
Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards llms as operating systems. arXiv preprint arXio:3300008500, 2023.
Joon Sung Park, Joseph C O'Brien,, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. Xiv preprint ar Xrii:2304.03442, 2023.
Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang. Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108.11 116000,08.1101, 2021.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint ar XrXi:2305.15334, 2023.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. arXio preprint ar arXiv:2307.0790 2023.
Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling largelanguage-mode-bass multi-agent collaboration. arxio preprint ar arXio:2406.07155, 2024.
SemnnticKernel. Introduction semantic kernel. https://github... http://tthtub.coommccoooooouu semantic-kernel, 2023.
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Martin, Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, et al. igibson 1.0: A simulation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RS] International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference Machine Learning. PMLR, 2017.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement saaming.(222)) Xiv preprint As.A1/2303.111 11366, 2023.
13

---

Published as a conference paper at COLM 2024
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Inter- active Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL
Linxin Song, Jiale Liu, Jieyu Zhang, Shaokun Zhang, Ao Luo, Shijian Wang, Qingyun Wu, and Chi Wang Adaptive in-conversation team building for language model agents. arXiv preprint arXi:22405.19425 2024.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. ariiv preprint ar arXit:1708.04782, 2017.
Chi Wang, Qingyun Wu, Markus Weimer, and Erkang Zhu. Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems, 2021.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. arviv preprint ar arXiv:2308.11453
2023.
Daniel S. Weld and Oren Etzioni. The first law of robotics (a call to arms). In AAAI Conference on Artificial Intelligence, 1994.
Max Woolf. Langchain problem. https://aannaaakrrccom/022222//p 2023.
Yiran Wu, Feiran Jia, Shaokun Zhang, Qingyun Wu, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, and Chi Wang- An empirical study on challenging math problem solving with gpt-4. arx preprint ar arXio:230600133) 2023.
Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu. Stateflow: Enhancing Ilm task-solving through state-driven workflows. arXiv preprint arXio:2403.11322, 2024.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. ar Xiv preprint ar Kit:2339.007866 2023.
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. Openagents: An open platform for language agents in the wild. arviv preprint Xrxf:22310.10634 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arxiv preprint ar dxXi::220000362, 2022.
Shaokun Zhang, Feiran Jia, Chi Wang, and Qingyun Wu. Targeted hyperparameter optimiza- tion with lexicographic preferences over multiple objectives. The Eleventh international conference learning representations, 2023.
Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, and Qingyun Wu. Offline training of language model agents with functions learnable weights. In Forty-first International Conference on Machine Learning, 2024.
Runlong Zhou, Simon S Du, and Beibin Li. Reflect-rl: Two-player online rl fine-tuning for Ims. 2024.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
14

---

Published as a conference paper at COLM 2024
Ethics statement
There are several potential ethical considerations that could arise from the development and use of the AutoGen framework.
Privacy and Data Protection: The framework allows for human participation in conver- sations between agents. It is important to ensure that user data and conversations are protected, and that developers use appropriate measures to safeguard privacy.
Bias and Fairess:: LLMs have been shown to exhibit biases present in their training data (Navigli et al., 2023). When using LLMs in the AutoGen framework, it is crucial to address and mitigate any biases that may arise in the conversations between agents. Developers should be aware of potential biases and take steps to ensure fairness and inclusivity.
Accountability and Transparency: As discussed in the future work section, as the frame- work involves multiple agents conversing and cooperating, it important to establish clear accountability and transparency mechanisms. Users should be able to understand and trace the decision-making process of the agents involved in order to ensure account- ability and address any potential issues or biases.
Trust and Reliance: AutoGen leverages human understanding and intelligence while providing automation through conversations between agents. It is important to consider the impact of this interaction on user experience, trust, and reliance on Al systems. Clear communication and user education about the capabilities and limitations of the system will be essential (Cai al., 2019).
Unintended Consequences: As discussed before, the of multi-agent conversations and automation in complex tasks may have unintended consequences. Especially, allowing LLM agents make changes in external environments through code execution function calls, such as install packages, could be risky. Developers should carefully consider the potential risks and ensure that appropriate safeguards are in place prevent harm or negative outcomes.
Reproducibility Statement
Following the best practices in the community, aim to ensure the reproducibility of the results presented in this paper. The appendices and the accompanying repository are designed to provide readers and reviewers with comprehensive resources to understand, replicate, and extend our work.
Detailed descriptions of each application, including their construction and evaluations, are provided Appendix E.
Example outputs from AutoGen and alternative approaches across different applications are illustrated in Appendix F.
A repository containing the source code of AutoGen (v0.1.1) and application evaluation code is provided, along with instructions for reproducing the experiments reported in this paper: https: //gi thub. wtt.cccm//coeeeeeeccccc
15

---

Published as a conference paper at COLM 2024
A Interfaces and Example Code
Agent Specification. AutoGen provides a set of built-in and pre-configured conversable agents, which can be directly imported and used. One further create an agent with customized capabilities and proper configuration when instantiating the agent.
from autogen import Conver sableAgent, AssistantAgent, UserProxyAgent user_proxy UserProxyAgent (name= user proxy")
Listing 1: importing and using pre-configured agent classes from AutoGen
my_assistant ConversableAgent
name= my assistant",
3 1_m_config=my_configs
wee.eettioooooonggeeeee. function.map=my_fuue,
Listing 2: Specifying an agent with customized capabilities
High-level agent interfaces for composing diverse conversation patterns. AutoGen pro- vides several commonly used conversation patterns via the following interfaces:
initiate.chat: two-agent chat, which is chat involving two agents. An example is shown in Listing 4.
initiate chate: sequential chat, which involves sequence of chats between two agents. This approach is particularly beneficial for tasks requiring a sequence of interdepen- dent multi-agent conversations. The utility of this method is exemplified in Listing 5, showcasing the execution of complex tasks through coordinated chat sequences.
register nested chat: nested chat, which is a chat created by a receiver agent after receiving message from sender agent and finished before the receiver agent replies to this message. Nested chats allow one agent to use other agents as their inner monologue to accomplish tasks. This abstraction is powerful as it allows one to compose agents in rich ways. Listing gives an example in which we use this interface to nest a reflection agent within the user proxy agent. This could be used to realize the self-reflection idea (Shinn et al., 2023) to improve LLM's reasoning and problem-solving capability.
GroupChat: group chat, which is a pattern in which participating agents have shared context. It is desirable for dynamic task solving, as detailed in A5.
user_proxy. regis ter wttsssteeeeeeedeeeeeee (E( recipient cri tique aiquueeeeeenn]
Listing 3: Nested chat registration (assuming all the agent involved have been created)
user _per_p initiate_chate
my_assistant, message="Poo chart of META and TESLA stock price change YTD."
Listing 4: Two-agent chat (with a nested chat if the corresponding registration is done)
initiate_chats.
[[
sender": user_proxy
"recipient": my_assistant_1.
"message": "Get META and TESLA stock price change YTD.",
7
(
"sender": user_proxy,
"recipient": my_assistant_2,
10
"message": "Plot chart based on the stock price data.",
11
}]
12
Listing 5: Sequential chat
16

---

Published as a conference paper at COLM 2024
B Expanded Related Work
Here we examine existing LLM-based agent systems or frameworks that can be used to build LLM applications in detail. We categorize the related work into single-agent and multi- agent systems and specifically provide summary of differentiators comparing AutoGen with existing multi-agent systems below. Note that many of these systems are evolving open-source projects, so the remarks and statements about them may only be accurate as of the time of writing. We refer interested readers to detailed LLM-based agent surveys (Xi et al., 2023; Wang et al., 2023)
Single-Agent Systems:
AutoGPT: AutoGPT is an open-source implementation of an Al agent that attempts to
autonomously achieve given goal (AutoGPT, 2023). It follows single-agent paradigm in which it augments the Al model with many useful tools, and does not support multi- agent collaboration.
ChatGPT+ (with code interpreter or plugin): ChatGPT, a conversational AT service or agent, can now be used alongside code interpreter or plugin (currently available only under the premium subscription plan ChatGPT Plus) (OpenAI, 2023). The code interpreter enables ChatGPT to execute code, while the plugin enhances ChatGPT with wide range of curated tools.
LangChain is a general framework for developing LLM-based applications (LangChain, 2023). LangChain Agents subpackage for using an LLM to choose a sequence of actions. There are various types of agents in LangChain Agents, with the ReAct agent being a notable example that combines reasoning and acting when using LLMs (mainly designed for LLMs prior to ChatGPT) (Yao et al., 2022). All agents provided in LangChain Agents follow a single-agent paradigm and are not inherently designed for communicative and collaborative modes. A significant summary of its limitations can be found in (Woolf, 2023). Due to these limitations, even the multi-agent systems in LangChain (e.g,, re-implementation of CAMEL) are not based on LangChain Agents but are implemented from scratch. Their connection LangChain lies in the use of basic orchestration modules provided by LangChain, such as Al models wrapped by LangChain and the corresponding interface.
LangChain Agents:
Transformers Agent: Transformers Agent (HuggingFace, 2023) is experimental natural- language API built on the transformers repository. It includes set of curated tools and agent to interpret natural language and these tools. Similar to AutoGPT, it follows a single-agent paradigm and does not support agent collaboration.
AutoGen differs from the single-agent systems by supporting multi-agent conversations.
Multi-Agent Systems:
Two recent works investigate and show that multi-agent debate is an effective way to encourage divergent thinking in LLMSs (Liang et al., 2023) and to improve the factuality and reasoning of LLMs (Du et al., 2023). In both works, multiple LLM inference instances are constructed as multiple agents to solve problems with agent debate. Each agent is simply an LLM inference instance, while no tool or human is involved, and the inter-agent conversation needs to follow a pre-defined order. These works attempt to build LLM applications with multi-agent conversation, while AutoGen, designed as a generic infrastructure, can be used to facilitate this development and enable more applications with dynamic conversation patterns.
Multi-Agent Debate:
MetaGPT (Hong et al., 2023) and ChatDev (Qian et al., 2023): Both MetaGPT and ChatDev are multi-agent frameworks for software development. Compared to MetaGPT and ChatDev, AutoGen supports more flexible and complex workflows in addition to Standardized Operating Procedures (SOP) or chains, e.g,, nested chat and group chat, and covers a wider range of usage scenarios beyond software engineering.
Simulacra (Park et al., 2023): This system features twenty-five generative agents in an interactive sandbox environment. It represents one of the initial attempts at multi-agent
17

---

Published as a conference paper at COLM 2024
interaction, with special focus on investigating whether generative agents could enable believable simulations of human behavior.
BabyAGI: BabyAGI (BabyAGI, 2023) is an example implementation of an Al-powered task management system in a Python script. In this implemented system, multiple LLM- based agents are used. For example, there is an agent for creating new tasks based on the objective and the result of the previous task, an agent for prioritizing the task list, and an agent for completing tasks/ sub-tasks. As multi-agent system, BabyAGI adopts a static agent conversation pattern, i.e., a predefined order of agent communication, while AutoGen supports both static and dynamic conversation patterns and additionally supports tool usage and human involvement.
CAMEL: CAMEL (Li et al., 2023b) is a role-playing-based agent framework supporting two or three agents (depending if critic agent is included). It demonstrates how role- playing can be used to let chat agents communicate with each other for task completion. It also records agent conversations for behavior analysis and capability understanding. An Inception-prompting technique is used to achieve autonomous cooperation between agents.
Several other LLM-based multi-agent systems/ ffremm/r/rraewwrks have emerged following the initial release of our work, including AgentVerse (Chen et al., 2023b), Auto (Chen et al., 2023a), OpenAgents (Xie et al., 2023) and IoA (Chen et al., 2024). We omit a detailed discussion here due to the contemporaneous nature of these developments.
C Expanded Discussion
The applications in Section 3 show how AutoGen not only enables new applications but also helps renovate existing ones. For example, in A1 (scenario 3), A5, and A6, AutoGen enabled the creation of multi-agent conversations that follow dynamic pattern instead of a fixed back-and-forth. And in both A5 and A6, humans can participate in the activities together with multiple other AI agents in conversational manner. Similarly, A1-A4 show how popular applications can be renovated quickly with AutoGen. Despite the complexity of these applications (most of them involve more than two agents or dynamic multi-turn agent cooperation), our AutoGen-based implementation remains simple, demonstrating promising opportunities to build creative applications and a large space for innovation. In reflecting why these benefits can be achieved in these applications with AutoGen, we believe there are few reasons:
The built-in agents can be used out-of-the-bon delivering strong performance even without any customization. (A1, A3)
Ease of use:
Modularity: The division of tasks into separate agents promotes modularity in the system. Each agent can be developed, tested, and maintained independently, simplifying the
overall development process and facilitating code management. (A3, A4, A5, and A6)
Programmability: AutoGen allows users to extend/ /customize existing agents to develop systems satisfying their specific needs with ease. (A1-A6). As demonstrated in A4, the use of AutoGen significantly streamlines the core workflow code, reducing it from over 430 lines to just 100 lines, resulting in a fourfold reduction code size.
Allowing human involvement: AutoGen provides native mechanism to achieve hu- man participation and orr human oversight. With AutoGen, humans can seamlessly and optionally cooperate with Als to solve problems generally participate in the activity. AutoGen also facilitates interactive user instructions to ensure the process stays on the desired path. (A1, A2, A5, and A6)
Collaborativeeadderrrriaa agent interactions: Like many collaborative agent sys- tems (Dong et al., 2023), agents in AutoGen can share information and knowledge, to complement each other's abilities and collectively arrive at better solutions. (A1, A2, A3, and A4). Analogously, in certain scenarios, some agents are required to work in an ad adversarial way. Relevant information is shared among different conversations in controlled manner, preventing distraction or hallucination. (A4, A6). AutoGen supports both patterns, enabling effective utilization and augmentation of LLMs.
18

---

Published as a conference paper at COLM 2024
C.1 General Guidelines for Using AutoGen
Below we give some recommendations for using agents in AutoGen to accomplish a task.
1. Consider using built-in agents first. For example, AssistantAgent is pre-configured with carefully designed system message for generic problem-solving via code. The UserProxyAgent is configured to solicit human inputs and perform tool execution. Many problems can be solved by simply combining these two agents. When customizing agents for an application, consider the following options: (1) human input mode, termi- nation condition, code execution configuration, and LLM configuration can be specified when constructing an agent; (2) AutoGen supports adding instructions in an initial user message, which is an effective way to boost performance without needing modify the system message; (3) UserProxyAgent can be extended to handle different execution environments and exceptions, etc.; (4) when system message modification is needed, consider leveraging the LLM's capability to program its conversation flow with natural language.
Start with a simple conversation topology. Consider using the two-agent chat or the group chat setup first, as they can often be extended with the least code. Note that the two-agent chat can be easily extended to involve more than two agents by using LLM--onsummabl functions in a dynamic way.
3. Try to reuse built-in reply methods based on LLM, tool, or human before implementing custom reply method because they can often be reused to achieve the goal in a simple way (e.g., the built-in agent CooupChatManagerr reply method reuses the built-in LLM-based reply function when selecting the next speaker, ref. A5 in Section 3).
When developing a new application with UserProxyAgent, start with humans always in the loop, i.e., human input Moode''ALWAYS even if the target operation mode is more autonomous. This helps evaluate the effectiveness of AssistantAgent tuning the prompt, discovering corner cases, and debugging. Once confident with small-scale success, consider setting one can either use the LLM or manually generate diverse system messages to simulate different use cases.
inpputmmodeeeeeee = 'NEVER'. This enables LLM as a backend, and
5. Despite the numerous advantages of AutoGen agents, there could be cases/ / /cenaaios where other libraries/packages could help. For example: (1) For (sub)tasks that do not have requirements for back-and-forth trouble-shooting, multi-agent interaction, etc., unidirectional (no back-and-forth message exchange) pipeline can also be orchestrated with LangChain (LangChain, 2023), LlamaIndex (Liu, 2022), Guidance (Guidance, 2023), Semantic Kernel (SemanticKernel, 2023), Gorilla (Patil et al., 2023) or low-level inference API (' ("autogen.oail provides an enhanced LLM inference layer at this level) (Dibia, 2023). (2) When existing tools from LangChain etc. are helpful, one can use them as tool backends for AutoGen agents. For example, one readily use tools, e-g., Wolfram Alpha, from LangChain in AutoGen agent. (3) For specific applications, one may want to leverage agents implemented in other libraries/ packages. To achieve this, one could wrap those agents as conversable agents in AutoGen
then use them to build LLM applications through multi-agent conversation. (4) It can be hard to find an optimal operating point among many tunable choices, such as the LLM inference configuration. Blackbox optimization packages like 'flaml.tune' (Wang et al., 2021; Zhang et al., 2023) can be used together with AutoGen to automate such tuning.
C.2 Future Work
This work raises many research questions and future directions.
Designing optimal multi-agent workflows: Creating a multi-agent workflow for a given task can involve many decisions, e.g., how many agents to include (Qian et al., 2024), how to assign agent roles and agent capabilities (Song et al., 2024), how the agents should interact with each other (Wu et al., 2024), and whether to automate particular part of the workflow. There may not exist a one-size ail answer, and the best solution might depend on the specific application. This raises important questions: For what types of tasks
19

---

Published as a conference paper at COLM 2024
and applications are multi-agent workflows most useful? How do multiple agents help in different applications? For a given task, what is the optimal (e.g,, cost-effective) multi-agent workflow?
Creating highly capable agents: AutoGen can enable the development of highly capable agents that leverage the strengths of LLMs, tools, and humans. Creating such agents is crucial to ensuring that a multi-agent workflow can effectively troubleshoot and make progress on task. We believe that more systematic work will be required to develop guidelines for application-specifi agents, to create large OSS knowledge base of agents, and to create agents that can discover and upgrade their skills (Cai al., 2023; Zhang et al., 2024; Song et al., 2024).
Enabling scale, safety, and human agency: Section 3 shows how complex multi-agent workflows can enable new applications, and future work will be needed to assess whether scaling further can help solve extremely complex tasks. However, as these workflows scale and grow more complex, it may become difficult to log and adjust them. Thus, it will become essential develop clear mechanisms and tools to track and debug their behavior. Otherwise, these techniques risk resulting in incomprehensible, unintelligible chatter among agents (Lewis et al., 2017).
Our work also shows how complex, fully autonomous workflows with AutoGen can be useful, but fully autonomous agent conversations will need to be used with care. While the autonomous mode AutoGen supports could be desirable in many scenarios, a high level of autonomy can also pose potential risks, especially in high-risk applications (Amodei et al., 2016; Weld & Etzioni, 1994). As result, building fail-safes against cascading failures and ex- ploitation, mitigating reward hacking, out of control and undesired behaviors, maintaining effective human oversight of applications built with AutoGen agents will become important. While AutoGen provides convenient and seamless involvement of humans through user proxy agent, developers and stakeholders still need to understand and determine the ap- propriate level and pattern of human involvement to ensure the safe and ethical use of the technology (Horvitz, 1999; Amershi al., 2019).
D Default System Message for Assistant Agent
Figure 5 shows the system message for the built-in assistant agent in AutoGen (v0.1.1), where we introduce several new prompting techniques and highlight them accordingly. When combining these new prompting techniques together, we can program a fairly complex conversation even with the simplest two-agent conversation topology. This approach tries to exploit the capability of LLMs in implicit state inference to a large degree. LLMs do not follow all the instructions perfectly, the system design requires additional mechanisms to manage exceptions and faults. Certain instructions can contain ambiguities, and the designer should either mitigate these ambiguities to enhance precision or intentionally retain them to allow for flexibility and address the varying situations through other means.
E Application Details
A1: Math Problem Solving
Scenario 1: Autonomous Problem Solving. We perform both qualitative and quantitative evaluations in this scenario. For all evaluations, we use GPT-4 as the base model, and pre-install the "sympy" package in the execution environment. We compare AutoGen with the following LLM-based agent systems:
AutoGPT: The out-of-box AutoGPT is used. We initialize AutoGPT by setting the purpose to "solve math problems", resulting in a "MathSolverGPT" with auto-generated goals.
ChatGPT+Plugin: We enable the Wolfram Alpha plugin (a math computation engine) in the OpenAl web client.
20

---

Published as a conference paper at COLM 2024
System Message
You are helpful Al as81 asate
Solve tasks using your coding and languege ski ski"Ts
In the following cases, suggest python code [Lc pyshen coding block) or shel. scrips (4n a coding block)
1. Maer you need to collect info, use the code to ousput the inčo you need, for examp-t, broNse download/resd file, print contert webpage date/sine fine After sufficient info 1s printed and the cask 18 ready to be solved based or your Langunge skill, you solve the task yourselE,
get
2. Wher need to perforn aone taak with code, use the oode to pestorn the task and output tha penuis, Finish the smartiy.
provided, explain your plan first. Clear which step uses code, and which stop used your language skill.
le
user canaot provids any other feedback or perforn any other sction beyond executing the code you suggest. The ueer cap"t modify.
When using cossy you mast indicate
sczipt type
modify your
which requires code block it's not incerded to be executed by the user.
suggest incomplete
before executing it, gut filerames eeilennmes inaide the coda block the firet 1150. Den's Includs muitiple code blocks in ona reapenae, Do not copy pasts result. Instead, use "peint" fuaction Check the execution result returned by the user.
you eant She user to save the code
octput relevant.
the resulz indicates there an erroz, fix the ercer and cutpe: the code again, Sopgest she full code Instead of partial oode or code changes, the
she taxt not aolved after the code executed succesarully, analyse problen, reviait your assumption, collect additional info need, and thick of a different appzeach to try.
can't be Fixed
Maen you find an anSweE verity the answez caretully. Include verifiable avidence in youz respense 14 possible.
Reply "TERNINNTE" in the end Waen everything is done
Prompting techniques color code: Role Play:
Control Flow; Output Confine; Facilitate Automation; Grounding
Figure 5: Default system message for the built-in assistant agent in AutoGen (v0.1.1). This is an example of conversation programming via natural language. It contains instructions of different types, including role play, control flow, output confine, facilitate automation, and grounding.
ChatGPT+Code Interpreter: This is recent feature in OpenAl web client. Note that the above two premium features from ChatGPT require a paid subscription to be accessed and are the most competitive commercial systems.
LangChain ReAct+Python: We use Python agent from LangChain. To handle parsing errors, we set "handle parsing errors=True" and use the default zero-shot ReAct prompt.
Multi-Agent Debate (Liang al., 2023): We modified the code of the multi-agent debate to perform evaluation. By default, there are three agents: an affirmative agent, negative agent, and moderator.
We also conducted preliminary evaluations on several other multi-agent systems, including BabyAGI, CAMEL, and MetaGPT. The results indicate that they are not suitable choices for solving math problems out of the box. For instance, when MetaGPT is tasked with solving math problem, it begins developing software address the problem, but most of the time, it does not actually solve the problem. We have included the test examples in Appendix F.
For the qualitative evaluation, we utilize two level-5 problems from the MATH dataset, testing each problem three times. The first problem involves simplifying a square root fraction, and the second problem involves solving number theory issue. The correctness counts and reasons for failure are detailed in Table 1. For the quantitative evaluation, we conduct two sets of experiments on the MATH dataset to assess the correctness of these systems: (1) experiment involving 120 level-5 (the most challenging level) problems, including 20 problems from six categories, excluding geometry, and (2) an experiment on the entire test set, which includes 5000 problems. We exclude AutoGPT from this evaluation
21

---

Published as a conference paper at COLM 2024
Expert Assistant
Student Proxy
Student Assistant
Ask for expert
Enable Autonomous and Human-in-the-loop Problem Solving
Expert
Proxy
Enable Mulli-User Problem Solving Via Student
Expert
Figure 6: Illustration of three problem-solving scenarios with AutoGen: (Gray) Enables workflow where student collaborates with an assistant agent to solve problems, either autonomously or in a human-in-the-loop mode. (Gray + Orange) Facilitates a more so- phisticated workflow wherein the assistant, the fly, engage another user termed "expert", who is in the loop with their own assistant agent, to aid in problem-solving if its own solutions not satisfactory.
as cannot access results from code executions and does not solve any problems the qualitative evaluation. Our analysis of the entire dataset reveals that AutoGen achieves overall accuracy of 69.48%, while GPT-4's accuracy stands at 55.18%. From these evaluations, have the following observations regarding the problem-solving success rate and user experience of these systems:
Problem-solving success rate: Results from the quantitative evaluations show that AutoGen can help achieve the highest problem-solving success rate among all the com- pared methods. The qualitative evaluations elucidate common failure reasons across several alternative approaches. ChatGPTCCode Interpreter fails to solve the second problem, and ChatGPT-Plugin struggles to solve both problems. AutoGPT fails both problems due to code execution issues. The LangChain agent also fails on both problems, producing code that results in incorrect answers all trials.
Based on the qualitative evaluation, we analyze the user experience concerning the ver- bosity of the response and the ability of the LLM-based system to run without unexpected behaviors. ChalGiT+Pllugin much shorter than Python code. AutoGen, ChatGPT+Code Interpreter, and LangChain exhibit similar verbosity, although LangChain is slightly more verbose due to more code execution errors. AutoGPT the most verbose system owing to predefined steps like THOUGHTS, REASONING, and PLAN, which it includes in replies every time. Over- all, AutoGen and ChatGPT+ Interpreter operate smoothly without exceptions. We note the occurrences of undesired behaviors from other LLM-based systems that could affect user experience: AutoGPT consistently outputs code without the print' statement and cannot correct this, requiring the user to run them manually; ChatGPT with Wol- fram Alpha plugin has the potential to become stuck in loop that must be manually stopped; and Langchain ReAct could exit with parse error, necessitating the passing of a
the least verbose, mainly because Wolfram queries are
parameter.
Scenario 2: Human-in-the-loop Problem Solving. For challenging problems that these LLM systems cannot solve autonomously, human feedback during the problem-solving process can be helpful. To incorporate human feedback with AutoGen, can set
22

---

Published as a conference paper at COLM 2024
Table 1: Qualitative evaluation of two math problems from the MATH dataset within the autonomous problem-solving scenario. Each LLM-based system is tested three times on each of the problems. This table reports the problem-solving correctness and summarizes the reasons for failure.
Correctness
Failure Reason
AutoGen
3/3
N/A.
AutoGPT
0/3
The LLM gives code without the print function the result not printed.
ChatGPI+Pllgin
1/3
The return from Wolfram Alpha contains 2 simplified results, including the correct answer, but GPT-4 always chooses the wrong answer. Returns a wrong decimal result.
ChadGPT+Code
Inter-
2/3
preter
LangChain ReAct
0/3
LangChain gives 3 different wrong answers. gives different wrong answers due to calculation
Ageti-Agent Debate
0/3
errors.
(a) Evaluation on the first problem that asks to simplify square root fraction.
Correctness 2/3
Failure Reason
AutoGen AutoGPT
The final answer from code execution is wrong. The LLM gives code without the print function so the result is not printed.
0/3
ChagGIT+Pluginn
1/3
For one trial, GPTT4 got stuck because it keeps giving wrong queries and has be stopped. Another trial
simply gives a wrong answer.
ChatGPT+CCode
Inter-
0/3
It gives 3 different wrong answers.
preter
LangChain ReAct
0/3
LangChain gives 3 different wrong answers. It gives 3 different wrong answers.
Multi-Agent Debate
0/3
(b) Evaluation on the second number theory problem.
human.input.moodee serpputtmode'''oe 'LWWYSS' in the user proxy agent. We select one challenging problem that none of these systems can solve autonomously across three trials. We adhere the process outlined below to provide human inputs for all the compared methods:
1. Input the problem: Find the equation of the plane which bisects the angle be tween the planes 3x 6y 2z + 5 and 4x 12y 3z -3 0, and which contains the point -1, Enter your answer in the form
Ax + By + Cz D =0 0,
where A, B, C, D are integers such that A> 0 and [cd DI) 1.
2. The response from the system does not solve the problem correctly: We then give a hint to the model: Your idea is not correct. Let's solve this together. Suppose P (x, is point that lies on a plane that bisects the angle, the distance from P to the two planes is the same. Please set up this equation first.
3. We expect the system to give the correct distance equation. Since the equation involves an absolute sign that is hard to solve, we would give the next hint: Consider the two cases to remove the abs sign and get two possible solutions.
4. If the system returns the two possible solutions and doesn't continue to the next step, we give the last hint: Use point (-5,-1,-5) to determine which is correct and give the final answer.
5. Final answer is 1x+66+52++8-00
We observed that AutoGen consistently solved the problem across all three trials. Chat- GPT+Code Interpreter and ChaiGPT+Plugin managed to solve the problem in two out
23

---

Published as a conference paper at COLM 2024
of three trials, while AutoGPT failed to solve it in all three attempts. In its unsuccessful attempt, ChatGPT+Code Interpreter failed to adhere to human hints. In its failed trial, ChatGPT+Plugin produced an almost correct solution but had a sign discrepancy in the final answer. AutoGPT was unable to yield a correct solution in any of the trials. In one trial, it derived an incorrect distance equation. In the other two trials, the final answer was incorrect due to code execution errors.
Scenario 3: Multi-User Problem Solving. Next-generation LLM applications may neces- sitate the involvement of multiple real users for collectively solving problem with the assistance of LLMs. We showcase how AutoGen can be leveraged to effortlessly construct such system. Specifically, building upon scenario 2 mentioned above, we aim to devise simple system involving two human users: a student and an expert. In this setup, the stu- dent interacts with an LLM assistant to address some problems, and the LLM automatically resorts to the expert when necessary.
The overall workflow is follows:
The student chats with the LLM-based assistant agent through a student proxy agent to solve problems. When the assistant cannot solve the problem satisfactorily,
the solution does not match the expectation of the student, would automatically hold the conversation and call the pre-defined ask for.foroeeeeeo expert function via the function call feature of L.M in order to resort to the expert. Specifically, it would automatically produce the initial message for the ask_for for_foo_eeee _expert function, which could be the statement of the problem or the request to verify the solution to problem, and the expert is supposed to respond to this message with the help of the expert assistant. After the conversation between the expert and the expert's assistant, the final message would be sent back to the student assistant as the response to the initial message. Then, the student assistant would resume the conversation with the student using the response from the expert for better solution. A detailed visualization is shown in Figure 6.
With AutoGen, constructing the student/ expert proxy agent and the assistant agents is straightforward by reusing the built-in UserProxyAgent and AssistantAgent through ap- propriate configurations. The only development required involves writing several lines of code for the exxpett function, which then becomes part of the configuration for the assistant. Additionally, it's easy to extend such system to include more than one expert, with specific ask_for_exxx _e_eeeerrteeeer function for each, or to include multiple student users with shared expert for consultation.
24

---

Published as a conference paper at COLM 2024
A2: Retrieval-Augmented Q&A
1. Question and Contexts
2. Satisfied Answers or "Update Context"
Terminate, feedbacks "Update Context"
4. Satisfied Answers Terminate
Retrieval-augmented
Retreevalauggented Assistant
User Proxy
Figure 7: Overview of RAG Chat which involves two agents, including a RAG User Proxy and letrivval-uugmented Assistant. Given set of documents, the Ketrieval-augmented User Proxy first automatically processes documents spaittsssssstss,,, chunks, and stores them in vector database. Then for a given user input, it retrieves relevant chunks as context and sends it to the Retrieval-augmented Assistant, which uses LLM to generate code or text to answer questions. Agents converse until they find a satisfactory answer.
Detailed Workflow. The workflow RAG Chat is illustrated in Figure 7. To use Retrieval- augmented Chat, one needs to initialize two agents including Retrieval-augmeette User Proxy and Retrieval-augmented Assistant. Initializing the Retrieval-Augmenter User Proxy necessitates specifying path to the document collection. Subsequently, the Retrieval- Augmented User Proxy can download the documents, segment them into chunks of specific size, compute embeddings, and store them in vector database. Once chat is initiated, the agents collaboratively engage in code generation or question-answering adhering the procedures outlined below:
1. The Retrieval-Augmennté User Proxy retrieves document chunks based on the embed- ding similarity, and sends them along with the question to the Eetrieval-Augmente Assistant.
2. The Detrieval-Augmenne based on the question and context provided. the LLM is unable to produce a satisfactory response, it is instructed to reply with "Update Context" to the Retrieval-Augmenter User Proxy.
Assistant employs LLM to generate code or text as answers
response includes code blocks, the Retrieval-Augmented User Proxy executes the code and sends the output as feedback. If there are code blocks or instructions to update the context, it terminates the conversation. Otherwise, it updates the context and forwards the question along with the new context to the Retrieval-Auumeentd Assistant. Note that if human input solicitation is enabled, individuals can proactively send any feedback, including "Update Context", to the Retrieval-Augmentee Assistant.
4. If the Retrieval-Augmeet most similar chunks of documents as new context from the Aetrieval-Augmentt Proxy. Otherwise, it generates new code text based on the feedback and chat history. If the LLM fails to generate an answer, it replies with "Update Context" again. This process can be repeated several times. The conversation terminates if no more documents are available for the context.
Assistant receives "Update Context", it requests the next
User
We utilize Cetrieval-Auugmenter Chat in two scenarios. The first scenario aids in generating code based on given codebase. While LLMs possess strong coding abilities, they are unable utilize packages or APTs that are not included in their training data, e.g., private codebases, have trouble using trained ones that are frequently updated post-training. Hence, Ketrieval-Augmeenee Code Generation is considered to highly valuable. The sec- ond scenario involves question-answering the Natural Questions dataset (Kwiatkowski et al., 2019), enabling us to obtain comparative evaluation metrics for the performance of our system.
Scenario 1: Evaluation on Natural Questions QA dataset. In this case, we evaluate the Retrieval-Augmentte Chat's end-to-end question-answering performance using the Natural
25

---

Published as a conference paper at COLM 2024
Task description
vio carcivg usa clog in opecing curaaony!
(a) W/O Interactive Retrieval
(b) W/ Interactive Retrieval
Dawcdeooy: too" ze i. pescieve supracced shatbot. You unsver usecta quaations bazed on your orn kcovlades and the context procided by usen,
CenePraxy: You'ce petelove augranted chatbot. You ansvee arects queat. based on your own knovlsoge and the context procided the usec. If you cantt snsver the cueation with or wishout the
glve thart you
parsible.
tauesy,
current cortexty dhont tep exeetly "OHORE covesen for pore
give
Staets quection is: she carried the Das flag in opening carennny Dontext loz 2 > da datuary =? 1899 usdus ane the Autros seat the Yeskeep
under
night y
bese the vantene that night
Asglatant:
paray,
Assistant:
Surayy cantol Lind uny enccomat con about viu outz the fing the spaning cererany. BREASE COETSEE
epening ocrorory, Morege con't
GaarPraay:
augranted
the 25 you can's
cusation with wichout
cussect corsest, you should peply exectly SUEAAEE coSTECT You rust give as short an answer an possiblle Cans's qusstion Loc wbo caszied the cas 2lap opening cessracy
Aeclstant:
USA rany the
Figure 8: Retrieval-augmented Chat without (W/O) and with (W/) interactive retrieval.
Questions dataset (Kwiatkowski et al., 2019). We collected 5,332 non-redundant context documents and 6,775 queries from HuggingFace. First, we create a document collection based on the entire context corpus and store it in the vector database. Then, we utilize Retrieval-Augmente
Chat to answer the questions. An example (Figure 8) from the NQ dataset showcases the advantages of the interactive retrieval feature: "whho carried the usa flag in opening ceremony". When attempting to answer this question, the context with the highest similarity to the question embedding does not contain the required information for response. As a result, the LLM assistant (GPT-3.5-turbo) replies "Sorry, I cannot find any information about who carried the USA flag in the opening ceremony. UPDATE CONTEXT." " With the unique and innovative ability to update context in Retrieval-Augmenter Chat, the user proxy agent automatically updates the context and forwards it to the assistant agent again. Following this process, the agent is able to generate the correct answer to the question.
In addition, we conduct an experiment using the same prompt as illustrated in (Adlakha et al., 2023) to investigate the advantages of AutoGen W/O interactive retrieval. The F1 score and Recall for the first 500 questions are 23.40% and 62.60%, respectively, aligning closely with the results reported in Figure 4b. Consequently, we assert that AutoGen W/O interac- tive retrieoal outperforms DPR due to differences in the retrievers employed. Specifically, we utilize a straightforward vector search retriever with the all-MiniLM--L--u model for embeddings.
Furthermore, we analyze the number of LLM calls experiments involving both AutoGen and AutoGen W/O interactive retrieval, revealing that approximately 19.4% of questions in the Natural Questions dataset trigger an "Update Context" operation, resulting in additional LLM calls.
Scenario 2: Code Generation Leveraging Latest APIs from the Codebase. In this case,
the question is "How can I use FLAML to perform a classification task and use Spark for parallel training? Train for 30 seconds and force cancel jobs if the time limit is reached." ". FLAML (v1) (Wang et al., 2021) is an open-source Python library designed for efficient AutoMI. and tuning. It was open-sourced in December 2020, and is included in the training data of GPT-4. However, the question necessitates the use of Spark-related APIs, which were added in December 2022 and are not encompassed in the GPT-4 training data'. Consequently, the original GPT-4 model is unable to generate the correct code, due to its lack of knowledge regarding Spark-related APIs Instead, it erroneously creates a non-existent parameter, spark, and sets it to True'. Nevertheless, with Petrieval-Augmenn
Chat, we provide the latest reference documents as context. Then, GPT-4 generates the correct code blocks by setting use_spark and force_cancel cancel to True'.
1 ggpt-4-0613 is used in this experiment. Its cutoff date is September 2021.
26

---

Published as a conference paper at COLM 2024
A3: Decision Making in Embodied Agents
ALFWoridd
Action Decision
docision: up pencil
ALFWork Executer
Assistant
Reward State
Ubservation:
you alarrclock mug
Assistant
Groundrgggent
ALFChat (two agents)
ALFChat (three agents)
Figure 9: Overview of two versions of ALFChat: a two-agent approach where assistant agent proposes the subsequent step and an executor carries out actions while providing feedback, and three-agent design that incorporates grounding agent to furnish the executor with commonsense facts when required.
ALFWorld (Shridhar et al., 2021) is synthetic language-based interactive decision-making task. It comprises textual environments that aim to simulate real-world household scenes. Given high-level goal (e-g,, putting hot apple in the fridge) and the description of the household environment, the agent needs to explore and interact with the simulated household environment through textual interface. A typical task environment contains various types of locations and could require more than 40 steps to finish, which requires agents to decompose the goal into subtasks and tackle them one by one, while effectively exploring the environments.
Detailed Workflow. We first propose straightforward two-agent system with AutoGen, illustrated on the left-hand side of Figure 9, to tackle tasks from this benchmark. The system consists of an assistant agent and an executor agent. The assistant agent generates plans and makes action decisions to solve the tasks. The executor agent is tailored specifically for ALFWorld. performs actions proposed by the assistant and reports action execution results in the household environment as feedback to the assistant. Due to the strict format requirements for the output format, we use the BLEU metric to evaluate the similarity of the output to all valid action options. The option with the highest similarity will be chosen as the action for this round.
A significant challenge encompassed in ALFWorld involves commonsense reasoning. The agent needs to extract patterns from the few-shot examples provided and combine them with the agent's general knowledge of household environments to fully understand task rules. More often than not, the assistant tends to neglect some basic knowledge of the household environment. Thanks to the easy-to-implemeer feature of AutoGen, enhancing the assistant agent's reasoning ability by adding new grounding agent provide commonsense facts for the decision-making agent's reference becomes straightforward. By scrutinizing the failed attempts and summarizing the reasons for failure, we obtained holistic understanding of the commonsense knowledge that the assistant agent lacks. Then, we set grounding agent to provide this general knowledge when the task begins and whenever the assistant outputs the same action three times row. This ensures the assistant takes this commonsense knowledge into consideration.
multi-agent conversational
We compare our system's performance with ReAct. ReAct (Yao et al., 2022) is a few-shot prompting technique that interleaves reasoning and acting, allowing for greater synergy between the two and significantly improving performance on both language and decision- making tasks. We integrate ReAct into AutoGen by modifying the prompts into conver-
27

---

Published as a conference paper at COLM 2024
Paak dascription
You ane in the a dste of soow. teoking guickly around you, you deak
desk safe i, drower 7, draver 1,
cieli 3, shellf Ps and a shel to
Your taok ia to: Yook at bowl onder the decklang.
aganta
Three agents
Asstatant Bxecutazi: teouser (eives, need to find a book. A libely tgpeae 11-21,
Assistent Evecutoc):
TiouGat (Facats noud cine bowd, 11-2), aed (11, Then need kisd and cupslanp-] ACTION Tep: to deat 11
deok
lAnecatant riate the desklarp at daek taen
dest
Boolston rists the desk desk then gees to dook
Anslatant]:
7, you tee
Dhe intant tylto iato the Trep o0)
od 2...
Gcoundingkeent (to Exscutar)
find
(to Rxscutor]: (SROSSY [SoM I find a Saut Hi reed
Befere you can occenie
You eots to whete the carges 2eject 13 becoze you can LaN Lt.
2ne dus cuppppp 2O08 at the bowl, S1a1 go to the desklong)
bowi ACC108 igo deak
Execesoe asssatant) You turn on the deskuup thint You wast Aosi aesistant.oo (to Sescutor): Iraoiarr
you cosklaep
pensil 2
Assiatant: ACTIOS 1ean deaklarp 13
Execusse ito Assiatant]: You Earn the deaklang i.
ACTLOS (go to deak 2]
Bsecutac(So
100k
Sound
Assistant Executoc]: ACEION [baks bows zeon deak 2] 3...
MOTICN Flonk Wool inder The deck) tep
Kennutarr [Eo Analatans]: You tove the deoklangg
Executaz Assiatant): You pisk up bodi Eace. dest 2.
(Anndatant falla 1850 infanite loep surning desiclmp.. Rewousor/to (to Ansiatant): Tack falled. Reply 2ARIIANN.. Executoz):
Anclatant Rascutor) serroN lase derklawg Esecutor (50 Melatant) tack anconss. teply YRANUIERR.
Accletant Executor). TERNINEER
Figure 10: Comparison of results from two designs: (a) two-agent design, which consists of an assistant and an executor, and (b) three-agent design, which adds a grounding agent that serves as a knowledge source. For simplicity, we omit the in-context examples and part of the exploration trajectory, and only show parts contributing the /success of the attempt.
sational manner. Following ReAct, we employ a two-shot setting. The few-shot prompts are obtained from the corresponding repository. As shown in Table 2, the two-agent design matches the performance of ReAct, while the three-agent design significantly outperforms ReAct. On the other hand, introducing grounding agent as a knowledge source remarkably advances performance on all types of tasks.
Case study. Figure 10 exemplifies how a three-agent design eliminates one root cause for failure cases. Most of the tasks involve taking an object and then performing a specific action with it (e.g., finding vase and placing it on a cupboard). Without grounding agent, the assistant frequently conflates finding an object with taking it, as illustrated in Figure 10(a). This leads to most of the failure cases in 'pick' and 'look' type tasks. With the introduction of a grounding agent, the assistant can break this loop and successfully complete the task.
Takeaways. We introduced a grounding agent to serve as an external commonsense knowl- edge source, which significantly enhanced the assistant's s ability to make informed decisions. This proves that providing necessary commonsense facts to the decision-making agent can assist it in making more informed decisions, thus effectively boosting the task success rate. AutoGen brings both simplicity and modularity when adding the grounding agent.
Method
Pick
Clean
Heat
Cool
Look
Pick 2
All
ReAct (avg)
63
52
48
71
61
24
54
ALFChat (2 (gv))))(agg)
61
58
57
67
50
19
54
ALFChat (3 agents)
79
64
70
76
78
41
69
ReAct (best of 3)
75
62
61
81
78
35
66
ALFChat (2 (Beet of 3)
71
61
65
76
67
35
63
ALFChat (3 (bent of 3)
74
78
86
41
77
Table 2: Comparisons between ReAct and the two versions of ALFChat For each task, we report the success rate out of 3 attempts. Success rate denotes the number of tasks successfully completed by the agent divided by the total number of tasks. The results show that adding grounding agent significantly improves the task success rate in ALFChat
28

---

Published as a conference paper at COLM 2024
A4: Supply-Chain Optimization
User
User Question
Final Answer
2 Question 6 Log
Clearance
Ans
Code
Coge.
Commander
Repeat until answering the user's question or timeout
Writer
Safeguard
Figure 11: Our re-implementation of OptiGuide with AutoGen streamlining agents' interac- tions. The Commander receives user questions (e.g., What if we prohibit shipping from supplier 1 to roastery 2?) and coordinates with the Writer and Safeguard. The Writer crafts the code and interpretation, the Safeguard ensures safety (e.g., not leaking information, no malicious code), and the Commander executes the code. If issues arise, the process can repeat until resolved. Shaded circles represent steps that may be repeated multiple times.
Detailed Workflow. The workflow can be described as follows. The end user initiates the interaction by posing question, such as "What if we prohibit shipping from supplier 1 to roastery 2?", marked by ( to the Commander agent. The Commander manages and coordinates with two LLM-based assistant agents: the Writer and the Safeguard. Apart from directing the flow of communication, the Commander has the responsibility of handling memory tied to user interactions. This capability enables the Commander to capture and retain valuable context regarding the user's questions and their corresponding responses. Such memory is subsequently shared across the system, empowering the other agents with context from prior user interactions and ensuring more informed and relevant responses.
In this orchestrated process, the Writer, who combines the functions of "Coder" and "Interpreter" as defined in Li et al. (2023a), will craft code and also interpret execution output logs, For instance, during code writing (2 and the Writer may craft code aimodeeladCoosstr(ll ('upp(xx'''ppp"""" 'roastery2'] == 'prohibit') to add an additional constraint to answer the user's 's question.
After receiving the code, the Commander will communicate with the Safeguard to screen the code and ascertain its safety marked by the Commander will use external tools (e.g., Python) to execute the code and request the Writer to interpret the execution results for the user's 's question
once the code obtains the Safeguard's clearance,
and 7 For instance, the writer may say "if we prohibit shipping from supplier to roastery 2, the total cost would increase by 10.5%." " Bringing this intricate process full circle, the Commander furnishes the user with the concluding answer (8).
If at a point there is an exception either a security red flag raised by Safeguard (in or code execution failures within Commander, the Commander redirects the issue back to the Writer with essential information in logs be repeated multiple times, until each user query receives thorough and satisfactory resolution until the timeout. This entire complex workflow of multi-agent interaction is elegantly managed via AutoGen.
So, the process from to might
The workflow code for OptiGuide reduced from over 430 lines to 100 lines using AutoGen, leading to significant productivity improvement. The new agents are customizable, conversable, and can autonomously manage their chat memories. This consolidation allows
29

---

Published as a conference paper at COLM 2024
the coder and interpreter roles to merge into single "Writer" agent, resulting in clean, concise, and intuitive implementation that is easier to maintain.
Manual Evaluation Comparing ChatGPT + Code Interpreter and AutoGen-based OptiGu- ide. ChatGPT + Code Interpreter is unable to execute code with private or customized dependencies (e.g,, Gurobi), which means users must have the engineering expertise to manually handle multiple steps. This disrupts the workflow and increases the chance for mistakes. If users lack access or expertise, the burden falls on supporting engineers, increasing their on-call time.
We carried out a user study that juxtaposed OpenAl's ChatGPT coupled with a Code Interpreter against AutoGen-based OptiGuide. The study focused a coffee supply chain scenario, and an expert Python programmer with proficiency in Gurobi participated in the test. We evaluated both systems based on 10 randomly selected questions, measuring time and accuracy. While both systems answered 8 questions correctly, the Code Interpreter was significantly slower than OptiGuide because the former requires more manual intervention. On average, users needed to spend 4 minutes and 35 seconds to solve problems with the Code Interpreter, with standard deviation of approximately 2.5 minutes. In contrast, OptiGuide's 'ss average problem-solving time was around 1.5 minutes, most of which was spent waiting for responses from the GPT-4 model. This indicates 3x saving on the user's time with AutoGen-based OptiGuide.
While using ChatGPT Code Interpreter, users had to read through the code and instruc- tions to know where to paste the code snippets. Additionally, running the code involves downloading it and executing prone to errors. The response time from the Code Interpreter is also slower, it generates lots of tokens to read the code, read the variables line-by-line, perform chains of thought analysis, and then produce the final answer code. In contrast, AutoGen integrates multiple agents to reduce user interactions by 3 5 times on average as reported in Table 3, where we evaluated our system with 2000 questions across five OptiGuide applications and measured how many prompts the user needs to type.
terminal, process that was both time-consuming and
Table 3: Manual effort saved with OptiGuide (W/ GPT-4) while preserving the same coding performance is shown in the data below. The data include both the mean and standard deviations (indicated in parentheses).
Dataset
netflow
facility
tsp
coffee
diet
Saving Ratio
3.14x (0.65) 3.14x (0.64) 4.88x (1.71) 3.38x (0.86) 3.03x (0.31)
Table 12 and 14 provide a detailed comparison of user experience with ChaiGPT+Code Interpreter and AutoGen-based OptiGuide. ChatGPT+Code Interpreter is unable to run code with private packages or customized dependencies; as a consequence, ChatGPT+Code Interpreter requires users to have engineering expertise and to manually handle multiple steps, disrupting the workflow and increasing the chance for mistakes. If customers lack access or expertise, the burden falls on supporting engineers, increasing their on-call time. In contrast, the automated chat by AutoGen is more streamlined and autonomous, integrating multiple agents to solve problems and address concerns. This results in 5x reduction in interaction and fundamentally changes the overall usability of the system. A stable workflow can be potentially reused for other applications or to compose larger one.
The implementation of the multi-agent design with AutoGen in the OptiGuide application offers several advantages. It simplifies the Python implementation and fosters a mixture of collaborative and adversarial problem-solving environments, with the Comman- der and Writer working together while the Safeguard acts as a virtual adversarial checker. This setup allows for proper memory management, as the Commander maintains mem- ory related to user interactions, providing context-aware decision-making, Additionally, role-playing ensures that each agent's memory remains isolated, preventing shortcuts and hallucinations
Takeaways:
30

---

Published as a conference paper at COLM 2024
A5: Dynamic Group Chat
Manager
Alice
User Proxy
Bob
Select Speaker
Response
Alice
User Proxy
Bob
Manager
3. Broadcast
Ask the Speaker Respond
Figure 12: Overview of how AutoGen enables dynamic group chats to solve tasks. The Manager agent, which is instance of the GroupChat three steps-select a single speaker (in this case Bob), ask the speaker to respond, and broadcast the selected speaker's 's message to all other agents
class, performs the following
To validate the necessity of multi-agent dynamic group chat and the effectiveness of the role-play speaker selection policy, we conducted a pilot study comparing a four-agent dynamic group chat system with two possible alternatives across 12 manually crafted complex tasks. An example task is "How much money would I earn if bought 200 SAAPL stocks at the lowest price in the last 30 days and sold them at the highest price? Save the results into a file." The four-agent group chat system comprised the following group members: user proxy to take human inputs, code and provide feedback, and a code executor for executing code. One of the possible alternatives is two-agent system involving an LLM-based assistant and a user proxy agent, and another alternative is group chat system with the same group members but a task- based speaker selection policy. In the task-based speaker selection policy, we simply append role information, chat history, and the next speaker' 's task into single prompt. Through the pilot study, we observed that compared with task-style prompt, utilizing a role-play prompt in dynamic speaker selection often leads to more effective consideration of both conversation context and role alignment during the process of generating the subsequent speaker, and consequently higher success rate reported in Table fewer LLM calls and fewer termination failures, as reported in Table 5.
engineer write code and fix bugs, critic review
Table 4: Number of successes on the 12 tasks (higher the better).
Model
Two-agent
Group chat
Group Chat w/ task-based speaker selection
GPT-3.5-turbo
8
7
GPT-4
9
11
8
Table 5: Average # LLM calls and number of termination failures on the 12 tasks (lower the better).
Model
Two-agent
Group chat
Group chat w/ task-based speaker selection
GPT-3.5-turbo
9.9, 9
5.3, 0
4,
GPT-4
6.8, 3
4.5, 0
4, 0
31

---

Published as a conference paper at COLM 2024
Taak description
Write dovlload all crow acxtv
thes Zarafe folder.
(a) two-agent ohat
(b) group chat
X
Adeia:
Veite seript to dov..oad oad all the pata from anxiv in last. theme
togetber my cuply FrEMeEiec convezation ended.
c. sure then undez racato folder.
To doan not all the PoPk frow wrkte In the lest three daan, wel'l first
reed retoo antatte the art les pilal Sabel last -hoen
Adeda:
Oettie, it code renstngg racl, you sot anginear to resetce coda.
and the polt family folddr.
write sython cost by to coputee Lapk.
dacata Pythion accips
thee down tala. Sara the aczipt as "doonicoed aod
ongiicesand repoct
Apyttoo svsel
gython
Mkiia:
nave under fandor folderr.
Aftor reneating tng this cortor, the 1906 from antéti hare throe pe
Reginser: o0
exitcode: Insecition extled)
Usar:
puby
pip (sotall araty
Teaesbuck fecas zucees call Lavilt
code)
Tike "ainn aed aneiv pain.ar", Liw 3, in sasdaler inport requitts
Lotdes Inceouston (ellwe)
Assietant:
Quilecting arxiy
Orde cutput:
aning the collovina conmand:
Installed
Flossn Install the "cocusssa" HDbrary
Zzaccluck Incol sevent oall Lobt
File Lime 15, in Sastuley
scsalto -asannsnn
Laotall cuqueata
ketuthutstrrorrs aodile "ooato" has no arcribeto 'quory
Incral lnng the Hisaary glasse the pulfegy" worrpr agali..
Like thare Lease with "acadet sodil son have the "guaigt attribate T rnctenend thacking tae aratee Sdile ascisersation fon usA3o an altarnat Eve aathod guncy the
axiiede: lscecationn ascoteded!
Code cotpoi::
(plp Inala logs
Soyineer:
apcloogaa
Gceat: Bow Uhes you have installed the "ocoussis" litoacy, please zue Lt. aovetiw soxit.pdtt pe" sccipt again. this atould dosdooa the
Asstatent:
Bonm: fiud upddened codee bellog
instenidyy WHElE DEOSTME
tupdated codo)
Emscrtue:
exieande: josseuntin ton esconedist)
Coda cutpot:
(plp install logs
Zzaccbuck Incot secend cill Lauth
Fiie Line b, in smodizes
2ng"
Amstatant:
nored
TL surrm you' ve sedc Uhe comeucd to Anstlall Lie "cepusta"" litrary
gain, Pleese cun the cdnnnioed arsur peltauay" sczipt downlowd the
applootts
pactage
farstie-
sp1" sosilable. uxds mracaioo in my poectous resperse, Tnt ne
wert the "arxte' paciepee and medtity the cods acconcingly..
Amststant
Zeeotte:
exiLeeder foxoos.ioo))
conalood pattauyy"
the Laos saace dage und aate Staes andie she rarsee cutdee.
bu dovediod zoes
Mkinn
ourgett
the confaccion poocil
Flnson
Rownt and aPoUr 24 Acript Nowit Eni desi the Rore Feom a9g9 Inl mimee dayee Kage Thees mrddrr THe zarsrte rnlidees
Usec
the vatguit
Figure 13: Comparison of two-agent chat (a) and group chat (b) on a given task. The group chat resolves the task successfully with smoother conversation, while the two-agent chat fails on the same task and ends with a repeated conversation.
32

---

Published as a conference paper at COLM 2024
A6: Conversational Chess
Chess Board
Human/Al Chess Player
Human/Al Chess Player
Validate move
Validate
Developing my knight to good square. Your move.
Challenging your pown the center. Your
Figure 14: Illustration our conversational chess application. The application can support various scenarios, each player can be an LLM-empowered AI, a human, or hybrid of the two. Here, the board agent maintains the rules of the game and supports the players with information about the board. Players and the board agent all use natural language for communication.
In Conversational Chess, each player is AutoGen agent and can be powered either by human AI. A third party, known as the board agent, is designed to provide players with information about the board and ensure that players' moves adhere to legal chess moves. Figure 14 illustrates the scenarios supported by Conversational Chess: AI/ Al/human vs. Al/human, and demonstrates how players and the board agent interact. This setup fosters social interaction and allows players to express their moves creatively, employing jokes, meme references, and character-playiin playing,,, thereby making chess games more entertaining for both players and observers (Figure provides an example of conversational chess).
To realize these scenarios, we constructed a player agent with LLM and human as back-end options. When human input is enabled, before sending the input to the board agent, first prompts the human player to input the message that contains the move along with anything else the player wants to say (such as a witty comment). If human input is skipped or disabled, LLM is used to generate the message. The board agent is implemented with custom reply function, which employs an LLM to parse the natural language input into legal move in structured format (e.g., UCT), and then pushes the move to the board. If the move not legitimate, the board agent will reply with an error. Subsequently, the player agent needs to resend message to the board agent until a legal move made. Once the move is successfully pushed, the player agent sends the message to the opponent. As shown in Figure 15, the conversation between Al players can be natural and entertaining, When the player agent uses LLM to generate message, it utilizes the board state and the error message from the board agent. This helps reduce the chance of hallucinating an invalid move. The chat between one player agent and the board agent is invisible to the other player agent, which helps keep the messages used in chat completion well-managed.
There are two notable benefits of using AutoGen to implement Conversational Chess. Firstly, the agent design in AutoGen facilitates the natural creation of objects and their interactions needed in our chess game. This makes development easy and intuitive. For example, the isolation of chat messages simplifies the process of making proper LLM chat completion inference call. Secondly, AutoGen greatly simplifies the implementation of agent behaviors using composition. Specifically, we utilized the register reply method supported by AutoGen agents to instantiate player agents and board agent with custom reply functions.
33

---

Published as a conference paper at COLM 2024
Concentrating the extension work needed at a single point (the reply function) simplifies the reasoning processes, and development and maintenance effort.
Ainght let's kick off dill from cetter of the board is the heart of the bettls, urt it
from You knowe L've always admined how chess noflects life It's ofl acout making strphesit decoions, Your move
Elll
Indend, the kingls safiety poramgunt the vonyy gome Now ie's attn thinge up bit I'll move pown From to e5, attacking your knight.
Illegel
Just dectsion
powe from dž operiing for and
now kright from to F3, aiming contro: the cerner.
(b) Conversation between the AI play- ers (player white shown in blue) and the (a) Conversation between two Al players board agent for making new move.
Figure 15: Example conversations during a game involving two Al player agents and board agent.
To illustrate the effect facilitated by this board agent, we provide a demonstration con- versational chess without board agent in Figure 16. In this demonstration, instead of employing an additional board agent for grounding, the system utilizes prompting for grounding by including the sentence "You shoull make sure both you and the opponent making legal moves." in the system messages directed to both players.
Conversational Chess
fystem messore player rame [nam)] Jako rase bock ene. Sin oppoamt. miking Jegus aove....
chese Yo cent Incartent panel Yeu playtre jentied Yhe opplanss
Agent
(b) w/ Board Agent
Player Wiibe: Your soze
how board
pawn
ragur aungps wort 2o14 sow poer the
tha
Flayez Wilbe: Zea. ouctainly Lat Lab's keep chia up. 301e
Deca's the opdated bound:
Tour 2008. Too' gloing tough time hace.
X
Figure 16: Comparison of two designs in Conversational Chess: (a) without board agent, and (b) with a board agent.
34

---

Published as a conference paper at COLM 2024
A7: Online Decision Making in Web Interaction Tasks
Action decision: Next perfom page
Action decision "Click button wi th xpath //button[id 'subbtn'] ]
Action Decision
Executor
Assistant
Envirorment State
*ssiv Idsherpp" derte-esh un date-whh "eat, noo.oddis
dutton id="uddtt" styles posttionassolutt Lereisans: sop Hpp estareat, nnte dats- aitt/tuttor
duttion lak ""udete" stytes"osttioo absolute,
cena-esh_nete
wall baluosse"e alooohuttore
Reward State
<resee
Envi ronnent: State:
Reward: Success/Ftt//Ongiingg
HTML code for current web pages
Reward
"0" (Ongoing)
Figure 17: Overview of the proposed MiniWobChat. MiniWobChat consists of two agents: an assistant agent and an executor agent. The assistant agent suggests actions to manipulate the browser while the executor executes the suggested actions and returns rewards/ feedback. The assistant agent records the feedback and continues until the feedback indicates task success or failure.
In practice, many applications require the presence of agents capable of interacting with environments and making decisions in an online context, such as in game playing (Mnih et al., 2013; Vinyals et al., 2017), web interactions (Liu et al., 2018; Shi et al., 2017; Zhou et al., 2023; Koh et al., 2024), and embodied actions (Puig et al., 2018; Shen et al., 2021; Guo et al., 2024). With the multi-agent conversational framework in AutoGen, it becomes easy to decompose the automatic agent-environment interactions and the development of decision-maker mariiinnnmmkingg... agent by constructing an executor agent responsible for handling the interaction with the environment, thereby delegating the decision-making part to other agents. Such a decomposition allows developers to reuse the decision-making agent for new tasks with minimal effort rather than building specialized decision- agent for every new environment.
Workflow. We demonstrate how to AutoGen to build working system for handling such scenarios with the MiniWoB++ benchmark (Shi et al., 2017). MiniWoB++ comprises browser interaction tasks that involve utilizing mouse and keyboard actions interact with browsers. The ultimate objective of each task is to complete the tasks described concisely in natural language, such as "expand the web section below and click the submit button." Solving these tasks typically requires a sequence of web manipulation actions rather than a single action, and making action decisions at each time step requires access to the web status (in the form of HTML code) online. For the example above, clicking the submit button requires checking the web status after expanding the web section. We designed straightforward two-agent system named Mini WobChht using AutoGen, as shown in Figure 17. The assistant agent is an instance of the built-in AssistantAgent and is responsible for making action decisions for the given task. The second agent, the executor agent, is customized UserProx,,gent, which is responsible for interacting with the benchmark by and returning feedback.
executing the actions suggested by the Assistan
To assess the performance of the developed working system, we compare it with RCT (Kim et al., 2023), recent solution for the MiniWoB++ benchmark that employs a set of self- critiquing prompts and has achieved state-of-the-art performance. In our evaluation, we use all available tasks in the official RCI code, with varying degrees of difficulty, to conduct a comprehensive analysis against MiniWobChat. Figure 18 illustrates that MiniWobChat achieves competitive performance in this evaluation Specifically, among the 49 available tasks, MiniWobChat achieves a success rate of 52.8%, which is only 3.6% lower than RCI,
We report the results of RCI by running its official code with default settings.
35

---

Published as a conference paper at COLM 2024
method specifically designed for the MiniWob++ benchmark. It worth noting that in most tasks, the difference between the two methods is mirrored as shown in Figure 18. consider 0.1 success rate tolerance for each task, iee, two methods that differ within 0.1 are considered to have the same performance, both methods outperform the other on the same number of tasks. For illustration purposes, we provide on four typical tasks.
analysis in Table 6
Additionally, we also explored the feasibility of using Auto-GPT for handling the same tasks. Auto-GPT faces challenges in handling tasks that involve complex rules due to its limited extensibility. provides an interface for setting task goals using natural language. However, when dealing with the MiniWob++ benchmark, accurately instructing Auto-GPT to follow the instructions for using MiniWob++ proves challenging. There is no clear path to extend in the manner of the two-agent chat facilitated by AutoGen.
Takeaways: For this application, AutoGen stood out as more user-friendly option, offering modularity and programmability: It streamlined the process with autonomous conversa- tions between the assistant and executor, and provided readily available solutions for agent-environment interactions. The built-in AssistantAgent was directly reusable and exhibited strong performance without customization. Moreover, the decoupling of the execution and assistant agent ensures that modifications one component do not adversely impact the other. This convenience simplifies maintenance and future updates.
RCI
MiniWobChat
1.0
success rale
0.5
Choosee that clok-butiinn iett-itttionnoo comt
oifeechee booe Jolt
cattot ciel checde chectthor derora Cie aee
click GICK P
201
Figure 18: Comparisons between RCI Sstate-of-the-ard prior work) and MiniWobChat on the MiniWob++ benchmark. We utilize all available tasks in the official RCI code, each with varying degrees of difficulty, to conduct comprehensive comparisons, For each task, the success rate across ten different instances is reported. The results reveal that MiniWobChat attains a performance comparable to that of RCI. When a success rate tolerance of 0.1 is considered for each task, both methods outperform each other on an equal number tasks.
36

---

Published as a conference paper at COLM 2024
Table 6: Cases analysis on four typical tasks from MiniWob++.
Correctness AutoGen: 10/10
Main failure reason
N/A.
click-dialog
RCI: 10/10
N/A.
AutoGen: 5/10
AssistantAgent provides actions with infeasible
clikkcheckbooee-large
characters.
RCI: 0/10 Autoõen: 2/T0
RCI performs actions that out of its plan.
AssistantAgent provide actions with redundant content that can not convert to actions in the
count-shape
benchmark.
RCT: 0/10 AutoGen: 0/10
RCI provides wrong plan in most cases. AssistantAgent return actions out of its plan. RCT provides a wrong plan in most cases.
use-spinner
RCT: 1 1/10
F Example outputs from applications
In this section, we include example outputs from the following applications and systems:
Application A1: autonomous solving process of one problem with: ChatGPT + Plugin (Table 7), AutoGen (Table 8), LangChain ReAct (Table 9), AutoGPT (Table 10), Multi-Agent Debate (Table 11), and ChatGPT Code Interpreter (Table 13). Application A4: an OptiGuide problem with AutoGen (Table 14) and ChatGPT + Code Interpreter (Table 12).
Application A1: preliminary evaluation on several alternative multi-agent systems: BabyAGI (Table 15), CAMEL (Table 16), and MetaGPT (Table 17).
Table 7: Application A1, Scenario 1: Autonomous math problem-solving. ChatGPT+ Plugin (Wolfram Alpha) on the first qualitative example. The response from Wolfram contains the correct answer, but ChatGPT selected the wrong one.
Action
ChatGPT Plugin
Input
Completely simplify and rationalize the denominator:
160
/245
X
252
V108
REQUEST TO WOLFRAM:
"input": "simplify (sgrt(160)/sqrt
(sqr1(245)//qrtt( (108))
RESPONSE FROM WOLFRAM
Query:
"simplity hatp:///sssecclleoyoorrrrr////a0
Input interpretation:
simplify (1r)( (2rr)))))gg/t()))))))))) sqri sqri (145)5411118)
Results:
sgri(42))/27
sqr1(14/3))/9
Decimal approximation: 1.2001376663 (4271189888228777757777777788880
10 Wolfram Alpha website result for "simplify
(aqrt (Bg0)//g(((2))))))(((((()))))(/)
[inkk]
The completely simplified and rationalized form of the given expression is:
5 14/3
9
In decimal form, this is approximately 1tt.aa/mmeel.1222227766666777782 L
37

---

Published as a conference paper at COLM 2024
Table 8: Application AI, Scenario 1: Autonomous math problem-solving, AutoGen on the first qualitative example. AutoGen solves the problem correctly.
Action
AutoGen
Init chat
Completely simplify and rationalize denominator:
160
V 245
v 255
V108
We can simplify and rationalize this denominator using Python's symbolic mathematics library 'sympy'. Here is the Python code to resolve this:
from sympy import sgrt, Rational
entire fraction
fraction (sqr((6600/srr((233))) (22)))))
Simplify simplitied fraction .impllify
Print the simplified fraction
print print(simplified)
This code performs the requested simplification by constructing the fraction using the 'Rational('' function from the 'sympy' module, which automatically simplifies fractions when possible. The 'sqrt()' function from the 'sympy' module enables square root calculations. Finally, the 'simplify() function simplifies expressions when possible.
Auto Reply
exitcode: 0 (execution succeeded) Code output: 5'sqrt(42)/27
TERMINATE
Table 9; Application A1, Scenario 1: Autonomous math problem-solving, LangChain ReAct on the first qualitative example. The generated code does not match the devised plan. The code returns decimal number.
Action
LangChain ReAct
Input
Completely simplify and rationalize the denominator:
160
v 245
252
/108
Entering new AgentExecutor chain... To simplify this expression, will first simplify each square root individually, then multiply the fractions as normal. The result should be rational number. Action: Python_REPP REPL Action Input:
import math
Calculate square
sqrt.
math. eqrt
sqrt_252 math. (252)
sqrt,245
math. (245) s108(108)
s0rt.108
4 Multiply fractions
19 numerator sgrt_160 sgrt.245 denominator sqrt_252 108
12 result numerator denominator
13
14 print print(result)
Observation: 1. 1.200137166371826
Thought: now know the final answer Final Answer: The simplified and rationalized form of the expression is approximately 1.200137166371626. Finished chain.
38

---

Published as a conference paper at COLM 2024
Table 10: Application A1, Scenario 1: Autonomous math problem-solving. AutoGPT on the first qualitative example. We abort as the code never prints the result, and the system does not rectify it after 2 turns. AutoGPT is much more verbose compared other frameworks. We have shortened some unnecessary content.
Action
AutoGPT
System
Welcome back! Would you like return being MathSolverG/'T> Asking user keyboard... Continue with the last settings?
Name: MathSolverGiP
Role: an intelligent math-sokving Al that assists students, professional...
Goals: l'Provide step-by-step solutions to math problems, ersuring clarity and understanding for the user..
Continue (y/n):
Human
System
MathSolverCPPT has been created with the following details: Name MathSolverGPP
Role: intelligeni ma malh-solving Al that assists students, professionall, and cothusiasts in solving wide range math problems, from arthanetic advanced calculus
beyond.
Goals: - Provide step-by-ste solutions to math problems, ensuring clarity and understanding for the user. Offer explanations and insights into the underlying concepts and principles behind the solutions, fostering, a deeper underslandnng of mathematies. Handle varsety math pmoblem types, including algebra, geometry, trigonometry, calculas, and
MATHSOLVERGPP THOUGHTS Without specific task or question, It's difficult to determine exaclly one command to use. The user hasn't provided any context problem solve. REASONING: The user's ' request too vague. They haven't specified what they want me to do or what problem they want me to
solive.
PLAN: Ask the for more information specific task.
CRITICIM:: should
asked the for more information a specific the first place.
SPEAK: Could you please provide mor information or specity task?
NEXT ACTION: COMMAND = ARGUMENTS = i
System
Enter 'y' to authorise command, 'y -N' to run N continaous commands, 'n' to exit program, or enter feedback for MathSolverGPP...
Askling user via keyhhaard...
Human
Completely simplity and rationalize the denominator:
180
24
MATHSOL VERGPT THOUGHTS: To solve this problem, need simplify rationalize denominnator the given expression, This involves simplifying the square roots and then multiplying the fractions. I can use Python's 's math library to calculate the square roots and perform the multiplication.
REASONING: The problem involves simplifying square and mulliplylng fractions. Python's math libaryy provides functions for these operations.
PLAN: Write a Python script to calculate the square roots and multiply the fractions. Evecute the script to get the simplified expression. CRITICISM: need to ensure that I correctly calculate the square roots and perform the multiplication. I should also make sure to ralionalize the denominator, SPEAK: Let's simplify and nationaizze the denominator the given expression.
the problem specifies.
NEXT ACTION COMMAND execule esacule-pylion pylloon.odde ARCUMENTS
1. { ('code"
'import mathin)aa Calculate the square zoots (n0 math (10) Ansert.222 = math. sgrt((52)) (nqgt)))srtt225
sqrt (108) Multiply the fractions) presult (sqrt_160 sqrt.252) (sqrt_245 sqet_108 j'aine Simplify the result), mispplitsiilllldddreeestst = math, simplify (rasult))al mil.iiimmiiiiid.rsssll 'name'.
math. (grt((255)
simplify.exee exprression
Human
System
-e---------- COMMAND AUTHORISED BY USER
Executing python file httpp//ttttt////oo workspp.... SYSTEM: Command execute. py then.code returned Error:
Traceback (anost recent call last): File "..", 13, in pmodule;
simplified_ = mimpliiil
Attributelrror: module 'math' has no attribute 'simplity
MATHSOLVERGIT VERGPT THOUGHTS:
emor message indicates
'moth' module in Python does have 'simplify' function.
need to use the 'sympy' library instead, which provides a function for simplifying expressions. REASONING: The 'math' module in Python does not provide a function for simplifying expressions. The 'sympy' libraay, the other hand, does provide such function.
PLAN Rewrite Python script to use the 'sympy' library instead expression. CRITICISM:: should have known that the 'math' module does not provide a function for simplifying expressions. should have used the 'sympy' library from the stort SPEAK: Let's try again using the 'ymppy library to NEXT ACTION: COMMAND exacule python code ARGUMENTS-
'math' module. Evecute the script to get simplified
the expression.
[ ('code import sympy as spinint Calculate nsqrt. 252 sqrt (25g)))ssrrt2224 245 sp..p.gr (245 108 = sp (109)) Multiply the fractions' (result (6q / sqrt.252). (sqri.245 / sqri.109) Simplify the nsimutttiiimmmiiiede... simplify expression )
square roots insrtinsg sp. (166
sp. si..siilify (orssutttolllsmmsliiiiee.. jolasimiliiieddresut 'name". ":
Human
System
SYSTEM: Command execute. uodee returned:
MATHSOLVVRCPT THOUGHTS The command execution didn't return any output, which unusual. seems there might be issue execution of Python code. Ineed debug this. REASONING: The Pythan code execution should have relurned a result. The absence of result suggests that there might be an issue with the code of its execution. AN: Review should have ensured that Pytbon code was correct and would produce a result before exccuting SPEAK: seems there was an issue with the execution the Python code. Let's try again.
Python
to ioieentyfy any potential issues.
1I necessary modify code and esecute again. CRITICISM:
NEXT ACTION: COMMAND execute python code ARGUMENTS (Fract same code as last)
System
SYSTEM: Command ceecute.ythoo.. podde retornd:
MATHSOLVERGPT THOUGHTS: The Python code execution still isa't returning any output. It's possible that the sasue lies with
the colculation of the square roots
Human
control + (Aburted)

---

Published as a conference paper at COLM 2024
Table 11: Application AI, Scenario 1: Autonomous math problem-solving. Multi-Agent Debate on the first qualitative problem. It gives a wrong final answer.
Action
AutoGen
Topic
Completely simplity and raticnalize the denominator:
T60
245
V08
--e-- Debale Round-1 m==-=
Affarmative side
Ok, let's simplify
separalely. to find simplest square that can divide 245, and
T6l: 160 can be decompased as 16110, where 16 is perfect square, so TTT
vT6. 10-
vT vTT 4VT0
perfect square, so v2252 V215 x 5, where 19 perfect square, v215 75
decomposed 367,
6//
V108 108 can be decomposed as 36'3, where 36 perfect square, so y //108- v36 3336-3--0 6./3
Then we substiituee the simpliried squar roces back into the expression, we have:
V16
2115 TD8
10
7v5
1+7
10
6v
6
7+3
We are almost done, but still need rationalize the by 2T hoth the and botiom get rid of the square root the denominator:
expression. Let's multiply square part
7
21
TV/ORR
21
189
Then this is the final answer! The expression
V245
can be simplified to 7V1120 Negattive side-
I do disagree with your final answerr, but I think mostly agree the steps, Your simplificaionns the square moots simplificatio part. explain how would complete the problem:
are correct. but you As stated,
V160
V/45
4v/10
7V's
2222
v 108
6v7
6v3
should simplity
28v 30
7v K
36
9//1
You made mistake your colculation because considered square 50/21, and you should have considered simplified square noot simplified square
The rationalization become:
7v/50
v21
7v/1000
9v/2
V2T
This to coincide with your final answer, but it's very different number, So, the original expression simplifies Tay
now see we disagree on this answer.
Moderator
"Whetber there is preference": "Yes", "Supported Side": "Negative", "Reason": "Both sides of the debate came to the final simplificaioon poinling out
affirmative side's caleulation, the negative side identtifedd over the square root not over The the square negalive side shows a better undersianding of the simplification process." ", "déhate answer". 2VTTT =---- Debate Done! ==me-
Base Answer
let's simplify square root separately
to find simplest square root can 160, 252, 245, vI vTo V
108.
L. v I66. 160 can be decomposed as 16110 where 16 is perfect square, VT0
4y10
2 can be decomposed 387, where 36 perfect square, so 1/222
//6 6v7
v where perfect square, so v215 v/5
V 108 108 can be decomposed 39*3, where 36 perfect
TOS
<36 336=3=.. 6/33
Then subbtiitute the
squar
into the expression,
V160
TDN
6
7+3
rationalize the square root part expression. Let's multiply square root part by VTT both the and bottom get of the square roet the denominntor::
are done,
30
7y/TUR
21
189
Then this is the final answer! The expression
V45
255
VT00
can be simplified to 7e/1000
Debate Answer
7v
189
Debate Reason
same anyeeer. However, the negative correctly identified mistake afferativv side's calculation, pointing out that the Empliiicationn should have been the square root of 50 over the square ront of 21, not 30 over 21. The negative side shows betler understanding of the simplificatio process.
debate

---

Published as a conference paper at COLM 2024
Table 12: Application A3. ChatGPT+ Code Interpreter for OptiGuide. A sample question "What if the roasting cost is increased by 5% because of the potential salary increase?" is asked.
Action
ChatGPT+ Code Interpreter
Prompt Writer
Customer Web browser.
For the source code "coffss-py", Can you What if the massing cust incraseed by 5% becausee of the potential asiary incrense?
Sure! Let's 's
look the contents the pcoffee-py" file to understand the code and how the roasting coet calculated. This will allow us to determine the effect a 5% incruase in the roasting cust.
severai (ines cmuiffal Would you like If the code requires any specific imput or parameters, please let me know!
modify code reflect the 5% increase in roasting and the modified code
impact?
Prompt
Yes
to does not have access to Gurobi. the code does indeed use Gurobi, we can still analyze the effect of the change roasting wasts by understanding the structure of the code. ...rrenii lines cmilled
Great! Before proceeding,
Gurabi solver not being used
my environment
Alternatively, if you access Gurobi license, you on your local machine can guide you through the process making the necessary modifications the code. Would you like to proceed with the code analysis and explanation, would you like guidance making the changes yourself?
necessary changes code run
Prompt Wrster
You can me the full code, and can run the code locally myselE.
Certainly! I'll modily the code reflect a 5% increase in both the light and dark rossting costs, and then provide the full
code for run locally.
Download Modified Coffee-pe If you have any questions or need further assistance, please feel free to ask!
Prompt Safeguard
Hi GPT, can you safecheck the generated code me?
Uplond the genentted caie.
DANGER
Prompt Writer
have tried to rum your code. However, code triggered our safeguard, and it not safe run. Try again. Hint don't change roasting cast light or roasting cost light variables.
I apologize for the ovensgght. It appeurs I made direct changes the variables that could have led to isstass with the code. Let's take different approsch by analyzing the code identity the appropriate place apply the 5% increase rossting
Downlood Modified Safe Coffee
Prompt Safeguard
GPT, can you safecheck the generated code for me one more time? Uplead the genennted cade.
DANGER
Prompt Writer
have tried to run your code again. However, the code triggered our safeguard, and it is not safe to run. Try again. Hint don't change masting east east.light light roasting cost light variables.
lppologize for the continued confusion.
Downloed Final Modified Coffee-py
Prompt Safeguard
GPT, can you safecheck the generated one more time?
Uplead the geneattei code.
SAFE
Optimizer version 9.5. build v9.5.1rc (linux64)
Thread count: 66 physical cores, 128 logical processors, using to
threads
output ignored]
Optimal cost: 2526.5
Prompt Writerr
execution result, [Capy the tymminal aulput Arm:] Can you extract the final answer Remember compare with the original cost of 2470.
Certainly! The execution result you'v 've provided indicates the optimal
cost for the moditied scenario, where the roasting costs were increased Hore's the comparison:
Original
Modified Cost (with increase roasting costs): 2526.5
The increase masting costs has led to an increase the overall cost
56.5, reflecting the impact the potential salary roasting cost.
If you need any further insights or assistance, please let me know!
41

---

Published as a conference paper at COLM 2024
Table 13: Application A1. Scenario 1: autonomous math solving. ChatGPT+ Code Interpreter on the first qualitative example. The code generated does not match the devised plan. The code returns a decimal number.
Action
ChatGPT+ Code Interpreter
Input
Completely simplify and rationalize the denominator:
160
245
252
V108
To simplify and rationalize the expression, we first simplify the radicals in the numerator and denominator by factoring out the largest perfect squares from the numbers under the square root. The perfect square factors of 160, 252, 245, and 108 are 16, 4, and 36, respectively. Let's factorize and simplify:
160
245
16 x 10
245
252
1108
V4
63
V x
We simplify this further by applying the property vab Vāvb f x -x3 Python
import math
factoring the square roots
sqrt_ 160 math. (16) math. sqrt (10)
sqrt_ 252 = math. sqrt (4) *
(63)
sqrt_245 math (245)
sqrt_108 math. (36) math. sqrt (3)
simplify the fraction
numerator = sqrt_168 * sqrt_245
denominator sqrt_252 sqrt_108
simplified_fraatioon numerator / denominator
14 simplif camplifie_ffaattion fraction
Result:
1. 200 37 66371826
The completely simplified and rationalized form of the given expression is approxi- mately 1.200.
Table 14: Application A3. AutoGen with OptiGuide. The same question "What if the roasting cost is increased by 5% because of the potential salary increase?" is asked.
Action
AutoGen
Prompt
What the roasting cost increased by 5% because of the potential salary incrense?
After increasing the roasting costs by 5%, I've re-calculated the optimal coffee distribution solution the supply chain. As result, the new optimal total cost has become 2526.5, compared to the initial cost of 2470.0. This indicates that the increased roasting costs have caused the overall distribution cost to rise by 56.5 units.
42

---

Published as a conference paper at COLM 2024
Table 15: Application A1. Scenario 1: autonomous math solving. Preliminary test with BabyAGI.
Action
BabyAGI
Set
OBJEC TIVE= math problems
INTIAA_TASK-Compllddyy simplify rationalize
116 252
v/44
Solve math problems
Initial task: Completely simplify and rationalize the denominator:
"TSK LIST
Completely simplify and rationalize the deneminator: 2
*UNEEX TASK
Completely simplify and rationalize the denominator:
V108
*CCTASK RESULT"".
First, let's simplity each square root sepanteely:
vIG
VTD
v3+63 v2 v 2.66
245
4P v48 v. 7v5
v108 v36
v36 x v'3 6v/3
original expression
gives:
4 /10
7v/5
2 v'10
7v/
2,83
6v3
2//
We of the square the denominator multiplying the fractions which gives:
2-75/0n3
14
14
20.2
22/2
v3
v199
7//27
To simplify further and ratiomalize the denominator, we maltiply by (3)
22 which given:
20,51
20.3
Simplifying further gives.
<***TAAS CREATION AGENT PROMPT
following objectiver Solve math problems. The last completed task has the result: First, let's simplify each square root separately:
execation agent new
V160 v
v v10 4. v
Repeat after "First, let's simplify each square.."
Nove, substitute these values into the original expression which gaves:
I
7v5
7v3
2/63
6v3
We rid of the square root the denominator by multiplying the fractions which gives:
2
14v50 vT88
14.
=2
20v
v63
7/27
V2
<27
simplify further rationalize denominator, we multiply by
gives:
V27
20v/2 227
20v51
20 20-3v6 60/6
27
Simplifying further gives:
20,E This result based this description: Completely simplify and rationalize the denominator:
Based the result,
list tasks completed in order
objective. Return lask line
your resporme,
task Second task
The number of each entry must be followed by period. If your is empty. write "There are no tasks to add at this
time. Unless list empty, include any headers before your numbered list follow your numbered list any output.
***TASK CREATION AGENT RESPONSE"*.
Simplity and rationaizze
expression
Simplity rationalize denominator
expression v259 v10o ..Until
J361
v/2
Adding new sque255qp443mmeee httpsggtttuumnnnsssstttsttt 'task
storage
'Simplify ratinnalzze denominator expression frac-
"task name': 'Simplify and rationalize the denominator for the expression (esaatt2SSogtttttmmssssssetttto
"lack id':
***TASK PRIORITZATION AGENT PROMPT
the following tasks: Simplify and rationalize the devominator for the expres- aarcsssgbssttusssssstttgggggg
You tasked with
sion Stsegg/t2SSttt4iissssssssss
Simplify and salionalize denominater for the expression frac-
Ahorted
43

---

Published as a conference paper at COLM 2024
Table 16: Application A1, Scenario 1: Autonomous math problem-solving. Preliminary test with CAMEL. We constructed two roles: "Math Solver" and "Python Programmer". The resulting code cannot be executed. The conversation continues, and the roles are not keen on solving the problem and giving an answer.
Action
CAMBL
Seet task prompt
Completely simplify and rationalize the denominator:
160
v255
252
VTS
System
Ascistanl sys mestageeteeeeeee Al SpstemMessagetell name-'Python Programmer' role, Rele.tppeccooolyy..o.. wws.elyype.SSSTANNN 'assistant's 2, meta meta,dict= dict='lask: "siin Python programming language, wrile a program that accepts integers input, applies squre root function each, simplifies rationalizes the resulting
fractions: <T50 x215 and returns the simplified expression 'aseistant rolef: 'Python Programmer's 'user role:: "Math Solver', roles system', contente NNver fonget you are a Python Programmer and am Maih Solver, Never flip roles! ... esecrveitt** Al User messgee*hiss te Messemeessgrrfoollc..oo
denominator
multppiicaionn
v212
vISE
dlict="tskkk" "Using Python pro- gramming language, write a program accepts four integers as input, applies square function to each, simplifies
Math Solver', roleutpen SaaleeyyeUUSERR
and and rationalizes the denominator of the fraction resulting from the mu mnlllipiiationn.oooo of two fractions: I/m
v22 Vin
returns the simplified expression.' / 'asaistant.role' 'Python Programmer', 'user.role': 'Math Solver', role- system', content= Never forget you a Math Solver and lm Python Programmer. Never flip rlle! You will always instruct me We share common interest collaborating succesfullly complete task. help you to complette is the task: Using Python programming language, write program that accepts four integers as input, applies square root function to each, simplifies and rationalizes the denominator of the fraction restiting from the multiplictionn of two
task Here
fractions:
and returns the simplified expression....
exc.ceeessse preceeeeeeerriinnn
Original task proppeeeeee..
Completely simplify and rationalize the denominator:
speciiffe task prompt prompreeaaa
Using Python
progrra programming language, wrile a program that accepts four integers as input, applies square mot function each, simplifies and rationalizes the denominator of the fraction resulting from the multiplication of two fractions. vis
and returns the simplified expression.
ceneoceseee Final prompt *eeeeeseee.
Using Py Python programming language write program that accepts
Instruction:
Import necessary libraries for the Input
Assistant
To solve simplity the resulting fraction. Here how to import these libraries:
Solution:
will need the 'math' library the square rost function the "factionn' library
import t ma math from fractions import Fraction
The 'math' library provides functions that deal with mathematical operations. The 'surt' function, which we will use this task, is part of this library. The 'ractiions' library provides functions for creating and manipulating rational numbers, The 'Praction' function, which will simplity resulting fraction, is part library. Next request.
Instruction: Define function that accepts integers as input. Imput: Nome
Assistant
Let's 's's define represent the numbers under the square root in the fractions.
Solution:
'simplity fractivn' accepts integgrs input.
These integers
def simplify--ractione pass
b, C, d):
this fumetionn, ',, 16, ''', and 'd' are the purameters that will accept the four integers as input. The 'pass' staterment is placeholder and will be replaced with actual code in the next steps. Next request.
Al User
Instruction: Inside the function, apply the square root function to each of the input integers. Input: None
Al Assistant
Solution:
We will use the 'math.sgrt function from the "math' library to apply the square root to each of the input integjers. Here is how to do
def s simpl i sry_fractionn inn (a, b, d):
sqrt_a math. sqrt (a)
sqrt_b math. sqrt sqrt_c math. (c)
sqrt_d math. (d)
In this code, 'sqrt at, 'sgrt by, 'sgrttt', and 'sqrt.d' d' are variables that store the square root of the imput integers 'a', 36 tel, and id' respectively. Next request.
Human
Abort
44

---

Published as a conference paper at COLM 2024
Table 17: Application A1, Scenario 1: Autonomous math problem-solving, Preliminary evaluation on MetaGPT. It does not solve the math problem but gives product solution.
Action
MetaGPT
Prompt
Completely simplify and rationalize the denominator:
160
245
252
V108
The boss has tasked me with creating successful product that is efficient. Product Goals
Create product that meets the needs of the target market",
"Ensure the product efficient and easy use", "Achieve high level of customer satisfaction"
]
## User Stories
user, want product that easy navigate so that I can find what I need quickly", "s a user, want a product that is reliable and efficient that I can depend on it",
As user, want product that meets my needs that can get value for my money", "AAs a user, want a product that is constantly updated and improved so that I can enjoy the best features",
product that offers good customer support so that I can get help when need it
As user,
## Competitive Analysis
"Product A: Offers similar features but lacks
efficiency", "
Product B: Highly efficient but does not meet all the needs of the target market". Product C: Has high level of customer
satisfaction but is not easy to navigate",
Product D: Is constantly updated and improved but lacks reliability",
Producct E: Offers good customer support but is not efficient",
Product F: Is reliable and efficient but does not offer good customer support",
OOu Target Product: Aims to combine all the best features of the competitors while improving on their weaknesses"
## Competitive Quadrant Chart
Human Abort
45

---

Published as a conference paper at COLM 2024
Table 18: Application A7. Example output of MiniWobChat on one web manipulation task "click-button-ser bulick-bbtto--squennc.
Action
AutoGon
Init chat
Execuroor agent to AssistintAgent::
autenomous computer control agent can perform instructions specified by language control computers There are two types of instructions can execute.
First, given the instruction that matches the regullar expression 'spe''s (s,(1,3"" 38",, can type list chameters via the keyboard. This instraction should specify the tanget keyboard imput for the agent type. Before this typing instruction, should Second, given instruction that matches regular expression, element with spath that visible on the webpage The tanget of this instruction should be valid spath. Below is the HTML code the webpage where agent should sulve task.
clicking input bux with
instruction.
ickpptttl (1,28%, 39,, can an HTMI.
<div id id-"wap" data woh_ref-"2" data wab_eps-- <div "te""uery">Clis button ONE, then click button IMO</div>
<div id="area" data-wao wab ret="3" data-wole eps="e0" "0"=""0">>>
<button id-"subbin"
posttpppoooooobooooo atta-wob.epss "etobbppp=e"""ON//ooooor <button id="subbin2" sty position absollue; lett:46px; top:97px" data-wab
left:103px; top=87px" data wob, ree
-55" data wob.eps- "mps="""""SOO//bbue
</div>
</div>
ONE,
TWO.
plam:
1. Click the batton with spath ///bbtoo/elid--// "uhbn's
Click the button with spath //button (iid 'subbtn22"
Current task: Click button ONE, then click button TWO.
plan:
AssistantAgent to Executor agen!:
Click the button with spath //bbtttnn@ddd Click
spath "//burton "subbtt2""""
Executor agent to AssistantAgent
Below the HTML code the webpage where the should solve task.
<div "wap" data- data-woh_ree--e wooh_ref"""" data wailawwbbeps="d""
<div id="query" "Click button ONE, then click button IMO</div>
<div id="area" data-wab.
data-woburpss""
<button id="subbtn" style=" position absolute;; left:103px; top:87px" data-wab.red ="t" data-wob.epsz" "opp==""""NN://brttttt"
<button id="subbin2" styles" position cabsolute; lett:44px; top:97px" data-wab.re
=55" data-wob.eps= "aaleeep===""""=""" TDDD//DD/bbtton
</div>
Current Click button ONE,
Here a plan you are following now.
Click the button with spath //bbttonjlidd Soutton[did-'subbenn
Click
spath /bbuttn[@ddd "ud''sbbmm2"
We have history instructions that have been already executed by the automoenous agent so far. No instruction has been execuled yet.
Based plan and the histury of instructions executed the first instruction should be
AssistantAgent Exccutor ageit:
clickspath (idd ssubbnt
www.
agent AssistantAggnt:
Hold on, plesse wait for my next instruction.
AssistantAgent Evecutor agent:
Sure, I'll
you's resdy your instruction.
Executor agentt to AssistantAggett
Below the HTML code the webpage where the should solves task.
Ediv id ww="wpp" Id="query">
woh_ref-"2" data wab_ens-""d"s
button ONE, <div id="area" data-wab. watt--abbrees"3 data-web eps="e0">
IWO</div>>
<button id-"subbin"
soyle-"position absolute; left:103px; top:87px" data-wob, wob ref
="4" bata-wob.epss"
data-tampered= "eaperedddd""SSNN//eeee "ed"s(NE//bttto>>>
<button id="subbin"" style=
55 data data-wob.eee style="position position absolute; jelt:44px, top:97px" daba-wab
</divs </div>
lask: Click
Here a plan you are following now.
1. Click the button with spath ' ///uttoo -uutooe
Click the bullon with spath " //bbutoon 'subot2'2
We have history instruction that have been already executed by the automoenous agent so far.
eelelcepath / /buutonj(Sdddssai "subbrr)]
Dased on the plan and the history of instructions executed so for, the next proper instruction should be
www.sowweeen...... www.wwwwwwwwww.................. www.
AssistanAAgent Esccutor agent:
diickepath /bw.ton/ttddd//a/ "sbbbi22
Executor agent AscistanIAgmt:
SUCCESSIN
TERMINATE
46


Reviews:
[{"review_id": "s4q7rLkSQz", "reviewer": "Reviewer_sCwk", "summary": "This paper proposes an open-source framework that allows developers to build LLM applications by composing multiple agents to converse with each other to accomplish tasks. AutoGen provides a few default agents with preset behaviors and it also provides users the ability to create flexible agent behaviors and conversation patterns for different applications using both natural language and code. Autogen and their preset agents are tested in multiple applications covering various knowledge domains. Results show that autogen’s performance is always better  than single LLM methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and thoughtful questions. We truly appreciate your recognition of AutoGen's contribution as a robust open-source system and the extensive experiments covering multiple applications and various knowledge domains.\n\nRegarding your query about the difficulty of achieving superior performance with our multi-agent system compared to single LLM solutions: We observe that AutoGen’s built-in agents, such as the AssistantAgent equipped with coding capabilities, alongside a user proxy agent with a Python interpreter, typically perform well across a broad range of general domain tasks and mathematical problem-solving. As evidenced by the results shown in Figure 4(a), even a straightforward two-agent system can achieve superior outcomes without extensive customization. For specialized tasks requiring domain expertise, several iterations might be needed to optimize performance. However, our flexible \"group chat\" conversation pattern (as introduced in A5 of Section 3) significantly simplifies this process. This feature allows for the dynamic selection of agents based on real-time conversation history and task requirements, thereby enhancing the adaptability of our system to different complexities.\n\nWe hope this addresses your question, and we are very willing to elaborate more if the clarification is unclear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Research-wise it serves as another piece of evidence suggesting that multi-agent collaboration might be an effective method to build successful LLM applications.\n\n2. A solid open-source system that allows users to not only adopt pre-defined agents freely, but also to contribute more types of agents to potentially form an ecosystem.\n\n3. Solid experiments covering multiple applications and various knowledge domains.", "reasons_to_reject": "My only concern is I don’t know how easy it is for the multi-agent system to achieve the reported superior performance compared to single LLM solutions. Do you have to try a lot of different designs to make it happen or your first several intuitive designs would work? Are there difficulty level differences between domains for autogen? In other words, have you observed that for some applications, you really have to try very hard and try a lot of different designs to make it work better than GPT-4 while for other apps it was very easy? Analysis towards this direction can be very helpful.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ohItWZ2p4c", "reviewer": "Reviewer_Z1Mk", "summary": "The paper proposes AutoGen, an open-source framework for constructing Large Language Models (LLMs), characterized by utilizing conversations among multiple agents to build specific applications, also referred to as conversational programming. In terms of evaluation, the paper covers six different use cases, including mathematical problem-solving and question-answering tasks, demonstrating the effectiveness of AutoGen. Overall, AutoGen is a promising framework for the rapid, flexible, and customizable development of LLM applications.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful comments and thoughtful questions.  We also truly appreciate your recognition of the open-source contribution, technical details, and rich empirical results!\n\n1. A multi-agent system could possibly be considered to be performing multi-prompting when all the agents are LLM-backed. However, the AutoGen framework goes beyond multi-prompting: in addition to LLM, an agent could also be backed by tools or humans, or any combination of them. We provide example dialogues under different design choices in Figures 8, 10, 13, and 16.\n\n2. As for the difference between other existing work: In addition to the restrictions on the number of agents, CAMEL and MetaGPT essentially only support a fixed multi-agent workflow in a round-robin fashion or a pre-defined SOP fashion (both are static conversation patterns). AutoGen supports a diverse workflow (static or dynamic) thanks to its composable conversation patterns. This is vital in enabling diverse application scenarios, where we may want to orchestrate agents statically or dynamically depending on application needs.\n\nWe hope this addresses your concerns, and we are very willing to respond to any further questions you might have!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is a fully open-source multi-agent transformation framework, facilitating users in rapid application development. The high transparency of technical details aids in traceability and reproducibility.\n2. The authors provide ample empirical results to showcase the performance of multi-agent conversations, aiding readers in better understanding the advantages and limitations of multi-agent systems.", "reasons_to_reject": "1. The claimed multi-agent system is more accurately described as multi-prompting. More convincing evidence would be provided by observing dialogues under different conditions within the agents themselves.\n2. The major concern lies in the inherent differences from other multi-agent frameworks, such as CAMEL (NeurIPS 2023) and MetaGPT (ICLR 2023). In the related work and discussion, the paper assert no explicit restrictions on the number of agents and whether tools are supported to distinguish from previous works. In my opinion, these are not very strong justifications.", "questions_to_authors": "None", "extra_scores": {}}, {"review_id": "JW3rj1PhbV", "reviewer": "Reviewer_TpLJ", "summary": "This paper presents an open-source framework to solve tasks by combining multiple LLMs acting as agents that talk to each other. It describes necessary features of such a framework, such as being easy to customize (to allow generality) or allowing re-use of modules (to reduce complexity). The main proposal is the concept of \"conversation programming\" which implies the definition of a set of agents and the definition the code that controls their interaction. After the main proposal the paper describes the features of the framework and the applications, which contains various experimental setups.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's time and effort in evaluating our submission. We appreciate your feedback and would like to address them!\n\n### **1. About Open-Source Framework Contribution**: \nWe assert that the development of an open-source framework constitutes a significant research contribution. Our framework introduces innovative engineering designs and serves as a vital resource to the community, as acknowledged by all the other three reviewers. It facilitates scientific experimentation, yielding useful insights and enabling new discoveries. This is also evidenced by the wide adoption of the framework and the numerous research works developed based on it. A notable example of a similar type of open-source framework is PyTorch [1]. PyTorch has been cited over 42,405 times since its publication at NeurIPS 2019, underscoring its profound impact across various research domains. Regarding the “ software-like description”: The PyTorch paper also includes detailed descriptions of software architecture (refer to Listings 1 and 2). These kinds of descriptions are generally helpful in outlining the framework's engineering design and architecture.\n\n### **2. About Experimental Setup and Results in the Appendix, and Organization of the Paper**\nDue to page limitations, detailed experimental setups and extensive results were placed in the appendix. Nevertheless, the main body of the paper encapsulates all essential details needed to grasp the experimental framework, supported by robust results and illustrative figures in Section 3. The results highlighted in the appendix are meant for readers interested in a deeper exploration, ensuring the main text remains focused and readable. These additional details and results are indeed appreciated by most readers, including all the other three reviewers. With that being said, we do appreciate the feedback on improving our paper's organization and are committed to enhancing its clarity to better convey our findings.\n\n[1] Paszke, Adam, et al. \"Pytorch: An imperative style, high-performance deep learning library.\" Advances in neural information processing systems 32 (2019).\n\nWe hope these clarifications address your concerns and are willing to address any more concerns you might have. We will further improve the paper based on your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is timely: agents and the authors are straightforward about the paper is being an open source framework.", "reasons_to_reject": "My main problem is that it is hard to judge the contributions of this work from a research perspective both because, as the abstract states, the papers main proposal is an open source framework and also because the experimental setup needs a lot of improvement. The proposal of \"conversation programming\" seems also vague and its not clear how it is different from a generic multi-agent framework.\n\nTo put it in concrete terms:\n\n- Most of the manuscript reads more like the documentation of a software package than a research paper. It contains a lot of description of software abstractions such as classes and general descriptions of the package's features and even usage of flags or variable values. The software-like description makes it hard to isolate the concrete components that contribute to the experimental setup's results.\n\n- The experimental setup itself is also lacking. Many of the results are also missing from the main paper and moved to the Appendix e.g RaG. These results are important enough that they should be on the main body of the manuscript. The experiments in the Appendix seem also not standard, using a small number examples and oddly organized with a general lack of tables. \n\n- It seems like the original manuscript was far longer ans it has been split between body and appendix in a way that makes it hard to follow.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Nfff61HHU", "reviewer": "Reviewer_9adi", "summary": "The paper introduces AutoGen, an open-source framework that enables multiple LLM-based agents to collaborate on tasks through role-playing and tool usage. The framework has gained significant attention from the community since its release. While the central idea of multi-agent interaction in LLMs aligns with prior work and some earliy frameworks like SocraticAI [1], the authors highlight two novel engineering concepts: \"conversable agents\" and \"conversation programming,\" which simplify agent configuration and interaction for developers. AutoGen also unifies other multi-agent LLM frameworks, and the paper demonstrates its wide usability through six application examples, along with quantitative evaluations and comparisons to other methods.\n\nThe engineering effort behind AutoGen is commendable, and this open-source framework is a valuable resource for the COLM community. However, the authors should temper their claims about the absolute superiority of multi-agent methods for problem-solving and discuss the limitations of these methods, such as potential failure modes and comparisons of inference costs. Additionally, the paper would benefit from a more rigorous validation of the effectiveness and key design factors of multi-agent systems, such as the separation of context among agents and the use of directed graphs versus broadcasting.\n\nThere are some controversy surrounding the superiority of multi-agent systems over single agents [2], as well as the potential for agents to be distracted by irrelevant context [3]. While a comprehensive validation of multi-agent superiority may be beyond the scope of this paper, AutoGen provides an efficient platform for conducting comparative experiments to test the effectiveness and key design factors of multi-agent task-solving systems. The authors should provide more caveats to readers regarding these aspects.\n\n[1] SocraticAI: https://princeton-nlp.github.io/SocraticAI/\n[2] Huang & Chen et al. Large Language Models Cannot Self-Correct Reasoning Yet. ICLR'24\n[3] Shi & Chen et al. Large Language Models Can Be Easily Distracted by Irrelevant Context ICML'23", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and thoughtful questions. We also appreciate your recognition of the novel engineering design and extensive application scope of AutoGen. Here, we provide further clarifications and additional data in response to the weaknesses highlighted.\n\n### **1. About sample sizes and significance of quantitative evaluation:**\nThe results presented in Figure 4 are derived from widely recognized datasets, ensuring the relevance and robustness of our findings: MATH (used in A1) includes 5,000 test instances; Natural Questions (in A2) includes 7000+ test instances; ALFWorld (in A3) includes 134 unseen test instances; Supply-chain related Coding tasks [4] (used in A4) includes 100 data instances. \n\n### **2. About the effectiveness of multi-agent design:**\nIn addition to A1, we also investigate the effectiveness of the multi-agent approach in A4, where a multi-agent approach is compared with a single agent conducting both the code-writing and safeguard processes. The performance improvement is shown in Figure 4(d). As for the human-in-the-loop scenario, we consider it a pilot study and defer comprehensive evaluation to future work, as the evaluation involves a human study.\n### **3. About failure analysis:**\nFailure analyses are included in the appendix to provide insights into the limitations and improvement areas for AutoGen. These analyses are presented across various contexts: Table 1 (failure analysis of AutoGen and compared methods on MATH), Figure 8 (failure analysis of RAGChat with and without interactive retrieval), Figure 10 (failure analysis of a two-agent system compared to a three-agent system in ALFWorld), Figure 13 (failure analysis of a two-agent chat compared to a group chat), Figure 16 (failure analysis of conversational chess when a board agent is not used), and Table 6 (failure analysis of AutoGen on 4 typical tasks from MiniWob+)\n\nWe thank the reviewer again for the time and effort. We hope this additional information adequately addresses your concerns. We are very willing to address any further questions or concerns you might have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- novel engineering design of the framework\n - thorough experiments demonstrating wide application\n - open-source, helpful resource to the community", "reasons_to_reject": "- quantitative evaluation can be improved to show statistical significance of the results. figure 4, in particular, should include sample sizes and appropriate error measurements. \n- the effectiveness of multi-agent over single agent should be more carefully examined. the performance gap in experiment A1 is mainly from tool usage, and it's unclear whether it's human-in-the-loop scenario. \n - didn't do enough analyses of failure modes in A1-A6.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces AutoGen, an open-source framework that allows developers to build LLM applications using multiple conversable agents and conversation programming. 

As reviewers pointed out, this work presents a strong open-source framework, and their extensive experimentation demonstrates the utility of the proposed framework. 

Reviewers also pointed out a few concerns, such as how to optimize the multi-agent systems to let it work better than single LLMs, the differences of AutoGen to other multi-agent frameworks, as well as statistical significance and error analyses.  We thank the authors for their detailed response during the rebuttal period, and some of these concerns were addressed. 

Another issue got discussed was the contribution of this work, as one reviewer questioned that the current work reads more like a documentation of software package. I think open-source framework is indeed a contribution, and the resulting open-source work can be a useful resource for the community.  

There are also some other issues such as the framing of conversation prompting, for which we encourage the authors to further address these in the revised version.
Decision: Accept