OpenAgents: An Open Platform for Language Agents in the Wild
Authors: Tianbao Xie
Fan Zhou
Zhoujun Cheng
Peng Shi
Luoxuan Weng
Yitao Liu
Toh Jing Hua
Junning Zhao
Qian Liu
Che Liu
Zeyu Liu
Yiheng Xu
Hongjin SU
Dongchan Shin
Caiming Xiong
Tao Yu
Venue: COLM (2024)
Keywords: Language Agents; Large Language Models (LLMs); Data Analysis; Web Browsing Automation; API Tools
TL;DR: OpenAgents is an open platform facilitating easy access to language agents, offering three agents for data analysis, API tools, and web browsing, aiming to foster innovation and real-world evaluations in the realm of language agents.

Abstract: Language agents show potential in being capable of utilizing natural language for varied and intricate tasks in diverse environments, particularly when built upon large language models (LLMs). Current language agent frameworks aim to facilitate the construction of proof-of-concept language agents while neglecting the non-expert user access to agents and paying little attention to application-level designs. We present OpenAgents, an open platform for using and hosting language agents in the wild of everyday life. OpenAgents includes three agents:  (1) Data Agent for data analysis with Python/SQL and data tools;  (2) Plugins Agent with 200+ daily API tools;  (3) Web Agent for autonomous web browsing. OpenAgents enables general users to interact with agent functionalities through a web user interface optimized for swift responses and common failures while offering developers and researchers a seamless deployment experience on local setups, providing a foundation for crafting innovative language agents and facilitating real-world evaluations. We elucidate the challenges and opportunities, aspiring to set a foundation for future research and development of real-world language agents.

Published as a conference paper at COLM 2024
OpenAgents: An Open Platform for Language Agents in the Wild
Tianbao Xie
Fan Zhou Zhoujun Cheng Peng Shi
Luoxuan Weng" o Yitao Liu"
Toh Jing Hua" Junning Zhao Qian Liu d0 Che Liu Leo Z. Liu'
Yiheng Xu" 9
Hongjin Su"
Dongchan Shin oo Caiming Xiong" Tao Yu
"The University of Hong Kong ' "LLang Lab Sea AI Lab Salesforce Research (tbxie, taoyds]@cs hku. (alla99..f@@gaiii com, z6cheng@ucsd.edu
Abstract
Language agents show potential in being capable of utilizing natural lan- guage for varied and intricate tasks in diverse environments, particularly when built upon large language models (LLMs). Current language agent frameworks aim to facilitate the construction of proof-of-concept language agents while neglecting the non-expert user access to agents and paying little attention to application-lvvel designs. We present OpenAgents, platform for using and hosting language agents in the wild of everyday life. OpenAgents includes three agents: (1) Data Agent for data analysis with Python/ SQL and data tools; (2) Plugins Agent with 200+ daily API tools; (3) Web Agent for autonomous web browsing. OpenAgents enables general users to interact with agent functionalities through a web user interface optimized for swift responses and common failures while offering developers and researchers a seamless deployment experience on local setups, providing foundation for crafting innovative language agents and facilitating real-world evaluations. We elucidate the challenges and opportunities and set foundation for future research and development of real-world language agents.
open
1 Introduction
Intelligent agents are broadly conceptualized autonomous problem solvers with the ability to sense their environment, decide, and act upon that environment (Wooldridge) & Jennings 1995 Sutton & Barto 2005 Russell 2010). With the advent of large language
models (LLMs) Brown et al., 2020; Chen et al.. 2021 Chowdhery et al. 2022 OpenAT 2023b Touvron et al. 2023), recent implementations from the academic, industry, and open-
source communities have leveraged this concept to create language agents. These agents are capable of utilizing natural language to perform variety of intricate tasks in diverse
environments, showcasing notable potentials Yao et al. 2022b Chase
2022
Gravitas 2023
OpenAl 2023a Wang al. 2023a)..
Meanwhile, current agent frameworks such as LangChain CCasse 2022 AutoGPT Gravitas
2023), Gentopia (Xu et al. 2023a Aaby (Nakajima, 2023), Agent (Chen et al. 2023), Agents [Zhou et 2023c provide proof-of-concept implementations and console interlaces largely tailored for developers. This often restricts access wider audience, particularly those not versed in programming or consoles. Current agent benchmarks are constructed within specific environments for deterministic evaluation, especially in scenarios involving coding (Yang et al. 2023a tool utilization et 2023c Patil et al. 2023 Qin et al. 2023b , web browsing (Shi et al. 2017 Yao et al. 2022a) Deng et al.) 2023 Zhou et al. 2023bb or
combination of above (Liu al.. 2023). These offer initiatives toward agent evaluations while general agent usage is bound to be connected to more open or unlimited environments
and real users with their special needs and materials.
Fquul contribution.

---

Published as a conference paper at COLM 2024
Language Agents
Researchers)
wab
Action Da cing
Agene
Agant
@ t
APriCa Todls Selection
vi by Sarencion
UX/UI
Smnoming Fror wardding
User Symtem
Frontend
Backend
a. Web UI for Agents (For Users)
b. Localhost Deployment (For Developers)
Figure 1: The OpenAgents platform for general users, developers, and researchers. (a) Gen- eral users can interact with the agents via our online web interface, instead of programmer- oriented console or packages. (b) Developers can effortlessly deploy the frontend and backend for further developments given our codes. (c) Researchers can build new language agents or agent-related methods given the examples and shared components, and see how they perform with the web UI. Our OpenAgents serves to be simple and versatile platform for using, developing, and evaluating language agents.
Aiming to develop LLM-powered offerings for a broader user base, OpenAl has crafted and deployed well-designed products specifically Advanced Data Analysis (previously known as Code Interpreter), Plugins, and Browse with b Bing, leveraging their further trained models (which has been reverse engineered and accepted by many (Zheng et al. 2023b
Gou et al. 2023)), ) business logic code, and a nurtured software community (e.g., OpenAl plugins store). Despite the significant success attained, the models and business logic code have not been open-sourced due to business considerations, hindering not only users from free access but also developers and researchers from further exploring, evaluating, and improving upon them.
Recognizing this, OpenAgents stems from the motivation to democratize agent access open-source platform for using and hosting agents, currently encompassing three integral agents: Data Agent for data analysis with Python and SQL,, Plugins Agent for 200+ tool usages, and Web Agent for autonomous web browsing. We believe that for LLMs to reach their full potential, they must transition from purely theoretical or developer-centri tools to dynamic, interactive systems that cater to a diverse user base. As depicted in Figure [ general users can readily explore agent capabilities via the online Web UI without any coding expertise required. In addition, OpenAgents provides full business logic and research codes for developers to easily deploy it locally and researchers to further introspect and build language agents. Lastly, given all provided above, OpenAgents is meant to be a realistic and holistic numan-in-the-looe
agent evaluation platform: out of real needs, real users interact with agents to fulfill their tasks, and the whole human-agent interaction traces and user feedback are recorded for further evaluation. Compared to existing benchmarks and platforms, OpenAgents provides in-the-wild environment where agents tackle variety of genuine user needs.
During building OpenAgents, we first underscore the significance of effectively specify- ing application requirements via LLM prompting, a process that often requires crafting instructions that cater to backend logic, enhance output aesthetics, and safeguard against adversarial inputs. Our findings indicate that the build-up of such instructions can, at times, be substantial, posing challenges in terms of token limitations and context handling for the LLMs. Additionally, for effective real-world deployment, agent models must not only
2

---

Published as a conference paper at COLM 2024
exhibit high performance but also be able to handle real-time, interactive scenarios, such as streaming, to provide an optimal user experience. Furthermore, our exploration reveals that current research often gravitates towards idealized performance metrics, sometimes sidelin- ing critical real-world considerations, such as the trade-offs between system responsiveness and accuracy, and the nuanced complexities introduced when application-baser failures arise, potentially obfuscating the true capabilities of the LLMs.
For future directions and extensions based on OpenAgents, we envision the expansion of new agents, agent-related methods, models, and tools for researchers. Plus, the built-in web UI paves the way towards human-in-the-loop agent and LLM evaluation and interaction in the wild under realistic needs, which could benefit the NLP and HCI communities, Finally, we hope that our codebase, which includes off-the-shelf frontend and backend codes and shared components of agents, will inspire the development of more innovative applications.
Related Works and Preliminaries
Building agents that operate intelligently in specific environments has a long history across the fields traditional artificial intelligence (Kaelbling et al., 1987 Maes| 1990: Brooks 1991
Russell 2010), reinforcement learning (Sutton & Barto 2005 Mnih et al. 2013: Finn et al.
2017), and recent language agents (Yao et al. which are built on LLMs.
2022b
Yang et al. 2023a Wang et al. 2023a
Generally, language agents can be formalized as a partially observable Markov decision process (POMDP) (H, A, O, TRR) with natural language space H. state space S, action space A, observation space O, transition function T S x A + S, and reward function R: S x A - [0, 1]. Given the dialogue history (user queries and agent responses) h E H, an agent generates executable action a; E A and interacts with the environment. An action is executable if it is within the action space A and incurs change in the state St+1 E S, and execution feedback as observation 0t+1 E O. The interaction loop repeats until certain stop tokens as action is generated. worth noting that currently most LLM-based language agents typically do not require training, and as result, rewards are often not regarded.
Table 1: Comparison between OpenAgents and current existing works building pro- totypes and benchmarks on agent concept. Online stands for whether it is deployable and can be online-hosted. UI stands for providing a user interface. # #Tools is the number of tools contained. Feedback stands for supporting feedback from users. Web. is short for web browsing. "Controlled" means the framework/ haanework/benchmale benchmark sets up deterministic environments ahead for the agent to operate actions, while "Wild" ones let the agent take actions in the open-ended real-world environment. The "*" marker indicates their agent only implements basic web browsing methods (e-g. web crawl) uses external APIs (e-g., google search). The "*" marker indicates OpenAgents offers additional features beyond just statistical numbers, for example automatic selection on tools. All statistics in the table are collected by September 2023.
Interface
Environment
Name
Online
Feedback
UI
Coding Env.
#Tools
Web.
AutoGPT
Gravitas
2023
CLI
Wild
15
BMTools
Oin et al.
2023a
X
Controlled
32
BabyAGI
Nakajima
2023
X
X
Controlled
Gentopia (Xu al.
2023a
V
CLI
Controlled
15
Open Interpreter [Lucas
2023
CLI
Wild
GAs (Park et al. | 20233)
X
Web
AgentVerse (Chen et
2023
X
Web
Camel (Li al, 2023b
X
Web
Agents (Zhou al. 2023c
V
Web
Wild Wild
11
AgentScope (ao
2024)
/
Web
OpenAgents (ours)
2
/
Web
Controlled & Wild
2200
ChatGPT Plus (closed-source)
Web
Controlled & Wild
2500
3

---

Published as a conference paper at COLM 2024
Numerous frameworks and codebases surrounding language agents have been established. One category comprises the construction of some proof-of-concep catchy prototype imple- mentations (Chase 2022 Gravitas 2023 Nakajima 2023). The core technique is prompting LLMs and demonstrating astonishing results in certain cases. On this basis, some improved and updated conceptual frameworks (Xu et al. 2023a Zhou et al. (023c) have been pro- posed. Another category emphasizes the evaluation of language agents. These evaluations are often conducted in fully simulated environments (self-hosted web pages (Yao et al., 2022a Zhou et al. 2023b), etc.), semi-real environments (for instance, utilizing real tool API calls QQn et al. 2023b)) or combination (Liu et al.) 2023 Wang al. 2023b). Additionally,
there are frameworks focusing on the study of multi-agen and their behaviors (Li et al.
2023b Park et al. 2023 Chen et al. 2023 Hong al. 2023 Wu el. 2023 Table shows
an overall comparison Of these frameworks and codebases.
Platform Design and Implementation
User Interface
Language Agent
AI
HWW
gurquag
Backend
Tools
Envrrnments
User System
Frenlend Website
Databasse Design
Prompting
Aaliing
Web Extorsion
Streaming
Backend Server
Action Parsing
Tools Scaling
Sandbox
Enor Handlnng
Agent Method
Tools Selection
Figure 2: System overview of OpenAgents' architecture comprising (1) User Interface, a bridge that facilitates communication between the user and the agent, and manages backend operations; and (2) Language Agent, encompassing the language model, tools, and environment, driving the agent's decision-making processes. The flow of interaction typically follows from the user needs, through the user interface, and culminates in the language agent taking specific actions using its various components.
3.1 Systematic Design
In this section, we introduce our systematic efforts in OpenAgents, aiming towards ad- dressing this gap between existing agent frameworks and real-world functional agent applications.
The OpenAgents architecture can be generally split into two parts corresponding to Figure 2 (1) User Interface, including both the frontend and backend; (2) Language Agent, including language models, tools, and environments. OpenAgents provides well-built interface for user-agent communication. Upon receiving instructions from the users, the agent will then plan, and take action in the environment via using tools.
User Interface We have devoted significant effort to crafting a user-friendly and highly functional User Interface. We have tackled a multitude of highly reusable business logic to use and host agents. This effort has resulted in robust support for various technical aspects within the demo, such as backend server operations, error handling, data streaming, etc. Our goal has been to make the OpenAgents not only user-friendly but also to provide enhanced usability. Although these aspects were not the central focus of the current agent frameworks (Chase 2022 Xu et al. 2023a Zhou et al., 2023c), we firmly believe that these schind-the-scemee efforts should not be. They hold significant meaning for future research, contributing to the overall robustness and usability of the system.
Language Agent In our design, the language agent comprises three essential components: the language model, the tool interface, and the environment. Currently, adopt our prompting method based on the approach introduced in ReAct (Yao et al. 2022b). The agent generally follows sequential process of Observation Deliberation Action in each
4

---

Published as a conference paper at COLM 2024
turn of interaction. We also prompt the language model to produce easily parsable text. The tool interface includes parsers capable of translating this text into executable actions, such as generating code or making API calls. Subsequently, these actions are executed within the corresponding environment. We've also invested considerable effort into sehind-the-sceen challenges, like stable API calling, tools scaling, and building sandbox environment.
3.2 Practical Implementation Challenges
The implementation journey unveiled several challenges essential for in-the-wild agent readiness. Below are the simplified insights and highlights into the core facets of our implementation. For more details, please kindly refer to Appendix
For User Interface implementation, we tackle the following challenges:
1. Adaptive Data Mapping 1A.1.
Drawing from database terminology, we employ the concept of DataModel. This model effectively converts various raw data types-such as text, codes, images, and tables-ints formats optimized for both human-side pretty formatting, LLM contexts, and persistent data storage, streamlining communication between system components and external entities.
2. Strategic Data Storage A.112 We adopt strategic data storage paradigm catering to the multi-user nature of OpenAgents. Utilizing in-memory storage for temporary variables, Redis (Carlson 2013) for global variables, and MongoDB (Banker et al. 2016) for user- centric data orchestrated an efficient data management and retrieval ecosystem.
3. User-Centric Interfuce $..1.3 An Adaptive User Interface was crafted, bridging the inter- action chasm between users and the system. This UI, catering to various operational environments, renders rich media and interactive content (images, code snippets, con- sole outputs, and interactive visualizations, etc.), augmenting user engagement and task efficiency.
4. Real-time Response Streaming A.1.4 A streaming approach adopted to mitigate the latency inherent in receiving long text completions. Based on streaming API and push-down automata, this innovation allowed for real-time parsing and rendering of generated tokens, significantly ameliorating the immediacy of user feedback.
5. System Robustmess A.1.
Enhancing agent robustness was identified as quintessential requirement for delivering a realistic user experience. Key areas including effective failure handling (e.g., API calling failure), prompt response generation (e.g., LLM calling APIs pool set and streaming handling), and token overflow management (chat history over length for LLM to handle) were delineated and addressed, ensuring reliable functionality under diverse real-world scenarios.
6. Browser Control via Chrome Extension A.1.6 A Chrome extension was engineered, en- dowing our web agent with direct browser control capabilities. This setup facilitates real-time user monitoring and intervention during web interactions, and enables web browsing commands on the user side, enhancing user control and trust the system.
For Language Agents, we provide solutions to the following challenges:
1. Data Grounding 1.2.1 Our system lets users upload files which agents can process by writing codes. We ve thus established grounding source pool to store user-uploaded data. With DataModel, each file type linearized and indexed by its name, allowing agents to retrieve and use content as instructed by humans.
2. Automatic Tool Selection g A.2.2 Users previously had to select plugin manually, like OpenAl Plugins Plugins A 2023a), for command execution. Recognizing the challenge of manual plugin selection from numerous options, we've integrated feature namely "Auto Selection", which auto-detects the most relevant tool based on user instructions, streamlining the process.

---

Published as a conference paper at COLM 2024
3. Automatic Tool Scaling $A.2.33 While tool creation poses unique challenges, especially for agent use within the LLM infrastructure, we've ve sourced API provider information from platforms like RapidAPI and OpenAl plugins store. This approach has led to over 200 high-quality plugins, albeit with challenges that occasionally require human oversight. Further exploration is needed for efficient scaling.
4. Executable Entoiromments A.2.4 By "executable", we imply the transformation of lan- guage model outputs into actionable tasks within a specific context. Constructing an application-level, multi-user executable environment poses challenges due to the need for safety, robustness, and functionality. We managed to implement sandbox environments catering to code execution, API interactions, and web navigation. These environments serve as a comprehensive testbed for agents, facilitating tasks like code generation, plugin interactions, and web manipulations.
Collectively, these advancements contribute towards architecting a more coherent, robust, and user-friendly platform. Moreover, We believe they also provide rich repository of learnings, promising to guide future implementations in similar domains.
4 OpenAgents
In OpenAgents, we develop three distinct agents: namely the Data Agent for data analy- sis, the Plugins Agent for plugin integration, and the Web Agent for autonomous web browsing. The three agents are experts in different domains, just like OpenAl's ChatGPT Plugins (OpenAT 2023a). Moreover, our implementation is purely based on the top of open language APIs (and theoretically any other open-sourced LLMs), such as GPT-4 (OpenAT 2023b), and Claude (Anthropic) 2023). We hope OpenAgents can serve as pioneer the move towards democratizing potent, real-world language agents. Further, we contend that OpenAgents can serve as an experimental testbed for researchers across various disciplines who share interest in the future trajectory of language agents.
4.1
Data Agent
ratachiv meca anocoso monno
the benderoet dlakk
Data
Agent
Generata
Execute
Falbee ioweiiin
Figure Pipeline (left) and demonstrations (mid and right) of Data Agent.
The data agent has been designed and implemented to deal with a wide spectrum of data- related tasks that real users encounter daily. We support code generation and execution in two programming languages: Python and SQL. We also integrate several data tools for the agent use: (1) Kaggle Data Search (datasets search on Kaggle via calling API); (2) Data Profiling (heuristic data profiling providing basic data information);
(3) ECharts Tool (interactive ECharts plotting). We prompt the agent to proactively use these data tools to respond to user requests. Recognizing ( the intensive coding requirements for data agent, have opted to embed language models in the tool, and let the tools generate code rather than the agent. Specifically, tools such as Python, SQL, and ECharts will generate code, thereby harnessing the language models' full programming prowess and alleviating the strain on the agent itself.
Equipped with these data tools, the agent adeptly manages various data-centric requests. It transcends the boundaries of mere text and code generation to proficiently perform

---

Published as a conference paper at COLM 2024
data queries, visualization, manipulation tasks, etc. An illustrative example is depicted in Figure 3) where users can upload custom files, such as tables and images, and make successive inquiries about the uploaded data. While we' ve highlighted the data agent's key capabilities, the full potential of our data agent extends beyond this brief overview. Much like OpenAT's Advanced Data Analysis(previously known as Code Interpreter) (OpenAT 2023a), we hope the data agent can serve as versatile tool for myriad user workflows; for more use cases, please see Appendix B.1
4.2 Plugins Agent
Zee
2 Used fen
Plugins
Agent
User Steme Cheading
iraak GG We
500.0 thi symthei borthe
API Cal
Figure Pipeline (left) and demonstrations (mid and right) of Plugins Agent.
The plugin agent has been meticulously designed to cater to the multifaceted requirements of users' daily tasks which necessitate additional plugins, such as shopping, searching, news reading, and website creation. We have integrated over 200 plugins from various sources, including prominent plugins like (1) Google Search; (2) Wolfram Alpha; (3) Zapier; (4) Klarna; (5) Coursera; (6) Show Me; (7) Speak; (8) AskYourPDF; (9) BizToc; and (10) Klook. Special attention has been paid to ping the APIs, function calling interface, and API response
length, enabling LLM-based agents to optimally leverage plugins.
Users can choose one or multiple plugins they would like to let their agent leverage based on the given instructions of their needs. Examples can be observed in Figure 4 in- stances where users are uncertain about the appropriate plugins for their needs, we have incorporated a feature that automatically selects the most relevant plugins based on their instructions. We posit that the plugin agent can be instrumental in myriad scenarios, enrich- ing various aspects users' daily lives. For a more comprehensive list of use cases, refer to Appendix B.2
4.3
Web Agent
+
Burat
micct
Web Agent (chat)
+
Web Agent (browse)
Act
Figure 5: Pipeline (left) and demonstrations (middle and right) of Web Agent.
We present the web agent specialized tool designed to enhance the capabilities of the chat agent. The main interaction interface still lies with the chat agent, but when necessary, it seamlessly involves the web agent. The web agent then delivers the final response to
7

---

Published as a conference paper at COLM 2024
the user, shown in Figure 5 implemented and demonstrates several notable advantages.
This design strategy is not just theoretical; it has been
First, the chat agent systematically processes important parameters like user inquiries or initiating URLs before transferring them to the web agent. This ensures that there is a strong alignment with user intentions and facilitates clearer communication. To clarify further, when dealing with complex tasks that may seem ambiguous or have multiple aspects, the chat agent uses decompositional approach. Herein, overarching directives from users are segmented into more digestible sub-tasks. The chat agent then interfaces with the web agent using these sub-instructions, sequentially, ensuring more granular and efficient problem resolution. Moreover, our configuration empowers dynamic interplay of multi-turn web navigation interspersed with chat dialogues, thereby accommodating more layered and adaptable user queries. By distinctly demarcating the roles and responsibilities of the web- browsing agent and the chat agent, we pave the way for the independent evolution and continual refinement of each module. For a deeper dive into practical applications and the potential of the web agent, we invite readers to peruse Appendix B33
From Research to Real-world Deployment
The section reflects on the crucial challenges encountered, the lessons imbibed from real- world scenarios, and the evaluation complexities that surfaced through this transition.
Challenges of Transforming LLMs into Real-world Apps through Prompting When building applications for real users based on LLM prompting, we specify certain require- ments through the instructions in the prompt. Some of these instructions are designed to ensure that the output of the LLMs conforms to specific format for our backend logic to process (e-g., output in the form of dictionary with specific keys). Others aim to en- hance the aesthetic appeal of the output (e.g., listing items separately whenever possible). Additionally, some instructions serve to defend against potential attacks (e.g., refuse ma- liciously crafted infinite loops in the program by users). As illustrated in the an] SE) the accumulation of these instructions often results several hundred tokens, thereby demanding certain requirements on the instruction tracking ability and supported context length of the LLMs. With inferior LLMs, sometimes the improper output format leads to unsatisfactory frontend appearance even inability to complete the response. A positive sign is that there have been significant improvements in these aspects in current open-source models (i.a. |Peng et al. 22023a Tworkowski et al. 2023 Zhou et al. 2023a Peng et al., 2023b Xiong et al. [203 Xu et al. 2023c). Additionally, a greater emphasis is required on the toundational development and research of agent models, and on training dedicated agent models (maybe not LLMs) that are tailored for specific domains and requirements. This approach may turn out be more effective and controllable than relying on purely prompting general powerful but fixed model.
Uncontrollable Real-world Factors Upon deployment in the real world, we encountered numerous uncontrollable factors triggered by users, internet infrastructure, business logic, and so forth, that haven't 't been well-modeled, necessitating reevaluation and often over- turning of many assumptions and forms adopted in past research. We had to assume the possibility of the server hosting the APIs we call crashing, monitoring and robustly complet- ing user commands during such crashes than those assumed in tool-use studies (i.a. Cheng et al. 2023 Schick et al., 2023 Qin et al. 2023a). ). There could be instances where users may become dissatisfied during the response generation, intervening and causing the language model to halt generation midway. Unpredictable occurrences like CAPTCHAs popping up advertisements altering the web page, could introduce degree of randomness even in relatively stable web structures, which is unconsidered in previous autonomous web browsing works (i.a. Shi et al., 2017: Zhou et al. 2023b).. These uncontrollable scenarios beckon a deeper exploration or the proposition of more realistic modeling approaches.
Extra Metrics from Real-world Scenarios Research primarily emphasizes petasssiisspeffrmfm metrics, frequently overlooking essential requirements derived from worldworld scenarios. An instance learn during implementation is streaming. It allows users to quickly
performance
8

---

Published as a conference paper at COLM 2024
perceive system responses instead of waiting for lengthy text to generate. Specially designed prompting makes the format of agent response look prettier. These significantly impact user experience. Existing methods (i.a. Yao et al. 2022b Xu et al.) 2023b; Lin et al. 2023) haven't t considered much of its effects, resulting in slower response times and bad user experience in practical applications, regardless of their superior performance metrics in accuracy. These findings should be contemplated in subsequent application-driven studies. And we also need to mention the trade-offs between the performance and user experience.
6 Discussions and Future Work
Agent Applications OpenAgents sets up whole pipeline of building application-level language agents, thus paving the way for other innovative applications such as customizable dialogue systems, multimodal interaction, and automated workflow integrations for end- users. Each of these applications not only offers unique advantages but also collectively contributes a richer and more user-centric agent application environment.
Tool and Component Integration OpenAgents explores and addresses the fundamental requisites for constructing practical-level agent application, laying down robust founda- lariii tion that allows the community to effortlessly expand horizontally by integrating additional
components such tools (e.g,, integrate from more diverse API sources like Public Pls extending more foundation models (e.g,, recent large multimodal models (i.a. Yang et al. 2023b)), adapting new UI designs, etc.
Li al. 2023a
Human-LM Interaction Benefiting from the easiness of building new LLM-based agent
applications from our platform, we believe OpenAgents can consequently be helpful in build- ing application demos for the studies of Human Coman-Computer Interaction (HCI) researchers to delve into the design of more intuitive and user-friendly interfaces (f.a. Suh al. 2023 Kim
let al. 2023
Angert et al., 2023), thereby enhancing user engagement and satistaction.
In-the-wild Evaluation of LLMs Establishing robust, impartial evaluation methodologies for LLMs is essential for an unbiased assessment of their capabilities and performance. Current agent-around benchmarks (i.a. Yang et al. 2023a Liu et al. 2023 Wang et al. 2023b) evaluate agents through benchmarks with pre-collected data and under controlled environ- ments. While these evaluations are crucial, they often do not fully represent the dynamic challenges encountered in real-world scenarios. Platforms like Vicuna Arena (Zheng et al. 2023a) have taken the lead in evaluating chatbots in the wild, marking a significant step towards more realistic evaluation settings. However, there's 's room for further development in metrics that cater to the nuances of language agents, especially those that tie language understanding with grounding in real-world contexts. Community contributions in ex- panding or refining these evaluation metrics and platforms are highly encouraged as they would significantly advance the field, providing more accurate insights into the practical performance and capabilities of LLMs.
7 Conclusion
In this work, we introduced OpenAgents, an open-source platform, tailored to meld the advancements of LLMs with practical, user-focused applications. OpenAgents has built agents for three typical applications data analysis, tool utilization, and web browsing, demonstrating its practical utility. During the development of OpenAgents, we identified numerous challenges that were unique to constructing in-the-wild agents. These challenges highlighted the complexities of transitioning from theoretical designs to fully operational agents that cater to actual user needs and interactions. By providing a holistic, transparent, and deployable platform, we aim not only to make the immense potential of LLMs accessible to a wider audience beyond developers and researchers but also to facilitate grounded research through exposure to real-world scenarios, empowering the community to harness and refine the capabilities of state-of-the of-the-art language agents. We sincerely hope OpenAgents can inspire and pave the way for a plethora of applications and platforms, streamlining the symbiotic relationship between humans and intelligent language agents.

---

Published as a conference paper at COLM 2024
Acknowledgments
We extend our profound gratitude to Google Research, Amazon AWS, and Salesforce Research. The munificent gift funds and computational resources they bestowed upon us have been instrumental in the realization of this project. Their unwavering support has been pivotal in the success of our endeavors. We also wish express our heartfelt appreciation to our contributors for their relentless commitment and insightful perspectives: Roxy Rong, Yixian Zhang, Haotian Li, Xingbo Wang, Chen Henry Wu, Toh Jing Qiang, Jansen Wong, and Jixuan Chen. Equally, our discourse was enriched by the sagacious discussions and invaluable advice from Yiru Chen, Bohan Zhang, Rui Zhang, Ruiqi Zhong, Michihiro Yasunaga, and Ansong Ni. Their collective wisdom has been invaluable cornerstone in shaping this research.
10

---

Published as a conference paper at COLM 2024
References
Tyler Angert, Miroslav Ivan Suzara, Jenny Han, Christopher Lawrence Pondoc, and Hariha- ran Subramonyam. Spellburst: node-based interface for exploratory creative coding with natural language prompts. ar Xiv preprint ar arXi::2008.03921 2023.
Anthropic. Introducing claude, 2023. URL
https://ww.artirrppcccco//////
claroducing-claude Accessed: October 11, 2023.
Jean-Michel Autebert, Jean Berstel, and Luc Boasson. Context-free languages and pushdown automata. Handbook of Formal Languages: Volume I Word, Language, Grammar, pp. 111-174, 1997.
Kyle Banker, Douglas Garrett, Peter Bakkum, and Shaun Verch. MongoDB in action: covers MBngoDB version 3.0. Simon and Schuster, 2016.
Rodney Brooks. Intelligence without representation. Artificial intelligence, 47(1-3):139-15,, 1991.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. ar Xiv preprint ar X12: 2023.
Josiah Carlson. Redis in action. Simon and Schuster, 2013.
Harrison Chase. Langchain: Building applications with Ilms through composability, 2022.
URL http:////gggttbboo.oo///aaaaaaaa
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arxiv preprint arXiv:2107.03374 2021.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent col- laboration and exploring emergent behaviors in agents. arXiv preprint arXiie2308..1844 2023.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Binding language models in symbolic languages. ICLR,
2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arviv preprint ar arXio:2204.02311, 2022.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023.
Roy Thomas Fielding. Architectural styles and the design of brtworr--bssed software architecfures. University of California, Irvine, 2000.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1126-1135. PMLR, 2017.
Dawei Gao, Zitao Li, Weirui Kuang, Xuchen Pan, Daoyuan Chen, Zhijian Ma, Bingchen Qian, Liuyi Yao, Lin Zhu, Chen Cheng, et al. Agentscope: A flexible yet robust multi-agen platform. arXio preprint arXim:2402. 10344, 2024.
11

---

Published as a conference paper at COLM 2024
Zhibin Gou, Zhihong Shao, Yeyun Gong, yelong shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving, 2023.
Significant Gravitas. Auto-gpt, 2023. URL Auto-GPT
tas/
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.0033 00352, 2023.
Leslie Pack Kaelbling et al. An architecture for intelligent reactive systems. Reasoning about actions and plans, pp. 395-410, 1987.
Tae Soo Kim, Yoonjoo Lee, Minsuk Chang, and Juho Kim. Cells, generators, and lenses: Design framework for object-oriented interaction with large language models. In The 36th Annual ACM Symposium on User Interface Software and Technology (3IST 2023.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general-purpose assistants. Xrx preprint ar ArX::22309. 10020, 2023a.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arviv preprint arXiv:2303.17766, 2023b.
Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented Ilms. ar preprint arXiv:2304.08244 2023c.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. Xrv preprint ar drXio:2305.17390, 2023.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Agents, August 2023.
Openais Killian Lucas. open-interpreeer: Openai's code interpreter in your terminal, running locally. http://iihthb.cooiiiiiiimaaa/app http://iiiaaaiicsoeereeeeeeeeeer 2023. URL htths::////ttttbbb com/ KilliianLucasoope--inneet interprettt GitHub repository.
Pattie Maes. Designing autonomous agents: Theory and practice from biology to engineering and back. MIT press, 1990.
Andrew McNutt, Chenglong Wang, Robert A Deline, and Steven M Drucker. On the de- sign of ai-powered code assistants for notebooks. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-16, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXio:1312.566 2013.
Yohei Nakajima. Babyagi, 2023. URL /ttp:///////ggtttbooo/////////i/
OpenAL. Chatgpt plugins. http:://oppaaa..coo////////hhhee
2023a.
OpenAI. GPT-4 Technical Report, March 2023b.
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior, August 2023.
12

---

Published as a conference paper at COLM 2024
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arviv preprint arXiv:2305.1533,, 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. ar preprini ar arXt::2304.03277, 2023a.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. ar preprint ar Xii::3309.00001, 2023b.
Cheng Qian, Chi Han, R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. Creator: Disentan- gling abstract and concrete reasonings of large language models through tool creation. arv preprint arXiv:2305.14315, 2023.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang. Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. ar preprint Xio:2330408854, 2023a.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arxiv preprint Xi::2307.16789 2023b.
Stuart J Russell. Artificial intelligence a modern approach. Pearson Education, Inc., 2010.
Timo Schick, Jane Dwivedi-Yis Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Xiv preprint arXiv:2302.04761 2023.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pp. 3135-3144. PMLR, 2017.
Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world applications via restful apis. Xiv preprint ar Xix::2306006224 2023.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction- finetuned text embeddings, 2023.
Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. Sensecape: Enabling multilevel exploration and sensemaking with large language models. arXiv preprint ar arXiv:2305.1148; 2023.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285-286, 2005. URL i
semanticscholar
org/CorpusID:: 9166388
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open Foundation and Tune-Tuned Chat Models, July 2023.
13

---

Published as a conference paper at COLM 2024
Szymon Tworkowski, Konrad Staniszewski, Mikotaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Milos. Focused transformer: Contrastive training for context scaling. arvio preprint ar arXo::237..03170 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arv preprint ar Xrx::3305.16691, 2023a.
Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating iims in multi-turn interaction with tools and language feedback. Xiv preprint ar Stt::209.11699, 2023b.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Michael Wooldridge and Nicholas R Jennings. Intelligent agents: Theory and practice. The knowledge engineering review, 10(2):115-152, 1995.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint ar arXiv:2308.08155, 2023.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, al. Unifiedskg: Unifying and
structured knowledge grounding with text-to-text language models. ar arxiv preprint Xtt::2201.0090 2022.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-contex: scaling of foundation models, 2023.
Binfeng Xu, Xukun Liu, Hua Shen, Zeyu Han, Yuhan Li, Murong Yue, Zhiyuan Peng, Yuchen Liu, Ziyu Yao, and Dongkuan Xu. Gentopia: collaborative platform for tool-augmented llms. Xrv preprint arXio:2308.04034 2023a.
Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. Rewoo: Decoupling reasoning from observations for efficient augmented language models. Xiv preprint arx::22305.1832. 2023b.
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, et al. Lemur: Harmonizing natural language and code for language agents. Xiv preprint ar Xio:22300006800, 2023c.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Stan- dardizing and benchmarking interactive coding with execution feedback. arxio preprint drXiv:2306.14898, 2023a.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arxiv preprint ar Xiv:2309.1 17441, 2023b.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ar Xiv preprint XxXi::210000362, 2022b.
14

---

Published as a conference paper at COLM 2024
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023a.
Shen Zheng, Yuyu Zhang, Yijie Zhu, Chenguang Xi, Pengyang Gao, Xun Zhou, and Kevin Chen-Chuan Chang. Gpt-fathom: Benchmarking large language models to decipher the evolutionary path towards gpt-4 and beyond, 2023b.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXit:2305.112000 2023a.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building autonomous agents. ar preprint arXiv:2307.3385. 2023b. URL https
//webarena. dev.
Wangchunshu Zhou, Yuchen Eleanor Jiang, Long Li, Jialong Wu, Tiannan Wang, Shi Qiu, Jintian Zhang, Jing Chen, Ruipu Wu, Shuai Wang. Shiding Zhu, Jiyu Chen, Wentao Zhang, Ningyu Zhang, Huajun Chen, Peng Cui, and Mrinmaya Sachan. Agents: An open-source framework for autonomous language agents, 2023c.
A Implementation Details
A.1
Implementation Details of User Interface
A.1.1 Data Model to Map Data
Taking database objects as an example, common encounter structures housing dozens or hundreds of tables with thousands or even millions of records. For humans, this data is often interpreted through organized, scrollable windows or analyzed utilizing data visualization tools such as Tableau
3 On the other hand, an LLM typically cannot process this vast amount of data end-to-end, necessitating more selective, formulated presentation, e-g., linearization of the first few rows as string. This discrepancy gives rise to the requirement for a more adaptable method of data storage, transcending a mere string mapping or adherence to specific prompting templates.
To address this, we use DataModel, which serves as an encapsulating entity designed to intake specific forms of raw data, be it tabular structures, databases, images. Its primary function is to process and transform this raw data into various output formats that are compatible and comprehensible to range of designated receivers, including humans, frontend interfaces, computational systems, and LLMs, among others. We establish a suite of over ten fundamental classes to encapsulate data in various formulations, ensuring that each entity involved receives an angle of data that is both relevant and manageable, enhancing efficiency in demo constructions and LLM prompting processes. Through the implementation of the DataModel concept, we anticipate transformative shift in the way data handled, fostering more coherent and dynamic interactions between humans and AI.
A.1.2 Strategic Data Storage
Due to the multi-user nature of the actual application, and the numerous conversation records associated with each user, we store data on different mediums based on varying requirements, instead of simply ignoring them in memory or hacking into some temporary files. Temporary variables such as constructed prompts are stored computer memory; global variables essential within the application are stored in Redis (Carlson 2013); while user conversation data and distinct user data are stored a database system, we utilize MongoDB (Banker et al. 2016).
https://wwwttableuu.cmmm
15

---

Published as a conference paper at COLM 2024
A.1.3 User-Centric Interface
Academic-oriented agents typically employ text-based or command-line interfaces, which can impede user interaction and practical task completion. Hence, we introduce an Adap- tive User Interface that seamlessly integrates user inquiries, agent actions, and associated outputs into the conventional chat workflow. This interface parses tool usage data from back- end outputs and responsively renders rich media. Specifically, we design and implement various interfaces that adapt to each environment.
For Data Agent, we interleave images, code snippets, console outputs, and interactive visualizations with markdown text altogether, echoing the structure of computational notebooks frequently employed in data science tasks (McNutt et al. 2023). This enables users to track data analysis workflows interactively and conversationally with our data agent, providing immediate and responsive feedback, thereby enhancing user engagement and sustaining analytical continuity.
For Plugins Agent, we support both user-selected and automated plugin selection. Given the diverse real-world tasks served by different plugins, we render their outputs accordingly, For example, we leverage Cards to hierarchically display relevant images and text, typically used for presenting websites, articles, and online products. Thus, we provide users with an immersive and intuitive task-focused experience, enhancing the efficiency and enjoyment of interaction with our plugin agent.
For Web Agent, we incorporate different interaction method via a Chrome extension that activates upon detection of potential user intents requiring web browsing by our web agent. This extension operates within the browser sidebar, sequentially inspecting, manipulating, and interpreting automatically opened web pages. Meanwhile, we display the execution plans and steps undertaken by our web agent, enabling users to comprehend and follow the process, and intervene if necessary. This liberates users from the manual and tedious task of inspecting and interacting with web pages by themselves.
A.1.4 Real-time Response Streaming
By default, when we request a text completion from a provider such as OpenAI, the entire completion is generated before being sent back in a single response. However, when we are generating long completions, waiting for the response can take many seconds. To get responses sooner, we can 'stream' the completion as it's being generated, considering the auto-regressive feature of GPT models. This allows us to start printing or processing the beginning of the completion before the full completion is finished. However, introduces significant challenge in delineating the specific roles of each generated token in real-time, iee. some tokens are plain text that can be directly printed out, some tokens serve as API calls that need to be parsed and then can be printed out, some are special tokens that need to be marked for special handling.
Traditionally, real-time identification of the distinct roles of individual tokens - be they directly displayed responses, undisclosed planning sequences, or internal thought pro- cesses (Wei et al. 2022 Yao et al. 2022b)- has not been focal point of research and develop- ment in dialogue systems and existing agent frameworks, such as LangChain (Chase 2022).
Yet, in practical applications, this facet is crucial in enhancing the user experience. ChatMI aiming at facilitating the determination of dialogue termination points, and possibly updat- ing grammar rules to handle complex role dynamics within response characters.
We recognize the correlation between this process and automata theory, specifically drawing parallels with pushdown automata (Autebert et al.) 1997). Our approach focuses on the early identification of the roles of characters in the data stream, thus enabling a prompt display to the user. This early interaction not only bridges the gap between generation and user receipt but also fosters a more interactive and engaging user experience.
16

---

Published as a conference paper at COLM 2024
A number of LLMs service providers self-hosting open-source frameworks provide streaming APIs, such as OpenAT as well as FastChat
For models solely used for chatting, the parsing is relatively simple. In fact, one might argue that parsing isn't required at all; one can merely render these tokens directly onto the
frontend using markdown. OpenAl designed a simple ChatML (Chat Markup Language) for this purpose. However, for language agent applications prompted using LLMs, the situation becomes more intricate. The tokens outputted by the LLM might be used to initiate a tool call, or they might signify the tool itself, or even parameters for the tool. They can mark the beginning of a displayed response or the content of the response itself. The roles are multifaceted. One direct approach involves setting markers for the beginning and end of these roles, and then prompting LLMs to output according to these markers. But this demands that LLMs have a strong ability to track instructions and generalize, especially when dealing with formats they have never encountered before. Hence, we shifted the pressure from generation to parsing, attempting to address this using automata. We discovered a strong correspondence between automata and the role identification of streaming tokens. Each state in the automaton can be mapped to role of token (though the granularity might differ slightly, requiring further code-based adjustments). Each character read results in a state transition described by
: Q x (Eu x P(QxI*)
leading to a new state s' The state s' indicates the current role of the chat.
A.1.5 System Robustness
The robustness of agents is important when it comes to realistic user experience, including but not limited to failure handling, in-time response, and token overflow. Though previous agents might shed light on some of these aspects, OpenAgents offer wrapped-up components or off-the-shelf examples, which developers can follow to build robustly functional agents.
Failure Handling is unpredictable, and may occur when utilizing real-world infrastruc- ture, such as various API services including the LLMs API service and Plugin API services from different providers. Identifying the types of these failures and deciding whether to retry or terminate with an error message problem that requires design and resolution. We have addressed this issue wherever there is potential for such problems to arise.
In-time Response is usually hindered by the LLM call overhead (both decoding time and rate limits) or different plugín APIs' response time. As these practical limits are bound by the service providers, we make some workarounds from the client side to either reduce the response time or improve the user experience. First, since multiple threads may call LLMs in parallel, we collect an LLM key pool (e.g., OpenAl API key) to be queried in turn, alleviating the rate limit burden of each single key. Second, we allow users to stop and retry each generation on will. This is crucial because it gives users the freedom to choose whether they would like to wait for the current response when the already generated tokens are not satisfied (also require the Streaming technique introduced later) or there is a block or stuck in remote API calling underneath. Without it, users may end up waiting so long for an unwanted response, if not no response. Third, we set maximum waiting time for each remote API call, and categorize detailed error information for each failed response, so that users will always know what happened inside the agent and how to interact next.
Token Overflow is a common situation in prompting LLMs where the input and output tokens exceed the max token limits of the LLM. We implement a MessigeDataMotee subclass to explicitly truncate message history tokens and extract the information needed (e-g., actions, tool responses) from history. Concretely, the whole chat history is in the agent
17

---

Published as a conference paper at COLM 2024
memory by default, and will be truncated from the beginning by round iteratively until the max token limit is met. Since information in all types is wrapped in DataModel, we can seamlessly handle the token overflow by calling the handling function anywhere in the data flow, otherwise developers have to design truncation and parsing APIs according to different data types and modalities.
A.1.6 Chrome Extension
It is difficult to control the browser the user side since the website or application does not have enough rights in most cases. Let applications run on the computer terminal (e.g. OpenInterpreter [] and open mock browser to navigate on the web is easier implement. However, this method could face some restrictions and some risks in the long shot. It also limits the development of large-scale web assistant applications for demonstration, verification or other purposes. To build publicly accessible and user-friendly web agent application for demonstration, the agent should be able to control the user's browser directly and can be interrupted or taken over easily by the user. With this in mind, we use the Chrome extension to implement our web agent. The agent can take actions on the user's
g browser through Chrome debugger API We parse the output of the agent to extract the action it takes and then use the sendCommand function in Chrome Debugger API to send the action to take to the browser, and then the browser will execute the action instead of the user. During the execution of Web Agent, the user can keep watch on the thinking and action process all the time and can easily interrupt the execution.
http:://ewweooeoooooocco/////cce /
18

---

Published as a conference paper at COLM 2024
A.2
Implementation Details of Language Agents
A.2.1 Grounding Source
Herein, the grounding source represents the data files that agents can utilize during interactions, especially structured knowledge form (Xie et al. 2022) like Excel files, CSV files and JSON files, etc. We integrate file-uploading system that allows users to seamlessly upload their files for the agents to process. Agents have the capability to either directly write code or employ alternative methods read, modify, and save these files. This enables agents to accomplish specific tasks as humans require, rather than merely generating single actions. Consequently, we have implemented a grounding source pool store and represent data uploaded by human users. We leverage the DataModel to provide a linearization method for each supported file type and use the file name as the key for these files. Agents can access the content corresponding to file names and actively invoke the file content as humans desire.
A.2.2 Auto-selection on Plugins
Currently, users need to first select a plugin, like the OpenAl Plugins (OpenAl 2023a), before the system can execute their commands using these plugins. However, real- world scenario, choosing appropriate plugin from hundreds of options can be a daunting task. Oftentimes, even if users know what they want, they might not know which specific plugin can best assist them. Therefore, we've introduced special feature. Each time the user provides new instruction, we employ a text embeddíng technique, specifically Instructor (Su et al. 2023), to identify which tools described match closely to the user's s instructions, eliminating the need for manual plugin selection.
A.2.3 Auto-scaling Plugins
While tools have been constructed under simpler models QQian et al. 2023 Cai et al. 2023
developing tools for agent use presents unique challenges, requiring attention to both the agent and the LLM infrastructure. Creating tools from scratch ensures quality but is resource-intnssive Qin et al. (2023a). We source API provider information from platforms
like RapidAPI 10] and the OpenAl plugins store, and test these providers for accessibility by LLMs, This method has enabled us to introduce over 200 high-quality plugins. However, using LLMs for plugin construction has its challenges and requires human intervention. Future work should investigate more efficient scaling techniques.
A.2.4 Executable Environments
Code Execution Environment We provide SQL and Python environments for code ex- ecution. We extract the SQL query from the agent output, and leverage SQL engine !T] to execute the query. For the Python environment, we largely referred to the official Kaggle 12 Dockerfile the docker container provides a safe and isolated sandbox for code execution.
API Calling Environment We make API calls to adhere to the principles of RESTful design (Fielding) 2000; Song et al. 2023). To make sure we're ready for the different needs of each API provider torehead, we organize API calls into separate functions. Within these functions, we manage specific exceptional conditions, such context overlength (we leverage DataModel to handle that) or particular invocation methods. This strategic encapsulation ensures that the agent can seamlessly operate without being encumbered by vendor-specific anomalies. Subsequently, when the agent elects to utilize a particular plugin, the designated plugin name cane-allog with requisite information like the API documentation (herein referenced as open API file)-is relayed to dedicated parsing and dispatching
10 "https:/rappidap.coom//bbb
11 https://wwwwss/lite.rrr///
12
19

---

Published as a conference paper at COLM 2024
module. This module, also powered by prompting LLMs, orchestrates the requisite function call and awaits its resultant output. Given that APIs are scattered across various servers on the internet, and their quality and reliability can be inconsistent, the interfaces may change unpredictably.
Web Environment manipulating real-world web navigation. We let the language agent control the web browser (Chrome in our case) via the Chrome extension built by us and navigate on any websites accessible by the browser. The extension can control the web page by performing actions 13
We also explore the possibility of whether the web agent is capable of
such as clicking an element or typing values into an element through Chrome Debugger
In this way, the web agent can act like real human user the webpage, and the human can also interact with it anytime and continue to act on the webpage. This environment makes the web navigation more real and visible and provides the platform to build generalist web assistant in the future. On the contrary, Browse with Bing can only act behind the screen and the navigation process actually occurs on the server side rather than the client side, which significantly limits the function of web navigation and can hardly be seen, controlled or taken over by the user. This makes it quite hard to build web assistant that can help users execute daily tasks on the user's computers, and enable more diverse scenarios for any website the wild unlike screenshotted or self-hosted ones (Shi et al. 2017) Yao al..
2022a Deng et al., 2023 Zhou et al. 2023b)). We also believe this feature can be beneficial to further research and development in the automatic web agent area.
20

---

Published as a conference paper at COLM 2024
B Use cases
For quick reference, we demonstrate the use cases of data agent, plugin agent, and web agent Figure Figure and Figure respectively.
B.1 Data Agent
(a) Kaggle Dataset Search: Input your sub- ject domain preference, and the Data Agent will traverse the Kaggle API to iden- tify datasets fitting your specifications.
(b) Python Code Execution: Utilize the Python tool to craft Python code and ex- ecute rudimentary data sanitization on given table.
(c) SQL Query Execution: Using the SQL tool, the Data Agent can execute data queries on an input table dataset.
(d) Interactive Echarts Visualization: The paccaee Data Agent employs the pyecharts) package, transforming it into JSON object. This ob- ject can rendered as an interactive Echarts plot, as facilitated by the ECharts library.
(e) Image Operations: Beyond textual and tabular data, the Data Agent can also per- form basic manipulations on images.
(f) Interesting Cases:
Demonstrate the earth's roundness, create GIFs with descend- ing letters, or construct a word cloud from an academic article.
Figure 6: Use cases for Data Agent
B.2 Plugins Agent
21

---

Published as a conference paper at COLM 2024
(a) Shopping with 'Klama Shopping': Spec- ify product and the Plugins Agent, using "Klarna Shopping', will locate it, presenting it in an aesthetically pleasing layout.
(b) Weather Updates via 'XWeather': Seeking weather information? Communicate your lo- cation to the Plugins Agent and 'Weetther' will fetch the latest updates.
(c) Concept Visualization with 'Show Me': Desire visual representation of idea? The "Show Me' plugin, upon receiving your input, will generate and elucidate the concept.
(d) Inquiry with 'Wolfram Alpha': Engage with the 'Wolfram' plugin to investigate and gain insights into diverse topics.
(e) Workflow Configuration via 'apier': Har- ness the 'Zapier' plugin to understand and manipulate your Zaps.
(f) Travel Planning with Multiple Plugins: Combine 'Klook,, 'Currency converter' and 'WeatherViz' for an enriched travel experi- ence.
(g) Automatic Plugin Selection: Uncertain about plugin functionalities choices? Con- vey your requirements to the Plugins Agent, and will autonomously select the optimal plugin, simplifying your experience.
Figure 7: Use cases for Plugins Agent
22

---

Published as a conference paper at COLM 2024
B.3
Web Agent
Save
off
(a) Flight Finder: Detail your travel informa- tion, and the Web Agent will scout for flights, for instance, the most economical ones.
(b) IMDb Comments Summarization: Con
sidering movie? The Web Agent will peruse
IMDb comments for your selected film, provid- ing a concise summary.
(c) Google Map Navigation: Supply your start- ing point and destination to the Web Agent, and it will delineate the route using Google Maps.
(d) Twitter Post Creation: Share your desired topic with the Web Agent, and it will curate and post content to Twitter on your behalf.
(e) Google Form Assistance: For club registra- tions or event sign-ups, provide the necessary form and details to the Web Agent, and it will adeptly complete the submission.
Figure 8: Use cases for Web Agent
23

---

Published as a conference paper at COLM 2024
C Key Prompts in OpenAgents (Agents)
The main technique behind OpenAgents LLMs prompting, it is crucial for building appli- cations upon LLMs and controls the quality (e.g. retrying after failure), functionalities (e.g. understanding the context and available tools clearly), user experience (e.g. whether the response is in good format) and even safety (e-g. refusing to respond to harmful questions or questions that attack the server) of agent responses. Here we show key prompts we explore and use to prompt LLMs to serve real-world agents. These prompts are combined with the true response of the user, dialogue history and environment. Refer to our code for more details about the variable inside the prompts and the running logic.
C.1
Data Agent
System Prompt
You are an open agent, friendly and intuitive interface to guide human through every stage of human data lifecycle, Whether human are loading, processing, interpreting data, open agent is always human's s fingertips through our interactive chat system.
Empowered by array of innovative tools that generate and execute code, open agent delivers robust, reliable answers to human queries. Whenever possible, employs these tools to give human rich insights, like dynamic code generation & execution and compelling visualizations. And You will always proactively and correctly using all tools to help with human.
Get ready for seamless and insightful journey with open agent, the personal assistant for things data!
TOOLS
have direct access to following tools.
Format Instructions
RESPONSE FORMAT INSTRUCTIONS
When tools or generate final answer, please output response in one of formats:
**Option Explain Use Tool**
If the response involves using a tool, you can start with a natural language explanation[Optiona]]] plus exactly one tool calling[MUST]. But *smake sure no words answer appended after tool calling json**, The tool calling format should be markdown code snippet with the following JSON schema:
--Jjoon
<(((
"action": string wrapped with \rV,, // The action take. Must be in the list [[tool_names]]
baction_input": string wrapped with \"\" // Natural language query to be input to the action tool.
}>>}
[##Restrictione+] Please note that ONLY one tool should be used per round, and MUST stop generating right after tool calling and make sure any text appended after tool calling markdown code snippet. Save your words.
24

---

Published as a conference paper at COLM 2024
NEVER EVER EVER make up tool not in [[tool_names]]
NEVER EVER EVER generate code as action input when using tool. Just input natural language by using/paraphrasing human query.
x*Option #2:**
Use this if you want respond directly to the human. you want to respond directly to the human plain natural language response. However, you initially generated a natural language response and then decide to use tool, make sure to include the tool action and input after the initial response.
using tool, provide
Note if the human asks for malicious code, and just respond directly deny the request and give your professional reason. Don 't use any tool.
The malicious code includes but not limited to:
Endless operations and excessive waiting (e.g., while True, long print, input
()
System crash (e.g., any risky system command)
3. Data loss (e.g., list or delete files)
4. Leak sensitive information (e.g., (e.getenv())
Establish network connections (e.g., requests. ())
Cause any other security issues
[Mandatory to notice] It is imperative and must to utilize tools whenever the human's query tasks that implies using tools, such searching online, generating code, executing code, any other complex functionalities. You must try use tools to solve human queries these cases.
Begin.
Suffix
(input)
Tool Response Template
TOOL RESPONSE:
(observation)
THOUGHT
Okay, So what's 's next? Let's assess if the tool response is enough to answer the human's initial query. Please follow these instructions:
Evaluate Tool Response [Mandatory]: Carefully evaluate the tool s response and determine if it sufficiently addresses the human's 's query. Consider the content and implications of the tool's 's response.
tool response does not fully address the query or if an error occurred during execution, you may proceed with additional tool usage. However, exercise caution and limit the number of iterations maximum of three. You start with natural language explanationn[Oationa]]]] plus exactly tool calling[MUST]. But **make any words answer appended after tool calling json**. Follow this format additional tool usage:
Consider Additional Tool Use [Optional or 3]:
jssoo
<C((
25

---

Published as a conference paper at COLM 2024
"action": string wrapped with V"V, // The action to take. Must be of [[ tool_names]]
"action_inpurs string wrapped with \"X" // Natural language query to be input to the action tool
})}}
[*<Restriction*]] Please note that only one tool should be used per round, and you MUST stop generating right after tool calling and make sure no any text appended after tool calling markdown code snippet.
Deliver Comprehensive Answer [Optional or 3]: sufficiently addresses the query, deliver a comprehensive answer to the human. Focus solely the content and implications of the tool's 's response. MUST NOT include explanations of the tool's 's functions.
tool response
3.1. Avoid Tables, Images, and Code [Mandatory]: MUST NOT generate tables or image links in the final answer, assuming the human has already seen them. Avoid generating code in the final answer as well. Instead, paraphrase the code into human query if you need explain it.
Note. you must do 1; For 2 and 3, You must choose one between them and generate output following the format.
Begin.
C.2
Plugins Agent
System Prompt
You open plugins agent, friendly and intuitive assistant to guide you through every aspects your work and your daily life. Open agent is always your fingertips through our interactive chat system.
You can aware of what plugins you have, and use the plugins properly in right order to finish what user wants.
Today is .n. .stri())
+ datetime. datetime.datetimm.. now() www.:nowwww.mnnennn..e wtt.stttttemmeeeeeeee ("XY-Xm-Xd"))
and you should adapt the input to fit into the date, for example, seasonal information, or today's s date as coordinate, etc.
To make response informative, always speak includes the following information in MARKDOWN format when responding message, that is: Natural language explanation, that make explain the API output in human readable way;
2. Organized information such as bullet points or MARKDOWN tables, followed by the links to the items (that in the API output), news etc. if API output contains the information;
The links should MARKDOWN format and have value in it. If reference information is provided in the API output, like links to the items, news etc.
Your explanation MUST provide the links each items and links can be clicked on when API output contains the information. The links better attach natural language explanation through MARKDOWN syntax, for example, [Renewable Energy Center renewable-energy/):
Climate and Energy Solutions] (https://wwwr) wwt.:.ccee..rr.... www.ees....oooee.././.
4. If there are image would like to display, please use MARKDOWN syntax to display it, for example, [[image](tttp::///i http://wwwwweeesg.g./e www.://www..ggottte.oo////eeeeea
26

---

Published as a conference paper at COLM 2024
5. Try to speak more and show all the information you in organized way, that will make you better assistant, especially when you giving the final answer
PLUGINS
The plugins you can use are:
Format Instructions
RESPONSE FORMAT INSTRUCTIONS
When you use tools or generate final answer, please output response in one of two formats:
*+Option 1: Explain and Use plugin**
the response involves using a plugin, you can start with a natural language explanation[[pttiona]]]] plus exactly one plugin calling(MUST], and ends with no more words. The plugin calling format should be markdown code snippet with the following JSON schema:
-Jjson
<(((
"action": string wrapped with V"V, // The action to take. Must be one in the list [[tool_names}]
"action_input": string wrapped with \"\" Query input the action plugin. Pass as much information as possible to the plugin from the history of the conversation.
})}}
NEVER EVER EVER make up plugin not [[tool_names}]
You MUST pass as much information as possible to the plugin from the history of the conversation. It could be natural Language or structured language like jsonl, csv, etc. BUT MUST in single line. (Please note that ONLY ONE plugin should be used per respanse.)
**Option #2: **
If want to respond directly to the human without using plugin, provide plain natural Language response. However, if you initially generated natural language response then decide to use plugin, make sure to include the plugin action and input after the initial response.
Begin.
Suffix
(input)
Tool Response Template
PLUGINS RESPONSE:
[observation]
THOUGHT
27

---

Published as a conference paper at COLM 2024
Okay, what's next? the plugins' response enough answer human's initial query? Please follow these instructions:
1. Evaluate plugin Response [Mandatory]: Carefully evaluate the plugin's response and determine if it sufficiently addresses the human's query. Consider the content and implications of the plugin's response.
2. Consider Additional plugin Use [Optional 2 or 3]: If the plugin response does not fully address the query or if an error occurred during execution, you proceed with additional plugin usage. However, exercise caution and limit the number of iterations to maximum of 5. You can start with natural language explanation[Optional]] plus exactly one plugin calling[MUsT]. Follow this format for additional plugin usage:
jsonn
{({(
"action": string wrapped with V"V,, // The action to take. Must be one of [{ tool_names]]
"action_input": string wrapped with \"X" // Query be input to the action plugin. Pass as much information as possible to the plugin from the history the conversation.
}>
(Please note that ONLY ONE plugin should be used per response.)
3. Deliver Comprehensive Answer [Optional 2 or 3]: If the plugin response sufficiently addresses the query, deliver a comprehensive answer to the human. Focus solely the content and implications of the plugin's response. MUST NOT include explanations of the plugin's functions.
Note. you must do 1; For 2 and 3, You must choose one from them.
Begin.
C.3
Web Agent
System Prompt
You are web agent, friendly and intuitive assistant to guide you through every aspects of your work and your daily life. Open agent is always your fingertips through our interactive chat system.
Here are detailed instruction for you. Each time you generate response, you should think step by step to follow instructions below. You are helpful assistant that is provided with plugin called "WeBot" which is web navigation agent tool and should leverage the power it help human fulfill their needs, such as booking a hotel, buying ticket, or searching for information, etc.
Human will ask you questions, and can use WeBot to help them, they are assumed know nothing about the WeBot.
Here are something you MUST remember:
1. After receiving output from the WeBot, you should check
1.1 whether WeBot interrupted, if so you should NEVER try again by yourself.
1.2 whether WeBot failed or had error (not because of interruption), so you should tell the human the error.
2. Today is strip()
datetime. datetime. www.ttiimmedtttmmm..mmmmm
dotetime.now() strftime ("XY-XX"="""""
28

---

Published as a conference paper at COLM 2024
and you should adapt the input to fit into the date, for example, seasonal information, or today's date as coordinate, etc.
NEVER EVER EVER use other plugins except WeBot.
TRY YOUR BEST to break the question down into several parts and answer them one one.
TRY YOUR BEST to use the WeBot help you answer the question, you don't 't need to mention that you will use which WeBot, just use it.
To make your response informative, always speak includes the following information in MARKDOWN format when responding message, that is: Natural language explanation, that make explain the API output a human readable way;
2. Organized information such as bullet points or MARKDOWN tables, followed by etc. if API output contains
the links to the items (that in the API output), the information;
3. The links should in MARKDOWN format and have value in it. If reference
information is provided in the API output, like links to the items, news etc. Your explanation MUST provide the links each items and links can be clicked on when API output contains the information. The links better attach natural language explanation through MARKDOWN syntax, Energy Center for Climate and Energy Solutions] (https://www. ortes..org/onteeeeeee renewable-energy/
example, [Renewable
4. If there are image would like to display, please use MARKDOWN syntax display it, for example, [ http:///wwww..eeee////otttttt/te orrg//coneenn//eenwbbbeeeeennrg
5. Try to speak more and show all the information you got in organized way, that will make you better assistant, especially when you are giving the final answer.
PLUGINS
The plugins you
Format Instructions
RESPONSE FORMAT INSTRUCTIONS
When tools or generate final answer, please output response in one of formats:
**Option Explain Use WeBot**
If the response involves using a WeBot, you can start with a natural language explanation[Optional]. plus exactly one WeBot calling[MUST], and ends with no more words. The WeBot calling format should be a markdown code snippet with the following JSON schema:
---json <(((
"action": string wrapped with \"\", // The action take. Must be WeBot paction_input": string wrapped with \"\" // Natural language query to input to the WeBot.
NEVER EVER EVER make up plugin except [[tool_nand [amo__]aame]]] NEVER EVER EVER generate code as action input when using WeBot. Just input natural language by using/paran paing//praphrasing human query.
(Please note that ONLY ONE WeBot should be used per response.)
**Option
29

---

Published as a conference paper at COLM 2024
If you want to respond directly to the human without using WeBot, provide plain natural language response. However, you initially generated natural
language response and then decide to use WeBot, make sure to include the WeBot action and input after the initial response.
Begin.
Suffix
(input)
Tool Response Template
PLUGINS RESPONSE:
{observation)
THOUGHT
Okay, So what's next? Are the WeBot's response enough to answer human's 's initial query? Please follow these instructions:
1. Evaluate WeBot Response [Mandatory]: Carefully evaluate the WeBot's s response and determine if it sufficiently addresses the human's query. Consider the content and implications of the WeBot response.
--jjoon
{(((
"action": string wrapped with V"V,, // The action to take. Must be of WeBot
faction_input": string wrapped with A"\" // Natural language query to be input to the WeBot
}}}}
(Please note that ONLY WeBot should used per response.)
3. Deliver Comprehensive Answer [Optional 2 or 3]: If the WeBot response sufficiently addresses the query, deliver comprehensive answer to the human. Focus solely the content and implications of the WeBot's response. MUST NOT include explanations of the WeBot's 's functions.
Note. you must do 1; For 2 and 3, You must choose one from them.
Begin.
D
Key Prompts in OpenAgents (Executors)
number of Executors OpenAgents are implemented by prompting as well. Here we show some of the prompts as exemplars. Due to space limitations, we only show few typical examples in the article. Refer to our code for more details about the variable inside the prompts and the running logic.
D.1 SQL Database
Prompt
30

---

Published as a conference paper at COLM 2024
Here chat histaries you may refer to, maybe empty. {chat_history)
Given an input question, first create a syntactically correct (dialect) query to run, then look at the results of the query and return the answer. Never query for all the columns from specific table, only ask for the few relevant columns given the question.
Pay attention to use only the column names that you can see the schema description. Be careful to not query for columns that do not exist. Also, remember to wrap the table names in double quotes. Use the following format:
Question: "Question here"
SQLQuery: "SQL Query to run"
SQLResult: "Result of the SQLQuery"
Answer: "Final answer here" Only the tables listed below.
(table_infs
Question: (question)
D.2 API Calling
System Prompt
You are acting like plugin system that understand user's needs and call APIs precisely for them.
User Prompt
Here the endpoints specs: (specs_str)
Here is the input string:
(input_str)
Select the right endpoint called that can process the input string. You need wrap the input into json object, that it could be fed into
function you selected. During wrapping, you should:
1. modify the value of each key so that it satisfies the requirements in function specs. For example, if the type of the value should be number, then you should modify it into number;
ignore the information that not useful or not applicable to the function you selected.
You fill values into some slots in the input_json, and then call the API. If the API returns valid output, then you succeed. Otherwise, you fail. json object in the following format:
Return function called
<(
"endpoint": "xxx", iinput_jsoo""(C "xxx": "xxx", "xxx": "xxx",
))
})
31

---

Published as a conference paper at COLM 2024
Retry Prompt
Here the function specs: (specs_str)
Here is the input string:
(input_str)
Select the right function called that can process the input string. You need to the input str into json object, that it could be fed into the function you selected. During wrapping, you should: modify the value each key so that satisfies the requirements in function specs. For example, if the type of the value should be number, then you should modify it into number.
ignore the information that is not useful not applicable to the function you selected.
Return the function called and the json object in the following format:
<
"endpoint": "xxx", "input__json "xxx": "xxx",
"xxx": "xxx",
}}
}}
You have tried to call function to process the input string but failed. The output do not have enough information to answer the tool input. Here is the history your trials, each element this list means trial:
(trial_history)
You should firstly analyze your trial history, find the value of the key "errors" in the output each trial check whether there are any errors Then you may consider changing the input_json or endpoint based on the error information in your trial history, function specs and the input string. Return the function called and the json object in the following format:
{ "endpoint": "xxx", "input__json" "xxx": "xxx",
"xxx": "xxx",
}}
}>
Stop Prompt
Here are the function specs:
(specs_str)
32

---

Published as a conference paper at COLM 2024
Here is the input string:
(input_str)
Here is the output that you get from calling the API:
(api_output)
You need to decide whether the returne_bblock contains valid information or not. Some returned_block example, the returned_block may be empty, return json that says kind answer.
not have enough information to answer the tool input, for
Answer only by 'yes' or 'no"
D.3 WeBot
System Prompt
You are browser automation assistant.
You will be given user request and DOM of current webpage time, you need one action at time finally finish the task.
The last page you visited will be further fed into another model who is responsible for chatting with the user.
You MUST take one the following actions. NEVER EVER make up actions that do not exist:
{formattedActions)
will be given task to perform and the current state of the DOM. You will also be given previous actions that you have taken. You may retry a failed action up to one time.
This is an example of action:
<Thought>I should click the add to cart button</Thought> <Actiom-clicc((3))<AAttee
MUST always include the <Thought> <Action> open/close tags or else your response will be marked as invalid.
Rules you MUST follow:
If you input something a search box, MUST FOLLOW:
1.1 YOU MUST convert the instruction into proper query into the box rather than directly input it. e.g. YOU MUST input New York rather than New York amantments in the input box of zillow.com when user request about New York tments.
1.1 there are some options pop out, you MUST NOT directly go next action You MUST click one of the options.
2. You must only take one step at time. You cannot take multiple actions in a single response.
You should check whether your action last time successful. If not, you should retry ance. If still fails, you should try another way.
example 1: The box should be clicked and choose from the options and you just setValue and failed, you may consider to use click and then click the option. example 2: You click button button not clicked, you should retry once.
but after checking the page you found that
33

---

Published as a conference paper at COLM 2024
4. You should not consider the action to present the result the user. You only need available actions. info in current page enough for the user solve the problem, you should finish. The content on the page you saw might not be in English, you should be aware of this.
(plan)
Remember: you do not need to follow this plan exactly, but you MUST follow the rules above.
YOU MUST MUST check whether there some options pop out if your last action is setValue. there are some options pop out, you MUST click one of the options rather than go to the next action.
The id of the elements can be different each time. If click(1) last time you should not assume is the element this time.
User Prompt
The user requests the following task:
{user_query)
(previlus_actioos_sstinng
Current time: {current_time)
Current page contents: (processed_html)
Retry Prompt
The user requests the following task:
{user_query]
(previous_actioossssting]
Current time: {current_time)
Current page contents: {processed_html)
Your last answer has some problem: (retry_message)
34

---

Published as a conference paper at COLM 2024
E
User Interaction Statistics
To evaluate the performance and efficiency of our OpenAgents, we collected and analyzed variety of interaction metrics. In total, OpenAgents has collected 10K conversations with 56K messages contributed by 7,832 registered users, This section presents detailed statistical overview of user interactions, including the distribution of conversation durations, the number of conversations initiated, and the frequency and diversity of tool usage during conversations.
E.1 Conversation Distribution
We analyzed the distribution of conversation lengths across our user base, focusing on the number of turns and tokens. Figure presents these distributions in detail.
Firstly, the average number of turns per conversation is 5.6. While over 35,000 conversations (35%) end within two turns, indicating a single interaction with the agents, the majority of conversations are more extensive, involving multiple-turn engagements with agents.
Secondly, tools play critical role in agent problem-solving, with an average of 1.95 tool uses per conversation. This includes the automatically triggered data profiling tool for uploaded data.
Lastly, the majority of token consumption (over 80%) occurs on the agent side, with the average conversation token length being 1,229.
35

---

Published as a conference paper at COLM 2024
252
D02
(a) Conversations per User
(b) Rounds per Conversation
000
(c) Tools per Conversation
(d) Tokens per Conversation
(e) Tokens per User Message
(f) Tokens per Agent Message
Figure 9: Distribution Statistics of User Interaction Data.
36


Reviews:
[{"review_id": "leN05omr1B", "reviewer": "Reviewer_Cqqw", "summary": "The authors propose an open platform OpenAgents for deploying and interacting with language agents in everyday contexts. It addresses the gap in current frameworks by focusing on accessibility for non-expert users and emphasizing application design. The platform comprises three key agents: a Data Agent for handling data analysis tasks using Python/SQL and other tools, a Plugins Agent equipped with over 200 daily utility APIs, and a Web Agent designed for autonomous web navigation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and appreciation of our work. We are glad you highlighted the importance of our studied topic, the quality of the illustrations, and the provision of an open-source codebase. We have addressed the specific points you mentioned and will update the paper accordingly if it is accepted.\n\n\n**W1: Writing Style & Technical Depth:**\n\nA: Thank you for highlighting the need for more detailed technical explanations. We have expanded several subsections to provide deeper insights into our method. Specifically, in Section A.1.2, we elaborate on the challenges of strategic data storage, detailing our hierarchical storage strategy (local memory -> Redis -> persistent database) with illustrations. We explain which variables are stored at each level and the benefits of this approach. Additionally, we provide more details on grounding sources in Section A.2.1 and auto plugin selection in Section A.2.2. We will update the paper accordingly if accepted.\n\n\n**W2: No Experiments & Analysis:**\n\nA: We agree that quantitative verification is essential for demonstrating the effectiveness of OpenAgents. In the revised manuscript, we will include human-in-the-loop experiments: 1) user satisfaction surveys, 2) average time to complete selected tasks, and 3) failed case studies. Evaluating human-agent interaction is challenging, but we have made significant efforts to improve platform quality. We have established Discord and Slack channels with over 6,000 users and addressed 87 issues and engaged 16 new contributors. Despite the lack of reliable automatic evaluation, we are committed to enhancing OpenAgents based on user feedback.\n\n\n**W3: Preciseness & Timeliness:**\n\nA: Thank you for the observation. We have updated Table 1 with the latest data as of April 2024 to ensure our comparisons are both fair and current. \n| Name                | Interface                        |          |          |  Environment            |          |     |\n|---------------------|-----------------------------------|----------|----------|-------|-------------------------|----------|\n|                     | Online                            | Human Feedback | UI           | Coding Env.            | #Tools   | Web      |\n| **AgentScope** (Gao et al., 2024)  | ✓ | ✓ | Web | Wild | - | ✗* |\n...\nWe will further review recent literature, including Data-Juicer, and discuss contributions and potential integration points within our platform in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tGood illustrations of the functional utilities of the proposed framework.\n2.\tThe studied topic is important and appeals to a broad audience.\n3.\tThe provision of an open-source codebase, developed with considerable dedication, is both welcomed and appreciated.", "reasons_to_reject": "1.\t[Writing Style & Technical Depth] The paper presents its content at a too high-level fashion, leaving ambiguities regarding its novelty from a methodological perspective. It fails to clarify the rationale behind the selection and integration of numerous tools, the features to be provided, and what makes the construction of such a platform technically challenging and non-trivial. For instance, the discussion on \"tackling the challenge of strategic data storage\" in Sec A.1.2 is too brief, offering only two sentences without adequate details. This lack of information does not sufficiently explain the issues related to `data storage` or how the approach is `strategic`.\n2.\t[No Experiments & Analysis] The paper does not adequately justify the proposed platform due to the absence of quantitative verification. The lack of empirical and numerical support makes it difficult for readers to grasp the effectiveness and operational mechanisms of the platform.\n3.\t[Preciseness & Timeliness] Assertions and arguments to current works require careful review and updates, given that LLM-Agent is an extremely active and rapidly evolving field of study. For example, in Table 1, some markers and specific numbers are outdated (gathered by September 2023, whereas the submission deadline for COLM is March 29, 2024), potentially misleading new readers and being unfair to the works being compared.\n4.\tIt is recommended to explore additional literature on data agents and multi-agent systems for possible improvements or integration. For instance, a data agent with enhanced data processing capabilities could offer significant benefits [1], and the proposed *three agents* could potentially be enhanced to cooperatively work better  through more customized multi-agent workflows [2].\n\n[1] (arXiv’23, SIGMOD’24) Data-Juicer: A One-Stop Data Processing System for Large Language Models\n\n[2] (arXiv’24) AgentScope: A Flexible yet Robust Multi-Agent Platform", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f5bUkIZJdn", "reviewer": "Reviewer_zoss", "summary": "- This paper introduces OpenAgents, a versatile and open platform designed for the deployment and management of language agents in real-world settings.\n- OpenAgents consists of three types of agents: the Data Agent, the Plugins Agent, and the Web Agent.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for recognizing the broad utility and detailed exploration presented in our OpenAgents platform. We appreciate your constructive and detailed feedback.\n\n**W1: Gathering feedback from actual users**\n\nA: We agree that user feedback is crucial for evaluating the platform's effectiveness. Although our main focus is the design philosophy and technical details of OpenAgents, we discuss the need for new language agent metrics and in-the-wild evaluation in Sec. 5 and 6. Evaluating human-agent interaction is challenging, but we have made significant efforts to improve platform quality. While automatic evaluation isn't reliable yet, we are committed to enhancing OpenAgents based on user feedback.\n\n**W2: Performance of the auto-selection feature for plugins**\n\nA: The goal of the OpenAgents paper is on how features like auto tool selection significantly enhance user experience compared to ChatGPT Plus, rather than examining the capabilities of individual components in detail, which is beyond the scope of this paper.\n\n**W3: Novel contributions in agents' design**\n\nA: While the primary focus of our work is to create an accessible and versatile platform, we also highlight our technical contributions in two key areas. \n1. **System design:** For Data & Plugin Agents, we follow the ReAct agent definition but introduce techniques such as adaptive data mapping, data grounding, and auto tool selection/tool scaling to transition from proof-of-concept to real-world applications. \n2. **Web Agent:** We use a Chrome extension for the Web Agent, enabling dialog-based assistance to help users complete tasks through interactive actions.\nWe will revise the manuscript to detail our agents' unique design aspects.\n\n\n**W4: Chat API service and handling large volumes of user queries**\n\nA: We appreciate your concern regarding the chat API service. In our released code, we expose the chat/ endpoint API on the server side for handling large concurrent user queries. Users can directly query this API  in their programs/applications for concurrent access instead of on the web demo.\n\n\n**Q1: Missing Plugins**\n\nA: We previously uploaded 16 best-performing tools in our user test as some other public tools suffered from connection or maintenance issues. We have been cleaning and validating the rest tools this month and have uploaded all 200 API tools these days.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The OpenAgents platform is engineered to serve a diverse audience, encompassing general users, developers, and researchers. \n- It provides a wide array of features, including a web user interface, streaming capabilities, and more.\n- The paper offers an in-depth exploration of agent-based systems, yielding numerous insights that substantially enrich the field.\n-  It details the implementation meticulously, ensuring clarity in the processes involved.", "reasons_to_reject": "- Gathering feedback from actual users is crucial for evaluating the platform's effectiveness and overall user experience. However, the paper does not discuss this aspect.\n- The performance of the auto-selection feature for plugins has not been evaluated.\n- In terms of agents' design, there are no significant novel contributions.\n- It seems that the platform lacks a chat API service capable of handling large volumes of user queries.", "questions_to_authors": "- Do you open-source all 200+ APIs for Plugins Agent?", "extra_scores": {}}, {"review_id": "c60Z3dQmLx", "reviewer": "Reviewer_8ELQ", "summary": "This paper presents OpenAgents, an open-source platform for real-life language agent applications. The authors design three agents: Data Agent, Plugin Agent and Web Agent, and provide interfaces for different types of users (general users, developers, and researchers). The platform's open-source nature may inspire broader community involvement and innovation, advancing the development and application of language agent technologies. Overall, \"OpenAgents\" represents a high-quality and practically significant work.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and constructive feedback. We appreciate your recognition of the significance and quality of our work on OpenAgents. We have addressed your concerns and provided further clarification on the points you raised.\n\n\n**W1: Emphasizing the difference between OpenAgents and ChatGPT Plus:**\n\nA: Thank you for highlighting this important point. While we don’t have detailed knowledge of ChatGPT Plus’s internals, we emphasize the unique aspects of OpenAgents, including its open-source nature, modular architecture, and wide API and tool integration. Unlike ChatGPT Plus, OpenAgents offers customizable interfaces for different user types and supports the integration of external agents and plugins, enhancing flexibility and extensibility.\n\n**W2: Functionality based on code or APIs:**\n\nA: Thank you for pointing out a common issue with current API-calling-based agents. Although OpenAgents straddles both approaches with our WebAgent, we believe these two methods address different angles. The performance of advanced models like GPT-4 on benchmarks like WebArena and OSWorld is only around 10%, which is not practical for the current users and developers. In comparison, the API-based modeling approach is more easily implemented by current language models, making it a tradeoff in the application.\n\n**Q1: Comparison with LangChain and integration possibilities:**\n\nA: LangChain aims to facilitate the construction of proof-of-concept language agents while neglecting non-expert user access to agents and paying little attention to application-level designs. We have integrated these aspects into the backend logic, frontend logic, and user interface, building a product-grade platform, and uncovered many issues that only emerge in real-world applications, such as the need for handling long text efficiently, and the expansion and maintenance of real tools, etc.\nOpenAgents is designed to be compatible with LangChain. Users can integrate the two platforms easily to further enhance functionality.\n\n**Q2: Prompting strategies in OpenAgents:**\n\nA: Prompting is indeed a critical part of OpenAgents. Our platform supports a diverse set of prompting strategies, including templates, chaining, and context-aware prompting. We have designed OpenAgents to be user-friendly, allowing users to add and customize their own prompting strategies easily. This flexibility ensures that users can tailor the platform to their specific needs and preferences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This presents an open-source implementation of ChatGPT Plus；\n2. This paper provides a well-defined architecture to guide the design of the language agent platform.\n3. The proposed platform ‘OpenAgents’ is designed to be scalable and flexible, supporting various research needs and adapting to various computational environments.", "reasons_to_reject": "1. This paper doesn’t emphasize the difference between ‘OpenAgents’ and ‘ChatGPT Plus’.\n2. Most functionality of ‘OpenAgents’ is based on code or APIs, which makes ‘OpenAgents’ like an assistant rather than an intelligent agent. In more complex scenarios, for example, in OS operation and Household tasks, agents need to consider more.", "questions_to_authors": "1. Compared to LangChain, what specific advantages do ‘OpenAgents’ offer? Additionally, can ‘OpenAgents’ be seamlessly integrated with LangChain?\n2. Prompting is a critical part of OpenAgents. Does OpenAgents support a diverse set of prompting strategies, or is it easy for users to add their own?", "extra_scores": {"ethics_concerns_details": "none"}}]

Author rebuttal:

Meta-review: This paper presents OpenAgents, an open-source platform to develop language agents for real-world applications. The supported applications include data agents, plugins agents and web agents.

I believe this open-source platform can be valuable for the community to develop and evaluate LLM-based agents, thus I recommend acceptance. However, the lack of evaluation results is a notable drawback of this work. The authors should revise the paper to include more details of the platform, update the discussion of related work, and add empirical results they promised to add.

[comments from the PCs] Please revise the paper following the AC recommendation, especially the empirical content promised for the next revision.
Decision: Accept