Towards large language model-based personal agents in the enterprise: Current trends and open problems
Authors: Vinod Muthusamy
Yara Rizk
Kiran Kate
Praveen Venkateswaran
Vatche Isahagian
Ashu Gulati
Parijat Dube
Venue: EMNLP 2023 Findings (2023)
Keywords: Large language models
task-oriented
chatbots
multi-modal
TL;DR: Using large language models as the brains of autonomous agents (especially in enterprise settings) still has open research challenges that must be addressed.

Abstract: There is an emerging trend to use large language models (LLMs) to reason about complex goals and orchestrate a set of pluggable tools or APIs to accomplish a goal. This functionality could, among other use cases, be used to build personal assistants for knowledge workers. While there are impressive demos of LLMs being used as autonomous agents or for tool composition, these solutions are not ready mission-critical enterprise settings. For example, they are brittle to input changes, and can produce inconsistent results for the same inputs. These use cases have many open problems in an exciting area of NLP research, such as trust and explainability, consistency and reproducibility, adherence to guardrails and policies, best practices for composable tool design, and the need for new metrics and benchmarks. This vision paper illustrates some examples of LLM-based autonomous agents that reason and compose tools, highlights cases where they fail, surveys some of the recent efforts in this space, and lays out the research challenges to make these solutions viable for enterprises.

There an emerging trend to use large lan- guage models (LLMs) to reason about complex goals and orchestrate a set of pluggable tools APIs to accomplish a goal. This function- ality could, among other use cases, be used to build personal assistants for knowledge work- Towards large language model-based personal agents in the enterprise: Current trends and open problems
Vinod
Yara Rizk
Kiran Kate IBM Research
Praveen Venkateswaran
Muthusamy
IBM Research
IBM Research
IBM Research
Vatche Isahagian
Ashu Gulati Persistent Systems
Parijat Dube
IBM Research
IBM Research
Abstract
tcals
(2) Restomitg at rugoot Upatte boaed
(S) Thot onlertion and roverter Shakke rIXt
[S] Tool aborcvation Integees
Tack Est
LLM bosed agent
External tooks
auised task fut
(2.4) Tork pelenthation Retedos rasks
kanwindge costeut nor ausiono rovis
While there impressive demos of LLMs being used as autonomous agents or for tool composition, these solutions are not ready for mission-criticall enterprise settings. For exam- ple, they are brittle to input changes, and can get stuck in reasoning loops. These use cases raise challenging problems opening up excit- ing areas of NLP research, such as trust and explainability, consistency and reproducibility, adherence to guardrails and policies, best prac- tices for composable tool design, and the need for new metries and benchmarks. This vision paper illustrates some examples of L.MM- autonomous agents that reason and compose tools, highlights cases where they fail, surveys some of the recent efforts in this space, and lays out the research challenges make these solutions viable for enterprises.
High level goal
knowledge and contect
Figure 1: An LLM-based agent autonomously reasons about tasks and composes external tools to complete tasks, and ultimately achieve the user's goal.
perform a sequence of actions (equivalent to com- posing external tools and APIs to carry out tasks) to achieve a new state in this world (Russell, 2010). Figure 1 depicts the iterative pattern that some LLM- autonomous agents are applying (Yao et al., 2022; Nakajima, 2023; Wang et al., 2023b; Brohan al., 2023).
Philosophically, it is perhaps not surprising that language models are being used for reasoning and carrying out complex goals. For example, the Sapir-Whorf Whorf Hypothesis (Kay and Kempton, 1984) suggests that language influences thought. Bet- ter language models then should be capable of more sophisticated thinking, including reasoning and planning. Another connection is to the modes of thinking in humans: System 1 for instinctive thought processes, and System 2 for more delibera- tive thought (Kahneman, 20.1). One consider NLP tasks such as entity recognition belonging to the System bucket, whereas tasks where LLMs are used to reason and plan belong to the System 2 bucket. As language models become more power- ful, perhaps, they acquire System abilities.
1 Introduction
The emergence of ChatGPT has put large language models (LLMs) (Bommasani et al., 2021) under the microscope to determine what they can and cannot do (Hu, 2023). With LLMs outperform- ing -kate-of-the-eee art approaches for traditional ar- eas of natural language processing (NLP) research (Yang et al., 2023a), the focus shifted to other ways that LLMS can be applied. The last few months, for example, have brought on a flurry of activity around LLMs as an orchestrator or brain of au- tonomous agents tasked with completing high level goals (Park et al., 2023; Wu et al., 2023a; Vemprala al., 2023; Singh al., 2022).
In enterprise settings, autonomous agents of- ten take the form of personal assistants capable of conversing in natural language and executing actions that alter the state of the world (Kephart,
Autonomous agents perceive the world they op- crate in, reason about the events within it and their goal, decompose this goal into sub-problems, and

---

Ampe Seman Dhatin
Research GPT
hapurti alsad into everi munds
woked calivaa banio Resech desaral anl (cheries lieee
lote cnd tne boputtil
yut stem, optendt
lang
-24 eap teroir sompherree elcckheriis, Nuthertu)
-XMaay frok catu, clepped
Dwiort ket: Hello, here service, shat woild
Disft Ougpst
Suggestions
Like del
Her:
pizza.
for
Prrlettoos to SIn' Nov the loguett wordd ontor loge losing and unli gudhe brows anderigg, about e-deenssa
Owesit lot: Whet typs of Pizce wolld you 1ite ordert
herr
pepporent piaze cruat.
elace le you
in exciag han cunditne die achonde jodt shree with houp and Irmoe Juice thy ench conin read laee the yeor cheos mesher.
Oweart Bet: have find plase plase calted Plaae.
Hiding my phone address
Trp dhr crecte with eardd heriio and esta.
Zare wwweeedddddd and your ditioos, salqu Soternecea
Tosks
(a) Call restaurant to order pizza. (@ All-Grroth-Strrtupp 2023)
(b) Perform reseurch to prepare pod- cast. j@jamsshbakerd, James Baker) event. (Sharma, 2023)
(c) Invent a receipt for an upcoming
Examples of autonomous agents that use LLMs to reason about high level goal and compose tools such as search engines. phone services, and text to speech models, to achieve the goal.
Figure 2:
2021; Rizk et al., 2020; Chakraborti et al., 2022). Among other functionality, these assistants carry out task-oriented dialog, further coupling language and reasoning. In addition to performing their functions accurately, enterprise-ready technology must be explainable and reliable. They must also adhere to strict regulations and governance, espe- cially around concerns such as privacy, bias, and auditability. We notice, however, that beyond im- pressive demos, these LLM orchestrators are brittle, fail in unpredictable ways, and are not consistent which is not ideal for enterprise applications.
2.1 Examples
An agent, inspired by Auto-GPT (Richards, 2023). was able to successfully order a pizza over the phone (@ Al-Growth-Startups, 2023; @ @rogerhamil-
ton Roger James Hamilton); GPT-4 reasoned about the goal, asked clarifying questions such the type of pizza, used tools to find nearby restau- rant and look up the phone number, made phone call, and conversed with a real human (who did not realize they were conversing with a bot). Fig- ure 2a shows some of the internal reasoning by this agent. While this was an impressive demo, even the author admitted that it took few attempts to successfully place order, illustrating the brittle nature of these agents. It was also custom lution with known task, and fixed set of tools including speech-to-ters
In this vision paper, discuss the readiness of LLM-based autonomous agents for enterprise ap- plications. First, we survey both anecdotal and aca- demic works that have implemented agents. Then, we show empirical evidence of LLMs as tool composers in an enterprise travel use case. We rely on both analyses to define a set of challenges that must be addressed to make LLMs resilient enough to use an orchestrator for mission-critical enterprise cases.
text-to-speech, and the Twilio APT for phone calls. It not clear how this would extend to support arbitrary goals and dynamically pluggable tools.
Another agent based on Auto-GPT and BabyAGI (Nakajima, 2023) concepts was given the task of preparing relevant topics for pod- cast (@amesbbaker4 James Baker). Given the names and expertise of the hosts, performed a se- ries of web searches for current events related to the hosts' expertise, reasoned about whether the topics were relevant and covered the interests of all the hosts, and performed further searches to improve on the topics. Figure 2b depicts a partial output from this agent. It was restricted to web search tool and hence avoided issues with tools that may have undesirable side effects.
2 Anecdotal uses of LLM--ased agents
ChatGPT created a lot of excitement around build- ing LLM-based autonomous agents, where re- searchers and non-researchers alike were inspired to create demos, applications, and services using LLMs. In this section, review anecdotal ac- counts of LLM-based autonomous agents and tool composers. They help stretch the imagination of what possible, but do not perform experiments to stress test their solutions and understand the strengths and limitations of LLMs this role.
An agent was used to "browse the web to dis- cover the next upcoming event and invent a unique

---

and original recipe that would suit it" (Sharma, 2023). It used GPT-4 devise five-step plan, including performing web searches to find an up- coming event, generating a relevant recipe, and writing the output to a file. Figure 2c shows the recipe this agent wrote to a file.
Doesan
Beorsngeer
Fiec 20
Ma
(b
Thacet 221 AuSTIue JChn at (hegera
plaered biaee you oit
Agasdi
There are other examples of LLM-based agents performing sales prospecting (@ompemi Omar Pera), autonomously carrying out tasks in a task list (Scott, 2023), and managing social media accounts (Sharma, 2023).
Uhache (Camm ina 23 Poae (2)
Tou iilheeeeed 201
Figure 3: Spectrum of actions per domain
understanding of the risks of actions taken with a tool and may autonomously perform actions such as installing certificates as a super user (Molony, 2023). These risks are somewhat mitigated by hav- ing agents prompt the user every step before acting, but this diminishes the value of these agents autonomously carrying out tedious work for users. There are also studies that present the limitations of the reasoning abilities of LLMs (Borji, 2023). Since reasoning is such a critical part of how agents use LLMs, these limitations will have direct effect on the performance of these agents.
2.2 Limitations
A common observation the LLM-based au- tonomous agent examples is that they tend to be one-off demos. Applying more scrutiny reveals issues around robustness, practicality, and trustwor- thiness. For example, an agent can get into loops where it continuously tries to decompose a task into smaller ones or generates new tasks that are duplicates of ones already in the task list or have been completed. This leads to agents performing work but not making progress towards the higher level goal (Xiao, 2023; Molony, 2023).
These anecdotal observations of current limita- tions of LLMM ---Meddaed agents present roadblocks to adoption for enterprise use cases, where risk, cost, and robustness are critical issues. We formulate a set of research challenges in Section to address these and other roadblocks.
These agents can be brittle (e.g., sensitive to small changes in how the goals are expressed) and sometimes even producing different outputs for the same input (Brandt, 2023). The authors have also observed such brittleness when ChatGPT Plu- gins (OpenAI, 2023) was used to compose tools, with inconsistent behavior for the same or similar inputs (Patil et al., 2023). The agents are also sen- sitive to the LLM that is used, making it difficult to predict their behavior with different models or even newer versions of the same models. This lack of robustness is challenge in enterprise applications. Another observation is that these can be slow and expensive. They can make hundreds of calls to LLMs as they reason about the goals and make observations on the tool outputs. For example, even relatively simple goals can take 50 calls to GPT-4 costing about $14 (Xiao, 2023). This is only expensive monetarily, but these iterative LLM calls can take several minutes or longer to accomplish a goal making them impractical for interactive apps.
A review of LLM-based agents
In the previous section, we provided evidence of the feasibility of LLM-based agents from non- academic sources such blogs, Twitter posts, GitHub repositories, and demos. While these gen- erated excitement around LLMss outside of the search community, this evidence does not allow us assess the true capabilities and limitations of LLMs. Next, we survey published literature that more systematically analyzes LLMs domains- agnostic and domain-specific settings. These pa- pers consider a spectrum of actions from no side- effects to irreversible as illustrated in Figure 3.
Domain-agnostie autonomous agents Recent papers extend the use of LLMs beyond simple tasks such as summarization or entity extraction handle complex reasoning and question answering. This includes augmenting L.LMs with the ability to query the internet (Nakano et al., 2021; Sem- nani et al., 2023) or to use of different types of
Allowing agents to decide how to compose tools that have side-effects can be dangerous. Some use cases only compose tools such as web search with little harmful effects, while others utilize tools to send emails, write to a file system, or manipulate calendars. The agents and LLMs have little or no

---

tools (e.g. calculators) or access external informa- tion (e.g. Google, or Wikipedia search) (Schick et al., 2023; Chen et al., 2023c). When (Sun et al., 2023) leveraged LLMs to decompose com- plex tasks into sequence of actions that have no side effects, TaskMatrix.AI and ReACT present a vision that utilizess GPT-4 to connect any APIs to complete tasks (Liang et al., 2023; Yao et al., 2022; Yang al., 2023b) (Luu al., 2023) focused plug and play when leveraging LLMs for tool composi- tion. Finally, (Li et al., 2023) created benchmark with 500+ APIs to evaluate the effectiveness of LLMs as tool composers.
if intervention was necessary. In the area of online games, Voyager (Wang al., 2023a) uses GPT-4 to build an agent to continuously explore Minecraft world with the goal of making novel discoveries. It uses a catalog of skills or tools, and can also learn new skills.
Use case: chatbot for enterprise travel
The integration of APIs with LLMs (Liang et al., 2023) has raised the question of whether LLMs can act as task-oriented chatbots, particularly on complex tasks that require the execution of a se- quence of APIs. Automating tasks in enterprise settings is particularly difficult since each company has custom processes in place and often little data to train fine-tune LLM to their peculiarities.
As depicted in Figure agents adds long term memory and task prioritiza- tion capabilities. An LLM decomposes high level goals to smaller tasks, a vector store is typically used for long term memory to recall outputs from long task lists, and an LLM is used to iteratively reprioritize existing tasks and create new tasks based on the progress towards the goal (Richards, 2023; Nakajima, 2023; Wang et al., 2023b).
evolution of these
4.1 Travel example
Let's consider travel example where LLM users may want book trip which includes making flight, hotel, and car rental urrangements. For en- terprises, additional steps are necessary such as es- timating travel costs and submitting a pre-approval with adequate justification (e.g., client event or con- ference travel), as shown in Table 1. Simply classi- fying these utterances into intent classes may not be enough some of the information to distinguish between these clusters is not found in the natural language utterances provided by the users.
Robotic agents Due to their emergent behav- iors (Bommasani et al., 2021), LLMs have been used to make goal-driven decisions. These ap- proaches mainly rely on LLMs accomplish tasks in the physical or virtual world using Internet-of- Things devices and robots (Huang et al., 2022; Singh et al., 2022; Vemprala et al., 2023). (Huang et al., 2022) evaluate whether L.MSs contain infor- mation necessary to accomplish goals without ad- ditional training or grounding. Followup efforts on grounding primitive actions include Text2Motion (Lin al., 2023a), AAto (Chen et al., 2023b) and SayCan (Brohan al., 2023) to guide LLM- based task and motion planning. TidyBot (Wu et al., 2023a) uses LLMS to infer generalized user pref- erences that are applicable to future interactions. LLM-based interactive agents mimicked human behavior in (Park et al., 2023).
4.2 Methodology
Historically, process knowledge was embedded into task-oriented dialog systems by system de- velopers. For example, a chatbot may ask the user (based on their underlying dialog tree) if their travel is a client event or a conference in cases where the process requires this information to determine the next step. By placing LLMs at the center of such dialog systems, how can we infuse such process knowledge into LLMs' decision making? In this case study, we focus on doing so through prompt- ing but fine-tuning and possible pre-training mod- els with appropriate data could also be considered.
Other domains of performing tasks such as finding restaurants or reserving hotels have been addressed in the NLP community (Gupta et al., 2022; Ren et al., 2018). For the banking sector, (Wu et al., 2023b) trained finance specific LLM. In the medical domain, (Au Yeung et al., 2023) tested ChatGPT for clin- (Jo et al., 2023) created a public health intervention system that can chat with patients and notify health care professionals
Dialog systems with the goals
The knowledge to make good decisions may come from multiple sources (Figure 4). For ex- ample, business process descriptions or standard operating procedures may include the necessary steps for business travel (e.g., submit a pre-approval first). Past process traces showing the execution of such processes can reveal information about best practices (e.g., most travel requests to conferences
ical

---

RESERVE MY TRIP TO PARIS FOR THE User Utterance
Cluster
Expected Sequence of APIs Slot filling -> Car rental
BOOK MY TRIP FROM BOSTON TO NYC FOR MAY
Business, Day trip
Business, Multi-day
Travel Estimation -> Travel Pre-approval
ACME WORKSHOP FROM MAR.
BOOK MY TRIP FROM BOS TO SFO FOR MAY 5-10 I'M ATTENDING EMNLP 2023
Personal
Slot filling -> Airline booking
Conference, Multi-day
Publication database -> Travel estimate API -> Travel pre-approval
Table 1: Travel Example
prox. 700 from Google employee travel (Emp. 2022) and 1200 from Frames (Fra, 2018). We man- ually labelled twelve phrases and then clustered the phrases to weakly label them.
Ulterance &
Canva content
System
nueinees process descriptions
Evaluation metrics To compare the various prompts, adopted the BLEU metric (Papineni et al., 2002) and calculated precision, recall and F1- score as follows. Precision is the total number of common APIs between the ground truth sequence and the model's output divided by the total number of APIs in the model's 's output sequence. Recall is the total number of common APIs between the ground truth sequence and the model's output di- vided by the total number of APIs in the ground truth sequence. We calculated them the and sequence levels. Set level metrics compare the set of APIs in the ground truth and the output sequence, whereas sequence level metrics are based on the longest common subsequence between the ground truth and output sequence.
APls
Standard spanaling procedures, pollcy documentt
Figure Sources of information feed into an LLM
include listing papers accepted at the conference).
4.3 Experimental setup
Model and prompt To circumvent the need for a multi-modal foundation model, we represent all the sources of information natural language, as in Figure 5. We experiment with a few variations of the prompt by paraphrasing some of its sections. Multiple versions of the prompts are fed into an instruction-tuned transformer model (FLAN-T5) (Longpre al., 2023).
4.4 Experimental results
Tables 2-5 summarize our results. A baseline prompt (vl) does not include any process knowl- edge or past process traces (i.c., only the blue sec- tions of Figure 5), whereas the remaining versions Business procees desiciption
FM's task
moald
(v2-v5) include all sections in Figure 5 but para- phrased to express the same information in differ- ent ways (see Appendix for an example). We can from the tables that including process informa- tion, not just users' natural language phrases and conversation context, outperformed the baseline prompt (in both set level and sequence level evalu- ations). Furthermore, the way that this information is included in the prompt matters (i.e., v4 and v5 perform better than other versions).
buand cot
Post
Inpul-Ouqpur examples
lable APts (outout)
Usar's inout
Figure 5: A prompt template for the travel use case
To evaluate LLMs' consistency, we prompted the model 30 times with fixed input using a tem- perature of 0.5. Prompt v5 was used as the template and we compared the output of the model to the output when the temperature is set to O. We did
Dataset We created natural language utterances to sequence of API pairs (similar to Table I) from two publicly available travel datasets (ap-

---

this for the 12 manually labeled phrases from the dataset, as shown in Table Ideally, we want the metrics be close to to have a consistent LLM. In this case, we see that the results are not very repeatable with an average BLEU score 0.76.
Prompt
Precision
Recall
Fi-score
BLEU
v1
0.32
0.39
0.35
0.38
V2
0.38
0.48
0.43
0.19
V3
0.61
0.76
0.68
0,55
v4 0.61
0.78
0.68
0.57
v5
0.65
0.79
0.71
0.62
5 Research challenges
Comparing prompts Google Employee Travel Dataset (averaged over instances). Precision, recall, are calculated at level.
Table 2:
We now draw on the limitations observed from anecdotal applications of LLM-based agents (Sec- tion 2), experimental evaluation of an enterprise travel case study (Section 4), and our own experi- ences developing enterprise application platforms' to define set of research challenges that need be solved before LLM-based agents can be applied to mission-critiees enterprise use cases.
Prompt
Precision
Recall
Fl-score
BLEU
vI
0.54
0.55
0.54
0.42
v2 0.57
0.55
0.56
0.40
v3 0,55
0.61
0.58
0.40
v4 0.68
0.60
0.64
0.56
v5 0.63
0.55
0.59
0.52
Table 3: Comparing prompts on Frames Dataset (av- eraged over instances). Precision, recall, and F-1 are calculated at set level.
5.1 Metries and benchmarks
The evaluation of personal agents depends on the problem formulation. As task-oriented dialog, it is commonly evaluated for accuracy of intent de- tection and accuracy of slot filling, i.c., how well the values of parameters detected from the natu- ral language utterance match the ground truth API call. AST (abstract syntax tree) sub-tree match- ing is another metric to mcasure the correctness API calls. The natural language utterance can be single-intent or multi-intent. This is the simplest form of personal agents and has existing bench- marks (Hemphill et al., 1990; Coucke et al., 2018; Qin al., 2020; Patil al., 2023). Most of these benchmarks contain a limited number (dozens) of APIs or tools with limited number of slots per API and some them are synthetically generated.
Prompt
Precision
Recall
FI-score
0.28
0.34
0.31
0.33
0.38
0.56
0.69
0.62
0.55
0.70
0.62
v5
0.61
0.73
0.69
Table 4: Comparing prompts Google Employee Travel Dataset (averaged over instances). Precision, recall, are calculated at sequence level.
Prompt
Precision
Recall
FI-score
vl
0.52
0.53
0.53
v2
0.54
0.52
0.53
0.53
0.59
0.56
v4
0.65
0.57
0,61
0.61
0.53
0.57
Table 5: Comparing prompts on Frames Dataset (av- eraged over instances). Precision, recall, and F-1 calculated at sequence level.
For a multi-turn dialog, it is also important to evaluate the number of turns required to achieve the goal. If the task requires executing sequence of dependent tools, then the order of the tools be- comes important. There are few benchmarks available for this setting (Rastogi et al., 2020; Li et al., 2023; Budzianowski al., 2018) with only one of them being a fully human annotated cor- pus (Budzianowski et al., 2018).
Phrase
BLEU
Precision
Recall
F1-
0.56
0.46
0.63
0.92
0.91
0.93
0.95
0.51
0.68
0.70
0.75
0.74
0.74
0,90
0.78
0.84
0.81
0.84
0.86
Furthermore, most of these datasets are simpli- fied and do not represent enterprise scenarios. For example, (Li et al., 2023) has well-documented handcrafted APIs with handful of parameters per API. Real tools the other hand have many con- figuration options and rarely well documented.
0.97
0.97
1.00
0.99
0.68
0.72
0.72
0.72
10
0.77
0.96
0.82
0.89
11
0.71
0.82
0.93
0.87
12
0.91
0.85
0.97
0.91
Average
+
0.76 +
0.80+
0.89+
0.83+
Standard Deviation
0.15
0.17
0.10
0.11
For a more complex setting of autonomous agents interacting with each other, the evaluation
Table Repeatability on Manually Labelled Phrases
Citatioon omitted for double-blind review.

---

is challenging, Human evaluation is critical part for building such agents (Park al., 2023). If the tasks change the state of the world, then comparing the state after each action is a potential metric.
efforts should improve the predictability and de- buggability of these systems. As touched on in Section 2, current agents are brittle and hard to test.
5.4 Pluggability
The development of representative benchmarks has accelerated the field of NLP. We believe this is an important area to focus on for enterprise per- sonal agents, especially ones that factor in business process information and other modalities to fully capture the domain knowledge.
An enterprise user needs to perform many tasks on a typical day. With an ever-evolving landscape of software tools, this set is large and anything but constant. Hence, personal assistants should allow for easy plugging in of new tools. Current prompt- based approaches such ChatGPT function call- ing (Eleti et al., 2023) are limited by the context length of the model to add more APIs. Approaches such as (Schick et al., 2023) can potentially scale to a large number of APIs but new APIs cannot be added dynamically without re-training the model. Ensuring pluggability will also make it difficult to purely rely on fine-tuning models since the rate of adding and removing plugins and data scarcity could make fine-tuning prohibitive.
5.2
Data for fine-tuning
Most of the LLM training efforts on tasks in the natural language domain have large amounts and variety of relevant and (un)labeled training data that has been collected and open-sourced by the larger research community. In most enterprise set- tings, there is either a lack of sufficient labeled open-source real-world data to fine-tune model due to their inherent proprietary nature, or lack of necessary infrastructure or training expertise that enables the fine tuning of such LLMs. One way to overcome this is by further advancements in prompt engineering (Wei et al., 2022) but also to provide a framework that efficiently guides devel- opers to structure their prompts, and effectively prompt the model to generate more accurate results and enhance the overall performance without the need fine-tune the models.
5.5 Reproducibility and reliability
The sensitive and business critical nature of enter- prise tools and systems requires consistency and reliability from their tools and compositions. How- ever, LLMs are capable of hallucinating predictions (Jiang et al., 2020), and could generate inconsis- tent and invalid tools and compositions, thereby having harmful consequences. Additionally, these models are brittle, where small variations in the prompt could result in different predictions, re- sulting in users' having very different experiences with the same model. Approaches like instruction- tuning (Ouyang et al., 2022) and chain-of saain-off-hure of-thouught prompting (Wei et al., 2022) alleviate some of these problems by breaking down prompts into stages to improve prediction consistency. However, these approaches do not provide any guarantees on the reliability and reproducibility of the predictions.
5.3 Composing tools
LLMs are not typically explicitly trained with the goal of composing tools and acting as the reason- ing engine of an autonmous agent. Likewise, tools such as web search APIs, databases, and file sys- tem primitives aren't 't designed to called by a dy- namic agent and I.LM. We see some evidence of this impedance mismatch with ChatGPT Plugins requiring plugin developers to author manifest files for their existing tools, as well as extensive guid- ance on how to describe the tools, so they can be composed by ChatGPT (OpenAI, 2023).
An inereasingly popular approach to enable re- liable LLM predictions is constrained decoding (Hokamp and Liu, 2017), which enforces the model to only consider certain outputs for predictions by modifying their log-probabilities based on the given constraints. This would enable enterprise systems to prevent hallucinated outputs. Addition- ally, enterprises could also represent their policies as constraints the model, to enforce compliance. However, as the number of constraints increases and in multi-modal settings, representing these con- straints and policies in a format that the model can consume presents a significant challenge.
More work is needed on improving the capabili- ties of LLMS to compose tools that take structured inputs and outputs, not just natural language. Fur- thermore, agents need an understanding of the risks and side-effects of performing actions with tools; tool to search the web should be treated differ- ently from one that performs financial transactions. This might require better interfaces or program- ming models to make these tools more consumable by agents and LLMs. A related aspect is these

---

5.6 Confidence and failing gracefully
5.9 Generalizability and scalability
Data sources used to train LLMs often restricted to positive knowledge and do not provide sufficient negative examples, including appropriate responses in case of failure or lack of knowledge, resulting in LLMs confidently producing incorrect answers
Most popular LLMs have a large number of pa- rameters, making them prohibitively expensive fine-tune for different tasks. Given the varied ob- jectives of enterprise use-cases, ensuring the gener- alizability of models to new tasks and domains, and unforeseen tools is essential. Approaches like few- shot prompting and prompt tuning have become more popular (Lester al., 2021; Gu et al., 2022), leveraging specific and well-crafted examples improve model generalizability. The growing size also inherently increases inference time and infras- tructure costs, and enterprise services that depend on these models often have latency constraints. Ad- dressing the scalability and costs of LLM-driven enterprise applications is another significant chal- lenge that requires attention.
(Chen et al., 2023a; Jiang et al., 2020). In enter- prise settings, it is critical for LLM based systems to be able to recognize their limitations and fail gracefully. This challenging to achieve, since the notion of uncertainty is dependent on the ap- plication domain, the knowledge sources used to train the model, and an evaluation the model's response. While there has been some initial efforts to develop methods to measure the confidence or uncertainty of black-box models (Lin et al., 2023b; Kuhn et al., 2023), they do not translate to many enterprise use-cases such as those described in this paper, necessitating specialized approaches.
5.10 General purpose or specialized models
5.7 Error handling and failure semantics
In Figure 1, single LLM is used for multiple tasks including task reasoning and composing tools. There is no reason to presuppose that single model well-suited every step in that flow. Thus, thought should be given to whether specialized models for each task perform better and how to chain them to complete the overall objective.
As indicated previously in Section 2.2, LLM agents get into loops trying to decompose task or perform actions that can have lasting side effects on their environment. Handling such situations will be necessary to provide a reliable and consistent output in an enterprise setting. Furthermore, as LLM's access to tools increases, it becomes critical for them to overcome errors such as page not found, non-responsive server, unauthorized access, that get them stuck and hinder their progress.
6 Final thoughts and next steps
Enterprises are on a continuous quest to opti- mize repetitive and tedious work. Advancements in LLMs have broadened imaginations on what knowledge work can be automated, and LLM- based autonomous agents can be the vehicle to deliver incredible productivity gains.
5.8 Multi-modality
Many enterprise applications have decades worth of information saved in various modalities such as spreadsheets, scanned documents in image format, unstructured emails, business process diagrams and others. One way address this diversity bring everything into the language space (per section 4). Prior work has shown that better performance can be achieved by learning a unified representation across modalities (Bao et al., 2022) opposed to treating these modalities independently (Tia et al., 2021) or converting into one modality where infor- mation will be inevitably lost (Xiang al., 2023), Creating a new class of foundation models that include L.MMs along with models for enterprise specific modalities like business processes is worth investigating (Rizk et al., 2022).
Attempts in the community build agents us- ing even state-of-the-art LLMs, as well our case study using LLMs for an enterprise travel use case, reveal how inadequate these solutions are for mission-critical enterprise use cases. We outline several research challenges that the NLP and other research communities can investigate.
While we describe individual challenges, such as robustness, multi-modality, and tool composi- tion, the community must take a holistic view of the problem, addressing not just LLM advances. but the entire software lifecycle including tool au- thoring, and debugging. Defining clear use cases and benchmarks will be important steps.

---

Limitations
@ Al- (Growth-Startups 2023. My ChatGPT GPT-4 Agent ordered me pizza BY PHONE!
On the survey and challenges contributions of this work: our coverage of the literature is incomplete due to the extraordinary fast pace and sheer volume of work posted on arxiv, blogs, etc. Furthermore, due to this rapid pace, we do cite a large number non- con--per-rrvieweee isting challenges and how to resolve them is also colored by our experience in industry and may not include some more theoretical challenges that the community should also solve in conjunction with the practical challenges.
Joshua Au Yeung. Zeljko Kraljevic, Akish Luintel, Al- fred Balston, Esther Idowu, Richard J Dobson, and James T Teo. 2023. Ai chatbots not yet ready for clinical use. Frontiers in Digital Health, 5:60.
work. Our discussion ex-
Hangbo Bao, Wenhui Wang. Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subho- jit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified laiguae pre-training with mixture-of- modality-expert Advances in Neural Information Processing Systems, 35:32897-32912.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosse- lut, Emma Brunskill, al. 2021. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07258.
On the experimental contributions of this work: our datasets are small in size compared to what may be commonly used, their labeling is noisy due to the pseudo-labeling approach we adopted and may not be very realistic due to the additional pro- cessing performed to get them to the format required. Furthermore, the metrics we calculated do not measure all characteristies that we may want to evaluate. Also, since we only used one ground truth to calculate the metrics, this may result less accurate values (e.g., BLEU becomes more accu- rate as more references are used). Finally, the ex- perimental analysis is not comprehensive. missing some ablation studies and other experiments that could help answer additional questions the per- formance on LLMs on task-oriented dialog tasks.
Ali Borji. 2023. A categorical archive of chatgpt fail- ures.
Craig Brandt. 2023. Ansible and ChatGPT: Putting it to the test.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. 2023. Do as i can, not as say: Grounding language in robotic affordances. In Conference on Robot Learn- ing, pages 287-318. PMLR.
Pawel Budzianowski, Tsung- Hsiee Wen, Bo-Hsiang Tseng, Inigo Casanueva, Ultes Stefan, Ramadan Os- man, and Milica Gašić, 2018. Multiwoz a large- scale multi multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing (EMNLP).
Ethics statement
Our work discusses how to enable LLMs to per- form actions (or compose tools) whose purpose is change the state of the real world. Given the emergent property of LLMs that may result in un- predictable behavior, allowing these LLMs to alter the state of the world by performing these actions could lead to irrevocable changes that could have negative impact (e.g., autonomous agents capable of stealing money from people's 's banks).
Tathagata Chakraborti, Yara Rizk, Vatche Isahagian, Burak Aksar, and Francesco Fuggitti. 2022. From natural language to workflows: Towards emergent in- telligence in robotic process automation. In Business Process Management: Blockchain, Robotic Process Automation, and Central and Eastern Europe Forum: BPM 2022 Blockchain, RPA, and CEE Forum, Mün- ster, Germany, September 11-16, 2022, Proceedings, pages 123-137. Springer.
References
2018.
Frames microsoft. www/enous///ecccocoooeeeww wwwp///eaach/proeeeeeeeee
Dataset.
https://www
Jiangjie Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023a. Say what you mean! large language models speak too positively about negative commonsense knowledge. arXiv preprint arXiv:2305.05976.
oadd/ Accessed on June
23, 2023.
2022.
About
open
data
city of
greater sudbury. greatersudbury. ca/ ca/documents/ /stpss//oopeddar 9dafb3c869754aca88 3daaa Accessed on June 21, 2023.
Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. 2023b. Autotamp: Autoregres- sive task and motion planning with 1lms as translators and checkers. arXiv preprint arX

---

Zhipeng Chen, Kun Zhou, Beichen Zhang, Zheng Gong, Wayne Xin Zhao, and JinRong Wen. 2023c. Chatcot: Tool-augmented chain-of-thought reason- ing on cchat-based large language models. arXiv preprint arXiv:2305. 1432:33300..33333.
Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun Hun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calta- girone, Thibaut Lavril, et al. 2018. Snips voice plat- form: an embedded spoken language understanding system for rrivate-by-design voice interfaces. arXiv preprint arXiv: /0190.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Grabam Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-436
Eunkyung Jo, Daniel A Epstein, Hyunhoon Jung, and Young-Ho Ho Kim. 2023. Understanding the benefits and challenges of deploying conversational lever- aging large language models for public health inter- vention. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1- 16.
Layla El Asni, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin D. Harris, Emery Fine, Rahul Mehro- tra, and Kaheer Suleman. 2017. Frames: A corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 207-219.
Atty Eleti, Jeff Harris, and Logan Kilpatrick. 2023. Function calling and other api updates. Accessed on June 23, 2023.
Daniel Kahneman. 2011. Thinking, fast and slow. macmillan.
Paul Kay and Willett Kempton. 1984. What is the sapir-whorf hypothesis? American anthropologist, 86(1):65-79.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2022. Ppt: Pre-trained prompt tuning for few-shot learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1410-8423,
Jeffrey O Kephart. 2021. Multi-modal agents for busi- ness intelligence. In Proceedings of the 20th Interna- tional Conference Autonomous Agents and Multi- Agent Systems, pages 17-22.
Raghav Gupta, Harrison Lee, Jeffrey Zhao, Yuan Cao, Abhinav Rastogi, and Yonghui Wu. 2022. Show, don't tell: Demonstrations outperform descriptions for schema-guides task-oriented dialogue. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4541-4549.
Lorenz Kuhn, Yarin Gal, and Schastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for un- certainty estimation in natural language generation. arXiv preprint 2xXiv2332.006644
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 3045-3059.
Charles Hemphill, John J. Godfrey, and George Doddington. 1990. The ATIS spoken language sys- tems pilot corpus. In Speech and Natural Language: Proceedings of Workshop Held Hidden Valley, Pennsylvania, June 24-27,1990.
Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api- bank: A benchmark for tool-augmented 1lms. arXiv e-prints, pages arXiv-2304.
Chris Hokamp and Qun Liu. 2017. Lexically con- strained decoding for sequence generation using grid beam search. arXiv preprint arXiv: 7704.07388
Krystal Hu. 2023.
Chatgpt sets record fastest growing user base: Analyst note.
Yaobo Liang, Chenfei Wu, Ting Song. Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji Shaoguang Mao, et al. 2023. Taskmatrix. ai: Com- pleting tasks by connecting foundation models with millions of apis. arXiv preprint arXiv: 164444
https://www.reuters.comm hww.:///ww..ooocc/oooo/oo//cceee
haa.rroorrdffsttttttrooooneeeeee haaa---s----a---aaaattttoooo 2023-02-01. Accessed on June 21, 2023.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, pages 91 18-9147. PMLR.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. 2023a. Text2motion: From natural language instructions to feasible plans. arXiv preprint 2rX
@ jamesbbaker4 (James Baker).
https:
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023b. Generating with confidence: Uncertainty quantifi- cation for black- box large language models. arXiv preprint arXiv:2305.19 19187.
/ //twitteerr.. wtt.////mmmmssaeeereeeeteeeeeeee
1 6448898646662722755.
Accessed on June 23,
2023.

---

Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung. Yi Tay, Denny Zhou, Quoc Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.
the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 0696-
Lillang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. To- wards universal dialogue state tracking. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2780-2788
Pan Lu, Baolin Peng. Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu. Song-Chun Zhu, and Jian- feng Gao. 2023. Chameleon: Plug-an and-play compo- sitional reasoning with large language models. arXiv preprint 230402204
Toran Bruce Richards. 2023. Auto-GPT: An Au- tonomous GPT-4 Experiment. https://gi h/pp:://iitub... thub. Sisiiiiiiiiatttrrattttatss/a/e
Rick Molony, 2023, Auto-GPT Prroiis versus Real- ity.
Yara Rizk, Vatche Isahagian, Scott Boag, Yasaman Khazaeni, Merve Unuvar, Vinod Muthusamy, and Rania Khalaf. 2020. A conversational digital assis- tant for intelligent process automation. In Business Process Management: Blockchain and Robotic Pro- cess Automation Forum: BPM 2020 Blockchain and RPA Forum, Seville, Spain, September 13-18, 2020, Proceedings 18, pages 150 Springer.
Yohei Nakajima, 2023. Task-driven autonomous agent utilizing gpt-4, pinecone, and langehain for diverse applications.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. 2021. Wehgpt: Browser-asiised question- answering with human feedback. arXiv preprint arXiv:21
Yara Rizk, Praveen Venkateswaran, Vatche Isahagian, and Vinod Muthusamy. 2022. A case for business process-specific foundation models. arXiv preprint arXiv:2210.14739.
@ompemi (Omar Pera). 2023. An ai agent that au- tonomously does sales prospecting on own with
@ rogerhamilton (Roger James Hamilton). 2023. This guy built a gpt4 ai that followed a prompt to find and call a pizza co and order pizza without them realizing it
gpt-4, powered by babyagi from @yoheinakajima run on @ replit. imagine once you integrate it with @ @langchainai tools like @hubspot or apollo.
bot. #ai is doubling its power every months. seary to think where we will be 6 months from now.. #gpt4 #chatgpt full vidco: bttps://youtu.b/88ppp be/jggf/2qqqq
OpenAI. 2023. ChatGPT plugins.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. Advances in Neural Information Processing Systems,
Stuart J Russell. 2010. Artificial intelligence a modern approach. Pearson Education, Inc.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 1-318.
tools.
Garrett Scott. 2023, Do Anything Machine.
Sina J Semnani, Violet Z Yao, Heidi C Zhang, and Mon- ica S Lam. 2023. Wikichat: A few-shot Ilm-based chatbot grounded with wikipedia. arXiv preprint 2.X.:23055.1292
Joon Sung Park, Joseph C. O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive sim- ulacra of human behavior.
Nitin Sharma. 2023. AutoGPT: The New Kid on the Al Block That's Changing Everything!
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305115334
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2022. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302
Libo Qin, Xiao Xu, Wanxiang Che, and Ting Liu. 2020. Agif: An adaptive graph-interactive framework for joint multiple intent detection and slot filling, arXiv preprint arXiv:2004.1008)
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of
Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, and Mohit lyyer. 2023. Pearl: Prompting large language models to plan and execute actions over long documents. arXiv preprint arXiv: 14564.

---

Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. 2023. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton, Syst. Robot. Res, 2:20.
Promet Rulos
Pollon below instructions:
Signting te with corporate 1D tnplite that it is bustnees trip-
Bustieess
$250 should approved.
30 teployass camott
similar acccenotation services
11 For conference triss, amployens should provice the paper titin,
Guanzhi Wang, Yuqi Xie, Yunfan Jiang. Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models.
For eltent trips, employose thould provtde the clfart
Prompt v4 Rules
<tzavel, type> businssa, follow belon zulest
Business traveis costing eore than $258 should be approved using Trave
1f it's conference trip, Paper Oatabase API teployee Directory
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023b. Plan-and-solve prompting: Improving zero- shot chain-of-thoughe reasoning by large language models.
client teips, use Ollent Databose IPI to verify the client.
12
stravel_types is pecsoral, follos the balon ruless
not Trave] Extimste Comsany Pre-ssproval Request
Figure 6: Rephrasing of rules in prompt v2 and v4
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1903..
B Datasets
Employee Travel/Golden dataset We have used freely available Employee Travel 2022 dataset from Google Dataset Search. This dataset contains infor- mation about the employee travel expenses for the year 2022. Details are provided on the employee (name, title, department), the travel (dates, location, purpose) and the cost (expenses, recoveries). Count of rows = 178.
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lep- ert, Andy Zeng, Shuran Song, Jeannette Bohg, Szy- mon Rusinkiewicz, and Thomas Funkhouser. 2023a. Tidybot: Personalized robot assistance with large language models. arXiv preprinl arXiv:2305.055 05658.
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam- badur, David Rosenberg, and Gideon Mann. 2023b. BloombergGPT: A large language model for finance. arXiv preprint arXiv:2303 17564.
For the purpose of generating NL utterances, we used four columns (termed entities) dates (start and end), location and purpose of travel. Using these entities, divide our NL utterance genera- tion into four categories:
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang. Zichao Yang, and Zhiting Hu. 2023. Lan- guage models meet world models: Embodied expe- riences enhance language models. arXiv preprint arXiv:2305. 10626.
random Bartoomssmmmm___ee
4: with all the four entities present (start date, end date, purpose of travel and location of travel)
Han Xiao, 2023. Auto-GPT Unmasked: The Hype and Hard Truths of Its Production Pitfalls.
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng. Haoming Jiang, Bing Yin, and Xia Hu. 2023a. Harnessing the power of Ilms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712
mondom_sample with any three random entities present
sandom_samples with any two random enti- ties present
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023b. Mm-rcact: Prompting chatgpt for multimodal rea- soning and action. arXiv preprint arXiv:2303. 11381.
random Lactloom_smmm_______ 1: with any one random en- tity present
Using OpenAl's text-davinci-003 model, we engi- neered a prompt after preprocessing the dataset, generate five distinct user utterances with the help of the entitics, in first person. The same process was repeated for all the defined categories above. This approach generated an overall count of 3560 utterances from the initial set of 178 rows.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting language models. In NewrIPS Foundation Models for Decision Making Workshop.
Appendix
A Prompts
Past Travel Requests We created a template, us- ing seven entities, craft travel requests made by employees in the past. These past travel requests are a part of our prompt.
Check supplemental material for complete prompts. Figure 6 shows an example of the difference in how the rules are phrased between v2 and v4.

---

The travel request by <name> (<ti- de>) was approved for <amount>; his trip to <desti- nation> from <date> to <date> was for the purpose of <purpose>
Template:
[Information Gathering API, Book Hotel API, Book Flight API]
Then, set level precision is:
beclooutput_sequeneee  setereeeeeeeeeeeeeellrooodddddd
Example: "The travel request by Adair, Brendan (Director of Transit) was approved for $1230.3; his trip to Toronto, ON from 2022-07-17 to 2022-07- 20 was for the purpose of Ontario Transportation Expo- OTE'
|output__squencc]
and level recall is:
[scl[output_sequence)) Secrreeneddssssggdouuuudt.o.oe
[ground_ truth
Frames Dataset Frames is a dialogues dataset which was collected in a Wizard-of-OP fashion by (El Asri et al., 2017). Two humans talked to each other via chat interface. One was playing the role of the user and the other one was playing the role of the conversational agent called wizard. The wizards had access database of 250+ packages, each composed of hotel and round-trip flights.
For sequence level metrics, we will look at the longest common subsequence (LCS) between the sequence of APIs in output_sequence and ground_ trunnd
LCS(output_ LCSSoutput_seeuencee _eeuuunceeeeeeeee ground_ sordddrruth-a [Infor- mation Gathering API, Book Hotel API).
Then, the sequence level precision is:
Frames is composed of 1369 human-human di- alogues with an average of 15 turns per dialogue. This corpus contains goal-oriented dialogues be- tween users who were given some constraints to book trip and assistants who search database to find appropriate trips.
(CCSooutput_sequencee ground 2 output_sequence
and the sequence level recall is:
CccS(output_sequeneee grobndd_trggoouuuuutuuteu 2
ground_truth
5
The Frames dialogues are in ISON format. Each dialogue has five main fields: user_id, wizard id, id, RerruurveRRating and turns. For our purpose. we extracted the first occurrence from every user- wizard dialogue or id. On postprocessing, we were able to create dataset of about 1200 user utter- ances. The postprocessing was done using HDB- SCAN to group similar sentences into clusters and then discarding those clusters which contained only greetings (such Hi, Hello there) without any re- quest for travel booking.
Observe that another common subsequence is (In- formation Gathering API, Book Flight API] which also length 2. If we use this instead, we will get the same sequence level precision and recall.
Metries
We will calculate our set level and sequence level metrics for an example utterance. Consider the following utterance with the corresponding ground truth and model output sequence.
utterance: I'm planning ski trip to Banff, AB in February. Can you help me plan it out? IIffrrmaiinn Gathering API, Book Hotel API, Book Flight API, Book Train API]
gound_truth: truth: Information Gathering API, Book Flight API, Book Hotel API, Rent Car API) For level metrics we will look at the intersec- tion of the set of APIs in output_sequence and ground_truth. We have: set (etupuutt_sqquencc)) n set(ground__rrute


Reviews:
[{"review_id": "5CMVEDUZeq", "reviewer": "Reviewer_AqLF", "summary": "", "questions": "In your classification of agents you have divided it into domain-agnostic autonomous agents, robotic agents, and others. What other kind of classification could be done to be more problem oriented than solution oriented?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree that this is a preliminary analysis of the space of LLM-based autonomuous agents. Please keep in mind that much of the work in this space is only a few months old, and we submitted this vision paper “with the goal of stimulating discussion around Large Language Models and the Future of NLP” (as stated in the CFP for the LLM theme track).\n\nIn terms of “current applications” in this space, we tried to survey both anecdotal accounts (Section 2) and the literature (Section 3). If the reviewer has important missing applications in mind, we would be happy to include them.\n\nAbout the “taxonomy of the problems”, we have dedicated Section 5 to research challenges, and have outlined 10 sets of problems that could each be future research topics. This is not explicitly  a “taxonomy” in terms of a structured hierarchy of problems, but we feel it still serves to classify the problem areas and provide some guidance to the research community. We will consider the reviewer’s advice for a more comprehensive problem taxonomy in a followup paper as we better understand the developments in this space.\n\nRegarding the reviewer’s question on classification schemes, note that we structured Section 3 into “domain-agnostic autonomous agents, robotic agents, and others” mainly because that reflects the balance of literature in the community. Figure 3 outlines another classification across two dimensions: the problem domain and the type of actions (from those with no side-effects such as searching the web, to those with irreversible side-effects). The classification by action type is important since it influences the solution design (e.g., whether the agent can explore and backtrack) and limits the use cases that a solution can be applied to (e.g., an agent that assumes actions have no side-effects may not be suitable for critical health care use cases). Another classification can be: information retrieval agents vs. personal assistant agents vs. task-oriented agents (Fig. 3 x-axis is a more specific view of this classification where info retrieval agents can be viewed as agents that take actions without any side effects or reversible actions). The reason we chose this specific classification in Fig. 3 was to lead to a conversation about challenges that may hinder LLM adoption in enterprises (e.g., many enterprises may care about task-oriented chatbot problems that may take irreversible actions and LLMs are error prone and hallucination prone which is problematic when their mistakes cannot be undone). We can make the link between the problem oriented classification and solution oriented classification more explicit in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The topic is extremely relevant as several other submissions to EMNLP precisely use the generic LLM solution to problems traditionally solved with specific methods. Even though the paper analyzes anecdotal cases to draw its conclusions, the analysis is systematic and interesting insights can be perceived by the experiments.  ", "reasons_to_reject": "The work is a preliminary analysis about the boundaries of effectiveness of the generic LLM approach, as much of the current applications is not discussed and probably a taxonomy of the problems would benefit the analysis. Yet the paper has the merit to call the reader the attention to the wide spread use of LLM without actually knowing how effective it might be to specific cases.", "questions_to_authors": "In your classification of agents you have divided it into domain-agnostic autonomous agents, robotic agents, and others. What other kind of classification could be done to be more problem oriented than solution oriented?", "paper_topic_and_main_contributions": "This paper draws a review of the performance of LLM to diverse tasks that were approached through specific AI solutions trying to indicating the boundaries of effectiveness of the generic LLM approach compared to the specific solutions.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "As far as I could tell, the presented references are enough.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "As far as I could tell, there were no typos or grammar issues.\n"}, "extra_text": {}}, {"review_id": "xMmIajfJn1", "reviewer": "Reviewer_Jf2r", "summary": "", "questions": "1-\tIn line #072 you say: “In addition to performing their functions accurately, enterprise-ready technology must be explainable and reliable”, but no discussion on explanability is done. Why?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We acknowledge that the paper does not present a novel solution. Please keep in mind that this is a vision paper “with the goal of stimulating discussion around Large Language Models and the Future of NLP” (as stated in the CFP for the LLM theme track). In any case, we think there is some non-obvious material in Section 5 where we define a set of 10 research topics that can provide guidance for the research community, including highlighting existing research to build on. We also think the paper brings attention to the community about how LLMs are being used as autonomous agents in practice (Section 2), in the literature (Section 3), and with an empirical use case (Section 4) that can be leveraged by the research community to improve LLM performance (especially in applications that are relevant for enterprises).\n\nRegarding the reviewer’s question about explainability and reliability, there is well-known literature about the pitfalls of relying on opaque AI models for use cases that have consequential effects on people [1], and the different dimensions of trust that users in high-stakes use cases expect [2]. We can include some of these references in the final version of the paper. Note, however, that several of the research areas identified in the paper are related to explainability and reliability, including Sections 5.5 (Reproducibility and reliability), 5.6 (Confidence and failing gracefully), and 5.7 (Error handling and failure semantics). Each of these subsections include discussions and relevant references.\n\n[1] Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner. “Machine Bias: There’s software used across the country to predict future criminals. And it’s biased against blacks.” ProPublica (2016). https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing\n[2] Varshney, Kush R. \"Trustworthy machine learning.\" Chappaqua, NY (2021). http://trustworthymachinelearning.com", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1-\tThe paper presents the SOTA on large language model-based personal agents in the enterprise;\n2-\tSince “LLM-based autonomous agents” is a new area of interest, the paper arrives at a good moment;\n3-\tThe paper raises good points of attention in the Research Challenges section.\n", "reasons_to_reject": "1-\tThe paper does not present a novelty but contributes to disseminating important facts on LLM in autonomous agents research.", "questions_to_authors": "1-\tIn line #072 you say: “In addition to performing their functions accurately, enterprise-ready technology must be explainable and reliable”, but no discussion on explanability is done. Why?", "paper_topic_and_main_contributions": "The paper presents examples of LLM-based autonomous agents that reason and compose tools, highlights cases where they fail, surveys recent efforts in this space, and lays out the research challenges to make these solutions viable for enterprises.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OrnDj3lDDF", "reviewer": "Reviewer_dPLF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the review and encouraging feedback.\n\nRegarding other domains beyond the travel example in Section 4 of the paper, we do survey other application use cases in Sections 2 and 3. The travel example serves to complement these surveyed domains with one that we can more deeply evaluate and further inform the research challenges listed in Section 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Visionary and grounded in enough evidence\n+ Lays out a number of interesting reseach direction focused on solving specific problems\n+ A joy to read", "reasons_to_reject": "- The paper may benefit from complementing the travel example with other cases in other domains.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This vision paper discusses the current state of LLMs as autonomus agents for enterprise appllications.  The authors survey both anecdotal and academic work that have implemented LLM-based to stuate the discussion. Then, empirical evidence on the use of LLMs as tool composers in an enterprise travel use case is provided. Finally, the insight obtain from those analyses is used to define a set of challenges and recommendations to enable LLMs to be used in mission-critical orchestration enterprise use cases. \n\nThe paper does a good job showing the current state of LLMs as agents for enterprise applications, including empirical evidence. Also in providing the necessary insight to motivate the final list of recommendations, which IMO look reasonable and point at interesting future work.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, this paper addresses the topic of LMs powering agents with clarity, providing a structured framework* for how to reason about it.

As the reviewers all note, the paper is well-written and provides good conceptual clarity. I am perhaps a bit more critical than the reviewers here on how much the paper has sharpened our view on the agent paradigm; I would encourage the authors to be a bit more incisive about how things should evolve and so forth (e.g. most of the guidance in the paper is not terribly sharp/fairly restrained). With that said, this paper is a good contribution to our field and hopefully will help the growth of this subarea!

* Remark: I am not sure I would call the paper a vision paper/it seems more like a framework (whereas I would expect a vision paper to make more speculation about the future/take a stronger stand on how the future should evolve), but that's fine/a minor nit.
Decision: Accept-Findings