WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting
Authors: Olly Styles
Sam Miller
Patricio Cerda-Mardini
Tanaya Guha
Victor Sanchez
Bertie Vidgen
Venue: COLM (2024)
Keywords: evaluation, work, LLMs, simulation
TL;DR: A benchmark dataset for evaluating agents in a realistic workplace setting.

Abstract: We introduce WorkBench: a benchmark dataset for evaluating agents’ ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents’ errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents’ ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/link_updated_upon_acceptance

Published as a conference paper at COLM 2024
WorkBench: Benchmark Dataset for Agents in a Realistic Workplace Setting
Olly Styles; Sam Miller Patricio Cerda-Mardini Mindsdb ollystylesd@mail. com,
sameartanis.ai, patriciommindsdb.com
Tanaya Guha
University of Glasgow
tanaya. augh@ggasggw...omm ac.
Victor Sanchez
University of Warwick
V.F. S....anhhez-Sil.. wwwwwwwwwww...a...... ac.
Bertie Vidgen Contextual AI
bvidgen@turing. ac
Abstract
We introduce WorkBench: a benchmark dataset for evaluating agents' ability execute tasks workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks repre- sent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for ro- bust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents' errors can result in the wrong action being taken, such an email being sent to the wrong person. Wench reveals weaknesses in agents' ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as free resource /thttps://github. http://thhttbbc.ooolllyyss//////
Introduction
Large language models (LLMs) excel at a broad range of tasks such as translation, summari- sation and sentiment analysis. However, they often fail tasks such as retrieving niche knowledge or recent information (Bubeck et al. 2023). For example: they may give wrong answer for this year's 's growth of the S&P 500. Proposed approaches overcoming these limitations include finetuning and Retrieval Augmented Generation (Lewis et al. 2020), or RAG, which gives LLMs access to data stored as embedded vectors. However, these approaches cannot deal with other LLM failures, such as simple arithmetic problems (Brown et al. 2020). LLMs' inability to take actions also limits their capabilities. While they can summarise email for user, they cannot reply to it.
'Fqual contributors

---

Published as a conference paper at COLM 2024
User
Can you email Sarah to arrange a catch-up if we haven't met the last week?
Agent 1:46pm
Sure! It's been 8 days since your last meeting with Sarah l've sent her an email.
Figure 1: Agents in the workplace. A sample task from WorkBench, the first cataset for evaluating autonomous agents on realistic workplace tasks.
Autonomous agents, powered by LLMs with access to tools, can overcome these limitations. Tools are functions that agents use to interact with their environment. Rather than relying solely on knowledge from training, agents engines to broaden their capabilities. With access to external systems, these agents can perform range of tasks such replying to emails and booking meetings.
tools such calculators and search
While agents are promising, they make mistakes and little is known about their efficacy in practice. Recent benchmark datasets cover tasks like solving logic problems (Mialon al., 2023), or playing video games (Wang et al. 2023). These help understand agents' broader capabilities, but do not represent tasks agent could be asked perform in workplace setting.
This paper introduces WorkBench: benchmark dataset for evaluating agents realistic workplace setting. Tasks in WorkBench represent common functions across businesses, such as sending emails. WorkBench includes complex, multi-step tasks that require using several tools. For example: the task in Figure requires an agent to review the user's 's calendar before sending an email.
For tasks in WorkBench, we introduce outcome-centric evaluation. Each task has a unique, unambiguous ground-truth outcome. Agents are evaluated on whether the outcome from their actions matches the ground truth outcome. This is a major step forward in enabling robust and automatic evaluation of action-taking agents.
We build a dataset of 690 unique tasks by combining human-curated task templates with a programmatic approach for creating multiple tasks per template. WorkBench tasks are challenging. We evaluate five different agents, finding that state-of-the-art agent (ReAct with GPT-4) only completes 43% of tasks correctly.
To summarise, the contributions of this work are:
An opensource dataset, WorkBench, which enables robust and automatic evaluation of agents in realistic workplace setting.
Our outcome-centren evaluation methodology for benchmarking agents.
Benchmark results showing what current state-of-the-at agents are capable of.
2 Literature Review
Tool Use Methods. Many prior works have investigated using LLMs as a planning agent for API-based tools (Qin (2023a) provide survey). Augmenting LLMs with tools enables them to interact with external systems, so end-users can interact with systems through a chat interface. Closed-source LLMs, like GPT-4 (OpenAl et al., 2023), are assumed to be fine-tuned led researchers to using closed-source LLMs generate the training data for opensource LLMMs (Liu et al. 2023).
tool-usage data, whereas opensource LLMs are generally not. This has
Yao et al. 2023 propose the ReAct framework, which has been highly influential for agent design. 1 Tis inspires approaches that i) transform the user's intent into high-level task, ii) make plan to achieve the task based on available tools, and finally iii) execute that plan (Qin
et al. 2023b Schick et al. 2023: Bubeck et al. 2023). Lu et al. (2023) show this approach is
2

---

Published as a conference paper at COLM 2024
effective when using tools across multiple domains, such maths and information retrieval, with GPT-4 as the LLM.
Many papers build the insight that adding examples of tool use to the prompt may improve performance (Schick et al. 2023). Hsieh et al. (2023) instead give tool documentation in the prompt, showing that this may be more effective than demonstrations. |Hao et al. (2023) overcome this by embedding each tool in a separate space, then using this embedding space choose tools. [Snn al. (023)) show that first forming a natural language plan and then forming an explicit call plan improves performance.
Tool Use Limitations. A limitation of some studies is assessing tool usage with GPT-3.5 as the only LLM, so others have analysed the impact of choosing different LLMs (Qin et al. 2023b: Hsieh et al. 2023; Schick et al.) 2023). Ruan al. (2023) find that most smaller LLMs cannot effectively use tools, particularly for multi-step problems, with only GPT-3 effectively using tools. Huang al. (2023) support this conclusion, finding that most LLMs other than GPT-4 cannot ettectively select the right tools on their benchmark dataset. This may be explained by the finding of et al. (2023) that GPT-4 can make correct API calls without in-context examples, whereas most models cannot. This suggests tool usage is part of the training data for GPT-4. Nevertheless, Hao et al. (2023) propose improving opensource models by giving them access to embeddings of tool descriptions. Ruan et al. (2024) study the risk of negative side effects when agents are used in real-world setting, which we build on in this work.
Tool Use Evaluation. There are many datasets for evaluating LLM capabilities, such as Big-Bench (Srivavastava et. all 2023 MMLU (Hendrycks et al. 2021) and HellaSwag (Zellers) et al. 2019). These feature range of topies such as coding, chess, and chemistry. However, they are of limited value for evaluating tool usage, given that questions are answerable without access to tools.
Other benchmarks have been proposed for specifically evaluating tool usage. Mialon al. 2023 propose GAIA, which contains 466 questions that test agents on tasks such as web search and coding. Each question has unique answer for robust, automatic evaluation. Patil eet al. (2023) present APIBench for evaluating the effectiveness of LLM agents in calling other Al models accessed through APIs. SLURP (Bastianelli et al. 2020) and TaskMaster (Byrne
al. 2019) focus on smart home tasks that only require single action. Other benchmarks use simulated environments, such as playing video games (Wang et al.) 2023), human behaviour in a home (Park et al. 2023 and chat dialogues (Budzianowski et al. 2018).
Xu al. 2023) propose ToolBench for evaluating the efficacy of LLMs in constructing real API calls for tasks such travel booking. They generate a large volume of tasks with ChatGPT. Zhuang et al. (2023) propose ToolQA, which combines human task creation with programmatic approach to scaling the dataset. However, both datasets are limited to in- formation retrieval tasks. Li al. (2023) propose API-Bank for evaluating agents using tools to take actions. Their automated approach enables very large number of tools. However, answers are not unique or unambiguous, They require an LLM to evaluate performance of agents, therefore the validity of their evaluation depends on the evaluator LLM not making mistakes. Our proposed outcome-centric evaluation methodology overcomes this issue. Table compares WorkBench against existing datasets.
Web Browsing Agents. Another relevant strand of literature is agents for web browsing,
Their tools are implicitly defined via possible interactions with a browser (Deng et al. 2023
He et al. 2024 Yao et al. 22022). Some web browsing benchmarks use text-based eval evaluation,
which shares properties with our evaluation methodology (Zhou et al. 2024 Koh et al. 2024).
Particularly relevant and concurrent with our work, Drouin et al. (024) also propose
robust method for evaluating retrieval tasks. Our benchmark builds on these papers by i)
evaluating side effects from unintended state changes outside the target, and ii) focusing workplace-relevant tasks.
3

---

Published as a conference paper at COLM 2024
Outcome-centric
Dataset
Tool usage Actions
evaluation
BigBench
Srivavastava al
2023
X
n/a
Datasets without tools
MMLU
Hendrycks et al.
2021
X
X
n/a
HellaSwag (Zellers et al.
2019
X
n/a
GATA Mialon et al.
2023
X
n/a
Retrieval-based
ToolQA Zhuang et al.
2023
X
n/a
datasets
ToolBench Qin al. 2023b
X
n/a
API-Bank Li et al.
2023)
X
Action-based
WebArena
Zhou et 2024
datasets
WorkArena Drouin et al. 2024
X
WorkBench (Ours)
Table 1: Dataset comparison. WorkBench is the first dataset built with outcome-centric evaluation. This enables robust, automatic evaluation of agents on tasks requiring actions.
Proposed Benchmark Dataset: WorkBench
Figure 2 shows our complete methodology for evaluating agents. Real-world agents typ- ically interact with data using external API calls. However, we do not use data from real APIs such as Gmail, as these require authentication and change over time. Instead, we create local environment with five sandbox databases. These represent the initial state of the agent's environment. In response to a task, the agent's actions can alter the sandbox databases. All tasks in WorkBench have a unique, unambiguous outcome, which is the expected state of the sandbox databases after successful completion of the task.
Sandbox environment
Task execution
Use tool
Outcome-centric evaluation
Initial state
Predicted
Ground Truth Outcome
Outcome
Task
Agent
User
Intermediate
Book meetings
Observe
Outcome
tomorrow with
Email
Calendar
Carlos and Sarah
Toolkits
Our complete pipeline for evaluating agents. 1) Sandbox ervironment: The sandbox has an initial state, defined by five databases. 2) Task: request is sent by the user. 3) Task execution: task sent to the agent, which has access toolkits in various domains. The agent takes actions using these tools, which may alter the sandbox databases. The agent observes the result of using the tool to determine if more actions are required. 4) Outcome-centric coaluation: the updated sandbox databases compared against the ground truth.
Figure 2:
3.1 Sandbox Environment
We create sandbox environment consisting of five simulated databases:
Calendar. Each event has an ID, name, participant email, start time and duration. There are 300 events in the database. See Table for snippet of Calendar data.
4

---

Published as a conference paper at COLM 2024
Template
Example
Task
Task
Cancel my meeting with (name).
Cancel my meeting with Carlos.
Outcome
Outcome
def vet_ootcome__ee tet.uuoooeeeeeeplttteeeeee (nane):
get_ cir__outcoe__ecc wtt._utttooe_eepplaeeeeee.e Carlos")
events calendar www.eeeeeaeeeeeeeeee (queryyyan
[E for E in events if E.tine now()] if len(events):
event_id events[o]["'d""
new_calendar calendar. delete (id=ovent_id)
new_calendar.ewe
Initial
Ground truth outcome
olse:
calendar. events
Figure 3: Task and outcome creation. The left side shows pair of task-and-outcomee templates. The outcome template is a function that returns the ground truth for the changes to the sandbox databases, given correct task completion. The right side shows task- outcome pair created from these templates. In this example, the correct outcome is that the next meeting with Carlos is no longer in the Calendar sandbox.
Email. Each email has an ID, sender, subject, body, and datetime. There are 500 emails in the database.
Website analytics. Each website visit has visitor ID, page view count, visit duration, traffic source, and visitor engagement score. There are 500 website visits in the database. Customer relationship management (CRM). Each customer has an ID, name, email, phone number, activity status, assigned employee, date of last contact, product interest, follow up deadline, and notes. There are 200 customers in database. Project management. Each task has an ID, name, due date, assigned employee, list name, and board name. There are 300 tasks in our database.
Event ID Event Name
Participant Email luis.ortiz@atlas.com
Start Time
Duration Minutes
000013
sync up
2023-08-01 09:00
90
000275
process review
fatima.khane taiima.khan@atlas.com
2023-08-01 11:30
30
000264
Onboarding akira.sato@atlas.com
2023-08-01 14:30
60
Table 2: Calendar sandbox database snippet. This a sample of data from the Calendar sandbox database, which has row for each event. We create sandbox databases for four other domains, which are further detailed in Appendix
3.2 Task and outcome pairs
We manually create task templates that represent realistic workplace tasks. Templates are split into two categories: 1) Single domain: these only require tools from one domain, such as Calendar, to complete. 2) Multi-domain: these require tools from multiple domains, such as Email and Calendar, to complete.
Figure shows how we programmatically create multiple tasks per template, and calculate the associated outcome. Across all domains, create 10 unique tasks for each template, yielding 690 tasks in total. Table 3 shows the number of templates for each domain in WorkBench. Our templates contain added linguistic variation, so that each task is phrased in three different ways. We provide examples of this variation and empirical justification for our template approach in Appendix A.3
We create some very challenging task templates by combining actions across multiple domains. Figure 4shows the distribution of the number of actions required to complete

---

Published as a conference paper at COLM 2024
Domain
Number of unique templates
Number of unique tasks
Analytics
12
120
Calendar
11
110
CRM
8
80
Email
9
90
Project Management
8
80
Multi-domain
21
210
Total
69
690
Table 3: Number of unique tasks and templates in each domain. Our template approach creates 690 varied, realistic tasks. Each has an associated unique outcome.
400
Frequency
300
200
100
1
2
4
6
7
9 10
11 12
Number actions
Figure 4: Ground truth number of actions required to complete task. 18% of tasks require no actions. Sometimes agent would use retrieval tools, but would not need to execute any actions. For example: request to cancel meetings on date when there aren't any scheduled. These tasks are easier, which we show in Appendix A.4
each task in database. Many tasks require multiple actions, with some requiring up to 12 actions. Here is an example of complex multi-domain task:
If our website page views fell by more than 10% in the past week, schedule 30-minute meeting with Sam called "Urgent Analytics Update" at the earliest free time tomorrow. Otherwise email them saying "Site traffic was stable the past week, nice work."
We can define simple function to automatically find the correct outcome. However, the agent must combine Analytics, Calendar and Email tools to complete this task. Ap- pendix A.2 provides more examples of tasks.
3.3 Task Execution
Agents execute tasks using 26 tools across the five domains, which are summarised in Table 4
Each tool consists of a function that interacts with the sandbox databases, and documentation (a docstring) showing the agent how use the tool. Each docstring contains high-level description, parameters, return values, an example of tool usage, and any limits. An example limit is the search. events (start_time, end. time), which returns a maximum of five events. If the agent needs to find a large number of events, it must search multiple times with different time windows and then concatenate the results. Appendix A.5 contains the full docstring for each tool.
3.4 Outcome-Centrr Evaluation
Figure shows that the correct outcome, which is the ground truth, is always known. We then evaluate whether the outcome resulting from the agent's actions matches the ground truth. The ground truth therefore includes the state of all five sandbox databases. We call this methodology outcome-centric evaluation Figure 5 compares our outcome-centric evaluation against prior works, which evaluate the agent's function calls.

---

Published as a conference paper at COLM 2024
Email
Calendar
Web Analytics get.visitor.infs
CRM
Projects
get.email innf
get_event_info search.events areate_event
weccuustomeriiiii
get.task.info
search.emails
count.traffic.source
sear search.cus tomer's
search.tasks
send_email
sount.engaged.users
update_custone comeeerr
create_task
delete_email delete_event count_total.visiie average.visi
add_customer
delete_task
forward.email
update.event
weeragevisiit.duraiio create_plot
delete_customee tomer
update_task
reply_email
Table 4: Summary of toolkits. We define 26 tools across domains.
An agent can follow any action path provided the resulting sandbox databases match the ground truth outcome. For example, sometimes the agent recovers from its error and takes the correct action:
Task:
Make a task on the Front end board for Sam to improve conversion.
Ground truth tool use:
create_ treake_task (name="improw conversion", board="Front end", "o=
Agent's tool use:
create task (name="improve conversion", board="Front End", assigned to="Sam") Observation: "'Front End' board does not exist, but 'Front end' does create_task (name="improve conversion", board="Front end", aasigned_to=="Se
Evaluation methods based matching the function calls could unfairly find the agent had failed due to the extra calls. However, outcome-centric evaluation recognises that the agent was able to recover because the final change in state matches the ground truth outcome. As a result, the agent is not unfairly penalised.
Some prior benchmarks such Gaia (Mialon et al., 2023 and ToolQA (Zhuang et al., 2023) also have tasks with unique outcomes, but they are limited to information retrieval. WorkBench is the first dataset to evaluate tasks that require actions in this manner, due to our outcome-centric evaluation methodology.
4 Results
We assess the performance of LLM agents using the ReAct framework (Yao et al. 2023). This enables the LLM to perform multiple action steps and update its action plan based on results from previous steps.
4.1 Performance Metrics
Our primary metric is accuracy. This is the % of tasks where the outcome from the agent's actions match the expected outcome, which is the ground truth.
Our secondary metric side effects Some tools have negative consequences if used incor- rectly, such as sending emails to the wrong person. If the agent's actions modify the sandbox databases in a way that does match the ground truth exactly, we consider this side effect. If the agent fails to complete the task, but does not alter the sandbox databases, then there are no side effects.
4.2 Comparing Large Language Models
Table compares five LLMss: GPT-3.5 (Brown et al. 2020 GPT-4 OpenAl et al.) 2023)
Claude-2 |Anthropic 2023), Llama2-70B [louvron et al. 2023 and Mixtral-8x71 (Jiang et al. 2024). GPT4 greatly outperforms other models. For the porse-performing models, the main errors are insufficient context window length and failing to follow the ReAct framework.
This computer science term refers program altering variables outside its local environment.
7

---

Published as a conference paper at COLM 2024
Tool Use Evaluation (Prior works)
Predicted Tool Use
Ground Truth Tool Use
update_tase tadeetttaaidddaaoaaaeae "00'd
update_taske task
"ielt""ttats",, Delue="Compless
field="Status", value="In II review")
apdate_taske task(id='000))
update_ Sasatt__tss((e task
field="Status",
value= review")
field="Status", value="In review")
Outcome Centric Evaluation (This work)
Predicted Outcome
Ground Truth Outcome
Task ID
Task name
Status
Task ID
Task name
Status
008
Design new logo
Complete
008
Design new logo
In review
Task ID
Task name
Status
Task ID
Task name
Status
008
Design new logo
In review
008
Design new logo
In review
Figure 5: Outcome-Centria Evaluation. We propose outcome-centric evaluation, where there is unique ground-truth outcome for each task (lower panel). We consider the task correctly executed the predicted outcome following the agent's actions matches this outcome. This allows the agent to find multiple paths to the correct outcome, unlike prior works (upper panel) which evaluate the agent's function calls.
GPT-4
GPT-3.5
Claude-2 Llama2-70B
Mixtral-8x7B
Accuracy (required tools)
49%
14%
23%
3%
20%
Accuracy (all tools)
43%
0%
26%
0%
16%
Table 5: Comparing accuracy across all models on WorkBench. We selected these models based on general usage and prevalence in other benchmark studies. Due to limits on context window size, the upper row shows when models are given only the toolkit from the domain needed to complete each task. For example: the task were "Book a 30-minute meeting with Sam for tomorrow at 9:30", we would provide all tools in the calendar toolkit and none from other toolkits. The lower row is when models are given all 26 tools. Scores of 0% occur when the context window is not long enough to include all 26 tool descriptions.
Our benchmark is challenging for all models, including GPT-4. Given how poorly other models perform, we restrict further analysis to GPT-4. When giving this agent all 26 tools, rather than just the required toolkits, and find accuracy falls from 49% to 43%. This suggests the agent is negatively affected by redundant tools, which is consistent with prior studies (Hao et al. 2023). The next sections explore in further depth why the GPT-4 agent fails.
4.3 Performance across domains
Table 0lmparce the GPT-4 agent's performance across our five individual domains, and tasks that require tools from multiple domains. Performance varies from 23% accuracy CRM tasks to 65% for Calendar tasks. The agent is capable of combining tools across multiple domains. Its performance (40%) on these tasks is similar to its average performance single-domain tasks (43%).
8

---

Published as a conference paper at COLM 2024
Project Management
Multi
Analytics
Calendar
CRM
Email
Domain
Number of tasks
120
110
80
90
80
210
Accuracy (1)
39%
65%
23%
48%
39%
40%
Side Effects (J)
54%
22%
6%
6%
4%
29%
Table 6: Performance of GPT-4 ReAct agent on WorkBench with all tools provided.. Higher is better (1) for accuracy and lower is better (J) for side effects. Side effects are particularly common in the Analytics domain, as the agent often plots data for the wrong time period.
4.4 Sources of Error
Figure 6h shows the prevalence of side effects. Side effects occur when the agent's actions modify the sandbox environment, but this change does not match the ground truth outcome. In this example from task in our dataset, the agent cancels the wrong meeting:
Task:
Cancel my next meeting with Nadia Ground truth: delete eeentt (event_id=000000005)) Prediction: delete_event (event.c==0000001111)
Errors without side effects occur when the agent fails to complete task, but there are no unintended modifications to the environment, In this example, the agent does not take any actions because it searches on the wrong date:
(Today's date, Monday 20th November)
Task: Cancel all my meetings on Tuesday
Ground Truth: delete event (event id=20000025)
Prediction: Searches for meetings on 28th November - no meetings are found
Figures 6p and 6 further break down errors by their most common sources. The most frequent error is failing to follow the ReAct framework. The agent must use the keyword ACTION followed by a JSON string with the tool name and arguments. The agent may omit the ACTION keyword, meaning actions are performed.
To address this source of error, we implement a re-sampling strategy. the agent does not correctly generate the ACTION keyword, repeat the same task up to 5 times. We use temperature of 0.5 for retries, rather than the initial temperature of With this strategy, the accuracy of the GPT-4 agent increases from 43% to 49%.
However, other common sources of error cannot be fixed with resampling. The agent often fails to find the correct email address when given names in the task. The agent may hallucinate an email address rather than using the search tool to find the correct email address. Here is an example, with intermediate steps hidden for concision:
Task: Forward all the emails from kofi last week about 'Staff Roster for Next Week' to fatina Ground truth: forward_erd deemaieealllllae (d"211 recipient="fatinee khan@atlas. com") Prediction:
forward email amail(emaill)) (emaii.id==2299"" id=="229",
com"
Errors also come from searching incorrectly. The agent may fail tasks such as "Cancel my next meeting with Sam" because it searches for events in the past. Similarly, the agent does not always account for the limits of its tools. It can fail on tasks such as "Cancel all future meetings with Sam" because it only cancels subset of future events after failing to factor in the limit on the number of results returned from searches. The agent could complete this task by repeating the search-and-delen process, but fails to do so.
Discussion and Future Work
We have introduced WorkBench the first benchmark that enables robust, automatic evalu- ation of agents in workplace setting. Our approach ensures that each task has unique

---

Published as a conference paper at COLM 2024
Falled to follow ReAct
Thied
future
Other
Correct
28%
43%
witnout Side Effects
12%
12%
Other
Wroon email
19%
19%
identily
10%
28%
9%
Incorrect with Side Effects
Wrong
Incorrecty
Updated
Misnteepreteed retrieved deta
(a) Overall Error Breakdown
(b) Errors without Side Effects
Errors with Side Effects
Figure 6: Error breakdown for GPT-4 across all domains with all tools provided. Break- down of errors with side effects vs those with no side effects (left). Detailed breakdown of errors with no side effects (middle) and errors with side effects (right).
outcome, which is the expected change to the state of the sandbox environment upon successful task completion.
One limitation of WorkBench is how well the sandbox environment represents real-world complexity, A real email inbox may contain tens of thousands of emails over many years, and emails are often spam, very long, and/or full of errors. Our initial results may therefore overestimate agents' current capabilities, so further studies could improve WorkBench by adding more challenging sandbox data.
While our tasks require multiple actions, they are limited to single-turn chat. Some longer queries may not represent realistic workplace interactions. Establishing a human baseline on our tasks would help determine the significance this, but is hard implement given our tasks would be completed by humans using a Graphical User Interface (GUT). Implementing a human baseline would measure GUI quality as well as task difficulty. Nevertheless, multi-turn chat setup may be more representative of real tasks and could build upon our work.
We also found the agent's performance dropped when it had greater number of tools could choose from, but we were not able to explore this relationship fully with just 26 tools. Future work could extend our benchmark by adding more tools from other real-world domains such as HR software. This would help assess the relationship between agent accuracy and breadth of tooling.
A final limitation is that we do not assess pure retrieval tasks. While retrieval tools are required intermediate step to complete many WorkBench tasks, we do not assess tasks that require solely retrieval such as finding a recent email. Future papers could build on our work with a method for assessing retrieval tasks, such as those proposed by other recent benchmarks (Drouin et al.) 2024 Zhuang et al. 2023).
Despite these limitations, WorkBench has large volume of high-quality, unique tasks. We include complex tasks that require planning, tool selection and analysing results across five domains. WorkBench is challenging, with the best agent achieving only 43% accuracy. We find the main sources of error are the agent failing to execute its plan, giving the wrong arguments to tools, and not understanding the limits of its tools. Furthermore, errors often have negative consequences like sending emails to the wrong people. Future work could study fine-tuning LLMs to improve performance, such as (Schick et al. 2023).
WorkBench is both scalable and extensible. Future researchers could extend our dataset to include new domains, such as accounting tasks, and build an even larger dataset using our scalable approach to task creation. This will enable the evaluation of agents in progressively more complex settings as they continue to improve in the future.
References
Anthropic. Model card and evaluations for claude models. http::/wwww.....ooooooaoooeaaaaa 2252
2023.
URL
https:// cdn. http::/www..dh.rrttrrrr......
Model-Card-Claude-
10

---

Published as a conference paper at COLM 2024
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A spoken language understanding resource package. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7252-7262, Online, November 2020. Association for Compulational Linguistics. doi: 10.18653 /v1 h /2000..mmlp-mmai..55. URL https
//aclanthology.org/ org/ wttp://re/.....mmmmmmaaaaaa 588
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voste Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc,, 2020. URL file://57700bbr00044111181122222
neur http://aann...uurrr......cppppea
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Pawet Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašic. MultiWOZ a large-scale multi-domain Wizard- of-Oz dataset for task-oriented dialogue modelling. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujüi Proceedinns of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5016-5026, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/ (18653//11/18-15//// URL httpss///aclamthoooo..oo/ggg/
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. Taskmaster-1: Toward a realistic and diverse dialog dataset. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 4516-4456444 5516-4525, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: (0.18653/v1/D19-11 http:://cc..aanhooooy.ro////1/10
URL
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 28091-28114 Curran As- sociates, Inc., 2023. URL https://proceedins.eeeee neuriss. wwwpppppapeeeeeepppppppeeepeee and Benchmarks.pdff
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks? In The International Conference on Machine Learning, 2024.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings, 2023.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models, 2024.
httgg/bb//401111/ 13919
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
11

---

Published as a conference paper at COLM 2024
Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li, Yasuhisa Fujii, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, and Tomas Pfister. Tool documentation enables zero-shot tool-usage with large language models, 2023.
Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. Metatool benchmark for large language models: Deciding whether to use tools and which to use, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arxiv preprint Xio:2401 Xit:2401.040888 2024.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024. URL https://arxiv
org/abs/2401. 13649
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9459-9474. Curran Associates, Inc., 2020. URL URLhtpps//prooceddgg.nn
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented Ilms, 2023.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating iims as agents, 2023.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Pluu-and-play compositional reasoning with large language models. In The 37th Conference on Neural Information Processing Systems (NeurIPS), 2023.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023.
OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim- ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiroo Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick,
12

---

Published as a conference paper at COLM 2024
Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023.
Joon Sung Park, Joseph C. O'Brien, Carrie]. J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023.
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong. Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models, 2023a.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toollim: Facilitating large language models to master 16000+ real-world apis, 2023b.
Jingqing Ruan, Yihong Chen, Bin Zhang, Zhiwei Xu, Tianpeng Bao, Guoqing Du, Shiwei Shi, Hangyu Mao, Ziyue Li, Xingyu Zeng, and Rui Zhao. Tptu: Large language model-based ai agents for task planning and tool usage, 2023.
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris] J Maddison, and Tatsunori Hashimoto. Identifying the risks of Im agents with an Im-emulated sandbox. In The Twelfth International Conference on Learning Representations, 2024.
Timo Schick, Jane Dwivedi-YYu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.
13

---

Published as a conference paper at COLM 2024
Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world restful apis, 2023.
Srivavastava et. al. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ar preprint arXiv:2307 (KKr:2207.00988,, 09288,, 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models, 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: To- wards scalable real-world web interaction with grounded language agents. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 20744-2075! Curran Associates, Inc., 2022. UR/httpss:/proocediing hewp:/oceeednnns.eeiiiii. neur ips. cc/paper files/paper/2822/ cb9101 hat.aa/offfffffcccccccaacoca1a6-
pdf
Shunyu Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2019.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. URL httpss///ppennreiiwwweeeerrrrrmm @Kn9c6ytLx
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for Ilm question answering with external tools, 2023.
A Appendix
A.1
Simulated database creation
Below is a sample of 5 rows from each of our sandbox databases.
A.1.1 Analytics
date viiist, visi .id, pwgpppageeeee.eeee
traffic unseeeeeuu
2023-10-22, 860, 8, referral,
2023-11-22, 214, 11, 1, search engine, False
2023-09-24, 130, 18, social media, False
2023-10-08, 385, 2, 4, direct, False
2023-09-23 252, 2, 11, search engine, False
14

---

Published as a conference paper at COLM 2024
A.1.2 Calendar
event_id, event_name, participant.emaii event.start, duration
00000013, sync up, quis.ortizéatlan com, 2023-08-01 09:00:00 00, 90
00000275, process review, aitima.khan@atlan com, 2023-08-01 11:30: 30: 00,
00000098, Data Security and Compliance Training, amir ali@atla com, 2023-08-02
11:00:00, 30
00000190, Product Launch Analysis, yuki tww.aaaaaallaaeooomm com, 2023-08-02 11:30:00,
30
00000071, daily stand-up, kofi. nofi.mensah@atllas.com com, 2023-08-02 13:30:00,
A.1.3 Customer Relationship Manager
customer_id, assigned assigned.to.emmal dast_contact_date 00000189, lena. scwmict@ttlass.....mmmmmo com, Taylor Jackson, taylor. myyloor.jcckoonennnoll.o
wettome__amee customer_emal _emall, customer.phon .phone, status, follow_up_bp notes
2023-11-30, Consulting, Lost, 2023-12-22, 2023-11-07: Had a call.
Had call. 2023-11-25:
00000107, sofia. sariosaaaatss com, Quinn Harris, quinn. haanmhaarrrissanooorrroooooiis 2023-11-30, Consul ting, Proposal, 2023-12-14, 2023-11-26:
Saw the demo.
2023-11-29: Had a call. 2023-10-27: Had call.
00000052, raj.patel@atlas com, Jaden White,
jaden.whi teiteerrrrcceeoodd 724-857-2625, 2823-11-30, Hardware, Won, 2023-12-13, 2023-10-17: Had a call.
00000102, vofia.santos@atti com, Alex Thomas, alex. thomas@proeeerry.
2023-11-30, Hardware, Qualified, 2023-12-22, 2023-10-15: On holiday.
00000187, lena. scwmidt@atlas. com, Quinn Robinson,
999-396-5380,
2023-11-30 Hardware, Lead, 2823-12-23,
A.1.4 Email
email.id, inbox/outbox, sender/recipient, subject, sent.datetime body 00000373, inbox, santiago.martinezzee waw.ineeettela......o com, Task Update on Develop prototype for payment gateway, 2023-10-01 09:15:02, "Sam, \n Completed task 'Develop prototype for payment gateway' ahead of schedule. Please review and let me know if any tweaks are needed. \n)n Best,'n 00000353, inbox, chenwei. shanwei.zhaggeatlas com, Update on Annual Budget Planning Session, 2023-10-01 09:40:01, "Sam, \n Encountered few challenges while working on the Annual Budget Planning Session. Could use your advice. \n'n Cheers, \n Chenwei" 00000013, inbox, . wwwsmmeatlas..oomm com, Task Update Fix alignment issue in homepage, 2023-10-01 50:50:46, "Dear Sam, \n Regarding task 'Fix alignment issue in homepage', I've made significant progress but have hit snag with third-party API compatibility. Could use brainstorm session. \n In Regards, \n Kofi"
Santiago"
00000103, inbox, Theewei.zhang@atte com, Update on Quarterly Sales Review, 2023-10-01 10:58:07, "Hey Sam, \n Encountered few challenges while working on the Quarterly Sales Review. Could use your advice. 'n \n Thanks, \n Chenwei" 00000295, inbox, andia.moreauaatlan com, Update on Year-End Performance Assessment, 2023-10-01 37, "Hey Sam, in Could you provide your input on the Year-End Performance Assessment planing? Your insights would be really valuable. \n |n Addi tionally, wanted to touch base on some other areas we' 'ee been focusing on lately. Our team has been working tirelessly on improving our project management workflows and enhancing collaboration across departments. This effort includes adopting new tools, refining our communication strategies, and ensuring that all team members fully aligned with objectives. \n \n Best, in Nadia"
15

---

Published as a conference paper at COLM 2024
A.1.5 Project management
task id, task nase, assigned to email, list.name, due date, board 00000149, Add animation to carousel, leila. aziziiallzsi.. com, Backlog, 2023-11-28
Front end 00000037, Add authentication for email notification, carlos. radr rorrigueeaai. com
Backlog, 2023-11-28, Back end 00000061, Update Flask to latest version, aisha, caaaaaacheatlascco Backlog, 2023-11-28, Back end
00000093, Optimize database query for search functionality, fatima. Keiima.khanettlas.com
Backlog, 2023-11-28, Back end 00000096, Add authentication for third-party login, carlos. rarlos..odrigueet rowrirezzeieeeallss...... com, Backlog, 2023-11-28, Back end
A.2 Example tasks
The following are randomly sampled tasks from each domain and randomly sampled multi-domain tasks.
A.2.1
Analytics
Please plot for me the distribution of engaged users and average session duration between October 14 and November
Can you make a line plot of the most popular traffic source since November 27?
Was total visits more than 10 at any time in the last 2 weeks? If so, please plot it as a line chart
you plot the distribution of both total visits and average session duration between October 12 and November 6?
Can you make line plot of the most popular traffic source since October 15?
A.2.2 Calendar
Create a 1.5 hour event called New Employee Onboarding on December at 3:30 with nia
Cancel my next meeting with yuki
Delete the next Annual Budget Planning Session meeting
have I met with carlos in the last 7 days? If not, schedule 30-minute meeting called 'catch-up' for my first free slot from tomorrow
something came up. Can you cancel my meetings on Friday before 10:30?
A.2.3 Customer Relationship Manager
Give Sofia all of Lena's s customers that are interested in training and are either qualified or in proposal in the crm
need to move all of Sofia's customers that are interested in training and are either qualified or in proposal to Nadia. Can you make that change in the crm?
Reassign all of Nadia's leads that are interested in training to Lena in the crm.
Move all customers that haven't responded to proposal for the consulting product in weeks to lost in the crm
Sofia is taking over all of Lena's 's customers that are interested in services and either qualified or in proposal. Can you reassign them in the crm?
16

---

Published as a conference paper at COLM 2024
A.2.4 Email
I need to reply to the latest email from kofi with 'Got it, thank you!".
Can you do that?
can you forward the latest email about 'Task Update on Design logo for blog' carlos
lena and aisha need the last email about 'Update Team Building Retreat'. Can you forward it?
Reply to yuki's 's last email about 'Update on Corporate Social Responsibility Initiative' with 'Thanks for the update will get back to you tomorrow.
Delete my last email from chenwei
A.2.5 Project Management
Move any of luis's tasks that in review completed
Give all the overdue tasks that fatima hasn't started to santiago
Move any of nia's tasks that are in review to completed
Give all the overdue tasks that chenwei hasn't started to amir.
can you move any of luis's s tasks that are review completed?
A.2.6 Multi-domain
I need to make sure everyone remembers to attend the first event on December 6. Can you send email to the attendees with the event name as the title and "Remember to attend this event.' in the email?
need to make sure everyone remembers to attend the first event on December
1. Can you send email to the attendees with the event name as the title and "Remember to attend this event.' in the email?
please check the percent growth of engaged users since Friday. If it grew by more than average session duration make front-end backlog task called Improve average session duration' for kofi that's due next Friday and schedule 30 minute meeting called "Discuss engaged users' for at the earliest slot i'm free tomorrow
think carlos might have some overdue tasks. Can you check and if so, send them an email titled 'Overdue tasks' saying 'You have few overdue tasks can you update me on them?' Otherwise email them with 'Nice work keeping top of your tasks this sprint!' titled 'Good work this sprint'
if fatima hasn't sent me any emails in the past days, schedule half hour meeting with them for Friday and call it 'Catch up with fatima'
A.3 Template dependence
As introduced in Section 3.2. use templates create a large number of tasks. Figure 7 shows the percentage of tasks within each template that were completed correctly by the GPT-4 agent. This varies between 10% and 90% for most tasks, indicating that our approach creates diverse set tasks for each template.
We introduce linguistic variation by writing three templates per tasks. The following is an example of the three templates used for task in the calendar domain:
Cancel all future meetings with {name}
{name} is leaving the company. Can you cancel all future meetings with them?
I need to cancel all future meetings with {name) Can you do that for me please?
17

---

Published as a conference paper at COLM 2024
Number of templates
10
8
2
10%
20%
30%
40%
50%
80%
70%
80%
90%
100%
Percentage tasks completed correctly
Figure 7: Template dependence. Our approach to task creation using templates creates diverse set of tasks. The agent achieves 0% or 100% accuracy on just 15 out of 69 templates, indicating that tasks within template are sufficiently diverse to evaluate agent performance.
A.4 Impact of "no action" tasks
As shown Figure 122 of the 690 tasks in WorkBench do not require any actions to complete. The following is example of such task:
If haven't t met with akira the last days, schedule a 30-minute meeting called 'catch-up' for first free slot from tomorrow
Using the calendar.. search_events tool, the agent can determine that the condition for scheduling meeting is not met and therefore no action is required. In Table 7 we compare the accuracy using subsets of WorkBench based on the number of actions required. All tools are provided in the prompt.
GPT-4
GPT-3.5
Claude-2
Llama2-70B
Mixtral-8x7B
Accuracy (all tasks)
43%
0%
24%
0%
16%
Accuracy (0 action tasks)
75%
0%
71%
0%
77%
Accuracy (1+ action tasks)
36%
0%
14%
0%
2%
Accuracy (2+ action tasks)
18%
0%
1%
0%
1%
Table 7: Performance comparision based number of actions. All agents perform better on tasks that require no actions. Note that all tools are provided the prompt, which cannot fit into the context window for GPT3.5 and Llama2-70B.
A.5 Prompt and tool descriptions
The following is the full prompt template provided to the agent. The description of each tool provided to the agent in the prompt.
Today's date is Thursday, 2023-11-30 and the current time is 00: 00:00:000 00. Remember the current date and time when answering queries. Meetings must not start before 9am or end after Respond to the human as helpfully and accurately as possible. You have access to the following tools:
ieet_email_infore iy email_id=None, field=None) Retrieves specific details of an email by its ID.
email.get_eme wiw.emmiiioooaaiiooouuu
Parameters
email str, optional
Unique ID of the Specific field to return. Available
email. field str, optional
18

---

Published as a conference paper at COLM 2024
fields: "email_id" iid "sender", "subject", "sent_date" "body", "inbox/outbox"
Returns
email_information dict specified email for the given ID and field.
Information of the
Examples
>>> email.get wew.tgeeeeeeinnnee
"subject") (("subject": "Project Update"))
email.
amail.get__eee get_eemil_informanne " "sent_date") {
"sent_date", "2024-01-10 09:30:00"11 args: (( (C'emall_id': (('title': 'Email Id' 'field' ((' (('title': 'Field']))}
email.search_eemails sewrsceaeceeemilsuueeeee
date_min=None,
date_max=None) Searches for emails matching the given query across subject, body, or sender fields.
The function matches an email if all words in the query appear in any of these fields.
Parameters
query str, optional
Search query, matching terms in subject, body, sender fields.
date_min str, optional
Lower date limit for the email's 's sent date (inclusive). Format: "YYYY-NM-DD"
date_max str, optional
Upper date limit for the email's sent date (inclusive). Format: "YYYY-MM-DD" Returns
emails list
List of emails matching the query criteria.
Examples
>>> searellesscccemmissss Update")
"[""mail_id": "12345678", "inbox/outbox": "inbox". "subject": "Project
Update", "sender/recipients "jane@example.cm com", bsent_dateting "2024-01-10 09:30:00". "body": "Please find the project update attached, "]]], args: {(
"query': : (['title': 'Query', 'default': "}), 'date_min': (('title': 'Date
Min') )), 'date_max'; (('title': 'Date Max }}}}
email. send_email. email, send_ (rreeemmileereiiiten=tonnen subject=None, body=None)
19

---

Published as a conference paper at COLM 2024
Sends email to the specified recipient.
Parameters
recipient str, optional
Email address of the recipient.
subject str, optional
Subject line of the email.
body str, optional
Body content of the email.
Returns
message str
Confirmation message of the email being sent.
Examples
>>> email. send_emaill(jjaeeexxmma com", "Meeting Reminder", "Don 't forget our meeting at 10am tomorrow.")
"Email sent successfully.", args: (C'recipinnt': (('title': 'Recipient') }), 'subject': (('title': 'Subject )),, 'body': (('itlee': 'Body') }}}}
email.delede emaite_emmiil email. (email_id=Noee)) - Deletes email by its
ID.
Parameters
email_id str, optional
Unique ID of the email to be deleted.
Returns
message : str
Message indicating whether the deletion was successful.
20

---

Published as a conference paper at COLM 2024
Examples
>>> email. www.eeeeeeeeaaleeeee11eee
"Email deleted successfully."" ", args: ((' (L'emailiid' ((titlee' 'Email Id' Id'
fillfoowwardemaaa email. www.wordeeaailllaaaaraoooeeeelee (emaai_idd-N-ons recipient-None)
Forwards an email to the specified recipient.
Parameters
email_id str, optional
Unique ID of the email to be forwarded.
recipient str, optional
Email address of the recipient.
Returns
message : str
Message indicating whether the email was forwarded successfully.
Examples
>>> email. (ttpr__emmma(11335777777" "jane@exxmplee.comm)
"Email forwarded successfully." ", args: (C'email_id': (('title': 'Email Id']), 'recippentt (('title': 'Recipient" }}}}
email. repl__eeply_ematte email: email. rwplyrepplyyeeee emai (email Ldd=None, body=None) Replies an email by its ID.
Parameters
email_id str, optional
Unique ID of the email to be replied.
body str, optional
21

---

Published as a conference paper at COLM 2024
Body content of the email.
Returns
message : str
Confirmation message of the email being replied.
Examples
>>> repil.rel ("12334666""""7""""""" "Thank you for the update.")
"Email replied successfully."" ", args: {( em'ail H ('title': 'body': (('title': 'Body' }}}}
'Email Id'}},
calendar get_eeent_innormmatiee event_id=None, field=None) Returns the event for given ID.
Parameters
event_id str, optional
8-digit ID of the event.
field str, optional
Field to return. Available fields are: "event_id", "event_name" "participant_email"" "event_start", "duration"
Returns
event dict
Event information for the given and field.
Examples
>>>
calendar.get get...eet_.ifoommttoooo.__lldde. "event_name"
(("event_name": "Meeting with Sam"))
calendar calendar.get.eeent_.nnoo get.e "event_start")
22

---

Published as a conference paper at COLM 2024
[["event_start": "2021-06-01 13:00:00"]}
>>> calendar ott.://eee.eee...ooooooooooooorr "duration")
(("duration": "60"}), args: (('event_id': (('title': 'Event Id' 'field': {E 'title': 'Field' }}}}
calendar calendar.search_eeenn
calendar.search. beeaccheeeettsuueery time_min=None, -ime_max=None) Returns the events for given query.
Parameters
query: str, optional
Query to search for. Terms will be matched in the event_name and participant_email fields.
time_min: str, optional
Lower bound (inclusive) for an event's ss end time to filter by. Format: "YYYY-MM- DD HH:MM:SS"
time_max: str, optional
Upper bound (inclusive) for an event's start time to filter by. Format: "YYYY-
MM-DD SS
Returns
events list
List of events matching the query. Returns at most events.
Examples
>>> calendar calendar.search_eeeeee wwwrchheeeeeeesssss..
[(["event_id) : "00000000", "event_name" "Meeting with Sam", "participant_emaill "sam@example, com", "event_starr"": "2821-96-01 13:00:00", "duration": "60"}),
(["event_idh) id": "00000001", Dovent_name"" "Lunch with Sam", "participant_emaii" "sam@example.com com", "event_start":: "2021-06-01 13:00:00", "duration": "30))"
], args: {(' query': (('title': 'Query', 'default'; "}, 'time_min': {('title': 'Time Min' }), 'time_max'; (('title': 'Time Max })))
calendar. www.teeeeente....eeeeeeeeeeeee.. calendar
23

---

Published as a conference paper at COLM 2024
participant_email=None,, event_start=None, duration=None)
Creates new event.
Parameters
event_name: str, optional
Name of the event.
participant_emill: str, optional Email of the participant.
event_start: str, optional
Start time of the event. Format: "YYYY-MM-DD HS:
duration: str, optional
Duration of the event minutes.
Returns
event_id str
ID of the newly created event.
Examples
>>> calendar calendar.creaate_evnn ("Meetin with Sam", "sam@example.com com", "2021-06-44 13: 00:00", "60")
"00000000", args: (L'event_name! {('iitle': 'Event Name }}, pnrticipanttemail" (('title': 'Participant Email')), 'event_start' (( 'title': 'Event Start')), 'duration': (('title': 'Duration }}}}}
calendar.delete.cee deletteeevent calendar dehendar.delete.ccm (even)_id=None) Deletes an event.
Parameters
event_id: str, optional
8-digit ID of the event.
Returns
24

---

Published as a conference paper at COLM 2024
message : str
Message indicating whether the deletion was successful.
Examples
calendar calendar.dellete_re. deelete_event("00000090
"Event deleted successfully." ", args: {('event_id': (C'title': 'Event Id })))
calendar. updatee__aateeeeeeeeeeeeeeeeeeee calendar
field=None,
new_value=None) Updates an event.
Parameters
event_id: str, optional
8-digit ID of the event.
field: str, optional
Field to update.
new_value: str, optional
New value for the field.
Returns
message str
Message indicating whether the update was successful.
Examples
>>> calendar Lppdate "event_name", "New Event Name")
"Event updated successfully.", ", args: {(' (d'event_id': {(title'' 'Event Id')),
'field': {{' title': 'Field' }}, 'new_value': (('title': 'New Value'})}
analytics. engaged serraged_uees__coo users_count:: analytics. engaged Lerraged_uses___ee www.uuueesuuuueeu cow.ootimennttttttiiee (time-_iim=ooeeee. time_max=None) Returns the number of engaged users within specified time range.
Parameters
25

---

Published as a conference paper at COLM 2024
time_min str, optional
Start date of the time range. Date format is "YYYY-NM-DD". time_max str, optional End date of the time range. Date format is "WYYY-MM-DD".
Returns
engaged_users dict
Number of engaged users in the specified time range.
Examples
>>>
"2023-10-06")
(("2023-10-01"; 1, "2023-10-02"; 2, "2023-10-03": 2, "2023-10-04": 1, "2023-10- 05": 0, "2023-10-06": 4)), args: [{'time_min' (('title': 'Time Min )), 'time_max': (('title': 'Time Max }}}})
weteiieeeetiiiiiireeeeerei tinoirraioot bidid: htw.aatoooob.idiiiiiiooooieeeeee Returns the analytics data for given visitor ID.
Parameters
visitor_id str, optional
ID of the visitor.
Returns
visitor_data : dict
Analytics data for the given visitor ID.
Examples
26

---

Published as a conference paper at COLM 2024
{["date sitee_oo_viii""" "2023-10-01", "visitor_id"; "000", "page_views". "3", currsstonnduratio_sseceee "10.0", "traffic_source".. "search engine", "user_engaged". ")) args: (('visitor_ide (('title': 'Visitor []}
analytics. traffic_source__omm (time_ Lome_max=None, traffic_source Returns the number of visits from specific traffic source within specified time range.
analytics.
(time_min=nonee mone
Parameters
time_min str, optional Start date of the time range. Date format is "YYYY-NM-DD". time_max str, optional End date of the time range. Date format is "WYY--MM-DD".
traffic_source str, optional
Traffic source to filter the visits. Available values are: "direct", "referral", "search engine", "social media"
Returns
traffic__surce_visiii dict
Number of visits from the specified traffic source in the specified time range.
Examples
>>> analytics, trafficc_source_contt ("20333000 "2023-10-06", "search engine")
{("2023-10-01"; 0, "2023-10-88 1, "2023-10-03": 0, "2023-10-04"; 3, "2023-10- 05": 2, "2023-10-06"; 4)}, args: (('timemin'' (('title': 'Time Min'}}, 'time_max': (('title': 'Time Max }), "traffic_source': {('title': 'Traffic Cource'])}}
analytics. wtteylliss.ta.ll......eoo total_visiis._.t.mtm.oi wwt._viisssttttseee analytics. total_visits_eeet (time_min=None time_max=None) Returns the total number of visits within a specified time range.
Parameters
time_min str, optional
27

---

Published as a conference paper at COLM 2024
Start date of the time range. Date format is "WWY-NMM-DD". time_max str, optional
End date of the time range. Date format is "WYY-NM-DD".
Returns
total_visits dict
Total number of visits in the specified time range.
Examples
>>> analytics. total_visits__count ("2033100000 "2023-10-06")
(("2023-10-01"; 1, "2023-10-02" 2, "2023-10-03": 3, "2023-10-04": 1, "2023-10- 05": 0, "2023-10-06"; 4)), args: (('time_min's (('title': 'Time Min '}), 'time_max': (('title': 'Time Max' }]]]
analytics. www.seeeeeeeeeeeeeeeeeeeeeeeeeee analytics. ctaplicsscrreee_peepeee_ (time_min==one, time_max= galle_to_plot-Wee plot_type=None) Plots the analytics data for a given time range and value.
Parameters
time_min str, optional Start date of the time range. Date format is "YYYY-MM-DD".
time_max str, optional
End date of the time range. Date format is "YYYY-MM-DD".
value_to_plot str, optional
Value plot. Available values are: "total_visits",
dbession_duration_secooo "user_engaged", "direct", "referral", "search engine", "social media"
plot_type str, optional
Type of plot. Can be "bar", "line", "scatter" or "histogram"
Returns
file_path str
28

---

Published as a conference paper at COLM 2024
Path to the plot file. Filename is _ng
Examples
>>> amalyyiis.crrreeppoo(((223312199 "2023-12-31", "total_visits")
args: (f'time_min': (('title':
'Time Min') 'time_max'; (f'title': 'Time Max')), value_to_plot's (('title':
"Value To Plot']]; 'plot_type': (('title': 'Plot Type }}}}
analytics, geee_averrge_eession_ddee
nime_min=None, time_max=None) Returns the average session duration within specified time range.
Parameters
time_min str, optional Start date of the time range. Date format is "YYYY-MM-DD". time_max str, optional
End date of the time range. Date format is "YYYY-MMDDD".
Returns
seerage_session_duuatioo float
Average session duration in seconds in the specified time range.
Examples
>>>
"2023-10-06")
(("2023-10-01"; 10.0, "2023-10-02": 20.5, "2023-10-03"; 32.8, "2023-10-04" 40.2, 2, "2023-10-05"; 5.3, "2023-10-06"; $3.03}, args: (('time_min': (('title': 'Time Min')) 'time_max's (('title': 'Time Max }}}}
geet_ttsk_informatiooo iiditttk_idddooeee field=None) sareect_maaagemmnt task__iformaatioi... siie__y__dd(ttttee Returns the task infomration for given ID.
Parameters
29

---

Published as a conference paper at COLM 2024
task_id str, optional
8-digit ID of the task.
field str, optional
Field return. Available fields are: "task_id", "task_name", tossiignd__t_email""" "list_name", "due_date", "board"
Returns
task dict
Task information for the given ID and field.
Examples
>>>
"task_name")
(("task_name"; "Refactor code")), args: (('task_id: (('title': 'Task Id' } 'field': (C'title': 'Field'))]
vroject_management search_tasks, besieett_mnageeeene search_ tasks task_name=None assigned_ter tatee__t__ealllNooee list_ kist_name=None, due due__dat=None,
board=None) Searches for tasks based on the given parameters.
Parameters
task_name str, optional
Name of the task.
essigned_tto_emile str, optional
Email address of the person assigned the task.
list_name str, optional
Name of the list the task belongs to.
due_date str, optional
Due date of the task in "YWYY-MM-DD" format. board str, optional
Name of the board the task belongs to.
30

---

Published as a conference paper at COLM 2024
Returns
tasks dict
Task information for the given parameters.
Examples
>>> sarrect_managemett. searcch progress", "2023-06-01", "Front end")
code", "tishtrya@examlee.com com" "In
(("task_id"; id": "00000000", "task_name" "Refactor code", "assigned_ to_ to__mail"; "tishtrya@example come", "list_name"s "In Progress", "due_date"" "2023-06-01", "board": "Front End")), args: {( (('task_name:: {('title': 'Task Name }}, dessigned_to_email" (('title': 'Assigned To }) 'list_name': {{ ((ttitle': 'List Name '), 'due_date': {('title', 'Due D) 'board': (('title': "Board'' }}}}
projectt_aneen begjjet_mmaagement..cm create_taske project_managemeet ceapmmeetttrtett.eeeeee create_ caate__tsscC paasssttaa_name=Nooom laapendd_t_eeaallleeeeeee list_name=None, due_date=None, board=None) Creates a new task.
Parameters
task_name str
Name of the task.
assigned_to_emm to_ str Email address of the person assigned to the task.
list_name str
Name of the list the task belongs to.
due_date
Due date of the task in "YWYY-MM-DD" format.
board str
Name of the board the task belongs to.
Returns
task_id : str
31

---

Published as a conference paper at COLM 2024
8-digit ID of the new task.
Examples
wwweaeeeetaanInntteeeeeee projeett_anngeettt.rrr......... API service with frontend", "Gam@example.com" "In progress", "2023-06-01", "Front end")
profect__anagement.
"00000001", args: (('task_name': (('title': 'Task Name }),
tessiigne_tt_emiill (('title': 'Assigned To Email'}), 'list_name': (('title': 'List Name }}, 'due_date': {['title', 'Due D)te')), 'board': : (('title':
'Board' }}}}
project__anagemeet wew.aneeeeeeeeeeeeeeeeeeeeeeeeee delete_ croject_managemeet tateeetttsttttkttttteeee Deletes task by ID.
Parameters
task_id str
8-digit ID of the task.
Returns
message str
Message indicating the status of the deletion.
Examples
>>>
"Task deleted successfully." args: {{ id': {{ s(title': 'Task Id (d'))))
project_managementt upwpeeetttudatt__asskks aaaangemnnt.updateetttttttttteoe
field=None, new_value=None) Updates task by ID.
Parameters
task_id str
8-digit ID of the task.
field str
32

---

Published as a conference paper at COLM 2024
Field to update. Available fields are: "task_name", Tassigned_to_emall "list_name", "due_date", "board"
new_value str
New value for the field.
Returns
message str
Message indicating the status of the update.
Examples
>>>
"task_name", "New Task Name
"Task updated successfully." ", args: {( ['skkkddd';; {('title': 'Task Id' }
'field': {('title': 'Field'J }}, 'new_value': (('title': 'New Value }}}}
mustommmr_rrelationhhii...me wewrcchccsttcccrerereeeee...seee wtt.:eaccccccccttoeeooocoooooooa
Mustomer__email=Nonee product_interestt:oon status=None,
follow_up the given parameters.
www_jp_by_minnNoore
follow.up.by.maxxNWe Searches for customers based on
Parameters
customer_name str, optional Name of the customer.
customer_email str, optional Email address of the customer. product_interest str, optional Product interest of the customer.
status str, optional
Current status of the customer.
issigned_tt__eaill str, optional
Email address of the person assigned to the customer.
33

---

Published as a conference paper at COLM 2024
last_contact__ate__tit str, optional
Minimum last contact date. Format: "YYYY-MM-DD"
last_contact_date_ str, optional
Maximum last contact date. Format: "YYYY-MM-DD"
follow_up_by_min str, optional
Minimum follow up date. Format: "WYYY-MM-DD"
follow_up_by_me str, optional
Maximum follow up date. Format: "WYY-MM-DD"
Returns
customers dict
Customer information for the given parameters. Returns at most records.
Examples
>>> crm. (cw.r/uccsssstttuststtteereeeeee
{("customer_id" "00000001", "assigned__to_mma" "sam@example.com "customer_name" "John Smi Smith",
"customer_email "john.sm smi wwwetthpeee.eooom com", "customer_pponee: "123-456-7890", "last_contact_datt"" "2023-01-01",
"product_interrss"" "Software", "status": "Qualified", "follow_up_by" "2023-01- 15", "notes": "Had call on 2023-01-01. "}}, args: {(' custcmerr_nmmee {{
'title': 'Customer Name }}, customer_emaill {('title': "Customer Email }),
ponduuttintteest'.. {('title': 'Product Interest')), 'status': (('title': 'Status }}, basiinne__to_emaill (('title': "Assigned To Email')). llast_contatt_dtt___nnn"" {('title': 'Last Contact Date Min
{('title': 'Last Contact Date Max M), 'follow_up" wwt.:ouupppbbmmii..._p {(' (('title': 'Follow Up By Min' Min')), 'follow_up_by_max" {(
llast_contact_dateeeoo
'title': 'Follow Up By Max })}}
mustomer_relationnee mmaiiier_mmggr...m. update_custone customrr::
cus customerr_rellaionnhii.mm.m paa.mmanager..em wwwaagrr.uddae.ee....... cendate_ccstomen (custocr tomtrtteerroe terstttoerrddeot id==one, field=None, new new_value=None) Updates customer record by ID.
Parameters
customer_id id str
34

---

Published as a conference paper at COLM 2024
ID of the customer.
field str
Field to update. Available fields are: "customer_nae" __mmmr,,___,,,, "assigned_ta" temrre______mma___,__________ "customer_phone", "last_contact_dae coattontttctdtte"",.a___ prrouct_interrest "status", "notes", "follow_up_by"
"customer_ email", petttmmer_mmi"",,
new_value str
New value for the field.
Returns
message str
Message indicating the status of the update.
Examples
>>>
cmm.uppatee_custoorrreo000 "status", "Won")
"Customer updated successfully." " args: ((' : {('title': 'Customer
Id' }), 'field' (('title': "Field' }}, 'new_value': (('title': "New Value }}}}
cus rustomerr_eelatiinshii.mmeee http://aaaaaaaaaaaaaaaaaaaaaaaaa
assigned__o__email=NMoo status=None, cesttmerr_emii-ttie customer__phon==None,
past_contact_date-Nonee product_interest-NNnn. notes="',
Adds new customer record.
Parameters
austomer__ame str Name of the customer.
assigned_ to__mailelllllmo str Email address of the person assigned to the customer.
status str
Current status of the customer. One of: "Qualified", "Won", "Lost", "Lead", "Proposal"
customer_email str, optional
Email address of the customer.
35

---

Published as a conference paper at COLM 2024
customer_phone str, optional
Phone number of the customer.
last_contact_datte str, optional
The last date the customer was contacted. Format: "YWY--NM-DD"
product_interest str, optional
Product interest the customer. One of: "Software", "Hardware", "Services", "Consulting", "Training"
notes str, optional, optional
Notes about the customer.
follow_up_by str, optional
Date for the next follow up. Format: "YYYY-MM-DO"
Returns
customer_id str ID of the new customer.
Examples
>>> crm. goddd_cssteer""" (""aa Smith", "sam@example com", "Lead", "sam. smi th@example com", "123-456-7890", "2023-01-01", "Software")
"00000201", args: ((' cusstomer_nmmee" {('title': 'Customer Name }),, "assigned tot__ttteeaall______ (('title': 'Assigned To Email'J] 'status': (('title': 'Status }}, customer_emaill (('title': 'Customer Email']), "customer_poone' {('title': 'Customer Phone')] ')), bast_contact__att": {( 'title': 'Last Contact Date')), "product_interes'': {('title': 'Product Interest' }), 'notes': (('title': 'Notes', 'default': "}), 'follow_up_by': {(
'title': 'Follow Up By }}}}
customer__eelationshipp.me mustomer_relationnine www.nnssiipaaaaaa....... delett_ccsttmer (customer_se (customer_id-None) Deletes customer record by ID.
Parameters
customer_id id str
ID of the customer.
36

---

Published as a conference paper at COLM 2024
Returns
message str
Message indicating the status of the deletion.
Examples
>>> crm. crm.delete_custent "wm.lelleeeccusttmrr"0000000""""
"Customer deleted successfully.", args: (('customer_id': (('t(tii title': Id' [d]}
Cus tomer
company_directio Loindeemil_addresse company_directory fwnd_eiainnnaieee [aail_sdddess name= Finds the email address of an employee by their name.
Parameters
name str, optional
Name of the person.
Returns
email_address str
Email addresses of the person.
Examples
directory.
"john. smi tw@exappee.eecoom com", args: {(' name":
(('title':
'Name
'default': "}}}))
Use json blob to specify tool by providing action key (tool name) and an action_input key (tool input).
Valid "action" values: "Final Answer" email.get_emaill
send_emll, email. delete_email, email. forward_email, maill.sercchemmill calendar calendar.get_event itte_eeent_innormmtiionnn
calendar wewrch searcheevnttss,, calendar create_eventt calendar.delete.eeee deente_event,
calendar. holeenar.uudate_cee www.uupdddaannnneeeeee
37

---

Published as a conference paper at COLM 2024
amalytics.ge...iittrr... .tt_.viiittr..nnnormttoooo ,id, analytics.trafee trafccsstttrffccccrre......tt
analytics
analytics.total_viiitttt
total_ Sea.yttics.creat__lloo..
arojeett.aangeemmn...crrtt....,t
project_management. www.deeeeeeeeeeeeeeeeeeeeeeeeeee task,
httprageerreeeleeeeettssttteeeee find Company.ddrreettryy.ffddeeeeem_.
Provide only ONE action per $JOON_BLO8, as shown:
[(
"action": NAME,_
paction_input": $INPUT
}}
Follow this format:
Question: input question to answer
Thought: consider previous and subsequent steps
Action:
$JSON_BI BLOB
Observation: action result
(repeat Thought//cttionnOOseevvaiio N times)
Thought: know what to respond
Action:
[(
"action": "Final Answer",
"action_input": "Final response to human"
38

---

Published as a conference paper at COLM 2024
}}
Begin! Reminder to ALWAYS respond with a valid json blob of a single action. Use tools if necessary. Respond directly if appropriate. Format is Action:" JSON_BLOB then Observation:
Thought:
39


Reviews:
[{"review_id": "6sjADFDEtE", "reviewer": "Reviewer_oM5y", "summary": "The paper proposes a new dataset called WorkBench for evaluating the how well agents can perform business-related tasks, such as sending e-mails, given a natural language instruction from the user. The instruction could be simple (such as \"cancel my next meeting with Carlos\") or quite complex (\"if our website page views fell by more than 10%, then schedule a meeting; otherwise, send an email\"). To evaluate whether the agent performed the correct actions, the authors provide a simulator for the actions and propose to measure whether executing the actions results in the correct changes to the environment (i.e. the correct updates to the databases). The authors evaluate the dataset on several language models using the ReAct framework.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The limits of outcome-centric evaluation.** Thank you for the helpful comments about this. Taking them in turn:\n\n  1. We acknowledge that our approach sometimes allows agents to get to the right answer with the wrong reasoning. However, this trades off against a major benefit: our approach enables the agent to correct its own mistakes and choose alternate, valid reasoning paths. Section 3.4 provides an example of where this is beneficial. We will discuss this trade-off explicitly to the paper and provide an example illustrating it.\n\n  2. We agree that our evaluation approach does not allow for pure QA tasks. We note that there are many other QA benchmarks, such as GAIA and ToolQA, in Section 2. A helpful extension of our work would be to add an evaluation method for QA tasks too. We will discuss this further in the final section.\n\n**Multi-turn implementation** would be an interesting extension. We will add a section to the discussion about building on our paper with multi-turn dialogues, while noting that our current implementation still allows for tasks that require planning and multiple actions.\n\n**Linguistic variation in templates** - we agree this is important. While we included 2-3 linguistic variations per template, this was not clearly communicated in the original paper. We included data in Appendix A.3 to justify the template approach, however we did not write about the linguistic variation that we included. In the revised version, we will write about this in the main body of the text, and add examples to the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a timely and relevant topic.\n- Even though the paper focuses on proposing a new dataset, it still evaluates the dataset on multiple models to provide relatively comprehensive results.\n- The authors annotated all of the tasks with a detailed way to measure whether the task was performed correctly, ensuring a more precise evaluation.", "reasons_to_reject": "- The outcome-centric evaluation has several limitations which seem under-studied in this work:\n  - The agents may perform actions which coincidentally happen to result in the same outcomes as what the user intended given the provided databases, but would not produce the desired result when executed in a different environment.\n  - QA-style tasks, not just performing actions, are still important for an agent assisting users in a workplace setting. The outcome-centric evaluation does not seem to be able to cover this use case, because it focuses on evaluating changes to the databases, and not what information is returned to the user.\n- The design of the dataset focuses on complex user requests specified in one turn, which is unnatural; a dialogue-style interaction where the user can use multiple turns, and also correct errors from previous turns, would be better.\n- The use of templates to create the dataset may have resulted in insufficient linguistic variation, which is not discussed in the paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XVXLG8L3VU", "reviewer": "Reviewer_EXCB", "summary": "This paper proposes an evaluation framework for LLMs on goal completion, especially focusing on tool usage. 5 domains are selected, email, calendar, web analytics, crm, and projects with several functions as tools for each. They evaluated GPT, Claude, and other open models using this framework.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Firstly, we thank the reviewer for highlighting these datasets. The main differences to emphasize are 1) our focus on practical business tasks and 2) our tasks have dependencies (e.g. needing to check the calendar before sending an email) which requires planning and actions across multiple domains. None of these papers are in a business context and none of them have tasks with dependencies.  \n\n**MultiWoz** - This dataset involves updating belief states, which could be turned into function calls. However, WorkBench is different because i) We evaluate side effects, which are critical to real-world applications ii) The domains in WorkBench are highly relevant to business environments and iii) Tasks in WorkBench have dependencies that make them challenging. The agent must manage these dependencies.\n\n**SLURP** and **Taskmaster** - these focus on smart-home applications and domestic tasks respectively. While there is some overlap (e.g. \"Make a calendar entry for brunch on Saturday morning\"), many of their tasks are unrealistic for business applications (e.g. \"brighten the lights\"). Furthermore, all tasks in SLURP and TaskMaster are single-step, so none require planning.\n\n**DialoGLUE** - This is a collection of 7 other datasets (MultiWoz is one of them), rather than being a dataset itself. None involve practical business settings AND tasks that require planning across multiple domains.\n\n**Gorilla** - We believe this comment refers to the APIBench dataset introduced in [1], but please correct us if this comment refers to another paper. We cite APIBench in our paper, which evaluates API calling for models on popular ML model repositories rather than business contexts. They also use function-calling accuracy rather than outcome-centric evaluation. Finally, their tasks do not require planning.\n\n**NexusFlow**. We couldn’t find a citation for this work. If possible, could the reviewer please provide us with a citation?\n\nThese papers are all relevant benchmarks for translating natural language tasks into actions. Some even have similar-sounding tasks to our work, such as “Make a calendar entry for brunch on Saturday morning” in SLURP. We will add citations for each of them.  \n\n**On multi-turn dialogues**\n\nWe acknowledge this could be a better format for complex requests. We will add discussion on how future work could build on WorkBench with this.\n\nWe would like to clarify that many WorkBench tasks require multiple function calls, as shown in Figure 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is complete in that it provides a whole benchmark with data, tools, and metrics. It can be a valuable resource for researchers working on goal oriented Conversational AI", "reasons_to_reject": "Interestingly the paper is totally unaware of the existing literature on this, such as the MultiWoz, SLURP, or TaskMaster like datasets, DialoGLUE, Gorilla and NexusFlow leaderboards and for the cited benchmarks they have weak arguments why the community needs Workbench. Furthermore, the framework is noly for single shot function calls, and does not support multi-turn task completion.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j1aqYaXH3w", "reviewer": "Reviewer_No8N", "summary": "This paper proposes WorkBench, a benchmark for evaluating the ability of LLMs to condition on natural language and generate programmatic API calls (\"use tools\") to query and modify workplace-related databases such as calendars, emails, and project management entries. A key aspect of the benchmark is that it evaluates by matching the state of the databases, accounting for the fact that there may be multiple sequences of API calls that can correctly carry out a given task. The paper benchmarks 5 LLMs, both open- and closed-source, using the ReAct prompting strategy, and performs an error categorization.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Human evaluation baseline.** This is hard to implement because humans would use GUIs, rather than APIs, to complete the tasks in WorkBench. To establish a human baseline, we could build a GUI for each tool. However, we would be testing the quality of the GUI too. Ultimately, we felt it was out of scope, but acknowledge this limit and will write about human baselines in the revision.\n\n**# of Multi-step tasks.** More may improve the paper, but the agents’ performance was already low on the current dataset. With our approach, future studies can easily create more multi-step tasks as agents improve.\n\n**Errors without side effects.** 49% are caused by not following the correct ReAct syntax. These are easier to fix than other errors via re-sampling. Fixing all of these would boost GPT-4 up to an upper bound of 59% (43+32 *.49). Re-sampling is a great suggestion and we will implement this in the revision.\n\nOther errors cannot be fixed with resampling. They are logic failures that lead to the agent not taking actions when it should. We will add this example to the revised version to clarify this error type:\n\n**Logic Failure Example** - *(Today's date, Monday 20th November)*\n\nTask: Cancel all my meetings on Tuesday\n\nAgent: Searches for meetings on 28th November. \n\nNo meetings are found, so no meetings are canceled.\n\n**Web browsing papers.** We agree these are relevant, particularly WebArena, as their evaluation has similarities to WorkBench. However, a major difference is they don't capture side effects. They solely evaluate the state of the target, but the agent may take undesired actions that affect other elements. WorkBench evaluates all elements, not just the target.\n\nWe agree our evaluation is a smaller step forward compared to WebArena than compared to other papers we cited. In the revision, we will discuss web browsing papers and highlight WorkBench’s differences, particularly i) multi-step tasks that require planning ii) application domains and iii) side effects.\n\n**Q1)** The ground truth is the entire database. This means we measure side effects occurring anywhere in the sandbox environment, unlike WebArena. We will clarify this in the revised paper.\n\n**Q2)** As discussed above, performance could improve by up to 16% via resampling.\n\n**Q3)** Correct, the agent either: i) Failed to make API calls e.g. because of correct syntax or invalid arguments. or ii) Used the wrong arguments, so it didn’t take actions when it should have (see example above).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1) The benchmark is very well motivated: a benchmark for workplace-oriented tasks is a good step toward developing LLM agents for real-world economically impactful tasks, whereas previous works have mainly focused on consumer tasks or very specialized domains.\n\nS2) The outcome-centric evaluation used here is an important feature of the benchmark, given that there are multiple possible ways that a task could be carried out. This addresses a key limitation of many existing benchmarks that evaluate using API call match.\n\nS3) The paper is overall well-written (although I have a few questions about the evaluation, see below).\n\nS4) The paper evaluated a range of reasonable models and did some interesting initial error analysis of model performance.", "reasons_to_reject": "W1) It's a bit hard for me to gauge the difficulty of the dataset: \n- The paper doesn't perform a human evaluation to see how easy or hard the examples are for people to carry out. This would be very valuable to make sure that there is some headroom beyond the performance currently achieved by GPT-4+ReAct (and see below for a concern about this setup), although given the task descriptions I expect that there will be some headroom.\n- From Figure 4, ~72% of the examples require 0 or 1 actions to solve, and the performance of models on these examples is much higher (from Table 7 in the appendix) than the tasks that require more actions.\n- From Figure 6, I have some doubts about how many of the errors of GPT-4 are serious errors that will be productive for future research to resolve. Many of the errors (32%) are \"Incorrect without side effects\", which if I understand correctly are all failures of the API calls or of the ReAct prompting itself (by simply not generating an ACTION keyword, if I understand Sec 4.4 correctly). It would be helpful to try simple baselines on top of GPT-4 that do things like re-sampling from the model if ReAct or the API calls fail. How much closer would this get to 75% (43+32) accuracy with GPT-4? \n\nHowever, this isn't an absolutely crucial weakness as the benchmark is still challenging for open-source models.\n\nW2) There are some missing comparisons to other works on LLM agent benchmarks / API calling. The below I think are the most relevant given this paper's focus on tool use and task automation in the introduction:\n\n- AgentBench, Liu et al. 2023 (which is cited in this paper, but  in a different context)\n- Mind2Web, Deng et al. 2023\n- WebArena, Zhou et al. 2023 \n- VisualWebArena, Koh et al. 2024 (but was released in Jan 2024, close to the CoLM deadline in March)\n- SMCalFlow, Andreas et al. 2020\n\nIn particular, WebArena and VisualWebArena use test-based evaluation which, like the outcome-centric evaluation proposed in this work, recognizes that multiple action paths can produce a correct outcome. So I don't think that the outcome-centric evaluation (while important!) is a key novelty of this work, if I'm understanding it correctly. But, I don't think this is a crucial weakness, as the focus on workplace-relevant tasks is important.", "questions_to_authors": "Q1) From Figure 3, it seems that the ground-truth outcome of a task is a subset of the items in the database (e.g., all events in the calendar after deleting the named event in the Fig 3 example). Is this correct that the ground-truth is a subset, or is it the entire database? If it's a subset, is the evaluation robust to the agent taking incorrect actions in other parts of the database (e.g. deleting a customer from the CRM database)? How does this interact with the \"side effects\" metric?\n\nQ2) Can you say more about the \"Failed to follow ReAct\" subcategory of Fig 6b? This seems to be a very large fraction of the overall errors of GPT-4 on the benchmark (~16%, from combining Fig6a and Fig6b) and if this were easily fixable then performance of GPT-4 on the benchmark would be quite a bit higher.\n\nQ3) The error analysis in Figure 6 indicates 32% \"Incorrect without side effects\". From the definition of side effects, does this mean that 32% had no effect on the environment (e.g., the API calls failed)?", "extra_scores": {}}, {"review_id": "k6tN98VMxC", "reviewer": "Reviewer_Defo", "summary": "The authors propose a benchmark dataset called WorkBench that consists of a variety of tasks that are typically performed in workplace settings.  Each task is outcome-centric in that the outcome is unique and unambigious to make evaluation easy and consistent. This benchmark is meant to serve as a testbed for agents to see how well they could automate business activities. They have tested 5 existing ReAct agents where Llama-2 performed poorly and GPT-4 performed significantly better and they have illustrated situations where agents can take the wrong action which could lead to disastrous outcomes (for example sending an email to the wrong person).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We thank the reviewer for suggesting many relevant benchmarks, which involve translating natural language tasks into actions. We will add citations in the revised version and will emphasize how WorkBench builds upon them.\n\n**WorkArena** - they have a similar evaluation approach for action tasks. They also propose a robust method for evaluating retrieval tasks, which could help build on our own work. We would like to note the following points about WorkArena:\n\n1) The preprint first went online 12 March 2024, concurrently with our work.\n2) WorkArena tasks focus on using a web browser, so they are often very different to tasks in our paper. For example: their task ‘Navigate to the Open\" module of the \"Problem\" application’ is not a natural-sounding business task for a human.\n3) WorkArena’s evaluation does not capture side effects. They do not link their domains, so tasks are limited to just one domain. The evaluation only accounts for the state of their target, rather than other parts of their system. Our evaluation considers side effects across the whole system.\n\nWe will discuss WorkArena in our revision, highlighting the similarities and differences with WorkBench.\n\n**Mind2web/WebArena/WebShop/MiniWob** - these are relevant papers, with an approach that has similar benefits to our outcome-centric evaluation. Unlike WorkBench, these are focused on web browsing rather than business tasks.\n\nAnother major difference is that these papers do not capture side effects. They solely evaluate the state of the target, so if the agent may take additional actions that are not desired and this would be missed in the evaluation. By comparison, WorkBench evaluates the state of all elements in our sandbox environment, not just the target element, so we can measure side effects too.\n\nWe agree our evaluation is a smaller step forward compared to the web browsing literature than the papers cited in our original submission. In the revision, we will cite these web browsing papers and highlight WorkBench’s differences, particularly i) multi-step tasks that require planning ii) focus on business tasks and iii) capturing side effects.\n\n**WebVoyager** - this is a web browsing paper but without reproducible, automatic evaluation. They use human evaluators, and LLM evaluators, as “questions in our benchmark have open-ended answers”. It is still a relevant paper and we will cite it in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A benchmark focused on business tasks is useful for a wide variety of industries and academics\nThe paper is well written, easy to read, and the figures illustrate the problem well\nThe paper provides a nice analysis on how different agents perform based on the model across different tasks", "reasons_to_reject": "The novelty of the paper is very weak as there are plenty of benchmarks out there that evaluate agents in similar sandbox environments including WorkArena (which tests agents in business settings as well) [1], and other outcome-centric benchmarks such as WebShop [6], MiniWob [4], WebArena [3], Mind2Web[4] , and WebVoyager [2].\nThis work does not include nor cite any of these benchmarks which are highly relevant.\nWhat does this benchmark give that others lack?\n\nIf we have an agent that performs well on these aforementioned benchmarks, do we really need the proposed benchmark?\n\n[1] WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?\n[2] WebVoyager: Buildingan end-to-end web agent with large multimodal models\n[3] Webarena: A realistic web environment for building\nautonomous agents. A\n[4] MiniWob: Reinforcement learning on web interfaces using workflowguided exploration\n[5] Mind2Web: Mind2Web: Towards a generalistagent for the web. \n[6] WebShop:Towards scalable real-world web interaction with grounded language agents.", "questions_to_authors": "Please see the weaknesses above", "extra_scores": {}}]

Author rebuttal:

Meta-review: Reviewers generally agree on the value of the WorkBench challenge set, and the authors have made available additional details about the evaluation in response to reviewer No8N in particular that should be built into the revised paper. The primary weakness of the work raised by reviewers is the lack of situated discussion within the other similar works available before as well as concurrent work that should be acknowledge in a revision. There should be space to clearly situate this work and add these additional experimental details in a revised paper.
Decision: Accept