WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting

Authors: olly  styles
sam  miller
patricio  cerda
mardini  mindsdb
tanaya  guha
victor  sanchez
bertie  vidgen
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 16:21:40

Abstract: We introduce WorkBench: a benchmark dataset for evaluating agents’ ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents’ errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents’ ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/link_updated_upon_acceptance

Main paper: 
 introduction: 
 Large language models (LLMs) excel at a broad range of tasks such as translation, summarisation and sentiment analysis. However, they often fail at tasks such as retrieving niche knowledge or recent information (Bubeck et al., 2023) . For example: they may give a wrong answer for this year's growth of the S&P 500. Proposed approaches to overcoming these limitations include finetuning and Retrieval Augmented Generation (Lewis et al., 2020) , or RAG, which gives LLMs access to data stored as embedded vectors. However, these approaches cannot deal with other LLM failures, such as simple arithmetic problems (Brown et al., 2020) . LLMs' inability to take actions also limits their capabilities. While they can summarise an email for a user, they cannot reply to it. Autonomous agents, powered by LLMs with access to tools, can overcome these limitations. Tools are functions that agents use to interact with their environment. Rather than relying solely on knowledge from training, agents can use tools such as calculators and search engines to broaden their capabilities. With access to external systems, these agents can perform a range of tasks such as replying to emails and booking meetings.While agents are promising, they make mistakes and little is known about their efficacy in practice. Recent benchmark datasets cover tasks like solving logic problems (Mialon et al., 2023) , or playing video games (Wang et al., 2023) . These help understand agents' broader capabilities, but do not represent tasks an agent could be asked to perform in a workplace setting.This paper introduces WorkBench: a benchmark dataset for evaluating agents in a realistic workplace setting. Tasks in WorkBench represent common functions across businesses, such as sending emails. WorkBench includes complex, multi-step tasks that require using several tools. For example: the task in Figure 1 requires an agent to review the user's calendar before sending an email.For tasks in WorkBench, we introduce outcome-centric evaluation. Each task has a unique, unambiguous ground-truth outcome. Agents are evaluated on whether the outcome from their actions matches the ground truth outcome. This is a major step forward in enabling robust and automatic evaluation of action-taking agents.We build a dataset of 690 unique tasks by combining human-curated task templates with a programmatic approach for creating multiple tasks per template. WorkBench tasks are challenging. We evaluate five different agents, finding that a state-of-the-art agent (ReAct with GPT-4) only completes 43% of tasks correctly.To summarise, the contributions of this work are:• An opensource dataset, WorkBench, which enables robust and automatic evaluation of agents in a realistic workplace setting. • Our outcome-centric evaluation methodology for benchmarking agents.• Benchmark results showing what current state-of-the-art agents are capable of. 
literature review: 
 Tool Use Methods. Many prior works have investigated using LLMs as a planning agent for API-based tools (Qin et al. (2023a) provide a survey). Augmenting LLMs with tools enables them to interact with external systems, so end-users can interact with systems through a chat interface. Closed-source LLMs, like GPT-4 (OpenAI et al., 2023) , are assumed to be fine-tuned on tool-usage data, whereas opensource LLMs are generally not. This has led researchers to using closed-source LLMs to generate the training data for opensource LLMs (Liu et al., 2023) . Yao et al. (2023) propose the ReAct framework, which has been highly influential for agent design. This inspires approaches that i) transform the user's intent into a high-level task, ii) make a plan to achieve the task based on available tools, and finally iii) execute that plan (Qin et al., 2023b; Schick et al., 2023; Bubeck et al., 2023) . Lu et al. (2023) show this approach is Other benchmarks have been proposed for specifically evaluating tool usage. Mialon et al. (2023) propose GAIA, which contains 466 questions that test agents on tasks such as web search and coding. Each question has a unique answer for robust, automatic evaluation. Patil et al. (2023) present APIBench for evaluating the effectiveness of LLM agents in calling other AI models accessed through APIs. SLURP (Bastianelli et al., 2020) and TaskMaster (Byrne et al., 2019) focus on smart home tasks that only require a single action. Other benchmarks use simulated environments, such as playing video games (Wang et al., 2023) , human behaviour in a home (Park et al., 2023) , and chat dialogues (Budzianowski et al., 2018) . Xu et al. (2023) propose ToolBench for evaluating the efficacy of LLMs in constructing real API calls for tasks such as travel booking. They generate a large volume of tasks with ChatGPT. Zhuang et al. (2023) propose ToolQA, which combines human task creation with a programmatic approach to scaling the dataset. However, both datasets are limited to information retrieval tasks. Li et al. (2023) propose API-Bank for evaluating agents using tools to take actions. Their automated approach enables a very large number of tools. However, answers are not unique or unambiguous. They require an LLM to evaluate performance of agents, therefore the validity of their evaluation depends on the evaluator LLM not making mistakes. Our proposed outcome-centric evaluation methodology overcomes this issue. Table 1 compares WorkBench against existing datasets.Web Browsing Agents. Another relevant strand of literature is agents for web browsing. Their tools are implicitly defined via possible interactions with a browser (Deng et al., 2023; He et al., 2024; Yao et al., 2022) . Some web browsing benchmarks use text-based evaluation, which shares properties with our evaluation methodology (Zhou et al., 2024; Koh et al., 2024) . Particularly relevant and concurrent with our work, Drouin et al. (2024) also propose a robust method for evaluating retrieval tasks. Our benchmark builds on these papers by i) evaluating side effects from unintended state changes outside the target, and ii) focusing on workplace-relevant tasks.BigBench (Srivavastava et. al, 2023) ✗ ✗ n/a MMLU (Hendrycks et al., 2021) ✗ ✗ n/a HellaSwag (Zellers et al.,  
proposed benchmark dataset: workbench: 
 Figure 2 shows our complete methodology for evaluating agents. Real-world agents typically interact with data using external API calls. However, we do not use data from real APIs such as Gmail, as these require authentication and change over time. Instead, we create a local environment with five sandbox databases. These represent the initial state of the agent's environment. In response to a task, the agent's actions can alter the sandbox databases. All tasks in WorkBench have a unique, unambiguous outcome, which is the expected state of the sandbox databases after successful completion of the task. 3) Task execution: a task is sent to the agent, which has access to toolkits in various domains. The agent takes actions using these tools, which may alter the sandbox databases.The agent observes the result of using the tool to determine if more actions are required. 4) Outcome-centric evaluation: the updated sandbox databases are compared against the ground truth.We create a sandbox environment consisting of five simulated databases:Calendar. Each event has an ID, name, participant email, start time and duration. There are 300 events in the database. See Table 2 for a snippet of Calendar data. We manually create task templates that represent realistic workplace tasks. Templates are split into two categories: 1) Single domain: these only require tools from one domain, such as Calendar, to complete. 2) Multi-domain: these require tools from multiple domains, such as Email and Calendar, to complete.Figure 3 shows how we programmatically create multiple tasks per template, and calculate the associated outcome. Across all domains, we create 10 unique tasks for each template, yielding 690 tasks in total. Table 3 shows the number of templates for each domain in WorkBench. Our templates contain added linguistic variation, so that each task is phrased in three different ways. We provide examples of this variation and empirical justification for our template approach in Appendix A.3.We create some very challenging task templates by combining actions across multiple domains. Figure 4 shows the distribution of the number of actions required to complete Table 3 : Number of unique tasks and templates in each domain. Our template approach creates 690 varied, realistic tasks. Each has an associated unique outcome.Figure 4 : Ground truth number of actions required to complete a task. 18% of tasks require no actions. Sometimes agent would use retrieval tools, but would not need to execute any actions. For example: a request to cancel meetings on a date when there aren't any scheduled. These tasks are easier, which we show in Appendix A.4.each task in our database. Many tasks require multiple actions, with some requiring up to 12 actions. Here is an example of a complex multi-domain task:If our website page views fell by more than 10% in the past week, schedule a 30-minute meeting with Sam called "Urgent Analytics Update" at the earliest free time tomorrow. Otherwise email them saying "Site traffic was stable the past week, nice work."We can define a simple function to automatically find the correct outcome. However, the agent must combine Analytics, Calendar and Email tools to complete this task. Appendix A.2 provides more examples of tasks.Agents execute tasks using 26 tools across the five domains, which are summarised in Table 4 . Each tool consists of a function that interacts with the sandbox databases, and documentation (a docstring) showing the agent how to use the tool. Each docstring contains a high-level description, parameters, return values, an example of tool usage, and any limits. An example limit is the search events(start time, end time), which returns a maximum of five events. If the agent needs to find a large number of events, it must search multiple times with different time windows and then concatenate the results. Appendix A.5 contains the full docstring for each tool.Figure 3 shows that the correct outcome, which is the ground truth, is always known. We then evaluate whether the outcome resulting from the agent's actions matches the ground truth. The ground truth therefore includes the state of all five sandbox databases. We call this methodology outcome-centric evaluation -Figure 5 compares our outcome-centric evaluation against prior works, which evaluate the agent's function calls. Table 4 : Summary of toolkits. We define 26 tools across 5 domains.An agent can follow any action path provided the resulting sandbox databases match the ground truth outcome. For example, sometimes the agent recovers from its error and takes the correct action:Task: Make a task on the Front end board for Sam to improve conversion.Ground truth tool use: create task(name="improve conversion", board="Front end", assigned to="Sam") Agent's tool use: create task(name="improve conversion", board="Front End", assigned to="Sam") Observation: "'Front End' board does not exist, but 'Front end' does..." create task(name="improve conversion", board="Front end", assigned to="Sam")Evaluation methods based on matching the function calls could unfairly find the agent had failed due to the extra calls. However, outcome-centric evaluation recognises that the agent was able to recover because the final change in state matches the ground truth outcome. As a result, the agent is not unfairly penalised.Some prior benchmarks such as Gaia (Mialon et al., 2023) and ToolQA (Zhuang et al., 2023 ) also have tasks with unique outcomes, but they are limited to information retrieval. WorkBench is the first dataset to evaluate tasks that require actions in this manner, due to our outcome-centric evaluation methodology. 
results: 
 We assess the performance of LLM agents using the ReAct framework (Yao et al., 2023) . This enables the LLM to perform multiple action steps and update its action plan based on results from previous steps.Our primary metric is accuracy. This is the % of tasks where the outcome from the agent's actions match the expected outcome, which is the ground truth.Our secondary metric is side effects1 . Some tools have negative consequences if used incorrectly, such as sending emails to the wrong person. If the agent's actions modify the sandbox databases in a way that does not match the ground truth exactly, we consider this a side effect. If the agent fails to complete the task, but does not alter the sandbox databases, then there are no side effects.Table 5 compares five LLMs: GPT-3.5 (Brown et al., 2020) , GPT-4 (OpenAI et al., 2023) , Claude-2 (Anthropic, 2023) , Llama2-70B (Touvron et al., 2023) and Mixtral-8x7B (Jiang et al., 2024) . GPT-4 greatly outperforms other models. For the worse-performing models, the main errors are insufficient context window length and failing to follow the ReAct framework. We propose outcome-centric evaluation, where there is a unique ground-truth outcome for each task (lower panel). We consider the task correctly executed if the predicted outcome following the agent's actions matches this outcome. This allows the agent to find multiple paths to the correct outcome, unlike prior works (upper panel) which evaluate the agent's function calls.GPT-4 GPT-3.5 Claude-2 Llama2-70B Mixtral-8x7B Accuracy (required tools) 49% 14% 23% 3% 20% Accuracy (all tools) 43% 0% 26% 0% 16%Table 5 : Comparing accuracy across all models on WorkBench. We selected these models based on general usage and prevalence in other benchmark studies. Due to limits on context window size, the upper row shows when models are given only the toolkit from the domain needed to complete each task. For example: if the task were "Book a 30-minute meeting with Sam for tomorrow at 9:30", we would provide all tools in the calendar toolkit and none from other toolkits. The lower row is when models are given all 26 tools. Scores of 0% occur when the context window is not long enough to include all 26 tool descriptions.Our benchmark is challenging for all models, including GPT-4. Given how poorly other models perform, we restrict further analysis to GPT-4. When giving this agent all 26 tools, rather than just the required toolkits, and find accuracy falls from 49% to 43%. This suggests the agent is negatively affected by redundant tools, which is consistent with prior studies (Hao et al., 2023) . The next sections explore in further depth why the GPT-4 agent fails. Figure 6a shows the prevalence of side effects. Side effects occur when the agent's actions modify the sandbox environment, but this change does not match the ground truth outcome.In this example from a task in our dataset, the agent cancels the wrong meeting:Task: Cancel my next meeting with Nadia Ground truth: delete event(event id=00000035) Prediction: delete event(event id=00000196)Errors without side effects occur when the agent fails to complete a task, but there are no unintended modifications to the environment. In this example, the agent does not take any actions because it searches on the wrong date: To address this source of error, we implement a re-sampling strategy. If the agent does not correctly generate the ACTION keyword, we repeat the same task up to 5 times. We use a temperature of 0.5 for retries, rather than the initial temperature of 0. With this strategy, the accuracy of the GPT-4 agent increases from 43% to 49%.However, other common sources of error cannot be fixed with resampling. The agent often fails to find the correct email address when given names in the task. The agent may hallucinate an email address rather than using the search tool to find the correct email address. Here is an example, with intermediate steps hidden for concision:Task: Forward all the emails from kofi last week about 'Staff Roster for Next Week' to fatima Ground truth: forward email(email id="0249",recipient="fatima.khan@atlas.com") Prediction: forward email(email id="0249", recipient="fatima@example.com")Errors also come from searching incorrectly. The agent may fail on tasks such as "Cancel my next meeting with Sam" because it searches for events in the past. Similarly, the agent does not always account for the limits of its tools. It can fail on tasks such as "Cancel all future meetings with Sam" because it only cancels a subset of future events after failing to factor in the limit on the number of results returned from searches. The agent could complete this task by repeating the search-and-deletion process, but fails to do so. 
discussion and future work: 
 We have introduced WorkBench -the first benchmark that enables robust, automatic evaluation of agents in a workplace setting. Our approach ensures that each task has a unique outcome, which is the expected change to the state of the sandbox environment upon successful task completion.One limitation of WorkBench is how well the sandbox environment represents real-world complexity. A real email inbox may contain tens of thousands of emails over many years, and emails are often spam, very long, and/or full of errors. Our initial results may therefore overestimate agents' current capabilities, so further studies could improve WorkBench by adding more challenging sandbox data.While our tasks require multiple actions, they are limited to single-turn chat. Some longer queries may not represent realistic workplace interactions. Establishing a human baseline on our tasks would help determine the significance of this, but is hard to implement given our tasks would be completed by humans using a Graphical User Interface (GUI). Implementing a human baseline would measure GUI quality as well as task difficulty. Nevertheless, a multi-turn chat setup may be more representative of real tasks and could build upon our work.We also found the agent's performance dropped when it had a greater number of tools it could choose from, but we were not able to explore this relationship fully with just 26 tools. Future work could extend our benchmark by adding more tools from other real-world domains such as HR software. This would help assess the relationship between agent accuracy and breadth of tooling.A final limitation is that we do not assess pure retrieval tasks. While retrieval tools are a required intermediate step to complete many WorkBench tasks, we do not assess tasks that require solely retrieval such as finding a recent email. Future papers could build on our work with a method for assessing retrieval tasks, such as those proposed by other recent benchmarks (Drouin et al., 2024; Zhuang et al., 2023) .Despite these limitations, WorkBench has a large volume of high-quality, unique tasks. We include complex tasks that require planning, tool selection and analysing results across five domains. WorkBench is challenging, with the best agent achieving only 43% accuracy. We find the main sources of error are the agent failing to execute its plan, giving the wrong arguments to tools, and not understanding the limits of its tools. Furthermore, errors often have negative consequences like sending emails to the wrong people. Future work could study fine-tuning LLMs to improve performance, such as (Schick et al., 2023) .WorkBench is both scalable and extensible. Future researchers could extend our dataset to include new domains, such as accounting tasks, and build an even larger dataset using our scalable approach to task creation. This will enable the evaluation of agents in progressively more complex settings as they continue to improve in the future. The following are 5 randomly sampled tasks from each domain and 5 randomly sampled multi-domain tasks.• Please plot for me the distribution of engaged users and average session duration between October 14 and November 6• Can you make a line plot of the most popular traffic source since November 27?• Was total visits more than 10 at any time in the last 2 weeks? If so, please plot it as a line chart• Can you plot the distribution of both total visits and average session duration between October 12 and November 6?• Can you make a line plot of the most popular traffic source since October 15?• Create a 1.5 hour event called New Employee Onboarding on December 8 at 3:30 with nia• Cancel my next meeting with yuki• Delete the next Annual Budget Planning Session meeting• have I met with carlos in the last 7 days? If not, schedule a 30-minute meeting called 'catch-up' for my first free slot from tomorrow• something came up. Can you cancel my meetings on Friday before 10:30?• Give Sofia all of Lena's customers that are interested in training and are either qualified or in proposal in the crm• I need to move all of Sofia's customers that are interested in training and are either qualified or in proposal to Nadia. Can you make that change in the crm?• Reassign all of Nadia's leads that are interested in training to Lena in the crm.• Move all customers that haven't responded to a proposal for the consulting product in 5 weeks to lost in the crm• Sofia is taking over all of Lena's customers that are interested in services and are either qualified or in proposal. Can you reassign them in the crm?• I need to reply to the latest email from kofi with 'Got it, thank you!'. Can you do that?• can you forward the latest email about 'Task Update on Design logo for blog' to carlos• lena and aisha need the last email about 'Update on Team Building Retreat'. Can you forward it?• Reply to yuki's last email about 'Update on Corporate Social Responsibility Initiative' with 'Thanks for the update -I will get back to you tomorrow.• Delete my last email from chenwei• Move any of luis's tasks that are in review to completed • Give all the overdue tasks that fatima hasn't started to santiago • Move any of nia's tasks that are in review to completed • Give all the overdue tasks that chenwei hasn't started to amir.• can you move any of luis's tasks that are in review to completed?• I need to make sure everyone remembers to attend the first event on December 6. Can you send an email to the attendees with the event name as the title and 'Remember to attend this event.' in the email?• I need to make sure everyone remembers to attend the first event on December 1. Can you send an email to the attendees with the event name as the title and 'Remember to attend this event.' in the email?• please check the percent growth of engaged users since Friday. If it grew by more than average session duration make a front-end backlog task called 'Improve average session duration' for kofi that's due next Friday and schedule a 30 minute meeting called 'Discuss engaged users' for us at the earliest slot i'm free tomorrow• I think carlos might have some overdue tasks. Can you check and if so, send them an email titled 'Overdue tasks' saying 'You have a few overdue tasks -can you update me on them?'. Otherwise email them with 'Nice work keeping on top of your tasks this sprint!' titled 'Good work this sprint'• if fatima hasn't sent me any emails in the past 3 days, schedule a half hour meeting with them for Friday at 12 and call it 'Catch up with fatima'As introduced in Section 3.2, we use templates to create a large number of tasks. Figure 7 shows the percentage of tasks within each template that were completed correctly by the GPT-4 agent. This varies between 10% and 90% for most tasks, indicating that our approach creates a diverse set tasks for each template.We introduce linguistic variation by writing three templates per tasks. The following is an example of the three templates used for a task in the calendar domain:• Cancel all future meetings with {name} • {name} is leaving the company. Can you cancel all future meetings with them?• I need to cancel all future meetings with {name}. Can you do that for me please?Figure 7 : Template dependence. Our approach to task creation using templates creates a diverse set of tasks. The agent achieves 0% or 100% accuracy on just 15 out of 69 templates, indicating that tasks within a template are sufficiently diverse to evaluate agent performance.As shown in Figure 4 , 122 of the 690 tasks in WorkBench do not require any actions to complete. The following is an example of such a task:If I haven't met with akira in the last 7 days, schedule a 30-minute meeting called 'catch-up' for my first free slot from tomorrow Using the calendar.search events tool, the agent can determine that the condition for scheduling a meeting is not met and therefore no action is required. In Table 7 , we compare the accuracy using subsets of WorkBench based on the number of actions required. All tools are provided in the prompt. Table 7 : Performance comparision based on number of actions. All agents perform better on tasks that require no actions. Note that all tools are provided in the prompt, which cannot fit into the context window for GPT3.5 and Llama2-70B.The following is the full prompt template provided to the agent. The description of each tool provided to the agent is in the prompt.Today's date is Thursday, 2023-11-30 and the current time is 00:00:00. Remember the current date and time when answering queries. Meetings must not start before 9am or end after 6pm.Respond to the human as helpfully and accurately as possible. You have access to the following tools:email.get_email_information_by_id: email.get_email_information_by_id( email_id=None, field=None) -Retrieves specific details of an email by its ID.Parameters ----------email_id : str, optional Unique ID of the email.field : str, optional Specific field to return. Available fields: "email_id", "sender", "subject", "sent_date", "body", "inbox/outbox".Returns -------email_information : dict Information of the specified email for the given ID and field.-------->>> email.get_email_information_by_id("12345678", "subject") {{"subject": "Project Update"}} >>> email.get_email_information_by_id("12345678", "sent_date") {{ "sent_date": "2024-01-10 09:30:00"}}, args: {{'email_id': {{'title': 'Email Id'}}, 'field': {{'title': 'Field'}}}} email.search_emails: email.search_emails(query='', date_min=None, date_max=None) -Searches for emails matching the given query across subject, body, or sender fields.The function matches an email if all words in the query appear in any of these fields. --------- [{{"email_id": "12345678", "inbox/outbox": "inbox", "subject": "Project Update", "sender/recipient": "jane@example.com", "sent_datetime": "2024-01-10 09:30:00", "body": "Please find the project update attached. field : str, optional Field to return. Available fields are: "event_id", "event_name", "participant_email", "event_start", "duration" Returns ------event : dict Event information for the given ID and field.-------->>> calendar.get_event_information_by_id("00000000", "event_name") {{"event_name": "Meeting with Sam"}} >>> calendar.get_event_information_by_id("00000000", "event_start") {{"event_start": "2021-06-01 13:00:00"}} >>> calendar.get_event_information_by_id("00000000", "duration") {{"duration": "60"}}, args: {{'event_id': {{'title': 'Event Id'}}, 'field': {{ 'title': 'Field'}}}} calendar.search_events: calendar.search_events(query='', time_min=None, time_max=None) -Returns the events for a given query.---------query: str, optional Query to search for. Terms will be matched in the event_name and participant_email fields. [{{"event_id": "00000000", "event_name": "Meeting with Sam", "participant_email: "sam@example.com", "event_start": "2021-06-01 13:00:00", "duration": "60"}}, {{"event_id": "00000001", "event_name": "Lunch with Sam", "participant_email": "sam@example.com", "event_start": "2021-06-01 13:00:00", "duration": "30}}" ], args: {{'query': {{'title': 'Query', 'default': ''}}, 'time_min': {{'title': Examples -------->>> calendar.create_event("Meeting with Sam", "sam@example.com", "2021-06-01 13: 00:00", "60") "00000000", args: {{'event_name': {{'title': 'Event Name'}}, 'participant_email': {{'title': 'Participant Email'}}, 'event_start': {{ 'title': 'Event Start'}}, 'duration': {{'title': 'Duration'}}}} calendar.delete_event: calendar.delete_event(event_id=None) -Deletes an event. Examples -------->>> analytics.get_visitor_information_by_id("000") {{"date_of_visit": "2023-10-01", "visitor_id": "000", "page_views": "3", "session_duration_seconds": "10.0", "traffic_source": "search engine", "user_engaged": "False"}}, args: {{'visitor_id': {{'title': 'Visitor Id'}}}} analytics.traffic_source_count: analytics.traffic_source_count(time_min=None, time_max=None, traffic_source=None) -Returns the number of visits from a specific traffic source within a specified time range. {{" 2023-10-01": 10.0, "2023-10-02": 20.5, "2023-10-03": 32.8, "2023-10-04": 40.2, "2023-10-05": 5.3, "2023-10-06": 53 Examples -------->>> project_management.search_tasks("Refactor code", "tishtrya@example.com" "In progress", "2023-06-01", "Front end") {{"task_id": "00000000", "task_name": "Refactor code", "assigned_to_email": "tishtrya@example.com", "list_name": "In Progress", "due_date": "2023-06-01", "board": "Front End"}}, args: {{'task_name': {{'title': 'Task Name'}}, 'assigned_to_email': {{'title': 'Assigned To Email'}}, 'list_name': {{'title': field : str Field to update. Available fields are: "customer_name", "assigned_to_email", "customer_email", "customer_phone", "last_contact_date", "product_interest", "status", "notes", "follow_up_by" new_value : str New value for the field. Examples -------->>> crm.add_customer("Sam Smith", "sam@example.com", "Lead", "sam.smith@example.com", "123-456-7890", "2023-01-01", "Software") "00000201", args: {{'customer_name': {{'title': 'Customer Name'}}, 'assigned_to_email': {{'title': 'Assigned To Email'}}, 'status': {{'title': 'Status'}}, 'customer_email': {{'title': 'Customer Email'}}, 'customer_phone': {{'title': 'Customer Phone'}}, 'last_contact_date': {{ 'title': 'Last Contact Date'}}, 'product_interest': {{'title': 'Product Interest'}}, 'notes': {{'title': 'Notes', 'default': ''}}, 'follow_up_by': {{ 'title': 'Follow Up By'}}}} customer_relationship_manager.delete_customer: customer_relationship_manager.delete_customer(customer_id=None) -Deletes a customer record by ID. Examples -------->>> directory.find_email_address_by_name("John") "john.smith@example.com", args: {{'name': {{'title': 'Name', 'default': ''}}}} Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input). Valid "action" values: "Final Answer" or email.get_email_information_by_id, email.search_emails, email.send_email, email.delete_email, email.forward_email, email.reply_email, calendar.get_event_information_by_id, calendar.search_events, calendar.create_event, calendar.delete_event, calendar.update_event, analytics.engaged_users_count, This computer science term refers to a program altering variables outside its local environment. 
 
 Appendix:  


Decision: Reviewers generally agree on the value of the WorkBench challenge set, and the authors have made available additional details about the evaluation in response to reviewer No8N in particular that should be built into the revised paper. The primary weakness of the work raised by reviewers is the lack of situated discussion within the other similar works available before as well as concurrent work that should be acknowledge in a revision. There should be space to clearly situate this work and add these additional experimental details in a revised paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a new dataset called WorkBench for evaluating the how well agents can perform business-related tasks, such as sending e-mails, given a natural language instruction from the user. The instruction could be simple (such as "cancel my next meeting with Carlos") or quite complex ("if our website page views fell by more than 10%, then schedule a meeting; otherwise, send an email"). To evaluate whether the agent performed the correct actions, the authors provide a simulator for the actions and propose to measure whether executing the actions results in the correct changes to the environment (i.e. the correct updates to the databases). The authors evaluate the dataset on several language models using the ReAct framework.
strength_weakness: strength_weakness: - The outcome-centric evaluation has several limitations which seem under-studied in this work:
  - The agents may perform actions which coincidentally happen to result in the same outcomes as what the user intended given the provided databases, but would not produce the desired result when executed in a different environment.
  - QA-style tasks, not just performing actions, are still important for an agent assisting users in a workplace setting. The outcome-centric evaluation does not seem to be able to cover this use case, because it focuses on evaluating changes to the databases, and not what information is returned to the user.
- The design of the dataset focuses on complex user requests specified in one turn, which is unnatural; a dialogue-style interaction where the user can use multiple turns, and also correct errors from previous turns, would be better.
- The use of templates to create the dataset may have resulted in insufficient linguistic variation, which is not discussed in the paper.
paper_summary: paper_summary: This paper proposes an evaluation framework for LLMs on goal completion, especially focusing on tool usage. 5 domains are selected, email, calendar, web analytics, crm, and projects with several functions as tools for each. They evaluated GPT, Claude, and other open models using this framework.
strength_weakness: strength_weakness: Interestingly the paper is totally unaware of the existing literature on this, such as the MultiWoz, SLURP, or TaskMaster like datasets, DialoGLUE, Gorilla and NexusFlow leaderboards and for the cited benchmarks they have weak arguments why the community needs Workbench. Furthermore, the framework is noly for single shot function calls, and does not support multi-turn task completion.
paper_summary: paper_summary: This paper proposes WorkBench, a benchmark for evaluating the ability of LLMs to condition on natural language and generate programmatic API calls ("use tools") to query and modify workplace-related databases such as calendars, emails, and project management entries. A key aspect of the benchmark is that it evaluates by matching the state of the databases, accounting for the fact that there may be multiple sequences of API calls that can correctly carry out a given task. The paper benchmarks 5 LLMs, both open- and closed-source, using the ReAct prompting strategy, and performs an error categorization.
strength_weakness: strength_weakness: W1) It's a bit hard for me to gauge the difficulty of the dataset: 
- The paper doesn't perform a human evaluation to see how easy or hard the examples are for people to carry out. This would be very valuable to make sure that there is some headroom beyond the performance currently achieved by GPT-4+ReAct (and see below for a concern about this setup), although given the task descriptions I expect that there will be some headroom.
- From Figure 4, ~72% of the examples require 0 or 1 actions to solve, and the performance of models on these examples is much higher (from Table 7 in the appendix) than the tasks that require more actions.
- From Figure 6, I have some doubts about how many of the errors of GPT-4 are serious errors that will be productive for future research to resolve. Many of the errors (32%) are "Incorrect without side effects", which if I understand correctly are all failures of the API calls or of the ReAct prompting itself (by simply not generating an ACTION keyword, if I understand Sec 4.4 correctly). It would be helpful to try simple baselines on top of GPT-4 that do things like re-sampling from the model if ReAct or the API calls fail. How much closer would this get to 75% (43+32) accuracy with GPT-4? 

However, this isn't an absolutely crucial weakness as the benchmark is still challenging for open-source models.

W2) There are some missing comparisons to other works on LLM agent benchmarks / API calling. The below I think are the most relevant given this paper's focus on tool use and task automation in the introduction:

- AgentBench, Liu et al. 2023 (which is cited in this paper, but  in a different context)
- Mind2Web, Deng et al. 2023
- WebArena, Zhou et al. 2023 
- VisualWebArena, Koh et al. 2024 (but was released in Jan 2024, close to the CoLM deadline in March)
- SMCalFlow, Andreas et al. 2020

In particular, WebArena and VisualWebArena use test-based evaluation which, like the outcome-centric evaluation proposed in this work, recognizes that multiple action paths can produce a correct outcome. So I don't think that the outcome-centric evaluation (while important!) is a key novelty of this work, if I'm understanding it correctly. But, I don't think this is a crucial weakness, as the focus on workplace-relevant tasks is important.
questions: questions: Q1) From Figure 3, it seems that the ground-truth outcome of a task is a subset of the items in the database (e.g., all events in the calendar after deleting the named event in the Fig 3 example). Is this correct that the ground-truth is a subset, or is it the entire database? If it's a subset, is the evaluation robust to the agent taking incorrect actions in other parts of the database (e.g. deleting a customer from the CRM database)? How does this interact with the "side effects" metric?

Q2) Can you say more about the "Failed to follow ReAct" subcategory of Fig 6b? This seems to be a very large fraction of the overall errors of GPT-4 on the benchmark (~16%, from combining Fig6a and Fig6b) and if this were easily fixable then performance of GPT-4 on the benchmark would be quite a bit higher.

Q3) The error analysis in Figure 6 indicates 32% "Incorrect without side effects". From the definition of side effects, does this mean that 32% had no effect on the environment (e.g., the API calls failed)?
paper_summary: paper_summary: The authors propose a benchmark dataset called WorkBench that consists of a variety of tasks that are typically performed in workplace settings.  Each task is outcome-centric in that the outcome is unique and unambigious to make evaluation easy and consistent. This benchmark is meant to serve as a testbed for agents to see how well they could automate business activities. They have tested 5 existing ReAct agents where Llama-2 performed poorly and GPT-4 performed significantly better and they have illustrated situations where agents can take the wrong action which could lead to disastrous outcomes (for example sending an email to the wrong person).
strength_weakness: strength_weakness: The novelty of the paper is very weak as there are plenty of benchmarks out there that evaluate agents in similar sandbox environments including WorkArena (which tests agents in business settings as well) [1], and other outcome-centric benchmarks such as WebShop [6], MiniWob [4], WebArena [3], Mind2Web[4] , and WebVoyager [2].
This work does not include nor cite any of these benchmarks which are highly relevant.
What does this benchmark give that others lack?

If we have an agent that performs well on these aforementioned benchmarks, do we really need the proposed benchmark?

[1] WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
[2] WebVoyager: Buildingan end-to-end web agent with large multimodal models
[3] Webarena: A realistic web environment for building
autonomous agents. A
[4] MiniWob: Reinforcement learning on web interfaces using workflowguided exploration
[5] Mind2Web: Mind2Web: Towards a generalistagent for the web. 
[6] WebShop:Towards scalable real-world web interaction with grounded language agents.
questions: questions: Please see the weaknesses above

Comments: title: On Linguistic Variation
Thanks for the response!
comment: Thanks for acknowledging the proposed additions will improve the paper.

**On linguistic variation** - we want to provide a specific example of how this is included in the current approach. The below shows two queries from the current dataset, generated from the same template but with linguistic variation:

**Example 1** "carlos is leaving the company. Can you cancel all future meetings with them?"

**Example 2** "Cancel all future meetings with anaya"

We were unable to include this in our last response due to space constraints. However, we think it is important to highlight given it featured as a reason to reject.

We appreciate you taking the time to leave detailed feedback, which will improve the paper.
Thank you for your thoughtful responses to the comments.

I agree that it is highly beneficial to allow the agent to correct its own mistakes as shown in Section 3.4, but there are other ways to avoid penalizing the agent incorrectly (for example, by disregarding calls which have no effect). No single metric will be superior to others in every way so I think it can still be useful to consider having multiple metrics, or discuss the tradeoff in greater detail as you mentioned you would do.

I agree with your other proposed additions as well.
Thanks for the rebuttal. I agree that having dependencies is valuable for dialogue evaluation. I raise my score to 5.
Thanks to the authors for the response, which concisely addressed all of my main concerns. In particular, it's a great point that the proposed approach tests for unintended side effects (which many other benchmarks don't). Thanks, too, for the upper bound on GPT-4 performance and the example of the logic failure example (which, while it too could potentially be fixed with resampling, I agree is a categorically different type of error than the syntax ones) -- these were helpful!

The major remaining limitation in my mind (as the other reviewers also pointed out) is the lack of acknowledgment of related work. But, I expect that this could easily be addressed in the camera ready especially since you have a nice description of the differences from this work in the rebuttals here.

I've raised my score to a 7.