Venue: NAACL 2025

# CRMArena: Understanding the Capacity of LLM Agents to

Perform Professional CRM Tasks in Realistic Environments

Kung-Hsiang Huang Akshara Prabhakar Sidharth Dhawan Yixin Mao Huan Wang Silvio Savarese Caiming Xiong Philippe Laban Chien-Sheng Wu Salesforce AI Research {kh.huang, akshara.prabhakar, sidharth, y.mao, huan.wang, ssavarese, cxiong, wu.jason}@salesforce.com

###### Abstract

Customer Relationship Management (CRM) systems are vital for modern enterprises, providing a foundation for managing customer interactions and data. Integrating AI agents into CRM systems can automate routine processes and enhance personalized service. However, deploying and evaluating these agents is challenging due to the lack of realistic benchmarks that reflect the complexity of real-world CRM tasks. To address this issue, we introduce **CRMArena**, a novel benchmark designed to evaluate AI agents on realistic tasks grounded on professional work environments. We worked with CRM experts to design nine customer service tasks distributed across three personas: service agent, analyst, and manager. We synthesize a large-scale simulated organization, populating 16 commonly-used industrial objects (e.g., account, order, knowledge article, case) with high interconnectivity, and uploading it into a real Salesforce CRM organization. UI and API access to the CRM is provided to systems that attempt to complete the tasks in **CRMArena**. Experimental results reveal that state-of-the-art LLM agents succeed in less than 58% of the tasks with ReAct prompting, and less than 65% even when provided manually-crafted function-calling tools. Our findings highlight the need for enhanced agent capabilities in function-calling and rule-following to be deployed in real-world work environment. **CRMArena** is an open challenge to the community: systems that can reliably complete tasks showcase direct business value in a popular work environment.1

Footnote 1: Our code and benchmark have been released at [https://github.com/SalesforceAIResearch/CRMArena](https://github.com/SalesforceAIResearch/CRMArena).

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 3830–3850 April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Customer Relationship Management (CRM) systems are pivotal in modern enterprises, serving as the backbone for managing interactions with current and potential customers Winer (2001); Payne and Frow (2005). The integration of intelligent agents based on large language models (LLMs) into CRM systems promises to automate routine tasks, enhance operational efficiency, and revolutionize customer experiences. However, evaluating LLM agents in real-world professional environments remains a challenge, due to the absence of robust benchmarks that faithfully capture the complexity of tasks encountered in real-world CRM environments, largely due to data privacy concerns within enterprises.

Prior benchmarks on evaluating LLM agents on work-related tasks, such as WorkArena Drouin et al. (2024), Workbench Styles et al. (2024), and Tau Yao et al. (2024) tend to focus on basic functionality, and fall short in two key areas. First, the complexity of the objects (e.g., tables in databases) and dependencies (e.g., foreign keys) between these objects is often overly simple, lacking the complexity of real-world scenarios. Second, the tasks included in the benchmarks, such as navigating web pages and filtering lists, are typically too straightforward and do not represent real-world work tasks.

To address these limitations, we introduce **CRMArena**, a comprehensive benchmark tailored to evaluate LLM agents on performing realistic CRM tasks in real-world work environments. **CRMArena** features a realistic sandbox environment modeled after Salesforce's schema, developed using an extensible data generation pipeline powered by LLMs (top left of Figure 1). Specifically, the pipeline tackles two key challenges: (1) _Object connectivity:_ reflecting the complex relationships between data objects (e.g., Account associated with Case and Order) by mirroring Salesforce's Service Cloud schema2. (2) Introducing _latent variables_ to better simulate realistic data dynamics, such as influenc

ing case-filing behavior and modeling deviations from company guidelines.

Moreover, CRMArena defines tasks based on actual customer service personas. By consulting CRM experts experienced with Salesforce, we identified nine tasks representative of CRM use cases (SS2.1). These tasks span three personas: Service Manager, Service Agent, and Service Analyst. For example, Service Managers focus on agent performance and strategic resource allocation. Table 1 compares CRMArena with previous datasets.

CRMArena seamlessly integrates with Salesforce,3 enabling interaction via both the user interface and API access (see bottom of Figure 1). This integration facilitated an expert study with CRM professionals to assess the quality of our synthesized organization (SS2.5). Study findings revealed that **90% of domain experts found the test environment to be _Realistic_ or better_**, underscoring the benchmark's fidelity to real-world CRM scenarios. Upon verifying the realism of CRMArena, we then assess various agentic systems through API access. We develop two sets of tools _generalpurpose_ vs. _task-specific_ tools, combine them with three agentic frameworks and various LLMs. Findings indicate that **all LLM agents struggle to reliably complete tasks when using general-purpose tools**, with top performing systems completing less than 40% of the tasks. Incorporating manually designed tools can enhance performance, with top LLM agents solving up to 55% of the tasks. However, we discover that **weaker LLMs often do not benefit from manually-crafted tools due to their weaker function calling capabilities**.

Footnote 3: [https://www.salesforce.com/crm/](https://www.salesforce.com/crm/)

In summary, our main contributions are:

* Introducing CRMArena, a realistic CRM agent benchmark with tasks validated by domain experts to evaluate LLM agents in real-world business scenarios.
* Developing a data generation strategy anchored in a real-world CRM schema, incorporating latent variables, deduplication, and rigorous data validation to ensure diversity and quality.
* Demonstrating through experiments that even state-of-the-art LLM agents do not reliably

Figure 1: An overview of the contribution of this work. We begin by generating realistic CRM data based on the Salesforce Service Cloud schema, ensuring both quality and diversity through rigorous verification processes. This verified data is then stored locally and uploaded to a Salesforce organization (Org). An expert study, conducted with domain experts, validated the data’s realism. Using this Org as a sandbox environment, we create query instances and benchmark various LLMs across different agentic frameworks.

 

complete CRMArena tasks, emphasizing the benchmark's value and challenges.

## 2 CRMArena

Motivated by tasks commonly addressed by CRM personas: service manager, service agent, and service analyst, CRMArena comprises nine tasks that reflect real-world CRM scenarios. Verified by domain experts as common occurrences in CRM, an overview of these tasks is presented in Figure 2. Below, we provide detailed illustrations of each task.

### Tasks

The tasks in CRMArena are designed to accurately reflect the variety of challenges encountered in real-world CRM environments. They span the responsibilities of three key personas: the Service Manager, who focuses on strategic resource allocation; the Service Agent, who addresses customer inquiries; and the Service Analyst, who analyzes data trends and performance metrics to improve service operations.

New Case Routing (NCR)The goal of this task is to assign the best human agent to an incoming case, aiming to optimize various performance metrics. The input consists of a case subject and description, and the output is the ID of the recommended human agent. This task assesses LLM agent's ability to match cases to the most suitable human agent based on case histories and the skills and availability of these agents.

Handle Time Understanding (HTU)This task involves identifying the agent with the shortest/longest average handle time. Given the case history data, the objective is to determine the human agent who handled the previous cases the fastest/slowest.

Transfer Count Understanding (TCU)In this task, the LLM agent must find out which human agent transferred cases to others the least/most given a period of case history. Both HTU and TCU evaluate LLM agent's capacity to analyze performance based on predefined metrics accurately.

Name Entity Disambiguation (NED)The LLM agent must disambiguate named entities related to customer transactions. Here, we focus on disambiguating product names. Given the query shown in Figure 2, the agent needs to identify the specific order corresponding to running shoes bought by the mentioned customer within the given time frame. This tests the understanding of product names and customer order histories.

Policy Violation Identification (PVI)In this task, the LLM agent is given a case with interaction between a customer and an agent and must determine if any company policies have been breached. This involves analyzing the case details and comparing them against policy rules outlined in knowledge articles to identify violations.

Knowledge Question Answering (KQA)The goal here is for the LLM agent to answer a specific question based on knowledge articles. This evaluates the agent's capacity to look for accurate and relevant information from the CRM knowledge repository.

Top Issue Identification (TII)This task requires the LLM agent to identify the most reported issue for a particular product. Given case history, the agent must determine which issue has the highest frequency. This tests the ability to analyze issue reports for trend analysis.

Monthly Trend Analysis (MTA)The LLM agent must determine which months have the highest number of cases for a given product and a given time period. By analyzing the case history in a given period of time, the LLM agent identifies the month with the most cases, demonstrating its ability to recognize trends and patterns over time.

Best Region Identification (BRI)In this task, the LLM agent's objective is to identify the regions where cases are closed the fastest. The agent must analyze case closure times across various regions and indicate which regions perform best.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Datasets** & **\# Objects** & **\# Dependencies/ Object** & **Real-world Environment** & **Realistic Work Tasks** & **Expert Validation** \\ \hline WordBench Styles et al. (2024) & 5 & 0 & ✗ & ✗ & ✗ \\ Tu-Bench Yao et al. (2024) & 3 & 0.67 & ✗ & ✗ & ✗ \\ WuForexma Drouin et al. (2024) & 7 & 0.86 & ✓ & ✗ & ✗ \\ \hline CRMArena Oursa & **16** & **1.31** & ✓ & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 1: **A comparison between our benchmark with prior datasets.**CRMArena is the most complex benchmark given the highest number of objects and object dependencies involved. Furthermore, CRMArena is the only expert-validated benchmark that encompasses both a real-world environment and realistic work tasks.

 



### Sandbox Environment

Creating a sandbox environment for CRMArena poses unique challenges, particularly related to privacy concerns and the need for realistic data without using real enterprise data. To this end, we develop a scalable data generation pipeline capable of producing diverse and realistic CRM data across various domains. In our setup, we model 16 business objects. The complete list of objects and their descriptions can be found in Appendix D. There are two major challenges for building such a pipeline: object connectivity and hidden casual relationship. In the following subsections, we illustrate how we address these challenges.

Object ConnectivityReal-world company data is characterized by complex interconnections between objects like Case and AccountOur data generation approach, based on Salesforce's Service Cloud schema, ensures high connectivity. For instance, the Case object is connected to objects like Account, Contact, and User. Figure 7 displays these interdependencies, creating meaningful data environments. Table 1 highlights our benchmark's much higher object connectivity compared to existing work.

Hidden Causal RelationshipReplicating the implicit causal relationships found in real-world data presents a significant challenge. To address this, we introduce latent variables that simulate various underlying factors, creating data that mirrors the subtle dependencies and patterns in authentic CRM databases. These latent variables are crucial for facilitating certain tasks and generating desired scenarios. As shown in the example in Figure 3, the ShoppingHabit variable allows us to more realistically simulate a customer's purchasing patterns based on time periods or holiday seasons. Similarly, the Skill latent variable for the User (Agent) object enables our simulations of EmailMessage and LiveChatTranscript to include situations where an agent is unable to resolve an issue and must transfer it to another agent. Without this latent variable, we would lack scenarios essential for our Transfer Count Understanding task. The full data generation flow is shown in Figure 8.

Quality and Diversity AssuranceWe generate data in JSON format, with each JSON object representing one entry of an object, to ensure higher controllability [24; 5]. Due to the large volume of objects (e.g., 500 Product entries paired with 40+ Pricebook entries resulting in over 20,000 PricebookEntry items) and the limited maximum output tokens of LLMs, directly prompting LLMs to generate all entries of an object is infeasible. To address this, we employ mini-batch prompting with a batch size of 10. However, this approach can lead to duplicated or highly similar content across batches. To mitigate this issue, we implement a two-phase de-duplication strategy. First, for all objects, we include all previously generated entries in the prompt during mini-batch prompting and instruct the LLM not to generate the same content. After data generation, we use string exact matching to remove duplicate entries for fields and objects crucial to certain tasks (e.g., the email of User).

Additionally, we subject the data to a rigorous quality assurance process involving a dual-layer verification. The _format verifier_ ensures all data entries conform to predefined schemas by checking whether each entry in the generated mini-batch contains all required fields for the object. Mini-batches that fail this verification are discarded and regenerated. The _content verifier_ checks for feasibility for

Figure 2: An overview of the nine distinct tasks introduced in CRMArena.



tasks, focusing on objects crucial for specific tasks. For example, in the Named Entity Disambiguation task, we verify that the paraphrased ambiguous product name (1) does not deviate too much from its original name and (2) is not too similar to other products the customer has purchased. In this scenario, the content verifier provides an LLM with a list of products the customer has purchased and the paraphrased product name. If the LLM correctly identifies the product, we retain the entry; otherwise, it is discarded.4

Footnote 4: We utilize gpt-4o as the LLM for data synthesis and content verification for its cost efficiency.

Upload to OrgOnce the data is generated, we populate it into a clean Simple Demo Org (SDO)5 on Salesforce without latent variables. This exclusion serves two purposes: it mirrors the typical scenario where companies do not have access to such information, thus providing a more realistic testing environment, and it adds an extra layer of challenge compared to testing on the generated databases. Moreover, utilizing Salesforce's SDO as the sandbox eliminates the necessity and complexity for local environment setup, which is required in many related work Styles et al. (2024); Drouin et al. (2024); Yao et al. (2024); Zhou et al. (2024). This approach not only facilitates testing but also encourages scientific rigor and future research on our benchmark. The details of the sandbox environment can be found in Appendix D.

Environment SpecificationThe input to our data generation pipeline are company name, company description, database schema, and the scale of the objects (e.g., number of cases and products). We choose to create an Org for a fictional shoe company due to the diverse product range and customer service scenarios typical in the footwear industry. The scale of our generated data is designed to reflect a mid-sized enterprise, with thousands of orders and hundreds of products and support cases spanning a 4-year period. The total number of entries per object is shown in Appendix D.

ExtensibilityOur data generation pipeline is designed for flexibility and can be easily adapted to other industries through changes in user-specified input parameters. For instance, by specifying the industry in the company description, our pipeline can automatically generate realistic CRM data tailored to that specific industry, such as finance. Furthermore, to accommodate other use cases beyond customer service, such as sales, users would only need to provide the corresponding schema (e.g., Salesforce Sales Cloud schema for sales). This flexibility ensures that the pipeline can be extended to meet a wide range of business needs and LLM agent benchmarking purposes.

Note that our current setup reflects a simplified version of CRM scenarios, where each Case is linked to both an Issue and a Product. This simplification helps manage the complexity of tasks like Top Issue Identification, which would otherwise require LLM agents to individually analyze every case, making the tasks too infeasible for the current state of LLMs. Our benchmark can be adjusted to create more complex settings by removing such dependencies as LLM capabilities advance.

### Query Instance Generation

Following the creation of the sandbox environment, we generate natural language query instances and their ground-truth answers to benchmark our tasks. For the Knowledge QA tasks, queries can be naively constructed by prompting an LLM each knowledge article to generate question answer pairs Laban et al. (2022); Huang et al. (2024). For the remaining tasks, we construct query instance through a four-step process: (1) seed query construction, (2) ground-truth computation, (3) ID mapping, and (4) query paraphrasing.

We manually create 14 seed queries in total with placeholders for corresponding variables, such as time period or product name. This facilitates the development of functions that compute the ground truth answers on the generated database by leveraging the latent variables that are only visible there. For example, an agent's policy violation during a live chat is verifiable only within the generated database. Upon obtaining the answers, we map the IDs in the generated database to their counterparts in Salesforce Org, thereby establishing the ground truths for our queries in the sandbox environment.

Figure 3: Examples of latent variables (gray) influencing object (blue) generation. (a) The ShoppingHabit variable affects when and what type of products a customer buys. (b) The Skill variable determines if an agent can handle a case or needs to transfer it.

 

Finally, to ensure diversity in the test queries, we employ an LLM to paraphrase the seed queries, enhancing the robustness and variety of our benchmarking process. An example of this process is shown in the top right of Figure 1.

Additionally, to simulate real-world scenarios where some questions may be _unanswerable_, we construct non-answerable queries. Inspired by the non-answerable question schema outlined in Brahman et al. (2024), we focus on _False Presuppositions_ queries, which are most relevant in CRM settings. For example, a query may request the identification of an agent who transfers the most cases during a given period, despite no agents transferring cases in that period. We include non-answerable queries in five tasks: Transfer Time Understanding, Handle Time Understanding, Top Issue Identification, Named Entity Disambiguation, and Policy Violation Identification. For these instances, we expect models to produce "None" as outputs. In summary, non-answerable queries account for 30% of the total queries per corresponding task. Overall, we produce 130 query instances per task, totaling 1,170 queries for CRMArena. Details and seed queries are provided in Appendix B.

### Tools: APIs and Functions

Salesforce Orgs naturally support a variety of _general-purpose_ APIs, such as the Apex API, REST API, and Tooling API, which are designed to cover a broad set of functionalities within the Salesforce ecosystem. For the scope of our tasks and their integration with a Python environment, we choose to utilize SOQL and SOSL queries6. SOQL queries are intended for obtaining a specific subset of objects using exact matches or filtering criteria, typically formatted as "SELECT Id ...", while SOSL queries enable fuzzy searching in objects like knowledge articles and product names, formatted as "FIND ...". These two types of queries can theoretically support a wide range of query instances, eliminating the necessity to manually design actions for function calls.

Footnote 6: [https://developer.salesforce.com/docs/atlas.en-us.soql_sscl.meta/soql_sosl/sforce_api_calls_sosl_sosl_intro.htm](https://developer.salesforce.com/docs/atlas.en-us.soql_sscl.meta/soql_sosl/sforce_api_calls_sosl_sosl_intro.htm)

In addition to general-purpose APIs, we also develop _task-specific_ tools in the form of Python wrapper functions to facilitate the evaluation of function-calling agents. These functions optimize task performance by providing structured and logical operations directly mapped to typical CRM tasks. We manually define 27 such Python wrapper functions on top of SOQL and SOSL (complete list in Appendix C) to streamline function calls and target the key components needed for each task. These task-specific functions are designed to maximize reusability across various tasks, minimizing the need for task-specific customizations.

### Expert Study

To ensure the realism and practicality of the sandbox environment we developed, we conducted a user study involving ten experts with diverse professional backgrounds who have experience working on Salesforce Orgs daily. These experts were recruited via the User Interviews platform7. Details of the expert study can be found in Appendix F.

Footnote 7: [https://www.userinterviewsw.com/](https://www.userinterviewsw.com/)

Each session of the expert study was structured into three parts. First, we provided the experts with an overview of our sandbox, highlighting key objects such as Case and Contact, and allowing them access through relevant URLs. This initial orientation was designed to familiarize them with the organization. Second, we assigned them five query instances sampled from CRMArena, each representing a different task, to complete. This task completion phase was aimed at evaluating the practical application and operational coherence of the sandbox in executing real-world CRM tasks. Finally, the experts rated the realism of our Org environment compared to the real-world systems they are accustomed to. They also provided detailed rationales for their ratings, giving insights into how our environment aligns with actual CRM scenarios.

The results of our expert study are presented

Figure 4: Expert study results. The plots illustrate domain experts’ realism ratings for the overall Org structure (top) and individual objects we generated (bottom).

 

in Figure 4. The findings are highly encouraging: **90% of the experts rated our populated Org as either _Realistic_ or _Very Realistic_. This positive assessment extended to the individual objects within the Org, with more than 77% of experts giving them similarly high ratings for realism. These results strongly suggest that our sandbox environment closely mirrors real-world CRM systems. We provide the qualitative feedback and rationale from the experts we interviewed in Table 14.

## 3 Benchmarking Experiments

### Experimental Settings

ModelsWe evaluate state-of-the-art proprietary and open-source LLMs, including gpt models (gpt-4o and gpt-3.5-turbo); claude models (claude-3.5-sonnet and claude-3-sonnet), and the llama models (llama-3.1-4@Sb and llama-3.1-70B Dubey et al. (2024)).8 Additionally, we tested inference-time scaling models for enhanced reasoning capabilities, including o1 and deepseek-r1 Guo et al. (2025). With these models, we tested three common agentic frameworks: Act, ReAct Yao et al. (2023), and Function Calling (FC). ReAct is a prompt-based method, with each step characterized by a _thought_ and _action_ process, while Act is ReAct without the _thought_ component. The details of these settings are described in the following paragraphs and Appendix G.

Footnote 8: We observed the native function-calling mode to perform poorly and hence report the prompt mode performance for Llama 3.1 models.

Action SpaceEvery task can be formulated as a Partially Observable Markov Decision Process (POMDP) \(({\cal U},{\cal S},{\cal A},{\cal O},{\cal T},{\cal R})\) with instruction space \({\cal U}\), state space \({\cal S}\), action space \({\cal A}\), observation space \({\cal O}\), transition function \({\cal T}:{\cal S}\times{\cal A}\rightarrow{\cal S}\), and reward function \({\cal R}:{\cal S}\times{\cal A}\rightarrow\{0,1\}\). In the Act and ReAct settings, the action space is rich, i.e. \({\cal A}=\{\text{execute}<\text{query}>,\text{submit}<\text{result}>\}\). Given a user query \(u\in{\cal U}\) in natural language, an agent can execute <query> to issue a SOQL or SOSL query to interact with the instance to receive the observation \(o_{t}\in{\cal O}\) of executing the query in the environment. This continues until the agent chooses to submit and receives a binary reward \(r={\cal R}(s_{T},\text{submit})\in\{0,1\}\). In the Function Calling setting, the agent interacts with the environment via API tools implemented as Python functions. In this case the agent is not directly exposed to the Salesforce environment and the object dependencies are kept hidden. Internally the APIs interact with the environment in a controlled manner defined by us. An action \(a\) is of the form tool_call{++kwargs}. The system prompts for these three setups are described in Appendix E.

Observation SpaceActions are executed on the sandbox environment through the Simple Salesforce package9. If an action succeeds, the environment will return the queried data in the CRM system; otherwise, an error message, such as incorrect function calling parameters, is returned.

Footnote 9: [https://github.com/simple-salesforce/simple-salesforce](https://github.com/simple-salesforce/simple-salesforce)

Evaluation MetricsFor the knowledge QA task, since it is an open-ended text generation task, we use F1 scores. For the remaining tasks, we only need to compare the predicted and ground truth object IDs; therefore, an exact match is used.

### Results

The main results are summarized in Table 2. We made the following observations. First, **real-world CRM tasks remain challenging for top LLVM agents**. Using the ReAct framework, the best model (o1) only achieves an overall score of 57.7%. Even when equipped with human-crafted functions, the overall performance is still only 64.3%. These findings highlight the challenges of our CRMrena. Second, **stronger and weaker LLMs show opposite trend on different agentic frameworks**. In particular, models like gpt-4o and Claude-3.5-sonnet score higher in the FC setting, while their weaker counterparts performs worse when equipped with function calling capabilities. This indicate that human-defined functions may not always help LLM agents, as weaker models may not be able to properly utilize the functions, resulting in lower performance. An intriguing exception is deepseek-r1. Though deepseek-r1 is recognized as a strong reasoning model, its tool-calling capabilities seem lacking, primarily due to its (1) inadequate adherence to user instructions and (2) poor ability to adjust previous responses based on external feedback. **function calling might be unnecessary with a sufficiently strong reasoning model**, as evidenced by o1 in the ReAct setting outperforming all other models in the FC setting. Nevertheless, integrating human-crafted functions can still offer performance benefits to strong reasoning models like o1. Finally, **open-source models 

are catching up the proprietary LLMs. Across three settings, we see the llama models score similar, and sometimes higher, than the gpt and claude models. This indicate a closing gap between the open and closed-source models. From Figure 6, we observe how llama models tend to show higher scope for error recovery based on execution feedback than the closed-source models.

### Discussions

What is the most cost-effective solution?Excluding the two reasoning models, in two-third of the agentic frameworks, gpt-4o performs the best. The efficiency of gpt-4o is also reflected in Table 3, which shows that gpt-4o has the lowest cost per instance and requires the least number of turns to complete a query. Therefore, the most cost-effective solution is using gpt-4o under the function calling setting.

How does the type of function affect model performance?In Table 2, we observe that equipping LLVM agents with function calling capabilities does not necessary results in increased performance. To better understand this phenomenon, we categorizes the functions based on two dimensions: functionality and functional dependency. Functionality refers to whether the function solely queries the CRM system via SOSL or SOQL (Query) or if it includes additional operations such as mathematical calculations or aggregations (Calculation). Functional dependency, on the other hand, classifies functions into those that rely on the outputs of other functions (Dependent)and those that are independent (Independent). This is crucial because our benchmark requires LLM agents to perform a sequence of calls, with each call dependent on the output of the previous ones Qin et al. (2023); Lu et al. (2024). Table 15 shows the list of functions and tasks we tested.

We sampled four function-task pairs from each category to evaluate the performance of gpt-4o, gpt-4o-mini, and claude-3-sonnet when specific functions were removed from the toolset, sub

\begin{table}
\begin{tabular}{l|r r r r r r r r r|c} \hline \hline Model & NCR & HTU & TCU & NED & PVI & KQA & TII & MTA & BRI & ALL \\ \hline \multicolumn{8}{c}{_Act_} \\ \hline gpt-4o & 43.1 & 10.0 & 17.7 & 30.8 & 28.5 & 29.3 & 68.5 & 29.2 & 7.7 & 29.4 \\ gpt-4o-mini & 0.8 & 38.5 & 23.8 & 9.2 & 0.0 & 43.1 & 26.9 & 3.8 & 3.8 & 16.7 \\ claude-3.5-sonnet & 78.5 & 24.6 & 15.4 & 51.5 & 28.5 & 44.7 & 45.4 & 20.8 & 26.9 & 37.4 \\ claude-3-sonnet & 9.2 & 26.9 & 24.6 & 30.8 & 23.8 & 16.6 & 16.2 & 1.5 & 0.0 & 16.6 \\ llama3.1-405b & 46.2 & 17.7 & 17.7 & 13.9 & 30.0 & 47.0 & 15.4 & 5.4 & 6.9 & 22.2 \\ llama3.1-7bb & 28.5 & 20.0 & 24.6 & 6.2 & 30.0 & 47.9 & 8.5 & 0.0 & 1.5 & 18.6 \\ llama3.1-8b & 0.0 & 3.1 & 0.0 & 6.2 & 4.6 & 4.5 & 2.3 & 0.0 & 1.5 & 2.5 \\ \hline \multicolumn{8}{c}{_ReAct_} \\ \hline gpt-4o & 70.0 & 39.2 & 22.3 & 30.8 & 35.4 & 50.2 & 64.6 & 20.9 & 10.8 & 38.2 \\ gpt-4o-mini & 40.8 & 36.9 & 25.4 & 31.5 & 24.6 & 52.8 & 30.0 & 6.2 & 6.2 & 28.3 \\ claude-3.5-sonnet & 62.9 & 20.0 & 11.5 & 52.3 & 30.0 & 45.0 & 43.9 & 20.8 & 21.5 & 34.3 \\ claude-3-sonnet & 7.7 & 24.6 & 26.9 & 29.2 & 28.5 & 16.0 & 22.3 & 0.8 & 0.0 & 17.3 \\ llama3.1-405b & 81.5 & 22.3 & 15.4 & 33.9 & 34.6 & 55.3 & 34.6 & 13.9 & 13.1 & 33.8 \\ llama3.1-7bb & 48.5 & 20.0 & 13.9 & 33.1 & 37.7 & 48.7 & 23.9 & 13.9 & 10.8 & 27.8 \\ llama3.1-8b & 0.0 & 0.0 & 1.5 & 6.2 & 15.4 & 4.0 & 0.0 & 0.0 & 0.8 & 3.1 \\ ol & 70.0 & 51.5 & 54.6 & 34.6 & 30.0 & 58.8 & **81.5** & 75.4 & 63.1 & **57.7** \\ deepseek-rl & 53.8 & 23.1 & 30.1 & 40.8 & 34.6 & 61.2 & 46.9 & 3.1 & 22.3 & 35.1 \\ \hline \multicolumn{8}{c}{_Function Calling_} \\ \hline gpt-4o & 60.0 & 47.7 & 81.5 & 46.2 & 39.2 & 30.4 & 97.1 & 27.7 & 59.2 & **54.4** \\ gpt-4o-mini & 0.8 & 10.8 & 10.8 & 17.7 & 13.8 & 39.7 & 60.0 & 0.0 & 21.5 & 19.5 \\ claude-3.5-sonnet & 4.6 & 33.1 & 82.3 & 52.3 & 30.0 & 40.5 & 69.2 & 26.9 & 36.9 & 41.8 \\ claude-3-sonnet & 0.8 & 1.5 & 30.0 & 25.4 & 41.5 & 23.2 & 12.3 & 1.5 & 0.0 & 15.1 \\ llama3.1-405b (prompt) & 16.2 & 31.5 & 64.6 & 50.0 & 26.9 & 47.6 & **95.4** & **86.9** & 42.3 & **51.3** \\ llama3.1-7bb (prompt) & 1.5 & 23.1 & 44.6 & 53.8 & 37.4 & 42.4 & 93.8 & 43.8 & 29.2 & 41.1 \\ llama3.1-8b (prompt) & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ o1 (prompt) & 60.8 & 68.5 & 66.9 & 60.0 & 24.6 & 39.2 & 99.2 & 84.6 & 74.8 & 64.3 \\ deepesek-rl (prompt) & 0.8 & 0.8 & 2.3 & 0.8 & 24.6 & 34.6 & 0.0 & 13.8 & 3.1 & 9.0 \\ \hline \end{tabular}
\end{table}
Table 2: Overall performance (%) of various LLMs under different agéntic frameworks on CRMArena. The evaluation metric is F1 score for the knowledge question answering (KQA) task and exact match for all other tasks. ALL represents the average performance across all tasks.

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **Model** & **\# Completion Tables** & **\# Terms** & **Cost (S)** \\ \hline \multirow{3}{*}{ReAct} & gpt-4o & 48.568.73 & 5.4 & 0.182 \\  & claude-3.5-sonnet & 70.814.75 & 6.9 & 0.228 \\  & llama3.1-405b & 35.647.29 & 7.3 & 0.125 \\ \hline FC & gpt-4o & 78.305.38 & 6.8 & 0.305 \\  & claude-3.5-sonnet & 105.248.43 & 8.1 & 0.371 \\ \hline \hline \end{tabular}
\end{table}
Table 3: The cost of top-performing agents averaged across queries and tasks.

 

stituting two generic functions, execute_soql and execute_sosl, to execute arbitrary queries. The findings, summarized in Table 4, indicate that while all function types enhance gpt-4o's performance, they do not have the same effect on gpt-4o-mini or claude-3-sonnet. This suggests that stronger models are better at utilizing human-crafted functions effectively, whereas weaker models might struggle. Interestingly, Calculation functions, hypothesized to benefit LLMs weak in mathematical operations, may actually decrease performance in weaker models due to their limited function calling capabilities.

How consistent are the agents across multiple trials?Consistency is important for LLM agents, especially when deployed in work environments. We evaluate the consistency of LLM agents through multiple trials of prompting. Here, we adapt the pass\({}^{\text{k}}\)k metric proposed by Yao et al. (2024). pass\({}^{\text{k}}\)k computes the probability that all k independent and identically distributed task attempts are successful, averaged over all tasks. We run ten trials across all tasks in CRMArena except for KQA, as the reward for KQA is not binary. The results are shown in Figure 5, we found that, surprisingly, pass\({}^{\text{k}}\)k for all three agentic frameworks we tested drop at the nearly same rate as k increases. This indicates that the consistency for these three frameworks are similar and that the top-performing LLM cannot reliably solve the tasks with any of the three agentic frameworks we evaluated.

## 4 Related Work

Agent BenchmarkSeveral benchmarks have been developed to evaluate LLM-based agents Yao et al. (2022); Liu et al. (2024); Jimenez et al. (2024). Recently, major efforts have focused specifically on web agents, which challenges LLMs to navigate and perform actions on websites. These websites are often about everyday scenarios, such as e-commerce, and social discussion form Deng et al. (2023); He et al. (2024); Zhou et al. (2024); Lu et al. (2024); Yoran et al. (2024). Another line of work focus on evaluating the safety of deploying agents Ruan et al. (2024); Yuan et al. (2024); Yin et al. (2024); Qiu et al. (2025).

Work-oriented DatasetsA few studies have developed datasets specifically for work-oriented tasks. The CRM Benchmark Salesforce (2024) aims to assess LLMs' text generation and summarization abilities in business applications. Workshopn Studies (Styles et al., 2024) consists of five databases designed to evaluate LLM agents' performance in simple work tasks, such as sending emails, creating calendar invites, and counting traffic sources for a website. \(\tau\)-Bench Yao et al. (2024) creates tasks that require interactions with users to obtain relevant information and authorization, achieved by using LLMs to simulate users. Work Arena Drouin et al. (2024) builds a web-based work environment that allows for testing agents with visual capabilities.

## 5 Conclusion

This work introduces CRMArena, a novel benchmark for evaluating LLM agents in performing realistic CRM tasks within professional work environments. By incorporating expert-validated tasks and modeling intricate data interconnections typical of CRM systems, CRMArena offers a comprehensive and realistic challenge for LLM agents. Our experiments demonstrate that even state-of-the-art LLMs struggle with these realistic tasks, achieving limited success rates even with function-calling capabilities. These findings highlight the gap between current LLM capabilities and the requirements of real-world CRM scenarios. CRMArena serves as a foundational step towards more sophisticated evaluations of LLM agents in realistic work environments.

Figure 5: The consistency of gpt-4o using different agentic frameworks.

 

Ethical Considerations

This work introduces a benchmark for evaluating LLM agents within the context of CRM systems. While the data used is synthetically generated, it is modeled after real-world CRM data structures and tasks. Thus, it is important to consider the ethical implications of this work, particularly regarding data biases and privacy concerns.

Data BiasAlthough synthetic, the data is generated by models trained on real-world data, which may contain inherent biases. These biases, related to customer demographics, purchase behavior, or case resolution, could be inadvertently reflected in the generated data, potentially perpetuating stereotypes or discriminatory practices. Thankfully, after conducting a thorough manual inspection of the generated data to identify potential biases, we did not observe such patterns.

Privacy ConcernsWhile our benchmark does not use any real customer data and therefore does not have access to personal information, the structure and nature of CRM data itself can raise privacy concerns. The tasks in our benchmark involve accessing sensitive customer information, albeit synthetic. To ensure responsible handling of this data, even though synthetic, we performed a thorough manual inspection to verify the absence of any personally identifiable information and to confirm that the data cannot be used to infer private information about individuals. This meticulous review process reinforces our commitment to ethical data practices and mitigates potential privacy risks.

## 7 Limitations

The CRMArena comprises nine tasks that thoroughly assess the ability of LLM agents to perform duties typically associated with three primary roles within a realistic environment: Service Manager, Service Agent, and Service Analyst. Nonetheless, this study does not encompass other common personas in CRM, such as sales representatives. We aim to incorporate these additional roles in our future studies.

## References

* Brahman et al. (2024) Faeez Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, et al. 2024. The art of saying no: Contextual noncompliance in language models. arXiv preprint arXiv:2407.12043. Cited by: SS5.
* Deng et al. (2023) Mind2web: towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: SS5.
* A. Drouin, M. Gasse, M. Caccia, I. Laradij, M. Del Verme, T. Marty, D. Vazquez, N. Chapados, and A. Lacoste (2024) Workarena: how capable are web agents at solving common knowledge work tasks?. In Forty-first International Conference on Machine Learning, Cited by: SS5.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahe, A. Lateman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: SS5.
* D. Guo, D. Yang, H. Zhang, J. Song, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llvms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: SS5.
* H. He, W. Yao, Kaixin Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu (2024) WebVoyager: building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangok, Thailand, pp. 6864-6890. External Links: Link, Document Cited by: SS5.
* K. Huang, P. Laban, A. Fabbri, P. K. Choubye, S. Joty, C. Xiong, and C. Wu (2024) Embrace divergence for richer insights: a multi-document summarization benchmark and a case study on summarizing diverse information from news articles. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico, New York, USA, pp. 570-593. External Links: ISBN 978-1-4503-3861-1, Link, Document Cited by: SS5.
* C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan (2024) SWE-bench: can language models resolve real-world github issues?. In The Twelfth International Conference on Learning Representations, Cited by: SS5.
* P. Laban, A. R. Fabbri, C. Xiong, and C. Wu (2024) Summary of a haystack: a challenge to long-context llvms and rag systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Section for Computational Linguistics, 

Philippe Laban, Chien-Sheng Wu, Lidiya Murakhov's ka, Xiang'Anthony' Chen, and Caiming Xiong. 2022. Discord questions: A computational approach to diversity analysis in news coverage. _arXiv preprint arXiv:2211.05007_.
* Liu et al. (2024) Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. Agentbench: Evaluating LLMs as agents. In _The Twelfth International Conference on Learning Representations_.
* Lu et al. (2024) Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2024. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. _Preprint_, arXiv:2408.04682.
* Lu et al. (2024) Xing Han Lu, Zdenek Kasner, and Siva Reddy. 2024. Weblinux: Real-world website navigation with multi-turn dialogue. _arXiv preprint arXiv:2402.05930_.
* Payne and Frow (2005) Adrian Payne and Pennie Frow. 2005. A strategic framework for customer relationship management. _Journal of marketing_, 69(4):167-176.
* Qin et al. (2023) Yujia Qin, Shihao Liang, Ying Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Toollm: Facilitating large language models to master \({}^{1}6000+\) real-world apis. _Preprint_, arXiv:2307.16789.
* Qiu et al. (2025) Haoyi Qiu, Alexander R. Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, and Chien-Sheng Wu. 2025. Evaluating cultural and social awareness of llm web agents. In _Findings of the Association for Computational Linguistics: NAACL 2025_.
* Ruan et al. (2024) Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2024. Identifying the risks of lm agents with an Implantated sandbox. In _The Twelfth International Conference on Learning Representations_.
* Salesforce (2024) Salesforce. 2024. Salesforce announces the world's first llm benchmark for crm.
* Styles et al. (2024) Olly Styles, Sam Miller, Patricio Cerda-Mardini, Tanaya Guha, Victor Sanchez, and Bertie Vidgen. 2024. Workbench: a benchmark dataset for agents in a realistic workplace setting. In _First Conference on Language Modeling_.
* Winer (2001) Russell S Winer. 2001. A framework for customer relationship management. _California management review_, 43(4):89-105.
* Yang et al. (2024) John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2024. Intercode: Standardizing and benchmarking interactive coding with execution feedback. _Advances in Neural Information Processing Systems_, 36.
* Yao et al. (2022) Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webbsp: Towards scalable real-world we interaction with grounded language agents. In _Advances in Neural Information Processing Systems_, volume 35, pages 20744-20757. Curran Associates, Inc.
* Yao et al. (2024) Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narayanhan. 2024. Tau-bench: A benchmark for tool-agent-user interaction in real-world domains. _arXiv preprint arXiv:2406.12045_.
* Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In _The Eleventh International Conference on Learning Representations_.
* Yin et al. (2024) Da Yin, Haoyi Qiu, Kung-Hsiang Huang, Kai-Wei Chang, and Nanyun Peng. 2024. Safeworld: Geo-diverse safety alignment. In _Thirty-eighth Conference on Neural Information Processing Systems_.
* Moran et al. (2024) Ori Moran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. Assistantbench: Can web agents solve realistic and time-consuming tasks? _Preprint_, arXiv:2407.15711.
* Yuan et al. (2024) Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruizjie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhousheng Zhang, et al. 2024. R-judge: Benchmarking safety risk awareness for llvm agents. _arXiv preprint arXiv:2401.10019_.
* Zhou et al. (2024) Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. Weabarena: A realistic web environment for building autonomous agents. In _The Twelfth International Conference on Learning Representations_ 



## Appendix A Further Discussions

Reward vs number of turnsIn Figure 6, we show the distribution of the number of turns it takes for agents to successfully complete an user query.

Non-answerable query analysisIn Table 5, we present the performance of each LLM agents. Overall, LLM agents are good at handling such queries, compared to standard queries. Interestingly, a trend shown in Table 2 is observed in this experiment as well: function calling only benefit stronger LLMs, while weaker LLMs like claude-3-sonnet and gpt-4o performs worse when equipped with function calling capabilities.

## Appendix B Query Generation Details

Table 6 show the complete list of seed queries used in our experiments. More examples of how the final queries are constructed can be found in Table 9.

## Appendix C Action Space Details

In the text-based agent settings (i.e. ReAct and Act), the actions include (1) executing SOSL queries, (2) executing SOQL queries, and (3) submitting the answer. In the function-calling settings, the actions are a list of carefully designed functions, shown in Table 7.

## Appendix D Sandbox Environment Details

We show the objects and dependencies in Figure 7. These objects, except for Knowledge__kav are densely connected, reflecting the complexity of real-world work environment. The total number of entry per objects is shown in Table 8. Our data generation flow is shown in Figure 8.

### Object Details

Below, we describe the details of each object.

* **ProductCategory**: Represents the category that products are organized in.

\begin{table}
\begin{tabular}{l|r r r r r} \hline \hline Model & HTU & TCU & NED & TII & PVI \\ \hline \multicolumn{6}{c}{_Act_} \\ \hline gpt-4o & 15.4 & 48.7 & 94.9 & 87.2 & 92.3 \\ gpt-4o-mini & 94.9 & 79.5 & 30.8 & 79.5 & 74.4 \\ claude-3.5-sonnet & 25.6 & 28.2 & 82.1 & 33.3 & 84.6 \\ claude-3-sonnet & 84.6 & 79.5 & 100.0 & 51.3 & 74.4 \\ llama3.1-4@5b & 56.4 & 51.3 & 46.2 & 38.5 & 0.0 \\ llama3.1-7@b & 46.2 & 76.9 & 20.5 & 20.5 & 100.0 \\ \hline \multicolumn{6}{c}{_ReAct_} \\ \hline gpt-4o & 64.1 & 48.7 & 100.0 & 84.6 & 74.4 \\ gpt-4o-mini & 97.4 & 82.1 & 97.4 & 61.5 & 71.8 \\ claude-3.5-sonnet & 12.8 & 7.7 & 87.2 & 30.8 & 82.1 \\ claude-3-sonnet & 79.5 & 84.6 & 94.9 & 69.2 & 94.9 \\ llama3.1-4@5b & 53.8 & 38.5 & 97.4 & 41.0 & 64.1 \\ llama3.1-7@b & 64.1 & 41.0 & 97.4 & 17.9 & 17.9 \\ \hline \multicolumn{6}{c}{_Function Calling_} \\ \hline gpt-4o & 59.0 & 84.6 & 74.4 & 100.0 & 35.9 \\ gpt-4o-mini & 15.4 & 7.7 & 0.0 & 0.0 & 0.0 \\ claude-3.5-sonnet & 52.6 & 74.4 & 100.0 & 100.0 & 100.0 \\ claude-3-sonnet & 2.6 & 15.4 & 59.0 & 38.5 & 56.4 \\ \hline \end{tabular}
\end{table}
Table 5: Performance (%) of various LLMs under different agentic frameworks on CRMArena for the non-answerable queries.

Figure 6: Distribution of the number of turns it takes for agents to reach the goal under _ReAct_.



* **Product2**: Represents a product that your company sells.
* **ProductCategoryProduct**: Holds the relation between product and product category to assign products to a category.
* **Pricebook2**: Represents a price book that contains the list of products.
* **Pricebook Entry**: Represents a product entry (an association between a Pricebook2 and Product2) in a price book.
* **Order**: Represents an order associated with a contract or an account.
* **Order Item**: Represents an order product that the company sells.
* **Knowledge**: Documentation or information articles that are accessible to users or customers.
* **Contact**: Refers to an individual or party related to an account.
* **Issue**: Represents a type of problem raised by a customer.
* **Account**: An entity, company, or individual your company does business with. In B2C setting, an account represents a customer.
* **User (agent)**: System user, often representing customer support agents.
* **Case**: A record that describes a customer inquiry or issue.
* **CaseHistory**: A log of the changes and updates made to a case over time.
* **EmailMessage**: Email communication related to cases or customer inquiries between an agent and a customer.
* **LiveChatTranscript**: A conversation from a live chat session between an agent and a customer.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Functionals** & **Description** \\ get_agents_stu_th_has_cases(subset_cases) & Returns a list of agent IDs with the maximum number of cases from the given subset of cases. \\ get_agents_stu_th_class(subset_cases) & Returns a list of agent IDs with the minimum number of cases from the given subset of cases. \\ calculate_average_paths(line(case) & Calculate the range based time for each agent based on a list of cases. \\ get_start_distracted data,get_t period,interval_conset_data) & Calculate the start one slot has the end date, period, and interval count. \\ get_port(top/out_for_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_per_perper_per_per_perper_per_perper_per_perper_perper_perper_perper_perper_perper_perperper_ 



## Appendix E Prompts

In this section, we display the prompts used in our experiments. Table 10, Table 11, Table 12 show the system prompt for the Act, ReAct, and Function Calling settings, respectively.



* Technical Support Agent
* Technical Support Expert
* Account Manager/Agent
* Account Manager/Analyst
* Customer Service Advisor/Customer Service Associate
* Customer Service Associate
* Customer Service Representative

In addition, we have created a screener survey. The most important question in the survey is "How often do you use Salesforce CRM?". The valid candidate must select the option "Several times a day" to be able to participate in our study.

### The study

We use Google Form to conduct expert studies due to its ease to use. The study is broken down into three parts:

* **Part 1**: Familiarizing the Org [5 minutes]. This is for a broad overview of some of the objects in this Org.
* **Part 2**: Task Completion [45 minutes]. At this stage, they are be given tasks regarding customer service. They should try to accomplish as many as possible within the 45-minute time frame.
* **Part 3**: Quality Rating [10 minutes]. Based on their experience with the first two parts of this study, rate the quality of the Org and objects.

Below, we illustrate how each part is executed.

Part 1In this part, we provide interviewee the log in credentials to our created Org (sandbox environment). Once they log in, they are instructed to spend 5 minutes to read the objects in the Org that are relevant to the tasks they will be completing later. The instructions and interface for this part are shown in Figure 9.

Part 2After familiarizing with our created Org, participants are then asked to complete the tasks. They are required to complete 5 query instances from CRMArena. An example of the query is shown in Figure 10.

Part 3Upon completing the first two parts of the expert study, in the final stage, participants are asked to rate the realism of our Orgs and data. In addition to providing ratings, they also need to provide rationales for their ratings. An example question is shown in Figure 11.

Below, we provide the rating and descriptions for participants to choose from.

Figure 8: Data generation overview. The generation of each object is conditioned on the previously generated objects with arrows pointing to it. Blue boxes represent standard object, while gray boxes denote latent variables that are not uploaded to the Salesforce Org.

 

You are an expert in Salesforce and you have access to a Salesforce instance.

**Instructions**

- You will be provided a question, the system description, and relevant task context.

- Interact with the Salesforce instance to build Salesforce Object Query Language (SOQL) or Salesforce Object Search Language (SOSL) queries as appropriate, to help you answer the question.

- Salesforce Object Search Language (SOSL) can be used to construct text-based search queries against the search index.

- Your generation should always be an Action command and NOTHING ELSE.

Generate only one Action command.

- DO NOT generate ANY system observation, you will receive this based on your Action command.

- If no record is found matching the requirements mentioned, just return 'None'.

Here is a description of how to use these commands:

**Action**

- Can be 'execute' or 'submit'.

- execute, to execute SOQL/SOSL that will return the

observation from running the query on the Salesforce instance.

- submit, to return the final answer of the task to the user.

- Format: <execute> a valid SOQL/SOSL query </execute> or <submit> response to user </submit>

**Guidelines**

- Execute SOQL/SOSL queries to understand the Salesforce instance

that will help you find the answer to the question.

- When you are confident about the answer, submit it.

- Always end with a submit action containing ONLY the answer, NO full sentences or any explanation.

**Example 1**

Question: What is the total number of opportunities?

Output:

\(<\)execute> SELECT COUNT() FROM Opportunity </execute>

(If the observation from the Salesforce instance 100, your next step can be)

\(<\)submit> 100 _c_/submit> NOT <submit> The total number of opportunities is 100 </submit>

**Example 2**

[... Hide details for space... ]

**Salesforce instance description**

The objects available in the Salesforce instance are:

User, Account, Contact, Case, Knowledge_kav, ProductCategory, Product2, ...

**The fields available for the objects along with their descriptions and dependencies are:**

User

- FirstName: First name of the agent

- LastName: Last name of the agent

- Email: Email address of the agent

[... Hide details for space.... ]

**Additional task context Handle/Transfer Times Policies**

[... Hide details for space... ]

### Org ratings

* Very Unrealistic: The organization structure and setup felt highly artificial, with no resemblance to typical Salesforce setups.
* Unrealistic: The organization had some familiar elements, but significant parts were not convincingly structured.
* Neutral: The organization felt somewhat realistic, with a mix of plausible and implausible elements.
* Realistic: The organization largely mirrored a real-world Salesforce setup, with minor inconsistencies.
* Very Realistic: The organization felt entirely authentic, closely resembling a real-world Salesforce configuration.

#### Object ratings

* I don't know/I'm not familiar with the object.
* Very Unrealistic: The objects seemed fundamentally flawed or invented with little regard for typical Salesforce objects.

\begin{table}
\begin{tabular}{l l}

 

* Unrealistic: The objects had recognizable features but were generally not representative of actual Salesforce objects.
* Neutral: The objects were moderately realistic, combining elements of both realistic and unrealistic features.
* Realistic: The objects were mostly realistic and aligned well with typical objects used in Salesforce, with minor issues.
* Very Realistic: The objects felt entirely authentic and perfectly matched real-world Salesforce objects.

\begin{table}
\begin{tabular}{l l}  & You are an expert in Salesforce and you have access to a Salesforce instance. \\  & **Instructions** \\  & - You will be provided a question, the system description, and relevant task context. \\  & - Think step by step and interact with the Salesforce instance to build Salesforce Object Query Language (SOQL) or Salesforce Object Search Language (SOSL) queries as appropriate, to help you answer the question. \\  & - Salesforce Object Search Language (SOSL) can be used to construct text-based search queries against the search index. \\  & - Your generation should always be a Thought followed by an Action command and NOTHING ELSE. \\  & Generate only one Thought and one Action command. \\  & - DO NOT generate ANY system observation, you will receive this based on your Action command. \\  & - If no record is found matching the requirements mentioned, just return ’None’. \\ Here i is a description of how to use these commands: \\  & **Thought** \\  & - A single line of reasoning to process the context and inform the decision making. \\  & Do not include any extra lines. \\  & - Format: <thought> your thought </thought> \\  & **Action** \\  & - Can be ‘execute’ or ‘submit’. \\  & - execute, we execute SOQL/SOSL that will return the \\  & observation from running the query on the Salesforce instance. \\  & - submitt, to return the final answer of the task to the user. \\  & - Format: <executes> a valid SOQL/SOSL query </execute> or ¡submit¿ response to user e/submit¿ \\  & **Guidelines** \\  & - Always start with a Thought and then proceed with an Action. \\  & - Generate only one Thought and one Action command at a time. \\  & - Execute SOQL/SOSL queries to understand the Salesforce instance \\  & that will help you find the answer to the question. \\  & - When you are confident about the answer, submit it. \\  & - Always end with a submit action containing ONLY the answer, NO full sentences or any explanation. \\  & **Example 1** \\  & Question: What is the total number of opportunities? \\  & Output: \\  & <thought¿ I need to find the total number of opportunities in the system. </thought¿ \\  & <execute¿ SELECT COUNT() FROM Opportunity </execute¿ \\  & (If the observation from the Salesforce instance 100, your next step can be) \\  & <thought¿ I have found the total number of opportunities. </thought¿ \\  & <submit¿ 100 </submit¿ NOT ¡submit¿ The total number of opportunities is 100 </submit¿ \\  & **Example 2** \\  & [... Hide details for space... ] \\  & **Salesforce instance description** \\  & The objects available in the Salesforce instance are: \\  & User, Account, Contact, Case, Knowledge\_kav, ProductCategory, Product2, ... \\  & **The fields available for the objects along with their descriptions and dependencies are:** \\  & User \\  & - FirstName: First name of the agent \\  & - LastName: Last name of the agent \\  & - Email: Email address of the agent \\  & [... Hide details for space.... ] \\  & **Additional task context** \\  & **Handle/Transfer Times Policies** \\  & [... Hide details for space.... ] \\ \end{tabular}
\end{table}
Table 11: The system prompt used in the ReAct setting.

 



### Qualitative Feedback

In Table 14, we present qualitative feedback and rationale from the experts we interviewed, as they determine whether our Organization and Object are perceived as Realistic or Unrealistic.

## Appendix G Implementation Details

We use the OpenAI API for the gpt models; Amazon Bedrock API for the claude models; and the Together API for the llama3.1 models. Below we provide the version of the model we tested:

* o1: o1-2024-12-17
* ggt-4o: gpt-4o-2024-08-06
* ggt-3.5-turbo: gpt-3.5-turbo-0125
* deepseek-r1: deepseek-ai/DeepSeek-R1
* claude-3.5-sonnet: anthropic.claude-3-5-sonnet-20240620-v1:0
* claude-3-sonnet: anthropic.claude-3-sonnet-2024029-v1:0
* llama3.1-405b: meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo
* llama3.1-70b: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo
* llama3.1-8b: meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Profession** & **Gender** & **Age** \\ \hline Customer Service Associate & Female & 23 \\ Customer Service Associate & Female & 25 \\ Customer Service Agent & Male & 39 \\ Customer Service Associate & Male & 29 \\ Customer Service Advisor & Male & 49 \\ Customer Service Manager & Male & 39 \\ Account Executive & Female & 25 \\ Technical Support & Female & 38 \\ Customer Service Advisor & Female & 25 \\ Customer Service Agent & Female & 35 \\ \hline \hline \end{tabular}
\end{table}
Table 13: The background of the participants in our expert study.



We choose the ReAct setting over Plan based approaches that decompose the task into more manageable steps as prior works showed that in SQL based database querying tasks, planning strategy is less flexible to altering its plan by adjusting to execution feedback (Yang et al., 2024). We set the max actions for each instance to 20, temperature to 0, and top_p to 1 for all experiments.

Figure 9: The instructions and interface of Part 1 of our expert study.

 

Figure 10: An example query instance for the part 2 of expert study.

Figure 11: An example question for the part 3 of our expert study.

 



\begin{table}
\begin{tabular}{l l c r} \hline \hline
**Functionality** & **Dependency** & **Function** & **Task** \\ \hline  & & get_order_item_ids_by_product(product_id) & MTA \\ Query & Independent & get_order_item_ids_by_product(product_id) & NCR \\  & & search_products(search_term) & NED \\  & & get_account_id_by_contact_id(contract_id) & NED \\ \hline Query & Dependent & get_non_transferred_case_ids(start_date, end_date) & HTU \\  & & get_cases(start_date, end_date, agent_ids, case_ids, & NCR \\  & & get_cases(start_date, end_date, agent_ids, case_ids, & BRI \\  & & get_cases(start_date, end_date, agent_ids, case_ids, & HTU \\ \hline Calculation & Independent & get_start_date(end_date, period, interval_count) & TCU \\  & & get_start_date(end_date, period, interval_count) & BRI \\  & & get_start_date(end_date, period, interval_count) & TI \\  & & get_start_date(end_date, period, interval_count) & TUI \\  & & get_period(period_name, year) & TCU \\ \hline Calculation & Dependent & calculate_region_average_closure_times(cases) & BRI \\  & & get_qualified_agent_ids_by_case_count(agent_handled_cases, n_cases) & TCU \\  & & calculate_average_handle_time(cases) & HTU \\  & & get_agents_with_max_cases(subset_cases) & NCR \\ \hline \hline \end{tabular}
\end{table}
Table 15: The list of functions and tasks tested in Table 4.