CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs
Authors: Jingzhe Shi
Jialuo Li
Qinwei Ma
Zaiwen Yang
Huan Ma
Lei Li
Venue: COLM (2024)
Keywords: LLM for Customer Service, LLM agents, benchmark
TL;DR: We introduce CHOPS, leveraging LLMs to seamlessly interface with existing customer profile systems, and validate its efficiency through experiments with our proposed dataset, CPHOS-dataset, thereby aiming to enhance customer service performance.

Abstract: Businesses and software platforms are increasingly utilizing Large Language Models (LLMs) like GPT-3.5, GPT-4, GLM-3, and LLaMa-2 as chat assistants with file access or as reasoning agents for custom service. Current LLM-based customer service models exhibit limited integration with customer profiles and lack operational capabilities, while existing API integrations prioritize diversity over precision and error avoidance that are crucial in real-world scenarios for Customer Service. We propose an LLMs agent called **CHOPS** (**CH**at with cust**O**mer **P**rofile in existing **S**ystem) that: (1) efficiently utilizes existing databases or systems to access user information or interact with these systems based on existing guidance; (2) provides accurate and reasonable responses or executing required operations in the system while avoiding harmful operations; and (3) leverages the combination of small and large LLMs together to provide satisfying performance while having decent inference cost. We introduce a practical dataset, *CPHOS-dataset*, including a database, some guiding files, and QA pairs collected from *CPHOS*, which employs an online platform to facilitate the organization of simulated Physics Olympiads for high school teachers and students. We conduct extensive experiments to validate the performance of our proposed **CHOPS** architecture using the *CPHOS-dataset*, aiming to demonstrate how LLMs can enhance or serve as alternatives to human customer service.

Published as a conference paper at COLM 2024
CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs
Jingzhe ShilA, Jialuo Li Qinwei MalA, Zaiwen Yang 1A Huan Ma 14, Lei LF
"singhua University, University of Copenhagen, University of Washington, 4 CPHOS
1 {shi-jz21, lijialuo21, mqw21, yangzw23, mah21) jallle@maillsssiggghheee binghasssn 2 lilei@di.ku.dk
Abstract
Businesses and software platforms are increasingly using large language models (LLMs) such as GPT-3.5, GPT-4, GLM-3, and LLaMa-2 as chat assistants with file access or as reasoning agents for custom service. Cur- rent LLM-based customer service models exhibit limited integration with customer profiles and lack operational capabilities, while existing API in- tegrations prioritize diversity over precision and error avoidance, which are crucial in real-world scenarios for Customer Service. We propose an LLMs agent called CHOPS (CHat with custOmer Profile in existing System) that: (1) efficiently utilizes existing databases or systems to access user in- formation or interact with these systems based on existing guidance; (2) provides accurate and reasonable responses or executing required opera- tions in the system while avoiding harmful operations; and (3) leverages the combination of small and large LLMs together to provide satisfying performance while having decent inference cost. We introduce a practi- cal dataset, CPHOS-datased including database, some guiding files, and
QA pairs collected from CPHOS'. We conduct extensive experiments to validate the performance our proposed CHOPS architecture using the CPHOS-dataset, aiming to demonstrate how LLMs can enhance serve alternatives to human customer service. Code for our proposed architecture and dataset can be found at https: //g/ wwt.tt........ Soobbcccmm/nngghhh//CHHPS..
1 Introduction
In most organizations with human customer service, there is usually system storing customer information. Responses are based on the customer's 's profile, like user type and purchase history, following set guidelines. Customer service can also update the customer's ss status upon request. Large language models (LLMs), such as GPT-3.5 (OpenAI, 2022), GPT-4.0 (OpenAI, 2023), GLM (Zeng et al., 2022; Du al., 2022) and LLaMa (Touvron al., 2023), have emerged as representative achievement of Al development in the past decade. With their mastering of common knowledge and their ability to understand prompts and generate contextually relevant answers, LLMs have been used as assistants across wide range of application scenarios, including chatting assistants, coding assistants, automatic assistant agents, etc (Kalla & Smith, 2023).
common paradigm for equipping LMs with external knowledge while avoiding long context lengths that are costly or technically hard is RAG (Lewis et al., 2021), represented by the widely used and efficient Vector Database using a sentence embedding model such as the Universal Sentence Encoder (Cer et al., 2018). Most publicly available architectures for utilizing LLMs as customer service mainly follow this paradigm, e.g. Databricks (Databricks) enables users to upload guiding files thus building Vector Database-based customer service agent to augment human customer service. However, for most software platforms
*CPHOS is an academic non-profit organization that employs an online platform to facilitate the organization of simulated Physics Olympiads for high school teachers and students. More information at https://cphos..
1

---

Published as a conference paper at COLM 2024
or businesses, to answer based series of guides is not enough: to query information or to manipulate the system using a set APIs is necessary in some scenarios.
Scenarios of Customer Service
Mistakes
Existing LLM helpers
Customer Service
Domain Specific Questions
Specific Questions
1. Go to "My" 2. Click "Add Candidate"
101
GUIDE FILES
Wrong Answers
"My" "Add Candidate"
FILES
QA
RAG, etc
System Specific Instruction
Wrong Operations
Command Exocuted:
System Apis
Various Instructions/ Queries many Apis
Command Executed:
System
>_
ChangeUtertMarknnguubjee
Apis
withargsquser 1d B,
Command Executed: Harmful Operations
Syster Apis
API Hub
Instruction
ToolsLLM, etc
Figure 1: Left: Existing scenarios for Customer Service require File QA and System Ma- nipulation. Middle: Possible mistakes in Customer Service. Accuracy is needed this scenario, especially to avoid those harmful operations. Right: existing methods to use LLM as assistants. LLMs for APIs like TooILLM (Qin et al., 2023) mainly focus on a large number of APIs in API hubs.
Previous works (involving models, agent architectures, and datasets) on LLMs using APIs (Patil et al., 2023; Qin et al., 2023; lang et al., 2023) mainly focus on the LLM's ability to choose between a vast amount (100 through 100,000+) APIs and to accomplish different tasks. However, for particular customer service scenario, much fewer APIs are needed but high accuracy is needed, especially for modifying user status in important aspects like banking. This difference in the focus point of Customer Service and previously defined APl-using tasks sheds light on different method needed for Customer Service compared to previous APl-using methods and new datasets for evaluation of such methods.
Online Customer Service, with its long history since the popularization the Internet and personal computers, requires accurate answers or modifications to user profiles in customer service based on the user's 's questions or requests and the guiding file. Unlike other scenarios for reasoning prompting, especially on math problem solving (Wei et al., 2023; Yao et al., 2023; Zhang et al., 2023), or utilizing numbers of APIs (Patil et al., 2023; Qin et al., 2023; Tang et al., 2023), focus on the new task of customer service, which sheds light the accuracy and admittance of the answer; moreover, the cost of the LLM needs to be controlled.
Previous efforts integrate Language Models (LMs) with databases have focused on generating SQL commands, which poses risks such as incorrect or harmful commands due to LMs' hallucination issues (Huang al., 2023). To address this, we employ APIs for database management, steering clear of direct SQL command generation. Platforms like LangChain (Chase, 2022), Gorilla (Patil al., 2023), and ToolLLM (Qin et al., 2023) offer extensive API libraries, which may exceed the needs of specific customer service applications where the accuracy and proper use limited set of APIs are paramount. Unlike scenarios requiring thousands of APIs, customer service systems prioritize precise and correct AP1 usage, highlighting the importance quality over quantity in API interactions.
Leveraging insights, shown in Figure. 3, from studies on using LLMs to verify responses of architecture, employing a verifier agent to
other LLMs, we introduce an Executor- assess and ensure the validity of commands executed by another LLM, termed the executor agent as Figure 3. This approach aims to enhance the accuracy of integrating LLMs with APIs by reiterating the execution process based on the verifier's 's feedback.
2

---

Published as a conference paper at COLM 2024
To address challenges when user requests involve both API use and information from guid- ing files, we evolved our approach into a Classifier- Exeeere-Eeeeut---eeeeeee
architecture, enhancing efficiency and reducing inference costs. This architecture first classifies user requests to determine if they require access to APIs, guiding files, or both, thus avoiding unnecessary processing and long, redundant texts from guiding files for API-only queries. Additionally, by segmenting guiding files into smaller, more focused documents and employing more nuanced classifier, we further minimize inference costs by ensuring that only the most relevant sections are utilized. This tailored architecture, specifically designed for Customer Service scenarios involving guiding files and user systems, optimizes performance while conserving computational resources.
In response to the lack of datasets for customer service that incorporate details on internal guiding files or systems for APT interactions within the customer service domain, we propose the EPHOS-datasee This dataset, derived from real-world scenarios the Cyber Physics Olympiad Simulations (CPHOS), a non-profit organization, includes databases, guiding files in PDF format, and QA pairs from actual interactions, aiming to bridge the gap in existing resources for LLM research in customer service. Carefully curated and anonymized, the CPHOS-dataset serves as a comprehensive tool for evaluating the effectiveness of LLMs in customer service environments, especially those involving complex systems and guiding documents (Vector, 2018; McAuley; Rajpurkar et al., 2016; Joshi et al., 2017; Patil et al., 2023; Qin et al., 2023; Tang et al., 2023; OpenAl, 2022; 2023).
To this end, we propose a general framework called CHOPS: CHat with custOmer Profile in existing System as Figure shown. In our CHOPS framework, we propose classifier- executor-verifiee based framework that makes the LLM better utilize tools, especially Database and guidance. Through the work of these three agents, the task of answering user's 's question is decomposed into (1) classifying the type theme of the user's 's question, (2) giving answers or decision operations to be executed and (3) verifying then reject or commit the result of the executor.
Our work makes several contributions to the application of LLMs, especially in the field of customer service with LLMs.
We propose framework CHOPS to embed LLM safely, effectively, high-cost- performancely into existing customer service systems.
Our experiments demonstrate that by using weaker LLMs in architecture, can achieve significantly better performance compared to naively using stronger LLMs while saving cost. Moreover, using gpt-4 as Executor while gpt-3.5-turbo as Classifier and Verifier can reach 98% accuracy with decent cost.
We proposed CPHOS-datasel, it is practical dataset collected in real scenarios that can be used to validate methods utilizing LLMs as customer service.
2 Related Works
Retrieval-Augmenntd Generation with LLMs. Incorporating external knowledge sources into LLMs for enhanced performance on knowledgg-intensive tasks has seen advancements Generation (RAG), with the use of vector-based databases for PDF files being notable example (Tripathi, 2023). This approach encodes user queries into vectors, using k-nearest neighbors (KNN) retrieve relevant information. In customer service, LLMs augmented with large databases have aimed to provide encyclopedic support for user inquiries (dat, 2022; Wulf, 2022; cms, 2022). However, such methods sometimes struggle in scenarios requiring modifications to user's profile within existing systemss, crucial aspect of customer service. This highlights the importance of integrating LLMs with software systems for direct interaction tasks, essential for operational efficiency in businesses.
through Retrieval-Augmente
The research on LLMs specialized agents is an evolving field in artificial intelligence and natural language processing. Initial research focused on using predefined prompts or fine-tuning to enhance LLMs for specific tasks, establishing their potential in
LLM Agents.
3

---

Published as a conference paper at COLM 2024
specialized applications like natural language understanding. Recent studies (Hu et al., 2023a; Doe & Smith, 2023) explore LLMs in agent-based architectures for complex problem solving, such mathematical puzzles, highlighting the importance of architectural design in improving performance. This progression suggests new avenues for leveraging LLMs in agent-based systems, offering insight into their capabilities for advanced interaction and problem-solving tasks.
LLMs tools. Recent research has explored the enhancement of LLMs with external tools to improve their performance for a variety of tasks. Studies like OpenAl (2023); Hu et al. (2023b) demonstrate that equipping LLMs with tools, including automated programming interfaces, database management systems, and coding environments, can significantly expand their capabilities. These advances illustrate the potential for LLMs to generate more accurate code, perform advanced database queries, and overall broaden their applicability by leveraging specialized tools.
3 CPHOS-dataset: A real-scene dataset for customer service
The CPHOS- is collected from an online platform of CPHOS, a non-profit organization dedicated to holding Simulated Physics Olympiads online through the online platform as Figure 2 shown.
Change marking
How to add students?
about Andy
>=
subfjcct to 7.
>_
Command Executed:
Command Exccuted:
Go co" My".. 2. Click 'Add Candidate"
admin
ChangsdilTyeccaaa kinguuutte
withargstuser Andy)
User Andl is team leader
Succesofully changed
school
Change my marking subject
about user Andy
>-
How much does CPHOS test cost?
Command Executed:
Command Executied:
Chengadll Syneeaarttteiinnggtugggess witharrgtueerj.d. id.
System Apis
GecToscheranteB/Name withergstuser Andly)
CPHOS tests are free to participate.
GUIDE FILES
because you haven't provided subject
Can't tell since you are not admin...
Figure 2: Dataset Examples include guide file-related QAs on the left; in the middle and right, there are system-related QAs and instructions. For the same query, results may differ based on the Query User Status (middle). Similarly, for the same API, the outcome of calling may vary.
3.1 Database
The online system of the Simulated Olympiad utilizes a MySQL database. We provided data desensitized tables. The detailed description can be found in the appendix A.1.
In short, given user's 's nickname, one can do series of queries on tables in the database to obtain or modify partially the profile of the user. The most important field of user profile includes (0) approved to use online platform; (1) user name; (2) school id; (3) user type: team leader, vise team leader, arbiter; (4) marking-queetion.id etc.
Unlike previous works (Hu et al., 2023a) directly using LLMs to generate SQL commands for query or modification the database, we wrapped the query and modification into series of python APIs, following the idea of Repository Pattern in software design. We provide 9 Data Managing APIs and 18 Data Query APIs, 10 of which are available to LLMs. We collect instructions and queries to the system and augment them with GPT-4 into 104 System-relatte queries and instructions. There are several advantages of wrapping SQL commands and LLMs manipulate databases through these APIs in that:
4

---

Published as a conference paper at COLM 2024
Properly named APIs are much easier for LLMs to understand and to generate compared to complicated table structures and SQL commands for the database.
By limiting APIs LLMs can use, or by checking status inside the APIs, one can prevent unwanted harmful operations that might be carried out by LLMs in extreme conditions.
In codes of software websites that are written following the Repository Pattern (or more broadly, the principle ''encasullation' in software architecture), a series of pre-defined APIs for database is likely to exist. Much less effort is needed modify these APIs into APIs suitable for LLMs than to check for correct SQL commands generated by LLMs.
The diversity of the APIs not only comes from the number of APIs. Calling them from different user and with different arguments would give different results as shown the middle and right part of Figure 2.
3.2 PDF-based guides
There are two main guiding files provided by CPHOS: the mini-program guiding file and questions that commonly asked by users, together with their answers. These QA files, together with the mini-program guiding file, are what we refer to as PDF-based guides. All files are translated by into English. In practice, we merge them into one file for RAG. Full PDF-based guides are appended in the supplementary materials. Collected QA pairs from real scene, we augment them through GPT-4 and repetition into 102 QA pairs on Guide Files, an example of which can be shown in the left part of Figure 2.
Methods
UserText Please
How to
System Apis
Carry Out
>
Final Answer
Operation
Instruction/ System Related Question
UserText & Operation/ Answer generated
UserText
UserText (w.nnallidRason)
Valid?
www.oon.ooo
Invalld Reason
Classifler
Executor
Verlfier
Data
Question
Reason?
Optional
GUIDE FILES
Control
Figure 3: Our CHOPS architecture including Classifer, Executor and Verifier.
Given the previous setup, in Figure 3, we define our task as follows. Given a user's nickname and its question, the task is to give proper answer appropriate execution command to the system, based on the status of the user in the existing system and the guiding files.
4.1 Framework Overview
We propose three-agent architecture for this task, termed the clasiffer-executor-eeiiee architecture. In this architecture, each agent is implemented using a Large Language Model (LLM). This three-stage architecture functions follows:

---

Published as a conference paper at COLM 2024
C: The Classifier is given the UserTexts, the System API descriptions and several relevant (and short) chunks from the guiding files. The Classifier classifies the UserTexts based on information needed for the following pipeline. The classi- fier itself does not output all relevant information but only indicates the type of information that would be beneficial for further processing.
E:
The Executor given the UserTexts and other information that the classifier classi- fies as helpful in the Classifier Stage. Note that the information can be richer than that given to the Classifier (e.g. longer retrieved chunks, more detailed descrip- tions, etc.). The Executor then gives proposed answer API call based on the information given.
V:
The Verifier is responsible for revisiting the Executor's result, verifying if it is valid or not and giving a reason as well. If valid, then the answer is returned or the execution is carried out, and a reply is generated based on the result of that execution. invalid, then the whole process will be redone, while the Classifier and Executor can see the invalid reason provided by the verifier as a reference.
4.1.1 Input Classifier
UserTextType Guide File Query
UserTextType System Query Instruction
UserToxtType Basic Info Query
UserTextType Guide File Query
UserTextType System Query Instruction
Case
Case 2
Case
Case
Case or
C
C1
Classifier
CERRITI For
Graaller FAT
UserText Please set...
UserTexx Please
(Cache Claoitier)
How to
GUIDE FILES
System Apis
GUIDE FILES
System Apis
Data
Info
Control
Figure 4: Classifier Architecture. Left: binary 1-level Classifier. Right: a 2-level Classifier
Previous work (Chen et al., 2023) has shown that the longer and more complicated the retrieved content, the more difficult for the Executor L.M to find the exact piece of information and return it. Also, feeding the Executor every time with retrieved chunks from guide file and API info is token-consuming. We utilize a Classifier to classify the information domain that needs to be given to the Executor in advance to solve these two issues.
simple and experimentally-prooee effective and efficient design for the Classifier binary classifier, only chooses between two categories: (1) that the UsseTextt are about a query to the guiding file, and (2) that the UserTexts
1-level Classifier shown in the left part of Figure 4. This classifier
instruction to the system. Non- classifiable cases, which are not related to the above two cases, are dealt with in the same way as (1) in the following pipeline. Note that although we use the same RAG method to retrieve file chunks for the Classifier and the Executor, the chunk length for Classifier is set to be lower save inference cost.
related to query
Moreover, we observe that many queries and questions are about basic information of the user the system that is much shorter and less token-consuming compared the retrieved chunks. Inspired by the idea of Cache, we further add one categorized: (0) Basic Info' apart from (1) 'Guide File', (2) 'System API. This leads the development of a two-level classifier architecture shown in the right part of Figure 4. One classifier will first decide whether the UserTexts are solvable only given the Basic Information (without the need for 'Guide Files' and 'System APls'). If it asserts yes, then the pipeline will go on and the Executor will only see the Basic Information. If not, then it goes to the second level classifier to categorize UserTexts into class (1) (2), and further provide the corresponding information to the Executor. The experiment indicates this architecture improves accuracy while saving token consumption.

---

Published as a conference paper at COLM 2024
4.1.2 Executor
The Executor needs return an answer or give an appropriate execution command given information provided. In practice, we do prompt-tuning separately for different cases given by the Classifier.
4.1.3 Verifier
Previous works (Weng et al., 2023; Zhang et al., 2023) have proved the effectiveness of Verifier for re-checking correctness.
In our work, the Verifier verifies the result and, if valid, summarizes the answer or generates a response the user based on the executed operation.
Like the Executor, we do prompt tuning separately for different cases given by the Classifier. For the Guide File cases, we feed the retrieved results the Verifier as well.
The Verifier is required to output valid score (1-10) and reasons at the same time. If the verification result invalid, then we redo the whole C-E-V process while giving the Classifier and Executor the invalid reason.
In practice ensure fast response, we restrict the loop iterations into 5. For latter iterations, lower score would be seen as valid: we use simple linear scheduling of the passing score with the iteration index. If all iterations fail, we ask the LLM to choose between one answer provided before. More details about failure cases and number of iterations in practice can be found in Appendix A.4.
4.2 Tools used
For PDF Retrieval, we follow the well-practiced method of using sentence encoder to encode chunks from guide files into a vector database and retrieve top-K closest chunks in latent space as related information given by the files. We follow Pdf-gpt (Tripathi, 2023) and utilize Universal Sentence Encoder (Cer al., 2018) as the embedding model. For Database manipulation, use the wrapped APIs available tools.
Experiments
5.1
Metrics
Our study evaluates the model's performance through several metrics: Instruction Set Accuracy, Guiding File Question Accuracy, and Input/Output character consumed per Question. More details ref Appendix A.3
We make comparison at the character level since different LLMs utilize different tokeniza- tion methods. Input characters and output characters are separated since generating output tokens is more resource-consuming than reading input tokens for LLMs (and more expensive in terms of API price). For clarity, we define the first two metrics mathematically.
Both Instruction Set Accuracy and Guiding File Question Accuracy measured as:
Neorract
Accuracy
(1)
Niotal
Where:
Neorrect is the number of questions correctly answered by the model.
Novtl is the total number of questions and instructions.
We validate whether the answer is correct by a combination of GPT4-based evaluation and human verification afterward.
7

---

Published as a conference paper at COLM 2024
This formula encapsulates the model's efficiency in accurately processing and responding to queries based on the instructions provided or the information contained within the guiding files. The metrics are instrumental in evaluating the model's adeptness at interpreting and acting upon specific sets of instructions and its capacity to extract and utilize knowledge from guiding documents, thereby providing multidimensional view of its performance in realistic scenarios.
5.2 Main Experiment Results
Architecture
LLM
AcCsys
AcCfile
rela. cost.
#char sog in
(k)
#char wrg out (k)
Executor Only
gpt-4*
85.6
83.3
100%
12.9
0.19
1-vote CoT
gpt-3.5-turbo
91.3
62.7
16.9%
14.51
0.5
4-vote CoT
gpt-3.5-turbo
90.2
77.4
61.0%
53.54
0.94
16-vote CoT
gpt-3.5-turbo
90.2
78.4
239.5%
211.58
2.59
C-E-V
gpt-3.5-turbo
95.2
90.2
34.4%
30.1
0.56
C-E-V
Mixed
98.0
99.0
96.6%
16.86+9.79*
0.33+0.21*
Table 1: Main Experiment Result gpt---0125-ppeevvee is used. t: gpt-3.5-turbo is used for Classifier and Verifier, while gpt-4-0125-prvieer is used for Executor. ": gpt-3-turbo characters gpt-4-0125-preview characters. Pricing for calculating relative cost is the price obtained from OpenAl in Mar.2024. See Appendix A.2 for more details about calculating relative cost.
We evaluate the proposed CHOPS agent architecture on the CPHOS -atoast We perform prompt tuning on gpt-4 18pt-4-0125-previets labeled Executor Only) and use multivote CoT (on gpt-3.5-turbo) as our baselines. We show our main experiment results in Table 1.
Compared to the CoT method, our method achieves higher accuracy while maintaining decent cost. Multi-vote CoT does see an improvement in performance when increasing the number of votes, but it cannot provide satisfactory performance even with more cost.
Moreover, proposed C-E-V agent architecture with gpt-3.5-turbo LLM backbones for all agents surpasses the Executor Only gpt-4 baseline by a large margin (86% + 95%, 83% > 90%) with 34.4% cost. Moreover, by substituting the Executor backbone with gpt-4, we can reach accuracy of above 98% on both accuracy metrics, hugely surpassing the plain gpt-4 baseline while even cost less than the gpt-4 baseline (see Appendix A.2 for more details about price estimation). Thus our architecture proven to be flexible and is token-efficient, reaching balance between cost and accuracy.
5.3
Ablation Studies
5.3.1
Effectiveness and Efficiency of our proposed clasifferrxecutorvveifeer architecture
plain Executor, we add all designed blocks and ablate the effectiveness. Finally, we substitute the Executor backbone with gpt-4 and make comparie comparison to plain Executor using gpt-4 as another baseline. Detailed experiment results and figure can be seen Table 2 and Figure 5.
Starting from the baseline using gpt-3.5-turie
Classifier is shown to both reduce token consumption and improve accuracy. A more complicated 2-Level Classifier can further improve accuracy while reducing cost. This free lunch can be explained follows. By adding classifier we can avoid sending long retrieved chunks into the Executor in some scenarios, thus improving RAG accuracy and reducing token consumption for LLMs (Chen et al., 2023). Note by shortening the chunks sent to classifier or by using weaker yet less expensive LLMs as Classifier we can save inference cost.
8

---

Published as a conference paper at COLM 2024
100
100
Arcrgga Accrooy fo 515 and PIL (N)
600
Ralette
60
Executer Only
Executor Only gpo
gpua.tturbo 9ph- Sturho based
got-1--turbo Iwsed
gpe 3--3 Sittrrto hased
Mise
Egot-4
Figure 5: Effectiveness and Efficiency of 2-level Classifier, Executor and Verifier in our proposed CHOPS-architecture Blue dots and lines: average accuracy for AcCsys and Accite Green bar chart: relative cost estimated compared to Executor only with 1pt-4-0125-previine backbone. Baselines: gpt-4-0125-preview and gpt-3.5-turbo.
Architecture
LLM
AcCsys
AcCfill
aug #char
#char oul
aug
in
E (1-L C+)-
gpt-3.5-turbo
38.5
82.4
12.8k
0.16k
gpt-3.5-turbo
90.4
81.2
5.98k
0.11k
E---V
gpt-3.5-turbo
96.1
80.4
32.9k
0.51k
(2-LC**)--V
8pt-3.5-turbo
95.2
90.2
30.1k
0.56k
E
gpt-4
85.6
83.3
12.9k
0.19k
16.86k+
0.33k+
(2-CC*))---V gpt-3.5-turbo + gpt-4"**
98.0
99.0
9.79kt
0.21k
Table 2: Effectiveness and Efficiency of 2-level C-E-V architecture and use of mixing LLMs. APIs of Mar:2024 version used. gpt---0125-previiie model is used for gpt-4. *: 1-Levll Classifier. ** Levell Classifier. *** gpt-3.5-turbo is used for Classifier and Verifier, while gpt-4-0125-preview is used for Executor. t; gpt-3.5-turbb character gpt-4-0125-preview character.
proven effective method improve accuracy previous works (Weng et al., 2023). In our experiments, it has been shown to improve accuracy while consuming more tokens. However, we find that in the case where gpt-4 is used for Executor, weaker and less expensive LLMs (ie. gpt-3.5-turbo) for Verification is enough for the architecture to produce results at very good accuracy (98%), while maintaining a decent cost the same time.
In general, the proposed Classifier and Verifier are dealing with easier tasks but can effec- tively improve accuracy. Even in scenarios where very high accuracy (98%) is required and we have to use more powerful LLMs as Executor, weaker and cheaper LLMs can still used Classifier and Verifier to reduce total cost while achieving satisfactory accuracy.

---

Published as a conference paper at COLM 2024
6 Conclusion
Targeting the important scenario of Customer Service, we have collected, processed related data and proposed our CPHOS-datasel Furthermore, we proposed CHOPS-architecturr, Mlassfier-Executorr- Verifier agent architecture that CHat with custOmer Profile in Systems, offering a flexible architecture for Customer Service scenarios. Our experiments have shown that this architecture (1) improves accuracy while controlling token consumption, achieving better accuracy compared to naively using state-of-the-art LLMs, and (2) provides flexible architecture to utilize different LLMs for agent tasks with different levels of requirements, thus achieving satisfying accuracy with decent cost.
However, though this architecture is flexible and is not domain-specific to Customer Service in Olympiad domain and we expect not hard to apply it to other Customer Service data, more datasets with QA pairs and Database for Customer Service are needed to further evaluate the effectiveness of our CHOPS-architectuue augment our CPHOS datasef based on the guide files, database and APIs we provide, or propose larger real-world datasets targeting the scenario of Customer Service. We are also looking forward to future work that further improves efficiency and effectiveness in the application of Customer Service with LLMs.
We hope future works may further
Acknowledgement
The authors acknowledge the help with the data source from CPHOS (https://cphos. cn), an academic non-profit organization dedicated to providing Physics Olympiad Simulations for high school students for free. Founded in late 2020 by around 10 Physics Olympiad contestants, it now has over 100 members with 1000+ students from 100+ high schools participating in most Olympiads held. The authors also acknowledge members from CPHOS with whom they have had meaningful discussions, including: Xiaoyu Xiong, Xiangchen Tian, Zicheng Huang, Hongyi Liu, Hanyi Li, etc.
In its development process, the project acted as a course project for the NLP course advised by Professor Zhilin Yang and the TAs at Tsinghua University. The authors also acknowledge the valuable advice they provided to the project.
The authors also thank anonymous reviewers, ACs and PCs at COLM who have provided valuable suggestions that helped to improve this work.
References
Customer interactions: Rocking them with Ilms. https://www. http://wwwwwwwiiiiiii//// wwwwwrre.........immm.m.mme.mwoe com/ http://prrreeeeeecuutttteeeeeeee wwwpriionnooooiinnntneeeeeeeeeee th-l1ms/, 2022.
Pioneering a new of automated customer service with large language
models (Ilms). -wwssteeeeeeiieeenn-----e
htttps//www/ www.databbicks..c wwtprriccs..oomoouuttiiiiiii www.soottioos//////ooooooooo accelerators/
2022.
Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder, 2018.
Harrison Chase. LangChain, October 2022. URL https://github. /ttpg/tthbb.commlmgggaaaiaaaaaaa langchain.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmennedd generation, 2023.
Databricks. Databricks: Llms for customer service and support. https: //ww:///wwww /ww. http:///www.aaabbiii...s. databricks. com/ssolutions/accellee. wtt.cceleaattrr//mmmmm. www.stteer-eeiiieeeee support
Jane Doe and John Smith. Exploring agent-based architectures for enhancing large language
model performance on mathematical puzzles. In Proceedings of the International Conference on Artificial Intelligence and Natural Language Processing, pp. 123-130, 2023.
10

---

Published as a conference paper at COLM 2024
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 320-335, 2022.
Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. Chatdb: Augmenting lims with databases as their symbolic memory, 2023a.
Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. Chatdb: Aug- menting ilms with databases as their symbolic memory. arv preprint ar Xix:2200.003901, 2023b.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension, 2017.
Dinesh Kalla and Nathan Smith. Study and analysis of chat gpt and its impact on different fields of study. International Journal of Innovative Science and Research Technology, 8(3), 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensivee nlp tasks, 2021.
Julian McAuley. Recommender systems and personalization datasets. https://cseweb. ucsd. edtu//-jiccaullyydaaaassss html Ntml##maazon_reiies
OpenAL. Chatgpt, 2022. URL https: http:///pppaaa/////a/eeeeeeeee/e wwtp///////o/tt///////ecoooo//ot
OpenAl. Gpt-4 technical report, 2023.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint ar 3rXim230551533.. 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2399 Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.1 (0.18653/vll DD16-111244
URL https:://acaattttttt wwwpoooooyyooggg/eeeee org/ 1264.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang,
11

---

Published as a conference paper at COLM 2024
Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Bhaskar Tripathi. Pdf-gpt. https///ggthtu...mmom//aaaarrrpp
2023.
Though Vector. Customer support on twitter. https://www. compstass/cm//ttttttss///s/ thouggttvetorr/uuusttorrrspppppp towtttttrr/t/tatta, 2018
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models, 2023.
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification, 2023.
Jocelyn Wulf. Pioneering a new era of automated customer service with large language models (Ims). https: //mediumm com/@jocwulf/, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. Xiv preprint ar Xtt::2210.02414, 2022.
Yifan Zhang, Jingqin Yang, Yang Yuan, and Andrew Chi-Chih Yao. Cumulative reasoning with large language models, 2023.
A Appendix
A.1 A Real-Scene dataset: CPHOS: Cyber Physics Olympiad Simulations
In the realm of customer service, datasets such as the Customer Support on Twitter dataset (Vector, 2018), which gathers over 3 million tweets and replies from prominent brands on Twitter, and the Recommender Systems and Personalization Datasets McAuley, which compile a variety of
interactions, ratings, and timestamps, have been instrumental. The core task within customer service be articulated as providing accurate responses or executing specific commands in response to a user's 's queries or directives. This involves leveraging guiding documents or appropriately utilizing APIs. Prior datasets in file-based question answering have concentrated on reading comprehension tasks, as seen in Rajpurkar al. (2016) and Joshi et al. (2017). Meanwhile, datasets focusing on API calls, such as Patil et al. (2023); Qin et al. (2023); Tang et al. (2023), have primarily emphasized the use of extensive API collections for task completion, intricate reasoning, and solving mathematical problems.
However, existing datasets in customer service predominantly lack detailed information about internal guiding documents or systems that can be interacted with. Similarly, datasets dedicated to file QA or API calling seldom address the customer service domain specifically.
To address this gap, we introduce the CPHOS-dalasel derived from the real-world context of the online platform CPHOS: Cyber Physics Olympiad Simulations. This dataset serves as the evaluation ground for our newly proposed CHOPS CHOPP-architecturr, emphasizing its practical application in the CPHOS-datasef.
CPHOS: Cyber Physics Olympiad Simulations is non-profit organization focusing physics education, predominantly operated by approximately 100 college student volunteers, pri- marily from China. It organizes simulated high school Physics Competitions, akin to the International Physics Olympiad (IPhO), attracting around 1000 participants from hundreds of schools annually. The organization utilizes online system and mini-program-bast
12

---

Published as a conference paper at COLM 2024
frontend for operational tasks such as uploading and marking answer sheets. The system's backend is supported by MySQL database, maintaining records on team leaders, vice team leaders, contestants, and examination details. Furthermore, CPHOS offers PDF guides on utilizing the mini-program for administrative purposes. The communication between team leaders and CPHOS liaison members typically relies on this documented information. After thorough data desensitization, the database, guiding documents, and QA pairs offer representative and functional scenario for integrating LLMs into customer service, utilizing existing systems and guiding documents.
The CPHOS-datasef comprises database, several guide files in PDF format, and QA pairs. These components collectively facilitate understanding how team leaders and vice team leaders can navigate the mini-program for various tasks, including uploading and grad- ing answer sheets and accessing student grades. QA pairs are derived from real-world interactions and are further augmented by both human efforts and LLMs (including GPT- 3.5 (OpenAI, 2022) and GPT-4 (OpenAI, 2023)), which enriches the dataset, making it robust resource for validation and experimentation. The dataset's inception, modification, translation, and augmentation have been meticulously undertaken by us, with raw data sourced from CPHOS.
It worth noting that while the dataset is domain-specific, the nature of the queries and instructions it encompasses-rangeee requests for system profile modifications-aal are emblematic of the broader customer service sector. This suggests that methodologies proven effective within this dataset have the potential for broader applicability across various customer service domains, contingent upon the adaptation of guiding documents and APIs to suit specific industry requirements.
from inquiries about the contents of PDF guides to
Database.
There are 9 tables in our database, listed as following Table 3:
table name
Description
Fields
id, p id, user name, school id, subject, status, type, limit, greate-time, nickname
cmf. anf.tp_memt ip_member
all users
umn.tp.admmnn
all admin users
id, user id
cmf schochool
all schools
id, area, school name
cmf.tp-area
all areas of schools
id, area
cmf tp correct
all answer sheets
id, user id, p id, grade, status, create time id, status, title, type, show, create time id, p-id, subject, image, grade, status, create_time
cmf tp exam
all exams
all answer subjects
id, p_id, .udd,userid, student_id id, score, eight, two, create.time
cmf.tp..est. tapper
all test papers
cmf.tp inf.tp.student
all students
id, user id, name, school, grade, prize
Table 3: Tables in the Database
There are 23 APIs. We list few here in the following table 4. Please refer to our provided codes for further information.
User Questions and Instructions. The questions are collected from CPHOS liaison mem- bers, who responsible for responding to (visit) team leaders. Most questions are about the use of the mini-program, and others are about requests to modify the marked problems or add/ modify their status in system.
13

---

Published as a conference paper at COLM 2024
Api
Args
Description
userldiint Namesstr AreaName:tr
add a new school given its name and area by admin
AddNew5ChoolByyNae
change a specific user into Arbiter
MakealllyyeeslokeAAtttee
ChangedUserldinnt
get user info by school name only admin user can call this successfully
userld:int SchoolName:tr
GefTeachernnodyySchoollane
change user's limit for its upload limit
ChangeAlTyyessppooddLitt
userld:int, Limit:in
Table 4: Tables in the Database
A.2 Pricing Estimation
Price information is obtained from OpenAl in March 2024. The prices are listed in the following Table 5.
model name
Input Price per 1M Token
Output Price per 1M Token
gpt-3.5-turbo gpt-4-0125-previer
$1.5
$2 $30.0
$10.0
Table 5: Pricing from OpenAI
Since token number is approximately proportional to the character number, we approximate the tokens by token_num R k char_num. Therefore, the final cost would be cost R output.char.num price-per.output...o .otpuut.oooot)) Since the gpt families are believed to use the same tokenization method, we use the same for gpt-3.5-turbo and spt-4-0125-previee to calculate their cost.
k* kinput.char-nunp
priceeper.input.ooke
A.3 Metrics
We make comparison at the character level since different LLMs utilize different tokeniza- tion methods. Input characters and output characters are separated since generating output tokens is more resource-consuming than reading input tokens for LLMs (and are more expensive in terms of API price). For clarity, we define the first two metrics mathematically.
Both Instruction Set Accuracy and Guiding File Question Accuracy measured as:
Neorract
Accuracy
(2)
N
Where:
Neorrett is the number of questions correctly answered by the model.
Niotal is the total number of questions and instructions.
We validate whether the answer is correct by a combination of GPT4-based evaluation and human verification afterward.
This formula encapsulates the model's 's efficiency in accurately processing and responding to queries based on the instructions provided the information contained within the guiding files. The metrics are instrumental in evaluating the model's adeptness at interpreting and acting upon specific sets of instructions and its capacity to extract and utilize knowledge
14

---

Published as a conference paper at COLM 2024
from guiding documents, thereby providing multiddmmensional view of its performance in realistic scenarios.
A.4 Failure and Iteration Number
In experiments we repeat the iteration for at most times before giving all previous answers to a summarizer for final answer. In practice, the first iteration will give the correct answer (and also terminates) in more than half cases, two iterations significantly boost accuracy for system instructions, and three iterations are enough for file-based QAs in most cases. At Table 6 we record the number of correct (positive) and wrong (negative) answers per iteration and the transition between iterations.
iteration index
1st
2nd
3rd
4th
5th
(Td(TP)
151
23
#pos in ith step/#pos *pos
168
13
30
6
14
5
4
4
#pos nee
#neg- >pos
17
8
#neg in ith step step/#neg >neg
38
18
22
10
11
7
7
end(FP)
Table 6: Number of Pos/ Pos/Neg in each iteration and transition between iterations.
A.5
More LLMs
We further do experiments on the robustness of our proposed architecture on different LLMs (GLM-3, llama-2-70b). We find hard to restrict GLM or LLaMa-based verifier to output results in the regulated format, hence we set the Verifier and Classifier into gpt-3.5-turbo and mainly focus on experimenting with the effectiveness of substituting the LLM backbone for the Executor. Please refer to Table 7 for the results.
As shown by the experiment, GLM-3 can provide rather decent performance in this case while 2-70-hht shows some difficulty generating answers in wanted format.
Executor Backbone
ACCsys
AcCfile
gpt-3.5-turbo
95.2
90.2
GLM-3-Turbo
93.3
83.3
-lama-2-7
87.5
58.8
gpt-44
98.0
99.0
Table 7: More LLMs: Classifier and Verifier are gpt-3.5-turbo. For all APIs, the version of Mar.28, 2024 APIs are used. t: The API version of 1lama-2-70b-chat is used. gpt-4-0125- preview is used.
specific example is LLaMA-2. During our experiments we find it hard to constrain LLaMA-2 to output certain format: classifier, it tends output multiple choices rather than one choice. As a supplement, we use the most frequent character in LLaMa output as choice and adopt LLaMa as Classifier for further experiments in Table 8.
15

---

Published as a conference paper at COLM 2024
Classifier Backbone
Accsys
AcC
LLaMa-2-7b-chat
7.7
78.4
LLaMa-2-70b-chat
97.1
80.4
gpt-3.5-turbo*
95.2
90.2
Table 8: LLaMa-2 as Classifier. The 70b-chat model gives relatively decent performance, but the 7b-chat model fails to correct itself given previous failure examples, resulting in low accuracy for system instructions.
16


Reviews:
[{"review_id": "sScT1kEQrS", "reviewer": "Reviewer_Utie", "summary": "The paper proposes a framework called CHOPS (CHat with custOmer Profile in existing System) that aims to integrate large language models (LLMs) into customer service systems that involve guiding files and databases. The key components are: 1) A new dataset called CHOPS-dataset derived from a real-world online platform for physics Olympiads, including a database, PDF guides, and user questions/instructions. 2) A classifier-executor-verifier architecture where the classifier determines what information (database, guides) is needed, the executor generates a proposed answer/operation using that information, and the verifier checks if the executor's output is valid. 3) Using different LLMs for each component to balance accuracy and cost - e.g. a powerful LLM like GPT-4 for the executor but cheaper models like GPT-3.5 for the classifier/verifier. The authors conduct extensive experiments showing their architecture can achieve high accuracy (98%) at reasonable cost on the CHOPS-dataset by combining LLMs in this way.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Utie,\n\nThank you for a detailed and inspiring review! We truly appreciate your advices and questions! Here are our responses:\n\n**Dataset limitation**: Our results are based on the specific dataset we proposed.\nWe hope our work may encourage more practical dataset with database, files and QA pairs for Customer Service, and we are looking forward to evaluating CHOPS architecture on future datasets.\n\n**Failure cases analysis**: A false positive in the Verifier stage leads to an incorrect output, while errors in the Classifier or Executor stages may be corrected by the Verifier in subsequent iterations. Classifier errors are the most frequent but are easy to identify and correct with the Verifier at the 2nd iteration. For difficult questions, the Executor may fail, and these failures are harder to correct and need more iterations: please refer to the following table about iteration number. We will add more analysis in the next version of our paper.\n\n**Complex setup procedure**: We recognize the database we include in our proposed dataset makes the setup with some workload, but using database like SQLAlchemy that can be easily installed will improve reproducibility, which will be a good choice for the next version of our code.\n\n**Iteration number used in practice**: More than half of the queries end at the first iteration, with few false positives. Two iterations significantly boost accuracy for system instructions, and three iterations do so for file questions. We use five iterations to ensure completion. Here is a table for reference:\n\n**Table: Pos/Neg in each Iteration and Transition between Iterations**\n\n||1st||2nd||3rd||4th||5th|\n|:----:|:-:|:--:|:---:|:--:|:---:|:-:|:--:|:-:|:--:|\n|end(TP)|151||23||9||2||4|\n|**\\#pos in ith step**/\\#pos->pos|**168**|13|**30**|6|**14**|5|**6**|4|**4**|\n|\\#pos->neg|/|4|/|1|/|0|/|0|/|\n|\\#neg->pos|/|17|/|8|/|1|/|0|/|\n|**\\#neg in ith step**/\\#neg->neg|**38**|18|**22**|10|**11**|7|**7**|7|**7**|\n|end(FP)|3||4||3||0||7|\n\n\n**Support for multiple languages**: Our architecture will only require slight modifications as long as Neural Networks with multilingual support are used. LLMs supporting targeted languages and a multilingual sentence encoder for file RAG are needed. Further edits may include prompt tuning for the Verifier to respond in the corresponding language. \n\nThank you again for your detailed review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a novel application of integrating LLMs into real-world customer service systems with databases and guidance in a accurate and cost-effective manner.\n2. New dataset (CHOPS-dataset) derived from practical scenarios to facilitate research in this area.\n3. Thorough experiments evaluating the different components and comparing performance/cost trade-offs.\n4. The classifier-executor-verifier architecture is a promising approach that could generalize beyond just customer service.", "reasons_to_reject": "1. Results are solely on the CHOPS dataset, which is quite specific and limited. More experiments on other customer service domains would strengthen the claims.\n2. Limited analysis of why the classifier-executor-verifier works well and what kinds of errors/failures occur.\n3. The database/API setup is complex and not easily reproducible from the paper.", "questions_to_authors": "1. The paper mentions iterating the pipeline up to 5 times if the verifier rejects outputs. How often do multiple iterations end up being required in practice?\n2. The results focus on English inputs/outputs. How might you handle customer service scenarios requiring other languages? Would the architecture need to be modified?", "extra_scores": {}}, {"review_id": "VEqSiNBwbI", "reviewer": "Reviewer_d31m", "summary": "This paper discussed the growing use of Large Language Models (LLMs) such as GPTs and LLaMas in businesses and software platforms for customer service. The authors identify a gap in the integration of LLMs with customer profiles and operational capabilities in customer service models. To address this, they propose an LLM agent called CHOPS that efficiently uses existing databases or systems to access user information and provide accurate responses while avoiding harmful operations. \n\nThe authors introduce a practical dataset, the CPHOS-dataset, collected from an online platform that organizes simulated Physics Olympiads for high school teachers and students. They propose a three-agent architecture for customer service tasks: Classifier, Executor, and Verifier (C-E-V). The Classifier classifies user requests, the Executor provides responses or executes operations, and the Verifier assesses the validity of commands executed by the Executor. \n\nThe paper also includes experiments to validate the performance of the proposed CHOPS architecture using the CPHOS-dataset. The results demonstrate that the CHOPS architecture can enhance or serve as an alternative to human customer service. The authors conclude by emphasizing the need for more datasets with QA pairs and Database for Customer Service to further evaluate the effectiveness of the CHOPS-architecture.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer d31m,\n\nThanks for the detailed and advisable review! We truly appreciate your advices!\n\nHere are our corresponding replies:\n\n**Description about our model**: We recognize that the current explanation is be a bit less detailed and we will add more descriptions in our next version of paper. We will add more overview and specific details, especially for the Executor and Verifier agent.\n\n**Real-world scenarios**: Our dataset is collected in a real-world scenario, and we think our model would be a real-world solution. Moreover, we wish our work may encourage more datasets with Database, APIs and related Guiding Files to be proposed in future works, and we wish to test our models to see its performance on a broader range of Customer Service. We hope our work may bring insights for LLM applications in Customer Service, not only for model structure but also for more possible real-world datasets.\n\n**Comparison with other existing models**: Recent works on agent or LLM reasoning (e.g. ToT, Cumulative Reasoning, etc) underlines efficient searching over a large state space for possible answer (e.g. math problems, etc) but Customer Service requires more about accuracy and less about reasoning skills compared to these complicated tasks, and these more complex models are hard to accommodate to this scenario. In comparison, our work addresses a key gap in integrating LLMs with customer profiles and operational capabilities in customer service models. As a supplement, we compare with CoT-SC method that is easier to adopt to the Customer Service scenario, and compare the CHOPS C-E-V architecture with it, as follows: \n\n**Table: Comparison with CoT-SC**\n\n|  | $Acc_{sys}$ | $Acc_{file}$ | $Acc_{avg}$ | #$char_{in}^{avg}$ | #$char_{out}^{avg}$ |\n| ---- | --- | ----- | ---- | -------- | ------- |\n| 1-vote CoT   | 91.2        | 62.7  | 77.0 | 14.51k   | 0.50k   |\n| 4-vote  CoT-SC  | 90.2| 77.4    | 83.8        | 53.54k | 0.94k     |\n| 16-vote CoT-SC  | 90.2  | 78.4   | 84.3        | 211.58k  | 2.59k     |\n| CEV (ours) | 95.2   | 90.2    | 92.7      | 30.1k      | 0.56k      |\n\n(We will include this table in our paper. All models use gpt-3.5-turbo)\n\nCompared to CoT-SC, our proposed architecture achieves higher accuracy with less cost.\n\nThank you for the detailed review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses a significant gap in the integration of LLMs with customer profiles and operational capabilities in customer service models. This is a relevant and timely topic given the increasing use of AI in customer service. The introduction of the CHOPS architecture and the C-E-V model is innovative and could potentially change the way customer service is handled in businesses. The authors have done a commendable job of validating their model using the CPHOS-dataset. The results demonstrate the effectiveness of the proposed architecture.", "reasons_to_reject": "1. The paper could benefit from a more detailed explanation of the CHOPS architecture and the C-E-V model. The current description may not be sufficient for readers unfamiliar with these concepts.\n2. The authors have used the CPHOS-dataset for validation. However, the effectiveness of the model in real-world scenarios remains untested. \n3. The authors could have included a comparison with other existing models. This would help readers understand how the proposed model stands against the current state-of-the-art.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qb6fpQKUS9", "reviewer": "Reviewer_T4SG", "summary": "The authors propose a framework for Retrieval-Augmented Generation based on LLMs in the domain of customer support. The framework utilises a series of smaller and larger LLMs to retrieve information from a database and to serve this information to the customer. The database is in the form of pdf guides and a MySQL database with data on users, schools, exams and answers. The developed framework is evaluated on the customer service of a Physics olympiad for teachers and students. A series of models and frameworks is evaluated, and the dataset used for evaluation is published.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Dear Reviewer T4SG:\n\nThank you for a detailed and constructive review! We truly appreciate your advice! Here are our responses:\n\n**Metric Section**: We will expand the experiment setting section in main paper to clearly differentiate the two cases regarding the database and the files.\n\n**Open-Source baselines**: Some Open-Source LLMs faced the difficulty to output in constrained format (e.g. LLaMa 2-70b will output multiple choices rather than one for Classifier task), making it hard to use them as Classifier or Verifier. As a supplement, we use the most frequent character in LLaMa output as choice and adopt LLaMa as our Classifier for the following experiment. The results will be included in our paper. (E,V: gpt-3.5-turbo).\n\n**Table: Performance of LLaMa and gpt-3.5-turbo as Classifiers**\n\n|Classifier|$Acc_{sys}$|$Acc_{file}$|\n|-|-|-|\n|LLaMa-2-7b|7.7|78.4|\n|LLaMa-2-70b| 97.1|80.4|\n|gpt-3.5-turbo|95.2|90.2|\n\nLlama2-70b model gives relatively decent performance, while llama2-7b model sometimes fails to correct itself given previous failure examples, resulting in a low accuracy for system instructions.\n\n**Security, inference time and cost**: We recognized that security is important, especially regarding malicious user behaviors. As mentioned in Section 3.1, we restrict the amount of APIs and their functionality. Moreover, particular APIs are parsed with an argument representing user identity not affected by LLMs, preventing attacks like admin impersonation.\n\nWe agree that inference time is crucial, but it varies due to network issues and we believe character count is a metric to measure it indirectly (which is included in Table 1, Table 2). As a supplement, we further measure average time per query with CoT-SC as baselines, for your reference: (LLM: gpt-3.5-turbo).\n\n**Table: Comparison with Baselines with Inference Time**\n| |$Acc_{sys}$|$Acc_{file}$ |#$char_{in}^{avg}$|#$char_{out}^{avg}$|$Time_{avg}$ |\n|-|-|-|-|-|-|\n|1-vote|91.2|62.7|14.51k|0.50k|5.40s|\n|4-vote|90.2|77.4|53.54k|0.94k|11.48s|\n|CEV(ours)|95.2|90.2|30.1k|0.56k|7.94s|\n\nFor inference cost, we estimated relative cost with character counting (please refer to Appendix A.2) and included it in Figure 5 in the main paper. We recognize its importance, hence we will add more about our estimation method in the main paper.\n\n**Miscues presents, typos and figures**: We have polished the paper and redrawn the figures. Our paper will be more accurate and readable.\n\nThank you again for the detailed review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A realistic and useful scenario is evaluated. In the examined scenario, a customer service agent is facilitating information retrieval between a customer and a database.\n\n2. The comparison of the different frameworks is highly informative, especially the compartmentalization of the different modules, such as the use of the Classifier, which greatly increases performance.", "reasons_to_reject": "1. Section 5.1 (Metrics) should be expanded. The evaluation methodology needs to be clearly set before the experiments and not in the Appendix. Both the metrics and the motivation behind them needs to be expanded upon in the main paper.\n\n2. Open-source baselines are missing. As mentioned by the authors, the Classifier and Verifier deal with easier tasks. It would be interesting to examine open-source LLMs for these tasks. (amended in rebuttal)\n\n3. Security and inference time are not discussed. The proposed framework does not currently deal with malicious user behaviour, which needs to be a greater focus. Further, for the proposed framework to be adopted, a study and comparison on inference cost is pivotal. (amended in rebuttal)\n\n4. The paper needs some further editing, there are multiple visual miscues present, such as:\n\n- Some citations are missing and some have spacing issues (no space between proceeding text)\n- Grammatical typos, such as: \"but high accuracy are needed\"\n- Figures 1, 2 and 4 are too small to read", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a system for using a Large Language Models (LLMs) in the application of customer service. Due to the number of customer support agents in the word, this could have a significant effect. The new framework, CHOPS, combines small and large LLMs to work with existing customer profiles, making customer service more efficient. They tested this with a new dataset from a real-world online platform for Physics Olympiads.

Pros:
* Novel Idea: It addresses a gap in current LLM-based customer service by integrating them better with customer profiles.
* New Dataset: The CPHOS-dataset is a valuable resource for future research.
* Thorough experiments evaluating the different components and comparing performance/cost trade-offs.

Cons:
* Further testing on other customer service domains would help show it works more broadly.
* Additional details on the CHOPS architecture would help readers understand it better.
* Additional editing is required for the paper.

In summary, all reviewers liked the paper and thought it should be accepted.
Decision: Accept