CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs

Authors: jingzhe  shi
jialuo  li
qinwei  ma
zaiwen  yang
huan  ma
lei  li
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 03:21:27

Abstract: Businesses and software platforms are increasingly utilizing Large Language Models (LLMs) like GPT-3.5, GPT-4, GLM-3, and LLaMa-2 as chat assistants with file access or as reasoning agents for custom service. Current LLM-based customer service models exhibit limited integration with customer profiles and lack operational capabilities, while existing API integrations prioritize diversity over precision and error avoidance that are crucial in real-world scenarios for Customer Service. We propose an LLMs agent called **CHOPS** (**CH**at with cust**O**mer **P**rofile in existing **S**ystem) that: (1) efficiently utilizes existing databases or systems to access user information or interact with these systems based on existing guidance; (2) provides accurate and reasonable responses or executing required operations in the system while avoiding harmful operations; and (3) leverages the combination of small and large LLMs together to provide satisfying performance while having decent inference cost. We introduce a practical dataset, *CPHOS-dataset*, including a database, some guiding files, and QA pairs collected from *CPHOS*, which employs an online platform to facilitate the organization of simulated Physics Olympiads for high school teachers and students. We conduct extensive experiments to validate the performance of our proposed **CHOPS** architecture using the *CPHOS-dataset*, aiming to demonstrate how LLMs can enhance or serve as alternatives to human customer service.

Main paper: 
 introduction: 
 In most organizations with human customer service, there is usually a system storing customer information. Responses are based on the customer's profile, like user type and purchase history, following set guidelines. Customer service can also update the customer's status upon request. Large language models (LLMs), such as GPT-3.5 (OpenAI, 2022) , GPT-4.0 (OpenAI, 2023) , GLM (Zeng et al., 2022; Du et al., 2022) and LLaMa (Touvron et al., 2023) , have emerged as a representative achievement of AI development in the past decade. With their mastering of common knowledge and their ability to understand prompts and generate contextually relevant answers, LLMs have been used as assistants across a wide range of application scenarios, including chatting assistants, coding assistants, automatic assistant agents, etc (Kalla & Smith, 2023) .A common paradigm for equipping LMs with external knowledge while avoiding long context lengths that are costly or technically hard is RAG (Lewis et al., 2021) , represented by the widely used and efficient Vector Database using a sentence embedding model such as the Universal Sentence Encoder (Cer et al., 2018) . Most publicly available architectures for utilizing LLMs as customer service mainly follow this paradigm, e.g. Databricks (Databricks) enables users to upload guiding files thus building a Vector Database-based customer service agent to augment human customer service. However, for most software platforms Previous works (involving models, agent architectures, and datasets) on LLMs using APIs (Patil et al., 2023; Qin et al., 2023; Tang et al., 2023) mainly focus on the LLM's ability to choose between a vast amount (100 through 100,000+) APIs and to accomplish different tasks. However, for a particular customer service scenario, much fewer APIs are needed but high accuracy is needed, especially for modifying user status in important aspects like banking. This difference in the focus point of Customer Service and previously defined API-using tasks sheds light on a different method needed for Customer Service compared to previous API-using methods and new datasets for evaluation of such methods.Online Customer Service, with its long history since the popularization of the Internet and personal computers, requires accurate answers or modifications to user profiles in customer service based on the user's questions or requests and the guiding file. Unlike other scenarios for reasoning or prompting, especially on math problem solving (Wei et al., 2023; Yao et al., 2023; Zhang et al., 2023) , or utilizing numbers of APIs (Patil et al., 2023; Qin et al., 2023; Tang et al., 2023) , we focus on the new task of customer service, which sheds light on the accuracy and admittance of the answer; moreover, the cost of the LLM needs to be controlled.Previous efforts to integrate Language Models (LMs) with databases have focused on generating SQL commands, which poses risks such as incorrect or harmful commands due to LMs' hallucination issues (Huang et al., 2023) . To address this, we employ APIs for database management, steering clear of direct SQL command generation. Platforms like LangChain (Chase, 2022 ), Gorilla (Patil et al., 2023) , and ToolLLM (Qin et al., 2023) offer extensive API libraries, which may exceed the needs of specific customer service applications where the accuracy and proper use of a limited set of APIs are paramount. Unlike scenarios requiring thousands of APIs, customer service systems prioritize precise and correct API usage, highlighting the importance of quality over quantity in API interactions.Leveraging insights, shown in Figure . 3, from studies on using LLMs to verify responses of other LLMs, we introduce an Executor-Verifier architecture, employing a verifier agent to assess and ensure the validity of commands executed by another LLM, termed the executor agent as Figure 3 . This approach aims to enhance the accuracy of integrating LLMs with APIs by reiterating the execution process based on the verifier's feedback.To address challenges when user requests involve both API use and information from guiding files, we evolved our approach into a Classifier-Executor-Verifier architecture, enhancing efficiency and reducing inference costs. This architecture first classifies user requests to determine if they require access to APIs, guiding files, or both, thus avoiding unnecessary processing and long, redundant texts from guiding files for API-only queries. Additionally, by segmenting guiding files into smaller, more focused documents and employing a more nuanced classifier, we further minimize inference costs by ensuring that only the most relevant sections are utilized. This tailored architecture, specifically designed for Customer Service scenarios involving guiding files and user systems, optimizes performance while conserving computational resources.In response to the lack of datasets for customer service that incorporate details on internal guiding files or systems for API interactions within the customer service domain, we propose the CPHOS-dataset. This dataset, derived from real-world scenarios at the Cyber Physics Olympiad Simulations (CPHOS), a non-profit organization, includes databases, guiding files in PDF format, and QA pairs from actual interactions, aiming to bridge the gap in existing resources for LLM research in customer service. Carefully curated and anonymized, the CPHOS-dataset serves as a comprehensive tool for evaluating the effectiveness of LLMs in customer service environments, especially those involving complex systems and guiding documents (Vector, 2018; McAuley; Rajpurkar et al., 2016; Joshi et al., 2017; Patil et al., 2023; Qin et al., 2023; Tang et al., 2023; OpenAI, 2022; 2023) .To this end, we propose a general framework called CHOPS: CHat with custOmer Profile in existing System as Figure 3 shown. In our CHOPS framework, we propose a classifierexecutor-verifier based framework that makes the LLM better utilize tools, especially Database and guidance. Through the work of these three agents, the task of answering a user's question is decomposed into (1) classifying the type or theme of the user's question, (2) giving answers or decision operations to be executed and (3) verifying then reject or commit the result of the executor.Our work makes several contributions to the application of LLMs, especially in the field of customer service with LLMs.• We propose a framework CHOPS to embed LLM safely, effectively, high-costperformancely into existing customer service systems. • Our experiments demonstrate that by using weaker LLMs in our architecture, we can achieve significantly better performance compared to naively using stronger LLMs while saving cost. Moreover, using gpt-4 as Executor while gpt-3.5-turbo as Classifier and Verifier can reach 98% accuracy with decent cost. • We proposed CPHOS-dataset, it is a practical dataset collected in real scenarios that can be used to validate methods utilizing LLMs as customer service. 
related works: 
 Retrieval-Augmented Generation with LLMs. Incorporating external knowledge sources into LLMs for enhanced performance on knowledge-intensive tasks has seen advancements through Retrieval-Augmented Generation (RAG), with the use of vector-based databases for PDF files being a notable example (Tripathi, 2023) . This approach encodes user queries into vectors, using k-nearest neighbors (KNN) to retrieve relevant information. In customer service, LLMs augmented with large databases have aimed to provide encyclopedic support for user inquiries (dat, 2022; Wulf, 2022; cms, 2022) . However, such methods sometimes struggle in scenarios requiring modifications to a user's profile within existing systems, a crucial aspect of customer service. This highlights the importance of integrating LLMs with software systems for direct interaction tasks, essential for operational efficiency in businesses.LLM Agents. The research on LLMs as specialized agents is an evolving field in artificial intelligence and natural language processing. Initial research focused on using predefined prompts or fine-tuning to enhance LLMs for specific tasks, establishing their potential in specialized applications like natural language understanding. Recent studies (Hu et al., 2023a; Doe & Smith, 2023) explore LLMs in agent-based architectures for complex problem solving, such as mathematical puzzles, highlighting the importance of architectural design in improving performance. This progression suggests new avenues for leveraging LLMs in agent-based systems, offering insight into their capabilities for advanced interaction and problem-solving tasks.Recent research has explored the enhancement of LLMs with external tools to improve their performance for a variety of tasks. Studies like OpenAI (2023); Hu et al. (2023b) demonstrate that equipping LLMs with tools, including automated programming interfaces, database management systems, and coding environments, can significantly expand their capabilities. These advances illustrate the potential for LLMs to generate more accurate code, perform advanced database queries, and overall broaden their applicability by leveraging specialized tools. 
cphos-dataset: a real-scene dataset for customer service: 
 The CPHOS-dataset is collected from an online platform of CPHOS, a non-profit organization dedicated to holding Simulated Physics Olympiads online through the online platform as Figure 2 shown. The online system of the Simulated Olympiad utilizes a MySQL database. We provided 9 data desensitized tables. The detailed description can be found in the appendix A.1.In short, given a user's nickname, one can do a series of queries on tables in the database to obtain or modify partially the profile of the user. The most important field of a user profile includes (0) approved to use online platform; (1) user name;(2) school id;(3) user type: team leader, vise team leader, arbiter; (4) marking question id, etc.Unlike previous works (Hu et al., 2023a) directly using LLMs to generate SQL commands for query or modification to the database, we wrapped the query and modification into a series of python APIs, following the idea of Repository Pattern in software design. We provide 9 Data Managing APIs and 18 Data Query APIs, 10 of which are available to LLMs.We collect instructions and queries to the system and augment them with GPT-4 into 104 System-related queries and instructions. There are several advantages of wrapping SQL commands and LLMs manipulate databases through these APIs, in that:• Properly named APIs are much easier for LLMs to understand and to generate compared to complicated table structures and SQL commands for the database.• By limiting APIs LLMs can use, or by checking status inside the APIs, one can prevent unwanted or harmful operations that might be carried out by LLMs in extreme conditions.• In codes of software or websites that are written following the Repository Pattern (or more broadly, the principle of 'encapsulation' in software architecture), a series of pre-defined APIs for a database is likely to exist. Much less effort is needed to modify these APIs into APIs suitable for LLMs than to check for correct SQL commands generated by LLMs.The diversity of the APIs not only comes from the number of APIs. Calling them from a different user and with different arguments would give different results as shown in the middle and right part of Figure 2 .There Given the previous setup, in Figure 3 , we define our task as follows. Given a user's nickname and its question, the task is to give a proper answer or an appropriate execution command to the system, based on the status of the user in the existing system and the guiding files. 
framework overview: 
 We propose a three-agent architecture for this task, termed the classifier-executor-verifier architecture. In this architecture, each agent is implemented using a Large Language Model (LLM). This three-stage architecture functions as follows:C: The Classifier is given the UserTexts, the System API descriptions and several relevant (and short) chunks from the guiding files. The Classifier classifies the UserTexts based on information needed for the following pipeline. The classifier itself does not output all relevant information but only indicates the type of information that would be beneficial for further processing.E: The Executor is given the UserTexts and other information that the classifier classifies as helpful in the Classifier Stage. Note that the information can be richer than that given to the Classifier (e.g. longer retrieved chunks, more detailed descriptions, etc.). The Executor then gives a proposed answer or API call based on the information given.V: The Verifier is responsible for revisiting the Executor's result, verifying if it is valid or not and giving a reason as well. If valid, then the answer is returned or the execution is carried out, and a reply is generated based on the result of that execution. If invalid, then the whole process will be redone, while the Classifier and Executor can see the invalid reason provided by the verifier as a reference. Previous work (Chen et al., 2023) has shown that the longer and more complicated the retrieved content, the more difficult it is for the Executor LLM to find the exact piece of information and return it. Also, feeding the Executor every time with retrieved chunks from guide file and API info is token-consuming. We utilize a Classifier to classify the information domain that needs to be given to the Executor in advance to solve these two issues.A simple and experimentally-proved effective and efficient design for the Classifier is a binary classifier, or a 1-level Classifier as shown in the left part of Figure 4 . This classifier only chooses between two categories: (1) that the UserTexts are about a query to the guiding file, and (2) that the UserTexts are related to a query or an instruction to the system. Nonclassifiable cases, which are not related to the above two cases, are dealt with in the same way as (1) in the following pipeline. Note that although we use the same RAG method to retrieve file chunks for the Classifier and the Executor, the chunk length for Classifier is set to be lower to save inference cost.Moreover, we observe that many queries and questions are about basic information of the user in the system that is much shorter and less token-consuming compared to the retrieved chunks. Inspired by the idea of Cache, we further add one categorized: (0) 'Basic Info' apart from (1) 'Guide File', (2) 'System API'. This leads to the development of a two-level classifier architecture shown in the right part of Figure 4 . One classifier will first decide whether the UserTexts are solvable only given the Basic Information (without the need for 'Guide Files' and 'System APIs'). If it asserts yes, then the pipeline will go on and the Executor will only see the Basic Information. If not, then it goes to the second level classifier to categorize UserTexts into class (1) or (2), and further provide the corresponding information to the Executor. The experiment indicates this architecture improves accuracy while saving token consumption.The Executor needs to return an answer or give an appropriate execution command given information provided. In practice, we do prompt-tuning separately for different cases given by the Classifier.Previous works (Weng et al., 2023; Zhang et al., 2023) have proved the effectiveness of a Verifier for re-checking correctness.In our work, the Verifier verifies the result and, if valid, summarizes the answer or generates a response to the user based on the executed operation.Like the Executor, we do prompt tuning separately for different cases given by the Classifier.For the Guide File cases, we feed the retrieved results to the Verifier as well.The Verifier is required to output a valid score (1-10) and reasons at the same time. If the verification result is invalid, then we redo the whole C-E-V process while giving the Classifier and Executor the invalid reason.In practice to ensure fast response, we restrict the loop iterations into 5. For latter iterations, a lower score would be seen as valid: we use a simple linear scheduling of the passing score with the iteration index. If all iterations fail, we ask the LLM to choose between one answer provided before. More details about failure cases and number of iterations in practice can be found in Appendix A.4.For PDF Retrieval, we follow the well-practiced method of using a sentence encoder to encode chunks from guide files into a vector database and retrieve top-K closest chunks in latent space as related information given by the files. We follow Pdf-gpt (Tripathi, 2023) and utilize Universal Sentence Encoder (Cer et al., 2018) as the embedding model. For Database manipulation, we use the wrapped APIs as available tools. 
metrics: 
 Our study evaluates the model's performance through several metrics: Instruction Set Accuracy, Guiding File Question Accuracy, and Input/Output character consumed per Question. More details ref to Appendix A.3We make a comparison at the character level since different LLMs utilize different tokenization methods. Input characters and output characters are separated since generating output tokens is more resource-consuming than reading input tokens for LLMs (and are more expensive in terms of API price). For clarity, we define the first two metrics mathematically.Both Instruction Set Accuracy and Guiding File Question Accuracy are measured as:EQUATIONWhere:• N correct is the number of questions correctly answered by the model.• N total is the total number of questions and instructions.We validate whether the answer is correct by a combination of GPT4-based evaluation and human verification afterward.This formula encapsulates the model's efficiency in accurately processing and responding to queries based on the instructions provided or the information contained within the guiding files. The metrics are instrumental in evaluating the model's adeptness at interpreting and acting upon specific sets of instructions and its capacity to extract and utilize knowledge from guiding documents, thereby providing a multidimensional view of its performance in realistic scenarios. We evaluate the proposed CHOPS agent architecture on the CPHOS-dataset. We perform prompt tuning on gpt-4 (gpt-4-0125-preview, labeled Executor Only) and use multivote CoT (on gpt-3.5-turbo) as our baselines. We show our main experiment results in Table 1 .Compared to the CoT method, our method achieves higher accuracy while maintaining a decent cost. Multi-vote CoT does see an improvement in performance when increasing the number of votes, but it cannot provide satisfactory performance even with more cost.Moreover, our proposed C-E-V agent architecture with gpt-3.5-turbo as LLM backbones for all agents surpasses the Executor Only gpt-4 baseline by a large margin (86% → 95%, 83% → 90%) with 34.4% cost. Moreover, by substituting the Executor backbone with gpt-4, we can reach accuracy of above 98% on both accuracy metrics, hugely surpassing the plain gpt-4 baseline while even cost less than the gpt-4 baseline (see Appendix A.2 for more details about price estimation). Thus our architecture is proven to be flexible and is token-efficient, reaching a balance between cost and accuracy.Starting from the baseline using gpt-3.5-turbo as plain Executor, we add all designed blocks and ablate the effectiveness. Finally, we substitute the Executor backbone with gpt-4 and make a comparison to plain Executor using gpt-4 as another baseline. Detailed experiment results and figure can be seen in Table 2 and Figure 5 .Classifier is shown to both reduce token consumption and improve accuracy. A more complicated 2-Level Classifier can further improve accuracy while reducing cost. This free lunch can be explained as follows. By adding a classifier we can avoid sending long retrieved chunks into the Executor in some scenarios, thus improving RAG accuracy and reducing token consumption for LLMs (Chen et al., 2023) . Note by shortening the chunks sent to classifier or by using weaker yet less expensive LLMs as Classifier we can save inference cost.Baseline1: Executor Only gpt-4 based Baseline2: Executor Only gpt-3.5-turbo based * * * : gpt-3.5-turbo is used for Classifier and Verifier, while gpt-4-0125-preview is used for Executor. † : gpt-3.5-turbo character + gpt-4-0125-preview character.(1-level C)-E gpt-3.5-turbo based (1-level C)-E-V gpt-3.5-turbo based (2-level C)-E-V gpt-3.5-turbo based (2-level C)-E-V Mixed C,V: gpt-3.Self-Verification is a proven effective method to improve accuracy in previous works (Weng et al., 2023) . In our experiments, it has been shown to improve accuracy while consuming more tokens. However, we find that in the case where gpt-4 is used for Executor, weaker and less expensive LLMs (i.e. gpt-3.5-turbo) for Verification is enough for the architecture to produce results at very good accuracy (98%), while maintaining a decent cost at the same time.In general, the proposed Classifier and Verifier are dealing with easier tasks but can effectively improve accuracy. Even in scenarios where very high accuracy (98%) is required and we have to use more powerful LLMs as Executor, weaker and cheaper LLMs can still be used as Classifier and Verifier to reduce total cost while achieving satisfactory accuracy. 
conclusion: 
 Targeting the important scenario of Customer Service, we have collected, processed related data and proposed our CPHOS-dataset. However, existing datasets in customer service predominantly lack detailed information about internal guiding documents or systems that can be interacted with. Similarly, datasets dedicated to file QA or API calling seldom address the customer service domain specifically.To address this gap, we introduce the CPHOS-dataset, derived from the real-world context of the online platform CPHOS: Cyber Physics Olympiad Simulations. This dataset serves as the evaluation ground for our newly proposed CHOPS-architecture, emphasizing its practical application in the CPHOS-dataset.CPHOS: Cyber Physics Olympiad Simulations is a non-profit organization focusing on physics education, predominantly operated by approximately 100 college student volunteers, primarily from China. It organizes simulated high school Physics Competitions, akin to the International Physics Olympiad (IPhO), attracting around 1000 participants from hundreds of schools annually. The organization utilizes an online system and a mini-program-based frontend for operational tasks such as uploading and marking answer sheets. The system's backend is supported by a MySQL database, maintaining records on team leaders, vice team leaders, contestants, and examination details. Furthermore, CPHOS offers PDF guides on utilizing the mini-program for administrative purposes. The communication between team leaders and CPHOS liaison members typically relies on this documented information. After thorough data desensitization, the database, guiding documents, and QA pairs offer a representative and functional scenario for integrating LLMs into customer service, utilizing existing systems and guiding documents.The CPHOS-dataset comprises a database, several guide files in PDF format, and QA pairs. These components collectively facilitate understanding how team leaders and vice team leaders can navigate the mini-program for various tasks, including uploading and grading answer sheets and accessing student grades. QA pairs are derived from real-world interactions and are further augmented by both human efforts and LLMs (including GPT-3.5 (OpenAI, 2022) and GPT-4 (OpenAI, 2023)), which enriches the dataset, making it a robust resource for validation and experimentation. The dataset's inception, modification, translation, and augmentation have been meticulously undertaken by us, with raw data sourced from CPHOS.It is worth noting that while the dataset is domain-specific, the nature of the queries and instructions it encompasses-ranging from inquiries about the contents of PDF guides to requests for system profile modifications-are emblematic of the broader customer service sector. This suggests that methodologies proven effective within this dataset have the potential for broader applicability across various customer service domains, contingent upon the adaptation of guiding documents and APIs to suit specific industry requirements.Database. There are 9 tables in our database, listed as following There are 23 APIs. We list a few here in the following table 4. Please refer to our provided codes for further information.User Questions and Instructions. The questions are collected from CPHOS liaison members, who are responsible for responding to (visit) team leaders. Most questions are about the use of the mini-program, and others are about requests to modify the marked problems or add/modify their status in system. Since the gpt families are believed to use the same tokenization method, we use the same k for gpt-3.5-turbo and gpt-4-0125-preview to calculate their cost. 
 
 Appendix: We make a comparison at the character level since different LLMs utilize different tokenization methods. Input characters and output characters are separated since generating output tokens is more resource-consuming than reading input tokens for LLMs (and are more expensive in terms of API price). For clarity, we define the first two metrics mathematically.Both Instruction Set Accuracy and Guiding File Question Accuracy are measured as:EQUATIONWhere:• N correct is the number of questions correctly answered by the model.• N total is the total number of questions and instructions.We validate whether the answer is correct by a combination of GPT4-based evaluation and human verification afterward.This formula encapsulates the model's efficiency in accurately processing and responding to queries based on the instructions provided or the information contained within the guiding files. The metrics are instrumental in evaluating the model's adeptness at interpreting and acting upon specific sets of instructions and its capacity to extract and utilize knowledge 


Decision: This paper introduces a system for using a Large Language Models (LLMs) in the application of customer service. Due to the number of customer support agents in the word, this could have a significant effect. The new framework, CHOPS, combines small and large LLMs to work with existing customer profiles, making customer service more efficient. They tested this with a new dataset from a real-world online platform for Physics Olympiads.

Pros:
* Novel Idea: It addresses a gap in current LLM-based customer service by integrating them better with customer profiles.
* New Dataset: The CPHOS-dataset is a valuable resource for future research.
* Thorough experiments evaluating the different components and comparing performance/cost trade-offs.

Cons:
* Further testing on other customer service domains would help show it works more broadly.
* Additional details on the CHOPS architecture would help readers understand it better.
* Additional editing is required for the paper.

In summary, all reviewers liked the paper and thought it should be accepted.

Reviews: review: paper_summary: paper_summary: The paper proposes a framework called CHOPS (CHat with custOmer Profile in existing System) that aims to integrate large language models (LLMs) into customer service systems that involve guiding files and databases. The key components are: 1) A new dataset called CHOPS-dataset derived from a real-world online platform for physics Olympiads, including a database, PDF guides, and user questions/instructions. 2) A classifier-executor-verifier architecture where the classifier determines what information (database, guides) is needed, the executor generates a proposed answer/operation using that information, and the verifier checks if the executor's output is valid. 3) Using different LLMs for each component to balance accuracy and cost - e.g. a powerful LLM like GPT-4 for the executor but cheaper models like GPT-3.5 for the classifier/verifier. The authors conduct extensive experiments showing their architecture can achieve high accuracy (98%) at reasonable cost on the CHOPS-dataset by combining LLMs in this way.
strength_weakness: strength_weakness: 1. Results are solely on the CHOPS dataset, which is quite specific and limited. More experiments on other customer service domains would strengthen the claims.
2. Limited analysis of why the classifier-executor-verifier works well and what kinds of errors/failures occur.
3. The database/API setup is complex and not easily reproducible from the paper.
questions: questions: 1. The paper mentions iterating the pipeline up to 5 times if the verifier rejects outputs. How often do multiple iterations end up being required in practice?
2. The results focus on English inputs/outputs. How might you handle customer service scenarios requiring other languages? Would the architecture need to be modified?
paper_summary: paper_summary: This paper discussed the growing use of Large Language Models (LLMs) such as GPTs and LLaMas in businesses and software platforms for customer service. The authors identify a gap in the integration of LLMs with customer profiles and operational capabilities in customer service models. To address this, they propose an LLM agent called CHOPS that efficiently uses existing databases or systems to access user information and provide accurate responses while avoiding harmful operations. 

The authors introduce a practical dataset, the CPHOS-dataset, collected from an online platform that organizes simulated Physics Olympiads for high school teachers and students. They propose a three-agent architecture for customer service tasks: Classifier, Executor, and Verifier (C-E-V). The Classifier classifies user requests, the Executor provides responses or executes operations, and the Verifier assesses the validity of commands executed by the Executor. 

The paper also includes experiments to validate the performance of the proposed CHOPS architecture using the CPHOS-dataset. The results demonstrate that the CHOPS architecture can enhance or serve as an alternative to human customer service. The authors conclude by emphasizing the need for more datasets with QA pairs and Database for Customer Service to further evaluate the effectiveness of the CHOPS-architecture.
strength_weakness: strength_weakness: 1. The paper could benefit from a more detailed explanation of the CHOPS architecture and the C-E-V model. The current description may not be sufficient for readers unfamiliar with these concepts.
2. The authors have used the CPHOS-dataset for validation. However, the effectiveness of the model in real-world scenarios remains untested. 
3. The authors could have included a comparison with other existing models. This would help readers understand how the proposed model stands against the current state-of-the-art.
paper_summary: paper_summary: The authors propose a framework for Retrieval-Augmented Generation based on LLMs in the domain of customer support. The framework utilises a series of smaller and larger LLMs to retrieve information from a database and to serve this information to the customer. The database is in the form of pdf guides and a MySQL database with data on users, schools, exams and answers. The developed framework is evaluated on the customer service of a Physics olympiad for teachers and students. A series of models and frameworks is evaluated, and the dataset used for evaluation is published.
strength_weakness: strength_weakness: 1. Section 5.1 (Metrics) should be expanded. The evaluation methodology needs to be clearly set before the experiments and not in the Appendix. Both the metrics and the motivation behind them needs to be expanded upon in the main paper.

2. Open-source baselines are missing. As mentioned by the authors, the Classifier and Verifier deal with easier tasks. It would be interesting to examine open-source LLMs for these tasks. (amended in rebuttal)

3. Security and inference time are not discussed. The proposed framework does not currently deal with malicious user behaviour, which needs to be a greater focus. Further, for the proposed framework to be adopted, a study and comparison on inference cost is pivotal. (amended in rebuttal)

4. The paper needs some further editing, there are multiple visual miscues present, such as:

- Some citations are missing and some have spacing issues (no space between proceeding text)
- Grammatical typos, such as: "but high accuracy are needed"
- Figures 1, 2 and 4 are too small to read

Comments: title: Summary of Revisions
Thanks for reviewer responses; Open for further discussions
Reviewer, please respond to rebuttal
Reviewer, please respond to rebuttal
comment: We sincerely thank all reviewers for their detailed comments, which are very constructive for our work.

In our work, we proposed a real-world Customer Service dataset with database, guiding files and QA pairs. We also proposed the CEV agent-based architecture for utilizing LLMs on Customer service and evaluated it on our proposed dataset. We hope our work may bring insight for LLM applications in Customer Service in the aspect of agent architecture and encourage the proposal of more real-world datasets with databases, APIs, related guiding files and other related materials as well.

The reviewers generally hold positive opinions to our paper, in that our work proposes a novel and timely application of LLMs in the real-world scenario of Customer Service, our experiments are informative that show the effectiveness of our proposed architecture on performance/cost trade-offs, and our proposed architecture provide insight that may generalize beyond Customer Service.

The reviewers have also raised detailed and constructive concerns that help us improve our work. Compared to the original version, our new version of paper will feature enhancements in the following aspects:

**Supplementary experiment results and analysis**

1. The table for Comparison with baselines will be added to the main paper. (Reviewer d31m, Reviewer T4SG)
2. The table for Performance of open-source llms as Classifiers as well as analysis about shotcomings of Llama2 will be added. (Reviewer T4SG)
3. More analysis about failure case and iteration number required in practice will be added. (Reviewer Utie)

**More detailed explanations in the main paper**

4. More detailed explanation about the CEV model will be added in the main paper. (Reviewer d31m)
5. More details about experiment settings will be added in the main paper. (Reviewer T4SG)
6. More details about estimation methods for inference cost will be added in the main paper. (Reviewer T4SG)

**Amendment for Miscues presents**

7. Miscues presents, typos will be corrected and figures will be redrawn for readability. (Reviewer T4SG)

We will also modify current codebase to simplify setup procedure for better reproducibility. (Reviewer Utie)

The valuable suggestions from reviewers are very helpful for us. We would be glad to have further discussions.
Thank you for the rebuttal! I have amended my score from a 4 to a 6. I particularly appreciate the note on open-source baselines, the finding on the shortcomings of LLama2 is important ("LLaMa 2-70b will output multiple choices rather than one for Classifier task").
Dear Reviewers,

Thank you for taking time to do reviews and responses! We hope our previous feedback provide more supplementary information about our work and address your concerns.

We are also looking forward to possible further discussions in the remaining discussion period!
Thanks for the response. It's helpful.
Reviewer, please respond to rebuttal even if your score hasn't changed. The discussion period ends Thursday
Reviewer, please respond to rebuttal even if your score hasn't changed. The discussion period ends Thursday
Thanks for providing more details.