Salespeople vs SalesBot: Exploring the Role of Educational Value in Conversational Recommender Systems
Authors: Lidiya Murakhovs'ka
Philippe Laban
Tian Xie
Caiming Xiong
Chien-Sheng Wu
Venue: EMNLP 2023 Findings (2023)
Keywords: conversational recommender systems
mixed-initiative conversational agents
mixed-type conversational dataset
TL;DR: This paper introduces SalesOps, a framework for enhancing conversational recommender systems by incorporating educational value, and evaluates its effectiveness through a human study comparing SalesBot against professional salespeople.

Abstract: Making big purchases requires consumers to research or consult a salesperson to gain domain expertise. However, existing conversational recommender systems (CRS) often overlook users' lack of background knowledge, focusing solely on gathering preferences. 
In this work, we define a new problem space for conversational agents that aim to provide both product recommendations and educational value through mixed-type mixed-initiative dialog. 
We introduce SalesOps, a framework that facilitates the simulation and evaluation of such systems by leveraging recent advancements in large language models (LLMs).
We build SalesBot and ShopperBot, a pair of LLM-powered agents that can simulate either side of the framework. 
A comprehensive human study compares SalesBot against professional salespeople, revealing that although SalesBot approaches professional performance in terms of fluency and informativeness, it lags behind in recommendation quality. We emphasize the distinct limitations both face in providing truthful information, highlighting the challenges of ensuring faithfulness in the CRS context. We release our code and make all data available.

Salespeople vs SalesBot:
Exploring the Role of Educational Value in Conversational Recommender Systems
Lidiya Murakhovs'ka
Philippe Laban
Tian Xie
Caiming Xiong Chien-Sheng Wu
Salesforce Al Research (1 murakkhoskka, plaban, t.xie, cxiong, wu ww..jason)esalesforcee.
Abstract
Seller
Hey! What can help you with?
g
Making big purchases requires consumers to re- search or consult a salesperson to gain domain expertise. However. existing conversational rec- ommender systems (CRS) often overlook users' lack of background knowledge, focusing solely gathering preferences. In this work. de- fine new problem space for conversational agents that aim to provide both product rec- ommendations and educational value through mixed-type mixed-initiative dialog. We intro- duce SalesOps, a framework that facilitates the simulation and evaluation of such systems by leveraging recent advancements large lan- guage models (LI (LL.Ms). We build SalesBot and ShopperBot, pair of LLM- -.M--poeere agents that can simulate either side of the framework. A comprehensive human study compares Sales- Bot against professional salespeople, reveal- ing that although SalesBot approaches profes- sional performance in terms of fluency and in- formativeness, it lags behind in recommenda- tion quality. We emphasize the distinct limita- tions both face in providing truthful informa- tion, highlighting the challenges of ensuring faithfulness in the CRS context. We release code and make all data available !
Hi, I'm looking
Buying
Product Catalog
Guide
Alright,
types
vacuums. [-.]] Do you know
which one like?
No, I'm not sure which one need
Vac
Vac
Do you budget in mind?
Shopper
Yes, ideally under $200.
8/6
For house with lots of carpets, recommend Vac 3.
Budget: 4100-200
3br
Thanks, I'll take it! [ACCEPS]
Flooza:
carpeta
Figure The SalesOps framework a conversational recommendation system simulation involving: (1)
Seller viewing a buying guide and product catalog, (2) Shopper gradually learning about shopping preferences. Either can be simulated using LLM-based system.
dialog and assist users with underspecified goals (Kim et al., 2020; Liu et al., 2020, 2022). How- ever, most CRS remain focused on domains in- volving content recommendation such movies, books, music, news, etc, which require a differ- ent type of recommendation strategy than the e- commerce space where consumers often need a certain level of background knowledge to under- stand their options (Laban et al., 2020; Papenmeier et al., 2022). In content recommendation domains, CRS achieve success by questioning user about previous content consumption and retrieving similar content. This strategy is not valid for the sale of complex products, as prior user habits do not inform a buyer's item-specific preferences.
1 Introduction
Conversational recommender systems (CRS) use multi-turn dialog to understand user preferences, gather information, and provide suitable recom- mendations (Gao et al., 2021). They have gained significant attention from academia and industry due to their flexibility compared to one-shot recom- mender systems. CRS allow users to dynamically refine their preferences and express feedback on recommended items through chat.
While many traditional CRS are built under a "System Ask-User Answer" paradigm (Zhang et al., 2018b), some recent works have identified the crit- ical need to support mixed-type, mixed inittaaivie
In this work. we focus on complex products with multiple attributes that would usually require sig- nificant expertise or a salesperson consultation to make an informed purchase decision on -e.g., TVs,
chttps://gittee http://gttb..comssssssssrrr//ss/

---

guitars, etc. We associate the ambiguous user goals often present in this real-life setting with the lack of knowledge about the product domain and sug- gest the need for the agent to provide educational value in addition to fulfilling the recommendation objective. We address this new problem space by introducing SalesOps. a framework that facilitates the recreation of realistic CRS scenarios and agent evaluation, as shown in Figure 1. Our method in- corporates several novel features.
2
Related Work
2.1 Conversational Recommendation
Since the introduction of conversational recommen- dation systems (Sun and Zhang, 2018), the field has grown to a wide range of task formulations, set- tings, and application scenarios (Gao al., 2021; Deng et al., 2023). Recent works explore conver- sational systems in the context of multi-type and mixed-initiative dialogs (Liu et al., 2020, 2022). They release sizable datasets in Chinese to advance the field toward supporting users with undefined goals. However, these primarily cover content rec- ommendation domains such movies, food, and news. As such, they do not directly address the challenges the e-commerce domain poses. In this work we extend these ideas to the specific context of e-commerce, where the focus is on providing recommendations for complex products.
We utilize existing buying guides as a knowl- edge source for the sales agent. We provide the seller with relevant guide in addition to product catalog, which enables them to edu- cate shoppers on the complex product space.
Shopping preferences are gradually revealed to the shopper during the course of conver- sation in order to simulate the underspecified goal scenario of a typical uninformed shopper.
In the e-commerce space, Fu et al. (2020) presented COOKIE CRS dataset constructed synthetically from user reviews. Most recently, Bernard and Balog (2023) presented a small multi- goal human-to-human conversational dataset with 64 chats for e-commerce. While closely related to our task, they do not aim to simulate ambiguous user preferences or target the educational objective; instead, they reveal all user requirements at once.
A mulli-dimenssonal evaluation framework is designed to measure sales agent performance in terms of (a) quality of final recommenda- tion, (b) educational value to the shopper, and (c) fluency and professionalism.
We leverage recent progress made in large lan- guage models (LLMs), which has enabled increas- ingly powerful conversational agents, to build pair of agents SalesBot and ShopperBot which can simulate either side in the SalesOps framework. Thus, facilitating evaluation in any of the following settings: human-human, human-bot, bot-human, and bot. We critically examine the components of these agents to understand where modern LLMs have the most impact on CRS.
While existing CRS are effective, criticism can be raised regarding the incorporation of expert knowledge into the user experience. While some systems provide explanations for recommendations (Chen et al., 2020; Wen et al., 2022), their primary focus is not on educating the user about the dif- ferent options available. This highlights the need for research into new methods of integrating expert knowledge into CRS.
We recruit 15 professional salespeople to study the gap between human experts and SalesBot at complex product conversational recommendations. The results reveal that although SalesBot matches professionals in tone politeness and educational quality, there remain minor gaps in the quality of recommended products. We also perform a prelim- inary analysis of faithfulness within the SalesOps framework, which is important in domains involy- ing generative AI as they enter the applied setting. Our NLI-based analysis (Laban et al., 2022) re- veals that salespeople can use strategies that may seem unfaithful, to upsell or simplify technical de- tails. These results highlight the practical chal- lenges of implementing robust faithfulness checks in domains like conversational sales.
2.2 Showledge-Groundde Dialog
Knowledge -rolugdddgr response generation in dia- logue systems has been explored for years in both task-oriented (Madotto et al., 2018: Li et al., 2022) and open-domain systems (Zhang et al., 2018a; Shuster et al., 2022a; Thoppilan et al., 2022; Shus- ter et al., 2022b). However, it is important to note that most existing approaches focus on passively responding to user queries rather than proactively conveying the knowledge.
A study by Cai et al. (2022) proposes teacher- bot that transmits information in passages to stu- dents, aiming to help them acquire new knowledge through conversation. While their work focuses

---

Unlike previous approaches that utilize databases with thousands of items, we deliberately limit the product catalog approximately 30 items per category to mirror the curated selection of a typical store (Fu et al., 2020; Bernard and Balog, 2023). This also significantly impacts human sellers' ability to complete the task as they need to familiarize themselves with the products available to effectively perform their role. However, the automated nature of the creation process would allow us to expand the product catalog efficiently. solely on the educational objective, our work com- bines it with CRS, suggesting suitable products while educating users about the relevant domain.
uct Catalog and the Buying Guide are the content elements accessible to the Seller, and the Shopping Preferences are accessible to the Shopper.
Some CRS have incorporated knowledge graphs (KGs) as complementary resources for analyzing item- and attribute-level user preferences (Zhang et al., 2021; Zhou et al., 2021; Ren et al., 2023). The primary objective of KGs is to enhance recom- mendations or aid preference elicitation rather than assisting sellers in answering inquiries or proac- tively conveying knowledge. Other works (Schick et al., 2023; Peng et al., 2023) focus on augmenting LLMs with external knowledge to address common pitfalls like hallucinations and lack of up-to-date information. This can greatly benefit our goal: thus, we leverage and augment LLMs with external tools build conversational agents in this work.
Importantly, we populated all content elements for the six product categories we include in our initial relcase, but we aim for the procedures to be automatable. so they can be expanded to new prod- ucts, unlike previous resources that can become outdated (Ni al., 2019).
3.1.1 Product Catalog
In the SalesOps framework, the Seller has access to fixed list of products that can be recommended to the Shopper. Each product consists of (I) unique ID, (2) product name, (3) price, (4) a product description, and (5) a feature set.
When creating the product catalogs, we initially considered leveraging the Amazon Product Re- views dataset (Ni et al., 2019). However, we found that the products in the dataset are outdated (1997- 2018), 8), which greatly impacts the conversations we obtain from a study with human participants in 2023. In many cases, product information and pric- ing are obsolete. products with the latest technology are missing (e.g., QLED TVs) and thus, the product catalog misaligns with updated Buying Guides and participant expectations.
2.3 Evaluation and User Simulation
Designing effective evaluation protocols has long been a challenge for various conversational systems (Deriu et al., 2021). Interacting with real users is costly, prompting the adoption of user simulators to assess proactive interactions in dialogue systems (Deng et al., 2023). Sekulic et al. (2022) utilized a GPT-2-based generative agent to simulate users in Conversational Question Answering (ConvQA) set- ting. We build upon their approach and incorporate the latest advancements in LLMs. We also intro- duce novel concept of gradually incorporating user preferences into the simulation process.
We generate synthetic product entries using an LLM since web-scraping an up-to-date product catalog can lead to limitations in terms of open sourcing. We first repeatedly prompt the LLM to generate diverse list of average of 30 product names for a given category (c.g., TVs). We then prompt the model for each product name to gen- erate realistic product metadata, including a title, description, price, and feature list. Appendix A presents more details of this process.
3 SalesOps Framework
We now describe the SalesOps framework, allow- ing to study CRS systems in terms of educational and recommendation objectives for complex prod- uct scenarios. SalesOps, two actors the Seller and the Shopper have a conversation that begins with a Shopper request and ends once the Seller makes a product recommendation that the Shopper accepts. Each actor gets access to specific content elements that assist them in completing the task. We first describe the three content elements and the procedure used to generate them at scale, and then we describe the constraint we put on actors' access to the content to create realistic sales conversations.
3.1 Content Elements
During a SalesOps conversation, the Seller can decide on their turn to recommend one or several items whose details will be included in a subse- quent message to the shopper (see Appendix C for
As illustrated in Figure 1, for any given product (i.c., vacuums), three content elements are required to instantiate the SalesOps framework. The Prod-

---

(e.g. ["1", "2-4", "5-9", "I0+"I), Although we attempt to have mutually exclusive questions, it is inevitable for some combinations to be improbable (e.g., a very high-capacity coffee maker for the smallest budget), thus, we leverage LLMs in a third step to select 10 diverse but realistic combinations of the preferences.
coéfee Jakore
1. Generated Quastions
Generated Options $150, 66001 [3, 2-4, 3-2, 151] [Linitess intintted
poir luigee? ausy zupa day de you deabk? lov waeh cocater space do you taow?
Shopping Preferences
Unlike prior work that reveals the shopper pref- erences in their entirety when the conversation is initiated (Liu et al., 2022; Bernard and Balog, 2023), we choose to reveal preferences to the Shop- per gradually during the conversation, providing more realistic simulation of an underspecified buying experience (Kim et al., 2020).
capa you space "Louited*
Figure 2: Shopper Preference Generation Pipeline using ChatGPT. Example for the Coffee Makers category.
To achieve this objective, for each Shopper turn in the conversation, we extract the last Seller mes- sage, and use a semantic similarity model* to detect whether the utterance corresponds to a question re- lated to one of the preferences. If the similarity passes a manually selected threshold, the related preference is revealed to the Shopper, and they can choose leverage the additional information in their response. We note that the system reveals most one preference per Shopper turn and does not enforce that all preferences are revealed. We intend these choices to simulate a realistic conversational experience for the Shopper and Seller.
an example conversation).
3.1.2 Buying Guide
Professional salespeople often receive training or rely on technical documentation to effectively sell complex products. We proxy this expert knowl- edge through leveraging publicly available buying guides. Buying guides, such as ones available on BestBuy² or Consumer Reports", are often writ- ten by professionals to help coach buyers on the decision-making process so that they can determine the best option for themselves. For each product category, we retrieve the top five articles from the C4 corpus (Raffel et al., 2019) that match the search query "[PRODUCT] Buying Guide", and select the most appropriate one.
3.2 SalesOps Actors
In SalesOps, the two actors the Seller and the Shopper - can either be simulated by an LLM- based system or enacted by person such as a sales professional or crowd worker. We briefly introduce the considerations for each actor.
On average, the buying guide we select for each product category is 2,500 words and 50 paragraphs long. Selected buying guides are diverse in their or- ganization, with some being organized by shopper persona, shopping budget, major item subcat- egories (e.g., drip vs. espresso machines). The heterogeneity the layout of buying guides goes towards creating realistic experimental setting in which the layout of knowledge documents for a particular product might not be known in advance.
The Seller has access to the Product Catalog and Buying Guide during SalesOps conversation, corresponding roughly 65 paragraphs which is a large amount of content both for humans and a system enacting the role. We estimate that human enacting this role would require roughly 30 min- utes at an average reading speed of 200 words per minute to read all the content. In our experiments with professional salespeople, they were each pro- vided period of reading time to get familiar with the content prior to participating in conversations. In our automated Seller implementation, we lever- age retrieval system to efficiently provide the con- tent to the LLM components with limited context lengths.
3.1.3 Shopping Preferences
Figure introduces the three-step method we use to obtain shopping preferences for a given product category, which relies on LLMs. First, we generate a list of five possible questions the Seller may ask based on the buying guide (e.g., "How many cups of coffee do you drink per day?"). Second, generate for each question set of answer options
Phttps://bestbur come com/
crntrrencettrrsssfrrrssss paraphrase mwweraassemmllttte ti
Phttps: //wwww www.umerreports.coooo /

---

Appendix D goes over the SalesOps Seller User Interface, built using the Mephisto library (Urbanek and Ringshia, 2023). The interface is extended for several aspects: (1) contains the Buying Guide and a search interface to the Product Catalog, (2) the Seller must select which paragraphs of the Buying Guide they are leveraging in crafting their response (if any), (3) the post-chat questionnaire which asks select utterances where Shopper revealed their preferences and rate the conversation partner.
Conversation History
Action Decision
Know Leggl
Recommend
Chat
Query Gen.
Query Gen.
Retrieval
Retrieval
Buying Guide
Product Catalog
Vac Vac
During a chat, we further log all product search queries. As a result, the metadata generated when implementing the SalesOps framework can be use- ful for tasks beyond the CRS setting, such as knowledge-groumdee response generation, conver- sational summarization, and query generation.
Vacc Var
Response Generation
Regeneration
SalesBot Response
the other hand, the Shopper only pro- vided with the product category they are tasked with shopping for at the initial stage of SalesOps chat. Shopping preferences are revealed based on the Seller's s questioning the conversation unfolds. The Shopper interface, presented in Appendix D requires fewer adaptations of Mephisto than the Seller interface: (1) we repurpose one-sided "Coor- dinator" messages reveal Shopping Preferences during the conversation, and (2) when a product recommendation is suggested by the Seller, the Shopper interface displays buttons to accept or re- ject the item.
Figure 3: SalesBot architecture to generate Seller response, based on two Content Elements and the con- versation history. LLM-bbased modules are in yellow.
scenario - i.e., if too few preferences were revealed at this point in the conversation, do not want the shopper accept an item that would not satisfy the whole set P.
We note that the Shopping Preferences are far from a comprehensive listing of all questions that could occur when shopping for a complex product category. The ShopperBot is instructed to make its own decisions when choices not in P (e.g., the preferred color of coffee machine) and fluently converse with the Seller.
4 Bots Implementation
We now present ShopperBot and SalesBot, the LLM-based implementations (ChatGPT our ex- periment) that can simulate both sides of the Sale- sOps framework.
Qualitatively, we have confirmed our experi- ments that ShopperBot can provide subjective and unique preferences, realistically simulating a hu- man shopper.
4.1 ShopperBot
ShopperBot's goal is to generate responses in ac-
4.2 SalesBot
cordance with the provided set of preferences (P), consisting of several question-answer pairs (q-a). We achieve this objective by prompting LLMs with (a) natural language instruction act shopper seeking [PRODUCT] (e.g., a TV), (b) a list of cur- rently revealed shopping preferences, and (c) the chat history, at every turn in the conversation. Full prompt can be seen in Appendix B.
SalesBot has access to two main external tools 1) knowledge search and 2) product search. As shown in Figure 3, it generates dialogue responses using a series of modules. Each module is independent component of the overall system.
4.2.1 Action Decision
This module decides which tool to use based on the current conversation history. Selecting from the following: Knowledge Search (details in section 4.2.2), Product Search (dctails in section 4.2.3). Response Generation (details section 4.2.4). We query an LLM to make this choice and pro-
When the latest seller's utterance includes rec- ommendation of an item, ShopperBot instructed include [ACCEPT] or [REJECT] token in its reply. It will base this decision on the whole set of prefer- ences (P) to ensure consistency with the simulated

---

vide natural language instructions on when to use each of the available tools our prompt.
to length. This forces SalesBot's responses be concise and contain full sentences.
4.2.2 Knowledge Search
5 Evaluation Criteria
Knowledge Search module's main purpose is to educate the user by incorporating expert domain knowledge into the conversation. It is made up query generation, and 2) retrieval. We ask the LLM to generate a query based on the chat history.
Along with the SalesOps framework, we propose multi-dimensional evaluation that defines suc- cess for the Seller along three axes: (1) recom- mendation quality, which verifies whether the recommendations of the Seller are compatible with the Shopper's preferences, (2) informativeness, which checks whether the Seller provides educa- tional value to the Shopper, and (3) fluency which evaluates the Seller's ability to communicate pro- fessionally and concisely. We next define the met- rics - both automatic and manual used to evaluate each axis.
of two components:
We then use a FAISS retriever to lookup rele- vant knowledge article paragraphs (Johnson et al., 2019). We concatenate top 3 paragraphs (separated by "An'n") and feed it external knowledge the Response Generation module described in section 4.2.4.
4.2.3 Product Search
Recent work shows the promise of using LLMs for evaluation across many NLP tasks (Liu et al., 2023; Fu et al., 2023; Laban et al., 2023). Fol- lowing this thread work, we leverage GPT-4 (OpenAI, 2023) for automatic evaluation purposes in several of our proposed metrics.
Product Search module's goal is finding relevant items to recommend to the user. Similar to the above, this module is made up of 1) query gener- ation, and 2) retrieval. We embed each product's information (i.e., title, description, price, and fea- ture list) using the Sentence Transformer model and retrieve the top products based on the query embedding (obtained using the same model). Same as in Knowledge Search, we concatenate the re- sults. Note that in some cases, we may not need all 4 products, for example. if the shopper asked follow-up question about certain product, the response should not include all 4 retrieved items. We leave up to the Response Generation module to determine which products should be mentioned based on the chat history.
5.1 Recommendation Quality
Accurate recommendations that match shopper preferences are core expectation of CRS. The authors of the paper manually annotated each prod- uet category and its 10 corresponding Shopping Preferences for all acceptable product recommen- dations as the ground truth. On average, given shopping preference configuration yielded accept- able products from a product catalog of 30 items. Thus, for completed SalesOps conversation, we compute recommendation accuracy (Rec).
4.2.4 Response Generation
5.2 Informativeness
Based on the Action Decision module, response generation can either include external information (e.g.. buying guide excerpts, product information) or not. We thus, write two separate prompts to re- spond to the shopper. Response Generation with External Knowledge based on: Chat History, Ac- tion Selected, Query Generated, Retrieved Results. Response Generation without External Knowledge is based solely the chat history.
We propose two metrics to measure the informa- tiveness of the Seller during a conversation.
First, we leverage NLI-based model mea- sure the content overlap between the Seller's utter- ances and the buying guide, as such model has been shown to perform competitively on tasks involving factual similarity (Laban et al., 2022; Fabbri et al., 2022). Specifically, we calculate the % of the buy- ing guide sentences that are entailed at least once by seller utterance (Infe).
We additionally implement a Regeneration sub- module to rewrite the final response if needed. when
Second, assess the shopper's knowledge through a quiz that we designed which consists of 3 multiple-choice questions that can be an- swered using the buying guide (examples in Ap- pendix C. We then ask croke-workers to an-
We place a limit on max_ prompting the LLM and ask it to rewrite the pre- viously generated response if it was cut off due
"sentence- wrw.ssoorees//lllmmpppe... base-v2

---

Agent
Flu. f
Flu; f
Infe f
Rec f
6.2
Results
Ada-RG
1.41
0.00
0.16
0.16
We generated 150 conversations on the six product categories between ShopperBot and the five ver- sions of SalesBots, and computed automatic results which are summarized in Table 1. Overall. the use of an LLM is most crucial in the Response Generation component but is beneficial across all components.
Rule-AD
4.82
0.59
0.86
0.40
Key-QG
4.97
0.72
0.71
0.36
No-ReGen 4.99
0.85
0.74
0.47
SalesBot
4.99
0.91
0.74
0.44
Table I: Results of the ablation study of SalesBot com- ponents using GPT4-based evaluation metrics.
Replacing ChatGPT with smaller LLM the Response Generation component leads to the largest degradation across the board, confirming that smaller models are unable to handle the com- plex task of tying together the conversation history and external knowledge to generate a concise, co- herent, fluent, and informative response.
swer each knowledge question solely consider- ing the conversation, with the option of choosing "Cannot answer based on the conversation", We report the % of correct answers on the knowl- edge questions (In.fg).
The rule-based action decision module leads to improved informativeness (as the system relies more heavily on the knowledge action), at the cost of fluency and recommendation quality.
5.3 Fluency
We frame questions to measure the fluency and professionalism of the Seller:
The keyword--ased retrieval ablation leads to lower a 20% decrease in recommendation qual- ity, confirming that generation of retrieval queries benefits from the generative flexibility of an LLM. Finally, adding the regeneration component leads to boost fluency, at the cost of minor recom- mendation quality drop.
Flue: How would you rate the salesperson's communication skills? (scale: 1-5)
Flu; Do you think the seller in the given chat is: (i) human or (ii) a bot? (Yes/No)
We perform annotation for the two fluency met- rics both manually by recruiting corkers (sce Appendix C) as well as by prompting GPT-4 answer both questions.
As a result, we recommend that designers of con- versational agents: (1) Leverage LLMs Response Generation, (2) Integrate Generative Flexibility in Retrieval Queries, and (3) Utilize Regeneration for Improved Fluency.
6 Ablation Study
SalesBot contains LLM-based components, which we swap with baselines to understand the impact of using an LLM for each component.
Salespeople SalesBot
We aim to study the qualitative differences between SalesBot and professional salespeople to compre- hend the effect of deploying such systems in real- world settings (Guo al., 2023), and perform extensive human evaluation of SalesBot and Sales- people within the SalesOps framework.
6.1 Baselines
We implement four ablations of SalesBot:
We replace ChatGPT with the smaller GPT3 text-ada-001 model in the response gener- ation module.
Ada-RG
7.1 Experiment Setup
Rule-AD We replace the Action Decision module with rule-based system: select the "Knowledge" action for the first 6 turns, and the "Recommend" action afterwards.
We recruit 15 professional salespeople across a di- verse set of industries (e.g.. insurance, retail, B2B sales) through Userlnterviews 6 They were given I-hour onboarding session covering the SalesOps framework, reading a Buying Guide and Product Catalog, and completing warmup conversations. Participants then completed up to 15 SalesOps con- versations with ShopperBot, which took an average of 3 hours.
Key-QG We replace Query Generation modules with keyword method: extract five keywords from the latest utterance and concatenate them query.
No-ReGen We experiment with removing Regen- eration module from response generation.
http://wwww.erirneeriieee.s.s

---

GPT-4 Explanation of Unfaithful Content
Statistics (avg)
Nb. words of Seller utterance
62.5
35.3
Salesperson. The salesperson incorrectly men-
Nb. words of Shopper utterance
20.8
20.7
tioned that the Cuisinart DCC-3200P1 has hot water dispenser, which is not mentioned in the doc- ument.
Nb. of turns
11.9
12.9
Nb. turns before the first rec.
6.0
7.9
Nb. of recommendations
2.6
2.4
Nb. of triggered revelations
1.7
2.8
SalesBot. The salesperson incorrectly stated that some drip coffee makers and espresso machines have the option for pre-measured pods or capsules, The document only mentions single-serve coffee makers as having the option for pre-measured pods.
% Correct recommendations (Rec)
44
Information Quiz Score (f)
32,9
31,8
Fluency Score (Flug)
4.4
% Is Human (Flu,)
55
80
Table 2: Comparison of I SalesBot and & Salesper- son conversation statistics (top) and human evalua- tion metrics (bottom).
Table 3: Example of GPT4-based detection of unfaithful behavior in both k SalesBot and & Salespeople.
cases and find that there are several reasons for mismatches, including salespeople upselling prod- ucts (i.e., convincing Shoppers to accept product beyond their initial budget). We study this phe- nomenon through the lens of faithfulness analysis. We provide GPT-4 with Buying Guides, the Product Catalog and the full conversation, and prompt it with identifying whether the Seller pro- vides advice that is inconsistent with existing in- formation. It detects that for both types of Sell- ers, roughly one in four chats contains unfaithful claims.
In parallel, we generated 150 SalesOps conver- sations between SalesBot and ShopperBot for the same set of preferences. Unlike the ablation study in which we perform the evaluation with GPT-4, we recruit crowd workers from Amazon M'Turk? 7
to complete evaluation (interface in Appendix C).
7.2 Results
Table 2 presents statistics and evaluation results of the comparison between professional Salespeople and SalesBot. Overall, SalesBot's utterances are almost twice long. It makes its first recommen- dation earlier and makes slightly more recommen- dations in total than professional salespeople.
We provide examples of identified inconsis- tencies in Table 3. Our findings suggest that while SalesBot may occasionally hallucinate due to known LLM-bound limitations, salespeople can also exhibit unfaithful behavior, usually in the form of (a) upselling and (b) answering follow-up questions without knowing the true answer. Both are motivated by the desire to close the sale; how- ever, this confirms the challenge of evaluating faith- fulness in the sales domain, for which successful sales strategies might require unbacked claims.
Looking at the human evaluation, crowd work- ers were largely able to distinguish between Sales- Bot and professionals, as they were much more likely to believe the Seller was human for profes- sionals (80%) than for SalesBot (55%), yet Sales- Bot achieved higher Likert fluency score. This is likely due salespeople acting more casual in conversations, making occasional typos which come across as less professional.
Finally, even though professionals write utter- ances that are nearly half the length, they achieve higher recommendation quality and almost equal informativeness. This pair of results confirms that there is still a large gap between LLMs and pro- fessional salespeople with respect to conciseness.
Conclusion
In this paper we introduce SalesOps, a flexible framework for simulating realistic CRS scenarios in the context of complex product sales, and pro- pose an evaluation paradigm for such systems. Our framework provides researchers and practitioners with valuable tool for assessing the effectiveness and performance of conversational agents in sales settings. By developing SalesBot and ShopperBot within this framework, we gain insights into the individual components and their impact on conver- sational performance. Through a comprehensive
7.3 Faithfulness
In absolute terms, both SalesBot and Salespcople achieve low rates of correct recommendations (< 55%). We perform qualitative analysis of failure
Phttps: / //wwww mturk. com/

---

human study, we identify gaps in product recom- mendation quality and provide faithfulness anal- ysis of both automated and human sellers. These contributions advance the understanding and devel- opment of CRS, paving the way for improved sales interactions and user experiences.
task instructions and provide sample chats. After- ward, the participants had the freedom to complete the study at their own pace. Additionally, human evaluation was conducted with Amazon Mechani- cal Turk workers, who were compensated for their contributions.
We discuss the potential impact of deploying conversational agents, such as SalesBot, in real- scenarios. The study highlights both the strengths and limitations of SalesBot compared to profes- sional salespeople. The evaluation reveals gaps in recommendation quality and examines the chal- lenges of ensuring content faithfulness in the CRS context. We emphasize the importance of consid- ering the implications of generative Al systems in domains like conversational sales, where there sev- eral ethical concerns may arise related to upselling and providing factually accurate information.
9 Limitations
In this work, we heavily on LLMs to build the conversational agents (SalesBot and ShopperBot) within our framework. While LLMs have shown significant advancements in generating human-like responses and engaging multi-turn conversations, they still suffer from hallucinations as shown in our Faithfulness analysis, and thus impact the overall performance of the system.
Additionally, our evaluation primarily focuses on three aspects: the quality of the final recom- mendation, educational value to the shopper, and fhenncy/roffsssionaiiss. While these aspects are important, there are other dimensions that could be relevant, such as user satisfaction, persuasion skills, and diversity recommendations. We leave further exploration into CRS evaluation to future work.
References
Nolwenn Bernard and Krisztian Balog. 2023. Mg- shopdial: A multi-goal conversational dataset for e-commerce. ArXiv, abs/2304. 126626
Pengshan Cai, Hui Wan, Fei Liu, Mo Yu, Hong Yu, and Sachindra Joshi. 2022. Learning as conversa- tion: Dialogue systems reinforced for information acquisition. ArXiv, abs/2205.
We limit our evaluation to the chat-based sales experience, even though most sales conversations involving a shopper and salesperson happen in au- dio form, either physical store or on the phone. Prior work has shown that adapting conversational content to the audio format is non-trivial (Kang and Gretzel, 2012), and requires modifications to remain natural and engaging (Laban et al., 2022). We leave it to future work to further adapt SalesOps and SalesBot to the audio setting.
Zhongxia Chen, Xiting Wang, Xing Xie, Mehul Parsana, Akshay Soni, Xiang Ao, and Enhong Chen. 2020. To- wards explainable conversational recommendation. In International Joint Conference Artificial Intelli- gence.
Yang Deng. Wenqiang Lei, Wai Lam, and Tat-Seng Chua. 2023. A survey proactive dialogue sys- tems: Problems, methods, and prospects. ArXiv, Abs/2305.0275m)
Ethical Statement
Jan Deriu, Alvaro Rodrigo, Aranixa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021. Survey evaluation methods for dialogue systems. Artificial Intelligence Review. 54:755-810.
The ethical considerations in this work primarily revolve around the interactions with human partici- pants and the potential implications of deploying conversational agents in real-life settings.
The study involved 15 professional salespeople who were recruited through User Interviews plat- form and participated voluntarily. The study aimed to ensure representation and inclusivity in its par- ticipant selection process. We recruited individuals of different genders, professionals of all levels (as- sociate, manager, director, VP), and spanning wide range of age groups from 19 to 65. They re- ceived compensation for their time and effort. An onboarding session was conducted to explain the
Alexander Fabbri, Chien-Shena Shien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA- based factual consistency evaluation for summariza- tion. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 2587-2601, Seattle, United States. Asso- ciation for Computational Linguistics.
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. ArXiv. abs/2302.04466.

---

Zuohui Fu, Yikun Xian, Yaxin Zhu, Yongfeng Zhang. and Gerard de Melo. 2020. Cookie: A dataset for con- versational recommendation over knowledge graphs e-commerce. ArXiv, abs/2008.092.
Zeming Liu, Jun Xu, Zeyang Lei, Haifeng Wang, Zheng- Yu Niu, and Hua Wu. 2022. Where to go for the holi- days: Towards mixed-type dialogs for clarification of user goals. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1024-1034, Dublin, Ireland. Association for Computational Linguisties.
Chongming Gao. Wengiang Lei, Xiangnan He, M. de Ri- jke, and Tat-Seng Chua. 2021. Advances and chal- lenges in conversational recommender systems: A survey. AI Open, 2: 120-
Andrea Madotto, Chien- Chien-Sheng Wu, and Pascale Fung. 2018. Mem2Seq: Effectively incorporating knowl- edge bases into end-to-end task brii-oriented dialog sys- tems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Vol- une 1: Long Pupers), pages 468-1478, Melbourne, Australia, Association for Computational Linguisties.
Biyang Guo, Xin Zhang. Ziyuan Wang. Minqi Jiang. Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection, ArXiv, abs/2301.07597.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 73))5555-477
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labell reviews and fine- ain--griied aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLPP) pages 188-197, Hong Kong, China. Association for Computational Lin- guistics.
M. Kang and U. Gretzel. 2012. Effects of podcast tours on tourist experiences in a national park. Tourism Management, 33:440-455.
Seokhwan Kim, Mihail Eric, Karthik Gopalakrishnan, Behnam Hedayatnia, Yang Liu, and Dilek Hakkani- Tur. 2020. Beyond domain APIs: Task-oriented con- versational modeling with unstructured knowledge access. In Proceedings of the 21th Annual Meet- ing of the Special Interest Group on Discourse and Dialogue, pages 278-289, 1st virtual meeting. Asso- ciation for Computational Linguistics.
OpenAL. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Andrea Papenmeier, Alexander Frummet, and Dagmar Kern. 2022. "mhm..." conversational strategies for product search assistants. In Proceedings of the 2022 Conference on Human Information Interaction and Retrieval, CHIIR '22, page 36-46, New York, NY, USA. Association for Computing Machinery.
Philippe Laban, John Canny, and Marti A Hearst. 2020. What's the latest? a question-driven news chathot. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 380-387.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng. Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Lidén, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feed- back. ArXiv, abs/2302. 12813.
Philippe Laban, Wojeiech Kryscinski, Divyansh Agar- wal, Alexander R. Fabbri, Caiming Xiong, Shafiq R. Joty, and Chien- Sheng Wu. 2023. Lims as factual reasoners: Insights from existing benchmarks and beyond, ArXiv, abs/2305. 14540.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with unified text-to-text trans- former. arXiv e-prints.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NLI- based models for inconsistency detection in summa- rization. Transactions of the Association for Compu- tational Linguistics, 10:163-177, 1777
Xuhui Ren, Tong Chen, Quoe Viet Hung Nguyen, Li zhen Cui, Zi-Liang Huang, and Hongzhi Yin. 2023. Explicit knowledge graph reasoning for conversa- tional recommendation. ArXiv, fos/2305.00783.
Miaoran Li, Baolin Peng. Jianfeng Gao. and Zhu Zhang. 2022. Opera: Harmonizing task-oriented dialogs and information secking experience. ArXiv, abs/2206.15 12449.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. ArXiv, abs/2302.04761.
Yang Liu, Dan Iter, Yichong Xu. Shuo Wang, Ruochen Xu, and Chenguang Zhu. 2023, G-eval: Nlg evalua- tion using gpt-4 with better human alignment. ArXiv. abs/2303.166344.
Zeming Liu, Haifeng Wang. Zheng- Yu Niu, Hua Wu. Wanxiang Che, and Ting Liu. 2020. Towards conver- sational recommendation over multi-type dialogs. In Annual Meeting of the Association for Computational Linguistics.
Ivan Sekulic, Mohammad Aliannejadi, and Fabio A. Crestani. 2022. Evaluating mixed-initiative conver- sational search systems via user simulation. Proceed- ings of the Fifteemth ACM International Conference on Web Search and Data Mining.

---

Kurt Shuster, Mojtaba Komeili. Leonard Adolphs. Stephen Roller, Arthur D. Szlam, and Jason We- ston. 2022a. Language models that seek for knowl- edge: Modular search generation for dialogue and prompt completion. In Conference on Empirical Methods in Natural Language Processing.
Conference on Information and Knowledge Manage- ment.
Jinfeng Zhou, Bo Wang, Ruifang He, and Yuexian Hou. 2021. Crfi: Improving conversational recommender systems via flexible fragments reasoning on knowl- edge graphs. In Conference on Empirical Methods in Natural Language Processing.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Bric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane. Morteza Behrooz, W.K.F. Ngan, Spencer Poff, Naman Goyal, Arthur D. Szlam, Y-Lan Boureau, Melanie Kam- badur, and Jason Weston. 2022b. Blenderbot 3: deployed conversational agent that continually learns to responsibly engage, ArXiv, abs/2208.03) 188
Yueming Sun and Yi Zhang. 2018. Conversational rec- ommender system. The 41 International ACM SI- GIR Conference Research & Development In- formation Retrieval.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee. Huaixiu Zheng. Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin. Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, A. Krivokon, Willard James Rusch, Marc Pick- ctt, Kathleen S. Mcier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Spraker, Ben Zevenhergen, Vin- odkumar Prabhakaran, Mark Díaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman- John, Josh Lee, Lora Aroyo, Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Areas
Claire Cui, Marian Croak, Ed Huai hsin Chi, and Quoe Le. 2022. Lamda: Language models for dialog applications. ArXiv, abs/2201.08239
Jack Urbanek and Pratik Ringshia 2023. Mephisto: framework for portable, reproducible, and iterative crowdsourcing.
Binghing Wen, Xiaoning Bu, and Chirag Shah. 2022. Eger: Explanation generation for conversational rec- ommendation. ArXiv, abs/2208.08033.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018a Sa. Per- sonalizing dialogue agents: I have dog, do you have pets too? arXiv preprint Ar0iv:/801. 07243..
Tong Zhang, Yong Liu, Peixiang Zhong, Chen Zhang, Hao Wang, and Chunyan Miao. 2021. Keers: To- wards knowledge-enriched conversational recommen- dation system. ArXiv, ths/210550826..
Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. 2018b. Towards conversational search and recommendation: System ask, user re- spond. Proceedings of the 27th ACM International

---

A Product Catalog Generation
For each product domain, we automatically generate Product Catalog following the procedure in Table 4.
1. Product Name Gen.
Generate list of top 30 [PRODUCT_NAME options in the following format: ("name": ...3
Include diverse list of options with different brands, sizes, price points for a variety of customers.
2. Product Metadata Gen.
Generate product description, features, and price based on the product name. Output should be the following json format:
"name":
"price": "..."
"description": "..."
"features": [...]
Table 4: Product Catalog Generation Prompts.
An example of the resulting product metadata is presented in Table 5.
name price
Samsung 55" Class S95B OLED 4K Smart Tizen TV
$1,699.99
description
Samsung OLED TV changes the game again with 8.3 million self lit pixels and ultra powerful 4K AI Neural Processing, all for a picture real, it's surreal. Add on Dolby Atmos @ sound built in, the latest Smart TV apps, and LaserSlim design and get viewing experience that's intensely cinematic.
features
["55 inch", "OLED Technology", "Neural Quantum Processor with 4K Upscaling". "Smart Calibration", "Connectivity with Bluetooth, RF, Wi-Fi, USB, HDMI, Ethernet (LAN), Digital Audio Out x (Optical)", "Supported internet services: Netflix, Google TV, Amazon Instant Video, YouTube, Browser"]
Table 5: Product Metadata Example the category.
B ShopperBot Design
B.1 Prompt
You are shopping online for (product). You haven 't done your research on this product and want to speak salesperson over chat to learn more and make an informed decision. Follow these rules:
- Chat with the salesperson to learn more about (product). They will be acting as a product expert, helping you make an informed purchasing decision. They may ask you questions to narrow down your options and find suitable product recommendation for you. Use your assigned preferences and incorporate them in your responses when appropriate but do not reveal them the salesperson right away or all at once. Only share maximum of assigned preference with the salesperson at a time. - Let the salesperson drive the conversation.

---

Ask questions when appropriate. Be curious and try learn more about [product] before making your decision.
Be realistic and stay consistent in your responses.
When the salesperson makes a recommendation, details with 'ACCEPT' and 'REJECT' in the message. Please consider whether the product satisfies your assigned preferences. If the recommended product meets your needs, generate [ACCEPT] token in your response. For example, "[ACCEPT] Thanks, I'll take it!".
see product
If the recommended product is not good fit, let the salesperson know (e.g. "this is too expensive") If you re not sure about the recommended product, ask follow-up questions (e.g. "could you explain the benefit of this feature ?") Do not generate more than response at time.
Your assigned preferences: preferences))
Follow the above rules to generate reply using your assigned preferences and the conversation history below:
Conversation history: )
Shopper:
B.2 Architecture
Figure 4 highlights ShopperBot's key components and the overall design flow.
Conversation History
salesperson recommending items?
Yea
No
|
Retrieval
Shopping Preferences
Relevant Shopping Preferences
Response Generation
ShopperBot
Response
Figure 4: ShopperBot architecture to generate response based on the selected shopping preferences.
Human Evaluation Questionnaire
We recruit 150 crowd workers from Amazon MTurk to complete a survey. The user interface is presented in Figure 5. Examples of True/False quiz questions per product category are shown Table 6.

---

Survey
Sales Chat Transcript
Miluk WorkerDD:
the carefully, ansveing questions.
X430YABCD
help you
transcript right
[Communication]
actually shopping sure what I'm loaking for: Can tell me bit more about the different types TV's you have available?
How would salesperson's communication skils??
Excellent
Do you think the salesperson ls:
your personal size preference, room size, and your mounting options. You'll want to keep in mind the width and weight the that you need an entertainment stand. Televisions can be as small about 30 inches large as inches moref), Do you have mind?
Certainly!
Human
your
you shopper
Human
[Informativeness]]
The resciution of TV relers the number vartical plsiss or that maki pictuns
I'm thinking something on the larger side, maybe between and 75 inches. What options do you have in that size range?
Cannot answer based the chat
True
False
display technology
have TV's ranging in price from $749 $9,809 that size range. What you be using TV mosty?
generally have better picture quality than 1080p TV with an LGD display tachnology
Cannot answer based on the chat True
False
bigger
quality
Well, I'm bit of shopper and want make sure got something that a while. I watch movies and sports, something good high refresh rate. Do you have any recommendations that might those proferences?
Cannot answer based on the chat True False
quality
[Product Recommendation)
your
"around
rate
for watching movies and sports!
size you looking toz?":
"a large size, naybe betveen. 65 and T5 inches"
"Is image quality priority for you?"
"very important"
Product Name: LG 66' Class NanoCell 99 Series LED 8K Price: 79.799
"Do you have any preferred
*I 1ike Samsung"
Description: Experience everything in more detail with LG NanoCell LG's composition, black, light and color scane, while the
let: natural 1ight?" "yes"
Figure S: Human Evaluation User Interface.

---

Product Category
Question
resolution of TV refers to number of vertical pixels that make up TV's picture. 4K with an OLED display technology and refresh rate 120Hz will generally have hetter picture quality than 1080p TV with an LCD display technology and refresh rate of 60Hz.
T
T
TV's with bigger screens always have better sound quality.
F
canister vacuum is best suited for large home with multiple floors und different types of flooring.
T
Vacuum Cleaners
you have pets, important choose
cleaner with HEPA filter
T
cupture pet hair dander.
Stick vacuums are not suitable for cleaning homes with stairs.
F
Memory foum mattresses mold to the shupe of your body.
T
Mattress
Gel memory foam mattresses are ideal for hot sleepers who want to stay cool night.
T
mattress with more coils always better than mattress with fewer coils.
F
The acoustic guitar the common guitar for first-time buyers. variety sizes,
T
Guitar
Acoustic guitars self-amplified instruments that most common shape is the Parlor.
F
Acoustic-ee elletricc guitars are easiest way to perform with an acoustic guitar, and they always come with an integrated pickup system.
F
laptop with a larger screen size is always better for gaming.
F
Laptop
laptop with dedicated gruphics card unnecessary for basic office tasks.
T
An SSD provides faster storage than HDD.
T
Drip coffee makers ideal making coffee person. Single-serve coffee makers are convenient because they require no measurement of coffee grinds or water.
F
Coffee Makers
T
Espresso machines can only make espresso shots.
F
Table 6: True/False quiz questions per product category.
SalesOps Framework User Interface
We present the SalesOps Framework's User Interface in Figure 6, displaying both the Seller View (a) and the Shopper view (b).

---

calsepasoon your touther buger thourr p protuct tor they insendeeii (LAPTOR) thenyying produzt detabeas bedldddbblloo
Cocrdinator:
Welcome onboard
PrODucT essage
panel zareluty technical dlralls section bearn to uso our task interfacc offootivoly.
Searc product OUFADASS)
Ovarch
Gocrcinetor:
Once vou/se dene medding trstructions, please start
assuming exampll,, you sond may holp you today?" lick of the chat.
laptop buying guide
Laokkags loow lsear oubarl teekitzz PDs clecaele, On the alsins lhey's uostadike having capabillyy number choices bicll sptop shappens better aver
You Hollo, how may help you Boday??
However, doeen't
anything you
sptiore isplope Chromebosks.
Your Partner: Hi then! I'm look for new laptop. Can you that?
Laptop Buying
purpose. general
faclor throad, can very diftrentt
capatil Bine, intended use prion.
Windows, MizBBooc Chromebook??
biggest
teptops Apple computer computer,
Sure thing! Do you have platoom prefarencee
Sand
(a) Seller View
Coandratto:
Task Instructions
proket edecstoons
Coordinet antor Please eus row shopper mamerted buyhng sppoadd
ceaae they my en auetie due you des un tu
Comernes, sssertivee mocerd snemank
spediic losking
by
cdeepronat.
maren ure Vie reporaes the oepenso roting hep you
an enee impaoten
partore
Your Portner hello
S me cas sia unei st miconne voso pictuae accosann you staplan
ovor canvertaie
fiyo beteeeses The
oafore udtmata
lobbiig cofiee
Techn cal detais
Your Portner now mach ach cs dinkk day?
sociee tha
Goundrattor Your soeig sclpeckocee many capa octhee do typics day?
cowe revere dery esl Plesee
the reedur button
esparan. earrale take thaiita accepaad ogpntti inacadd
wn wals. Aetir ooc natoa vitte snlleer woblling cuftee wis fyrvar ara suss atvalle youraoe condera.
hoi geniee is suscciant reperd,
massage
(b) Shopper View
Figure SalesOps Framework User Interface.


Reviews:
[{"review_id": "JnSdZn7gcX", "reviewer": "Reviewer_fUC4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for a thorough review of our work! We have carefully reviewed your feedback and would like to address the following points.\n\nRegarding the readability, we will modify and rethink the presentation of Figure 1 which depicts the overall framework. Figure 3 currently provides the technical components of SalesBot, and we will add a similar Figure for the ShopperBot system in the final version.\n\nRegarding sample sizes: we have 6 product categories (e.g., vacuum, TV, etc) each containing 30 products (e.g., iRobot Model X, etc), so in total 180 products. Limiting the number of products allowed salespeople to familiarize themselves with the product offering prior to selling.\nWe recruited 15 professional salespeople to conduct experiments. We preferred to recruit fewer people and train them well in order to obtain high quality chats. With our current number of participants, we observe meaningful quantitative differences between professionals and SalesBot. The goal of our work is not to collect a large-scale dataset or conduct a large-scale user study, instead we focus on the overall task definition, SalesOps framework design, and a first glance at sales automation.\n\nWe agree with the reviewer that the presentation of our Ablation results (Section 6.2) can be improved by providing concrete recommendations for designers of conversational agents. Namely, we will add the following recommendations: (1) Leverage LLMs in Response Generation (2) Balance Informativeness and Fluency through Action Decision Module (3) Integrate Generative Flexibility in Retrieval Queries (4) Utilize Regeneration for Improved Fluency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors present a novel idea of exploring the role of educational value in CRS, and the idea of utilizing LLMs to build this CRS seems technically sound.\n- The authors use a combination of human evaluation and NLI-based analysis to compare SalesBot and professional salespeople, providing a comprehensive evaluation of the agents. \n- The authors provide insights of the limitations of both SalesBot and professional salespeople in providing truthful information and how these challenges can be addressed in the CRS context. \n", "reasons_to_reject": "- The content organization can be better improved to increase the readability. More details on the SalesBot and ShopperBot design and development should be given to increase the clarity. A technical figure describing the proposed framework is missing, and thus fails to better elaborate how different component could cooperate with each other. \n- The sample size of products and professional salespeople used in the work is relatively small, which may limit the generalizability of the results. \n- The result of the ablation study can be better analyzed, and the potential usage of the findings for conversational agent design should be clearly addressed. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a conversational recommender systems (CRS) providing both product recommendations and educational value through mixed-type mixed-initiative dialog in e-commerce domain. By analyzing the necessary components of CRS, the authors utilize the LLM technology to build SalesBot and ShopperBot to simulate both sides in product purchasing. Experimental results reveal that SalesBot approaches professional performance in terms of fluency and informativeness, but lags behind in recommendation quality. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MEysn2obek", "reviewer": "Reviewer_LBGM", "summary": "", "questions": "1. Have you compared your system with other sales chatbots or recommender systems, including the ones that you have mentioned in your Related Work sections?\n2. In Table 1, it seems that using rule-based increase the informativeness of your system, why didn't you use this as the final configuration?\n3. What retrieval system did you use for both the Seller and the Shopper? \n4. Have you done ablation study on your Shopper Bot?\n5. What do you think we should do to improve the Seller Bot and Shopper Bot performace?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and provide actionable feedback!\n\nRegarding the fact that the SalesBot system does not achieve on-par performance with expert human sellers. This is indeed a major finding of our work and showing that AI can outperform human sales experts is not the goal of this work. Instead, we hope that the identified gap and our opensource framework can help direct future research towards the most important problems in this space. We will address this point in our Discussion section.\n\nWe believe that our novelty comes from the SalesOps framework, the overall task definition, and the findings from our user study. To the best of our knowledge, autonomous LLM-based agents like SalesBot have not yet been applied to CRS. Additionally, our SalesOps framework goes beyond prior work in terms of setup (with the gradual preference revelations and several knowledge sources provided) and the multi-aspect evaluation. We will improve the Related Work section to clearly reflect the novelty in our framework and individual system implementations.\n\nQ1: Regarding comparison to other chatbots/recommender systems, we agree that we could add more detailed information on the related work. In our preliminary experiments, we included a baseline which relied on BlenderBot3 3B (Shuster et al., 2022), however due to its model’s limited context length, we found it underperformed text-ada-001. Only recent models have the capacity to handle large context length windows required in our framework (due to Product DB), which limits our ability to compare to older systems. The ablation study we conducted includes common baselines as replacements to individual components of SalesBot, which provide insights into the necessity of the LLM in the components of SalesBot. \n\nQ2: We found that the rule-based action decision module leads to improved informativeness (as the system relies more heavily on the knowledge action), but at the cost of fluency and recommendation quality. We argue that recommendation quality should not be compromised to provide educational value, so we steer away from hard coding a set of knowledge-based turns into a system. This is also shown in a decrease in fluency, meaning that users of the system would quickly recognize these rules which might lead to unsatisfactory outcomes. Diving deeper into user satisfaction would be a promising direction of future research for CRS evaluations.\n\nQ3: We implemented retrieval using FAISS and \"sentence-transformers/all-mpnet-base-v2\" embedding model (sections 4.2.2 (knowledge search) and 4.2.3 (product search)).\n\nQ4: No, we mainly focus on SalesBot performance and leave the ShopperBot optimization to future work in User Simulation research.\n\nQ5: We will add a subsection to our Discussion section dedicated to Future Work. This section will propose 3 directions that we believe would lead to further improvements in systems similar to SalesBot. In summary, they are (1) applying faithfulness methods to CRS, (2) optimizing retrieval for recommendation quality, and (3) measuring user satisfaction to enable more iterative improvements of CRS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The author has done a thorough literature review and a thorough evaluations on their ideas.\n* The idea of using a ShopperBot and a SellerBot to help a customer in eCommerce situation is interesting.", "reasons_to_reject": "* Although the authors' proposed framework, as expected, provides a fluent chat responses, the quality of it's recommendation and informativeness are significantly below human salesperson\n* The proposed framework is not that novel\n* The authors have not compared their framework with other chatbots or recommender systems\n* The authors have not done a thorough discussions on what need to be done to improve the performance of a salesbot in this area", "questions_to_authors": "1. Have you compared your system with other sales chatbots or recommender systems, including the ones that you have mentioned in your Related Work sections?\n2. In Table 1, it seems that using rule-based increase the informativeness of your system, why didn't you use this as the final configuration?\n3. What retrieval system did you use for both the Seller and the Shopper? \n4. Have you done ablation study on your Shopper Bot?\n5. What do you think we should do to improve the Seller Bot and Shopper Bot performace?", "paper_topic_and_main_contributions": "In this paper, the authors have experimented with utilising LLM and Information Retrieval system to create a SalesBot, a chatbot that could help a customer choosing the right product to buy, and a ShopperBot, an LLM module that learns about the customer's preference based on the customer's answer and the SalesBot information. \nThis paper is a position papers on NLP engineering experiment.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zIxLqaRpJS", "reviewer": "Reviewer_Uvkj", "summary": "", "questions": "Question A: What are the Inf_q questions?\nQuestion B: What is the number of annotators for Rec. quality (*Rec*)?\nQuestion C : What is the annotator agreement for *Rec*?\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments, questions, and feedback on this work!\n\nRegarding human evaluation details (Questions A/B/C), these are all great points. We will modify the Appendix to include all of the Inf_q \n quiz questions (e.g., \"Acoustic guitar is the most common guitar for first-time buyers. [TRUE] or [FALSE]\") and add the number of MTurk workers who completed the survey (150).\n\nRegarding the evaluation criteria, in Section 5 we briefly provide the motivation behind our metrics of: (1) informativeness - driven by our educational objective, (2) fluency - standard for any generative system, and (3) recommendation quality - standard for any recommendation system. We will also add Future Work subsection under Discussions which will include user satisfaction as an additional evaluation criteria to enable more iterative improvements of CRS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors demonstrate that a completely LLM-based development and evaluation of chatbots for the sales domain is possible and leads to good results\n- The authors present a well planned evaluation of SalesBot and Shopper which incorporates a comparison of the the setup with human salesman\n- The paper gives important stimuli for the generation and evaluation of chatbots using LLMs in a zero-shot learning manner in other domains", "reasons_to_reject": "- The evaluation covers all important general aspects but is not described in all necessary details (the quiz questions are not presented)\n- A motivation and derivation of the evaluation criteria and metrics is missing\n- The title of the paper puts the focus on educational value, which is finally only one (side) aspect among many other topics in the paper\n- It is not evaluated if the metric measures the relevant factors to evaluate interactions in the sales domain\n", "questions_to_authors": "Question A: What are the Inf_q questions?\nQuestion B: What is the number of annotators for Rec. quality (*Rec*)?\nQuestion C : What is the annotator agreement for *Rec*?\n", "paper_topic_and_main_contributions": "The author describe the setup and analysis of an NLP engineering experiment. I general, the paper describes the usage of a LLMs to generate both interlocutors for a dialog system (here a chatbot) and to use a LLM (large language model) for the evaluation of generated dialogs. \n\nMore specific, the authors have created a chatbot (SalesBot) for the sales domain and a user simulation (Shopper) which can interact with the each other. For both, the system and the user simulation, LLMs are used in an instructive way, using prompting only. A special goal, in the shopping domain, is to educate user about (type of) products while asking for recommendations.\n\nTo evaluate the generated interactions, evaluation criteria and related metrics have been developed. The metrics can be used by humane evaluators as well as again a LLM.\n\nFor evaluation 150 dialogs have been generated and rated by a LLM - the recommendation quality was rated by the authors. Furthermore, additional 150 interactions have been generated using 15 human salesman (replacing SalesBot) and  Shopper. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The sentence starting in line 661 needs a verb.\n"}, "extra_text": {"Missing_References": "The evaluation criteria need more background and some more reasoning why the are useful to evaluate the interactions.\n\nStarting points could be the ITU-T Recommendation P.852 : Subjective quality evaluation of text-based chatbots (https://www.itu.int/rec/T-REC-P.852-202207-I) and Supplement 24 to ITU-T P series P.Sup24 : Parameters describing the interaction with spoken dialogue systems (https://www.itu.int/rec/T-REC-P.Sup24-200510-I)\n"}}]

Author rebuttal:

Meta-review: This work utilizes large language models (LLMs) to construct a SalesBot and ShopperBot to simulate both sides in product purchasing. One aim of the work is to introduce a new paradigm for conversational recommender systems (CRS) which, in addition to providing product recommendations, includes customer education.

This work makes several contributions, including:
 - Presentation of a novel idea of exploring the role of educational value in CRS
 - Demonstrates utilization of LLMs to build CRS
 - Conducts a study that compares SalesBot against professional salespeople

The result are however largely in line with expectations for LLMs, i.e, while it provides fluent chat responses, the quality of it's recommendations and informativeness are significantly below those of a human salesperson. Additionally, there is a lack of comparisons with other relevant chatbot/CRS systems and analysis of the ablation studies to better inform conversational agent design.|meta review
Decision: Accept-Findings