Conversational Recommender System and Large Language Model Are Made for Each Other in E-commerce Pre-sales Dialogue
Authors: Yuanxing Liu
Weinan Zhang
Yifan Chen
Yuchi Zhang
Haopeng Bai
Fan Feng
Hengbin Cui
Yongbin Li
Wanxiang Che
Venue: EMNLP 2023 Findings (2023)
Keywords: Conversational recommendation
large language model
collaboration method
TL;DR: Collaboration between LLM and CRS in E-commerce pre-sales dialogue

Abstract: E-commerce pre-sales dialogue aims to understand and elicit user needs and preferences for the items they are seeking so as to provide appropriate recommendations. Conversational recommender systems (CRSs) learn user representation and provide accurate recommendations based on dialogue context, but rely on external knowledge. Large language models (LLMs) generate responses that mimic pre-sales dialogues after fine-tuning, but lack domain-specific knowledge for accurate recommendations. Intuitively, the strengths of LLM and CRS in E-commerce pre-sales dialogues are complementary, yet no previous work has explored this. This paper investigates the effectiveness of combining LLM and CRS in E-commerce pre-sales dialogues, proposing two collaboration methods: CRS assisting LLM and LLM assisting CRS. We conduct extensive experiments on a real-world dataset of E-commerce pre-sales dialogues. We analyze the impact of two collaborative approaches with two CRSs and two LLMs on four tasks of E-commerce pre-sales dialogue. We find that collaborations between CRS and LLM can be very effective in some cases.

Conversational Recommender System and Large Language Model Are Made for Each Other in E-commerce Pre-sales Dialogue
Yuanxing Liu'
Wei-Nan Zhang
Yifan Chen'
Yuchi Zhang'
Haopeng Bai
Fan Feng²
Hengbin Cui Yonghin Li Wanxiang Chel
l Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China IIndepeddent China
(yxliu, wnzhang, yfchen, yczhang, hpbai, yarr wiw.cuu.ccun (fengfan. blender, alexcui. chb, liyb821 p0gmail. com
Abstract
- A00 7 1 (Which of your
thermal underwear is the warmest? Recommend one7)
E-commerce pre-sales dialogue aims to under- stand and elicit user needs and preferences for the items they are seeking so as provide appropriate recommendations. Conversational recommender systems (CRSs) learn user rep- resentation and provide accurate recommen- dations based on dialogue context, but rely on external knowledge. Large language mod- els (LLMs) generate responses that mimic pre-sales dialogues after fine-tuning, but lack domain speciffc knowledge accurate rec- ommendations. Intuitively, the strengths of LLM and CRS in E-commerce pre-sales di- alogues are complementrry, yet no previous work has explored this. This paper investigates the effectiveness combining LLM and CRS in E-commerce pre-sales dialogues, proposing two collaboration methods: CRS assisting LLM and LLM assisting CRS. We conduct exten- sive experiments on real-world dataset of E- commerce pre-sales dialogues. We analyze the impact of two collaborative approaches with two CRSs and two LLMs on four tasks of E- commerce pae-sales dialogue. We find that col- laborations between CRS and LLM can be very effective in some cases.
D: : SeERttS ;
Baak (Functional
requirement: wermest, Category: thermal underwear)
# mi- ta 222 ( (t ? (What are requirements for the thermal underwear?!
28 tkk.
(Catagory: thermal undenwear)
'd like thermal underwear for winter since feel the cold.)
SHE - 1119.
1e0 :
deFais 2 (Functional requirement:
thermal: Usage Period winter)
-R- & (First-dass Category, undenwear)
07 (Price Range, low)
3: (488 (Functioral requirement, thermal)
5:45 S: (Seeson, winter)
53.15 32 (Season, auturm))
10 99.0 (Materiel, cotton)
Stat. 90S (Style, loose)
$6 RE: (Category, thermal set)
655399641290
(Recommended item
63338864329) 3
made double fleace. You a look)
Figure 1: Example of E-commerce pre-sales dialogue.
Conversational recommender systems (CRSs) aim to learn relationships between user preferences and candidate product representations to provide accurate recommendations (Li et al., 2018) and generate responses related to recommended prod- uets (Liang et al., 2021). However, understanding user preferences from dialogues relies heavily on external knowledge (Chen et al., 2019), e.g. DBpe- dia and ConceptNet. With an external knowledge base, CRS is able to recognize the entities present in the context, but it still has difficulty understand- ing the semantic information within the context.
1 Introduction
E-commerce pre-sales dialogue refers to a dialogue between a user and a customer service staff be- fore the purchase action (Chen et al., 2020; Zhao et al., 2021). A high-quality pre-sales dialogue can greatly increase the purchase rate of user. However, there many challenges to providing high-quality pre-sales dialogue service (Liu et al., 2023b). Refine. Figure shows an example of an e-commerce pre-sales dialogue. The bot needs to interact with the user, understanding the user's needs and responding with understandable words. Additionally, it should offer appropriate recommen- dations and elicit further preferences from the user.
Large language models (LLMss), which have nu- merous parameters pre-trained on a large amount of data, possess wealth of knowledge that en- ables people to interact with them using natural language (Brown et al., 2020; Ouyang et al., 2022). With supervised fine-tuning tasks, LLMs can han- dle diverse user needs descriptions in pre-sales di- alogues. However, LLMMs lack information about candidate products, which makes them not suitable for providing domain specific recommendations.
*Corresponding author.

---

What will happen when LLMs and CRSs col- laborate? In this paper, we explore two types of collaborations between LLMs and CRSs commerce pre-sales dialogues: (i) LLM assisting CRS. (ii) CRS assisting I.M.M When a LLM assists a CRS, we append the generated response of the LLM to the input of the CRS. For the recommenda- tion task, we incorporate the representation of the product predicted by the LLM into the caleulation of the user representation. When CRS assists a LLM, we append the predictions of the CRS the input the LLM. For the recommendation task, we insert the recommendation list, while the other tasks insert the text.
2.1 Conversational recommendation
Conversational recommender systems (CRSs) aim to provide real-time recommendations based on users' dynamic preferences through natural lan- guage interactions (Gao et al., 2021; Jannach al., 2021). Early work focus on: (i) question- question-based user preference elicitation (Zou et al., 2020; Hu et al., 2022a), (ii) multi-turn conversational recommenda- tion strategies (Lei et al., 2020a, (iii) exploration- exploitation trade-offs (Fu et al., 2021; Wong et al.. 2021; Zhang et al., 2020), (iv) user preference mod- eling with external knowledge (Zhou al., 2022: Chen et al., 2019; Ma et al., 2021; Zhou et al., 2021; Ren et al.. 2022), (v) dialogue strategies (Liu et al.. 2020; Zhou et al., 2020; Hayati et al., 2020) and (vi) generating persuasive responses (Liang et al., 2021). Recently, some work (Deng et al., 2023; Wang et al., 2022a.b) utilize pre-trained language models (PLMs) as the foundation to build unified CRSs, capable of performing various tasks using a single model, instead of multiple components.
Specifically, we explore the effectiveness of col- laborations on real-world dataset of E-commerce pre-sales dialogues, namely U-NEED (Liu al., 2023b). U-NEED contains pre-sales dialogues in five top categories and supports four key tasks in E-commerce pre-sales dialogue: (i) pre-sales dialogue understanding (ii) user needs elicita- tion (iii) user needs-based recommendation and (iv) pre-sales dialogue generation. We select two popular open source LLMs, ChatGLM-6B and Chinese-Alpaca-75 well as two latest CRSs, Bart-based CRS and CPT-based CRS. We report experimental results for each combination of col- laborations on the four challenging tasks. Experi- mental results demonstrate that the collaboration between LLM and CRS is effective on three tasks: pre-sales dialogue understanding, user needs elici- tation and user needs-based recommendation. Main contributions of this paper are as follows:
The emergence of LLMs has undoubtedly im- pacted CRS-related researches. However, previous work barely explores the collaboration between conversational language models and CRSs on tasks related to conversational recommendation. We in- vestigate the collaboration between LLM and CRS in E-commerce pre-sales dialogues.
2.2 LLMs for recommendation
Large language models (LLMs), such as GPT- (Brown et al., 2020), InstructGPT (Ouyang et al., 2022), PaLM (Chowdhery et al., 2022), Bloom (Scao al., 2022), LLaMA (Touvron al., 2023) and GLM (Du et al., 2022), have gained at- tention for their natural language understanding and generation capabilities (Zhao et al., 2023). Re- cent studies have examined the performance of ChatGPT (OpenAl, 2022) in tasks such as pas- sage re-ranking (Sun al., 2023) and recommenda- tion (Wang al., 2023; Liu al., 2023a). ChatGPT has also been applied to domains like augmenting recommender systems (Gao et al., 2023). Addition- ally, Friedman et al. (2023) propose a roadmap for utilizing LLM to build a controllable and explain- able CRS for YouTube videos.
To the best our knowledge, we are the first to explore collaboration between LLM and CRS in real-world scenario, namely E- commerce pre-sales dialogue.
We propose methods for two types of collab- orations between LLMs and CRSs, i.e., CRS assisting L.M and I.M assisting CRS.
Extensive experimental results on real-world E-commerce pre-sales dialogue dataset indi- cate the effectiveness and potential of collabo- rations between LLMs and CRSs.
Different from previous work using LLM to en- hance CRS, we systematically investigate the ef- fectiveness of combining L.LM and CRS, i.e., LLM assisting CRS and CRS assisting LLM, which pro- vides insights for future research on CRSs.
2 Related Work
We review the related work along two lines: (i) con- versational recommendation and (ii) large language models (LLMss) for recommendation.

---

CRS
Train
Trained CRS
CRS
Train
Trained CRS collaborated with LLM
CRS
Train
Trained CRS
Prediction
Pre-sales dialogues
Pre-sales dialogues
Prompt and vector
Pre-sales dialogues
t Prediction
Instruction and input
LLM
Fine-tune
Fine tieettuned LLM
LLM
Fine-tune
Fine-tuned LLM
LLM
Fine-tune
Fine-tuned LLM collaborated with CRS
No collaboration
LLM assisting CRS
CRS assisting LLM
Figure 2: A comparison the three types of collaboration between a CRS and LLM. We explore the collaboration between LLM and CRS, i.e., LLM assisting CRS and CRS assisting LLM, and we compare the three in detail in $5.
3 Method
prises several sentences that introduce the task's s objective. The "input" contains the necessary in- formation for completing the task. For instance, in the case of user needs-based recommenda- tion task, the "input" encompasses the user's needs, candidate products, and related product knowledge. The "output" remains consistent with the original task. Figure 3 shows an example of the instruction data corresponding to the user needs elicitation task. Additional examples of various tasks can be found in Appendix F. Note that the original user needs-based recommendation task involves numer- candidates, with each product possessing exten- sive attribute knowledge, the "input" surpasses the maximum input length permitted by LLMs. Con- sequently, in practice, we limit the number can- didates to 20.
3.1 Overview
this paper, we explore the collaboration of conversational recommender system (CRS) and a large language model (LLM). Figure 2 provides an illustration of our collaboration framework.
LLM assisting CRS. We leverage the prediction results of LLM support a CRS. Initially, we fine-tune a LLM using pre-sales dialogues, follow- ing the method described in Section 3.2. Subse- quently, we incorporate the prediction results of the fine-tuned large model into the training process of the CRS, via prompts and vectors. For detailed description, refer to Section 3.5.
CRS assisting LLM. We utilize the prediction re- sults of CRS to assist a LLM. Initially, we train a CRS using pre-sales dialogues, following the ap- proach outlined in Section 3.3. Subsequently, we integrate the prediction results of the trained CRS into the instructions and inputs optimize the pro- cess of fine-tuning the LLM. For further details, see Section 3.4.
Base LLMs and fine-tuning. We select ChatGLM and Chinese- Mainese-Alpaca-71 as base LLMs due to their openness and commendable performance in Chi- nese basic semantic understanding. ChatGLM is an open bilingual language model built upon General Language Model (GLM) framework (Zeng et al., 2023), with 6.2 billion parameters. LLaMA (Tou- vron et al., 2023) is decoder-only. foundational large language model based on the transformer chitecture (Vaswani al., 2017). The Chinese LLaMA model an extension of the original LLaMA model, incorporating an expanded Chinese vocabulary and undergoing secondary pre-training using Chinese data (Cui et al., 2023). We adopt the Chinese Alpaca model, which builds upon the aforementioned Chinese LLaMA model by incor- porating instruction data for fine-tuning. 2 To carry out the fine-tuning process, follow the official method provided by ChaGLLM6B/Chiness-AAppe Plus-7B 7B, using LoRA (Hu et al., 2022b).
In this paper, we explore the effectiveness of col- laboration by analyzing the impact of collaboration between CRS and LLM on the performance of four tasks in E-commerce pae-sales dialogue (Liu et al., 2023b). These tasks include: (i) pac-sales dialogue understanding (ii) user needs elicitation (iii) user bases-based recommendation and (iv) pre-sales di- alogue generation. Due to space constraints, we provide detailed definitions of these tasks in Ap- pendix
3.2 LLMs for E-commerce pre-sales dialogue We introduce the method of fine-tuning a large language model (LLM) using pre-sales dialogues. Instruction data. Fach sample within the train- ing, validation, and test sets consists of "instruc- tion", "input" and "output". The "instruction" com-
htttps://gi thub. COm/THUDM/CCatGLM hhttps://gi thub. com/ymcui/ Chinese-LLaMA- Alpaca

---

3.3 CRSs for E-commerce pre-sales dialogue We introduce the method of train a conversa- tional recommender system (CRS) for pre-sales di- alogues. We adopt UnIMIND (Deng ct al., 2023) as our base CRS, as it focus on multiple tasks in con- versational recommendation. The recommendation candidates for UniMIND movies. The movie ti- tle can be generated based on the prompt. However, in E-commerce pre-sales dialogue, the recommen- dation candidate the product ID, which is diffi- cult to be decoded directly. Therefore, for the user needs-based recommendation task we follow a tra- ditional user representation-bane approach (Kang and McAuley, 2018).
Loss functions. Following UniMIND (Deng al., 2023), we design a unified CRS with prompts learn- ing and multitask learning for pre-sales dialogues.
Ca C (XYY,--D log Poly v.)...2)
(1)
where D = (D. DA, Dc} denote the train-set for three tasks: pre-sales dialogue understanding, user needs elicitation and pre-sales dialogue generation. And I. is the length of the generated sequence. For user needs-based recommendation task, the recom- mendation probability and loss function are defined as follows:
Prompts. Following Deng et al. (2023), we de- fine the inputs and outputs of the four tasks us- ing a unified sequence-tor certice-to-eqqeenee sequence paradigm. We use five special tokens to indicate information seg- ments: (i) [user] indicates the utterance of the user. (ii) [system] indicates the response from customer service staff. (iii) [understand] indicates the needs contained in the user utterance, i.e., attributes and attribute values. (iv) [elicit] indicates the attributes that the customer service staff plans to ask about (v) [recommend// indicates the items that have been recommended by customer service staff. For instance, the original input X can be represented follows:
ri CLS(ei, Enc(XR))
(2)
[8] e logri,
LR =
(3)
where is the trainable item embedding, E is collection of candidate products and r, is recom- mendation probability. CLS(-) classifier, we apply linear layer practice. And Enc(-) the en- coder, we adopt two versions: BART (Lewis et al.. 2020) and CPT (Shao et al., 2021).
user preferences.
Training process. We train a CRS in two stages. In the first stage we train a CRS only on the user needs-based recommendation task, i.e., L = LR- Since BART and CPT do not have pre-sales conver- sation knowledge, this step aims to warm up CRS. In the second stage we continue to train a CRS on all four tasks, i.e., L LR + Co.
Xy [user] un [understand] d1 [system] 81 [understand]
d2 [user] M4
Xg [user] in [understand] dj [system] 81 [understand]
d2 ... [system] 83
3,4 Collaboration 1: CRS assisting LLM
XA = [user] 41 [understand] dj [elicit] 01 [system] 81
We introduce the method of CRS assisting LLM. Initially, we train CRS model following the ap- proach outlined Section 3.3. Subsequently, we enrich the original instructions and inputs of a LLM by incorporating the prediction outcomes from the CRS model. Finally, we fine-tune the LLM, explained in $ 3.3. utilizing the augmented instruc- tions and inputs.
[understand] d2 ... [user] U;
XR [user] U1 [understand] dj [recommend] E1
[system] 81 [understand] d2 ... [user] U4
XG [user] U1 [understand] dj [system] $1 [understand]
dg [user] u; [elicit] 01 [recommend] 61
where is the i-th utterance of the user, is the i-th response of customer service staff, d; is the i-th user needs, a; is the i-th attribute to be asked, and e,is the i-th recommended product. We adopt natural language prompt (Raffel et al., 2020) to indicate each task:
Enhanced instruction and input. We convert the output of a trained CRS into text and incorporate it into the input of LLM. We add additional instruc- tion, i.c., completing the task requires considering the results of the CRS. An example is shown in the upper right corner of Figure 3. Note that in the user needs-based recommendation task, CRS can output a list of recommendations along with cor- responding recommendation scores. We rank the
Zu "Identify attributes and v
Z "Select attribute to ask:"
Zg "Generate response:"

---

User needs elicitation task
No collaboration
CRS assisting LLM
Input
Instruction and imput for LLMs
Instructood B6 SE
(NRIS 2016 REIEINE SI A
User utteerence
Instructions Fol stfy ReReetare SIS S.TTE aattt #* & 1E 13 1J A-S & M E (Based on pre sales dialogue in the fastion domain. select series of attribuces to guie the user in providing additinal information values not)
9e S Cattttt--ctttteeeee 7 T. (nlxch your ihemral underwoor & the
(Based pre-saess daloque the fashion domain and the attributts prodiction resuts of CRS select series of attributes to guiee the user in providing additional informatio about hi preferences. The results may incldde attribute values not)
User needs
praterences.
results may
Jrunsioonll requaemmntt
: #6dt
Input Mellet5: [Round 0lv/R : HN TA 15 7 HeTF FFy (Pre sales didague (Round Ofrisser Which of your thermal underwesr is the warmest7 Recommend one? (n)
InputiBliR |Round de22
MARGE Rink an Aed (Pro sales dialogue [Round DiviUsor Which of thermal undenwezr the werment Recommend one?yn The attributes predicted by CRS categorst
Category: thermaal underwsar? )
Output
BBX (Categoory)
Prompt for CRSs
LLM assisting CRS
annotated the response.
[user] (8-11 (A t 18 15 # undersandiSittteee Sorte 1otalii lile tik, (ittaa letiite:
[uset] 1H43 i4 t A888 ta 7 1879 (understandjs M. : -ia0ffi Ri4k & S St P9 underwear
RUS 6:13. 8. 22 (Whor and your rEQurTmens the
(luser/Which your thermal underwear
22 # (JuserWhich
warmest! Recommend
thermal
funderstandjfuncti requiremntt warmest Category
Furdersondd]funccioes
wornest Cetegooy:
underwear. The
thermal attributes)
(LAMjmickness The query
query attrbbuee))
Figure 3: An example of collaboration between CRS and LLM on the user needs elicitation task. Left side shows the input and output the task. The middle displays data used to fine-tune a LLM and train a CRS independently. The right side shows two cases of combining the two. Collaboration content is highlighted in red italics.
candidates according to their scores and include the ranking to the input of LLM. For other tasks, we only consider the final output of the trained CRS.
recommended by the fine-tuned LLM:
r CLS(ei, Enc(EEeeeece (Xn), éx)
(4)
[8]
Cn - - log r.
3.5 Collaboration 2: LLM assisting CRS
(5)
We introduce the method of LLM assisting CRS. Initially, we fine-tune LLM model using the tech- nique described Section 3.2. Subsequently, we employ the prediction outcomes from LLM to en- hance the prompt and user representation in the CRS model. Finally, we train a CRS model as outlined in $ 3.3.
where é; is the embedding of the product recom- mended by the fine-tuned LLM.
Experimental Settings
4.1 Research questions
To guide the remaining part of this paper, we set up two research questions:
Enhanced prompts. We use the prediction results of LLM to enhance the prompts of CRS.
Are LLM and CRS complementary? Does the combination of LLM and CRS improve performance?
X [user] [understand] dj [system] $1 [understand]
d2 [user] uy [LLM] d,, 6
X = [user] [understand] dj [system] [understand]
How does the combination of CRS and LLM perform on different tasks and different cat- egories? What are the differences between different collaboration methods?
d2 [system] 8; [LLM) d;, 23
X = [user] [understand] d, [elicit] [system] 81
[understand] d2 [user] u; [LLM] d;
In the Results and Analysis section, we systemati- cally examine the outcomes of each task to answer the aforementioned two research questions.
X [user] 8n [understand] dj [system] 81 [understand]
d2 ... [user] un [elicit] d1 [recommend] é1 [LLM] &;
For X and X's. di and ú, are attributes and values involved in utterance identified by LLM for the i-th turn. For X á; is the attribute user needs elici- tation for the i-th turn. For X & is the response for the pre-sales dialogue generation task generated by LLM.
4.2 Dataset
We conduct experiments on U-NEED (Liu et al., 2023b). U-NEED consists of 7,698 fine-grained an- notated pre-sales dialogues. which consist of 1662, 1513, 1135, 1748, and 1640 dialogues in Beauty, Phones, Fashion, Shoes and Electronics categories respectively. We follow the partition of the training set, validation set and test set proposed in U-NEED.
Enhanced representation. For recommendation task, we consider the embedding of the product

---

Table 1: Performance of baseline methods on pre-sales dialogue understanding task in 3 typical categories: Beauty,
Fashion and Shoes. Baseline results marked with are taken from U-NEED (Liu et al., 2023b). CLLM is short for ChatGLM and ALLM is short for Chinese-Alpacee BCRS is short for UniMIND(BART) and CCRS is short for UniMIND(CPT). The best results are highlighted bold.
Beauty
Shoes
Phones
All 5 categorics
Methods
R
F1
P
R
F1
P
R
P
R
Bert*
0.5355
0.6284
0.5782
0,5851
0.7020
0.6382
0.4212
0.5384
0.4726
0,4549
0.5652
0.5041
BeR+CREE
0.6731
0.6802
0.6766
0.7302
0.7703
0.7497
0,5620
0.5923
0.5768
0.6688
0.6530
0.6608
BOTH+BILSTM=CRF**
0.7282
0,7481
0.7380
0.7870
0.8101
0.7984
0.6701
0.6990
0.6843
0.6892
0.6875
0.6884
No collaboration
UniMIND(BART)
0.6443
0.6000
0.6085
0.7711
0.7417
0,7483
0.7522
0.7406
0.7407
0.7188
0.6933
0.6978
UniMIND(CPT)
0.5994
0,5420
0.5565
0.7468
0.6836
0,6889
0.7110
0.6907
0.6959
0.6807
0.6451
0.6539
ChatGLM
0.7858
0.7797
0,7777
0,8265
0.8307
0.8248
0.7805
0.7792
0,7760
0.7968
0,7936
0.7910
Chinesc-Alpaca
0.7409
0.7310
0.7316
0,8032
0.7868
0,7899
0.7363
0,7178
0.7238
0.7568
0.7378
0.7425
LLM assisting CRS
CLLM-BCRS
0.6502
0,5848
0.6004
0.7725
0.7171
0.7318
0.7504
0.7152
0.7246
0.7173
0.6665
0.6796
CLLM-CCRS
0.6101
0.5346
0.5545
0.7663
0.7218
0.7338
0.7183
0.6976
0.7018
0.7023
0.6539
0.6666
ALLM-BCRS
0.6255
0,5688
0.5835
0.7746
0.7120
0.7302
0.7311
0.7108
0.7159
0.7088
0.6653
0.6765
ALL.M-CCRS
0.5307
0.5410
0.7048
0.6658
0.6833
0.6635
0.6691
0.6629
0.6277
0.6369
CRS asvisting ILM
BCRS-CLLM
0.7900
0.7879
0.7824
0.8521
0.8511
0.8473
0.8222
0.8220
0.8179
0,8105
0.8065
0,8033
CCRS-CLLM
0,7940
0.7926
0.7878
0.8372
0.8378
0.8326
0.7911
0.7866
0.7847
0.7927
0,7897
0.7864
BCRS-ALLM
0.7772
0.7462
0.7542
0,8062
0.7690
0.7770
0.7662
0,7160
0.7311
0.7700
0,7310
0.7414
CCRS-ALLM
0,7600
0.7272
0,7354
0,8036
0,7621
0.7738
0.7392
0.6978
0.7086
0.7569
0,7168
0.7276
4.3 Baseline methods
adopt automatic and human evaluation for pre-sales dialogue generation task. For automatic evaluation, we use Distinct-n. And for human evaluation, we measure the informativeness and relevance of gen- erated response. For the limited space, we put the description of each metric in the Appendix D.
For each task, baseline methods consist of typical methods, CRS methods and LLM methods.
Typical methods. We select typical meth- ods for the four tasks following (Liu et al.. 2023b). Specifically,
select Bert, Bert+CRF, Bert+BiLSTM+CRR baselines for pre-sales di- alogue. For user needs elicitation task, we se- lect DiaMultiClass and DiaSeq as baselines. For user needs-based recommendation task, we choose Bert, SASRec, TG-CRS. And we select GPT-2 and KBRD as baseline methods for pre-sales dia- logue generation task. For the limited space, we put the description of cach typical methods in the Appendix B. We select UniMIND(BART) (Deng al.. 2023) and (niM as CRS meth- ods. For LLLM methods, we select ChatGLM (Zeng et al., 2023) and Chinese-Alpaca (Cui et al., 2023). For combination of LLM and CRS, we define eight variants. We put the description of each variant in the Appendix C.
Results and Analysis
We conduct extensive experiments to explore the performance of collaborations of LLMs and CRSs on four tasks. We analyze impacts of collaborations on each task in turn.
5.1 Impacts of collaborations on pre-sales dialogue understanding
Based on Table I we have the following observa- tions: (i) LLMs perform well in understanding user needs. ChatGLM substantially outperforms classical baseline methods and CRSs on all met- rics in all categories. The second best method is Chinese-Alpaca, which outperforms the strongest baseline method, Bert--RLLTTM-RRR on most metrics. We attribute this to the strong capabil- ity of the LLMS for dialogue understanding. (ii) In collaborations with CRSs, LLMs perform even better in understanding user needs. BCRS-CLLM outperforms ChatGLM in all category and all met- rics, especially in Shoes and Phones. Similarly we observe that -ELM-ALLM outperforms Chine- se-Alpaca in some metrics. By carefully comparing
4.4 Evaluation metrics
We adopt the evaluation metrics used in U- NEED (Liu et al., 2023b). Specifically, we select precision, recall and fl score as evaluation metrics for pre-sales dialogue understanding and user needs clicitation. For user needs-based recommendation task, we choose Hit@k K and MRR @K K. And we

---

Table 2: Performance of baseline methods on user needs elicitation task in typical categories: Beauty, Fashion and Shoes. Baseline results marked with * taken from U-NEED (Liu et al., 2023b). CLLM is short for ChatGLM and ALLM is short for Chinese-Alppe Alpaca. BCRS is short for UniMIND(BART) and CCRS is short for UniMIND(CPT), The best results are highlighted in bold.
Beauty
Shoes
Phones
All 5 categories
P
F1
R
P
F1
R
DiaMultiClass*
0.4037
0.7228
0.5054
0.3361
0,4131
0.3423
0.4534
0.5212
0.4585
0.3222
0.4966
O. 0,3662
DiaSeq
0.4761
0.4272
0.4424
0.3992
0.3305
0.3498
0.4414
0.3789
0.3966
0.3555
0.2996
0.3153
No collaboration
UNiMIND(BART)
0.4979
0,4305
0.4518
0.4367
0.3827
0.3927
0.5513
0,4973
0.5061
0.4301
0.3881
0.3943
UniMIND(CPT)
0.4022
0.3575
0.3657
0.4388
0.3774
0.3906
0.4946
0.4432
0.4515
0.4021
0.3575
0.3657
ChaiGLM
0.4348
0.4057
0.4055
0.4054
0.3496
0.3621
0.4712
0.4541
0.4441
0.3683
0,3422
0.3380
Chinese-Alpaca
0.2723
0.2362
0.2475
0.4020
0.3440
0.3588
0.4946
0.4613
0.4604
0.3513
0.3132
0.3191
LLM assisting CRS
CLLM-BCRS
0.5106
0.4413
0.4617
0.4510
0.3837
0.4016
0.5703
0.4982
0,5168
0.4518
0.3963
0.4095
CLLM-CCRS
0.4128
0.3405
0.3583
0.4531
0.3735
0.3955
0.5243
0.4541
0.4737
0.4056
0,3405
0,3583
ALLM-BCRS
0.4702
0.4149
0.4289
0,4490
0.3827
0,3969
0,5297
0,4865
0.4863
0,4258
0.3866
0,3903
ALLM-CCRS
0.4043
0.3362
0.3574
0.4490
0.3661
0.3884
0.5505
0.4928
0.5009
0,4138
0,3519
0.3671
CRS assisting L.M
BCRS-CLLM
0.4908
0.4319
0.4450
0.3918
0.3280
0.3429
0.4923
0.4788
0.4645
0.4190
0.3723
0.3796
CCRS-CLLM
0,4092
0.3766
0.3783
0.4082
0.3439
0.3590
0.4495
0.4356
0.4211
0.4020
0.3710
0,3704
BCRS-ALLM
0.2660
0,2383
0.2428
0.2490
0.1914
0.2067
0.2568
0,2432
0.2378
0,2350
0.1904
0.1994
CCRS-ALIM
0.1681
0.1489
0.1539
0.2388
0.1826
0.1951
0.2027
0.1874
0.1823
0,2071
0.1632
0.1730
the results of the four combinations of "CRS assist- ing LLM" with the results of the four methods of "No collaboration", we find that a better perform- ing CRS improves the performance, while a worse performing CRS degrades the performance. Since LLMS are usually very sensitive to inputs, we think that the former may provide useful reference infor- mation to complement what the LLMs do not take into account. The latter, on the other hand, may bring in noises that disturb the judgments of the LLMs. (iii) The improvement in understanding user needs brought by the collaboration of CRS to LLMs varies across categories. In the Shoes and Phones categories, BCRS-CLLM significantly outperforms ChatGLM with the collaborations of CRSs. While in the Beauty category, has only a minor improvement compared to Chat- GLM. We think this may be due to the fact that user needs in the Beauty category are usually focused on specific attributes such as "skin type".
rics in Beauty, Shoes, and Mobile categories. This indicates that in E-commerce pre-sales dialogue, the performance of methods that make decisions with the help of generative models, e.g. BART and LLMs, is somewhat limited. DialMultiClass doesn't 't require large number of parameters and doesn't 't need to be trained for a long period of time. Compared to making decisions with LLMs, DialMultiClass still has considerable strengths in real-world production environments. (ii) Collabo- rations between ChatGLM and UniMIND (BART) can improve their respective performance in user needs elicitation. Specifically, CLLM-BCRS out- performs UniMIND((ART) on all metrics for all categories. Moreover, CLLM-BCRS beats all methods on six metrics. Similarly, BCRS- CLLM outperforms ChatGLM in all categories ex- cept Shoes. Specifically, BCRS-CILLN achieves a I 12.3% improvement over ChatGLM on the av- erage FI score across all 5 categories. Based on this, we see that the output of ChatGLM are beneficial for UniMIND(BART) and vice versa. (iii) Chinese-Alpaca and ChatGLM exhibit ma- jor differences in their performance in collabo- rations with CRSs. Specifically, CCRS-ALLM and BCRS-ALLM achieve the worst and second- worst performance on almost all metrics in all cat- egories. This implics that the predicted results of CRSs cause large disruption to Chinese-Al- paca. The two combinations of CRSs assisting
5.2 Impacts of collaborations on user needs elicitation
Based on Table 2 we have the following observa- tions: (i) LLMs do not exhibit superb performance on user needs elicitation. On the average results of the 5 categories, UniMIND(BART) achieves the best performance, followed by the classical method DialMultiClass and UnPMIND(CPT). Morcover, DiaMultiClass beats all the methods on Recall met-

---

Table Performance of baseline methods on user needs-based recommendation task in 3 typical categories: Beauty, Fashion and Shoes. H@ K and M@ K refer Hit@K and MRR @ K. Acc. refers to accuracy. CLLM short for
ChatGLM and ALLM is short for Chinese-Alpar UniMIND(CPT), Since LLM methods only recommend 1 product, H@5 and M@5 cannot calculated. The best results are highlighted in bold.
BCRS is short for UniMIND(BART) and CCRS is short for
Beauty
Shoes
Phones
All 5 categories
Acc.
H@5
M@5
Acc.
H@5
M@5
Acc.
H@5
M@5
Acc.
H@5
M@5
Bert
0.0123
0.0185
0.0141
0.0046
0.0138
0.0072
0.0326
0.0688
0.0463
0.0118
0,0215
0.0151
SASRec
0.1108
0.2831
0.1711
0.0399
0.1121
0.0668
0.0761
0.2681
0.1475
0.0976
0.2532
0.1556
TG-CRRS
0.1323
0.3354
0.2034
0.1275
0.2396
0.1692
0.2564
0.3347
0.1744
0.3074
0.2244
No collaboration
UnMMIND(BART)
0.2154
0.6246
0.3654
0.2458
0.5315
0.3483
0.3478
0.6449
0.4608
0.2398
0.5440
0,3510
UniMIND(CPT)
0.2554
0.6246
0.3915
0.2826
0.5438
0.3779
0.3043
0.6594
0.4460
0,26.79
0.5617
0.3737
ChaGLM
0,2123
0.3810
0.0580
0.2226
Chinesc-Alpace
0.1415
0.3917
0.0036
0.2157
assisting
CLLM-BCRS
0.2462
0.6062
0.3741
0.2565
0.5376
0.3594
0.3804
0.7536
0.5204
0.2623
0.5617
0.3694
CLLM-CCRS
0.2585
0.6308
0.3913
0.2657
0.5438
0.3690
0.4058
0.7138
0.5319
0.2768
0.5665
0.3835
ALLM-BCRS
0.2554
0.6092
0.3822
0,2550
0.5223
0.3554
0.3587
0.7681
0.5196
0.2623
0.5606
0.3721
ALLM-CCRS
0.2430
0.6369
0.3747
0.2750
0.5515
0.3788
0.4312
0.7609
0.5959
0,2822
0.5832
0.3919
CRS assisting LLM
BCRS-CLLM
0.1600
0.2442
0.3478
0.2264
CCRS-CLLM
0.1785
0.2642
0.3043
0.2479
BCRS-ALLM
0,2000
0.2458
0.3297
0.2307
CCRS-ALLM
0.2369
0.2688
0.2754
0.2532
ChatGLM, i.e., BCRS-CLLM and CCRS-CLLM, however, perform well. We think that the differ- ences between ChatGLM and Chincse-Alpaca in collaborating with CRSs come from their base mod- els and fine-tuning data.
respectively, when compared UniMIND(BART), Note that LLMs provide recommendations in a different way than CRSs do. LLMs provide rec- ommendations relying on given inputs, i.e., a rec- ommended product is semantically related to user needs in some way. CRSs, on the other hand, model representation of both and learn the im- plicit relationship between the two compute the probability of product being recommended. The above improvements are only that CRSs consider the representations of the recommended products given by the LLMs. We believe that collabora- tions between LLMs and CRSs on recommenda- tion tasks far beyond this and are a direction worth exploring. (iii) With collaborations with CRSs, LLMs can achieve comparable recommen- dation performance. Specifically, in the Phones category, ChatGLM and Seinese-Alpace have very poor recommendation performance. In contrast, the four methods of "CRS assisting LLM" achieve the performance close to that of CRSs. Based on this, we think that when the recommendation per- formance of LLMs is very poor in certain do- main, collaborative approach could make LLMs to achieve performance close to that of CRSs.
5.3 Impacts of collaborations on user needs-based recommendation
Based on Table have the following observa- tions: (i) LLMs show the potential for user need- s-based recommendation. Specifically, LLMs, i.c., Chinese-Alpar Alpaca and ChatGLM, achieve the best and second best performance significantly outper- forming all the methods on the Accuracy in Shoes category, respectively. They also achieve perfor- mance over classical methods the results of all 5 categories. Based on this, we think that LLMs can somewhat provide suitable recommendations when the candidate range is small (the number of candidate products in Table 3 is 20). (ii) With the collaboration of LLMs, the recommendation performance of CRSs can be improved. Specifi- cally, on the average results across all 5 categories, ALLM-CCRS achieves 6.9%, 3.8%, and 4.9% im- provements Accuracy, Hit@ 5, and MRR @5 respectively, compared to UniMIND (CPT). Sim- ilarly, on average results across all 5 categories, ALLM-BCRS achieves 9.4%, 3.0%, and 6.0% im- provements on Accuracy, Hit@ 5, and MRR@5.

---

Table 4: Performance of bascline methods on pre-sales dialogue generation task in 3 typical categories: Beauty, Fashion and Shoes. CLLM is short for ChatGLM and ALLM is short for Chinese-Alpaca. BCRS is short for UniMIND(BART) and CCRS is short for UniMIND(CPT). Info. and Rel. refer to informativeness and relevance. The best results are highlighted in bold.
Beauty
Shoes
Phones
All categorics
Dist-1
Rel.
Info.
Dist-1
Rel.
Info.
Dist-1
Rel.
Info.
Dist-1
Rel.
Info.
GPT-2
0.6195
1.8800
1,8400
0.6504
2.1933
2.1467
0.6305
1.9867
1.8067
0.6335
2,0200
1,9311
KBRD
0,5753
2,9933
2,5067
0.5639
3.3467
2. 2,8800
0.6034
3,1000
2.7533
0.5809
3.1467
2,7133
No collaboration
UNiMIND(BART))
0.8611
3.7933
3.3600
0.8897
3.9133
3.6933
0.8299
3.6667
3.6200
0.8521
3.7911
3.5578
UniMIND(CPT)
0,8578
3.7333
3.2600
0,9080
3.9333
3.7800
0.8472
3,8000
3.7400
0.8583
3.8222
3,5933
ChaiGLM
0.8169
3.7467
3,4533
0.8594
3,7000
3,4400
0.8411
3,6200
3.4533
0.8152
3.6889
3,4489
Chinese-Alpaca
0.9131
3.6667
3.2467
0.9124
3.8733
3.6667
0.9109
3.6733
3.6133
0.8927
3.7378
3,5089
LLM assisting CRS
CLLM-BCRS
0.8686
3.8467
3.4933
0.8986
3.9133
3.7600
0.8460
3.7267
3,6000
0.8611
3.8289
3.6178
CILM-CCRS
0.8633
3.7667
3.5000
0.8975
3.9267
3.7200
0.8568
3.6267
3.5267
0.8587
3.7734
3.5822
ALLM-BCRS
0.8700
3.7667
3,4000
0.8959
3,9200
3.7867
0.8505
3.7333
3.6533
0,8650
3.8067
3.6133
ALLM-CCRS
0.8792
3.8467
3.4467
0.9086
3,9933
3.7867
0.8640
3,6400
3,5933
0.8699
3,8267
3.6089
CRS assisting LLM
BCRS-CLLM
0.8371
3 3.6867
3.3867
3,9400
3.7667
0.8544
3.5800
3.7667
3,5778
CCRS-CLLM
0.8324
3,5000
3.0867
0.8714
3.9200
3,7800
0.8355
3,5933
3.4533
0.8256
3.6711
3,4400
BCRS-ALLM
0.8910
3.6933
3.2200
0.9011
3.7867
3.6133
0.9040
3.6067
3,5400
0,8804
3.6956
3,4578
CORS-ALLM
0.8898
3.8600
3.3400
0.8951
3.9333
3.6733
0.8988
3.7400
3.5667
0.8826
3.8444
3.5267
5.4 Impacts of collaborations on pre-sales dialogue generation
recommended items in the decoding process. In- tuitively, collaborations between such CRSs and LLMs may work out well.
Based on Table 4 we have the following obser- vations: (i) CRSs and LLMs show comparable performance in pre-sales dialogue generation. In Shoes, Beauty, and Phones categories, Chinese-Al paca achieves the best performance on Dist-1. This indicates that Chinese-Alpaa can generate more diverse responses. While in most cases, the re- sponses generated by CRSs are more relevant and informative than those generated by LLMs. In ad- dition, neither the LLMs nor the CRSs generate responses that beat the ground truth responses pro- vided by customer service staff. (ii) Collaborations between L.MSs and CRSs show marginal effects on pre-sales dialogue generation. Specifically, the methods of collaborations between LLMs and CRSs, i.e., "LLM assisting CRS" and "CRS assist- ing LLM", achieve the best performance on most the metrics. However, the improvement from collaboration is marginal compared to CRSs or LLMs. We believe this may be due to the fact that LLMs and UniMIND are relatively close in their approaches to generating responses, i.e., both are based on pre-trained language models and prompts. Therefore, collaborations between two similar ap- proaches does not have much impact. In future work, we plan to consider CRSs that focus on gen- crating persuasive reasons for recommendations, e.g., NTRDs that introduce words related to the
6 Conclusions and Future Work
In this paper, we investigated the integration of con- versational recommender systems (CRS) and large language models (LLM) in E-commerce pre-sales dialogues. Specifically, we proposed two collabo- ration strategies: "CRS assisting LLM" and "LLM assisting CRS". We evaluate the effectiveness of these collaborations between two LLMs and two CRSs on four tasks related to E-commerce pre- sales dialogues. Through extensive experiments and careful analysis, we found that the collabora- tion between CRS and L.M can be highly effective in certain scenarios, providing valuable insights for both research and practical applications in E- commerce pre-sales dialogues. Additionally, our findings can inspire exploration of collaborations between general large models and private small models in other domains and areas.
For future work, we plan to examine the collab- oration between LLMs and CRSs across various categories. For instance, CRS within the Shoes category could provide information to LLM in the Fashion category, resulting in recommended combination such as dress, pants, and shoes.

---

Limitations
Qibin Chen, Junyang Lin, Yichang Zhang. Ming Ding. Yukuo Cen, Hongxia Yang, and Jie Tang. 2019. To- wards howweedeebbssed recommender dialog system. In Proc. of EMNLP.
One limitation of our work is that our findings may only apply to large language models (LLMs) around 7B parameters. In this work, we select two LLMs that are widely used, namely ChatGLM-61 and Chinese-Alpaca-7B. Related work reveals that there is some variation in the capability of L.MSs with different parameter sizes (Wei et al., 2022). For LLMs with more parameters, such as 100B, more GPU resources and time are needed to ex- plore the effects of combining CRSs and LLMs. In addition, LLMs are constantly being updated. Re- cently ChatGL.M2-6B and Chinese-Alpaca2-113 have been open sourced." They show better perfor- mance than ChatGLM-6B and Aeiinse-Alpaca-11 multiple benchmarks, and may have higher re- sults on E-commerce pre-sales dialogues as well. However, we believe that the combination of LLMs and CRSs is still worth researching.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways.
Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca.
Zhenjin Dai, Xutao Wang, Pin Ni, Yuming Li, Gangmin Li, and Xuming Bai. 2019. Named Entity Recogni- tion Using BERT BiLSTM CRF for Chinese Elec- tronic Health Records. Proc. 2019 12th Int. Congr. Image Signal Process. Biomed. Eng. Infor- matics, CISP-BMEI 2019, pages 0-4.
Yang Deng, Wenxuan Zhang, Weiwen Xu, Wenqiang Lei. Tat-Seng Chua, and Wai Lam. 2023. unified multi-task learning framework for multi-goal conver- sational recommender systems. ACM Transactions Information Systems, 41(3):1-25.
Ethics Statement
In this paper, we explore the effectiveness of com- bining LLM and CRS e-commerce pre-sales conversations. We are committed to using and mod- ify U-NEED dataset only for research purposes and not for commercial exploitation. Our proposed ap- proach does not generate harmful content or raise ethical issues.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. NAACL HLT 2019 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. Proc. Conf. 1:4171-4186,
Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregres- sive blank infilling. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320-335
Acknowledgments
We thank the anonymous reviewers for their help- ful comments. This work supported by the Na- tional Key Research and Development Program (No. 2022YFF0902100) and National Natural Sci- ence Foundation of China (No. 62076081 and No. 61936010).
Luke Friedman, Sameer Ahuja, David Allen, Terry Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, al. 2023. Lever- aging large language models in conversational rec- ommender systems.
Zuohui Fu, Yikun Xian, Yaxin Zhu, Shuyuan Xu, Ze-
long Li, Gerard de Melo, and Yongfeng Zhang. 2021. Hoops: Human-in-the-loon graph reasoning for con- versational recommendation. In Proc. of SIGIR.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Chongming Gao, Wengiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and challenges in conversational recommender systems: A survey, Al Open, 2:100-126.
Yunfan Gao, Tao Sheng, Youlin Xiang. Yun Xiong, Haofen Wang, and Jiawei Zhang. 2023. Chat-rec: Towards interactive and explainable llms-augmented recommender system.
Meng Chen, Ruixue Liu, Lei Shen, Shaozu Yuan, Jingyan Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020. The JDDC corpus: A large-scale multi-turn Chinese dialogue dataset for E-commerce customer service. In Proc. of LREC.
Shirley Anugrah Hayati, Dongyeap Kang, Qingxi- aoyang Zhu, Weiyan Shi, and Zhou Yu. 2020. IN- SPIRED: Toward sociable recommendation dialog systems. In Proc. of EMNLP.
Phttps://gittee httpa//.ccmmmmiD////aaaaaaaga https:// //github....... www.://wcul/hhhieeesLLLaa----apa

---

Chenhao Hu, Shuhua Huang, Yansen Zhang, and Yubao Liu. 2022a. Learning to infer user implicit preference in conversational recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Reirieval, SIGIR '22, page 256-266,, New York, NY, USA. As- sociation for Computing Machinery.
Yuanxing Liu, Weinan Zhang. Baohua Dong, Yan Fan, Hang Wang, Fan Feng. Yifan Chen, Ziyu Zhuang. Hengbin Cui, Yongbin Li, and Wanxiang Che. 2023b, U-need: fine- line-grained dataset for user needs-centric e-commerce conversational recommendation. In Pro- ceedings of the 46th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, SIGIR '23, page 2723-2732 New York, NY, USA. Association for Computing Machinery.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022b. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Zeming Liu, Haifeng Wang, Zheng-Yu Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020. Towards conver- sational recommendation over multi-type dialogs. In Proc. of ACI.
Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021. A Survey on Conversational Recom- mender Systems. ACM Comput. Surv., 54(5).
Wenchang Ma, Ryuichi Takanobu, and Minlie Huang. 2021. CR-walker: Tree-structured graph reasoning and dialog acts for conversational recommendation. In Proc. of EMNLP.
Wang Cheng Kang and Julian McAuley. 2018. Self- Attentive Sequential Recommendation. In Proc. IEEE Int. Conf. Data Mining, ICDM, volume 2018- Novem, pages 197 206. IEEE
OpenAL. 2022. Introducing chatgpt.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong. Min-Yen Kan, and Tat-Seng Chua. 2020a. assimation-action--ellcttto Towards deep interaction between conversational and recom- mender systems. In Proc. of WSDM.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever. et al. 2019, Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Wenqiang Lei, Gangyi Zhang, Xiangnan He. Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. 2020b. Interactive path reasoning on graph for con- versational recommendation. In Proc. of KDD.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 11(5)554855551.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke 7eulemoyer. 2020. Bart: Denoising
pre-training for nat- ural language generation. translation, and comprehen- sion. In P Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 7871-7880.
Zhaochun Ren, Zhi Tian, Dongdong Li, Pengjie Ren, Liu Yang. Xin Xin, Huasheng Liang, Maarten de Ri- jke, and Zhumin Chen. 2022. Variational reasoning about user preferences for conversational recommen- dation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 165-175, New York, NY, USA. Association for Computing Machinery.
Raymond Li. Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommenda- tions. In Proc. of NeurlPS.
Ziming Li, Julia Kiseleva, and Maarten de Rijke. 2020. Rethinking supervised learning and reinforeement learning in task-oriented dialogue systems. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 3537-3546, Online. Association for Computational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvo,, Matthias Gallé, al. 2022. Bloom: 176b- parameter open-access multilingual language model.
Zujie Liang, Huang Hu, Can Xu, Jian Miao, Yingy- ing He, Yining Chen, Xiuho Geng, Fan Liang, and Daxin Jiang. 2021. Learning neural templates for recommender dialogue system. In Proc. of EMNLP.
Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang. Li Zhe, Hujun Bao, and Xipeng Qiu. 2021. Cpt: A pre-trained unbalanced transformer for both chinese language understanding and generation.
Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang. 2023a. Is chatgpt a good recommender? a preliminary study.
Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2019. Portuguese Named Entity Recognition using BERT-CRF.

---

Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agent.
Xiaoying Zhang, Hong Xie, Hang Li, and John C. S. Lui. 2020. Conversational contextual bandit: Algorithm and application. In Proc. of WWW.
Nan Zhao, Haoran Li, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2021. The jdde 2.0 corpus: A large- scale multimodal multi-turn chinese dialogue dataset for e-commerce customer service.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong. Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS.
Jinfeng Zhou, Bo Wang. Ruifang He, and Yuexian Hou. 2021. CRFR: Improving conversational rec- ommender systems via flexible fragments reasoning on knowledge graphs. In Proc. of EMNLP.
Lingzhi Wang, Huang Hu, Lei Sha, Can Xu, Daxin Jiang, and Kam-Fai Wong. 2022a. RecInDial: A uni- fied framework for conversational recommendation with pretrained language models. In Proceedings of the 2nd Conference of the Paciffc Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Lan- guage Processing (Volume Long Papers), pages 489-500, Online only. Association for Computational Linguistics.
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji- Ji-Rong Wen. 2020. Towards topic-guided conversational recommender system. In Proc. of COLING.
Yuanhang Zhou, Kun Zhou, Wayne Xin Zhao, Cheng Wang. Peng Jiang, and He Hu. 2022. CR-crs: Coarse- to-fine contrastive learning for conversational recom- mender system. In Proc. of WSDM.
Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao, Jingyuan Wang, and Ji- JinR Wen. 2023. Rethinking the evalu- ation for conversational recommendation in the era of large language models.
Jie Zou, Yifan Chen, and Evangelos Kanoulas. 2020.
Towards question-based recommender systems. Proc. of SIGIR.
Xiaolei Wang. Kun Zhou, JinRong Wen, and Wayne Xin Zhao. 2022b. Towards unified conversational rec- prompt
A Pre-sales Dialogue Tasks
ommender systems via knowledge-enhaneee learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing. KDD 22, page 1929-1937, New York, NY, USA. Association for Computing Machinery.
To evaluate the performance of CRSs, LLMs and their collaborations in E-commerce scenarios, we adopt four challenging tasks proposed in U-NEED dataset (Liu et al., 2023b). The four challeng- ing tasks are: (i) pre-sales dialogue understand- ing (ii) user needs elicitation (iii) user needs-based recommendation and (iv) pre-sales dialogue gener- ation.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals. Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica- tion.
Pre-sales dialogue understanding aims to under- stand utterances of both users and customer service staff. First, identify the attributes that are related to products. And second, extract preferences related to the identified attributes. For example, when a user says, "Which of your thermal underwear is the warmest? Recommend one?" This task aims to ob- tain semantic frames ("Functional requirement", "Warmest"), ("Catcgory", "Thermal underwear") where "Functional requirement" and "Category" are attributes related to products. "Warmest" and "Thermal underwear" are preferences.
Chi-Man Wong, Fan Feng, Wen Zhang, Chi-Man Vong, Hui Chen, Yichi Zhang, Peng He, Huan Chen, Kun Zhao, and Huajun Chen. 2021. Improving conver- sational recommender system by pretraining billion- scale knowledge graph. In 202i IEEE 37th Interna- tional Conference Data Engineering (ICDE).
Aoban Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2023. Glm-130h 130h: An open bilingual pre-trained model. In Proc. of ICLR. International Conference on Learning Repre- sentations, ICLR.
User needs elicitation aims to select attributes that can elicit more information about user needs.

---

The inputs for this task are the dialogue context and the identified user needs, i.e. ("Functional requirement", "Warmest"), ("Category", "Thermal underwear") ). The output is a set of attributes, e.g., ("Category", "Price").
lect GPT-2 (Radford et al., 2019) and KBRD (Chen et al.. 2019) baselines for the pre-sales dialogue generation task.
For the pre-sales dialogue understanding task, Bert, Bert+CRF, and Bert+BiLSSMM-CFF adopt the sequence labeling approach to identify the se- mantic frames. Bert considers only the representa- tion of the input utterance. Bert+CRI takes into ac- count the sequential relationships of the predicted tags in addition to the representation of the input utterance. Whereas Bert+BiLSTM++C adds bidi- rectional information encoding after obtaining the representation of the input and considers the se- quential relationships of the predicted tags to com- pute the probability of each tag.
User beeds-based recommendation aims rec- ommend products that satisfy explicit and implicit user needs. Explicit user needs refer the needs and preferences expressed by the user in the on- going dialogue, i... ("Functional requirement", "Warmest"), ( ("Category"". "Thermal underwear") J. Implicit user needs are related to user behav- iors outside of the pre-sales dialogue. Users usu- ally view some items before starting dialogue with the customer service staff. In addition, they browse through items while talking customer service staff. Such behaviors can reflect implicit user needs to some extent. The inputs to this task are explicit and implicit user needs, i.e., identified semantic frames and user behaviors, and the output is a collection of items.
For the user needs elicitation task, DiaMulti- Class and DiaSeq employ a multi-label classifi- cation approach to determine the collection of at- tributes. tributes. DiaMulticlass computes the probability of each attribute based on the representation of the inputs. DiaSe computes the probability of each at- tribute based on the sequential relationship between semantic frames.
Pre-sales dialogue generation aims to generate a response based on given information about user needs. Information consists of a collection of at- tributes and a collection of items. The collection of attributes is the output of the user needs elici- tation task, i.e., the attributes that may elicit more information about the user's needs. The collec- tion of items is the output of the user geeds-based recommendation task, i... items that satisfy the current user needs. The inputs to this task are the dialogue context, the collection of attributes and the collection of items. The output is response, which may be query asking a question about an attribute, "hat are your requirements for the thermal underwear?" or a recommendation reason for recommending an item, e.g., "Recommended item: 655398643290. This one is seamless and made of double-sided fleece. You can take look."
The baseline for the user needs-based recom- mendation task is Bert, SASRec and TG-CRS. Bert calculates the probability of an item based on the representation the input. SASRec calculates the probability of an item based on the sequential re- lationships of user behaviors. TG-CRS considers both dialogue context and sequential user behav- iors.
For the pre-sales dialogue generation task, the baselines are GPT-2 and KBRD. GPT-2 is a com- monly used pre-trained language model for the dia- logue generation task. KBRD utilizes switching mechanism to introduce tokens related to the rec- ommended items during the decoding of responses.
C Combinations of LLMs and CRSs
We define four variants of LLM assists CRS:
B Baselines for Pre-sales Dialogue Tasks
CLLM-BCRS refers that ChatGLM assists BART-based CRS.
Following Liu et al. (2023b), we (i) select Bert (De- vlin et al., 2019), Bert+CRF (Souza et al., 2019) and Bent+BiLSTM+CRR (Dai et al., 2019) as base- lines for the pre-sales dialogue understanding task; (ii) select DiaMultiClass (Li et al., 2020) and Di- aSeq (Li et al., 2020) baselines for the user needs elicitation task; (iii) select Bert (Devlin et al.. 2019), SASRec (Kang and McAuley, 2018) and TG-CRS (Zhou et al., 2020) as basclines for the user needs- needs-based recommendation task; (iv) and se-
CLLM-CCRS refers that ChatGLM assists CPT-based CRS.
ALLM-BCRS refers that Chinese-Alpaca as- sists BaRT-based CRS.
ALLM-CCRS refers that Chincse-Alpaca as- sists BART-based CRS.

---

We define four variants of CRS assists LLM:
The annotators evaluate the extent which gener- ated response includes information about the prod- uct, as compared to the ground truth. The score of informativeness and relevance ranges from to 5, and we calculate the average score from all annotators to obtain the final score.
BCRS-CLLM refers that BART-based CRS assists ChatGLM.
CCRS-CLLM refers that CPT-based CRS as- sists ChatGLM.
BCRS-ALLM refers that BART-based CRS assists Chinese-Alpann Alpaca.
E Implementation Details
We implement CRSs based on UniMIND + The code available online. For CRSs, we use a NVIDIA A A100-SXM4-80GB gpu and train model for 10 epochs, with a duration of approximately 12 hours. For LLMs, we use NVIDIA A100-SXM4- 80GB gpu and train model for 3 epochs, with duration of approximately 9 hours.
CCRS-ALLM refers that CPT-based CRS as-
sists Aliaca.-Alpaca.
D Evaluation Metrics
Following Liu al. (2023b), for the pre-sales di- alogue understanding and user needs elicitation tasks, we set Precision, Recall and Fl score as evaluation metrics. Precision is the proportion of correctly selected tags to the total number of se- lected tags. Recall is the ratio of correctly selected tags to the original number of correct tags. The Fl score is calculated by taking the harmonic mean of precision and recall.
F Examples of Fine-tuning LLMs
We give examples of the instructions, inputs, and outputs used to fine-tune the LLMs for each task in Tables 5, 6, 7, and 8, respectively.
G Examples of Collaborations of CRSs and LLMs
Regarding the user greds-based recommenda- tion task, the evaluation metrics in U-NEED (Liu et al., 2023b) are @it@ 10, Hit 50 and MRR @ 50. Due to the limitation of the input length of LLMs, where each product contains attributes and attribute values, we can provide a maximum of 20 candidate products. Therefore, in order to compare whether the collaborative approach improves the perfor- mance of CRSs, measure Accuracy (Hit@) 1), H and MRR @5. The Hit@ metric represents the proportion of relevant items that are present in the top-K results out of all the relevant items. The MRR @ K score is determined by taking the aver- age of the reciprocal ranks of the top-K items in a ranking. If an item does not appear in the top-K positions, its reciprocal rank is set O.
We show examples of inputs (instructions) for col- laborations between CRSs and LLMs on pre-sales dialogue understanding and generation tasks in Fig. 4 and Fig. 5, respectively.
The evaluation metrics for the pre-sales dialogue generation task are Distinct@) Informativeness and Relevance. Distinct@ @1 computed as the av- erage of the fraction distinct 1-grams out of all 1-grams in a response. Distinct@ 1 measures the diversity of generated responses. Informativeness and relevance are for human evaluation. We ran- domly sample 100 dialogues and recruit 12 an- notators to evaluate 1400 responses from 14 meth- ods on these 100 dialogues. Informativeness is calculated as the average informativeness of all gen- crated responses. Relevance determined as the average relevance degree of all generated responses.
h
Pnttess//iitibb.com/eeeeeeel

---

No collaboration
Collaborations of CRS and LLM
Instruction and input for LLMs
CRS assisting LLM
( to ae A 36 @fi u it emmesni unnt, V+4 99X, We Ne LI SEBM & an W (Based on pre-sales dialogue in the fashion domain, identify the item related attribute values and comerpanding attributes mentioned the
Instruction 4 ta F the Preaata ar ae s @F A cagg RAREEERR RI MAR 2i8 W te ReRdll l2 41 R.. S Ri 17 (Based pre sales CRS. related attribute values and corresponding attributes current imput user customer service rep. VnUser input requires identification attributes and their values, while customer service rep input may have empty attribute values)
dialogue
fashion domain and the identification
or oustomer rep, VnUser ingut requires identificatin attributes and their values. while customer service rep input may have empty attribute values.)
input
identiyy mentioned
Inpur [Round djvi@:: (0 AC FinStt 555 S 2e
RJ:: FRSE $x08 162 (Pre-sales disiogue, [Round diviser: Which of your thamal undenwear is the warmest? Recommend one? in Customer Service Rep: What are your requirements for the thermal undervear?y/Round 1] ViCurrent User: I 'd like themaal underwear for winter wear, since feel the cold.)
inpu [Round O waate AAN H24 Finss:: Sek. WME d Re Bonn of
RPF: 19at. #x : SUUS MMA; ALS: Preeer-sddes dialogue: [Round ojvnUser. Which of your thermal underwear is the warmest? Recommend one? in Customer Service Rep: What are requirements for the themal underear/Vv I 'd like thermal underwear for winter wear, since I feel the cold. VnThe identification results of CRS skin isvve stinging skin type: itching)
1) VnCumeent Input User
2h (u:;; (2 : 2 (Functional requirement
thermall ; Usage Penod :iiter)
Outpuetiee::::: (R:) dbaall::: #* (Functional requirement: thermal Usage Period: winter)
Prompt for CRSs
LLM assisting CRS
( 1M Mi Ktt.
# S:ll Funddrstand [ystem! #8,, 98 &.5 A Et fell8 ( aE:: Sel (AL 27900 131. 02
A AR
2:4 Un (R
tEl: Fitt PXB Isystentarar
BA A50 understandl
dideerstandlaltt 8xck $1
x: 8.: litt ie 14 #: SARRRRREE
2296. EE (LLMUIREE sall de
[[Uer]Which of your themaa underwear is the wameet? Recommend one? UUndestandifunction requirement underwear, [System]Whatt are your requirements Category: thermal undenwaar? Lunderstanjcateggoyy thermal underwear themaal underwear for winter wear, since the cold. Identify the user requirements)
([ser/Whicc of your thermal underwear is the warmest? Recommend one? [nnderstaddFuunctinnn requirementt warmest 1ysttem|Whatt are your requirements for the themnal underwear? [UnderstandyCatgoor thermal undervear [User] underveer vinter wear, since requirement thernal: Seeson winter. Identily the user requirements)
thermal 'd like
like thermal
the [LLM Functional
Figure 4: An example of collaboration between CRS and LLM on the pre-sales dialogue understanding task. Left side displays data used to fine-tune LLM and train CRS independently. The right side shows two cases of combining the two, Collaboration content is highlighted in red italics.
No collaboration
Collaborations of CRS and LLM
Instruction and input for LLMs
CRS assisting LLM
Instruction fetsaatiionnttt (e www.aaiaaaaaeeoeaa m SImF E0 PRRAR
Instruction-1eee KS Mi ae S1m Ra e Pmptterr Ei Hil 2e SRM. 12 8 35 (
[Based information obtained, user guiding attributes items that meet user requirements pre sale dialogue in fashion domain, generate user-ffl resporse that address user requirements)
information obtained. user guidig autributes and items that meet user requirements pre sale dialogue the fashion domain as well as the response generated by CRS generate Usee-liiendly resporse that address user requirements)
(Based
( [Round Olvife MINRRRE ABNKOUE # #- Finittt nit, WiG
Input Mo (Round Qunee: RORR 3 m- FFnttt:: Sekk mG & &o Bnjouund 1InEA: WA8 3:55 to Rommen 7 8 NYSER: SHERER (REE mi:: Raat Se: S my 4 #H00 R BRAMEM 1E AS
1/HHP & g. Sx0M E (E NYEER Giitts
(MC M8: bal 9- Exnntt Ai M. MHAREE PRD KiMIMME MRR #ime B. HIGEE 2t & S Pre sales dislogue: (Round Wuner Which of your themaal undenwear is the warnes? Recommend one? Customer Service Rep: What underwear? vioouud ThuUser: your requirements for thermal wear, since foel the cold. InObtained
DESSSER Tome B. RAAKG KERRS S6 Aist Rot A #aW Mer saless dialogue: (Round DjvnUser. Which of your thermal undenwear the warmest? Recommend one? Customer Service Rep What
requirements the thermal underwea?? VilRound Tuuuser: 'd like thermal underwear for winter since feel the cold. VnOetained thermal Category: themmal underwear, Seeson: requirement. meet requirements: Item meet
lke thermal underwear for winter
preference: functional requirement:: warnest thermat Category: thermal underwear, Seasor: winter
preforence: Functional
in items requrements. the price range is low, functional requirement thermal, seasons are winter auturnn, material is cotton, style thermal setly Response generated by CRS: What functions you need the product?)
that requirements, the
range requiements: item
functional requirement the seasons are winter autumn, material cotton, the style loose, the category themaal set)
calegory
Output wa8,, *tlll Tag/This one is seamlless and made of double--ided fleece You can take a look)
TTittThi one is searniess and made of double-Sided fleece, You can take look))
Outout: 1 3:00
Prompt for CRSs
LLM assistingg CRS
(user/( 11 Hets tftunddesstandhhhhssssss (8 fi.6: : 8x-8 lsystem & MUL a t te
[user] (user)e(1Swaattt ta ME 2:0 Tlunderst stand] WWSS:R SAE
138: Ris: StStsystem) SO Mi
unnuudetn understand Ride:
(understandd :1 KR: hi AlBckluse Hill.
2 2708. fill.
Maa R8::: (R $7799 198. [understandi w9: 3i
[understand] it 3t to MB xe &
(R it (oommneend6333986432200 $ ai m (Wser of your Recommend Category: themal underwear [SysleniWhhtt your requirements for the thermal undenwear? Sonderstandj/attgoong themaal underwear [User] 'd like underwear winter wear,
$ $ X
(aat: # 4x,
4D AD 3 ttll R R:M:: (User/Whic
thermal
themmal underwear is the warmest?
underwear is the warmest? Recommennd onee[Understarte innderstandFuunctioon. requirement wermest:
vee//Undeestandjjuntiooo requirement
thermal underwear WSystemWWhht your requirements for the themmal underwear? UinderstandjCaleggnn underwear [User]
like thermal underwear for
feel
the cold. [Understand] Functional requirement thermal Usage
Usage Period cold. [Understand] (553396433990LLMMM Take look it. winter (eccommeed)655396666 ancteracterial and anti-mite function. System respanse generation))
requirement
(ecomineedB$333944220.0 System response response generation))
An example of collaboration between CRS and LLM on the pre-sales dialogue generation task. Left side displays data used to fine-tune I.LM and train a CRS independently. The right side shows two cases of combining the two. Collaboration content is highlighted in red italics.
Figure 5:

---

Table 5: Three examples of fine-tuning LLMs for pre-sales dialogue understanding task in Appliance, Beauty and Fashion categories.
Example
Translation
Instruction
Combined with the pre-sales dialogue in Appliance category, identify the product-related attribute values and corresponding attributes involved in the input of the current user customer service. For user input, attributes and attribute values need to be identified. The attribute value in the customer service input may be empty.
4
&
N ft
म Hi :
Input
2 :6: | #
Pre-sale dialogue: User: Help me recommend an or- dinary washing machine with high cost performance, durable leather, and no drying function Current input: Customer service: Wave wheel or drum
X  AL 1. A
क AJ М
पे A: 3IR: M
34
Output
#2.0t
washing machine type: wave wheel; washing machine type: drum
T. MM
Instruction
Combined with the pre-sales dialogue in Beauty cate- gory, identify the product-related attribute values and corresponding attributes involved in the input of the current user or customer service. For user input, at- tributes and attribute values need to be identified. The attribute value in the customer service input may be empty.
N +
>
Input
T iN X :5 = 17
Pre-sale dialogue: User: Do you have set in your store? [SEP] Customer service: What type do you want? [SEP] User: Hydration [SEP] Customer service: Do you want milk? Kiss [SEP] User: Well, need water Milk(Seee service: Kiss, do you need to get rid of acne(SEP]User:: Need get rid of acne Current input: Customer service: Do you want to solve red, swollen, pimples keep your mouth shut?
# 2 # : 13
=
# > A * 3L 18
/े
[SEP] P. 0 मे
E *
[SEPP IR: *
S [SEP] F. - 1
3 A: 1 3
Output
UL M1 BU
skin problem: redness, swelling and acne; skin prob- lem: shut up
H.
M
Instruction
#
Combined with the pre-sales dialogue in Fashion cate- gory, identify the product-related attribute values and corresponding attributes involved in the input of the current user or customer service. For user input, at- tributes and attribute values need to be identified. The attribute value in the customer service input may be empty.
f - _ #
#
Input
* X :5: AP:
Do you have any other rec- ommendations? [SEP] Customer service: What ma- terial do you want, dear? [SEP] User: It's the ice silk one [SEP] Customer service: Only send product links [SEP] Customer service: This is made of modal mate- rial with ice silk touch [SEP] Customer service: Only send product links [SEP] Customer service: This is made of icc silk [SEP] User: Which one better to wear in winter Current input: User: My son like to wear the brand of Jinlilai.
Pre-sale dialogue: User:
# # [SEP] 11
#3yE & S/PPH// #1
2 BOE
1
ISEP E H I #S[SEP]
HR: 0 EN [
HE: B
A. || #1
Output
A#F
; *
Crowd: Son; Brand: Goldlion

---

Table 6: Three examples of fine-tuning LLMs for user needs elicitation task in Appliance, Beauty and Fashion categories.
Example
Translation
Instruction
According to the pre-sales dialogue in Appliance cat- egory, select a series of attributes to guide users to provide more preference information about needs. At- tribute values may or may not be included in the result.
-
E. #
Input
: # #
Pre-salc conversation: User: Help me recommend an ordinary washing machine with high cost perfor- mance, durable leather, and no drying function [SEP] Customer service: Wave wheel or drum [SEP] User: Drum [SEP] User: Simple function [SEP] Customer service: Only send product links [SEP] Customer ser- vice: Only send product links [SEP] User: Do you have Little Swan? Azimuth massage, as gentle as washing by hand, reshape the cleanliness and softness of the clothes between rubbing: (2)Boil and wash at 95 degrees high temperature, sweep away the viruses and bacteria hidden in the fibers of the clothes, long-term sterilization and disinfection, 99.99 99.9% healthy steriliza- tion (3) mobile phone remote control, anytime, anywhere, you can wear it as you want W4) the special down jacket, enter the water in multiple stages, the washing cycle is soft, prevent the down jacket from floating on the water or damage, even washing and care, caring (5)BLDC inverter motor, dehydration is faster and more thorough, clean and less residue [SEP User: Which one of the wave wheel more cost-effective! Durable
X BE 11 H.
S M T 11
SISEPP : WH 7M 191
[[SEP) N [SEP]
=: 3b ft AA (S[SE) HR
X ( 1 [SEP) ~ l ix
A: #
E U B5 [ HK (1)
TE tu F
0 X H ME
. (2)95 #it.
, 99.9% W (b
M M0 E tM - C
=: 1h E
F
Output
tz.
Price
Instruction
tx
According to the pre-sales dialogue in Beauty category, select a series of attributes to guide users to provide more preference information about needs. Attribute values may or may not be included in the result.
Input
3 AAA
Pre-salc dialogue: User: Do you have any suits in your store? Ellicacy
$
Output
31%
Instruction
According to the pre-sales dialogue in Fashion cat- egory, select series of attributes to guide users to provide more preference information about needs. At- tribute values may or may not be included in the result.
*
Input
; HJ.
T
Pre-sale dialogue: User: Do you have any other rec-
k #
ommendations?
Output
Material

---

Table 7: An example of fine-tuning LLMs for user needs- recommendation task in Fashion category.
Example
Translation
Instruction
According to the demand and preference information expressed by the user in the pre-sales dialogue in Fashion category and the candidate product informa- tion, the product that is most likely to meet the user's needs and preferences is selected from the candidate products A-T and recommended to the user.
E
Input
AU X :
3 ltb
Pre-sale dialogue: User: Do you have any other rec- ommendations [SEP] Customer service: What ma- terial do you want? Dear [SEP) User: is the kind of ice silk The attributes and attribute values corre- sponding to each candidate product: A The price range is high, the functional requirement is comfort- able, the season is summer, the gender is male, the clothing thickness is thin, the material is ice silk, cot- ton, modal, the style boxer, no trace, simple, and the category is men's boxer underwear [SEP] The price range of B is high [SEP] The price range of C is high [SEP] The price range of D is high, the gender is male, and the category is pajamas/home service sets [SEP] The price range of is high [ The price range SEPJF is medium, the functional requirement to keep warm, the scason autumn, the gender is female, and the category is thermal suits. The price range of [SEP]G is high. The price range of [SEP]H is high. The price range is high, the functional requirement is to keep warm, the gender is female, the clothing thickness is thin, and the category ther- mal suit [SEP]J, the price range is high[SEP]K, the price range is high|SEP]L. is high [SEP] the price range of M is high [SEP] the price range of N is high [SEP] the price range of O is high [SEP] the price range of P is high [SEP] the price range of Q is high [SEP) The price range of R is high [SEP] the price range of S high [SEP] the price range of T is high, the style is v-neck, and the category is thermal suit A
म (SEP)
D क # A: M (1
# М 3
R , #
# T [SEP]B
th K & # 61 # X
[=]
A D1 7
11:
BU 말े 5 X E HR
it # X E
A RA [SPP]GM f1 # IX [R] A
(SE)H N3
(SEP]I
it th K M E
3 & AR I7 DE E N के.
1 (E D1 NY K [7]
K 60 IX (H) F [SEP]LA
SSEPIN b0 &
t & [H] BY /I K3 K (H)
| 10 NY E I
t (n) P TJ SEP (A)
[SEP]TA Di HS E (I] E m.
*
E
Output A

---

Table 8: Three examples of fine-tuning LLMs for pre-sales dialogue generation task in Appliance, Beauty and Fashion categories.
Example
Translation
Instruction
According to the information obtained in the pre-sales dialogue in Appliance category, the attributes that guide the user's needs, and the product information that meets the user's needs, generate a popular reply that responds to the user's needs and is easy for the user to understand. Pre-sale conversation: User: Help me recommend an ordinary washing machine with high cost performance, durable leather, and no drying function [SEP] Customer service: Wave wheel or drum [SEP] User: Drum [SEP] User: [SEP] Customer service with simple functions: only send product links [SEP] customer service: only send product links [SEP] user: do you have Little Swan? The all-round massage is gentle as hand washing, and the rubbing will reshape the cleanliness and softness of the clothes; (2)Boil and wash 95 degrees high temperature, sweep away the virus bacteria hidden in the fibers of the clothes, long- sterilization and disinfection. 99.9% healthy sterilization (3)wifi mobile phone Remote con- trol. you can wear it anytime, anywhere (4)Special down jacket washing, multi- stage water intake, gentle washing cycle, prevent down jacket from floating damaged, even washing and care, caring (5)BLDC frequency conversion motor, dehydration is faster and more thorough, clean and less residue [SEP] User: Which one of the wave wheel is more cost-effective? Durable Leather Acquired user demand preference information: category: washing ma- chine, functional requirements: durable leather, no drying, simple function, durable leather, style: cost cost-effective, cost- effective, washing machine type: drum, pulsator, brand: Little Swan Attributes to guide user needs: Price: How much is the budget?
a
S
Input
+ HU E #
#P #E
#X 15 11
* #1 # t4 H E 38 mi H #1.
T #tt + 5 RE ( Il. IT
b म SELEP) P: 1 H1[
SE 1* DS : X EH IN ME
#ISEP| :R: X IN R RE [SSPPj/B
H: /
(1)
#
&
#0
$ (2)95 A.
K
* 99.9% (3) FL
#2 SART HU, # (F [5
SAREEE B VEAA 11
FA
A III A2 H0, 19
J.
AL.
Rbck
# H. H
3 2 HK
M #1
:E H T
AJ # #3
1 BA M 8EF2
: 15
fi
fi:
Output
Instruction
# Ml #
According to the information obtained in the pro-sales dialogue Beauty category, the attributes that guide the user's needs, and the product information that meets the user's neods, generate a popular reply that responds to the user's needs and is casy for the user to understand.
E.
Input
S SE #
Pre-sales dialogue: User: Do you have suits in your store Acquired user demand preference information: Category: Set Attributes to guide user needs: Function: what type do you want
(.
Output
Instruction
#C HK
According to the information obtained in the pre-sales dialogue Fashion category, the attributes that guide the user's needs, and the product information that meets the user's needs, generate popular reply that responds the user's needs and is easy for the user to understand. Pre-sale dialogue: User: Do you have any other recommen- dations? Obtained user demand preference information: Style: Attributes that guide user demand: Material: What material do you want dear?
H. "
J HE. 12
H
Input
#ct; 5]
5 :t: Mi
Output


Reviews:
[{"review_id": "UEu3QA18iD", "reviewer": "Reviewer_j99w", "summary": "", "questions": "1. What are the specific properties of e-commerce pre-sale dialogues have been explored by the proposed model?\n\n2. Why using different evaluation metrics in Table 1, 2, and 3?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for your careful review. We really appreciate your efforts in reviewing our paper.\n\nOur work provides many interesting experimental findings, mainly including but not limited to:\n1. Main finding of our work：CRSs and LLMs can benefit each other. \nOn the four tasks of Conversational Recommendation, the state-of-the-art experimental results of the collaborative approach compared with the non-collaborative approach always appear in the collaborative approach, and this phenomenon strongly proves the effectiveness of improving the performance of the conversational recommendation system through collaboration between the traditional CRS approach and the LLMs approach.\n2. Effectiveness of collaborations varies across domains.\nFor the results in the Shoes and Phones domains in Tasks 3 and 4, some of the metrics appear state-of-the-art in the non-collaborative approach. This indicates that our proposed collaborative methods have different effectiveness for different distribution data. The reason for this may be that the corresponding conversations under the Shoes and Phones domains are more homogeneous and the product information is scarce, so it is difficult to highlight the advantages of the model in utilizing the information under the collaborative approach. This suggests that the richness of information under the corresponding scenarios of the data needs to be evaluated beforehand when applying our proposed collaborative approach.\n3. Effectiveness of collaborations varies across tasks.\nFor the two different collaboration approaches, XLLM-YCRS and YCRS-XLLM, the strengths and limitations of the approaches are observed based on the experimental results of Task 1, 2 and 3. \nFrom Table 1-3, we observe that (i) XLLM-YCRS > YCRS-XLLM on Task 1 and (ii) XLLM-YCRS < YCRS-XLLM on Tasks 2 and 3. The experimental results show consistency with the model results under the non-collaborative approach (CRS > LLM on Task 1; CRS < LLM on Tasks 2 and 3). That is, when the better model is used as the subject and the other type of model is used for collaborative assistance, an overall improved result will be obtained.\n4. Main finding on Task 1: LLMs show superb performance in understanding user needs.\nTable 1 shows that the LLMs approach achieves a performance that exceeds that of the CRSs and the classical approach. Moreover, the performance of LLMs is further improved by considering the prediction results of CRSs. We attribute this to the strong capability of the LLMs for dialogue understanding. Note that traditional CRS methods struggle to model the fine-grained needs and preferences in a dialogue. The emergence of LLMs, on the other hand, makes it possible for a pre-sales shopping robot to truly understand users' needs and preferences. In practice,  \"CRS assisting LLM\" approach may be a viable solution based on the results in Table 1.\n5. Main finding on Task 2: CRSs and LLMs show limited performance in user needs elicitation.\nThe capacity of LLMs to provide decision-making information in certain ways (e.g., chain of thoughts) is a key point of exploration in LLMs. In this work, we explore whether LLMs can select attributes that can elicit users' preferences. The results in Table 2 show the state-of-the-art performance of LLMs in assisting CRSs on the Precision metric. However, in the Recall metric it is the classical approach applied in industry that achieves state-of-the-art performance. Note that ChatGLM assisting CRSs improves performance, while Chinese-Alpaca assisting CRSs decreases performance. This may be an accumulation of errors due to the limited performance of Chinese-Alpaca on Task 2. The collaboration between LLMs and CRSs in terms of strategies leaves a lot of room for exploration. We plan to explore more sophisticated ways of collaboration in future work, such as LLMs and CRSs engaging in debates to decide which attributes should be chosen at the moment.\n6. Main finding on Task 3:  LLMs (i.e. generative recommendation) can boost CRSs to achieve the best performance. \nNote that although we fine-tuned the LLMs with training data, the LLMs providing product recommendations is not a classification task. LLMs select and recommend product from a given candidate list based on understanding the given instruction and input. CRSs, on the other hand, model the product recommendation as a classification task, and its prediction is computed based on the relations between user representation and candidate product representation. We consider these to be two different approaches to providing recommendations. The results in Table 3 show that modeling the relations between user representations and candidate item representations is still effective, and the two ways of providing recommendations are mutually helpful. In detail, LLMs can enhance CRSs to achieve state-of-the-art performance, while CRSs can also help LLMs to get moderate performance. We believe that this is inspiring to study generative recommender systems for E-commerce scenarios.\n7. Main finding on Task 4: CRSs and LLMs show comparable performance in pre-sales dialogue generation.\nIt is widely recognized that the LLMs are much better at generating responses than the smaller models. However, Table 4 demonstrates that the performance of the two is comparable and does not exceed the human standard response. In the human evaluation: a score of 4 (or 5) means that a model-generated response meets (or exceeds) the ground truth response, respectively. And there is no major improvement in collaboration between the two. This means that there is still some room for improvement in pre-sales dialog generation. And the small model is still worth exploring because it can achieve comparable performance without spending a lot of resources and time for both training and inference.\n\n### Thank you, Reviewer j99w. Here are the responses to your concerns/questions.\n> ... the proposed two collaboration models can also be applied to other conversational recommendation scenarios, e.g., movie recommendation.\n\nA: Yes, You're right. However, user needs-based recommendation is very different from movie recommendation. The LLMs has movie knowledge, so many studies have found that the LLMs can do well on the movie recommendation task even in the 0-shot case[1, 2]. We focus on product recommendation, which contains knowledge that LLMs do not have. From the experimental results, we also see that the LLMs has good performance on pre-sales dialog understanding (Table 1) and more limited performance on user needs-based recommendation (Table 3). Even after fine-tuning, it could not outperform the CRS SOTA approach. This shows that how to make LLMs model structured product attribute information is an important challenge for LLMs applied in e-commerce recommendation.\nThis paper is trying to show the insight that CRS and LLMs are complementary on pre-sales dialogue tasks. This finding may be instructive for recommendations in other domains, but may not be entirely applicable.\nFrom our experience, it depends on the knowledge of ITEM. Movies, with a lot of textual information, movie summary, movie reviews, etc., are suitable for LLMs to \"read\". Products, on the other hand, relate to a lot of structured knowledge, such as user actions, product attributes, etc., which are not easily understood by LLMs. Moreover, when we transform them into instructions and inputs, there is also missing information.\n\n> ... only ... on one e-commerce pre-sale dialogue dataset. Thus, the experimental analysis is not sufficient.\n\nA:  For the exploration of the field of E-commerce pre-sales dialogue, we believe that our experiments are sufficient and provide a lot of interesting findings. Thank you for your suggestion, and in future work we will explore whether similar findings exist in other domains of recommendations.\n\n> What are the specific properties of e-commerce pre-sale dialogues have been explored by the proposed model?\n\nA：We list the main findings of our experimental results above, which can hopefully answer your question.\n\n> Why using different evaluation metrics in Table 1, 2, and 3?\n\nA: For the metrics in Tables 1 and 2, we adopt the benchmark and evaluation metrics proposed in the U-NEED dataset paper[3]. Regarding the user needs-based recommendation task, the evaluation metrics in [3] are Hit@10, Hit@50 and MRR@50. Due to the limitation of the input length of LLMs, where each product contains attributes and attribute values, we can provide a maximum of 20 candidate products. Therefore, in order to compare whether the collaborative approach improves the performance of CRSs, we measure Acc, Hit@5 and MRR@5.\n\n[1] Wang, Xiaolei, et al. \"Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models.\" arXiv preprint arXiv:2305.13112 (2023).\n[2]Hou, Yupeng, et al. \"Large language models are zero-shot rankers for recommender systems.\" arXiv preprint arXiv:2305.08845 (2023).\n[3] Liu, Yuanxing, et al. \"U-NEED: A Fine-grained Dataset for User Needs-Centric E-commerce Conversational Recommendation\" . In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23). Association for Computing Machinery, New York, NY, USA, 2723–2732. https://doi.org/10.1145/3539618.3591878", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose to study the combination of large language model and conversational recommendation system in pre-sales dialogue in e-commerce. This idea seems novel.\n\n2. The authors propose two collaboration methods: 1) conversational recommendation system assisting large language model, and 2) large language model assisting conversational recommendation system.\n\n3. The authors have performed extensive experiments on a real-world pre-sale dialogue dataset in e-commerce.\n\n4. This paper is well-structured and clearly written.", "reasons_to_reject": "1. The main objective of this work is to explore the integration of conversational recommendation system with large language model in e-commerce pre-sale dialogues. However, in the proposed models, the specific properties of e-commerce pre-sales dialogues are not explored.\n\n2. According to my understanding, the proposed two collaboration models can also be applied to other conversational recommendation scenarios, e.g., movie recommendation. However, in this work, the authors only study the performance of the proposed methods on one e-commerce pre-sale dialogue dataset. Thus, the experimental analysis is not sufficient.", "questions_to_authors": "1. What are the specific properties of e-commerce pre-sale dialogues have been explored by the proposed model?\n\n2. Why using different evaluation metrics in Table 1, 2, and 3?", "paper_topic_and_main_contributions": "In this paper, the authors study the effectiveness of combing large language model with conversational recommendation system in e-commerce pre-sales dialogues. They propose two collaboration methods, i.e., conversational recommender systems assisting large language model and large language model assisting conversational recommender systems. The authors have performed extensive experiments on real datasets to demonstrate the effectiveness of the proposed methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N.A.", "Typos_Grammar_Style_and_Presentation_Improvements": "N.A."}, "extra_text": {}}, {"review_id": "Lq9nCsXTw3", "reviewer": "Reviewer_MtkJ", "summary": "", "questions": "Please refer to **Reasons To Reject**.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for your careful review. We really appreciate your efforts in reviewing our paper.\nLarge language models(LLMs) have become a hot topic in AI research. On most tasks, LLMs can easily achieve performance far beyond that of the state-of-the-art methods with only a small amount of fine-tune data. Many researchers have begun to explore the capabilities and limitations of LLMs models for various tasks and scenarios. \n\nWe would like to emphasize that our work can provide valuable insights for the future research and application of LLMs in E-commerce scenarios. \n1. We explore the capacities of LLMs in real-world E-commerce scenarios, i.e. E-commerce pre-sales dialogues.\nOnline shopping has become an inaccessible part of life. Pre-sales dialogue is a key factor in improving user experience and increasing purchase rates. The dataset we use is constructed based on real conversations between users and customer service staff, rather than conversations simulated by crowdsourcers. Our work explores four important tasks in E-commerce pre-sales dialogue that can provide insights for applying LLMs in e-commerce scenarios as well as developing LLMs-based customer service assistants. \n2. We explore collaborations between LLMs and conversational recommender systems(CRSs).\nLLMs generate answers based on massive pre-trained knowledge and understanding of the given instruction and input. CRSs, on the other hand, learn representations of users, items, and attributes to compute probabilities. We view these as two different approaches to solving questions. Our work reports the results of combining these two types of approaches, which is instructive for subsequent research (open-domain LLMs base + small models with domain-specific knowledge).\n3. We conduct extensive experiments that show many interesting findings.\nWe explore 2 types of collaboration on 4 important tasks, 4 product categories, 2 LLMs and 2 CRSs.\nBased on the experimental results, we can analyze the differences and impacts of (1) tasks (2) collaboration methods (3) base models (4) product categories. See below for details.\n\nOur work provides many interesting experimental findings, mainly including but not limited to:\n1. Main finding of our work：CRSs and LLMs can benefit each other. \nOn the four tasks of Conversational Recommendation, the state-of-the-art experimental results of the collaborative approach compared with the non-collaborative approach always appear in the collaborative approach, and this phenomenon strongly proves the effectiveness of improving the performance of the conversational recommendation system through collaboration between the traditional CRS approach and the LLMs approach.\n2. Effectiveness of collaborations varies across domains.\nFor the results in the Shoes and Phones domains in Tasks 3 and 4, some of the metrics appear state-of-the-art in the non-collaborative approach. This indicates that our proposed collaborative methods have different effectiveness for different distribution data. The reason for this may be that the corresponding conversations under the Shoes and Phones domains are more homogeneous and the product information is scarce, so it is difficult to highlight the advantages of the model in utilizing the information under the collaborative approach. This suggests that the richness of information under the corresponding scenarios of the data needs to be evaluated beforehand when applying our proposed collaborative approach.\n3. Effectiveness of collaborations varies across tasks.\nFor the two different collaboration approaches, XLLM-YCRS and YCRS-XLLM, the strengths and limitations of the approaches are observed based on the experimental results of Task 1, 2 and 3. \nFrom Table 1-3, we observe that (i) XLLM-YCRS > YCRS-XLLM on Task 1 and (ii) XLLM-YCRS < YCRS-XLLM on Tasks 2 and 3. The experimental results show consistency with the model results under the non-collaborative approach (CRS > LLM on Task 1; CRS < LLM on Tasks 2 and 3). That is, when the better model is used as the subject and the other type of model is used for collaborative assistance, an overall improved result will be obtained.\n4. Main finding on Task 1: LLMs show superb performance in understanding user needs.\nTable 1 shows that the LLMs approach achieves a performance that exceeds that of the CRSs and the classical approach. Moreover, the performance of LLMs is further improved by considering the prediction results of CRSs. We attribute this to the strong capability of the LLMs for dialogue understanding. Note that traditional CRS methods struggle to model the fine-grained needs and preferences in a dialogue. The emergence of LLMs, on the other hand, makes it possible for a pre-sales shopping robot to truly understand users' needs and preferences. In practice,  \"CRS assisting LLM\" approach may be a viable solution based on the results in Table 1.\n5. Main finding on Task 2: CRSs and LLMs show limited performance in user needs elicitation.\nThe capacity of LLMs to provide decision-making information in certain ways (e.g., chain of thoughts) is a key point of exploration in LLMs. In this work, we explore whether LLMs can select attributes that can elicit users' preferences. The results in Table 2 show the state-of-the-art performance of LLMs in assisting CRSs on the Precision metric. However, in the Recall metric it is the classical approach applied in industry that achieves state-of-the-art performance. Note that ChatGLM assisting CRSs improves performance, while Chinese-Alpaca assisting CRSs decreases performance. This may be an accumulation of errors due to the limited performance of Chinese-Alpaca on Task 2. The collaboration between LLMs and CRSs in terms of strategies leaves a lot of room for exploration. We plan to explore more sophisticated ways of collaboration in future work, such as LLMs and CRSs engaging in debates to decide which attributes should be chosen at the moment.\n6. Main finding on Task 3:  LLMs (i.e. generative recommendation) can boost CRSs to achieve the best performance. \nNote that although we fine-tuned the LLMs with training data, the LLMs providing product recommendations is not a classification task. LLMs select and recommend product from a given candidate list based on understanding the given instruction and input. CRSs, on the other hand, model the product recommendation as a classification task, and its prediction is computed based on the relations between user representation and candidate product representation. We consider these to be two different approaches to providing recommendations. The results in Table 3 show that modeling the relations between user representations and candidate item representations is still effective, and the two ways of providing recommendations are mutually helpful. In detail, LLMs can enhance CRSs to achieve state-of-the-art performance, while CRSs can also help LLMs to get moderate performance. We believe that this is inspiring to study generative recommender systems for E-commerce scenarios.\n7. Main finding on Task 4: CRSs and LLMs show comparable performance in pre-sales dialogue generation.\nIt is widely recognized that the LLMs are much better at generating responses than the smaller models. However, Table 4 demonstrates that the performance of the two is comparable and does not exceed the human standard response. In the human evaluation: a score of 4 (or 5) means that a model-generated response meets (or exceeds) the ground truth response, respectively. And there is no major improvement in collaboration between the two. This means that there is still some room for improvement in pre-sales dialog generation. And the small model is still worth exploring because it can achieve comparable performance without spending a lot of resources and time for both training and inference.\n\n### Thank you, Reviewer MtkJ. Here are the responses to your concerns/questions.\n\n> The motivation for using CRSs for recommendation is not clear. \n\nA: CRSs can do all the tasks of E-commerce pre-sales dialogue. We guess you might be asking why we chose CRSs to collaborate with LLMs. We focus on exploring the capabilities of LLMs for E-commerce pre-sales dialogue, which is a specific scenario of conversational recommendation. \nCRSs learn user, item, and attribute representations to compute recommendation probabilities. While LLMs generate recommended products directly based on the understanding of the given instruction and input.\nWe focus on exploring the performance of the combination of these two types of approaches for four important tasks in E-commerce pre-sales dialogue.\n\n> The authors should justify why it is not sufficient to fine-tune an LLM on the recommendation task. \n\nA: First the performance of LLMs on user needs-based recommendation is relatively limited. We think that the possible reason is that LLMs do not have the product-related knowledge. Although fine-tuning methods enable LLMs to be more familiar with the instructions and inputs of the tasks, it is still difficult for LLMs to retain the knowledge of the related attributes of the products. In addition, in real scenarios, the number of related candidate products may be more than 50,000 and the products are updated frequently, fine-tuning is not feasible as it consumes a lot of resources.\n\n> The setting of no collaboration in Table 1,2,3,4 shows that the proposed method has only a marginal advantage over the single LLM, despite the incorporation of the collaboration mechanism.\n\nA: In Table 3, on all five categories, ALLM-CCRS shows a considerable improvement in Acc (0.2639->0.2822), H@5 (0.5617->0.5832), and M@5 (0.3737->0.3919) compared to the CRS method without collaboration. In addition, we list the main findings of the experimental results above.\n\n> The combination of the CRSs and the LLM is kind of simple and ad hoc. The authors should explain whether there are any alternatives or ablations to the proposed interaction mechanism.\n\nA: The collaboration approach we used can be seen as a simplified version of a mixture of experts. As we state in the response to the first question, LLMs and CRSs have different approaches to completing tasks. And we have considered other ways for LLMs to collaborate with CRS, such as joint training. However the time and GPU resources required by fine-tuing LLMs don't really allow LLMs to have complex interactions with CRSs. Thank you for your advice. We will explore more ways for LLMs to collaborate with smaller models based on the findings of this work.\n\n> The term task 3 in Line 401 is undefined and confusing. The authors should clarify what it refers to and how it relates to the previous tasks or sections of the paper.\n\nA: Task 3 refers to user needs-based recommendation task. The corresponding experimental results are in Table 3.\n\n> It might be inappropriate to move part of the analysis of the main result into the appendix. The authors should include the most important findings and insights in the main paper. Moreover, the figure/table referred in Appendix A is missing or mislabeled. The writing of the experiment section is messy and hard to follow. The authors should improve the clarity and coherence of their presentation, and avoid grammatical errors and typos.\n\nA: Thanks for pointing out this problem. We appreciate your time. We have reorganized the findings of the experimental results. We list the main findings above. We will rewrite the section of experimental results analysis in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors explore a novel setting in conversational recommendation, i.e., LLM collaborating with CRS.", "reasons_to_reject": "1. The motivation for using CRSs for recommendation is not clear. The authors should justify why it is not sufficient to fine-tune an LLM on the recommendation task. The setting **no collaboration** in Table 1,2,3,4 shows that the proposed method has only a marginal advantage over the single LLM, despite the incorporation of the collaboration mechanism.\n2. The combination of the CRSs and the LLM is kind of simple and ad hoc. The authors should explain whether there are any alternatives or ablations to the proposed interaction mechanism.\n3. The term **task 3** in Line 401 is undefined and confusing. The authors should clarify what it refers to and how it relates to the previous tasks or sections of the paper.\n4. It might be inappropriate to move part of the analysis of the **main** result into the appendix. The authors should include the most important findings and insights in the main paper. Moreover, the figure/table referred in Appendix A is missing or mislabeled.\n5. The writing of the experiment section is messy and hard to follow. The authors should improve the clarity and coherence of their presentation, and avoid grammatical errors and typos.", "questions_to_authors": "Please refer to **Reasons To Reject**.", "paper_topic_and_main_contributions": "This work proposes to combine a large language model (LLM) and a conversational recommender system (CRS) for precise recommendation. Experiments show the effectiveness of this kind of collaboration.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None."}, "extra_text": {}}, {"review_id": "FaymUJy3en", "reviewer": "Reviewer_FfbT", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for your careful review. We really appreciate your efforts in reviewing our paper.\nLarge language models(LLMs) have become a hot topic in AI research. On most tasks, LLMs can easily achieve performance far beyond that of the state-of-the-art methods with only a small amount of fine-tune data. Many researchers have begun to explore the capabilities and limitations of LLMs models for various tasks and scenarios. \n\nWe would like to emphasize that our work can provide valuable insights for the future research and application of LLMs in E-commerce scenarios. \n1. We explore the capacities of LLMs in real-world E-commerce scenarios, i.e. E-commerce pre-sales dialogues.\nOnline shopping has become an inaccessible part of life. Pre-sales dialogue is a key factor in improving user experience and increasing purchase rates. The dataset we use is constructed based on real conversations between users and customer service staff, rather than conversations simulated by crowdsourcers. Our work explores four important tasks in E-commerce pre-sales dialogue that can provide insights for applying LLMs in e-commerce scenarios as well as developing LLMs-based customer service assistants. \n2. We explore collaborations between LLMs and conversational recommender systems(CRSs).\nLLMs generate answers based on massive pre-trained knowledge and understanding of the given instruction and input. CRSs, on the other hand, learn representations of users, items, and attributes to compute probabilities. We view these as two different approaches to solving questions. Our work reports the results of combining these two types of approaches, which is instructive for subsequent research (open-domain LLMs base + small models with domain-specific knowledge).\n3. We conduct extensive experiments that show many interesting findings.\nWe explore 2 types of collaboration on 4 important tasks, 4 product categories, 2 LLMs and 2 CRSs.\nBased on the experimental results, we can analyze the differences and impacts of (1) tasks (2) collaboration methods (3) base models (4) product categories. See below for details.\n\nOur work provides many interesting experimental findings, mainly including but not limited to:\n1. Main finding of our work：CRSs and LLMs can benefit each other. \nOn the four tasks of Conversational Recommendation, the state-of-the-art experimental results of the collaborative approach compared with the non-collaborative approach always appear in the collaborative approach, and this phenomenon strongly proves the effectiveness of improving the performance of the conversational recommendation system through collaboration between the traditional CRS approach and the LLMs approach.\n2. Effectiveness of collaborations varies across domains.\nFor the results in the Shoes and Phones domains in Tasks 3 and 4, some of the metrics appear state-of-the-art in the non-collaborative approach. This indicates that our proposed collaborative methods have different effectiveness for different distribution data. The reason for this may be that the corresponding conversations under the Shoes and Phones domains are more homogeneous and the product information is scarce, so it is difficult to highlight the advantages of the model in utilizing the information under the collaborative approach. This suggests that the richness of information under the corresponding scenarios of the data needs to be evaluated beforehand when applying our proposed collaborative approach.\n3. Effectiveness of collaborations varies across tasks.\nFor the two different collaboration approaches, XLLM-YCRS and YCRS-XLLM, the strengths and limitations of the approaches are observed based on the experimental results of Task 1, 2 and 3. \nFrom Table 1-3, we observe that (i) XLLM-YCRS > YCRS-XLLM on Task 1 and (ii) XLLM-YCRS < YCRS-XLLM on Tasks 2 and 3. The experimental results show consistency with the model results under the non-collaborative approach (CRS > LLM on Task 1; CRS < LLM on Tasks 2 and 3). That is, when the better model is used as the subject and the other type of model is used for collaborative assistance, an overall improved result will be obtained.\n4. Main finding on Task 1: LLMs show superb performance in understanding user needs.\nTable 1 shows that the LLMs approach achieves a performance that exceeds that of the CRSs and the classical approach. Moreover, the performance of LLMs is further improved by considering the prediction results of CRSs. We attribute this to the strong capability of the LLMs for dialogue understanding. Note that traditional CRS methods struggle to model the fine-grained needs and preferences in a dialogue. The emergence of LLMs, on the other hand, makes it possible for a pre-sales shopping robot to truly understand users' needs and preferences. In practice,  \"CRS assisting LLM\" approach may be a viable solution based on the results in Table 1.\n5. Main finding on Task 2: CRSs and LLMs show limited performance in user needs elicitation.\nThe capacity of LLMs to provide decision-making information in certain ways (e.g., chain of thoughts) is a key point of exploration in LLMs. In this work, we explore whether LLMs can select attributes that can elicit users' preferences. The results in Table 2 show the state-of-the-art performance of LLMs in assisting CRSs on the Precision metric. However, in the Recall metric it is the classical approach applied in industry that achieves state-of-the-art performance. Note that ChatGLM assisting CRSs improves performance, while Chinese-Alpaca assisting CRSs decreases performance. This may be an accumulation of errors due to the limited performance of Chinese-Alpaca on Task 2. The collaboration between LLMs and CRSs in terms of strategies leaves a lot of room for exploration. We plan to explore more sophisticated ways of collaboration in future work, such as LLMs and CRSs engaging in debates to decide which attributes should be chosen at the moment.\n6. Main finding on Task 3:  LLMs (i.e. generative recommendation) can boost CRSs to achieve the best performance. \nNote that although we fine-tuned the LLMs with training data, the LLMs providing product recommendations is not a classification task. LLMs select and recommend product from a given candidate list based on understanding the given instruction and input. CRSs, on the other hand, model the product recommendation as a classification task, and its prediction is computed based on the relations between user representation and candidate product representation. We consider these to be two different approaches to providing recommendations. The results in Table 3 show that modeling the relations between user representations and candidate item representations is still effective, and the two ways of providing recommendations are mutually helpful. In detail, LLMs can enhance CRSs to achieve state-of-the-art performance, while CRSs can also help LLMs to get moderate performance. We believe that this is inspiring to study generative recommender systems for E-commerce scenarios.\n7. Main finding on Task 4: CRSs and LLMs show comparable performance in pre-sales dialogue generation.\nIt is widely recognized that the LLMs are much better at generating responses than the smaller models. However, Table 4 demonstrates that the performance of the two is comparable and does not exceed the human standard response. In the human evaluation: a score of 4 (or 5) means that a model-generated response meets (or exceeds) the ground truth response, respectively. And there is no major improvement in collaboration between the two. This means that there is still some room for improvement in pre-sales dialog generation. And the small model is still worth exploring because it can achieve comparable performance without spending a lot of resources and time for both training and inference.\n\n\n### Thank you, Reviewer FfbT. Here are the responses to your concerns/questions.\n\n> The paper's writing needs improvement. Four different tasks are defined in the paper, but the details of each task are not clearly explained. For specifics, see the Typos, Grammar, Style, and Presentation Improvements section.\n\nA: We appreciate your time and constructive review. We will revise the writing of this paper following your suggestions. Also, we will rewrite the Results and Analysis section to include the findings we listed above. We will also tweak other sections, such as the Appendix, to make this paper easy to understand.\n\n> The proposed method only achieves improvements on some specific CRS tasks.\n\nA: We list the main findings of the experimental results above. \n\n> There is a lack of explanation for instances where no improvement is observed. For example, in line 406, \"In contrast, the performance in task 1 across all categories declines.\"\n\nA: Thanks for your suggestion, we will rewrite the analysis of experimental results in the final version and add more explanations.\n\n> The authors only conducted experiments on a single dataset, which could be a limitation.\n\nA: To the best of our knowledge, there are no other accessible datasets in the field of E-commerce pre-sales dialogue.  The dataset we use contains five product categories, i.e. Beauty, Phones, Fashion, Shoes and Electronics, which somewhat alleviates this limitation.\n\n> No analysis of computational overhead is provided.\n\nA: This is due to the limitation of the length of the submission. For CRSs, we use a NVIDIA A100-SXM4-80GB gpu and train model for 10 epochs, with a duration of approximately 12 hours. For LLMs, we use a NVIDIA A100-SXM4-80GB gpu and train model for 3 epochs, with a duration of approximately 9 hours. Thank you for your suggestion and we will add implementation details in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Using LLMs to assist in recommendation tasks is an interesting problem.\n2. The authors' proposal to combine LLMs with traditional CRS models is well-motivated.", "reasons_to_reject": "1. The paper's writing needs improvement. Four different tasks are defined in the paper, but the details of each task are not clearly explained. For specifics, see the Typos, Grammar, Style, and Presentation Improvements section.\n2. The proposed method only achieves improvements on some specific CRS tasks.\n3. There is a lack of explanation for instances where no improvement is observed. For example, in line 406, \"In contrast, the performance in task 1 across all categories declines.\"\n4. The authors only conducted experiments on a single dataset, which could be a limitation.\n5. No analysis of computational overhead is provided.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores the collaboration of conversational recommender systems (CRSs) and large language models (LLMs) in E-commerce pre-sales dialogues. The paper proposes two methods of collaboration: CRS assisting LLM and LLM assisting CRS. The paper evaluates the effectiveness of these methods on a real-world dataset of pre-sales dialogues, involving four tasks: dialogue understanding, needs elicitation, recommendation, and dialogue generation. The paper shows that the collaboration of CRS and LLM can improve the performance of these tasks.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper's writing and the content of the corresponding figures need revision to make them easier to understand. For example,\n\n1. Section 3.3 does not introduce each task in multi-task training, and there is no specific explanation or example for the design of prompts. Figure 3 and the Figures in the Appendix are also difficult to understand.\n2. The variable name definitions in Section 3.3 are problematic. $X_S$ is defined but not used. $X_R$ and $Z_R$ are not defined. The user needs-based recommendation task corresponding to Eq. 2 and Eq. 3 is not defined. Section 3.5 has similar issues.\n3. In the experimental analysis section, the author uses tasks 1, 2, 3, and 4 to represent the tasks. Using task in Table 1, 2, 3, 4 or directly using the task name are more suitable."}}, {"review_id": "umVjrSPuCP", "reviewer": "Reviewer_n8cG", "summary": "", "questions": "1. Given the observations in Reasons to Reject (#1), how do you view the difference in model capacity between the proposed methods and the baselines?\n2. Lines 473-474 claim that \"CRSs assisting LLMs\" can improve the \"robustness\" of the LLMs alone. Are CRSs improving LLMs' robustness or are they improving their domain knowledge?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for your careful review. We really appreciate your efforts in reviewing our paper.\n\nOur work provides many interesting experimental findings, mainly including but not limited to:\n1. Main finding of our work：CRSs and LLMs can benefit each other. \nOn the four tasks of Conversational Recommendation, the state-of-the-art experimental results of the collaborative approach compared with the non-collaborative approach always appear in the collaborative approach, and this phenomenon strongly proves the effectiveness of improving the performance of the conversational recommendation system through collaboration between the traditional CRS approach and the LLMs approach.\n2. Effectiveness of collaborations varies across domains.\nFor the results in the Shoes and Phones domains in Tasks 3 and 4, some of the metrics appear state-of-the-art in the non-collaborative approach. This indicates that our proposed collaborative methods have different effectiveness for different distribution data. The reason for this may be that the corresponding conversations under the Shoes and Phones domains are more homogeneous and the product information is scarce, so it is difficult to highlight the advantages of the model in utilizing the information under the collaborative approach. This suggests that the richness of information under the corresponding scenarios of the data needs to be evaluated beforehand when applying our proposed collaborative approach.\n3. Effectiveness of collaborations varies across tasks.\nFor the two different collaboration approaches, XLLM-YCRS and YCRS-XLLM, the strengths and limitations of the approaches are observed based on the experimental results of Task 1, 2 and 3. \nFrom Table 1-3, we observe that (i) XLLM-YCRS > YCRS-XLLM on Task 1 and (ii) XLLM-YCRS < YCRS-XLLM on Tasks 2 and 3. The experimental results show consistency with the model results under the non-collaborative approach (CRS > LLM on Task 1; CRS < LLM on Tasks 2 and 3). That is, when the better model is used as the subject and the other type of model is used for collaborative assistance, an overall improved result will be obtained.\n4. Main finding on Task 1: LLMs show superb performance in understanding user needs.\nTable 1 shows that the LLMs approach achieves a performance that exceeds that of the CRSs and the classical approach. Moreover, the performance of LLMs is further improved by considering the prediction results of CRSs. We attribute this to the strong capability of the LLMs for dialogue understanding. Note that traditional CRS methods struggle to model the fine-grained needs and preferences in a dialogue. The emergence of LLMs, on the other hand, makes it possible for a pre-sales shopping robot to truly understand users' needs and preferences. In practice,  \"CRS assisting LLM\" approach may be a viable solution based on the results in Table 1.\n5. Main finding on Task 2: CRSs and LLMs show limited performance in user needs elicitation.\nThe capacity of LLMs to provide decision-making information in certain ways (e.g., chain of thoughts) is a key point of exploration in LLMs. In this work, we explore whether LLMs can select attributes that can elicit users' preferences. The results in Table 2 show the state-of-the-art performance of LLMs in assisting CRSs on the Precision metric. However, in the Recall metric it is the classical approach applied in industry that achieves state-of-the-art performance. Note that ChatGLM assisting CRSs improves performance, while Chinese-Alpaca assisting CRSs decreases performance. This may be an accumulation of errors due to the limited performance of Chinese-Alpaca on Task 2. The collaboration between LLMs and CRSs in terms of strategies leaves a lot of room for exploration. We plan to explore more sophisticated ways of collaboration in future work, such as LLMs and CRSs engaging in debates to decide which attributes should be chosen at the moment.\n6. Main finding on Task 3:  LLMs (i.e. generative recommendation) can boost CRSs to achieve the best performance. \nNote that although we fine-tuned the LLMs with training data, the LLMs providing product recommendations is not a classification task. LLMs select and recommend product from a given candidate list based on understanding the given instruction and input. CRSs, on the other hand, model the product recommendation as a classification task, and its prediction is computed based on the relations between user representation and candidate product representation. We consider these to be two different approaches to providing recommendations. The results in Table 3 show that modeling the relations between user representations and candidate item representations is still effective, and the two ways of providing recommendations are mutually helpful. In detail, LLMs can enhance CRSs to achieve state-of-the-art performance, while CRSs can also help LLMs to get moderate performance. We believe that this is inspiring to study generative recommender systems for E-commerce scenarios.\n7. Main finding on Task 4: CRSs and LLMs show comparable performance in pre-sales dialogue generation.\nIt is widely recognized that the LLMs are much better at generating responses than the smaller models. However, Table 4 demonstrates that the performance of the two is comparable and does not exceed the human standard response. In the human evaluation: a score of 4 (or 5) means that a model-generated response meets (or exceeds) the ground truth response, respectively. And there is no major improvement in collaboration between the two. This means that there is still some room for improvement in pre-sales dialog generation. And the small model is still worth exploring because it can achieve comparable performance without spending a lot of resources and time for both training and inference.\n\n### Thank you, Reviewer n8cG. Here are the responses to your concerns/questions.\n\n> these marginal gains are due to the additional parameters introduced by the CRS \n\nA: We do not think that collaboration \"increases\" the number of parameters in LLMs. We think this collaboration \"considers\" results from other perspectives. CRSs learn user, item, and attribute representations from E-commerce structured information to compute recommendation probabilities. While LLMs generate recommended products directly based on the understanding of the given instruction and input. \n\n> --- in this case, would a slightly larger LLM be enough to achieve similar gains?\n\nA: There might be. But that's not the same contribution as this paper. \n\n> Methodology aside, I am not sure how novel and practically helpful the findings are. \n\nA: We list the main findings above.\n\n> This takeaway has already been contributed by various previous works, ... for example: a) \"What does BERT know about books, movies and music? Probing BERT for conversational recommendation\" (Penha et al., 2020) shows how LLMs can be helpful in recommendation-making; \n\nA: BERT is a pre-trained language model, but not a large language model in terms of parameter scales.\n\n> However, these previous contributions are not properly acknowledged in the paper.\n\nA: We think that none of the language models (with ~100 million parameters) used in the literature listed may be viewed as LLMs (with at least 1 billion parameters). \n\n> The paper makes too big of a claim in that it is the first to investigate the combination of LLMs + CRSs (e.g., line 14, lines 97-100). Lines 135-137 are not accurate (see #2 above).\n\nA: We indicate the scope of this paper's contribution, E-commerce pre-sales dialogue, both in the title and in the end part of the Introduction. Thank you for your feedback. We will revise the statements in the corresponding parts to avoid ambiguity.\n\n> The majority of the Appendices is copied verbatim from \"U-NEED: A Fine-grained Dataset for User Needs-Centric E-Commerce Conversational Recommendation\" (Liu et al., 2023), even duplicated in lines 979-994.\n\nA: We apologize for this mistake. We will rewrite the experimental settings in the final version. It will not affect the experimental results and findings.\n\n> Given the observations in Reasons to Reject (#1), how do you view the difference in model capacity between the proposed methods and the baselines?\n\nA: In terms of the number of parameters, they may be incomparable. The number of LLMs parameters may be far more than the classical approach. But for a specific task, the difference in their performance can provide some insights into applying LLMs in real-world scenarios.\n\n> Lines 473-474 claim that \"CRSs assisting LLMs\" can improve the \"robustness\" of the LLMs alone. Are CRSs improving LLMs' robustness or are they improving their domain knowledge?\n\nA: Sorry for the ambiguity. The collaborative approach of this paper does not augment the domain knowledge of LLMs. From our point of view, it is difficult to make LLMs memorize a large amount of domain knowledge. A recent research hotspot is to enhance the domain knowledge of LLMs by means of retrieval. However, in this way, LLMs still need to process the information retrieved back. We would like to have small models like CRSs that can learn the structured information of real scenarios to process some information on complex relationships. And the LLMs can directly utilize this information in a collaborative way.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper focuses on a relevant topic with direct, real-world applicability.\n2. Experiments contemplate five product categories, and include sub-tasks that are consensually important to conversational recommendations. The product recommendation sub-task has received increasing attention from the NLP community, particularly in how to leverage LLMs for the task. \n3. Interestingly, it is for the product recommendation sub-task that authors report the most promising gains (as seen in aggregated accuracy of ALLM-CCRS, Table 3). A paper focused on \"LLMs assisting CRSs\" applied to the product recommendation sub-task could lead to potentially helpful findings to the community.", "reasons_to_reject": "1. Since there are smaller LMs behind UniMIND (e.g., BART) and larger LMs representing LLMs, the experimental set up makes it hard to understand if the gains are due to the proposed methods or simply due to increased model capacity. Comparing the combination of LLMs + CRSs to the LLMs alone shows small gains (e.g., on Tables 1 and 4), so one conclusion can be that these marginal gains are due to the additional parameters introduced by the CRS --- in this case, would a slightly larger LLM be enough to achieve similar gains?\n2. Methodology aside, I am not sure how novel and practically helpful the findings are. Essentially, the main takeaway is that it depends on the nature of the sub-task how to combine LLMs and CRSs. This takeaway has already been contributed by various previous works, for example:\na) \"What does BERT know about books, movies and music? Probing BERT for conversational recommendation\" (Penha et al., 2020) shows how LLMs can be helpful in recommendation-making;\nb) \"'It doesn't look good for a date': Transforming Critiques into Preferences for Conversational Recommendation Systems\" (Bursztyn et al., 2021) shows how LLMs can be helpful in preference interpretation;\nc) \"ReXPlug: Explainable Recommendation using Plug-and-Play Language Model\" (Hada and Shevade, 2021) shows how LLMs can be helpful in recommendation explanation.\nHowever, these previous contributions are not properly acknowledged in the paper.\n3. The paper makes too big of a claim in that it is the first to investigate the combination of LLMs + CRSs (e.g., line 14, lines 97-100). Lines 135-137 are not accurate (see #2 above).\n4. The majority of the Appendices is copied *verbatim* from \"U-NEED: A Fine-grained Dataset for User Needs-Centric E-Commerce Conversational Recommendation\" (Liu et al., 2023), even duplicated in lines 979-994.", "questions_to_authors": "1. Given the observations in Reasons to Reject (#1), how do you view the difference in model capacity between the proposed methods and the baselines?\n2. Lines 473-474 claim that \"CRSs assisting LLMs\" can improve the \"robustness\" of the LLMs alone. Are CRSs improving LLMs' robustness or are they improving their domain knowledge?\n", "paper_topic_and_main_contributions": "This paper focuses on the combination of LLMs and CRSs in the context of product recommendations. Authors approach the topic from two viewpoints --- either \"LLMs assisting CRSs\", or \"CRSs assisting LLMs\" --- and across four sub-tasks implied in recommendation-making: intent understanding, preference elicitation, product recommendation, and response generation. Evaluation measures depend on the sub-task: precision, recall, and F1 for intent understanding and preference elicitation; accuracy, hit@5 and MRR@5 for product recommendation; and distinct@1, informativeness, and relevance for response generation.\nAuthors rely on the U-NEED dataset, which comprises 1135-1748 dialogues organized across five product categories: Beauty, Phones, Fashion, Shoes, and Eletronics. More simple baselines vary for each sub-task, but more complex baselines generally include both CRSs (variations of UniMIND) and LLMs (ChatGLM-6B and Chinese-Alpaca) in isolation, as well as the proposed methods: LLMs combined into CRSs (ChatGLM/Alpaca combined into UniMIND), or CRSs combined into LLMs (UniMIND combined into ChatGLM/Alpaca). Findings are somewhat unclear when comparing the proposed methods against the more complex baselines, with marginal gains associated to either \"LLMs assisting CRSs\" or \"CRSs assisting LLMs\" depending on the sub-task.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "As described in Reasons to Reject:\n\"What does BERT know about books, movies and music? Probing BERT for conversational recommendation\" (Penha et al., RecSys 2020);\n\"'It doesn't look good for a date': Transforming Critiques into Preferences for Conversational Recommendation Systems\" (Bursztyn et al., EMNLP 2021);\n\"ReXPlug: Explainable Recommendation using Plug-and-Play Language Model\" (Hada and Shevade, SIGIR 2021).\n", "Typos_Grammar_Style_and_Presentation_Improvements": "1. In Table 4, Dist-1 for Chinese-Alpaca should be marked in bold --- not CCRS-ALLM.\n2. Duplicated information in lines 309-313 and lines 979-994.\n3. Verb \"guess\" used twice in lines 407 and 485 --- less colloquial verbs like \"hypothesize\" would be more appropriate.\n4. Distinct-1 should be supported by a citation."}}]

Author rebuttal:

Meta-review: This paper introduces approaches for combining the training of conversational recommender systems (CRS) and large language models (LLM) for pre-sales dialogue. They look at how CRS can assist LLM and vice versa. They look at four subtasks; intent understanding, preference elicitation, product recommendation, and response generation. They use the U-NEED dataset which has five product categories; beauty, phones, fashion, shoes, and electronics. The reviewers agree that this is a well motivated and interesting problem that has real world applications. The reviewers disagree on the clarity of the paper and have provided helpful suggestions for improvements. The authors provide detailed results and analysis that provide insights that future work may build off of. The reviewers range in excitement. The novelty is limited in comparison to recent works pointed out by the reviewers. Small improvements are seen in the results. Reviewer n8cG points out the potential confound in the number of model parameters that may contribute to the improvements. However, the work provides insights that will help inform future work in this area. The overclaim of novelty should be corrected in the next revision. Similarly, the plagiarized portion of the appendix must be rewritten and/or properly attributed in the next revision.
Decision: Accept-Findings