Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
Authors: Xiaolei Wang
Xinyu Tang
Xin Zhao
Jingyuan Wang
Ji-Rong Wen
Venue: EMNLP 2023 Main (2023)
Keywords: Conversational Recommendation
Large Language Model
Evaluation

Abstract: The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs), which rely on natural language conversations to satisfy user needs. In this paper, we embark on an investigation into the utilization of ChatGPT for CRSs, revealing the inadequacy of the existing evaluation protocol. It might overemphasize the matching with ground-truth items annotated by humans while neglecting the interactive nature of CRSs.

To overcome the limitation, we further propose an **i**nteractive **Eva**luation approach based on **L**L**M**s, named **iEvaLM**, which harnesses LLM-based user simulators. Our evaluation approach can simulate various system-user interaction scenarios. Through the experiments on two public CRS datasets, we demonstrate notable improvements compared to the prevailing evaluation protocol. Furthermore, we emphasize the evaluation of explainability, and ChatGPT showcases persuasive explanation generation for its recommendations. Our study contributes to a deeper comprehension of the untapped potential of LLMs for CRSs and provides a more flexible and realistic evaluation approach for future research about LLM-based CRSs.

Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
Xiaolei Wang 13 Xinyu Tangl, Wayne Xin Zhao Jingyuan Wang' and Ji-Rong Wen 1.2.3
Gaoling School of Artificial Intelligence, Renmin University of China
School of Information, Renmin University of China
Beijing Key Laboratory of Big Data Management and Analysis Methods
*School of Computer Science and Engineering, Beihang University wx1 wx119990foxmail.. com, com, txy2001033 001 63. com, batmanfly@gmaill com
Abstract
Since CRSs rely the ability to understand and generate natural language conversations, capable approaches for CRSs have been built on pre-trained language models in existing literature (Wang et al., 2022c; Deng et al., 2023). More recently, large lan- guage models (LLMs) (Zhao et al., 2023a), such as ChatGPT, have shown that they are capable of solving various natural language tasks via conver- sations. Since ChatGPT has acquired wealth of world knowledge during pre-training and is also specially optimized for conversation, to be an excellent CRS. However, there still lacks a comprehensive study of how LLMs (e.g., Chat- GPT) perform in conversational recommendation.
The recent success large language mod- els (LLMs) has shown great potential de- velop more powerful conversational recom- mender systems (CRSs), which rely on natural language conversations satisfy user needs. In this paper, we embark on an investigation into the utilization of ChatGPT for CRSs, revealing the inadequacy of the existing evaluation proto- col. might overemphasize the matching with ground-truth items annotated by humans while neglecting the interactive nature of CRSs.
expected
To overcome the limitation, pose an interactive Evaluation approach based on LLMs, named iEvaLM, which harnesses LLM- user simulators. Our evaluation ap- proach can simulate various system- inter- action scenarios. Through the experiments on two public CRS datasets, we demonstrate no- table improvements compared to the prevailing evaluation protocol. Furthermore, size the evaluation of explainability, and Chat- GPT showeases persuasive explanation gen- eration for its recommendations. Our study contributes to a deeper comprehension the untapped potential LLMs for CRSs and provides more flexible and realistic evalu- ation approach for future research about LLM- based CRSs. The code is available at https: //gi www.hh....o/.u....... cww/RUCATBox/ iiw.o//-EEvaaMMM.S.a.
further pro-
To investigate the capacity of LLMs in CRSs, we conduct empirical study on the performance of ChatGPT on existing benchmark datasets. We fol- low the standard evaluation protocol and compare CharGPT against state-of-the-ar CRS methods. Surprisingly, the finding is rather counter-intuitive ChatGPT shows unsatisfactory performance in this empirical evaluation. To comprehend the reason behind this discovery, we examine the failure cases and discover that the current evaluation protocol is the primary cause. It relies on the matching be- tween manually annotated recommendations and conversations and might overemphasize the fitting of ground-truth items based on the conversation context. Since most CRS datasets are created in a chit-chat way, we find that these conversations are often vague about the user preference, making difficult to exactly match the ground-truth items even for human annotation. In addition, the current evaluation protocol is based on fixed conversations, which does not take the interactive nature of con- versational recommendation into account. Similar findings have also been discussed on text gener- ation tasks (Bang et al., 2023; Qin et al., 2023); traditional metries (e.g., BLEU and ROUGE) may not reflect the real capacities of LLMs.
empha-
1 Introduction
Conversational recommender systems (CRSs) aim provide high-quality recommendation services through natural language conversations that span multiple rounds. Typically, in CRSs, a recom- mender module provides recommendations based on user preferences from the conversation context, and a conversation module generates responses given the conversation context and item recommen-
dation.
"Equal contribution.
Corresponding author.
Considering this issue, we aim to improve the

---

evaluation approach, make it more focused on the interactive capacities of CRSs. Ideally, such an evaluation approach should be conducted by hu- mans, since the performance of CRSs would finally be tested by real users in practice. However, user studies are both expensive and cume-conuu,ing making them infeasible for large-scale evaluations. As surrogate, user simulators can be used for evaluation. However, existing simulation methods are typically limited pre-defined conversation flows or template-basee based utterances (Lei et al., 2020; Zhang and Balog, 2020). To address these limita- tions, a more flexible user simulator that supports free-form interaction in CRSs is actually needed.
Dataset
#Dialogues
#Utterances
Domains
ReDial
10,006
182,150
Movie
Movie, Book,
OpenDialKG
13,802
91,209
Sports, Music
Table I: Statistics of the datasets.
the two datasets. It demonstrates the superiority of ChatGPT across different scenarios, which is expected for general-purpose CRS.
We summarize our key contributions as follows: (1) To the best of our knowledge, time that the capability of ChatGPT for conver- sational recommendation has been systematically examined on large-scale datasets.
the first
To this end, this work further proposes an interactive Evaluation approach based on LLMs, named iEvaLM, in which LLM-based user sim- ulation is conducted to examine the performance. Our approach draws inspiration from the remark- able instroctionn-following capabilities exhibited by LLMs, which have already been leveraged for role- play (Fu et al., 2023). With elaborately designed instructions, LLMs can interact with users in a highly cooperative manner. Thus, we design our user simulators based on LLMs, which can flexi- bly adapt to different CRSs without further tuning. Our evaluation approach frees CRSs from the con- straints of rigid, Humann-rrtten conversation texts, allowing them interact with users more natu- ral manner, which close to the experience real users. To give a comprehensive evaluation, we also consider two types of interaction: strribute-based question answering and free-form chit-chat.
(2) We provide detailed analysis of the limita- tions of ChatGPT under the traditional evaluation protocol, discussing the root cause of why it fails on existing benchmarks.
(3) We propose a new interactive approach that employs LLM-basee user simulators for evaluating CRSs. Through experiments conducted on two pub- lic CRS datasets, we demonstrate the effectiveness and reliability of our evaluation approach.
2 Background and Experimental Setup
In this section, we describe the task definition and experimental setup used in this work.
2.1 Task Description
Conversational Recommender Systems (CRSs) are designed to provide item recommendations through multi-turn interaction. The interaction can be di- vided into two main categories: question answering based on templates (Lei et al., 2020; Tu et al., 2022) and chit-chat based on natural language (Wang et al., 2023; Zhao al., 2023c). In this work, we consider the second category. At each turn, the sys- tem either presents a recommendation or initiates a new round of conversation. This process continues until the user either accepts the recommended items or terminates the conversation. In general, CRSs consist of two major subtasks: recommendation and conversation. Given its demonstrated prowess in conversation (Zhao et al., 2023b), we focus our evaluation of ChatGPT its performance in the recommendation subtask.
With this new evaluation approach, we observe significant improvements in the performance of ChatGPT, as demonstrated through assessments conducted on two publicly available CRS datasets, Notably, the Recall @ 10 metric has increased from 0.174 to 0.570 on the REDIAL dataset with five- round interaction, even surpassing the Recall@50 50 result the currently leading CRS baseline. More- over, in our evaluation approach, we have taken the crucial aspect of explainability into consideration, wherein ChatGPT exhibits proficiency in providing persuasive explanations for its recommendations. Besides, existing CRSs can also benefit from the in- teraction, which is important ability overlooked by the traditional evaluation. However, they per- form much worse in the setting of attribute-based question answering on the OPENDIALKG dataset, while ChatGPT performs better both settings on
2.2 Experimental Setup
Datasets. We conduct experiments on the RE- DIAL (Li et al., 2018) and OPENDIALKG (Moon

---

al., 2019) datasets. REDIAL is the most com- monly used dataset in CRS, which is about movie recommendations. OPENDIALKG
hello I'm open to any movie
would suggest some comethes you could watch, have you seen The Woolding Singer (19982
a multi- domain CRS dataset covering not only movies but also books, sports, and music. Both datasets are widely used for CRS evaluation. The statistics for them are summarized in Table 1.
have not seen it but watched American Pie 2 (2001). watched Avengers: Infinity (2013) liked
instruction) Recemmend 10 items that consistent with user preference in the dialogue.
It seems like you enjoy both
action und comeddy movies.
Response as query
Baselines. We present comparative analysis of ChatGPT with a selection of representative super- vised and unsupervised methods:
(Furmat guideline) The format is: no. tite.
Recommendation Models
Item
Recommend by itself
Sure. here are movies:
RED (2000)
1. The Avengers (2012)
Lethal Weapon (1987)
KBRD (Chen et al., 2019): It introduces DBpe- dia to enrich the semantic understanding of entities mentioned in dialogues.
2. Avengers: Endgame (2019)
(b) Integrating Recommendation Models
Zero-shot Prompting
KGSF (Zhou et al., 2020): It leverages two KGs to enhance the semantic representations of words and entities and use Mutual Information Maximiza- tion to align these two semantic spaces.
Figure 1: The method adapting ChatGPT for CRSs.
set temperature=0 when calling the API. All the prompts we used are detailed in Appendix C.
CRFR (Zhou et al., 2021a): It performs flex- ible fragment reasoning on KGs to address their inherent incompleteness.
ChatGPT for Conversational Recommendation
BARCOR (Wang et al., 2022b): It proposes unified CRS based on BART (Lewis et al., 2020), which tackles two tasks using a single model.
In this section, we first discuss how to adapt Chat- GPT for CRSs, and then analyze its performance.
MESE (Yang et al., 2022): It formulates the recommendation task as a two-stage item retrieval process, i.e. candidate selection and ranking, and introduces meta- incermmnformtioon when encoding items. UniCRS (Wang et al.. 2022c): designs prompts with KGs for DialoGPT (Zhang et al., 2020) to tackle two tasks in a unified approach. (Neelakantan et al., 2022): It is a powerful model provided in the Ope- nAT APT to transform each input into embeddings, which can be used for recommendation.
3.1 Methodology
Since ChatGPT is specially optimized for dialogue, possesses significant potential for conversational recommendation. Here we propose two approaches to stimulating this ability, as illustrated Figure 1.
Zero-shot Prompting. We first investigate the ability of ChatGPT through zero-shot prompting (see Appendix C.I). The prompt consists of two parts: task instruction (describing the task) and format guideline (specifying the output format).
rext-embedding-uua--00
Among the above baselines, text eext-embedding ida-002 an unsupervised method, while others are supervised and trained on CRS datasets.
Integrating Recommendation Models. Although ChatGPT can directly generate the items, it is not specially optimized for recommendation (Kang et al., 2023; Dai et al., 2023). In addition, it tends to generate items that are outside the eval- uation datasets, which makes it difficult to directly assess the predictions. To bridge this gap, we incorporate external recommendation models to constrain the output space. We concatenate the conversation history and generated responses inputs for these models to directly predict tar- get items calculate the similarity with item candidates for matching. We select the CRS model MESE (Yang et al., 2022) as the supervised method (ChatGPT + embeddiemdddinaaaeeee ada-002) and the wewttemmedding-aad ada-003 (Neelakantan et al.,
Following existing work (Zhang et al., 2023; Zhou et al., 2022), we adopt Recall @ & evaluate the recommendation subtask. Specifically, we k 1, 10, 50 follow- ing Zhang et al. (2023) for the RF.DTAL. dataset, and 1, 10, 25 following Zhou ct al. (2022) for the OPENDIALKG dataset. Since requiring too many items can sometimes be refused by ChatGPT, we only assess Recall@1 and Recall @ 10 for it.
Evaluation Metrics.
Model details. We employ the publicly available provided in the OpenAT API, which the underlying model of ChatGPT. To make the output as deterministic as possible, we
model 3.t-3.5-tur

---

Datasets
ReDial
OpenDialKG
Models
Recall
Recall 10
Recall
Recall
Recall@
Recall 25
KBRD
0.028
0.169
0.366
0.231
0,423
0.492
KGSF
0.039
0.183
0.378
0.119
0,436
0.523
CRFR
0.040
0.202
0.399
0.130
0,458
0.543
BARCOR
0,031
0.170
0.372
0.312
0,453
0.510
UniCRS
0.050
0.215
0.413
0,308
0.513
0.574
MESE
0.056
0.256*
0,455
0.279
0.592*
0.666
text-embedding ada-002
0.025
0.140
0.250
0.279
0.519
0.571
ChatGPT
0.034
0.172
0,105
0.264
+ MESE
0.036
0.195
0.240
0,508
0.037
0.174
0.310
0.539
Table 2: Overall performance of existing CRSs and ChatGPT. Since requiring too many items at once can sometimes be refused by ChatGPT, we only assess Recall@ and Recall 10 for it, while Recall @50 marked as "-". Numbers marked with indicate that the improvement is statistically significant compared with the best baseline (I-test with p-value 0.05).
Partially relevant
Highly relevant
3.2.2 Explainability
Dataset
Irrelevant
To better understand how ChatGPT conducts the recommendation, we require to generate an ex- planation to examine the inner working princi- ples. Then, we employ two annotators to judge the relevance degree (irrelevant, partially relevant, or highly relevant) of the explanation to the con- versation context 100 randomly sampled failure examples. The Cohen's Kappa between annotators is 0.77, indicating good agreement. The results in Table indicate that ChatGPT can give highly relevant explanations most of the cases. Here is one example:
ReDial
8%
20%
72%
OpenDialKG
20%
16%
64%
Table The relevance degree of the explanations gen- erated by ChatGPT to the conversation context.
2022) model provided in the OpenAl API as the unsupervised method (CharGPT + MESE).
3.2 Evaluation Results
We first compare the accuracy of ChatGPT with CRS baselines following existing work (Chen et al., 2019; Zhang al., 2023). Then, examine the inner working principles ChatGPT, we show- case the explanations generated by it to assess its explainability as suggested by Guo et al. (2023).
[Conversation Historyl
User:
movie like Super Troopers (2001)
[Label] Police Academy (1984)
[Prediction ChatGPT] Beerfest (2006), The Other Guys (2010), Hot Fuzz (2007).
3.2.1 Accuracy
[Explanation of ChatGPT]... .... have recommended movies that share similar themes of comedy, law en- forcement, and absurdity. Some of the movies the list are also from the same creators or feature of the
The performance comparison of different methods for CRS shown in Table 2. Surprisingly, Chat- GPT does not perform well we expect. When using zero-shot prompting, ChatGPT only achieves average performance among these baselines and far behind the top-performing methods. When inte- grating external recommendation models, its per- formance can be effectively improved. In particu- lar, the OPENDIALKG dataset, the performance gap is significantly reduced. It indicates that the responses generated by ChatGPT models understand the user preference. However, there is still a noticeable performance gap on the REDIAL dataset.
Super Troopers...
As see, ChatGPT understands the user preference and gives reasonable explanations, sug- gesting that it can contradicts its poor performance in accuracy. It motivates us to investigate the reasons for failure.
good CRS. However, this
help external
3.3 Why does ChatGPT Fail?
In this part, we analyze why does ChatGPT fail terms accuracy. Through inspecting the incor-

---

[Conversation History]
[Conversation History]
User: Hello, am looking for movies for a night with friends that have coming up. Any sugges- tions?
Bot: What genre of movie do you like?
User: like comedies
[Response in the Dataset] Have you seen
[Label] Black Panther (2018)
Talludega Nights: The Ballad of Ricky Bobby (2006)
[Prediction of ChatGPT) The Hangover (2009), Bridesmaids (2011), Superbad (2007), Mean Girls (2004), The Princess Bride (1987), Clueless (1995),
or The Hangover (20099}
[Response by ChatGPT] That's great! Comedies are fun and lightheurted genre that can make you laugh and feel good. Do you have a favorite comedy movie or actor?
The Breakfast Club (1985), Ferris Bueller's Day Off (1986), Pitch Perfect (2012), 21 Jump Street (2012)
(a) Lack of explicit user preference
(b) Lack proactive clarification
Figure 2: Two failure examples of ChatGPT for conversation recommendation.
rect recommendations (evaluated according to the annotated labels), we identify two main causes and detail them in the following part.
ment, it is reasonable to seek clarification before making recommendation. However, such cases cannot be well handled in the existing evaluation protocol since no more user responses are available in this process. To verify this, we randomly sample 100 failure examples for two annotators to classify the responses generated by ChatGPT (clarification, recommendation, chit-chat). We find that 36% of them are clarifications, 11% chit-chat, and only 53% are recommendations, suggesting the im- portance of considering clarification in evaluation. The Cohen's Kappa between annotators is 0.81.
Lack of Explicit User Preference. The exam- ples in this class typically have very short con- versation turns, in which CRSs may be unable to collect sufficient evidence to accurately infer the user intention. Furthermore, the conversations are mainly collected chit-chat form, making it vague to reflect the real user preference. To see this, we present an example in Figure 2(a). As we can see, the user does not provide any explicit information about the expected items, which is a common phe- nomenon as observed by Wang et al. (2022a). To verify this, randomly sample 100 failure amples with less than three turns and invite two annotators to determine whether the user prefer- ence is ambiguous. Among them, 51 examples are annotated as ambiguous, and the rest 49% are considered clear, which confirms our speculation. The Cohen's 's Kappa between annotators is 0.75, Compared with existing models trained on CRS datasets, such an issue is actually more serious for ChatGPT, since it not fine-tuned and makes pre- diction solely based on the dialogue context.
To summarize, there are two potential issues with the existing evaluation protocol: lack of ex- plicit user preference and proactive clarification. Although conversation-leva evaluation (Zhang and Balog, 2020) allows system-user interaction, it is limited to pre-defined conversation flows or template-based utterances (Lei et al., 2020; Zhang and Balog, 2020), failing to capture the intricacies and nuances of real-world conversations.
A New Evaluation Approach for CRSs
Considering the issues with the existing evaluation protocol, in this section, we propose alterna- tive evaluation approach, iEvaLM,, which features LLM-based interactive evaluation with LLM- based user simula- tion, as illustrated in Figure 3. We demonstrate its effectiveness and reliability through experiments.
Lack of Proactive Clarification. A major lim- itation in evaluation is that it has to strictly fol- low existing conversation flows. However, in real- world scenarios, a CRS would propose proactive clarification when needed, which is not supported by existing evaluation protocols. To see this, present an example in Figure 2(b), As can see, the response in the dataset directly gives recommen- dations, while ChatGPT asks for detailed user pref- erence. Since so many items fit the current require-
4.1 Overview
Our approach is seamlessly integrated with exist- ing CRS datasets. Each system-user interaction extends over one of the observed human-annotated conversations. The key idea our approach is to conduct re-ael user simulation based on the

---

User Simulator
4.3
User Simulation
(LLM)
Instriction
init
Exisling CRS datheets
train
Conversational Recommendation System
To support the interaction with the system, we em- ploy LLMs for user simulation. The simulated user can take on one of the following three behaviors:
Persono
Beluvior roles
Start from the existing convensaiion
Talking about preference. When the system makes a clarification or elicitation about user pref- erence, the simulated user would respond with the information about the target item.
Free-form
Astribuse- hased
chit-chut
question anowering
Existing conversation
Existing conversation
Chil-clat
Ask about attribute
Providing feedback. When the system recom- mends an item list, the simulated user would check each item and provide positive feedback if finding the target or negative feedback if not.
Chit-chat
Answer with preference
Invoke clarification
Recommend
Talk preference
Provide foedlback
Recommend
Recommend
Accept and complete
Refuse and complete
Completing the conversation. If one of the target items is recommended by the system the interaction reaches certain number of rounds, the simulated user would finish the conversation.
User Simulator
Conversational Recommenderion System
Figure 3: Our evaluation approach M. It is based on existing CRS datasets and has two settings: free-form chit-chat (left) and attribute-based question answering (right).
Specifically, we use the ground-trun items from existing datasets to construct realistic personas for simulated users. This is achieved by leveraging the text-davinci-00) (Ouyang et al., 2022) model provided in the OpenAT API, which demonstrates superior instruction following capacity as an eval- uator (Xu et al., 2023; Li al., 2023). To adapt text-davinci-003 for user simulation, we set its behaviors through manual instructions (see Ap- pendix C.3). In these instructions, we first fill the ground-trute items into the persona template and then define their behaviors using a set of manually crafted rules. At each turn, append the conver- sation the instruction input. When calling the API, we tax_tokens to 128, temperature ture 0, and leave other parameters at their default values.
excellent role-play capacities of LLMs (Fu et al., 2023). We take the ground-truth items as the user preference and use them to set up the persona of the LLM-based simulated user via instructions. After the interaction, we assess not only the accuracy by comparing predictions with the ground-truth items but also the explainability by querying an LLMM- based scorer with the generated explanations.
4.2 Interaction Forms
To make a comprehensive evaluation, we consider two types of interaction: attribute- question answering and free-form chit-chat.
The maximum number of interaction rounds is set to 5.
In the first type, the action of the system is re- stricted choosing one of the k pre-defined at- tributes ask the user making recommenda- tions. At each round, we first let the system decide on these A + 1 options, and then the user gives the template-basee based response: answering questions with the attributes of the target item or giving feed- back on recommendations. An example interaction round would be like: "System: Which genre do you like? User: Sci-fi and action."
4.4 Performance Measurement
We consider both subjective and objective metrics to measure the recommendation performance well as the user experience. For the objective met- ric, we use recall as stated in Section 2.2 to evalu- ate every recommendation action in the interaction process. For the subjective metric, following Chen et al. (2022), we use persuasiveness to assess the quality of explanations for the last recommendation action in the interaction process, aiming to evaluate whether the user can be persuaded to accept recom- mendations. The value range of this metric is (0. 1, 2). To reduce the need for humans, we propose an LLMM-based scorer that can automatically give the score through prompting. Specifically, we use the text-davinci i000 (Ouyang et al., 2022) model
In contrast, the second type does not impose any restrictions on the interaction, and both the system and user are free to take the initiative. An example interaction round would be like: "System: Do you have any specific genre in mind? User: I I'm looking for something action-packed with a lot of special effects.." "

---

Single-turn Usefulness
Multi-turn
tions with each user simulator. The interactions are based on the first round of conversations from 100 randomly sampled examples. Then, we employ an- other two annotators to make pairwise evaluations, where one is generated by our simulator and the other comes from DialoGPT or the dataset. We count a win for a method when both annotators agrec that its utterance is better; otherwise, we count tie. The Cohen's Kappa between annota- tors 0.73. Table demonstrates the results. We can see that our simulator significantly outperforms DialoGPT, especially terms of naturalness in the multi- turn setting, which demonstrates the strong language generation capability of LLMs. Further- more, the usefulness of our simulator is better than others, indicating that it can provide helpful infor- mation to cooperate with the system.
Settng
Naturalness
Naturalness
Usefulness
DialoGPT
13%
23%
11%
31%
IEvaLM
36%
43%
55%
38%
Tie
51%
34%
34%
31%
Human
10%
34%
28%
1EvaLM
39%
33%
35%
40%
Tic
51%
33%
48%
32%
Table 4: Performance comparison in terms of natural- ness and usefulness in the single-turn and multi-turn settings. Each value represents the percentage of pair- wise comparisons that the specific model wins ties.
provided in the OpenAI API as the scorer with the conversation, explanation, and scoring rules con- catenated as prompts (see Appendix C.4). Other parameters remain the same as the simulated user.
5 Evaluation Results
5.2 The Performance of CRS
In this section, we assess the quality of the user simulator and the performance of CRSs using our proposed evaluation approach.
In this part, we compare the performance of ex- isting CRSs and ChatGPT using different evalua- tion approaches. For ChatGPT, we use ChatGPT + eex-embbeddin--ada-00 due to its superior perfor- mance in traditional evaluation (see Appendix C.2).
5.1 The Quality of User Simulator
To evaluate the performance of CRSs in an inter- active setting, we construct user simulators based on ground-trutt items from existing datasets. The simulated users should cooperate with the system to find the target item, e.g., answer clarification questions and provide feedback on recommenda- tions. However, is not easy to directly evaluate the quality of user simulators.
5.2.1 Main Results
The evaluation results are presented Table and Table 6. Overall, most models demonstrate im- proved accuracy and explainability compared to the traditional approach. Among existing CRSs, the order of performance is UniCRS BARCOR KBRD, Both UniCRS and BARCOR utilize pre-trained models to enhance conversation abili- ties. Additionally, UniCRS incorporates KGs into prompts to enrich entity semantics for better under- standing user preferences. It indicates that existing CRSs have the ability to interact with users for bet- ter recommendations and user experience, which is an important aspect overlooked in the traditional eva evaluation.
Our solution is to make use of the annotated conversations in existing datasets. We first use the ground-truth items to set up the persona of the user simulator and then let them interact with the sys- tems played by humans. They are provided with the first round of annotated conversations to com- plete the rest. Then, conversations with the annotated ones for evalua- tion. Following Sekulić et al. (2022), we assess the naturalness and usefulness of the generated utter- ances in the settings single-turn and multi-turn free-form chit-chat. Naturalness means that the utterances are fluent and likely to be generated by humans, and usefulness means that the utterances are consistent with the user preference. We com- pare user simulator with fine-tuned version of DialoGPT and the original conversations in the REDIAL dataset.
compare the completed
For ChatGPT, there significant performance improvement in both Recall and Persuasiveness. and the Recall 10 value even surpassing the Re- call @ 25 or Recall 50 value of most CRSs on the two datasets. This indicates that ChatGPT has su- perior interaction abilities compared with existing CRSs and can provide high-quality and persuasive recommendations with sufficient information about the user preference. The results demonstrate the effectiveness of iEvaLM evaluating the accuracy and explainability of recommendations for CRSs, especially those developed with LLMs.
Specifically, we first invite five annotators to play the role of the system and engage in interac-

---

Model
KHRD
BARCOR
UnICKS
ChaiGPT
i
I
iEvaLMM M
ieval.M MM
ibxaLM
IEva.M
iEvaLM M
ieval.M
Evalustion Appeoah
Original
Original
Original
Original
(ntrr)
(free)
(arr]]
(Tres)
(atr)
(froc)
(amr)
(free)
0.039
0.035
0,191
R81
0.028
(439.355)
(+25.0%)
0.031
(+9.7%)
(49.758)
0.050
(46.0%)
(+114.0%)
0.037
(4416.2%)
(+204
0.198
0.201
0.317
0.536
ReiDial
0.169
(+16.0%)
(+17.2%)
(+18.25)
(+11.89) 1.8%
(+10.7%)
(+47.4%)
(+208 0200000
(+1 (+152.9%)
0.436
0,453
0.366
(+19.198)
(+23.8%)
(+14.8%)
(+25.55))
0.413
(125.95%)
(+45.8%%
0.131
0.234
0.400
0.231
(-43.3%)
(+1.3%%
(-15.4%%
(41.69%)
0.308
(-41.666)
(+1.9%)
0,310
(-3.5%)
(+29.0%)
0.293
01411
0.423
0.458
0,538
0.604
0.715
OpenDalKG
0:423
(-30.7%)
(+1.9%)
0.453
(-6.7%)
(+1.1%)
0.513
(-23.458)
(+4.9%)
0.539
(+12.15)
(+32.79)
0.509
0,482
0.609
0.492
(-23.4%)
(+3.5%)
(-5,5%)
(+3.998)
(-20.29) 258)
(+6.19)
Table 5: Performance of CRSs and ChatGPT under different evaluation approaches, where "attr" denotes attribute- based question answering and "free" denotes free-form chit-chat. "R@k" k" refers to Recall @ Since requiring too many items can sometimes be refused by ChatGPT, we only assess Recall@ and 10 for it, while Recall is marked as *_", Numbers marked with * indicate that the improvement is statistically significant compared with the rest methods (t-test with p-value < 0.05).
Evaluation Approuch
Partially persuasive
Highly persuasive
Model
ReDial
OpenDialKG
Method
Unpersuasive
Original
0.638
0.824
iEvaL 1%
5%
94%
KBRD
0.766
0,862
Human
4%
7%
891
iEvaLM
(+20.1%)
(+4.6%)
Table 7: The score distribution of persuasiveness ("un- persuasive" for O, "partially persuasive" for 1, and "highly persuasive" for 2) from our I.LM-based scorer and human on a random selection of 100 examples from the REDIAL dataset.
Original
0.667
1.149
BARCOR
0.795
1,211
iEvaLM
(+19.2%)
(+5.4%)
Original
0.685
1.128
UniCRS
1.015
1.314
iFvaLM
(+48.2%%
(+16.5%)
Original
0.787
1.221
more holistic assessment of CRSs, providing valu- able insights into their strengths and weaknesses across different types of interactions.
ChanGPT
1
1.513*
iEvaLM
(+69.15)
(+23.9%)
Table 6: The persuasiveness of explanations. We only consider the setting of free-form chit-chat in iEvaLM. Numbers marked with indicate that the improvement statistically significant compared with the rest methods (t-test with p-value < 0.05).
5.2.2 The Reliability of Evaluation
Recall that LLMs are utilized in the user simulation and performance measurement parts of iEvaLM as alternatives for humans in Section 4. Considering that the generation of LLMs can be unstable, in this part, we conduct experiments to assess the reliability of the evaluation results compared with using human annotators.
Comparing the two interaction settings, Chat- GPT has demonstrated greater potential general-purpose CRS. Existing CRSs perform much worse in the setting of attribute-based ques- tion answering than in the traditional setting on the OPENDIALKG dataset. One possible reason is that they are trained on datasets with natural lan- guage conversations, which is inconsistent with the setting of attribute-based question answering. In contrast, ChatGPT performs much better in both settings on the two datasets, since it has been spe- cially trained on conversational data. The results indicate the limitations of the traditional evaluation, which focuses only on single conversation sce- nario, while our evaluation approach allows for a
First, recall that we introduce the subjective met- ric persuasiveness for evaluating explanations in Section 4.4. This metric usually requires human evaluation, and we propose an LLM-bbased scorer as an alternative. Here we evaluate the reliability of our LLM-based scorer by comparing with human annotators. We randomly sample 100 examples with the explanations generated by ChatGPT and ask our scorer and two annotators rate them sep- arately with the same instruction (see Appendix C). The Cohen's 's Kappa between annotators is 0.83. Ta- ble 7 demonstrates that the two score distributions are similar, indicating the reliability of our LLM-

---

Evaluation Approach
KBRD
BARCOR
UniCRS
ChaiGPT
ity of our evaluation approach.
Recall/ 10
0.180
0.210
0.330
0.460
Overall, our work contributes to the understand- ing and evaluation of LLMs such as ChatGPT for
IEvaLM
Persuasiveness
0.810
0.860
1,050
1.330
Recall 1o
0.210
0,250
0.370
0.560
Human
Persuasiveness
0.870
0.930
1,120
1,370
conversational recommendation, paving the way for further research in this field in the era of LLMs.
Table 8: The evaluation results using simulated and real users on a random selection of 100 examples from the REDIAL dataset.
Limitations
A major limitation of this work the design of prompts for ChatGPT and LLM-based user simu- lators. We manually write several prompt candi- dates and select the one with the best performance on some representative examples due to the cost of calling model APIs. More effective prompting strategies like chain-of-thougle can be explored for better performance, and the robustness of the eval- uation framework different prompts remains be assessed.
based scorer as a substitute for human evaluators. Then, since we propose an LLM-based user sim- ulator as a replacement for humans to interact with CRSs, we examine the correlation between the values of metrics when using real vs. simulated users. Following Section 4.3, both real and sim- ulated users receive the same instruction (see Ap- pendix C) to establish their personas based on the ground-truth items. Each user interact with different CRSs for five rounds. We randomly select 100 instances and employ five annotators and our user simulator to engage in free-form chit-chat with diffcrent CRSs. The results are shown in Table 8.
In addition, our evaluation framework primar- ily focuses on the accuracy and explainability of recommendations, but it may not fully capture po- tential issues related to fairness, bias, or privacy concerns. Future work should explore ways in- corporate these aspects into the evaluation process to ensure the responsible deployment of CRSs.
We can see that the ranking obtained from our user simulator is consistent with that of real users, and the absolute scores are also comparable. It suggests that our 4LM-basee user simulator is capable of providing convincing evaluation results and serves as reliable alternative to human evaluators.
A Acknowledgements
This work was partially supported by National Nat- ural Science Foundation of China under Grant No. 62222215 and 72222022, Beijing Natural Science Foundation under Grant No. 4222027, and the Outstanding Innovative Talents Cultivation Funded Programs 2022 of Renmin University of China. Xin Zhao is the corresponding author.
6 Conclusion
In this paper, we systematically examine the capa- bility of ChatGPT for conversational recommenda- tion on existing benchmark datasets and propose an alternative evaluation approach, iEvaLM. First, we show that the performance of ChatGPT was unsatisfactory. Through analysis of failure cases, the root cause is the existing evaluation protocol, which overly emphasizes the fitting of ground-truth items based on conversation context. To address this issue, we propose interactive evaluation ap- proach using LLM-based user simulators.
References
Jafar Afzali, Aleksander Mark Drzewiecki. Krisz- tian Balog, and Shuo Zhang. 2023. Usersim- ers: A user simulation toolkit for evaluating versational recommender systems. arXiv preprint arXiv:2301.05544.
Krisztian Balog and ChengXiang Zhai 2023. User simulation for evaluating information access systems. arXiv prepriat arXiv:2306.08533
Through experiments with this new approach, we have the following findings: (1) ChatGPT is powerful and becomes much better in our evalua- tion than the currently leading CRSs in both accu- racy and explainability; (2) Existing CRSs also get improved from the interaction, which is impor- tant aspect overlooked by the traditional evaluation; and (3) ChatGPT shows great potential as a general- purpose CRS under different settings and datasets. We also demonstrate the effectiveness and reliabil-
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.00 302004023......
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Re-

---

cent advances and new frontiers. Acm Sigkdd Ex- plorations Newsletter, 19(2):25-35.
Wenqiang Lei. Xiangnan He. Yisong Miao, Qingyun Wu, Richang Hong. Min- Yen Kan, and Tat-Seng Chua. 2020. cedimatioo-actioo-rellll deep interaction between conversational and recom- mender systems. In Proceedings of the 1 13th Interna- tional Conference on Web Search and Data Mining. pages 304-312.
Towards
Qibin Chen, Junyang Lin, Yichang Zhang. Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019. To- wards knowledge-bere -baed recommender dialog system. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL pages 1803-1813.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising eqquence-t--equeeeee
Xu Chen, Yongfeng Zhang, and Ji-R Ji-Rong Wen. 2022. Measuring" why" in recommender systems: com- prehensive survey on the evaluation of explainable recommendation. arXiv preprint arXiv:2202 06466.
pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the A Association for Computational Linguistics, pages
Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu. 2023. Uncovering chatgpt's capa- bilities in recommender systems. arXiv preprint arXiv:2305.02/82.
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommenda- tions. Advances in neural information processing systems, 31,
Yang Deng. Wenxuan Zhang. Weiwen Xu, Wenqiang Lei, Tat-Seng Chua, and Wai Lam. 2023. A unified multi-task learning framework for multi-goal conver- sational recommender systems. ACM Transactions on Information Systems, 4 -33:
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpacaeval: An au- tomatic evaluator of cestructiio-followiinn models. https://gi thub. wwtp:/ttotttte/e.cett -aba hat.lllalaaaaaaaaaaaaea elaac__eval.
Yao Fu, Hao Peng. Tushar Khot, and Mirella Lapata. 2023. Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.100 10142.
Seungwhan Moon, Pararth Shah, Anuj Kumar, and Ra- jen Subba. 2019. Opendialkg: Explainable conver- sational reasoning with attention -bsed walks over knowledge graphs. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 845-854.
Chongming Gao, Wengiang Lei. Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and challenges in conversational recommender systems: A survey. Al Open, 2: 1206
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak. Jong Wook Kim, Chris Hallacy, et al. 2022. Text and code embeddings by contrastive pre- training. arXiv preprint arXiv:2201 10005.
Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neu- ral approaches to conversational ai. In The s1 In- ternational ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1371 1374.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Caroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 13:27730-27744
Shuyu Guo, Shuo Zhang, Weiwei Sun, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. 2023, Towards explainable conversational recommender systems. arXiv preprint arXiv:2305.1836.
Gustavo Penha and Claudia Hauff. 2020. What does bert know about books, movies and music? probing bert for conversational recommendation. In Proceed- ings of the I 14th ACM Conference on Recommender Systems, pages 388-397.
Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021. A survey on conversational recom- mender systems. ACM Computing Surveys (CSUR). 54(5):1-36.
Wang Cheng Kang, Jianmo Ni, Nikhil Mehta, Mah- eswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. 2023. Do Iims understand user preferences? evaluating Ilms on user rating pre- diction. arXiv preprint 2305.023050
Chengwei Qin. Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpos natural language process- ing task solver? arXiv preprint arXiv:2302.0642 06476.
Ivan Sekulić. Mohammad Aliannejadi, and Fabio Crestani. 2022. Evaluating mixed-initiative conversa- tional search systems via user simulation. In Proceed- ings of the Fifteemth ACM International Conference on Web Search and Data Mining, pages $88-896.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of nla.L-IHLLT volume I, page 2.

---

Quan Tu, Shen Gao. Yanran Li, Jianwei Cui, Bin Wang. and Rui Yan. 2022. Conversational recommendation via hierarchical information modeling. In Proceed- ings of the 45th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, pages 2201-2205
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William B Dolan. 2020. Dialogpt: Large- scale generative pre-training for conversational re- sponse generation. In Proceedings of the 58th An- nual Meeting of the Association for Computational System Demonstrations, pages 270-278.
Linguistics:
Lingzi Wang. Huang Hu, Lei Sha, Can Xu, Daxin Jiang, and Kam-Fai Wong. 2022a. Recindial: A uni- fied framework for conversational recommendation with pretrained language models. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Juint Conference on Natural Language Processing, pages 489-500.
Yongfeng Zhang. Xu Chen, Qingyao Ai, Liu Yang, and W Bruce Croft. 2018. Towards conversational search and recommendation: System ask. user respond. In Proceedings of the 27th aem international conference on information and knowledge management, pages 177-186.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023a. A survey of large language models. arXiv preprint arXiv:2303. 230.11333..
Ting-Chun Wang, Shang-Yu Yu Su, and Yun-Nung Chen. 2022b. Barcor: Towards a unified framework for con- versational recommendation systems. arXiv preprint 222.:
Xiaolei Wang, Kun Zhou, Xinyu Tang, Wayne Xin Zhao, Fan Pan, Zhao Cao. and Ji-Rong Wen. 2023. Improving conversational recommendation systems via counterfactual data simulation. In Proceedings of the 29th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023, pages 2398- 2408. ACM.
Weixiang Zhao, Yanyan Zhao, Xin Lu, Shilong Wang. Yanpeng Tong, and Bing Qin. 2023b. Is chat- gpt equipped with emotional dialogue capabilities? arXiv preprint arXiv: 095882.
Zhipeng Zhao, Kun Zhou, Xiaolci Wang, Wayne Xin Zhao, Fan Pan, Zhao Cao, and Ji-Rong Wen. 2023c. Alleviating the long-tail problem in conversational recommender systems. In Proceedings of the 17th ACM Conference Recommender Systems, pages 374-385.
Xiaolei Wang. Kun Zhou, Ji-Rong Wen, and Wayne Xin Zhao. 2022c. Towards unified conversational rec-
ommender systems via knowledge-enhancedd prompt learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, pages 1929-1937.
Jinfeng Zhou, Bo Wang. Ruifang He, and Yuexian Hou. 2021a. Crfr: Improving conversational recommender systems via flexible fragments reasoning on knowl- edge graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 4324 4334.
Le Wu, Xiangnan He, Xiang Wang. Kun Zhang, and Meng Wang. 2022. A survey on eecuracy-orientee neural recommendation: From collaborative filtering to information-rich recommendation. IEEE Transac- tions Knowledge and Data Engineering.
Jinfeng Zhou, Bo Wang, Minlie Huang. Dongming Zhao, Kun Huang, Ruifang He, and Yuexian Hou. 2022. Aligning recommendation and conversation via dual imitation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 549-561, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the tool manipulation capability of open-source large lan- guage models. arXiv preprint 2rxiv:2305.16300
Bowen Yang, Cong Han, Yu Li. Lei Zuo, and Zhou Yu. 2022. Improving conversational recommenda- tion systems' quality with context-aware item meta- information. In Findings of the Association for Com- putational Linguistics: NAACL 2022, pages 38 48.
Kun Zhou, Xiaolei Wang, Yuanhang Zhou, Chenzhan Shang, Yuan Cheng, Wayne Xin Zhao, Yaliang Li, and Ji-Rong Wen. 2021b. Crslab: An open source toolkit for building conversational recommender sys- tem. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing: System Demonstrations, pages 185-193.
Shuo Zhang and Krisztian Balog. 2020. Evaluating con- versational recommender systems via user simulation. In Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining, pages 1512-1520.
Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020. Improv- ing conversational recommender systems via knowl- edge graph based semantic fusion. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1006- 1014.
Xiaoyu Zhang, Xin Xin, Dongdong Li, Wenxuan Liu, Pengjie Ren, Zhumin Chen, Jun Ma, and Zhaochun Ren. 2023. Variational reasoning over incomplete knowledge graphs for conversational recommenda- tion. In Proceedings of the Sixteenth ACM Interna- tional Conference on Web Search and Data Mining, pages 231-239.

---

Recalle:10
B Related Work
0.65
In this section, we summarize the related work from the following perspectives.
0.55
0.45
B.1 Conversational Recommender System
0.35
The fields of conversation intelligence (Chen et al.. 2017; Gao et al., 2018) and recommendation systems (Wu et al., 2022) have seen significant progress in recent years. One promising develop- ment is the integration of these two fields, leading to the emergence of conversational recommender systems (CRSs) (Jannach et al., 2021; Gao et al.. 2021). CRSs provide recommendations to users through conversational interactions, which has the potential to significantly improve the user experi- ence.
free
0.05
Figure 4: The performance of ChatGPT with different interaction rounds under the setting of attribute-based question answering (attr) and free-form chit-chat (free) on the REDIAL dataset.
A The Influence of the Number of Interaction Rounds in iEvaLM
One popular approach (Lei et al., 2020; Tu al., 2022) assumes that interactions with users primar- ily take the form of question answering, where users are asked about their preferences for items and their attributes. The goal is to learn an optimal interaction strategy that captures user preferences and provides accurate recommendations in as few turns possible. However, this approach often re- lies
Interacting with the user for multiple rounds typi- cally leads to more information and improved rec- ommendation accuracy. However, users have lim- ited patience and may leave the interaction when they become exhausted. important to investi- gate the relationship between the number of inter- action rounds and performance. Following the set- ting in our approach, the interaction between Chat- GPT and users is start from the observed human- annotated conversation in each dataset example, and set the maximum interaction rounds to val- from (1, 2, 3, 4, 5), in order evaluate the changes in recommendation accuracy.
templates and does not explic- itly model the language aspect of CRSs. Another approach (Zhang al., 2023; Zhou al., 2022) fo- cuses on engaging users more free-form natural language conversations, such as chit-chat. The aim is to capture user preferences from the conversa- tion context and generate recommendations using persuasive responses.
Figure shows the results of Recall@ 10 on the REDIAL dataset. attribute-based question an- swering, the performance keeps increasing and reaches saturation at round 4. This observation aligns with our conversation setting, since the RE- DIAL dataset only has three attributes inquire about. In free-form chit-chat, the performance curve is steep between rounds 1 and 3, while it is relatively flat between rounds 3 and 5. This pat- tern may attributed to insufficient information in the initial round and marginal information in the last rounds. Since the user will gradually get exhausted with the progress of the interaction, how to optimize the conversation strategy remains to be further studied.
Our work belongs the second category. In this work, we systematically evaluate the performance of large language models (LLMs) like ChatGPT for conversational recommendation on large-scale datasets.
B.2 Language Models for Conversational Recommendation
There have been recent studies on how to integrate language models (LMs) into CRSs. One notable investigation by Penha and Hauff (2020) evalu- ates the performance of the pre-trained language model (PLM) BERT (Kenton and Toutanova, 2019) in conversational recommendation. Other stud- ies (Wang al., 2022c; Yang al., 2022; Deng et al., 2023) primarily utilize PLMs the foun- dation to build unified CRSs, capable of perform- ing various tasks using single model instcad multiple components. However, the current ap-

---

proaches are mainly confined to small-size LMs like BERT (Kenton and Toutanova, 2019) and Di- aloGPT (Zhang et al., 2020).
C
Prompts Used in the Paper
C.1 Prompts for ChatGPT in the Traditional Evaluation
In this paper, we focus on the evaluation of CRSs developed with not only PLMs but also LLMs and propose a new evaluation approach iEvaLM.
We the following prompts for zero-shot prompt- ing in section 3.1.
B.3 Evaluation and User Simulation
ReDial
The evaluation of CRSs remains area that has not been thoroughly explored in existing literature. Previous studies have primarily focused turn- level evaluation (Chen al., 2019), where the sys- tem output of a single turn is compared against ground-trutt truth labels for two major tasks: conver- sation and recommendation. Some researchers have also adopted conversation-Ievel evaluation to assess conversation strategies (Lei et al., 2020; Zhang et al., 2018; Balog and Zhai, 2023; Afzali et al., 2023). In such cases, simulation is often employed
Recommend 10 items that are consistent with user preference. The recommendation list can contain items that the dialog mentioned before. The format of the recommendation list is: title (year). mention anything other than title items your recommendation
OpenDialKG
Recommend 10 items that are consistent with user preference. The recommendation list can contain items that the dialog mentioned before. The format of the recommendation list is: title. Don't t mention anything other than title of items your recommendation list.
substitute for human evalua- tion. These approaches typically involve collecting real user interaction history (Lei et al., 2020) or reviews (Zhang et al., 2018) to represent the pref- erences of simulated users. Zhou al. (2021b) de- velop spen-source toolkit called CRSLab, which provides extensive and standard evaluation proto- cols. However, due to the intricate and interactive nature of conversational recommendation, the eval- uation often constrained by pre-defined conver- sation flows or template-base utterances. Conse- quently, this limitation hinders the comprehensive assessment of the practical utility CRSs.
C.2 Prompts for ChatGPT in iEvaLM
We use the following prompts for ChatGPT in our new evaluation approach.
C.2.1 Recommendation
Free-Form Chit-Chat.
In our work, we propose an interactive evalua- tion approach iEvaLMM with LLM-based user sim- ulators, which has strong instruction-following ability and can flexibly adapt to different CRSs based on the instruction without further tuning.
ReDiall
You are recommender chatting with user to provide recommendation. You must follow the instructions below during chat. If you do not have enough information about user preference, should ask the user for his preference.
If you have enough information about pref- erence, you give recommendation. The ommendation list must contain 10 items that are consistent with user preference. The rec- ommendation list can contain items that the dialog mentioned before. The format of the recommendation list is: no. title (year). Don't mention anything other than the title of items in your recommendation list.
OienIi

---

are a recommender chatting with the user provide recommendation. You must follow the instructions below during chat. you do not have enough information about user preference, you should ask the user for his preference.
Free-Form Chit-Chat, "()" refers to the item labels of each example in the datasets.
You seeker chatting with recommender for recommendation. Your target items: {)- You follow the instructions below during chat.
recommender recommends (). you should ac- cept.
you have enough information about pref- erence, you give recommendation. The rec- ommendation list contain items that are consistent with user preference. The rec- ommendation can contain items that the dialog mentioned before. The format of the recommendation list is: no. title. Don't men- tion anything other than the title of items in your recommendation list.
If the recommender recommends other items, you should refuse them and provide the information about (). You should never directly tell the target item title. If the recommender asks for your preference, you should provide the information about (). You should never directly the target item title.
Attribute-Based Question Answering.
Attribute-Ban -aaeed Question Answering. "()" refers to the options that have been selected.
When the recommended item list contains least one of the target items:
ReDial
That's perfect, thank you!
To recommend me items that will accept, you can choose one of the following options.
When the recommended item list does not contain any target item:
A: usk my preference for genre
B: ask my preference for actor
don't like them.
C: ask my preference for director can directly give recommendations You have selected (}. do not repeat them. Please enter the option character.
When the system asks about the preference over pre-defined attributes, we use the at- tributes of target items as the answer they exist, otherwise:
OpenDialKG
Sorry, information about this.
recommend items will accept, you can choose one of following options.
C.4 Prompts for the LLM-based Scorer in iEvaLM
A: ask my preference for genre
We the following prompts for next-davinci-00. to score the persuasive- ness of explanations, "I "()" refers the item labels of each example the datasets.
B; ask my preference for actor
C: ask my preference for director
D: ask my preference for writer
E: I can directly give recommendations
You have selected (), do repeat them. Please enter the option character.
Does the explanation make you want accept the recommendation? Please give your score.
mention one [(]]. give 2.
C.2.2 Explainability
Elsc you think recommended items are worse than [(}), give
Please explain your last time of recommen- dation.
Else you think recommended items are comparable
to (() according to the explanation, give 1.
C.3 Prompts for the User Simulator in iEvaLM
Flse you think recommended items are better than
[ill according to the explanation, give 2.
Only the numher.
use the following prompts for text-davinci-003 to play the role the user during interaction.
We


Reviews:
[{"review_id": "RjvhbPCKWr", "reviewer": "Reviewer_UZ7K", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your valuable feedback on our paper. We have carefully considered your comments and concerns, and we provide our response below to address each of them.\n\n**1. Since the user simulator is based on text-davince-003, will iEvaLM be biased towards ChatGPT during evaluation?**\n\nWe appreciate your concern regarding bias in the evaluation process. However, we would like to clarify that the user simulator is not designed to score CRSs; its purpose is solely to interact with the system, while performance is assessed using metrics such as Recall based on the interaction. Therefore, selecting text-davinci-003 as the user simulator will not introduce bias towards ChatGPT during the evaluation.\n\n**2. The robustness of the evaluation framework is unclear. Minor changes in prompts could lead to significant differences in results.**\n\nThanks for the insightful comment! Robustness is a widespread concern for the evaluation of LLMs since prompts for LLMs exist in the space of natural language. We follow common practice (e.g., Open LLM Leaderboard by HuggingFace) to report the evaluation results using a standard prompt for all models. However, in consideration of robustness, we have also tested some prompts with minor changes as the reviewer suggested.\n\n\\\\begin{array} {|l|l|}\n\\\\hline\n\\\\textrm{Prompt No.} & \\\\textrm{Prompt for the user simulator} \\\\\\\\\n\\\\hline\n1 (\\\\textrm{we show it in the paper}) & \\\\textrm{You are a seeker chatting with a recommender for recommendation. Your target items: \\\\{\\\\textit{target items}\\\\}. You must follow the instructions below during chat.}\n\\\\newline\n& \\\\textrm{If the recommender recommends \\\\{\\\\textit{target items}\\\\}, you should accept.}\n\\\\newline\n& \\\\textrm{If the recommender recommends other items, you should refuse them and provide the information about \\\\{\\\\textit{target items}\\\\}. You should never directly tell the target item title.}\n\\\\newline\n& \\\\textrm{If the recommender asks for your preference, you should provide the information about \\\\{\\\\textit{target items}\\\\}. You should never directly tell the target item title.} \\\\\\\\\n\\\\hline\n2 (\\\\textrm{add a number before each rule}) & \\\\textrm{You are a seeker chatting with a recommender for recommendation. Your target items: \\\\{\\\\textit{target items}\\\\}. You must follow the instructions below during chat.}\n\\\\newline\n& \\\\textrm{1. If the recommender recommends \\\\{\\\\textit{target items}\\\\}, you should accept.}\n\\\\newline\n& \\\\textrm{2. If the recommender recommends other items, you should refuse them and provide the information about \\\\{\\\\textit{target items}\\\\}. You should never directly tell the target item title.}\n\\\\newline\n& \\\\textrm{3. If the recommender asks for your preference, you should provide the information about \\\\{\\\\textit{target items}\\\\}. You should never directly tell the target item title.} \\\\\\\\\n\\\\hline\n3 (\\\\textrm{paraphrase the first sentence}) & \\\\textrm{You are inquiring with a recommender for recommendations. Your target items: \\\\{\\\\textit{target items}\\\\}. You must follow the instructions below during chat.}\n\\\\newline\n& \\\\textrm{If the recommender recommends \\\\{\\\\textit{target items}\\\\}, you should accept.}\n\\\\newline\n& \\\\textrm{If the recommender recommends other items, you should refuse them and provide the information about \\\\{\\\\textit{target items}\\\\}. You should never directly tell the target item title.}\n\\\\newline\n& \\\\textrm{If the recommender asks for your preference, you should provide the information about \\\\{\\\\textit{target items}\\\\}. You should never directly tell the target item title.} \\\\\\\\\n\\\\hline\n4 (\\\\textrm{remove ``You must''}) & \\\\textrm{You are a seeker chatting with a recommender for recommendation. Your target items: \\\\{\\\\textit{target items}\\\\}. Follow the instructions below during chat.}\n\\\\newline\n& \\\\textrm{If the recommender recommends \\\\{\\\\textit{target items}\\\\}, you should accept.}\n\\\\newline\n& \\\\textrm{If the recommender recommends other items, you should refuse them and provide the information about \\\\{\\\\textit{target items}\\\\}. You should never directly tell the target item title.}\n\\\\newline\n& \\\\textrm{If the recommender asks for your preference, you should provide the information about \\\\{\\\\textit{target items}\\\\}. You should never directly tell the target item title.} \\\\\\\\\n\\\\hline\n\\\\end{array}\n\nAs shown below, there are no significant differences in the evaluation results, and the ranking of models remains consistent across different prompts.\n\n- OpenDialKG (Since requiring too many items at once can sometimes be refused by ChatGPT, following our paper, we only assess Recall@1 and Recall@10 for it)\n\\\\begin{array} {|c|c|c|c|c|c|}\n\\\\hline\n\\\\textrm{Metric} & \\\\textrm{Prompt No.} & \\\\textrm{KBRD} & \\\\textrm{BARCOR} & \\\\textrm{UniCRS} & \\\\textrm{ChatGPT} \\\\\\\\\n\\\\hline\n\\\\textrm{Recall@1} & 1 & 0.234 & 0.314 & 0.314 & 0.400\n\\\\newline\n& 2 & 0.232 & 0.314 & 0.313 & 0.410\n\\\\newline\n& 3 & 0.232 & 0.313 & 0.321 & 0.414\n\\\\newline\n& 4 & 0.234 & 0.313 & 0.320 & 0.409 \\\\\\\\\n\\\\hline\n\\\\textrm{Recall@10} & 1 & 0.431 & 0.458 & 0.538 & 0.715\n\\\\newline\n& 2 & 0.428 & 0.458 & 0.536 & 0.716\n\\\\newline\n& 3 & 0.434 & 0.461 & 0.547 & 0.727\n\\\\newline\n& 4 & 0.433 & 0.458 & 0.545 & 0.726 \\\\\\\\\n\\\\hline\n\\\\textrm{Recall@50} & 1 & 0.509 & 0.530 & 0.609 & -\n\\\\newline\n& 2 & 0.508 & 0.528 & 0.611 & -\n\\\\newline\n& 3 & 0.517 & 0.529 & 0.618 & -\n\\\\newline\n& 4 & 0.511 & 0.527 & 0.617 & - \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n- ReDial (Since requiring too many items at once can sometimes be refused by ChatGPT, following our paper, we only assess Recall@1 and Recall@10 for it)\n\\\\begin{array} {|c|c|c|c|c|c|}\n\\\\hline\n\\\\textrm{Metric} & \\\\textrm{Prompt No.} & \\\\textrm{KBRD} & \\\\textrm{BARCOR} & \\\\textrm{UniCRS} & \\\\textrm{ChatGPT} \\\\\\\\\n\\\\hline\n\\\\textrm{Recall@1} & 1 & 0.035 & 0.034 & 0.107 & 0.146\n\\\\newline\n& 2 & 0.032 & 0.033 & 0.112 & 0.144\n\\\\newline\n& 3 & 0.037 & 0.034 & 0.121 & 0.150\n\\\\newline\n& 4 & 0.035 & 0.035 & 0.124 & 0.149 \\\\\\\\\n\\\\hline\n\\\\textrm{Recall@10} & 1 & 0.198 & 0.190 & 0.317 & 0.440\n\\\\newline\n& 2 & 0.198 & 0.188 & 0.319 & 0.430\n\\\\newline\n& 3 & 0.207 & 0.185 & 0.325 & 0.450\n\\\\newline\n& 4 & 0.206 & 0.188 & 0.322 & 0.445 \\\\\\\\\n\\\\hline\n\\\\textrm{Recall@50} & 1 & 0.453 & 0.467 & 0.602 & -\n\\\\newline\n& 2 & 0.455 & 0.465 & 0.600 & -\n\\\\newline\n& 3 & 0.460 & 0.462 & 0.606 & -\n\\\\newline\n& 4 & 0.470 & 0.468 & 0.610 & - \\\\\\\\\n\\\\hline\n\\\\end{array} \n\nAdditionally, you may notice that our paper has limited page space, which necessitates focusing on the most important parts. Given that the robustness of evaluation regarding LLMs is still an open problem without standard and reliable evaluation approaches, we do not include this aspect in our submission. We promise to add a discussion about it in the revised version.\n\n**3. Even though the code is open-sourced, there are concerns about the transparency of the evaluation framework given that text-davince-003 is a closed API.**\n\nWe appreciate your concern regarding the transparency of the evaluation framework. We chose text-davinci-003 to implement the user simulator in our work due to its powerful capacity for human alignment. As shown in the evaluation of model-based evaluators from AlpacaEval [1], text-davinci-003 performs much better than ChatGPT in agreement with human evaluators, only falling behind Claude and GPT-4. Since we do not have access to Claude and GPT-4 (as they require application at the time of submission), text-davinci-003 is the best model we can choose for the user simulator. Such a powerful capacity for human alignment is important for simulating realistic dialogues to evaluate the performance of CRSs in close-to-real scenarios.\n\nIn addition, the user simulator is seamlessly integrated with public CRS datasets. The preference of the user simulator is set to the ground-truth items from datasets, which are sourced from real human annotators and publicly available.\n\nWhile it is true that text-davinci-003 is a closed API, we would like to clarify that the evaluation framework itself is independent of any specific API and can be adapted to any model with capable conversational abilities. Here, we report the evaluation results obtained by using the open-sourced model Vicuna-7B to implement the user simulator. It can be observed that the ranking of models remains consistent between text-davinci-003 and Vicuna-7B.\n\n- ReDial (Since requiring too many items at once can sometimes be refused by ChatGPT, following our paper, we only assess Recall@1 and Recall@10 for it)\n\\\\begin{array} {|c|l|c|c|c|c|}\n\\\\hline\n\\\\textrm{User simulator} & \\\\textrm{Metric} & \\\\textrm{KBRD} & \\\\textrm{BARCOR} & \\\\textrm{UniCRS} & \\\\textrm{ChatGPT} \\\\\\\\\n\\\\hline\n& \\\\textrm{Recall@1} & 0.035 & 0.034 & 0.107 & 0.146\n\\\\newline\n\\\\textrm{text-davinci-003} & \\\\textrm{Recall@10} & 0.198 & 0.190 & 0.317 & 0.440\n\\\\newline\n& \\\\textrm{Recall@50} & 0.453 & 0.467 & 0.602 & - \\\\\\\\\n\\\\hline\n& \\\\textrm{Recall@1} & 0.023 & 0.024 & 0.097 & 0.117\n\\\\newline\n\\\\textrm{Vicuna} & \\\\textrm{Recall@10} & 0.167 & 0.166 & 0.298 & 0.419\n\\\\newline\n& \\\\textrm{Recall@50} & 0.401 & 0.425 & 0.578 & - \\\\\\\\\n\\\\hline\n\\\\end{array}\n\n- OpenDialKG (Since requiring too many items at once can sometimes be refused by ChatGPT, following our paper, we only assess Recall@1 and Recall@10 for it)\n\\\\begin{array} {|c|l|c|c|c|c|}\n\\\\hline\n\\\\textrm{User simulator} & \\\\textrm{Metric} & \\\\textrm{KBRD} & \\\\textrm{BARCOR} & \\\\textrm{UniCRS} & \\\\textrm{ChatGPT} \\\\\\\\\n\\\\hline\n& \\\\textrm{Recall@1} & 0.234 & 0.314 & 0.314 & 0.400\n\\\\newline\n\\\\textrm{text-davinci-003} & \\\\textrm{Recall@10} & 0.431 & 0.458 & 0.538 & 0.715\n\\\\newline\n& \\\\textrm{Recall@50} & 0.509 & 0.530 & 0.609 & - \\\\\\\\\n\\\\hline\n& \\\\textrm{Recall@1} & 0.228 & 0.302 & 0.309 & 0.383\n\\\\newline\n\\\\textrm{Vicuna} & \\\\textrm{Recall@10} & 0.425 & 0.441 & 0.530 & 0.689\n\\\\newline\n& \\\\textrm{Recall@50} & 0.495 & 0.527 & 0.592 & - \\\\\\\\\n\\\\hline\n\\\\end{array}\n\nFurthermore, we have made our best efforts to ensure the transparency and reproducibility of our work.\n\n- We set the temperature to 0 for text-davinci-003, as suggested by OpenAI, to make the generation mostly deterministic. \n- We have open-sourced the code, which includes the implementation of the evaluation framework. Researchers can utilize our code, replace text-davinci-003 with open-sourced conversation models, and replicate the evaluation process, thereby promoting transparency and enabling further research in this domain.\n\n[1] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https://github.com/tatsu-lab/alpaca_eval#evaluators\n\n**4. Reproducibility: 2: would be hard pressed to reproduce the results.**\n\nWe would like to clarify that, as stated in the response to Q3, we have open-sourced all of our code and data. The evaluation framework itself is independent of any specific API or model. Furthermore, we have followed OpenAI's suggestion to set the temperature to 0 for all the provided APIs, which makes the generation mostly deterministic. Therefore, you can easily reproduce the results following the instructions for running the code. Reviewers P19e and ANzj have also given high scores for the reproducibility of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Strengths:**\n\n1. This paper addresses an important problem in a timely manner. Evaluation is vital in research, especially at a time when LLM is widely used in real-life applications. Traditional evaluation protocols, like matching with ground-truth items, cannot effectively measure the performance of current CRSs.\n2. The proposed approach of adopting LLMs as user simulators for evaluating CRSs is intriguing.\n3. This paper conducts a systematic analysis of ChatGPT’s capabilities in conversational recommendation and highlights its limitations under traditional evaluation methods.", "reasons_to_reject": "**Weaknesses：**\n\n1. Since the user simulator is based on text-davince-003, will iEvaLM be biased towards ChatGPT during evaluation?\n2. The robustness of the evaluation framework is unclear. Minor changes in prompts could lead to significant differences in results.\n3. Even though the code is open-sourced, there are concerns about the transparency of the evaluation framework given that text-davince-003 is a closed API.", "questions_to_authors": "", "paper_topic_and_main_contributions": "To address the limitations of existing evaluation protocols, which might overemphasize matching with ground-truth items and overlook the interactive nature of CRSs, this paper introduces iEvaLM, which leverages LLM-based user simulators.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3VxG7DyIb8", "reviewer": "Reviewer_P19e", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. In response to your comments, we have carefully considered each point and addressed them as follows.\n\n**1. Some experimental designs are confusing: GPT-3.5-turbo is used for CRS while text-davinci-003 is used for user simulator.**\n\nThanks for the comment! For CRS, we use gpt-3.5-turbo since it has acquired a wealth of world knowledge during pre-training and is also specially optimized for conversation, which is expected to be an excellent LLM-based CRS.\n\nFor the user simulator, we use text-davinci-003, as it demonstrates a capable ability to follow instructions during multi-turn dialogue, which is what we need for a user simulator with expected behaviors set in Section 4.3. In contrast, gpt-3.5-turbo is specially trained for dialogue, which tends to generate responses more influenced by the dialogue history rather than adhering to the provided instructions.\n\nAs shown in the example below, text-davinci-003 perfectly follows the instruction to refuse the recommended items since they are not the target ones in the instruction, while gpt-3.5-turbo starts to chit-chat with the system without following the instruction.\n\n\\\\begin{array} {|l|l|}\n\\\\hline\n\\\\textrm{\\textbf{Instruction}} & \\\\textrm{You are a seeker chatting with a recommender for recommendation. Your target items: \"Click (2006)\", \"It (2017)\". You must follow the instructions below during chat.} \n\\\\newline\n& \\\\textrm{If the recommender recommends \"Click (2006)\", \"It (2017)\", you should accept.}\n\\\\newline\n& \\\\textrm{If the recommender recommends other items, you should refuse them and provide the information about \"Click (2006)\", \"It (2017)\". You should never directly tell the target item title.}\n\\\\newline\n& \\\\textrm{If the recommender asks for your preference, you should provide the information about \"Click (2006)\", \"It (2017)\". You should never directly tell the target item title.} \\\\\\\\\n\\\\hline\n\\\\textrm{\\textbf{Recommender}} & \\\\textrm{Tell me what would you like to watch?} \\\\\\\\\n\\\\hline\n\\\\textrm{\\textbf{Seeker}} & \\\\textrm{Easy A (2010)} \\\\\\\\\n\\\\hline\n\\\\textrm{\\textbf{Recommender}} & \\\\textrm{Did you watch Avengers: Infinity War (2018) ? Or The Black Panthers: Vanguard of the Revolution (2015) ? i really liked it} \\\\\\\\\n\\\\hline\n\\\\textrm{\\textbf{Seeker}} & \\\\textrm{No , I think I would like to watch Easy A (2010) It's funny} \\\\\\\\\n\\\\hline\n\\\\textrm{\\textbf{Recommender}} & \\\\textrm{Great! Based on your preference for Easy A (2010), here are some movie recommendations for you:}\n\\\\newline\n& \\\\textrm{1. The DUFF (2015)}\n\\\\newline\n& \\\\textrm{2. Mean Girls (2004)}\n\\\\newline\n& \\\\textrm{3. 10 Things I Hate About You (1999)}\n\\\\newline\n& \\\\textrm{4. Clueless (1995)}\n\\\\newline\n& \\\\textrm{5. Legally Blonde (2001)}\n\\\\newline\n& \\\\textrm{6. She's the Man (2006)}\n\\\\newline\n& \\\\textrm{7. The Edge of Seventeen (2016)}\n\\\\newline\n& \\\\textrm{8. The Perks of Being a Wallflower (2012)}\n\\\\newline\n& \\\\textrm{9. Juno (2007)}\n\\\\newline\n& \\\\textrm{10. The Fault in Our Stars (2014)}\n\\\\newline\n& \\\\textrm{I hope you find these recommendations helpful. Let me know if you need any more suggestions!} \\\\\\\\\n\\\\hline\n\\\\textrm{\\textbf{Seeker (gpt-3.5-turbo)}} & \\\\textrm{Thanks for the recommendations! I have already watched some of them, but I will definitely check out the ones I haven't seen yet. By the way, have you watched any good movies lately?} \\\\\\\\\n\\\\hline\n\\\\textrm{\\textbf{Seeker (text-davinci-003)}} & \\\\textrm{These look great, thank you! I'm actually looking for something a bit different though. I'm looking for a comedy and a horror movie. Do you have any recommendations for those genres?} \\\\\\\\\n\\\\hline\n\\\\end{array}\n\nFurthermore, text-davinci-003 has been shown to outperform GPT-3.5-turbo on the WebShop task [1]. The WebShop task involves an agent purchasing a product based on user instructions through multi-turn interactions with the web. It evaluates the agent's ability to follow instructions in the context of multi-turn interactions. This ability aligns with our needs for the user simulator, and the superior performance of text-davinci-003 further reinforces our experimental designs.\n\nWe apologize for any confusion caused by this experimental design and will provide more thorough explanations in the revised version.\n\n[1] Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, Jian Zhang. On the Tool Manipulation Capability of Open-source Large Language Models. arxiv 2023.\n\n**2. The reasons in Section 3.3 can not fully explain why ChatGPT fails to outperform other methods.**\n\n- **For the first reason, other baseline methods are also unable to collect enough evidences.**\n\nThanks for the insightful comment! Regarding the first reason (\"Lack of Explicit User Preference\"), you correctly mentioned that baseline methods also face the challenge of collecting enough evidence. We agree with your observation; however, this issue is actually more serious for ChatGPT. \n\nExisting work [1, 2] has shown that benchmark datasets like ReDial have a long-tail distribution over items (Figure 2(a) in [1]), and CRSs suffer from popularity bias [2]. Since baseline methods are trained on these datasets, this issue results in unreasonably high scores for examples whose labels are items with high frequency (Figure 3(a) in [1]), even if these examples lack explicit user preference. \n\nIn contrast, ChatGPT is not fine-tuned in our evaluation. Therefore, when it comes to examples without explicit user preference, ChatGPT cannot utilize the item distributions of the datasets like baseline methods. As a result, the \"Lack of Explicit User Preference\" becomes an important reason for the failure of ChatGPT.\n\n[1] Lingzhi Wang, Huang Hu, Lei Sha, Can Xu, Kam-Fai Wong, Daxin Jiang. RecInDial: A Unified Framework for Conversational Recommendation with Pretrained Language Models. AACL 2022.\n\n[2] Allen Lin, Jianling Wang, Ziwei Zhu, James Caverlee. Quantifying and Mitigating Popularity Bias in Conversational Recommender Systems. CIKM 2022.\n\n- **For the second reason, I suppose forcing ChatGPT to make recommendation at that step with prompt design can solve the problem.**\n\nThanks for the comment! Regarding the second reason (\"Lack of Proactive Clarification\"), we have considered the method of forcing ChatGPT to make recommendations in our paper, as suggested by the reviewer.\n\n- For directly making recommendations by ChatGPT itself, we consider this approach in Section 3.1 (\"Zero-shot prompting\"), where ChatGPT is prompted to generate recommendations. As shown in Table 2 (the first line of ChatGPT), such an approach only achieves average performance and falls far behind the top-performing methods.\n\n- For making recommendations by external recommendation models, we consider this approach in Section 3.1 (\"Integrating Recommendation Models\"), where the response by ChatGPT is used as the input for recommendation models. Here, we would like to clarify that the key problem is not the intention of the response (recommendation or clarification) but the limited information of the user preference in the dialogue context.\n\n  - As shown in the example below, even if ChatGPT generates a response with the intention of recommendation, it only summarizes the user preference in the dialogue history. However, since so many items fit the requirement (\"musical movies\"), it is still hard to give accurate recommendations.\n\n    \\\\begin{array} {|l|l|l|}\n    \\\\hline\n    & \\\\textrm{Seeker} & \\\\textrm{I like The Cabin in the Woods. do you have it or something   similar}\n    \\\\newline\n    \\\\textrm{Dialogue history} & \\\\textrm{Recommender} & \\\\textrm{That's a great movie. Do you like Kristen Connolly? She starred in Cabin in the Woods.}\n    \\\\newline\n    & \\\\textrm{Seeker} & \\\\textrm{actually I like something like High School Musical 2.} \\\\\\\\\n    \\\\hline\n    \\\\textrm{Response by ChatGPT} & \\\\textrm{Recommender} & \\\\textrm{If you enjoyed High School Musical 2, you might like other musical movies.} \\\\\\\\\n    \\\\hline\n    \\\\hline\n    \\\\end{array}\n\n  - Instead, our evaluation approach, iEvaLM, allows further interaction between the user and system, where ChatGPT can gather the necessary information to differentiate among candidates. Table 5 demonstrates the significant improvement of baseline methods and ChatGPT in iEvaLM, further justifying the second reason (\"Lack of Proactive Clarification\").\n\nWe have provided sufficient explanations for the two mentioned reasons for the failure of ChatGPT and promise to add more thorough explanations to fully address your concerns in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes LLMs as user simulators to evaluate the performance of CRS methods, which is reasonable.\n2. Several human evaluations have been done to validate the quality of the LLM-based user simulator and consistency with human preferences.\n3. The paper is well-written and easy to understand.", "reasons_to_reject": "1. Some experimental designs are confusing\n    1.  GPT-3.5-turbo is used for CRS while text-davinci-003 is used for user simulator.\n2. The reasons in Section 3.3 can not fully explain why ChatGPT fails to outperform other methods.\n    1. For the first reason, other baseline methods are also unable to collect enough evidences.\n    2. For the second reason, I suppose forcing ChatGPT to make recommendation at that step with prompt design can solve the problem.\n\n--------------\nThese issues are well addressed in the authors' response.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is about evaluating conversational recommender systems (CRSs) in the era of large language models (LLMs) such as ChatGPT. The authors first investigate the performance of ChatGPT on existing CRS datasets and find that it is unsatisfactory due to the limitations of the current evaluation protocol. Then, they propose a new interactive evaluation approach, iEvaLM, which leverages LLM-based user simulators to examine the accuracy and explainability of recommendations. Through experiments, they show that ChatGPT significantly improves its performance under iEvaLM and outperforms existing CRSs in both aspects. They also demonstrate the effectiveness and reliability of their evaluation approach. The paper contributes to the understanding and evaluation of LLMs for conversational recommendation.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jclaIFdVJ9", "reviewer": "Reviewer_uFwV", "summary": "", "questions": "1. How do you ensure the reproducibility of results? Given that ChatGPT is not entirely open-source and undergoes frequent updates, and considering recent findings [1] suggesting a significant performance drop compared to versions from a few months ago, how can you guarantee the reproducibility of your approach that heavily relies on ChatGPT?\n2. What does \"LLM-based scorer\" refers to in lines 371-372? How was it implemented?\n3. When testing ChatGPT, did you provide a candidate movie list? If not, how did you ensure that the recommended movies fall within the evaluation scope? For instance, if the target movie is \"The Avengers,\" but ChatGPT recommends \"The Avengers 4.\"\n\n[1] Lingjiao Chen, Matei Zaharia, James Zou, \"How Is ChatGPT’s Behavior Changing over Time?\" https://arxiv.org/pdf/2307.09009.pdf", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for reviewing our paper and providing valuable feedback. We have carefully considered your comments and have addressed each of your concerns below.\n\n**1. How do you ensure the reproducibility of results? Given that ChatGPT is not entirely open-source and undergoes frequent updates, and considering recent findings [1] suggesting a significant performance drop compared to versions from a few months ago, how can you guarantee the reproducibility of your approach that heavily relies on ChatGPT?**\n\nWe appreciate your concern regarding the reproducibility of our results related to ChatGPT. We have tried our best to ensure the reproducibility of the experimental results.\n\n- As stated in Section 2.2, lines 219-224, we have followed existing work [1, 2] to utilize the publicly available model gpt-3.5-turbo-0301 provided in the OpenAI API instead of the web version, which does not receive updates.\n\n- Additionally, we have set the temperature to 0 for gpt-3.5-turbo-0301, as suggested by OpenAI, to ensure the most deterministic generation.\n\nMoreover, we have provided all of our code and data via the anonymous link in the paper. Reviewers P19e and ANzj also gave high scores for the reproducibility of our work.\n\n[1] Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang. A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets. ACL 2023.\n\n[2] Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, Zhaochun Ren. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. arxiv 2023.\n\n**2. What does \"LLM-based scorer\" refers to in lines 371-372? How was it implemented?**\n\nWe have introduced the LLM-based scorer in Section 4.4, lines 430-443. The LLM-based scorer is proposed to reduce the need for human scoring of the subjective metric persuasiveness. It was implemented using the text-davinci-003 model provided in the OpenAI API. For a detailed description of the definition and implementation of the LLM-based scorer, please refer to Section 4.4, lines 430-443.\n\n**3. When testing ChatGPT, did you provide a candidate movie list? If not, how did you ensure that the recommended movies fall within the evaluation scope? For instance, if the target movie is \"The Avengers,\" but ChatGPT recommends \"The Avengers 4.\"**\n\nThanks for the comment! We do not provide a candidate movie list when testing ChatGPT since it has been shown that ChatGPT can exhibit unexpected behaviors like repetition and missing when required to rank a list [1].\n\nConsidering this, in Section 3.1 (\"Integrating Recommendation Models\"), we follow existing work [2] to utilize external recommendation models to constrain the output space. We concatenate the conversation history and responses generated by ChatGPT as inputs for these recommendation models. We consider two models: MESE [3] and text-embedding-ada-002.\n\n- MESE is a classification-based CRS model trained with the training set of the dataset for evaluation, so its predictions will definitely fall within the evaluation scope.\n\n- Text-embedding-ada-002 is a model from the OpenAI API that can be used to translate item descriptions into embeddings. Then, the recommendation can be made through calculating the similarity between dialogue context and candidate items. We constrain the candidate items to those from the dataset for evaluation; thus, the matched items definitely fall within the evaluation scope.\n\nTherefore, the recommended items will definitely fall within the evaluation scope, and the approach we take can avoid the pitfalls of providing a candidate movie list.\n\n[1] Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, Zhaochun Ren. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. arxiv 2023.\n\n[2] Junling Liu, Chao Liu, Peilin Zhou, Renjie Lv, Kang Zhou, Yan Zhang. Is ChatGPT a Good Recommender? A Preliminary Study. arxiv 2023.\n\n[3] Bowen Yang, Cong Han, Yu Li, Lei Zuo, Zhou Yu. Improving Conversational Recommendation Systems' Quality with Context-Aware Item Meta Information. NAACL 2022.\n\n**4. This paper focusing solely on ChatGPT's capacities in conversational recommendation tasks. It would be valuable to explore the performance of other open-source large language models in conversational recommendation settings. Does the number of parameters such as 7B, 13B affect the findings? Providing demonstrations or examples to ChatGPT might be beneficial, such as 1-shot, 3-shot scenarios, considering that experimental results are obtained on a 0-shot scenario.**\n\nThanks for the comment! Since this was an initial investigation into using LLMs for CRS, we focused specifically on ChatGPT, which is believed to be the most powerful LLM that is publicly available when submitting the paper (GPT-4 requires an application). \n\nWe appreciate your suggestion to explore other LLMs with different scales and the few-shot setting for ChatGPT. However, you may notice that our paper has very tight page space, with extensive analysis and experiments. Since the results of other LLMs and the few-shot setting (as shown below) do not affect the rethinking of the traditional evaluation approach, we have to keep the most important model, ChatGPT, as shown in Table 2.\n\nHere, as requested by the reviewer, we report the evaluation results of Vicuna 7B and 13B, Baize 7B and 13B (we did not have enough resources to test larger models like 33B), and the 1-shot and 3-shot scenario for ChatGPT.\n\n- ReDial\n  - Open-source LLM Vicuna 7B and 13B, Baize 7B and 13B\n  \\\\begin{array} {|l|c|c|}\n  \\\\hline\n  \\\\textrm{Models} & \\\\textrm{Recall@1} & \\\\textrm{Recall@10} \\\\\\\\\n  \\\\hline\n  \\\\textrm{ChatGPT} & 0.034 & 0.172 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.040 & 0.198 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.058 & 0.179 \\\\\\\\\n  \\\\hline\n  \\\\textrm{Vicuna-7B} & 0.025 & 0.138 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.039 & 0.193 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.042 & 0.131 \\\\\\\\\n  \\\\hline\n  \\\\textrm{Vicuna-13B} & 0.032 & 0.161 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.037 & 0.189 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.046 & 0.127 \\\\\\\\\n  \\\\hline\n  \\\\textrm{Baize-7B} & 0.034 & 0.158 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.028 & 0.144 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.045 & 0.134 \\\\\\\\\n  \\\\hline\n  \\\\textrm{Baize-13B} & 0.030 & 0.167 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.027 & 0.140 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.046 & 0.131 \\\\\\\\\n  \\\\hline\n  \\\\end{array}\n  - 1-shot and 3-shot for ChatGPT\n  \\\\begin{array} {|l|c|c|}\n  \\\\hline\n  \\\\textrm{Models} & \\\\textrm{Recall@1} & \\\\textrm{Recall@10} \\\\\\\\\n  \\\\hline\n  \\\\textrm{(0-shot) ChatGPT} & 0.034 & 0.172 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(0-shot) ChatGPT + MESE} & 0.040 & 0.198 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(0-shot) ChatGPT + text-embedding-ada-002} & 0.058 & 0.179 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(1-shot) ChatGPT} & 0.031 & 0.148 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(1-shot) ChatGPT + MESE} & 0.038 & 0.190 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(1-shot) ChatGPT + text-embedding-ada-002} & 0.058 & 0.165 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(3-shot) ChatGPT} & 0.037 & 0.176 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(3-shot) ChatGPT + MESE} & 0.038 & 0.195 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(3-shot) ChatGPT + text-embedding-ada-002} & 0.061 & 0.174 \\\\\\\\\n  \\\\hline\n  \\\\end{array}\n\n- OpenDialKG\n  - Open-source LLM Vicuna 7B and 13B, Baize 7B and 13B\n  \\\\begin{array} {|l|c|c|}\n  \\\\hline\n  \\\\textrm{Models} & \\\\textrm{Recall@1} & \\\\textrm{Recall@10} \\\\\\\\\n  \\\\hline\n  \\\\textrm{ChatGPT} & 0.105 & 0.264 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.258 & 0.531 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.370 & 0.633 \\\\\\\\\n  \\\\hline\n  \\\\textrm{Vicuna-7B} & 0.237 & 0.437 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.245 & 0.523 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.332 & 0.579 \\\\\\\\\n  \\\\hline\n  \\\\textrm{Vicuna-13B} & 0.221 & 0.438 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.246 & 0.527 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.346 & 0.588 \\\\\\\\\n  \\\\hline\n  \\\\textrm{Baize-7B} & 0.184 & 0.397 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.229 & 0.530 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.316 & 0.572 \\\\\\\\\n  \\\\hline\n  \\\\textrm{Baize-13B} & 0.235 & 0.440 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ MESE} & 0.240 & 0.508 \\\\\\\\\n  \\\\hline\n  \\\\textrm{+ text-embedding-ada-002} & 0.311 & 0.545 \\\\\\\\\n  \\\\hline\n  \\\\end{array}\n  - 1-shot and 3-shot for ChatGPT\n  \\\\begin{array} {|l|c|c|}\n  \\\\hline\n  \\\\textrm{Models} & \\\\textrm{Recall@1} & \\\\textrm{Recall@10} \\\\\\\\\n  \\\\hline\n  \\\\textrm{(0-shot) ChatGPT} & 0.105 & 0.264 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(0-shot) ChatGPT + MESE} & 0.258 & 0.531 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(0-shot) ChatGPT + text-embedding-ada-002} & 0.370 & 0.633 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(1-shot) ChatGPT} & 0.077 & 0.211 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(1-shot) ChatGPT + MESE} & 0.212 & 0.487 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(1-shot) ChatGPT + text-embedding-ada-002} & 0.374 & 0.648 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(3-shot) ChatGPT} & 0.100 & 0.227 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(3-shot) ChatGPT + MESE} & 0.255 & 0.523 \\\\\\\\\n  \\\\hline\n  \\\\textrm{(3-shot) ChatGPT + text-embedding-ada-002} & 0.374 & 0.651 \\\\\\\\\n  \\\\hline\n  \\\\end{array}\n\nAs you can see, the performance of Vicuna, Baize, and few-shot ChatGPT also falls behind the top-performing method. Therefore, these results do not affect the rethinking of the traditional evaluation approach.\n\nHowever, we believe it would be valuable to explore other LLMs and settings. So in the revised version, we will move some analysis to the appendix and expand Table 2 to include these additional results.\n\n**5. The reasons behind ChatGPT's poor performance in conversational recommender tasks lack evidence.**\n- **Line 303. Traditional CRSs are also limited by explicit user preferences. This may be why ChatGPT does not do well. But the analysis in lines 313-320 does not seem to substantiate this claim convincingly.**\n\nThanks for the insightful comment! You correctly mentioned that traditional CRSs are also limited by explicit user preferences. However, this issue is actually more serious for ChatGPT. \n\nAs shown in existing work [1, 2], benchmark datasets like ReDial have a long-tail distribution over items (Figure 2(a) in [1]), and CRSs suffer from popularity bias [2]. Since traditional CRSs are obtained from supervised training, these issues lead to unreasonably high scores for examples whose labels are items with high frequency (Figure 3(a) in [1]), even if these examples lack explicit user preference. \n\nIn contrast, ChatGPT is not fine-tuned in our evaluation. Therefore, for examples without explicit user preference, ChatGPT cannot utilize the item distributions of the datasets like traditional CRSs to generate recommendations. \n\nConsequently, lack of explicit user preference may not be very serious for traditional CRSs but is crucial for the failure of ChatGPT.\n\n[1] Lingzhi Wang, Huang Hu, Lei Sha, Can Xu, Kam-Fai Wong, Daxin Jiang. RecInDial: A Unified Framework for Conversational Recommendation with Pretrained Language Models. AACL 2022.\n\n[2] Allen Lin, Jianling Wang, Ziwei Zhu, James Caverlee. Quantifying and Mitigating Popularity Bias in Conversational Recommender Systems. CIKM 2022.\n\n- **Line 321. I don't think this is the reason why ChatGPT doesn't do well with conversational recommendation tasks. Traditional datasets are manually annotated with fixed settings and rules. So without giving ChatGPT examples, it is likely that it will not be able to follow the patterns of the conversations in the dataset to make recommendations.**\n\nThanks for the comment! We agree with you that traditional datasets are annotated with fixed settings and rules. Actually, our approach is seamlessly integrated with these datasets, which keeps the original labels and takes them as the preference of simulated users. However, we would like to clarify that the key problem is not following the patterns but the limited information about the user's preference.\n\nAs shown in the example below, even if ChatGPT follows the patterns to generate a response with the intention of recommendation, it only summarizes the user's preference in the dialogue history. However, since there are so many items that fit the requirement (\"musical movie\"), it is still challenging to provide accurate recommendations.\n\n\\\\begin{array} {|l|l|l|}\n\\\\hline\n& \\\\textrm{Seeker} & \\\\textrm{I like The Cabin in the Woods. do you have it or something similar}\n\\\\newline\n\\\\textrm{Dialogue history} & \\\\textrm{Recommender} & \\\\textrm{That's a great movie. Do you like Kristen Connolly? She starred in Cabin in the Woods.}\n\\\\newline\n& \\\\textrm{Seeker} & \\\\textrm{actually I like something like High School Musical 2.} \\\\\\\\\n\\\\hline\n\\\\textrm{Response by ChatGPT} & \\\\textrm{Recommender} & \\\\textrm{If you enjoyed High School Musical 2, you might like other musical movies.} \\\\\\\\\n\\\\hline\n\\\\hline\n\\\\end{array}\n\nFurthermore, as the reviewer suggested in Q4, we tested the performance in the 1-shot and 3-shot scenarios using responses with the intention of recommendation as demonstrations. You can refer to Q4 for the performance table. It can observed that the performance of ChatGPT in the few-shot setting is comparable with the zero-shot setting and still falls behind the top-performing methods, which further reinforces the importance of the reason \"Lack of Proactive Clarification\" instead of following the patterns.\n\n**6. The paper lacks theoretical support for the reliability of the evaluation method. In dialogue evaluation-related research, correlation coefficients are often computed to measure the effectiveness of an evaluation method. I don't think the experiments carried out in the paper show that iEvaLM is objective and fair. Moreover, the reproducibility of iEvaLM's evaluation results also needs to be ensured as an evaluation method.**\n\nThanks for the comment! We acknowledge the importance of correlation coefficients. However, commonly used correlation coefficients in dialogue evaluation-related research, such as Pearson, Spearman, and Kendall's tau, cannot accurately reflect the correlation in our scenario. \n\nAs stated in Section 5.2.2, we compare our LLM-based scorer with human annotators to evaluate its reliability for scoring the subjective metric persuasiveness. This metric belongs to ordinal data since its value range is {0, 1, 2} with an inherent order (2 > 1 > 0). However, since Pearson correlation coefficient is used for quantitative data, it cannot be used to effectively measure the correlation. While Spearman and Kendall's tau correlation coefficients are used for ordinal data, they can only measure the monotonic relationship between variables, which does not apply to the comparison between scores of persuasiveness. For example, if the human gives a score of 1 for persuasiveness, giving a score of 0 and 2 are equally bad for the LLM-based scorer, only 2 is good. Our explanation is also supported by the discussion on using correlation measures to evaluate NLG metrics [1].\n\nIn fact, we have tried using these metrics but obtained abnormal values. As shown in the example below (taking from our collected comparison data), it is evident that these two variables are correlated (with only 14 differences out of 100 observations). However, the values of these coefficients indicate that they are not correlated at all (very close to 0). This discrepancy suggests that these correlation coefficients are not suitable for our scenario.\n\n```\nimport numpy as np\nfrom scipy import stats\nx = np.array([2.0, 2.0, 1.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 1.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0])\nprint(len(x))  # 100\ny = np.array([2.0, 2.0, 2.0, 2.0, 0.0, 2.0, 2.0, 2.0, 2.0, 1.0, 2.0, 2.0, 0.0, 2.0, 2.0, 2.0, 1.0, 2.0, 2.0, 2.0, 2.0, 2.0, 1.0, 2.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 1.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 0.0, 2.0, 2.0, 2.0, 2.0, 1.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 0.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0, 2.0])\nprint(len(y))  # 100\nprint((x != y).sum())  # 14 differences\nprint(stats.pearsonr(x, y).statistic)  # -0.05506796823644914\nprint(stats.spearmanr(x, y).statistic)  # -0.06171640287008385\nprint(stats.kendalltau(x, y).statistic)  # -0.06069243286297407\n```\n\nConsidering this, we follow existing work on user simulation in CRS [2] to compare the score distributions between human and LLM-based scorers, as shown in Table 7. The similarity between these two distributions demonstrates the reliability of our LLM-based scorer for the Persuasiveness metric. Therefore, our performance measurement, which consists of Recall and Persuasiveness, is reliable, given that Recall itself is an objective metric.\n\nBuilding upon this, we further examine the LLM-based user simulator, which is a replacement for humans to interact with the system. We follow the approach outlined in the existing work [2] to compare the performance obtained using the user simulator and humans. As show in Table 8, the ranking obtained from our user simulator is consistent with that of real users, and their absolute scores are also comparable. These results suggest that our user simulator is also reliable.\n\nAfter evaluating both components, the user simulator, and the performance measurement, we can conclude that iEvaLM provides an objective and fair evaluation of CRS.\n\nRegarding reproducibility, we provide detailed information in Q1. Please refer to that for the response.\n\n[1] Ananya B. Sai, Akash Kumar Mohankumar, Mitesh M. Khapra. A Survey of Evaluation Metrics Used for NLG Systems. ACM Computing Surveys 2022.\n\n[2] Shuo Zhang, Krisztian Balog. Evaluating Conversational Recommender Systems via User Simulation. KDD 2020.\n\n**7. Reproducibility: 1: Could not reproduce the results here no matter how hard they tried.**\n\nWe detail this in Q1. Please refer to that for the response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper investigates ChatGPT's capabilities in completing conversational recommender tasks on two datasets.\n2. This paper introduces an interactive approach using LLM-based user simulators to evaluate CRSs.\n3. This paper is well written and organized.", "reasons_to_reject": "1. This paper focusing solely on ChatGPT's capacities in conversational recommendation tasks. It would be valuable to explore the performance of other open-source large language models in conversational recommendation settings. Does the number of parameters such as 7B, 13B, or even larger affect the findings? Providing demonstrations or examples to ChatGPT might be beneficial, such as 1-shot, 3-shot scenarios, considering that experimental results are obtained on a 0-shot scenario.\n2. The reasons behind ChatGPT's poor performance in conversational recommender tasks lack evidence.\n\n- Line 303. Traditional CRSs are also limited by explicit user preferences. This may be why ChatGPT does not do well. But the analysis in lines 313-320 does not seem to substantiate this claim convincingly.\n- Line 321. I don't think this is the reason why ChatGPT doesn't do well with conversational recommendation tasks. Traditional datasets are manually annotated with fixed settings and rules. So without giving ChatGPT examples, it is likely that it will not be able to follow the patterns of the conversations in the dataset to make recommendations.\n\n3. The paper lacks theoretical support for the reliability of the evaluation method. In dialogue evaluation-related research, correlation coefficients are often computed to measure the effectiveness of an evaluation method. I don't think the experiments carried out in the paper show that iEvaLM is objective and fair. Moreover, the reproducibility of iEvaLM's evaluation results also needs to be ensured as an evaluation method.", "questions_to_authors": "1. How do you ensure the reproducibility of results? Given that ChatGPT is not entirely open-source and undergoes frequent updates, and considering recent findings [1] suggesting a significant performance drop compared to versions from a few months ago, how can you guarantee the reproducibility of your approach that heavily relies on ChatGPT?\n2. What does \"LLM-based scorer\" refers to in lines 371-372? How was it implemented?\n3. When testing ChatGPT, did you provide a candidate movie list? If not, how did you ensure that the recommended movies fall within the evaluation scope? For instance, if the target movie is \"The Avengers,\" but ChatGPT recommends \"The Avengers 4.\"\n\n[1] Lingjiao Chen, Matei Zaharia, James Zou, \"How Is ChatGPT’s Behavior Changing over Time?\" https://arxiv.org/pdf/2307.09009.pdf", "paper_topic_and_main_contributions": "This paper primarily focuses on evaluating conversational recommender systems (CRSs) using large language models (LLMs).\n\nIts main contributions involve the exploration of ChatGPT's performance in conversational recommendation tasks and the proposal of a method that employs large language model to construct a user simulator for evaluating CRSs performance through simulated dialogues.\n\nHowever, the proposed method iEvaLM seems less practical in the research. ChatGPT's performance is inconsistent[1], raising concerns about ensuring consistent evaluations from the LLM-based user simulator.\n\n[1] Lingjiao Chen, Matei Zaharia, James Zou, How Is ChatGPT’s Behavior Changing over Time?, https://arxiv.org/pdf/2307.09009.pdf", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q1v3ritiqt", "reviewer": "Reviewer_ANzj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on our paper. We appreciate the time and effort you have put into reviewing our work. We have carefully considered your comments and have addressed each of your concerns below.\n\n**1. You can use a bar chart as an alternative to a performance table.**\n\nThanks for the suggestion! We use the performance table because it offers a comprehensive and detailed view of the results. However, we acknowledge your great suggestion of using a bar chart as an alternative to the performance table. It can provide a visual representation of the performance metrics, enhancing the visualization of our results. We will use the bar chart instead in the revised version.\n\n**2. Taking into account the grammatical concern, when referring to \"objective metrics,\" it's important to note that the term \"objective\" refers to a singular metric. Likewise, the same consideration applies to \"subjective\" metrics. Furthermore, you need to provide additional information about subjective metrics, such as how its formula is defined.**\n\nThanks for your careful reading! We apologize for the grammatical error and will rectify it in the revised version. Regarding the subjective metric persuasiveness, we have provided a detailed explanation of its definition, implementation, and scoring rules in Section 4.4 (Lines 430-443) and Appendix D.4. Since it is a simple and subjective metric (e.g., \"if you think recommended items are worse than the ground-truth items, give 0\" in Appendix D.4), it does not have a strict formula and we believe scoring rules are enough.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe authors thoroughly analyse the current issues and challenges, and presents a novel approach to address them.\n\n2.\tThis paper goes beyond theoretical discussions, providing thorough validation through detailed experiments and case studies.\n\n3.\tThis paper is well-written in clear and concise language, demonstrating excellent organization and structure.", "reasons_to_reject": "1.\tYou can use a bar chart as an alternative to a performance table.\n\n2.\tTaking into account the grammatical concern, when referring to \"objective metrics,\" it's important to note that the term \"objective\" refers to a singular metric. Likewise, the same consideration applies to \"subjective\" metrics. Furthermore, you need to provide additional information about subjective metrics, such as how its formula is defined.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. This work is the first to systematically examine the capability of ChatGPT for conversational recommendation on large-scale datasets.\n\n2. The authors provide a detailed analysis of the limitations of ChatGPT under the traditional evaluation protocol, discussing the root cause of why it fails on existing benchmarks.\n\n3. They propose a new interactive approach that employs LLM-based user simulators for evaluating CRSs, and the experiments conducted on two public CRS datasets also demonstrate the effectiveness as well as reliability of their evaluation approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: To address the limitations of existing evaluation protocols, which might overemphasize matching with ground-truth items and overlook the interactive nature of CRSs, this paper introduces iEvaLM, which leverages LLM-based user simulators.

This paper has gone through a thorough discussion among reviewers and many concerns about it have been well addressed. Nonetheless, the paper can still be improved by addressing the following issues:
1. The proposed iEvaLM might be biased towards the LLM used during evaluation.
2. Add a discussion of the potential risks of using iEvaLM, as reviewers have concerns about the objectivity and fairness of the proposed method. It would be nice if the authors could add some use hints or guides for readers or practitioners.

The authors are also encouraged to include all results listed in the rebuttal in their paper if accepted.
Decision: Accept-Main