LLM-REDIAL: A Large-Scale Dataset for Conversational Recommender Systems Created from User Behaviors with LLMs (2024)

Abstract: The large-scale conversational recommendation dataset is pivotal for the development of conversational recommender systems (CRS). Most existing CRS datasets suffers from the problems of data inextensibility and semantic inconsistency. To tackle these limitations and establish a benchmark in the conversational recommendation scenario, in this paper, we introduce the LLM-REDIAL dataset1 to facilitate the research in CRS. LLM-REDIAL is constructed by leveraging large language models (LLMs) to generate the high-quality dialogues. To provide the LLMs with detailed guidance, we integrate historical user behavior data with dialogue templates that are carefully designed through the combination of multiple pre-defined goals. LLM-REDIAL has two main advantages. First, it is the largest multidomain CRS dataset which consists of 47.6k multi-turn dialogues with 482.6k utterances across 4 domains. Second, dialogue semantics and the users’ historical interaction information is highly consistent. Human evaluation are conducted to verify the quality of LLM-REDIAL. In addition, we evaluate the usability of advanced LLM-based models on LLM-REDIAL.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 8926–8939 August 11-16, 2024 ©2024 Association for Computational Linguistics
LLM-REDIAL: A Large-Scale Dataset for Conversational Recommender Systems Created from User Behaviors with LLMs Tingting Liang1,2, Chenxin Jin1, Lingzhi Wang3, Wenqi Fan1
Congying Xia4, Kai Chen1, Yuyu Yin1,2* 1Hangzhou Dianzi University, China
2Zhoushan Tongbo Marine Electronic Information Research Institute of Hangzhou Dianzi University, China 3The Chinese University of Hong Kong, Hong Kong, China
4Salesforce Research, Palo Alto, USA {liangtt,chenxinkim,wenqifan397,chenkai,yinyuyu}@hdu.edu.cn
lzwang@se.cuhk.edu.hk , c.xia@salesforce.com Abstract
The large-scale conversational recommendation dataset is pivotal for the development of conversational recommender systems (CRS). Most existing CRS datasets suffers from the problems of data inextensibility and semantic inconsistency. To tackle these limitations and establish a benchmark in the conversational recommendation scenario, in this paper, we introduce the LLM-REDIAL dataset1 to facilitate the research in CRS. LLM-REDIAL is constructed by leveraging large language models (LLMs) to generate the high-quality dialogues. To provide the LLMs with detailed guidance, we integrate historical user behavior data with dialogue templates that are carefully designed through the combination of multiple pre-defined goals. LLM-REDIAL has two main advantages. First, it is the largest multidomain CRS dataset which consists of 47.6k multi-turn dialogues with 482.6k utterances across 4 domains. Second, dialogue semantics and the users’ historical interaction information is highly consistent. Human evaluation are conducted to verify the quality of LLM-REDIAL. In addition, we evaluate the usability of advanced LLM-based models on LLM-REDIAL.

heading: 1 Introduction
text: In recent years, conversational recommender systems (CRS) have been widely explored in both academia and industry (Zhou et al., 2020a; He et al., 2023; Zhou et al., 2021), which leverage natural language conversations to provide users with personalized and context-aware recommendations. Unlike the conventional recommender systems that rely solely on user-item interactions, CRS incorporates the conversational aspect, allowing users to interact with the system through natural language.
The existing CRS methods are primarily data-driven, requiring large-scale conversational *Corresponding author 1The dataset is available at https://github.com/ LitGreenhand/LLM-Redial
datasets for model training. In this connection, an increasing emphasis has been placed on dataset construction in the field of CRS. There are a few efforts to build datasets for conversational recommendation (Li et al., 2018; Zhou et al., 2020b; Liu et al., 2020; Manzoor and Jannach, 2022). Table 1 lists some commonly known CRS datasets. The REDIAL dataset (Li et al., 2018) consisting of over 10,000 dialogues was released to the community for conversational movie recommendation. REDIAL was collected by pairing up Amazon Mechanical Turk (AMT) workers and guiding them to engage in a dialogue with the purpose of recommending movies. A topic-guided CRS dataset named TGReDial (Zhou et al., 2020b) was constructed with the topic threads-based utterance retrieval and human annotation. DuRecDial (Liu et al., 2020) is a human-to-human recommendation oriented multitype dialog dataset which was created by manual annotation with pre-defined goals.
While these existing datasets have propelled the development of conversational recommendation to some extent, there are still the following limitations of two aspects: (1) Data inextensibility. Most of previous dataset construction require a lot of human annotations significantly limiting the dataset scalability. Additionally, the quality of dialogue texts obtained through sentence retrieval or crowd-workers can not be guaranteed. Insufficient quantity and quality of dialogues would hinder the training of CRS models. Even with the emergence of Large Language Models (LLMs), this situation persists. While current LLMs demonstrate superior capabilities in text generation and data annotation (Wang et al., 2021; Ding et al., 2023), they exhibit less promising performance in conversational recommendation. Consequently, large-scale conversational recommendation datasets remain a bottleneck in the development of CRS. (2) Semantic Inconsistency. The surge of LLMs making the response generation in CRS less challenging, and the
8926
Table 1: Comparison of LLM-REDIAL with other datasets for conversational recommendation.
Datasets #Dialogues #Utterances #Tokens #4-Grams Domains User-Centric
REDIAL 10k 182k 4.5k 58k Movie No TG-REDIAL 10k 129k 50k 7.5k Movie No DuRecDial 10.2k 156k 17.6k 461k Movie, music, food, etc No INSPIRED 1k 35k 11k 182k Movie No OpenDialKG 15k 91k 22k 547k Movie, book No LLM-REDIAL 47.6k 482.6k 124.2k 4.6M Movie, book, sport, etc Yes
Conversation 1:
Conversation 2:
[User] Hi, I recently watched Mission Impossible on VHS and it was a fantastic high-tech spy movie!TomCruise gave… [Agent] That's great to hear! I have a movie recommendation for you based on your likes.How about checking outWorldWarZ? [User] Oh, I'm definitely interested. Can you tell me more about the movie? [Agent] Certainly!WorldWar Z is a good zombiewar flick…
Historical Interactions: ["Robin Williams: Live On Broadway", "Mission Impossible on VHS", "Solaris", "Elysium", "Wall Street", "Mystic River", …]
User ID: A1EXXXXXDUE6B0
[User] Hi, I recently watched the movie "Solaris" and I have to say, I didn't enjoy it at all. It felt overly melodramatic and lackedsubstance… [Agent] I understand why you didn’t like “Solaris”. I can recommend a movie you might enjoy instead. How about “Elysium”? It’s a bilingual film that… [User] Oh, I've actually already watched "Elysium" and it was better than I expected, but still not great.… [Agent] How about giving "Wrecking Crew" a try? It's another movie you might like based on yourprevious preferences… [User] Sure, that sounds interesting…
Figure 1: An example of user profile including dialogues generated by LLM in LLM-REDIAL. One user might be associated to multiple conversations. The items mentioned in the conversations are consistent with user’s historical interactions.
research focus is gradually shifting towards the recommendation aspect. The consistency between dialogues and users’ actual behaviors is a choke point for the assessment of recommendation. Neither the simulated dialogue generated by crowd-workers nor the user profile-based semi-automatic dialogue generation can maintain semantic consistency between the conversation content and users’ historical behaviors. Because these generation methods typically only specify the start of dialogues and the final goal or topic of recommendation, they fail to fully leverage the users’ truely historical behaviors to present the recommendation process. Consequently, a dataset that aligns the semantics in dialogue texts with users’ behavior is indispensable for the thorough evaluation of conversational recommendation.
To address the above limitations, in this paper, we construct a new large-scale dataset for CRS created from user behaviors through LLMs (LLMREDIAL). For the first limitation, we introduce the LLMs to generate a large quantity of highquality dialogue sentences under the guidance of pre-defined dialogue templates. To the best of our
knowledge, this is the largest conversational recommendation dataset with multiple domains. Table 1 shows that our LLM-REDIAL contains 47.6k multi-turn dialogues with 482.6k utterances across 4 domains. For the second limitation, we create a collection of templates by assigning each turn a goal in the dialogues. By filling these dialogue templates with the users’ behaviors including both positive and negative feedbacks along with review information, the prompts are derived for the LLMs to generate the complete multi-turn dialogues covering the recommendation process. In this manner, the consistency between the dialogue semantics and the users’ actual interactions can be effectively guaranteed.
As Table 1 shows, compared with the previous datasets, our LLM-REDIAL has a significant advantage in scale, with a much larger number of dialogues, utterances, and tokens. Our dataset contains more diverse dialogues, mainly characterized by involvement in multiple domains and richer semantics. The higher 4-grams value of our dataset indicates the more complex patterns and semantics contained in the conversational texts. Furthermore, LLM-REDIAL is user-centric, which means the user of each dialogue can be identified and all the dialogues and historical interactions associated with one specific user can be located in our dataset as shown in Figure 1. The multiple conversations of one specific user are beneficial to the capturing of user preferences and behaviour features. We provide an example of REDIAL in Appendix A to compare the text quality of the dialogues. The more specific quantitative comparison on the dialogue quality can be found in Section 4.1.

heading: 2 Dataset Construction

heading: 2.1 Data Source
text: To approach the realistic conversational recommendation scenario as closely as possible, we construct the dataset based on authentic user historical behaviours. In addition, we aim at naturally incorporating relevant item details, making the dialogues appear more reasonable and real. Therefore, we
select the product reviews from Amazon2 (He and McAuley, 2016) as the database. The review data contains user reviews along with rating information from Amazon platform. Specifically, the ratings of each user are used to identify the preference which would be combined with the corresponding review texts to generate the dialogues. In this manner, each dialogue is associated with one user’s historical interactions. The combination of these elements forms a complete data for conversational recommendation.

heading: 2.2 Overview of Dataset Construction
text: As Figure 2 shows, the overall process of the dataset construction sequentially consists of data preprocessing, template construction, and dialogue generation. First, the raw data of Amazon reviews are processed through the operations of data filtering, grouping, and splitting to obtain the historical interactions and the item list to be predicted for each user. The following template construction module designs the multiple goals for utterances and formulates templates for multi-turn dialogues by combining these goals. In the dialogue generation phase, the LLM is invoked to generate the dialogues implying the recommendation process based on pre-designed prompts which are derived by filling the dialogue templates with users’ behaviors and reviews.

heading: 2.3 Data Preprocessing
text: In order to smoothly utilize the raw review data to generate dialogues that centered around the func2http://jmcauley.ucsd.edu/data/amazon
tion of providing recommendations, we design a series of data preprocesses to filter out the interactions that meets the requirements. The details of the data preprocesses can be found in Appendix B.

heading: 2.4 Template Construction

heading: 2.4.1 Goal Design
text: To make the dialogue proceed along the lines of recommendations, we design multiple kinds of primary goals for the utterances referring to the communicative functions from the international standard ISO 244617-2. We design total 8 primary goals based on which detailed sub-goals are provided. Table 2 shows a part of the sub-goals and the complete 30 sub-goals can be found in Appendix C. The primary goals are used to decide the function of each utterance. Under each primary goal, there are several sub-goals of two types. One is the fixed instruction that indicates the more specific aspect (e.g., “Ask for recommendation”). The other type is the flexible instruction, consisting of the fixed instruction and a slot to be filled, such as “Recommend[USER_HIS_LIKES]”, where [USER_HIS_LIKES] would be filled with an item randomly sampled from the LIKES set collected from historical items with positive feedbacks.

heading: 2.4.2 Template Construction
text: To offer the LLMs the more instructive inputs for the generation of fluent and natural conversations, we construct various dialogue templates each of which is composed of multiple sub-goals. Specif-
ically, to enhance the diversity of dialogues, we set different templates based on the frequency of recommendations with the count restricted to 1-3 times. For the settings where recommendations are made 2 or 3 times, except for the final recommendation, all preceding recommendations are assumed to be rejected. Correspondingly, based on the three setting types, the ranges of dialogue lengths are also restricted differently. Referring to the dialogue lengths of most existing CRS datasets that range around 6-16, such as the datasets listed in Table 1, we constrained the dialogue lengths of all the settings within the same range. In setting with a higher number of recommendations, the dialogue length is extended accordingly. The combinations of goals are manually and carefully designed and finally 168 dialuge templates are obtained. Figure 3 (a) displays an example of the template that makes recommendation once with 8 utterances.

heading: 2.5 Dialogue Generation

heading: 2.5.1 Generation with LLMs
text: The prompt that fed into the LLMs is formed by integrating a pre-defined static prompt and a concretized template. The static prompt provides the task description and requirements with simple plain language statements as shown in Figure 3 (b). It is worth noting that, to establish a strong connection between dialogue content and item information, we introduce the real users’ reviews of the historical interactions to enrich the dialogue, while avoiding verbatim replication of review content. To prevent the dialogue from becoming overly verbose and ensure the quality of sentence generation, we limit the length of each sentence to 60 words.
The concretized template is achieved by filling in user information into the slots of the dialogue template. Specifically, for the generation of each dialogue, user information is obtained by sampling interactions and review texts from the historical behavior of one specified user. Figure 3 (c) shows an example of user information which is structured in a JSON file. By concatenating the static prompt
and the concretized template, the complete prompt to be fed into the LLMs are constructed.
To facilitate reproducibility, we adopt the static version of ChatGPT3, i.e., GPT-3.5-turbo, to generate the dialogues for conversational recommendation. Based on the integrated prompt shown in Figure 3 (b) and (c), Figure 3 (d) presents the complete dialogue output by GPT-3.5-turbo. It can be observed that the dialogue flow smoothly follows the designed dialogue template, and the key steps such as requesting recommendations, providing recommendations, and accepting recommendations are well reflected in the dialogue (the underlined words). Benefitting from the powerful generation capabilities of LLMs, the generated sentences seamlessly incorporate the item information from the relevant review texts and express in a natural and coherent manner, enhancing the diversity and authenticity of the dialogue. More importantly, the incorporation of items and related review information effectively strengthens the consistency between dialogue content and users’ historical behaviors, which is more aligned with the scenario of conversational recommender systems.

heading: 2.5.2 Dialogue Filtering
text: Due to the randomness of LLMs and the long review texts that may confuse the model, the conversations directly generated by ChatGPT contains some invalid, noisy cases, which might has detrimental impact on the research using this dataset for conversational recommendation. To tackle this issue, we design the following automatic data filtering processes to filter out the high-quality multiturn dialogues step by step: (1) We remove the dialogues that are not completely generated; (2) We ignore and discard the dialogues containing garbled or unreadable characters. (3) We remove the dialogues that contains the template information, i.e., the slots in the templates are not successfully filled with the user information. (4) We discard the dialogues that are inconsistent in length with
3https://openai.com/blog/chatgpt
the related dialogue templates. Through the above data filtering procedure, the final large-scale CRS dialogues could be better utilized to investigate the conversational recommendation methods.

heading: 2.6 Dataset Construction Cost Analysis
text: The main cost in the proposed construction process of LLM-REDIAL is the invocation of the GPT-3.5turbo-16k API during dialogue generation. Generating one dialogue requires 10-20 seconds. Generally, the fees of LLMs are measured as the dollar cost which are proportional to the number of tokens in the input and output. GPT-3.5-Turbo-16k is priced at $0.003 per 1K input tokens and $0.004 per 1K output tokens. In this work, we make approximately 100,000 calls to the GPT-3.5-turbo-16k API to generate the dialogues of conversational recommendation. Finally, it costs ∼$750 to generate the preliminary dialogues for the subsequent filtering.

heading: 3 Dataset Statistics
text: Our LLM-REDIAL is constructed based on the Amazon review dataset. There are 24 different domains and this work selects 4 of them to be the data sources. More domains will be used to generate more conversations in our future work. The LLM-REDIAL consists of 47,651 dialogues with
482,684 utterances across 4 domains. The statistics of our LLM-REDIAL are shown in Table 3. On average, each dialogue session in 4 domains has 9-10 utterances since we design three kinds of dialogue template with fixed ranges of dialogue length. One distinctive character of our dataset is its user-centric focus, each user has two corresponding dialogue sessions on average. Compared to Sports and Electronics categories, users in the Books and Movies categories have the higher average numbers of dialogues, possibly due to longer historical interaction sequences for book and movie purchases.

heading: 4 Evaluation

heading: 4.1 Human Evaluation on Dataset Quality
text: To perform a thorough and direct assessment of the quality of our curated dataset, we choose three representative CRS dataset in English for comparative analysis. Specifically, we conduct a human evaluation to measure the effectiveness and reliability of our constructed dataset, incorporating assessments at both the utterance and conversation levels. The potential limitation of the human evaluation process lies in the subjectivity and bias that annotators may introduce. Factors such as individual prefer-
ences and mood can influence how the annotators perceive and rate the quality of generated dialogues. To mitigate the limitation, we carefully recruit and select seven volunteer annotators to evaluate the dataset. All the annotators are Chinese graduate students in our institution. They are informed of the ongoing research and well known the evaluation rules and standards. 4.1.1 Utterance-Level Evaluation In the utterance-level evaluation, we randomly sample 10 dialogues from each of the compared datasets and our LLM-REDIAL. The order of these 40 dialogues is randomly shuffled. The selected dialogues of LLM-REDIAL totally contain 112 utterances with given contexts while these of REDIAL (Li et al., 2018), INSPIRED (Hayati et al., 2020), and OpenDialKG (Moon et al., 2019) respectively have 103, 208, 76 utterances. In total, each annotator is required to score 1,996 utterances for the 40 dialogues.
We then evaluate the quality of utterances based on four aspects: (1) Fluency: Assessing whether a response is organized in regular English grammar and is easy to understand. (2) Informativeness: Determining whether a response is meaningful and not a “safe response”, with repetitive responses considered uninformative. (3) Logicality: Evaluating the logical consistency of a response by assessing whether it aligns with common sense reasoning and follows a logical flow. (4) Coherence: Ensuring that a response is coherent with the previous context. The annotators are asked to evaluate re-
sponses on these four aspects, using a scale of 0, 1, 2 (a more detailed rating scheme can be found in Appendix D.1). We use Kendall’s coefficient of concordance (W) to measures the agreement of the seven annotators. The standard procedure for testing Kendall’s W concordance coefficient involves Chi-square statistics. With the calculated Kendall’s W coefficient of 0.312, the Chi-square value calculated is 4353.788. According to the degree of freedom (1996−1) = 1995, the Chi-square boundary value can be found to be X 20.01,1995 = 2150.66 which is smaller than 4353.788, then P < 0.01. Therefore, at the significance level of α = 0.01, it can be considered that the seven annotators achieve agreement in their ratings of the 1996 samples.
The results of the human evaluation on four datasets are presented in Table 4. The utterances of our LLM-REDIAL dataset achieve the higher scores than those in three compared datasets in terms of all the four metrics. The utterances in our dataset exhibit extremely high fluency, logicality, and coherence, which benefits from the strong generation capability of LLMs. Compared to the other datasets, the superiority in information expression of utterances in LLM-REDIAL is significant. It is mainly because we incorporate the users’ historical interactions with review information in the dialogue templates for LLMs-based generation, while the compared datasets rely on the temporarily paired two crowd-workers to generate dialogues, making it challenging to delve into detailed and in-depth topics.

heading: 4.1.2 Conversation-Level Evaluation
text: For the conversation-level evaluation, we assess quality through direct pair comparisons, asking annotators to determine which of the two provided conversations (note that the order of sourced datasets is randomized) exhibits higher quality. Specifically, we set three groups each of which is composed of our LLM-REDIAL and one compared dataset. For each group, we randomly select 50 dialogues from each of two datasets, forming 50 pairs through random matching. Seven annotators are asked to annotate a total of 150 pairs of dialogues from the three groups and select the
one with overall higher quality for each pair. Finally, we obtained 350 annotations for each group. The annotation results are shown in Figure 4. It shows that the proportions of annotations in all the three groups believe our LLM-REDIAL has the better quality are higher. An interesting finding is that OpenDialKG’s utterance-level results are quite good on all four aspects while approximately 88% of the annotators believe that the overall quality of dialogues in OpenDialKG is inferior to that in LLM-REDIAL. This is mainly because some dialogues in OpenDialKG end abruptly, and some dialogues lack recommendations.

heading: 4.2 Evaluation on Conversational Recommendation
text: We conduct a series of experiments on the dataset of Movie domain to show the applicability of LLM-REDIAL on the task of conversational recommendation and emphasize the importance of user-centric dialogues with interactions. Since generating dialogue texts is not a particularly challenging task for LLMs, our focus is on the recommendation task. We use Recall@K and NDCG@K (K = 5, 10, 50) as evaluation metrics.

heading: 4.2.1 Baselines
text: To verify the practicable of the constructed LLM-REDIAL, we consider the following LLMbased baselines for comparison: ChatGPT-based model uses GPT-3.5-turbo from OpenAI4 as recommender. Vicuna-based, Baize-based, and Guanaco-based models use the representative open-sourced LLMs fine-tuned based on LLAMA13B (Touvron et al., 2023), namely Vicuna-7B (Chiang et al., 2023), Baize-v2-7B (Xu et al., 2023), and Guanaco-7B (Dettmers et al., 2023), to be the recommenders. The decoding temperature is set to 0 for all models.
All the models take the preceding context of each dialogue as input to predict the item that will appear in the next response. Specially, we consider three settings which are zero-shot, few-shot, and fine-tuning. For the ChatGPT-based model, we randomly select 200 dialogues for testing. In the few-shot setting, we offer 5 case as examples. In the fine-tuning setting, we use 200 training examples to fine-tune. For the other three models, we randomly select 1,500 dialogues for testing. In the few-shot setting, we offer 5 case as examples. In the fine-tuning setting, we use the remaining 4https://openai.com/
8,593 training examples to fine-tune. The prompts for three settings are shown in Appendix D.2. As the LLM-based models offer the recommendation through the way of generative retrieval, we follow (He et al., 2023) and apply a fuzzy matching to transfer the generated textual recommendation list to a item ranking list.

heading: 4.2.2 Results and Analysis
text: Table 5 reports the performance of different baseline models on recommendation task. ‘Dial. Only’ indicates that only the dialogue texts are fed into the LLMs to generate the results, and ‘Dial. + H. I’ represents that both dialogue texts and users’ historical interactions are considered to be the inputs. It can be observed that all the baseline models obtain poor performance in the zero-shot and few-shot settings on both datasets, which indicates that the pretrained LLMs can not be directly applied for conversational recommendation without fine-tuning. All the models achieve a little improvements in the few-shot setting and significant improvements from fine-tuning on training data. The ranking of these models in terms of their performance under fine-tuning settings is consistent with the leaderboard ranking on AlpacaEval5. This indicates that our dataset is able to test the abilities of different models. The incorporation of users’ historical interactions effectively improves the recommendation performance for all the three settings, with the most significant enhancement in the fine-tuning setting. The experimental results demonstrates that the users’ historical interaction records are quite crucial in the scenario of CRS. However, most existing CRS datasets predominantly focus on the dialogue text. The conversations in these datasets often can not associated with the specific users, making it impossible to identify the corresponding historical interaction information.

heading: 4.2.3 Case Study
text: To more intuitively explore the effect of response generation for recommendation based on the LLMs under different settings, we provide an example of generating response with recommendation by ChatGPT-based model in Figure 6 of the Appendix E. In both the zero-shot and few-shot settings, the generated responses are coherent and natural while the recommendation performance is relatively poor. In other words, the introduction of LLMs makes the task of response generation in
5https://tatsu-lab.github.io/alpaca_eval/
traditional CRS more straightforward, while there is still significant room for improvement for the recommendation task. After fine-tuning, it is more likely to make recommendations meeting users’ requirements in the generated responses.

heading: 5 Related Work

heading: 5.1 Conversational Recommender Systems
text: Dialogue recommendation systems can be classified based on the number of dialogue turns, distinguishing between single-turn dialogue recommendation and multi-turn dialogue recommendation. This paper is focus on multi-turn dialogue recommendation systems (Srivastava et al., 2023; Balaraman and Magnini, 2020; He et al., 2023). The multi-round dialog recommendation system was studied by (Lei et al., 2020a) which allowing the CRS to pose multiple questions or recommend items across turns until the user accepts or exits the recommendation. To address challenges associated with multi-turn CRS, (Lei et al., 2020b) leveraged knowledge graphs to select more relevant attributes for cross-turn inquiries. (Xu et al., 2021) dynamically adjusted user embeddings based on user feedback on attributes and items, extending the work of (Lei et al., 2020a; Deng et al., 2021; Chu et al., 2023) unified the problem selection module and recommendation module in a reinforcement learning-based CRS solution. However, all the aforementioned works rely on carefully designed heuristic reward functions, which may lead
to strategies deviating from the optimal solution.

heading: 5.2 Datasets for Conversational Recommendation
text: In order to enhance the performance of CRS, many researchers have curated dialogue datasets based on specific rules (Li et al., 2018; Chen et al., 2019; Jannach et al., 2021; Lu et al., 2021; Hayati et al., 2020) manually annotated each utterance using social strategies to validate the effectiveness of social recommendation strategies in CRS. (Liu et al., 2020) created a multi-type dialogue dataset, aiming for bots to naturally guide conversations from nonrecommendation types to recommendation types. Similarly, (Zhou et al., 2020b) introduced a topicguided dialogue recommendation dataset to facilitate the transition of dialogue topics. However, Some studies (Liu et al., 2016; Novikova et al., 2017; Gao et al., 2021) pointed out that existing datasets lack the qualification to develop CRS that meet industrial application requirements due to: 1) these datasets are insufficient in scale to cover real-world entities and concepts; 2) dataset construction is carried out under strict conditions, making it challenging to generalize to complex and dynamic real-world dialogues. Therefore, developing a large-scale, generalizable, and naturally occurring dialogue dataset is a crucial task.

heading: 6 Conclusion
text: This paper presents a large-scale multi-turn dialogue dataset for conversational recommendation
which is constructed with LLMs based on the users’ historical behaviours. We fill the user behaviour data into the well-designed dialogue template to guide the LLMs to generate high-quality dialogues. Benefitting from the powerful generation capability of LLMs, LLM-REDIAL is the largest multidomain CRS dataset with 47.6k dialogues covering recommendation process. Comprehensive experiments are conducted to verify the quality and usability of our LLM-REDIAL. We believe that LLM-REDIAL can serve as a rich resource for advancing research in CRS, assisting the community in proposing better methods for conversational recommendation within the context of LLMs.
Limitations
Besides its merits, this work still has limitations that could be further improved. Firstly, the quality of generated dialogues, including the content, fluency, and relevance, is greatly influenced by the design of prompts as the prompts play a crucial role in shaping the output of LLMs. Recent works (Liu et al., 2022; Gu et al., 2022) on prompt tuning try to improve the output quality of LLMs by adjusting and optimizing prompts. However, this work mainly focuses on generating a large-scale dialogue dataset for conversational recommendation based on the users’ historical behaviours and does not consider the prompt tuning. It would be of interest to explore to find whether it is possible to achieve a optimized prompt for dialogue generation in the scenario of conversational recommendation. Secondly, the template construction in the pipeline of the LLM-REDIAL generation highly relies on manual design, which to some extent limits the efficiency and quality of dataset construction. We leave the question of how to reduce human intervention in the process of goal design and template construction as a direction for future research. Additionally, the biases present in Amazon review dataset are prone to lead to the potential bias in the dataset construction process. The Amazon review dataset mainly contains two types of biases: user rating bias and review bias. User rating bias arises from different users having different rating standards, while the positive and negative interactions for dialogue generation are splitted based on a uniform standard. Review bias is mainly reflected in the content being polarized, with possible cases of exaggeration or depreciation. Dialogues generated based on such review content may also be
biased. Detecting bias in our dataset that is generated by the conversational LLM is non-trivial, mainly due to the diverse outputs. There is a need for more nuanced and sophisticated process that can correct user rating and review bias before the user behaviour based dialogue generation.
Ethics Considerations
The LLM-REDIAL dataset is constructed based on the Amazon review dataset which contains the authentic user historical reviews collected from Amazon platform. Amazon review dataset is an open source which is commonly used for research. It is collected following strict legal and ethical guidelines and respecting user privacy. Therefore, all the collected review data, including user profiles, is publicly available and does not contain private information of the reviewers (consumers), such as real user names, phone numbers, and addresses. Moreover, for the user-related information in the raw data, we only use the processed reviewer IDs to ensure there is no disclosure of private information and the identity of the consumers can not be inferred.
For the data access, we establish a strict application process6 for the further privacy protection that requires users to provide application information, including name, organization, professional direction, position, reason for application, and email address. The dataset will be shared by e-mail once the application is approved. It should be noted that LLM-REDIAL is only for research purposes. Without permission, it can not be used for any commercial purposes or distributed to others.

heading: Acknowledgements
text: This work is supported by National Science and Technology Major Project (No.2022ZD0116700), Yangtze River Delta Project (No.2022CSJGG1000 and No.2023ZY1068), "Pioneer" and "Leading Goose" R&D Program of Zhejiang Province under No.2024C01166, Zhejiang Province High-Level Talent Special Support Program-Leading Talent of Technological Innovation under No.2022R52043.
6https://github.com/LitGreenhand/ LLM-Redial/blob/main/dataset_access.md

heading: A Comparison with REDIAL
text: Figure 5 shows the examples of dialogues in LLMREDIAL and REDIAL. It can be found that the conversations in LLM-REDIAL are are more fluent, with the smoother context transitions and richer semantic information in the text. The main difference is that LLM-REDIAL is user-centric and REDIAL is dialogue-centric. User-centric means the user of each conversation could be identified and all his or her conversations and the historical interactions can be found in LLM-REDIAL. REDIAL contains a collection of conversations which are independent with each other. The user in each conversation is just an agent played by a worker hired through AMT and do not have information such as the historical interactions or other related conversations.

heading: B Data Preprocessing
text: Due to the presence of non-word tokens in the review texts, we firstly tokenize the texts and remove those irregular tokens. After that, to guarantee the usability of the review content while avoiding excessively long text that may not provide accurate semantic information, we filter the review texts and retain records with a word count between 20 and 400. Besides, to ensure that the interaction quantity for each user is sufficient to support the generation of dialogues representing the recommendation process, we impose restrictions on the number of interactions. Specifically, we remove the users and items with less than 10 interactions. To make the dialogue content more diverse, it is expected to not only reflect situations where users accept recommendations but also those where users reject recommendations. Therefore, we intend to incorporate interactions of both user likes and dislikes into the dialogue. Ratings equal to or higher than 4 are picked out as positive feedbacks, while those equal to or lower than 2 are used as negative ones. Finally, the positive and negative interactions are sorted chronologically to form two collections (LIKES and DISLIKES) that prepare for generating prompts in the subsequent dialogue generation step. It should be noted that the last 10% of the positive interactions of each user are moved to a new collection (MIGHT_LIKES) from which the items are selected as the final golden recommendation in the dialogues.

heading: C Complete Goals
text: Table 6 shows total 8 primary goals and 30 subgoals with the related descriptions.

heading: D Experimental Settings
text: D.1 Rating Scheme In this manual evaluation, four key metrics are employed to assess the quality of a dialogue, namely: 1) Fluency; 2) Informativeness; 3) Logical; and 4) Coherence. Each metric is graded on a scale from 0 to 2, with 0 indicating poor performance, 1 signifying moderate performance, and 2 denoting excellent performance. The specific grading criteria for each metric are delineated below: Fluency: 0 (poor): The dialogue exhibits severe grammatical errors, spelling mistakes, vocabulary issues, or incoherent expressions, rendering it difficult to comprehend. 1 (normal): The dialogue contains some grammar errors, spelling mistakes, vocabulary problems, or lacks fluency, yet remains generally understandable. 2 (good): The dialogue is fluent, devoid of noticeable grammar errors, spelling mistakes, or vocabulary issues, presenting clear and comprehensible expression. Informativeness: 0 (poor): A dialogue lacking information, essentially falling into ‘safe response’ territory or consisting of repeated answers. 1 (normal): The dialogue offers some information but still lacks detailed or in-depth responses, requiring additional information for user understanding. 2 (good): The dialogue furnishes rich, detailed, and in-depth information, answering user queries while providing additional relevant content. Logical: 0 (poor): The dialogue features severe logical errors, responses or suggestions are unrelated to user queries or context, or there are blatant selfcontradictions. 1 (normal): The dialogue has some logical issues, with responses or suggestions insufficiently related or reasonable, though still understandable. 2 (good): The dialogue maintains logical coherence, with responses or suggestions logically related and reasonable in the context of user queries. Coherence: 0 (poor): The dialogue is highly incoherent, lacking clear contextual connections, with no logical
links between various parts of responses or suggestions. 1 (normal): The dialogue is moderately coherent, exhibiting some coherence but with occasional ruptures or insufficient logical connections between contexts. 2 (good): The dialogue is highly coherent, with clear logical connections between responses or suggestions, ensuring smooth transitions between contexts.
D.2 Prompts for LLM-based Baselines
D.2.1 Zero-shot and Fine-tuning Settings
Pretend you are a movie recommender system. I will give you a conversation between a human and assistant. Based on the conversation, you reply me with 50 recommendations without extra sentences. Here is the conversation: { }.
D.2.2 Zero-shot and Fine-tuning Settings with Historical Interactions
Pretend you are a movie recommender system. I will give you a conversation between a human and assistant and human’s history item lists. Based on the conversation and item lists, you reply me with 50 recommendations without extra sentences. Here is the item lists: { } and here is the conversation: { }.
D.2.3 Few-shot Setting Pretend you are a movie recommender system. I will give you a conversation between a human and assistant and 5 correct examples. Based on the conversation and examples, you reply me with 50 recommendations without extra sentences. Here is the examples: { }, and here is the conversation: { }.
D.2.4 Few-shot Setting with Historical Interactions
Pretend you are a movie recommender system. I will give you a conversation between a human and assistant, 5 correct examples and human’s history item lists. Based on the conversation, examples, item lists, you reply me with 50 recommendations without extra sentences. Here is the examples: { }, here is the item lists: {}, and here is the conversation: {}.

heading: E Case Study
text: Figure 6 shows an example of generating response with recommendation by ChatGPT-based model.

heading: Future Work
text: We leave the question of how to reduce human intervention in the process of goal design and template construction as a direction for future research. Additionally, the biases present in Amazon review dataset are prone to lead to the potential bias in the dataset construction process. The Amazon review dataset mainly contains two types of biases: user rating bias and review bias. User rating bias arises from different users having different rating standards, while the positive and negative interactions for dialogue generation are splitted based on a uniform standard. Review bias is mainly reflected in the content being polarized, with possible cases of exaggeration or depreciation. Dialogues generated based on such review content may also be
biased. Detecting bias in our dataset that is generated by the conversational LLM is non-trivial, mainly due to the diverse outputs. There is a need for more nuanced and sophisticated process that can correct user rating and review bias before the user behaviour based dialogue generation