Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-Enhanced Data Augmentation
Authors: Xi Wang
Hossein A. Rahmani
Jiqun Liu
Emine Yilmaz
Venue: EMNLP 2023 Findings (2023)
Keywords: Conversational Recommendation
Bias Mitigation
Generative Data
Data Augmentation
TL;DR: The use of synthetic recommendation dialogues with movie variants to address various biases within conversational recommendation as well as the introduction of additional evaluation metrics, joint with rich analysis.

Abstract: Conversational Recommendation System (CRS) is a rapidly growing research area that has gained significant attention alongside advancements in language modelling techniques. However, the current state of conversational recommendation faces numerous challenges due to its relative novelty and limited existing contributions.  In this study, we delve into benchmark datasets for developing CRS models and address potential biases arising from the feedback loop inherent in multi-turn interactions, including selection bias and multiple popularity bias variants. Drawing inspiration from the success of generative data via using language models and data augmentation techniques, we present two novel strategies, 'Once-Aug' and 'PopNudge', to enhance model performance while mitigating biases. Through extensive experiments on ReDial and TG-ReDial benchmark datasets, we show a consistent improvement of CRS techniques with our data augmentation approaches and offer additional insights on addressing multiple newly formulated biases.

Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-nnhanee Data Augmentation
Xi Wang Hossein A. Rahmani Jiqun Liut Emine Yilmaz'
University College London, London, UK *The University of Oklahoma, OK, USA
(xi-wang, hossein. rahmani 22, emine. wemiee.yllmaaa.............aae.e silma ac. uk jiqunliu@ou edu
Abstract
In-Context
LLMs
Conversational Recommendation System (CRS) is rapidly growing research area that has gained significant attention alongside ad- vancements language modelling techniques. However, the current state of conversational recommendation faces numerous challenges due its relative novelty and limited existing contributions. In this study, we delve into benchmark datasets for developing CRS models and address potential biases arising from the feedback loop inherent in multi-turn interactions, including selection bias and multiple popularity bias variants. Drawing inspiration from the success of generative data via using language models and data augmentation techniques, we present two novel strategies, 'Once-Aug' and 'PopNudge', to enhance model performance while mitigating biases. Through extensive experiments on ReDial TG-ReDial benchmark datasets, we show consistent improvement of CRS techniques with our data augmentation approaches and offer additional insights on addressing multiple newly formulated biases.
Learning Prompt
GPT-3.5
Aug Data
Movie Name
Model Train
Train Data
Aug Strategy
Figure 1: Data Augmentation Pipeline
Hence, in the development of conversational rec- ommenders, we identify two primary streams of research directions: optimising natural responses and improving recommendation results. While re- cent advancements in language models have proven effective in generating natural utterances (Wang et al., 2022), enhancing recommendation results based on user preferences inferred from their utter- ances remains a challenge. These include skewed data interactions with limited coverage and miss- ing evaluation based on the multi-turn interaction nature of CRS. In addition, due to the nascent na- ture of conversational recommendation techniques, there has been limited investigation into uncover- ing the potential biases within the feedback loop of their development. Left unchecked, the biases could lead to unfair information presentation, inef- fective explorations, and poor decisions.
1 Introduction
Conversational Recommendation System (CRS)
is a growing research topic and application area, along with the recent advance in Natural Lan- guage Processing (NLP) and Conversational tech- niques. In contrast to traditional recommendation approuches, which provide item suggestions non-interactive manner, conversational recommen- dation involves multi-turn and mix-initiative inter- actions between users and the system (Jannach and Chen, 2022). Hence, a growing body of research studies (Zhang et al., 20180 8b; Chen et al., 2019; Li et al., 2022) has introduced diverse conversational recommendation models, which address the natural language understanding from user utterances, user intent estimation, user preference estimation and generation of uppropriate responses that encapsu- late the recommended items.
To address this research gap. we first characterise various potential biases in conversational recom- mendation systems, including selection bias in data and popularity-orient oriented biases based on the multi- turn interaction nature of CRS. Through a detailed analysis of CRS models, we identify these biases and their impact. Subsequently, by leveraging the advancements in Large Language Models (LLMs) and in-context learning approaches, we introduce novel data augmentation techniques, namely Once- Aug and PopNudge, enhance the recommen- dation results and mitigate the effects of biases. Figure provides an overview of the data augmen- tation pipeline that we applied this study, which illustrates the generation of synthetic data followed

---

by the application of our novel augmentation strate- gies. These strategies enrich the training data, lead- ing to improved performance of CRS models. To validate the effectiveness of our data augmentation strategies, we conduct extensive experiments us- ing popular CRS baselines, such as KGSF (Zhou et al., 2020a) and KBRD (Chen et al., 2019), on two benchmark datasets: ReDial (Li et al., 2018) and TGReDial (Zhou et al., 2020b). The experimental results consistently demonstrate the performance improvement achieved by our data augmentation strategics, particularly PopNudge. Specifically, we evaluate the improvement based on recommenda- tion accuracy, selection bias as well as various appularity-orienter biases and effects.
tigated popularity bias (Zhao al., 2022; Naghiaei et al.. 2022a), which arises from the concern about the growing applications of recommendation sys- tems and the Matthew effect (cf. Wang et al., 2018) that reinforce interactions with popular items. Con- sequently, several strategies have been proposed to mitigate or alleviate bias effects in developing rec- ommendation techniques (Wang al., 2021; Liu et al., 2021; Naghiaei al., 2022b). For example, Wang et (2021) addresses selection bias by vali- dating the learned model on a small set of unbiased item ratings to enforce unbiased rating predictions.
In the context of Conversational Recommen- dation Systems (CRSs), which are the focus of this study, there a growing concern about biases within the feedback loop, but limited research has been conducted to explore and mitigate these bi- ases (Gao et al., 2021). Similar to the concerns regarding popularity bias in conventional recom- mendation systems, Lin et al. (2022a) and Fu et al. (2021) have investigated and contributed to mit- igating popularity bias in CRSs. In addition popularity bias, Shen al. (2023) have explored the impact unintended biases, such as racial and gender biases, on CRS recommendations. How- ever, existing studies have relied on semi-synthetic conversations generated using user-posted reviews and developed using the system ask-user response method (Zhang al., 2018b), rather than using real recomeendation-oriinntedd conversations to exam- inc and evaluate bias effects. As a result, research on the effect of potential biases on CRS develop- ment remains underexplored, lacking the use of real user-involved conversation data.
The main contributions of this paper are three- fold: (I) a comprehensive investigation into poten- tial biases and effects affecting the performance of CRS models, (2) novel data augmentation strate- gies for improving CRS models from multiple per- spectives, and (3) extensive experiments demon- strating the effectiveness of the proposed data aug- mentation strategies and providing additional in- sights with discussion novel biases and effects.
2 Related Work
This section offers an overview of the biases from data collection and the development of recommen- dation systems to the conversational recommen- dation system. In addition, we further discuss the existing bias mitigation strategies that leverage gen- erative data or so-called data imputation.
2.1 Biases in Recommendation System
A recommendation model is designed to suggest items of high interest to given user based on their preferences learned from historical interactions and associated information in the collected data. How- ever, when examining the feedback loop involving the interactions between the user, data, and the model, it becomes apparent that recommender sys- tems can suffer from various biases (Chen et al., 2022; Mehrabi et al., 2021; Wang et al., 2023; Salu- tari et al., 2023; Liu, 2023). For example, biased interactions between users and items, known as se- lection bias EHernández-Lobate et al., 2014; Steck, 2013)) or the biased representation of items (i.e.. ex- posure bias (Liu al., 2020)) can result skewed data distribution that makes challenging to accu- rately capture user interests and ensure fair item representations. In particular, the commonly inves-
2.2 Generative Data for Bias Mitigation
In previous literature, one commonly applied so- lution address biases caused by missing data is data imputation (Hernández-Lobato et al., 2014; Steck, 2013). Data imputation estimates and gen- erates pseudo-labels or ratings for users' unseen items, aiming to mitigate the selection bias, result- ing from unfair data collection strategies. However, the performance of bias mitigation strategies based on data imputation is hindered by the issue of im- putation inaccuracy, which introduce larger biases (Wang al., 2019). In the field of machine learning, generative neural models have advanced significantly, and the use of the generative data has become widespread for developing debiased training data (Wu et al., 2022; Lee et al., 2021).

---

Consequently, generative data holds the potential mitigate biases in CRS models but has not been fully explored in previous experiments.
3.2 Selection Bias for CRS
In the existing literature on conventional recom- mendation systems, selection bias is a widely exam- ined phenomenon (e.g. Ovaisi et al., 2020; Wang et al., 2021; Liu et al., 2022). It highlights that users' behaviour in observed user-item interactions, as captured in the collected model development datasets, is often skewed, with users primarily in- teracting with only a small subset of items (Chen et al., 2022). Consequently, the items that are in- teracted within these datascts do not represent the full spectrum of available items, thereby limiting the ability of the learned model to make promis- ing recommendations and can also exaggerate the impact of other biases, such as the popularity bias. However, the selection bias in CRSs datasets has not been thoroughly examined and discussed.
3 Bias Analysis, Formulation and Mitigation for CRS
Here we formally define the conversational rec- ommendation task and its associated components, susceptible various biases originating in the feed- back loop. We then propose to consider the im- pact of multiple biases, and account for the unique characteristics of CRSs. Finally, we introduce two novel data augmentation strategies, Once-Aug and PopNudge, to mitigate the impact of biases.
3.1 Problem Statement
In this study, we address this gap by examining and statistically revealing the presence of selection bias in conversational recommendation datasets, focusing their Initial Item Coverage (IIC). That is formulated follows:
A Conversational Recommendation System (CRS) is an extension of conventional recommendation systems that enables interactive and personalised recommendations through multi-turn interactions and user feedback collection. The goal of CRS is to effectively understand users' personal prefer- ences and immediate intent for item exploration in order to provide natural responses that include accu- rate and personalised recommendations. Formally, at specific turn I, for given user among M users (i.e., U {uo, U1, ..... uM }), CRS model utilises information from the previous conversa- tional utterances (i.e., d {co, C1, C22 .... ct-1)) ) from a dialogue corpus D, to generate ranked list items r; from an item corpus / = {io. i1,.... in) and formulates a natural language response a, In this study, we introduce the concept of a "conver- sational episode", dº inspired by reinforcement learning, which represents the conversations from when user initiates conversation until the recom- mendation is accepted by the user or the maximum interaction turns of the agent are reached (Sun and Zhang, 2018; Chu et al., 2023). The index e denotes a specific episode. Consequently, we investigate the bias effects on conversational recommendations at different levels, including individual turns of rec- ommendations (r:), recommendations based on the context of a conversational episode (r*), recom- mendations considering historical conversations Ti|eo previous episodes rele and corpus-level bi- ases (rp). By examining the bias effects from var- ious perspectives, we analyse the advantages and potential impacts of the dynamic nature of CRSs.
IIC = |Tyaain
(1)
[7]
where |- represents the number of unique items in the corresponding set. By calculating the IIC, we can assess the extent to which the training dataset covers the entire item space, revealing the presence and magnitude of selection bias in CRS datasets.
3.3 Popularity Bias
Besides selection bias, popularity bias also a widely studied aspect when evaluating recommen- dation systems (Chen et al., 2023). Popularity bias arises from the tendency of users to interact more frequently with popular items, resulting in a skewed distribution where popular items receive more at- tention. This bias lead to reduced coverage, diversity, and potentially lower user satisfaction lev- els in personalised recommendation results (Zhao et al., 2022; Abdollahpouri et al., 2017). Over time, can also contribute to the Matthew effect, further amplifying the popularity gap between items.
A recent formalisation of popularity bias in the context of CRSs considers the recommended item list Ta, turn in dialogue (Lin et al., 2022a). The bias effect quantified by the multiplication of two components, a ranking utility that assigns weights to items based on their ranking positions and a popularity coverage measure that indicates

---

the proportion of popular items included in the recommendation. The ranking utility x(ra,)) is calculated as:
recommendations and the ones from the previous episodes. Diverse popularity distributions among recommended items are expected due to the differ- ent target items in each episode. A lower coeffi- cient score indicates a weak correlation between previous episodes and current recommendations, resulting in highly diversified recommendations within conversation interaction.
T(i E ( (n)) (ra.)) log(rank(i) +1
(2)
here, r (1) represents the set of popular items based on a popularity threshold 7 determined by the items' interaction frequency. The popularity coverage Pre., is calculated as:
3.3.2 User Intent-Oriented Popularity (UIOP) In addition to the CEP effect discussed earlier, we emphasize the importance of CRS model in ef- fectively capturing user intent, whether it leans towards popular items or specific preferences. By identifying and addressing user intent, CRSs can offer tailored suggestions that increase user satisfac- tion. To quantify the user intent- popularity (UIOP) effect, we leverage the popularity of target items (pop(ia:)) and formulate it as follows:
P(rux)= card(ru, (1))
card(ru)
where card(.) indicates the cardinality of set. The above formation, well the existing approaches in the literature (Fu al., 2021; Lin al., 2022b), overlook the influence of contextual user-system interactions and the varying user intent during a conversation. Considering the multi-ture interaction nature of CRS is crucial for enhancing the user experience. In this study, we extend the formalisation of popularity bias by analysing novel factors, including cross-episode popularity and user intent-oriented popularity effects.
EuSopIaá = (pop(ta,) (ra..) (4)
This equation measures the proximity between the popularity of the user's target item and the recom- mended items. A higher score is obtained only if the recommendations deviate from the target items. By comprehensively evaluating multiple poten- tial biases and effects in the development of CRSs, we enable a comprehensive assessment of conver- sational recommendation results. These measures promote the advancement of CRSs that prioritize fairees and unbiased recommendations, well as boosted user experience.
3.3.1 Cross-Episode Popularity (CEP)
Regarding the CEP effect, we assume that a rational CRS model should provide relatively independent suggestions within each individual episode of con- versational recommendations This assumption is supported by an example from an existing CRS benchmark dataset, ReDial, which is provided in the Appendix (see Figure 7). In essence, during multi-turn conversations, users often seek multiple recommendations, and we observe that the pref- erence modelling for accurate recommendations tends to be relatively independent. Therefore, con- sidering the dynamic preferences of users, be- comes crucial to capture their immediate interest within each episode, which may involve both pop- ular and less popular items, to make appropriate suggestions and enhance users' satisfaction levels. To explicitly model the CEP effect, we extend the formulation of popularity bias as follows:
3.4 Bias Mitigation via Generative Recommendation Dialogues
In the preceding discussion, have identified several biases that can impact the performance of learned CRSs, which could cause an imbalanced training corpus. mitigate the bias effects, we pro- pose novel data augmentation approaches, joined with the generation of synthetic dialogues.
Synthetic Data Generation. In our generation strategy, inspired by the recent success of in- context learning on LLMs, we develop effecttve prompts' as input the recent advanced GPT3.5 model (cf. Brown et al., 2020) to successfully gen- crate synthetic movie recommendation dialogues. To control the frequency of item mentions, we use specific movie names as variables and ask the LMs
(r(r)) Prs. (p(pop(pp))))))).mm.))
poptr 1))
where -) represents the absolute value of the Pearson correlation coefficient between the pop- ularity levels (denoted as pop(-), which nor- malised score based on item frequency) of current
Detailed prompt variants can he found in Appendix (sce Figures 5 and 6).

---

generate complete conversations about suggest- ing those movies. The generated dialogues are then reformatted align with the input instances in the chosen benchmark.
Table A statistical comparison among conversational and popular coneeevvesaaio datasets. The number of dia- logues and ratings are counted for conversational and non- conversational datascts, respectively. Popular items are the ratio of items with more than interactions.
Dataset
Train
Valid
Test
Items
IIC
Popular Items
Data Augmentation Strategies. We propose two data augmentation strategies to enhance model per- formance and mitigate multiple biases when train- ing CRS models using the available synthetic di- alogues. Previous studies, as discussed in Sec- tion 2.1, have not examined the bias effect in publicly available conversational recommendation datasets. Additionally, these studies faced chal- lenges related to generalizability, either due to the attribute-basse based definition of popularity (Li et al., 2018) or the specific formulation for the Bayesian Pairwise Ranking loss (Lin et al., 2022a).
Conversational Recommendation Datacets
ReDial
8,103
900
1,341
6,112
72.75%
26.55 %
TGReDial 8,494
34, 82%
3.74%
Onn-Conversational Recommmecddatin Datasets
ML-100k
100,000
1,682
100%
80,20%
M1.-20m
20,000,263
62,423
94.59%
52.41%
potential relationship between popularity and rat- ings (Zhao et al., 2022). Algorithm 1 outlines the procedure for applying PopNudge. augmenting the model's training batches with dialogues featuring less popular items. For every batch, we initialise an empty augmentation set (line 4). Next, for each dialogue instance, identify its included item and prepare the of available items with lower popularity (line 6). Then, we perform weighted sampling to select k: items, in addition to the aug- mentation set, based on item popularity (line 7). The resulting sampled items are finally augmented in the training corpus for model learning (line 1 I1). PopNudge offers several ad advantages despite its simplicity. First, it maintains the long-tail distri- bution of item mentions, which consistent with natural human behaviour. This preserves the over- all item distribution pattern. Secondly, it effectively increases the exposure of less popular items, lead- ing to a more balanced item distribution. Impor- tantly. PopNudge seamlessly integrates with exist- ing methodologies without requiring any model updates. It selectively incorporates augmented data for model training. making it applicable in various contexts for bias mitigation purposes.
Algorithm 1 PopNudge Data Augmentation
Require: Dirain: Daug
1: POPaug t- pop( (op(Da
Item popularity
2: for batches do
3: Derain batch-sample Deroin
4: Doarpple 0
5: for d, E birain do
6:
Diet, + Daug - {dj.popj > popi) lagg
Daample += W WS(Dree aug: POPnggg! k)
7:
8:
D WSSC,WV W, k) is weighted sampling items from corpus C with weights W.
9:
10: end for
11: Dirain + Dirain Daag sample
12: model update
13:
end for
In this study, we propose two data augmentation strategies. The first strategy, called 'Once-Aug' (OA), involves adding all synthetic dialogues to the training data, evenly increasing the exposure of items in the corpus. However, Once-Aug does not consider item popularity or allow control of the presence of different items. To address these issues and model wider range of users' preferences, we introduce the 'PopNudge' strategy.
Experimental Setup
Our objective is improve the evaluation of CRSs techniques based on various bias effects while vali- dating the performance of our proposed data aug- mentation approaches. To achieve this, we conduct a series of experiments using recent CRS models on two benchmark datasets: ReDial (Li et al., 201 8) and TGReDial (Zhou et al., 2020b). These experi- ments allow us access the recommendation per- formance and bias effect on existing approaches and determine the effectiveness of our proposed approach. Table presents a statistical summary of the two datasets, comparing them to the popular MovieLens (ML) datasets. The comparison reveals
PopNudge (PN) inspired by the data-agnostic data augmentation strategy MixUp (Zhang et al., 2018a), which augments training data with 'vicin- ity" instances improve the model's ability to han- dle unseen inputs. Similarly, PopNudge augments training batches with dialogues recommending sim- ilar but less popular items, aiming to 'nudge' the model towards bias mitigation and avoid the fre- quent use of lower-rated items by assuming the

---

0.05
Basic
0.175
Basic
0.25
Basic
QA
0.150
QA
PK-K1
Dopularity Blas
0.20
PR-K5
PALKS
PR--3
PN-K10
Nitto
0.100
0.15
PN-K10
PN-K5D
0.02
0.30
0,050
0.01
0.025
0.000
ReDial
KGSF
KBRD
TGReDial
KGSF
TGReDial
Rebial
KGSF
KBRD
TGReDial
Figure 2: Performance comparison between conversational recommendation models using Hit@ 10 and 50, together with the scores of popularity bias on the ReDial dataset. OA and PN refer to the Once Aug and PopNudge data augmentation strategies.
0.010
Basic
Basic OV
1.2
QA
PN-KS
0.025
PN-KS
1.0-
PN-K1
PN-K30
Niteso
0.020
PN-K10
Popularity
0.8
PN-KLO
0.015
0.004
0.4
0.002
0.005
0.2
0.000
0.000
0.0
ReDial
KGSF
KBRD
TGReDial
Figure 3; Performance comparison between conversational recommendation models using Hit@ 10 and H0,@50, together with the scores of popularity bias the TGRcDial dataset.
Evaluation of Initial Item Coverage two datasets Table 2:
Bacc
and after applying data augmentation.
PIL CI PIU S
Dutuset
ReDial
TGReDial
500
PM PM C53
<13 Brwounnn
Ph, <53
Basic DA
72.75%
34.82%
300
1006
T000%
2:0
PX-Li
98.11%
666.3-9%
PN-45
100%
97.81%
6
1060
730 2000 <600
720
6000
530 133013416-0000000 J60EE
PN-150
100%6
100%
Figure 4: Mitigated Long-tail effect after applying PopNudge.
that both ReDial and TGReDial datasets are sub- ject to selection bias, as indicated by their relatively low initial item coverage (IIC) values. For instance, only 72.75% and 34.82% of items are mentioned in the context of training dialogues in the ReDial and TGReDial datasets, respectively. Furthermore, when compared non-conversational recommen- dation datasets, the benchmark conversational rec- ommendation data contains significantly fewer in- teraction data. This finding further emphasizes the importance of data augmentation in addressing these limitations.
Pedia) and employs self-attention mechanism to learn entity representations for recommendations. 3) Re-Rl (Zhou et al., 2020b) uses the pre- trained BERT (Kenton and Toutanova, 2019) and the sequential recommender SASRec (Kang and McAuley, 2018)) to model the historical utterances and user interactions, respectively. 4) KGSF (Zhou et al., 2020a) leverages both entity-oriented and word-oriented to enrich data representations and en- hance item recommendations The implementation and evaluation of these models are supported by the open- CRSLab (Zhou et al., 2021) toolkit?, For recommendation accuracy evaluation, low (Zhou et al., 2020b) and apply set of com- monly used evaluation metrics, including Hit Ratio (HIT), NDCG and MRR with cutoffs and 50. To ensure fair performance comparisons, our data
In our analysis, we consider several commonly used CRS models that serve baselines in the lit- crature. We summarise the recommendation com- ponent of these models: ReDial (Li et al., 2018), an autoencoder conversational recommender and uses sentiment-analyeee user opinions as input. 2) KBRD (Chen et al., 2019), links entities dia- logue content to an external knowledge graph (DB-
fol-
'Implementation is availahle via: https: htt.ttggttttb.ooo com/ wangxieric/Bias-CC

---

augmentation strategies solely enrich the training data and do not modify the validation and test data, Specifically, for the PopNudge strategy, we explore augmentation effects with varying numbers sam- pled dialogues (k) ranging from 10,5,)
Model
ReDial
TGReDial
CEP
UTOP
CEP
UIOP
ReDial
0.2000
0.5827
0.0994
0.6553
Once-Aug
0.0411
0.7693
0,0607
0.2354
PopNudge kl
0.0524
0.7111
0.0464
0.7482
PopNudge_ A5
0.1195
0.6858
0.0412
0.7516
PopNudge_
0,0778
0.8123
PopNudge A50
0.0119
0.7976
0.0228
0.7811
5 Result Analysis
KGSP
0.0658
0.6891
0.4301
0,4861
Once-Aug
0.0743
0.6694
0,3860
0,5069
In this section, we present and discuss the experi- mental results to illustrate the effectiveness of ap- plying Once-Aug and PopNudge to baseline mod- els while examining the recommendation accuracy, the effect on selection and popularity biases.
PopNudge_ kl
0.0670
0.6848
0.4686
0.6325
PopNudge AS
0.0710
0.6616
0.4996
0.6574
PopNudge__ A10
0.0042
0.6733
0,5099
0.6142
PopNudge__ss
0.0650
0.6856
0.6673
0.6123
KBRD
0.0574
0.7058
0,4092
0.4949
Once Aug
0,0758
0.6750
0.4033
0.5097
PopNudge__ All
0.0595
0.6989
0.3405
0,4558
PopNudge AS
0.0684
0.6832
0.4966
0.5681
5.1 Recommendation Accuracy Evaluation
PopNudge_ k10
0.0657
0.6903
0.6563
0.8557
PopNudge D.0659
0.6900
1.0047
For conversational recommendation models, accu- rately estimating users' preferences and providing appropriate recommendations are critical. There- fore, as discussed in Section 4, in line with the existing literature, we evaluate the recommenda- tion performance using Hit ratio, NDCG and MRR. Figures 2 and depict the results of Hit@ 10 and Hit@ 50 on the ReDial and TiReDial datasets. They show that our data augmentation strategies consistently improve the Hit ratio scores, indicat- ing enhanced recommendation accuracy. However, Once-Aug improves recommendation accuracy the ReDial dataset but does not guarantee improve- ment the TGReDial dataset. In contrast, Pop- Nudge consistently enhances the performance of bascline CRS models on both datasets. This conclu- sion also supported by the complete experimen- tal results, joined with the evaluation using NDCG and MRR, in the Appendix (Tables 4 and 5). Thus, we conclude that PopNudge is an effective data augmentation strategy that consistently enhances recommendation accuracy with the increased expo- sure of available items across all four CRS models on both benchmark datasets.
TGRedial
0.0527
0.7047
0.4565
Once-Aug
0,0529
0.7039
0.8876
0.5437
PopNudge__k All
0.0519
0.7076
0:4626
0.4654
PopNudge__K A5
0.0554
0.7009
0.3584
0.4116
PopNudge_ A10
0.0603
0.6876
0.3724
0.4218
PopNudge__A0 0.0564
0.6972
0.3076
0.3719
Table 3: Evaluation of Cross- -piss-E Popularity (CEP) and
User Intent- Oriented Popularity (UIOP) effects on models and after applying data augmentation.
observe that Once-Aug easily achieves 100% cov- erage of all available items with equal additional exposure, while the PopNudge approach gradually increases item coverage with a growing number of sampled dialogues. Furthermore, Figure 4 demon- strates the frequency of items mentioned in the training dialogues before and after applying the PopNudge approach. It shows that the varying number of sampled dialogues (kt) aligns with our objective discussed in Section 3.4, which aims to adjust item exposure without disrupting the long- tail distribution of natural user-item interactions. Specifically, increasing the number of sampled di- alogues includes more items, with higher fre- quency of popular items and gradual inclusion of less popular items. Hence, we conclude the ef- fectiveness of both data augmentation strategies in addressing the selection bias, and PopNudge is well- peel-perfor in gradually mitigating the selec- tion bias while retaining the long-tail distribution of user-item interactions.
5.2 Analysis on Selection Bias
According to the statistical summary of benchmark datasets in Table 1, CRS models are affected by selection bias in the training data. In particular, the initial CRS datasets (ReDial and TGReDial) only cover 72.75% and 34.82% of the available items, which hinders the accurate estimation of users' preferences. Therefore, we aim compar- atively evaluate the effectiveness of our data aug- mentation strategies in mitigating selection bias. Table 2 presents the calculated IIC scores after ap- plying various data augmentation strategies. We
5.3 Analysis on Popularity Bias
Next, in this study, we extensively discuss the value of investigating the popularity bias and propose ad- ditional Cross-Episode Popularity (CEP) and User Intent-Orient Oriente Popularity (UIOP) to further exam- ine the effectiveness of CRS models towards the

---

corresponding perspectives. At first, in Figure and 3, we depict the scored popularity bias of sev- eral CRS models before and after applying our data augmentation strategies on ReDial and Dill datasets. The exact scores are also available in the Appendix (see Tables 6 and 7). According to the experimental results, we observe that a lower popularity score does not necessarily leads to higher recommendation accuracy. For example, for the experimental results on the ReDial dataset, among the four CRS models, KGSF is the best per- formed with @it 10 and 50 scores at 0.0414 and 0. 1496, respectively. However, it is measured with a higher popularity bias score (i.e.. 0.1382) than both KBRD (0. 240)) and TGReDial (0.1312). Therefore, a marginal increase in having popular items could potentially benefit the improvement of the recommendation performance of CRS models. This also has been validated in the literature (Zhao et al., 2022). In addition, as for the performance of our data augmentation strategies, we observe two main findings: (1) PopNudge has the potential to improve the model's performance when model extremely suffers from popularity bias. This can be found in the improved recommendation accu- racy but significantly lower popularity bias of the ReDial model on the ReDial dataset. (2) The in- creasingly sampled dialogues do not significantly increase the impact of popularity biases and can also improve the model with higher recommen- dation accuracy. In contrast, Once-Aug is rather unstable, such as the examined performance of the ReDial model on the TGReDial dataset.
approach. By comparing the scores of Hit ratio and CEP, is likely that the ReDial model suf- fers from a repetitive recommendation of similar items, which not only lowers the recommendation accuracy but can also harm users' experience.
At last, examine the UTOP scores, which examine if the predictions are presenting close pop- ularity with the target items. Similar the find- ings of CEP scores, we also observe a consistent improvement towards lower UIOP scores after ap- plying our bias mitigation strategies. In addition, we also observe that UIOP has a certain connec- tion with recommendation accuracy. For example, KGSF boosted by PopNudge with 5 sampled di- alogues is the best-performed variant per ommendation accuracy among the basic and data- augmented models. Meanwhile, we also observe the lowest UIOP score for the identical model vari- ant. Therefore, UTOP has the potential to serve as a metric in evaluating recommendation accuracy and joined with the consideration of item popularity.
In summary, when examined by various popularity- measures and effects, our intro- duced data augmentation strategies, especially Pop- Nudge, can effectively mitigate various biases and boost the models' performance in serving diversi- fied and accurate recommendations. In addition, the introduction of the CEP and UIOP effects adds additional insights to the performance evaluation of CRS techniques and can also be leveraged in the future development of CRS models.
Conclusions
On the other hand, we also extend the evaluation of popularity bias with two additional measures: CEP and UIOP. In Table 3, we share the full ex- perimental results when evaluated by the CEP and UTOP scores. Accordingly, we observe that by ap- plying our data augmentation strategies, we can lower the CEP scores in most cases apart from KBRD on the ReDial dataset. As discussed in Section 3, the CEP score can also reflect the diversity of recommendation results across conver- sational episodes. Hence, the observed lower CEP scores also indicate the value of our data augmenta- tion strategies in diversifying the recommendations, which is promising in improving users' experience. Specifically, according the experimental results in Figure 2, the initial ReDial model does not per- form well on the ReDial dataset with high accuracy, which is significantly improved by our PopNudge
Our study investigates several biases within the feedback loop of developing conversational rec- ommendation system, including selection bias, pop- ularity bias, cross-episode popularity (CEP) effect and user intent-oriented popularity (UIOP) effect. These proposed biases and effects shed light on CRS model performance and provide valuable in- sights for effective model development. Addition- ally, we leverage the recent advance of large lan- guage models to generate effective synthetic data and propose novel and effective data augmenta- tion strategies, Once-Aug and PopNudge, to en- hance model performance. Our experimental sults demonstrate the effectiveness of data aug- mentation strategies in improving CRS model per- formance while considering the recommendation accuracy as well as the mitigation across selection bias and popularity biases.

---

Limitations
Zhendong Chu, Hongning Wang. Yun Xiao, Bo Long. and Lingfei Wu, 2023. Meta policy learning for cold- start conversational recommendation. In Proc. of WSDM.
this paper, we make significant contributions by utilissing generative dialogues to augment the training data and mitigate biases. However, we ac- knowledge two main limitations of our approach. Firstly, our generative dialogues only involve sin- gle item per dialogue, which deviates from the ma- jority of dialogues in the initial training corpus and may impact cross-episode recommendations. Sec- ondly, we primarily focus on improving recommen- dation performance and overlook the evaluation of responses, an essential aspect of conversational techniques. Addressing these limitations will re- quire further dedicated efforts in future work.
Zuohui Fu, Yikun Xian, Shijie Geng, Gerard De Melo, and Yongfeng Zhang. 2021. Popcorn: Human-in-the loop popularity debiasing in conversational recom- mender systems. In Proc. of CIKM.
Chongming Gao, Wenqiang Lei, Xiangnan Hc, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and challenges in conversational recommender systems: A survey. A/ Open, 2.
José Miguel Beemánddez-Lbbtoo, Neil Houlsby, and Zoubin Ghahramani. 2014. Probabilistic matrix fac- torization with non-random missing data. Proc. of ICML.
A Acknowledgements
Dietmar Jannach and Li Chen. 2022. Conversational recommendation: A grand ai challenge. Al Maga- zine, 43(2):151-163.
This research is supported by the Alan Turing In- stitute under the EPSRC grant [EP/N510129/1] 10129/1], the EPSRC Fellowship titled "Task Based Infor- mation Retrieval" [EP/P024289/1] and the Engi- neering and Physical Sciences Research Council [BP/S021566/1], Jiqun Liu's participation in this project is partially supported by the National Sci- ence Foundation (NSF) Award IIS-2106152 and Junior Faculty Fellowship Program Award from the University of Oklahoma Office of the Vice Presi- dent for Research and Partnerships.
Wang-Cheng Kang and Julian McAuley. 2018. Self- attentive sequential recommendation. In Proc. of ICDM.
Jacob Devlin Ming-Wei Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre- Pre-training of deep bidirec- tional transformers for language understanding. Proc. of NAACL-HLT.
Minwoo Lec, Seungpil Won, Juae Kim, Hwanhee Lee, Cheoneum Park, and Kyomin Jung. 2021. Crossaug: A contrastive data augmentation method for debias- ing fact verification models. In Proc. of CIKM.
References
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommenda- tions. In Proc. of NeurlPS.
Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2017. Controlling popularity bias in learning-to-rank recommendation. Proc. of Rec- Sys.
Shuokai Li, Ruobing Xie, Yongchun Zhu, Xiang Ao,
Fuzhen Zhuang, and Qing He. 2022. User-centric conversational recommendation with multi-aspect user modeling. In Proc. of SIGIR.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et 2020. Language models are few-shot learners. Proc. of NeurIPS.
Allen Lin, Jianling Wang. Ziwei Zhu, and James Caver- lec, 2022a. Quantifying and mitigating popularity bias in conversational recommender systems. In Proc. of CIKM.
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng. Meng Wang. and Xiangnan He. 2023. Bias and de- bias recommender system: A survey and future directions. ACM Transactions on Information Sys- tems, 1133)11-39.
Allen Lin, Ziwei Zhu, Jianling Wang, and James Caver- lee. 2022b. Towards fair conversational recom- mender systems. CoRR, 0bs/2208.03854.
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan Hef. 2022. Bias and debias recommender system: A survey and fu- ture directions. ACM Transactions on Information Systems.
Dugang Liu, Pengxiang Cheng. Zhenhua Dong, Xi- uqiang He, Weike Pan, and Zhong Ming. 2020. A general knowledge distillation framework for coun- terfactual recommendation via uniform data. In Proc. of SIGIR.
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019. To- wards knowledge-based recommender dialog system. In Proc. of EMNLP-IJCNLP.
Haochen Liu, Da Tang, Ji Yang, Xiangyu Zhao, Hui Liu, Jiliang Tang, and Youlong Cheng. 2022. Rating distribution calibration for selection bias mitigation in recommendations. In Proc. of WWW.

---

Haochen Liu, Da Tang. Ji Yang, Xiangyu Zhao, Jiliang Tang, and Youlong Cheng. 2021. Self-supervised learning for alleviating selection bias in recommen- dation systems. In Proc. of IRS.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. 2021. Combating selection biases in recommender systems with a few unbiased ratings. In Proc. of WSDM.
Jiqun Liu. 2023. A Behavioral Economics Approach to Interactive Information Retrieval: Understand- ing and Supporting Boundedly Rational Users, vol- ume 48.
Yifan Wang, Weizhi Ma, Min Zhang. Yiqun Liu, and Shaoping Ma. 2023. A survey on the fairness of recommender systems. ACM Transactions on Infor- mation Systems, 41(3):1-43.
Yuxiang Wu, Matt Gardner, Pontus Stenetorp. and Pradeep Dasigi. 2022. Generating data to mitigate spurious correlations in natural language inference datasets. Proc. of ACL.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. sur- vey on bias and faimess in machine learning. ACM Computing Surveys (CSUR), 54(6):1-35.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 8a. mixup: Beyond empirical risk minimization. In Proc. ICLR.
Mohammadmehdi Naghiaei, Hossein A Rahmani, and Mahdi Dehghan. 2022a. The unfaimess popularity bias in book recommendation. In Proc of BIAS.
Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang. and Bruce Croft. 2018b. Towards conversational search and recommendation: System ask, user re- spond. In Proc. of CIKM.
Mohammadmehdi Naghiaci, Hossein A Rahmani, and Yashar Deldjoo. 2022b. Cpfair: Personalized con- and producer fairness re-ranking for recom- mender systems. In Proc. of SIGIR.
Zihao Zhao, Jiawei Chen, Sheng Zhou, Xiangnan He, Xuezhi Cao, Fuzheng Zhang, and Wei Wu. 2022. Popularity bias is not always evil: Disentangling benign and harmful bias for recommendation. IEEE Transactions on Knowledge and Data Engineering.
Zohreh Ovaisi, Ragib Ahsan, Yifan Zhang, Kathryn Vasilaky, and Elena Zheleva. 2020. Correcting for selection bias in learning-to-ran to-nkkkktt-a-kkk--- systems. In Proc. of WWW.
Kun Zhou, Xiaolei Wang. Yuanhang Zhou, Chenzhan Shang, Yuan Cheng, Wayne Xin Zhao, Yaliang Li. and Ji-Rong Wen. 2021. Crslab: An open-source toolkit for building conversational recommender sys- tem. In Proc. of ACL-IJCNLP.
Flavia Salutari, Jerome Ramos, Hossein Rahmani, Leonardo Linguaglossa, and Aldo Lipani. 2023. Quantifying the bias of transformer-based language models for african american english in masked lan- guage modeling. In Proc. of PAKDD, pages 532-543.
Kun Zhou, Wayne Xin Zhao, Shuging Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020a. Im- proving conversational recommender systems via knowledge graph based semantic fusion. In Proc. of KDD.
Tianshu Shen, Jiaru Li, Mohamed Reda Bouadjenek. Zheda Mai, and Scott Sanner. 2023. Towards un- derstanding and mitigating unintended biases in language model-driven conversational recommen- dation. Information Processing & Management, G0(1): 103139.
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. 2020b. Towards topic- guided conversational recommender system. In Proc. COLING.
Harald Steck. 2013. Evaluation of recommendations: rating-prediction and ranking. In Proc. of RecSys.
Yueming Sun and Yi Zhang. 2018. Conversational rec- ommender system. In Proc. of SIGIR.
Hao Wang, Zonghu Wang. and Weishi Zhang. 2018. Quantitative analysis of matthew effect and sparsity problem of recommender systems. In Proc. of ICC- CBDA), pages 78-82.
Lingzhi Wang, Huang Hu, Lei Sha, Can Xu, Daxin Jiang, and Kam-Fai Wong. 2022. Recindial: A unified framework for conversational recommenda- tion with pretrained language models. In Proc. of AACLL-AACL.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. 2019. Doubly robust joint learning for recommen- dation on data missing not at random. In Proc. of ICML.

---

A In-Context Learning Prompts for Dataset Generation
You help in generating mixiinitiative synthetic multi-turn conversational recommendation dialog, each utterance starts with [user] or [system). by leveraging mentioned movie input.
Example movie: [The Shawshank Redemption]
Dialogue:
[user]: I'm looking for a good movie. Any suggestions?
In-Context Prompt
[system]: Absolutely! highly recommend The Shawshank Redemption. It's critically acclaimed film known for its powerful storytelling and exceptional performances.
[user]; Oh, thinn I've heard of it. What's 's it about?
[system]: The Shawshank Redemption tells the story of Andy Dufresne, banker wrongly convicted of murder and his experiences in Shawshank State Penitentiary. It's compelling tale of hope, friendship, and the strength of the human spirit.
[user]: That sounds intriguing. Is it drama more of thriller?
[system]: It's primarily drama, but has elements of both genres. The film balances intense
moments with emotional depth, creating captivating and thought-prowoking experience. [user]: Great! I'll give watch then. Thanks for the recommendation! [system]: You's welcome! I'm sure you'll Il enjoy "The Shawshank Redemption." you have any more questions or need further recommendations, feel free to ask, Enjoy the movie!
Input
Input: [movie name]
Figure In-Context Learming Prompts generating synthetic recommendation dialogue for augmenting ReDial dataset.
# ml
#
# 1
:
[AAA R -00
In- In-Context Prompt
[A B 1
21
[ H 2f
STI (+EEA) B A
[A 7
1 11
XI 16 3T.
XI
32 99 K 3t fR
1
5
A
FS #2
Input
:
Figure 6: In-Context Learning Prompts for generating synthetic recommendation dialogue for augmenting TG-ReDial dataset. TiiReDial includes dialogues in the language of Chinese, which leads to the use of Chinese prompt.

---

B Dialogue Example for Motivation
Example dialogue from the ReDial dataset
Conversation ID: 20001
[seeker]: Hi, am looking for a movie like super troopers.
[recommender]: You should watch Police Academy. [seeker]: Is that a great one? have never seen have seen @192131.
Episode 1
[seeker]: mean American
frecommender]: Yes, Police Academy very funny and so Police Academy 2: Their First Assignment. [seeker]: It sounds like need check them out.
Irecommender]: Yes, you will enjoy them.
Episode 2
[seeker]: appreciate your time. will need to check those out. Are there any others would recommend?
[recommender]: Yes, Lethal Weapon.
[seeker]: Thank you. I will watch that too.
Episode 3
Irecommender]: And, also Beverly Hills Cop. [seeker]: Thanks for the suggestions.
[recommende]]: You are welcome. [recommender]: And, also 48 Hrs. [seeker]: Thanks, goodbye.
Episode
Note: This dialogue re-formatted from Text-only for improved readability.
Figure 7: Dialogue example ReDial dataset, which shows the rather independent episode-wise interaction between user and system, for the motivation of investigating cross-episode popularity effect.

---

C Collection of Full Experimental Results
Table 4: Initial Experimental results of conversational recommenders on the ReDial dataset.
Hit@50
NDCG@10
NDCG@50
ReDial
0.0032
0.0262
0.0010
0.0061
0.0004
0.0016
Once Aug
03006
0.0444
6X0042
0.0141
0.0033
6.0061
PopNudge
0.0129
0.0M0
0,0067
0,0126
0.0088
0.0039
PopNudge
0.0220
0.0584
0.1033
0.0179
0.0067
0.0081
PopNudge k10
0,0219
0,0517
0.0107
0.0172
0.0072
0.00KS
PopNodge__ae 450
0.0017
0.0109
0.0006
0.0026
0.0002
0.0007
KGSF
0.0414
0.1496
0.0229
0.0476
0.0168
0.0227
Once Aug
0.0467
0.1658
000257
0.0529
0.0189
0.0252
PopNudge__i
0.4227
0.1554
0,0238
0.0196
0.0176
0.0237
PopNudge
0.0254
PopNudge k10
0.0427
0.1569
0.0235
0.0493
0,0173
0.0232
PopNodge_k50
0.0394
0.1496
0.02188
0.0470
0,0161
0.0220
KBRD
0.0387
0.1471
0.0209
0.0459
0.0151
0.0211
Onne Aug
0.0514
0.1749
0.0287
0,0569
0,0212
0.0278
PopNudge_M k1
0.0457
0.1621
0.0254
0.0518
0,0187
0.0249
PopNudge_
0,0484
0.1631
0,0272
0,0536
0.0202
0.0265
PopNudge
0.0257
PopNudge k50
0.0483
0.1599
0.0272
0.0826
0.0202
0.0262
TGReDial
0.0344
0.1442
00214
0.0458
0.0158
0.0217
Once-Ang
0.0387
0.1466
0.0216
0.0864
0.0159
0.0219
PopNodge_Ki
0.0492
0,1541
0.0271
0.0512
0.0199
0.0257
PopNudge k5
0.0539
0.1590
0.0306
0.0550
0.0228
0.0289
PopNudge_ k-10
0.0494
0.1574
0.0272
0.0523
0.0200
0.0261
PopNudgee_s5 150
0.1638
0.0289
Table 5: Experimental results of conversational recommenders the TGReDial dataset.
Model
Hit@ 10
Hir@50
NDCC@ 10
NDCC@50
MRR@10
MRR@50
ReDial
0.0008
0.0022
0.0004
0.0007
0.0002
0.0003
Once Once-Aug
0.0022
0.0089
0,0009
0.0025
0.0006
0.0009
PopNuddee ki
0.0009
0.0027
0.0005
0.0009
0.0064
0.0005
PoopNddee
0.0006
0.0037
0,0002
0.0009
0.0001
0.0003
PPopNudge_ k10
0.0004
00010
0.0001
0.0003
PopNNudge 650
0.0004
0.0022
0,0002
0,0006
0.0001
0,0022
KGSF
0.0040
0.0231
0.0022
0,0063
0.0016
0.0025
Once-Aug
0.0040
0.0236
0.0033
12000
0.0688
0.0018
0.0028
PopNudge 3T
0.0066
0.0231
00003
0,0024
0.0035
PopNudge_ 85
0.0053
0.0245
0,031
0,0075
0.0024
0.0034
PopNudge_ 110
0.0017
0.0142
0,0008
000037
0,0005
0,0012
PopNudge A50
0.0057
0.0007
0,0010
0.0005
0.0006
0.0062
00077 0
0.0024 1
4
Once-Aug
0.0062
0.0227
00076
0,0028
8E000
Popiuudge 7:1
0.0062
0.0182
00002
PoopNudge_K 85
0.0066
0.0261
000000
10000
0,0032
0,041
PopNudge A10
0.0026
0.0147
0.0014
0,0466
0.0011
0,0019
PooNNdgee A50
0.0057
0.0160
0,0011
0,0054
0.0023
0.0027
TGReDial
0.00800
0.0213
0,0045
0,0776
0.0033
0.0041
Once-Ang
0.0613
0.0075
0.0007
10000
0.0008
6.0008
PoppNudge RT
00001
000006
00005
000064
6.6007
-PopNudge__ 85
0.0057
0.0311
0.0033
0,0091
0.0025
0.0039
-PopNudge__i R10
0.0098
0.0329
0.0056
0.0108
0.0042
0.0054
PopNuddee A30
0.0102
0.0267
0.0059
0.0096
0.0045
0.0053

---

Table 6: Popularity hias analysis of Conversational Recom-
mendations on the ReDial Dataset
Model
Pop Bias (Mean)
Pup Bias (Sid)
ReDial
0.2601
0.0005
Once-Aug
0.0523
0.0001
PopNudge__i
0.1149
0.0364
PopNudge
0.1419
0.0001
PopNudge_kll
0.0940
0.0045
PopNudge_k50
0.0229
0,0005
KGSF
0.1382
0.1183
Once-Aug
0.1634
0,1501
PopNudge__i =1
0.1499
0.1364
PopNudge
0.1540
0.1463
PopNudge_ki =10
0.1576
0,1471
PopNudge_k50
0.1483
0.1513
KBRD
0.1240
0.1208
Once-Aug
0.1365
0,1142
PopNudge__i
0.1308
0.1257
PopNudge
0.1484
0.1279
PopNudge__t
0.1403
0,1195
PopNudge_A50
0.1398
0.1128
TGReDial
0.1312
0.1155
Once-Aug
0.1320
0,1064
-PopNudee_it
0.1292
0.1074
PopNudgee
0.1367
0.1099
- PoopNdge k10
0.1509
0,1259
PopNudge_AA00
0.1388
0.1178
Table 7: Popularity hias analysis of Conversational Recom- mendations the TGReDial Dataset
Model
Pop Bias (Mean)
Pop Bias (Sid)
ReDial
0.1626
0.0013
Once-AAg
1.0534
0.0307
- PopNudgee__ k1
0.0697
0.0001
PopNudge A5
0,0663
0,0321
PopNudge k10
0.0056
0.0001
PopNudge__ss A50
0.0368
KGSF
1.0847
0.5789
Once-Aug
T.0438
0.6240
PopNudge__t k1
1.2668
0.6837
PopNudge A5
1.3174
0,6899
PopNudge k10
1.2844
0.6122
PopNudge_Ass A50
1.3958
0.5049
KBRD
1.1012
0.5925
Once-Aug
1.1005
0.6144
- PopNudgee_ k1
0.8892
0.5747
PopNudge A5
1.1514
0,6588
PopNudge k10
1.1700
0.7493
Poopuddge_As A50
1.2343
0.7804
TGReDial
1.0943
0.5372
Once-Aug
1.3614
0.1649
- PopNudgee_ kl
1.2120
0.3565
Poppuudee A5
0.9601
0.5202
Poopudge k10
1.0031
0.5239
PopNuddge_AS A50
0.8259
0.4630


Reviews:
[{"review_id": "f1LE4R7JV4", "reviewer": "Reviewer_PYLM", "summary": "", "questions": "1. The symbols used in this paper can sometimes be confusing, like in function (2) the symbol r_u is not defined. What dose it stand for? And in line 6 of Algorithm 1, if d_j stands for a dialogue, what’s the meaning of {pop}_j>{pop}_i?\n2. In line 496-498, this paper says that Pop-Nudge consistently enhances the performance of baseline CRS models on both datasets. However, the performance of TGReDial model with PN-K1 on TGReDial dataset has a significant decrease in Hit@10 and Hit@50. Why do you compare the result of the basic model and the best result of all the data augmentation methods, rather that analyze the result respectively? For example, on the TGReDial dataset, PN-K10 with TGReDial base model surpasses all the other methods, but achieves the worst performance with KBRD model, how is that “consistently”?\n3. The synthetic data are generated using GPT-3.5, is it possible that the test dataset or valid dataset are included in the pretraining process of GPT-3.5? How can you make sure that there is no data leakage problem?\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nWe would like to express our sincere gratitude for your time and effort in reviewing our paper. We highly value your comments aimed at improving the quality of the manuscript. In response to your specific concerns about the clarity of symbols used in the paper, performance improvement consistency and the potential data leakage via using language model-generated synthetic data, we provide our detailed clarifications as follows:\n\n1. Regarding the symbols:\n\n(1) Symbol $r_u$: We understand your concern about the definition of $r_u$ in Equation (2). However, the text following this equation does provide a detailed explanation. Specifically, the term $r_u^{pop}(\\eta)$ is described to “represent the set of popular items based on a popularity threshold \\eta determined by the items’ interaction frequency”. Additionally, in Section 3.1 of the paper, the symbol $u$ is clearly defined to refer to the user u. For added clarity, we have made sure to refer to the paper by Lin et al. (2022a), where the equation for calculating the ranking utility is originally introduced. By taking these points together, the term $r_u$ can be understood as the recommendations made to a given user $u$.\n\n(2) Symbol $d_j$: You query pertains to the interpretation of $d_j$, which positioned together with $pop_j > pop_i$ in line 6 of Algorithm 1. As outlined in the manuscript, the term $D_{aug}^{ret} \\leftarrow D_{aug} - \\{d_j, pop_j - pop_i\\}$ is described in detail in the accompanying text. Specifically, the manuscript states, “we identify its included item and prepare the set of available items with lower popularity”. Thus, $d_j$ refers to dialogues that contain items with higher popularity than pop_i, and the algorithm proceeds by excluding these from $D_{aug}^{ret}$. This effectively sets a focus on items with popularity less than or equal to $pop_i$ for later steps in the algorithm.\n\nWe aim for a rigorous yet accessible presentation, and we believe that the referenced sections and additional explanations add the necessary context to comprehend the symbols $r_u$ and $d_j$.\n\n\n2. Regarding the consistency of PopNudge’s impact on various baseline conversational Recommendation Systems (CRS) models:\n\n(1) Concerning the term “Consistently” in Line 496-498. \n\nThe term “consistently” is used in our paper to indicate that the PopNudge strategy, when aptly tuned, enhances the performance of baseline CRS models across various datasets. This is based on tuning the value of k, which is elucidated in Section 4 starting from line 473.\n\n(2) On the Decrease in Hit@10 and Hit@50 for the TGReDial model with PN-K1 and the Role of Tuning in Deep Learning Models. \n\nWe acknowledge the specific point you raised concerning the TGReDial model's performance with PN-K1 on the TGReDial dataset. This observation is indeed important, and we wish to emphasise the crucial role of hyperparameter tuning, particularly in deep learning models. \n\nDeep learning models are often sensitive to hyperparameter choices, and the value of k in our PopNudge strategy can be viewed as another hyperparameter. The process of tuning k can be likened to adjusting other crucial deep learning hyperparameters like learning rates, regularisation terms, or batch sizes. Proper tuning is often essential for realising the potential of deep learning models, and the same applies to our PopNudge strategy. In the case of the TGReDial model, different k values produced better performance, underscoring the need for careful tuning, which we consider a standard practice in the application of deep learning techniques.\n\nHence, the term \"consistently\" in our paper is used to signify that PopNudge can, under proper tuning, enhance the performance of different baseline CRS models. The importance of tuning, a practice that is almost ritualistic in the deep learning community, cannot be overstated and directly correlates with our results.\n\n3. Data leakage via using language model-generated synthetic data: \n\nAs specified in Section 4 of our paper, beginning on line 470, we delineate that our data augmentation techniques, including the usage of GPT-3.5, are strictly applied to enrich the training data. The validation and test sets remain unmodified. This approach inherently minimises the risk of data leakage affecting the evaluation of the model performance on unseen data. While GPT-3.5 is used to generate synthetic dialogue, it's worth noting that our primary evaluation metric focuses on recommendation performance rather than the quality of the generated dialogue per se. Given that GPT-3.5 is trained mainly on textual data, the potential for data leakage from the test collection to the training corpus impacting the recommendation results is further minimised. This point is clarified further in our paper to mitigate concerns about data leakage. \n\nIt's theoretically possible, albeit highly unlikely, that the training corpus for GPT-3.5 includes dialogues from the test or validation sets of our specific datasets. However, given that we are not evaluating dialogue quality and that GPT-3.5 is designed to generalise across a wide array of text, the risk of significant data leakage is low.\n\nIn summary, we have put thought into our experimental design to minimise the risk of data leakage, and our evaluation strategy further mitigates this risk. We trust that these explanations adequately address your concerns, and we would be pleased to discuss any further questions or suggestions you may have.\n\nThank you once again for your valuable comments, and we look forward to any further feedback you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper conducts a comprehensive investigation into potential biases and effects affecting the performance of CRS models and proposes several evaluating metrics to quantify the biases for CRS.\n2. This paper proposed a data augmentation method of using LLMs to generate synthetic dialogue data to mitigate bias problem of CRS.\n3. The experiment results show that the proposed method can be effective on ReDial dataset.\n", "reasons_to_reject": "1. The symbols used in this paper can sometimes be confusing, like in function (2) the symbol r_u is not defined.\n2. The proposed method is simple and can be effective to some extent on the ReDial dataset, but can be very unstable on the TGReDial dataset. It can sometimes cause significant performance decrease, like the TGReDial model with PN-K1 where the Hit@10 score drops from 0.008 to 0.0013.\n3. Some statements in this paper don’t have enough supports or even can be wrong, like line 496-498. The performance of TGReDial model with PN-K1 on TGReDial dataset has a significant decrease in Hit@10 and Hit@50, which is contrary to the original statement that Pop-Nudge consistently enhances the performance of baseline CRS models on both datasets.\n", "questions_to_authors": "1. The symbols used in this paper can sometimes be confusing, like in function (2) the symbol r_u is not defined. What dose it stand for? And in line 6 of Algorithm 1, if d_j stands for a dialogue, what’s the meaning of {pop}_j>{pop}_i?\n2. In line 496-498, this paper says that Pop-Nudge consistently enhances the performance of baseline CRS models on both datasets. However, the performance of TGReDial model with PN-K1 on TGReDial dataset has a significant decrease in Hit@10 and Hit@50. Why do you compare the result of the basic model and the best result of all the data augmentation methods, rather that analyze the result respectively? For example, on the TGReDial dataset, PN-K10 with TGReDial base model surpasses all the other methods, but achieves the worst performance with KBRD model, how is that “consistently”?\n3. The synthetic data are generated using GPT-3.5, is it possible that the test dataset or valid dataset are included in the pretraining process of GPT-3.5? How can you make sure that there is no data leakage problem?\n", "paper_topic_and_main_contributions": "This paper conducts a comprehensive investigation into potential biases and effects affecting the performance of CRS models and proposes several evaluating metrics to quantify the biases for CRS. It also proposes a data augmentation strategy for improving CRS models, conducting experiments over two datasets ReDial and TGReDial with four different base models. The results show that the proposed method can be effective on the ReDial dataset, while not so effective on the TGReDial dataset, even causing significant performance decrease with TGReDial base model.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MCTvL9gFqq", "reviewer": "Reviewer_EaFv", "summary": "", "questions": "Question 1: The authors use their designed CEP and UIOP metrics to evaluate the proposed data augmentation strategies. Did the authors test the conventional popular bias metric in the experiment?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer:\n\nThank you for your time and the critical feedback provided on our manuscript. We appreciate your comment regarding the perceived limited novelty of the proposed method. Allow us to clarify the unique contributions that set our work apart in the field.\n\n(1) **Novelty.** Our work introduces a unique data augmentation approach that considers item popularity,  which is underexplored in the research community. We systematically explore and evaluate understudied biases in conversational recommendation systems, contributing new dimensions for analysis in this emerging field.  \n\nUnlike existing literature, which often confines itself to analysing traditional popularity bias, our work goes a step further. We introduce the concept of \"cross-episode popularity,\" examining the dynamic shifts in recommendations across different episodes within a dialogue. Here, episodes refer to the varying user preferences for target items during the course of a single dialogue, a term that has also been employed in existing research (as detailed in Section 3.1).\n\nAdditionally, we propose a measure, User Intent-Oriented Popularity, for popularity alignment between the user's target items and the items recommended by the system. This serves as an evaluative metric to gauge the system's proficiency in recommending items that align well with the user’s preference for popularity. These innovations are specifically designed to accommodate the unique characteristics of conversational systems and represent novel contributions to this emerging field.\n\n(2) **Recommendation-focus analysis.** Thank you for your comments highlighting the emphasis of our work on the recommendation component, as opposed to the conversational aspect. We appreciate the opportunity to address this observation.\n\nWhile it is true that the current study primarily focuses on biases and effects related to recommendation, this was an intentional design decision. The scope was narrowed to gain a comprehensive understanding of this specific yet vital facet of conversational recommendation systems. By delving deeply into this area, we can create a robust foundation that future research, including our own, can build upon to integrate the dialogue aspects. We have recognised this limitation in the \"Limitations\" section of the paper. The next phase of our research series aims to extend the current findings by incorporating dialogue performance metrics, thereby creating a more holistic model.\n\nOur present focus on the recommendation aspect is not an oversight but a strategic choice that allows us to provide valuable insights in this particular dimension, contributing to the nuanced understanding of conversational recommendation systems.\n\n(3) **Compare with existing bias mitigation strategies and very recent CRS models.**\nThank you again for your constructive feedback on the need for a more comprehensive comparison with other bias mitigation methods and state-of-the-art CRS baselines in our experiments.\n\nYour point is well-taken; however, we'd like to clarify that our focus on pioneering data augmentation techniques for addressing popularity bias presents a challenge in identifying appropriate baselines for comparison. While there are some related works like Lin et al., 2022a and Fu et al., 2023, they are not entirely applicable for fair comparison for the following reasons:\n\nLin et al., 2022a: Though relevant, this work specialises in an updated BRP-based loss function that may not be directly applicable to various CRS models without tailored modifications.\n\nFu et al., 2023: This study proposes a popularity mitigation strategy based on a Markov decision process, which similarly lacks broad applicability to all CRS techniques.\n\nIn light of these constraints, our experimental design emphasises multiple novel strategies for bias mitigation and benchmarks them against widely recognised CRS models that have gained consensus for their reliability in the literature. We believe this approach effectively demonstrates the efficacy of our bias mitigation strategies in a meaningful context.\n\n(4) **Test the conventional popular bias:** Thank you for your inquiry concerning the metrics used to evaluate our data augmentation strategies. To clarify, we indeed incorporate conventional popularity bias metrics in our evaluation. These results are comprehensively presented in Figures 2 and 3, as well as Tables 6 and 7 in the paper's appendix. Additionally, we provide an in-depth discussion of these findings within the main text of the paper. We appreciate your attention to the methodological rigor of our work, and we assure you that both our novel CEP and UIOP metrics, as well as the conventional metrics for popularity bias, were utilized to furnish a well-rounded evaluation of our proposed strategies.\n\nIn summary, thank you for your valuable comments. We have endeavoured to create a foundation upon which more expansive comparisons can be conducted as the field matures, and future research can incorporate more diversified baselines for a comprehensive assessment. We also look forward to your further feedback if you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written, which is easy to read and clear to grab the key idea of the model.\n2. This paper addresses an area that received recent traction in the field.\n3. The authors propose a simple but effective data augmentation strategy to mitigate selection and popularity biases. \n4. The paper presents several evaluations and performance improvements over comparison methods.", "reasons_to_reject": "1. Limited novelty of the proposed method.\n2. All the proposed biases and effects only focus on the recommendation part. They did not consider the possible impact factors from the conversation part.\n3. For the experiment, the authors did not compare their method with the other bias mitigation methods.\nThe experiments could be more solid if they are compared with the other bias mitigation methods and more state-of-the-art CRS baselines such as  UniCRS [1].\n\n[1] Wang, Xiaolei, Kun Zhou, Ji-Rong Wen, and Wayne Xin Zhao. \"Towards unified conversational recommender systems via knowledge-enhanced prompt learning.\" In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1929-1937. 2022.", "questions_to_authors": "Question 1: The authors use their designed CEP and UIOP metrics to evaluate the proposed data augmentation strategies. Did the authors test the conventional popular bias metric in the experiment?", "paper_topic_and_main_contributions": "This paper investigates selection and popularity biases within the feedback loop of developing a conversational recommendation system. Considering the influence of contextual user-system interactions and the varying user intent during a conversation, the authors extend the formalisation of popularity bias and explore the Cross-Episode and user intent-oriented popularity effects. Besides, two data augmentation strategies are proposed to enhance the model performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. to mitigate or alleviate bias effects (page 2) -> mitigate and alleviate have the same meaning\n2. KGSF (Zhou et al., 2020a) leverages both entity-oriented and word-oriented to enrich (page 6)"}, "extra_text": {}}, {"review_id": "bzW0nb2rcE", "reviewer": "Reviewer_ZYGt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for bringing to our attention the issue of clarity and ease of understanding in our paper, specifically regarding terminology and formulation. We understand the importance of making research accessible and transparent for all readers.\n\nIn response to your feedback, we plan to undertake a thorough review of the manuscript to enhance the clarity of terminology and the presentation of mathematical formulations. Our aim is to make the paper more reader-friendly while maintaining its academic rigour.\nWe appreciate your constructive comments, which allow us to improve the quality and accessibility of our work. Should the paper be accepted, please be assured that we will implement the necessary revisions to address these concerns in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting method for the data augmentation\n2. Solve the important bias in the recommendation area: selection bias and popularity bias\n3. Design the quantitate metric to evaluate the selection bias: Initial Item Coverage (IIC)\n4. It's possible to use GPT for other data augmentation", "reasons_to_reject": "1. The paper is not very easy to understand, the terminology and formulation can be better clarified", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors study different bias existing in the Conversational Recommendation System, and they propose a data augmentation technique based on generative language model. Through extensive experiments on two benchmark datasets, they show a consistent improvement of CRS techniques with our data augmentation approaches and offer additional insights on addressing multiple newly formulated biases.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an investigation into potential biases and effects that may affect the performance of models for conversational recommendation.  A novel metric for quantifying selective bias is proposed, as well as a data augmentation method that addresses these biases.  Experiments on two datasets show how the proposed method can improve performance (in some cases).  The reviewers agree that overall, the paper is well-written and the motivation is clear.  There is some concern about the limited novelty of the work and that the evaluations focus on recommendation only and not on the conversational component, which was intentionally left for future work (according to the rebuttal).  This does however make for a weaker paper in a 'dialogue and interactive systems' track.|This paper presents an investigation into potential biases and effects that may affect the performance of models for conversational recommendation. A novel metric for quantifying selective bias is proposed, as well as a data augmentation method that addresses these biases. Experiments on two datasets show how the proposed method can improve performance (in some cases). The reviewers agree that overall, the paper is well-written and the motivation is clear. There is some concern about the limited novelty of the work and that the evaluations focus on recommendation only and not on the conversational component, which was intentionally left for future work (according to the rebuttal). This does however make for a weaker paper in a 'dialogue and interactive systems' track.
Decision: Accept-Findings