Towards a Unified Conversational Recommendation System: Multi-task Learning via Contextualized Knowledge Distillation
Authors: Yeongseo Jung
Eunseo Jung
Lei Chen
Venue: EMNLP 2023 Main (2023)
Keywords: Recommendation
Multi-task learning

Abstract: In Conversational Recommendation System (CRS), an agent is asked to recommend a set of items to users within natural language conversations. To address the need for both conversational capability and personalized recommendations, prior works have utilized separate recommendation and dialogue modules. However, such approach inevitably results in a discrepancy between recommendation results and generated responses. To bridge the gap, we propose a multi-task learning for a unified CRS, where a single model jointly learns both tasks via Contextualized Knowledge Distillation (ConKD). We introduce two versions of ConKD: hard gate and soft gate. The former selectively gates between two task-specific teachers, while the latter integrates knowledge from both teachers. Our gates are computed on-the-fly in a context-specific manner, facilitating flexible integration of relevant knowledge. Extensive experiments demonstrate that our single model significantly improves recommendation performance while enhancing fluency, and achieves comparable results in terms of diversity.

Towards a Unified Conversational Recommendation System: Multi-task Learning via Contextualized Knowledge Distillation
Yeongseo Jung Eunseo Jung Lei Chen
The Hong Kong University of Science and Technology (yjungag, ejungab, leichen)@cse. ust. hk
Abstract
Biaing History
Sysbem: How new you todlgg?
User I'm havingg bad
don's recommend morte you
In Conversational Recommendation System (CRS), an agent is asked to recommend set items to users within natural language con- versations. To address the need for both con- versational capability and personalized recom- mendations, prior works have utilized sepa- rate recommendation and dialogue modules. However, such approach inevitably results in a discrepancy between recommendation re- sults and generated responses. To bridge the gap, we propose a multi-task learning for unified CRS, where single model jointly learns both tasks via Contextualized Knowledge Distillation (ConKD). We intro- duce versions of ConKD: hard gate and soft gate. The former selectively gates between task-specific teachers, while the latter in- tegrates knowledge from both teachers. Our gates are computed on- the-fle-l fly in a context- specific manner, facilitating flexible integration of relevant knowledge. Extensive experiments demonstrate that our single model significantly improves recommendation performance while enhancing fluency, and achieves comparable results in terms of diversity.'
User: I eojoy romanooe movies.
Dakingas Modicl
Revemmensaation Modill
Tesingon TTechhr
Rewommerdationn Thatthee
Cosy
Lihh of Items: Sitande Love Actually
@rudent
Hare you seeo The Avrengorw??
Hespomasc
Have Eitanie?
Itvaitate dailog- perttreannce
Evshasta r0commendatlon parformaccos
Evahusta chaiogin Necommasdation parfarmance
Baselines
Ours
Figure 1: Illustration of evaluation mismatch. Previous methods evaluate recommendation and dialogue perfor- mance independently as separate models are trained for each task. In contrast, a single model is utilized for both tasks ConKD.
items to users through natural language conver- sations, and the model-generated conversation is expected to be fluent and suit the context. Unlike traditional recommendation systems, the interac- tive nature of multi-turn dialogue allows the agent to explore explicit user interests that may be present in the user's history. This advantage particu- larly stands out compared to the traditional models that have access to only few purchase histories or implicit feedback (e.g. click) from users.
1 Introduction
Natural language dialogue systems generally fall into either task-oriented system (Wen et al., 2016; Henderson al., 2019; Peng al., 2020) or open- domain dialogue system (Xing et al., 2016; Zhang et al., 2020; Adiwardana et al., 2020). Despite the same modality (conversation), the tasks differ in their objectives; the former aims to achieve certain tasks (e.g. booking hotels), while the latter engage in open-ended dialogue.
To generate appropriate responses containing recommended items aligned with a user's taste, an agent needs to possess both recommendation and dialogue capabilities. Previous works addressed the issue with separate modules (Li et al., 2018; Chen et al., 2019; Zhou et al., 2020; Lu et al., 2021). A recommendation module learns user preferences based conversation history and retrieve relevant items, while dialogue module generates the final response sentences. In Conversational Recommen- dation System (CRS), a major problem lies in- corporating the two separate modules. Common
Conversational Recommendation (CR) an emerging task natural language dialogue, which combines task-oriented and open-domain (Gao et al., 2021). The task aims to recommend proper
'The code is available at yeangseoj/ConKD
com/

---

strategies for injecting recommendation ability into the responses include copy mechanism (Gu al..
Distillation.
We introduce two versions of ConKD, employ- ing different gate mechanisms: hard gate and seft gate.
2016) and pointer network (Gulcehre al., 2016).
Despite these efforts, the prior approaches (Li et al., 2018; Chen al., 2019; Zhou et al., 2020; Lu et al., 2021) have demonstrated a discrepancy between the separate modules: the results of a rec- ommendation model are not integrated in the gener- ated response as depicted in Figure 1. The dialogue module suggests "The Avengers", while the recom- mendation module recommends "Titanic", reveal- ing clear disagreement between the two modules. Accordingly, the probability distribution of the rec- ommendation model is not directly reflected in the output of the dialogue model. Such mismatch is inevitable when CR formulated as two separate tasks, failing to serve the original purpose of the task.
Our approach surpasses strong baseline mod- els in making coherent recommendations and fluent responses, while competitive results are observed in response diversity.
Preliminaries & Related Work
2.1 Open-Ended Conversational Recommendation System
Conventional recommendation systems mainly fo- cus building static user preference based previous histories, such as clicks, purchases, and ratings (Sarwar et al., 2001; Koren et al., 2009; Kuchaiev and Ginsburg, 2017; Kang and McAuley, 2018). such environment, where feedback from users is static, implicit, and sparse, recommen- dation systems have difficulty reflecting dynamic changes in users' preferences as well as suffer the cold-start problem (Gao et al., 2021).
To address this challenge, we propose a multi- task learning approach for a unified CRS using Contextualized Knowledge Distillation (ConKD). We build a CRS with a single model via knowledge transfer by two teacher models, a dialogue teacher and recommendation teacher. However, combin- ing the knowledge is not straightforward due to the nature of CRS; the task differs in each time step depending the context.
ReDial (Li al., 2018) is one of the first attempts at handling such issue; the work combines open- ended chit-chat with recommendation task, called Conversational Recommendation System (CRS). Specifically, let yx,) be a dialogue sample, where = {x x² .... is a set of previous dialogue turns. m is the lengths of turns in the dialogue his- tory and y is the corresponding response (ground truth). In each turn, recommendation module is expected to provide an item set Ia for user u E U. while a dialogue module produces a response y based on a dialogue history x.
In this light, we introduce two gating mecha- nisms, hard gate and soft gate, to effectively fuse teachers' knowledge. With hard gate, knowledge transfer comes solely from either of the teachers, while soft gate integrates both sources of knowl- edge. We introduce an adaptive nature in the gates which is context-specific and computed on-the-fly during forward propagation. To our knowledge, this is the first work to explicitly demonstrate the existence of the discrepancy, and provide a ded- icated training mechanism to address it. More- over, our approach offers the flexibility in selecting model architectures, enabling integration of diverse language and recommendation models.
To incorporate recommended items into a re- sponse, a copy mechanism (Gu et al., 2016) pointer network (Gulcehre et al., 2016) is generally adopted in prior works (Li et al., 2018; Chen et al., 2019; Zhou al., 2020; Lu et al., 2021: Zhou al., 2023). In such methods, additional networks are trained to predict whether the next token is word or an item by aggregating representations from rec- ommendation and dialogue modules.
Extensive experiments conducted on widely used benchmark dataset (Li et al., 2018) demon- strate that our single model significantly outper- forms baselines in terms of recommendation per- formance and response fluency, while achieving comparable results in response diversity.
In the aim of improving CRS, previous studies leverage external knowledge in training. In KBRD (Chen et al., 2019), an item-oriented knowledge graph is introduced as an auxiliary input rec- ommendation module. KGSF (Zhou et al., 2020) utilizes both word-level and item-level knowledge graphs in training a CRS. In addition to the knowl-
The contributions of our work can be summa- rized as follows:
We propose a multi-task learning approach for a unified CRS using Contextualized Knowledge

---

Models
Mismatch
R@1
ReR@1
R@10
ReR@ 10
R@50
ReR@50
KBRD
0,931
0.034
0.008
(76.47%)
0.168
0.040 (76.19 19
0.360
0.096 (74.17%)
KGSF
(78.95%)
0.043 (76.50%)
(73.33%)
RevCore
0.971
0.052
0.006 (88.46%)
0.195
0.031 (8%))
0.077 (77.42%)
Table 1: Comparisons of Recall (R@k) by recommendation modules and Recall in Response (ReR @)) by dialogue modules. Numbers in parenthesis indicate relative decrease recall score by dialogue module compared to the recall score by recommendation module. The scores are averaged over three runs with random seeds.
edge graphs movie review data is utilized in RevCore (Lu et al., 2021), 1) and the meta informa- tion of items encoded in (Yang et al., 2022) to enrich item representation. However, these works employ separate modules to manage CRS, which inevitably introduces discrepancy issues.
3.1 Preliminary Experiments
In our preliminary experiments on REDIAL (Li al., 2018) dataset?, we aim to identify the mismatch problem in evaluation. In Table 1, we compare the recommendation results from two separate modules using recall metrics: R @ k (Recall) and ReR @ k (Recall in Response) which evaluates the top-k items predicted by recommendation module and dialogue module respectively. In all metries, a sig- nificant degradation is observed when recall is com- puted on the dialogue response. The relative de- creases in the performance are ranged from 73.33% to 88.1%%, implying that large discrepancy exists between the outputs of recommendation modules and generated responses. However, incorporating the recommendation module's outputs during in- ference does not provide the fundamental solution to the problem. Further discussion on this issue is provided in Appendix D.
To address this problem, prompt-based learning strategies are introduced in (Wang al., 2022b; Deng et al., 2023). Despite the unified architecture, these approaches fail to dynamically incorporate multiple recommendations into a response. RecIn- Dial (Wang et al., 2022a) introduces a vocabulary pointer and knowledge bias to produce a unified output by combining two modules.
2.2 Knowledge Distillation
The core idea behind Knowledge Distillation (KD) (Hinton et al., 2015) is transferring knowledge of high-capacity teacher network to relatively smaller student model. In knowledge distillation, student network is guided by only one-hot en- coded ground-truth but also soft target mapped by the teacher network (probability distribution). This is known to transfer a ttass-wise relation mapped by a teacher is commonly termed the dark knowl- edge. Given data sample from a joint distribution (x, y) C X x V, student model is optimized by combining two cross-entropy terms.
To address the discrepancy, we propose a multi- task learning approach for a unified conversa- tional recommendation system via Contextualized Knowledge Distillation (ConKD). ConKD consists of three key components: a dialogue teacher and recommendation teacher experts on each task, and student model multi-task learner, de- scribed in Figure 2.
LKD(#)
Σ Yue log (o( 1=1
3.2 Recommendation Teacher
(1)
(1 - P log (xn)l)))
A recommendation teacher models the item-user joint distribution and provides set of items that suit a user's preference. We adopt the model struc- ture of (Zhou et al., 2020), where an item-oriented Knowledge Graph (KG) (Bizer et al., 2009) and word-oriented KG (Speer et al., 2016) are encoded to build user preference. To learn item represen- tations with structural and relational information, R-GCN (Schlichtkrull al., 2017) adopted for
where |Y| and Sh denote the number of classes and a k-th target label (one-hot encoded) respectively. y, and P denote balancing parameter, and prob- ability distribution scaled with temperature. 0 and @ are parameters of student and teacher network
respectively.
3 Unified CRS via ConKD
In this section, we first demonstrate the mismatch issue with preliminary experiments and introduce our approach that mitigates such problem.
'Li et al. (2018) proposed a dataset and a model, which we term the dataset as REDIAL and the model as ReDial here- inafter.

---

115
M(x,))
V
Ground Truth
Ground Truth
D-Tescher
R-Teacher
Bbudent
D-Tescher
R-Tescher
Bbudant
"If you romantle movies, would recommend Titante"
you like romantle movies, I would recommend Titente"
(a) Learning from dialogue teacher (A: 0)
(b) Learning from recommendation teacher (A+
The main structure of the proposed contextualized knowledge distillation with the hard gate. D-Teacher and ReTeccher denote dialogue teacher recommendation teacher respectively. I and V denote item space and vocabulary space respectively. The dashed arrow indicates backward propagation. One can easily extend the above to the soft gate, where d is continuous.
Figure 2:
the item-oriented KG as follows.
3.3
Dialogue Teacher
h (++1) = a E Σ
1 Zer
w m
To handle the chit-chat task, we train conditional language model that intakes dialogue history and generates a sontext-aware response. We explore two primary structural variations for our language model:
FERE'EEE
(2)
where h denotes the representation of node at (I)-th layer and (()) is the initial node embedding. E is the set of neighbor nodes for node under the
KGSF (Zhou et al., 2020): A standard trans- former (Vaswani et al., 2017) and knowledge- enhanced transformer (Zhou et al., 2020) are utilized for each encoder and decoder.
relation r. (!()) and W are learnable matrix for handling various edge types and self-loop respec- tively. Zer is normalization constant. Similarly, word-oriented KG is encoded with the GCN (Kipf and Welling, 2016) and the description in detail is illustrated Appendix B.
DialoGPT (Zhang et al., 2020): A transformer- based pre-trained language model (PLM) trained on a large-scale dialogue dataset.
Given the learned node embeddings, a user repre- sentation Pu is acquired by aggregating words (xx) and items n (x) that appear in previous dialogue turns as follows3.
Both dialogue models are trained to maximize the likelihood of predicting the ground truth response as follows:
T
(()) = ΣΣ
T Ut. log Polunglyia-1,
Pu = 3 (x) + (1 - -3 (x)
(3)
E=
8= o (W (n)
(5)
where T and j are the length of a response y and token index respectively. Y is union of the vocabulary set and item set (Y = V I), and ø denotes the parameters of the dialogue model.
where W are learnable parameters for computing the balancing parameter 3. Finally, a matching score between a user u and an item is calculated as follows.
3,4 Contextualized Knowledge Distillation
(i) sof tmo((p))
(4)
We claborate how student model learns both the recommendation and dialogue tasks. Specifi- cally, two gating mechanisms are introduced: dis- crete and continuous gate, which integrate knowl- edge between the two teachers in an adaptive man- ner.
where is model parameters optimized max- imize the likelihood of predicting ground-truth truth items.
For the detailed aggregation prncess, please refer to Sec- tion 4.3 in (Zhou et al., 2020)

---

Hard Gate
applied under the assumption that sum of item probabilities mapped by the dialogue teacher re- flects the extent to which recommendation is ex- pected. Therefore, the aggregated mass answers the question of 'how much to learn the recommenda- tion ability at the time step". Based on the intuition, we introduce soft gate as follows.
A student model is trained to minimize the gap between its conditional probability distribution and the conditional probabilities mapped by the teacher networks. However, it not ideal to equally weight the knowledge from both teacher models at every time step as seen in the following response.
yl: If you like romance movies, would recommend Titanic.
d - Σ P J'EZ
(7)
At the last time step, where the item Titanic ap- pears, knowledge of recommendation teacher can help a student learn the recommendation ability. On the contrary, student can learn to generate a coherent and suitable utterance by accessing knowl- edge from the dialogue model except the time of recommendation.
where the gate X takes continuous values within the range of [0, 1]. A gate close indicates that the system should focus more on recommendation, while a gate close to 0 suggests that the agent is expected to prioritize the conversation task.
To validate assumption regarding the behav- ior of the dialogue teacher, we conducted prelimi- nary experiment using smaller model, KGSF. We computed the average sum of item probabilities a dialogue time step A, and in a recommendation time step Ay The computed value of X, was found to be 0.653, while Ao was measured to be 0.023. These results support our assumption that the dia- logue teacher assigns relatively large probability mass item indexes in recommendation time. We provide further discussion on the validity of X in Appendix C.
Taking all these factors into account, we intro- duce a token-level hard gate between teacher net- works, where supervision solely comes from either of the teachers each time step. To distinguish which knowledge of the two teachers to be trans- ferred, we aggregate the probability mass of item indexes mapped by the dialogue teacher in each time step. This is built on an assumption that the di- alogue model assigns a relatively large probability mass on item indexes at the time of recommenda- tion.
The gating computation differs from the previ- ous gating approaches (Zhou et al., 2020; Li et al., 2018; Chen et al.. 2019; Lu et al., 2021) in two ways 1) leverage teacher distribution as signal of gating, where the gates can be discrete X {0, 1} or continuous A E [0, 1]. 2) our gates are not learned but a simple sum of probabilities.
Given the distribution P=(M,ly:-11,x) mapped by the dialogue teacher, we calculate sum of item prohabilities which answers the question of "is this time to recommend an item?". Therefore, a time step-specific hard gate is computed on-the-fly dur- ing forward propagation as follows.
0, if Ever A
<n
(6)
Contextualized Knowledge Distillation Loss
1, otherwise
With the two pre-trained teachers and the gating mechanisms, we now introduce loss for contextual- ized knowledge distillation.
where I denotes the of items in the vocabulary, and ry predefined threshold. When A is com- puted to be 1, it is a clear indication that CRS is expected to output a recommendation result. On the contrary, when the hard gate is O, a dialogue teacher defines the current time step as dialogue time step; hence, the CRS is expected to make coherent chit-chat.
KD losses for each task are formulated as fol- lows.
Y
(DIAL-D(()
T
log Po(yr.eliaa
Soft Gate
(8)
The hard gate inevitably introduces hyper- parameter 7 due to the thresholding approach. To remove the haper-parameter search on 17, we intro- duce a continuous gating mechanism. This can be
T
CREC-KD(B) = ΣΣ
(o(xxx))x
T
(9)
k=1
log Po(yr.)l1a-11xx)

---

where 0 is the parameter of the student. Then, the final KD losses for each task are derived follows.
transformer layers and optimized follows.
K
[(0ass) = IB (ka))Dl log P P(kg x 0cds)]
LDIAL(0) = (1- + DDDIL--D (8) - -1- + CRECKD
(10)
(12)
where K = {0, 1 is the label set, and fet denotes the classifier's s parameters. The model learns to classify whether the current turn is intended for chit-chat or recommending an item, based on the dialogue history x.
where is the balancing parameter between ground truth and teacher distribution, and LNL. is the cross entropy with ground truth. Finally, the losses are aggregated with our gate > per time step.
4
Experiments
(()) - (1-A)CCILLL + AcCREC
(11)
4.1
Dataset
When the hard gate is applied, supervision is made by either of the teachers with the discrete Àt- On the other hand, the soft gate fuses knowl- edge from the two teachers with the Xe being the weight. By optimizing the combined objective, single model is capable of learning the dialogue and recommendation tasks simultaneously. allevi- ating the mismatch that comes from two separate modules in previous methods.
The proposed approach is tested on the recently introduced REDIAL (Recommendation Dialogues) dataset (Li et al., 2018). REDIAL is conversation dataset which the dialogues are centered around
recommendation, and the subject of the recommen- dation movie. REDIAL contains 10,006 multi-turn dialogues, which amount to 182,150 utterances. The total number of unique movies in the dataset is 6.924, and the size of vocabulary is 23,928.
An evident advantage of employing contextu- alized knowledge distillation lies in taking the class-wise relation into consideration beyond the observed data (Hinton et al., 2015). With a one- hot encoded label, neural network is trained to maximize the difference between the ground-truth and remaining classes; the dark knowledge is over- looked with one-hot supervision where single index is set to and the remaining indexes to Os. In our work, the dark knowledge from both teachers is engaged in an adaptive manner to generate fluent and user-specific response.
4.2 Baselines
ReDial (Li et al., 2018) consists of dialogue, rec- ommendation, and sentiment analysis modules. Pointer network (Gulcchre et al., 2016) is intro- duced to bridge the modules. KBRD (Chen et al., 2019) introduces item-oriented KG (Bizer et al., 2009) and the KG representation is added when building logit for the dialogue module (Michel and Neubig, 2018). KGSF (Zhou et al., 2020) inte- grates word cord-oriented KG (Speer et al., 2016) and item-oriented KG (Bizer et al., 2009) for seman- tic alignment. KG-enhanced transformer and copy network (Gu al., 2016) are employed. RevCore (Lu al., 2021) incorporates movie-review data for review- review-enriched item representations and uti- lizes a copy network (Gu et al., 2016). DialoGPT (Zhang et al., 2020) is fine-tuned on the REDIAL dataset. RecInDial (Wang et al., 2022a) finetunes DialoGPT-small with R-GCN(Schlichtkrull et al., 2017) and introduces vocabulary pointer and knowledge-awar bias to generate unified outputs.
Special Tokens
Under CR, dialogue turn falls into either a turn with recommendation result or a turn for querying a user preference. In this light, to inject an extra signal to the student model, we add two special tokens, [REC] and [GEN], at the beginning of each turn. During training, the ground truth prefix starts with either [REC] if the response includes an item.. or [GEN] if is chit-chat. This explicit scheme en- ables the model learn turn-specific actions based on the preceding context and generate appropriate sequences.
4.3 Evaluation Metrics
To validate the recommendation performance, we employ top-k evaluation approach with k val- ues of 1, 10, and 50. Consistent with prior re- search (Zhou et al., 2020; Lu et al., 2021), we re- port Recall@k k (R @ k) in Section 3.1. However,
During inference, we employ pre-trained classi- fier to predict one the two special tokens at each dialogue turn. The classifier built with standard

---

Models
ReR@1
ReR@10 10
ReR@50
PrR@1
PrR@10
PrR@50
F1@1
F1@10
F1650
Rec Ratio
REDIAL
0.002
0.017
0,002
0,021
0.048
0.002
0.043
KBRD
0.126
0.009
KGSF
0.008
0.043
0.123
0.009
0.045
0.445
REVCORE
0.006
0.031
0.077
0.014
0,075
0.183
0.008
0.044
0,109
0,203
DialoGPT
0,011
0.070
0.172
0.011
0.174
0.011
0.070
0.173
0.462
RecInDial
0.017
0.088
0.203
0.022
0.114
0.264
0.020
0.099
0.229
0.438
KGSF ConKD (harch
0.023
0.110
0.299
0.024
0.113
0.257
0.023
0.111
0.253
0.499
KGSF ConKD (soft)
0.022
0.105
0.241
0.023
0.111
0.255
0.023
0.108
0.248
0,479
DialoGPT ConKD (hard)
0.022
0.120
0,250
0.021
0.112
0.235
0.022
0.116
0.243
0.525
DialoGPT ConKD (soft)
0.019
0.101
0.219
0.02
0.226
0.019
0.102
0.222
0,505
Automatic evaluation results on recommendation task. The scores are averaged over three runs with random seeds. ReR@k & and PrR@ k are recall and precision response for the top k items. Fl @ is the harmonic of ReR @+ k and PrR@k k. Rec Ratio is the ratio of recommendation turns to total turns. All scores are computed on final oulputs predicted by dialogue modules. Bold and underlined numbers denote best and second-best performance, respectively.
Table 2:
Qualitative
Quantitative
Models
Flu
Info
Cohe
Average
PPI.
DIST-2
DIST-3
DIST-4
REDIAL
1.772
0.334
I.194
1.100
17.577
0.075
0.123
0.166
KBRD
1.640
0.729
1.321
1.230
19.039
0.123
0.223
0.304
KGSF
1.811
0.502
1.433
1.249
11.191
0.168
0.305
0.420
REVCORE
1.854
0.512
1.187
1.184
9.283
0.098
0.170
0.235
DialoGPT
1.766
0.781
1,580
1.376
17.552
0.206
0.419
0.595
RecioDial
1.812
0.726
1.606
1,381
5,858
0.065
0.124
0.183
KGSF ConKD (hard)
1.831
0.856
1.571
1.419
8.886
0.138
0.249
0.344
KGSF ConKID (sofl)
1.858
0,878
1.644
1.460
8.689
0.132
0.236
0.326
DialoGPT ConKD (hard)
1.894
0.859
1,688
1.480
12.412
0.179
0.344
0.489
DialoGPT ConKD (soft)
1.870
0.829
1.591
1.430
12.336
0,180
0.350
0.505
Table 3: Qualitative and quantitative evaluation results on conversation task. The qualitative scores are averaged over three hired annotators. Flu, Info and Cohe indicate fluency, informativeness, and coherence of model-generatee responses. Average is the average of the quantities. In quantitative method, scores are averaged over three runs with random seeds. PPL is the perplexity of dialogue calculated by language model, and DIST-n the distinct n-gram in corpus level.
given the conversational nature of the task, it is crucial to evaluate recommendation performance within generated responses. We introduce Recall in Response (ReR @ k) following Liang et al. (2021) and Wang et al. (2022a), with a refined calcula- tion approach to ensure scores range within [0, 1]. Specifically, take the average of correct item pre- dictions the total number of item predictions instead of responses containing items. Addition- ally, we introduce Precision in Response (PrR @ k) and compute the harmonic mean of ReR @k and PrR @ kt denoted as Fl @ k. Furthermore, we assess the system's 's ability to make active recommenda- tions by introducing the recommendation turn ratio, calculated as the number of dialogue turns with recommended items over the total dialogue turns.
at the corpus-level and averaging them over sen- tences, which can lead to scores greater than To address this, we have updated the metric calcula- tion to count distinct n-grams and calculate rates at the corpus-level, ensuring the scores fall within the range of 0 to 1. The results evaluated on original metrics are illustrated in Appendix F.
Recent studies find that the n-gram based evalu- ation methods may not be sufficient to assess the performance of a language model (Zhang et al., 2020). Therefore, we conduct human evaluation to comprehensively assess model performance done in previous works (Zhou al., 2020; Wang et al., 2022a). Detailed human evaluation setup is described in Appendix E.
4.4 Results
To evaluate dialogue performance, we report per- plexity (PPL) and distinct n-grams (Li al., 2016) (DIST), assessing the fluency and the diversity of generated responses, respectively. In prior studies, DIST was computed by counting distinct n-grams
Evaluation of Recommendation
In Table 2, we present the recommendation perfor- mance of the models. The results clearly demon- strate that models with ConKD (hard) consistently

---

achieve the highest scores in F1 @ k; indicating su- perior performance in the recommendation task. Notably, KGSF integrated with ConKD doubles the scores compared to KGSF, while DialoGPT with ConKD achieves the scores 1.5 times as high as DialoGPT. These improvements are observed not only in single predictions, but also in top-10 and top-50 predictions, indicating superior user-item mapping. We hypothesize that such gain stems from the "dark knowledge" distilled from the rec- ommendation teacher within our framework. This knowledge encompasses inter-class relations that are absent in the one-hot encoded hard targets but are expressed through the probability distribution provided by the recommendation teacher. Further- more, the models with ConKD make active recom- mendations, depicted by the high recommenda- tion ratio, reflecting their focus on task-oriented conversation. Among our gating mechanisms, the hard gate outperforms the soft gate, which can be attributed to the stronger supervision made by the hard gate; a student is guided solely by the recom- mendation teacher in recommendation time.
Models
F1@1
@1@10
FI@
RecInDial
0.087
0.283
0.424
DialoGPT ConKD (hard)
0.124
0.331
0.465
Table 4: Recommendation results evaluated on contex- tullay relevant items.
dialogue teacher, which influences the gate value. It suggests that the choice between hard and soft gates depends on the capacity of the dialogue teacher.
4.5 Quality of Recommendation
In CRS, evaluating recommendations based solely single ground-truth item may not fully capture model's potential, user preferences wide range of items. To address this, we expand the evaluation by considering contextually relevant items. These relevant items are those located within a 2-hop distance from previously mentioned items in the item-oriented KG (Bizer et al., 2009), shar- ing attributes such as genre and actor. We compare two models, RecInDial and DialoGPT ConKD (hard), both of which use the same backbone model to produce unified outputs. Table 4 shows that ConKD consistently outperforms RecInDial across all metrics, with a significant improvement over the single-item evaluation results Table 2. This high- lights ConKD's capability not only to recommend gold item but also to comprehend the underly- ing knowledge structures, even in the absence of knowledge graph.
span
Evaluation of Dialogue Generation
addition to the recommendation performances, ConKD exhibits comparable results conversa- tion metrics, as shown in Table 3. Under quanti- tative evaluation, the proposed models outperform the backbone models in PPL., indicating enhanced fluency of responses. We observed that RecIn- Dial tends to generate relatively simple responses without active engagement, resulting lower PPL scores. Regarding the slight decrease in the DIST metric compared to the backbone models our results, two important observations should be high- lighted: 1) The base models fail to effectively ad- dress the recommendation task in their responses, and 2) DIST scores alone are insufficient for evalu- ating the quality of model-generated responses.
4.6 Efficiency Comparison
Ensuring real-time efficiency is of importance in enhancing the user experience in CRS. This see- tion compares the inference speeds of three mod- els: DialoGPT, DialoGPT + ConKD (hard), and RecInDial. DialoGPT and DialoGPT+ConKD (hard) achieve inference latencies of 5.464ms and 5.306ms per token, respectively. In contrast, RecIn- Dial incurs a slightly higher latency of 6. 1000s per token. This additional latency in RecInDial can be attributed to the computation of the knowledge- aware bias and vocabulary pointer. The compo- nents involve making decisions between generat- ing items or words in every time step. In ConKD, a language model handles the knowledge-awarr recommendations without sacrificing efficiency.
These findings are supported by the results of qualitative evaluation, where the single models with ConKD outperform all baselines in average scores. Specifically, when applied to KGSF, the proposed methods significantly enhance informa- tiveness and coherence. This implies our training mechanism performs consistently regardless of the model size, aligning with the automatic evaluation results. We also observe that ConKD-soft outper- forms ConKD-hhrd with KGSF as the backbone, and the opposite holds true for DialoGPT. This dis- crepancy is attributed to the model capacity of the
4.7 Ablations
To verify the efficacy of each component intro- duced in this work, we conduct ablation studies

---

Models
FI@1
F1@10
F1@50
Rec Ratio
Case 1
Vanilla
0.012
0.062
0.155
0.337
Context
D
0.017
0.063
0.165
0.334
User
Hi!
(
0.019
0.066
0.146
0.293
Agent
How are you?
(+) DAR
0.020
0.093
0,200
0.313
User
I like action movies. T like Bruce Willis. He a great actor. Or Tom Cruise. I am doing well, thanks for asking.
(+) D&R&ST
0.023
0.111
0.253
0.499
A 0.3..
0.021
0.245
0.471
Model Ourput
Table 5: Ablations on the recommendation task. (+)
ReDial
What can I help you find today
indicates adding corresponding components Vanilla, model without ConKD, D and R refer to the dialogue teacher and recommendation teacher. ST is the special token mechanism, and + indicates replacement. Hard gate is applied for combining the KGSF teachers.
KBRD
What would you like?
KGSF
I'm doing well. Edge Tomorrow
RevCore
not sure, but have to give you some good ideas.
DialoGPT
Have you soen the movic Mri& Mrs.Smith
really good movie...!
RGSF
Have you seen Edge of Tomorrow!
Models
DIST-1
DIST-2
DIST-3
DIST-4
PPT
KGSF
Tlkee action movics too. Have you socn Edge of Tomorrow?
Vanilla
0.100
0.246
7.538
(4)D
0.029
0.100
0.172
0.24T
7.071
DialoGPT
Have you seen Mission: Impossible"
(+) R
0,019
0.066
0,115
0.164
12.331
DialoOPT
Have you scen Mission: Impossible?
(+) D&R
0.028
0,097
0.166
0,232
6.349
Ground Truth
Have you Gladiator!
(+) D&R&ST
0.035
0.138
0.249
0.344
8.886
Au 6+ 0..
0.031
0.121
0.219
0.306
9.211
Table 7: Sample outputs generated by the baselines and ConKD. * and ** indicate model with ConKD (hard) and ConKD (soft) respectively. The last three dialogue turns are depicted for context. Bold words indicate user's preference and recommended items.
Table 6: Ablations on the dialogue task.
using several variants of our model. The results are depicted in Table 5 and 6.
We observed that the role of teachers signifi- cantly affects the performance of student. Learn- ing solely from the recommendation teacher en- hances recommendation performance but comes at the cost of PPL and DIST scores. In contrast, learning from the dialogue teacher improves flu- ency. When both teachers are present within the training framework, the student's performance im- proves in both tasks, highlighting the effectiveness of knowledge transfer through our gating mecha- nism. Scores decline in all metrics when we replace our gate with a static value of 0.5. Lastly, the spe- cial token also brings meaningful gains, depicted with the increased F1 @ k and DIST scores.
both fluent and informative; it makes user-specific recommendations, suggesting Edge of Tomorrow and Mission: Impossible, both of which are action movies featuring Tom Cruise. Notably, these rec- ommendations are more aligned with the user's ex- pressed preferences compared to the ground-truth movie "Gladiator", which is action movie with- out the mentioned actors. Additional examples are provided in the Appendix G.
5 Conclusion
In this study, we introduce contextualized knowl- edge distillation with hard gate and soft gate. In hard gate, a student is either learned from rec- ommendation teacher from dialogue teacher with discrete value, while the soft gate fuses knowledge from both teachers, removing hyper- parameter from the hard gate. The gates are com- puted in a context-specific manner by aggregating the probability mass on the interest set (a movie item set in our experiment). Our work verifies the idea in the popular benchmark dataset and the result illustrates the superior performance of our approach compared to strong baselines. In addi- tion, human evaluation mainly conforms with the automatic evaluation, demonstrating that the pro- posed approach a well-balanced model with both recommendation and chit-chat ability.
4.8
Case Study
The responses generated by the baselines and ConKD are shown Table 7. In the conversa- tion, the user has expressed clear preference for action movies and has previously mentioned actors "bruce willis" and "Tom Cruise". However, the three baselines generate chit-chat responses with- out recommendations. Although KGSF suggests a movie, the response lacks coherence and fluency. DialoGPT generates a coherent response but recom- mends "Mr.& Mrs.Smith". which does not align with the user's preference; neither Bruce Willis nor Tom Cruise are associated with it. On the other hand, our models produce apt responses that are

---

Limitations
Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding. Yukuo Cen, Hongxia Yang, and Jie Tang. 2019. To- wards knowledge-based recommender dialog system.
This work is grounded on the student-teacher framework, hence requiring additional computa- tion when obtaining knowledge of teacher; our approach requires two teachers, one for dialogue and one for recommendation. This extra compu- tation can be a burden in an environment lack of resource. Nonetheless, the proposed approach uti- lizes single model for inference. Furthermore, our approach requires the teachers to be well- cell-trained. This, however, is also shared problem within KD training.
Yang Deng, Wenxuan Zhang, Weiwen Xu, Wenqiang Lei. Tat-Seng Chua, and Wai Lam. 2023. A unified multi-task learning framework for multi-goal con- versational recommender systems. ACM Trans. Inf. Syst, 41(3).
Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and challenges in conversational recommender systems: A survey. Al Open, 2: 2:100-126.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence- to-sequence learning. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume I: Lonng Papers), pages 1631- 1640. Berlin, Germany, Association for Computa- tional Linguisties.
Ethical Consideration
Since REDIAL dataset (Li et al., 2018) contains multi-turn dialogue histories, the dataset, by nature, may pose privacy issue. If a dialogue teacher in our framework learns such information, the student in our framework can also learn to output private information while in conversation. Such issue may possibly be handled by employing a privacy clas- sifier model, where a model is trained to identify certain outputs containing private information of user.
Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati. Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 140-149, Berlin, Germany. Association for Computational Lin- guistics.
Matthew Henderson, Ivan Vulić, Daniela Gerz, Tñigo Casamueva, Pawel Budzianowski, Sam Coope, Geor- gios Spithourakis. Tsung- Wen, Nikola Mrkšić. and Pei-Hao Su. 2019. Training neural response lection for task-oriented dialogue systems.
Acknowledgements
Lei Chen's work is partially supported by National Science Foundation of China (NSFC) under Grant No. U22B2060, the Hong Kong RGC GRF Project 16213620,, CRF Project C2004-21GF, RIF Project R6020-19, 19, AOE Project AoE/E-603/18, Theme- based project TRS T41-603/20R, China NSFC No. 61729201, Guangdong Basic and Applied
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.
Wang-Cheng Kang and Julian McAuley. 2018. Self- attentive sequential recommendation. In 2018 IEEE International Conference Data Mining (ICDM). pages 197-206
Basic Research Foundation 2019B 1510000,,, Hong Kong ITC ITF grants MHX/078/21 and PRP/004/22FX, Microsoft Research Asia Collabo- rative Research Grant and HKUST-Webber joint research lab grants.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization.
Thomas N. Kipf and Max Welling. 2016. Semi- supervised classification with graph convolutional networks.
Yehuda Koren, Robert Bell, and Chris Volinsky. 2009. Matrix factorization techniques for recommender sys- tems. Computer, 40083333
References
Daniel Adiwardana, Minh-Thang Luong, David R. So,
Oleksii Kuchaiev and Boris Ginsburg. 2017. Training deep autoencoders for collaborative filtering.
Jamie Hall, Noah Fiedel, Romal Thoppilan. Zi Yang. Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoe V. Le. 2020. Towards a human-like open- domain chatbot.
Jiwei Li. Michel Galley, Chris Brockett, Jianfeng Gao. and Bill Dolan. 2016. A diversity-promoting ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics.
Christian Bizer, Jens Lehmann, Georgi Kobilarov, Sören Auer, Christian Becker, Richard Cyganiak, and Se- bastian Hellmann. 2009. Dbpedia crystallization point for the web of data. Journal of Web Semantics, 7(3):154-166 The Web of Data.

---

Raymond Li, Samira Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. To- wards deep conversational recommendations.
Chen Xing. Wei Wu, Yu Wu, Jie Liu, Yalou Huang. Ming Zhou, and Wei- Ying Ma. 2016. Topic aware neural response generation.
Zujje Liang, Huang Hu, Can Xu, Jian Miao, Yingy- ing He, Yining Chen. Xiubo Geng, Fan Liang, and Daxin Jiang. 2021. Learning neural templates for recommender dialogue system.
Bowen Yang, Cong Han, Yu Li, Lei Zuo, and Zhou Yu. 2022. Improving conversational recommenda- tion systems' quality with context-awand item meta- information. In Findings the Association for Com- putational Linguistics: NAACL 2022, pages 38-48, Seattle, United States. Association for Computational Linguisties.
Yu Lu, Junwei Bao, Yan Song, Zichen Ma. Shuguang Cui, Youzheng Wu, and Xiaodong, He. 2021. Revcore: Review-angmented conversational recom- mendation.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Paul Michel and Graham Neubig. 2018. Extreme adap- tation for personalized neural machine translation.
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020. Few-shot natural language generation for task- oriented dialog.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. Dialogpt: Large-scale generative pre-training for conversational response generation.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th International Conference on World Wide Web. WWW '01, page 285-295, New York, NY, USA. Association for Computing Machinery.
Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020. Improv- ing conversational recommender systems via knowl- edge graph based semantic fusion.
Yuanhang Zhou, Kun Zhou, Wayne Xin Zhao, Cheng Wang. Peng Jiang, and He Hu. 2023. C2-crs: Coarse- to-fine contrastive learning for conversational recom- mender system.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem. Rianne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling relational data with graph convolu- tional networks.
A Implementation Details
Robyn Speer, Joshua Chin, and Catherine Havasi. 2016. Conceptnet 5.5: An open multilingual graph of gen- cral knowledge.
Combinations of KGSF and ConKD. We mainly follow KGSF (Zhou et al., 2020) for the basic model structure of the two teacher and student models. For the dialogue teacher and student, we employ an encoder-decoder structure, with each module consisting of 2 transformer layers. The hidden dimension size of both models is 300. Our work excludes the copy mechanism used in KGSF, In the recommendation model, we utilize 1-layer GNN networks trained with word-oriented and item-triented KGs inputs. The normalization constant Ze, is set to 1. The token classifier is wansformer-basee encoder with a classification head. We employ the Adam optimizer (Kingma and Ba, 2014) and apply gradient clipping for sta- ble training. Our chosen hyper-parameters include batch size of 32, learning rate of le-3, and train- ing for 200 epochs. Furthermore, for ConKD, we set the 7) and 7 to 0.3 and 0.6, respectively. Combinations of DialoGPT and ConKD. We utilize DialoGPT-small (Zhang al., 2020) as the backbone for the dialogue teacher and student model. DialoGPT consists of 12 transformer lay- ers and the hidden dimension size of the model is
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need.
Lingzhi Wang, Huang Hu, Lei Sha, Can Xu, Daxin Jiang, and Kam-Fai Wong. 2022a. RecInDial: A uni- fied framework for conversational recommendation with pretrained language models. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 389-500, Online only. Association for Computational Linguisties.
Xiaolei Wang, Kun Ji-Rong Wen, and Wayne Xin Zhao. 2022b. Towards unified conversational rec- ommender systems via knowledge-enhanced prompt learning. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Min- ing. ACM.
Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Mil- ica Gasic, Lina M. Rojas-Barahona Pei-Hao Su, Ste- fan Ultes, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system.

---

768. The recommendation teacher and token clas- sifier in the DialoGPT+ConKD models follow the same settings as those the KGSF+ConKD mod- els. For hyper-parameters, we use a batch size of 32, a learning rate of le-3, and train for 20 epochs. For ConKD, we set the 17 and y to 0.6 and 0.4, respectively.
response. Additionally, the output does not reflect the dynamic changes of the context, which fails provide context-aware recommendation as seen in the following responses.
y4: Blended with Adam Sandler is a fav of mine.
During inference, we only use the student model for the response generation in both settings.
ys: Love Actually with Adam Sandler is a fav of mine.
B Graph Convolutional Network (GCN)
y/ and ys are generated under the original setting and the two-step inference setting respectively The dialogue module generates the coherent and in- formative response describing the actor Adam Sandler who stars in the movie Blended. On the other hand, the item refilling fails to incorporate suitable item into the response, which causes fac- tual error; there is no relationship between Adam Sandler and the movie Love Actually. This in- dicates that the simple integration cannot be the fundamental solution for the mismatch issues; it leads to semantic discrepancy between the recom- mendations and generated responses. In our unified system, the output distribution over items dynami- cally changes depending on the context, generating coherent and user-specific responses. This is done by single model in single step, thereby reducing the model size and inference time.
GCN is adopted to encode the word-oriented KG
ConceptNet (Speer et al., 2016). A triple in the KG is denoted by (w1, W2). where W1 and w2 are word nodes, and r is relationship between the nodes. The node features are updated with aggregation function as follows:
H ReLU(D AD W (13)
H E Rad and ( denote the node represen- tations and a learnable matrix at the I-th layer re- spectively. n is the number of nodes and d denotes the dimension size of node features. Á = A + is the adjacency matrix of the graph with self-loop, where I is the identity matrix. D E, Aij refers degree matrix.
Validity of
Human Evaluation Setup
To explore the validity of the X, we illustrate varia- tions of the response yl introduced in 3.4.
We engaged three annotators who were tasked with assessing model outputs given dialogue history. 100 model outputs from each model are randomly sampled and collected, the total being 1000 dia- logue turns. The annotators score each model out- put from the range of on the level of informa- tiveness, coherence, and fluency of model output.
y2: If you like romance movies, would recommend you Titanic. y3: If you like romance movies, I would recommend some romantic comedies.
After the word recommend, the vocab (you and some) can be replaced with the item Titanic in the yl. Therefore, the average value xy of 0.653 is acceptable in soft gate, indicating the mass reflects the level to which recommendation is expected.
The following instructions were provided to guide annotators in their assessments:
Fluency: Fluency encapsulates the naturalness of the generated text. It involves an assessment of how the output adheres to linguistic standards, avoiding grammatical flaws. Annotators should evaluate the syntactic flow, word choice, and over- all readability. The scores should be shown as 0, 1. and 2, where each indicates "not fluent", "readable but with some flaws", and "fluent", respectively. Informativeness: The informativeness encom- passes the model's 's ability to convey relevant and
Item Refilling
We discuss two-step inference, in which a re- sponse is first generated, and items in the response are refilled with the output of a recommendation module. The recommendation module predicts a probability distribution in each dialogue turn, which remains fixed at each time step during the response generation. Hence, the module cannor directly handle the variable number of items in a
"The samples are generated by KGSF. We leverage top k sampling to handle the variahle number of items in responses, where & is set to 1.

---

accurate information. Annotators should assess the depth and accuracy of the conveyed information. The scores need to be displayed O, I, and where each corresponds to "information missing or in- correct", "information is included but insufficient partially inaccurate", and "comprehensive and accurate information", respectively.
Coherence: Coherence entails the harmonious integration of the model's output within the evolv- ing conversation. Annotators should assess how well the model comprehends and adheres to the conversation's 's theme, avoiding abrupt shifts and ensuring a natural conversational flow. The scores should be valued using O, 1, and 2. Each rating represents "awkward conversation flow", "make sense but somewhat disconnected", and "coherent", respectively.
Context
Agent User
Hey! What is movie you love?
Hello, I am looking for old classic. What would you suggest??
Model Output
ReDial
Hello
KBRō KGSF
Have you Jumanji?
I can recommend some movies for you?
RevCore
T love comedy movies!
DialoGPT
Do you have a favorite actor actress? love
how many types of movies today.
KGSP
Have you seen Gome with the Wind! Of maybe
something like It's Wonderful Life?
F Results Evaluated Using Original Metrics
KGSF
Have you seen The Big Lebowski?
DialoGPT
I would suggest The Outsiders It's clussic!
DiuloGPT
Have you Driving Miss Duisy?
Ground Truth
The Gold Rush and Invasion of the Body
Snatchers are classics.
Modds
DIST-2
DIST-3
DIST44
RcR@1
ReR@
Reke30
Case 3
RacinDial
0,13
0.663
0.315
0.023
0.118
0.273
Ours
1.326
2,217
2,704
0.03
0.161
0.337
Context
User
Hello! Do you know of any family friendly movies?
Table 8: Dialogue and recommendation performance evaluated on the conventional metric. DialoGPT + ConKD (hard) employed for ours.
Agent
Good Moming have you seen The Sixth Sense?
User
Yes, but it is too seary for my children, somerhing
like, Peter Rabbit we saw that and was cute.
Ageni
A old classic will he Finding Nemo.
User
Oh yes, and Finding Dory.
Model Output
ReDial
liked it lot.
G Additional Cases
KBRD
Glad to helg. Have a great day!
KOSF
also recommend watching Captain Underpaans:
Our models generate diverse movies that differ from the ground truth but align with the user's pref- erences. For example, in Case 2, when the user re- quests old classics, our models suggest Gone with the Wind (1939), It's a Wonderful Life (1946), The Big Lebowski (1998), The Outsiders (1967) and Driving Miss Daisy (1989), all of which are considered old classics. In contrast, other baselines fail provide recommendation, except for KBRD. the Case 3, when the user expresses a prefer- ence for family friendly movies like Peter Rabbit, and Finding Dory, our models recommend Beauty and the Beast, Jumanji, Coco, and Troll all of which are family-friendly, with three of them being animations. This contrasts with other baselines that produce contextually incorrect responses without recommendations or mention Peter Rabbit again, which the user had previously mentioned in the dialogue context.
The First Epic Movie again. It's pretty good can
think some people laavnn seen
ResCore DialoCP
T think you would like it_
I have not seen Peter Rabbit but have heard good things about it
KGSF* KGSP
I also really liked Beauty und the Beast
I also recommend watching the new Jumanji. It was cute.
DiskGPT Di
Coco was good too
T also recommend Troli. "I your looking at ammmily that one is really good too'.
Ground Truth
T saw Peter Rubbit with my daughter she loved it"
Table Additional sample outputs generated by the baselines and ConKD, and ** indicate model with ConKD (hard) and ConKD (soft) respectively. The last four dialogue turns are depicted context. Bold words indicate user's preference and recommended items.


Reviews:
[{"review_id": "EuE9Cy18hN", "reviewer": "Reviewer_hs9f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and recognition of the rationale behind our system design, as well as the effectiveness of our approach. We have given careful thought to all your concerns and questions.\n\n**1. Add a baseline that fuses knowledge graph inputs and pre-trained language model.**\n\nThank you for your suggestion in adding a baseline. We have implemented RecInDial [1], which closely aligns with the points raised in your suggestion as well as our research direction. [1] integrates knowledge graph inputs and a pre-trained language model by introducing a knowledge-aware bias, computed through the aggregation of node representations. Additionally, [1] employs vocabulary pointer to control when to predict items or words during response generation. \n\n**Performance comparisons**\n\nFor a fair comparison, we have chosen to compare DialoGPT + ConKD (hard) with RecInDial in this response, given that DialoGPT-small serves as backbone model for RecInDial. Both models were finetuned on the Redial dataset, and the reported scores are averaged over three runs. \n\n|          Models         |   DIST-2  |   DIST-3  |   DIST-4  |    PPL    |    F1@1   |   F1@10   |    F@50   | Rec Ratio |\n|:-----------------------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|        RecInDial        |   0.065   |   0.124   |   0.183   | **5.858** |   0.020   |   0.099   |   0.229   |   0.438   |\n| DialoGPT + ConKD (hard) | **0.179** | **0.344** | **0.489** |   12.412  | **0.022** | **0.116** | **0.243** | **0.525** |\n\nThe results of ConKD are from the current version of our paper, where F1@k represents the harmonic mean of the ReR@k and PrR@k.\n\nIn terms of the recommendation metrics (F1@k), it is evident that the results of RecInDial are inferior to ours, with a notably lower recommendation ratio. We hypothesize that our contextualized knowledge distillation concept enables the student model to learn class-    wise relations which are not addressed by the addition of knowledge bias and vocabulary switching [1]. \n\nFurthermore, our results clearly show that the simple integration with vocabulary pointer [1] fails to generate diverse responses, as indicated by the DIST-n scores. We have observed that RecInDial tends to generate relatively simple and concise responses, leading to lower PPL score. While these responses exhibit fluency, they lack the desired level of engagement, as indicated by the low recommendation ratio.   \n\n**Efficiency considerations** \n\nIn terms of efficiency, we conducted a comparison of inference speeds. On the Redial dataset, DialoGPT and DialoGPT+ConKD (hard) achieved inference latencies of 5.464ms and 5.306ms per token, respectively. In contrast, RecInDial incurred a slightly higher latency of 6.100ms per token. This additional latency in RecInDial can be attributed to the **computation of the knowledge-aware bias** and **vocabulary pointer**, which controls the choice between generating items or general words **in every time step**. Importantly, our adaptive gates empower a language model to acquire such capability during training, eliminating the need for additional components in inference.\n\nIn addition, maintaining a knowledge graph for building user representations during inference introduces scalability challenges; the graph size grows with the addition of items and attributes. This is in contrast to our approach, where **a single language model handles the knowledge-aware recommendations.**\n\nIn conclusion, our approach outperforms the baseline and shows superior results in recommendation performances without sacrificing efficiency. We will add the results in the final version of our paper.\n\n**2.\tLack of discussion in the Case Study: All models fail to recommend the appropriate item compared to ground truth. What to digest in this section?**\n\nWe would like to clarify that evaluating solely based on ground-truth recommendations may not capture the full potential of a conversational recommendation system, as there can be a wide array of items that align with a user's preferences. \n\nFor instance, in Table 6, the ground-truth recommended movie **“Gladiator”** aligns only with the user’s preferred genre **“action”**, while the user’s preferred actors **“Bruce Willis”** or **“Tom Cruise”** are not associated with the movie **“Gladiator”**. However, our models recommend movies **“Edge of Tomorrow”** and **“Mission: Impossible”** which are action movies featuring **“Tom Cruise”**, as described in the Line 514-519.\n\nIn Appendix H, our models generate diverse movies that differ from the ground truth but align with the user’s preferences. For example, in Case 2, when the user requests **“old classics”**, our models suggest movies **“Gone with the Wind (1939)”**, **“It’s a Wonderful Life (1946)”**, **“The Big Lebowski (1998)**”, **“The Outsiders (1967)”** and **“Driving Miss Daisy (1989)”**, all of which are considered old classics. In contrast, other baselines fail to provide recommendation, except for KBRD.\n\nIn the Case 2, when the user expresses a preference for “**family friendly”** movies like **“Peter Rabbit”**, and **“Finding Dory”**, our models recommend **“Beauty and the Beast”, “Jumanji”, “Coco”**, and **“Troll”** all of which are **family-friendly**, with three of them being animations. This contrasts with other baselines that produces contextually incorrect responses without recommendations and the ground truth that includes the movie **“Peter Rabbit”** again, which the user had previously mentioned in the dialogue context. We will include this evaluation approach and further discussion on the Case 2 and Case 3 in Appendix H.\n\n**3.\tAdd the “GoRecDial” dataset**\n\nThank you for your helpful comment. We have carefully reviewed the dataset, and while it presents an interesting approach, it primarily focuses on a game-like scenario where the goal is to \"find\" the correct movie among a set of five options within minimal dialogue turns, with the dialogue concluding once the correct movie is successfully identified. This setup closely aligns with policy-based reinforcement learning methods, which are typically adopted for such tasks, rather than our task, which involves a combination of open-ended chitchat and recommendation. Additionally, the GoRecDial dataset is also centered around a single domain-movie recommendation, and is relatively small in scale compared to the Redial dataset we used. \n\nHowever, we recognize that additional datasets will be helpful to validate the generalization ability of our model. We will actively explore multi-domain datasets during the rebuttal period.\n\n[1] Recindial: A unified framework for conversational recommendation with pretrained language models, AACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Section 3.1 preliminary experiments are well laid-out and the results justify the challenge tackled by the paper. The discrepancies between recommendations made by the recommender module and those present in the generated dialog for several multi-stage ConvRec systems is substantial (70-90%) and demonstrates a large gap.\n- The system design should be clarified - using task-specific signal tokens ([REC]/[GEN]) etc. can all be done with a language model alone, and one could perform multi-task learning with the recommender system or teacher-forcing alone with the recommender system without the dialog system. However, the authors do justify their system design via ablations in 4.3\n- The ConKD approach significantly improves response entity precision and recall compared to baselines (Table 2).", "reasons_to_reject": "- The \"comprehensive evaluation\" by using KGSF/DialoGPT is a bit of a strange assertion; the effect of pre-trained language models are orthogonal to the effect of semantic fusion with knowledge graph inputs (a la KGSF). It would have been nice to see a knowledge graph fused to DialoGPT or some similar method (e.g. via prompting prepended with a subgraph or other KG features).\n- The case study is confusing and not particularly useful; all models evaluated demonstrate a failure to recommend the appropriate item compared to ground truth, and there isn't enough of a discussion to figure out what we're supposed to digest from this section.\n- The ReDial dataset is an interesting benchmark but limited to a single domain with a relatively small corpus of items and straightforward user behavior. The paper would be better served by evaluating on a dataset like GoRecDial (Kang 2019), which demonstrates more complex recommendation/querying behavior.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Positioning: the paper is about conversational recommender systems - systems that recommend items or entities to users through the course of a dialog. Specifically, the authors note consistency challenges between recommendation and dialog modules in multi-part ConvRec systems. They propose to learn an E2E conversational recommender by distilling recommender systems and dialog models into a single \"Unified\" model.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The diagram on Page 1 is difficult to read; the font is quite small and it's difficult to parse. Perhaps it could be tidied up."}, "extra_text": {}}, {"review_id": "0Lx6HtyyNJ", "reviewer": "Reviewer_nTHZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and comments. We have taken each of your concerns and questions into careful consideration.\n\n**1. Comparison to “RecInDial [1]” which performs significantly better than this work.**\n\nThank you for pointing out the comparison with RecInDial [1]. We would like to highlight two key aspects: (1) **The marked variations in performance** between our work and [1] can be attributed to **changes we made in metrics calculation** (2) We have implemented [1], and **the results confirm the first point**.\n\n**(1) Changes in metrics calculation**\n\n- DIST-n in Table 3\n\n    In Table 3, the baselines utilize the KBRD approach for computing the DIST-n scores. This methodology involves counting the distinct n-grams at the **corpus-level** and then averaging them over **sentences**. Such method can lead to scores greater than 1, as mentioned in the following GitHub issue ([https://github.com/neural-dialogue-metrics/Distinct-N/issues/2](https://github.com/neural-dialogue-metrics/Distinct-N/issues/2)). To address this, we have updated our evaluation code to count distinct n-grams and **calculate rates at the corpus-level**, ensuring the scores fall within the range of 0 to 1.\n\n- ReR@k in Table 2\n\n    In the context of ReR@k (Recall in Response) metrics shown in Table 2, we counted the correct movies and averaging them in a token-level. The reported results in [1] were replicated by counting the correct movies in a token-level and averaging them in a sentence-level.\n\n**(2) Experiment results**\n\n- Performance comparison using different metric calculation methods\n\n    Considering that DialoGPT-small serves as a backbone model for RecInDial, we have compared DialoGPT + ConKD (hard) to the baseline for a fair assessment. Both models were finetuned on the Redial dataset, and the reported scores are averaged over three runs. The results are presented in the tables below:\n    \n    **[Baseline metrics calculation method]**\n\n    |          Models         |   Dist-2  |   Dist-3  |   Dist-4  |   ReR@1  |   ReR@10  |   ReR@50  |\n    |:-----------------------:|:---------:|:---------:|:---------:|:--------:|:---------:|:---------:|\n    |        RecInDial        |   0.413   |   0.663   |   0.815   |   0.023  |   0.118   |   0.273   |\n    | DialoGPT + ConKD (hard) | **1.326** | **2.217** | **2.704** | **0.03** | **0.161** | **0.337** |\n\n    **[Our metrics calculation method]**\n\n    |          Models         |   Dist-2  |   Dist-3  |   Dist-4  |   ReR@1   |  ReR@10  |  ReR@50  |\n    |:-----------------------:|:---------:|:---------:|:---------:|:---------:|:--------:|:--------:|\n    |        RecInDial        |   0.065   |   0.124   |   0.183   |   0.017   |   0.088  |   0.203  |\n    | DialoGPT + ConKD (hard) | **0.179** | **0.344** | **0.489** | **0.022** | **0.12** | **0.25** |\n\n    In both methods used for metric calculations, it shows that our approach consistently outperforms RecInDial. In terms of the recommendation metrics (ReR@k), despite having a gain over DialoGPT as shown in Table 2, the results of RecInDial are inferior to those of ours. We hypothesize that *our contextualized knowledge distillation concept enables the student model to learn class-wise relations, which are not addressed by the addition of knowledge bias and vocabulary switching [1]*. Furthermore, it is evident that the simple integration with vocabulary pointer [1] fails to generate diverse responses, as indicated by the DIST-n scores.\n\n- Overall performances\n\n    We provide overall performances with additional evaluation metrics outlined in our paper, including DIST-n and PPL for dialogue performance, as well as the F1@k (harmonic mean of ReR@k and PrR@k) and Rec Ratio for recommendation performance. \n\n    |          Models         |   DIST-2  |   DIST-3  |   DIST-4  |    PPL    |    F1@1   |   F1@10   |    F@50   | Rec Ratio |\n    |:-----------------------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n    |        RecInDial        |   0.065   |   0.124   |   0.183   | **5.858** |   0.020   |   0.099   |   0.229   |   0.438   |\n    | DialoGPT + ConKD (hard) | **0.179** | **0.344** | **0.489** |   12.412  | **0.022** | **0.116** | **0.243** | **0.525** |\n\n    We have observed that RecInDial tends to generate relatively simple and concise responses, resulting in lower PPL scores. While these responses exhibit fluency, they lack the desired level of engagement, as indicated by the low recommendation ratio.   \n\n- Efficiency considerations\n\n    In terms of efficiency, we compared the inference speeds of unified models capable of handling the recommendation and generation tasks in the final outputs.  On the Redial dataset, DialoGPT and DialoGPT+ConKD (hard) achieved inference latencies of 5.464ms and 5.306ms per token, respectively. In contrast, RecInDial incurred a slightly higher latency of 6.100ms per token. This additional latency in RecInDial can be attributed to the **computation of the knowledge-aware bias** and **vocabulary pointer**, which controls the choice between generating items or general words **in every time step**. Importantly, *our adaptive gates empower a language model to acquire such capability during training, eliminating the need for additional components in inference*.\n\n    In addition, maintaining a knowledge graph for building user representations during inference introduces scalability challenges; the graph size grows with the addition of items and attributes. This is in contrast to our approach, where *a single language model handles the knowledge-aware recommendations.*\n\n    In conclusion, our approach outperforms the baseline and shows superior results in recommendation performances without sacrificing efficiency. We will add the results and changes in metric calculations in the final version of our paper.\n\n**2. Other recent baselines** \n\nWe appreciate your mention of other recent baselines, and we have carefully reviewed the ones you referenced. Upon our review, we observed that [2] is particularly well-aligned with our research direction and has achieved the promising performance results. \nDespite our efforts to experiment with the source code provided by [2], we encountered substantial challenges during this process; mainly stemming from the extensive training time required and encountered errors within their codebase.  The process involves sequential stages, encompassing semantic fusion and pre-training, conversation prompt training and recommendation prompt training, which cannot be executed in parallel. These factors made it difficult for us to successfully reproduce their results within the limited time.\n\nWe are fully committed to replicating their work during the remaining rebuttal period. \n\n**3.\tLack of explanation regarding the rationale behind the integration of knowledge distillation and gated switching mechanisms.**\n\nThank you for your feedback and for recognizing the integration of knowledge distillation and gated switching mechanisms in our work. We would like to clarify the rationale behind the integration.\n\nIn the context of natural language generation, especially in Conversational Recommendation, different knowledge is required at each time step to complete a proper response. *The core motivation behind incorporating adaptive gate mechanism into knowledge distillation is to address two fundamental questions: **“What knowledge should we distill”** and **“How much knowledge to be distilled”** at this time step.* Within our framework, a dialogue teacher assesses these aspects dynamically at every time step. *This enables us to adaptively measure the quantity of each knowledge to distill in a contextual manner.* This is built on an assumption that the sum of item probabilities mapped by the dialogue teacher reflects the extent to which recommendation is expected. The validation of the assumption is also provided in Line 343-354 of our paper.  \n\nAdditionally, our mechanism offers the advantage of *extending knowledge transfer beyond the specific movie indexes*. In our approach, the contextualized gates are computed by aggregating the probability mass on the interest set, rather than identifying specific indexes. Thus, the knowledge transfer can be performed across multiple domains in a contextual manner. Furthermore, our mechanism eliminates the need for additional components such as vocabulary pointer with knowledge-aware bias [1], and avoids the complexity of a two-step inference (generate and revise) [2].    \n\nWe will add this detailed explanation in the final version of our paper.\n\n**4.\tThe performance improvements resulting from the proposed method are not significant.**\n\nWe would like to clarify that, while our models do not always achieve the top rankings in certain dialogue metrics within automatic evaluation results, it's essential to recognize that *our approach achieves a well-balanced combination of the dialogue and recommendation capabilities with significant improvement in recommendation metrics*. In many cases, achieving the best performance in one specific metric can sometimes come at the cost of sacrificing performance in other areas.\n\n**5.\tThe experimental results reported in this paper is not convincing**\n\nWe have clarified the details of evaluation in the answer of the question 1.\n\n**6.\tClarify the inconsistency between qualitative and quantitative evaluation results.**\n \nWe appreciate your observation regarding the potential inconsistency between our qualitative and quantitative evaluation results. We would like to clarify the distinction between these two evaluation approaches. The qualitative evaluation assesses more nuanced properties of generated responses, such as informativeness and coherence. On the other hand, quantitative evaluation focuses on assessing response diversity. It’s essential to recognize that *quantitative results can indicate positive performance even if a language model generates responses that are meaningless or lack fluency, as long as the generated tokens are diverse*. \nIn terms of the fluency which is used in both evaluations (Flu in Qualitative and PPL in Quantitative) as shown in Table 3, we would like to highlight that our KGSF+ConKD (hard) model consistently performs the best. Furthermore, our KGSF+ConKD (soft) model ranks as the second-best model.\n\n**7.\tProvide details of the human evaluation scheme.**\n\nThank you for pointing out the lack of details in human evaluation scheme. In addition to Appendix G, we provide the detailed guildelines provided to the human annotators below.\n\nFluency: Fluency encapsulates the naturalness of the generated text. It involves an assessment of how the output adheres to linguistic standards, avoiding grammatical flaws. Annotators should evaluate the syntactic flow, word choice, and overall readability. The scores should be shown as 0, 1, and 2, where each indicates “not fluent”, “readable but with some flaws”, and “fluent”, respectively.\n\nInformativeness: The informativeness encompasses the model's ability to convey relevant and accurate information. Annotators should assess the depth and accuracy of the conveyed information. The scores need to be displayed 0, 1, and 2 where each corresponds to “information is missing or incorrect”, “information is included but insufficient or partially inaccurate”, and “comprehensive and accurate information”, respectively.\n\nCoherence: Coherence entails the harmonious integration of the model's output within the evolving conversation. Annotators should assess how well the model comprehends and adheres to the conversation's theme, avoiding abrupt shifts and ensuring a natural conversational flow. The scores should be valued using 0, 1, and 2. Each rating represents  “awkward conversation flow”,  “make sense but somewhat disconnected”, and “coherent”, respectively.\n\nWe will add this detailed description in the final version of our paper.\n\n[1] Recindial: A unified framework for conversational recommendation with pretrained language models, AACL 2022\n\n[2] Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning, KDD 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The knowledge distillation is first adopted to CRS.", "reasons_to_reject": "The main issue with this problem is that the authors ignore a crucial and related baseline [1]. To my knowledge, the work titled \"RecInDial\" [1] delves into the concept of a vocabulary pointer, integrating recommendation and generation modules, and proposing an end-to-end evaluation. Both this work and [1] conduct experiments on the ReDial dataset and evaluate the recommendation performance in an end-to-end manner, with [1] demonstrating significantly better performance than this work.\n\nIn addition to [1], which presents the most analogous idea and evaluation framework, there exist other recent publications in the field of Conversational Recommender Systems that have not been compared or reviewed, namely [2], [3], and [4].\n\nHere are the references for the mentioned publications:\n\n[1] Recindial: A unified framework for conversational recommendation with pretrained language models, AACL 2022\n\n[2] C²-CRS: Coarse-to-Fine Contrastive Learning for Conversational Recommender System, WSDM 2022\n\n[3] A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systems, TOIS 2023\n\n[4] Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning, KDD 2022\n\nAlthough I appreciate the integration of knowledge distillation and gated switching mechanisms in this work, the rationale behind incorporating these mechanisms remains insufficiently explained. Additionally, the performance improvements resulting from the proposed method are not significant.\n\nWhat’s more, the experimental results reported in this paper is not convincing. \n\n1. the evaluation results in the orginal papers (KGSF, KBRD, RevCore) is much higher than the results listed in this paper, the author did not give a clear explanation.\n\n2.the evaluations of generation are not consistent in Qualitative and Quantitive results, see Table 3. For example, KGSF + KonKD(hard) shows generally the best performance in Qualitative results, but it shows almost the worst among the variants.\n\n3.The human evaluation lacks credibility due to the inadequately described valuation scheme.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new method for conversational recommendation, which integrates the recommendation and generation with gated knowledge distillation. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Recindial: A unified framework for conversational recommendation with pretrained language models, AACL 2022\n\n[2] C²-CRS: Coarse-to-Fine Contrastive Learning for Conversational Recommender System, WSDM 22\n\n[3] A Unified Multi-task Learning Framework for Multi-goal Conversational Recommender Systems, TOIS 2023\n\n[4] Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning, KDD 2022"}}, {"review_id": "LjtxB1DSF3", "reviewer": "Reviewer_mGM1", "summary": "", "questions": "NOTE: ALL my questions were lifted by authors' responses. I do not have further questions. \n\n1. Please elaborate on how the authors compare between their methodology and Xiaolei Wang et al.'s work titled \"Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning.\" Specifically, emphasize the distinct contributions of the proposed approach over a more direct and explainable knowledge graph-based method. \n\n2. Upon thorough examination of the dataset, a noticeable semantic linkage between the prompt and the recommendation, represented by attributes, becomes evident. For instance, instances where the prompt \"romantic\" corresponds to the recommendation \"Titanic.\" In light of this observation, please justify the advantages of the proposed approach in contrast to semantic-aware recommendation techniques, underscoring its appropriateness and value in the field.\n\n3. Please provide valuable insights into how this research could extend its contributions to other domains of recommendation, such as E-commerce, wherein customers' prompts may lack sufficient informativeness. Consider scenarios such as search queries encountered on platforms like Amazon and eBay.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedbacks and acknowledging the novelty of our approach. We have taken each of your concerns into careful consideration.\n\n**1.\tInclude empirical validation of real-time application efficiency & comparison with incorporating knowledge graph (KG) approach such as [1]**\n \nWe appreciate your emphasis on efficiency evaluation and comparing our approach to KG-incorporating method models such as [1]. Despite our efforts to experiment with the source code provided by [1], we encountered substantial challenges during this process; mainly stemming from the extensive training time required and encountered errors within their codebase. The process involves sequential stages, encompassing semantic fusion and pre-training, conversation prompt training, and recommendation prompt training, which cannot be executed in parallel.\n\nWhile we remain committed to replicating their work during the remaining rebuttal period, we have also **introduced an alternative baseline, RecInDial [2]**. RecInDial closely aligns with the points raised in your suggestion as well as our research direction. To be specific, a knowledge-aware bias learned from a knowledge graph is incorporated into the language model. Additionally, [1] employs a vocabulary pointer to control predicting either items or words during response generation.\n\n**Performance comparisons**\n\nFor a fair comparison, we have chosen to compare DialoGPT + ConKD (hard) with RecInDial in this response, given that DialoGPT-small serves as backbone model for RecInDial. \n\n|          Models         |   DIST-2  |   DIST-3  |   DIST-4  |    PPL    |    F1@1   |   F1@10   |    F@50   | Rec Ratio |\n|:-----------------------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|\n|        RecInDial        |   0.065   |   0.124   |   0.183   | **5.858** |   0.020   |   0.099   |   0.229   |   0.438   |\n| DialoGPT + ConKD (hard) | **0.179** | **0.344** | **0.489** |   12.412  | **0.022** | **0.116** | **0.243** | **0.525** |\n\nBoth models were finetuned on the Redial dataset, and the reported scores are averaged over three runs. \nThe results of ConKD are from the current version of our paper, where F1@k represents the harmonic mean of the ReR@k and PrR@k.\n\nIn terms of the recommendation metrics (F1@k), it is evident that the results of RecInDial are inferior to ours, *with a notably lower recommendation ratio.* We hypothesize that our contextualized knowledge distillation concept enables the student model to learn class-wise relations which are not addressed by the addition of knowledge bias and vocabulary switching [2]. \n\nFurthermore, our results clearly show that the simple integration with vocabulary pointer [2] fails to generate diverse responses, as indicated by the DIST-n scores. We have observed that RecInDial tends to generate relatively simple and concise responses, leading to lower PPL score. While these responses exhibit fluency, they lack the desired level of engagement, as indicated by the low recommendation ratio.   \n\n**Efficiency Considerations**\n\nIn terms of efficiency, we compared the inference speeds of unified models capable of handling the recommendation and generation tasks in the final outputs. On the Redial dataset, DialoGPT and DialoGPT+ConKD (hard) achieved inference latencies of 5.464ms and 5.306ms per token, respectively. In contrast, RecInDial incurred a slightly higher latency of 6.100ms per token. This additional latency in RecInDial can be attributed to the **computation of the knowledge-aware bias** and **vocabulary pointer**, which controls the choice between generating items or general words **in every time step**. Importantly, our adaptive gates empower a language model to acquire such capability during training, *eliminating the need for additional components in inference.*\n\nIn addition, maintaining a knowledge graph for building a bias during inference introduces scalability challenges; the graph size grows with the addition of items and attributes. This is in contrast to our approach, where **a single language model handles the knowledge-aware recommendations.**\n\nIn conclusion, our approach outperforms the KG-incorporating baseline [2] and shows superior results in recommendation performances without sacrificing efficiency. We will add the results in the final version of our paper.\n\n**2.\tWhat are the distinct contributions of this approach compared to [1]? Specifically, emphasize the contributions over a more direct and explainable KG-based method.**\n \nThe distinct contribution of our method mainly lies in “flexibility” and “efficiency”. Our work is not limited to specific model structures or input formats, allowing flexible adaptation of large language model and recommendation model. \n\nIn contrast, [1] introduces a pre-training stage for semantic fusion between word and item embeddings, and the resulted representations are leveraged in conversation and recommendation prompt learning. During inference, they generate a response template first, and then generate a recommendation, which requires to run the language model twice. These sequential procedures during both training and inference not only hinder efficiency but also limit the flexibility of integrating two architectures.\n\n Furthermore, maintaining a knowledge graph introduces scalability issues as discussed in the answer of question 1. Since the knowledge information is transferred during training, our mechanism eliminates the need for additional components such as vocabulary pointer with knowledge-aware bias [2], and avoids the complexity of a two-step inference [1].\n\n\n**3. What are the advantages of the proposed method over the semantic-aware recommendation (prompting “romantic” may lead to recommend a romance movie)?** \n \nWe would like to highlight that the Conversational Recommendation System (CRS) is not merely “controlled text generation”, but a recommendation task within multi-turn dialogues.\n\nFor example, not only the top-1 recommendation but also the top-k items are crucial aspect to evaluate the recommendation performance. ConKD enables a single model to learn class-wise relationships (relationship between items) given a dialogue context, e.g., “User: I like romance movie and Leonardo dicaprio” -> “System: I would recommend a romance movie, ***“Titanic:0.8, Lalaland: 0.1, About time: 0.1”***. These relations are expressed through the probability distribution provided by the recommendation teacher, which are not addressed in semantic-aware recommendation.\n\nIn addition, when user preferences are represented by multiple attributes or not explicitly stated in a multi-turn dialogue, it is difficult to select relevant attributes to be prompted. ConKD eliminates the need for attribute extraction, enabling contextualized recommendations in multi-attribute or no-attribute scenarios. \n\n**4. How this research could extend its contributions to other domains of recommendation (E-commerce). Consider scenarios such as search queries encountered on Amazon.**\n\n Our current approach **incorporates knowledge** from the recommendation and dialogue teachers using **adaptive gates**. In e-commerce scenarios, we can adapt this approach in the following ways.\n\n**Explicit and implicit search queries require different knowledge:**\n\nThe gate computation can be performed by comparing the information contained in the dialogue teacher and recommendation teacher, taking into account the models’ uncertainty. Specifically, when the user query is vague (resulting in higher uncertainty of the recommendation model due to the absence of specified attributes), the language model can take a more prominent role by providing context-aware recommendations. Conversely, when users provide rich attributes (resulting in lower uncertainty of the recommendation model), the recommendation model can be more engaged to provide an accurate recommendation.\n\n**Handling multiple domains in a contextual manner:**\n\n Moreover, the gate values can be utilized to effectively manage multiple domains in the e-commerce context. \nFor instance, consider a situation where there are distinct recommendation teachers for different product domains. The language model produces contextualized item probabilities given the user query, which can be used as weights for each recommendation teacher from various domains, ensuring that the most relevant suggestions are provided to the user based on the context and preferences. \n\nThank you for the constructive feedbacks and suggestions. Currently, the public datasets are centered around movie recommendation, but we recognize that e-commerce datasets will be helpful to validate the generalization ability of our model. We will actively explore multi-domain e-commerce datasets during the rebuttal period.\n\n  \n\n[1] Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning, KDD 2022\n\n[2] Recindial: A unified framework for conversational recommendation with pretrained language models, AACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This article offers a well-structured and motivated research topic, offering clear and novel insights through comprehensible figures and detailed explanations of experimental setups and methodologies, thereby enhancing clarity and reproducibility.\n\nThe proposed teacher-student model structure for multi-task learning in a unified conversational recommendation system, which simultaneously addresses recommendation and dialogue tasks, outperforms existing methods in terms of recommendation performance and response fluency, while achieving comparable results in response diversity. This approach demonstrates its unique novelty. The authors rigorously validate the effectiveness of their approach through extensive experiments, along with a detailed ablation study that convincingly justifies the efficacy of the proposed teacher model. Lastly, the qualitative results remain intuitive.", "reasons_to_reject": "NOTE: The information below were lifted by authors' justification. I do not see such concerns anymore. \n\nThe proposed model architecture shows superior performance compared to existing solutions; however, empirical validation of its real-time application efficiency is lacking. For a dialogue-based recommender system, assessing feasibility in real-time application is crucial. To address this, conducting benchmarking against other solutions to evaluate its computational efficiency is recommended. This foundational step will serve as a guide for future research directions.\n\nFurthermore, it is noteworthy that a similar work titled \"Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning\" exists, which utilizes knowledge graphs instead of introducing additional learnable modules. This approach also demonstrates promising performance, but its methodology for incorporating knowledge priors requires further evaluation and comparison. Although the ablation study shows the effectiveness of the introduced method for learning contextual knowledge, additional evidence is necessary to establish its appropriateness as the optimal approach.\n\nFinally, questions persist regarding the significant contributions of this approach to the field. Existing similar works have evaluated on the same dataset, limited to the domain of movie recommendations. It is essential to demonstrate the generalization ability of this approach to other domains, such as E-commerce recommendation, to establish its broader applicability and impact.", "questions_to_authors": "NOTE: ALL my questions were lifted by authors' responses. I do not have further questions. \n\n1. Please elaborate on how the authors compare between their methodology and Xiaolei Wang et al.'s work titled \"Towards Unified Conversational Recommender Systems via Knowledge-Enhanced Prompt Learning.\" Specifically, emphasize the distinct contributions of the proposed approach over a more direct and explainable knowledge graph-based method. \n\n2. Upon thorough examination of the dataset, a noticeable semantic linkage between the prompt and the recommendation, represented by attributes, becomes evident. For instance, instances where the prompt \"romantic\" corresponds to the recommendation \"Titanic.\" In light of this observation, please justify the advantages of the proposed approach in contrast to semantic-aware recommendation techniques, underscoring its appropriateness and value in the field.\n\n3. Please provide valuable insights into how this research could extend its contributions to other domains of recommendation, such as E-commerce, wherein customers' prompts may lack sufficient informativeness. Consider scenarios such as search queries encountered on platforms like Amazon and eBay.", "paper_topic_and_main_contributions": "The authors present a novel approach proposing a multi-task learning method for a unified conversational recommendation system. Unlike prior research that employs separate modules for recommendation and dialogue tasks, the proposed approach jointly learns both tasks for recommendation and dialogue semantics respectively. This work also introduces two versions of contextualized knowledge distillation to integrate knowledge from task-specific teachers, which represents a significant contribution to the field. Furthermore, the paper addresses the crucial issue of the discrepancy between recommendation and dialogue tasks in conversational agents, a facet that has not been explicitly explored in previous works. The experimental results provided in the paper demonstrate the efficacy of the proposed approach, showcasing substantial improvements in recommendation performance and dialogue fluency when compared to existing methodologies.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. Kindly provide methodology comparison to a similar work - Xiaolei Wang, et at., \"Towards Unified Conversational Recommender Systems via\nKnowledge-Enhanced Prompt Learning\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for conversational recommendation, which integrates recommendation and generation with gated knowledge distillation.

The authors carefully addressed most concerns (e.g., missing crucial baselines and unconvincing experimental results) from the reviewers in the rebuttal. The authors are encouraged to add all the new experimental results in their paper if accepted. The authors should also make it extremely clear that they have revised the evaluation metric. It would be nice to add results corresponding to the original metric in the appendix as well.
Decision: Accept-Main