Reinforced Target-driven Conversational Promotion
Authors: Huy Quang Dao
Lizi Liao
Dung D. Le
Yuxiang Nie
Venue: EMNLP 2023 Main (2023)
Keywords: Conversational Promotion
Conversational Recommendation
Target-driven Recommenders
TL;DR: In this work, we focus on a challenging setting called target-driven conversational promotion and introduce RTCP - a novel reinforced promotion framework.

Abstract: The ability to proactively engage with users towards pitching products is highly desired for conversational assistants. However, existing conversational recommendation methods overemphasize on acquiring user preferences while ignore the strategic planning for nudging users towards accepting a designated item. Hence, these methods fail to promote specified items with engaging responses. In this work, we propose a Reinforced Target-driven Conversational Promotion (RTCP) framework for conversational promotion. RTCP integrates short-term and long-term planning via a balanced gating mechanism. Inside which, the dialogue actions are predicted via a knowledge-integrated multi-head attention and guided via reinforcement learning rewards. RTCP then employs action-guided prefix tuning to generate relevant responses. Experimental results demonstrate that our model outperforms state-of-the-art models on both automatic metrics and human evaluation. Moreover, RTCP has a strong capability in quickly adapting to unseen scenarios just by updating prefix parameters without re-training the whole model.

Reinforced Target-driven Conversational Promotion
Huy Quang Dao' 12, Lizi Liao", Dung D. Le", Yuxiang Nie 'FTT Software AI Center, "Singapore Management University College of Engineering and Computer Science, VinUniversity "Hong Kong University of Science and Technology huydao98 uet@gmail. com, izo@smu. edu. sg dung. leevinuni..n.....mmmmmmmm...i.ii edu. vn, ynieae@connect.....mmmmmmmm ust. hk
Abstract
Targeted hem:: King Pheris
The ability to proactively engage with users towards pitching products is highly desired for conversational assistants. However, exist- ing conversational recommendation methods overemphasize on acquiring user preferences while ignore the strategic planning for nudg- ing users towards accepting designated item. Hence, these methods fail to promote speci- fied items with engaging responses. In this work, we propose Reinforced Target-driven Conversational Promotion (RTCP) framework for conversational promotion. Specifically, RTCP integrates short-term and long-term plan- ning via a balanced gating mechanism. In- side which, the dialogue strategies are pre- dicted via knowledge-integrated tention and guided via reinforcement learning rewards. RTCP then employs an action-guided prefix tuning method to generate relevant re- sponses. Experimental results demonstrate that our model outperforms state-of-the-ar models on both automatic metrics and human evalua- tion. Moreover, RTCP has strong capability in quickly adapting to unseen scenarios just by updating prefix parameters without re-training the whole model. Code and data are here! !
Converaation
Dialogue Actions
Bot: Good momning Mt XXXX
Coul: Croelinga
Toptc: Homs
Y Bet: Fime's nowrything going mocntyy
User Good moming
User (telimemend very enjpystle
Bot: Let's belt about buy Chos. Jay Cheus tn the bret-ood ling
Chirsow arkal World Music Awards for lout larane Her
good
Geal: Oue about voure
scrape
Tople: Chou
Bot: Juy Chou only good ainging but gooe acting
He onco won the Chincse Film Media Awards for Best New Actor
Usar: only know that be singe wel but howen' novr the mouine
Bott you want see Hung Ru Fandy 3 caarnd Joy Chou.
User: Haooert sases the moves, that good
Goat Modee Becommercaion Topric Kangha Panda
Bot: if good movio can called a tar plopp
fimms
recommendationn
Goal Say Goood Paye Toppi:t None
Bot em. wont diaauts you arymarre Good Bye
User: Bye
integrated multi-head at-
Figure I: An illustration of the target-driven conversa- tional promotion setting.
However, a more challenging scenario would be target-driven conversational promotion, where the system mimics a salesperson to nudge the users towards accepting a pre-given item. As illustrated in Figure 1, the bot tries to recommend given target 'Kung Fu Panda 3' to the user via multiple turns of conversation. It gradually nudges the user via chatting about 'JayChou', on his well-known achievements in music, and then recommends his movie, which helps the user to get interested and ac- cept the target with higher chance. Such scenario is highly desired due to the potential for boosting sales revenue and reducing human costs but also calls for higher requirements on wisely promoting items and proactively engaging with users (Liao et al., b023a, Deng et al., 2023a). Although ex- isting CRS models (Chen et al.. 2019; Zhou et al.. 2020a; Lu et al., 2021; Zhou et al., 2021a; Liang et al., 2021; Zhou et al., 2022; Wang et al., 2022c) have achieved promising results, they tend to adopt relatively passive
1 Introduction
In recent years, conversational recommender sys- tems (CRSs) (Li et al., 2018; Kang et al., 2019; Zhou et al., 2020b; Ma et al., 2021; Zhou et al., 2021b; Li et al., 2022; Ren et al., 2022; Zou et al.. 2022; Chu et al., 2023) have gained considerable attention from both academic researchers and in- dustrial practitioners. Inside such systems, most CRS models will first infer user preferences via multi-turn conversations and then recommend set of potential items when appropriate. The essential goal for such models is to solicit user interests accu- rately and then map such interests to some specific items in repository.
guessing user preferences via conversations, hence cannot be di- rectly applied to the new proactive nudging setting. Generally speaking, the new conversational pro-

---

motion setting brings in two new obvious chal- lenges that press for solutions. Firstly, as the virtual assistant needs to converse gradually to convince the user in accepting the target item, it is essential to produce an appropriate plan, e.g. in a sequence of action tuples, that leads to the target item while can also keep the user interested in the conversation. Secondly, once the plan is predicted, the other im- portant requirement is to generate proper responses that are in line with the plan accurately. Also, likely that the model would encounter new conver- sation scenarios with unseen dialogue plans. The model should be able to adapt to such scenarios quickly and handle them properly, in order to avoid disappointing the users.
2
Related Work
2.1 Conversational Recommender Systems
The goal of CRS models is to offer personalized recommendations via interactive dialogues. One line of CRS methods (Lei al., 2020a.b; Ren al., 2021; Deng et al., 2021; Hu et al., 2022; Tu et al., 2022) mainly focuses on improving the perfor- mance of item recommendation, where they ask clarifying questions to gradually find an optimal candidate set. Therefore, the quality of generated responses is less emphasized as these works only leverage pre-defined response templates (Lei et al., 2020a,b; Ren et al., 2021) to interact with the users.
Another line of CRS research integrates recom- mendation modules into dialogue systems so that the item recommendation and the response gener- ation objectives could be jointly optimized. Some efforts (Chen et al., 2019; Zhou et al., 2020a; Liao et al., 2020; Liang et al., 2021; Lu et al., 2021; Wang et al., 2022b; Zhou et al., 2022; Zhang et al., 2023) utilize different knowledge resources, such as knowledge graphs (Bizer al., 2009; Wu et al., 2022) and user reviews to enrich extracted informa- tion from the conversations. However, such CRS systems lack appropriate conversational strategies to interact with users. Therefore, other recent meth- ods (Liu et al., 2020, 2021; Hayati et al., 2020; Zhang et al., 2021) aim to develop goal-oriented CRS models that can converse with users by using different strategies (e.g chitchat, asking questions or recommendation). Liu et al. (2020) make the first attempt by introducing a goal-oriented CRS dataset called DuRecDial. State-of-the-at goal- driven CRS approaches (Wang et al., 2022a; Deng et al., 2023b) attempt to improve their short-term planning abilities with either target-side informa- tion or multi-task learning.
To address the aforementioned challenges, we propose systematic framework-Reinforee Target-driven Conversational Promotion (RTCP) for the new setting. For planning, RTCP integrates long-term planning and short-term planning via a balanced gating mechanism. The former takes care of achieving the predefined target, while the latter looks into engaging the user and the smoothness of the conversation. Specifically, RTCP predicts action tuples in a plan via knowledge-integrated multi-head attention and is guided via estimated Reinforcement Learning rewards. For response generation, RTCP employs novel action-guided prefix tuning method to regulate the response gen- eration being in line with the plan. In particular, it pushes action plan factors into different prefix parameters, while leaves the main generation task the pre-trained language model. This helps the model to learn and adapt better, especially when new action plans happen. To sum up, our contribu- tions are threefold:
We emphasize on new Larget-driven conver- sational promotion setting where the agent nudges the user towards accepting a specified item via multiple turns of conversation.
However, since these models mainly produce items according to the user's preference, is non- trivial for these methods to transfer into a setting when target item is given by the system before- hand. In this work, we utilize both short-term and long-term planning and combine these two strate- gies via flexible balancing mechanism.
We propose a reinforced target-driven conver- sational promotion framework that balances between short-term and long-term planning, while further integrates action-driven prefix tuning better guide the generation of rele- vant responses.
2.2 Controlled Response Generation
Experiments show that RTCP outperforms the baselines on both automatic metrics and hu- man evaluation. It also has a strong ability to quickly adapt to unseen conversation scenar- ios without re-training the whole model.
Recently, surge of works (Zhong et al., 2021: Qin and Eisner, 2021; Ye et al., 2022b,a) focuses on leveraging the power of pre-trained language models for improving response generation. Among them, prefix-based methods aim to tune a set of

---

Planning Module
Long-Term Planning
Generation Module
Environment
State Sc-1 (arli=i
Reward = f
Agent
Action a,
Goal Prefix
Topic Prefix
Task Prefix
Dialogue History
Target =(0.0 Ø)
Ta ===vv v gt and P: else 0
Gating
Knowledge
Short-Term Planning
Mechanism +1 0
Context C
Integrade Attention Model
Action dgg
Previous Actions A
Generated Response 'gen
The architecture of the proposed RTCP model. The planning module predicts action tuples via balancing between short-term and long-term planning to keep user engaged and approach the target. With predicted action tuples, the generation module employs action-guided prefix tuning generate relevant responses and realize fast adaptation to unseen scenarios.
Figure 2:
trainable parameters, which are prepended to the input and control text generation (Li and Liang, 2021). Lester al. (2021) propose similar method, where they prepend the input sequence with special tokens and then they directly train the embeddings of these tokens. Such methods side- walk the heavy burden of training separate large models for different settings. Inspired by these, we incorporate predicted strategies into separate prefix parameters and give different inputs to these prefixes for controlled response generation.
action tuples to please the users and recommend the targeted item when appropriate.
Response Generation: For the 1-th turn, given historical context X,, knowledge base K. and pre- dicted action tuple ât, we aim to generate coher- ent response Y, to the user.
4 Methodology
We show the overall framework in Figure 2. Our RTCP consists of two components namely planning in Section 4.1 and generation in Section 4.2.
3 Preliminary
4.1 Long Short-Term Strategic Planning
Notations. We denote by G the set of all goals (e.g. chit-chat, movie recommendation, etc), and denote by P the set of all topics (e.g The Conjur- ing, Jaychou, etc). At the 1-th turn, we represent an action tuple by pair of goal and topic Pt (i.e (9),Pt)). Each conversation D = (C, K, A) is a tuple of three elements namely the conversa- tion content C, set of all associated knowledge K
At each turn, the planning module needs to pre- dict an action tuple à; (Ôt: (t). For target-driven recommendation setting, it is crucial to balance between two objectives: (1) engaging users with relevant topics; (2) recommending the target item while avoiding lengthy conversations. Correspond- ingly, we adopt short-term and long-term planning to achieve these two aforementioned objectives.
Nx { (NR is the size of the knowledge M (M is the number of turns). The conversation con- text C {(,,, is set of all historical user utterances and corresponding responses. The goal of conversational promotion is to interact with the users to persuade them to accept a targeted item. We decompose the problem into sub-tasks includ- ing (1) the dialogue strategy planning and (2) the response generation tasks.
base) and a sequence of action tuples 1 = {
4.1.1 Short-Term Planning
Similar to existing works (Zhang et al., 2021; 1; Deng et al., 2023b), we consider short-term planning as the task of predicting the next action tuple.
Knowledge-integraated Attention To effectively leverage different kinds of inputs, we propose utilize Knowledggeinntegrated Attention block gradually incorporate the previous action tuple se- quence A, historical context X, and knowledge base K in order to produce the final prediction (as
Dialogue Strategy Planning: Formally, given a designated item v E P. We aim to produce a plan A {āi, â2 a., 3m3 which is a sequence of

---

described as follows):
sequence of states, actions, and rewards. For- mally, MDP consists of tuple of five elements M = (S, P, R, 7). At the 1-th turn, we de- fine the state St E as the sequence of previ- ous t 1 action tuples in chronological order (i.e 8, = {aı 02 ..... 4-1 The action space A con- sists of all available action tuples (i.e A {aj (9g. p})pp P, P }). For state transition, after employing the action tuple at, we append the action tuple at to the end of the current state 8t construct the next state 81 Finally, Y E [0, 1] is the discount factor.
50 MHA (FL ,,,'' 1)((-1) F. ; 1
F(] MHA (FO)) FA, FAD;
PO
F() MHA F Ex, Fx),
PO
F/) MHA E EK, Ex):
where E is the output of the (-th attention block while EA, Ex Æx are representations of the ac- tion path, dialogue context, and knowledge base, which are computed by using separated BERT mod- els (Devlin et al., 2019). MHA and FFN are the multi-head attention and feed-forward neural func- tions (Vaswani al., 2017) respectively.
Reward R. To train the RL agent, we define the following reward function:
[ Ta + Ts:
Prediction via Short-Term Planning. We uti- lize the representation of the [CLS] token of the last layer to simultaneously predict the next goal and topic. Formally, compute probability distri- butions of goals and topics as follows:
where is the final reward and B is hyper- parameter while we refer to Ta: T' target and intermediate rewards respectively. On the one hand, use the former to encourage the agent when it successfully recommends the targeted item. The target reward is computed as follows:
(1))) pg = Softmax (W, O.(ds) + bg);
Pp = Softmax (.) + bp);
(L) (L)
v. if gl = 9.P. - 8, Sts t2. Ta
O,
0, otherwise,
where Pg Pp are probability distributions for the goals and topics respectively while Wg. Wp, by. bp are model's parameters. To train goal and topic planning, we minimize the negative log likelihood functions of the ground-truth goal g" and topic p*:
where is a predefined value, 9, 8 are the targeted goal and item of the current conversation respec- tively. f1, 12 are two ssper-parameters utilized to determine whether the system should recom- mend the targeted item. On the other hand, we adopt the intermediate reward to preserve the rel- evance of topical transitions during the conversa- tions and compute it by applying a binary classifi- cation model on the state st 1 as follows:
N
bg
log P (0 [x, K₁, A;, Qy),
N
Lp log Pp (p; X1, KG, AG, Op),
=aWWa + by),
where N is the total number of training examples and 0g. Op are parameters associated with the goal and topic prediction task respectively.
where hays is the latent representation of the state 8t41, a is the Sigmoid activation function and Ws. by are parameters. To compute h we feed the next state 8141 through BERT model and utilize the output corresponding the [CLS] to- ken. Additionally, train the binary classification model, we consider sequences of strategies from the training set as positive instances and randomly replace some elements in the positive instances to produce negative ones.
4.1.2 Long-Terme Tong-TermBl Planning
this work, we adopt Reinforcement Learning (RL) to enhance the long-term planning ability of our system. The goal is to encourage the system to focus more on recommending the targeted item while avoiding long-lasting conversations.
A MDP Viewpoint of Conversational Planning. We re-formalize conversational planning as a Markov Decision Process (MDP) which includes
Prediction via Long-Terie Planning. At each turn, we compute a policy [aa]st which is a

---

probability distribution over all available action tuples by using the following formulation:
generation model, we prepend learnable continu- ous prompts representing the predicted goal and topic to the input sequence. Formally, given spe- cific action tuple å = (gt, Pt), we construct the input sequence I for the i-th training example as follows.
T aaai St, 0) fo(st):
where function fe parameterized by a neural network and 8 are its corresponding parameters. Finally, following (Zhao et al., 2017), we utilize actor-critic framework to train our RL agent.
Ie GG, Ps, Toen. Xel.
where X, is the historical context while G. Pr, Tr,Toa gen are corresponding continuous prompt tokens representing the goal, topic and the generation task respectively. Noticeably, in contrast these existing prompt tuning methods (Li and Liang. 2021), our proposed approach is data-driven which means the model can adapt better to different individual training examples.
4.1.3 Long Short-Term Strategic Balancing
Based on the conversation situation, the system will need to balance short-term and long-term planning for better performance. Therefore, we propose an adaptive gating mechanism that allows the system to flexibly combine the two aforementioned abil- ities. Specifically, during the inference step, compute weighted combination of two probabil- ity distributions. Formally, the final distribution on action tuples is computed as follows:
Parameter Optimization. We fix the parameters of pretrained GPT2 model and only learn the ad- ditional latent prompts. That means only the goal prompt G, topic prompt Pt, and task specifcc to- kens T Ten are updated during the training process. To learn the response generation task, we optimize the following loss function:
P(a+): = (1 - a) P (at) + a * Piong(a) (at) = (1 - a) * (9)) * Pp(pt) + (at),
where is hyper-parameter that be flexibly chosen by using cross- volss-vliiationn. P Pshort (atorr(at), Piong (ae) are probability distributions computed by the short-term and long-term parts respectively. Here, we assume that goal and topic are two independent random variables. Therefore, the short-term distribution (at) can be computed by taking the product of the two factorial distributions. By varying a, the planning strategy of the system also changes accordingly. A larger value of might encourage the system quickly suggest the targeted item the user. In contrast, by using a smaller a, the system focuses more on maintaining the relevance of topical transitions during the conversations. Consequently, choosing suitable values for a is not trivial and largely depends on the situation at hand.
Lgen
log P gen (Vi.) (e, |G.,
j=1
P T gens X1, Opim).
where N the total number training examples, the length the output sequence and Opm are fixed parameters of the pretrained language model.
5 Experiments
5.1 Experimental Setup
Dataset. this work, we utilize DuRecDial (Liu et al., 2021) and INSPIRED (Hayati et al., 2020) to conduct our experiments. Both of them are orig- inally designed for goal-driven CRS setting. It is worth noticing that for INSPIRED, we only con- duct the goal planning due lacking annotations of topics. We show the detailed statistics of these two datascts the Appendix A.2. In this work, we regard the item which the user will accept at the end of each conversation as the targeted item.
4.2 Action-Guided Response Generation
Given predicted action tuple à that consists of goal Ôr and a topic f.. from the planning part, the language generation model is expected to produce appropriate responses accordingly.
Action-guided Prefix Tuning. Inspired by meth- ods from controlled generation, we enhance the standard prefix tuning (Li and Liang, 2021) with action-driven contents to further calibrate the gen- eration process. Specifically, to train the language
Baselines. In this work, we compare RTCP with several representative baseline methods including general pre-trained language models( GPT2 (Rad- ford et al., 2019), DialoGPT (Zhang et al., 2020), BART (Lewis et al., 2020) ) and state-of-the-..

---

goal-driven CRS methods (KERS (Zhang al.. 2021). TCP (Wang et al., 2022a) and UNIMIND (Deng et al., 2023b)). Moreover, we also report the results of variants of RTCP including RTCP with- out knowledge-inteerraee attention (RTCP-KIA), RTCP without goal-topic prompts (RTCP-GP) and RTCP without task-specific prompt (RTCP-T gen)). We describe the details of baselines in Appendix A.3.
DuRedDul Topic Prediction
INSPIRED
Goal Prediction
Joant Acc.
Goal Prediction
Model
(%)
Aec. (8)
KERS
96. 10
78.41
77.83
19.64
BERT
97.81
68.03
89.61
27.21
97.88
93.00
21:12
UNIMIND
97.36
92.67
92.37
RTCP
98.41
96.27
95.12
30.57
RTCP XIA
98.28
93.08
92.83
24.17
Table I: Results on the short-term planning, demon- strated via goal and topic prediction (t-test, p < 0.05).
Implementation Details We use Pytorch frame- work 2 to implement our conversational promo- tion framework and train the framework on IGPU NVIDIA A100 40G card. We run the model 4 times with different random seeds and compute the aver- aged results. this work, we use the GPT2-base (1 14) and Bert-base (114M) as our backbone models, We set the dimension of hidden vectors to 768. For the knowledge attention model, set the number of layers to 12 (4 for INSPIRED), and the number of attention heads 8. To train our RL module, we the target reward and weighted parameter y to 3.0 and 1.0 respectively. To balance between short-term and long-term planning, we perform cross- cross-validation and empirically set the value of a to 0.6 and 0.9 for DuRecDial and INSPIRED respectively. We use a learning rate of le-5 to train the actor network while the learning rate of the critic is set to 5e-5. For the response generation part, use 50 soft tokens for training the task-specific prefix and 2 for both the goal and topic prefixes respectively. Fi- nally, we train the generation model with a learning rate of 5e-5 with 5 epochs till converge.
DuRecDial (t- 0.6)
INSPIRED (t= 0.9)
Model
Avg. Turnst.)
Avg. Turns())
DialoGPT
17.06
17.08
GPT?
14.77
67.22
4.26
3.33
17.53
BART
22.17
57.67
3.95
2.61
17.75
KERS
1.65
19.72
6.33
1.33
17.75
TCP
7/45
75:43
4.26
8.33
17.23
UNIMIND
1)
16.82
RTCP =
9.21
86.78
3.80
9.16
16.91
RTCP 0.0) GP
X.91
57.44
3.77
7.49
17.16
Table 2: Results on the long-term planning, demon-
strated via the target achievement (t-test, p < 0.05).
sample 20 dialogues and invite two annotators to score those dialogues. We report the results in both turn-level and dialogue-level. For turn-level, we concern three aspects, Fluency, Proactivity and Informativeness, while for dialogue-level we compare user Satisfaction and conversation Co- henrency. The range of scores is from 1 to 3. We measure the inter-annotator agreement by Fleiss' Kappa (McHugh, 2012).
5.2 Main Results
Evaluation Metrics. We evaluate the models on
5.2.1 Results on Short-term Planning
both dialogue planning as well as response gen- eration aspects. In particular, we utilize accuracy (Acc) and joint accuracy (Joint Acc) metrics for short-term planning. For the target accomplish- ment, we report the dialogue-leve success rate (SR), success rate at k-thh turn (SR@k) (Lei et al., b020a,b) and the averaged number of conversation turns (Avg. Turns) needed to successfully recom- mend the targeted item. For response generation, we utilize both automatic and human evaluation. For automatic evaluation, we use perplexity (PPL), word-level F1 (F1), BLEU-N (N=1,2) (Papineni et al., 2002), Dist-N (N = 1,2) (Li et al., 2016) and Knowledge Fl (Know. F1)(Wang et al., 2022a; Zhang al., 2021). For human study, we randomly
Table 1 shows the empirical results of the goal and topic prediction tasks. Overall, pre-trained models such BERT and TCP perform better than KERS. This is because such models are pre- trained on massive amounts of text, hence could produce more meaningful representations for down- stream tasks. Second, we observe that our proposed method RTCP significantly outperforms all base- line methods across all metrics, which indicates the superior short-term planning ability of our model compared to others. The superior performance of our proposed RTCP mainly comes from the knowledgg-inteeraaee attention mechanism. This is supported by the substantial drop in the perfor- mance for the variant RTCP-KIA, in which we removed the inooledgeiintegraeed attention.
hattps:/ppytorc..oo

---

DuRedDial (t= 0,6)
INSPIRED (t= 0.9)
Modd
PPL
F1
BLEL-1
BLEU-2
DIST-1
DIST-1
Know. F1
PPL 11)
F1
BLKU-1
BLEU-2
DIST-1
DIST-2
Know. F1
BART
6.88
28.1
0418
0.361
0.101
41.06
19:47
10.21
0.292
0.112
0.088
0.198
10.16
DialoiPT
0.314
42.30
14.58
0.117
12:42
GPT2
4.09
39.21
0.195
0.319
0.035
0.111
18.19
15.72
13.71
0.356
0.153
0.085
0.218
8.11
KERS
6.69
35.41
0.455
0,278
0.089
0,127
33.19
24.11
10.54
0.228
0.302
0.013
00000
1.23
TCP
4.15
41.11
0.507
0.302
0.035
0.107
60.51
17.11
13.50
0.352
0.149
0.087
0.227
7.38
UNIMIND
0.530
0.381
68.66
15,69
0.159
11.85
RTCP
0.524
0.376
0.034
63.13
14.27
0.351
0:353
0.085
0.211
14.24
RTCP (o 0.0)
3,69
45.39
0.542
0.402
0.036
0,109
70.35
18.61
15.39
0.371
0.168
0.091
0.211
10.21
RTCP 0.00--G GP
3.89
44.66
0.529
0,391
0.034
0.100
05.04
13:48
17.31
0.355
0.167
0.077
02317
17.16
RTCP (o 0.00 Taes
5.10
36.38
0.482
0.326
0.051
0.104
46.36
33.78
13:25
0.346
0.152
0.063
0.154
4.41
Table 3: Automatic evaluation results on the response generation task (t-test, < 0.05). We empirically set a to 0.6 and 0,9 for DiReeDial and INSPIRED respectively.
DuRecDial
Turn-level results
Dialog-level results
Turn-level results
Dialug-level results
Model
Fluency
Infor:
Proactivity
Satisfaction
Coherency
Kappa
Fluency
Infor
Proactivity
Satisfaction
Coberency
Kappa
GFT2
2.935
2.194
1.623
1,95
1.75
0.72
2.921
1,242
1,079
1.10
1.10
0.84
KERS
2.577
1.759
1,376
1.55
1.45
0.75
2.881
1.079
1.019
1.00
1.00
0.88
2.967
2.25
2,895
1.123
1.20
UNIMIND
2,971
2,236
1.659
2.52
0.79
2,932
1.504
1.50
RTCP
2,981
2.246
1.676
2.55
2.45
0.74
2.971
1.752
1.237
1.90
1.30
0.86
Table 4: Human evaluation results the generated responses (Infor. stands for informativeness).
5.2.2 Results on Target-achievvmmnt Task
to effectively guide the response generation task. Second, our RTCP variants significantly outper- form baseline methods on several metrics (6 out of 7 metrics on DuRecDial and 5 of 7 metrics on INSPIRED). This mainly comes from: (I): RTCP could produce more accurate conversational plans as demonstrated Section 5.2.1. (2) RTCP utilizes action action-guided guided prompt tuning which parameterizes dialogue strategies with learnable parameters and adjusts them to better calibrate the generation pro- cess. This is further demonstrated by removing the goal-topic prompts (RTCP (a = -- GP) results in considerable drop in the results. Similarly, we also experience performance degradation when removing the task-specific prompt T yea, which in- dicates the general task-specific tokens are indeed important for the generation task. Finally, we show the performance of RTCP with different values of a. RTCP (a - 0) (only short-term planning) achieves the best results. This is as expected since this evaluation scheme is carried out turn-by-tur and hence favors those methods that directly opti- mize turn-level objectives.
Table 2 shows the performance comparison on the larget-achievement aspect. First, on DuReeDial, relatively general models such as DialoGPT, GPT2, and BART achieve rather good results on SR @1 1. The reason is these models tend to recommend the target item at the beginning state of the conversa- tion since the target item is a part of their input sequences. However, such behavior is less pre- ferred in terms of user engagement and satisfaction. On dialogue-level success rate (SR), target-driven methods such as TCP and UNIMIND perform significantly better than the aforementioned pre- trained language models. Noticeably, our RTCP consistently outperforms all baseline methods and achieves state-of-the-an results. Moreover, com- pared to other baseline approaches, the proposed RTCP manages complete the task in relatively shorter conversations. This can be attributed to RTCP's long-term planning part which is effec- tively guided by reinforcement learning rewards.
5.2.3 Automatic Evaluation on Responses
Table 3 shows the performance comparison on the response generation task based on automatic met- rics. First, target-driven methods such TCP and UNIMIND perform better than pre-trained language models (i.e. BART, GPT2, and Dialog- GPT). Thanks to their short-term planning part, such CRS methods could produce interactive plans
5.2.4 Human Evaluation on Responses
Table 4 shows the results of human evaluation. Overall, compared to other baseline methods, our RTCP framework achieves better performance on both turn-level and dialogue-level assessments. In- terestingly, in the dialogue level, despite achieving a high SR @ as shown in Section 5.2.1, 1, the GPT2

---

baseline still performs worse than TCP and TCP on the satisfaction and coherency metrics. This
12.0
45.0
TCP
40.0
indicates that the annotators are not quite satisfied with responses generated by the GPT2 model. Such an interesting insight further demonstrates our ob- servation that pre-trained language models (GPT2, BART, and DialoGPT) tend to recommend the tar- geted item at the early stage of the dialogues, which is not preferred in the conversational promotion set- ting. Unsurprisingly, our RTCP presents the best results on dialogue-level metrics. This could be attributed to our planning strategy, which could produce appropriate dialogue strategies to manage the conversations.
10.0
8.0
25.0
RTCP
20.0
GPT2
BART
100 5s 80 %6
60
40% 20
100 % 80
60 56
40 95
20
(a) PPI. (L)
(b)
70.0
60.0
70.0
50.0
50.0
40.0
RTCP
RTCP
30.0
TCP
TCP
20.0
GPT2
30.0
GPT2
BART
BART
100 % % 60 15 40 20
20.0
100 % 80 %
% 40 20
(c) Know.
(d) SR
11.9
68.0
Figure 4: Performance comparison on DuRecDial w.r. different ratio the original training data.
10.
87.5
87.25
16.
9.75
86.75
5.3.2 Effectiveness of Training Data Size
923
88:35
Figure 4 illustrates the performance of BART, GPT2, TCP, and the proposed RTCP on different training data sizes. Overall, we can see that RTCP consistently outperforms other baseline approaches in all metrics across all different training sizes. It indicates that RTCP learns faster and more effee- tively than other baselines. This might be due the fact that RTCP only tunes prefix parameters for generation. The backbone generation model inside RTCP is pre-trained GPT2 model which is fixed and will not be updated. For baselines such as BART, GPT2, and TCP, they have large number of tunable parameters, which requires more train- ing data to approach good local optima position. Hence, when the ratio of training data decreases, these methods' performance drops relatively faster.
86.0
(a) SR1
(b) SR
46.0
0.55
0.0
0.2
0.4
42.5 0.6
0.6
1.0
66.6
0.1
30.3
BLEU-I
Figure 3: Performance of RTCP DuRecDial with
different values of balancing parameter a.
5.3 Detailed Analyses
5.3.1 Effectiveness of Balancing Parameter
Figure 3 shows the results on $R@1, 1, SR, Fl and BLEU-1 metrics of our RTCP model with differ- ent values of balancing hyper-parameter a. First, that
5.3.3 Fast Adaptation to Unseen Plans
We further conduct experiment to demonstrate the adaptability of our RTCP model to unseen narios. First, from the of all available topics, we randomly select a subset as unseen topics which is approximately 20% of all topics. Then we divide the original training set into two parts: train_seen and train_wnsedd where the train_unseed set con- tains all the unseen topic samples. We do the same for the original testing set and obtain test_ rest_seen and test unseen. Then we train the proposed RTCP and the comparing TCP model train_seen. For the Without Training setting, we directly apply trained RTCP and TCP models on the test test_unseen for re- sults. For the With Training setting, we further fine-tune RTCP and TCP models on train unseen
increase the value of o,, the @ and SR results gradually increase. This is reasonable since a large value of a means we focus more on recommending the target item. However, it also indicates that over-emphasizing long-term planning would also hurt the model's performance (as SR decreases). Moreover, we notice that for generation metrics such as Fl and BLEU-1, the increases. The
performance result decreases main reason is that the F1 and BLEU-1 metrics work in turn level. When long-term planning con- tributes more, very likely that the model would choose new plans which are significantly different from ground-truie trunh plans.

---

and test on the uns__n
INSPIRED, etc.), the users are likely accept items mentioned at the end of conversations, which eventually limits the difficulty of the new nudg- ing setting. To make the task more challenging, it might be better to construct new promotion- oriented datasets where the systems are given "hard target items" that are less likely to be accepted by the users, and the systems need several rounds of promotions convince the users. (2) Usage of LLMs. Recently, large language models (LLMs) such as ChatGPT or LLaMA have exhibited their out-standing performance on generation. We plan to further explore how to make use of such pow- erful LLM models on the target-driven promotion setting. (3) Reliance on dialogue actions. Exist- ing goal-driven CRS models (KERS, UNIMIND) and our model heavily rely a pre-defined set of dialogue actions, which inherently restricts their usages to real-world scenarios. Improving action- free conversational models will a more potential solution to alleviate such reliance.
Model
F1
BLEU-1
BLEU-2
Know, Pl
TCP
36.15
0.499
0.298
55.04
Without Training
RTCP
43.51
0.532
69.38
RTCP-
0.514
66.07
TCP
42:27
0.500
0.362
56.07
With Training
RTCP
47.54
0.568
0.425
74.31
RTCP-
GP
45.72
0.541
0.401
71.31
Table 5: Performance comparison on unseen dialogue examples (results DuRocDial) (t-test, < 0.05).
The results are shown in Table 5. We observe that even when the models have not seen such scc- narios (without training), RTCP manages to per- form better than TCP by large margins across sev- eral metrics. When get exposed to new training cases and get fine-tuned, both TCP and RTCP ob- tain better performance results. However, there are two important things to note: (1) In order to fine- tune the model for such new scenarios, TCP needs to update the whole model while RTCP only needs to tune the prefix parameters, and the backbone GPT2 part remains unchanged. This is of much lower cost; hence RTCP is much more efficient as compared to other fine-tuning based approaches. (2) In the With Training setting, removing goal and topic prefixes results in a considerable drop in the performance, which indicates the impacts of these action-driven parameters to our RTCP when it is required to adapt to new plans.
A Acknowledgements
This research was funded by Vingroup Inno- vation Foundation (VINIF) under project code VINIF.2022.DA00088 and the Singapore Ministry of Education (MOE) Tier 1 Academic Research Fund No. 21-SIS-SMU-038, MSS22C003.
References
6 Conclusion
Christian Bizer, Jens Lehmann, Georgi Kobilarov, Sören Auer, Christian Becker, Richard Cyganiak, and Se- bastian Hellmann. 2009. Dbpedía-a crystallization point for the web of data. Journal of web semanties.
this work, we focused on a promising yet under- explored setting called target-driven conversational promotion. We proposed a novel RTCP framework to produce dialogue action tuples which direct the conversation to targeted item while also please the user with interesting and relevant topics. It man- ages to properly generate responses that are closely in-line with the plan while also can adapt quickly to new scenarios. We conducted extensive exper- iments to compare with a bunch of representative baselines over a rich set of metrics. Both automatic evaluation and human evaluation results showed that RTCP significantly outperformed state-of-the. art approaches on many important aspects.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Qibin Chen, Junyang Lin. Yichang Zhang. Ming Ding. Yukuo Cen, Hongxia Yang. and Jie Tang. 2019. To- wards mowledge-basee recommender dialog system. In EMNLP-IJCNLPP pages 1803-1813.
Limitations
Zhendong Chu, Hongning Wang, Yun Xiao, Bo Long, and Lingfei Wu. 2023. Meta policy learning for cold- start conversational recommendation. In WSDM. page 222-230.
We discuss the limitations from the following perspectives: (1) Difficulty level of promotion datasets. In existing CRS datasets (DuRecDial,

---

Yang Deng, Wenqiang Lei, Lizi Liao, and Tat-Seng Chua. 2023a. Prompting and evaluating large lan- guage models for proactive dialogues: Clarification, target-guided, and non-collaboration. arXiv preprint arXiv:2305.133 13626.
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommenda- tions. In NeurIPS.
Shuokai Li, Ruobing Xie, Yongchun Zhu, Xiang Ao.
Yang Deng. Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. 2021. Unified conversational recommendation policy learning via graph- based reinforeement learn- ing. In SIGIR, pages 143 1-14
Fuzhen Zhuang, and Qing He. 2022. User-centric conversational recommendation with multi-aspect user modeling. In SIGIR.
Xiang Lisa Li and Percy Liang, 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACI-IJJNNL,, pages 4582-4597
Yang Deng, Wenxuan Zhang. Weiwen Xu, Wenqiang Lei. Tat-Seng Chua, and Wai Lam. 2023b. A unified multi-task task learning framework for multi-godi conver- sational recommender systems. ACM Transactions on Information Systems, 41(3).
Zujie Liang, Huang Hu, Can Xu, Jian Miao, Yingy- ing He, Yining Chen, Xiubo Geng, Fan Liang, and Daxin Jiang. 2021. Learning neural templates for recommender dialogue system. In EMNLP, pages 7821-7833.
Jacob Devlin, Ming-Wei. Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACL, pages 4171-4186.
Lizi Liao, Ryuichi Takanobu, Yunshan Ma, Xun Yang. Minlie Huang, and Tat-Seng Chua. 2020. Topic- guided conversational recommender in multiple do- mains. TKDE, 34(5):2485-244 2496.
Shirley Anugrah Hayati, Dongyeop Kang, Qingxi- aoyang Zhu, Weiyan Shi, and Zhou Yu. 2020. IN- SPIRED: Toward sociable recommendation dialog systems. In EMNLP, pages 1122-8152.
Lizi Liao, Grace Hui Yang, and Chirag Shah. 2023a. Proactive conversational agents. In WSDM, pages 1244 1247.
Chenhao Hu, Shuhua Huang, Yansen Zhang, and Yubao Liu. 2022. Learning to infer user implicit preference in conversational recommendation. In SIGIR, page 256-266.
Lizi Liao, Grace Hui Yang, and Chirag Shah. 2023b. Proactive conversational agents in the post-chatgpt world. In SIGIR, pages 1452-3455.
Dongyeop Kang, Anusha Balakrishnan, Pararth Shah, Paul Crook, Y-Lan Boureau, and Jason Weston. 2019. Recommendation as a communication game: Self- supervised bot- pot-play for goal-orian oriented dialogue. In EMNLP-UJNNLP, pages 1951-1961.
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang. Yuanyuan Yang, Jiaming Tian, Hao Hc, Antong Li. Mengshen Hc, Zhengliang Liu, Zihao Wu, Lin Zhao, Dajiang Zhu, Xiang Li, Ning Qiang, Din- gang Shen, Tianming Liu, and Bao Ge. 2023. Sum- mary of ChatGPT-related research and perspec- tive towards the future of large language models. Reta-Radiology. 12):1000117
Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Towards
Chua. 2020a. iilimaaion-atiion-efeeetiii deep interaction between conversational and recom- mender systems. In WSDM, page 304-312.
Zeming Liu, Haifeng Wang, Zheng-Yt Yu Niu, Hua Wu, and Wanxiang Che. 2021. DuRecDial 2.0: A bilin- gual parallel corpus for conversational recommenda- tion. In EMNLP, pages 4335-43477
Wenqiang Lei, Gangyi Zhang. Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. 2020b. Interactive path reasoning on graph for conversational recommendation. In KDD, page 2073-2083.
Zeming Liu, Haifeng Wang, Zheng- Yu Niu, Hua Wu,
Wanxiang Che, and Ting Liu. 2020. Towards conver- sational recommendation over multi-type dialogs. In ACL, pages 1036-1049.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP, pages 3045-3059.
Yu Lu, Junwei Bao, Yan Song, Zichen Ma, Shuguang Cui, Youzheng Wu, and Xiaodong He. 2021. RevCore: Review-augmented conversational recom- mendation. In Findings of ACL-UJCLP, pages 1161-1173.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-fo-sequence pre-training for natural language generation, translation, and com- prehension. In ACL, pages 7871-7880.
Wenchang Ma, Ryuichi Takanobu, and Minlie Huang. 2021. CR-walker: Tree-structured graph reasoning and dialog acts for conversational recommendation. In EMNLP, pages 1839-1851.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-rromoting ob- jective function for neural conversation models, In NAACL-HLT, pages 11-1119.
Mary McHugh. 2012. Interrater reliability: The kappa statistic. Biochemia medica časopis Hrvatskoga društva medicinskih biokemičara HDMB, 22:276 82.

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalua- tion of machine translation. In ACI, pages 311 1-318.
Chenchen Ye, Lizi Liao, Suyu Liu, and Tat- Tat-Seng Chua. 2022b. Reflecting on experiences for response gener- ation. In ACM Multimedia, pages 5265-5273.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In NAACL, pages 5203-5212.
Jun Zhang, Yan Yang, Chencai Chen, Liang He, and Zhou Yu. 2021. KERS: A kmowlede--nnhanee framework for recommendation dialog systems with multiple subgoals. In Findings of EMNLP, pages 1092 1101.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Iyya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18)):
Xiaoyu Zhang, Xin Xin, Dongdong Li, Wenxuan Liu, Pengjie Ren, Zhumin Chen, Jun Ma, and Zhaochun Ren. 2023. Variational reasoning over incomplete knowledge graphs for conversational recommenda- tion. In WSDM.
Xuhui Ren, Hongzhi Yin, Tong Chen, Hao Wang. Zi Huang, and Kai Zheng. 2021. Learning to ask appropriate questions in conversational recommenda- tion. In SIGIR, pages 908-817.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT Large--scale generative pre-training for conversational response generation. In ACL, pages 270-278
Zhaochun Ren, Zhi Tian, Dongdong Li, Pengjie Ren, Liu Yang, Xin Xin, Huasheng Liang, Maarten de Ri- jke, and Zhumin Chen, 2022. Variational reasoning about user preferences for conversational recommen- dation. In SIGIR, SIGIR '22, page 165-175.
Xiangyu Zhao, Liang Zhang, Long Xia, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2017. Deep reinforce- ment learning for list-wise recommendations. arXiv preprint 1 1000090
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Zexuan Zhong. Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In NAACL, pages 5017-5033.
Quan Tu, Shen Gao, Yanran Li, Jianwei Cui, Bin Wang. and Rui Yan. 2022. Conversational recommendation via hierarchical information modeling. In SIGIR, SIGIR '22, page 2201-2205
Jinfeng Zhou, Bo Wang, Ruifang He, and Yuexian Hou. 202 CRFR: Improving conversational recom- mender systems via flexible frugments reasoning on knowledge graphs. In EMNLP, pages
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, f. ukasz Kaiser, and Ilaia Polosukhin. 2017. Attention is all you need. In NeurIPS.
Kun Zhou, Xiaolei Wang. Yuanhang Zhou, Chenzhan Shang. Yuan Cheng. Wayne Xin Zhao, Yaliang Li, and Ji-Rong Wen. 2021b. CRSLab: An open-source toolkit for building conversational recommender sys- tem. ACL-LICNLP pages 185-193.
Jian Wang, Dongding Lin, and Wenjie Li. 2022a. Fol- low me: Conversation planning for target-driven rec- ommendation dialogue systems.
Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020a. Im- proving conversational recommender systems via knowledge graph based semantic fusion. In KDD, pages 1006-1014.
Lingzhi Wang, Huang Hu, Lei Sha, Can Xu, Daxin Jiang, and Kam-Fai Wong. 2022b. RecInDial: A uni- fied framework for conversational recommendation with pretrained language models. In AACL, pages 489-500.
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xi- aoke Wang, and Ji-Rong Wen. 2020b. Towards topic-guided conversational recommender system. COLING, pages 4128 4139.
Xiaolei Wang. Kun Zhou, Ji-Rong Wen, and Wayne Xin Zhao. 2022c. Towards unified conversational rec- ommender systems via knowledge-enhanced prompt learning. In KDD, page 1929-1937.
Yuanhang Zhou, Kun Zhou, Wayne Xin Zhao, Cheng Wang, Peng Jiang, and He Hu. 2022. C2-ers: Coarse- to-fine contrastive learning conversational recom- mender system. In WSDM, page 1488-1496.
Yuxia Wu, Lizi Liao, Gangyi Zhang, Wenqiang Lei, Guoshuai Zhao, Xueming Qian, and Tat-Seng Chua. 2022. State graph reasoning for multimodal con- versational recommendation. IEEE Transactions on Multimedia.
Jie Zou, Evangelos Kanoulas, Pengjic Ren, Zhaochun Ren, Aixin Sun, and Cheng Long. 2022. Im- proving conversational recommender systems via transformer-basce sequential modelling. In SIGIR, SIGIR '22 page 2319-2324.
Chenchen Ye, Lizi Liao, Fuli Feng, Wei Ji, and Tat- Seng Chua. 2022a. Structured and natural responses co-generation for conversational search. In SIGIR, pages 155-164.

---

A Appendix
mendation setting and there are 19 different social strategies in INSPIRED. our setting, we regard these social strategies as goals and attempt to pre- dict them at each turn. Finally, we use the default data splits of INSPIRED to conduct our experi- ments.
A.1 Additional Comparison with Large L Models (LLMs)
Recently, LLMs such GPT3 (Brown et al., 2020), ChatGPT (Liu et al., 2023) and Llama (Touvron et al., 2023) have shown out-standing performance on a variety of NLP tasks. Hence, in this work, we also report the performance comparison between our RTCP model and these LLM-based target- driven recommender systems. In particular, we adopt ChatGPT and leverage zero-shot prompting produce responses using the LLM model. We construct the input prompt by prepending the target item the current dialogue context and ask the model to generate the corresponding response. As can be seen in Table 6, We can observe that RTCP significantly achieves better performance on sev- eral metrics including BLEU-N (N=1,2), Fl and Know. F1. This is reasonable since RTCP is opti- mized to mimic responses in the corpus. However, ChatGPT performs better than our RTCP DIST- N metrics (N=1,2). This can be attributed to the fact that ChatGPT have been pre-trained on a mas- sive amount of texts. Therefore, it could generate more diverse responses. For target-achievement as- pects, we can observe that our RTCP outperforms ChatGPT on all metrics. This can be attributed to the planning module of RTCP which could pro- duce appropriate plans direct the conversations towards the target item, while ChatGPT tends to passively answer user queries. This can be seen in Table which shows the results of human eval- uation with RTCP and ChatGPT on DuRccDial. Athough ChatGPT is trained with large corpus and with huge amount of parameters, our model still manages to outperform it in Proactiveness.
A.3 Additional Details of Baseline Methods In this work, we compare our RTCP framework against several representative baselines including:
BERT (Devlin al., 2019) is widely used baseline model for text classification tasks. We adapt it to show the performance compar- isons on goal and topic prediction.
GPT2 (Radford et al., 2019) a basic but strong text generation baseline which gains from large pretrained language modeling.
BART (Lewis et al., 2020) is a more recent denoising autoencoder pretrained model for language generation. It can be as gen- cralizing BERT, GPT, and many other more recent pretraining schemes.
DialoGPT (Zhang et al., 2020) is a dialogue generative pre-trained GPT model trained on large-scale conversation-Hee exchanges from Reddit, which helps to generate more relevant and context-consistent responses.
KERS (Zhang al., 2021) is knowledge- enhanced multi-subgoal driven recommender which predicts a sequence of subgoals and use them guide the dialog model to select knowledge from a sub-set of existing knowl- edge graph.
TCP (Wang et al., 2022a) is a target-driven recommender. It predicts a sequence of goals and topics in short-term planning manner which further guide the response generation. UNIMIND (Deng et al., 2023b) is the state- of-the-an art goal-driven CRS model which lever- ages multi-task learning and a prompt-based approach to unify sub-tasks of multi-goal CRS setting.
A.2 Detailed Statistics of Benchmark Datasets
Detailed statistics of datasets are described in Table 8. In particular, the DuRecDial 2.0 dataset (Liu et al., 2021) consists of 16.5K English-Chinese par- allel dialogues and approximately 55K natural lan- guage utterances belong to seven different domains such as Movie/PO//Foo//MMsic///eee this work, we utilize the English version of the dataset. We adopt the default data split 6.5 1 2.5 for trainingddevelopmeenttess respectively. On the other hand, INSPIRED dataset (Hayati al., 2020) consists of 1001 conversations (35K utterances) that belong to the movie recommendation domain. The dataset supports sociable conversation recom-
A.4 Instructions for Human Evaluation
etc. In
Given generated conversations, we ask the annota- tors to evaluate those dialogues in both turn-level and dialogue-level aspects. For turn-level results, report three metries namely Fluency, Informa- tiveness, and Proactivity. On the dialogue level, we report Satisfaction and Coherency respectively. The rating scores must be in the range of [1.3] and a higher score is corresponding to a better example.

---

Response Quality
Target-accheveemet
Model
F1
BLEU-1
BLEU-2
DIST-I
DIST-2
Know. F1
SR@1
SR
Avg. Turns(])
ChatGPT
12.26
0.317
0.176
0.048
0.205
16.20
5.19
85.09
3.97
RTCP (alpha 0)
45.39
0.542
0.402
0.036
0.109
70.35
9.21
86.78
3,80
Table 6: Automatic evaluation between RTCP and ChatGPT the DuRecDial dataset.
Turn-level results
Dialog-level results
Model
Fluency
Infor.
Proactivity
Satisfaction
Coherency
Kappa
ChatGPT RTCP
3.000
2.376
1.577
2.65
2.40
0.69
2.981
2.246
1.676
2.55
2.45
0.74
Table Human evaluation on the DiRecDial by RTCP and ChatGPT (Infor, stands for informativeness).
DuRecDial
INSPIRED
or recommend items) to interact with the users. The instructions are as follows:
of convs
16.5K
1001
of utterances
255K
35,811
(1): The generated sentences are utilized to just passively respond to the user's last utter- ance (e.g.: "T haven't seen that movie."; "It is interesting movie", etc.).
# of goals
14
19
of topics
646
# of domains
Table 8: The detailed statistics of DuRecDial and IN- SPIRED datasets.
(2): The goal of the generated responses is obtain further information from the user. (e.g.)
"What are your favorite singers ?"; "Do you
For consistent evaluation, carry set of detailed instructions to guide the annotators. For fluency, we ask the annotators to check whether the generated responses are grammar- correct and understandable. The detailed instruc- tions are follows:
like horror movies ?" etc.).
(3): The generated responses are utilized to of- fer recommendations or to provide relevant in- formation about specific topic (e.g: "Speak- ing of Jacky Chung, he is also the Most Pop- ular Asian Artist of Jade Solid Gold Best 10 Awards": "Revolution also very good. It's a song of Leehom Wang."; etc).
(1): If the sentences are strictly grammar- incorrect and the annotators can not under- stand the meaning or the intention of the sen- tences (e.g.: "I 'm not sure it is it, but I have n't seen it 's it 's a big").
For satisfaction, we ask the annotators to evaluate how successfully the surget-drived promotion task is fulfilled by considered methods. The detailed criteria are as follows:
(2): The sentences might have some minor mistakes but they are still understandable.
(1): The systems fail to recommend the tar- geted item. That means the targeted item never appear in the generated dialogues.
(3): The generated sentences grammar- correct and the annotator can understand its meaning or intention (e.g.: "How about The Conjuring (2013) ?")
(2): The systems successfully recommend the targeted item but they do not provide any con- vincing reasons persuade the user. (for in- stance, they might intermediately suggest the designated item).
For informativeness, the annotators need to check if the generated responses contain topics or relevant information. The instructions are as follows.
(1): The generated responses do contain any useful information such as topics or re- lated knowledge.
(3): The systems not only recommend the tar- geted item successfully but they also convince the users with additional justifications.
(2): The generated responses contain either
For coherency, we ask the annotators to check if generated responses are relevant to their corre- sponding dialogue contexts and these responses with the user's utterances together they form a meaningful conversation about the targeted item. (1): Major generated responses are not rele- vant to their historical contexts. (for instance, the models might frequently produce simple
topics or related information.
(3): The generated responses not only contain relevant topics but also provide additional re- lated knowledge about the mentioned topics. For proactivity, the annotators are required to check if the systems can use diverse intents (c.g.: chit- chat about a specific topic, ask clarifying questions,

---

(2): Generated responses are relevant to their dialogue contexts. However, they together with the user's utterances do not form a mean- ingful conversation about the targeted item. responses such as "It is good movie" or "What kinds of movic do you like ?" for every dialogue context).
(3): Dialogues that satisfy the two criteria.


Reviews:
[{"review_id": "Ce1csj9JGJ", "reviewer": "Reviewer_pkX6", "summary": "", "questions": "See W1-W3.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and valuable comments. We answer the questions as below.\n\n**Q1: A related baseline is not compared. 'Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning' (UNICORN) is cited but not compared in the experiment.**\n\n**A1:** We sincerely appreciate your suggestion. We indeed considered it. However, UNICORN is designed for scenarios in which system attempts to ask clarifying questions about attributes/properties to gradually search for an optimal candidate set of items. In contrast, our desired setting is to nudge users to accept a predefined target item, which is significantly different from UNICORN’s setting. Besides, UNICORN is a question-based CRS model that is trained with historical ratings of the users. Their benchmark datasets are recommendation datasets like Yelp and LastFM, which is rather different from our natural conversation dataset.  Moreover, UNICORN uses predefined templates to interact with the users which inherently can not handle arbitrary responses by the users.\n\n**Q2: Evaluation criteria. BLEU-n and DIST-n may not be comprehensive as n only takes 1 and 2. It is recommended to set n as 3 and 4 for generative tasks. Besides, there are some advanced evaluation metrics apart from n-gram-based metrics.**\n\n**A2:** Thank you for your suggestions about evaluation metric. Due to space limitation, we only reported BLEU-N, DIST-N (N=1,2). We additionally report the performance on BLEU-N, DIST-N (N=3,4) here, which will be added to the appendix. As can be seen in Table 1 and 2, compared to other existing methods,  RTCP achieved better performance on 3 out-of 4 and 2 out-of-4 metrics for DuRecDial and INSPIRED respectively, which shows a similar trend with the reported results in the paper. Indeed, there are also advanced evaluation metrics such as embedding based metrics (e.g. BERTScore, BARTScore etc). We will add the results and adjust the tables accordingly.\n\n```other\nTable 1: Performance on DuRecDial (t-test, p-value < 0.05):\n```\n\n| **Model**        | **BLEU-3** | **BLEU-4** | **DIST-3** | **DIST-4** |\n| ---------------- | ---------- | ---------- | ---------- | ---------- |\n| BART             | 0.198      | 0.167      | 0.162      | 0.222      |\n| DialogGPT        | 0.252      | 0.221      | 0.160      | 0.235      |\n| GPT2             | 0.277      | 0.242      | 0.182      | **0.26**   |\n| KERS             | 0.198      | 0.164      | 0.046      | 0.064      |\n| TCP              | 0.294      | 0.257      | 0.179      | 0.258      |\n| UNIMIND          | 0.310      | 0.273      | 0.176      | 0.254      |\n| RTCP (alpha = 0) | **0.343**  | **0.307**  | **0.184**  | 0.259      |\n\n```other\nTable 2: Performance on INSPIRED (t-test, p-value < 0.05):\n```\n\n| **Model**        | **BLEU-3** | **BLEU-4** | **DIST-3** | **DIST-4** |\n| ---------------- | ---------- | ---------- | ---------- | ---------- |\n| BART             | 0.057      | 0.039      | 0.288      | 0.386      |\n| DialogGPT        | 0.081      | 0.06       | 0.188      | 0.256      |\n| GPT2             | 0.083      | 0.059      | 0.341      | 0.445      |\n| KERS             | 0.032      | 0.021      | 0.037      | 0.043      |\n| TCP              | 0.087      | 0.063      | 0.359      | 0.46       |\n| UNIMIND          | 0.086      | 0.063      | **0.382**  | **0.486**  |\n| RTCP (alpha = 0) | **0.101**  | **0.077**  | 0.333      | 0.433      |\n\n**Q3: Generative PLMs are not enough. It is recommended to compare it with ChatGPT since it is free and available.**\n\n**A3:** We appreciate your suggestions and additionally report the performance of ChatGPT. First,  we show the generation performance of RTCP and ChatGPT on DuRecDial in Table 3.  We can observe that RTCP significantly achieves better performance on several metrics including BLEU-N (N=1,2,3,4), F1 and Know. F1. This is reasonable since RTCP is optimized to mimic responses in the corpus. However, ChatGPT performs better than our RTCP on DIST-N metrics (N=1,2,3,4). This can be attributed to the fact that ChatGPT have been pre-trained on a massive amount of texts. Therefore, it could generate more diverse responses. Second, we show the target-achievement results of RTCP and ChatGPT in Table 4. We can observe that our RTCP outperforms ChatGPT on all metrics. This can be attributed to the planning module of RTCP which could produce appropriate plans to direct the conversations towards the target item, while ChatGPT tends to passively answer user queries. This can be seen in Table 5 which shows the results of human evaluation with RTCP and ChatGPT on DuRecDial. Athough ChatGPT is trained with large corpus and with huge amount of parameters, our model still manages to outperform it in Proactiveness.\n\n```other\nTable 3: Generation performance of RTCP and ChatGPT on DuRecDial.\n```\n\n| **Model**        | **F1**    | **BLEU-1** | **BLEU-2** | **BLEU-3** | **BLEU-4** | **DIST-1** | **DIST-2** | **DIST-3** | **DIST-4** | **Know.F1** |\n| ---------------- | --------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ----------- |\n| RTCP (alpha = 0) | **45.39** | **0.542**  | **0.402**  | **0.343**  | **0.307**  | 0.036      | 0.109      | 0.184      | 0.259      | **70.35**   |\n| ChatGPT          | 12.26     | 0.317      | 0.176      | 0.081      | 0.048      | **0.048**  | **0.205**  | **0.37**   | **0.49**   | 16.2        |\n\n```other\nTable 4: Target-achievement Results of RTCP and ChatGPT on DuRecDial Dataset.\n```\n\n| **Model**        | **SR@1** | **SR**    | **Avg_turn** |\n| ---------------- | -------- | --------- | ------------ |\n| RTCP (alpha = 0) | **9.21** | **86.78** | **3.80**     |\n| ChatGPT          | 5.19     | 85.09     | 3.97         |\n\n```other\nTable 5: Human Evaluation on DuRecDial for RTCP and ChatGPT.\n```\n\n| **Model**        | **Fluency** | **Informativeness**    | **Proactivity** | **Statisfaction**    | **Coherency** | **Kappa** |\n| ---------------- | -------- | --------- | ------------ | --------- | ------------ | ------------ |\n| RTCP  | 2.981 | 2.246 | **1.676**     | 2.55 | **2.45**     | **0.74**|\n| ChatGPT   | **3.000**     | **2.376**     | 1.577         | **2.65** | 2.40     | 0.69|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. This work focused on a neglected problem in CRS, which may promote the application of CRS.\n\nS2. Well-organized and well-written.\n\nS3. Comprehensive comparison to the related works and PLMs.", "reasons_to_reject": "W1. A related baseline is not compared. 'Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning' (UNICORN) is cited but not compared in the experiment. As far as I know, UNICORN also has an RL pipeline to control what attributes to ask, which items to recommend, and when to ask or recommend. Based on the paper's scope, it is also a competitive baseline to be considered.\n\nW2. Evaluation criteria. BLEU-n and DIST-n may not be comprehensive as n only takes 1 and 2. It is recommended to set n as 3 and 4 for generative tasks. Besides, there are some advanced evaluation metrics apart from n-gram-based metrics.\n\nW3. Generative PLMs are not enough. It is recommended to compare it with ChatGPT since it is free and available.", "questions_to_authors": "See W1-W3.", "paper_topic_and_main_contributions": "This work proposes an RL-based framework for conversational recommender systems emphasizing modeling the plan to make users accept the designated item. This topic is interesting in the IR community. This work focuses on a neglected problem in the CRS field.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GX0vm3zNKF", "reviewer": "Reviewer_N7Uc", "summary": "", "questions": "Not clear why parameters are learnt in line 354  shouldnt the values learnt from previous steps be directly used?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below we address your concerns.\n\n**Q1: The paper lacks clear explanation and detail regarding the metrics used for evaluation. Also not clear how much data was used in human eval.**\n\n**A1:** We appreciate your suggestion and will additionally describe explanations regarding the evaluation metrics. In particular, for the response generation task, we utilize both automatic and human evaluation. For automatic evaluation, we use perplexity (**PPL**), word-level F1 (**F1**), **BLEU-N** (N=1,2), **Dist-N** (N =1,2) and Knowledge f1 (**Know. F1**).  Specifically, we use the perplexity (**PPL**) to quantify how uncertain the model is about its predicted responses. We use distinct (**Dist-N**) to evaluate the diversity of generated responses. **F1** and **BLEU-N** measure the similarity between the generated responses and ground truth sentences. The **Know. F1** shows the matching performance between generated knowledge and the ground truth knowledge triples. We will add more details in Appendix.\n\nFor human study, we randomly sample 20 dialogues generated by RTCP and baseline models (shown in line 406). We then invite two annotators to score the responses in both turn-level and dialogue-level. The range of score is 1 to 3. The final performance is calculated using the average scores of all annotators. We measure the inter-annotator agreement by Fleiss’ Kappa. Human evaluation instruction details are described in Appendix A.6.\n\n**Q2: The performance results, especially in Table 1, show marginal differences from baselines.**\n\n**A2:** Thank you for pointing this out. We need to highlight that Table 1 (in paper) results are only for short-term planning. For maintaining smooth and coherent conversations, it is important for the agent to ensure not moving far away from the context. Also, the performance improvement are not marginal actually. For instance, in Table 1 (in paper) , compared to the best performing UNIMIND model, our RTCP achieve considerable improvement on both DuRecDial (+2.67 \\% on Topic Accuracy,  + 2.75 \\% on Joint Accuracy) and INSPIRED (+ 7.64 \\% on Goal Accuracy). Additionally, we also ensure the statistical significance of the reported results via t-test with p-value < 0.05.\n\n**Q3: Not clear why parameters are learnt in line 354 shouldn't the values learnt from previous steps be directly used?**\n\n**A3:** Please let us provide some clarifications about our method here and we will make it more clear in the draft later. There are two separate components in our proposed RTCP framework namely a planning module and a response generation model. For the planning module, we utilize encoder models based on BERT to encode knowledge, previous planning sequence, and conversation context. For the response generation part, we adopt a GPT2 model as our main pretrained backbone. Regarding line 354 in the paper, to train our response generation module, we freeze the parameters of the GPT2 model and only learn parameters of the action-topic prefixes $G_{t}$, $P_{t}$ and task-specific prompt tokens $T_{gen}$. Hence, we cannot use the values learnt from the previous BERT model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The RTCP framework outperforms other baselines on the datasets considered.\n- Ablation show the balanced gating mechanism and action-guided prefix tuning contribute to effective short-term and long-term planning and relevant response generation.\n", "reasons_to_reject": "- The paper lacks clear explanation and detail regarding the metrics used for evaluation. Also not clear how much data was used in human eval.\n- The performance results, especially in Table 1, show marginal differences from baselines.\n- Not clear why parameters are learnt in line 354  shouldnt the values learnt from previous steps be directly used?\n", "questions_to_authors": "Not clear why parameters are learnt in line 354  shouldnt the values learnt from previous steps be directly used?\n", "paper_topic_and_main_contributions": "The paper introduces an approach to conversational recommendation by focusing on the strategic planning of nudging users towards accepting specific items. To address this, the authors propose the Reinforced Target-driven Conversational Promotion framework, which integrates short-term and long-term planning through a gating mechanism. This approach predicts dialogue actions using knowledge-integrated multi-head attention and also reinforces them through reinforcement learning rewards. Additionally, RTCP employs action-guided prefix tuning to generate relevant responses. Experimental results showcase the performance of RTCP over state-of-the-art models in terms of automatic metrics and human evaluations.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "joXQLto0Tq", "reviewer": "Reviewer_nCqR", "summary": "", "questions": "A. What does the task-specific token $T_{gen}$ refer to in Line 357? How important is it during prefix tuning?\n\nB. In target-driven conversations, how to measure the average number of conversation turns? Are shorter conversations better?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. Below we address your concerns.\n\n**Q1: The term use of \"dialogue actions\" in this paper needs careful consideration.**\n\n**A1:** We appreciate your suggestions and will utilize the suggested term “a pair of a goal $g_t$ and a topic $p_t$” in our modified version.\n\n**Q2: Some details of the proposed approach are not clear enough. (1) It lacks necessary descriptions of how $h_{s_{t+1}}$ is obtained when computing the reward $r_s$. (2) The two loss functions $L_g$ and $L_p$ in Lines 243-244 are not correct.**\n\n**A2:** Please let us provide some clarifications to address these two concerns. For (1), to compute the representation of the next state $h_{s_{t+1}}$, we feed the next state $s_{t+1}$ (which is a sequence) through a BERT model and utilize the output corresponding to the [CLS] token. Additionally, to train the binary classification model, we consider sequences of actions from the training set as positive instances and randomly replace some elements in the positive instances to produce negative ones. We will make these details more clear in the paper. For (2), we appreciate your thoughtful comment and will correct these loss functions in our modified version. Actually, our current equations such as the one in line 361, it focuses on the dimension in the probability vector corresponding to the groundtruth token $y_{t,j}^*$ , because other dimentions will multiply with zeros. But we will make the loss functions more clear following common practice.\n\n**Q3: What does the task-specific token $T_{gen}$ refer to in Line 357? How important is it during prefix tuning?**\n\n**A3:** $T_{gen}$ refers to task-specific soft tokens which are automatically learned to instruct the generation task. We got inspired from the work [1], which shows that adding such Task-Specific Prompt enhances model performace. Different from the action and topic prefixes $G_{t}$ and $P_{t}$ , we randomly initialize $T_{gen}$ and gradually optimize its parameters during training. To show the impact of $T_{gen}$, we additionally report the performance of RTCP wihout $T_{gen}$ on both the DuRecDial and INSPIRED datasets. As can be seen in Table 6 and Table 7 below, removing the task-specific prompt tokens lead to significant drops in the performance, which demonstrates that these prompt tokens are indeed crucial to the performance of RTCP.\n\n```other\nTable 6: Ablation study on response generation for DuRecDial (t-test, p-value < 0.05).\n```\n\n| **Model**                | **PPL**  | **F1**    | **Bleu-1** | **Bleu-2** | **Dist-1** | **Dist-2** | **Know.F1** |\n| ------------------------ | -------- | --------- | ---------- | ---------- | ---------- | ---------- | ----------- |\n| RTCP (alpha = 0)         | **3.69** | **45.39** | **0.542**  | **0.402**  | **0.036**  | **0.109**  | **70.35**   |\n| RTCP - t_gen (alpha = 0) | 5.10     | 36.38     | 0.482      | 0.326      | 0.031      | 0.104      | 46.86       |\n\n```other\nTable 7: Ablation study on response generation for INSPIRED (t-test, p-value < 0.05).\n```\n\n| **Model**                | **PPL**   | **F1**    | **Bleu-1** | **Bleu-2** | **Dist-1** | **Dist-2** | **Know.F1** |\n| ------------------------ | --------- | --------- | ---------- | ---------- | ---------- | ---------- | ----------- |\n| RTCP (alpha = 0)         | **18.64** | **15.39** | **0.371**  | **0.168**  | **0.091**  | **0.211**  | **10.21**   |\n| RTCP - t_gen (alpha = 0) | 33.78     | 13.25     | 0.346      | 0.152      | 0.063      | 0.154      | 4.41        |\n\n[1] Wang, Xiaolei, et al. \"Towards unified conversational recommender systems via knowledge-enhanced prompt learning.\" *In KDD*. 2022.\n\n**Q4: In target-driven conversations, how to measure the average number of conversation turns? Are shorter conversations better?**\n\n**A4:** This is indeed a good question. In target-driven setting, we define the number of conversation turns as the number of turns needed to successfully recommend the target item. Shorter conversations does not mean they are better. A desired target-driven model needs to engage the users with relevant and interesting information to gain their attention before suggesting the target item. If a system constantly recommends the target items in the early turns of the conversations, it might make the users annoyed, which inherently hurts their experience. This is supported via experiment results reported in the paper. Specifically, in Table 2 in the paper, one can observe that some general pretrained language models namely GPT2 and BART achieve relatively good performance on the SR@1 metric. However, in Table 4 in the paper, the satisfaction score of GPT2 is significantly lower than other target-driven models such as RTCP and UNIMIND, which indicates that the annotators might not be so satisfied with responses generated by vanilla GPT2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-motivated. Target-driven conversational promotion is an interesting yet challenging scenario, where how to conduct effective planning to promote target items proactively is a noteworthy research focus.\n2. The proposed RTCP framework is reasonable and in line with the motivation. I really like the idea of making a balance between short-term planning and long-term planning.\n3. Experiments are solid, with representative baseline models included and widely-used metrics for evaluation. Besides, it's good to see that the authors conduct additional analysis and show that RTCP can adapt to unseen conversation scenarios without whole re-training.", "reasons_to_reject": "1. The term use of \"dialogue actions\" in this paper needs careful consideration. It might not be appropriate enough to define a dialogue action $a_t$ as a pair of goal $g_t$ and topic $p_t$. Generally, a dialogue topic outlines the subject or theme at a turn, while a dialogue action denotes the action or behavior performed by a speaker, e.g., asking for clarification, informing the user, making recommendations, etc. Actions and topics are two types of ingredients in the dialogue setting. Actually, I think the \"goals\" in the context of this work should be called \"actions\", though the authors follow the term use of UNIMIND (Deng et al., 2023). Nonetheless, I suggest the authors take a more appropriate term to denote \"a pair of goal $g_t$ and topic $p_t$\".\n\n2. Some details of the proposed approach are not clear enough. (1) It lacks necessary descriptions of how $h_{s_{t+1}}$ is obtained when computing the reward $r_s$. (2) The two loss functions $L_g$ and $L_p$ in Lines 243-244 are not correct, since $P_{g}(\\cdot)$ and $P_{p}(\\cdot)$ are estimated probabilities, rather than ground-truth distributions. They should be in the form of $P(p^{*})\\log P(p)$ when the authors use ground-truth distributions.", "questions_to_authors": "A. What does the task-specific token $T_{gen}$ refer to in Line 357? How important is it during prefix tuning?\n\nB. In target-driven conversations, how to measure the average number of conversation turns? Are shorter conversations better?", "paper_topic_and_main_contributions": "This paper explores a promising yet under-explored conversational task, namely conversational promotion. It aims to promote specified items through engaging conversations. The authors propose a Reinforced Target-driven Conversational Promotion (RTCP) framework to perform long short-term strategic planning to predict a dialogue action (a pair of goal and topic), then employ prefix-tuning to generate proper responses guided by the planned actions. Experiments on two datasets demonstrate the effectiveness of the proposed approach. Overall, this paper is well-written, with clear motivations and good experimental settings.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a conversational nudging technique that allows the system to proactively plan and suggest items, improving the final outcome of item selection by the user. The paper has done the necessary evaluation and their method is sound.
Decision: Accept-Main