Joint Semantic and Strategy Matching for Persuasive Dialogue
Authors: Chuhao Jin
Yutao Zhu
Lingzhen Kong
Shijie Li
Xiao Zhang
Ruihua Song
Xu Chen
huan chen
Yuchong Sun
Yu Chen
Jun Xu
Venue: EMNLP 2023 Findings (2023)
Keywords: Persuasive dialogue
Application of dialogue
Retrieval-based dialogue

Abstract: Persuasive dialogue aims to persuade users to achieve some targets by conversations.
While previous persuasion models have achieved notable successes, they mostly base themselves on utterance semantic matching, and an important aspect has been ignored, that is, the strategy of the conversations, for example, the agent can choose an \textit{emotional-appeal} strategy to impress users.
Compared with utterance semantics, conversation strategies are high-level concepts, which can be informative and provide complementary information to achieve effective persuasions.
In this paper, we propose to build a persuasion model by jointly modeling the conversation semantics and strategies, where we design a BERT-like module and an auto-regressive predictor to match the semantics and strategies, respectively.
Experimental results indicate that our proposed approach can significantly improve the state-of-the-art baseline by 5\% on a small dataset and 37\% on a large dataset in terms of Recall@1. Detailed analyses show that the auto-regressive predictor contributes most to the final performance.

Joint Semantic and Strategy Matching for Persuasive Dialogue
Chuhao Jin', Yutao Zhu', Lingzhen Kong', Shijie Li, Xiao Zhang', Ruihua Song'
Xu Chen', Huan Chen', Yuchong Sun', Yu Chen', Jun Xu'
'Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China MMeituan, Beijing, China
{jinchuhao, ytzhu, konglingzhen, rsong] (ernn) edu. cn, (lishijiel1 chenhuan1 5, chenyu1 e tuan com
Abstract
Have you ever heard charity Save Children?
Aesurd--f the-ary
Persuasive dialogue aims to persuade users to achieve specific goals through conversations. While previous models have achieved notable successes, they mostly rely on matching utter- ance semantics and neglect important as- pect: the strategies of conversation, such as emotional-appest and foot-in-door. In contrast to utterance semantics, conversation strategies are high-level concepts, which can be informa- tive and provide complementary information, contributing to more effective persuasion. This paper proposes a novel persuasion framework that combines the modeling of conversation se- mantics and strategies. To accomplish this ob- jective, design BERT-like module and an auto-regressive predictor that match the seman- ties and strategies, respectively. Experimental results indicate that our proposed approach significantly improve the state-of of-the-art base- line by 5% on a small dataset and 37% on large world dataset in terms of Recall @ 1. The online evaluation shows that our approach improves the ultimate goal of persuasion in real-world applications.
M4. No, it sounds terrible and dull.
ug: It provides food hungry children. Mayhe
can change you mind today.
provider facts
That's parents job. am not interested.
Mg Some of these poor children do not have parents
their parents afford food. These kids break
my beart.
conraiionn apppeui
My Maybe you are right, but don't think children's 's charities are worth my donation.
M Think about donating little bit of 50 cents?
foot-it-door
My understand, but imagine child you help overcomes adversity ends up growing into productive and valuable member. Maybe just this once, would you be willing to donate 10 cents?
lugical- appeea & fooi-in-dooor
Not Persuaded (R
Figure I: Part of persuasive dialogue session from PersuasionForGond with strategies labelled in color.
engaged but also adapting effective strategies to achieve the intended goal.
To realize effective persuasive dialogue systems, recent years have witnessed several promising per- suasion models. For example, Shi et al. (2021) utilizes reinforcement learning methods to avoid repetition and inconsistency in the persuader's 's re- sponses; Chen et al. (2022) integrates factual infor- mation and social content into persuasive dialogue systems; Samad et al. (2022) developed empa- thetic persuasive dialogue system, etc. Although these models have achieved remarkable success, they primarily focus on designing improved algo- rithms from a semantic perspective to generate or match responses, rather than efficiently capitalizing on the vital role of conversational strategies.
1 Introduction
Persuasive dialogue has recently attracted increas- ing attention from the research community and shown great potential in real-worid scenarios. aims to simulate the persuaders to communicate with the users for achieving a given goal (e.g., rais- ing charitable donations). As demonstrated in Fig- ure 1, the agent (left side) must not only introduce the charitable organization, but also consider the user's emotions and encourage reasonable dona- tion amount in order to persuade the user (right side) donate. Compared with the other conversa- tion tasks, persuasive dialogue can be particularly challenging as involves not only keeping the user
Skilled persuaders may high-level strategies attract and guide the users to achieve the prede- fined goals. As shown in Figure when continuing the persuasive conversation the fourth turn, the agent may improve the effectiveness of the conver- sation by first using the phrase "I understand" to
This work was performed when Chuhao Lingzhen Kong were visiting Meituan as research interns. "Coresponniig author.

---

address the user's emotions. Subsequently, utiliz- ing a logical-appeal strategy before foot-in- is more likely to result the person accepting a donation proposal, rather than solely relying on a foot-in-door strategy by asking "think about do- nating little bit of 50 cents". The conversation strategy and utterance play different yet comple- mentary roles in persuasion modeling. The strategy abstracts the conversations in more direct and global manner. Successful persuasions are usually determined by effective transition patterns among different strategies. The utterances are the specific outcomes leveraged to interact with the users. They contain more detailed information that is not cov- ered in the strategies. As a result, it is essential to consider both conversation strategies and utter- ances jointly to promote better persuasion models. Inspired by the above analysis, we propose to build a persuasion dialogue model by simultane- ously matching strategies and semantic. Specifi- cally, we propose using an auto-regressive decoder to generate next strategies sequentially. Then we propose new framework that combines both se- mantic and strategy matching scores the final score for response candidates. We evaluate our proposed methods and state-of-the-arr baselines on a small publicly available dataset (Wang et al., 2019) and a newly collected dataset that are 30 times larger. Experimental results indicate that our proposed approach significantly outperforms all baselines in terms of Recall @ 1, by 5% on the small dataset and 37% the large dataset.
response for the persuader in order continue the conversation and try to achieve persuasion goal. Formally, we define a session of persuasive dialogue as a pair (D, R). D is the dialogue con- tent, and R is a binary return indicating whether the persuasion is ultimately successful (R = 1) or not (R 0). The signal R is only available during the training stage. The dialogue content is given by D {ui "1 :..... uN= the persuader's 's and persuadee's 's utterances the t-th turn, respectively. Each uy contains 4, tokens.
"N); where ut and u, are
There are K atomic strategies that can be used by the persuader. Note that the persuader can use multiple strategies in single utterance. In this paper, we train multi-label classifiers from the data with strategy labels (sce Section 4.1) and use them to automatically label all utterances. We further denote the strategy set at the 1-th turn vector Ar E IR. where Ar, E {0, 1} indicating whether the i-th strategy used or not. Based on the above notations, we learn a matching model f with the following input and output:
binary
"T.,.... Input: dialogue context {NT "T ...., ntt " associated with strategy set {A11,.., ..., Ar}, and response-strategy pair (u,; Ac)
Output: a matching score between (0, 1).
Based on f. we select the next response with the largest matching score.
Our Approach
The main contributions of this paper are sum- marized as follows: (1) We propose to build per- suasion models by jointly considering the conver- sation strategies and semantic, which, to our best knowledge, is the first time in this domain. (2) To achieve the above goal, propose auto- regressive model generate next conversation strategy from previous strategies and utterances. (3) We collected large dataset from a real-world application and performed extensive offline experi- ments and an online evaluation, which demonstrate the effective use of our model in persuasion.
Effective persuasive conversation requires next utterance that is fluent, appealing and presents strategies that are successful in influencing users to achieve the persuasion goal. As shown in Fig- ure 2, we propose a new framework called SARA (Semantic And stRategy mAtching) that assesses the quality of response candidates from both utter- ance and strategy perspectives. For the utterances, we calculate the matching score between the candi- date u, and context by fine-tuned BERT and two auxiliary tasks. As for the strategies, we de- sign an ruto-segres model to generate the next strategies, which are expected to capture the strat- egy transition patterns, and estimate the matching score between the strategies of candidate A, and the generated strategies. Finally, aggregate the two scores as the final score for ranking candidates and select the most appropriate response. In the following, we introduce our model more in detail.
2 Problem Formulation
This paper investigates the persuasive dialogue be- tween two interlocutors, referred to as the per- suader or "** party and the persuadee """ party. The task is formulated as a retrieval- problem, in which the model should select an appropriate
party

---

Final atching Score
SHUP
Aggregation
NUP
CSP
RP
Strakegy Classifier
Cosine Similarity
Uc
A₂
BERT Encoder
A
A
CLS
SEP
u
SEP
Audllary Tasks
Trranffomer Demodr
CSP
BERT Encoder
pos.
E
E
CLS
SEP
Post
SEP P
BERT Enooder
Our BERT Encoder
Our B BERT Encoder
K
T
CLS
SEP
Le Cor.
CLS
BOR
SEP
CLS
BOB
SEP
STLt
Semantic M atching
Strategy M atching
Figure 2: Overview of our SARA framework. At the 1-th turn, uz"" and u, persuader's and persuadee's 's utterances, Ey and Eg are the semantic representation of uf and uy A is the used strategies, G3 is the expect gain.
3.1 Semantic Matching
used to predict the matching score, as follows:
Recent retrieval- based dialogue systems commonly use BERT for the task of choosing response that fits naturally with the preceding context. In SARA, we also employ BERT to measure the semantic sim- ilarity between dialogue context and a response candidate. Before fine-tuning the model, the BERT model parameters address potential do- main differences by implementing the post-training tss-trraning proposed by Han et al. (2021) over our datasets. The key fine-tuning objective, as shown in Fig- ure 2, is to determine the extent to which a candi- date response wt matches the semantic content of the conversation context C. We refer to this task Next Utterance Prediction (NUP) task. Fol- lowing the design of BERT, we construct the input sequence UUP as:
SNUP = a (W NWNUPENUP + bNUP).
where NUP E 768x1 and UNUP E IR are parame- ters. (-) is the sigmoid activation function.
adapt
To enhance the model's capacity for detecting long- ming-term dependencies in dialogues, we propose two auxiliary tasks: Context Segment Prediction (CSP) and Return Prediction (RP). For CSP, we di- vide the conversation context C into two segments at the randomly selected turn The resulting segments are Cpre = {u² u1 ...., ur u, and Grost {ut+1: "1+11--.. 24 n- " We then construct the input #CSP as follows:
JCSP = [[CLS]; [SEP];Cpont [SEP]].
NUP [[CLS];C] C; [SEP]; t [SEP]],
Negative samples were randomly selected from other dialogues' context segments. The BERT model uses a linear layer with sigmoid function that takes the representation of the [CLS] token as input to predict whether two segments belong to consecutive context. For RP, the model predicts the return R, i.e., whether the persuasion goal was achieved or not, using partial dialogue context as input. We randomly select subset from C, de- noted as Cuub = { u7, ...., u U Then, we construct the input ØRP as:
[CLS] and [SEP] are special tokens of BERT, and [:] is concatenation operation.
Different from previous works, we add special tokens [BOR] and [BOE] to indicate the role of the utterance in the context C. These tokens are placed at the beginning of persuadeR's and persuadeE's utterances, respectively, before concatenating them into long sequence. We do the same thing for concatenating multiple utterances context in the remaining part of this paper. Next, the sequence is fed into BERT, and the [CLS] token, i.e., ENUP, is
TRP [[CLS]] Cuub: [SEP]].

---

The BERT model is trained predict the return R based on the representation of the [CLS] token. Training The three fine-tuning tasks are based on binary classification and learned jointly using multi-task learning. The loss function for the three tasks is defined as:
The representations E and E, are obtained by using the outputs of [CLS] tokens. To save com- putation cost, the parameters are shared with our fine-tumed BERT in Section 3.1.
As yet, the four elements have been represented as vectors with different dimensions. We apply a group of linear transformations align them into the same space. For example, for the utterance representations E,, we apply T', I R compute a d-dimensional vector. Finally, we obtain a quadruple (A,, G2, E, E, that corresponds to the f-th turn, where all elements have the same d dimension (to avoid redundancy, we do not use additional symbols for utterance representations after transformation).
L LNUP + 'n LCSP + 72 LRP;
where 71 and 722 are hyper-parameters control the impacts of two auxiliary tasks. Cross-entropy loss is used to implement LUUP, LCSP, and LRP.
3.2 Strategy Matching
In persuasive dialogue, the primary goal is to per- suade the other interlocutor, in addition to contin- uing dialogue. People often employ series of persuasion strategies. For instance, in the case of persuading others to donate, one can first describe the situation of catastrophe victims to elicit sympa- thy, inquire about their career, and finally suggest a suitable donation amount. We propose a sequential strategy generation method, inspired by recent study on offline reinforcement learning (Chen et al., 2021), model such persuasion process and pre- dict future strategics in an auto regressegee manner using the previous turns as input.
Auto-regressive Strategy Prediction After rep- resenting dialogues as turn sequences, we pro- pose to learn the persuasion process by predict- ing the strategies at each turn in an auto-regressive manner based on Transformer decoder (Vaswani et al., 2017). To elaborate, we structure all com- ponents in sequence for N turn dialogue: Ti [A,, G2 E, E, and Tn [FI,..., T.]] where TN RANxx To enable sequential learn- ing in an auto-regressive manner, we add position embeddings upon the quadruple T; to indicate the turn's 's index and feed Tr as input to a Transformer decoder. The generation process's 's objective is:
Turn Representation To model the persuasion process sequentially, we view a dialogue session as several turns and regard cach turn as basic representation unit. Within the 1-th turn, we con- sider four elements: the persuasion strategy set, A,, the expected gain, Gr, and representations of the utterances u, and u denoted asE, and E, (I) Strategy Vector: A, E R. is a binary vector, where the size K is the number of atomic strategies. (2) Expected Gain: The return, Rt. is divided into rewards from each turn, from which calculate the expected gain. At each turn t, we calculate the reward as r R/N, where N is the total number of turns. We then calculate the expected gain G3 by summing the rewards from the current turn up to the final turn: G, = Ei Ti.
N II (T:
arg maxo
(3)
where p is the generation probability, and Q de- notes the Transformer decoder's parameters. Training We implement the objective by the fol- lowing network:
Á+11 = sigmoid WH B7 + b),
(4)
where H is the hidden state of E, We E RRdx
b and b E R are learnable parameters. Á E
R is a distribution vector within the range of (0. 1), indicating the probabilities of selecting various strategies at the turn / + 1.
(3) Utterances: We represent the text uf and u4 by our BERT model with the input format as:
Then the learning objective is attained by opti- mizing binary cross-entropy loss as follows:
x [[CLS]; [OR]] uf : [SEP]],
xt [[CLS]; [OO]] n4 [SEP]].
N
Cg
Arj log Arj
'Experimental results show that this method is the most stable for the training process. A more efficient mothod for distributing the return can be explored in future work.
j=1
+ (1 Ary) 1og(1 A.j)l,
(5)

---

Table 1: Statistics of two persuasive dialogue datasets.
group of individuals persuades others donate to the charity organization named Save the Chil-
Statistics
PersuasionForGood DebtRiskAlert
dren. The sessions are labeled as 1, i.e., R= 1, if
Dialogue sessions
1,017
30,000
the persuadee donated or otherwise O. Persuaders
Annotated dialogues
300
4,000
can use 27 atomic strategies, i.e., K = 27, such
Atomic strategies
27
83
Turns per dialogue
10.43
4.98
as self-modeling and aaptiln-appeal, and confirm- donation.As there are only 29.5% of dialogues are labeled with persuasion strategy, we fine-tune
Words per utterance
19.36
23.11
Strategies per turn Persuaded
1.82
2.38
545 (53.6%)
17,947
(59.8%)
Not persuaded
472 (46.4%% 12,053 (40.2%)
RoBERTa model (Liu et al., 2019) these anno- tated data as classifier, and use it to obtain pre- dicted strategies for the unlabeled dialogues. This approach achieves an accuracy of 72.9%.
Matching inference phase, we use the cosine similarity between the predicted A++1 and the can- didate strategy set A of ue to calculate the persua- sion strategy matching score Sp as follows:
DebtRiskAlert We collect another dataset from real-world persuasion scenario to support this study on a commercial platform, where skilled debt re- minders alert users with overdue debts by calls and try to persuade them to repay their debt so as to reduce financial risk. We collect 30,000 dialogues, which are labeled as 1, i.e., R = 1, if the users finally make repayments within several days or oth- erwise O. A total of 83 strategies, i.e., K = 83, are used by the persuaders, such credit- protectoro partial-repay. and amoout-notiffcation. 4,000 dia- logues are manually annotated with strategy labels. Similar to the PersuasionFor ForliooooFoFFo dataset, for unla- beled data, we obtain predicted labels by a trained classifier with an accuracy of 85.0%.
Ar-1
8p |
(6)
|| l22
It is worth noting that we cannot obtain the final persuasion return R in advance during inference. Therefore, set N to the average number of turns per session and R = 1, to encourage the model to generate the corresponding persuasion strategies and pursue the persuasion goal.
3.3 Fusion for Ranking
The final score is computed based on how good a response candidate is as the next utterance for the given context and how well it matches the predicted strategies. Concretely, we linearly combine SNUP and 8p together for joint ranking score:
Comparing the two datasets (See Table 1), we find that DebtRisk has longer utterances but shorter sessions than PersuadeForGGood. This indi- cates that AleeR call receivers have less pa- tience, although the success rate in DebiRiskAlert is bit higher than PersuadeForGood. Each turn in the two datasets has 1.8 and 2.4 strategies on average, which indicates that applying multiple strategies is common in persuasive dialogue.
(1 - 8)- SNUP + 5 Spp
where 8 is a hyper-parameee to balance the scores. Such fusion widely used and can easily inter- pret the importance of two factors.
Although we attempt to optimize both L1 and L2 jointly, the results are unsatisfactory and exhibit instability during the training process. Therefore, we separately optimize the semantic matching loss L1 and strategy matching loss L2, and empirically set the optimal 5 by using validation set finally.
4.2 Evaluation
Retrieval-based dialogue systems are often eval- uated as follows (Han et al., 2021: Yuan et al., 2019). First, given dialogue context, the orig- inal response from humans is regarded positive utterance. Then, the randomly sampled M utter- ances (M = 99 in our paper) from other dialogue sessions are regarded as negative candidates, al- though some of them may be false negative. Next, all the compared methods rank the M + 1 candi- dates, and the performance is evaluated by the rank of the ground-truth response. Recall at k (R @. (=1,,)))) is used the evaluation metric. Different from an open-domain chitchat system,
4 Data and Evaluation
4.1 Datasets from Applications
To better evaluate approaches in real applications, we built an additional large dataset called Deb- tRiskAlert, which is 30 times larger than the public dataset PersuasionForGood as shown in Table 1. PersuasionForGood This dataset (Wang et al., 2019) simulates a persuasive scenario which a

---

Table 2: Comparing dialogue models with our approach two benchmarks. The best results in bold, and i indicates significant improvement over all baselines with p-value < 0.05.
Table 3: Comparing strategy selection models with our approach two benchmarks. ACC denotes accuracy when all predicted labels exactly match with ground- truth. The best results are bold.
PersuasionForGood
DebtRiskAlert
PersuasionForGood
DebtRiskAlert
R@2
R@5
R@1
R@2
R@5
ACC
Fi-score
ACC
Fi-score
SMN
4.5
8.4
16.7
20.5
34.1
56.8
BCQ
11.0
22.4
41.9
44.8
MSN
11.7
17.5
29.3
29.3
42.1
61.4
CQL
19.3
31.5
21.5
12.2
BERT
31.7
46.5
65.5
32.3
49.1
75.6
SARA Strategy
20.9
40.1
45.8
52.8
SA-BERT
29.7
44.7
65.4
33.6
50.1
78.2
w/o G
20.8
39.6
45.8
52.2
BERT-FP
37.8
53.1
71.3
47.9
68.2
87.0
w/o E
20.5
33.9
39.5
47.0
CQL
38.1
53.3
71.2
47.3
68.1
87.0
E
20.9
34.6
32.6
BCQ
38.1
53.3
71.3
61.2
72.7
87.3
w/o E E
&
20.0
31.3
30.3
32.1
SARA (ours)
39.81
53.6
71.7
65.81
75.6"
89.21
w/o Strategy
38.3
53.3
71.7
50.6
70.1
88.6
the interaction information for ranking. We choose two typical methods, SMN (Wu et al., 2017) and MSN (Yuan et al., 2019), as baselines.
a persuasive dialogue system in our real application requires the model to select appropriate strategies first and then sample some human reviewed utter- ances of the strategies to compose a final utterance, in order control data quality and non-compliance risk. Thus, we randomly sample M utterances from other strategies than the ground-truth truth strategy as negative samples for evaluating persuasive systems, Moreover, this way can also reduce the number of negatives that are actually appropriate responses. We manually label 400 negative samples ranked at the top one, based on the PersuasionFFrroow dataset, and find that sampling negative utterances results in 12% false negative rate, while sampling utterances from negative strategies yields only 7%. Please note our strategies are automatically classi- fied and thus we cannot guarantee the utterances with negative strategy labels are 100% negative.
PLM--ased Models The manner of pre-trained language model (PLM)-based methods is: 1) choos- ing pre-trained language model (e.g., BERT) the backbone; 2) concatenating the context and can- didate response into a long sequence as the input; and 3) using the output representation to compute the ranking score by a neural network layer. We choose BERT (Devlin et al., 2019), SA-BERI (Gu et al., 2020), and BERT-FP (Han et al., 2021) as PaaM- baselines, where BERT-FP is the state- of-the-art method in the response selection task. Strategy Selection Models Offline Reinforce- ment Learning can be used sequentially pre- dict strategies, and thus we choose Conservative Q-Learning (CQL) (Kumar et al., 2020) and Batch Constrained Q-learning (BCQ) (Fujimoto et al., 2019) as baselines. Their implementation details are provided in Section A.2 of Appendix.
5 Experiments
5.2 Comparison with Dialogue Models
We conduct experiments to compare model with two categories of retrieval-based dialogue models, and compare our strategy matching module with strategy selection baselines. Moreover, we do analyses to gain in-depth insights. We also conduct an online evaluation by deploying our SARA model to real users. Implementation details of SARA are described in Section A.1 of Appendix.
We compare our SARA model with the dialogue baselines and show results in Table 2. SARA per- forms the best in all metries over the two datasets, The improvements are also statistically significant in terms of R 1, the major metric for the target applications. Due to the small size of the Persua- sionForGood dataset, non-pretrained methods, i.e., SMN and MSN, cannot be fully optimized and result in low performance. Our strategy genera- tive model may also not be fully optimized, but still significantly achieves 5% relative improve- ment in R @ On the larger DebtRiskAlert dataset, our method has relative improvement of 37% in R@ and the improvements in R@2 and R@5 (36% and 1 1 1)) are also significant. These results
5.1 Baselines
Interaction-based Models The paradigm of interaction-based retrieval dialogue methods is: 1) individually encoding each utterance in the dia- logue context and the candidate response; 2) fully interacting the representation of each utterance with that of the candidate response; 3) aggregating

---

demonstrate the best effectiveness of our proposed method, particularly when data is plentiful and the strategy generative model can be fully optimized.
64
65.4
65.7
65.8
65.7
63.5
63.9
63.4
Table also shows that all proposed ideas have positive contributions. Using the two auxiliary tasks, i.e., Context Segment Prediction and Re- turn Prediction, to fine-tune BERT can improve the baseline slightly from 37.8% to 38.3% in R @ on the small dataset and dramatically from 47.9% to 50.6% on the large dataset. Without strategy match- ing the performance drops by 1.5% and 15.2% in @@ on the small dataset and the large dataset.
DehtRiskAler K@
50.8
48-
40-
36-
38.5
39.8
39.1
38.5
57.7
36.8
35.7
34.3
PerTorGion R@
24.0
Figure 3: Impact of hyper-paramees on the perfor- mance the proposed method.
The previous dialogue models did not consider persuasive strategies. To be fair, we also combine the best baseline model BERT-FP with two base- line strategy selection methods, i.e., BCQ and CQL. The results presented in Table 2. It indicates that BERT-FP+BCQ better than BERT-FP+CQL, but our SARA method still significantly outper- forms both of them.
SARA R&B
BERT-FF
15000
Training Data
Figure 4: Performance according to amount of train- ing data in DebtRiskAlert.
5.3 Comparison of Strategy Models
the semantic matching module. When they are combined, we can obtain significantly better results when is in the wide range from 0.2 to 0.6. indi- cates that combining the two modules stably leads to significant improvement in performance. Over the small dataset, although the strategy matching module cannot be fully optimized and get lower per- formance than the semantic matching module, the fused results are still better than semantic matching alone when 8 is from 0.1 to 0.2. Again, indicates that our SARA framework is consistently effective.
We evaluate how well our proposed method, two of- fline reinforcement learning baselines, and several variants of ours, on predicting the next persuasion strategies. We use accuracy for multi- classifi- cation and average Fl-score for different labels as metrics. Table shows the results.
As shown in the table, our proposed method is the best among all methods. Our method improves the best performance of two baselines by 8% in accuracy and 27% in Fl -c-oe on the Persuasion- ForGood data and by 9% in accuracy and 17% in Fl-score on the DebuRiskAlert data. This indicates that our proposed strategy generation is the most effective in predicting the next strategies. When remove the expected gain, the performance slightly drops; whereas when removing E E, or both, the Fi-score dramatically decreases. This indicates that predicting the next strategies heavily relies on previous utterances of persuaders and persuadees. Previous strategies are the main contributors as using them and gains (w/o E. & E, can still achieve 31.3 and 32.1 in Fl-score on two datasets.
5.5 Impact of Training Data Scale
We investigate whether the performance difference is caused by the scale of training data. Thus, we randomly sample 1,000 sessions from the Deb- tRiskAlert dataset, which is the same scale the PersuadeForGoodd dataset, to train model and then increase the scale of training data little by little. We draw their performance in Figure 4. When us- ing 1,000 sessions, the performance drops to 35.6 in R @ 1, which is even lower than that on the Per- suadeForGood. With increasing training data, the performance of our model continues to improve. This indicates that we do need large-scale data to train models and draw convincing conclusions. In addition, although the performance of the BERT- FP baseline improves with more training data well, the increased gap between the two curves indicates that more training data benefit strategy matching more than semantic matching.
5.4 Effeet of the Fusion Weight
Figure illustrates the effect of the fusion weight 8 on the performance of proposed method. When 0. it equals semantic matching only; whereas it equals to strategy matching only when 8 1. On the DebtRiskAlert dataset, the strategy matching module has significantly outperformed

---

5.6 Online Evaluation
Jain and Srivastava, 2021). Some work (Wu et al., 2021) assumes that persuasion strategies are hid- den in the persuader's utterances, and thus a sep- arate conversation model is learned from the per- suader's s content, which reinforced by the feed- back from a model learned from the other per- son. Some works (Shi et al., 2021; Samad et al., 2022) build persuasion strategy classifier and add the loss of matching the strategies of generated and ground truth responses during the training, but they do not explicitly use strategies in inference. Some work (Chen et al., 2022) assumes that the proper strategy is given and investigate how to take the strategy as condition to generate better re- sponse. Different from the previous works, to our best knowledge, we are the first to explicitly pre- dict the proper strategies by the whole sequence of previous strategies, utterances from the persuader and the other person, and explicitly integrate the predicted strategies in scoring the best response. Offline Reinforcement Learning (RL) Ap- proaches provide offline learning paradigm for constructing sequential decision-making engines based on the fixed dataset without online environ- ment interaction. Recently, offline RL approaches have been used in lask-oriented dialogue systems. which were used to optimize the policy of response selection to achieve user goals (Jang et al., 2021; Wen et al., 2017: Peng et al., 2018). Since some representative offline RL algorithms, such as Deci- sion Transformer (Chen et al., 2021) can conduct sequential action selection conditioned on preset target, they provide a promising way model the sequential strategy generation process and inspire our ideas to improve persuasive dialogue.
To deploy the proposed SARA model on the debt risk alert platform, we make several additional efforts to ensure quality, safety and user experi- ence. First, we acquired 388 human confirmed high-quality utterances, which is much fewer than the number of candidate utterances in offline evalu- ation, covering all strategies. Then we distilled the SARA model to a smaller size to comply with latency requirements. To ensure speech quality, we invited skilled human speakers to convert those 388 textual utterances into spoken utterances. Finally, we use the distilled SARA model to select appropri- ate responses and concatenate their corresponding spoken utterances when talking to real users.
Among the 9,396 calls that SARA made within five days, 2,204 users repay their debts on the day they receive the call, resulting in a repayment rate of 23.5%. During the same period, the best online model based on semantic matching and heuristic rules achieves repayment rate of 21.9%, and the repayment rate for those who do not receive any call is 15.8% This indicates that our SARA model improves the effectiveness of achieving the ulti- mate goal of persuasion in real-world applications.
6 Related Work
Retrieval-based Dialogue Systems aim to select proper responses from large-scale response repos- itory by estimating the matching degree between context and response (Wu al., 2017: Yuan et al., 2019). Pre-trained methods have become the domi- nant approach (Gu et al., 2020; Xu al., 2021; Han et al., 2021). Directly fine-tuning the pre-trained model as a dialogue model is less effective because of domain gap. A common practice is to further train the model with the language modeling objec- tives using texts from the target domain to reduce the negative impact (Han al., 2021; Gu al., 2020). As a preliminary study of modeling persua- sion strategy sequentially in persuasive dialogue, we follow the retrieval-basee paradigm. In addition to being easier to evaluate, it is more controllable and thus can respond ethically in real applications. Persuasive Dialogue Systems aim to influence the views, attitudes, or bchaviors of people through conversation. Compared to task-oriented and chitchat-oriented dialogue, identifying and apply- ing persuasion strategies are unique and essential in developing persuasive dialogue systems (Carlile et al., 2018; Wang et al., 2019; Tian et al., 2020;
7 Conclusion
In this paper, we propose new framework called SARA that jointly considers semantic and strat- egy matching in persuasive dialogue. For strategy matching, we propose using an auto-regressive gen- erator to sequentially predict the next appropriate strategies based on all previous utterances, strate- gies, and expected gain to return. We also collect a new large-scale dataset from real application. Ex- perimental results on the dataset and small public dataset indicate that integrating strategy matching can significantly improve the state-of-the-art dia- logue baselines in terms of R@ Our proposed strategy generation model performs the best among strategy selection baselines. Online evaluation also

---

indicates the effectiveness of our model in achiev- ing the ultimate goal of persuasion. The big impact of training data scale suggests using enough data necessary in future work. We plan to improve our persuasive dialogue systems by supervised data, digiffferetii high-quality training data from others by the average return of a caller.
References
Winston Carlile, Nishant Gurrapadi, Zixuan Ke, and Vincent Ng. 2018. Give me more feedback: Amno- tating argument persuasiveness and related attributes in student essays. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics, ACI. 2018, Melbourne, Australia, July 15-20, 2018, Volume I: Long Papers, pages 621 621-631, Association for Computational Linguisties.
Acknowledgements
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. 2021. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Pro- cessing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 15084-1509)
This work is supported by the Fundamental Re- search Funds for the Central Universities, and the Research Funds of Renmin University of China (21XNLG28), and National Natural Science Foun- dation of China (No. 62276268).
Maximillian Chen, Weiyam Shi, Feifan Yan, Ryan Hou, Jingwen Zhang, Saurav Sahay, and Zhou Yu. 2022. Seamlessly integrating factual information and social content with persuasive dialogue. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joins Conference on Natural Language Processing, AACL/UCNI.P 2022 Volume 1: Long Papers, Online Only, November 20-23, 2022, pages 399-413, Association for Computational Lin- guistics.
Limitations
Due to concerns about the compliance and qual- ity of online experiments, it is difficult to directly evaluate the persuasion outcomes of different meth- ods online. However, preliminary exploration of supervised retrieval-based persuasive dialogue systems, there is still a gap between the current persuasive dialogue system and human generated supervised data. Thus, it is reasonable to evaluate the persuasive dialogue system with real human responses. Considering user privacy, we cannot di- rectly release the DebtRiskAlert dataset. However, it is not difficult to collect and construct similar persuasive dialogue datasets when researchers co- operate with the industry.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutamova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLE 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pages 4171-4186. Association for Computational Linguistics.
Scott Fujimoto, David Meger, and Doina Precup. 2019. Off-policy deep reinforcement learning without ex- ploration. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2052-2062. PMLR.
Ethical Statement
The study of persuasive dialogue systems is still in its early stages. As the system is designed to achieve some given purposes, such changing the user's attitude. it should be carefully checked to avoid providing offensive utterances. In this study, we formulate the research problem in a retrieval- based setting. Therefore, we have all utterance candidates manually checked and ensure that all of them are proper for an online dialogue. Besides, in our collected dataset (hebRRiskAlert) we have removed all user-related information to keep user privacy. It is worth noting that, though our system can be applied to various scenarios, should not be used replace humans directly. All applications should be performed under human supervision.
Jia- Chen Gu, Tianda Li. Quan Liu, Zhen-Hua Ling, Zhiming Su, Si Wei, and Xiaodan Zhu. 2020. Speaker-aware BERT for multi-turn response selec- tion in retrieval -based chatbots. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Viriual Event, Ireland, October 19-23, 2020, pages 2041-2044 ACM.
Janghoo Han, Tacsuk Hong, Byoungjae Kim, Youngjoong Ko, and Jungyun Sco. 2021. Fine- grained post-training for improving retrieval-based dialogue systems. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2021, Online, June

---

6-11, 2021, pages 1549-1558. Association for Com- putational Linguistics.
Associaatioo
Weiyan Shi, Yu Li, Saurav Sahay, and Zhou Yu. 2021. Refine and imitate: Reducing repetition and inconsis- tency in persuasion dialogues via reinforcement learn- ing and human demonstration. In Findings of the Association for Computational Linguistics: EMNLP 2021, Viriual Event / Punta Cana, Dominican Re- public, 16-20 November, 2021, pages 3478-3492. Association for Computational Linguistics.
Ayush Jain and Shashank Srivastava. 2021. Does social pressure drive persuasion in online fora? In Proceed- ings of the 2021 Conference Empirical Methods in Natural Language Processing, EMNLP 2021, Vir- tual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 9201-9208. Association for Computational Linguistics.
Youzhi Tian, Weiyan Shi, Chen Li, and Zhou Yu. 2020. Understanding user resistance strategies in persuasive conversations. In Findings of the Association for Computational Linguistics: TMNNP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 4794 4798. Association for Computational Linguistics.
Youngsoo Jang. Jongmin Lee, and Kee-Eung Kim. 2021. GPT-critic: Offline reinforcement learning for end-to- end task-oriented dialogue systems. In Proceedings of the 9th International Conference Learning Rep- resentations.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- Learning Representations. ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pages 5998-6008.
national Conference
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conservative q-learning for offline reinforcement learning, In Advances in Neural In- formation Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang. Jingwen Zhang, and Zhou Yu. 2019. Per- suasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019, Florence, Italy, July 28- Augusi 2, 2019, Volume 1: Long Papers, pages 5635-5649, Association for Computational Linguis- tics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zetllemyee, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary Vetioo, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, gegg-heeffrmmnce deep learning li- brary. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurlPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada, pages 8024 8035.
Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Mil- ica Gasic, Lina Maria Jojas-Barahons. Pei-Hao Su, Stefan Ultes, and Steve J. Young. 2017. network- based end-to-end trainable task-oriented dialogue sys- tem. In Proceedings of the 15th Conference of the European Chapter of the Association for Computa- tional Linguistics, EACI.2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers, pages 438-449, Association for Computational Linguistics.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2021. Alternating recurrent dialog model with scale pre- trairra language models. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 1292- 1301, Association for Computational I Linguisties.
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. 2018. Deep dyna-q: Integrating planning for task- completion dialogue policy learn- ing. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 1 15-20, 2018, Vol- ume 1: Long Papers. pages 2182-2192. Association for Computational I. Linguistics.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new archi- tecture for multi-turn response selection in retrieval- based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACI. 2017, Vancouver, Canada, July 30- August 4, Volume 1: Long Papers, pages 496-505. Association for Computational Linguistics.
Azlaan Mustafa Samad, Kshitij Mishra, Mauajama Fir- daus, and Asif Ekbal. 2022. Empathetic persuasion: Reinforcing empathy and persuasiveness in dialogue systems. In Findings of the A Association for Computa- tional Linguistics: NAACL 2022, Seattle, WA, United States, July 15-15, 2022, pages 844-854 856. Association for Computational Linguistics.
Ruijian Xu, Chongyang Tao, Daxin Jiang, Xueliang
Zhao, Dongyan Zhao, and Rui Yan. 2021. Learn- ing an effective context-response matching model

---

with seff-supervised tasks for retrieval-basee dia- logues. In Thirty-Fiftht AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thise Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Vir- tual Event, February 2-9, 2021, pages 14156-14166.
previous turn
0
1
2
3
4
5
6
7
8
1.25
1.00
current turn
AAAI Press.
0.75
Chunyuan Yuan, Wei Zhou, Mingming Li, Shangwen Lv, Fuqing Zhu, Jizhong Han, and Songlin Hu. 2019. Multi-hop selector network for multi-turn response selection in retrieval-based chatbots. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internu- tional Joint Conference on Natural Language Pro- cessing, EMNLP-IJCNI 2019, Hong Kong. China, November 3-7, 2019, pages 111-120. Association for Computational Linguistics.
0.50
8
0.25
10
0.00
Figure 5: Average normalized attention weights of dif- ferent turns over the entire PersuasionForGood dataset.
issue lies in designing the state, action, and reward. When predicting the action for the + turn, we take the following design into account:
A Appendix
A.1 Implementation Details of SARA
We implement our model by PyTorch (Paszke et al., 2019). The maximum length of the input sequence is set 256. If the length of the input sequence exceeds the maximum length, we remove tokens from the shorter one of the context and the candi- date response. 7/1 and 72 are both set to 0.1. The BERT model for dialogue modeling is trained for 10 epochs with a batch size of 96. Adam opti- mizer (Kingma and Ba, 2015) is applied with learning rate of le-5. The strategy decision model is implemented based on Decision Transformer. The maximum length of the input sequence is set as 120. The representation dimension d is set to 128. The number of hidden layers and heads is set to 12 and 8, respectively. The model is also optimized by the Adam optimizer with a learning rate set of le-5. The training epochs is 50 with a batch size of 8. In the inference stage, we set the re- and 10, respectively. Following
Action: We use the binary strategy vector
A++1 as the action the + turn;
State:
We consider using persuadee's utter- ances u, persuader's utterance u, and cor- responding persuasion strategies Ar as states. So we concatenate the feature representations E E, and A, as input:
We calculate the reward as T R/N, where R represents the return and denotes the total number of turns.
Reward:
is worth mentioning that, based on this design,
the offiine RL baselines align with the proposed strategy selection model, as they also integrate se- mantic and strategy information as input.
A.3 Attention Weight Analysis of Persuasion In Figure 5, we normalize the attention weights different turns based on the PersuasionFor Good dataset. The brighter yellow color means higher attention weight while the darker blue color means lower. We find that for any turn, all previ- ous turns are somehow useful except for the first turn, which most likely greeting and mention- ing the topic. The most interesting finding is that the fourth-turn is the most important for the later strategy generation. It may be caused by that the persuadee having obvious attitude whether he/she wants donate. The latter strategies are chosen around this key information.
turn and turns previous work (Wang et al., 2019), perform five- fold cross-validations on the PersuasionForGood dataset and use the average scores of each fold to compare different models. For the DebtRiskAlert dataset, it divided into 20k, 5k, and 5k dialogues as training, validation, and testing sets, respectively. We train our model on 2 100 GPUs, and it takes about 16 hours to run all the experiments on both datasets.
A.2 Details of Offline RL Baselines
When considering the offline reinforcement learn ing methods CQL and BCQ baselines, the key
httt:::///ttuub.coomm


Reviews:
[{"review_id": "b2TUzESAWu", "reviewer": "Reviewer_pxCx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewer for providing valuable feedback. We are encouraged that Reviewer\\#pxCx acknowledged our work. We have carefully considered the concerns raised and have prepared a response to address them.\n\n**Q1: Limited improvement in R@2 and R@3 on PersuasionForGood**\n\nWe have the same observations as the reviewer does but the improvements are significant on DebtRiskAlert. As Figure 3 shows, the strategy-only model performs much worse than the semantic-only model on the small PersuasionForGood dataset; whereas, the strategy model alone can beat the semantic model on the large DebtRiskAlert dataset. These indicate that the strategy model cannot be sufficiently trained on such a small dataset as PersuasionForGood. We need large enough datasets to draw reliable conclusions or discover effective ideas for improving persuasive dialogues.\n\n**Q2: Non-public availability of the DebtRiskAlert dataset**\n\nDue to privacy issues, we cannot release the dialogues in the DebtRiskAlert dataset. We are trying to figure out other ways for safe sharing, e.g., releasing only token ids. Fortunately, many companies have their own logs of persuasion dialogues and thus they could reproduce or follow up on our work. In addition, building a larger public dataset with at least 20,000 dialogues is possible when our community pays more attention to the dialogues with a session-based goal.\n\n**Q3: How to solve the problem of multiple choice of strategies in an utterance**\n\n$A$ is a distribution vector indicating the probabilities of different strategies. For the candidate $u_c^+$, we apply the strategy classifier to get the distribution $A_c$, where multiple dimensions may have non-zero values. When we use our proposed Transformer decoder to predict $A_{t+1}$, which is the distribution of the next strategies, we calculate the cosine similarity between the predicted strategies and the candidate response strategies (Equation 6). If the predicted next strategies have multiple choices, the candidate with the same distribution of multiple strategies would have higher scores. We will clarify this in the revised paper.\n\n**Q4: The SOTA baseline is proposed in 2021**\n\nDifferent from generation-based dialogue models, retrieval-based dialogue models have not made much progress in recent years. However, retrieval-based dialogue systems are still widely used in industry, in particular when the product is required to be trustworthy. To our best knowledge, BERT-FP is still one of the best-performing methods. Some follow-up works have shown relatively small improvements over it or the improvements are not applicable to our focused problem. Our focus is on how combining the SOTA semantic models with strategy matching works for persuasive dialogue applications. Therefore, it is appropriate to choose BERT-FP as the baseline. We will add a more detailed explanation in the related work section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written. The motivation is clearly stated, and the contribution and methods are clearly described.\n2. This paper proposed a novel framework called SARA to assess the quality of response candidates from both utterance and strategy perspectives.\n3. In contrast to prior work, the authors considered the strategy matching degree in response selection, and developed an auto-regressive decoder to generate persuade strategies sequentially. This is a novel attempt in the persuasive dialogue task.\n4. The authors constructed a large-scale dataset DebtRiskAlert to facilitate the persuasive dialogue research field. Extensive experimental results on both the public benchmark dataset and self-collected dataset verify the effectiveness of the proposed network in this paper.", "reasons_to_reject": "1. The R@2 and R@3 results of experiment show that the proposed model is only 0.3-0.4 points higher than SOTA on PersuasionForGood dataset. Meanwhile, the w/o strategy result is the same as the main results on the R@5 metrics, which may be difficult to convince the effectiveness of the proposed method.\n2. The collected dataset DebtRiskAlert is not public-available, which hinders the reproducibility of the experiments.\n3. The author mentions that the persuader can use multiple strategies in a single utterance. The authors could clarify how they solve the problem of multiple choice of strategies in the method\n4. The SOTA model BERT-FP used by the authors is from year 2021. The authors could chose a more up-to-date model for the comparison.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper mainly studies persuasive dialogue which aims to persuade interlocutors to achieve specific goals through conversations. This paper proposes a novel dialogue framework that simultaneously matching strategies and semantic. This network utilizes a BERT model to measure the semantic similarity and an auto-regressive decoder to generate next conversation strategies. Extensive experimental results on a benchmark dataset and a self-constructed dataset verify the effectiveness of the proposed framework.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fBYEXE72Wh", "reviewer": "Reviewer_urL8", "summary": "", "questions": "1. You mention in line 325 that \"we separately optimize the semantic matching loss L1 and strategy matching loss L2\". Can you elaborate how did you do this?\n2. Why is the drop in performance for the w/o Strategy setting in Table 4 much higher for DebtRiskAlert compared to PersuasionForGood?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude to the reviewer for providing valuable feedback. We are encouraged that Reviewer\\#urL8 acknowledged our approach and dataset. We have carefully considered the concerns raised and have prepared a response to address them. \n\n**Q1: The variables table and Figure 2**\n\nWe will add a table in the revised paper to explain our variables. We will also re-design Figure 2 by including an example for better illustration. \n\n**Q2: How separately optimize the semantic matching loss $L_1$ and strategy matching loss $L_2$**\n\nWe first optimize the semantic matching loss $L_1$, then freeze the trained BERT model and optimize the strategy matching loss $L_2$. We have tried to optimize them together, but experimental results show that such a separate training strategy is more stable.\n\n**Q3: Drop in performance for the w/o Strategy setting in Table 4 (Figure 4) much higher for DebtRiskAlert**\n\nWe have the same observations as the reviewer does. As Figure 3 shows, the strategy-only model performs much worse than the semantic only model on the small PersuasionForGood dataset; whereas, the strategy model alone can beat the semantic model on the large DebtRiskAlert dataset. These indicate that the strategy model cannot be sufficiently trained on such a small dataset as PersuasionForGood. We need large enough datasets to draw reliable conclusions or discover effective ideas for improving persuasive dialogues. \n\nIn addition, compared to the PersuasionForGood dataset, DebtRiskAlert has finer-grained strategy annotations (83 atomic strategies vs. 27 atomic strategies), and the strategy classifier provides more accurate annotations in terms of accuracy (85.0% vs. 72.9%). This makes our strategy selection module more effective on the DebtRiskAlert dataset.\n\n**Q4: Move the related work section after introduction**\n\nWe will follow this suggestion and move the related work section after the introduction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a novel approach that jointly models strategic and semantic matching. Several baselines, offline and online experiments have been conducted to establish the efficacy of the proposed framework. \n2. The authors construct a new dataset that is 30 times larger than the Persuasion For Good to conduct thorough experiments. The dataset would be valuable to the community if released.\n3. The paper is well-written and easy to follow.", "reasons_to_reject": "1. Given the multiple variables introduced in section 3, it'll help improve the readability if the authors include a notations/nomenclature table.\n2. Figure 2 can be improved. Currently, it's a bit hard to understand the flow chart properly. It'll be helpful to add an example within the figure to explain its working.", "questions_to_authors": "1. You mention in line 325 that \"we separately optimize the semantic matching loss L1 and strategy matching loss L2\". Can you elaborate how did you do this?\n2. Why is the drop in performance for the w/o Strategy setting in Table 4 much higher for DebtRiskAlert compared to PersuasionForGood?", "paper_topic_and_main_contributions": "The paper proposes a novel approach to generate persuasive text that jointly models semantic and strategy matching to create a persuasive dialogue system. The joint modeling of semantics and strategies has not been extensively attempted in the literature previously. The proposed method is evaluated on the Persuasion For Good and Debt Risk Alert datasets, where the reported results surpass previous state-of-the-art results. \n\nThe paper is well-written, and the proposed approach appears sound. If the paper gets accepted, I recommend the authors release their dataset and implementation for community use.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. A suggestion: the Related Works section can be moved up right after the introduction or problem formulation since it helps give the readers an idea about the landscape before diving into the proposed approach."}}, {"review_id": "PoUZf1pDwD", "reviewer": "Reviewer_xjnT", "summary": "", "questions": "\n\n1. Why do you need L_RP in semantic matching? What is the purpose of this loss?\n2. In Problem Formulation, what is the merit of defining the task as a retrieval-based task? Please correct me if I'm wrong, but my understanding is that this definition assumes there is already a pool of candidate responses available. Does it reflect realistic scenarios?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive feedback and provide the following responses to the concerns. \n\n**Q1: Couldn't find links of these numbers to the results in evaluation sections**\n\nThe improvements we report are relative to the baseline BERT-FP: the 5% is calculated by (39.8-37.8)/37.8, and the 37% is calculated by (65.8-47.9)/47.9. We will add relative improvements to Table 2 and revise our paper accordingly.\n\n**Q2: Why we need the two auxiliary losses in semantic matching training**\n\nFor our addressed persuasive dialogue problem, not only do we need to select a relevant next utterance, but we also need to pay attention to the long-term goal of persuading the other party. Therefore, we design the Context Segment Prediction task to improve the model's ability to predict the next half-session, which somehow captures the relation between utterances in a higher resolution. The Return Prediction task is used to predict whether the persuadee is persuaded, i.e., the ultimate goal, and thus can help the model know which status it stands on in the progress of persuasion and choose a more suitable response for the current status. As Table 2 shows, with auxiliary tasks (Line SARA w/o strategy) our method outperforms the baseline BERT-FP with the next utterance prediction task only. We will add some explanations in the revised paper.\n\n**Q3: What is the merit of defining the task as a retrieval-based task?**\n\nIn our target realistic applications, it require that models give correct, relevant and compliant responses, which current generation-based models can difficult to meet. That is why retrieval-based systems are in demand and widely used. A retrieval-based system can better guarantee the quality of responses and is easy to control. For example, in the DebtRiskAlert scenario, we can review all response candidates to ensure their quality and minimize risks. The retrieval-based systems are also superior in latency to generation-based systems like LLMs.\n\nIn addition, our aim is to address the important and challenging problem of how to consider the long-term goal, e.g., persuading the other party, in dialogue systems. This is difficult for both retrieval and generation-based systems. As a good start, it is easier to control variables to investigate the problem in retrieval-based dialogue systems. Once we demonstrate that strategy prediction is effective for persuasive dialogues, we are more confident to explore how to integrate long-term goals into the LLM-based generation models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Main reasons to accept:\n\n1. The proposed model combines both strategy and semantic into account for persuation. The application of reinforcement learning is an interesting use in this study. \n2. The result looks significantly better than baselines on both datasets.", "reasons_to_reject": "Main reasons to reject:\n\n1. Paper is a little hard to follow. \n     a.  In abstract, it claims that the method improves the state-of-the-art baseline by 5% on a small dataset and 37% on a large real-world dataset in terms of Recall@1. I couldn't find links of these numbers to the results in evaluation sections. \n     b. The 2 auxiliary losses in semantic matching training is not very clear to me about how helpful they are. \n \n2. This task is defined as retrieval-based task, which would limit its use to the dialogue systems in general. ", "questions_to_authors": "\n\n1. Why do you need L_RP in semantic matching? What is the purpose of this loss?\n2. In Problem Formulation, what is the merit of defining the task as a retrieval-based task? Please correct me if I'm wrong, but my understanding is that this definition assumes there is already a pool of candidate responses available. Does it reflect realistic scenarios?", "paper_topic_and_main_contributions": "This work proposes a method for persuasive dialogues by using a generator to generate a proper strategy for the current turn based on context semantics. Specifically, at each persuader's turn, the strategy matching module would pick the best strategy candidate and then , based on the strategy, the semantic matching module would pick the best response candidate. Evaluation results show significantly better results with the proposed methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LI6V56sqcL", "reviewer": "Reviewer_zkSK", "summary": "", "questions": "\t1. On the top left in Fig.2, the \"CSP\" and \"RP\" appeared. Is it right?\n\t2. Compared with ChatGPT, how is the performance? Have tried?\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive feedback and provide the following answers to the concerns. \n\n**Q1: The proposed model is not novel enough...not meet the high criteria of EMNLP**\n\nAs Conference on Empirical Methods in Natural Language Processing (EMNLP) emphasizes \"Empirical\" in its name, being \"novel\" covers not only a new model architecture but also new empirical ideas that are proved to be a significantly better solution to an important task. We shall also measure whether a method is novel by considering how much impact it makes. Our community has too many published papers today but few of them do work in real applications and widely used. The latter influences the impact of a conference indeed.\n\n**Here we clarify our novelty and contributions again:**\n\n1.  We address an important but difficult problem. Even for ChatGPT, the persuasive dialogue task is difficult and important because it requires the ability to achieve a goal via a long conversation. It is still a challenging task for our community and it is the core ability of many real applications on automatic calling. \n2.  Our paper is the first work to propose combining strategy selection with semantic matching, which brings significant improvements in the effectiveness of persuasive dialogue systems; \n3.  We are the first to discover that the small dataset is not enough to verify the effectiveness of an idea on persuasive dialogue and also provide insights on how many is good enough; \n4.  In addition to offline evaluation, we made great efforts to push our ideas online. Our proposed method serves thousands of users and makes a real impact.\n\n**Q2: The baseline method is outdated**\n\nDifferent from generation-based dialogue models, retrieval-based dialogue models have not made much progress in recent years. However retrieval-based dialogue models are still widely used in industry, in particular when the product is required to be trustworthy. To our knowledge, BERT-FP is still one of the best-performing methods. Some follow-up works have shown relatively small improvements over it or the improvements are not applicable to our focused problem. Our focus is on how combining the SOTA semantic models with strategy matching works for persuasive dialogue applications. Therefore, it is appropriate to choose BERT-FP as the baseline. We will add a more detailed explanation in the related work section. \n\n**Q3: The two reference papers the reviewer provided**\n\nWe appreciate the advice and will include these two papers in our related work session. However, these two papers do not weaken our contributions:\n\n- In ``Do You Know My Emotion? Emotion-Aware Strategy Recognition Towards a Persuasive Dialogue System``, authors investigate strategy recognition, i.e., predicting which strategies the persuader used in historical dialogues. It is related to our strategy classifier, but our task is to plan appropriate persuasive strategies for the persuader for future dialogues. \n- In ``Understanding User Resistance Strategies in Persuasive Conversations``, authors focus on annotating resistance strategies based on historical dialogues (similar to strategy recognition based on historical dialogues mentioned above). This is related but different from our work to predict appropriate next strategies and select the best response.\n\n**Q4: On the top left in Fig.2, the \"CSP\" and \"RP\" appeared**\n\nWe intend to show the simultaneous training of three tasks in the larger figure at the top, and then show the details of two auxiliary tasks at the bottom. We will improve the illustration in the revised version.\n\n**Q5: Compare with ChatGPT**\n\nAfter the rise of ChatGPT, we also try to apply LLMs to our persuasive dialogue scenarios. We attempt to use ChatGPT and fine-tune open-source LLMs on our data, but the experimental results were disappointing. Although LLMs can generate seemingly good responses, some of them contain hallucinations and it is difficult to control LLMs to generate trustworthy replies only. Moreover, it is difficult to evaluate the generation models. We are still doing research on how to improve ChatGPT-like models in our domain and how to effectively consider long-term goals in LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\t1. The proposed model outperforms some baselines from 2019-2021.\n        2. The paper considers a real-world dataset, looking forward to the dataset and code.", "reasons_to_reject": "\t1. The proposed model is not novel enough, it seems combining the work A and work B, which is not meet the high criteria of EMNLP.\n\t2. The compared baselines are too old, which comes from 2019, 2020, 2021. More new models and more analyses could be better.\n\t3. There are two another papers which are relevant with strategy prediction.\"Do You Know My Emotion? Emotion-Aware Strategy Recognition towards a Persuasive Dialogue System\",  \"Understanding User Resistance Strategies in Persuasive Conversations\" \n", "questions_to_authors": "\t1. On the top left in Fig.2, the \"CSP\" and \"RP\" appeared. Is it right?\n\t2. Compared with ChatGPT, how is the performance? Have tried?\n", "paper_topic_and_main_contributions": "Persuasive dialogue aims to persuade users to achieve specifc goals through conversations. While previous models mostly rely on matching utterance semantics and neglect the strategies of a conversation. Therefore, the paper proposes a novel persuasion framework that combines the modeling of conversation semantics and strategies, which consists of a BERT-like module and an auto-regressive predictor that match the semantics and strategies, respectively. Experiments have shown that the model improve some baselines on two datasets.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See above.", "Typos_Grammar_Style_and_Presentation_Improvements": "See above."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Jointly modeling of conversation semantics and strategies improves final model performance. The paper is well-written and well-motivated; however, it should incorporate suggested changes by our reviewer.
Decision: Accept-Findings