Stylized Dialogue Generation with Multi-Pass Dual Learning
Authors: Jinpeng Li
Yingce Xia
Rui Yan
Hongda Sun
Dongyan Zhao
Tie-Yan Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Dual learning
Stylized dialogue generation
TCFC
SDGC

Abstract: Stylized dialogue generation, which aims to generate a given-style response for an input context, plays a vital role in intelligent dialogue systems. Considering there is no parallel data between the contexts and the responses of target style S_1, existing works mainly use back translation to generate stylized synthetic data for training, where the data about context, target style S_1 and an intermediate style S_0 is used. However, the interaction among these texts is not fully exploited, and the pseudo contexts are not adequately modeled. To overcome the above difficulties, we propose multi-pass dual learning (MPDL), which leverages the duality among the context, response of style S_1 and response of style S_0. MPDL builds mappings among the above three domains, where the context should be reconstructed by the MPDL framework, and the reconstruction error is used as the training signal. To evaluate the quality of synthetic data, we also introduce discriminators that effectively measure how a pseudo sequence matches the specific domain, and the evaluation result is used as the weight for that data. Evaluation results indicate that our method obtains significant improvement over previous baselines.


Stylized Dialogue Generation with Multi-Pass Dual Learning
Jinpeng Li', Yingce Xia?, Rui 3.5 Hongda Sun Dongyan Zhaol Tie-Yan Liu?
Wangxuan Institute of Computer Technology, Peking University, Beijing, China
Microsoff Research Asia, Beijing, China 3 Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China Institute for Artificial Intelligence, Peking University, Beijing, China Beijing Academy of Artificial Intelligence, Beijing, China
1ij i.jinpeng@stu. pku. edu. en, {yingce. xia, tyliu)@microsoo com, {ruiyan, sunhongda98) &ruc. edu cn, zhaody@pku. edu. cn
Abstract
Stylized dialogue generation, which aims to generate given-style response for an input context, plays vital role in intelligent dialogue systems. Considering there no parallel data between the contexts and the responses of target style S₁, existing works mainly use back translation to generate stylized synthetic data for training, where the data about context, target style S1 and an intermediate style So is used. However, the interaction among these texts is not fully exploited, and the pseudo contexts are not adequately modeled. To overcome the above difficulties, we propose multi-pass dual learning (MPDL), which leverages the duality among the context, response of style S, and response style So. MPDL builds mappings among the above three domains, where the context should be reconstructed by the MPDL framework, and the reconstruction error is used as the training signal. To evaluate the quality of synthetic data, also introduce discriminators that effectively measure how a pseudo sequence matches the specific domain, and the evaluation result is used the weight for that data. Evaluation results indicate that our method obtains significant improvement over previous baselines.
Introduction
In recent years, dialogue systems have attracted surge of research interest and achieved great The dialogue generation task can be formulated as a sequence-to-sequuecc
success 2 3
learning problem 6). where the source sequence is the given context, and the target sequence is the response. Stylized dialogue generation, which automatically generates response following the desired style, obtains significant progress in open-domain dialogue systems. The styles could be serious/humor, etc.
formal/informal, Soodern/Shakespeareee
A core challenge of this task is the lack of parallel training data between the contexts and responses of desired styles (denoted as style S1), which limits the performance models. A natural solution to establish the pseudo pairs between contexts and responses of style S₁ using unsupervised semi-supervised learning. The parallel data between contexts and responses of style So (denoted as style So) is leveraged. Note the conversations corpus (context, style So) are relatively easy obtain. After that, the stylized dialogue model can be trained on the synthetic data. Specifically, Su et al. [7] propose diversifying dialogue generation model based on iterative back translation (8). Zheng et al. [9] propose a style routing approach with a joint training process, where an inverse model with style embeddings is used generate pseudo pairs and train the stylized dialogue generation model.
corresponding authors: Rui Yan (ruiyan@e @ ruc.edu.cn) and Dongyan Zhao (zhaody @ pku.cdu.cn).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Although the above methods achieved success in the stylized dialogue generation, there still exists lots of room to improve. First, the interaction between context, style So and style S is not fully exploited. An example is that a pseudo context can he explicitly obtained by two-hop process, Si + So -pseedo context instead of using style embedding only. Second, in the aforementioned methods, the quality of the pseudo contexts generated through back translation is not properly modeled. By manually checking the generated data, we find that there are quite a few pseudo contexts that contain irrelevant contents of the response. Several common patterns in the pscudo contexts and responses include "I don't know", "I'm so happy", etc. previous methods, each pseudo data is treated equally regardless of their quality, which introduces bias to the training set and hurts the performances of model.
In this paper, we propose new framework, multi-pass dual learning (MPDL), for stylized dialogue generation. Inspired by dual learning [00] that can automatically extract information through two dual tasks in an unsupervised manner, we extend it to multi-pass version. In MPDI,, there are three dual tasks in total, including contexte-style S1, context So and style S. <+styyle St. The unlabeled texts in style S and contexts are leveraged. We also introduce two discriminators to evaluate the quality of the generated data. The discriminators are used to measure the similarity of generated data to the corresponding domains. The similarity is used the weight of the synthetic data for training. by which we can adaptively use the unlabeled data. We conduct experiments with benchmarks of stylized response generation in formal English and Shakespearean English. We achieved state-of of-ee----rerrrr results in terms of both automatic evaluation and manual judgment.
Our contributions are summarized as follows:
(1) We propose multi-pass dual learning (MPDL) framework for stylized dialogue response generation, that can effectively leverage the unlabeled data; (2) Compared with standard dual learning, we introduce two discriminators to evaluate the quality of the pseudo parallel data. This is new attempt for the general dual learning framework; (3) We provide new dataset for this task and set several benchmarks using our method; (4) We empirically verify the effectiveness of MPDL on two datasets with formal and Shakespearean response generation.
2 Related work
Text style transfer aims to re-write a sentence with given stylistic properties. Some early works try supervised approaches, where work is the transfer between modern English and Shakespearean style sentences [2] Nevertheless, more and more recent research begins to pay attention to the unsupervised approach due to the lack of parallel corpora [13] [4]. In general, there are three typical approaches. The first is to learn disentangled representations of content and style. Shen et al. [15) learned a shared representation of context across different styles and then leveraged the refined alignment of latent representations for style transfer. Fu et al. [T6] utilize a multi-decoder with a style embedding model to learn separate content representations, where style representations are obtained via adversarial networks. Another is to generate stylized texts with back translation [17 [8], which produces pseudo parallel data to jointly train the forward and backward model. The third is based on the template, which tries to replace original stylistic words with the target stylistic words 19 14.
Stylized dialogue generation refers to generate response in the target style. Zhou et al. [20] propose an Emotional Chatting Machine (ECM) model for the emotional response. An intuitive solution generates response with a pre-trained style language model [21], which builds response generation model STYLEDGPT on top of pre-trained language model DialoGPT (22 and devise both word-level loss and sentence-level loss to fine-tune the DialoGPT towards the target style. Niu and Bansal (23] propose three weakly supervised models that can generate diverse, polite (or rude) dialogue responses without parallel data. Some work bridge conversation modeling and non-parallel style transfer by sharing a structured latent space (24. Recently, some work iteratively tuned to find the optimal mapping relation between conversational context and non-conversational utterances via iterative back translation
77 These works have achieved good performance, but we have achieved better diversity and efficiency through auxiliary domain.
Back translation is to train target-to-source seq2seq model for producing source sentences, and then construct pseudo parallel datasets. This approach is widely used in machine translation B 25 text
style transfer 26 27 and dialogue generation 7 [ Similar concurrent work is Dual learning 110.
which is first proposed for neural machine translation, involves two tasks with immediate reward.

---

fyg
Dg
Fxy!
fyx
Dy
Figure 1: The framework multi past dual learning. f denotes the encoder and decoder model constructed by GPT-2. x. and denote context, style So and style S respectively. D2 and D3 are discriminators.
Dual learning has proved to be very effective in neural machine translation and image translation 28 Our work also based on the idea of them. However, most of the recent works have focused on two domain mapping. We employ dual learning work three-domain related problem, then the contents of non-conversational text can be effectively utilized to enrich the dialogue generation.
Method
Let C, Sp. S1 denote the domains of context, response in style Su and response in style S₁ respectively. Our task is to learn a mapping. that can map a sequence from context domain to a stylized response domain S1. core challenge is that there no parallel data between C and S1, but there some related parallel corpora: (i) A dialogue dataset Daia = {(ri, W where I; E C and E So: {Vi, ū.)) where y: E So and y; Sj. The corpus sizes of the above two datasets are n and m. The sequences in style So are not the same between Dais and Dira. i.c., {yl (z, y) Daia n { (y. 5) Dua) 0. In addition to parallel data, there another unlabeled stylized corpus Daty (ūr Hi with size where Ut S1.
(ii) A style transfer dataset Dera
3.1 Multi-past dual learning framework
To fully utilize the parallel and unlabeled data, we propose multi pass dual learning (briefly, MPDL), which is illustrated in Figure [ There are three pairs of dual tasks involved in our framework: (i) Dialogue generation and the inversed task, fzy :C + Su. far So C: (ii) Style transfer between
S1 + Su: (iii) Stylized dialogue response generation and the inversed task, faj C + St. Jix SS C.
So and $1, fer : So + S1. föy
Our ultimate goal is to obtain the stylized dialogue response generation model saj. while the remaining tasks all auxiliary tasks. For multi-pass dual learning, given any ÿ Daty= could obtain reconstruction ÿ' fug (far (f (ū))) by sequentially applying föv- fyz and fzg. The reconstruction error between ÿ' and y determines the ability of model reconstruction. That is, want to maximize the logarithmic probability:
log P(ū' = ūlū: Jüy. Syz: Sağ).
(I)
Directly optimization the above probability is diffieult since there three models be updated. Consequently, we derive a relaxed version of the above probability. We have that:
log P(j' = Glū: Jog- Jyz Juū) = log Σ
Σ
P( y, o: Jüs Syx Szū)
DEC ME
(2)
= log
P(ū',,, D: Juy Syz fag) P(), ælūs fays Syn, fas).
xEC VESo
Considering that J' is only related to and Saz, we have that: P( y. x. it Jöys Syn San) P(ğ' xr, Jzg). Due to the concavity log(...) .). we can convert the Eqn. ( to:
log P( = 5lū: Juy. Jyz, Jan) > E Σ P(y, xlū; Jūn Swz: Say) log P(y' [r, Sag)
#EC S
(3)
Σ Σ
P(aly: P(sl) faw) log P( fxa),
FEC yEoo

---

where the last equation is obtained by the fact that x: is only related to and fyz, and y is only related to y and fay.
Instead of maximizing log P( أ| ..... directly, we maximize a lower bound as Eqn. 3 suggests. We use sampling-baser method to solve this problem: (i) Given D. we sample y accord- ing to P(-i; faw); (ii) Based on y, we sample a x' according to P(- fyz): (iii) We maximize log P(yl='s fae). We can use either beam search quences. On the other hand, we can pick an x from the set {æ|æ E Ddia sequentially use two sampling steps y P(-[2;; fay) and y' p Pl-ly: fug) and then maximize log P(zlğ', fee). ). In addi- tion to the loss for unlabeled data, we propose loss functions on the parallel data. Therefore, there are four loss functions total:
random sampling to get the intermediate se-
Ldia E
(log Plyla; fay) log P(xly; fy=)),
(2,g)~Dam
Lim E (v.
(log P(ū Jwn) + log P(ulūs fāy)),
(4)
Ldual,1 (0) log P0] faū); 2 = log P(z|ū': fgæ),
where x' and y/ are generated by the MPDL model as the above steps.
3.2 Network architecture
All the six models f's are built on Transformer-based architecture [29] The encoder and decoder of f's are initialized using pre-trained DialoGPT weights trained on conversation corpus of Reddit community [22]. To reduce the memory cost, (i) the parameters of all forward models, including say- fuy and sag are shared; (ii) the parameters of all backward models, including fur- Jōy and f are shared; (iii) the both encoder and decoder (excluding the encoder-decoder attention modules) share the same parameters too.
Taking the forward models as an example, to distinguish the models from each other, we introduce several domain embeddings T {Ec, Ex E,y} J for context domatin (i.e., C). style Sy (i.e., Sn) and style S₁ (i.e., S₁) domains respectively. Given an input and the domain T. the encoder of the forward model works as follows:
H ency + W(x)),
(5)
where Wx:) is the word embedding of input sequence x, and enc, is encoder module of the forward model. Before feeding the representation /1 (x) into the decoder, according to the target domain (So or 81), Hx)x) is concatenated with another style embedding:
(x()) [E,, H
(6)
where & E {C, }. In this way, the hidden representations are shifted according to the target domains. We can obtain H (x) for dialogue response generation, H (z) for stylized dialogue generation and S (y) for text style transfer. Similarly, for the backward model (marked by subscipt o), we can obtain the features as follows: H[(()) for inverse dialogue response generation, H (ū) for inverse stylized dialogue generation and 11,5 (ī) for inverse text style transfer. For ease of reference, we set that all the H's are d-dimensional vectors.
3.3 Quality discriminators
Inpul p:1 believe that Brazilin are very sexy.
When analyzing the intermediate results, we find that previous work simply which use iteratively back translation cannot obtain proper context sequences because of the poor data quality. In many scenarios, examples of the synthetic con- text include:
BBrckoodd
Outputx' : I'm so excited for Brazil to come out. [D. - 0.45
Input
x: D: you to go work right away?
Foowword
Outputy I'm gomntt hevearest.
[
0.13
I don't know
I'm so happy ...
The example of MPDL model discrimi- nators in context and formal style.
Figure 2:
I 'm not sure if I should

---

based on which the model cannot generate meaningful response. The inadequate text modeling leads to a peak reduction model performance. As shown in Fjgure To evaluate the data quality of the intermediate data (i.e., the pseudo contexts x and responses in style S1), we employ two diseriminators D₂ IRd + [0. 1] and D3 : TR + [0, 1], which are used to classify whether an input text is real one (e.g., written by people) or synthetic one (e.g., generated by model).
The discriminators will give weight the pseudo sequence generated by model, which can be used to judge the degree to which the model parameters can be updated. D2 and Dy are implemented as 3-layer feed-fon forward networks with LeakyReLU activation function, whose input are H {x) and (t) obtained according to Eqn.(S). The loss function for discriminators is as follows:
dis log Dr (1H + ( - D2 (Ho(()))
dis log D3 (()) + log(1 (y(HH()))), Diis (z.y))-aan:(0)-D..
E
- -feiis + llle)
where Hy(y) is used to generate pseudo context z' for inverse dialogue response generation task, (y) is used to generate pseudo response j' for text style transfer task. Note that at inference time, the discriminator is not used, which means that MPDL will not introduce additional inference cost. The output of the discriminators could regarded as confidence score for the sample, which is used further to improve the dual loss in the previous section. Mathematically,
Cduul,1
E -
DD log P(i)='; fag).
Jr-Day
(8)
E D-Daii
(H)(()) log P fge),
In training stage, loss function of MPDL model consists of the standard maximum log likelihood loss and the multi-pass dual learning loss. The mixed learning objective function is defined as follows:
L = Ldia + Fira + Ldual,1 + 2
4 Experiment
4.1 Experiment setup
Datasets. We train and evaluate our method on two benchmarks, TCFC 30 whose responses are of formal style, and Shakespearean Dialogue Generation Corpus (SDGC whose responses are of Shakespearean style. TCFC is constructed by crawling text from Twitter. Following [9), we sample 217.2k (context, informal response) dialogue pairs as Daia and 500k: formal texts as Daty from the TCFC dataset. The test set contains 978 manually-crafted dialogue pairs. For the Dera. sample 105.5k informal-formal parallel sentences from GYAFC [31] SDGC is constructed by Twitter, Yahoo Answers and Shakespeare [I]] Shakespeare is collection of Shakespearean plays with aligned modern version on supervised style transfer task. For the test set, we manually craft 1,000 dialogue pairs and we will release them to the community. The Daia is the same as that for TCFC. We extract 18.4k (modern English, Shakespearean English) pairs from [T] as Dera- All Shakespearean English used Day- The statistic of datasets is shown in Table 5 of Appendix.
Baselines. We compare our proposed model with six baselines, including standard Transformer- based dialogue model, multi-task learning method, language model method, hidden state fusion method, pipeline method and the latest end-to-end method. Specifically, (1) S2S: a Transformer-based model trained on Daia only, where the weights are initialized by DialoGPT (22). (2) MTask: Luan et al. (32]
propose a vanilla multi-task learning model, which is trained with both Daia and Daty. (3) SLM: Niu
and Bansal (23) propose the fusion model by merging the decoder of a sequence-to-sequeee seee--eece-ee model trained on Daia and a language model trained on Daty. (4) SFusion: a multi-task learning model proposed by [24], which builds structured latent space bridge the conversation modeling and style transfer through jointly learned with Daia and Daty- (5) S2S+BT: the responses with style So of pipeline approach is produced by the S2S baseline, and then text style transfer model proposed by B3 is used to get a response with style S1. (6) SRJT.: a Style Routing approach by employing an
"Our code and dataset publicly available http:://gittbbcccmm/CCocleaaaaau
5

---

Table I:
Automatic and manual evaluation results on TCFC dataset. The stylized accuracy of the BERT and SVM classifiers on the holdout test is 93.98% and 89.57% respectively.
Automatic Metrics
Manual Metries
Model
BLEU-1
BLEU-2
Distinct
BERT
SVM
Fluency
Relevance
Style-Con.
The target style is formal response (i.e., style Si)
MTask
6.35
0.50
29.3
37.3
0.78
0.33
0.58
SLM
12.6
0.99
42.5
85.6
87.2
0.83
0,45
0.87
SFusion
5.51
0.28
61.0
21.9
39.0
0.77
0.32
0.57
S2S+BT
12.1
1.25
42.0
86.3
86.8
0.79
0.31
0.65
SRIT
15.1
1.71
43.4
0.85
0.55
0.89
MPDL
16.5
2.07
51.3
98.6
97.1
0.88
0.64
0.91
Human
62.7
89.6
85.8
0.88
0.65
0.90
The target style is informal response (i.e., style So)
S2S
6,92
0.61
54.8
70.1
60.9
0.75
0.46
0.66
SFusion
4.61
0.22
62.8
70.3
61.1
0.66
0.34
0.78
SRJT
6.96
0.67
49.4
69.4
59.2
0.81
0.57
0.73
MPDL
7.12
0.69
49.5
70.3
60.7
0.83
0.55
0.75
Human
72.6
72.0
72.1
0.79
0.56
0.78
inverse dialogue model with Joint Training process proposed by [9] in which it is based back translation. More baselines and evaluating metrics [34 are available in Appendix.
Training details. The DialoGPT (small-size 117M) 22 with 12 layers and 768 hidden sizes is used to initialize the multi-pass dual learning modules, which is released by Transformer Our model has 50,257 vocabulary size and was trained on Nvidia GTXI080T, machines with batch size 10. The maximum input length and maximum output length are set as 45. We choose the Adam optimizer. The learning rate of generators is 2.25 10 10-4 with warm-up steps x 10 while that for discriminators is x 10-4. We use the grid search to tune the hyper-parameters.
4.2 Main results
Automatic evaluation. In this work, we evaluate the generation quality of both style S₁ and style So. following the evaluation metries in [9]. Content relevance, diversity and style consistency are primary consideration. For the content relevance, choose the BLEU score 35] between the generated response and the ground-truth one. We focus on the unigram precision (BLEU-1) and bi-gram precision (BLEU-2). For diversity, choose the distinct between the number of unique n-grams and the number of all n n-grams in the generated responses. In this work, is 2. For style consistency, we an SVM, to evaluate the style consistency, both of which are trained on the GYAFC and Shakespeare datasets. The accuracy of the BERT and SVM classifiers the holdout test set 93.98% and 89.57% respectively for the TCFC experiments, and 95.79% and 82.28% respectively for the SDGC dataset. Each setting is independently run five times and we report the mean value of each metric.
[36, which is the ratio
text style classifiers, BERT-based model [37] and
The automatic results are summarized in the left part of Table [ (for TCFC) and Table 2 (for SDGC). In each table, the generation quality of both styles S1 and So evaluated. Overall, achieves the highest BLEU-1 and BLEU-2 scores on the two tasks, which shows the superiority of method. For formal response generation task the TCFC dataset, compared with the previous state-of-the-art algorithm SRJT, MPDL improves the BLEU-1, BLEU-2 and distinct scores by 1.4, 0.36 and 7.9. The reasons two-fold: (1) multi-pass dual learning framework significantly improves the learning ability since bridges the relation between three domains; (2) the discriminator controls the quality pseudo data. SFusion gets a better distinct score than MPDL, but has worse BLEU-1 and BLEU-2 scores. This is because the discriminator tends to maintain the consistency of the generated responses and the target style, at the cost of reducing diversity. The BERT and SVM scores of our method are also notable, which indicates that our model can generate target style response
method

---

Table 2: Automatic and manual evaluation results on SDGC dataset. The stylized accuracy of the BERT and SVM classifiers on the holdout test is 95.79% and 82.28% respectively.
Automatic Metrics
Manual Metries
Model
BLEU-1
BLEU-2
Distinct
BERT
SVM
Fluency
Relevance
Style-Con.
The target style is Shakespearean response (i.e, style Si)
MTask
5.64
0.15
30.2
18.4
20.7
0.45
0.23
0.19
SFusion
6.76
0.21
41.6
20.1
16.0
0.50
0.27
0.21
S2S+BT
9.68
0.38
37.4
84.7
73.0
0.64
0.33
0.70
SRJT
12.2
0.89
43.1
65.0
53.4
0.67
0.59
0.71
MPDL
13.9
1.43
53.2
64.3
54.1
0.79
0.71
0.80
Human
81.2
92.9
66.0
0.84
0.75
0.83
The target style is modern response (i.e., style So)
S2S
9.33
0.90
43.2
93.1
94.3
0.64
0.31
0.55
SFusion
5.71
0.12
45.7
91.5
93.9
0.45
0.41
0.63
SRIT
8.90
0.63
40.5
96.9
95.4
0.71
0.54
0.64
MPDL
10.4
1.19
46.7
97.8
96.7
0.73
0.57
0.70
Human
79.9
92.9
92.0
0.80
0.69
0.77
better with the auxiliary text transfer task. On informal response generation, by using MPDL, the performances are also improved, where we also achieved the best BLEU-1 and BLEU-2 scores. We observe that compared the formal case, the BERT score similar to previous best result, and the SVM score is lower, which is caused by the lack of discriminator for the informal domain. We will explore the future.
For the SDGC dataset, MPDL model also shows great advantages in most metrics. We observed that S2S+BT achieved the highest BERT and SVM scores Shakespearean style. This shows that the unsupervised pipeline method does help to migrate text to the target style, but at the same time, it will lose the conversation information since the style transfer model does not take the context into consideration. MPDL achieves the second achievement in the style classifier while other metrics are improved significantly. SFusion has much higher Distinct score for the TCFC dataset than it does for SDGC. Because SFusion learns the patterns from the data of the style S domain, and more data helpful to increase the diversity. TCFC contains 500 non-parallel data style S₁ domain while SDGC has only 18.4k. Therefore, MPDL has better distinct score on the SDGC dataset. We further investigate the generalization ability of MPDI. by evaluating modern responses, and find that the model generalizes well not only over different styles but also different datasets.
Manual evaluation. Complementary to automatic evaluation, also carry out the manual evalua- tion. We focus on three aspects: (1) fluency which indicates how smooth the sentence is; (2) relevance, which shows the content of the response consistent with the context: (3) style consistency, which evaluates how the response style matches the target domain. For a fair comparison, all generated responses and human responses are re-capitalized, de-tokenized and shuffled order. We sample 100 instances from the text set and employ three graduated annotators rate each context-response pair. The score range of sample is from I, where 1 indicates the best score. We provide the responses (generated by models and ground-truth by humans) for each example to our annotators. Every sample can obtain three human judgments and the final scores are averaged across different judges.
The manual evaluation scores are summarized in the right part of Table and Table where we can see that MPDL significantly outperforms previous baselines. Specifically, for formal response generation on TCFC, the scores of our method are close to the human score. On informal text generation, SFusion and SRJT achieved slightly better results on style consistency and relevance than our method. On SDGC, our method achieves the best human evaluation scores, which in general shows the effectiveness our method. Compared to SFusion and SRJT in style So, the relevance and style consistency scores of MPDL decrease on TCFC but increase on SDGC. The reason is that we use more unlabeled target domain data for TCFC than SDGC (500k v.s. 18. 18.4k). Considering that share the parameters of all tasks, the model for SDGC is less specialized to the target domain

---

Table 3: Automatic evaluation average results of ablation models for responses with style So and S on TCFC dataset.
Model
BLEU-1
BLEU-2
Distinet
BERT
SVM
Ours
12.0
1.38
50.1
84.4
79.2
w/o Discriminator
10.9
1.33
52.8
83.1
79.4
w/o Multi-Pas
9.89
0.89
45.1
82.3
75.4
w/o DialoGPT
10.3
0.87
33.6
84.0
79.7
1.0
Context Porma Informal
1.6
Contoat Formal
nnternll
2.4
0.4
2.0
(b) With discriminators. Figure 3: Visualizing the representations of the context, informal and formal text from TCFC dataset.
(a) Without discriminators.
and maintains the generalization ability across different domains. More details and the variance are available in the Appendix.
4.3 Discussions
Ablation studies. To understand the roles of discriminator, multi-pass dual learning (specifically, Lira in Eqn.(4), and pre-trained model learning to generate stylized responses, we conduct ablation study by removing each component from our framework, and the results are in Table 3 We have the following observations: (1) Without the discriminator, all pseudo responses and context are equally treated. Both the BLEU and BERT scores drop, but the distinct scores increased. This is consistent with our previous explanation, since there is no discriminator to enforce that the styles of generated sequences should match the target domain. (2) Without the Lura in Eqn. the results of all metrics significantly drop due to the lack closed loop: (3) The pre-training is also helpful to the whole scheme, without which all metries drop. The ablation study shows the importance of each component in our model. Moreover, we also provide some examples in Table 7 of Appendix to show how the discriminator weights the sequences. Specifically, in dialogue generation of Shakespearean style, the discriminator assigns a score 0.99 to the sentence "Tis a brave lady", where "Tis" frequently appeared in Shakespearean play. In contrast, the response "T m sure what do with it", which is quite general, obtains 0.14 weight.
Latent space visualization. To further study the effect of the discriminators, visualize the hidden state on TCFC dataset, including the context, informal responses and formal responses. We use the output of the last layer in decoder as the representation of the sequences and utilize t-SNE algorithm (8 for visualization. We compare the models trained without the discriminator and with the discriminator. The results in Figure 3] the remaining visualization results SDGC dataset and more details are included in the Appendix. As shown in Figure 3(),, after using discriminators, the three types of sequences are properly separated. Without the discriminators (see Figure a)), can see that the boundary of context and the informal text is not quite clear, which verifies that the quality the synthetic data low, and we need module re-weight them.
Ja)).

---

Table 4: The example responses produced by MPDL model and the baselines on TCFC dataset.
Context
wedding ring? But why would you throw it away?
The target style is formal response (i.e., style S1)
Human
Are you sure? You must have surely least pawned it. wish I had a chance to see that.
MTask
SFusion
They already let me set up! :)
SRJT
Yes, they should have given it to you. It is wedding ring.
Ours
I have never thrown away wedding ring before.
The target style informal response (i.e., style So)
Human S2S
Right? Surely you would at least have pawned it.
I don't 't want to talk about it.
SFusion
I've emailed the proof when picks in. I'll send it shortly.
SRJT
I'm sure you didn't throw it.
Ours
You know you wouldn't I throw the engagement ring away.
Parallel data size. To show the effect of the parallel conversation data Diia size, we re-train MPDL, MPDL w/o discriminator and the SRJT baseline on TCFC dataset. The BLEU-1 shown in Figure 4 The performances of all three meth- ods increase w.r.t. to the labeled data size. In com- parison, MPDL is less sensitive to the amount of labeled data compared to SRJT, and achieves the best results in BLEU-1, BLEU-2 and Dis- tinct. Moreover, the MPDL with 50k labeled dialogue conversation data achieves comparable (even slightly better) results to the SRJT with 200k labeled data, demonstrating that our method can effectively utilize data. Experiment par- allel style transfer data Dem with different sizes are provided in the Appendix.
BLRU-2
15.0
14.5
MPOL we
SRJT Model
50K
100K
150K
200K
Sumberr Paallll
Figure 4: The impact of parallel conversation data quantity on model performance of BLEU-1.
Case study. We show some examples of the stylized response in Table and make comparison of the previous algorithms. For formal English generation (no parallel data between context and the formal response), the MTask model, whose weights are not initialized from the pre- mre-trained model, tends to generate general and bland responses because of the lack of prior knowledge in the dialogue model. SFusion generates the wrong style, where there is an emoji ":)" at the end, since there is no discriminator. For SRJT, although it generates formal sentence, the response is not consistent the context, because would like to the response to "why would throw it away". Overall, our model performance the best, capturing more input information and the target style. For informal English generation where parallel data is available, by using our training scheme, we also observe some improvement over the baselines. MPDL model reduces the use of words like "I don't or "T m sure " and tends to generate more meaningful words to make the response more interesting.
Conclusions and future work
In this paper, we propose novel stylized dialogue generation model with multi-pass dual learning (MPDL), that can effectively leverage unlabeled data. Compared with standard dual learning, we employ discriminator to ensure that the responses move towards the target style while the content relevant. Experiments on two stylized response generation benchmark datasets demonstrate that our method has significant advantages over the recent works, and our model achieves state-of-the of-the--rt results in the formality and Shakespearean style response generation. For future work, first we can extend the stylized dialogue from single style to multiple styles without too much additional cost; second, we will increase the robustness of our model; third, we will further explore how to effectively more unlabeled data.

---

Broader Impact
Generating stylized responses is an important step towards real artificial intelligence. can be pre- dicted that its achievements can significantly improve the experience of human- human-computer interaction, and can be applied in various fields, such as education, finance, and so on. This work has the potential positive impact on an intelligent and engaging dialogue system. At the same time, this work may have some negative consequences in social interaction. Besides, if the pre-training model covered by malicious and vulgar information, it will bring bad information feedback to users. Therefore, we should be cautious of these advantages and disadvantages.
Acknowledgments
This work was supported by the National Key Research and Development Program of China (No. 2020AAA0106000) and NSFC Grant (No. 62122089 and No. 61876196).
References
[1] Alan Ritter, Colin Cherry, and W. Dolan. Data-driven response generation in social media. In EMNLP, 2011.
[2] L. Shang, Z. Lu, and Hang Li. Neural responding machine for short-text conversation. ArXiv, 023444 2015.
[3] Yansen Wang, C. Liu, Minlie Huang, and L. Nie. Learning to ask questions in open- open-domain conversational systems with typed decoders. In ACL, 2018.
[4] Xueliang Zhao, Wei Wu, Chongyang Tao, Can Xu, Dongyan Zhao, and Rui Yan. Low-resource knowledge-grounded dialogue generation. abs/2002. 103448 2020.
[5] lya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence sequence learning with neural networks. In NIPS, 2014.
[6] Kyunghyun Cho, B. V. Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- decoder for statistical machine translation. ArXiv, abs/1406. 1078, 2014.
[7] Hui Su, Xiaoyu Shen, Sanqiang Zhao, Xiao Zhou, Pengwei Hu, Randy Zhong, Cheng Niu, and Jie Zhou. Diversifying dialogue generation with non-conversational text. In ACL., 2020.
[8] Rico Sennrich, B. Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In ACL, 2016.
[9] Yinhe Zheng. Zikai Chen, Rongsheng Zhang, Shilei Huang, Xiao-Xi Mao, and Minlie Huang. Stylized dialogue response generation using stylized unpaired texts. In AAAI, 2021.
[10] Di He, Yingce Xia, Tao Qin, L. Wang, N. Yu, T. Liu, and W. Ma. Dual learning for machine translation. In NIPS, 2016.
[11] Wei Xu, Alan Ritter, W. Dolan, R. Grishman, and Colin Cherry. Paraphrasing for style. In COLING, 2012.
[12] Harsh Jhamtani, Varun Gangal, E. Hovy, and Eric Nyberg. Shakespearizing modern language using copy-enriched sequence-to-sequees models. ArXiv, abs/1707.01161. 2017.
[13] Yi Zhang, Jingjing Xu, Pengeheng Yang, and X. Sun. Learning sentiment memories for sentiment modification without parallel data. In EMNLP, 2018.
[14] Chen Wu, Xuancheng Ren, Fuli Luo, and Xu Sun. A hierarchical reinforced sequence operation method for unsupervised text style transfer. In ACL, 2019.
[15] Tianxiao Shen, Tao Lei, R. Barzilay, and T. Jaakkola. Style transfer from non-parallel text by cross-alignment. ArXiv, 0bs/1705500955, 2017.
10

---

[16] Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and R. Yan. Style transfer in text: Exploration and evaluation. In AAAI, 2018.
[17]
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, M. Zhou, and E. Chen. Style transfer as unsupervised machine translation. ArXiv, abs/1 808.07894, 2018.
[18] Guillaume Lample, Sandeep Subramanian, Eric Michael Smith, Ludovic Denoyer, Mare' Aurelio Ranzato, and Y-Lan Boureau, Multiple-attribute text rewriting. In ICLR, 2019.
[19] Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: A simple approach to sentiment and style transfer. In NAACL-IILT, 2018.
[20] Hao Zhou, Minlie Huang. T. Zhang, Xiaoyan Zhu, and Bing Liu. Emotional chatting machine: Emotional conversation generation with internal and external memory. In AAAI, 2018.
[21] Ze Yang, Wei Wu, Can Xu, Xinnian Liang, Jiaqi Bai, Liran Wang, Wei Wang, and Z. Li. Styledgpt: Stylized response generation with pre-trained language models. In EMNLP, 2020.
[22] Yizhe Zhang, S. Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and W. Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. In ACL, 2020.
[23] Tong Niu and Mohit Bansal. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics, 6:373-389, 2018.
[24] Xiang Gao, Yizhe Zhang, Sungjin Lee, Michel Galley, Chris Brockett, Jianfeng Gao, and W. Dolan. Structuring latent spaces for stylized response generation. In EMNLP/IJCNLP. 2019.
[25] M. Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In ICLR, 2018.
[26] Sandeep Subramanian, Guillaume Lample, Erie Michael Smith, Ludovic Denoyer, Mare' Aurelio Ranzato, and Y-Lan Boureau. Multiple-atrribtte text style transfer. ArXiv, abs/18 0bs//881,0055, 1,00552, 2018.
[27] Ning Dai, Jianze Liang, Xipeng Qiu, and X. Huang. Style transformer: Unpaired text style transfer without disentangled latent representation. In ACL, 2019.
[28] Jianxin Lin, Yingce Xia, Yijun Wang, T. Qin, and Zhibo Chen. Image-to-image translation with multi-path consistency regularization. In IJCAI, 2019.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L. Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[30] Y. Wu, Yunli Wang, and Shujie Liu. A dataset for low-resource stylized sequence- generation. In AAAI, 2020.
[31] Sudha Rao and J. Tetreault. Dear sir or madam, may introduce the yafc corpus: Corpus, benchmarks and metries for formality style transfer. ArXiv, abs/1803 btss/8833.0635, 2018.
[32] Yi Luan, Chris Brockett, W. Dolan, Jianfeng Gao, and Michel Galley. Multi-task learning for speaker-role adaptation in neural conversation models. In IJCNLP, 2017.
[33] Junxian He, X. Wang, Graham Neubig, and Taylor Berg- Kirkpatrick. A probabilistic formulation of unsupervised text style transfer. ArXiv, abs/2002.03912 2020.
[34] Kalpesh Krishna, John Wieting, and Mohit lyyer. Reformulating unsupervised style transfer as paraphrase generation. In EMNLP, pages 737-762, 2020.
[35] Kishore Papineni, S. Roukos, T. Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
[36] Li, Michel Galley, Chris Brockett, Jianfeng Gao, and W. Dolan. A diversity-promoting objective function for neural conversation models. ArXiy, abs/1510.03005 2016.
11

---

[37] J. Devlin, Ming- Wei Chang, Kenton Lee. and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, In NAACL-HLT. 2019.
[38] L. V. D. Maaten and Geoffrey E. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2608 2008
12


Reviews:
[{"review_id": "hvp85AgF7a", "reviewer": "Reviewer_vM8L", "summary": "This paper studies the stylized dialog generation problem, in which the problem setup contains contexts C, responses with the original style S0, and responses with the desired style S1. The contributions of this paper includes:\n- A novel multiple-pass dual learning framework that explores the interaction between the pairwise relationship among C, S0 and S1.\n- The authors propose an additional discrimination to determine the text quality generated by the backward models, which has been shown to be helpful for the task.\n- Evaluations are compared to state-of-the-arts on two large text style transfer datasets, and show good amounts of improvements on both automatic and manual metrics.\n- Ablation studies further demonstrated the necessity of each component, especially convinced the capacity of discriminator (which is pretty clear from Figure 3). \n- The authors also release their code and data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Originality: There have been many work exploring the stylized dialog generation problem without parallel data. But it's new for this paper to use dual learning to explore the interactions between the pairwise relationship between context C, source style S0 and target style S1. The proposed discriminator component is also intuitive and effective (verified by the experiments), though I am wondering if pretraining such discriminator component might be more effective than optimizing it jointly with the dialog generation loss. Overall, the method is novel and technically sound.\n- Quality & Clarity: In general, the paper's writing is clear and easy to follow. There are a few questions to me still: 1) In Equation 9, the optimization loss is defined as the sum of losses from the four components, which gave me the impression that the sum of losses is jointly optimized. However, from Algorithm 1 in the Appendix section, the different components are optimized sequentially. I would love to see the confusion could be resolved. 2) In Table 2, the style consistency scores by auto metrics (e.g., BERT and SVM) don't seem correlate well with the manual judgements, when you compare MPDL with S2S+BT. Is that because the small samples in manual testing, or inaccurate consistency prediction by BERT or SVM?\n- Significance: Evaluations are compared to state-of-the-arts on two large text style transfer datasets, and demonstrate significant improvements over strong baselines on a subset of metrics. Human evaluation confirmed that the model generates more style-constrained responses. Ablation studies also provide insights about why this method work. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are a few flaws in the evaluation part:\n- Instead of reporting BLEU-1 and BLEU-2, it's better to report BLEU-4, which is much widely adopted. \n- The author could consider to increase the number of samples for human eval to increase the evaluation significance. Also, it would be good to disclose the variance of manual's judgements in each metric.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "T2ikjIJ68m8", "reviewer": "Reviewer_FhxC", "summary": "This paper focuses on proposing multi-pass dual learning (MPDL) to build mappings among dialogue context and responses in two styles. The proposed model is reasonable and is the extension of the classical dual learning method, which usually maps between two domains. The idea is reasonable and novel, and the experiments show the improved performance in terms of both automatic and human evaluation for both datasets.\n\nA main concern is about the paired style transfer data $D_{tra}$, which limits the flexibility of usage. Also, an intuitive baseline may be needed for fair comparison. The further suggestion is to remove the \"paired style transfer data\" constraint, considering that the proposed MPDL may still work.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes multi-pass dual learning (MPDL) to build mappings among three domains (dialogue context $C$, response of style $S_0$, response of style $S_1$) through duality. The target scenario is lack of $(C, S_1)$ parallel data for training, but there is unlabelled data with the style $S_1$. \n\nIn the proposed MPDL, the main idea is to utilize the reconstruction property of the response with the style $S_1$, which is mapped to the response with the style $S_0$ via a style transfer model, then mapped to the context via an inverse dialogue generation model, and then finally mapped to the original one (the response with the style $S_0$) via a dialogue generation model. In addition to minimizing the reconstruction error through multiple passes, two discriminators are introduced to ensure the good quality of pseudo contexts and responses in style $S_1$. By jointly learning the mixed learning objective, the proposed method outperforms multiple baselines for two benchmark datasets and most metrics in the conducted experiments.\n\nThe idea about using multi-pass dual learning is reasonable and original, where the unlabelled data in style $S_1$ can be effectively utilized and the mappings among three domains can be obtained. However, there are some concerns to be addressed.\n\n- In the experiments, three datasets are used: $D_{dia}$: $(C, S_0)$ paired data, $D_{tra}$: $(S_0, S_1)$ paired data, and $D_{sty}$: unlabelled $S_1$ data. An intuitive baseline can be a pipelined method, where the first model is to generate the response in style $S_0$ and the second model is to transfer it into the response in style $S_1$ in a supervised manner. However, this baseline is not compared in the experiments. Although the unlabelled data $D_{sty}$ is not used, it still needs to be shown in the table so that we can see how much improvement the unlabeled data can bring in. (The S2S+BT baseline uses \"unsupervised\" text style transfer model [33] instead of \"supervised\" learning, where the performance is expected worse than the supervised one.)\n\n- In the proposed MPDL framework, it may be possible to use the unpaired $D_{tra}$ for learning $f_{\\tilde{y}y}$ and $f_{y\\tilde{y}}$ and add a discriminator to check the quality of $y$. Hence, the paired style transfer data may not be required, making the target scenarios more practical. Have the authors tried this setting? (Comparing the results of this setting with S2S+BT is fair.)\n\n- In the discussions, the authors analyze the sensitivity of the size of the parallel data $D_{dial}$. The results are good, but it is also useful to show the impact of the size of $D_{tra}$. Because the main idea is to better utilize unlabelled $S_1$, so the paired style transfer data $D_{tra}$ may not be available/enough. Showing the results with different sizes of $D_{tra}$ may be more informative from my perspective.\n\n- Are the numbers shown in Table 3 for the TCFC dev set? Because the numbers are quite different from ones in Table 1, can they be compared directly?\n\n- For ablation study, removing discriminators shows the degraded performance in Table 3. Can the authors show the results of removing only one discriminator (either $D_x$ or $D_\\tilde{y}$), so that we can see which one contributes more?\n\n- The diversity (Distinct) is sometimes lower in the proposed method. Is it because $f_{yx}$ and $f_{x\\tilde{y}}$ are not 1-1 mappings and the reconstruction $\\tilde{y}'=f_{x\\tilde{y}}(f_{yx}(f_{\\tilde{y}y}(\\tilde{y})))$ forces the mappings to be less diverse? Do the authors have any explanation about this observation?\n\nIn sum, the proposed MPDL is reasonable and seems useful in the conducted experiments. The paper is clear. The experiments can be improved in order to better convince the readers about the model's effectiveness. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "QaXWYZrjQYh", "reviewer": "Reviewer_tHTV", "summary": "This paper studies the task of stylized dialogue generation, where the goal is to generate dialogue responses in a particular target style (formal / informal / Shakespeare-like). The authors operate in a setting where they do not have access to any dialogue training data in the target style, but instead have access to dialogue data in a complementary style, along with parallel data to transfer between the two styles. For instance, the authors study dialogue generation in formal style, and assume access to dialogue data in informal style, parallel data between formal / informal sentences, and a corpus of unpaired formal sentences.\n\nThe authors model this task as a three-way dual learning problem, a technique which has proven useful for machine translation [1]. They jointly training their model to perform 1) style transfer in both directions; 2) dialogue / inverse-dialogue generation to the complementary style; 3) dialogue / inverse-dialogue generation to the target style. 1 / 2 act as auxiliary tasks to aid (3) via shared models. Since no parallel data is available for (3), the authors perform sampling (using current version of model for 1 / 2) to create pseudo parallel data for (3). Since sampling can be noisy, the authors additionally use discriminators to weight samples, which judge whether the samples are real/fake (akin to adversarial training).\n\nThe authors evaluate their approach on formal dialogue generation and Shakespearan dialogue generation. They compare their approach to 5-6 baselines, along with human upperbounds. Both automatic and human evaluation is conducted. The authors additionally conduct ablation studies, analyze their discriminator, and show the effect of the size of parallel data.\n\n[1] - https://arxiv.org/abs/1611.00179", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n1. Style-controlled text generation is an important problem in natural language generation technology, and is far from solved. This has several important applications in chatbots and coversation agents.\n\n2. The authors use several interesting ideas in this paper, which were new to me. The idea to use dual learning to boost style-controlled generation as well as discriminators for weighing poor synthetic data were interesting and could be more widely applicable in style transfer literature.\n\n3. The authors approach beats several strong baselines on automatic & human evaluation. Several good analysis experiments have been conducted in the paper including ablations, analysis of the effect of discriminators and the effect of the size of available parallel data.\n\n**Weaknesses**\n\n1. My biggest issue with the paper is the usage of parallel data. Most style transfer literature [1, 2, 3, 4] operates in an unsupervised / unpaired setting (with access to D_sty only), and recent work [5] is assuming access to only 5-10 exemplars from D_sty. Access to parallel data is a strong assumption --- besides formality, shakespeare, sentiment transfer, almost no parallel resources exist. Moreover, all these datasets are in English, and no parallel style transfer resources exist for other languages. Even assuming access to dialogue data in a single style (`D_dia`) is strong assumption in my opinion. Indeed, the authors show that their method's performance reduces significantly if you don't have access to this parallel data (Table 3 w/o Multi-Pass, Figure 4), which reduces the applicability of this work.\n\n2. The evaluation in this paper could be improved. Here are some specific thoughts ---\n\n* Since there are three orthogonal metrics [7] for both human / automatic evaluation (style accuracy, semantic correctness, fluency), it's hard to compare between systems when only one metric improves (and the other does not), for instance SFusion vs MPDL in Table 1; S2S+BT vs MPDL in Table 2; experiments in Table 3. I suggest using the aggregation strategy from [2] to combine the metrics into a single score, making it easy to compare between systems.\n\n* The BLEU scores are quite low overall for all systems --- which is not surprising given the large space of possible outputs which makes evaluation challenging and easy to game [8]. I suggest using other automatic metrics too, such as those based on natural language inference [9] or the recent BEGIN benchmark [10].\n\n* Since automatic metrics can be flawed, human evaluation is critical. While the authors conduct human evaluation, more details will be helpful. Specifically, i) Were the annotators familiar with the research goals? (ideally they should be unfamiliar to avoid bias) ii) Were they shown outputs from different systems in a blind manner (not aware which system produced which output) & shuffled order? (consecutive samples from different systems) iii) What's the agreement between annotators & variance / error-bars of their annotation? Are the gains statistically significant? iv) How much were the annotators paid for this experiment?\n\n* More qualitative outputs from the system would be very helpful. Currently only two outputs are shown in Table 4. In both examples it's not really clear to me if the outputs are really formal / informal, which makes me doubt the quantiative evaluation in this work (In second example, \"wouldn't\" has an apostrophe which shouldn't be there in an informal style generation).\n\n**Minor**: The notation in Section 3 is confusing, primarily due to lots of inverted commas and tildes. I suggest avoiding these and using subscripts with short natural language identifiers (like you did for `D_dia / D_tra / D_sty`). Also it will be good to mention `BERT / SVM` in Table 1 refer to classification accuracy (my initially incorrect thought reading this was a reference to BERTScore [6], which confused me).\n\n**Overall**\n\nIn terms of the NeurIPS reviewer guidelines, I think the paper has decent originality (strength #2), average quality (strength #3, weakness #2), average clarity (minor #1), and average significance (strength #1, weakness #1). Overall, I liked the ideas in the paper, but I think weakness #1 is the biggest deal-breaker for me which will prevent me from giving a higher score. I encourage the authors to improve this in the next version of the paper.\n\n[1] - https://arxiv.org/abs/2011.00416  \n[2] - https://arxiv.org/abs/2010.05700  \n[3] - https://arxiv.org/abs/1804.09000  \n[4] - https://arxiv.org/abs/1811.00552  \n[5] - https://arxiv.org/abs/2010.03802  \n[6] - https://arxiv.org/abs/1904.09675  \n[7] - https://arxiv.org/abs/1910.03747  \n[8] - https://arxiv.org/abs/2103.06332  \n[9] - https://arxiv.org/abs/1904.03371  \n[10] - https://arxiv.org/abs/2105.00071\n\n-------\n\n**After author response**: Thanks for the excellent response. I've raised my score to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5 hours"}, "extra_text": {}}, {"review_id": "-0wmPlMB0Wt", "reviewer": "Reviewer_nC2K", "summary": "The authors propose a method for learning stylized dialogue response generation: given a context, generate a response conditioned on a particular style domain (e.g. modern vs. shakespearean english). The method, Multi-Pass Dual Learning (MPDL), leverages large corpora of non-parallel data in tandem. Specifically, the framework involves several forward and backward models between the context and two styles (one which has labeled data from the context to the style, and one that does not); the models produce intermediate data that is used to train the desired style generation model. To make sure that the intermediate data is being used appropriately, the authors introduce two discriminators that evaluate the quality of the data and determine whether it falls in the respective style/domain. The scores of these discriminators weight the importance of the loss computed on that data to ensure that higher-quality pseudo tuples are more impactful in the learning process. The authors evaluate the framework on two style-transfer datasets, and show that the method outperforms 5 baselines, achieving state-of-the-art results in automated metrics and outperforming nearly all models in human evaluations. A case study and ablations are presented as well that highlight and support the authors’ design decisions.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*: The authors build upon the concept of dual learning and apply it to dialogue response generation; the framework is novel in the sense that the dual learning is required for three domain mappings, and is additionally novel in its use of discriminators to measure the quality of the intermediate data; related work is generally adequately cited, though there are a few cites that are not included (e.g. [1] https://arxiv.org/abs/2009.10855). In general, it is the first paper I have seen applying such a method to dialogue.\n\n*Quality*: The claims are generally well supported, with the presented results clearly identifying MPDL as the preferred method amongst several interesting baselines. Relevant ablations and case studies are considered that highlight the impact of various components in the framework (e.g., the multi-pass and discriminator components are important indeed). Some of the results are not clearly justified or explained, however; see limitations below.\n\n*Clarity*: The theoretical derivations are presented cleanly and concisely; although several pieces of notation are introduced, the proposed method and loss functions are all generally understandable. The results tables and figures are very clearly presented as well. However, I had several questions regarding the presentation of the overall method; see limitations below.\n\n*Significance*: The method is understandable and reproducible and appears to bring strong gains to style generation tasks. What is especially interesting and important is the way that the method can leverage unlabeled data, which makes the method quite generalizable (which is supported by its use on two completely different style-transfer domains). \n\n\n[1] Smith et. al., \"Controlling Style in Generated Dialogue\", https://arxiv.org/abs/2009.10855", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- I have several questions for the authors; in general, the presented equations all made sense, but required that I re-read them several times to resolve the notation. \n    - On line 56, the third contribution is stated that they provide a new dataset for this task; it is unclear what dataset they have provided (is it just the sampled set of data?)\n    - The derivation of Pr(y\\~|x, f_xy\\~) in lines 112-113 is not super clear to me (why is y~ only related to x)? (similarly for lines 114-115)\n- The scores presented in figure 2 do not seem to align with the quality of the input/output sentences (are they swapped?)\n- I believe the first line of equation 7 has a typo, which confused me for quite a bit: should the last part read log(1-Dx(Hf(x)), rather than Hf(y)?\n- Where does the informal response in “(context, informal response) dialogue pairs” (line 176) come from? \n- What does TCFC stand for?\n- How does the model seem to have higher performance on the style classifiers than human responses for the formal response target in the TCFC task?\n- Why does SFusion have a much higher Distinct score for the TCFC dataset than it does for SDGC (such that MPDL is now better)?\n- I think a qualitative case study of the shakespeare data would be interesting to evaluate the qualitative performance of the model\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.75 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for stylized dialog generation. The task is that one learns a model to automatically generates a stylized response given a context in dialogue. The authors apply the technique of dual learning to address the problem by learning bi-directional transformations of texts between the contexts C, the original responses S0, and responses with the desired style S1.

Pros
* The paper is generally clearly written.
* A new approach has been proposed, which appears to be reasonable and technically sound.
* The proposed approach appears to be new.
* Experimental results have shown that the proposed method outperforms the baselines on two large datasets. An ablation study has also been conducted.
* The authors also plan to release their code and data.

Cons
* There are some issues with the presentation. The authors have addressed some of them in the rebuttal. They are encouraged to reflect the discussion results in the new version of the paper.
* The authors also conducted additional experiments during the rebuttal. They are asked to include the results in the new version of the paper.
* The proposed method appears to be complicated. One may doubt whether it can be really used in practice.* 
Decision: Accept (Poster)