SlowGuess/ACL_2020_OCR huggingface dataset:
Hui Su $^{1*}$ , Xiaoyu Shen $^{2*}$   
Sanqiang Zhao $^{3}$ , Xiao Zhou $^{1}$ , Pengwei Hu $^{4}$ , Randy Zhong $^{1}$ , Cheng Niu $^{1}$  and Jie Zhou $^{1}$ $^{1}$ Pattern Recognition Center, Wechat AI, Tencent Inc, China
$^{2}$ MPI Informatics & Spoken Language Systems (LSV), Saarland Informatics Campus
$^{3}$ University of Pittsburgh  ${ }^{4}$ The Hong Kong Polytechnic University, Hong Kong
aaronsu@tencent.com,xshen@mpi-inf.mpg.de
Neural network-based sequence-to-sequence (seq2seq) models strongly suffer from the low-diversity problem when it comes to open-domain dialogue generation. As bland and generic utterances usually dominate the frequency distribution in our daily chitchat, avoiding them to generate more interesting responses requires complex data filtering, sampling techniques or modifying the training objective. In this paper, we propose a new perspective to diversify dialogue generation by leveraging non-conversational text. Compared with bilateral conversations, non-conversational text are easier to obtain, more diverse and cover a much broader range of topics. We collect a large-scale non-conversational corpus from multi sources including forum comments, idioms and book snippets. We further present a training paradigm to effectively incorporate these text via iterative back translation. The resulting model is tested on two conversational datasets and is shown to produce significantly more diverse responses without sacrificing the relevance with context.
Seq2seq models have achieved impressive success in a wide range of text generation tasks. In open-domain chitchat, however, people have found the model tends to strongly favor short, generic responses like "I don't know" or "OK" (Vinyals and Le, 2015; Shen et al., 2017a). The reason lies in the extreme one-to-many mapping relation between every context and its potential responses (Zhao et al., 2017; Su et al., 2018). Generic utterances, which can be in theory paired with most context, usually dominate the frequency distribution in the dialogue training corpus and thereby pushes the model to
Conversational Text
Context(Translation)	暗恋的人却不喜欢我
The one I have a crush on doesn’t like me.
Response	摸摸头
Head pat.
Non-Conversational Text
ForumComments	暗恋这碗酒，谁喝都会醉啊
Crush is an alcoholic drink, whoever drinks it will get intoxicated.
Idiom	何必等待一个没有结果的等待
Why wait for a result without hope
BookSnippet	真诚的爱情之路永不会是平坦的
The course of true love never did run smooth (From A Midsummer Night’s Dream)
Table 1: A daily dialogue and non-conversational text from three sources. The contents of non-conversational text can be potentially utilized to enrich the response generation.
blindly produce these safe, dull responses (Su et al., 2019b; Csaky et al., 2019)
Current solutions can be roughly categorized into two classes: (1) Modify the seq2seq itself to bias toward diverse responses (Li et al., 2016a; Shen et al., 2019a). However, the model is still trained on the limited dialogue corpus which restricts its power at covering broad topics in open-domain chitchat. (2) Augment the training corpus with extra information like structured world knowledge, personality or emotions (Li et al., 2016b; Dinan et al., 2019), which requires costly human annotation.
In this work, we argue that training only based on conversational corpus can greatly constrain the usability of an open-domain chatbot system since many topics are not easily available in the dialogue format. With this in mind, we explore a cheap way to diversify dialogue generation by utilizing large amounts of non-conversational text. Compared with bilateral conversations, non-conversational text covers a much broader range of topics, and can be easily obtained without further human annotation from multiple sources like forum comments, idioms and book snippets. More importantly, non-conversational text are usually more interesting and contentful as they are written to convey some spe
7087
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7087-7097 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
cific personal opinions or introduce a new topic, unlike in daily conversations where people often passively reply to the last utterance. As can be seen in Table 1, the response from the daily conversation is a simple comfort of "Head pat". Nonconversational text, on the contrary, exhibit diverse styles ranging from casual wording to poetic statements, which we believe can be potentially utilized to enrich the response generation.
To do so, we collect a large-scale corpus containing over 1M non-conversational utterances from multiple sources. To effectively integrate these utterances, we borrow the back translation idea from unsupervised neural machine translation (Sennrich et al., 2016; Lample et al., 2018b) and treat the collected utterances as unpaired responses. We first pre-train the forward and backward transduction model on the parallel conversational corpus. The forward and backward model are then iteratively tuned to find the optimal mapping relation between conversational context and non-conversational utterances (Cotterell and Kreutzer, 2018). By this means, the content of non-conversational utterances is gradually distilled into the dialogue generation model (Kim and Rush, 2016), enlarging the space of generated responses to cover not only the original dialogue corpus, but also the wide topics reflected in the non-conversational utterances.
We test our model on two popular Chinese conversational datasets weibo (Shang et al., 2015a) and douban (Wu et al., 2017). We compare our model against retrieval-based systems, style-transfer methods and several seq2seq variants which also target the diversity of dialogue generation. Automatic and human evaluation show that our model significantly improves the responses' diversity both semantically and syntactically without sacrificing the relevance with context, and is considered as most favorable judged by human evaluators<sup>1</sup>.
The tendency to produce generic responses has been a long-standing problem in seq2seq-based open-domain dialogue generation (Vinyals and Le, 2015; Li et al., 2016a). Previous approaches to alleviate this issue can be grouped into two classes.
The first class resorts to modifying the seq2seq architecture itself. For example, Shen et al. (2018a); Zhang et al. (2018b) changes the train
ing objective to mutual information maximization and rely on continuous approximations or policy gradient to circumvent the non-differentiable issue for text. Li et al. (2016d); Serban et al. (2017a) treat open-domain chitchat as a reinforcement learning problem and manually define some rewards to encourage long-term conversations. There is also research that utilizes latent variable sampling (Serban et al., 2017b; Shen et al., 2018b, 2019b), adversarial learning (Li et al., 2017; Su et al., 2018), replaces the beam search decoding with a more diverse sampling strategy (Li et al., 2016c; Holtzman et al., 2019) or applies reranking to filter generic responses (Li et al., 2016a; Wang et al., 2017). All of the above are still trained on the original dialogue corpus and thereby cannot generate out-of-scope topics.
The second class seeks to bring in extra information into existing corpus like structured knowledge (Zhao et al., 2018; Ghazvininejad et al., 2018; Dinan et al., 2019), personal information (Li et al., 2016b; Zhang et al., 2018a) or emotions (Shen et al., 2017b; Zhou et al., 2018). However, corpus with such annotations can be extremely costly to obtain and is usually limited to a specific domain with small data size. Some recent research started to do dialogue style transfer based on personal speeches or TV scripts (Niu and Bansal, 2018; Gao et al., 2019; Su et al., 2019a). Our motivation differs from them in that we aim at enriching general dialogue generation with abundant non-conversational text instead of being constrained on one specific type of style.
Back translation is widely used in unsupervised machine translation (Sennrich et al., 2016; Lample et al., 2018a; Artetxe et al., 2018) and has been recently extended to similar areas like style transfer (Subramanian et al., 2019), summarization (Zhao et al., 2019) and data-to-text (Chang et al., 2020). To the best of our knowledge, it has never been applied to dialogue generation yet. Our work treats the context and non-conversational text as unpaired source-target data. The back-translation idea is naturally adopted to learn the mapping between them. The contents of non-conversational text can then be effectively utilized to enrich the dialogue generation.
We would like to collect non-conversational utterances that stay close with daily-life topics and can
1Code and dataset available at https://github. com/chin-gyou/Div-Non-Conv
Resources	Size	Avg. length
Comments	781,847	21.0
Idioms	51,948	18.7
Book Snippets	206,340	26.9
Table 2: Statistics of Non-Conversational Text.
be potentially used to augment the response space. The utterance should be neither too long nor too short, similar with our daily chitchats. Therefore, we collect data from the following three sources:
1. Forum comments. We collect comments from zhihu  $^{2}$ , a popular Chinese forums. Selected comments are restricted to have more than 10 likes and less than 30 words  $^{3}$ .
2. Idioms. We crawl idioms, famous quotes, proverbs and locutions from several websites. These phrases are normally highly-refined and graceful, which we believe might provide a useful augmentation for responses.
3. Book Snippets. We select top 1,000 favorite novels or prose from wechat read  $^{4}$ . Snippets highlighted by readers, which are usually quintessential passages, and with the word length range 10-30 are kept.
We further filter out sentences with offensive or discriminative languages by phrase matching against a large blacklist. The resulting corpus contains over 1M utterances. The statistics from each source are listed in Table 2.
Let  $\mathcal{D} = \{(X_1,Y_1),(X_2,Y_2),\ldots ,(X_N,Y_N)\}$  denote the parallel conversational corpus.  $X_{i}$  is the context and  $Y_{i}$  is the corresponding response.  $\mathcal{D}_T = \{T_1,T_2,\dots ,T_M\}$  denotes our collected corpus where  $T_{i}$  is a non-conversational utterance. As the standard seq2seq model trained only on  $\mathcal{D}$  tends to generate over-generic responses, our purpose is to diversify the generated responses by leveraging the non-conversational corpus  $\mathcal{D}_T$  , which are semantically and syntactically much richer than
responses contained in  $\mathcal{D}$ . In the following section, we first go through several baseline systems, then introduce our proposed method based on back translation.
4.1 Retrieval-based System
The first approach we consider is a retrieval-based system that considers all sentences contained in  $\mathcal{D}_T$  as candidate responses. As the proportion of generic utterances in  $\mathcal{D}_T$  is much lower than that in  $\mathcal{D}$ , the diversity will be largely improved. Standard retrieval algorithms based on context-matching (Wu et al., 2017; Bartl and Spanakis, 2017) fail to apply here since non-conversational text does not come with its corresponding context. Therefore, we train a backward seq2seq model on the parallel conversational corpus  $\mathcal{D}$  to maximize  $p(X_i|Y_i)$ . The score assigned by the backward model, which can be seen as an estimation of the point-wise mutual information, is used to rank the responses (Li et al., 2016a)
The major limitation of the retrieval-based system is that it can only produce responses from a finite set of candidates. The model can work well only if an appropriate response already exists in the candidate bank. Nonetheless, due to the large size of the non-conversational corpus, this approach is a very strong baseline.
4.2 Weighted Average
The second approach is to take a weighted average score of a seq2seq model trained on  $\mathcal{D}$  and a language model trained on  $\mathbf{D}_T$  when decoding responses. The idea has been widely utilized on domain adaptation for text generation tasks (Koehn and Schroeder, 2007; Wang et al., 2017; Niu and Bansal, 2018). In our scenario, basically we hope the generated responses could share the diverse topics and styles of the non-conversational text, yet stay relevant with the dialogue context. The seq2seq model  $S2S$  is trained on  $\mathcal{D}$  as an indicator of how relevant each response is with the context. A language model  $\mathcal{L}$  is trained on  $\mathcal{D}_T$  to measure how the response matches the domain of  $\mathcal{D}_T$ . The decoding probability for generating word  $w$  at time step  $t$  is assigned by:
$$
p _ {t} (w) = \alpha S 2 S _ {t} (w) + (1 - \alpha) L _ {t} (w) \quad (1)
$$
5The backward seq2seq model measures the context relevance better than forward models since the latter highly biases generic utterances (Li et al., 2016a; Zhang et al., 2018b)
<sup>2</sup>https://www.zhihu.com
3The posts are usually very long, describing a specific social phenomenon or news event, so building parallel conversational corpus from post-comment pairs is difficult. Nonetheless, these high-liked comments are normally high-quality themselves and can be used to augment the response space.
4https://weread.qq.com/
7089
(a) Retrieval-based System
(b) Weighted Average
(c) Multi-task
(d) Back Translation
Figure 1: Comparison of four approaches leveraging the non-conversational text.  $S2S_{fw}$ ,  $S2S_{bw}$  and  $LM$  indicate the forward, backward seq2seq and language model respectively. (d) visualizes the process of one iteration for the back translation approach. Striped component are not updated in each iteration.
where  $\alpha$  is a hyperparameter to adjust the balance between the two. Setting  $\alpha = 1$  will make it degenerate into the standard seq2seq model while  $\alpha = 0$  will totally ignore the dialogue context.
4.3 Multi-task
The third approach is based on multi-task learning. A seq2seq model is trained on the parallel conversational corpus  $\mathcal{D}$  while an autoencoder model is trained on the non-parallel monologue data  $\mathcal{D}_T$ . Both models share the decoder parameters to facilitate each other. The idea was first experimented on machine translation in order to leverage large amounts of target-side monolingual text (Luong et al., 2016; Sennrich et al., 2016). Luan et al. (2017) extended it to conversational models for speaker-role adaptation. The intuition is that by tying the decoder parameters, the seq2seq and autoencoder model can learn a shared latent space between the dialogue corpus and non-conversational text. When decoding, the model can generate responses with features from both sides.
4.4 Back Translation
Finally, we consider the back translation technique commonly used for unsupervised machine translation (Artetxe et al., 2018; Lample et al., 2018a). The basic idea is to first initialize the model properly to provide a good starting point, then iteratively perform backward and forward translation to learn the correspondence between context and unpaired non-conversational utterances.
Initialization Unlike unsupervised machine translation, the source and target side in our case
come from the same language, and we already have a parallel conversational corpus  $\mathcal{D}$ , so we can get rid of the careful embedding alignment and autoencoding steps as in Lample et al. (2018b). For the initialization, we simply train a forward and backward seq2seq model on  $\mathcal{D}$ . The loss function is:
$$
\mathbb {E} _ {X _ {i}, Y _ {i} \sim \mathcal {D}} - \log P _ {f} (Y _ {i} | X _ {i}) - \log P _ {b} (X _ {i} | Y _ {i}) \tag {2}
$$
where  $P_{f}$  and  $P_{b}$  are the decoding likelihood defined by the forward and backward seq2seq model respectively. We optimize Eq. 2 until convergence. Afterwards, the forward and backward seq2seq can learn the backbone mapping relation between a context and its response in a conversational structure.
Backward After the initialization, we use the backward seq2seq to create pseudo parallel training examples from the non-conversational text  $\mathcal{D}_T$ . The forward seq2seq is then trained on the pseudo pairs. The objective is to minimize:
$$
\begin{array}{l} \mathbb {E} _ {T _ {i} \sim \mathcal {D} _ {T}} - \log P _ {f} (T _ {i} | b (T _ {i})) \\ b \left(T _ {i}\right) = \underset {u} {\arg \max } P _ {b} (u \mid T _ {i}) \tag {3} \\ \end{array}
$$
where we approximate the arg max function by using a beam search decoder to decode from the backward model  $P_{b}(u|T_{i})$ . Because of the non-differentiability of the arg max operator, the gradient is only passed through  $P_{f}$  but not  $P_{b}$ .
As  $P_{b}$  is already well initialized by training on the parallel corpus  $\mathcal{D}$ , the back-translated pseudo
As also noted in Lample et al. (2018b), backpropagating further through  $P_{b}$  brings no improvement.
7090
(Initialization) Train by minimizing Eq. 2 until convergence;
(Backward) Train by minimizing Eq. 3 until convergence;
(Forward) Train by minimizing Eq. 4 until convergence;
Algorithm 1: Model Training Process
pair  $\{b(T_i), T_i\}$  can roughly follow the typical human conversational patterns. Training  $P_f$  on top of them will encourage the forward decoder to generate utterances in the domain of  $T_i$  while maintaining coherent as a conversation.
Forward The forward translation follows a similar step as back translation. The forward seq2seq  $P_{f}$  translates context into a response, which in return form a pseudo pair to train the backward model  $P_{b}$ . The objective is to minimize:
$$
\begin{array}{l} \mathbb {E} _ {X _ {i} \sim \mathcal {D}} - \log P _ {b} \left(X _ {i} \mid f \left(X _ {i}\right)\right) \\ f (X _ {i}) = \underset {v} {\arg \max } P _ {f} (v | X _ {i}) \tag {4} \\ \end{array}
$$
where the arg max function is again approximated with a beam search decoder and the gradient is only backpropagated through  $P_{b}$ . Though  $X_{i}$  has its corresponding  $Y_{i}$  in  $\mathcal{D}$ , we drop  $Y_{i}$  and instead train on forward translated pseudo pairs  $\{X_i,f(X_i)\}$ . As  $P_{f}$  is trained by leveraging data from  $\mathcal{D}_T$ ,  $f(X_{i})$  can have superior diversity compared with  $Y_{i}$ .
The encoder parameters are shared between the forward and backward models while decoders are separate. The backward and forward translation are iteratively performed to close the gap between  $P_{f}$  and  $P_{b}$  (Hoang et al., 2018; Cotterell and Kreutzer, 2018). The effects of non-conversational text are strengthened after each iteration. Eventually, the forward model will be able to produce diverse responses covering the wide topics in  $\mathcal{D}_T$ . Algorithm 1 depicts the training process.
We conduct our experiments on two Chinese dialogue corpus Weibo (Shang et al., 2015b) and Douban (Wu et al., 2017). Weibo is a popular Twitter-like microblogging service in China, on which a user can post short messages, and other
users make comment on a published post. The postcomment pairs are crawled as short-text conversations. Each utterance has 15.4 words on average and the data is split into train/valid/test subsets with  $4\mathrm{M} / 40\mathrm{k} / 10\mathrm{k}$  utterance pairs. Douban  $^{8}$  is a Chinese social network service where people can chat about different topics online. The original data contains 1.1M multi-turn conversations. We split them into two-turn context-response pairs, resulting in 10M train, 500k valid and 100K test samples.
5.2 General Setup
For all models, we use a two-layer LSTM (Hochreiter and Schmidhuber, 1997) encoder/decoder structure with hidden size 500 and word embedding size 300. Models are trained with Adam optimizer (Kingma and Ba, 2015) with an initial learning rate of 0.15. We set the batch size as 256 and use the gradients clipping of 5. We build out vocabulary with character-based segmentation for Chinese. For non-Chinese tokens, we simply split by space and keep all unique tokens that appear at least 5 times. Utterances are cut down to at most 50 tokens and fed to every batch. We implement our models based on the OpenNMT toolkit (Klein et al., 2017) and other hyperparameters are set as the default values.
5.3 Compared Models
We compare our model with the standard seq2seq and four popular variants which were proposed to improve the diversity of generated utterances. All of them are trained only on the parallel conversational corpus:
Standard The standard seq2seq with beam search decoding (size 5).
MMI The maximum mutual information decoding which reranks the decoded responses with a backward seq2seq model (Li et al., 2016a). The hyperparameter  $\lambda$  is set to 0.5 as suggested. 200 candidates per context are sampled for re-ranking
Diverse Sampling The diverse beam search strategy proposed in Vijayakumar et al. (2018) which explicitly controls for the exploration and exploitation of the search space. We set the number of groups as 5,  $\lambda = 0.3$  and use the Hamming diversity as the penalty function as in the paper.
<sup>7</sup>http://www.weibo.com/
<sup>8</sup>https://www.douban.com/group
7091
Nucleus Sampling Proposed in Holtzman et al. (2019), it allows for diverse sequence generations. Instead of decoding with a fixed beam size, it samples text from the dynamic nucleus. We use the default configuration and set  $p = 0.9$ .
CVAE The conditional variational autoencoder (Serban et al., 2017b; Zhao et al., 2017) which injects diversity by imposing stochastical latent variables. We use a latent variable with dimension 100 and utilize the KL-annealing strategy with step  $350\mathrm{k}$  and a word drop-out rate of 0.3 to alleviate the posterior collapse problem (Bowman et al., 2016).
Furthermore, we compare the 4 approaches mentioned in §4 which incorporate the collected non-conversational text:
Retrieval-based (§4.1) Due to the large size of the non-conversational corpus, exact ranking is extremely slow. Therefore, we first retrieve top 200 matched text with elastic search based on the similarity of Bert embeddings (Devlin et al., 2019). Specifically, we pass sentences through Bert and derive a fixed-sized vector by averaging the outputs from the second-to-last layer (May et al., 2019)<sup>9</sup>. The 200 candidates are then ranked with the backward score<sup>10</sup>.
Weighted Average (§4.2) We set  $\lambda = 0.5$  in eq. 1, which considers context relevance and diversity with equal weights.
Multi-task ((§4.3)) We concatenate each context-response pair with a non-conversational utterance and train with a mixed objective of seq2seq and autoencoding (by sharing the decoder).
Back Translation (§4.4) We perform the iterative backward and forward translation 4 times for both datasets. We observe the forward cross entropy loss converges after 4 iterations.
As for the experiment results, we report the automatic and human evaluation in §6.1 and §6.2 respectively. Detailed analysis are shown in §6.3 to elaborate the differences among model performances and some case studies.
6.1 Automatic Evaluation
Evaluating dialogue generation is extremely difficult. Metrics which measure the word-level overlap like BLEU (Papineni et al., 2002) have been widely used for dialogue evaluation. However, these metrics do not fit into our setting well as we would like to diversify the response generation with an external corpus, the generations will inevitably differ greatly from the ground-truth references in the original conversational corpus. Though we report the BLEU score anyway and list all the results in Table 3, it is worth mentioning that the BLEU score itself is by no means a reliable metric to measure the quality of dialogue generations.