SlowGuess/NAACL_2022_OCR huggingface dataset:
Qingfeng Sun Can Xu Huang Hu Yujing Wang Jian Miao Xiubo Geng Yining Chen Fei Xu Daxin Jiang*
Microsoft, Beijing, China
{qins, caxu, huahu, yujwang, jianm,
xigeng,yinichen,fexu,djiang}@microsoft.com
Current Knowledge-Grounded Dialogue Generation (KDG) models specialize in producing rational and factual responses. However, to establish long-term relationships with users, the KDG model needs the capability to generate responses in a desired style or sentiment. Thus, we study a new problem: Stylized Knowledge-Grounded Dialogue Generation (SKDG). It presents two challenges: (1) How to train a SKDG model where no <context, knowledge, stylized response> triples are available. (2) How to cohere with context and preserve the knowledge when generating a stylized response. In this paper, we propose a novel disentangled template rewriting (DTR) method which generates responses via combing disentangled style templates (from monolingual stylized corpus) and content templates (from KDG corpus). The entire framework is end-to-end differentiable and learned without supervision. Extensive experiments on two benchmarks indicate that DTR achieves a significant improvement on all evaluation metrics compared with previous state-of-the-art stylized dialogue generation methods. Besides, DTR achieves comparable performance with the state-of-the-art KDG methods in standard KDG evaluation setting.
Every good conversational agent needs the ability to generate good responses, which are not only knowledgeable and coherent with contexts but also have abundant and desirable styles and sentiments (Rashkin et al., 2018; Smith et al., 2020; Zhou et al., 2020). Such an agent can deliver depth dialogues on various topics and yield more engaging and vivacious conversations to attract more users. In other words, rational and perceptual thought are all necessary for a perfect dialogue agent. Nevertheless, most existing Knowledge-Grounded Dialogue Generation (KDG) methods (Dinan et al., 2019;
Figure 1: The KDG models only produce a pedantic response, which lacks emotion and attraction compared with the responses with polite style, positive and negative sentiments.
Kim et al., 2020; Zhao et al., 2020b) pay more attention to the former and ignore the latter. Specifically, let's claim our motivation: The previous KDG works mainly focus on selecting knowledge and expressing knowledge in response accurately. However, the excessive emphasis on knowledge makes the KDG models tend to mechanically copy large sections from the unstructured knowledge (e.g., Wikipedia). As a result, the responses from the KDG models reflect a "pedantic" style (i.e., use very technical terms and language), making the conversation less engaging and less natural.
In this paper, we are aiming to have the first attempt to incorporate stylized-text-generation into KDG to tackle the above challenge. As shown in Figure 1, the KDG model takes the context and related document as input and outputs a knowledgeable but pedantic response corresponding to the polite one, which makes people feel respected and comfortable. In the meanwhile, the polite, positive responses all show bright and lively styles which not only are able to condense the core meaning of the response, but also sound appealing to the users for more exposure and memorableness.
Specifically, we formulate a new problem: Styl-
Human Language Technologies, pages 3304 - 3318
Figure 2: Overview of DTR. The sequential style disentangler can find out and replace the style-related tokens in generated response with  $[^{*}]$  and produce a template. Then, the style rewriter transfers the template to a new response in the target style.
Figure 3: Our model combines the style-related fragments from the style corpus and the knowledge fragments from the KDG response to generate a response in a desired sentiment or style.
ized Knowledge-Grounded Dialogue Generation (SKDG). That is, the responses provided by a model should be coherent with the dialogue contexts and be consistent with the given knowledge and a designated style or sentiment. The challenges lie in two aspects: (1) As lacking stylized knowledge-grounded dialogue triples (i.e., <context, knowledge, stylized response>), we need to train the SKDG model jointly by both independent knowledge-grounded dialogues and monolingual corpus with a target style or sentiment. (2) In addition to being coherent with context and consistent with target style / sentiment, a good response from SKDG needs to ensure objective correctness in the knowledge section. Especially when the given knowledge contains style-related content, existing stylized dialogue generation (SDG) models (Zheng et al., 2020; Ze et al., 2020) may undermine the correctness of knowledge section. For example, in case of negative-to-positive sentiment transfer shown in Figure 3, the first two negative fragments of KDG response - "dislike cruel" and "horrible" should be modified to positive fragments, but the third "bad" should be retained to maintain the
original meaning of knowledge section.
Hence, our motivation is: on the one hand, bridging the separate knowledge-grounded response generation and stylized rewriting by sharing a disentangled template (addressing challenge (1)); on the other hand, enhancing the fidelity regarding to given knowledge by using a reinforcement learning approach (addressing challenge (2)).
To achieve this goal, we propose a new paradigm: Generate-Disentangle-Rewrite. Firstly, given a dialogue context and the associated external knowledge, a KDG model is adopted to generate a response. Then as shown in Figure 2 and 3, we leverage a sequential style disentangler to delete style-related fragments from the KDG response to form a style-agnostic template. Then the rewritten rewrites the entire template token-by-token, injecting style-related fragments in the process, to generate a vivid and informative response in the desired style. As there is no supervision on the style disentangler and the style rewriter, we propose a reinforcement learning-based method to train the style disentangling and style rewriting in an end-to-end manner using a style intensity reward and a semantic similarity reward. The huge joint action space of the two modules fragile the training, thus we propose a novel weakly supervised stylistic template disentangle method to initialize both the disentangler and the rewriter. As a result, our method successfully produces the knowledgeable response in the desired style without any paired training data.
We name our model DTR standing for "Disentangled Template Rewriting". We demonstrate this approach using knowledge-grounded dialogues from Wizard of Wikipedia (Dinan et al., 2019) and Topical Chat (Gopalakrishnan et al.,
2019) with three sets of sentences with distinct sentiments (positive, negative) and styles (polite). Automatic and human evaluations show that our method significantly outperforms competitive baselines with a large margin in generating coherent and knowledgeable dialogue responses while rendering stronger stylistic features.
Our contributions are three-fold: (1) To the best of our knowledge, it is the first work on the generation of stylized knowledge-grounded responses without any labeled paired data for style-specific context-knowledge-response. (2) We proposed a stylized knowledge-grounded dialogue generation method via disentangled template rewriting. To optimize the model, we propose a reinforcement learning approach with a novel weakly supervised method to guide the learning of both disentangler and rewriter. (3) Extensive experiments on two benchmarks indicate that DTR significantly outperforms previous state-of-the-art SDG methods on all evaluation metrics. Besides, DTR achieves comparable performance with the state-of-the-art KDG methods in the standard KDG evaluation setting. Our source code will be released at https://github.com/victorsungo/SKDG-DTR.
Knowledge-Grounded Dialogue Generation has attracted broad interest in recent years, where the knowledge could be obtained from documents (Dinan et al., 2019; Kim et al., 2020; Rashkin et al., 2021) and images (Shuster et al., 2018; Yang et al., 2020a; Liang et al., 2021). Our study considers document-grounded dialogue generation. With the rapid development of pre-training techniques, Zhao et al. (2020a) proposes pre-trained disentangled decoder, and Li et al. (2020) proves that the KDG models could achieve comparable performance with state-of-the-art supervised methods through an unsupervised learning method. Rather than testing new architectures on the benchmarks, our main contribution lies in the investigation of transferring the pedantic and factual knowledge-grounded responses into a desired style or sentiment, which roots in the requirement from practice.
Text Style and Sentiment Transfer was inspired by visual style transfer (Gatys et al., 2016; Zhu et al., 2017), and many methods have made remarkable work in text style transfer, which aims to alter the style attributes of text while preserving the content. A prevalent idea of style transfer
is to disentangle the content and style of text (Fu et al., 2018; Li et al., 2018; Jin et al., 2020; Wen et al., 2020; Zhu et al., 2021) or leverage the backtranslation (Lample et al., 2019; Li et al., 2021). Stylized dialogue generation has attracted numerous attention in recent years (Niu and Bansal, 2018; Gao et al., 2019). Different from style transfer, stylized dialogue generation requires that the response is also coherent with its context.
Stylized Dialogue Generation refers to generate a dialogue response in the target style. Akama et al. (2017) first train a response generation model on a dialog corpus then use a style corpus to fine-tune the model. Yang et al. (2020b) builds a pretrained language model and devise both a word-level loss and a sentence-level loss to fine-tune the pre-trained model towards the target style. Su et al. (2020a) proposes an information guided reinforcement learning strategy to better balance the trade-off between the stylistic expression and the content quality. Sun et al. (2021) blends textual and visual responses to make the dialogue style more attractive and vivid. Zheng et al. (2020) captures stylistic features embedded in unpaired texts, Su et al. (2020b) uses the pointwise mutual information (PMI) to determine stylistic word, Ze et al. (2020) adopts pre-trained models to tackle the open-domain stylized response generation.
We propose a novel disentangled template rewriting approach as the first attempt to study stylized knowledge-grounded dialogue generation without any supervised style-specific context-knowledge-response triples data.
For the SKDG task, our model is trained on a dialogue dataset  $\mathcal{D}_c = \{(K_i,U_i,Y_i)\}_{i = 1}^N$  and a style corpus  $\mathcal{D}_s = \{T_i\}_{i = 1}^M$ , where  $\forall (K_i,U_i,Y_i) \in \mathcal{D}_c$ ,  $U_i$  is a dialogue context,  $K_i$  a external document that contains relevant knowledge regarding to  $U_i$  and  $Y_i$  a response to  $U_i$ , and  $\forall T_i \in \mathcal{D}_s$ ,  $T_i$  is a piece of text in the target style  $\mathcal{S}$ . We don't assume that there exists triples  $\{(K,U,Y')\}$  with  $Y'$  expressed in the style or sentiment  $\mathcal{S}$ , e.g.,  $\mathcal{S} = \{\text{"polite","positive","negative"}\}$ . Our goal is to learn a generation method  $P(Y|K,U,S)$  with  $\mathcal{D}_c$  and  $\mathcal{D}_s$ , thus given a document  $K$  and a context  $U$ , one can generate a response  $Y$  following the desired style  $\mathcal{S}$ , where  $Y$  also coheres with context and preserves the knowledge.
Heading for learning an effective disentangled template rewriting model for SKDG task, we need to deal with several challenges: (1) how to distinguish the style-related fragments from a given sentence without any supervision;(2) how to retain the style-related fragments in knowledge section to defend the completeness;(3) how to rewrite the disentangled template holistically instead of inserting a few style words, thus to enhance fluency and diversity.
Our DTR model is made up of a knowledge-grounded response generator  $\mathcal{G}_G$ , a sequential style disentangler  $\mathcal{F}$  and a style rewriter  $\mathcal{G}_R$ . Given a dialogue context  $U$  and its associated knowledge  $K$ , we first use  $\mathcal{G}_G$  to generate a response  $\overline{Y}$ . Figure 2 illustrates the cooperation of  $\mathcal{F}$  and  $\mathcal{G}_R$ . The former reads  $\overline{Y}$  and disentangles the style-related content from  $\overline{Y}$  to form a style-agnostic template sequence  $\widetilde{Y}$ , which is further provided as input to  $\mathcal{G}_R$  to generate the transferred response  $\hat{Y}$  in a target style. Since  $\widetilde{Y}$  is discrete, the major hinder of learning  $\mathcal{F}$  lies in the gradient is not differentiable. To cope with the challenge, we exploit a reinforcement learning approach to optimize  $\mathcal{F}$  leveraging the signals from  $\mathcal{G}_R$ .
So why do we need a Disentangler + Rewriter architecture? The previous SDG methods fuse the knowledge and style into mixed representation and decode a response. Due to the difficulty of mixing knowledge and style implicitly under the unsupervised setting, it is possible to lose knowledge or style in the decoding stage. Motivated by this, we propose to decouple the response generation into two relatively independent processes: 1) knowledge fragments generation 2) style fragments generation. The knowledge fragments and style fragments are explicitly composited into the response in the final stage. Such a method ensures the knowledge is successfully presented in the final output. The disentangler plays a central role in decoupling and composition. In the following, we will elaborate details of each component.
4.1.1 Knowledge-Grounded Response Generator
The generator  $\mathcal{G}_G$  is a sequence-to-sequence model based on the Transformer architecture (Vaswani et al., 2017), it consists of a 6-layers encoder and decoder with a hidden size of 768. Given a dialogue context  $U = \{u_{1},\dots ,u_{i},\dots ,u_{l}\}$  with
$u_{i}$  the  $i$ -th utterance, and a document  $K = \{k_{1},\ldots ,k_{i},\ldots ,k_{h}\}$  with  $k_{i}$  the  $i$ -th sentence. We concatenate  $U$  and  $K$  as a long sentence as the input of the encoder, then the decoder generates a response  $\overline{Y}$  as output
$$
\bar {Y} = \left\{w _ {1}, \dots , w _ {i}, \dots , w _ {m} \right\} = \mathcal {G} _ {G} (U, K) \tag {1}
$$
4.1.2 Sequential Style Disentangler
To identify and disentangle the style-related fragments from  $\overline{Y}$ , we employ a sequence labeling module named Sequential Style Disentangler  $\mathcal{F}$  to model the probabilities  $\{x_{i}\}_{i = 1}^{m}$  of being style-related token at each position in  $\overline{Y}$ . The formulations are as follows:
$$
P _ {\mathcal {F}} (A | \bar {Y}, U, K) = \prod_ {i = 1} ^ {m} P \left(a _ {i} | \bar {Y}, U, K\right) \tag {2}
$$
$$
P \left(a _ {i} | \bar {Y}, U, K\right) = x _ {i} = \operatorname {s i g m o i d} \left(\boldsymbol {W} e _ {i}\right) \tag {3}
$$
$$
\left\{e _ {1}, \dots , e _ {m} \right\} = \operatorname {B E R T} (\bar {Y}, U, K) \tag {4}
$$
where  $\mathbf{W} \in \mathbb{R}^{v \times 1}$  and  $e_i \in \mathbb{R}^v$ ,  $v$  is representation dimension,  $a_i \in \{\text{replace}, \text{retain}\}$ ,  $A = \{a_i\}_{i=1}^m$ . Then when generating  $\widetilde{Y}$  if  $x_i > \varepsilon$ ,  $a_i$  will be operation "replace" indicating  $w_i$  is a style token and needs to be replaced with a tag token [*], and viceversa for  $x_i < \varepsilon$ ,  $a_i$  will be operation "retain" indicating  $w_i$  remains unchanged. Threshold  $\varepsilon$  is equal to top  $P_r\%$  percentile of  $\{x_i\}_{i=1}^m$ , where  $P_r$  is a hyper parameter. Finally, we perform the predicted sequence of operations on  $\overline{Y}$  to obtain style-agnostic template  $\widetilde{Y}$ . As the style disentangler tags each word in a sentence, it captures the style fragments (e.g., words, phrases, sub-sequences, or even the whole sentence) rather than only style tokens. The learning detail is presented in Appendix A.1.
4.1.3 Style Rewriter
With  $\widetilde{Y}$  as input, the style rewriter  $\mathcal{G}_R$  generates a new  $\hat{Y}$  word-by-word in the target style.  $\mathcal{G}_R$  has the same architecture as  $\mathcal{G}_G$ . The generation process of  $\mathcal{G}_R$  is formulated as:
$$
P _ {R} (\hat {Y} | \widetilde {Y}) = \prod_ {t = 1} ^ {h} P _ {R} (\hat {w} _ {t} | \widetilde {Y}) \tag {5}
$$
where  $\hat{w}_t$  is the  $t$ -th token of  $\hat{Y}$  whose length is  $h$ .
4.2 Reinforcement Learning
Neither style disentangler nor style rewriter has supervision for training. Moreover, We need to ensure the correctness of  $\hat{Y}$  without any modifications of the original content in the knowledge
section of  $\overline{Y}$ . To cope with the challenges, we exploit REINFORCE (Sutton et al., 2000) to train  $\mathcal{F}$  and  $\mathcal{G}_R$  jointly with a total reward determined by the semantic similarity with the ground-truth response and the consistency with the desired style. Specifically, we maximize the expected reward as:
$$
\mathcal {R} _ {\mathcal {R L}} = \mathbb {E} _ {\hat {Y} \sim P _ {R} (\hat {Y})} \mathbb {E} _ {\widetilde {Y} \sim P _ {\mathcal {F}} (A)} [ R (\widetilde {Y}, Y) ] \quad (6)
$$
where  $P_{R}(\hat{Y})$  and  $P_{\mathcal{F}}(A)$  stand for  $P_{R}(\hat{Y}|\widetilde{Y})$  and  $P_{\mathcal{F}}(A|\overline{Y},U,K)$  respectively,  $R(\widetilde{Y},Y) = \mathrm{Sim}(\hat{Y},Y) + \mathrm{Cls}(\hat{Y})$ ,  $\mathrm{Sim}(\cdot)$  is embedding cosine similarity which supervises the knowledge regularization,  $\mathrm{Cls}(\cdot)$  is the style intensity predicted by a classifier. We subtract the mean value of rewards  $R$  in a batch to reduce the variance of gradient estimation (Clark and Manning, 2016). In order to avoid the destroying issue of RL, we fix the parameters of  $\mathcal{G}_R$ , then only optimize the  $\mathcal{F}$ .
4.3 Weakly Supervised Learning
Since the style disentangler and style rewriter need to be carefully synchronized, ideally we hope they can benefit each other in learning. However, in the early stage as the parameters of  $\mathcal{F}$  and  $\mathcal{G}_R$  are far from optimal. It is possible that, on the one hand, the templates that are not decoupled successfully hinder  $\mathcal{G}_R$  learning from rewriting style fragments accurately. On the other hand, noise signals from rewards computed with the low-quality responses generated by  $\mathcal{G}_R$  flow to the learning of  $\mathcal{F}$ , resulting in inferior  $\mathcal{F}$ . To alleviate error accumulation in joint training, we propose a novel weakly supervised stylistic template disentangle method to assist the learning of  $\mathcal{F}$  and  $\mathcal{G}_R$ .
4.3.1 Weakly Supervised Disentangler
Intuitively, style fragments dominate the distribution of style corpus  $\mathcal{D}_s$  compared with content fragments, thus the style fragments are easier to be reconstructed than content fragments by the denoising autoencoder trained on  $\mathcal{D}_s$ . As shown in Figure 4, a denoising reconstruction model  $\mathcal{G}_D$  reconstructs the style word "good" successfully but fail to do that for content word "pizza" in the same response from  $\mathcal{D}_c$ . Particularly, we randomly divide  $\mathcal{D}_s$  into two halves with equal probability:  $\mathcal{D}_s^1$  and  $\mathcal{D}_s^2$ , then  $\mathcal{D}_s^1$  is used to train the denoising reconstruction model  $\mathcal{G}_D$ . The reconstruction objective  $\mathcal{L}_{\mathcal{S}}$  is formulated as:
$$
\mathcal {L} _ {\mathcal {S}} = \mathbb {E} _ {T \sim \mathcal {D} _ {s} ^ {1}} [ - \log p (T | \widetilde {T}) ] \tag {7}
$$
Figure 4: The positive sentiment word "good" is easier to be reconstructed than the knowledge word "pizza" in the sentence, where the wrong prediction "beef" would hurt the knowledge preservation and confuse the dialogue theme.
where  $\widetilde{T}$  is the corrupted version of  $T$  by randomly mask  $15\%$  tokens.
Then for each sentence  $T = \{t_i\}_{i=1}^m$  (with  $t_i$  the  $i$ -th token in  $T$ ) in  $\mathcal{D}_s^2$ , we sequentially mask one token each time to construct its denoising versions  $\{\widetilde{T}_i\}_{i=1}^m$ , then  $\{\widetilde{T}_i\}_{i=1}^m$  are inferred by  $\mathcal{G}_D$  to reconstruct  $\{\hat{T}_i\}_{i=1}^m$ . We acquire a distance sequence  $\pmb{d} = \{d_i\}_{i=1}^m = \{\mathrm{Dis}(t_i, \hat{t}_i)\}_{i=1}^m$  where  $\mathrm{Dis}(\cdot, \cdot)$  denotes a distance function. Based on above intuition, lower  $d_i$  means  $t_i$  is more preferable as a style-related token, thus for  $t_i$  and  $t_j$ , if  $d_i < d_j$ , we define the label  $y = 1$ , and viceversa for  $d_i > d_j$ ,  $y = -1$ . We aggregate all  $< t_i, t_j, y >$  triples to construct  $\mathcal{D}_{s\_t}$  to optimize the style disentangler via the pairwise ranking loss:
$$
\mathcal {L} _ {\mathcal {P}} \left(t _ {i}, t _ {j}, y\right) = \max  \left(0, - y * \left(t _ {i} - t _ {j}\right) + \mu\right) \tag {8}
$$
where  $\mu$  is a hyper parameter. The action space of token-level pairwise ranking is large, so for each sentence in  $\mathcal{D}_{s\_ t}$ , we randomly sample  $Z$  non-repetitive  $< x_{i},x_{j},y>$  triples to optimize  $\mathcal{L}_{\mathcal{P}}$ , where  $Z$  is a hyper parameter. The style tokens in various style corpus found by the style disentangler is presented in Appendix B.6.
4.3.2 Weakly Supervised Rewriter
The training data for the rewriter are also constructed by an unsupervised method: Optimized style disentangler  $\mathcal{F}$  (Eq.8) infers the style corpus  $\mathcal{D}_s = \{T_i\}_{i=1}^M$  and generates a disentangled template set  $\widehat{\mathcal{D}}_s = \{\widetilde{T}_i\}_{i=1}^M$ . Then the rewriter takes paired  $<\widetilde{T}, T>$  as input and output respectively. Since  $\widetilde{T}$  is style-agnostic, the rewriter would focus on transferring a factual sentence to a desired sentence with target style. The loss function for the rewriter  $\mathcal{G}_R$  is:
$$
\mathcal {L} _ {\mathcal {R}} = - \frac {1}{M} \sum_ {l = 1} ^ {M} \left(\prod_ {i = 1} ^ {| T _ {l} |} p \left(t _ {l, i} \mid t _ {l, 1}, \dots , t _ {l, i - 1}; \widetilde {T} _ {l}\right)\right) \tag {9}
$$
where  $t_{l,i}$  is the  $i$ -th token in  $l$ -th sentence. Specifically, the rewriting  $\mathcal{G}_R$  has a same architecture as  $\mathcal{G}_G$ .
Algorithm 1 Optimization Algorithm.
1: Input: Datasets  $\mathcal{D}_c, \mathcal{D}_s$ ; Models  $\mathcal{G}_G, \mathcal{F}, \mathcal{G}_R$ .
2: Optimize  $\mathcal{G}_G$  using  $\mathcal{D}_c$ .
3: Construct  $\mathcal{D}_{s\_ t}$ .
4: Optimize  $\mathcal{F}$  using  $\mathcal{D}_{s,t}$  (Eq.8).
5: Construct  $\widetilde{\mathcal{D}}_s$  using  $\mathcal{F}$
6: Optimize  $\mathcal{G}_R$  using  $\mathcal{D}_s$  and  $\widetilde{\mathcal{D}}_s$  (Eq.9).
7: Further Optimize  $\mathcal{F}$  using  $\mathcal{D}_{conv}$  (Eq.6).
8: return  $\mathcal{G}_G, \mathcal{F}, \mathcal{G}_R$ .
We conduct experiments on Wizard of Wikipedia (Wizard) and Topical Chat with positive and negative sentiments, and polite style.
KDG Corpus Wizard consists of 1365 topics, and each conversation happens between a wizard who has access to Wikipedia paragraphs and an apprentice who talks to the wizard. Topical Chat utilizes wiki articles, Washington Post, and Reddit fun facts as the knowledge source. The participants play symmetric and asymmetric roles according to the knowledge. Wizard and Topical Chat are split as training, valid and test set respectively. We compare our method with baselines on Wizard Test Seen and Topical Chat Test Freq. More details are described in Appendix B.1.
Style Corpus We use Amazon dataset published in Juncen et al. (2018) and Politeness published in Madaan et al. (2020) for style transfer. Amazon consists of product reviews from Amazon for flipping sentiment, and it contains 27800 positive sentences and 27700 negative sentences. For Politeness, We use the P9-bucket as the polite dataset, which consists of 27000 polite sentences.