SlowGuess/NAACL_2022_OCR huggingface dataset:
Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018b. Common-sense knowledge aware conversation generation with graph attention. In *IJCAI*, pages 4623-4629.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018c. A dataset for document grounded conversations. arXiv preprint arXiv:1809.07358.
$$
\begin{array}{l} \log p (R | U, K) \\ = \log \sum_ {(M, Z)} p (R, M, Z) \\ = \log \sum_ {(M, Z)} q (M, Z | R) \frac {p (R , M , Z)}{q (M , Z | R)} \\ = \log \mathbb {E} _ {(M, Z) \sim q (M, Z | R)} \frac {p (R , M , Z)}{q (M , Z | R)} \tag {10} \\ \geq \mathbb {E} _ {(M, Z) \sim q (M, Z | R)} \log \frac {p (R , M , Z)}{q (M , Z | R)} \\ = \mathbb {E} _ {(M, Z) \sim q (M, Z | R)} \log p (R | M, Z) \\ - \mathbb {E} _ {(M, Z) \sim q (M, Z | R)} (\log q (M, Z | R) \\ - \log p (M, Z)). \\ \end{array}
$$
According to the mean-filed approximation,  $q(M,Z)\approx q(M)q(Z)$  Therefore,  $\mathbb{E}_{(M,Z)\sim q(M,Z|R)}\log p(R|M,Z)$  and  $\mathbb{E}_{(M,Z)\sim q(M,Z|R)}\bigl (\log q(M,Z|R) - \log p(M,Z)\bigr)$  can be re-written as:
$$
\begin{array}{l} \mathbb {E} _ {(M, Z) \sim q (M, Z | R)} \log p (R | M, Z) \\ = \mathbb {E} _ {M \sim q (M | R)} \left(\mathbb {E} _ {Z \sim q (Z | M, R)} \sum_ {t = 1} ^ {l _ {r}} \log p \left(r _ {t} \mid r _ {<   t}, z _ {t}\right)\right) \tag {11} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {(M, Z) \sim q (M, Z | R)} (\log q (M, Z | R) - \log p (M, Z)) \\ = \mathbb {E} _ {M \sim q (M | R)} \left(\mathbb {E} _ {Z \sim q (Z | M, R)} \big (\log q (M | R) - \log p (M) \big) \right. \\ \left. + \mathbb {E} _ {Z \sim q (Z | M, R)} \big (\log q (Z | M, R) - \log p (Z) \big)\right) \\ = \mathbb {E} _ {M \sim q (M | R)} \left(\log q (M | R) - \log p (M)\right) \\ + \mathbb {E} _ {M \sim q (M | R)} \left(\mathbb {E} _ {Z \sim q (Z | M, R)} \left(\log q (Z | M, R) - \log p (Z)\right)\right) \\ = \sum_ {t = 1} ^ {l _ {r}} \left(\mathbb {E} _ {M \sim q (M | R)} \left(\log q (m _ {t}) - \log p (m _ {t})\right)\right) \\ + \mathbb {E} _ {M \sim q (M | R)} \left(\sum_ {t = 1} ^ {l _ {r}} m _ {t - 1} \cdot \mathbb {E} _ {Z \sim q (Z | M, R)} \left(\log q (z _ {t}) - \log p (z _ {t})\right)\right) \\ = \sum_ {t = 1} ^ {l _ {r}} D _ {\mathrm {K L}} (q (m _ {t}) \| p (m _ {t})) \\ + \mathbb {E} _ {M \sim q (M | R)} \left(\sum_ {t = 1} ^ {l _ {r}} m _ {t - 1} \cdot D _ {\mathrm {K L}} \left(q \left(z _ {t}\right) \| p \left(z _ {t}\right)\right)\right). \tag {12} \\ \end{array}
$$
A.2 Details about the Construction of  $\tilde{\mathbf{M}}$  and  $\tilde{\mathbf{Z}}$
In this section, we provide more details about of construction of  $\tilde{M}$  and  $\tilde{Z}$ . For every response in the training set, we parse it as a syntax tree using StanfordNLP toolkit (Manning et al., 2014). The
syntax tree we obtain is in a hierarchical and nested structure. The root node of the tree represents the whole response sentence and the root node of every subtree represents a corresponding phrase, a small part of a sentence. For example, if a phrase could be divided into three parts, then the node representing the phrase has three child nodes and each represents a part of the phrase. After we acquire the parsing tree, segmentation is then carried out recursively. To be concrete, we traverse the parsing tree by deep-first search order. Every time we arrive at a node, compute the similarity $^{10}$  between the knowledge and the phrase represented by the node. If the similarity is above the threshold  $\mu_{seg}$ , we mark the phrase as a segment and search in this branch terminates. Else we continue to search the child nodes of the current node to segment at a more refined level. We use  $\tilde{M} = \{\tilde{m}_t\}_{t=1}^{l_r}$  to denote the results of segmentation labeling.
The pseudo label of module choice  $\tilde{Z} = \{z_t\}_{t=1}^{l_r}$  is tagged in a similar way to multiclass classification. Specifically, for a segment  $(r_s, \dots, r_e)$  where  $s$  and  $e$  are the start and end position of a segment respectively. If the similarity between this segment and the knowledge falls below a threshold  $\mu_{knl}$ , its pseudo label  $(z_s, \dots, z_e)$  will be set to 0. Otherwise we send the segment to a series of style discriminators one after another until the classification confidence given by a discriminator is above  $\mu_{sty_i}$  and pseudo module choice label will be set to  $i + 1$ . If all discriminators fail to classify the segment at a confidence greater than  $\mu_{sty_i}$ ,  $(z_s, \dots, z_e)$  are all 1, indicating knowledge should be expressed without particular style.
A.3 Learning Algorithm
The learning algorithm is summarized in Algorithm 1.
A.4 Details of Datasets
Training Data. We choose the Reddit Corpus published by (Li et al., 2020) as  $\mathcal{D}$  for pre-training. The data contains 842,521 context-knowledge-response triples for training and 2,737 context-knowledge-response triples for validation. On average, each dialogue contains 3.1 utterances in both sets, and the average length of the utterance is 16.0 in training and is 16.1 in validation.
Evaluation Data. We test our model on benchmarks of knowledge-grounded dialogue generation,
<sup>10</sup>We use unigram Precision to calculate the similarity.
including Wizard of Wikipedia (Wizard) (Dinan et al., 2019) and CMU Document Grounded Conversations (CMU_DOG) (Zhou et al., 2018c). Both datasets are split into training sets, validation sets, and test sets by the data owners. We follow Dinan et al. (2019) and conduct the pre-processing with the code published on  $\mathsf{ParlAI^{11}}$ . Topics in Wizard cover a wide range (1, 365 in total), and each conversation happens between a wizard who has access to the knowledge about a specific topic and an apprentice who is just eager to learn from the wizard about the topic. The test set is split into two subsets. Test Seen only contains dialogues with topics that have already appeared in the training set, while topics in Test Unseen never appear in the training set and the validation set. Different from Wizard, CMU_DOG focuses on movie domain, and besides wizard-apprentice conversations, the data also contain conversations between two workers who know the document and try to discuss the content in depth. In both datasets, only the turns where knowledge is accessible are considered in response generation. Table 5 reports the statistics of the Wizard data and the CMU_DOG data
A.5 More Implementation Details
We employ a knowledge selection (KS) module to select the top 7 related sentences in knowledge. The KS module is implemented based on Roberta-base (125M) and trained on the Reddit Corpus. Specifically, we treat the sentence which has the
highest F1 score with the response as the positive sample, and the negative sample is randomly sampled from all the other knowledge sentences. We train the KS module via maximum likelihood estimation (MLE) with a batch size of 64 and an initial learning rate of  $1e - 5$ . The threshold  $\mu_{seg}$ ,  $\mu_{knl}$ ,  $\mu_{pos}$  and  $\mu_{neg}$  in weak supervision are set as 0.9, 0.5, 0.8 and 0.8, respectively. The encoder-decoder architecture is implemented on the basis of Bart-base (139M) and trained on the Reddit Corpus with a batch size of 64 and an initial learning rate of  $5e - 6$ . The parameters for prior and posterior distributions of  $Z$  and  $M$  (i.e.,  $\theta_z$ ,  $\theta_m$ ,  $\phi_z$  and  $\phi_m$ ) are initialized randomly, and optimized with a learning rate of  $1e - 4$ . The parameters for adapters are initialized randomly and optimized with a learning rate of  $2e - 3$ . We only train the adapters for the first 1000 steps. We utilize gated recurrent units (GRUs) as the basic units in  $f_{z - \mathrm{rnn}}$ . We set the hidden size and the number of layers of RNN in our model (i.e.,  $f_{z - \mathrm{rnn}}$  and  $\psi(\cdot)$ ) as 128 and 1 respectively. The embedding size for  $Z$  is set as 128 and the adapter size is set as 64. When finetuning the model on the Wizard and CMU_DOG datasets, the learning rate and the batch size are set as  $5e - 5$  and 32 respectively. We employ greedy search in response decoding. All models are learned with Adam (Kingma and Ba, 2015) optimizer with  $\beta_1 = 0.9$  and  $\beta_2 = 0.999$ . We increase the learning rate linearly for the first 200
<sup>12</sup>We consider positive and negative sentiment style in our experiments.
11 https://github.com/facebookresearch/ParlAI/blob/master/projects/wizard_of_wikipedia
Wizard of Wikipedia	CMU_DOG
Train	Valid	Test Seen	Test Unseen	Train	Valid	Test
# Utterances	166,787	17,715	8,715	8,782	74,717	4,993	13,646
# Conversations	18,430	1,948	965	968	3,373	229	619
# Topics/Documents	1,247	599	533	58	30	30	30
Avg. # of Turns	9.0	9.1	9.0	9.1	22.2	21.8	22.0
Table 5: Statistics of the Wizard data and the CMU_DoG data.
Training Data	Models	Wizard Seen	Wizard Unseen	CMU_DOG
PPL	BLEU-1	ROUGE-L	METEOR	PPL	BLEU-1	ROUGE-L	METEOR	PPL	BLEU-1	ROUGE-L	METEOR-L
Reddit Corpus	BART	40.1	0.206	0.164	0.099	42.9	0.202	0.167	0.099	75.8	0.141	0.117	0.060
ZRKGC	41.1	0.225	0.163	0.099	42.7	0.220	0.164	0.100	53.8	0.161	0.128	0.075
Ours	35.9	0.235	0.168	0.095	38.4	0.232	0.169	0.095	60.4	0.166	0.131	0.069
Reddit Corpus +10% training data	BART	32.7	0.203	0.174	0.103	35.0	0.199	0.175	0.103	49.5	0.141	0.122	0.067
ZRKGC	29.1	0.227	0.175	0.101	31.6	0.222	0.176	0.100	38.0	0.173	0.139	0.083
Ours	28.6	0.237	0.181	0.103	30.7	0.231	0.176	0.101	40.8	0.182	0.139	0.080
Table 6: More Results about Automatic Evaluation.
steps and decrease it thereafter proportionally to the inverse square root of the step number. Early stopping on validation is adopted as a regularization strategy. All models are trained on a  $8 \times$  RTX 2080 Ti machine.
A.6 More Results about Automatic Evaluation
Table 6 reports more results about the automatic evaluation, from which we can see that our model still outperforms the baselines.
A.7 Human Evaluation
We randomly sample 300 examples from Test Seen of Wizard, and the test set of CMU_DOG respectively, and recruit 6 well-educated native speakers to do qualitative analysis on the responses generated by our model and all baselines. For each of the 300 examples, an annotator is provided with the context, the ground-truth knowledge, model responses and the associated style types. For evaluation of structure style, we defined two kinds of structure styles based on two datasets, namely the Wizard-like style  $S_{\text{wizard}}$  and the CMU_DOG-like style  $S_{\text{cmudog}}$ . While for evaluation of content style, we roughly divide content styles in two categories,  $S_{\text{pos}}$  and  $S_{\text{neg}}$  for convenience. The responses provided by different models are randomly shuffled to hide their sources. The annotators need to judge the quality of the responses from four aspects: (1) fluency: whether the response is fluent without any grammatical errors; (2) context coherence: whether the response is coherent with the context; (3) knowledge relevance: whether the re
sponse is relevant with the knowledge; and (4) style consistency: whether the response exhibits the desired style. Each annotator assigns a score from  $\{0,1,2\}$  (representing "bad", "fair" and "good" respectively) to each response for each aspect. Each response obtains four scores for aforementioned four aspects, and the agreement among all annotators is measured via Fleiss' kappa (Fleiss, 1971).
Results on Learning content style. Table 7 reports the human evaluation results on learning content style. The three models are trained on the Reddit Corpus. We can conclude that: (1) by introducing two latent variables and a number of adapters for different styles, our model can generate responses in desired content style (i.e.,  $S_{pos}$  and  $S_{neg}$ ) more accurately and achieve significant improvement on style consistency, which is consistent with the results in Table 4; (2) our model also outperforms ECM and DialogGPT on fluency, context coherency and knowledge relevance thanks to the capacity of large-scale pre-trained language models and the introduction of external knowledge respectively.
A.8 Comparison with More Baselines
We compare with models trained on full training data, and Table 8 shows the evaluation results. First, it is noted that our model outperforms KnowledgeGPT in terms of F1 by using only  $10\%$  training data on CMU_DOG, which provides a strong support for the effectiveness of the proposed model.
13The  $10\%$  training data is randomly sampled. The result is an average value of three repetitive experiments on every dataset
Models	Wizard Seen	CMU_DoG
Fluency	Context Coherence	Knowledge Relevance	Style Consistency	Kappa	Fluency	Context Coherence	Knowledge Relevance	Style Consistency	Kappa
ECM	0.85	0.94	1.02	1.24	0.65	0.96	0.95	1.18	1.08	0.72
DialoGPT	1.57	1.41	1.19	1.26	0.75	1.55	1.62	1.09	1.02	0.65
Our	1.64	1.60	1.78	1.72	0.76	1.59	1.63	1.51	1.69	0.62
Table 7: Human evaluation results on learning content style.
Training Data	Models	Wizard Seen	Wizard Unseen	CMU_DOG
F1	D-1	D-2	F1	D-1	D-2	F1	D-1	D-2
100% annotated data	TMN	15.9	0.041	0.176	14.3	0.025	0.106	9.9	0.003	0.008
SKT	19.3	0.085	0.300	16.1	0.056	0.188	-	-	-
DRD	19.3	0.065	0.252	17.9	0.046	0.177	10.7	0.010	0.044
KnowledGPT	22.0	0.141	0.431	20.5	0.094	0.290	13.5	0.023	0.113
Our Model	22.0	0.128	0.415	20.8	0.090	0.278	15.3	0.031	0.121
Reddit Corpus	BART	18.4	0.076	0.355	18.4	0.049	0.237	9.8	0.021	0.131
ZRKGC	18.9	0.055	0.246	18.8	0.037	0.179	12.2	0.015	0.094
Our Model	19.3	0.082	0.383	19.2	0.060	0.292	12.2	0.028	0.186
Reddit Corpus + 10% annotated data	Our Model	20.4	0.073	0.366	20.0	0.052	0.270	14.4	0.015	0.122
Reddit Corpus + 100% annotated data	Our Model	21.9	0.134	0.453	21.2	0.103	0.302	15.5	0.041	0.134
Table 8: Automatic evaluation results.
Second, by adjusting the structure style on a small amount of data, the gap between our model and KnowledgeGPT is further narrowed, while the improvements on ZRKGC and BART are trivial.
A.9 Ablation over Weak Supervision
To have more insights into the impact of weak supervision on the performance of our model, we compare the proposed model with the following variants: (1)-weak supervision on  $Z$ : the weak supervision on module indicator  $Z$  is removed; (2)-weak supervision on  $Z$  and  $M$ : the weak supervision on module indicator and boundary indicator is removed. Table 9 reports the evaluation results. We can conclude that (1) the weak supervision objectives significantly improve model performance; (2) the weak supervision objectives play a more crucial role on CMU_DOG, as removing them causes a dramatic drop in performance. The reason is that this dataset has more sophisticated expression styles and it is difficult to learn these styles without auxiliary supervision signals.
A.10 Ablation over Boundary Indicator
Since the module indicator is conditioned on the boundary indicator, we are curious about what will happen if the  $M$  is removed. The ablation result is shown on Figure 4. There is an evident drop on Wizard Seen and Wizard Unseen, verifying the effect of boundary indicator in assisting the module indicator. The margin is tiny on CMU_DOG,
Figure 4: Ablation results on Wizard and CMU_DOG.
perhaps because its structure feature is easier to capture, so the module indicator could works properly itself.
A.11 Case Study
This section mainly studies how different models vary in knowledge expression for the same context and background knowledge. Table 10 shows an example from the test set of CMU_DOG. This example contains the background knowledge which gives a plot from the movie, and the dialogue context which is generated by discussing the content in the knowledge. We choose the following four models to generate the response in corresponding style given the dialogue context and knowledge, and all models are pre-trained with the Reddit Corpus: (1) Wizard Model for  $S_{\text{wizard}}$ : the model fine-tuned with  $10\%$  training data in Wizard; (2) CMU_DOG Model for  $S_{\text{cmu dog}}$ : the model fine-tuned with  $10\%$  training data in CMU_DOG; (3) Positive Model for  $S_{\text{pos}}$ : the model forced to express knowledge with
Training Data	Models	Wizard Seen	Wizard Unseen	CMU_DOG
F1	D-1	D-2	F1	D-1	D-2	F1	D-1	D-2
Reddit Corpus	Our model	19.3	0.082	0.383	19.2	0.060	0.292	12.2	0.028	0.186
-weak supervision on Z	19.1	0.077	0.362	19.1	0.056	0.270	10.2	0.027	0.155
-weak supervision on Z and M	19.1	0.083	0.382	18.8	0.058	0.270	9.5	0.023	0.147
Reddit Corpus + 10% annotated data	Our model	20.4	0.073	0.366	20.0	0.052	0.270	14.4	0.015	0.122
-weak supervision on Z	19.5	0.072	0.354	19.3	0.051	0.250	13.2	0.014	0.115
-weak supervision on Z and M	19.5	0.077	0.366	19.2	0.054	0.258	13.5	0.013	0.091
Table 9: Ablation study over the weak supervision. Numbers in bold means that the improvement to variants is statistically significant (t-test with  $p$ -value  $< 0.05$ )
Knowledge	• MovieName: How to Train Your Dragon
• Back at the village, Hiccup subdues a captive dragon in his final training test in front of his father instead of killing it, but Stoick inadvertently angers the dragon into attacking.
• Toothless attempts to protect Hiccup in the ensuing panic but is instead captured by the Vikings.
• Hiccup accidentally reveals to Stoick that Toothless is capable of locating the dragons' nest.
• Stoick disowns his son and sets off for the nest with Toothless chained to the lead ship as a guide.
• The Vikings expel most of the dragons but are overwhelmed by the Red Death until Hiccup, Astrid and their fellow pupils fly in riding the training dragons from the academy and provide cover fire.
• Hiccup almost drowns trying to break Toothless free from a sinking ship but Stoick saves them both and then reconciles with his son.
• Toothless and Hiccup destroy the Red Death but Hiccup is injured in the fight.
• Hiccup regains consciousness on Berk where his lower left leg has been replaced by a prosthesis, and the Vikings and the dragons now live in harmony.
Context	User1: Now I keep thinking about phantom pain and whatnot... will he walk with his new prothesis? Was does viking prothesis look like?!
User2: ...but now they live there together! Schweet.
User1: Yes!
Ground Truth	Probably sharp and imposing!
Wizard Model	They live in a house together . It 's the same as before but with toothless and hiccup now living on berk where his lower left leg has been replaced by a prosthesis.
CMU_DOG Model	What is the new prothesis ?! It 's a Viking
Positive Model	Vikings live in harmony now have the dragons living together and are all still alive to be united.
Negative Model	Though Vikings and the dragons live in harmony, Hiccup can no longer walk with this lower left leg.
Table 10: A case from test set of CMU_DoG.
positive sentiment; (4) Negative Model for  $S_{neg}$ : the model forced to express knowledge with negative sentiment. We can see that the knowledge expression style of the Wizard Model and CMU_DOG Model are quite different. The central part of the Wizard Model response is copied from the background knowledge, which is consistent with the style of Wizard data. The response generated by CMU_DOG Model is more casual in knowledge expression, and the content is mainly related to the conversation context. Besides, responses generated by the Positive Model exhibit evident positive sentiment, while responses generated by the Negative Model show relatively negative sentiment.