SlowGuess/NAACL_2021_OCR huggingface dataset:
Shrimai Prabhumoye $^{1*}$ , Kazuma Hashimoto $^{2}$ , Yingbo Zhou $^{2}$ , Alan W Black $^{1}$ , Ruslan Salakhutdinov $^{1}$
$^{1}$ Carnegie Mellon University,  $^{2}$ Salesforce Research
{sprabhum@cs.cmu.edu，khashimoto@salesforce.com}
Document grounded generation is the task of using the information provided in a document to improve text generation. This work focuses on two different document grounded generation tasks: Wikipedia Update Generation task and Dialogue response generation. Our work introduces two novel adaptations of large scale pre-trained encoder-decoder models focusing on building context driven representation of the document and enabling specific attention to the information in the document. Additionally, we provide a stronger BART baseline for these tasks. Our proposed techniques outperform existing methods on both automated (at least  $48\%$  increase in BLEU-4 points) and human evaluation for closeness to reference and relevance to the document. Furthermore, we perform comprehensive manual inspection of the generated output and categorize errors to provide insights into future directions in modeling these tasks.
Natural language generation (NLG) systems are increasingly expected to be naturalistic, contentful, and situation-aware due to their popularity and pervasiveness in human life (Reiter and Dale, 2000; Mitchell et al., 2014). This is particularly relevant in dialogue systems (Zhang et al., 2018a; Niu and Bansal, 2018), machine translation systems (Mirkin and Meunier, 2015; Rabinovich et al., 2017), story generation (Fan et al., 2018; Yao et al., 2019), and question answering systems (Gatius, 2017; Reddy et al., 2019).
Despite these mainstream applications, NLG systems face the challenges of being bland, devoid of content, generating generic outputs and hallucinating information (Wiseman et al., 2017; Li et al., 2016; Holtzman et al., 2020; Welleck et al., 2020). Grounding the generation in different modalities
like images (Huang et al., 2016; Mostafazadeh et al., 2017; Shuster et al., 2018), videos (Palaskar et al., 2019; Regneri et al., 2013), and structured data (Banik et al., 2013; Gardent et al., 2017) alleviates some of these issues. Generating natural language from schematized or structured data such as database records, slot-value pair, and Wikipedia Infobox has been explored in prior work (Mei et al., 2016; Wen et al., 2015; Lebret et al., 2016). Although useful, these tasks encounter difficulties such as general applicability (databases may not be available for all domains) and are constrained by the available resources (size of the database).
Document grounded generation mitigates these applicability issues by exploiting the vast availability of data in unstructured form (e.g. books, encyclopedias, news articles, and Wikipedia articles). This enhances the applicability of document grounded generation to a wide range of domains with limited (or no) availability of structured data. Hence, recent work has focused on defining new tasks and carving the scope of the problems (Liu et al., 2018; Prabhumoye et al., 2019; Faltings et al., 2020; Zhou et al., 2018; Dinan et al., 2018).
We focus on two different document grounded generation tasks: (1) Wikipedia Update Generation task (Prabhumoye et al., 2019) and (2) Dialogue response generation (Zhou et al., 2018; Dinan et al., 2018). Prior work has studied these two tasks independently and focused on task specific modeling techniques (Zhao et al., 2020a,b; Prabhumoye et al., 2019). Our work unifies these tasks and formally shows the similarity in them: presence of a context and a document to ground the information in the generation process.
Our work introduces two novel improvements to the architectures of large scale pre-trained models (Lewis et al., 2019; Raffel et al., 2019): (1) we focus on building context driven representation of the document, where the context is taken into account while building the representation of the
* Work done during internship at Salesforce.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4274-4287 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Document Grounded Generation: An example of a conversation that is grounded in the given document (text in green shows information from the document that was used to generate the response).
document, and (2) during generation we provide specific attention to the information in the document. We provide a stronger BART-based (Lewis et al., 2019) baseline for these tasks. This work shows that pre-trained models albeit good at text generation, can be further improved by providing grounding specific improvements.
Our main contributions are the two new proposed techniques for the document grounded generation tasks (§3.2 and §3.3). We also provide a new baseline which is stronger than the previous state-of-the-art methods (Zhao et al., 2020b; Prabhumoye et al., 2019) for the two tasks. We formally show how the two independent tasks studied in this paper are identical and similar modeling techniques can be used to solve them (§3). Automated and human evaluation results on three different datasets demonstrate substantial improvements (§5.1 and §5.2). Specifically, we achieve an improvement of 19.7 BLEU-4 points compared to Zhao et al. (2020b) on the dialogue generation task. Additionally, significant gains are observed in BLEU-4 compared to BART-based baseline. A comprehensive manual analysis of the generated output is presented in this work which paves way for future work (§6). We will release our code on Github.
Our task is to generate text given a context and a source of content (document). Additionally, the generated text should coherently fit the context and contain information from the document. We focus on content present in unstructured form in documents to ground text generation. Figure 1 illustrates such an example. Dialogue response generation is traditionally conditioned on the dialogue context (Vinyals and Le, 2015; Li et al., 2016).
As Figure 1 demonstrates, the generative model is conditioned on both the document as well as the dialogue context. Note that the context and document play different roles in impacting the generation – the context sets the background while the document provides the content necessary to generate the text.
Formally, each sample  $\mathbf{i}$  of our task is defined as a tuple  $(\mathbf{d}_i,\mathbf{c}_i,\mathbf{x}_i)$  containing context  $\mathbf{c}_i$ , document  $\mathbf{d}_i$  and text  $\mathbf{x}_i$  to be generated. Note that each  $\mathbf{d}_i$  can be a single document or a set of documents. The task is to generate  $\mathbf{x}_i$  such that it coherently follows  $\mathbf{c}_i$  and contains information from  $\mathbf{d}_i$ . The task can be modeled as the following conditional text generation model:  $p_{\theta}(\mathbf{x}_i|\mathbf{c}_i,\mathbf{d}_i)$ , where  $\theta$  is a set of model parameters.
Figure 1 illustrates that the generator has to account for two inputs the dialogue context  $\mathbf{c}_i$  (shown in blue) and the document  $\mathbf{d}_i$  (shown in red) to generate the response  $\mathbf{x}_i$  grounded in  $\mathbf{d}_i$  (text shown in green). If the generative model was only conditioned on dialogue context, then it could produce generic responses like "Do you think they did the right thing?" or "Yes, I agree." or hallucinate information like "Yes, and the Times published it on the front page." These which would be appropriate to the given context but are devoid of content or contain wrong information. Document grounded models are capable of responding with interesting facts like "Yes, but it was dangerous for the white house to ban the post from the white house."
A natural way to model  $p_{\theta}(\mathbf{x}_i|\mathbf{c}_i,\mathbf{d}_i)$  is to train an encoder-decoder model using cross-entropy loss  $-\log p_{\theta}$  with respect to the ground-truth output text. We discuss two ways of building effective representations for encoder-decoder models to focus
on  $\mathbf{d}_i$ : (1) combine encoder representations of  $\mathbf{c}_i$  and  $\mathbf{d}_i$ , (2) include an additional attention multi-head at each layer of the transformer to specifically focus on the content in  $\mathbf{d}_i$ .
3.1 Baselines
Low-Res: Zhao et al. (2020a) introduce the state-of-the-art model for document grounded dialogue generation. As described in (§2), the chat history serves as the context  $\mathbf{c}_i$  and  $\mathbf{x}_i$  is the response to be generated. Zhao et al. (2020a) pre-train their architecture on the dialogue specific Reddit (Dziri et al., 2018) dataset and learn separate parameters for encoding  $\mathbf{c}_i$  and  $\mathbf{d}_i$ . Zhao et al. (2020a) further has three components—context processor, knowledge processor and the language model, each of which build distributions over the vocabulary space. A decoding manager is then trained to generate a token based on these three distributions.
Instead, we employ the recent success of the pre-trained encoder-decoder models (Lewis et al., 2019; Raffel et al., 2019) by using BART (Lewis et al., 2019). One key component of solving this task is to build a representation of the content in the document/s  $\mathbf{d}_i$  that is not present in the context  $\mathbf{c}_i$ . We want to leverage the SelfAttention feature of transformers (Vaswani et al., 2017) to build such a representation. Since, we use a pre-trained language model as our baseline architecture, we don't use a separate language model component. Instead, we direct our efforts to focus on effectively combining  $\mathbf{c}_i$  and  $\mathbf{d}_i$ .
Content Transfer: Prabhumoye et al. (2019) provide benchmark numbers for the Wikipedia Update Generation task (§2). They explore multiple generative as well as extractive models with and without context. We use their best performing Context Informed LSTM-based encoder-decoder model as baseline. This model concatenates the tokens of the context  $\mathbf{c}_i$  and the document  $\mathbf{d}_i$  and passes the concatenated sequence to the encoder.
BART: The most straightforward way of using BART for modeling  $p_{\theta}(\mathbf{x}_i|\mathbf{c}_i,\mathbf{d}_i)$  is to concatenate the tokens of the context  $\mathbf{c}_i$  and the document  $\mathbf{d}_i$  and pass the concatenated sequence  $([\mathbf{c}_i;\mathbf{d}_i])$  to the BART encoder, and then the decoder generates  $\mathbf{x}_i$ . This is our BART baseline; it already has the advantage of the highly contextualized representations of  $\mathbf{c}_i$  and  $\mathbf{d}_i$  in comparison with Zhao et al. (2020a). However, fully relying on the self-attention mech
anism over the concatenated text would lack the explicit distinction between  $\mathbf{c}_i$  and  $\mathbf{d}_i$ .
Below, we describe two techniques to efficiently build document focused representations. In Figure 1, the method which adds an additional CrossAttention multi-head sub-layer to each layer of the transformer is shown. This attention multi-head specifically focuses on the document  $\mathbf{d}_i$ .
3.2 Context Driven Representation
We propose to use two encoder representations for  $\mathbf{c}_i$  and  $\mathbf{d}_i$ . We first define  $\mathbf{h}_d = \mathrm{Encoder}([ \mathbf{c}_i; \mathbf{d}_i ])$  to get a contextualized representation of  $\mathbf{d}_i$ , conditioning on the context  $\mathbf{c}_i$ .  $\mathbf{h}_d$  is equivalent to the representation used in the BART baseline. We then apply the same BART encoder to the context alone:  $\mathbf{h}_c = \mathrm{Encoder}(\mathbf{c}_i)$ . We finally concatenate the encoder outputs  $\mathbf{h} = [\mathbf{h}_c; \mathbf{h}_d]$  before passing them to the BART decoder. This  $\mathbf{h}$  is Context Driven Representation (CoDR). This method does not require any model architectural modification, and instead the encoder and decoder are fined-tuned to use the multiple input representations.
3.3 Document Headed Attention
In this section, we describe Document Headed Attention (DoHA) to further enhance the use of the multiple input representations. A decoder in transformer encoder-decoder models (Vaswani et al., 2017) has two types of multi-head attention mechanism, SelfAttention and CrossAttention with the source sequence. SelfAttention module allows each position in the decoder to attend to all positions in the decoder up to and including that position. CrossAttention module performs multi-head attention over the output of the encoder stack and attends over the source sequence. While our CoDR method uses the two different source representations,  $\mathbf{h}_c$  and  $\mathbf{h}_d$ , CrosstAttention is still shared over the concatenated representation  $\mathbf{h}$ .
In this work, we add an additional multi-head attention CrossAttention_Doc to specifically attend over the tokens of the document, while the original CrossAttention (named as CrosstAttention_Cxt), only attends over the tokens of the context. Each of the multi-heads are of the form:
$$
\operatorname {M u l t i H e a d} (Q, K, V) = \left[ \mathbf {H} _ {1}; \dots ; \mathbf {H} _ {m} \right] \mathbf {W} ^ {\mathbf {o}},
$$
$$
\mathbf {H} _ {j} = \operatorname {A t t e n t i o n} (Q \mathbf {W} _ {\mathbf {j}} ^ {\mathbf {Q}}, K \mathbf {W} _ {\mathbf {j}} ^ {\mathbf {K}}, V \mathbf {W} _ {\mathbf {j}} ^ {\mathbf {V}}).
$$
The multi-head function receives three inputs - a query  $Q$ , key  $K$  and value  $V$ .  $\mathbf{W}^{\mathrm{o}}$  is an output projection of the concatenated outputs of the attention
heads. Each  $\mathbf{H}_j$  is the output of a single attention head and  $\mathbf{W_j^Q}$ ,  $\mathbf{W_j^K}$  and  $\mathbf{W_j^V}$  are head-specific projections for  $Q$ ,  $K$ , and  $V$ , respectively.
Hence, the multi-head CrossAttention_Doc is defined by:
$$
\begin{array}{l} \text {C r o s s A t t e n t i o n \_ D o c} (Q, K, V) \\ = \left[ \mathbf {H} _ {1}; \dots ; \mathbf {H} _ {m} \right] \mathbf {W} ^ {\mathrm {d o}}, \\ \mathbf {H} _ {j} = \text {A t t e n t i o n} (Q \mathbf {W} _ {\mathbf {j}} ^ {\mathrm {d Q}}, K \mathbf {W} _ {\mathbf {j}} ^ {\mathrm {d K}}, V \mathbf {W} _ {\mathbf {j}} ^ {\mathrm {d V}}), \\ \end{array}
$$
where  $\mathbf{W}^{\mathrm{do}}$ ,  $\mathbf{W}_{\mathbf{j}}^{\mathrm{dQ}}$ ,  $\mathbf{W}_{\mathbf{j}}^{\mathrm{dK}}$  and  $\mathbf{W}_{\mathbf{j}}^{\mathrm{dV}}$  are parameters trained specifically to focus on document. The parameters of CrossAttention_Doc are initialized with those of CrossAttention_Cxt.
Each decoder layer follows the following sequence of functions:
$$
\begin{array}{l} \mathbf {h} = \mathcal {F} (\text {S e l f A t t e n t i o n} (\mathbf {h} _ {\mathbf {x}}, \mathbf {h} _ {\mathbf {x}}, \mathbf {h} _ {\mathbf {x}})), \\ \mathbf {h} = \mathcal {F} (\text {C r o s s A t t e n t i o n} _ {-} C x t (\mathbf {h}, \mathbf {h} _ {\mathbf {c}}, \mathbf {h} _ {\mathbf {c}})), \\ \mathbf {h} = \mathcal {F} (\text {C r o s s A t t e n t i o n \_ D o c} (\mathbf {h}, \mathbf {h} _ {\mathbf {d}}, \mathbf {h} _ {\mathbf {d}})), \\ \mathbf {h} = \mathcal {F} (\mathsf {F F N} (\mathbf {h})), \\ \end{array}
$$
where  $\mathcal{F}(\mathbf{h})$  is a sequence of LayerNorm(residual + dropout(h)), followed by residual  $= \mathbf{h}$ . We integrate the additional attention head CrossAttention_Doc by passing the output of the previous attention head CrossAttention_Cxt as query. Unlike the weighted attention fusion techniques (Cao et al., 2020), this technique of fusing the additional attention head is novel and useful as it does not require any additional parameters for the fusion.
4 Document Grounded Generation Tasks
Document grounded generation can leverage unstructured data as a source of grounding and can hence be applied to a variety of generation tasks such as dialogue responses, Wikipedia articles, reports and legal argument. This work focuses on Wikipedia Update Generation and Dialogue Response Generation which have been studied independently in prior work. We discuss the similarities in these two tasks and design a common modeling technique for them.
4.1 Wikipedia Update Generation
This task involves generating an update for Wikipedia context given a news article (Prabhumoye et al., 2019). The dataset was collected by parsing Wikipedia articles and Common Crawl for news articles. It consists tuples of the form  $(\mathbf{d}_i, \mathbf{c}_i, \mathbf{x}_i)$ , where the grounding document  $\mathbf{d}_i$  is
the news article which contains information for the reference update  $\mathbf{x}_i$ .  $\mathbf{x}_i$  is written by a Wikipedia editor as an update to the Wikipedia context  $\mathbf{c}_i$ . The goal of the task is to generate  $\mathbf{x}_i$  given the context  $\mathbf{c}_i$  and the document  $\mathbf{d}_i$ .
4.2 Dialogue Response Generation
Goal oriented dialogues have been traditionally grounded in structured sources like slot-value pairs and databases (Wei et al., 2018; Rastogi et al., 2020). Open domain dialogue generation on the other hand faces the issue of "hallucinating" information (Ghazvininejad et al., 2018). Hence we study open domain dialogue generation which is grounded in documents as a source of information.
CMU_DOG: The CMU Document Grounded Conversations dataset consists of human-human conversations collected over Amazon Mechanical Turk (Zhou et al., 2018). The conversations are grounded in a document provided to the crowd-workers and focuses only on movies. The dataset uses Wikipedia descriptions of movies for grounding the conversations. The dataset consists tuples of the form  $(\mathbf{d}_i, \mathbf{c}_i, \mathbf{x}_i)$ , where  $\mathbf{d}_i$  is a section (or passage) extracted from Wikipedia,  $\mathbf{c}_i$  is dialogue history (or context) and  $\mathbf{x}_i$  is the reference response. The response  $\mathbf{x}_i$  is grounded in  $\mathbf{d}_i$  and coherently follows the conversation  $\mathbf{c}_i$ .
Wizard of Wikipedia: This dataset also consists of human-human conversations collected over Amazon Mechanical Turk and are grounded in passages extracted from Wikipedia (Dinan et al., 2018). These conversations are grounded in a diverse range of topics (totally 1365) which are further split into seen and unseen topics during training and validation. At each step of the dialogue the wizard has access to a set of passages of knowledge which may be relevant to the given dialogue context. The dataset is created by retrieving the top 7 articles (first paragraph only) that are most relevant to the last two turns of dialogue (by wizard and apprentice). Hence, the dataset consists tuples of the form  $(\mathbf{d}_i,\mathbf{c}_i,\mathbf{x}_i)$ , where  $\mathbf{d}_i$  is a list of 7 passages relevant to the conversation,  $\mathbf{c}_i$  is dialogue history (or context) and  $\mathbf{x}_i$  is the reference response.
The above three tasks consists tuples of the form  $(\mathbf{d}_i, \mathbf{c}_i, \mathbf{x}_i)$ , where  $\mathbf{x}_i$  coherently follows  $\mathbf{c}_i$  and is grounded in  $\mathbf{d}_i$ . Hence, we can use common modeling techniques (§3) for these tasks.
<sup>1</sup>Data statistics are shown in Appendix (§A)
Model	BLEU-1	BLEU-2	BLEU-3	BLEU-4	Rouge-L	Meteor	F1
Wikipedia Update Generation
Content Transfer (Prabhumoye et al., 2019)	10.18	4.42	2.20	1.23	10.08	6.21	12.6
BART (baseline)	21.72	14.71	11.28	9.20	22.39	12.90	27.5
CoDR	25.15	17.33	13.56	11.31	23.48	14.38	29.0
DoHA	25.11	17.04	13.17	10.86	23.49	14.28	29.1
CMU_DoG
Low-Res (Zhao et al., 2020a)	15.00	5.70	2.50	1.20	-	-	10.7
BART (baseline)	23.78	19.27	17.66	16.91	19.30	12.59	21.7
CoDR	26.86	22.75	21.30	20.68	20.41	14.47	22.7
DoHA	27.33	23.05	21.55	20.90	20.44	14.55	22.8
Wizard of Wikipedia (Seen)
Low-Res (Zhao et al., 2020a)	21.80	11.50	7.50	5.50	-	-	18.0
BART (baseline)	23.92	14.62	10.24	7.75	21.41	15.45	31.1
CoDR	24.00	14.98	10.64	8.18	21.82	15.71	31.8
DoHA	24.14	15.08	10.68	8.18	21.76	15.89	31.8
Wizard of Wikipedia (Unseen)
Low-Res (Zhao et al., 2020a)	20.70	10.10	6.20	4.30	-	-	16.5
BART (baseline)	21.88	12.54	8.44	6.23	19.14	14.03	28.2
CoDR	21.84	12.74	8.60	6.35	19.50	14.22	29.0
DoHA	22.31	13.04	8.89	6.60	19.62	14.47	29.0
Table 1: Results on the automated metrics for the three datasets
We implement all our models with the transformers tool (Wolf et al., 2019), and the details are in  $\S A$ .
5.1 Automated Evaluation
Following prior work (Prabhumoye et al., 2019; Zhao et al., 2020a), we evaluate our system-generated sentences against the reference sentences on Rouge-L (Lin, 2004), BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2011) metrics. Rouge-L measures the longest common subsequence between the generated sentence and the reference, capturing both lexical selection and word order. METEOR also uses synonyms and stemmed forms of the words in candidate and reference sentences, and thus may be better at quantifying semantic similarities. Additionally, we present F1 which indicates the unigram overlap between the generated output and the reference sentence.
Table 1 shows that the BART baseline outperforms previous state-of-the-art models (Zhao et al., 2020a; Prabhumoye et al., 2019) on all three tasks. It demonstrates that both our improvements DoHA and CoDR perform better than our BART baseline on all metrics and for all three tasks. Notably, we see an improvement of 19.7 BLEU-4 points
on the CMU_DOG dataset compared to Zhao et al. (2020a) which was pre-trained on dialogue specific data; and an improvement on 8.9 BLEU-4 points on the Wikipedia Update Generation compared to (Prabhumoye et al., 2019). We also see substantial improvements (23.6% increase in BLEU-4 for CMU_DOG) compared to the simple BART baseline for the three tasks. In general, DoHA performs slightly better than CoDR on the three tasks.
We follow the human evaluation guidelines mentioned in (Prabhumoye et al., 2019) and evaluate the system generated sentences on three dimensions: (1) closeness of the generated sentences to the references, (2) relevance of the generated sentences to the context and document, and (3) fluency of the generated sentences.
Closeness: The automatic metrics like BLEU, METEOR, and Rouge-L may not be tolerant towards linguistic variations in generated outputs. Hence, we perform a human evaluation to measures how accurately the generated sentence reflects the information in the reference. The annotators are provided with the reference sentence and the generated outputs of two systems labeled  $A$  and  $B$  in a randomized order. The annotators were instructed to "Pick the option which is closest in meaning with the reference option." The annotators could
2We use NLG evaluation toolkit (Sharma et al., 2017) from https://github.com/Maluuba/nlg-eval  
3We use the code published at https://github.com/facebookresearch/ParlAI/blob/master/parlai/core/metrics.py to calculate unigram F1.
4We use NLG eval script for (Prabhumoye et al., 2019)
Task	BART v CoDR	BART v DoHA	DoHA v CoDR
BART	NoPref	CoDR	BART	NoPref	DoHA	DoHA	NoPref	CoDR
Wikipedia Update Generation
Closeness	33.3	36.7	30.0	25.5	46.7	27.8	32.2	42.2	25.6
Relevance	18.9	54.4	26.7	24.4	45.6	30.0	33.3	38.9	27.8
CMU_DOG
Closeness	15.6	58.8	25.6	30.0	42.2	27.8	33.3	44.5	22.2
Relevance	22.2	43.4	34.4	23.3	42.3	34.4	34.4	42.3	23.3
Wizard of Wikipedia (seen)
Closeness	36.7	40.0	23.3	28.9	31.1	40.0	40.5	31.7	27.8
Relevance	24.2	51.6	24.2	32.2	35.6	32.2	28.9	46.7	24.4
Wizard of Wikipedia (unseen)
Closeness	23.3	47.8	28.9	44.4	20.0	35.6	21.1	63.3	15.6
Relevance	27.8	47.8	24.4	30.0	43.3	26.6	23.3	41.1	35.6
Table 2: Human evaluation results depicting percentage of times a model was picked (NoPref=No Preference)
select system  $A$  or  $B$ , or indicate that neither was preferred by picking the third option  $C$ . This is a simple evaluation task though potentially biased toward the sole reference.