SlowGuess/NAACL_2022_OCR huggingface dataset:
Sha Li $^{1*}$ , Madhi Namazifar $^{2}$ , Di Jin $^{2}$ , Mohit Bansal $^{2}$ , Heng Ji $^{2}$ , Yang Liu $^{2}$ , Dilek Hakkani-Tur $^{2}$
<sup>1</sup>University of Illinois at Urbana-Champaign, <sup>2</sup>Amazon Alexa AI shal2@illinois.edu {madhinam, djinamzn, mobansal, jihj, yangliud, hakkanit}@amazon.com
Providing conversation models with background knowledge has been shown to make open-domain dialogues more informative and engaging. Existing models treat knowledge selection as a sentence ranking or classification problem where each sentence is handled individually, ignoring the internal semantic connection among sentences in background document. In this work, we propose to automatically convert the background knowledge documents into document semantic graphs and then perform knowledge selection over such graphs. Our document semantic graphs preserve sentence-level information through the use of sentence nodes and provide concept connections between sentences. We apply multitask learning for sentence-level knowledge selection and concept-level knowledge selection jointly, and show that it improves sentence-level selection. Our experiments show that our semantic graph based knowledge selection improves over sentence selection baselines for both the knowledge selection task and the end-to-end response generation task on Holle (Moghe et al., 2018) and improves generalization on unseen topics in WoW (Dinan et al., 2019).
Natural language generation models have seen great success in their ability to hold open-domain dialogues without the need for manual injection of domain knowledge. However, such models often degenerate to uninteresting and repetitive responses (Holtzman et al., 2020), or hallucinate false knowledge (Roller et al., 2021; Shuster et al., 2021). To avoid such phenomena, one solution
is to provide the conversation model with relevant knowledge to guide the response generation (Parthasarathi and Pineau, 2018; Ghazvininejad et al., 2018; Dinan et al., 2019). Figure 1 illustrates such knowledge grounded generation.
Relevant knowledge is often presented in the form of documents (Moghe et al., 2018; Zhou et al., 2018b; Ghazvininejad et al., 2018; Dinan et al., 2019; Gopalakrishnan et al., 2019) and the task of identifying the appropriate knowledge snippet for each turn is formulated as a sentence classification or ranking task (Dinan et al., 2019). Although more advanced methods have been proposed by modeling knowledge as a latent variable (Kim et al., 2020; Chen et al., 2020), or tracking topic shift (Meng et al., 2021), they abide by the setting of sentence-level selection. This setting has two inherent drawbacks: (1) it ignores the semantic connections between sentences and (2) it imposes an artificial constraint over the knowledge boundary.
A document is not simply a bag of sentences, in fact, it is the underlying semantic connections and structures that make the composition of sentences meaningful. Two examples of such connections are coreference links and predicate-argument structures. These connections are vital to the understanding of the document and also beneficial to knowledge selection. In many cases, we can draw information from multiple sentences to create the response, breaking the knowledge boundary. For instance, in Figure 2, the connections among the character "Rango", the plot point "water shortage" and the name "Django" help us generate a response with a smooth topic transition.
A related line of work (Liu et al., 2018; Moon et al., 2019; Xu et al., 2020; Young et al., 2018; Zhou et al., 2018a) that seemingly overcomes the aforementioned issues is knowledge selection from existing knowledge graphs (KGs) such as
Another example would be discourse relations between sentences, which we do not explore here.
Work done as an intern at Amazon Alexa AI.
<sup>1</sup>See https://www.amazon.science/publications/enhanced-knowledge-selection-for-grounded-dialogues- via-document-semantic-graphsfor an updated paper with information about code and resources.
Human Language Technologies, pages 2810 - 2823
Figure 1: An example of knowledge-grounded dialog. Semantic connections between sentences improve coherence and not imposing knowledge boundaries allows the system to utilize multiple knowledge snippets. The used knowledge is in bold. *The jaguar shark is a character.
Wikidata (Vrandecić and Krötzsch, 2014), DBpedia (Lehmann et al., 2015), and ConceptNet (Speer et al., 2017). If the character "Rango" were in the KG, it would have been represented as an entity node and be connected to respective events. On the KG, we are also free to select as many concepts as needed, without being restricted to a single sentence as the source. However, KGs are known to have limited coverage of real world entities, let alone emerging entities in works of fiction such as books and movies (Razniewski et al., 2016).
Hence, to bridge these two worlds of sentence-based knowledge selection and KG-based knowledge selection, we introduce knowledge selection using document semantic graphs. These graphs are automatically constructed from documents, aiming to preserve the document content while enhancing the document representation with semantic connections. To create such a document semantic graph, we first obtain the Abstract Meaning Representation (AMR) (Banarescu et al., 2013) for each sentence. AMR detects entities, captures predicate-argument structures, and provides a layer of abstraction from words to concepts. Compared to existing knowledge base construction methods, AMR covers a wide range of relations and fine-grained se
matic roles, and can fully reflect the semantics of the source text. Since AMR graphs only represent single sentences, we utilize coreference resolution tools to detect coreferential entity nodes and merge them to build graphs for documents. On top of this content representation, we also add sentence nodes and passage nodes to reflect the structure of the document. This allows for traversal across the graph by narrative order or concept association.
Given the document semantic graph, knowledge selection can be seen as identifying relevant nodes on the graph, sentence nodes or concept nodes. As knowledge selection in dialog models is conditioned on the dialog context, for each dialog turn, we create a dialog-aware graph derived from the document graph. It contains context nodes representing contextualized versions of the sentence and concept nodes. We design an edge-aware graph neural network model to propagate information along the dialog-aware graph and finally score the context nodes (or concept nodes) on their relevance to the dialog turn (as shown in Figure 4).
We validate our model on two widely used datasets HollE (Moghe et al., 2018) and Wizard of Wikipedia (Dinan et al., 2019) by constructing a semantic graph from relevant background documents for each dialog. The use of document graphs improves both knowledge selection and response generation quality on HollE and boosts generalization to unseen topics for WoW. From our ablation tests we find that in terms of the graph structure, the key component is the use of coreference edges that stitches sentences together.
Our contributions include: (1) We propose to perform knowledge selection from document semantic graphs that are automatically constructed from source documents and can reflect the implicit semantic connections between sentences without being limited to a pre-defined set of entities and relations as KGs do. Our approach bridges the gap between sentence-based knowledge selection and KG-based knowledge selection. (2) We show that joint selection over sentences and concepts can model more complex relations between sentences and boost sentence selection performance. (3) We build a pipeline for converting documents (document collections) into semantic graphs through the use of AMR parsing and coreference. We hope that
<sup>4</sup>On WoW we use the passages retrieved from the first turn for graph construction. Our method will need to be extended to online graph construction to support per-turn retrieval of documents.
3In AMR, every node is a concept. This includes events, objects, attributes, etc.
Figure 2: The pipeline for generating responses based on a given knowledge source.
our tool can facilitate future work on graph-based representations of documents.
We show an overview of our knowledge-grounded dialog system in Figure 2. The system consists of three modules, namely semantic graph construction, knowledge selection and response generation.
2.1 Document Semantic Graph Construction
We first process the sentences in the background knowledge documents using the Stack Transformer AMR parser (Fernandez Astudillo et al., 2020) to obtain sentence-level AMR graphs. Based on the AMR output, we consider all of the concepts that serve as the core roles (agent, recipient, instrument etc.) for a predicate as mention candidates. Then, we run a document-level entity coreference resolution system (Wen et al., 2021) to resolve coreference links between such mentions. When joining sentence-level AMR graphs to form the document graph, entity mentions that are predicted to be coreferential are merged into one node, and we keep the longest mention as the node's canonical name. We show an example of our constructed document semantic graph in Figure 3.
On top of this content representation, we also add additional nodes to represent documents (or passages) and sentences. A document (or passage) node is linked to sentence nodes that are from this document (or passage). Each sentence node is directly connected to all the concept nodes that originate from that sentence. In addition, we add edges
Figure 3: Part of the document semantic graph for the shown plot. The graph includes the source node (white rectangle), the sentence nodes (green circles), and the concept nodes (yellow and blue rectangles). Directional edges with labels (e.g., A0, A1) are from AMR parsing, dotted edges are from the document structure.
between neighboring sentences following the narrative order in the document.
Since each node is grounded in text, in order to create embeddings for the document semantic graph, we initialize the embedding of each node with their contextual embeddings from a frozen pretrained language model RoBERTa (Liu et al., 2019a). For sentence nodes, we use the embedding of the [CLS] token. For concept nodes, we average the embeddings of the tokens in the span. Note that the document semantic graphs can be created offline and indexed by topics to be used at knowledge selection inference time.
2.2 Knowledge Selection
The task of knowledge selection is to identify relevant knowledge snippets that can be used to produce an appropriate and informative response for each turn. Since our document semantic graph is based on the background knowledge source alone, we first create a dialog-aware graph that is conditioned on the given dialog turn. We then encode the dialog-aware graph by an edge-aware graph attention network and predict relevance scores for sentences and concepts as shown in Figure 4.
Dialog-Aware Graph. The dialog-aware graph is a copy of the document semantic graph with additional context nodes  $(c)$ , each representing a dialog-contextualized knowledge sentence. For each candidate knowledge sentence  $s_i$ , to obtain the em
Figure 4: The knowledge selection model. We encode the dialog context using a pretrained language model and represent the dialog context along with each candidate sentence as a context node. We then use an edge-aware graph attention network to encode the dialog-aware graph. Finally, we classify each node on the graph to be relevant or not based on the learned node embedding, effectively performing both sentence selection and concept selection. The selected nodes are outlined in black.
bedding  $h_c$  of the context node  $c_i$ , we encode the dialog context  $x$  and the candidate knowledge sentence  $s_i$  through a pretrained language model  $f_{\mathrm{LM}}$ . We define the dialog context as the most recent two turns in the dialog history.
$$
h _ {c _ {i}} = \text {P o o l i n g} \left(f _ {\mathrm {L M}} ([ s _ {i}; x ])\right) \tag {1}
$$
For the pooling operation, we simply take the first token (namely the [CLS] token) as the representation for the sequence. Since we want to enable message passing between the context node and the rest of the graph, we add an edge between the context node  $c_{i}$  and the sentence node  $s_{i}$ .
Edge-Aware Graph Attention Network. At this point, although our dialog-aware graph captures both the dialog context and the knowledge source, there is no interaction between the two. To this end, we apply an edge-aware graph attention network (EGAT) model to allow information to be propagated along the graph. Note that our dialog-aware graph is a heterogeneous network with multiple node types and edge types. To capture the semantics of the node and edge types, we use an extension of the graph attention network (Velickovic et al., 2018) that includes edge type embeddings  $h_{T(e)}$  and node type embeddings  $h_{T(v)}$  (Yasunaga et al., 2021). These embeddings are learnt along with the model parameters and are used to compute the vector "message" that is passed along edges.
In general, a graph neural network consists of  $L$  layers with shared parameters. We denote the initial embeddings for each node as  $h^0$ . Each layer  $l$  involves a round of nodes sending out "messages" to their neighbors and then aggregating the received "messages" to update their own embeddings from
$h^l$  to  $h^{l + 1}$ . Consider a pair of nodes  $s$  and  $t$  with embeddings  $h_s$  and  $h_t$  respectively, the message  $m_{s\rightarrow t}$  that is passed from  $s$  to  $t$  through edge  $e$  is computed as the sum of the edge-aware message and the node-aware message, where  $W_{v}$  and  $W_{e}$  are projection matrices.
$$
m _ {s \rightarrow t} = W _ {v} ([ h _ {s} ^ {l}; h _ {T (v)} ]) + W _ {e} h _ {T (e)} \quad (2)
$$
Then we compute the attention weight  $\alpha_{s\rightarrow t}$  from node  $s$  to node  $t$  as:
$$
q _ {s} = W _ {q} ([ h _ {s} ^ {l}; h _ {T (s)} ])
$$
$$
k _ {t} = W _ {k} \left(\left[ h _ {t} ^ {l}; h _ {T (t)}; h _ {T (e)} \right]\right) \tag {3}
$$
$$
\alpha_ {s \rightarrow t} = \operatorname {S o f t m a x} _ {s \in \mathcal {N} _ {t}} \left(\frac {q _ {s} ^ {T} k _ {t}}{\sqrt {D}}\right)
$$
Here  $W_{q}$  and  $W_{k}$  are learnt projection matrices and  $D$  is the embedding dimension of  $h_{s}$ .  $\mathcal{N}_t$  is the neighbor node set of node  $t$ . Finally, the messages from the surrounding neighbors are aggregated to compute the updated node embedding  $h_t^{l + 1}$ .
$$
h _ {t} ^ {l + 1} = \mathrm {G E L U} \left(\mathrm {M L P} (\sum_ {s \in \mathcal {N} (t)} \alpha_ {s \rightarrow t} m _ {s \rightarrow t}) + h _ {t} ^ {l}\right)
$$
After  $L$  layers, we obtain embeddings for our context nodes  $h_c^L$ , sentence nodes  $h_s^L$  and concept nodes  $h_n^L$ .
Knowledge Selection Training. For each context node  $c$  that represents a pair of the knowledge sentence and dialog context, we compute their relevance score as
$$
\operatorname {s c o r e} (c) = \operatorname {M L P} \left(\left[ h _ {c} ^ {L}; h _ {c} ^ {0} \right]\right) \tag {4}
$$
For each concept node  $n$ , we compute its relevance score as
$$
\operatorname {s c o r e} (n) = \sigma \left(\operatorname {M L P} \left(h _ {n} ^ {L}\right)\right) \tag {5}
$$
where  $\sigma$  is the sigmoid function.
Each context node  $c$  needs to be encoded with the language model  $f_{LM}$ , but we are unable to fit all context nodes into memory. Hence, during training, we randomly sample  $k$  negatives for each positive knowledge sentence and compute cross-entropy loss over the samples.
$$
\mathcal {L} _ {c} = - \log \frac {\exp \left(\operatorname {s c o r e} \left(c ^ {+}\right)\right)}{\exp_ {c \in \{c ^ {+} \} \cup C ^ {-}} \left(\operatorname {s c o r e} (c)\right)} \tag {6}
$$
For concept nodes, we treat knowledge selection as a binary classification problem and compute the binary cross entropy loss.
$$
\mathcal {L} _ {n} = - \frac {1}{N} \sum_ {n \in G} r _ {n} \log \operatorname {s c o r e} (n) \tag {7}
$$
Here  $r_n \in \{0,1\}$  is the relevance label for the vertex  $n$ , and  $N$  is the total number of concept nodes. When the dataset does not directly provide concept-level labels for training, we derive them from the ground truth knowledge snippet by assigning any concept that is mentioned in the snippet with a relevant label  $r_n = 1$ . The overall loss is the weighted sum of the above sentence-level and the concept-level loss:
$$
\mathcal {L} = \mathcal {L} _ {c} + \beta \mathcal {L} _ {n} \tag {8}
$$
During inference, we compute  $\mathrm{score}(c)$  for all knowledge sentence candidates and take the highest scored sentence for knowledge grounded response generation.
2.3 Response Generation
We fine-tune a left-to-right language model GPT2 (Radford et al., 2019) to perform response generation given the dialog context  $x$  and the chosen knowledge snippet  $\hat{s}$ .
$$
y = \operatorname {G P T 2} ([ \hat {s}; x ]) \tag {9}
$$
During training we use teacher-forcing and use the ground truth knowledge snippet. This response generation model is independent from the knowledge selection model and trained with negative log-likelihood loss.
We evaluate our model on two publicly available datasets: Wizard of Wikipedia (Dinan et al., 2019)
Dataset	Train	Dev	Test
HollE	Dialogs	7,228	930	913
# turns	34,486	4,388	4,318
WoW	Dialogs	17,629	941/936	924/952
# turns	22,715	3257/3085	3104/3298
Table 1: Dataset statistics for WoW and HollE. For WoW, the first column is the seen split and the second column is the unseen split.
and Holl-E (Moghe et al., 2018). Both datasets are in English.
Wizard of Wikipedia (WoW) is an open-domain dialog dataset, spanning multiple topics including famous people, works of art, hobbies, etc. The test set in WoW consists of two splits that are named "Test Seen" and "Test Unseen" based on the overlap of topics with the training set. In order to build our document graph, we use the selected topic passage and the passages retrieved in the first turn as background knowledge.
Holl-E is a movie domain dialog dataset. Each dialog discusses one movie, and the background knowledge includes the plot, reviews, comments and a fact table. Holl-E additionally provides multiple references for the test set so we report performance for both single and multiple references.
Knowledge Selection. We only use the turns that utilize knowledge for training and prediction. To map the ground truth knowledge to a set of concept nodes, we choose all nodes with mention offsets contained within the span. We acquire the sentence-level labels following (Kim et al., 2020).
We use Roberta-base (Liu et al., 2019a) as the language model  $f_{LM}$ . We set  $k = 5$  for negative sampling. The EGAT model is trained with 200 hidden dimensions and 2 layers. Edge features and node features are represented with 20 dimensional vectors. We train our model with a batch size of 16 and learning rate  $3e^{-5}$  for 3 epochs.
Response Generation. Our response generation model is based on GPT2 (Radford et al., 2019) and is further fine-tuned with a batch size of 16 and learning rate of  $3e^{-5}$ . We truncate the dialog context to 128 tokens. During inference, we adopt top-k and top-p sampling with  $k = 20$  and  $p = 0.95$ . The maximum generation length is limited to 286 tokens, including the input tokens.
On average, we have 60 knowledge sentences per turn.
For knowledge selection, we also implemented the following two baseline methods:
- Roberta Ranking. We use a cross-encoder based on Roberta to represent the dialog context and the knowledge candidate, and a classification layer on top of it.
- Graph Paths. This model is built on top of the previous model. The graph paths are from the document semantic graph and obtained by breadth-first traversal starting at the candidate context node. In order to utilize the graph paths, we linearize it into tuples of (subject, predicate, object) or (modifier, subject) according to the AMR edge label and concatenate it with the candidate sentence.
For the end-to-end pipeline, we use the GPT2 response generation with our knowledge selection module and the two methods above. In addition, we compare against the following previous methods:
- Transformer MemNet (Dinan et al., 2019) is the combination of a Transformer memory network for knowledge selection and another Transformer decoder for generation.
- E2E BERT is a variant of the previous model using BERT (Devlin et al., 2019).
- Sequential Knowledge Transformer (SKT) (Kim et al., 2020) models knowledge as a latent variable and considers the posterior distribution of knowledge given the response.
- SKT+PIPM+KDBTS (Chen et al., 2020) is an improvement upon SKT with an additional Posterior Information Prediction Module (PIPM) and trained with knowledge distillation.
- Mixed Initiative Knowledge Selection (MIKe) (Meng et al., 2021) uses two knowledge selection modules to capture user-driven turns and system-driven turns respectively.
3.4 Evaluation Metrics
Knowledge Selection. To compare with previous methods, we use Accuracy, or Precision@1 as the main metric for evaluating knowledge selection. Additionally, we compute sentence ranking metrics, namely the mean average precision (MAP) and mean reciprocal rank  $(\mathrm{MRR})^6$  for more fine-grained analysis of knowledge selection quality.
Model	Single Reference	Multiple Reference
MAP	Acc	MAP	MRR	Acc
Ranking	0.493	34.3	0.527	0.526	45.3
Graph Paths	0.497	35.0	0.527	0.579	45.8
Ours	0.513	37.7**	0.514	0.580	46.1
Table 2: Knowledge selection results on the HollE dataset. For single references, MRR is the same as MAP. Acc is reported in percentage%. ** indicates significance compared to the second best model with  $p < 0.005$  under the paired t-test.
Model	Test Seen	Test Unseen
MAP	Acc	MAP	Acc
Ranking	0.472	30.1	0.436	26.3
Graph Paths	0.469	29.5	0.436	26.4
Ours	0.469	29.4	0.486	30.8**
Table 3: Knowledge selection results on WoW using the topic passage and passages retrieved at the first turn. Acc is reported in percentage%. ** indicates significance compared to the second best model with  $p < {0.005}$  under the paired t-test.
Response Generation. For automatic evaluation of responses, we use ROUGE-1, ROUGE-2 and ROUGE-L metrics (Lin, 2004).<sup>7</sup> As our response generation model is trained with gold-standard knowledge, we only report perplexity scores when using gold-standard knowledge, as a measure for the quality of the response generator alone.
For our human evaluation, we randomly sample 200 turns from the output of MIKe (Meng et al., 2021), our ranking model and our graph-based model. Annotators are asked to select which system's response is the best among the three (allowing for ties), and which system's knowledge is the most relevant. In addition, annotators score each response based on whether it is appropriate, knowledgeable and engaging on a scale of 1-4. Our annotators agreed with each other  $54.2\%$  on a single system and  $91.7\%$  when accounting for ties. The Krippendorff's alpha score for the normalized appropriate/knowledgeable/engaging scores is 0.537/0.634/0.470.
3.5 Main Results
We show our knowledge selection results in Table 2 and 3, and end-to-end results in Table 4 and 6.
From Table 2 we can see that our document semantic graph is helpful for the knowledge se
https://github.com/usnistgov/trec_eval
<sup>7</sup>We use the torchmetrics package, which follows rouge-score package Python ROUGE implementation.
Model	Single Reference	Multiple Reference
R1	R2	RL	R1	R2	RL
Transformer MemNet (Dinan et al., 2019)	20.1	10.3	-	24.3	12.8	-
E2E BERT †	25.9	18.3	-	31.1	22.7	-
SKT (Kim et al., 2020)	29.8	23.1	-	36.5	29.7	-
SKT+PIPM+KDBTS (Chen et al., 2020)	30.8	23.9	-	37.7	30.7	-
MIKe (Meng et al., 2021)	37.78	25.31	32.82	44.06	31.92	38.91
GPT2 + Ranking	40.22	31.78	38.73	47.53	39.31	45.89
GPT2 + Graph Paths	40.76	32.32	39.12	47.71	39.33	45.90
GPT2 + Graph Selection	42.49	34.37	41.01	47.89	39.58	46.25
GPT2 + Gold knowledge	75.92	72.82	75.37	75.92	72.82	75.37