SlowGuess/NAACL_2022_OCR huggingface dataset:
Deeksha Varshney†, Akshara Prabhakar‡* Asif Ekbal†
$^{\dagger}$ Department of Computer Science and Engineering, Indian Institute of Technology Patna, India  $^{\ddagger}$ Department of Information Technology, National Institute of Technology Karnataka, Surathkal
{1821cs13,asif}@iitp.ac.in
akshblr555@gmail.com
Grounding dialogue on external knowledge and interpreting linguistic patterns in dialogue history context, such as ellipsis, anaphora, and co-references is critical for dialogue comprehension and generation. In this paper, we present a novel open-domain dialogue generation model which effectively utilizes the large-scale commonsense and named entity based knowledge in addition to the unstructured topic-specific knowledge associated with each utterance. We enhance the commonsense knowledge with named entity-aware structures using co-references. Our proposed model utilizes a multi-hop attention layer to preserve the most accurate and critical parts of the dialogue history and the associated knowledge. In addition, we employ a Commonsense and Named Entity Enhanced Attention Module, which starts with the extracted triples from various sources and gradually finds the relevant supporting set of triples using multi-hop attention with the query vector obtained from the interactive dialogue-knowledge module. Empirical results on two benchmark dataset demonstrate that our model significantly outperforms the state-of-the-art methods in terms of both automatic evaluation metrics and human judgment. Our code is publicly available at https://github.com/deekshaVarshney/CNTF; https://www.iitp.ac.in/-ai-nlp-ml/resources/codes/CNTF.zip.
Neural language models usually focus on fewer language components such as sentences, phrases, or words for text analysis. However, language acts on a much broader scale - there is frequently a central theme to a conversation, and the speakers share common information in order to comprehend one another. Information is frequently reused, however to avoid overuse, same things and persons are
referred in the dialogue multiple times by using relevant expressions. A dialogue becomes coherent and speakers can understand each other when all of this information is delivered in a structured, logical, and consistent manner.
Semantic understanding of dialogues can be aided by commonsense knowledge or world facts. Additionally, as a key human language phenomena, co-reference simplifies human languages while being a significant barrier for machines to understand, particularly for pronouns, which are difficult to parse due to their weak semantic meanings (Ehrlich, 1981). Grounded response generation approaches (Ghazvininejad et al., 2018; Dinan et al., 2018) can provide replication of facts in open-domain settings, whereas commonsense knowledge is critical for creating successful interactions since socially constructed commonsense knowledge is the collection of contextual details that humans are expected to understand and use during a conversation.
Despite demonstrating efficacy in empirical evaluation, past work has a few significant drawbacks. There is no explicit representation of entities, semantic relations, or conversation structures, in particular. To solve such restrictions, asking a conversation model to identify relevant structures in dialogue histories can be used to directly test the level of dialogue understanding. We focus on named entity level knowledge in this paper, and analyze references to entities in a dialogue history context.
To ensure the generalizability of our model, we directly incorporate entities in the form of triplets, which is the most common format of modern knowledge graphs, instead of encoding it with features or rules as in conventional approaches. Take, for example, Figure 1, where the dialogue consists of eight utterances. In the third utterance, to know if there exists any relation between the director "Micheal Mann" and the movie "The Last of the Mohicans", we need to resolve the co-reference
*Work done during an internship at IIT Patna
Human Language Technologies, pages 1322 - 1335
Topic-Specific Knowledge
The Last of the Mohicans is a 1992 American epic historical drama, set in 1757 during the French and Indian War. It was directed by Michael Mann, based on James Fenimore Cooper's eponymous 1826 novel and George B. Seitz's 1936 film adaptation, owing more to the latter than the novel. The film stars Daniel Day-Lewis, Madeleine Stowe, and Jodhi May, with Russell Means, Wes Studi, Eric Schweig, and Steven Waddington in supporting roles. It was produced by Morgan Creek Pictures. The soundtrack features music by Trevor Jones and Randy Edelman, and the song "I Will Find You" by Clannad. The main theme of the film is taken from the tune "The Gael" by Scottish singer-songwriter Dougie MacLean.
Agent 1: [The Last of the Mohicans] is a 1992 American epic historical drama [it] is also one of my favorite movies.
Agent 2: Me too. I love [that movie] and the soundtrack in particular.
Agent 1: [It] was directed by Michael Mann, based on [James Fenimore Cooper's eponymous 1826] novel
Agent 2: I tried to read [the book] but gave up. I can't remember what other movies Michael Mann has directed.
Agent 1: I never got to reading [the book] but I am defenently interested.
Agent 2: I couldn't get into the rhythm of the writing. A little bit different in 1826!
Agent 1: [The movie] was produced by [Morgan Creek Pictures] and i think [they] did a great job.
Agent 2: Is [Morgan Creek] still producing movies?
Knowledge Triples
Figure 1: An example of named entity and concept based knowledge triples being used for grounding dialogues in addition to topic-specific knowledge sentences. In the conversation, various shades indicate the different co-reference clusters obtained. Blue nodes correspond to the concepts obtained from ConceptNet, red nodes correspond to the named entities obtained from the utterances in the dialogue. Named Entities other than the ones present in co-reference chains are highlighted in green in the conversation.
relationship between the pronoun [It] and the entity [The Last of the Mohicans]. Using co-reference resolution, we get an important triple for the movie "The Last of the Mohicans" viz. (The Last of the Mohicans, RelatedTo, Micheal Mann). Similarly, from the second last utterance, we obtain another triple as (The Last of the Mohicans, RelatedTo, Morgan Creek Pictures). Thus, for instance, to generate the fourth utterance "I tried to read the book but gave up. I can't remember what other movies Michael Mann has directed.", it is important for the model to know that there is a relation between the concept word "movie" and the named entities "Micheal Mann", "The Last of the Mohicans", to get a correct understanding of the dialogue context.
We create a conversational model called CNTF, Commonsense, Named Entity and Topical Knowledge Fused neural network to generate successful responses by leveraging both topic-specific document information and using structured entity and commonsense knowledge. We first construct triples based on named entity after resolving co-references in the dialogues to enhance the already existing commonsense triples obtained from the ConceptNet (Speer and Havasi, 2012). We use multi-hop attention to iterate over the multi-source information. We obtain a weighted query vector from the interactive dialogue-knowledge module, which is used to query over the dialogue, topical knowledge and the corresponding triples. In each round,
CNTF reasons on the dialogue history and knowledge sentences, using which we filter out relevant information from the dialogue context and topical knowledge. Similarly, to reason over the triples, we again iterate in multiple rounds, masking out irrelevant triples.
1. We propose CNTF, a novel knowledge grounded dialogue generation model that utilizes dialogue context, unstructured textual information, and structural knowledge to facilitate explicit reasoning.
2. We enhance the commonsense triples extracted from the ConceptNet database with named entity-aware structures using co-reference resolution.
3. We define an effective sliding window mechanism in order to remove irrelevant information from longer dialogue context and ensure efficient memory utilization. We use an interactive dialogue-knowledge module to generate a weighted query vector which captures the interactions between the conversation and the topical knowledge.
4. Through extensive qualitative and quantitative validation on publicly available datasets, we show that our model outperforms the strong baselines.
Sequence-to-sequence models (Vinyals and Le, 2015; Sutskever et al., 2014) have long been used for natural language generation (NLG) tasks. Stemming off the vanilla encoder-decoder architecture - introduced initially for neural machine translation (Shang et al., 2015), a variety of models have been developed to enhance the quality of the responses generated (Li et al., 2016a; Zhao et al., 2017; Tao et al., 2018); to effectively select the conversational context in multi-turn dialogues (Serban et al., 2016, 2017; Xing et al., 2017; Zhang et al., 2019); and to model persona while conversing (Li et al., 2016b; Zhang et al., 2018). Recent advances on dialogue systems aim at enhancing dialogue generation by making it more humanized by means of incorporating knowledge based on the dialogue context or from external sources, such as unstructured documents (Li et al., 2019; Qin et al., 2019) or knowledge graphs (Moon et al., 2019; Tuan et al., 2019).
Numerous pre-trained language models (Devlin et al., 2019; Radford et al., 2019) have been utilized for dialogue generation (Edunov et al., 2019; Zhang et al., 2020). They have been extended to leverage the knowledge from the unstructured documents and other auxiliary sources via knowledge selection and various attention fusion techniques (Zhao et al., 2020c; Cao et al., 2020). The task was explored in low-resource setting (Zhao et al., 2020b) using a disentangled response decoder, and the usability of language models itself as a knowledge base has also been investigated in Zhao et al. (2020d). An issue with language models is the noise which these introduce during knowledge selection. In order to limit the noise by generative models, term-level weighting (Zheng et al., 2021) for response generation after knowledge selection were studied. Zhao et al. (2020a) proposed a pre-training based multiple knowledge syncretic transformer that uses a single framework to integrate knowledge from multiple sources. Knowledge based end-to-end memory networks have been developed for task-oriented dialogue generation (Raghu et al., 2019; Reddy et al., 2019; Chen et al., 2019; Wang et al., 2020; Varshney and Singh, 2021) using multi-level, working, and dynamic types of memory. In Dual Dynamic Memory Network (DDMN) (Wang et al., 2020), the flow of history information during conversations is dynamically tracked to retain the important parts from both dialogue and KB, using a memory manager for each.
Prior studies (Young et al., 2018; Zhou et al., 2018a; Wu et al., 2020b) have demonstrated the feasibility of including commonsense knowledge into the dialogue systems. Further, in ConKADI (Wu et al., 2020a), felicitous facts highly relevant to the context were selected and effectively integrated in the generated response by means of fusion mechanisms. Recently, co-reference resolution has been utilized for obtaining coref-informed pre-trained models (Ye et al., 2020), task-oriented dialogue generation (Quan et al., 2019), and dialogue understanding (Zhang et al., 2021). Further, (Huang et al., 2021) demonstrated the improvement upon explicitly incorporating co-reference information to enhance the attention mechanism for the reading comprehension task.
In this paper, we show how both structured and unstructured knowledge can be used to improve the task of document-grounded dialogue generation. We propose an effective knowledge-grounded dialogue model named CNTF, which is built with multi-source heterogeneous knowledge. Experiments on knowledge-based dialogue generation benchmark datasets, viz. Wizard of Wikipedia and CMU_DOG, have shown the efficacy of our proposed approach. Our method employs a large-scale named entity enhanced commonsense knowledge network as well as a domain-specific factual knowledge base to aid in the comprehension of an utterance as well as the generation of a response using a novel mutli-hop attention based model.
Formally, let  $D = \{d_i\}_{i=1}^K$  denote a conversation composed of  $K$  dialogue turns, where  $d_i = (a_i^1, a_i^2)$  is an exchange of dialogues between the two agents. Associated with each utterance  $a_i^1$  and  $a_i^2$  are the relevant documents  $S_i^1$  and  $S_i^2$  with topic-specific knowledge. We utilize common sense and named entity oriented knowledge by creating the set of triples  $\tau = \{\tau_1, \tau_2, \dots, \tau_{|\tau|}\}$ , where  $\tau_i$  is of the form (head, relation, tail), from the following sources:
(a) extracting relations from ConceptNet for every word in the utterances (if the word is a concept-word from ConceptNet), and
(b) forming named entity based triples by using co-reference resolution method
For any arbitrary turn  $k$ , given the dialogue history  $\{d_j\}_{j=1}^{k-1}$ , the associated documents as well as
Figure 2: Proposed CNTF architecture. The dialogue encoder encodes the dialogue context in multi-turn conversation. Similarly, the knowledge encoder takes as input the document(s) associated with the utterances in the conversation. The multi-hop attention modules are used to extract relevant information from dialogue and knowledge whereas the Commonsense and Named Entity Enhanced Attention module is used to effectively incorporate the knowledge triples.
the target document  $\{S_j^1,S_j^2\}_{j = 1}^k$  , and the associated knowledge triples  $\tau$  , the objective is to generate an appropriate response  $Y = \{y_{1},y_{2},\dots,y_{|Y|}\}$  The architecture of CNTF is shown in Figure 2.
3.2.1 Dialogue Encoder
The Dialogue Encoder, that keeps track of the dialogue context in multi-turn conversations, encodes the utterances turn by turn. The input at each turn is a sequence of tokens  $x = (x_{1}, x_{2}, \dots, x_{n})$ , where  $n$  is the number of tokens. For the first turn,  $a_{1}^{1}$  is fed as input, while for the subsequent turns ( $j > 1$ ), the input is the concatenation of the previous turn's second agent's response and current turn's first agent's utterance,  $[a_{j-1}^{2}; a_{j}^{1}]$ . The encoder then exploits Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) to obtain the representations  $H_{D} = \{h_{i}\}_{i=1}^{n}$ .
Using the dialogue representations, we maintain two different states for the dialogue,  $D_{S}$  and  $D_{H}$ , which are both initialized with the encoder hidden states  $H_{D}$ , of the first turn. We then follow a sliding window mechanism to update both  $D_{S}$  and  $D_{H}$  for the succeeding turns. A window of size "l" means we concatenate hidden states of only the previous "l-1" turns. This helped in removing noise for longer dialogue contexts and saving memory.  $D_{H}$  remains fixed and stores the hidden states for the dialogue
context, while  $D_S$  gets updated at each turn, with the goal of capturing proper history information for accurate response generation.
3.2.2 Knowledge Encoder
Similarly, the Knowledge Encoder takes as input the document(s) associated with the utterances  $\nu iz$ .  $[S_{j-1}^2; S_j^1]$  for turn  $j > 1$ , else  $S_1^1$  for the first turn, truncated to a max token count of 400. We then again employ a BERT model and obtain the encoded features  $H_{Kb} = \{h_i\}_{i=1}^m$ , where  $m$  is the number of tokens in the document(s). To incorporate the external topic-specific knowledge effectively, we have knowledge states  $Kb_S$  and  $Kb_H$ . Similar to the dialogue states  $D_S$  and  $D_H$ , these are initialized with the hidden states  $H_{Kb}$  of the relevant documents associated with each utterance. Unlike the sliding window mechanism used for the dialogue states for the upcoming turns,  $Kb_S$  and  $Kb_H$  store only the current turn's hidden states obtained from the BERT based knowledge encoder.
3.3 Multi-hop Attention
We adopt the dual and dynamic graph attention mechanism (Wang et al., 2020) to mimic human's step-by-step exploring and reasoning behavior. In each step, we assume that the dialogue and knowledge states have some information to disseminate. At each hop  $r$ , we compute an attention vector  $\alpha_{t}^{(r)}$  using the query embedding  $q_{t}$  at the  $k$ -th turn us-
ing  $D_S^{(r - 1)}$  at time step  $t$ .  $D_{H,k,t}$  and the attention scores are used to obtain the context representation  $c_t^{(r)}$ .
$$
\alpha_ {k, t} ^ {(r)} = \operatorname {s o f t m a x} \left(e _ {k, t}\right) \tag {1}
$$
$$
e _ {k, t} = \left(v _ {1} ^ {(r)}\right) ^ {\prime} \tanh  \left(W _ {1} ^ {(r)} q _ {k, t} + W _ {2} ^ {(r)} D _ {S, k, t} ^ {(r - 1)}\right) \tag {2}
$$
$$
c _ {k, t} ^ {(r)} = \sum_ {j = 1} ^ {K} a _ {k, t} ^ {(r)} D _ {H, k, t} \tag {3}
$$
where  $v_{1}^{(r)}$ ,  $W_{1}^{(r)}$  and  $W_{2}^{(r)}$  are the learnable parameters.
$D_{S}$  is updated using the forget and add operations. To find more details on updating  $D_{S}$  refer to Appendix A.
3.4 Constructing Named Entity based Triples using Co-reference Resolution
To add more useful links to the already existing commonsense triples, we use the co-reference chains and named entities extracted from the dialogues. Firstly, we use AllenNLP co-reference resolution module to identify co-reference chains in the dialogue. For example, in the dialogue shown in Fig. 1, using the first co-reference chain: [The Last of the Mohicans: it, that movie, It] we rewrite the dialogue with resolved mentions in the utterances as: "[The Last of the Mohicans] is a 1992 American epic historical drama [The Last of the Mohicans] is also one of my favorite movies. Me too. I love [The Last of the Mohicans] and the soundtrack in particular. [The Last of the Mohicans] was directed by Michael Mann, based on [James Fenimore Cooper's eponymous 1826] novel and so on". We then use Spacy Named Entity tagging module to recognize named entities from the augmented dialogue. Simultaneously, we also identify all the concept words using ConceptNet in the newly formed dialogue.
The new set of triples is obtained using the named entities and concepts as nodes, and the corresponding edges are built as follows:
(a) between every pair of named entities that appear in the same dialogue, and
(b) between a named entity node and other concepts within the same dialogue.
We may note that resolving the co-references first and then extracting named entities ensures that entities across multiple utterances are connected in a certain way. Also, we explicitly form a triplet having the RelatedTo relation as it suits well for
most of the cases because it indicates a relation between the two named entities and their different references or aliases across the conversation.
3.5 Commonsense and Named Entity Enhanced Attention Module
For each dialogue, the final set of triples is composed of both commonsense and named entity based triples. We obtain triples' head and tail entity embedding from the trainable embedding layers i.e.  $E = \mathrm{emb\_layer}(\tau)$ . Formally, a query is used to loop over the triple embedding and compute the attention weights at each hop  $p$ .
$$
\alpha_ {k, t} ^ {(p)} = \operatorname {s o f t m a x} \left(q _ {t} ^ {(p - 1)} E ^ {(p - 1)}\right) \tag {4}
$$
Finally, the weighted context for knowledge triples,  $(c^T)^{(p)}$ , is obtained by weighting the current set of triple embedding,  $E^{(p)}$  using the attention scores,  $\mathbf{a}^{(p)}$ . A query update mechanism is used, where the query embeddings are updated using the weighted triple embeddings of the current step.
$$
\left(c _ {k, t} ^ {T}\right) ^ {p} = \sum_ {j = 1} ^ {n} \mathbf {a} _ {k, t} ^ {p} E ^ {p} \tag {5}
$$
$$
q _ {t} ^ {p} = q _ {t} ^ {p - 1} + \left(c _ {k, t} ^ {T}\right) ^ {p} \tag {6}
$$
3.6 Decoder
3.6.1 Interactive Dialogue-Knowledge Module
As each utterance is linked to topic-specific unstructured knowledge, we employ an interactive mechanism to attend to both the dialogue and the knowledge sentences. We can improve information extraction from dialogue as well as knowledge hidden states for response generation by using the encoded weighted dialogue context as the initial query vector  $q_{t}$ . To obtain the weighted dialogue context  $WH_{D}$ , we apply the multi-hop attention as described in Section 3.3 between  $H_{D}$  and  $H_{K}$  which are the hidden states received from the dialogue and knowledge encoder, respectively.
We use a GRU based decoder to generate responses word by word, and initialize the initial hidden states of the decoder with  $WH_{D}$ . Then at time step  $t$ , the decoder state  $s_t$  can be updated as
$$
s _ {t} = \operatorname {G R U} \left(e \left(y _ {t - 1}\right), s _ {t - 1}\right) \tag {7}
$$
where  $e(y_{t - 1})$  is the embedding of the previous word  $y_{t - 1}$ . Here,  $s_t$  is regarded as the updated
query vector, which is used to attend to the dialogue, topic-specific knowledge and the structured knowledge triples, and obtain the weighted context, knowledge and triple representation as  $c_t^D$ ,  $c_t^K$  and  $c_t^T$ , respectively.
3.6.2 Fusion Block
The probability distribution over the vocabulary  $P_{g}(y_{t})$  words is obtained by fusing  $c_{t}^{D}, c_{t}^{K}, c_{t}^{T}$  and the decoder state,  $s_{t}$ , and then passing them through a softmax layer.
$$
P _ {g} (y _ {t}) = \operatorname {s o f t m a x} \left(W _ {5} \left[ s _ {t}; c _ {t} ^ {D}; c _ {t} ^ {K}; c _ {t} ^ {T} \right]\right) \tag {8}
$$
where  $W_{5}$  is a trainable parameter.
3.6.3 Copy Block
In particular, a word at time step  $t$  is either generated from the vocabulary or copied from either the dialogue history, knowledge history, or using entities from the triples. Following the copy mechanism (Gulcehre et al., 2016), the attention scores are viewed as the probability to form the copy distribution. We use the attention score  $\alpha_{k,t}^{D}$  of the dialogue and  $\alpha_{k,t}^{Kb}$  of the unstructured knowledge at the last round viz.  $P_{D}(y_{t} = w) = \sum_{tj: w_{tj} = w} \alpha_{k,t}^{D}$ ;  $P_{Kb}(y_{t} = w) = \sum_{tj: w_{tj} = w} \alpha_{k,t}^{Kb}$ . The copy distribution over the triples is given by  $P^{T}(y_{t} = w) = \sum_{tj: w_{j}^{t} = w} \alpha_{k,t}^{T}$ . We use the soft gates  $g_{1}, g_{2}$  and  $g_{3}$  to control whether a word is generated from the vocabulary or it is being copied by combining  $P_{g}(y_{t})$ ,  $P_{D}(y_{t})$ ,  $P_{Kb}(y_{t})$ , and  $P_{T}(y_{t})$ :
$$
g _ {1} = \operatorname {S i g m o i d} \left(W _ {8} \left[ s _ {t}; c _ {t} ^ {D} \right] + b _ {2}\right) \tag {9}
$$
$$
P _ {k n} (y _ {t}) = g _ {1} P _ {g} (y _ {t}) + (1 - g _ {1}) P _ {D} (y _ {t}) \tag {10}
$$
$$
g _ {2} = \operatorname {S i g m o i d} \left(W _ {9} \left[ s _ {t}; c _ {t} ^ {K} \right] + b _ {3}\right) \tag {11}
$$
$$
P _ {t p} (y _ {t}) = g _ {2} P _ {K b} (y _ {t}) + (1 - g _ {2}) P _ {k n} (y _ {t}) \tag {12}
$$
$$
g _ {3} = \operatorname {S i g m o i d} \left(W _ {1 0} \left[ s _ {t}; c _ {t} ^ {T} \right] + b _ {4}\right) \tag {13}
$$
$$
P \left(y _ {t}\right) = g _ {3} P _ {T} \left(y _ {t}\right) + \left(1 - g _ {3}\right) P _ {t p} \left(y _ {t}\right) \tag {14}
$$
where,  $W_{8}, W_{9}, W_{10}$  are the parameters to be learned.
Therefore, the decoder loss is the cross-entropy between the predicted distribution  $P(y_{t})$  and the reference distribution,  $p_t$ , denoted as  $Loss = -\sum p_t \log (P(y_t))$ .
4 Datasets and Experimental Setup
In this section, we present the details of the datasets and the other experimental setups. Implementation details can be found in Appendix C.
4.1.1 Knowledge Grounded Dialogue Dataset