SlowGuess/NAACL_2022_OCR huggingface dataset:
Rachit Bansal $^{1*}$  Milan Aggarwal $^{2}$  Sumit Bhatia $^{2}$  Jivat Neet Kaur $^{3*}$  Balaji Krishnamurthy $^{2}$
$^{1}$  Delhi Technological University  $^{2}$  Media and Data Science Research Lab, Adobe, India  $^{3}$  Microsoft Research, India
racbansa@gmail.com, {milaggar, sumbhati, kbalaji}@adobe.com  
jivatneet@gmail.com
Pre-trained Language Models (PTLMs) have been shown to perform well on natural language tasks. Many prior works have leveraged structured commonsense present in the form of entities linked through labeled relations in Knowledge Graphs (KGs) to assist PTLMs. Retrieval approaches use KG as a separate static module which limits coverage since KGs contain finite knowledge. Generative methods train PTLMs on KG triples to improve the scale at which knowledge can be obtained. However, training on symbolic KG entities limits their applicability in tasks involving natural language text where they ignore overall context. To mitigate this, we propose a CommonSense Contextualizer (CoSe-Co) conditioned on sentences as input to make it generically usable in tasks for generating knowledge relevant to the overall context of input text. To train CoSe-Co, we propose a novel dataset comprising of sentence and commonsense knowledge pairs. The knowledge inferred by CoSe-Co is diverse and contain novel entities not present in the underlying KG. We augment generated knowledge in Multi-Choice QA and Open-ended CommonSense Reasoning tasks leading to improvements over current best methods on CSQA, ARC, QASC and OBQA datasets. We also demonstrate its applicability in improving performance of a baseline model for paraphrase generation task.
While dealing with natural language text, commonsense allows humans to expand salient concepts and infer additional information. For example, by reading a sign like Men at Work on a road, we implicitly know to slow down our vehicles, look carefully for workers. This implicit process of using common sense to make logical inferences is critical to natural language understanding (Xie and
Pu, 2021). A natural question to ask then is how we can incorporate common sense in now-ubiquitous language models (LMs) (Devlin et al., 2019; Radford et al., 2018a; Raffel et al., 2019).
There have been various efforts (Bao et al., 2016; Feng et al., 2020; Wang et al., 2020b) to leverage structured knowledge present in commonsense knowledge graphs - KGs (we use KG as a shorthand for Commonsense Knowledge Graph) (Xie and Pu, 2021). Such works have primarily focused on either retrieving or generating required knowledge. Retrieval methods rely heavily on structure of downstream task like multi-choice question answering (QA) to leverage knowledge in a KG (Yasunaga et al., 2021) and are not applicable beyond a specific task. Further, retrieval can restrict total knowledge that can be garnered since static KGs lack coverage due to sparsity (Bordes et al., 2013; Guu et al., 2015). The other body of work addresses this comprising of generative methods that learn commonsense through training a LM on symbolic entities and relations between them in a KG. They have either been designed for KG completion (Bosselut et al., 2019), i.e. generate tail entity of a KG triple given head entity and relation, or to generate commonsense paths connecting a pair of entities which suffer from two shortcomings. Firstly, applying such methods in downstream tasks require entity extraction from text as a prerequisite step and secondly, they generate knowledge between entity pairs ignoring overall context of sentence (Wang et al., 2020b). Hence, applying such methods is sub-optimal since most NLP tasks comprise of sentences. Further, being trained on entities, applying them directly on sentences is infeasible and lead to train-inference input type mismatch.
To address these limitations, we propose CommonSense Contextualizer - CoSe-Co, a generative framework which generates relevant commonsense knowledge given natural language sentence as input. We condition it on sentences to
*Work done as an intern at the Media and Data Science Research Lab, Adobe, India
Human Language Technologies, pages 1128 - 1143
make it learn to incorporate overall text context and enable it to dynamically select entities/phrases from an input sentence as well as output novel yet relevant entities as part of commonsense inferences generated. We consider commonsense knowledge in the form of paths, i.e., sequence of entities connected through relations. We first create sentence-path paired dataset by - 1) sampling paths from an underlying KG; 2) sampling a subset of entities from a path; and 3) retrieving & filtering sentences (from a sentence corpus) that are semantically similar to the path. The paired data is then used to train a generative language model to generate a path given a sentence as input.
To analyse the usefulness of generated commonsense, we augment it in various downstream tasks. The reasoning ability of NLP systems is commonly analysed using QA. Hence, we choose two such tasks: 1) Multi-Choice QA, where given a question and set of choices, the model has to identify the most appropriate answer choice. However, often more than one choice is a suitable answer. To mitigate this, 2) OpenCSR (Open-ended CommonSense Reasoning) (Lin et al., 2021a) was proposed, where each question is labeled with a set of answers which have to be generated without choices. We also show applicability of CoSe-Co in improving performance on paraphrase generation task ( $\S 4.5$ ).
Our contributions can be summarised as:
1. We propose a CommonSense Contextualizer (CoSe-Co) to generate knowledge relevant to overall context of given natural language text. CoSe-Co is conditioned on sentence as input to make it generically usable in tasks without relying on entity extraction.
2. We devise a method to extract sentence-relevant commonsense knowledge paths and create the first sentence-path paired dataset. We release the dataset and make it available to the community along with the trained models and corresponding code<sup>1</sup>.
3. Since CoSe-Co is based on generative LM, it infers relevant and diverse knowledge containing novel entities not present in the underlying KG (§4.2). Augmenting generated knowledge in Multi-Choice QA (§4.3) and OpenCSR (§4.4) tasks leads to improvements over current SoTA methods. Further, it is observed that CoSe-Co helps in generalising better in low training data regime.
Commonsense Knowledge Graphs (KGs) are structured knowledge sources comprising of entity nodes in the form of symbolic natural language phrases connected through relations (Speer et al., 2017; Sap et al., 2019a; Ilievski et al., 2021; Zhang et al., 2020). The knowledge in KGs is leveraged to provide additional context in NLP tasks (Bao et al., 2016; Sun et al., 2018; Lin et al., 2019) and perform explainable structured reasoning (Ren* et al., 2020; Ren and Leskovec, 2020). Additionally, a variety of Natural Language Inference (NLI) and generation tasks requiring commonsense reasoning have been proposed over the years (Zellers et al., 2018; Talmor et al., 2019; Sap et al., 2019b; Lin et al., 2020, 2021a,b). Pre-trained language models (PTLMs) (Devlin et al., 2019) trained over large text corpus have been shown to possess textual knowledge (Jiang et al., 2020; Petroni et al., 2019; Roberts et al., 2020) and semantic understanding (Li et al., 2021). Consequently, they have been used for reasoning where they perform well to some extent (Bhagavatula et al., 2020; Huang et al., 2019). However, it remains unclear whether this performance can be genuinely attributed to reasoning capability or if it is due to unknown data correlation (Mitra et al., 2019; Niven and Kao, 2019; Kassner and Schütze, 2020; Zhou et al., 2020).
Due to this, various LM + KG systems have been explored (Feng et al., 2020; Wang et al., 2019; Lv et al., 2020) to combine broad textual coverage of LMs with KG's structured reasoning capability. Early works on KG guided QA retrieve sub-graph relevant to question entities but suffer noise due to irrelevant nodes (Bao et al., 2016; Sun et al., 2018). Hybrid graph network based methods generate missing edges in the retrieved sub-graph while filtering out irrelevant edges (Yan et al., 2020). Graph Neural Networks (GNNs) have been used to model embeddings of KG nodes (Wang et al., 2020a). More recently, Yasunaga et al. (2021) proposed an improved framework (QA-GNN) leveraging a static KG by unifying GNN based KG entity embeddings with LM based QA representations. Although, such frameworks extract relevant evidence from a KG, it undesirably restricts knowledge that can be garnered since knowledge source is static and might lack coverage due to sparsity (Bordes et al., 2013; Guu et al., 2015). Contrarily, we train a generative model on a given KG to enable it to dynamically generate relevant commonsense infer
'https://linktr.ee/coseco
Figure 1: Our proposed approach consists of: (1) Path to Sentence Alignment to create the training data for CoSe-Co, (2) Training a CommonSense Contextualizer (CoSe-Co) to generate commonsense inferences relevant to a given natural language sentence. CoSe-Co can be used to infer knowledge in downstream task.
ences making it more generalizable and scalable.
Bosselut et al. (2019) cast commonsense acquisition by LMs as KG completion. They propose COMET, a GPT (Radford et al., 2018b) based framework to generate tail entity given head and relation in a KG triple as input. Owing to training on symbolic KG nodes, using COMET in downstream tasks involving natural language text is not straightforward. Specifically, it requires extracting entities from text as a prerequisite (Becker et al., 2021). Further, training on single triples makes its application in tasks requiring multi-hop reasoning challenging due to large relation search space (Bosselut et al., 2021). To address this, Path Generator (PGQA) was proposed to generate commonsense paths between entities pair (Wang et al., 2020b). Designed for multi-choice QA, they extract question entities and generate paths between each question entity and answer choice pair. Even though generated paths are multi-hop, training on entities limits applying it directly on sentences due to train-inference input type mismatch. Further, being conditioned only on question-choice entity pairs, paths are generated ignoring overall question context. To mitigate this, we design CoSe-Co as a generic framework to dynamically generate multi-hop commonsense inference given natural language sentence as input. Separately, retrieval methods have been explored to search relevant sentences to generate text corresponding to concepts (Wang et al., 2021). Different from this task, we retrieve sentences relevant to paths in a KG to create paired sentence-path data.
3 Proposed CoSe-Co Framework
Problem Setting Given a commonsense knowledge graph  $\mathcal{G} = (\mathcal{E},\mathcal{R})$  , where  $\mathcal{E}$  is the set of entity nodes and  $\mathcal{R}$  is the set of labeled directed rela
tional edges between entities, we aim to model a CommonSense Contextualizer (CoSe-Co) which generates a set of commonsense inferences in the form of paths derived using  $\mathcal{G}$ , that are relevant to a natural language text given as input. It is desirable that such a generative commonsense knowledge model should be generic, task agnostic, and takes into account the overall context of language input while generating commonsense. Since most tasks comprise of text in the form of sentences, we model the input to CoSe-Co as a sentence. In order to train such a model, a dataset is required which comprises of mappings of the form  $\{(s_1,p_1),(s_2,p_2),\ldots,(s_N,p_N)\}$ , where  $s_j$  and  $p_j$  are relevant sentence-commonsense inference path pair. However, no existing dataset consists of such mappings. To bridge this gap, we first devise a methodology to create a dataset  $\mathcal{D}$  comprising of sentences paired with relevant commonsense inference paths. Broadly, we first extract a large corpus  $\mathcal{C}$  constituting sentences  $\{s_1,s_2,\dots,s_{|C|}\}$ . Subsequently, we sample a set of paths  $\mathcal{P} = \{p_1,p_2,\dots,p_{|\mathcal{P}|}\}$  from  $\mathcal{G}$  such that each  $p\in \mathcal{P}$  is of the form  $p = \{e_1,r_1,e_2,r_2,\dots,e_{|p| + 1}\}$ , where  $e_i\in \mathcal{E}$  and  $r_i\in \mathcal{R}$ . For each  $p\in \mathcal{P}$ , a set of contextually and semantically relevant sentences  $S\subset \mathcal{C}$  is retrieved and mapped to  $p$ . We then train a generative LM based commonsense knowledge model using  $\mathcal{D}$ . During inference, given a sentence  $s'$ , it generates commonsense paths of the form  $p' = \{e_1',r_1',e_2',r_2',\dots,e_{|p'| + 1}'\}$  such that  $e_i'\in \mathcal{E}'$  and  $r_i'\in \mathcal{R}$ . Here,  $\mathcal{E}' = \mathcal{E} \cup \mathcal{E}_{novel}$  where  $\mathcal{E}_{novel}$  are novel entities not present in  $\mathcal{G}$ . These include phrases present in an input sentence but not in  $\mathcal{E}$  as well as entirely novel entities which the pre-trained LM based backbone enables it to generate through transfer learning. The generated commonsense inference paths from CoSe-Co can then be used to augment context in downstream tasks. An overview
Figure 2: Obtaining the sentence-path paired dataset. We begin with paths from the knowledge graph and employ a two-step matching and filtering process to obtain relevant paired sentences from the given text corpora. Here we accompany each step with corresponding examples that we observed.
of our framework is shown in Figure 1.
3.1 Sentence-Path Paired Dataset Creation
In order to train CoSe-Co, we create a novel dataset comprising of related sentence-commonsense path pairs. To obtain set  $\mathcal{P}$ , we perform random walk in  $\mathcal{G}$  to extract multi-hop paths of the form  $p = \{e_1, r_1, e_2, r_2, \dots, e_{|p| + 1}\}$ , where the number of hops, denoted as path length  $|p|$ , is in range  $[l_1, l_2]$ . To avoid noisy paths which do not convey useful information, we employ relational heuristics in  $\mathcal{P}$  (described in appendix E.1). Separately, the sentence corpus  $\mathcal{C}$  is indexed using Apache Solr which is queried to retrieve sentences relevant to a path. We now explain this in detail.
Broadly, we map each path  $p \in \mathcal{P}$  to a set of sentences  $S \subset \mathcal{C}$  based on semantic similarity and overlap between entities in  $p$  and sentences. For this, consider a path  $p = \{e_1, r_1, e_2, \dots, e_{|p| + 1}\}$ . To ensure that retrieved sentences are similar to  $p$ , we devise two types of query templates -  $Q1$  and  $Q2$  which are used to create multiple queries per path while querying Solr. We design  $Q1$  to capture relation information between entities in  $p$  in addition to entities themselves. Specifically, we extract non-contiguous entity-relation triples of the form  $\{(e_i, r_i, e_{i + 2})\}$  and  $\{(e_i, r_{i + 1}, e_{i + 2})\}$ . Here, we do not query entire path while retrieving sentences to ensure better coverage since we observed that no sentence exists which contains all entities and relations present in a given path. In  $Q2$ , we extract queries comprising of connected entities pairs  $\{(e_i, e_{i + 1})\}$ . For each query  $q$  obtained from  $p$  according to  $Q1$  and  $Q2$ , we query Solr and select sentences containing entities present in  $q$ . Subsequently, we rank retrieved sentences based on similarity between sentence embedding and embedded representation of the corresponding query  $q$  (including the relation in case of  $Q1$ ). The embeddings are obtained using SBERT (Reimers and Gurevych, 2019) since it is trained on siamese
objective to learn semantically meaningful representations. Based on the ranking, we retain a maximum of top  $\mathrm{K}^{\prime}$  ( $= 10$ ) sentences to ensure most semantically relevant sentences-path pairs are obtained and also to prevent CoSe-Co from getting biased towards generating particular paths. One thing to notice is that even though sentences are retrieved using templated sub-parts within a path, the retrieved sentences are finally paired up with the entire path and later used to train a generative commonsense model that learns to generate the path given sentence as input. Figure 2 illustrates the entire sentence-path pairing process using an example from the dataset.
Using queries of type  $Q1$  templates enables us to retrieve sentences that are relatively more semantically related to the overall path. For instance, consider a path 'violin hasproperty strings _hasprequisite guitar atlocation concert'. Sentences retrieved using queries like {strings, atlocation, concert} (of the form  $(e_i, r_{i+1}, e_{i+2})$ ) are more likely to be related to other entities in the path such as 'guitar'. Further, sentences that contain entities that are not directly connected in the corresponding path induce an inductive bias in CoSe-Co to generate paths that consist of intermediate entities which connect them. We perform ablations regarding query templates in §4.3.1. We study quality of the generated dataset to check for possible data leaks and relevance between sentence-path pairs
We determine the extent of n-gram overlap between questions in the CSQA test set and sentences in our sentence-path training set as indicators of any possible data leakage. For this, we obtain the set of n-grams in a question, determine the sentence in the training set with which the question has maximum matching n-grams and divide the matching n-gram count with the total number of n-grams in the question. Finally, this fraction is averaged over all the questions in the test split of CSQA. Following this scheme, an overlap of 0.15 is observed for
1-grams, 0.07 for 2-grams, 0.002 for 3-grams, and 0.00 for 4-grams which shows that the extent of overlap is very less (on a scale of 0 to 1). Further, we noted that 1-gram overlap does not necessarily indicate leakage. For instance, consider CSQA test question - 'If a person is tired how can they be refreshed?' . Even though, it has matching 1-grams with the sentence- 'a person may feel tired without having engaged in any physical activity', but it can be noted that they have an entirely different context. From the low n-gram overlap values, we conclude that extent of leakage is negligible.
To gauge the degree of relevance between the final set of sentence-path pairs, we measure the cosine similarity between the S-BERT embeddings of the complete path and the corresponding sentence in the dataset. We observe a high normalized cosine similarity score of 0.783 when averaged over all sentence-path pairs in training dataset which shows that sentence and corresponding path pairs are semantically related.
3.2 Sentence  $\rightarrow$  Commonsense Generator
The sentence-commonsense paired dataset  $\mathcal{D}$  obtained in §3.1 is used to train a path generator model CoSe-Co $\theta$  to generate commonsense inference path  $p$  relevant to the input sentence  $s$ . For this, we initialise the parameters  $\theta$  of CoSe-Co with weights of a generative pre-trained LM as backbone (eg. T5, GPT etc). Consider T5-base (Raffel et al., 2019) as backbone, given a sentence  $s = \{x_1^s,x_2^s,\dots,x_{|s|}^s\}$  comprising of a sequence of tokens, it is processed by T5 encoder  $E_{\theta_1}$  to give a sequence of outputs  $O_{E} = \{o_{1}^{E},o_{2}^{E},\dots,o_{|s|}^{E}\}$ . T5 decoder  $D_{\theta_2}$  is trained to sequentially generate the corresponding path tokens  $p = \{x_1^p,x_2^p,\dots,x_N^p\}$ . During the decoding phase at time step  $t$ ,  $D_{\theta_2}$  is jointly conditioned on encoder outputs  $O_{E}$  and past tokens  $x_{< t}^{p}$  in the path  $p$  while generating current path token  $x_{t}^{p}$ .  $E_{\theta_1}$  and  $D_{\theta_2}$ , where  $\theta = \theta_1\bigcup \theta_2$ , are jointly optimized by minimizing loss  $\mathcal{L}$ :
$$
\begin{array}{l} \mathcal {L} = - \sum_ {t = 1} ^ {N} \log P \left(x _ {t} ^ {p} \mid x _ {<   t} ^ {p}, O _ {E}\right), \text {w h e r e} \\ P (x _ {t} ^ {p} | x _ {<   t} ^ {p}, O _ {E}) = \mathsf {C o S e - C o} _ {\theta} (s, x _ {<   t} ^ {p}) \\ \end{array}
$$
We design a variant where given a sentence-path pair, we randomly select an entity that co-occurs in sentence and path and mask it in the sentence. Whether a sentence is masked during training is controlled by a probability  $p_{mask}$ . The model is then trained to generate path containing masked entity given masked sentence as input. The intuition is to enforce CoSe-Co to capture context
better through identifying masked entity during path generation. We discuss and perform ablations to compare masked CoSe-Co with varying values of  $p_{mask}$  in §4.3.1. Separately, we discuss and observe that using GPT-2 as backbone LM for CoSe-Co performs similar to T5-base in Appendix B.
3.3 Path Decoding During Inference
As in most sequence generation tasks, teacher forcing is used to train the model, while a decoding strategy is used to generate diverse outputs during inference (Vaswani et al., 2017). To maximise contextual knowledge obtained from paths for each sentence in a downstream task, we generate multiple paths. To improve diversity between paths while not losing relevance, we implement a path-specific variant of beam search, diverse-path search. Diversity is ensured by sampling top- $k$  most probable tokens at first generation step followed by decoding most probable sequence for each of them, thus returning  $k$  paths. This approach is motivated by observation that when generating a path, initial entity guide overall decoding of path.
We choose Wikipedia as the sentence corpus  $\mathcal{C}$ , and ConceptNet (Speer et al., 2017) as the knowledge graph  $\mathcal{G}$ . The subset of Wikipedia that we use comprises of  $\sim 5\mathrm{M}$  articles, from which we extract  $\sim 92.6\mathrm{M}$  sentences. ConceptNet comprises of  $\sim 8$  million nodes as concepts linked through 34 unique commonsense relations with  $\sim 21$  million links in total. We sample  $\sim 28\mathrm{M}$  paths that have a length  $|p|$  in the range  $l_{1} = 2$  and  $l_{2} = 5$ . We obtain a total of  $\sim 290\mathrm{K}$  sentence-path pairs. CoSe-Co is trained until validation loss across an epoch does not increase, with maximum number of epochs  $= 5$ .  $p_{mask}$  is set to 0.33 based on tuning on CSQA dev set and number of paths per sentence  $k = 5$  during inference. AdamW optimizer (Loshchilov and Hutter, 2017) is used to train parameters with a learning rate of  $5e - 4$ , weight decay of 0.01 and epsilon of  $1e - 8$  using a single A-100 GPU with batch-size 8 and 4 gradient accumulation steps.
4.2 Analysing Generated Paths
We analyse quality of generated paths on three aspects - Relevance, Diversity and Novelty, evaluated on test split of our sentence-path dataset. We estimate Relevance by treating each triple in generated
(a) Relevance: BLEU score of generated paths computed using ground truth paths
(b) Diversity: Compliment of fraction overlap between top-5 sampled paths.
Figure 3: Analysis and comparison of generated paths across different decoding strategies
Input	CoSe-Co Outputs
What do people typically do while playing guitar?	(playing_guitar causes singing usedfor people capable of feeling_sad)
(playing_guitar hassubevent sing _causesdesire singing_occupation musician genre_folk-rock)
(play_guitar _usedfor guitar atlocation symphony_halls_or_musical_instruments_or_bands_atlocation people )
Where are you likely to find a hamburger?	(burger_isa hamburger atlocation fast_food_restaurant usedfor eating食物)
(burger_king _usedfor hamburger atlocation fast_food_restaurant isa place capableof take_car_fordrive)
(fast_food_restaurant_isa_taco_bell product hamburger madeof wheat/flour_and_salt)
In what Spanish speaking North American country can you get a great cup of coffee?	(bretagne_partof north_america_atlocation cup_of_coffee hascontext usa isa country)
(hot_beverage_isa coffee atlocation cup_of_coffee hascontext north_america_partofgrenada)
(good_coffee_hasa caffeiene_in_milk_and_sugar atlocation in canada)