SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SITUATEDGEN: Incorporating Geographical and Temporal Contexts into Generative Commonsense Reasoning
Yunxiang Zhang
Ann Arbor, USA
yunxiang@umich.edu
Xiaojun Wan
wanxiaojun@pku.edu.cn
Recently, commonsense reasoning in text generation has attracted much attention. Generative commonsense reasoning is the task that requires machines, given a group of keywords, to compose a single coherent sentence with commonsense plausibility. While existing datasets targeting generative commonsense reasoning focus on everyday scenarios, it is unclear how well machines reason under specific geographical and temporal contexts. We formalize this challenging task as SITUATEDGEN, where machines with commonsense should generate a pair of contrastive sentences given a group of keywords including geographical or temporal entities. We introduce a corresponding English dataset consisting of 8,268 contrastive sentence pairs, which are built upon several existing commonsense reasoning benchmarks with minimal manual labor. Experiments show that state-of-the-art generative language models struggle to generate sentences with commonsense plausibility and still lag far behind human performance. Our dataset is publicly available at https://github.com/yunx-z/situated_gen.
In recent years, there has been substantial growth in new benchmarks evaluating commonsense reasoning for natural language processing (NLP) models, especially large-scale Pretrained Language Models (PLMs). Most existing commonsense reasoning benchmarks adopt natural language understanding formats due to easy evaluation (e.g., accuracy), including multiple-choice question answering [44, 41, 20, 24], natural language inference [4], and detecting true/false statements [33, 43]. However, datasets measuring commonsense knowledge in natural language generation are still relatively scarce. We aim to fill this research gap with a novel benchmark since real-world users of NLP systems would expect the generated outputs from LMs to be not only grammatically correct but also adhere to commonsense knowledge.
COMMONGEN [25], a generative commonsense reasoning challenge, has attracted wide attention recently. Given a set of keywords (e.g., {dog, frisbee, catch, throw}), the task requires models to compose a plausible sentence describing everyday scenario using all the provided keywords (e.g., "The dog catches the frisbee when the boy throws it"). While COMMONGEN focuses on social and physical commonsense in everyday life, it is unclear how well current commonsense generation models reason with factual knowledge about specific entities, which is referred to as entity commonsense [33]. In this work, we mainly consider geographical and temporal entities, as they provide extra-linguistic contexts [52] for commonsense reasoning and appear in a significant proportion of existing commonsense benchmarks (Section 4.2). To the best of our knowledge, we are the first to incorporate these situations into generative commonsense reasoning.
Furthermore, we argue that geographical and temporal contexts are important for commonsense reasoning. On the one hand, basic knowledge about geography and time is part of human commonsense [1, 6], such as "Earth rotates on its axis once in 24 hours." On the other hand, certain types of commonsense knowledge are correlated with specific situations [50]. For example, "July is summer" is true for people living in the northern hemisphere, while those living in the southern hemisphere would agree that "July is winter".
Our proposed task SITUATEDGEN (Situated Generative Commonsense Reasoning) requires the machines to generate a pair of contrastive sentences (formally speaking, antithesis) with commonsense plausibility, given a group of keywords including geographical or temporal entities. For example, when provided with [July, United States, winter, Australia, summer, July], a reasonable output could be "July is summer in the United States. July is winter in Australia.", while a slightly different version "July is summer in Australia. July is winter in the United States." does not adhere to commonsense.
The main challenge for machines to solve the SITUATEDGEN task lies in situated semantic matching. In order to generate a pair of contrastive sentences, machines need to split the keywords into two groups (either explicitly or implicitly) based on geographical/temporal relevance and perform relational reasoning [31] within/between the keyword groups.
To study the challenging SITUATEDGEN task, we construct a corresponding large-scale English dataset containing 8,268 pairs of situated commonsense statements. We design an automatic pipeline to collect data at scale with quality assurance and minimal human annotation efforts. Concretely, we derive commonsense statements with geographical or temporal contexts from existing commonsense benchmarks and mine contrastive sentence pairs based on entity-masked sentence similarity. We further manually filter out invalid examples in the test set to ensure the evaluation soundness. To assess the difficulty of our dataset, we conduct automatic evaluations on various generative (large) language models, including BART [22], T5 [39], and InstructGPT [34]. Results show these models lag far behind human performance, indicating that current models struggle to generate sentences adhering to commonsense under the SITUATEDGEN setting. We believe that SITUATEDGEN could serve as a complement to COMMONGEN and enrich the resource for evaluating constrained commonsense text generation in a more realistic setting.
The contributions of this work are three-fold:
- Task. We incorporate geographical and temporal contexts into generative commonsense reasoning and propose a novel task SITUATEDGEN.
- Resource. We construct a large-scale dataset in a non-trivial way to facilitate the studies of situated generative commonsense reasoning. The dataset is released and will contribute to the commonsense reasoning community.
- Evaluation. We benchmark the performance of state-of-the-art generative language models on our dataset and demonstrate the difficulty of the task with a significant gap between machine and human performance.
Constrained Commonsense Text Generation. Constrained Commonsense Text Generation [5] requires PLMs to generate commonsense text subject to a set of constraints. Commonsense generation models are currently evaluated by three tasks. First, COMMONSENSE EXPLANATION aims to generate an explanation for why a model selects a candidate answer to a given question. Second,  $\alpha$  NLG [4] is another commonsense generation task. The artificial intelligence models are provided with two observations in chronological order and need to generate a plausible hypothesis/explanation describing what happened between the observations. Third, in COMMONGEN [25], models should compose a plausible sentence describing everyday scenarios using all the provided concepts. This task has attracted much attention recently, and researchers advance machine performance on the dataset with contrastive learning [23], prototype editing [28], scene knowledge graph [47], etc. Our proposed task differs from these tasks with a focus on composing a pair of contrastive sentences instead of a single sentence and incorporating extra-linguistic contexts.
NLP Benchmarks with Geographical and Temporal Contexts. There are many emerging benchmarks in NLP that incorporate extra-linguistic contexts such as geographical and temporal contexts. TEMPLAMA [15] and GEOMLAMA [49] probe language models with masked text prompts to query geographical and temporal knowledge. In question answering, MCTACO [54], TORQUE [32] and TIMEQA [9] contains challenging questions involving temporal commonsense reasoning over the duration, frequency, temporal order, and other various aspects of events. SITUATEDQA [52] is made up of open-domain questions whose answers vary across different geographical and temporal contexts. TIMEDIAL [38] studies temporal reasoning in dialogues with a multiple-choice cloze task. In vision-and-language tasks, GD-VCR [50] and MaRVL [27] aim to collect commonsense questions and statements that are visually grounded and geographically diverse. Previous work mainly focuses on how well language models trained on a specific snapshot of corpus can adapt to different contexts. While our dataset SITUATEDGEN also considers such geographical and temporal contexts in language, we probe LMs for a new skill of reasoning for the commonsense relationship among extra-linguistic contexts. We also choose a different task format of generative commonsense reasoning, pioneered by [25], as it focuses on the commonsense reasoning capabilities of generative models rather than NLU models, which is under-researched by the community.
3 Task Definitions and Challenges
We use antithesis generation for evaluating generative commonsense reasoning under extra-linguistic contexts. In this section, we first introduce the definitions of our proposed task, followed by an analysis of the main challenges.
Antithesis. Antithesis refers to a figure of speech that expresses an opposition of ideas with a parallel grammatical structure of words, clauses, or sentences [29, 8]. An example of antithesis could be Neil Armstrong's famous quote "That's one small step for a man, one giant leap for mankind". In this work, we adopt a narrow sense of sentence-level antithesis, which means that two simple sentences with similar syntactic structures create a contradiction in semantics. Intuitively, the qualifying two sentences can be connected into a coherent sentence via conjunction words such as "while", "yet", and "whereas" (e.g., "July is summer in the United States, while July is winter in Australia"). We emphasize commonsense plausibility rather than the rhetorical effect of antithesis within the scope of this paper.
Extra-Linguistic Contexts. Following [52], we focus on two context types: geographical (GEO) and temporal (TEMP). GEO defines each context value as a geopolitical entity ("GPE"). TEMP defines each context value as timestamp ("DATE", "TIME", "EVENT").
Contextual Dependence. We define that a contrastive sentence pair is context-dependent if swapping any of the GEO or TEMP entities between the two sentences could lead to a contradiction with commonsense yet grammatical correctness. For example, for the sentence pair "July is summer in China. July is winter in Australia," if the two GEO entities "China" and "Australia" are swapped, the resulting sentences do not adhere to commonsense anymore: "July is summer in Australia. July is winter in China." This indicates that they are context-dependent.
Contextual dependence is crucial for a proper evaluation of the generation results. Because sentence pairs that do not satisfy context dependence may have multiple valid answers (swapping the entity words leads to an extra correct answer), the metrics introduced in Section 6 cannot make a sound evaluation with only a single reference.
Situated Generative Commonsense Reasoning. We modify the mathematical formulation of the task COMMONGEN to define SITUATEDGEN. The input of the task is a multiset consisting of  $k$  keywords  $x = [c_{1}, c_{2}, \dots, c_{k}] \in \mathcal{X}$ , where each keyword  $c_{i} \in \mathcal{C}$  is a noun or entity, a single word or phrase. We denote  $\mathcal{X}$  as all possible combinations of keywords and  $\mathcal{C}$  as the vocabulary of keywords.
1 Multiset is a set that allows multiple instances for each of its elements.
Figure 1: An overview of data collection pipeline. Inside the dotted box is a final example in the dataset.
Keywords in  $x$  should contain at least two GEO or TEMP entities of the same type and two other keywords<sup>2</sup>.
The output of the task is an unordered pair of coherent and plausible sentences  $y = \{s_1, s_2\} \in \mathcal{V}$  that satisfies the following conditions: 1) the sentence pair includes all keywords in  $x$ ; 2) each sentence has at least one GEO or TEMP keyword; 3) each sentence is geographical-temporal-semantically correct; 4)  $s_1$  and  $s_2$  form a pair of contrastive sentences, or antithesis; 5)  $s_1$  and  $s_2$  are context-dependent. The goal of the task is to learn a function  $f: \mathcal{X} \to \mathcal{V}$  that maps a group of keywords  $x$  to a pair of sentences  $y$ .
3.2 Challenges: Situated Semantic Matching
As the goal of our task is to generate a pair of sentences instead of a single sentence, machines need to explicitly or implicitly classify the keywords into two subgroups based on their geographical and temporal semantic relevance, so as to generate one commonsense sentence with each subgroup. For example, given [July, China, winter, Australia, summer, July], the resulting keyword subgroups should be {July, China, summer} and {July, winter, Australia}.
During the process of keyword grouping and matching, machines need to make connections among keyword concepts with relational reasoning [31] over factual knowledge about these nouns and entities, a.k.a. entity knowledge [52], such as geographical location, temporal order, physical rules, social customs, etc. The matching process is important since wrong grouping results will lead to generated sentences without commonsense plausibility<sup>3</sup>.
We require that the two sentences should have similar syntactic structures and express similar relationships (e.g., "X lives in Y"). This is important for securing the difficulty of the task as it prevents models from learning shortcuts [18, 45] to group keywords based on trivial syntactic (e.g., POS tag of the word) and semantic (e.g., two different kinds of relationship) information. For example, if the two sentences have different syntactic structures (e.g. "X lives in Y" and "Z eats W"), then the model could simply put a city name in Y and a food name in W for keyword grouping and ignore the commonsense connection with X/Z. This type of shortcut reduces the task difficulty.
2We do not explicitly provide the types of keywords in our dataset. The models are expected to infer which keyword is GEO or TEMP if needed.
<sup>3</sup>We note that under certain circumstances, wrong grouping results might produce correct answers via negative sentences. For example, the machine could generate "July is not summer in Australia" with {July, Australia, summer}. However, we observe that these are rare scenarios in our datasets, so we do not consider their confusing effects in our study.
4 Dataset Collection
To study the SITUATEDGEN challenge, we construct a large-scale English dataset. We design a pipeline to collect high-quality data at scale with minimal manual annotation efforts. Figure 1 illustrates the overall pipeline for dataset collection, which consists of three steps:
1. QA-to-statement. Converting question-answer pairs of existing commonsense question answering benchmarks into corresponding statements.
2. Contexts Identification. Identifying all entities in a statement with a NER tagger and removing those statements without GEO and TEMP entities.
3. Contrastive Sentences Mining. Automatically mining contrastive sentence pairs (antithesis) from the remaining commonsense statements based on entity-masked sentence similarity.
4.1 QA-to-Statement
Our dataset is composed of commonsense statements, which are simple sentences describing commonsense knowledge, e.g., "You would find many canals in Venice." In recent years, numerous commonsense reasoning benchmarks have been proposed and they form a potentially available commonsense knowledge base with high quality and diverse content. Inspired by recent benchmarks that are sourced from existing datasets [52, 37], we aim to extract commonsense statements from these commonsense benchmarks. We assume that the knowledge in these commonsense benchmarks is actually commonsense instead of encyclopedic knowledge, though they might not be shared locally in certain groups of people due to a lack of geographical diversity. That being said, we adopt and follow the concept of "commonsense" widely used in existing works.
We conduct a holistic study of commonsense reasoning datasets to date and select five different data sources after considering their size, annotation quality, and reasoning difficulty. They are CREAM [33], StrategyQA [17], CommonsenseQA [44], ARC [11] and OpenbookQA [30], respectively. We briefly introduce the nature of each dataset in Appendix A.1. Since the raw data come in different formats such as multiple-choice questions and Yes/No questions, we apply a specific preprocessing method for each dataset to transform them (i.e., question-answer pairs) into statements. The transformation details are also included in Appendix A.1. In general, we collected 35,997 commonsense statements from the five source datasets (statistics in Table 1).
4.2 Contexts Identification
We now filter out commonsense statements without geographical or temporal contexts. Following [52], we identify sentences with extra-linguistic contexts by GEO and TEMP entities. We use FLERT[42], a named entity recognition (NER) model, to extract all entities from a sentence and remove those statements without any GEO ("GPE") or TEMP ("DATE", "TIME", "EVENT") entities.
Table 1 shows that of all the commonsense statements extracted from the five source datasets,  $6.6\%$  sentences have GEO contexts and  $5.5\%$  have TEMP contexts, which we count as a significant proportion. Finally, we obtain 4,038  $(11.2\%)$  commonsense statements with extra-linguistic contexts.
4.3 Contrastive Sentences Mining
We aim to automatically mine contrastive sentence pairs from the commonsense statement corpus. Antithesis mining has not been studied in the existing literature, so we propose a pilot algorithm. We observe that after removing keywords from contrastive sentences, the remaining parts are very similar since antithesis sentences have parallel syntactic structures [8]. Based on this observation, we design the antithesis mining algorithm illustrated in Figure 2 consisting of three steps:
1. Keyword Masking. We extract all entities and other nouns as keywords in the sentence and replace each keyword with a [UNK] token, telling the pretrained language models to neglect the meaning of these keywords.
<sup>4</sup>https://huggingface.co/flair/ner-english-onthonotes-large
Table 1: Statistics of contexts identification results. "Sent" means the commonsense statements collected in Section 4.1. "GEO"/"TEMP" refer to statements with only geographical/temporal entities. "GEO & TEMP" refers to statements with both geographical and temporal entities. "Valid Sent" means the commonsense statements with GEO or TEMP contexts.
Dataset	# Sent	# GEO	# TEMP	# GEO & TEMP	# Valid Sent
CREAK	5,779	868	552	153	1,573
StrategyQA	4,976	501	366	86	953
CommonsenseQA	10,962	487	215	12	714
ARC	7,787	165	426	52	643
OpenbookQA	6,493	31	119	5	155
Total	35,997	2,052	1,678	308	4,038
Figure 2: An illustration of the contrastive sentence mining algorithm.
2. Masked Sentence Similarity Matching. We obtain the embedding of the keyword-masked sentence from a pretrained language model and calculate the cosine similarity between all possible sentence pairs.
3. Rule-based Filtering. We filter out invalid sentence pairs based on a fixed threshold of masked sentence similarity, number of keywords, and entity types.
We introduce the implementation of our antithesis mining algorithm in Appendix A.2. In this way, we efficiently extracted large-scale contrastive sentence pairs from all possible pairwise combinations of the aforementioned commonsense statements with extra-linguistic contexts<sup>5</sup> (Section 4.2). For each contrastive sentence pair, we merge the keywords from each statement and randomly shuffle them to get the input data. The output is the concatenation of two statements.
4.4 Dataset Splitting
When splitting the data into training, validation, and test set, we explicitly require that one statement cannot appear simultaneously in any two sets. Consequently, there is no overlap of the single sentence (or sentence-level keyword combinations) among the training, validation, and test data. This requirement forces machines to reason over new combinations of keywords during the inference stage instead of memorizing existing keywords matching results. Statements with similar syntactic structures will also be divided into the same set to reduce overlap of syntactic templates across different sets.
Specifically, we treat dataset splitting as a community structure [7] discovery problem. Community structure refers to a group of tightly connected nodes that have a high density of internal connections and a low density of external connections. We regard a single sentence as a node in the graph. If two single sentences can be matched into a pair of contrastive sentences, an undirected edge will connect the corresponding nodes of these two single sentences. In this way, we obtain an undirected graph describing the dataset structure. A subset of a dataset (such as a training set) is equivalent to a subgraph containing all sentence pairs (edges) and single sentences (nodes) of that subset.
In order to prevent the same sentence from appearing across different sets, we require that the subgraph node sets of the training set, validation set, and test set are disjoint. We use a community structure detection algorithm to meet this requirement. We use the community as the basic unit of dataset splitting, putting all the edges (sentence pairs) in one community into a certain dataset split. Connecting edges between communities (two vertices belonging to different communities) are removed. We note that sentences with similar syntactic structures tend to be connected to each other in the graph and thus fall into the same community, which ensures the syntactic variability between train/dev/test splits.
One statement might be paired with multiple statements, formulating multiple contrastive sentence pairs.
Table 2: The basic statistics of the SITUATEDGEN dataset. "Sent" means commonsense statement.
Statistics	Train	Dev	Test
Size (# Sent Pairs)	5,641	1,407	1,220
# Unique Sents	788	309	341
per Sent Pair	0.14	0.22	0.28
# Unique Keywords	1,847	725	851
# Avg. Input Keywords	7.34	6.96	6.89
# Avg. Output Tokens	20.89	24.08	20.61
Figure 3: Distribution of numbers of input keywords.
We use the Louvain [7] community structure detection algorithm and divide our graph into 79 communities. The largest community contains 3,273 edges, accounting for about  $26\%$  of the total data. We remove edges connecting different communities and then randomly divide the communities of contrastive sentence pairs into training set, validation set or test set.
To ensure the evaluation soundness, we manually filter out invalid examples in the test set that are not fluent antitheses or context-dependent.  $13.6\%$  of test data is removed and the final dataset has 8,268 examples in total. See additional details of manual filtering in Appendix B.
5.1 Quality Analysis
To measure the quality of our automatically collected data, we randomly select 100 examples (i.e. sentence pairs) from the validation set (which is not manually filtered) and annotate each example for whether it is actually 1) (fluent) antithesis and 2) context-dependent. We find that  $87\%$  of the data are real antitheses with fluency and  $80\%$  of the data satisfy both of the two requirements. Considering that our dataset is constructed through a fully automatic pipeline, this quality is pretty satisfying and can meet the needs of training and evaluation. As we have discussed in Section 3.1, test examples not satisfying contextual dependence can fool the evaluation metrics, since there are multiple valid references despite the single one provided in the test set. Thanks to the additional manual filtering at the end of Section 4.3, the test set is now qualified for evaluation. As for the unfiltered training set, even if a contrastive sentence pair is not context-dependent, it is still valuable training data, satisfying the other requirements for the target side (Section 3.1). Reduced size of training data after potential manual filtering is also unfavorable to the learning of models. As a result, we retain all the examples in the training set.