SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now we analyze the error cases in detail, including non-contrastive and non-context-dependent sentence pairs. The main explanation that accounts for the production of non-contrastive sentence pair is that the remaining verbs after keyword masking may have lexical ambiguity, e.g. "play" in "Slaves play a role in the history of the united states." and "A team sport played mostly in Canada is Lacrosse." Although the pretrained language models could infer the meaning of a word according to its context [14], the contexts are lost after keyword masking. As a result, two sentences with different syntactic structures are matched together, thus violating the antithesis rule. This poses a limitation of our antithesis mining algorithm.
In addition,  $7\%$  of the sentence pairs are antitheses yet not context-dependent. Take the following sentence pair as an example: "You could find millions of brownstone in New York City." One can find a Holiday Inn inside the United States. After swapping the GEO entity "New York City" and "United States" in these two sentences, they still conform to commonsense. The reason for this phenomenon is that New York City is part of the United States, and thus the "brownstone" related to New York will also be related to the United States. However, we would like to point out that contextual dependence
<sup>6</sup>https://github.com/shobrook/communities
<sup>7</sup>As background knowledge, there are many historical buildings in New York City whose facades are made of brown sandstone, see https://bungalow.com/articles/what-exactly-is-a-brownstone.
is not an absolutely strict condition. Although this example still holds after swapping the GEO entities, it is not the optimal answer, because "brownstone" is more a typical thing in New York City and thus more suitable for a match with "New York City".
5.2 Dataset Statistics
Table 2 includes the basic statistics of the SITUATEDGEN dataset. If we use the ratio of unique statement count to sentence pair count ("# Unique Sents per Sent Pair") to represent the content/keyword diversity of the dataset, the validation set, and the test set are relatively high (0.22/0.28), compared to the training set (0.14).
Distribution of Numbers of Input Keywords. Figure 3 shows the distribution of numbers of input keywords for all examples in the dataset. Intuitively, more input keywords imply an increased number of possible combinations, making it more difficult for the models to handle. The average number of input keywords is 7.21 and the distribution is fairly symmetrical (skewness  $= -0.25$ ), suggesting that the SITUATEDGEN has a reasonable difficulty.
Distribution of Context Types. Here we define three context types of pairs of contrastive sentences: a GEO pair of sentences contain only GEO entities; a TEMP pair of sentences contain only TEMP entities; If both sentences contain GEO and TEMP entities, the pair of sentences belongs to the type of GEO & TEMP. We find that  $78\%$  of all sentence pairs are GEO,  $21\%$  are TEMP and the rest  $1\%$  are GEO & TEMP.
6 Methods
Baseline Models. We benchmark the performance of three prominent pretrained language generation models with encoder-decoder architecture — BART [22], T5 [39], FLAN-T5 [10] — and a decoder-only large language model (LLM) — InstructGPT [34] with 175B parameters. We train BART, T5, and FLAN-T5 models in a fully supervised setting with the seq2seq format and expect that the models can learn to group keywords implicitly. Specifically, for the input of BART, we concatenate all shuffled keywords with a comma as the separation token “ $c_1, c_2, \ldots, c_k$ ”. Regarding the input format of T5/FLAN-T5, we preprocess the keyword sequence with a simple task description to align with its pretraining objective: “generate two sentences with:  $c_1, c_2, \ldots, c_k$ ”. The outputs of all models are simple concatenations of the two target sentences  $s_1$  and  $s_2$ . Since the output is an unordered pair, we feed two examples “ $x \rightarrow s_1 s_2$ ” and “ $x \rightarrow s_2 s_1$ ” to the model for each original training example. As for InstructGPT, we evaluate it in a few-shot setting. We build prompts with instruction and in-context demonstrations. For each test example, we randomly select 10 training examples as in-context demonstrations. We report the model hyper-parameters and GPT prompt format in Appendix C.1.
Evaluation Metrics. [25] have well established the automatic evaluation protocol of the generative commonsense reasoning task. They demonstrated a strong correlation between automatic metrics and human evaluation results. Since SITUATEDGEN adopts a similar format of keyword-to-text generation to COMMONGEN, we follow the evaluation protocol of COMMONGEN and do not include an extra manual evaluation in our study.
Concretely, we employ several widely-used automatic NLG metrics based on n-gram overlap — BLEU [36], ROUGE [26], METEOR [3] — and image caption metrics that focus on the consistency of keywords and their relationships — CIDEr [46] and SPICE [2]. In order to assess the validity of the generated outputs, we include BERTScore [53], a content-oriented and semantic metric. We also adopt COVERAGE, which is the average percentage of input keywords that are present in lemmatized outputs. Additionally, we report the accuracy of keyword grouping results<sup>8</sup> as MATCH, which serves as a good indicator of the commonsense plausibility of the generated texts. See Appendix C.2 for the implementation details of these evaluation metrics.
Keywords appearing in the same lemmatized output sentence are considered to be grouped together by models. In particular, if a keyword does not appear in the output, we treat it as unmatched.
Table 3: Experimental results on the test set of SITUATEDGEN. The best model performance is in bold. Human performance is tested on a subset of 100 random samples.
Model (# parameters)	COVERAGE	MATCH	BLEU-4	ROUGE-2	METEOR	CIDEr	SPICE	BERTScore
BART-base (140M)	78.3	60.5	22.7	29.9	29.6	18.3	53.9	48.4
BART-large (400M)	73.3	63.1	23.7	31.6	29.2	18.5	55.3	48.1
T5-base (220M)	75.6	55.3	21.9	28.7	29.8	17.4	53.6	46.2
T5-large (770M)	81.3	67.8	26.6	33.5	31.9	21.2	57.8	51.9
FLAN-T5-base (220M)	78.0	58.7	22.3	29.5	30.6	18.2	54.7	47.6
FLAN-T5-large (770M)	83.1	70.3	27.4	34.8	32.6	22.4	58.8	53.6
GEO	83.1	70.8	26.8	33.9	32.4	21.9	58.2	52.8
TEMP	83.1	67.0	31.2	40.4	34.1	22.7	62.5	59.1
InstructGPT (175B, 10-shot)	91.8	79.6	28.4	36.3	36.1	23.4	60.9	56.4
Human	98.1	92.9	39.9	46.9	40.4	39.7	71.4	65.0
Table 4: Case studies of machine generations. Keywords appearing in the generation results are underlined.
Input Keywords	24 hours, axis, one month, Earth, axis, Moon
Reference	It takes one month for the Moon to rotate on its axis. Earth rotating on its axis takes 24 hours.
BART-base	The axis of the Moon is 24 hours. One month is one month.
BART-large	There are 24 hours in one month.
T5-base	Earth has a 24 hour axis. One month is one month.
T5-large	One month is one month on Earth. The Moon is 24 hours away from the axis of the Earth.
FLAN-T5-base	The Moon is the axis of the Earth. One month is one month.
FLAN-T5-large	The Moon is 24 hours away from Earth. One month is one month.
InstructGPT	The Earth takes 24 hours to rotate on its axis. The Moon takes one month to orbit around the Earth.
Input Keywords	Paul, Emperor, China, Qin, Russia, dynasty
Reference	The Qin dynasty reigned in China. Paul I of Russia reigned as the Emperor of Russia.
BART-base	The Emperor of China worked in China. Paul served as the first emperor of the dynasty Qin.
BART-large	Emperor of the Qin dynasty. Paul existed in Russia.
T5-base	China is a dynasty of China. Paul Qin is the Emperor of China.
T5-large	Paul was the Emperor of Russia. The Qin dynasty ruled China.
FLAN-T5-base	Paul was the emperor of China. The history of Russia includes the history of Qin.
FLAN-T5-large	The Emperor of Russia was Paul the Great. Qin dynasty existed in China and had history in Russia.
InstructGPT	Emperor Paul was part of the Russian dynasty. Qin was part of the Chinese dynasty.
In Table 3, we report the experimental results of different baseline models on the test set of SITUATEDGEN. We approximate human performance with 100 randomly sampled examples from the test set which are annotated by the authors of this paper. We observe that larger models tend to have better performance than smaller ones, as larger parameters store more commonsense knowledge and provide better language generation quality. Notably, the few-shot InstructGPT surpasses other fully-supervised models in every metric, demonstrating its strong reasoning ability. Nevertheless, it still lags far behind human performance. For example, there is a difference of 13.3 points in MATCH, indicating the lack of commonsense in machine generations. The large gap of keyword-oriented metrics (CIDEr and SPICE) also suggests that models find it difficult to infer the relationship between keywords. The significant gap between models and humans demonstrates the difficulty of SITUATEDGEN and leaves much room for improvement in future research.
Performance across Different Context Types. Table 3 reports the performance of the FLAN-T5-large model across different context types. The results show that the matching accuracy of TEMP
type is lower than GEO, indicating that temporal-dependent test examples are more challenging. However, the amount of TEMP data is less than GEO in the training set, which may also give rise to the performance difference. Interestingly, the generation fluency of GEO type is worse than TEMP, suggesting that it is more difficult to use GEO entities to compose sentences smoothly.
Case Study. Table 4 shows two groups of generation examples by different models. The first example belongs to TEMP type ("24 hours" and "one month") and the second one is GEO ("Russia" and "China"). We find that models are prone to omit keywords in their outputs. For example, BART-large only covers 2 out of 6 keywords in the first example. Besides, most of the observed generated outputs are not commonsensical due to incorrect keyword grouping results, e.g., "There are 24 hours in one month". InstructGPT results seem to have the best generation quality and commonsense plausibility among other models, but it still demonstrates incompetence in handling the contrastive relationships between the two sentences.
In this paper, we introduce the challenging task SITUATEDGEN to incorporate geographical and temporal contexts into generative commonsense reasoning. We build a corresponding testbed to evaluate the situated reasoning capabilities of state-of-the-art text generation models. The benchmark performance shows that models struggle to generate commonsensical sentences and lag far behind humans. Altogether, our data will serve as a challenging benchmark for measuring commonsense knowledge in generative language models and support research progress of constrained commonsense text generation in a more realistic situation.
1. Since our dataset is derived from existing commonsense benchmarks, we may inherit their annotation artifacts [18] and contain certain types of spurious lexical patterns (e.g., "A lived in B").
2. We do not provide an automatic evaluation of the aspect of contrast between the sentences. A possible solution is to compute the similarity between the entity-masked sentences. This is similar to how we mine contrastive sentences during dataset collection (Figure 2).
3. We could also conduct an extra manual evaluation on the machine generations, so as to gauge its correlation with automatic metrics, though this has been verified by [25] on the original generative commonsense reasoning task.
4. Recently, a lot of work has developed new retrieval-augmented commonsense text generation models [51, 19], which could also be included as baseline models for a more comprehensive benchmark.
Our data is built upon publicly available datasets and we will follow their licenses when releasing our data. There is no explicit detail that leaks an annotator's personal information. The dataset has very low risks of containing sentences with toxicity and offensiveness. Since our data is sourced from existing datasets, we may inherit geographical biases [16] that result in an uneven distribution of commonsense knowledge about western and non-western regions. The commonsense statements may not sound familiar to people who live in locations that are poorly represented in the source datasets. Therefore, models developed on our dataset may preserve biases learned from the annotators of the source datasets. We note that pretrained language models may also inherit the bias in the massive pretraining data. It is important that interested parties carefully address those biases before deploying the model to real-world settings.
We thank the anonymous reviewers for their helpful comments and suggestions.
[49] Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. Geom-lama: Geo-diverse commonsense probing on multilingual pre-trained language models. CoRR, abs/2205.12247, 2022.
[50] Da Yin, Liunian Harold Li, Ziniu Hu, Nanyun Peng, and Kai-Wei Chang. Broaden the vision: Geo-diverse visual commonsense reasoning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November; 2021, pages 2115-2129. Association for Computational Linguistics, 2021.
[51] Wenhao Yu, Chenguang Zhu, Zhihan Zhang, Shuohang Wang, Zhuosheng Zhang, Yuwei Fang, and Meng Jiang. Retrieval augmentation for commonsense reasoning: A unified approach. CoRR, abs/2210.12887, 2022.
[52] Michael J. Q. Zhang and Eunsol Choi. Situatedqa: Incorporating extra-linguistic contexts into QA. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7371-7387. Association for Computational Linguistics, 2021.
[53] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[54] Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. "going on a vacation" takes longer than "going for a walk": A study of temporal commonsense understanding. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3361-3367. Association for Computational Linguistics, 2019.
A Additional Details of Dataset Collection
Table 5: Source dataset examples. Correct answers are in bold and underlined.
Dataset	Size	Format	Raw Data → Statement Conversion Example
CREAK [33]	13,418	True/False statement	In the calendar year, May comes after April and before June. (True/False) → In the calendar year, May comes after April and before June.
StrategyQA [17]	5,111	Yes/No Question	Are more watermelons grown in Texas than in Antarctica? (Yes/No) → More watermelons are grown in Texas than in Antarctica.
CommonsenseQA [44]	12,247	Multiple-choice Question	Where in Southern Europe would you find many canals? (A) Michigan (B) New York (C) Amsterdam (D) Venice (E) Sydney → You would find many canals in Venice, Southern Europe.
ARC [11]	7,787	Multiple-choice Question	How long does it take for Earth to rotate on its axis seven times? (A) one day (B) one week (C) one month (D) one year → It takes one week for Earth to rotate on its axis seven times.
OpenbookQA [30]	6,493	Commonsense Statement	You wear shorts in the summer. → You wear shorts in the summer.
A.1 Commonsense Statement Collection
We briefly introduce the nature of each source dataset in Section 4.1.
- CREAK [33] is a commonsense fact verification dataset featuring entity commonsense, which includes 13,418 true or false statements about entity knowledge written by crowd-workers.
- StrategyQA [17] is a commonsense question answering dataset that requires multi-hop implicit reasoning. It consists of 5,111 questions whose answers are either Yes or No. Machines need to decompose a question into multiple atomic questions to arrive at an answer.
- CommonsenseQA [44] is a commonsense question answering dataset of 12,247 five-way multiple-choice questions with a focus on knowledge in everyday life.
- ARC [11] is a commonsense question answering dataset. It has 7,787 four-way multiple-choice natural science questions collected from grade-school standardized tests.
- OpenbookQA [30] is a commonsense question answering dataset that simulates openbook test. The data set is made up of 5,957 multiple-choice questions, accompanied by 6,493 commonsense statements about science facts. Since there is a significant overlap between the knowledge in questions and statements, we only use the statements data for simplicity.
We now detail the specific preprocessing method for each source dataset to convert them (i.e., question-answer pairs) into statements.
- If the raw data comes in the statement format (CBREAK and OpenbookQA), we obtain the true statements (part of CBREAK and all of OpenbookQA) without extra processing.
- If the raw data comes in Yes/No question format (StrategyQA), we leverage a POS-rule-based open-sourced system question_to_statement<sup>9</sup> to transform a pair of question and Yes/No answer into a statement.
- If the raw data comes in multiple-choice format (CommonsenseQA and ARC), we utilize a neural model to convert a pair of question and correct choice  $(q, a)$  into a statement in a sequence-to-sequence fashion. Concretely, we use the QA-to-statement model checkpoint
<sup>9</sup>https://github.com/SunnyWay/question_to_statement
released by [35], which is a BART [22] model finetuned on QA2D [13], a dataset of human-annotated statements for QA pairs.
Converting QA pair to statement is not a difficult task for pretrained seq2seq models. We observe that the generated statements are mostly fluent and faithful to the input. Additionally, we have manually filtered out unnatural examples in the test set. We summarize the basic information of these datasets and provide an example of statement conversion for each dataset in Table 5.
A.2 Antithesis Mining
Keyword Masking. We use entities and other nouns as the keywords of sentences because as a pilot study, we only consider the relationships between spatio-temporal contexts and nouns and ignore the influence of other part-of-speech categories such as verbs, adjectives, and prepositions. We use the same NER tagger in Section 4.2 to extract entities. We leverage  $\mathsf{spaCy}^{10}$  to extract all the nouns (including proper nouns) from a sentence. We merge the entities and nouns as keywords after removing duplicates. In particular, if a noun and an entity partly overlap (e.g., "month" and "a lunar month"), we retain the entity when deduplicating.
Masked Sentence Similarity Matching. We use the pretrained language model all-MiniLM-L6-v2 $^{11}$  released by Sentence Transformers [40] to obtain high-quality embeddings of keyword-masked sentences. We calculate the cosine similarity to pair highly similar masked sentences. Computing the similarity of all possible sentence pairs requires  $\mathcal{O}(n^2)$  time complexity. To accelerate this process, we use the paraphrase_mining API of Sentence Transformers [40].
Rule-based Filtering. We devise the following rules to filter invalid sentence pairs based on iterative observation of the data:
- The masked sentence similarity exceeds a certain threshold<sup>12</sup>, which indicates parallel sentence structure of antithesis.
- The number of masked keywords ([UNK]) of every single sentence should not be more than 5 and less than 2, which controls for a reasonable difficulty of the keyword-to-text generation task.
- Any entity in one sentence does not appear in the other sentence within a pair (including the deformation of entity words, such as singular/plural form, upper/ lower case, etc.). This is to avoid both sentences expressing the information of the same entity, while contrastive sentences should describe two opposite things.
- Both of the two sentences contain either GEO entities or TEMP entities (GEO+GEO or TEMP+TEMP), which avoids sentences comparing GEO context to a non-parallel TEMP context (GEO+TEMP).
B Dataset Quality Analysis
B.1 Manual Filtering of the Test Set
To ensure the high quality of the dataset, we manually filter out invalid examples in the test set that are not fluent antitheses or context-dependent. This process is important for the very high human performance shown in Table 3. Table 6 shows the instructions for annotators. We first ask two graduate students with proficiency in English to annotate 100 examples as valid or invalid. They agree with each other (i.e., give the same label) on  $88\%$  of examples. The inter-annotator agreement in terms of Cohen's Kappa [12] is 0.76, which indicates substantial agreement [21]. Since the agreement ratio is satisfactory, we ask one of the annotators to complete the rest of the filtering process.
10https://spacy.io/models/en#en_core_web_sm
<sup>11</sup>https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
12We set the threshold as 0.8 via manual inspection.
Table 6: Annotator instructions for manual filtering of our dataset.
Goal: The objective of our project is to generate high-quality contrastive sentence pairs (antithesis) that incorporate geographical and temporal contexts. These sentence pairs will serve as a means to evaluate machines' commonsense reasoning abilities under different extra-linguistic contexts. We aim to create sentences that require a deep understanding of real-world geographical and temporal entities but can be reasonably confirmed without resorting to external sources like Google or Wikipedia.
Instructions: We show a set of keywords and a pair of sentences containing these keywords. Your task is to determine whether this sentence pair satisfies all of the following criteria:
1. The sentence pair includes all of the given keywords.
2. Each sentence has at least one entity related to geography or time.
3. Each sentence is fluent and adheres to commonsense knowledge.
4. The two sentences have similar syntactic structures and create a contradiction in semantics.
- Intuitively, the qualifying two sentences can be connected into a coherent sentence via a conjunction word such as "while", "yet", and "whereas" (e.g., "July is summer in the United States, while July is winter in Australia").
5. Swapping any of the geographical or temporal entities between the two sentences could lead to a contradiction with commonsense yet grammatical correctness.
- For example, for the sentence pair "July is summer in China. July is winter in Australia," if the two geographical entities "China" and "Australia" are swapped, the resulting sentences do not adhere to commonsense anymore: "July is summer in Australia. July is winter in China."
Keywords: morning, night, sunrise, sunset
Sentence 1: "The sky is bright with the sunrise in the early morning."
Sentence 2: "The sky is dark with the sunset in the late night."
Criterion 1: Both sentences include the keywords "morning" and "night."
Criterion 2: Each sentence contains a geographical or temporal entity ("sunrise" and "sunset") related to the context.
Criterion 3: Both sentences are fluent and adhere to commonsense knowledge.
Criterion 4: The sentences have a similar syntactic structure and create a semantic contradiction: "The sky is bright with the sunrise in the early morning, while the sky is dark with the sunset in the late night."
Criterion 5: Swapping the temporal entities "early morning" and "late night" would result in a contradiction:
"The sky is bright with the sunrise in the late night, while the sky is dark with the sunset in the early morning."
This example demonstrates how the sentence pairs satisfy the specified criteria of the task.
C.1 Baseline Models