Contextualized Multi-Step Commonsense Reasoning through Context Extension†‡††Hecong Wang ; Erqian Xu ; Pinxin Liu ; Zijian Meng ; Zhen Bai††Department of Computer Science, University of Rochester;‡Warner School of Education and Human Development, University of Rochester

Common Sense in AI
Common sense is a quintessential human capacity but a fundamental challenge ofArtificial Intelligence (AI). Human-like commonsense reasoning lies at the center toenable AIs to seamlessly augment human capability while maintaining trust andtransparency.
There has been a massive resurgence of interest in commonsense reasoning researchin recent years, caused by the rapid advancements in deep learning techniques,
the developments of large language models
such as OpenAI’s GPT-3, and the availabilityof large-scale commonsense knowledge resources such as ConceptNet.
Multi-step commonsense reasoning is key toachieving human-level performance in AIs.
It also enables us to explain, in a humanunderstandable way, the functioning of AIs.
However, making multi-step commonsense
reasoning is particularly difficult due to several properties of common sense.
AIs particularly struggle with social commonsense reasoning, which is essential forthem to behave as responsible social partners (Sap et al., 2022; Mahowald et al.,2023). The lack of social commonsense reasoning in AIs is largely due to their heavy
reliance on static text, which fails to capture the contextual nature of common
sense and omits important commonsense knowledge (Sap et al., 2022).

Contextualized Multi-Step Commonsense Reasoning through Context Extension

Reasoning without Context Extension
The man made pancakes. He
poured syrup on the pancakes. He
took a bite. He accidentally had
used raspberry syrup. He gave therest of the pancake to his sister.

The man made pancakes. He
poured syrup on the pancakes. He
took a bite. He accidentally had
used raspberry syrup. He gave therest of the pancake to his sister.

The man made pancakes. He
poured syrup on the pancakes. He
took a bite. He accidentally had
used raspberry syrup. He gave therest of the pancake to his sister.

GLUCOSEGLUCOSEGLUCOSEThe man accidentally uses
raspberry syrup.

The man feel(s) regret.

Causes

CausesThe man feel(s) regret.

Results in

The man posses(es)pancakes.

Reasoning with Context Extension (Random Insertion)
The man made pancakes. He
poured syrup on the pancakes. Hetook a bite. He accidentally had
used raspberry syrup. He gave therest of the pancake to his sister.

The man made pancakes. He
poured syrup on the pancakes. Theman eats. He took a bite. He accidentally had used raspberry syrup.
He gave the rest of the pancake tohis sister.

The man made pancakes. He
poured syrup on the pancakes. Theman eats. He took a bite. He accidentally had used raspberry syrup.
The man has a bad taste. He gave
the rest of the pancake to his sister.

GLUCOSEGLUCOSEGLUCOSEThe man takes a bite.

Causes/Enables

The man eats.

CausesThe man feel(s) regret.

Results inThe man posses(es)pancakes.

Contextualized Common Sense
Most existing methods for making multi-step commonsense reasoning suffer from two issues:• Local contextualization: Existing commonsense knowledge resources ignored common sense's contextual na-ture, causing reasoners to apply knowledge inappropriate for the context.
• Global contextualization: Existing commonsense reasoners lack the mechanism for effective information shar-ing, leading to repetitive or incoherent reasoning.
Images generated using OpenAI DALL-E 2 using the prompts “Alex is at the airport for a business trip.” (left)
and “Zac is at the airport for a Hawaii vacation.” (right).

AI-Child Collaborative Reasoning

By enabling collaboration andfeedback between humans andAIs, AIs can more directly acquire
the knowledge and reasoning
ability from humans, and humans
will also better understand thecapabilities and limitations of AIs.

Example entries from GLUCOSE for the story context: “Gage was riding his bike. A car turned in front of him. Gage
turned his bike sharply. He fell off of his bike. Gage skinned his knee.”

Experiments and Results
Four random-walk-based multi-step commonsense reasoners with different context
extension mechanisms were implemented and we randomly sampled 50 stories; foreach story, five three-step reasoning paths were sampled from each reasoner. Eachreasoning path is rated a score from 0 (worst) to 3 (best).

To address the first issue, we adopted GLUCOSE: a contextualized commonsense knowledge resource(Mostafazadeh et al., 2020). For the second issue, we introduced context extension: a simple yet effective mechanism for contextualization. In short, the mechanism maintains an informative reasoning context by inserting inferred information into the reasoning context in a coherent manner.
Context extension:• sustains contextualization over multiple reasoning steps.

Common sense will allow AIs tobetter understand and respond tohuman interactions and enable
AIs to explain themselves in a human-understandable way.

GLUCOSE (Mostafazadeh et al., 2020)• allows a reasoning step to leverage all previous steps.
• can be easily adopted by any search-based reasoner.

References
Davis, Ernest, and Gary Marcus. "Commonsense reasoning and commonsense knowledge in artificial intelligence." Communications of the ACM 58.9 (2015): 92103.

Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901.
Mostafazadeh, Nasrin, et al. "GLUCOSE: GeneraLized and COntextualized Story Explanations." Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.
Sap, M., LeBras, R., Fried, D., & Choi, Y. (2022). Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (arXiv:2210.13312). arXiv. https://doi.org/10.48550/arXiv.2210.13312
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2023). Dissociating language and thought in large language models:A cognitive perspective (arXiv:2301.06627). arXiv. https://doi.org/10.48550/arXiv.2301.06627

We observed a statistically significant improvement in reasoning quality when context extension is used. Additional experiments also showed that GLUCOSE benefits
from the sustained contextualization enabled by context extension. Although context extension is an effective information-carrying mechanism, care should be taken
to ensure the coherency of the information.