Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? (2024)

Abstract: Building machines with commonsense has been a longstanding challenge in NLP due to the reporting bias of commonsense rules and the exposure bias of rule-based commonsense reasoning. In contrast, humans convey and pass down commonsense implicitly through stories. This paper investigates the inherent commonsense ability of large language models (LLMs) expressed through storytelling. We systematically investigate and compare stories and rules for retrieving and leveraging commonsense in LLMs. Experimental results on 28 commonsense QA datasets show that stories outperform rules as the expression for retrieving commonsense from LLMs, exhibiting higher generation confidence and commonsense accuracy. Moreover, stories are the more effective commonsense expression for answering questions regarding daily events, while rules are more effective for scientific questions. This aligns with the reporting bias of commonsense in text corpora. We further show that the correctness and relevance of commonsense stories can be further improved via iterative self-supervised fine-tuning. These findings emphasize the importance of using appropriate language to express, retrieve, and leverage commonsense for LLMs, highlighting a promising direction for better exploiting their commonsense abilities.

heading: 1 Introduction
text: Building machines with commonsense has been a longstanding goal in AI and NLP (McCarthy, 1959; Brachman and Levesque, 2023). Despite advancements in large language models (LLMs), incorporating commonsense knowledge in these models remains a significant challenge (Ismayilzada et al., 2023; Bian et al., 2023; Li et al., 2022), due to the reporting bias of commonsense knowledge and the exposure bias of commonsense reasoning (Gordon and Van Durme, 2013; Shwartz and Choi, 2020).
*Corresponding Author
The reporting bias arises because many aspects of commonsense are rarely stated explicitly in language. For example, “A person is late” may appear more frequently than “A person arrives on time” in text corpora (Gordon and Van Durme, 2013). Furthermore, commonsense rules are often left implicit and omitted in human language reasoning, leading to exposure bias. For example, the commonsense rule “humans need air to breathe” is usually ignored in cases like “The room was getting too stuffy, and I opened the windows” as it is commonly known.
To enhance the commonsense ability of NLP models, current studies usually express commonsense as rules. For instance, commonsense rules structured as knowledge graphs of concepts and events (Ilievski et al., 2021; Hwang et al., 2021; Sap et al., 2019a; Speer et al., 2017) are incorporated to support rule-based logical reasoning (Zhang et al., 2023b; Wang et al., 2023c,d,e,f; Liu et al., 2023c). Recently, as studies reveal that LLMs like GPT3 (Brown et al., 2020) and ChatGPT (OpenAI, 2022) have already learned abundant commonsense (Shwartz et al., 2020), there is a current trend to
4023
extract commonsense knowledge from the models’ memory, also expressed as rules like in Figure 1, and enhance LLMs by reintegrating this knowledge into the models (Liu et al., 2023a; Yao et al., 2023; Liu et al., 2022b; West et al., 2022).
However, commonsense is more than just rules (Brachman and Levesque, 2022; Mostafazadeh et al., 2020). Humans acquire commonsense by recognizing prototypical patterns, extracting memories of similar past experiences, and contrasting them with the current novel situation to make decisions, as supported by psychological studies (Schacter and Addis, 2007; Klein, 2004; Tulving, 2002; Schank, 1983; Schank and Abelson, 1977). Our commonsense is often conveyed and passed down through stories such as myths and fairy tales (Cassirer et al., 1946), with only a limited portion expressed in rules. Renowned AI theorist and cognitive psychologist Roger Schank argues in his book “Tell Me a Story: Narrative and Intelligence” that “knowledge is stories” (Schank, 1995). He emphasizes that humans struggle to learn and remember abstract rules derived from past experiences but can more easily remember a good story, because “stories give life to past experience”.
As a result, human-written text corpora mainly convey commonsense through stories, with limited instances of explicit rules and logical reasoning. In this way, models trained on these corpora acquire commonsense and reasoning abilities implicitly. Studies show that LLMs exhibit a strong storytelling ability, generating narratives that adhere to real-world logic (Bhandari and Brennan, 2023; Eldan and Li, 2023; Wen et al., 2023; Jiayang et al., 2023). However, these models may not effectively learn commonsense rules and explicit reasoning through mimicking human behaviors, as shown by recent studies (Bian et al., 2023; Li et al., 2022).
These observations lead to a critical question: Which is the better commonsense expression for talking with LLMs—rule or story? Specifically, this paper aims to answer the following two questions: (1) Which expression is more effective for retrieving commonsense from the memory of LLMs? (2) Which expression is more suitable for LLMs to leverage commonsense in solving problems?
To answer the questions, we systematically compare stories and rules as commonsense expressions for talking with LLMs. We use a total of 28 commonsense QA datasets for experiments. For the first question, we instruct LLMs to generate stories and rules based on commonsense questions,
as shown in Figure 1. We compare the confidence and the accuracy of commonsense generation using stories and rules, showing that LLMs are more confident and more accurate at retrieving commonsense as stories than as rules. For the second question, we compare the confidence of generating the correct answers with stories or rules as contexts, showing that LLMs can more confidently leverage stories than rules for reasoning. The QA accuracy results further demonstrate that the story is a more effective commonsense expression for answering questions regarding daily events, while the rule is more effective for scientific commonsense QA. This phenomenon aligns with the reporting bias of commonsense in the text corpora. Moreover, stories and rules can complement each other, i.e., combining them can further enhance the answer accuracy.
In-depth analyses reveal two main issues in generating commonsense stories: commonsense hallucination and semantic drifting. To address these problems, we propose an iterative self-supervised fine-tuning (self-SFT) method. We ask the model to generate stories given the training set of 8 datasets and design a scoring method to rank the stories based on their consistency with commonsense and similarity with the question. We filter the stories based on the scores and use them to fine-tune the model. The tuned model is then used to generate stories in the next iteration. Experimental results show that the self-SFT method leads to further accuracy improvements, highlighting the potential for LLMs to self-improve their commonsense abilities.
The main contributions of this paper are:
1. We systematically investigate and compare the effects of using stories and rules as commonsense expressions for retrieving and leveraging commonsense in LLMs. To our best knowledge, this is the first study to investigate the effects of specific commonsense expressions in LLMs.
2. We show that the story is a more effective expression for retrieving commonsense from LLMs and for leveraging commonsense in answering questions regarding daily events.
3. We identify two main issues that hinder commonsense story generation: commonsense hallucination and semantic drifting, and propose an iterative self-SFT method to improve the accuracy and relevance of stories generated by LLMs.

heading: 2 Background
text: Commonsense QA. Answering commonsense questions has become one of the standard tasks for evaluating LLMs (Srivastava et al., 2022). In this paper, we use 28 commonsense QA datasets for experiments, covering different domains of commonsense. These datasets are CommonsenseQA (Talmor et al., 2019), OpenBookQA (Mihaylov et al., 2018), Winograd Schema Challenge (WSC) (Levesque et al., 2012), PIQA (Bisk et al., 2020), SocialIQA (Sap et al., 2019b), ARC (easy and challenge set) (Clark et al., 2018), QASC (Khot et al., 2020), HellaSWAG (ActivitiNet and WikiHow set) (Zellers et al., 2019), NumerSense (Lin et al., 2020), AI2 Science Questions (AI2Sci, elementary and middle school set) (AllenAI, 2017), CommonsenseQA 2.0 (Talmor et al., 2021), SWAG (Zellers et al., 2018), WinoGrande (Sakaguchi et al., 2020), Com2Sense (Singh et al., 2021), SciQ (Welbl et al., 2017), QuaRel (Tafjord et al., 2019a), QuaRTz (Tafjord et al., 2019b), CycIC, ComVE (Task A) (Wang et al., 2019), COPA (Roemmele et al., 2011), PROST (Aroca-Ouellette et al., 2021), CODAH (Chen et al., 2019), Story Cloze Test (SCT) (Mostafazadeh et al., 2016), αNLI (Bhagavatula et al., 2020), and WinoVenti (Do and Pavlick, 2021). We use their development set for evaluation.
Commonsense knowledge and knowledgeaugmented reasoning. In commonsense research, there is a growing consensus that integrating knowledge can improve the commonsense ability of NLP models (Bian et al., 2021). Typically, commonsense knowledge is expressed by concise and clear rules as either triples of <head, relation, tail> like in ConceptNet (Speer et al., 2017), or simple sentences like in Open Mind Common Sense (Singh et al., 2002). Recently, researchers turn to retrieving commonsense knowledge from pre-trained LLMs like GPT-3, assuming LLMs have already learned abundant commonsense from large-scale human-written text corpora (Wang et al., 2023a; Chen et al., 2023b; Liu et al., 2023a; Li et al., 2023; Yao et al., 2023; Wang et al., 2023b; Zhou et al., 2022; Wang et al., 2022a; Wei et al., 2022; Liu et al., 2022a,b; Gu et al., 2022; Yu et al., 2022; Paranjape et al., 2021; West et al., 2022; Bosselut et al., 2021; Shwartz et al., 2020; Latcinnik and Berant, 2020; Rajani et al., 2019). They instruct LLMs with prompts and examples to generate commonsense rules as concise sentences. In this paper, we follow their assumption and compare stories
and rules as the commonsense expression for retrieving commonsense knowledge from LLMs.
There have been numerous works that inject commonsense rules into NLP models to improve commonsense reasoning, either by pre-training on knowledge bases (Ma et al., 2021; Chang et al., 2020; Mitra et al., 2019; Zhong et al., 2019), or incorporating knowledge rules in the input of language models (Shi et al., 2023; Wang et al., 2023c,f; Lal et al., 2022; Bian et al., 2021). There is also a chain of works that use graph-based reasoning for inference (Wang et al., 2023e; Yasunaga et al., 2021; Lv et al., 2020; Lin et al., 2019). In this paper, we exploit the commonsense knowledge in the memory of LLMs as stories or rules to support commonsense reasoning.
Story generation by LLMs. Generating fluent and coherent stories using NLP models has been a crucial component of computational creativity. Recent studies have found that LLMs can perform well on story generation (Eldan and Li, 2023; Wen et al., 2023; Peng et al., 2022). Bhandari and Brennan (2023) compare stories generated by OPT (Zhang et al., 2022), LLaMA (Touvron et al., 2023), and Alpaca (Taori et al., 2023) with human-written stories, showing that these two kinds of stories exhibit a remarkable similarity in terms of readability and topics. The LLM-generated stories are even more accessible than traditional children’s stories. Xie et al. (2023) compared GPT-3 with story generation models before LLMs, demonstrating that LLMs generate stories of significantly high quality, even comparable with human authors. In this paper, we analyze and exploit the inherent commonsense embedded in the storytelling ability of LLMs.
Large language models. This study focuses on three LLMs: ChatGPT (OpenAI, 2022), Alpaca (Taori et al., 2023), and Vicuna (Chiang et al., 2023; Zheng et al., 2023). ChatGPT was developed by OpenAI and is one of the state-ofthe-art LLMs demonstrating robust abilities for generating human-like text. Our experiments are conducted using the gpt-3.5-turbo API. Alpaca is an open-source LLM with 7B parameters that achieves a good balance between performance and efficiency. It was fine-tuned from the LLaMA-7B model (Touvron et al., 2023) using 52K instructions gathered via a “self-instruct” methodology (Wang et al., 2022b). Vicuna is another open-source LLM trained by fine-tuning the LLaMA2 model on about 125,000 user-shared conversations with ChatGPT from ShareGPT. We use Vicuna v1.5 with 7B pa-
rameters. During experiments, we set the temperature of these LLMs to 0 when answering questions and to default when generating stories and rules.

heading: 3 Commonsense Retrieval from LLMs as Stories and Rules
text: In this section, we answer the first question: Which expression, story or rule, is more effective for retrieving commonsense from the memory of LLMs? First, we compare the confidence in generating stories and commonsense rules. Then, we employ an automatic evaluation to assess the accuracy of commonsense within the generated stories and rules.

heading: 3.1 Confidence of Commonsense Generation
text: To assess the confidence in generating stories and rules using LLMs, we ask LLMs to write corresponding stories of past experiences and commonsense rules given questions from commonsense QA datasets as input (as shown in Figure 1). Specifically, we randomly select 100 questions from each dataset and instruct Alpaca and Vicuna models to generate 5 stories and 5 rules using specific prompts (shown in Table 4 in Appendix A). We use perplexity to indicate the generation confidence of LLMs, which has been a longstanding confidence measure for language models (Jiang et al., 2021). However, there is a notable difference in word usage between stories and rules. Stories tend to incorporate less common words such as people’s names and specific scenes, while rules typically consist of more general and common words. To account for this variation in word frequencies, we subtract the text perplexity with the perplexity of randomly shuffled word lists, which is a common practice in psychological linguistic studies to account for the word frequency effects (Humphries et al., 2006; Pallier et al., 2011; Zaccarella et al., 2017; Labache et al., 2019; Zhang et al., 2023a). The confidence is measured by the “Perplexity Reduction (PR)”:
PR(t) = PPL(shuffle(t))− PPL(t) (1) Here, PPL(·) denotes the perplexity calculation function, and shuffle(t) refers to the shuffling of the text t by words. A higher PR indicates that the LLM is more confident with the text.
Finding 1. When retrieving commonsense from LLMs, stories can result in more confident commonsense generation than rules. The results in Figure 2 show that stories have significantly higher PR than rules for both Vicuna and Alpaca
models (p ≪ 0.001). This effect is more obvious for Vicuna, and we believe this is because Vicuna has better storytelling and instruction-following abilities than Alpaca. These observations align with the reporting bias, where commonsense rules are less prevalent in the training text of LLMs, resulting in lower confidence in generating them. Conversely, human language is more likely to convey commonsense as stories, so LLMs develop an ability to generate stories with more confidence.

heading: 3.2 Accuracy of Commonsense Generation
text: To assess which expression incorporates more accurate commonsense knowledge, we ask ChatGPT to determine if each story or rule aligns with commonsense, responding with either “yes” or “no” (prompt shown in Table 6 in Appendix A). We use the same 100 questions for each dataset in Section 3.1, randomly selecting one story and one rule for evaluation, which results in a total of 2,800 stories and 2,800 rules for each model. It is important to note that these annotations are used solely for the purpose of comparing stories and rules regarding commonsense accuracy under the same conditions, both annotated by ChatGPT, rather than implying absolute accuracy. Further human evaluation results for a small set of stories and rules are shown in Table 9 in Appendix C.1.
Finding 2. LLMs generate more accurate stories than rules in terms of commonsense. Table 1 shows that the commonsense accuracy of stories is higher than that of rules for all three mod-
els (p < 0.0005). This verifies that the story is a more accurate commonsense expression than the explicit commonsense rule for retrieving commonsense knowledge from LLMs, highlighting the potential of stories as valuable commonsense sources.

heading: 4 Leveraging Commonsense in Stories and Rules for Problem Solving
text: This section answers the second question: Which expression, story or rule, is more suitable for LLMs to leverage commonsense in solving problems? We compare the confidence of reasoning with stories or rules as contexts. Then, we assess the performance of commonsense QA by employing either stories or rules as contexts and perform detailed analyses.

heading: 4.1 Confidence in Commonsense Reasoning
text: To evaluate the confidence of reasoning and generating the correct answer in commonsense QA given stories and rules as contexts, we compare the perplexity reduction of sequence “context, question (and options if applicable), correct answer” with context as either stories or rules. Specifically, we employ the same set of 100 questions, the stories, and the rules of each dataset in Section 3.1. The perplexity reduction is calculated similarly:
PR([c, q, a])
= PPL([shuffle(c), q, a])− PPL([c, q, a]) (2)
where c, q, and a are the context, question, and correct answer, respectively.
Finding 3. LLMs are more confident in commonsense reasoning based on stories than on rules. Figure 3 shows a significantly higher perplexity reduction when generating the correct answers for commonsense questions using stories

heading: Model = Vicuna

heading: Acc < -1
text: than using rules as contexts (p < 0.0003). This discrepancy reflects the exposure bias in commonsense reasoning: explicit rules are seldom used by people to reason and solve commonsense problems, resulting in more sparse examples in the text corpora for training LLMs. This finding affirms that LLMs can more naturally leverage stories of past experiences for reasoning in commonsense QA.

heading: 4.2 Effectiveness in Commonsense QA
text: Next, we assess the accuracy of zero-shot commonsense QA using stories or rules, comparing them with a baseline setting without contextual commonsense. For each question, LLMs first generate five stories or rules, which are then concatenated as context for answering questions using the same model (with prompts shown in Table 5 in Appendix A). The results are presented in Table 2, and the accuracy differences between using stories and rules are shown in Figure 4 for Vicuna and Figure 7 in Appendix C.2 for the other two models.
Finding 4. Story is a more effective commonsense expression for answering questions regarding daily events, while the rule is more effective for scientific commonsense QA, which aligns with the reporting bias of commonsense. As shown in Table 2 and Figure 4, for datasets like HellaSWAG, SWAG, SCT, and αNLI, which involve daily events and stories, leveraging stories as context leads to higher accuracies across all models (only except ChatGPT on the SCT dataset). These datasets involve tasks like selecting correct followup behaviors for sequences of events (HellaSWAG and SWAG), choosing appropriate story endings (SCT), or determining events between the beginning and end of a script (αNLI). We attribute this
Datasets ChatGPT (gpt-3.5-turbo) Vicuna AlpacaBase Story Rule Both Base Story Rule Both Base Story Rule Both
to the influence of reporting bias of different types of commonsense in text corpora, which shapes the commonsense ability of LLMs. Commonsense in our daily life events, such as “how to add toothpaste onto a toothbrush”, is subject to more pronounced reporting bias than other forms of commonsense (Shwartz and Choi, 2020). This type of commonsense is inherently more implicit and, consequently, more suitable to be expressed as stories.
In contrast, datasets focusing on scientific commonsense at the elementary or middle school level, including OpenBookQA, ARC, QASC, AI2Sci, and SciQ, show better performance when provided with commonsense rules. This is because scientific knowledge is typically structured as rules in textbooks and encyclopedias. The questions and options often include scientific terms, and scientific concepts and descriptions are not commonly presented in the form of stories in the training corpora.
Finding 5. Stories and rules can complement each other, further enhancing the QA accuracy of LLMs. As shown in Table 2, when employing both stories and rules as contextual inputs, LLMs can achieve higher QA accuracy on 10, 12, and 13 datasets compared to using either stories or rules alone. This highlights that the combination
of both stories and rules enables LLMs to leverage the unique strengths of each, resulting in a more comprehensive and precise understanding of the presented questions and underlying commonsense. For example, although rules are better for expressing scientific commonsense, stories still play a crucial role in scientific commonsense QA by providing essential contextual information. As shown in Table 2, using both stories and rules on the OpenBookQA dataset further improves answer accuracy for all three models.

heading: 4.3 Analyses

heading: 4.3.1 Error Analysis
text: To gain deeper insights into the influence of generated stories, we conduct an error analysis. We focus on questions that are initially answered correctly by the Vicuna model, but the answers change to incorrect when considering the stories as context. We randomly select 10 such questions from each dataset, except for the AI2Sci-elem dataset which has only 8 such questions. Each question comprises five stories, resulting in a total of 1390 stories for error analysis. We manually classify error cases into 5 primary types: Semantic Drifting (34.0% – the story drifts away from the question), Uncom-
mon or Incorrect Scenarios (26.6% – the story does not represent common real-world situations or contains errors), Incorrect Answering (18.6% – the story is accurate, but the predicted answer is wrong), Inconsideration of Options (16.2% – the story does not align with any answer options), and Inclusion of Wrong Options (4.6% – the story emphasizes a wrong answer). The pie chart is shown in Figure 8 in Appendix C.3.
This analysis highlights two main issues of the stories: commonsense hallucination and semantic drifting. A commonsense hallucination occurs when LLMs are misled by incorrect answer options, generating stories that are uncommon or against commonsense, leading to the uncommon or incorrect scenarios error. Semantic drifting refers to LLMs generating stories whose topics deviate from the question, making them unhelpful in answering the question. These two error types are the primary reasons for incorrect model answers, accounting for over 60% of total errors. Besides, 18.6% of the stories are correct and relevant, yet the model fails to effectively use them to answer questions correctly, suggesting room for further improvement in LLMs’ ability to leverage contextual information.
To quantitatively analyze the generated stories, we employed two scoring methods: commonsense scores and BERT similarity scores. For the commonsense score, we use the Vera model (Liu et al., 2023b), a T5-based model trained on extensive commonsense statements from knowledge bases. This model outputs a score of the correctness for a given text according to commonsense. For the BERT similarity score, we calculate the cosine similarity between semantic representations of stories and questions using the BERT-large model (Devlin et al., 2019). The two scores range between 0 and 1.
There is a correlation between error types and
the corresponding scores. Figure 5 shows that stories with semantic drifting have significantly lower semantic similarities with the questions than the overall average and other error types. Moreover, stories describing uncommon or incorrect scenarios show both lower similarity and commonsense scores in contrast to the overall average and other error types except for semantic drifting. These findings further support the commonsense hallucination and semantic drifting issues.

heading: 4.3.2 Influence of Story on Answer Accuracy
text: Further analysis shows notable correlations between answer accuracy and the two scores, the commonsense score and BERT similarity score, at the dataset level (shown in Figure 9 and 10 in Appendix C.4). Specifically, answer accuracy demonstrates a robust correlation with the commonsense score (Pearson coefficient 0.612, p < 0.001). In comparison, the correlation with the BERT similarity score is weaker but still positive (Pearson coefficient 0.226, p = 0.003). This is because a story involving commonsense hallucination may mislead the model by providing incorrect information, leading to wrong answers, while a story deviating from the question merely offers no relevant information, resulting in a weaker influence. These observations underscore the substantial influence of semantic drifting and commonsense hallucination issues on commonsense QA based on stories.

heading: 4.3.3 Analysis of Datasets Related to Negation
text: An interesting phenomenon of leveraging commonsense in stories is that LLMs are still not good at handling negations in commonsense. On datasets that involve negation, including CommonsenseQA 2.0 and Com2Sense which require models to assess statement correctness, and ComVE (Task A) which requires identifying statements contradicting commonsense, using rules demonstrates higher accuracy compared to using stories (Table 2). Specifically, the Vicuna model tends to more frequently give incorrect “yes” responses to questions with a correct answer of “no” (69.1% of error cases in CommonsenseQA 2.0 and 90.2% in Com2Sense) than the opposite cases. Furthermore, for error cases where the correct answer is “no”, the model generates stories with more commonsense errors (the average commonsense scores of stories for incorrectly answered “no” questions are significantly lower than the opposite with p < 0.003). This disparity indicates a challenge for LLMs in han-
dling negations within commonsense (Chen et al., 2023a). Training corpora include few negative commonsense examples like “a stapler is not used for sewing”, resulting in LLMs generating more hallucinations misled by the given incorrect statements.

heading: 5 Iterative Self-Supervised Fine-Tuning
text: After identifying the two issues in generating stories of past experiences, this section presents our iterative self-SFT approach to address these issues.

heading: 5.1 Method
text: Our self-SFT method contains three steps in each iteration: generating, filtering, and training.
In the generating step, we use LLMs to generate stories for questions in the training set. Specifically, we generate five stories for each question. In the filtering step, we first select the generated stories based on changes in the responses to the questions, i.e., a story is considered helpful if it can rectify an initially incorrect response as correct given it in the context. These helpful stories are then scored using a scoring method. The design of the scoring method is crucial for mitigating the commonsense hallucination and semantic drifting issues. We employ the two scores in Section 4.3: the commonsense score by the Vera model for commonsense correctness and the BERT-based semantic similarity for relevance. The total score is the sum of the two scores. We retain the K% top-scored stories for the subsequent training step. In the training step, we fine-tune an LLM using the filtered stories as output, with inputs following the prompt in Section 3.1. The fine-tuned model is used for the generating step in the next iteration.

heading: 5.2 Experiments
text: Following Liu et al. (2023a) and our analyses, we train our model on 8 datasets: OpenBookQA, AI2Sci (elementary and middle school set), WinoGrande, HellaSWAG (ActivitiNet and WikiHow set), SWAG, and αNLI. The other datasets are used for unseen evaluation. We fine-tune the Vicuna model using LoRA tuning (Hu et al., 2022), with
hyper-parameters in Appendix D. The commonsense ability of LLMs can be further self-improved via iterative self-supervised fine-tuning. Table 3 shows that on the datasets used for fine-tuning (seen), the average accuracy at all three iterations outperforms the original Vicuna model without SFT. The most significant improvement is at iteration 2, rising from 51.96% to 53.26%. This verifies the effectiveness of selfSFT in enhancing the quality of generated stories. Furthermore, self-SFT demonstrates improvements on unseen datasets at iteration 1, suggesting the method’s ability to generalize to other commonsense QA datasets not seen during training. Performance decreases at later iterations can be attributed to over-fitting on seen datasets. Ablation and hyperparameter studies are shown in Appendix E.
Our approach is effective in addressing the issues and improving the quality of generated stories. To further assess the effect of our method in mitigating commonsense hallucination and semantic drifting issues during story generation, we analyze the score variations across training iterations. Figure 6 shows that, along with the training iterations, the total score (sum of commonsense score and BERT similarity) consistently improves for both the training and testing phases.

heading: 6 Conclusion
text: This paper systematically compares stories and rules as commonsense expressions to retrieve and leverage commonsense knowledge in LLMs. Experimental results show that the story is a better expression for retrieving commonsense from LLMs. LLMs generate stories with more confidence and higher commonsense accuracy than rules. Moreover, the story is a more effective commonsense expression for answering questions regarding daily events, while the rule is more effective for scientific commonsense QA. This phenomenon aligns
with the reporting bias of commonsense. Stories and rules can complement each other to further enhance answer accuracy. We provide further insights through in-depth analyses, highlighting two challenges in generated stories: commonsense hallucination and semantic drifting. We show that the correctness and relevance of commonsense stories can be improved via iterative self-supervised fine-tuning, underscoring the potential for selfimprovement in the commonsense ability of LLMs.
This paper suggests a new perspective, going beyond the common practice of expressing commonsense as rules. Our results and findings emphasize the importance of using the appropriate language to express, retrieve, and leverage commonsense for LLMs to further exploit their potential. The full extent of LLMs in handling commonsense is not yet fully realized, calling for future research to refine and improve the commonsense abilities of LLMs.

heading: Limitations
text: This study specifically investigates several popular LLMs, including ChatGPT, Vicuna, and Alpaca, while excluding the exploration of other LLMs such as GPT-4, Mistral (Jiang et al., 2023), and Google’s Bard (Thoppilan et al., 2022). The selection of models is based on considerations of popularity, availability, and cost. Future research could provide valuable insights by examining whether similar findings hold for these models and conducting performance comparisons with the models included in this study.
The assessment of commonsense accuracy in Section 3.2 relies on automatic labeling by ChatGPT. Therefore, the accuracy presented in Table 1 serves solely for comparing stories and rules, and should not be regarded as absolute accuracy. To ensure the quality of this automatic evaluation, we conduct a manual evaluation of a small set of stories and rules, as shown in Table 9 in Appendix C.1. However, due to its labor-intensive nature, manually labeling all generated stories and rules is impractical for us. Future studies should incorporate more extensive human evaluations to provide a more comprehensive and nuanced understanding of the commonsense knowledge generated by LLMs.
This paper employs two scores to assess the quality of generated stories and to filter training data. However, these two scores—the commonsense score and BERT similarity score—may inherently exhibit biases as they rely on model-based
scoring methods. For instance, the Vera model for commonsense score is fine-tuned from the T5 model using commonsense statements synthesized from commonsense knowledge bases, potentially leading the score to favor statements aligned with these knowledge bases. We anticipate detailed analyses of potential biases in these models and scores.
Lastly, in the manual error analysis process, we discover that some commonsense QA datasets are not actually asking about commonsense, despite being recognized as commonsense questions. Some datasets are automatically constructed based on knowledge graphs, probably leading to unreasonable questions or insufficient information to answer the question. Other manually constructed datasets may face the challenge that different annotators may have entirely different understandings of what commonsense is. We follow the common practices in commonsense studies and use these datasets in our commonsense QA experiments. Further investigation into the existing commonsense QA datasets is a task for future studies.

heading: Acknowledgements
text: We sincerely thank all anonymous reviewers for their insightful comments and valuable suggestions. This research work is supported by the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDA27020200, and the National Natural Science Foundation of China under Grants No. 62122077, 62106251, and 62306303.

heading: A Prompts
text: The prompts we use in this paper for instructing LLMs to generate stories and rules are shown in Table 4. The common prefix which contains the question and the answer options (if applicable) is added before the two prompts below. To avoid responses such as “As an AI language model, I do not have past experiences”, the specific name “Jane” is used instead of “you” in our prompts. It is worth noting that the choice of the name is arbitrary, and any name can be used. The potential influence of name biases in LLMs is a topic for future study.
The prompts for answering questions are shown in Table 5. The prompt for automatically evaluating the stories and rules is shown in Table 6.
These prompts are constructed through a prompt engineering process. This involves testing and comparing different prompt variations to select the most effective ones. Through this process, we can find the prompts to ensure optimal performance in guiding LLMs to generate high-quality stories and rules and to answer questions effectively.

heading: B Details of Commonsense QA Datasets
text: Table 7 provides information on the number of questions, question types, and accuracy when randomly selecting an answer option for each commonsense QA dataset. All these datasets are in English. It is important to note that, for HellaSWAGact.net, HellaSWAG-wikihow, SWAG, and PROST datasets, we randomly sample 1,000 questions from each of their development sets. This decision is made due to the excessively large number of questions in their development sets, which would have required significant time and computational resources for evaluation (3,243 for HellaSWAGact.net, 6,799 for HellaSWAG-wikihow, 20,006 for SWAG, and 18,736 for PROST). An example from each dataset is presented in Table 8.

heading: C Further Analyses of Generated Stories

heading: C.1 Manual Evaluations of Commonsense Accuracy in Generated Stories and Rules
text: To further support the commonsense accuracy results presented in Section 3.2, we conduct a manual evaluation of the generated stories and rules. For each model, we sample 50 stories and 50 rules from the Commonsense QA dataset and manually label each one for commonsense correctness. The annotators are the authors of this paper. A story or rule is marked as incorrect if it contains a description that contradicts real-world commonsense. The results, shown in Table 9, indicate that the commonsense accuracy of stories is higher than that of rules across all three models. This further supports our findings in Section 3.2, where ChatGPT is used to automatically assess the commonsense accuracy
of a larger set of stories and rules.

heading: C.2 Accuracy Differences between Using Stories and Rules for ChatGPT and Alpaca
text: We show the accuracy differences between using stories and rules for the ChatGPT and Alpaca models in Figure 7, as supplementary to Figure 4.

heading: C.3 Error Analysis
text: We show the pie chart of our error analysis in Figure 8. From the figure we can see that uncommon or incorrect scenarios and semantic drifting are the primary reasons for incorrect model answers, accounting for over 60% of total errors.

heading: C.4 Correlation Between Answer Accuracy and the Two Scores of Stories
text: We plot the correlation between answer accuracy and the commonsense score in Figure 9, and the BERT similarity score in Figure 10. The dashed lines are connected between the results of using stories and rules of the same datasets, which further reveal positive correlations of the two scores between using stories and using rules on most datasets

heading: C.5 Length of Stories and Rules
text: We calculate the average lengths of a single story and rule generated by the models used in our study. Lengths are calculated as the number of words while excluding stop words. The results are shown in Table 10. This analysis reveals that, on average, stories are longer than rules. However, this difference in length does not imply a difference in the amount of information conveyed. Stories tend to include longer sentences with detailed descriptions and personal feelings, incorporating adjectives and adverbs, while rules are typically more concise and straightforward. Despite these stylistic differences, both formats provide a comparable amount of commonsense information.

heading: D Hyper-parameters for Self-SFT
text: For the filtering step, we randomly select 200 questions initially answered incorrectly by the Vicuna and can be corrected with at least one generated story for each dataset. The filter ratio K is set to 50%, resulting in 2,269, 2,300, and 2,447 training stories across the 8 datasets for three iterations.
For the training step, we use LoRA tuning (Hu et al., 2022) for training the Vicuna model. LoRA is a parameter-efficient fine-tuning method that has
become a common practice in the LLM era to reduce the overhead of expensive adaptations. The
hyper-parameters for LoRA are rank r = 16 and α = 16. Models are fine-tuned for 3 epochs in each iteration with a batch size of 64 and a learning rate of 3e-4.

heading: E More Results and the Ablation Study of Self-SFT

heading: E.1 Full Accuracy on All Datasets
text: We show the accuracy before and after self-SFT of each dataset in Table 11. Across 22 datasets, our self-SFT method consistently outperforms the original Vicuna model without SFT (7 seen and 15 unseen) and shows the most accuracy improve-
ments at iterations 1 and 2.

heading: E.2 Effect of Iterative Self-SFT
text: We compare our self-SFT method with a naive SFT method without iteration and scoring. In this ablated method, we only filter the stories generated by Vicuna on the training sets that can rectify an initially incorrect answer to correct, and use these stories for fine-tuning. The results are shown in Table 11.
From the results, we can see that the ablated method performs worse than our self-SFT method on all datasets, and even worse than the original Vicuna model without fine-tuning on some datasets. This further verifies that our iterative scoring and filtering mechanism is crucial for alleviating the semantic drifting and commonsense hallucination issues of story generation.

heading: E.3 Effect of Hyper-parameter: Filter Ratio
text: We investigate the influence of the filter ratio K% by experimenting with various values of K. We assess the average answer accuracy with K taking values from the set {10, 30, 50, 70, 90, 100} using a single iteration, testing on randomly selected 100 questions from each dataset (2,800 questions in total). We maintain a consistent number of training steps across different K values. As illustrated in Figure 11, our method achieves optimal results when K% is set to 50%.

heading: E.4 An Example of Generated Stories Alongside the Self-SFT Iterations
text: We show an example of the generated stories by the Vicuna model before (iteration 0) and alongside (iteration 1-3) the self-SFT iterations. The commonsense score and BERT similarity both increase along with the iteration, and the answer becomes
correct after iteration 2. This further demonstrates the effectiveness of our self-SFT method in generating better stories.

heading: F Computational Costs
text: For the QA evaluations across the 28 datasets, we use an RTX 3090 GPU. Generating the five stories or rules for a single commonsense question typically takes about 20-30 seconds. Once the stories or rules are generated, answering a question with them as context takes less than a second. Therefore, a complete evaluation of all datasets consumes approximately 170 hours, which spans slightly over a week.
In the iterative self-SFT method employed for fine-tuning the Vicuna model, we use an A100 GPU (80G). The training step in each iteration of the self-SFT method takes around 12 minutes. Additionally, the generation step in each iteration requires about 70 hours to process the 8 datasets.

heading: G Licenses of Datasets and Models
text: The CommonsenseQA, NumerSense, HellaSWAG, SWAG, and WinoVenti datasets are licensed under the MIT License. The OpenBookQA, QASC, WinoGrande, and PROST datasets are licensed under the Apache-2.0 license. The PIQA dataset is licensed under the Academic Free License v. 3.0 license. The SocialIQA, CommonsenseQA2.0, QuaRel, and QuaRTz datasets are licensed under the CC-BY license. The SciQ dataset is licensed under the CC-BY-NC-3.0 license. The ARC and ComVE (Task A) datasets are licensed under the CC-BY-SA license. The αNLI dataset is licensed under the CC-BY-NC-4.0 license. The COPA dataset is licensed under the BSD 2-Clause license. The CODAH dataset is licensed under the Open Data Commons Attribution license. The AI2Sci, Com2Sense, CycIC, and SCT datasets have unknown licenses.
The terms of use for ChatGPT API are in https: //openai.com/policies/terms-of-use. The Alpaca and Vera models are licensed under the MIT License. The Vicuna model is licensed under the Llama 2 Community License Agreement. The BERT model is licensed under the Apache-2.0 license.

heading: Future Work
text: The potential influence of name biases in LLMs is a topic for future study. The prompts for answering questions are shown in Table 5. The prompt for automatically evaluating the stories and rules is shown in Table 6. These prompts are constructed through a prompt engineering process. This involves testing and comparing different prompt variations to select the most effective ones. Through this process, we can find the prompts to ensure optimal performance in guiding LLMs to generate high-quality stories and rules and to answer questions effectively