Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations (2024)

Abstract: We introduce CHARM, the first benchmark for comprehensively and in-depth evaluating the commonsense reasoning ability of large language models (LLMs) in Chinese, which covers both globally known and Chinese-specific commonsense. We evaluated 7 English and 12 Chinese-oriented LLMs on CHARM, employing 5 representative prompt strategies for improving LLMs’ reasoning ability, such as Chain-of-Thought. Our findings indicated that the LLM’s language orientation and the task’s domain influence the effectiveness of the prompt strategy, which enriches previous research findings. We built closely-interconnected reasoning and memorization tasks, and found that some LLMs struggle with memorizing Chinese commonsense, affecting their reasoning ability, while others show differences in reasoning despite similar memorization performance. We also evaluated the LLMs’ memorizationindependent reasoning abilities and analyzed the typical errors. Our study precisely identified the LLMs’ strengths and weaknesses, providing the clear direction for optimization. It can also serve as a reference for studies in other fields. We will release CHARM at https://github.com/opendatalab/CHARM.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11205–11228 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction Commonsense reasoning is important for the enhancement of the large language models (LLMs) (Bommasani et al., 2021; Achiam et al., 2023) towards artificial general intelligence (AGI) (Davis and Marcus, 2015), therefore requires thorough evaluations. Numerous benchmarks evaluate the commonsense reasoning of LLMs, but most are English-based, limiting non-English evaluations (Davis, 2023). This paper focuses on assessing
∗Equal contribution. †Project lead. ‡Corresponding author.
LLMs’ commonsense reasoning in a Chinese context. Currently, some commonsense reasoning benchmarks in Chinese are simply English translations (Conneau et al., 2018; Ponti et al., 2020; Lin et al., 2022), which overlooks unique Chinese cultural, linguistic, regional, and historical aspects. These factors matter when Chinese users use the LLM, hence should be included in benchmarks. To effectively tackle this, we introduce CHARM, the benchmark designed to thoroughly and in-depth assess the abilities of LLMs in Chinese commonsense reasoning. It covers two domains: globally accepted commonsense (global domain) and Chinese-specific commonsense (Chinese domain). The latter includes 7 aspects: History (H), Traditional Culture and Arts (CA), Daily Life and Customs (LC), Entertainment (E), Public Figures (F), Geography (G), and Chinese Language (L). Therefore CHARM allows a thorough evaluation of LLMs’ reasoning in a Chinese context.
Prompt strategies like Chain of Thought (CoT) (Wei et al., 2022) can significantly improve LLMs’ reasoning performance (Wang et al., 2022, 2023b). Particularly, as the training corpus of LLMs is primarily in English (Touvron et al., 2023a), studies (Shi et al., 2022; Huang et al., 2023a; Zhang et al., 2023a) have shown that for non-English reasoning tasks, some LLMs perform better when reasoning in English than the native language. We evaluated 7 English and 12 Chinese-oriented LLMs on CHARM, employing 5 representative prompt strategies. The result showed that prompt strategies’ effectiveness depends on the LLMs’ orientation and the benchmark task’s domain, which enriches prior research and guides performance assessment and strategy choice for non-English LLMs.
LLMs’ commonsense reasoning relies on memorization. Exploring the correlation between memorization and reasoning offers insights into LLMs, aiding deeper understanding and suggesting ways to enhance these abilities(Bian et al., 2023). Some
11205
benchmarks (Yu et al., 2023; Wang et al., 2023a; Fei et al., 2023) aid the research of memorizationreasoning relationships by incorporate tasks for assessing knowledge memorization and application (like reasoning). However, they used the existing and disparate datasets for different tasks, resulting in a lack of intrinsic connections between these tasks. For instance, the question Qrea tests the LLM’s reasoning with the knowledge piece K. However, in memorization tasks, there probably is not any matching questions to determine if the LLM has effectively memorized K. Hence, if the LLM fails on Qrea, it’s unclear whether due to poor reasoning or forgetfulness of K. This results in the disjointed evaluation of memorization and reasoning, failing to uncover their intrinsic links. To address this limitation, we selected suitable reasoning tasks from CHARM’s Chinese domain, and built related memorization questions for each reasoning question (see Figure 1). This design produces the closelyinterconnected reasoning and memorization tasks, therefore allows for not only the concurrent evaluation of the two abilities, but also the assessment of memorization-independent reasoning, providing the clear guidance for the LLMs’ enhancement. The contributions of this paper are as follows: • We present CHARM, the first benchmark for comprehensively evaluating the LLMs’ commonsense reasoning ability in Chinese, by encompassing not only the global but also the Chinese-specific commonsense. • We evaluated the representative prompt strategies on CHARM. Results showed that LLMs’ orientation and the task’s domain affect prompt strategy performance, which enriches previous research findings.
• In CHARM, we built closely-interconnected
reasoning and memorization tasks in Chinese commonsense domain, allowing for in-depth understanding the correlation between these abilities and precisely identifying the LLMs’ strengths and weaknesses. The design approach could serve as the reference for other fields.
2 Related Work Commonsense Reasoning Benchmarks There are lots of commonsense reasoning benchmarks, most of them are in English (Davis, 2023). Some Chinese commonsense reasoning benchmarks are directly translated from English benchmarks (Conneau et al., 2018; Ponti et al., 2020; Lin et al., 2022), which lack the Chinese specifics. There are some native Chinese benchmarks that include some Chinese-specific factors and involve commonsense reasoning to a certain extent, such as LogiQA (Liu et al., 2020, 2023), CLUE (Xu et al., 2020) and CMMLU (Li et al., 2023). However, they are not designed for commonsense reasoning, therefore containing the large portion of irrelevant tasks and questions. CORECODE (Shi et al., 2023) is the benchmark for Chinese commonsense reasoning and commonsense conflict detection, but it is not strictly designed to distinguish the Chinese-specific and global domains when compared with CHARM. In addition, CHARM has the closely-interconnected reasoning and memorization tasks, which are not included in previous commonsense reasoning benchmarks. The comparison of CHARM with previous commonsense reasoning benchmarks is shown in Table 1.
Prompt Strategy Prompt strategies such as CoT (Wei et al., 2022) can effectively boost the reasoning capabilities of LLMs (Wang et al., 2022, 2023b). Notably, as the LLM training corpus is primarily in English (Touvron et al., 2023a), research revealed that for reasoning tasks in non-English languages, some LLMs exhibit superior performance when reasoning in English as opposed to the native language (Shi et al., 2022; Zhang et al., 2023a; Huang et al., 2023a). (Kim et al., 2023) proposed a novel cross-language transfer prompt method, which uses both the source and target languages to construct examples.
Benchmarks on Correlations of Memorization and Reasoning There are benchmarks which assess both the knowledge memorization and reasoning capabilities of the LLMs within specific domains. For instance, KoLA (Yu et al., 2023),
with its focus on world knowledge, includes tasks related to knowledge memorization and application (reasoning). SeaEval (Wang et al., 2023a), emphasizing cross-language consistency and multicultural reasoning, involves tasks for cultural understanding and complex reasoning. There are also benchmarks aimed at specialized fields, like LawBench (Fei et al., 2023), which include tasks for both memorization and application.
3 CHARM
CHARM is built for comprehensive and in-depth evaluation of LLMs in Chinese commonsense reasoning and revealing the intrinsic correlation between memorization and reasoning. Therefore, CHARM covers two domains, global and Chinese, using carefully selected tasks for comprehensive coverage. In addition, we chose reasoning tasks and constructed the closely-tied memorization tasks. The construction and main features of CHARM are in Figure 1. The detailed composition of CHARM is in Table 2.
3.1 Commonsense Domain Global commonsense domain consists of universally understood commonsense. It covers objects and aspects of modern life that an individual should be aware of. It includes foundational knowledge that someone with a basic modern education is expected to know. When it involves individuals, they are globally recognized figures.
Chinese commonsense domain encompasses Chinese-specific elements. We categoried them into 7 aspects:
History (H) includes important events and figures in Chinese history, China’s dynasties, and other basic facts and shared knowledge about the history of China.
Traditional Culture and Arts (CA) encompasses Chinese traditional cultural arts, literary works, and traditional lifestyles.
Daily Life and Customs (LC) includes modern Chinese daily routines, clothing, food, housing, transportation festivals and so on.
Entertainment (E) includes the movies, television programs, music, and other entertainments in modern Chinese daily life.
Public Figures (F) encompasses the public figures well-known in Chinese society.
Geography (G) includes China’s geographical distribution, natural landscapes, and characteristic
regional cultures. Chinese Language (L) includes the fundamentals of the Chinese language, such as Chinese characters, idioms and so on.
For the two domains, especially for the above 7 aspects, we collected corresponding entities, forming the lists as shown in the Figure 1 and 5. Most of the entities were selected from Gaokao Bench1(Zhang et al., 2023b), Douban2, Hupu3. Some entities were collected with the help of searching engines. We only collected the entities that are well-known in China. These entities were then used to create the commonsense reasoning questions, which belong to the corresponding domain and aspect.
3.2 Reasoning Tasks When designing the reasoning tasks in CHARM, we beared two criteria in mind. First, the tasks should span both commonsense domains, particularly the 7 Chinese aspects. Second, the global and Chinese tasks should have identical types and settings, differing only in their commonsense domains. From the existing English commonsense reasoning datasets (Davis, 2023; Suzgun et al., 2022), we selected the following 7 tasks:
Anachronisms Judgment (AJ) necessitates the LLM to identify anachronisms in provided sentences. This involves the LLM understanding the era associated with well-known figures, items, and events to facilitate commonsense-based reasoning. Global domain questions are the mix of translations4 and handcrafted, while all Chinese domain questions are handcrafted.
Time Understanding (TU) requires the LLM infers a time (including year, date, moment, etc.) based on a given context, which necessitates the fundamental understanding of time-related commonsense and the capacity for mathematical reasoning. All question in the global domain are translations5 and all in Chinese domain are handcrafted.
Sequence Understanding (SqU) requires the LLM sort a series of entities according to time or
1Gaokao Bench is the collection of China’s university entrance exam questions, which contributes to all the 7 aspects.
2https://www.douban.com/ is the popular user-centric cultural review platform in China, which mainly contributes to the Entertainment aspect.
3https://www.hupu.com/ is the large sports community popular in China, which mainly contributes to the Public Figures aspect.
4https://github.com/google/BIG-bench/tree/ main/bigbench/benchmark_tasks/anachronisms
5https://github.com/google/BIG-bench/tree/ main/bigbench/benchmark_tasks/date_understanding
occurrence order, requiring logical reasoning based on commonsense. The global domain questions are the mix of translations6 and handcrafted; while all in the Chinese domain are handcrafted.
Movie and Music Recommendation (MMR) necessitates the LLM identifies the most similar matches to a variety of movies or music tracks, requiring the understanding of these popular movies and music and ability to identify their commonalities. All global domain questions are translations7, and all in the Chinese domain are handcrafted.
Sport Understanding (SpU) involves a crafted sentence with a famous athlete and a common sport action, and the LLM must assess its credibility, which demands understanding of sports and commonsense judgement. The questions in both domains are handcrafted, refering (Suzgun et al., 2022) .
Natural Language Inference (NLI) gives two sentences and asks the LLM to classify their relationship as entailment, contradiction, or neutral, necessitating commonsense-based reasoning and judgement. All global domain questions are selected from CLUE (Xu et al., 2020); the questions in the Chinese domain are partly from CLUE, and partly handcrafted.
Reading Comprehension (RC) gives a passage of text, and the LLM is required to reason based on it. All question in both domains are selected from LogiQA (Liu et al., 2020, 2023).
The chosen tasks adequately cover both the commonsense domains, particularly the 7 aspects of the Chinese commonsense domain. This coverage enables a comprehensive assessment of LLMs’ commonsense reasoning ability in Chinese. Moreover, the Chinese-domain questions could be created following the similar types and settings as their global counterpart, facilitating the cleaner comparison of the LLMs’ performance across the domains.
All questions in the CHARM reasoning tasks are multiple-choice questions. Detailed information is in Table 2. Question examples of the tasks are in Figure 6 in Appendix B. We used regular expressions to extract the preferred choice from the generation of the LLMs (Huang et al., 2023b; Li et al., 2023) and used accuracy as the metric.
6https://github.com/google/BIG-bench/tree/ main/bigbench/benchmark_tasks/logical_sequence
7https://github.com/google/BIG-bench/ tree/main/bigbench/benchmark_tasks/movie_ recommendation
3.3 Construction of Reasoning Tasks The construction of CHARM reasoning tasks involved the following three methods:
Translation [T] was applied to some global domain reasoning tasks. We translated the English commonsense reasoning benchmarks mentioned in §3.2 using GPT-3.5. Then we replaced the English names with commonly used Chinese names and manually screen the translated questions, retaining those without translation errors and accepted as commonsense globally.
Selection [S] We selected the excellent native Chinese datasets, LogiQA (Liu et al., 2020), and CLUE (Xu et al., 2020), and chose the questions that meet the requirements for CHARM.
Handcraft [H] was mainly applied to the most Chinese domain reasoning tasks. We used the entities in §3.1, and referred to the corresponding global domain task questions (from [T] or [S]) to construct questions with the same type and style. This ensured that the same reasoning task in two domains only differs in the commonsense domain, thus facilitating cleaner comparative analysis, as shown in Figure 6.
Detailed construction information of all reasoning tasks are shown in Table 2.
3.4 Memorization Tasks Shared commonsense knowledge pieces serve as links between reasoning and memorization questions. From the 7 reasoning tasks, we chose 4 that can be readily associated in this manner, AJ, TU, MMR, SpU, referred as the MemorizationReasoning-Interconnected (MRI) tasks, and built the related memorization questions.
Construction We first extracted the commonsense knowledge pieces related to the entities in
70.3%
9.0%
20.7%
Handcraft [H] Translation [T] Selection [S]
0.00 0.25 0.50 0.75 1.00
0.0
0.2
0.4
0.6
0.8
1.0
Figure 2: Distribution of CHARM construction.
the corresponding reasoning questions. Information about each entity was collected to the degree sufficient to address the associated reasoning question, and then used to formulate the memorization questions. Following the Knowledge Memorization task in KoLA (Yu et al., 2023), we chose freeform QA instead of multiple-choice or true/false questions, which can effectively avoid the impact of randomness. All memorization questions were handcrafted [H]. Question examples are in Figure 7 in Appendix C. The averaged number of related memorization questions for each reasoning question are shown in Table 3.
Judgement and Metric For the memorization task of MMR, we used a rule-based matching method for evaluation; for the other three tasks, we used GPT-3.5 for judgement. We used accuracy as the metric.
3.5 Quality Assurance The distribution of the contruction methods for CHARM is shown in Figure 2. After construction of CHARM, we conducted the quality assurance to ensure the quality of the questions. We hired professional NLP annotators to review the questions. The
quality assurance process involved five steps: (1) We prepared and assigned annotation task packages to annotators; (2) We trained annotators, emphasizing the avoidance of social bias; (3) We conducted a trial review on a random 20% of questions to fine adjust the review process; (4) Two annotators independently reviewed each question and provided answers without seeing our answers, and a question passed only if both annotators’ answers matched ours and they found no issues with the question; (5) For questions that failed in step 4, authors discussed whether to retain, discard, or correct them based on the nature of the issues identified.
Details about the quality assurance are in Appendix D.
4 Experimental Setup 4.1 Language Models We evaluated the currently commonly used LLMs, which can be divided into two categories: (1) 7 English LLMs, including GPT series (Achiam et al., 2023), LLaMA-2 (Touvron et al., 2023b), and Vicuna8. (2) 12 Chinese-oriented LLMs, including ChatGLM39, Baichuan2 (Yang et al., 2023), InternLM2 (Team, 2023), Yi10, DeepSeek(Bi et al., 2024) and Qwen (Bai et al., 2023). For open-source models, we chose the chat version instead of the base version. For closed-source models, we used the official API11. Detailed information is in Table 4.
We used opencompass12 in all our experiments. For all LLMs, the maximum out length was set to 512. For all open-source LLMs, we used the default settings in opencompass: the decoding temperature was the default value of the huggingface transformers library13, which is 1.0; do_sample was set to False; the PyTorch numerical type was bf16. For closed-source models (GPT-3.5 and GPT-4), we used the default settings in opencompass: the temperature was set to 0.7.
4.2 Prompt Strategies We selected 5 commonly used prompt strategies, and assessed the performance of the 19 LLMs on
8https://huggingface.co/lmsys/vicuna-7b-v1. 5-16k and https://huggingface.co/lmsys/ vicuna-13b-v1.5-16k
9https://huggingface.co/THUDM/chatglm3-6b-32k 10https://github.com/01-ai/Yi 11We used the gpt-3.5-turbo-1106 version for GPT-3.5 and
the gpt-4-1106-preview version for GPT-4. 12https://github.com/open-compass/opencompass 13https://github.com/huggingface/transformers
CHARM reasoning task: Direct: The LLM does not perform intermediate reasoning and directly predicts the answer. ZH-CoT: The LLM conducts intermediate reasoning (Wei et al., 2022) in Chinese before producing the answer.
EN-CoT: The reasoning process of CoT is in English for the Chinese questions(Shi et al., 2022).
Translate-EN: We used the DeepL api14 to translate our benchmark into English, and then used English CoT for reasoning (Zhang et al., 2023a).
XLT: The template prompt (Huang et al., 2023a) was used to change the original question into an English request, solve it step by step, and finally format the answer for output .
The examples for each prompt strategy are in Figure 8 in Appendix E. For all prompt strategies, we use the 3-shot setting.
5 Results and Analysis 5.1 Integrated Reasoning Performance We show the performance of the 19 LLMs on CHARM reasoning tasks in Table 5. We only choose one representative prompt strategy: XLT for English LLMs and ZH-CoT for Chinese-oriented LLMs, which is based on our empirical conclusion in §5.2. The LLMs’ performance on the 7 aspects of the Chinese commonsense domain are shown in Table 11 in Appendix G.
Commonsense Domain We found that the LLMs exhibit inconsistent performance in the global and Chinese commonsense domains. The rankings of the English LLMs dropped in the Chinese domain compared to the global domain. For instance, GPT-4 ranks first in the global domain, but in the Chinese domain, Qwen-72B outperforms all, pushing GPT-4 to the second. In the Chinese domain, the performance of LLaMA-2-70B is even worse than many Chinese-oriented LLMs in the 6B-7B size range.
14https://www.deepl.com/translator
However, in the global domain, LLaMA-2-70B is better than all Chinese-oriented LLMs up to 20B in size, except for Qwen-14B.
5.2 Prompt Strategy Selection We tested the combinations of the 19 LLMs and the 5 prompt strategies in CHARM reasoning tasks. Detailed results are in Table 9 and Table 10 in Appendix F. To draw some empirical conclusions, we analyzed along the following two dimensions: • Dim1: global or Chinese commonsense domain. • Dim2: English or Chinese-oriented LLMs.
We averaged the 19 × 5 LLM-prompt combinations along the above two dimensions, and the obtained results are in the Table 6. From the LLM dimension, it’s clear that various LLMs prefer different prompt strategies: XLT consistently excels for English LLMs among the 5 strategies, while
for Chinese-oriented LLMs, despite some complexity, ZH-CoT generally performs best. From the commonsense domain dimension, strategies that use English for reasoning (like XLT, Translate-EN, etc.) are suitable for the global domain; however, ZH-CoT generally performs better in the Chinese domain.
The conclusion here differs from previous studies (Shi et al., 2022; Huang et al., 2023a), which suggested that employing English for non-English reasoning tasks was more effective than using the native language. These previous studies had limitations, focusing only on English LLMs and neglecting the many Chinese-oriented LLMs developed since 2023. Furthermore, most benchmarks in these studies were merely translations from English, lacking unique cultural and linguistic characteristics in Chinese. The empirical findings with CHARM in this paper have somewhat alleviated those limitations, leading to more current and comprehensive conclusions, and of course still have the limitations, which are detailed in section Limitations.
5.3 Integrated Reasoning vs Memorization We evaluated the correlation between the integrated reasoning and the memorization on the MRI tasks, as mentioned in §3.2. The average performance of the LLMs on the 4 MRI tasks is in Figure 3. Detailed performance on each task is in Figure 9 in Appendix H.1.
As shown in Figure 3, the 19 LLMs can be roughly divided into the three types: • Type I: Low memorization and low integrated
reasoning ability. We found that apart from OpenAI’s GPT series, all other English LLMs belong to this type. • Type II: High memorization and medium integrated reasoning ability. GPT-3.5 and all Chinese-oriented LLMs below 30B belong to this type. It’s worth noting that some LLMs have high memorization performance, but relatively poor integrated reasoning ability. • Type III: Ultra-high memorization and high integrated reasoning ability. This category includes GPT-4 and the three Chinese-oriented LLMs that exceed a size of 30B.
The above findings offer clear guidance for the enhancement of LLMs’ reasoning abilities in Chinese commonsense domain. For Type I, the limitation lies in the memorization. For Type II, there should be further improvement in understanding, applying knowledge, and reasoning abilities.
In addition, we also evaluated the correlation between memorization and integrated reasoning during the LLM pre-training process, details can be found in Figure 10 in Appendix H.2.
The results clearly indicate that strong memorization is the foundation of integrated reasoning. Weak memorization leads to poor reasoning, as shown by Type I LLMs. Also, factors other than memorization can cause significant differences in reasoning abilities among LLMs with similar memorization.
5.4 Memorization-Independent Reasoning Methods We proposed two methods, FRMM and MIB, to compare the LLMs’ memorizationindependent reasoning on the MRI tasks. The results are in Table 7. • Filtering Reasoning questions based on Mono-
LLM-Memorization (FRMM) For each LLM, we selected reasoning questions based on its performance in memorization tasks: only retaining reasoning questions for which all related memorization questions were answered correctly. Then we calculated the accuracy of the retained reasoning questions for each LLM. The LLMs were then ranked based on the accuracy, producing the leaderboard shown in the penultimate column of Table 7. The detail of the FRMM is in Appendix I.1. • Memorization-Independent Battles among LLMs (MIB) Inspired by the pairwise battle method adopted in LLM evaluation (Zheng et al., 2023), we tallyed each LLM’s performance in a “roundrobin” tournament of pairwise match-ups and then ranked the performance of the LLMs. Specifically, we selected two LLMs at each time and filter the MRI task’s reasoning questions based on the performance of these two LLMs in memorization tasks. We only retained the reasoning questions whose related memorization questions were correctly answered by both LLMs. In this way, the two LLMs were battled under fair conditions. Then we calculated the accuracy of the two LLMs on the retained reasoning questions, and computed the difference in accuracy as the battle score between the two LLMs. For a total of 19 LLMs, we averaged each LLM’s scores from the 18 battles they participated in as their final scores. The LLMs were then ranked based on these final scores, producing the leaderboard shown in the last column of Table 7. The detail of the MIB is in Appendix I.2.
Analysis As shown in Table 7, when comparing the leaderboards for integrated and memorizationindependent reasoning, Type III LLMs rank at the forefront and the Type I rank at the end in all leaderboards. There is a slight variation in the ranking order within the three types of LLMs.
Error Types For the in-depth analysis, we chose Vicuna-13B, Qwen-7B, Qwen-72B as representatives for Type I, II, and III LLMs, and filtered out the reasoning questions in the MRI tasks, only keeping those with correct answers to the related memorization questions, same as the FRMM in Appendix I.1. This ensured the LLM had sufficiently memorized the commonsense knowledge required for the retained reasoning questions, thereby minimizing the impact of memorization on reasoning. There are totally 500 reasoning questions in the 4 MRI tasks, and the numbers of the retained are 106, 323 and 402 for Vicuna-13B, Qwen-7B and Qwen-72B respectively, as shown in Table 8.
If the LLMs provided incorrect answers for the retained reasoning questions, these errors can be referred to as memorization-independent reasoning errors. We conducted the manual review and analysis of their reasoning process, and classified the errors into 4 main categories. • Understanding Error In this case, the LLM was unable to accurately comprehend the question, including misunderstanding the content, ignoring or even modifying important information in the premise, and failing to grasp the core query of the question. • Knowledge Error The LLM incorporated inaccurate knowledge during the reasoning process. It’s important to highlight that the knowledge pieces related to the reasoning question were previously examined in the related memorization questions, which the LLM answered correctly. However, the LLM output incorrect information during the reasoning phase. • Logical Error The LLM made logical reasoning errors, such as mathematical reasoning errors, inability to reach the correct conclusion based on sufficient information, or reaching the correct conclusion but outputing the wrong option. • Other Errors These are other scattered, relatively rare types of errors.
We show examples of each type of errors in Figure 12 in Appendix J. The distribution of these error types are shown in Figure 4.
Discussions Obviously, the majority of errors are from logical reasoning mistakes and knowledge

heading: Models LLM type # Original # Retained # Incorrect

heading: Understanding Error Knowledge Error Logical Error
text: inaccuracies, which further provides the directions for LLMs’ enhancement. As for knowledge errors, prior studies (Bian et al., 2023; Allen-Zhu and Li, 2023) have indicated that the way LLMs remember and master knowledge is a relatively complex topic. Simple memorization doesn’t guarantee that LLMs can apply this knowledge accurately and skillfully during the reasoning process.
6 Conclusion
This paper introduces CHARM, the first benchmark designed to comprehensively and thoroughly evaluate LLMs’ commonsense reasoning in Chinese. CHARM encompasses two counterpart commonsense domains, global and Chinese-specific, with the carefully selected tasks. We evaluated the representative prompt strategies for improving LLMs’ reasoning ability, and the empirical findings significantly enhances and supplements the conclusions of previous studies. CHARM comprises closely-interconnected reasoning and memorization tasks, helping to reveal the intrinsic correlation between memorization and reasoning of LLMs. We evaluated the strengths and weaknesses of different LLMs and conducted the detailed analysis of memorization-independent reasoning abilities. We hope that CHARM’s approach to studying the correlations between memoriztion and reasoning can serve as a reference for similar research in other fields.
Limitations
This study conducted tests on combinations of the 19 LLMs and the 5 prompt strategies, resulting in empirical conclusions. However, many existing LLMs and prompt strategies have not yet been tested. Furthermore, the best prompt strategy for the commonsense reasoning task for the LLMs, particularly in Chinese or other non-English languages, is not static and should progress with LLM technology. This is influenced by three elements: (1) The new prompt strategies are continuously proposed, which are likely more effective. (2) The new LLMs may have different prompt strategy preference, or be less sensitive to prompt. (3) For other non-English languages with high resources, future LLMs would be continuously evolving and updating, and necessitate ongoing updates in evaluation.
The automation of the construction and evaluation of CHARM needs further improvement, including the following: (1) Most of the questions in CHARM Chinese domain are manually constructed by the author. This limits the number of benchmark questions and the range of knowledge pieces covered. (2) Regarding memorization-independent reasoning, we chose only 3 LLMs as representative and manually categorized the types of errors within CHARM. In future research, we could employ robust LLMs, like GPT-4, for automated error classification and statistical analysis.
Ethical Consideration
This work involved human annotation. We have provided appropriate compensation for all annotators. The total cost of annotation for the project is about 2.2k RMB. For all annotators, we explicitly informed them about the use of the data and required them to ensure that the questions included in CHARM do not involve any social bias, ethical issues or privacy concerns during the annotation process.
Acknowledgements
We thank Dr. Qipeng Guo of Shanghai AI Lab for his invaluable suggestions. We additionally thank our crowd annotators for their diligent work and the anonymous reviewers for their constructive comments.
References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama
Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
Zeyuan Allen-Zhu and Yuanzhi Li. 2023. Physics of language models: Part 3.2, knowledge manipulation. arXiv preprint arXiv:2309.14402.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954.
Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. 2023. Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. arXiv preprint arXiv:2303.16421.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating crosslingual sentence representations. arXiv preprint arXiv:1809.05053.
Ernest Davis. 2023. Benchmarks for automated commonsense reasoning: A survey. arXiv preprint arXiv:2302.04752.
Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58(9):92– 103.
Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289.
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023a. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv preprint arXiv:2305.07004.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng
Lv, Yikai Zhang, Jiayi Lei, et al. 2023b. Ceval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322.
Sunkyoung Kim, Dayeon Ki, Yireun Kim, and Jinsik Lee. 2023. Boosting cross-lingual transferability in multilingual models via in-context learning. arXiv preprint arXiv:2305.15233.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2022. Fewshot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052.
Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. 2023. Logiqa 2.0—an improved dataset for logical reasoning in natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. Xcopa: A multilingual dataset for causal commonsense reasoning. arXiv preprint arXiv:2005.00333.
Dan Shi, Chaobin You, Jiantao Huang, Taihao Li, and Deyi Xiong. 2023. Corecode: A common sense annotated dialogue dataset with benchmark tasks for chinese large language models. arXiv preprint arXiv:2312.12853.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261.
InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal
Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and finetuned chat models. arXiv preprint arXiv:2307.09288.
Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, Ai Ti Aw, and Nancy F Chen. 2023a. Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning. arXiv preprint arXiv:2309.04766.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023b. Planand-solve prompting: Improving zero-shot chain-ofthought reasoning by large language models. arXiv preprint arXiv:2305.04091.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.05986.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.
Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zĳun Yao, Xiaohan Zhang, Hanming Li, et al. 2023. Kola: Carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296.
Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023a. Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7915–7927.
Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. 2023b. Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685.
A Entity and Question Examples of the 7 Chinese Commonsense Aspects
Figure 5 shows the number of questions and partial entities of each Chinese commonsense aspect we propose, as well as corresponding question examples.
B Question Examples of the Reasoning Task in CHARM
Figure 6 shows the question examples of the 7 reasoning tasks in CHARM, including both Chinese and global domains.
C Question Examples of the Memorization Task in CHARM
Figure 7 shows the questions examples of the memorization tasks in CHARM.
D Details of Quality Assurance
D.1 Quality Assurance Procedures After construction of CHARM, we conducted the quality assurance to ensure the quality of the questions. We hired professional NLP task annotators to review the questions we have constructed. The entire quality assurance includes the following steps:
Step(1) Annotation Task Submission and Assignment: We packaged the constructed questions into annotation task packages. Usually, one annotation package corresponds to all the questions of one reasoning or memorization task in Table 2. Before submitting the annotation tasks, we had written the annotation requirement document, which includes key requirements and typical examples. After the annotation tasks were submitted to the online annotation platform, they were assigned to suitable annotators. Typically, a task would be assigned to either two or four annotators.
Step(2) Annotator training: After the annotation task was assigned, we organized an online meeting to train the annotators. We specifically asked annotators to avoid social bias and sensitive issues such as morality.
Step(3) Trial review: Before the official review begun, we randomly selected 20% of the questions for the annotators to try annotating. We reviewed the results of the trial annotation, corrected any detailed issues or understanding deviations in the review process in a timely manner, to ensure the quality of subsequent reviews.
Step(4) Official review: For each question, whether handcrafted or translated, we had two annotators do independent reviews. We only provided the question to the annotators, not our answers. Using external resources fully (such as search engines, online encyclopedias, etc.), the annotators would provide the answers. Annotators could also provide feedback on issues with the question itself (for example, translation errors or questions that do not meet the commonsense standards). Only when both annotators believed that there was no problem with the question, and their provided answers were consistent with our previous answers, did we consider the question to have passed the review.
Step(5) Consultation and correction: For questions that did not pass the step(4) review, several authors would hold a meeting to discuss the questions. There are three cases in total: • Case(1) Retain: There was no problem with the question itself, but the annotator answered incorrectly. We retained these questions. • Case(2) Discard: The question had significant errors due to translation issues or problems with the question itself. We discarded these questions. • Case(3) Correct: There were minor issues with the question or the answer, the multiple authors would discuss together and complete the correction of the questions.
D.2 Information of the Annotators We submitted the annotation task online to the professional data annotation company, which organized the annotators to complete the annotation work. A total of 30 professional annotators, all native Chinese speakers with extensive experience in natural language processing tasks, were involved in this project. They possess the expertise to discern and comprehend commonsense knowledge pertinent to both global and Chinese-specific contexts. Here is the specific information about these annotators.
Education and profession: Of the 30 annotators, 11 have a bachelor’s degree and 19 have an associate degree. Regarding their fields of study, 9 are in humanities (4 specialized in design, 3 in business, 2 in language), while 21 are in science and engineering (11 specialized in computer science, 4 in automation, 3 in medicine, 2 in architecture, 1 in mathematics).
Age: 14 annotators are aged 20-25, 8 are aged 26-30, 7 are aged 30-35, and 1 is aged 35-40.
The annotators are all located in Changzhi City, Shanxi Province, China. We offer the hourly wage
of 23.75 RMB for each annotator, which is higher than the local minimum wage standard.
D.3 Other Details During the construction of the CHARM project, we submitted a total of 22 annotation tasks, which together contained approximately 3.55k questions, and took a total of approximately 93 hours of the annotators’ working time. The entire process for each annotation task, from step(1) to step(4), typically required half a day. It needs to be emphasized that the annotators were only tasked with checking and answering questions that we have already created; they were NOT responsible for creating questions from scratch.
The annotation work for the entire project spanned from January 2 to January 26, 2024. Each annotation task required only a few annotators to complete, rather than all 30 annotators. During the project period, the annotators participating in this project were also undertaking other data annotation tasks in the companies. In fact, our annotation tasks only accounted for a very small part of their working time.
The statistics results of the quality evaluation process are as followings: • For reasoning tasks, the average failure rate in step(4) is 0.19, and the average ratio of case(3) in step(5) is 0.04. • For memorization tasks, the average failure rate in step(4) is 0.04, and the average ratio of case(3) in step(5) is 0.02.
E Examples of Prompt Strategies Figure 8 shows the examples of the 5 prompt strategies.
F Detailed Evaluation Results of 19 LLMs with 5 Prompt Strageties on Reasoning Tasks
We conducted a detailed evaluation of 19 different LLMs using 5 distinct prompt strategies. Table 9 and Table 10 respectively display the performance of various prompt strategies on 7 reasoning tasks in the CHARM’s Chinese commonsense domain and global commonsense domain.
G Performance of LLMs on Chinese Commonsense Knowledge Aspects
Table 11 displays the performance of LLMs in the 7 Chinese commonsense aspects. We only
choose one representative prompt strategy: XLT for English LLMs and ZH-CoT for Chinese LLMs, which is based on our empirical conclusion in §5.2.
H Correlation of Memorization and Integrated Reasoning
H.1 Detailed Correlations of Memorization and Integrated Reasoning on the 4 MRI Tasks
The detailed performances of the 19 LLMs on the 4 MRI tasks are in Figure 9.
H.2 Correlation of Memorization and Integrated Reasoning throughout the LLM pretraining
We tested the intermediate checkpoint models of Baichuan2 and DeepSeek on the memorization and reasoning questions on the 4 MRI tasks. The results are shown in Figure 10.
With the increase in the number of tokens during the training process, the model’s memorization ability quickly reach a high level (in fact, there is no particularly obvious difference between the results of the first checkpoint and the final results). This is because the knowledge involved in our task setting is the most basic commonsense, and thus widely and abundantly exists in various Chinese training corpora.
However, the improvement in reasoning performance significantly lags behind memorization. This is because to complete a reasoning task in CHARM is actually a multi-step process, requiring memorization of relevant knowledge, understanding of the question, use of knowledge for reasoning, and answering according to the requirements of the question and the demonstration of few-shot examples, etc. If an error occurs in any step of the above complex process, the reasoning task will fail.
I Leaderboard of Memorization-Independent Reasoning
It is non-trivial to acquire and compare the memorization-independent reasoning abilities of the LLMs. Intuitively, we can filter the reasoning questions by only retaining those whose related memorization questions are all correctedly answered by every LLMs. This approach ensures that each LLM has memorized the commonsense knowledge necessary for the retained reasoning questions. However, when we applied this process to all the 19 LLMs, only 28 reasoning questions
remained out of the original 500 in the MRI tasks, which was obviously insufficient in number and lacks diversity, thereby introducing a high degree of uncertainty due to randomness.
Therefore, we proposed two slightly more complex methods, one called Filtering Reasoning Questions based on Mono-LLM-Memorization (FRMM), the other is Memorization-Independent Battles among LLMs (MIB).
I.1 Filtering Reasoning Questions based on Mono-LLM-Memorization (FRMM)
This method is relatively simple, but has some flaws to a certain extent. For each LLM, we selected reasoning questions based on its performance in memorization tasks: only retaining reasoning questions for which all related memorization questions are answered correctly. It’s clear that, after individual filtration, different LLMs would retain different reasoning questions, and even differ in the number of retained reasoning questions, as shown in the “# retained” column in Table 12.
Then, we calculated the accuracy of the retained reasoning questions for each LLM, and the results are shown in the “Retained Acc” column in Table 12. The LLMs were then ranked based on the accuracy, producing the leaderboard shown in the penultimate column of Table 7.
As mentioned above, while this method can reflect the memorization-independent reasoning abilities of LLMs to some extent, its drawback lies in that the denominator used in calculating the final ranking accuracy differs for different LLMs.
To overcome this, we proposed the MIB method.
I.2 Memorization-Independent Battles among LLMs (MIB)
To overcome the shortcomings of the FRMM method, we referred to the pairwise battle method adopted in LLM evaluation (Zheng et al., 2023). By tallying each LLM’s performance in a “round-robin” tournament of pairwise match-ups, we ranked the performance of the LLMs.
Specifically, we selected two LLMs at each time and filtered the MRI task’s reasoning questions based on the performance of these two LLMs in memorization tasks. We only retained the reasoning questions whose related memorization questions were correctly answered by both LLMs. In this way, the two LLMs are battled under fair conditions. We then calculated the accuracy of these two LLMs on the retained reasoning questions separately, and
compute the difference in accuracy as the battle score between the two models.
As shown in Figure 11, the element Eij represents the accuracy of LLM i minus the accuracy of LLM j during the battle between the two LLMs. For a total of 19 LLMs, we averaged each LLM’s scores from the 18 battles they participated in as their final scores, as shown in Table 13.
Finally, we ranked the LLMs based on these scores to produce the leaderboard shown in the last column of Table 7.
J Memorization-Independent Reasoning Errors
LLMs can answer memorization questions correctly, but they make mistakes when it comes to reasoning problems composed of these knowledge points. Figure 12 shows the examples of three memorizationindependent reasoning errors of LLMs.

heading: Future Work
text: (3) For other non-English languages with high resources, future LLMs would be continuously evolving and updating, and necessitate ongoing updates in evaluation. The automation of the construction and evaluation of CHARM needs further improvement, including the following: (1) Most of the questions in CHARM Chinese domain are manually constructed by the author. This limits the number of benchmark questions and the range of knowledge pieces covered. (2) Regarding memorization-independent reasoning, we chose only 3 LLMs as representative and manually categorized the types of errors within CHARM. In future research, we could employ robust LLMs, like GPT-4, for automated error classification and statistical analysis. Ethical Consideration
This work involved human annotation. We have provided appropriate compensation for all annotators. The total cost of annotation for the project is about 2. 2k RMB. For all annotators, we explicitly informed them about the use of the data and required them to ensure that the questions included in CHARM do not involve any social bias, ethical issues or privacy concerns during the annotation process