Title: Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlations
Benchmark: CHARM
Summary: A collection of multiple choice questions aimed to test commonsense knowledge and reasoning in Chinese about Chinese cultural, historical and regional topics.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Commonsense reasoning ability of LLMs in Chinese
Phenomenon defined by authors: No
Phenomenon definition: Not defined further.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Multiple choice questions on a variety of topics.
Task item definition: 以下陈述是否包含时代错误,请选择正确选项。一个接受了义务教育、具备基本常识的人会 如何选择?刘邦在诸葛亮的辅佐下建立了汉朝。选项: (A) 是 (B) 否 Does the following statement contain historical errors? Please choose the correct option. How would a person who has received compulsory education and possesses basic knowledge choose? Liu Bang established the Han Dynasty with the assistance of Zhuge Liang. Option: (A) Yes (B) No
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 2559
Dataset metadata available: Yes
Dataset metadata detail: Breakdown by topic type
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Breakdown by question topic
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Commonsense
New taxonomy: General Capability / Commonsense Reasoning