Title: WenMind: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Classical Literature and Language Arts

Benchmark: WenMind
Summary: WenMind is a benchmark for Chinese Classical Literature and Language Arts (CCLLA). It spans 42 tasks across three sub-domains (Ancient Prose, Ancient Poetry, Ancient Literary Culture), in both domain- and capability-oriented formats.

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Understanding, generation, and knowledge of Chinese classical language and literature

Phenomenon defined by authors: Yes
Phenomenon definition: Understanding, generating, and applying knowledge of ancient Chinese texts across prose, poetry, and literary culture 

Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: 42 distinct tasks derived from classical Chinese language skills, such as translation, comprehension, poetry writing, idiom interpretation, etc.

Task item definition: Each item is a QA pair with metadata, eg. "Translate the following ancient Chinese sentence into modern Chinese."

Task definition detail: Tasks categorized by sub-domain and cognitive ability (understanding, generation, knowledge).

Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Internet exam databases, open-source Chinese text corpora (C2MChn, WYWEB), and LLM generations (ERNIE-3.5), all standardized and filtered into Q&A formats

Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Maybe representative of how LLMs may be used in educational and cultural settings, but mostly a "knowledge test".

Dataset size: 4875
Dataset metadata available: Yes
Dataset metadata detail: Domain, capability, question format, task name (coarse/fine-grained), question, and answer

Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: Stratified human agreement evaluation on LLM-graded items; comparisons to BLEU/F1 for scoring validity.

Metric subscores: Yes
Metric subscores detail: Scores per task, domain (Prose/Poetry/Culture), and capability (Understanding/Generation/Knowledge)

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors report 89.4% agreement between ERNIE-3.5 scoring and human evaluation across representative models and tasks 

Taxonomy: Knowledge / Cultural
New taxonomy: General Application / Cultural Knowledge