Title:  CMB: A Comprehensive Medical Benchmark in Chinese 

Benchmark:  CMB 
Summary: CMB (Chinese Medical Benchmark) is a large-scale Chinese-native benchmark designed to evaluate LLMs on both theoretical knowledge (via multiple-choice exams) and practical reasoning (via multi-turn dialogues on clinical cases). 

Contribution: CMB covers the entire professional spectrum in Chinese medicine (physicians, nurses, pharmacists, technicians, etc.) and includes realistic multi-turn diagnostic cases.

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: medical knowledge, diagnostic reasoning 

Phenomenon defined by authors: Yes
Phenomenon definition: Factual medical knowledge (CMB-Exam) and reasoning/application of knowledge in medical consultations (CMB-Clin). 

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition:  2 task types: answering multiple-choice questions (CMB-Exam) and engaging in multi-turn Q&A based on clinical cases (CMB-Clin). 

Task item definition: CMB-Exam: A multiple-choice question with 4–6 answers, one or more correct. CMB-Clin: A question based on a clinical case in a simulated dialogue with reference answers.

Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues)
Task source detail: Exams from public Chinese medical licensing materials and textbook clinical cases.

Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: CMB-Exam aligns with actual licensing processes (but is just a medical exam); CMB-Clin simulates real-world diagnostic consultations.

Dataset size: 11200
Dataset metadata available: Yes
Dataset metadata detail: Professional category, subcategory, subject, case context, question/answer pairs, reference solutions

Sampling method: Random sample (creators defined a task space and sampled from it), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric definition detail: CMB-Exam: Accuracy. CMB-Clin: Scores for fluency, relevance, completeness, and medical proficiency

Metric aggregation: Simple Mean
Metric statistics: Means, standard deviations, and Spearman/Pearson correlations with expert rankings.

Metric subscores: Yes
Metric subscores detail: Per discipline, professional category, and evaluation dimension (CMB-Clin).

Metric metascoring: consensus@k (plurality vote over k trials)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Comparison between GPT-4 auto-eval and expert annotations (high correlation)

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine