Title: AlignBench: Benchmarking Chinese Alignment of LLMs
Benchmark: AlignBench
Summary: AlignBench is a 683‑query, eight‑category benchmark that tests how well Chinese‑supported LLMs satisfy user intent (“alignment”) in realistic, open‑ended settings. The authors supply a human‑in‑the‑loop curation pipeline, reference answers with evidence links, and a rule‑calibrated, multi‑dimensional GPT‑4‑as‑judge evaluation scheme, then benchmark 17 popular LLMs.
Contribution: - Introduces the first Chinese, multi‑dimensional alignment benchmark grounded in real user queries.
- Proposes “rule‑calibrated” point‑wise scoring that narrows GPT‑4/human agreement gaps vs. prior MT‑Bench prompts.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Alignment to human intent & preferences in Chinese
Phenomenon defined by authors: No
Phenomenon definition: The ability of LLMs to follow human instructions and reflect human intentions and preferences, typically achieved through supervised fine-tuning and RLHF.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a Chinese user query, generate a helpful, correct, preferred response in free text.
Task item definition: A single row comprises of {question, category, subcategory, reference_answer, evidences[]}
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Real user questions were filtered, de‑identified, and de‑sensitised; ~50 % easiest items were dropped after pilot LLM scoring to keep difficulty high.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 683
Dataset metadata available: Yes
Dataset metadata detail: category (8), subcategory, evidence URLs/quotes, difficulty filter flag
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: Mean, sample‑level Pearson r, system‑level Pearson r, pairwise win‑rate % (for agreement studies).
Metric subscores: Yes
Metric subscores detail: eight category scores & multi‑dimensional (correctness, logic, creativity, etc.).
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: 400‑item human study shows r ≈ 0.63 sample‑level, 0.998 system‑level and 75 % pairwise agreement, demonstrating high construct validity of the rule‑calibrated GPT‑4 judge.
Taxonomy: Alignment / Alignment
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Alignment