Title: SafetyBench: Evaluating the Safety of Large Language Models
Benchmark: SafetyBench
Summary: The paper introduces SafetyBench, a comprehensive benchmark for evaluating the safety of LLMs through MCQs spanning a range of safety concerns. It incorporates both Chinese and English data and tests both Chinese and English LLMs. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: safety
Phenomenon defined by authors: No
Phenomenon definition: It's defined as derived from "8 typical safety scenarios" proposed in another paper (page 3). Safety itself is not defined. 
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: answer MCQ questions that concern various safety scenarios
Task item definition: question, options, safety issue category
Task source: Human exam questions (e.g. GRE questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 11,435
Dataset metadata available: Yes
Dataset metadata detail: safety category, language
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: safety issue category
Metric access: Model access required (e.g. logits)
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: They run a small study where they compare their MCQs against open-ended responses from LLMs on the same scenarios, discussing differences in benchmark format and understanding vs generation of unsafe content.
Taxonomy: Alignment / Safety
New taxonomy: General Capability / Safety