Title: COLD: A Benchmark for Chinese Offensive Language Detection
Benchmark: COLD
Summary: COLDATASET is dataset of Chinese sentences with binary offensive/non-offensive labels covering topics of race, gender, and region. COLDETECTOR is a baseline detector trained on this dataset.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Offensive language detection in Chinese
Phenomenon defined by authors: Yes
Phenomenon definition: "The contents with any form of targeted offense to individuals or groups are consid- ered offensive language. It includes veiled or di- rect offensive content expressing rudeness, disrespect, insults, threats and profanity based on as- pects such as race, religion, sex, or sexual orientation" 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task involves classifying Chinese text samples as either "offensive" or "non-offensive," where offensive/toxic language, and hate speech are not distinguished.
Task item definition: Chinese Social Media text sample with a binary label (offensive/non-offensive), and in the test set, a more fine-grained label classifying it into one of four subcategories.
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: The classification of content as offensive/non-offensive directly mirrors a real-world content moderation task performed on social media platforms. However, in production systems, this task would likely include additional context (like user history, reports, engagement metrics).
Dataset size: test: 5,323
Dataset size notes: train/dev: 32,157
Dataset metadata available: Yes
Dataset metadata detail: topic, fine-grained labels, average character length, topic-related keywords presence
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Reporting accuracy, precision, recall, and F1 scores, both as macro averages across all categories and separately for offensive and non-offensive classes.
Metric subscores: Yes
Metric subscores detail: Four subcategories (attacking individuals, attacking groups, anti-bias, other non-offensive) and also by topic categories.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: First, report high inter-annotator agreement. Second, compare against the translated Jigsaw dataset. Third, conduct an ablation study.
Taxonomy: Alignment / Safety
New taxonomy: General Capability / Safety