COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for
  Alignment with Human Values
Authors: name: M-A-P Team, name: Siwei Wu, name: Jincheng Ren, name: Xinrun Du, name: Shuyue Guo, name: Xingwei Qu, name: Yiming Liang, name: Jie Liu, name: Yunwen Li, name: Tianyu Zheng, name: Boyu Feng, name: Huaqing Yuan, name: Zenith Wang, name: Jiaheng Liu, name: Wenhao Huang, name: Chenglin Cai, name: Haoran Que, name: Jian Yang, name: Yuelin Bai, name: Zekun Moore Wang, name: Zhouliang Yu, name: Qunshu Lin, name: Ding Pan, name: Yuchen Jiang, name: Tiannan Wang, name: Wangchunshu Zhou, name: Shenzhi Wang, name: Xingyuan Bu, name: Minghao Liu, name: Guoyin Wang, name: Ge Zhang, name: Chenghua Lin

Aligning large language models (LLMs) with human preferences has achieved
remarkable success. However, existing Chinese preference datasets are limited
by small scale, narrow domain coverage, and lack of rigorous data validation.
Additionally, the reliance on human annotators for instruction and response
labeling significantly constrains the scalability of human preference datasets.
To address these challenges, we design an LLM-based Chinese preference dataset
annotation pipeline with no human intervention. Specifically, we crawled and
carefully filtered 92k high-quality Chinese queries and employed 15 mainstream
LLMs to generate and score chosen-rejected response pairs. Based on it, we
introduce COIG-P (Chinese Open Instruction Generalist - Preference), a
high-quality, large-scale Chinese preference dataset, comprises 1,009k Chinese
preference pairs spanning 6 diverse domains: Chat, Code, Math, Logic, Novel,
and Role. Building upon COIG-P, to reduce the overhead of using LLMs for
scoring, we trained a 8B-sized Chinese Reward Model (CRM) and meticulously
constructed a Chinese Reward Benchmark (CRBench). Evaluation results based on
AlignBench liu2024alignbenchbenchmarkingchinesealignment show that that
COIG-P significantly outperforms other Chinese preference datasets, and it
brings significant performance improvements ranging from 2% to 12% for the
Qwen2/2.5 and Infinity-Instruct-3M-0625 model series, respectively. The results
on CRBench demonstrate that our CRM has a strong and robust scoring ability. We
apply it to filter chosen-rejected response pairs in a test split of COIG-P,
and our experiments show that it is comparable to GPT-4o in identifying
low-quality samples while maintaining efficiency and cost-effectiveness. Our
codes and data are released in
https://github.com/multimodal-art-projection/COIG-P.