CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for
  pre-training large language models
Authors: name: Liangdong Wang; name: Bo-Wen Zhang; name: Chengwei Wu; name: Hanyu Zhao; name: Xiaofeng Shi; name: Shuhao Gu; name: Jijie Li; name: Quanyue Ma; name: TengFei Pan; name: Guang Liu
Published: 2024-10-24 07:50:07

Abstract: We present CCI3.0-HQ (https://huggingface.co/datasets/BAAI/CCI3-HQ), a
high-quality 500GB subset of the Chinese Corpora Internet 3.0
(CCI3.0)(https://huggingface.co/datasets/BAAI/CCI3-Data), developed using a
novel two-stage hybrid filtering pipeline that significantly enhances data
quality. To evaluate its effectiveness, we trained a 0.5B parameter model from
scratch on 100B tokens across various datasets, achieving superior performance
on 10 benchmarks in a zero-shot setting compared to CCI3.0, SkyPile, and
WanjuanV1. The high-quality filtering process effectively distills the
capabilities of the Qwen2-72B-instruct model into a compact 0.5B model,
attaining optimal F1 scores for Chinese web data classification. We believe
this open-access dataset will facilitate broader access to high-quality
language models.