Qwen2.5 Technical Report
Authors: name: Qwen; name: An Yang; name: Baosong Yang; name: Beichen Zhang; name: Binyuan Hui; name: Bo Zheng; name: Bowen Yu; name: Chengyuan Li; name: Dayiheng Liu; name: Fei Huang; name: Haoran Wei; name: Huan Lin; name: Jian Yang; name: Jianhong Tu; name: Jianwei Zhang; name: Jianxin Yang; name: Jiaxi Yang; name: Jingren Zhou; name: Junyang Lin; name: Kai Dang; name: Keming Lu; name: Keqin Bao; name: Kexin Yang; name: Le Yu; name: Mei Li; name: Mingfeng Xue; name: Pei Zhang; name: Qin Zhu; name: Rui Men; name: Runji Lin; name: Tianhao Li; name: Tingyu Xia; name: Xingzhang Ren; name: Xuancheng Ren; name: Yang Fan; name: Yang Su; name: Yichang Zhang; name: Yu Wan; name: Yuqiong Liu; name: Zeyu Cui; name: Zhenru Zhang; name: Zihan Qiu
Published: 2024-12-19 17:56:09

Abstract: In this report, we introduce Qwen2.5, a comprehensive series of large
language models (LLMs) designed to meet diverse needs. Compared to previous
iterations, Qwen 2.5 has been significantly improved during both the
pre-training and post-training stages. In terms of pre-training, we have scaled
the high-quality pre-training datasets from the previous 7 trillion tokens to
18 trillion tokens. This provides a strong foundation for common sense, expert
knowledge, and reasoning capabilities. In terms of post-training, we implement
intricate supervised finetuning with over 1 million samples, as well as
multistage reinforcement learning. Post-training techniques enhance human
preference, and notably improve long text generation, structural data analysis,
and instruction following. To handle diverse and varied use cases effectively,
we present Qwen2.5 LLM series in rich sizes. Open-weight offerings include base
and instruction-tuned models, with quantized versions available. In addition,
for hosted solutions, the proprietary models currently include two
mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both
available from Alibaba Cloud Model Studio. Qwen2.5 has demonstrated top-tier
performance on a wide range of benchmarks evaluating language understanding,
reasoning, mathematics, coding, human preference alignment, etc. Specifically,
the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and
proprietary models and demonstrates competitive performance to the
state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5
times larger. Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness
while performing competitively against GPT-4o-mini and GPT-4o respectively.
Additionally, as the foundation, Qwen2.5 models have been instrumental in
training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and
multimodal models.