Aquila2 Technical Report
Authors: name: Bo-Wen Zhang; name: Liangdong Wang; name: Jijie Li; name: Shuhao Gu; name: Xinya Wu; name: Zhengduo Zhang; name: Boyan Gao; name: Yulong Ao; name: Guang Liu
Published: 2024-08-14 09:34:19

Abstract: This paper introduces the Aquila2 series, which comprises a wide range of
bilingual models with parameter sizes of 7, 34, and 70 billion. These models
are trained based on an innovative framework named HeuriMentor (HM), which
offers real-time insights into model convergence and enhances the training
process and data management. The HM System, comprising the Adaptive Training
Engine (ATE), Training State Monitor (TSM), and Data Management Unit (DMU),
allows for precise monitoring of the model's training progress and enables
efficient optimization of data distribution, thereby enhancing training
effectiveness. Extensive evaluations show that the Aquila2 model series
performs comparably well on both English and Chinese benchmarks. Specifically,
Aquila2-34B demonstrates only a slight decrease in performance when quantized
to Int4. Furthermore, we have made our training code
(https://github.com/FlagOpen/FlagScale) and model weights
(https://github.com/FlagAI-Open/Aquila2) publicly available to support ongoing
research and the development of applications.