Phi-4 Technical Report
Authors: name: Marah Abdin; name: Jyoti Aneja; name: Harkirat Behl; name: Sébastien Bubeck; name: Ronen Eldan; name: Suriya Gunasekar; name: Michael Harrison; name: Russell J. Hewett; name: Mojan Javaheripi; name: Piero Kauffmann; name: James R. Lee; name: Yin Tat Lee; name: Yuanzhi Li; name: Weishung Liu; name: Caio C. T. Mendes; name: Anh Nguyen; name: Eric Price; name: Gustavo de Rosa; name: Olli Saarikivi; name: Adil Salim; name: Shital Shah; name: Xin Wang; name: Rachel Ward; name: Yue Wu; name: Dingli Yu; name: Cyril Zhang; name: Yi Zhang
Published: 2024-12-12 03:37:41

Abstract: We present phi-4, a 14-billion parameter language model developed with a
training recipe that is centrally focused on data quality. Unlike most language
models, where pre-training is based primarily on organic data sources such as
web content or code, phi-4 strategically incorporates synthetic data throughout
the training process. While previous models in the Phi family largely distill
the capabilities of a teacher model (specifically GPT-4), phi-4 substantially
surpasses its teacher model on STEM-focused QA capabilities, giving evidence
that our data-generation and post-training techniques go beyond distillation.
Despite minimal changes to the phi-3 architecture, phi-4 achieves strong
performance relative to its size -- especially on reasoning-focused benchmarks
-- due to improved data, training curriculum, and innovations in the
post-training scheme.