Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language
  Models via Mixture-of-LoRAs
Authors: name: Abdelrahman Abouelenin; name: Atabak Ashfaq; name: Adam Atkinson; name: Hany Awadalla; name: Nguyen Bach; name: Jianmin Bao; name: Alon Benhaim; name: Martin Cai; name: Vishrav Chaudhary; name: Congcong Chen; name: Dong Chen; name: Dongdong Chen; name: Junkun Chen; name: Weizhu Chen; name: Yen-Chun Chen; name: Yi-ling Chen; name: Qi Dai; name: Xiyang Dai; name: Ruchao Fan; name: Mei Gao; name: Min Gao; name: Amit Garg; name: Abhishek Goswami; name: Junheng Hao; name: Amr Hendy; name: Yuxuan Hu; name: Xin Jin; name: Mahmoud Khademi; name: Dongwoo Kim; name: Young Jin Kim; name: Gina Lee; name: Jinyu Li; name: Yunsheng Li; name: Chen Liang; name: Xihui Lin; name: Zeqi Lin; name: Mengchen Liu; name: Yang Liu; name: Gilsinia Lopez; name: Chong Luo; name: Piyush Madan; name: Vadim Mazalov; name: Ali Mousavi; name: Anh Nguyen; name: Jing Pan; name: Daniel Perez-Becker; name: Jacob Platin; name: Thomas Portet; name: Kai Qiu; name: Bo Ren; name: Liliang Ren; name: Sambuddha Roy; name: Ning Shang; name: Yelong Shen; name: Saksham Singhal; name: Subhojit Som; name: Xia Song; name: Tetyana Sych; name: Praneetha Vaddamanu; name: Shuohang Wang; name: Yiming Wang; name: Zhenghao Wang; name: Haibin Wu; name: Haoran Xu; name: Weijian Xu; name: Yifan Yang; name: Ziyi Yang; name: Donghan Yu; name: Ishmam Zabir; name: Jianwen Zhang; name: Li Lyna Zhang; name: Yunan Zhang; name: Xiren Zhou
Published: 2025-03-03 17:05:52

Abstract: We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable
language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language
model trained on high-quality web and synthetic data, significantly
outperforming recent open-source models of similar size and matching the
performance of models twice its size on math and coding tasks requiring complex
reasoning. This achievement is driven by a carefully curated synthetic data
recipe emphasizing high-quality math and coding datasets. Compared to its
predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of
200K tokens to better support multilingual applications, as well as group query
attention for more efficient long-sequence generation. Phi-4-Multimodal is a
multimodal model that integrates text, vision, and speech/audio input
modalities into a single model. Its novel modality extension approach leverages
LoRA adapters and modality-specific routers to allow multiple inference modes
combining various modalities without interference. For example, it now ranks
first in the OpenASR leaderboard to date, although the LoRA component of the
speech/audio modality has just 460 million parameters. Phi-4-Multimodal
supports scenarios involving (vision + language), (vision + speech), and
(speech/audio) inputs, outperforming larger vision-language and speech-language
models on a wide range of tasks. Additionally, we experiment to further train
Phi-4-Mini to enhance its reasoning capabilities. Despite its compact
3.8-billion-parameter size, this experimental version achieves reasoning
performance on par with or surpassing significantly larger models, including
DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.