Defeating the Training-Inference Mismatch via FP16
Authors: name: Penghui Qi, name: Zichen Liu, name: Xiangxin Zhou, name: Tianyu Pang, name: Chao Du, name: Wee Sun Lee, name: Min Lin
Organization: Sea AI Lab

Reinforcement learning (RL) fine-tuning of large language models (LLMs) often
suffers from instability due to the numerical mismatch between the training and
inference policies. While prior work has attempted to mitigate this issue
through algorithmic corrections or engineering alignments, we show that its
root cause lies in the floating point precision itself. The widely adopted
BF16, despite its large dynamic range, introduces large rounding errors that
breaks the consistency between training and inference. In this work, we
demonstrate that simply reverting to FP16 effectively eliminates this
mismatch. The change is simple, fully supported by modern frameworks with only
a few lines of code change, and requires no modification to the model
architecture or learning algorithm. Our results suggest that using FP16
uniformly yields more stable optimization, faster convergence, and stronger
performance across diverse tasks, algorithms and frameworks. We hope these
findings motivate a broader reconsideration of precision trade-offs in RL
fine-tuning.