Sophia's adaptive optimization approach significantly impacts long-term training stability by providing a more efficient and effective way to navigate the complex landscape of language model pre-training. By incorporating a diagonal Hessian-based pre-conditioner and per-coordinate clipping mechanism, Sophia ensures that updates are tailored to the curvature of different parameter dimensions. This adaptability helps prevent issues such as slow convergence in flat dimensions, bouncing in sharp dimensions, and potential divergence due to negative curvatures or rapid changes in the Hessian.
The use of an EMA for both gradients and diagonal Hessian estimates also contributes to stability by smoothing out noise and fluctuations in these values over time. Additionally, the clipping mechanism controls the worst-case update size, safeguarding against inaccuracies in Hessian estimation or sudden changes in curvature along the trajectory. This not only improves convergence speed but also prevents large updates that could lead to instability or overshooting.
Overall, Sophia's adaptive optimization approach enhances long-term training stability by ensuring that updates are well-calibrated based on local curvatures, mitigating potential issues that could hinder convergence or cause divergence during training.