Title: Lions and Muons: Optimization via Stochastic Frank-Wolfe
Primary area: optimization
Keywords: stochastic Frank-Wolfe, stochastic optimization, Lion optimizer, Muon optimizer, heavy-tailed noise, gradient clipping, variance reduction

Abstract: Stochastic Frank-Wolfe is a classical optimization method for solving constrained optimization problems. On the other hand, recent optimizers such as Lion and Muon have gained quite significant popularity in deep learning. In this work, we provide a unifying perspective by interpreting these seemingly disparate methods through the lens of Stochastic Frank-Wolfe. Specifically, we show that Lion and Muon with weight decay can be viewed as special instances of a Stochastic Frank-Wolfe, and we establish their convergence guarantees in terms of the Frank-Wolfe gap, a standard stationarity measure in non-convex optimization for Frank-Wolfe methods. We further find that convergence to this gap implies convergence to a KKT point of the original problem under a norm constraint for Lion and Muon. Moreover, motivated by recent empirical findings that stochastic gradients in modern machine learning tasks often exhibit heavy-tailed distributions, we extend Stochastic Frank-Wolfe to settings with heavy-tailed noise by developing two robust variants with strong theoretical guarantees, which in turn yields new variants of Lion and Muon.

TL;DR: We present a Stochastic Frank-Wolfe framework, encompassing Lion and Muon, and develop variants leveraging gradient clipping and variance reduction techniques to handle heavy-tailed noise in stochastic gradients.