Title: Sharpness-Aware Minimization Can Hallucinate Minimizers
Primary area: optimization
Keywords: Optimization for deep learning, sharpness-aware minimization

Abstract: Sharpness-Aware Minimization (SAM) is a widely used method that steers training toward flatter minimizers, which typically generalize better. In this work, however, we show that SAM can converge to hallucinated minimizers---points that are not minimizers of the original objective. We theoretically prove the existence of such hallucinated minimizers and establish conditions for local convergence to them. We further provide empirical evidence demonstrating that SAM can indeed converge to these points in practice. Finally, we propose a simple yet effective remedy for avoiding hallucinated minimizers.

TL;DR: We show that SAM can fail by converging to hallucinated minimizers, and provide theory, experiments, and a practical fix.