To identify the conditions under which Sharpness-Aware Minimization (SAM) operates at its edge of stability, similar to the results obtained for Gradient Descent (GD), a theoretical analysis is required. This analysis would involve studying the dynamics of SAM in relation to the operator norm of the Hessian and the gradient updates.
One approach could be to analyze the convergence properties of SAM under certain assumptions and constraints. This analysis would need to consider the impact of the learning rate (η), the neighborhood offset (ρ), and the norm of the gradient on the behavior of SAM. By studying how these parameters interact with the loss landscape and the Hessian matrix, it may be possible to derive conditions under which SAM operates at its edge of stability.
Additionally, empirical experiments can be conducted to validate the theoretical findings. By training neural networks with SAM using different hyperparameters and monitoring the behavior of the Hessian's operator norm, researchers can observe whether SAM consistently converges towards the edge of stability under specific conditions.
Overall, a combination of theoretical analysis and empirical validation is essential to identify the precise conditions under which SAM operates at its edge of stability, similar to the established results for GD.