The improved alignment between the gradients used by Sharpness-Aware Minimization (SAM) and the principal eigenvector of the Hessian can be attributed to the specific update mechanism of SAM and its impact on the optimization process.
SAM updates its parameters by computing the gradient at a neighbor of the current solution, which is obtained by taking a step in the gradient direction with an offset ρ. This update strategy allows SAM to navigate the loss landscape in a way that influences the alignment of the gradients with the principal eigenvector of the Hessian.
Under certain conditions, such as when the norm of the gradient is appropriately scaled relative to the learning rate and the neighborhood offset, SAM tends to align the gradients more closely with the principal direction of the Hessian. This alignment can lead to more efficient optimization, as it helps the algorithm navigate towards regions of the parameter space that correspond to smoother and more stable minima.
The improved alignment between the gradients and the principal eigenvector of the Hessian occurs when SAM is operating in a regime where the update mechanism effectively guides the optimization process towards regions of the parameter space that lead to better convergence and generalization performance.