When training with stochastic gradients of intermediate batch sizes, the edge-of-stability phenomenon and its interaction with Sharpness-Aware Minimization (SAM) can exhibit different characteristics compared to training with full batch or single examples.
Impact on Edge of Stability: The edge-of-stability phenomenon may manifest differently with intermediate batch sizes. The convergence behavior of SAM towards the edge of stability, as determined by the operator norm of the Hessian, may vary based on the batch size used during training. Intermediate batch sizes introduce noise in the gradient estimates, which can affect the dynamics of SAM and its proximity to the edge of stability.
Stability and Generalization: Training with stochastic gradients of intermediate batch sizes can influence the stability and generalization capabilities of SAM. The noise introduced by the stochastic gradients may impact the algorithm's ability to reach the edge of stability consistently and maintain stable convergence towards optimal solutions.
Optimization Dynamics: The interaction between the edge-of-stability phenomenon and SAM may be more nuanced with intermediate batch sizes. The algorithm's behavior in terms of oscillations, convergence speed, and final solution quality could be influenced by the stochasticity introduced by the batch sampling process.
Overall, training with stochastic gradients of intermediate batch sizes introduces a level of randomness and variability that can affect how SAM approaches the edge of stability and its optimization performance. Further research and experimentation are needed to fully understand the implications of using intermediate batch sizes on the edge-of-stability phenomenon and its interplay with SAM.