Title: ON THE ROLE OF IMPLICIT REGULARIZATION OF STOCHASTIC GRADIENT DESCENT IN GROUP ROBUSTNESS
Primary area: learning theory
Keywords: Spurious Correlations, Stochastic Gradient Descent (SGD), Implicit Regularization

Abstract: Training with stochastic gradient descent (SGD) at moderately large learning rates has been observed to improve robustness against spurious correlations, strong correlation between non-predictive features and target labels. Yet, the mechanism underlying this effect remains unclear. In this work, we identify batch size as an additional critical factor and show that robustness gains arise from the implicit regularization of SGD, which intensifies with larger learning rates and smaller batch sizes. This implicit regularization reduces reliance on spurious or shortcut features, thereby enhancing robustness while preserving accuracy. Importantly, this effect appears unique to SGD: gradient descent (GD) does not confer the same benefit and may even exacerbate shortcut reliance. Theoretically, we establish this phenomenon in linear models by leveraging statistical formulations of spurious correlations, proving that SGD systematically suppresses spurious feature dependence. Empirically, we demonstrate that the effect extends to deep neural networks across multiple benchmarks. Moreover, standard SGD with small batch sizes, while not requiring prior knowledge of spurious correlations or underrepresented groups, achieves group-robust generalization that surpasses specialized debiasing methods in the more challenging setting with multiple spurious correlations, while remaining simpler and more computationally efficient. For the experiments that support these findings, please refer to \href{https://github.com/ICLR2026-submission/implicit-regularization-in-group-robustness}{https://github.com/ICLR2026-submission/implicit-regularization-in-group-robustness}.