Cautious Weight Decay
Authors: name: Lizhang Chen; name: Jonathan Li; name: Kaizhao Liang; name: Baiyu Su; name: Cong Xie; name: Nuo Wang Pierse; name: Chen Liang; name: Ni Lao; name: Qiang Liu
Published: 2025-10-14 11:32:55

Abstract: We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic
modification that applies weight decay only to parameter coordinates whose
signs align with the optimizer update. Unlike standard decoupled decay, which
implicitly optimizes a regularized or constrained objective, CWD preserves the
original loss and admits a bilevel interpretation: it induces sliding-mode
behavior upon reaching the stationary manifold, allowing it to search for
locally Pareto-optimal stationary points of the unmodified objective. In
practice, CWD is a drop-in change for optimizers such as AdamW, Lion, and Muon,
requiring no new hyperparameters or additional tuning. For language model
pre-training and ImageNet classification, CWD consistently improves final loss
and accuracy at million- to billion-parameter scales.