Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation
  Learning
Authors: name: Siyuan Li; name: Juanxi Tian; name: Zedong Wang; name: Luyuan Zhang; name: Zicheng Liu; name: Weiyang Jin; name: Yang Liu; name: Baigui Sun; name: Stan Z. Li
Published: 2024-10-08 21:14:23

Abstract: This paper delves into the interplay between vision backbones and optimizers,
unvealing an inter-dependent phenomenon termed
\textbf{backbone-optimizer coupling bias}
(BOCB). We observe that canonical CNNs, such as VGG and ResNet, exhibit a
marked co-dependency with SGD families, while recent architectures like ViTs
and ConvNeXt share a tight coupling with the adaptive learning rate ones. We
further show that BOCB can be introduced by both optimizers and certain
backbone designs and may significantly impact the pre-training and downstream
fine-tuning of vision models. Through in-depth empirical analysis, we summarize
takeaways on recommended optimizers and insights into robust vision backbone
architectures. We hope this work can inspire the community to question
long-held assumptions on backbones and optimizers, stimulate further
explorations, and thereby contribute to more robust vision systems. The source
code and models are publicly available at https://bocb-ai.github.io/.