Unveiling the Backbone-Optimizer Coupling Bias in Visual Representation Learning

Authors: Siyuan Li, Juanxi Tian, Zedong Wang, Luyuan Zhang, Zicheng Liu, Weiyang Jin, Yang Liu, Baigui Sun, Stan Z. Li
Primary area: optimization
Keywords: Network Design, Optimization, Bias, Backbone, Transformer, Benchmark

Abstract:
This paper delves into the interplay between vision backbones and optimizers, revealing an inter-dependent phenomenon termed backbone-optimizer coupling bias} (BOCB). Notably, canonical CNNs, such as VGG and ResNet, exhibit a marked co-dependency with SGD, while recent architectures, including ViTs and ConvNeXt, share a strong coupling with adaptive learning rate optimizers. We further show that strong BOCB may result in extra tuning efforts and poor generalization ability for pre-trained neural networks, substantially limiting their real-world applications. Through in-depth analysis and apples-to-apples comparisons, however, we surprisingly observed that certain types of network architecture could significantly mitigate BOCB, which might serve as practical takeaways for backbone design. We hope this work can inspire the community to rethink the long-held assumptions on backbones and optimizers, consider their interplay in future studies, and contribute to more robust vision systems. The source code and models are publicly available.

Decision: reject
Reviewer scores: 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729717541615

thread_type:
official_review

<thread_object>
creator: reviewer_fcxi

<content>
summary:
This paper introduces the concept of Backbone-Optimizer Coupling Bias (BOCB) -- i.e., strong dependencies between neural network backbones and specific optimizers -- which affects performance and generalization. To demonstrate this, a comprehensive benchmark is provided, evaluating various vision backbones and popular optimizers across different tasks.

strengths:
* I believe the problem statement of the paper is promising. Even though it is known that some modern architectures need specific optimizers to achieve high performance (e.g., ViTs use AdamW by default these days), there are only a few comprehensive studies on the relationship between architectures and optimizers.
* This paper provides results using a wide range of optimizers and architectures, and it reports the average performance over three runs.
* This paper not only reports the phenomenon but also tries to reveal the underlying causes.

weaknesses:
* One of the major weaknesses is soundness. I am not fully convinced by some statements in the introduction, as they seem somewhat overstated. As a result, the contributions are somewhat limited.
  * The results suggest that the backbone is not necessarily dependent on a specific optimizer, but rather that certain types of optimizers tend to be generally effective. For example, Table 1, which is one of the main results, seems to simply show that Adam-like optimizers are generally effective. AdaGrad-like optimizers consistently achieve poor results.
  * Likewise, I don’t believe we could derive strong and consistent results from Figures 3 and 4. In these figures, I don’t find significant differences in the variances, except in a few cases.
  * In Figure 5, even though AdamW and LAMB show favorable robustness, this may simply indicate that they are inherently robust optimizers, and we cannot strongly conclude that Adam-like optimizers are robust since other optimizers do not achieve significantly good robustness. Also, I am not convinced that Adagrad-like optimizers have heavy BOCB (L371), as RMSProp achieves good BOCB even though it yields poor performance.
  * Clarifying the optimizer and backbone design recommendation rules might strengthen the paper. As I mentioned before, the takeaway from the results would be choosing a good optimizer (e.g., AdamW) and a modern backbone, which is a straightforward conclusion.

* The paper does not provide a comprehensive analysis of the architectural components introduced in Section 2 and Figure 1. For example, I would expect a thorough analysis of the influence of architectural choices (e.g., block designs and isotropic vs. hierarchical structures). The discussions in Section 4.1 are useful, but more in-depth analysis would improve the paper.

* I understand conducting experiments on ImageNet might not be easy, but I am not fully convinced that the conclusions would hold on larger datasets. This is because the performance of ViT families highly depends on the training datasets. It is acceptable if we analyze just the behaviors and properties of ViTs on smaller datasets like CIFAR, but the main results depend on the classification accuracy and variances. For example, the problem mentioned in L266 might be resolved in large data regimes.

* In L322, the paper claims that “while weaker coupling offers more user-friendliness, stronger coupling can potentially lead to better performance and generalization.” However, I couldn’t find strong evidence. I am not convinced that strong coupling is a direct reason for better performance and generalization.

* Similarly, in L354, “stage-wise hierarchical and attention mechanisms can more effectively navigate complex optimization landscapes, unlocking superior performance and generalization capabilities.” However, I feel this claim is somewhat contrary to the observation that modern architectures, including attention mechanisms, have BOCB, as it implies that they require specific optimizers to navigate optimization landscapes. Some research, e.g., Park, Namuk, and Songkuk Kim. "How do vision transformers work?" arXiv preprint arXiv:2202.06709 (2022), claims that ViTs have many non-convex points that lead to poor optimization properties. 

In summary, I believe the questions this paper raises are meaningful. However, some statements lack strong evidence, and the novelty is limited as the takeaways are straightforward. Therefore, I lean toward rejecting this manuscript.

* Minor: I couldn’t find Table 3 for a while. Improving the table layout would enhance readability. Also, a more comprehensive analysis of training dynamics would strengthen the paper. It seems that the transfer learning experiments are among the key results, but they are in the appendix.

questions:
Please refer to the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730615719792

thread_type:
official_review

<thread_object>
creator: reviewer_jghs

<content>
summary:
This work provides an empirical study of the interplay between vision backbones and optimizers. It experiments on common vision datasets like CIFAR-100, ImageNet-1K, and COCO, to document the performance of different architectures (CNN, transformer, etc) and optimizers (SGD-based with momentum, Adam, RMSProp, etc).

strengths:
* The paper seems to tackle a less discussed aspect about the interplay of different optimizers and architectures
* A comprehensive study on the impact of many hyperparameters
* Great summary of existing architectures and optimizers

weaknesses:
* Overall I am not sure the significance of the contribution of this paper is for a conference paper. In my humble opinion, it leans toward a workshop technical report -- certainly empirical study is valuable but I did not find much insights from there. 
* The claims and explanations about BOCB are basically hand-waving like section 4; more rigorous derivations of the inductive bias will greatly improve
* All reasoning is empirical and narrows down to the impact of the accuracy. It is not very convincing to get a clear conclusion from such a study. Section 4 is more like reverse-reasoning from the empirical study from the previous sections. This may be acceptable for a paper that proposes a new method that shows better performance and leaves the community to figure out the true reasons. Still, I am not confident about it for a study paper, which I would expect some theoretical induction or in-depth analysis to make it more convincing.

questions:
What are the authors' thoughts about each of the specific update steps in section 2.2 affecting the performance of different architectures?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731177655725

thread_type:
official_review

<thread_object>
creator: reviewer_orer

<content>
summary:
The paper studies the interaction between popular optimizers and vision backbone architectures. The experiments are conducted over the product of 16 architectures and 20 optimizers on CIFAR-100, ImageNet-1K and COCO.

The authors find that there are notable patterns characterising  which combinations of optimizers + architectures work well together, where some architectures are more sensitive to the choice of optimizer -- which they refer to as “Backbone Optimize Coupling Bias” (BOCB).
From this perspective, the authors thoroughly discuss the various popular vision architectures, based on the design of their components as well as their overall architecture.

strengths:
* The motivation of the paper is clear, it's important to understand the relationship between vision backbones and optimizers.
* The authors have conducted extensive experiments across a large range of optimizers and vision backbones.
* The experiments reveal many interesting observations about how different architectures perform under different optimizers.
* The authors are releasing the benchmarking results and code, which could allow for further analysis by the community.

weaknesses:
* There is no discussion or references to related works studying optimizers for vision backbones and/or transformers. A brief search finds a couple of relevant papers:
- Scaling Vision Transformers, https://arxiv.org/pdf/2106.04560 which discusses e.g. how to adapt Adafactor for ViTs
- How to Fine-Tune Vision Models with SGD, https://arxiv.org/pdf/2211.09359 which studies SGD vs AdamW for finetuning.
- Since most vision backbones are transformer based, other transformer optimizer literature is also relevant -- such as https://arxiv.org/pdf/2002.04745



* In L 197 the authors write "Generally speaking, it is assumed that backbones and optimizers should be unbiased and generalized
components that can be combined freely without significant interdependence". But I don't believe anyone thinks this is the case -- it's well known that optimizer details very often need to adjusted for new architectures.  Furthermore, there are other factors such as model size and overfitting that come to play:
i) a new improved optimizer applied on a an overparameterized architecture may cause it to overfit and hence hurt performance. Why only look at top-1 accuracy but not also training loss?
ii) An architecture may be designed taking an improved optimizer for granted, so why should we expect it to work well with plain old SGD?
ii) A lot of gains can be had from tuning hparams. The authors use the NNI toolbox to pick hparams -- but why not also look at the hparams the authors themselves have proposed for each architecture?

* All of the analysis is of the form "this group of optimizers is working well/poorly for this method or group of methods" but does not really go deeper to explain actually why they perform so differently. E.g. in L260 just tell us that SGD + DeiT-S performs poorly but why? Is it consistent with other findings in the literature? etc

* Some claims made in the paper do not seem to be clearly justified. E.g. 
in L376 the write "The trajectory of vision backbone macro design has significantly sculpted the optimization landscape, progressing through distinct phases that reflect the intricate relationship between network architectural complexity and training challenges  (as illustrated in Figure 1)". I am not sure what exactly the authors are trying to claim here but surely Figure 1 does not illustrate it.
- in L417 they write: "This innovative macro design [of MetaFormer] refines the optimization landscape by harmonizing with
optimizers, leading to reduced BOCB and enhanced performance." How does it "refine the optimization landscape by harmonizing with optimizer"?
- in L445 they write "AttenFormer effectively mitigates BOCB due to its MetaFormer architecture, which incorporates balanced structural designs and residual scaling across stages, enhancing optimization and generalization." How is the "balanced structural design and residual scaling" "enhancing optimization and generalization" here?
- In L456 they write "BOCB in CNNs is often linked to the design of FFN blocks, which are pivotal in models like ConvNeXt" without pointing to any section/table justifying the claim.

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732494036366

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to extend our sincere thanks to Reviewers oReR, JghS, and FcXi for their insightful comments and constructive feedback. In response to their suggestions, we have made significant revisions to the manuscript to enhance its clarity, empirical robustness, and the depth of its contributions. The key points of revision are highlighted in $\color{Brown}brown$ color.

**1. Clarifying the Research Problem and Contributions**

$\quad$ This study addresses a crucial yet under-explored phenomenon in visual representation learning, termed Backbone-Optimizer Coupling Bias (BOCB). This phenomenon has been overlooked for years since Transformers came out and is widely used in various scenarios (as illustrated in Figure 2 in the revision). BOCB reflects the inherent dependencies between specific vision backbones, such as CNNs and ViTs, and the choice of optimizers, significantly affecting training stability and model generalization. Through a comprehensive empirical framework, we evaluated 20 vision backbones against 20 optimizers across multiple datasets (e.g., CIFAR-100 and ImageNet-1k), uncovering patterns of architectural and optimization interdependence. Key findings include the strong coupling of classical CNNs with SGD optimizers and the pronounced reliance of modern architectures like ViTs on adaptive optimizers such as AdamW. These insights not only deepen the theoretical understanding of backbone-optimizer interplay but also provide actionable guidelines for designing architectures and selecting optimizers that mitigate BOCB, ensuring more robust and generalizable vision models.

**2. Summary of Revision Updates**

$\quad$ In this revision, we refined our analysis of how architectural choices in Vision Backbones interact with optimizers to influence BOCB. By systematically evaluating architectural refinements, we demonstrated how BOCB can be mitigated while preserving competitive performance. This work elucidates the mechanisms underlying BOCB and offers practical guidelines for designing resilient vision models. Additionally, we provide a deeper analysis of how BOCB manifests across architectures, emphasizing its implications for model generalization and robustness.

$\quad$ To further strengthen the manuscript, we developed a refined methodology for detecting and characterizing BOCB. This includes a detailed evaluation of optimizers, with a ranking of 20 widely used methods based on their effectiveness. Our analysis shows that certain optimizers, such as AdamW, are more robust against BOCB across diverse architectures. These findings, supported by extensive empirical evidence, offer generalizable recommendations for practitioners and researchers.

$\quad$ Lastly, we expanded our experimental scope to include additional datasets and architectures, enhancing the consistency and generalizability of our conclusions. Experiments on ImageNet, alongside CIFAR-100, confirm that BOCB is a pervasive issue in vision models and that our proposed solutions are effective across different scales and complexities of data. These revisions significantly advance the manuscript's contributions, providing a comprehensive framework for understanding and mitigating BOCB while addressing the reviewers' concerns.

Sincerely,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732811575307

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Encouraging Final Check and Feedback

comment:
Dear Esteemed Reviewers,

We hope this message finds you well. We are writing to express our profound gratitude for the invaluable feedback and thoughtful discussions we have engaged in over the past few weeks. Your insights have been pivotal in significantly elevating the quality of our manuscript, and the improvements we have made are substantial. We believe the current version of the manuscript now stands as a testament to the collaborative effort and dedication we have all invested.

We understand that we are currently unable to submit further revisions, but we want to assure you that we remain fully committed to considering any additional suggestions or concerns you may raise. Should you find the revised manuscript satisfactory, we kindly request that you consider increasing the score accordingly. We are also eager to continue our dialogue and are open to further discussions to address any lingering doubts or questions you might have.

Moreover, we wish to emphasize that we are willing to make further refinements to the manuscript based on any subsequent discussions. Your continued engagement and feedback are of utmost importance to us, and we are dedicated to ensuring that our work meets the highest standards of quality and relevance.

Your participation and the time and effort you have invested in reviewing our work are deeply appreciated. We are hopeful that this revised manuscript will gain the recognition it deserves within the community and contribute meaningfully to the field.

Thank you once again for your unwavering support and constructive feedback. We look forward to the possibility of further collaboration and discussion, and we remain committed to refining our work to meet your expectations.

Warmest regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733210209158

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comments for Serious Concerns Regarding the Irresponsibility of Reviewer JghS:  A Call for Fair and Responsible Evaluation

comment:
Dear (Senior) Area Chairs,

We are writing to express our serious concern regarding the review process of our submission, specifically regarding Reviewer JghS's non-engagement throughout the entire rebuttal stage.

We feel compelled to bring to your attention that despite our comprehensive responses to each initial concern and the substantial improvements acknowledged by other reviewers, Reviewer JghS maintained **complete silence throughout the entire author-reviewer discussion period**, only to provide **a brief, sweeping comment at the very end of the rebuttal phase**.

The **ICLR 2025 Reviewer Guidelines (https://iclr.cc/Conferences/2025/ReviewerGuide)** clearly outline several key responsibilities that we believe have not been met in this case:

1. **Active engagement** during the discussion phase
2. Maintaining **openness to changing initial recommendations** based on author responses
3. Providing **constructive, thorough, and timely** feedback

In particular, Reviewer JghS's **final-hour comment** makes broad claims about analysis depth without providing any specific examples or engaging with our detailed responses and revisions. This type of feedback, delivered at the last possible moment without any prior engagement, **undermines the responsible attitude and collaborative nature** of peer review that ICLR strives to maintain. 

We respectfully request that ACs consider these **concerning circumstances** when evaluating our manuscript. The **stark contrast** between Reviewer JghS's conclusions and that of other engaged reviewers (such as Reviewer FcXi) raises **serious questions** about the fairness of this particular review. We believe our submission deserves evaluation based on **thorough engagement** with our responses and revisions, rather than isolated, last-minute comments that **completely ignore** the discussion phase.

We remain confident in the significant contributions of our paper and its value to the ICLR community. Once again, we appreciate your attention to this matter and **trust in your professionalism and commitment** to maintaining the high quality of ICLR 2025 review process.

Respectfully,

Authors of Submission #2246
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the interaction between vision backbone architectures and various optimizers, introducing the concept of Backbone-Optimizer Coupling Bias to explain how certain combinations of optimizers and architectures perform better than others. Through a comprehensive benchmark across popular vision backbones and various optimizers the authors aim to uncover patterns that characterize effective optimizer-architecture pairings. They conduct experiments on well-known datasets and provide a detailed analysis of how different architectures exhibit sensitivity to specific optimizers, revealing important observations about the optimization landscape and performance trade-offs.

The reviewers acknowledge the authors' efforts to improve their manuscript, including adding updated analysis and visualizations, but significant concerns remain: (1) lack of depth in the analyses and inability to substantiate the BOCB claim with clearer evidence, (2) lack of significant insights into optimization-architecture interplay, (3) overstating contributions, lack of theoretical depth, and limited experimental support for several claims. While the study contributes an extensive empirical analysis across a range of datasets, the findings are seen as somewhat expected or insufficiently justified.