Title: Stepwise Feature Learning in Self-Supervised Learning
Venue: NeurIPS 2025
Accepted: no
Keywords: shortcut learning

self-supervised learning

stepwise learning

feature learning

learning dynamics
Abstract: Recent advances in self-supervised learning (SSL) have shown remarkable progress in representation learning. However, SSL models often exhibit shortcut learning phenomenon, where they exploit dataset-specific biases rather than learning generalizable features, sometimes leading to severe over-optimization on particular datasets. We present a theoretical framework that analyzes this shortcut learning phenomenon through the lens of $\textit{extent bias}$ and $\textit{amplitude bias}$. By investigating the relations among extent bias, amplitude bias, and learning priorities in SSL, we demonstrate that learning dynamics is fundamentally governed by the dimensional properties and amplitude of features rather than their semantic importance. Our analysis reveals how the eigenvalues of the feature cross-correlation matrix influence which features are learned earlier, providing insights into why models preferentially learn shortcut features over more generalizable features.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical framework to explain shortcut learning in self-supervised learning (SSL), specifically through the lens of stepwise feature learning dynamics. The authors propose that during SSL training, models preferentially learn features with higher spatial extent (extent bias) or larger amplitude (amplitude bias), regardless of their semantic relevance. By analyzing the eigendecomposition of the feature cross-correlation matrix in toy Barlow Twins models, the paper demonstrates how features with larger eigenvalues are learned earlier due to the dynamics of gradient flow and weight singular value evolution. These findings are validated in both linear and non-linear (MLP) settings, and further tested on a Colored-MNIST dataset, where models are shown to initially rely on background features before eventually learning object-relevant ones. The work provides theoretical and empirical insights into the biases driving SSL training, with implications for designing future SSL methods that avoid such shortcuts and prioritize generalizable features.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper provides a clear and original theoretical framework that connects shortcut learning in self-supervised learning (SSL) to stepwise feature learning governed by feature extent and amplitude. The use of eigendecomposition and gradient dynamics to explain learning order is both elegant and insightful. The work is well-structured, and its conclusions are consistently validated across linear models, MLPs, and a synthetic Colored-MNIST dataset. These findings offer meaningful insights into the mechanisms underlying spurious feature learning in SSL.\n\nWeaknesses:\n\nThe primary limitation lies in the reliance on toy models and synthetic datasets, which may reduce the applicability of the conclusions to real-world SSL settings. The practical implications—how to mitigate extent or amplitude bias—are not fully explored. Additionally, the theoretical framework assumes idealized conditions (e.g., feature independence, aligned initialization) that may not hold in complex, modern architectures. More empirical evidence on realistic datasets would strengthen the impact.\n\nQUESTIONS:\n(1) The organization of the amplitude-bias section could be improved\n\nThe transition from frequency to amplitude bias in Section 5 is somewhat abrupt. Although both arise from the same motivation and experimental setup, the narrative jumps between them without first clearly introducing amplitude bias. We suggest restructuring the section to improve clarity. \n\n(2) Interpretability and Practical Implications\n\nWhile the paper provides strong theoretical insights, its practical implications remain unclear. Could the authors summarize key takeaways for practitioners designing or training SSL models? Additionally, although extent and amplitude bias are well-diagnosed, the paper does not propose mitigation strategies. It would be helpful to discuss how the theoretical framework might inform methods (e.g., regularization or augmentation) to encourage learning of more generalizable features. \n\n(3) Broader Empirical Validation\n\nThe analysis primarily relies on toy settings with synthetic data. To strengthen external validity, could the authors include or comment on experiments using more realistic or semi-synthetic datasets? A systematic study across multiple datasets or SSL frameworks (e.g., SimCLR, BYOL) would help verify whether extent and amplitude bias persist in practice and enhance the relevance of the findings.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors extend the theoretical analysis by Simon et al. to demonstrate the feature learning biases that exist in self-supervised learning setups, e.g. BarlowTwins. The work is well-positioned with references to empirically-observed feature learning issues in supervised learning, and introduces well-constructed theoretical settings to study equivalent pitfalls in self-supervised learning. Moreover, the authors verify their theoretical analysis with empirical results in simplistic linear network settings, as well as non-linear network settings. Overall, this work presents a neat theoretical analysis to study commonly observed pitfalls in feature learning, which might be interesting to the community.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is very well-written and easy to follow. The claims are described clearly and supported with proper theoretical analysis and empirical results. \n2. The theoretical problem setup is nicely chosen and explained. While the data model might be restrictive for practical purposes, it helps the reader appreciate the theoretical contributions and gain an intuitive understanding of the underlying phenomenon. \n3. The empirical results in non-linear setting are a good addition, and the colored-MNIST experiment is a fantastic example of a controlled experiment. Unfortunately, such experiments are rare in the deep learning literature, so kudos to the authors for crafting such an experiment and demonstrating their core claim.\n\nWeaknesses:\n1. I think a core issue with this work (that limits me from rating it higher) is its applicability. The current results are limited, both in terms of experimental evaluation in larger networks and complex datasets as well as the theoretical setup. Specifically, the authors extended the dynamics studied by Simon et al. for the BarlowTwins loss with a high redundancy reduction coefficient. In practice, the redundancy reduction coefficient is significantly smaller, and the resulting dynamics are distinct. Recent work [1] has studied these distinctions, which might be of interest to the authors. Therefore, I would recommend the authors to extend their analysis following the results of Ghosh et al. (see Theorems 3.2 & B.4). I believe this work would greatly benefit from comparing the feature learning behaviour studied by the authors in their setup for the high vs low redundancy reduction coefficient cases. \n2. While the data model presented here is better for theoretical advances, I feel it is restrictive to study the feature learning properties in realistic datasets. As a result, I am unsure how useful the results of this study will be to advance practical SSL pipelines. For instance, can the authors provide recommendations on how to improve the feature learning properties by changing the data augmentation strategy or with some dataset preprocessing? \n3. While the non-linear network experiments demonstrate that the theoretical results from linear networks hold for non-linear networks as well (which is very interesting and great to see), the eigenvalue dynamics seem to be different. Unfortunately, the authors do not comment on this discrepancy in the current version of the work. Also, I would recommend the authors to consider moving these results to the main text instead of the appendix. Furthermore, I also feel this is a great opportunity to talk about the effect of non-linearity on feature learning in intermediate layers of the network and study the distinctions in learning dynamics of features vs embeddings in BarlowTwins. Recent work [2] has made some progress in this direction, which might be of interest to the authors.\n\nOverall, I feel this work is exciting (both personally and overall for the field) and presents some interesting insights for the self-supervised learning community. However, the discrepancy from practical settings limits the utility of the results presented here. If the authors can resolve some of these discrepancies, I will be happy to reconsider my score.\n\n- [1] Ghosh et al. Harnessing small projectors and multiple views for efficient vision pretraining. NeurIPS 2024.\n- [2] Xue et al. Investigating the Benefits of Projection Head for Representation Learning. ICLR 2024.\n\nQUESTIONS:\n1. While the data model in Section 4.1 and 5.1 are interesting and makes the theoretical analysis easy, how general do you think they are for realistic datasets? Can you provide an example where this can be applied?\n2. The notations of $f_h$ and $f_l$ are confusing because it makes the reader think that they correspond to high and low frequency respectively. Is there a specific reason to use this notation? Can you change it to $f_{ha}$ and $f_{la}$ to indicate high and low amplitude, respectively?\n3. In the derivation in Appendix B.2, is there a missing $f$ in the denominator (after integrating $cos(2fx)$)? I don't think it changes the conclusion, but curious if it's a typo or something that I missed.\n4. The discussions section talks about the absence of frequency shortcut in BarlowTwins models, as opposed to supervised learning models. Can you comment more on the reason for this? Is it possible to talk about this as a pro over supervised learning (or maybe cross-entropy type losses)? Could this be related to work from Balestriero & Lecun [3]?\n\n- [3] Balestriero et al. How Learning by Reconstruction Produces Uninformative Features For Perception. ICML 2024.\n\nLIMITATIONS:\nYes, addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work studies shortcut learning in SSL, in relation to the step-wise learning dynamics of a linear Barlow Twins model, under strong orthogonality constraints.\n\nThe authors consider two models of data features inducing shortcut learning: those with large spatial extent, and those with large amplitude. Both kinds of features correspond to eigendirections with large eigenvalues in the data covariance matrix, which dominate learning under a gradient-flow model. The resulting theory provides a formal model of extent and amplitude bias, which is matched empirically in toy settings, as well as with non-linear models (MLPs) and ColoredMNIST.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper provides a clear theoretical formalization of two settings contributing to shortcut learning. Formally understanding this problem in the context of SSL is novel and very important, for both practitioners as well as theoretically.\n\n2. Results are presented clearly and concisely, and the paper is generally well written and easy to follow.\n\n3. Results are validated on small non-linear models and toy empirical settings.\n\n**Weaknesses**\n\n1. While the linear model is extremely useful for drawing intuitions, the authors fall short of providing a “kernelized” version. The work of Simon et al. (2023), which the present submission builds upon, provides a kernel version of the toy Barlow Twins model, which could have been exploited by the authors.\n\n2. The proposed linear model cannot account for architecture bias, which heavily influences which eigendirections are learned first and the speed of convergence. This has been studied for SSL in the kernel setting.\n\n3. The problem of subspace alignment is heavily influenced by the loss function, as well as optimizer hyperparameters. Specifically, I would have appreciated a discussion of the role of the redundancy term in the Barlow Twins loss (see Questions).\n\n4. The role of the noise covariance (data augmentation) is not discussed (see Questions). \n\n5. Some results (Theorem 4.2 and 4.3) are direct consequences of Simon et al. (2023), and thus constitute a validation of the prior work’s theory rather than a novel contribution. Such validation could have been moved to the appendix, in favour of additional discussion (e.g. on kernels).\n\n6. The validity of the feature orthogonality assumption in practical settings should be discussed and should ideally be supported empirically.\n\nQUESTIONS:\n1. When Barlow Twins (or other equivalent losses) are kernelized, it is possible to study the bias arising from model architectures (which essentially encodes which eigendirections are preferred). Could the authors relate their findings to Cabannes et al. (2023), Proposition 4 (and related discussions)?\n\n2. Again in the kernel setting, the interplay between data augmentation and the redundancy term (controlled by $\\lambda$) of the Barlow Twins loss has been discussed in the literature (Ghosh et al. 2024). Could Theorem 4.5 be adjusted to account for Barlow Twins under weaker orthogonality constraints?\n\n3. Data augmentation influences the structure of the data covariance matrix. Several covariance operators have been theoretically discussed in the literature (Lin et al., 2024) beyond the isotropic setting, in a similar toy linear regression model (albeit for supervised learning). How does the noise covariance affect feature learning? Particularly, shouldn’t $\\Gamma$ also depend on the noise covariance in Equation 5?\n\n4. In the high-dimensional setting, whereby feature covariances exhibit heavy-tailed behaviour (Agrawal et al., 2022), how likely is the assumption that large superpixels correspond to large-eigenvalues to hold in practice, once features are averaged over a large dataset? \n\n**Minor Questions**\n\n5. In section 4.8, what is the standard digit-to-pixel ratio of ColoredMNIST? \n\n6. Can Theorem 4.3 predict how long the test accuracy plateaus for in Figure 2?\n\n\n**Minor mistakes**\n\n7. The work of Doersch, et al. appears both as citation 10 and 11.\n\n**References**\n\n* The SSL Interplay: Augmentations, Inductive Bias, and Generalization. Cabannes et al. ICML 2023.\n* Harnessing small projectors and multiple views for efficient vision pretraining. Ghosh et al. NeurIPS 2024.\n* The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective. Lin et al. JMLR 2024.\n* alpha-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay. Agrawal et al. NeurIPS 2022.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors build upon a previously established theoretical framework for stepwise feature learning in self-supervised models and examine two illustrative toy examples to reveal extent bias and amplitude bias. They derive theoretical predictions for feature extraction under a specific initialization of a linear Barlow Twins model and support these findings with synthetic experiments. Finally, they demonstrate the practical relevance of their analysis through controlled Colored-MNIST trials.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**S1.** The paper presents rigorous theoretical results. The proofs I have verified are correct.\n\n**S2.** The Colored-MNIST experiment provides a clear and insightful demonstration of the model’s tendency to first learn background features before capturing the semantic content.\n\n**Weaknesses**\n\n**W1. (significance)** The scope of the theoretical results is somewhat narrow, being confined to linear encoders, the Barlow Twins method, and highly simplified synthetic datasets, which may limit their generalizability.\n\n**W2. (significance)** The analysis is based on a carefully selected initialization without quantifying the impact of deviations or error terms around this initialization, potentially restricting the robustness of the findings.\n\n**W3. (clarity)** Certain subsections in Section 4 appear misaligned with the central narrative: subsection 4.5 seems tangential (at least in the context of Sec. 4), and subsection 4.8 employs a supervised learning setup rather than applying Barlow Twins. I recommend revising Section 4 to establish a clear and coherent logical flow from the outset.\n\n**W4. (clarity)** Definition 4.4 raises concerns about potential basis dependence in the subspace alignment metric. The authors should clarify this point or provide a formal proof of its invariance under different basis choices.\n\nQUESTIONS:\n**Q1.** Could the authors please clarify the precise definition and role of Equation (6), specifically explaining the notation $e=e_l,e_s$?\n\n**Q2.** In the theoretical analysis, the predicted critical-time behavior of the dynamics of $s_t$ doesn't actually hold up if we were to plot the function. Despite this, the empirical curve follows the prediction and Figure 1 seems to exhibit the predicted dynamics. Could the authors comment on this apparent discrepancy?\n\n**Q3.** In Figure 1, the trajectory of the second eigenvalue appears to pause for an extended interval. Could the authors explain the underlying reason for this prolonged halting behavior?\n\n**Q4.** Theorem 4.2 posits that the derivative at non-matching critical points is approximately zero, yet no formal proof is provided. Could the authors supply a justification or reference for this claim?\n\n**Q5.** Subsection 4.8 employs a supervised learning setup rather than the Barlow Twins method. Could the authors explain the relevance of this experiment to self-supervised feature learning and its connection to the main SSL framework?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}