Title: Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Self-Supervised Learning

Representation Learning
Abstract: Reconstruction and joint-embedding have emerged as two leading paradigms in Self‑Supervised Learning (SSL). 
Reconstruction methods focus on recovering the original sample from a different view in input space. On the other hand, joint-embedding methods align the representations of different views in latent space.
Both approaches offer compelling advantages, yet practitioners lack clear guidelines for choosing between them.
In this work, we unveil the core mechanisms that distinguish each paradigm. 
By leveraging closed-form solutions for both approaches, we precisely characterize how the view generation process, e.g. data augmentation, impacts the learned representations.
We then demonstrate that, unlike supervised learning, both SSL paradigms require a minimal alignment between augmentations and irrelevant features to achieve asymptotic optimality with increasing sample size. 
Our findings indicate that in scenarios where these irrelevant features have a large magnitude, joint-embedding methods are preferable because they impose a strictly weaker alignment condition compared to reconstruction-based methods. 
These results not only clarify the trade-offs between the two paradigms but also substantiate the empirical success of joint-embedding approaches on real-world challenging datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors compare two popular self-supervised learning (SSL) families—reconstruction (e.g., MAE) and joint-embedding (e.g., BYOL, SimCLR, BarlowTwins).\nBy analysing a simplified linear setting they prove that which family works better depends on how strongly “unwanted” input variations (noise, corruptions, background) show up in the data and how well the chosen augmentations imitate those variations. They then confirm the theory with small synthetic tasks and with ImageNet-C / CIFAR-10-C experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe research question is very practical and important: Both loss families are widely used, yet few papers ask which one to pick and when.\nThe theory is clean: A straightforward linearization yields transparent derivations and an easy-to-remember rule of thumb: if the unwanted variations are mild, reconstruction suffices; if they are strong, joint-embedding performs better.\nThe paper has decent empirical support:  Simple linear demos and modern vision models on corrupted ImageNet/CIFAR largely follow the predicted pattern.\n\nWeaknesses:\nThe promised guidance for practitioners is thin. The paper never demonstrates how to estimate augmentation–noise alignment or noise strength on real, unlabelled data.\nAll proofs rely on additive Gaussian noise and scaled augmentations. Real augmentations (crops, colour jitter, CutMix, masking) are more complex, so it is unclear how well the conclusions generalise.\nThe theory assumes that signal and noise occupy orthogonal directions, an assumption that seldom holds for natural images or text, where the two are usually entangled.\n\nQUESTIONS:\nHow could one estimate alignment or noise magnitude from raw, unlabelled data in practice?\n\nCan the analysis be extended to non-Gaussian augmentations such as random crops, colour jitter or CutMix? In particular, how might you handle the patch-masking used in MAE?\n\nDo you have intuition or experiments for settings where signal and nuisance features overlap in the same subspace?\n\nLIMITATIONS:\nNo.\nThe paper never talks about what it can’t do or any downsides. It should briefly add:\nSimplifying assumptions – It assumes linear models and simple Gaussian noise. Real data and augmentations are messier, so results may change.\nSmall-data limits – All proofs need huge sample sizes; no clue if things hold for typical datasets.\nHow to use it – Readers still don’t know how to measure “noise strength” or “alignment” in practice.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the differences in the learning dynamics and performance of reconstruction-based and joint embedding-based self-supervised approaches in an attempt to enable and guide practitioners to make an educated choice of one over the other.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1) Well motivated and overall writing is good.\n2) A variety of experiments backup authors’ claims.\n3) Theoretical results consistent with and further explain results and patterns in previous work.\n4) The work would equip researchers and practitioners to better be able to decide on approaches\n\n**Weaknesses**:\n1) Unclear exactly what a good alignment of noise and augmentation means in practice. As far as I could tell, in real-world datasets when the high-frequency details are somewhat irrelevant, joint-embedding techniques are better. However, how do we judge this on other datasets? How can we judge if specific augmentations would be effective?\n2) Please refer to questions below.\n\nQUESTIONS:\n1) Are linear models used in section 3 a good proxy for the complex non-linear models that are used in practice? How do we ensure that the findings transfer to these more complex non-linear models?\n2) I understand how increasing alpha in Eq. 7 results in a more aligned augmentation with the noise added. However, how do the authors control for “unaligned” noise? It seems to me that the extremes of alpha either result in a dominant aligned augmentation with high alpha or just the theta term with alpha as 0. Shouldn’t we also control for the magnitude of theta?\n3) Propositions 4.2-4: How do we conclude that the limit holds (almost surely) as either alpha or n goes to positive infinity? It is unclear to me from the main text without any pointers to appendix.\n4) Figure 2, bottom right (Strong noise, reconstruction). Do the authors have insight why the model with strong alignment lags behind at a low sample size despite high alignment between the noise and the augmentation?\n5) For Table 1, I would highly suggest color-coding or adding a symbol to help readers separate joint embedding (BYOL, DINO) methods from reconstruction (MAE) methods at a glance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors theoretically analyze a linear implementation of two well-known self-supervised learning paradigms: reconstruction and joint-embedding based. The theoretical analysis is made under the assumption of existing data augmentations, i.e., the existence of a distribution of transformations. This idea has been previously explored in supervised learning, where the results conclude that the quadratic risk minimization problem is equivalent to a Tikhonov regularized problem. The authors derive, using similar principles, closed-form solutions for reconstruction and joint-embedding problems, where the results clearly indicate the tendency of the optimal solutions to align with the principal axes of variation in the data. Finally, under asymptotic regimes of infinite data and augmentation alignment, they derive critical criteria regarding the alignment of the models with the signal components that are important for solving the task. Experiments on synthetic and real datasets (with well-known SSL  implementations) validate the theoretical conclusions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- The study presents the first attempt at characterizing the differences between joint-embedding and reconstruction/generative models in closed form under a distribution of augmentations. This is an interesting step ahead in a field that is becoming increasingly more relevant in practice.\n- The closed form results for both paradigms are derived cleanly, and they align with intuition, for example a reconstruction based method will have an optimal encoder that captures data covariance and the covariance due to augmentations, project it on the top-k directions and then freely transform it (linearly) to maintain this k-rank projection. This corresponds with the intuition practitioners have regarding generative models as \"closely fitting the details of the data distribution.\"\n- The derivation of the two regimes of $\\alpha$ and which type of model can perform better under different noise and augmentation regimes is an important result. \n- The experiments, although few, confirm the theoretical results both on ideal (synthetic) and real-world cases. This is an optimal result, especially considering that the analysis is done on linear models and linear optimization principles. \n\nWeaknesses and Questions:\n- One of the main assumptions of the theoretical analysis is that the labels are considered to be continuous in order to frame a least squares task. While this obviously allows for an in-depth theoretical exploration, how would it affect the performance of SSL models downstream on categorical matching tasks?\n- In Equation 7, where the augmentation distribution is formalized, is my understanding correct that the directions that are unimportant to the task are augmented more, i.e., there is more noise added, as alpha grows, in the directions with $\\lambda_i^\\Gamma = 0$. Is this correct? \n- As a minor element and more of a suggestion, I believe the proofs would benefit more people looking to understand theoretical research in SSL if some of the identities, e.g., regarding matrix traces and norms, were expanded more in the first few derivations. This greatly helps the readability of the more technical material.\n\nQUESTIONS:\nPlease consult the weaknesses section above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the trade-offs between two major self-supervised learning paradigms (reconstruction-based and joint-embedding-based methods) by analysing how these methods respond to noise and data augmentation. The authors derive closed-form solutions for linear models under both objectives, enabling a precise comparison of how augmentation alignment with irrelevant (noisy) input features affects learning. A central finding is that joint-embedding methods are more robust in high-noise scenarios, as they require weaker alignment conditions to achieve good performance. Conversely, reconstruction-based methods are shown to be preferable in low-noise settings, where they benefit from the ability to directly reconstruct relevant features without needing heavy augmentation tuning. These insights are validated across synthetic datasets, MNIST-style tasks, and ImageNet experiments. The paper provides useful theoretical grounding for an important design choice in SSL.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\nThe theoretical results are clearly stated, with assumptions that are reasonable in the linear setting. Proofs in the appendix are detailed and rigorous. On the experimental side, the setup is comprehensive, covering both synthetic and real-world data, and includes both linear and deep model evaluations. One area that could benefit from further discussion is the reconstruction-based method's behaviour in Figure 3. Its performance does not always improve with stronger augmentation alignment (higher $\\alpha$), which seems at odds with the theoretical predictions. This may reflect finite-sample effects or practical limitations, and would be worth addressing more directly.\n\nClarity:\nThe writing is clear, and the paper is logically organised. The theoretical development is presented in a way that is accessible to readers familiar with SSL, and the motivation for each component is well articulated. Figures are thoughtfully chosen and support the key messages. The appendix is well structured and easy to navigate.\n\nSignificance:\nThe paper offers a thoughtful theoretical perspective on when to prefer joint-embedding versus reconstruction-based SSL methods. It adds clarity to a design choice that is often made based on empirical benchmarks or intuition. The analysis of how noise and augmentation alignment influence performance is particularly insightful and contributes to a deeper understanding of SSL behaviour. That said, in real-world scenarios, it may not always be clear whether one is operating in a low- or high-noise regime, which could limit the immediate practical applicability of the guidance. Nonetheless, the theoretical framework is likely to inform future research and help shape more principled approaches to SSL model development.\n\nOriginality: A key strength of the paper is its analytical treatment of the impact of augmentations. By explicitly modelling the augmentation distribution, the authors derive closed-form solutions that make it possible to analyse how SSL objectives respond to alignment between irrelevant features and augmentation. This leads to actionable insights, such as when reconstruction-based SSL is preferable and when joint-embedding is more robust. This kind of augmentation-aware analysis is novel in the SSL literature and adds a useful new angle to our theoretical understanding of self-supervised learning.\n\nQUESTIONS:\n1. In Appendix Figure 3, the probing error for the reconstruction-based method under high alignment ($\\alpha = 10$) is not consistently lower than that of $\\alpha = 1$ or even $\\alpha = 0.1$. This appears to deviate from Proposition 4.3, which suggests that performance should improve with stronger alignment. Could the authors elaborate on whether this is due to finite sample effects, model instability under large augmentations, or something else? A brief discussion of this discrepancy and its implications would be valuable.\n\n2. Could the authors clarify what latent dimensionality (i.e., value of $k$) was used in the synthetic experiments? While the theoretical derivations describe closed-form solutions in terms of the top‑$k$ components, I couldn't find an explicit mention of the actual value chosen. Since this dimensionality affects the probing performance and the comparison between methods, it would be helpful to know how $k$ was selected and whether it was consistent across settings.\n\n3. The experimental settings for the reconstruction-based method do not mention the use of masking, which is a common component in modern reconstruction-based SSL approaches (e.g., MAE). Could the authors clarify whether any form of input masking was used in their reconstruction experiments? If not, do they expect their conclusions about the comparative performance of reconstruction vs. joint-embedding methods to hold if masking were incorporated?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}