Title: Contrastive Self-Supervised Learning As Neural Manifold Packing
Venue: NeurIPS 2025
Accepted: yes
Keywords: self-supervised learning

neural manifold

vision

statistical physics in learning

computational neural science
Abstract: Contrastive self-supervised learning based on point-wise comparisons has been widely studied for vision tasks. In the visual cortex of the brain, neuronal responses to distinct stimulus classes are organized into geometric structures known as neural manifolds. Accurate classification of stimuli can be achieved by effectively separating these manifolds, akin to solving a packing problem. We introduce Contrastive Learning As Manifold Packing (CLAMP), a self-supervised framework that recasts representation learning as a manifold packing problem. CLAMP introduces a loss function inspired by the potential energy of short-range repulsive particle systems,
such as those encountered in the physics of simple liquids and jammed packings. In this framework, each class consists of sub-manifolds embedding multiple augmented views of a single image. The sizes and positions of the sub-manifolds are dynamically optimized by following the gradient of a packing loss. This approach yields interpretable dynamics in the embedding space that parallel jamming physics,
and introduces geometrically meaningful hyperparameters within the loss function. Under the standard linear evaluation protocol, which freezes the backbone and trains only a linear classifier, CLAMP achieves competitive performance with state-of-the-art self-supervised models. Furthermore, our analysis reveals that neural manifolds corresponding to different categories emerge naturally and are
effectively separated in the learned representation space, highlighting the potential of CLAMP to bridge insights from physics, neural science, and machine learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new objective criterion for contrastive self-supervised learning. The key idea is based on: (i) a novel representation that treats different augmentations of the same image as submanifolds parameterized by ellipsoids, and (ii) a criterion that quantifies the overlap between these ellipsoids to select negative pairs. The selected ellipsoids are then used in the objective function, encouraging the network to produce similar representations within each ellipsoid while repelling the centroids of different ellipsoids. Overall, the proposed strategy can be seen as a method to effectively harness negative sample pairs.\nThe experimental analysis is conducted by first evaluating linear probing performance on ImageNet-100/1K, comparing the proposed method against several self-supervised baselines, and then by analyzing the properties of the learned representations along with their neuroscientific interpretations. The results demonstrate that the proposed approach performs comparably to state-of-the-art self-supervised learning methods and shows strong alignment with neuroscientific findings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**\n- The paper is well written and easy to follow **Clarity**\n- The idea of using ellipsoids and quantifying their overlap is intuitive, simple, yet original **Originality**\n- The experimental methodology on linear probing evaluation is sound **Quality**.\n- The experiments to evaluate the alignment with neuro-scientific findings are highly appreciated **Quality**\n- Code is provided along with the submission **Code availability**\n\n**Weaknesses**\n- The main weakness lies in the significance of the proposed solution **Significance**. Since the method achieves performance comparable to the baselines in linear probing, it remains unclear whether the observations reported in Sections 5, 6, and 7 also hold for the baselines. The claims and analysis would be much stronger and more convincing if comparable results were provided for those cases.\n- There is extensive literature on harnessing negative samples in contrastive learning, e.g. [1], as well as approaches that enforce clustered representations without explicit clustering, e.g. [2] **Quality**. A discussion of related work is necessary to better contextualize and highlight the novelty of the work.\n\n**References** \\\n[1] Contrastive Learning with Hard Negative Samples. ICLR 2021 \\\n[2] Collapse-Proof Non-Contrastive Self-Supervised Learning. ICML 2025\n\nQUESTIONS:\nPlease refer to the above mentioned weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new approach for self-supervised learning of manifolds, based on the idea of packing problem - the idea that different \"objects\" cannot occupy the same space, so that if \"overlapping\" objects repel each other, we may find a packing of the objects in space in which they do not overlap. Here, \"objects\" are actually \"object manifolds\", the collections of neural responses to the same objects. If objects overlap, they cannot be classified correctly; thus, a physical-simulation-like learning where object manifolds repel each other leads to representations where objects can be reliably classified from one another.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n 1. Great idea and clear presentation of the physics intuition, with our understanding of jammed objects really helping to visualise the process.\n 2. The results are interesting, showing comparable performance to other methods (what the authors call \"competitive\").\n 3. Provide a very simple framing which pushes the representation toward both uniformity (large distances between manifolds) and small size (small distances within manifolds).\n\n## Weaknesses\n 1. The \"with only local operations\" argument is weird and misleading: while only \"near-by\" objects get pushed, this push would change the representation of ALL objects (because of changes to the encoder). \n 2. It is not clear which changes happen in the encoder and which happen in the embedding. As the embedding is not used for classification problems, some of the learned information is actually lost!\n 3. The difference between MMCR and CLAMP is that the former consider all binary linear classifications, which CLAMP consider all binary non-linear classifications. The benefit of \"local interactions\" (lines 11-113) is silly as noted above. \n 4. The results on brain-score are basically null-results (except for V2 maybe), showing SOME correspondence between the learned representations and the brains' representations, without a clear rule. Section 6 could have been omitted.\n 5. The argument on a similar exponent like V1 is extremely weak, as there are probably 4-5 controllable parameters which affect the exponent, making the results meaningless. Section 7 could have been omitted.\n\nQUESTIONS:\n1. Do CLAMP have meta-parameters which control the balance between alignment and uniformity (lines 119-121)? i.e., control the resulting manifold uniformity despite the locality of all interactions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Contrastive Learning As Manifold Packing (CLAMP), a new self-supervised learning method for visual representations. The key idea is to model the embeddings from augmented views of an image as a \"sub-manifold,\" which is then approximated by a high-dimensional ellipsoid. Learning is driven by a single loss function inspired by short-range repulsive potentials in physical systems. This loss function simultaneously encourages the separation of sub-manifolds from different images (negative pairs) and the contraction of individual sub-manifold volumes (positive pairs). The authors argue this unified approach is elegant, interpretable, and effective at preventing representational collapse. The method is evaluated on CIFAR-10, ImageNet-100, and ImageNet-1K using linear probing and semi-supervised learning protocols , where it demonstrates promising, and in one case, state-of-the-art results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.\tThe main contribution is the novel framing of SSL as a manifold packing problem. This is a refreshing and intuitive departure from point-wise contrastive methods. \n2.\tThe physics-based motivation provides a clear mental model, and the resulting loss function (Eq. 1) is elegant. Its hyperparameters, like the scaling factor have a tangible physical meaning, which is a commendable feature.\nWeaknesses\n1.\tThis is the most significant weakness. The paper's empirical support is not comprehensive enough to demonstrate the superiority or generalizability of the proposed method.\n2.\tThe evaluation is restricted to linear and semi-supervised classification. Modern SSL methods are expected to be benchmarked on a wider array of downstream tasks, such as object detection and semantic segmentation on datasets like COCO, to prove the robustness and richness of the learned features. Without this, it is difficult to ascertain if CLAMP's representations are truly general-purpose.\n3.\tWhile ImageNet and CIFAR-10 are foundational, the SSL field is moving towards more complex and varied datasets. Demonstrating strong performance exclusively on these benchmarks is no longer a sufficient condition for claiming SOTA status for a general vision backbone.\n4.\tThe ablation study for key hyperparameters (Appendix K) was performed on a 30% subset of the ImageNet-1K training data. While the authors cite computational constraints, this reduces the reliability of the findings and makes it difficult to assess the true sensitivity of the model to parameters like and the number of views m.\n5.\tThe framework is built on strong simplifying assumptions whose validity is not sufficiently proven by the limited experiments. The approximation of a complex augmentation distribution as a single ellipsoid is a major simplification. The paper does not provide empirical analysis to justify that this geometric approximation is robust across different augmentation strategies or that it captures the essential structure needed for learning. The limited experiments do not provide enough evidence to bridge the gap between the elegant theory and the practical implementation.\n\nQUESTIONS:\n1.\tGiven that the central claim is about learning better-structured representations through geometric packing, wouldn't evaluation on tasks that heavily rely on rich spatial features (e.g., object detection, segmentation) be the most convincing way to validate this? \n2.\tThe paper's claims hinge on the ellipsoid approximation being a valid model for augmentation sub-manifolds. What empirical evidence can the authors provide to show this is a robust choice? For instance, how do the results change if a more complex geometric model is used, or when faced with augmentations that might produce a non-convex or multi-modal distribution of embeddings?\n3.\tIn the training dynamics (Figure 3b), the sub-manifold size decreases as expected. What mechanism in the framework prevents a trivial solution where the radii collapse to near-zero values for all sub-manifolds, thus minimizing the loss? \n4.\tThe performance of many contrastive methods is highly dependent on batch size. Since CLAMP's loss is computed over all pairs in a batch, it likely shares this dependency. How does CLAMP's performance scale with varying batch sizes?\n\nLIMITATIONS:\n1.\tThe primary limitation is the scope of the evaluation. As detailed in the weaknesses, the lack of experiments on diverse downstream tasks makes it difficult to fully assess the quality of the representations learned by CLAMP. This should be explicitly stated as a key limitation of the current work.\n2.\tThe simplification of manifold geometry by ignoring orientation and using an isotropic radius approximation is a significant constraint, as the authors themselves note. This limits the \"packing\" from achieving its full theoretical potential and makes the physics analogy less direct than it could be.\n3.\tThe framework's effectiveness is intrinsically tied to the choice of data augmentations, as these define the very geometry of the sub-manifolds being packed. The paper would benefit from a more thorough discussion of this dependency and its implications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel reframing of contrastive self-supervised learning (CSSL) by casting it as a manifold packing problem in representation space. Instead of viewing representation learning as a purely statistical or mutual information maximization task, the authors approach it geometrically: they introduce a new loss function inspired by the short-range repulsive forces in particle physics, encouraging the learned representations to compactly fill space with minimal overlap. The paper offers an elegant and theoretically grounded framework for understanding representation learning through the lens of neural manifold packing. The fact that it achieves the best brain score for V2 and IT suggests a potential computational-level convergence between this framework and the primate visual systems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. A major contribution is the theoretical and empirical demonstration that this formulation leads to the emergence of low-dimensional, well-separated manifolds, each corresponding to different semantic image categories—even in the absence of labels. This structure supports linear decodability and provides a geometric rationale for the effectiveness of self-supervised representations.\n2. Experiments on image datasets (CIFAR-10, STL-10, etc.) confirm that learned representations exhibit this packing behavior. Even without labels, the manifolds align with semantic classes.\n3. One of the most striking results is that the learned representations achieve state-of-the-art brain scores in both area V2 and IT cortex when evaluated against neural data, suggesting a degree of alignment between the structure of artificial representations and those found in the primate visual system. Similar geometric structuring of representations (low-dimensional manifolds, separation of class-specific activity) has been observed in neural population codes in primate cortex. While the biological plausibility of the learning mechanisms remains uncertain, the computational-level similarity (in Marr's terms) offers intriguing evidence that efficient manifold packing may be a shared organizational principle.\n4. The paper brings much-needed theoretical clarity to the success of contrastive learning. It contributes not only a practical improvement in self-supervised learning but also a conceptual bridge between deep learning and neuroscience. It provides a unifying geometric perspective that connects contrastive objectives, biological representations, and manifold theory. \n\nWeaknesses:\n1. While the geometric structure aligns well with cortical representations, the underlying mechanisms—such as particle-inspired repulsion—are not themselves biologically grounded, and more work is needed to connect this theory to known neurophysiological learning rules.\n2. Evaluation: The evaluation presumes alignment between data structure and downstream labels. It's less clear how CLAMP behaves when class structure is ambiguous or hierarchical.\n3. While packing helps linear separability, how does it generalize to unseen concepts or transfer across tasks? \n4. The practical benefit is modest. The experiments are solid but limited to standard benchmarks; it would be valuable to see if the same geometric structure emerges in larger-scale vision-language or multimodal settings.\n\nQUESTIONS:\n1. Does manifold packing scale to hierarchical or compositional concepts?\n2. What practical benefits does this new perspective offer for self-supervised learning?  Improvement on the loss function? Does it lead to the design of new objective?\n\nLIMITATIONS:\n1. While similar geometric structuring of representations (low-dimensional manifolds, separation of class-specific activity) has been observed in neural population codes in primate cortex, it is not certain how this can be implemented using biological mechanisms. Is this primarily a computational theory level description of the principle?\n2. It is not certain how manifold the packing framework scales to hierarchical or compositional concepts, or large dataset.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}