Contrastive Self-Supervised Learning As Neural Manifold Packing
Authors: Guanming Zhang
David Heeger
Stefano Martiniani
Venue: NeurIPS 2025 poster ()
Keywords: self-supervised learning
neural manifold
vision
statistical physics in learning
computational neural science
Primary area: general_machine_learning
TL;DR: We recast contrastive self‑supervised learning as neural‑manifold packing, employing a physics‑inspired loss to separate sub‑manifold embeddings during pretraining and achieve high accuracy under linear evaluation.

Abstract: Contrastive self-supervised learning based on point-wise comparisons has been widely studied for vision tasks. In the visual cortex of the brain, neuronal responses to distinct stimulus classes are organized into geometric structures known as neural manifolds. Accurate classification of stimuli can be achieved by effectively separating these manifolds, akin to solving a packing problem. We introduce Contrastive Learning As Manifold Packing (CLAMP), a self-supervised framework that recasts representation learning as a manifold packing problem. CLAMP introduces a loss function inspired by the potential energy of short-range repulsive particle systems,
such as those encountered in the physics of simple liquids and jammed packings. In this framework, each class consists of sub-manifolds embedding multiple augmented views of a single image. The sizes and positions of the sub-manifolds are dynamically optimized by following the gradient of a packing loss. This approach yields interpretable dynamics in the embedding space that parallel jamming physics,
and introduces geometrically meaningful hyperparameters within the loss function. Under the standard linear evaluation protocol, which freezes the backbone and trains only a linear classifier, CLAMP achieves competitive performance with state-of-the-art self-supervised models. Furthermore, our analysis reveals that neural manifolds corresponding to different categories emerge naturally and are
effectively separated in the learned representation space, highlighting the potential of CLAMP to bridge insights from physics, neural science, and machine learning.

Contrastive Self-Supervised Learning As Neural Manifold Packing
Guanming Zhang 1.2
David J. Heeger
Stefano Martiniani 1,44555
Center for Soft Matter Research, Department of Physics, New York University Center for Neural Science, New York University
Department of Psychology, New York University
Simons Center for Computational Physical Chemistry, New York University
Courant Institute of Mathematical Sciences, New York University
fgz2241, david. heeger, sm7683)/8nyu. edu
Abstract
Contrastive self-supervised learning based on point-wise comparisons has been widely studied for vision tasks. In the visual cortex of the brain, neuronal responses to distinct stimulus classes are organized into geometric structures known as neural manifolds. Accurate classification of stimuli can be achieved by effectively separat- ing these manifolds, akin to solving a packing problem. We introduce Contrastive Learning As Manifold Packing (CLAMP), self-supervised framework that recasts representation learning manifold packing problem. CLAMP introduces loss function inspired by the potential energy of short-range repulsive particle systems, such as those encountered in the physics of simple liquids and jammed packings. this framework, each class consists of sub-manifolds embedding multiple aug- mented views of single image. The sizes and positions of the sub-manifolds are dynamically optimized by following the gradient of packing loss. This approach yields interpretable dynamics in the embedding space that parallel jamming physics, and introduces geometrically meaningful hyperparameters within the loss function. Under the standard linear evaluation protocol, which freezes the backbone and trains only a linear classifier, CLAMP achieves competitive performance with state-of- the-art self-supervised models. Furthermore, analysis reveals that neural manifolds corresponding to different categories emerge naturally and are effectively separated in the learned representation space, highlighting the potential of CLAMP to bridge insights from physics, neural science, and machine learning.
1 Introduction
Learning image representations that are both robust and broadly transferable remains major chal- lenge in computer vision, with critical implications for the efficiency and accuracy of training across downstream tasks. State-of-the- aht models address this problem by employing contrastive treated vectors and optimized using
self-supervised learning (SSL), wherein embeddings pairwise losses to draw together positive samples, augmented views of the same image, and push apart negative samples, views of different images | cantly, and in some cases, surpassed supervised methods 7 g its geometric foundations remain largely underexplored. In visual cortex, neural activity often resides within geometric structures known as neural manifolds which span the embedding space (9) and may exhibit low intrinsic dimensionality [TO] that capture much of the observed variability in neural activity. Neural manifolds have been characterized in number of neural systems supporting a variety of cognitive processes
2
3 4 5
6 While SSL has advanced signifi-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(in addition to vision) including neural activity coordination, motor control, and information coding I 12 T3 14] the context of SSL, neural representations of similar and dissimilar samples are frequently conceptualized as distinct sub-manifolds in the representation space, where sub-manifold corresponds the encoding of single data sample and its transformations, and a manifold to an entire class. In classification, the objective is to separate these class-level manifolds so that each maps cleanly to a distinct category under a readout function. When manifolds corresponding to different classes, for example, manifolds representing "cats" and "dogs" overlap in neural state space, the resulting ambiguity can lead to misclassification. This viewpoint recasts classification as the problem of efficiently arranging manifolds in representation space to minimize overlap and enhance separability [[5].
Our main contributions are as follows:
We propose a loss function inspired by the interaction energy of short-range repulsive particle systems such simple liquids and jammed packings, endowing each hyperparameter with clear physical interpretation.
We demonstrate that CLAMP achieves state-of-the-ari image classification accuracy, match- ing state-of-the-an SSL methods and even setting
SOTA ImageNet- 100, under the standard linear evaluation protocol (a frozen backbone plus a linear readout), all while being remarkably simple and interpretable thanks to its single projection head and one-term, physics-grounded loss comprising single term.
We show that the dynamics of SSL pretraining under CLAMP resemble those of the corresponding interacting particle systems, revealing connection between SSL and non- equilibrium physics.
We compare CLAMP's internal representations to mammalian visual cortex recordings and find that CLAMP offers compelling account of how complex visual features may self-organize in the cortex without explicit supervision.
2 Related work
2.1 Interacting particle models in physics
particularly insightful analogy for CLAMP comes from random organization models, originally introduced to describe the dynamics of driven colloidal suspensions [16] 17 [8]. In these models, overlapping particles are randomly "kicked" while isolated particles remain static. As the volume fraction is increased, the system undergoes phase transition from an absorbing state (where no particles overlap) to an active steady state (where overlaps cannot be resolved under the prescribed dynamics) in two and three dimensions [19 When kicks between overlapping particle pairs are reciprocal, meaning that each particle is displaced by an equal but opposite amount, the critical and active states are characterized by anomalously suppressed density fluctuations at large scales, property known as "hyperuniformity ([00 or "blue noise" 221 22). This absorbing-to-Bae to-active transition coincides with random close packing (RCP) when the noisy dynamics are reciprocal [17]. Recent work further establishes an equivalence between this process and a stochastic update scheme that minimizes repulsive energy under multiplicative noise [1]. We note that even beyond the hyperuniformity observed for dense active states of random organization systems, for a high-dimensional system of short-range repulsive particles, such as those interacting via exponential potential (c -or >> 0), confined to a closed surface, a uniform spatial distribution minimizes the system's 's energy in the large density limit 23 24 25). Drawing on these insights, CLAMP employs a short-range repulsive potential over augmentation sub-manifolds as its self-supervised learning loss, favoring the separation of embeddings.
2.2 Feature-level contrastive learning
Following recent SOTA self supervieed learning models, we use image augmentations and contrastive losses. Most self-supervised learning frameworks treat features as points or vectors the embedding space, contrasting positive Samplle--emeeddingg of augmented views of the same image with
negative samples. They typically use pairwise contrastive loss such as InfoNCE 2 K 6. or
nnighted-sample InfoNCE loss where negative samples are drawn from tailored Boltzmann-like like

---

distributions to sharpen sample-wise contrasts [26]. Here, CLAMP treats each image as its own class; however, instead of applying the contrastive loss directly to individual feature vectors, it operates on the entire augmentation sub-manifolds surrounding each input embedding.
2.3 Clustering methods for self-supervised learning
Clustering-basee self-supervised learning methods assign images to clusters based on similarity criteria, rather than treating each image as distinct class. For instance, The DeepCluster 27 applies the k-means algorithm to assign pscudo-labels to embeddings, which serve as training targets in subsequent epochs. Collapse-resisant non-contrastive methods prevent feature collapse by using fixed dictionary and enforcing augmentation invariance without negative samples (28]. However, the DeepCluster approach requires k k-means classification on the entire dataset, making it unsuitable for online, batih, learning. SwAV [29] overcomes this limitation by using soft assignments and jointly learning cluster prototypes and embeddings, enabling flexible online updates. contrast to clustering methods, our approach groups each image with its augmentations to learn the geometry of the augmentation sub-manifold without clustering across different samples.
2.4 Alignment-nnfformit and maximum manifold capacity methods
The Alignment- Uniimrtmt-nnffomityy method 30 constructs the loss function by incorporating two components: an alignment term and uniformity term. The alignment term maximizes the cosine similarity of the embedding vectors between positive pairs, thereby encouraging consistency. The uniformity term acts as repulsive pairwise potential over all pairs of embeddings, driving the representations to be uniformly distributed on the hyper-sphere and mitigating representational collapse (1
Maximum manifold capacity theory measures the linear binary classification capacity of manifolds with random binary labels. For P d-dimensional manifolds embedded in D dimensions, ae measures the maximum capacity such that for all P/D < ae, these manifolds can be linearly classified into two categories with probability in the limit where P + oc, D - co, d/ D/D- - 0, and P/ D remains Maximum manifold capacity representation (MMCR) maximizes this capacity on augmentation sub-manifolds in the framework of multi-view self- suff-super learning and achieves high accuracy classification performance [34..
finite 32 33
Unlike MMCR, which expressly maximizes the representation's 's binary lineur classification capacity, CLAMP optimizes the embeddings for an n-ary nonlinear classification problem, making the two approaches fundamentally different. Moreover, CLAMP's loss function is based on short-range interactions: the contribution from each augmentation uub-manifold depends only on its neighboring sub-manifolds, whereas MMCR requires global information from all sub-manifolds. Finally, although MMCR has comparable computational complexity to CLAMP, MMCR is less efficient to train, because it requires one SVD for each learning step, compared to CLAMP, and its loss function and optimization dynamics lack the clear, physics-basee based interpretability that CLAMP provides.
Compared to the cesimmenn--niffrmiie method, our approach differs in three key aspects: (1) The Uniifrttttniffoiity loss is defined over feature vectors, whereas our loss operates directly on manifolds; (2) Our method employs single loss term that simultaneously enforces alignment and uniformity, avoiding the need to manually balance separate terms as required Alignment- Uniformity; (3) Our pairwise repulsion is short-ranged, involving only neighboring embeddings, which reduces computational complexity compared to the global (long-ranged) repulsion used in Alignment-Unifon
To summarize, both the MMCR and Alignment- Unnirrrttt methods enhance mutual information between positive pairs by pulling together augmentations of the same image (positive samples) and pushing apart those of different images (negative samples) 35 30 In CLAMP, these properties emerge naturally from the short-range repulsive dynamics described in Sec. without relying on global information or separate alignment and uniformity terms that need to balanced, making CLAMP particularly clegant, efficient, and interpretable solution.
3

---

3 Method
3.1 Deseription
Inspired by physical packing problems and recent advances self-supervised learning, CLAMP learns structured representations by minimizing energy- loss that encourages optimal packing of neural manifolds in embedding space. Our framework is designed specifically for vision tasks that use images input. As shown in Fig. D given batch of images, denoted 22.............................. .22,,..,2,,
where b represents the batch size, we adopt the standard contrastive learning setup, in which positive samples are generated through image augmentations. For each input image I;, we apply random transformations to obtain m augmented views, denoted as xz. 22.
To establish precise terminology, define augmentation sub-manifold as the set of embeddings produced from a finite number of augmentations of single image. In the limit infinite augmenta- tions, this becomes sub-manifold 'he union of sub-manifoles corresponding to all images in given class defines a class manifold, or simply a neural manifold.
We employ backbone encoder (a) as the feature extractor to get the rep- resentations, vf fo(=;), ). To map these representations into the embed- ding space, apply multilayer perceptron (MLP) projection head, =t go (v("). ). Normalizing embed- ding vectors on the unit hypersphere has been shown be effective in self- supervised learning and variational au-
batch of images
pocking
((-)
1
bock
encoder
projector
each
toencoders 34 30 36) Therefore,
Figure CLAMP architecture. The CLAMP framework pro- cesses a batch of b input images by applying augmentations to generate wn views for each image. These augmented views are then encoded and projected into shared embedding space. Within this space, the augmented embeddings corresponding to cach input form distinct sub manifold, resulting b such sub-manifolds. Then, pairwise packing loss is applied to minimize overlap between these sub-manifolds. The gradient of the loss is subsequently backpropa- gated to optimize the model.
we normalize the embeddings to lie on the unit hypersphere by subtract- ing the global center and projecting to unit norm, ズト (2) cll2. where global mean of all the projected vec-
c)/
mb Σ En z is the
tors in the batch.
For each image 24, its m augmentations form a cluster of projected vectors that we refer as an augmentation sub-manifold, 22, ..!! centered at Z, Σx = We approximate the shape each augmentation sub-manifold with a high-dimensional ellipsoid, as described next in Sec. 3.1.I This procedure yields b ellipsoidal sub-manifolde one per image. We treat sub- manifolds from different images repelling entities and define the loss as L log Coverlap) where Lavertap = Ei, E(Z,, Zj) penalizes overlap between sub-manifolds. E(Z;, Zÿ) is an energy-based pairwise repulsive potential that pushes ellipsoids and j away from one another if they overlap,
(1- -ل-ي,--ة = 2 if 72 751l2 <:: <
where r, = ToV Tr(A.)
Loverlay
(I)
0. otherwise
Note that even though the loss function acts locally on neighboring uub-manifolds in the embedding space, each weight update affects all embeddings. Since the logarithm is monotonic function, applying it to Covettap preserves the locations of its minima while compressing large loss values, stabilizing the training process.
Following the convention established 2, we refer to the output space of the encoder fo(-) as the representation space, while the output space of the projection head go (fo(-)) is designated as the embedding space.
3.1.1 Approximate augmentation sub-manifolds as ellipsoids
We approximate the distribution of embedding vectors p(2:) generated from random augmentations
of input image as a multivariate Gaussian with mean 2; and covariance A,, ž, (N2,, A:). We assume that A, is full-rank and invertible (see Appendix K for a treatment of the general case). To represent the augmentation sub-manifold in embedding space, define the corresponding enclosing

---

ellipsoid with Mahalanobis distance
(3- 7 7,)² =
where ya is scaling hyperparameter that controls the effective size of the manifold. A large 7s corresponds to larger ellipsoids, and an appropriate choice rs ensures that the ellipsoid encloses the majority of the augmentation sub-manifold (Fig. a)). The lengths of the ellipsoid's semi-axes
are given by the square roots of the nonzero eigenvalues of A, scaled by r& To r. A V 1,20
(k) is the k-th eigenvalue of A; To estimate the effective size of the ellipsoid, we compute where X
the square root of the expected squared semi-axial length, v ([((raV))) yielding the sub-manifold
radius, Vi = V/ Σ, X
The radius r; provides an upper bound on the volume of the ellipsoid i
Pank[A.]
(V)), 2 eV.. mi/((, (see Appendix K for details). In practice, the number of augmentations m O(10) is much smaller than the embedding dimension D O(100), making the empirical covariance A; lower rank or singular. In such cases, the expression for r; remains valid with ra,)(A,) serving as a proxy (see Appendix for discussion).
3.1.2 Pairwise potential enhances similarity between positive samples and prevents embeddings from collapsing
CLAMP models sub-manifoll repulsive particles interacting via short-range potential E E(Z,, Z,). where Zí Zj are the centroids of sub-ma mob--anffold and j. The potential attains its maximum at ||4 Z, 2/ 0) corresponding complete overlap between two nub-manifolds and drops to Zj1l2 > r; + rj, ensuring that non-overlapping sub-manifolds do not interact. Eq.
when 2 implies that increasing the inter-center distance between sub-manifolds and reducing their sizes leads to lower loss.
Similarity. Similarity requires that augmentations of the same image (positive samples) be mapped to nearby points in the feature space, thereby making them robust to irrelevant variations. CLAMP allows the manifold sizes to vary dynamically, unlike fixed-size muttice-bbsed models in physics. This flexibility is crucial in self-supervised representation learning, where small augmentation sub- manifold sizes r, are desirable to enhance similarity across positive samples.
Avoiding representational collapse. Representational collapse refers to a degenerate solution in which all network outputs converge to constant vector, regardless of the input [11. In CLAMP; the repulsive term in the loss function avoids such collapse by explicitly pushing sub-manifolds of different images apart.
Separability. Separability encourages the negative embeddings to be distant and separable on the unit hypersphere. Analogous repulsive interactions in physical systems are known to produce mear-uniform distributions. Consistent with this, we observe that the embeddings progressively become more separuble throughout training. To quantify this effect, we track both similarity (distance among positives) and separability (distance among negatives) throughout pretraining, confirming that CLAMP systematically enhances class-level separation.
(a) augmentation
(b)
=10
T. 3.0
embedding space
Figure 2: Sub-manifold and visualization of the embedding space. (a) Schematic for approximating aug- mentation sub-manifolds ellipsoids with different scale factor r.. (b) We selected 10 images from the MNIST dataset, one for each digit from 0 and applied Gaussian noise augmentation. These augmented images were then encoded into a 3-dimensional emhedding space for visualization. Solid dots represent the embedding points of each augmented view, while the shaded regions denote circumserihed cllipsoids defined by (8 Z:)(A.)-1( 7x) = 2 Left: the initial embeddings. Right: the trained embeddings. For this toy example,
3.0
5

---

To visualize CLAMP and provide proof of concept, we developed a toy model using an 18- layer ResNet as the backbone, fa. and an MLP, 9o. that maps the resulting representation to three- dimensional embedding space. We loaded one image per class from the MNIST dateset [37 and applied Gaussian noise to generate 60 augmented views per image, and set T, 3.0. Fig 2 b) shows the manifold repulsion and size shrinkage dynamics of CLAMP.
3.2
Implementation details
The code for CLAMP is available at http::// http:////ggguuuu.oooomaggaaaiii com/guanming- -aaigg/haggg/aa aaaan/ccampp clamp
3.2.1 Image augmentation
During data loading, each image was transformed into augmented views using asymmetric augmentation consisting of two augmentation pipelines each sampled from different transformation distributions and applied to one half of the views. We used the following augmentations: random cropping, resizing. random horizontal flipping, random color jittering, random grayscale conversion, random Gaussian blurring (see Appendix B for details).
3.2.2 Architecture
We used a ResNet network, fo(#), as the backbone and a MLP, (s(x), as the projection head. Following [I] and (2, the projection head MLP was discarded after pretraining.
CIFAR10 For the CIFARIO dataset [38). we used the ResNet-18 [3] network as the backbone and two-layer MIP of size [2048,128] the projection head. Following
2. we removed the max pooling layer and modified the first convolution layer to be kernel_ (tazz=,,sstriee1,,aadaa
ImageNet-100/IK For the ImageNet dataset [40], we used the ResNet-50 [39 network as the backbone and a three-layer MLP of size 199,511 as the projection head.
3.2.3 Optimization
We used the LARS optimizer 41 for training the network. For CIFAR10, we trained the model for 1000 epochs using the warmup-decay learning rate scheduler, and with 10 warmup steps and a cosine decay for rest of the steps. For ImageNet- 100, trained the model for 200 epochs with 10 steps of warmup and 190 steps of cosine decay for the learning rate. For ImageNet-IK: we trained the model for 100 epochs with 10 steps of warmup and 90 steps of cosine decay for the learning rate. Training on 8 A100 GPUs with batch size 1024 for views with distributed data parallelization for 100 epochs took approximately 17 hours. See Appendix for details.
Evaluation
4.1 Linear evaluation
Following standard linear evaluation protocols, froze the pretrained ResNet-50 backhone encoder and trained linear classifier on top of the representation. Training was conducted for 100 epochs on ImageNet-IK and 200 epochs on ImageNet-100. Classification accuracies are reported on the corresponding validation sets (Table []. We find that CLAMP achieves competitive performance relative existing SSL algorithms and even sets new state of the on ImageNet- model performance under linear evaluation depends only weakly on batch size but improves with an increased number of augmented views as shown Appendix [ See Appendix [ for details about linear evaluation.
Our
4.2 Semi-supervised learning
We evaluated the model using semi-supervised setup with 1 1% and 10% split (the same as II] of the ImageNet-1K IK training dataset. During semi-supervised learning, both the backbone encoder and the appended linear classifier were updated for 20 epochs. The results shown in Table [ Again,

---

Linear evaluation ImageNet-
Semi-supervised
Method
ImageNet- -K
1%
10%
SimCLRD
79.64
66.5
42.6
61.6
SwAV 29
72.1
49.8
66.9
Barlow Twins 42
80.38*
68.7
45.1
61.7
BOLL
80.32
69.3
49.8
65.0
VICReg 6
79,4
68.7
44.75
62.16
CorInfoMax
80,48
69.08
44.89
64.36
MoCo-V213
79,28*
67.4
43.4
63.2
SimSiam
81.6
68.1
A&U [30]
74.6
67.69
MMCR (4 vicws+ME)
82.88
71.5
49.4
66.0
CLAMP (4 views)
85.12 +505
69.50 + 14
47.38
+.56
65.10
+.30
CLAMP (8 views)
85.10 1.15
70.04 1.16
47.87 #.03
65.96 +.04
Table I:
Linear evaluation accuracy on ImageNet-100 dataset for 200-epoch pretraining: For SimCLR the higher accuracy between 34 and [43] is reported. For Barlow twins, BYOL, and MoCo-V2, the accuracies using Re as the backbone (labeled with #) in 4 are reported because either ResNet-50 results are absent respectively. Linear evaluation accuracy on ImageNet-1k dataset for 100-epoch pretraining: For SimCLR, SWAV, Barlow Twins, BYOL, VICReg, MarlufoMax, MoCo-V V2 and SimSiam, we reported the highest accuracy for each method among (34] and [43]. We report the 200-epoch pretraining result for A&U, as the 100-epoch result is unavailable 30 Semi-supervised learning: The sup-suuer learning results for VICReg and CarInfoMax
ResNet-18 performs better in 4 A&U and MMCR results taken from 3D and 34
the results in the solo-learn library [44]. VISSL library [45
taken from 4 those for other methods are from [34]. In MMCR with 4 views and a momentum encoder (4 views ME), the effective 8 views, from backbonc and 4 from the encoder, achieve the same linear accuracy (71.5% as using views without the momentum encoder 34
we find that CLAMP achieves competitive performance relative to existing SSL algorithms. See Appendix G for details.
4.3 Transfer to object detection tasks
To evaluate CLAMP's generalizability beyond linear classification, we apply the pretrained model object detection. Specifically, we report results using Faster R-CNN architecture with a C4 backbone, pretrained with CLAMP on ImageNet- IK using 8 views and batch size of 512 for 100 epochs. The model is fine- tuned on the VOC2007+2012 training set and eval- uated on the VOC2007 test (Table. 2 CLAMP delivers stronger detection performance than other self-supervised learning baselines, demonstrating that its learned manifold structure transfers effectively to spatially structured vision tasks. See Appendix for
Methods
mAP 1
AP50
AP75 1
SimCLR
54.4
81.6
61.0
Barlow Twins
53.1
80.9
57.7
BYOL
55.6
82.3
62.0
MoCo
54.7
81.7
60.2
MMCR
54.6
81.9
60.0
CLAMP
55.7
82.3
62.4
Table 2: Object detection on VOC datasets:
The benchmarks (mAP, AP50 and AP75) for the baseline methods are from (34). Models pre- trained on ImageNet-IK for epuchs before fine-tuning.
details.
5 Training dynamics reflect geometrical changes in the embedding space
We analyze manifold packing dynamics by tracking the evolution of neighbor count and manifold size during ImageNet-IK training. Two sub-manifolde distance between their centroids is smaller than the sum of their radii (lqq: 93 2 < + ry). At each epoch, we compute the average number of neighbors (for m views) and the average manifold size, BEiddatlVAA//aa using a 1% validation split. As shown in Fig. 3 both metrics decrease over training, indicating that embeddings evolve from an initial collapsed state toward more structured representations with increased pairwise distances. This mirrors random organization models [5), where local density and spacing show similar temporal behavior, reflecting improved feature discrimination over time.
considered neighbors the Euclidean
7

---

(a)
number of neighbouf
1000
wwwm
sub-manifold size
(b)
0.3
(c)
untrained
trained
800
probability
0.2
600
w
400
0.1
50
100
50 epoch
100
8.0
0.5
1.0
1.5
epoch
centroid distances
Figure Training dynamics: (a) Number neighbours as a function epochs. sub-mun sub-manifold sizes as the function of epochs. (c) Distances between pair of embeddings for untrained and trained networks.
(b) Average embedding
10.
alsss inter-cless
0.20
Intra-class inter-class
inter-class
probabiity
109
10-7
10-3
0.2
0.4
manifald orientations similarities
sub-manifold centroids
sub-manifold centroids similiarties
Figure 4: The properties of sub-manifolds in the embedding space for the pretrained ResNet-18 network are characterized by: (a) Orientation similarity: the squared cosine similarity between the principal orientations of the sub-manifolds. (h) Centroid distances: the Fuclidean distances between the centroids of different sub- manifolds. (c) Centroid similarity: the cosine similarity between the centroid points of sub-manifolds.
Properties of the representations
To quantify the geometric properties of the learned rep- resentations, we analyzed the CLAMP representation ob- tained for CIFAR-10 using ResNet-18 18 backbone archi- tecture. As shown in Appendix F we achieve 90% top-1 accuracy (without hyperparameter tuning) which is competitive with alternative SSL frameworks. For each sub-manifold, we computed its centroid and alignment vector (defined as the principal eigenvector of its covari- ance matrix), grouping them according to their class labels. We compared the geometric propertics such as centroid distances, centroid cosine similarity, and alignment cosine similarity of representations across inter-class and intra- class sub-manifoll pairs. To ensure robust estimation, randomly sampled 800 images from the test dataset and generated 100 augmentations per image compute the manifold properties. This process was repeated 10 times.
Figure 5: t-SNE visualization of the rep- resentations. Visualization of the 256- dimensional representation space by t-SNE method. Each color shows the representa- tion corresponding to different category in CIFAR-10 dataset.
As shown in Fig. intra-class and inter-class sub- manifolds exhibit clear differences across all metrics, in- dicating that manifolds of different classes both position and orientation, reinforcing the potential for linear separability. Notably, the inter-class alignment similarity peaks strongly at 0, suggesting that sub-manifolds of different classes are almost orthogonal to one another.
distinct in
To demonstrate the emergence of category speccfcc manifolds, we visualize the representation space of the CIFAR- 10 test dataset (without augmentation) using t-SNE [46], method that preserves local neighborhood structure while projecting high-dimensional data into lower-dimensional space. As shown Fig. representations from different classes form distinct, well-separated clusters, indicat- ing the formation of class-specific manifolds in the learned representation space. This clear clustering underscores the structured organization induced by CLAMP and demonstrates its effectiveness in producing representation that are separable in representation space.
Biological implications

---

Methods
V1 1
V2 f
V4 1
IT
SimCLR
0.224
0.288
0.576
0.552
SwAV
0,252
0.296
0.568
0,533
Barlow Twins
0.276
0.293
0.568
0.545
BYOL
0.274
0.291
0.585
0.55
MMCR (8 views+ME)
0.270
0.311
0.577
0.554
CLAMP (4 views)
0.258 +:00133
0.336 10.017
0.558 +00004
0.570 1:0005
Table 3: Brain-score: We use public data from 47 for V1 and V2, and from
48 for V4 and IT to evaluate the
brain-score 49 50) The henchmarks for all baseline methods are from 34
In the MMCR method, ME refers
the momentum encoder.
Neuroscience has long served as a source of inspiration for advancements in artificial neural networks, prompt- ing the question of whether SSL models can replicate response statistics observed in cortex. Stringer al. [5]] analyzed stimulus-evoked activity mouse VI (primary
eigenvalue
10-1
a= 1.013
visual cortex) and found that the eigenspectrum of its variance matrix follows a power-law decay, x n -a with 1.04, and further proved that > is indeed necessary to ensure differentiability of the neural code. After training CLAMP the eigenspectrum of its feature covariance and observed power law decay À x -1.03 matching the cortical exponent and theoretical predictions [51].
10-2
10-3
10
ImageNet-1
measured
10
10
10a
103
eigenvalue rankings
Figure 6: The cigenspectrum of the covari- ance matrix for the responses of 24000 im- ages randomly selected from the ImageNet- 1K dataset follows power law decay À x n 11003 Our model pretrained on ImageNet-IK for 100 epochs. The inset is adapted from the eigenspectrum of the stimulus-evoked activity in mouse VI ported [5T
We then evaluated representational alignment using the Brain-Score benchmark [49 50], which measures how well model activations predict primate neural record- ings via valsdaval linear regression. We found that CLAMP achieves the highest Brain-Score in V2 and IT compared to the selected self-supervised models. These results suggest that CLAMP 's representation geometry not only supports downstream classification performance but also offers a compelling account of how complex visual features may self-organize in cortex without explicit supervision.
Discussion
We have introduced CLAMP, a novel self-supervised learning framework that recasts contrastive learning as ceural-manifole packing problem, guided by a physics-inspired loss. By promoting similarity of positive pairs and repulsion of negatives through purely short-range, multi-view repul- sive interactions, CLAMP achieves competitive, and in some cases state-of-the-art, performance across standard downstream benchmarks. Importantly, our analysis of training dynamics reveals progressive structuring of the embedding space. mub-manifolds that are initially overlapping evolve into well- separated compact structures. We further show that class-level manifolds emerge from the aggregation of sub-manifolds. which interact purely through repulsion, phenomenon reminiscent of phase separation statistical physics. The mechanism underlying this emergent organization should be elucidated in future theoretical work. It is also interesting to test if hierarchical cluster emerges during the learning process in the future.
We demonstrate that CLAMP offers a compelling account of how complex visual features might self-organize in the cortex without explicit supervision, based on direct comparisons between its learned representations and experimental neural data. However, these findings should be interpreted with caution. Brain-Score's 's linear mapping between deep nets output and experimental activity may overlook non-linear relationships between model and neural representation. It also does not guarantee that CLAMP implements the same computations or dynamics as cortex. Moreover, CLAMP's use of large-batch contrastive optimization with backpropagation is unlikely to reflect biologically realistic learning mechanisms. In contrast, Hebbian-style self-supervised learning frameworks with local update rules are considered biologically plausible [52]. Developing local self seff-supervised learning

---

dynamics inspired by manifold packing, without relying on backpropagation, is an important direction for future research.
Our loss function leverages the sizes and locations of sub-manifolds within the embedding space. Incorporating orientation alignment among sub-manifolds may enable denser packing, potentially improving downstream task performance. However, we currently omit orientation information due to its high computational cost: accurately estimating sub-manifold's orientation requires on the order of O()) ~ 100 points, ~ 100 augmentations per input, which is infeasible at the scale of CIFAR-10 or ImageNet.
Acknowledgments
We thank Shivang Rawat, Mathias Casiulis and Satyam Anand for valuable discussions. This work was supported by National Science Foundation grant IIS-2226387, the National Institutes of Health under award number R01MH137669, the Simons Center for Computational Physical Chemistry, and NYU IT High Performance Computing resources, services, and staff expertise. We also gratefully acknowledge the use of research computing resources provided by the Empire Al Consortium, Inc., with support from the State of New York, the Simons Foundation, and the Secunda Family Foundation.
References
[1] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[2] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 212244 2020.
[3] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738 2020.
[4] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0429: 2020.
[5] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 15750-15758, 2021.
[6] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: ization for self-supervised learning. arXiv preprint arXiv:2105.04906 2021.
regular-
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[8] Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05/19 2022.
[9] John P Cunningham and Byron M Yu. Dimensionality reduction for large-scale neural record- ings. Nature neuroscience, 17 15:0-5001 2014.
[10] Mehrdad Jazayeri and Srdjan Ostojic. Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity. Current opinion in neurobiology, 10:113-120, 2021.
[11] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Charu Bai Reddy, Matteo Carandini, and Kenneth Harris. Spontaneous behaviors drive multidimensional, brainwide activity. Science, 364(6437) 2019.
10

---

[12] Juan A Gallego, Matthew G Perich, Lee E Miller, and Sara A Solla. Neural manifolds for the control of movement. Neuron, 94(5):978-99) 984, 2017.
[13] Rishidev Chaudhuri, Berk Gerçek, Biraj Pandey, Adrien Peyrache, and Ila Fiete. The intrinsic attractor manifold and population dynamics of canonical cognitive circuit across waking and sleep. Nature neuroscience, 22(9):1512-1520 2019.
[14] Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. Nature Reviews Neuroscience, 22(11):703-718, 2021.
[15] Guanming Zhang and Stefano Martiniani. Absorbing state dynamics of stochastic gradient descent. arXiv preprint arXiv:2411.11844, 2024.
[16] Laurent Corte, Paul M Chaikin, Jerry P Gollub, and David Pine. Random organization in periodically driven systems. Nature Physics, 4(5):420 424, 2008
[17] Sam Wilken, Rodrigo E Guerra, David Pine, and Paul M Chaikin. Hyperuniform structures formed by shearing colloidal suspensions. Physical Review Letters, (25(14):14800, 2020.
[18] Elsen Tjhung and Ludovic Berthier. Hyperuniform density fluctuations and diverging dy- namic correlations in periodically driven colloidal suspensions. Physical review letters, 114(14):148301, 2015.
[19] Daniel Hexner and Dov Levine. Noise, diffusion, and hyperuniformity. Physical review letters, 118(2)022601, 2017.
[20] Salvatore Torquato. Hyperuniform states of matter. Physics Reports, 745:1-95, 2018.
[21] Mark AZ. Dippé and Erling Henry Wold. Antialiasing through stochastic sampling. In Proceed- ings of the 12th annual conference on Computer graphics and interactive techniques, pages 69-78, 1985.
[22] Hongxing Qin, Yi Chen, Jinlong He, and Baoquan Chen. Wasserstein blue noise sampling. ACM Transactions on Graphics (T0G), 36(5):1-13, 2017.
[23] Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy rectifiable sets, volume Springer, 2019.
[24] András Sütö. Ground state at high density. Communications in mathematical physics, 305:657- 710, 2011.
[25] Cameron Davies, Tongseok Lim, and Robert J McCann. Classifying minimum energy states for interacting particles: regular simplices. Communications Mathematical Physics, 399(2))577- 598, 2023.
[26] Joshua Robinson, Ching-Yao Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.0459) 2020.
[27] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[28] Emanuele Sansone, Tim Lebailly, and Tinne Tuytelaars. Collapse-proof non-contrastive self- supervised learning. In Forty-second International Conference on Machine Learning, 2025.
[29] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 99244, 2020.
[30] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference machine learning. pages 9929999399.....199999999999999.9 PMLR, 2020.
[31] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint 21X0v:2110.09348, 2021.
11

---

[32] YeoYeon& Chung, Daniel D Lee, and Haim Sompolinsky. Linear readout of object manifolds. Physical Review E, (3(6):06030,, 2016.
[33] Yuoo Chung, Daniel D Lee, and Haim Sompolinsky, Classification and geometry of general perceptual manifolds. Physical Review X, 8(3):031003, 2018.
[34] Thomas Yerxa, Yilun Kuang, Eero Simoncelli, and Yeon Chung. Learning efficient coding of natural images with maximum manifold capacity representations. Advances in Neural Information Processing Systems, 36:24103-24128, 2023.
[35] Rylan Schaeffer, Victor Lecomte, Dhruv Bhandarkar Pai, Andres Carranza, Berivan Isik, Alyssa Unell, Mikail Khona, Thomas Yerxa, Yann LeCun, YueYeon Chung, et al. Towards an improved understanding and utilization of maximum manifold capacity representations. arXiv preprint arXiv:2406.094 09366, 2024.
[36] Jiacheng Xu and Greg Durrett. Spherical latent spaces for stable variational autoencoders. arXiv preprint a8Xiv:1808. 10805, 2018.
[37] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmistl, 1998.
[38] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). URL http://www. cs. toronto. edu/kriz/cifar html, 5(4):1, 2010.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[40] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2 2522 2015.
[41] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 111.:07088..8888 2017.
[42] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International conference on machine learning. pages 12310-12320. PMLR, 2021.
[43] Serdar Ozsoy, Shadi Hamdan, Sercan Arik, Deniz Yuret, and Alper Erdogan. Self-supervised learning with information maximization criterion. Advances in Neural Information Process- ing Systems, 04555400-52533 2022.
[44] Victor Guilherme Turrisi Da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):1 2022.
[45] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl.
2021.
[46] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008
[47] Jeremy Freeman, Corey M Ziemba, David Heeger, Eero P Simoncelli, and Anthony Movshon. A functional and perceptual signature of the second visual area in primates. Nature neuroscience, 16(7))974-981, 2013.
[48] Najib J Majaj. Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 13402-11140 2015.
12

---

[49] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott- Roy, Franziska Geiger. et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
[50] Martin Schrimpf, Jonas Kubilius, Michael Lee, N Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking advance neurally mechanistic models of human intelligence. Neuron, 108(3):413-44) 2020.
[51] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Har- ris. High-dimensional geometry 365, 2019.
geomerer of population responses in visual cortex. Nature, 571(7765):361-
[52] Manu Srinath Halvagal and Friedemann Zenke. The combination of hebbian and predictive plasticity learns invariant object representations deep sensory networks. Nature neuroscience, (((11):990-11155 2023.
[53] Alexander Buslaev, Vladimir I Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A Kalinin. Albumentations: fast and flexible image augmentations. Information, 1122)122, 2020.
13

---

A
Algorithm
Algorithm I CLAMP algorithm
Input: batch size b, number of views m, number of optimization steps size scaling factor rs
initialize feature extractor network
initialize projection head De
initialize the distribution random transforms Ti and 72
for t = 1 to n. do
sample batch from the dataset , 22, ... for i to do
# for each image, augment half of the views with T1
,. m/2 = t (x;))..,tt*) (x;)) t t!...tm/2 Ti
#augment the other half of the views with T2
x
m/2++ .... /m/2+1 (or))..."""()
tm/2+11 ".. T2
y
-. V.th
fo(x)) fo..)((x;") # extract features
= (v) ) "..." projection
end for # normalization
mb Σ 1i=1 Em
=k
s/ (2, c)/ll= k=
el12 for 1...b, 1...m
for i = =1 to do
9: "Ex zk # center of each sub-manifold
A, E (2 q:) covariance matrix
raV (r(A)) # manifold size
# Note: in practice only the diagonal elements of A; are calculated for Tr(A:).
end for calculate the loss function
lla:-2. 2,11222 )a if llq. - -,ll2 < r. +
Σ
Loveriap
0.
otherwise
= log(Loverlap)
update the network fo, 96 to minimize C end return fo and discard go
14

---

B Image augmentation
During self-supervised training. CLAMP employs collection of image augmentations, drawn from the broader set introduced in [ and [2]. We adopt the following image augmentations: randomly sized cropping, Gaussian blur, random gray-scale conversion(color dropping), random color jitter,randolly horizontal flipping.
parameter
value (Ti T2)
random sized cropping output size
32 32
random sized cropping scale
[0.08,10]]
Gaussian blur probability
Gaussian blur kernel size
3 3
color drop probability
0,2
color jitter probability
0.8
color jitter- brightness adjustment max intensity
0.8
color jitter contrast adjustment max intensity color jitter - saturation adjustment max intensity color jitter hue adjustment max intensity horizontal flipping probability
0.8
0.2
0.5
Table 4; Image augmentation parameters for CIFARIO
parameter
Ti
T2
randomly sized cropping output size
224 x 224
x 224
randomly sized cropping scale
10.08, 1.0]
[0.08,1.0]
Gaussian blur probability
0.8
0.8
Gaussian blur kernel size
23 23
23 23
color drop - probability
color jitter probabiiity
0.8
0.8
color jitter brightness adjustment max intensity
0.8
0.8
color jitter contrast adjustment max intensity color jitter saturation adjustment max intensity
0.8
0.8
color jitter huc adjustment max intensity horizontal flipping probability solarization prohability
0.1
0.5
0.5
0.0
0.2
Table 5: Image augmentation parameters for ImageNet- 1K/100
As the last step of the augmentation process, we apply channel-wise normalization using dataset- specific statistics. Each image channel is standardized by subtracting the dataset mean and dividing by the standard deviation. For CIFAR-10, 10, we use (0.4914, 0.4822, 0.4465) as the mean values and 01.247,0.243.0 243)0.261) for the standard deviation. For ImageNet- 100 and ImageNet- 1K, the normal- ization parameters are (0.485, 0,456, 0.406) for the mean and (0.229, 0.224, 0.225) for the standard deviation. During resizing, employ bicubic interpolation preserve image quality. We use Albumentaions library for fast image augmentation 53
C Time complexity for the loss function
Note that only the diagonal part of the covariance matrix A; is required in the loss function, it is not necessary to calculate the complete covariance matrix. Therefore, each iteration, the time complexity for computing sub-manifold sizes is O(bmD), the time complexity for the pairwise loss function O(b² D), therefore the time complexity for each iteration O(bl) max (m, b)). Since the number of iteration is N/b where N is the size of the dataset, the complexity for calculating the loss function for one epoch is O(ND max(m, b)).
D Self-supervised pretraining
We employ the LARS optimizer for self-supervised learning with its default trust coefficient (0.001). Our learning-rate schedule begins with 10-step linear warm-up, after which apply cosine decay.
15

---

To enable fully parallel training, we convert all BatchNorm layers to synchronized BatchNorm and aggregate embeddings from all GPUs when computing the loss. addition, we exclude every bias term and all BatchNorm parameters from weight decay and use weight decay parameter = 10-6, For CIFAR-10 dataset, we removed the max pooling layer and modified the first convolution layer to be kernel boyr==1,strrdee .padding-2. We use = 16 views for CIFAR-10 10 dataset and m = 4, 8 for BongeNett-K/IO0e re = 8.5 is applied for CIFAR-I0 and ImageNet- 100//K. Other parameters are shown in Table 6 where the base learning rate = Ir x batch size/256.
dataset
backbone
MLP head
batch size
momentum parameter
epochs
CIFAR-10
ResNet-18
[2048,256] [8192.8192.512]
128
2.0
0.9
1000
ImageNet-100
ResNet-50
512
2.0
0.9
200
ImageNet-IK
ResNet-50
[8192.8192.512]
512
0.9
100
Table 6: supervidd pretraining setups for CIFAR- 10 and ImageNet-IK/100
E Linear evaluation
We discard the projection head and freeze the backbone network during linear evaluation.
CIFAR10 dataset: We use the Adam optimizer with learning rate 0.05 and batch size/ 256.0 (batch size 1024) to train the linear classifier for 100 epochs. We apply cosine learning rate schedule.
ImageNet-100 100 dataset: We use SGD optimizer with Nesterov momentum and weight decay to train the linear classifier where the learning rate 0.05x batch size / 256, batch size 1024, momentum =0.9 and weight decay le-5. During training, we transform an input image by random cropping. resizing 224 224, and flipping the image horizontally with probability 0.5. At test time we resize the image to 256 x 256 and center-crop it to size of 224 x 224. We apply a cosine learning rate schedule train the linear classifier for 100 epochs.
ImageNet-1K dataset:we use the same setup as ImageNet- 100 but base learning rate = 1.6x batch size / 256, batch size 1024.
F Linear evaluation on CIFAR10
Method
Top1 accuracy
Top5 accuracy
SimCLR
90.74
99.75
SwAV
89.17
99,68
DeepCluster
88.85
99.58
Barlow Twins
92.10
99.73
BYOL
92.58
99.79
CLAMP (ours)
90.21
99.60
Table 7: Linear evaluation on CIFARIO dataset using resmet-18 18 as the backbone encoder. Results for Sim-
CLR [I], SwAV (29), DeepCluster V2 (2). Barlow twins 42 and BYOL (2) methods are taken from solo-learn
benchmarking 44
Note that we adopt the same learning rate used for ImageNet-100 and do not perform further hyperparameter tuning.
G Semi-supervised learning
We use semi-s cemi-supervised learning via fine tuning on the backbone network with a linear classifier for 1 and 10% splits of the ImageNet- 1K dataset. The same splits are applied as [ We use the SGD optimizer with momentum of 0.9 but no weight decay for cross entropy loss. We adopt the same data augmentation procedure during training and testing as in the linear evaluation protocol. We scale down the learning rate of the backbone parameters by factor of 20. Both the backbone and classifier use the cosine-decay learning rate schedule. We use batch size of 256 and sweep the initial learning rate over (0.1, 0.2, 0.4, 0.6).
16

---

H Object detection
apply the Faster R-CNN detector with R50-C4 backbone (pretrained on ImageNet-Il I with views and batch size 512 for 100 epochs), fine-tuned on VOC2007+2012 training dataset and tested on VOC2007 test dataset. Our implementation follows the procedure of [3] exceept that set the base learning rate to 0.07.
Power law exponent for the eigen-spectrum
In analysis of experimental data in [51], the exponent is estimated using log-log scale liner fit on cigenvalues ranked from 11 to 500. Here, we use eigenvalues ranked from 15 to 1400 to fit the exponent.
J Brain-Score
We use the datasets with identifiers, PreemannZemmb22033pbbicccc V1-pls, FreemanZZemma2213ppuee V2-pls, MajajHong220 Spublicc, (4-bll,.V--lls, V4-pls, http:///onnn0Suuliccc.c.. for the neural activity recorded from V1,V2, and respectively.
Qualitative understanding of sub-manifolds in the embedding space K
K.1 Manifold radius for singular A4
As A is positive semi-definite, it can be decomposed as
(1) G)T
A
with (x > where K = rank(A) (A,) and ()) are orthonormal vectors. We then enclose the sub-manifoll as the ellipsoid whose equation reads,
I(= 2,) Tr) 1 [ - 2,) x (1)
[ - 2,) (K) 04 ,r2,, (K) 1
r²2
This equation represents the K-dimensional ellipsoid centered Z, embedded the D-dimensional
space with semi-axial lengths r, / s/A"".... (K)) We compute the sub-manifold radius as the square root of average semi-axial length square,
r2 +... + + (K)
K
Even if A; is not full-rank matrix, Tr(A,) = x!1) +... +
(K) is well defined. We get
Tr(A,)
7= -= K
A is the covariance matrix of m embedding vectors embedded in D-dimensional space, There- fore rank(A,) S min(m, D) m (in practice, the number of views is much smaller than the dimensionality. & D). Here, we use m, the upper bond, to estimate rank(A;).
K.2 The relation between estimated sub-manifold radius and the estimated sub-manifold volume
We enclose the sub-manifold by K (K rank(A,)) dimensional ellipsoid embedded in the D-dimensional space with semi-axial lengths r ((1)) ...Ts (K), Its volume reads
K/2
T 1
r(K + 1)
17

---

From the inequality of arithmetic and geometric means, we have
K
K/2
V
+1)
K
By Souchy-Schward inequality
K/2
1
Tr(A,)
T K/2
π xK/2
V
T(
+1)
K
To
K
+1)
r( K
K
Therefore r; is related to the upper-bound of the estimated sub-manifold volume.
L Hyperparameters
We conducted an ablation study investigate the role of different hyperparameters, Due to limited computational resources, ablation study on parameter r, and number of views m, 30% of the training data from ImageNet-IK were selected and pretrained for 100 epochs (which is consistent with our baseline where the whole training set pretrained for 100 epochs). For the ablation batch size, we employed the entire ImageNet-Ile IK dataset. We used the frozen-backbone linear evaluation accuracy the entire test dataset as the criterion for ablation.
L.1 Scale parameter Tss
As shown in Fig. / accuracy increases with until it reaches plateau at approximately r, < 6.0. This result further supports our qualitative finding that setting v. > 3 is necessary to compensate for the underestimation of manifold size. A higher r, increases computational cost because more sub-manifold pairs are involved in the loss function and the corresponding optimization process.
0.58
top 1 accuracy
0.56
0.54
0.52
0.50
8
10
rs
Figure Linear evaluation accuracy as the function of the size scale factor r, on 30% ImageNet- KK datasct.
use views.
L.2 Number of augmentations m
We observed that increasing the number of data augmentations leads to higher linear evaluation accu- racy. This indicates that greater augmentation diversity helps the model learn better representations.
18

---

views
4 views
8 views
top 1 accuracy
55.17
58.29
58.95
Table 8: Linear evaluation using different numher of views for ablation study on 30% ImageNet-IK dataset
L.3
Batch sizes
In contrast to SimCLR, where large batch sizes critical for achieving high accuracy, find that the linear evaluation accuracy of CLAMP depends only weakly on batch size.
batch size
256
512
1024
top accuracy
69.02
69.43
69.11
Table 9: Linear evaluation using different batch sizes for views
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims presented in the abstract and introduction substantiated by
our methodology, evaluation results, and supporting analyses, including figures and tables.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitation in the discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
Please see the section method and the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We show the details of the model, parameters, datasets in section implementa- tion details and appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We include our code in the supplementary material and all the datasets we use are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli///uui guwde//ddee/eee Coc//ooodeeesiseeeeeeo wwwpobmmsssioooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have presented the benchmark and implementation details of our algorithms in in section implementation detaily and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the error (two-sigma for three independent experiments with different random seeds) in the table. The error in Brain-Score is provide by the library.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have these details in section implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research conforms in every points with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: discuss broader impacts in our abstract and introduction.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:The paper has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:All the assets properly credited and respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are created in this paper.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
25

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
26


Reviews:
[{"review_id": "vkVnwmFyTB", "reviewer": "Reviewer_AN7f", "summary": "This paper proposes a novel reframing of contrastive self-supervised learning (CSSL) by casting it as a manifold packing problem in representation space. Instead of viewing representation learning as a purely statistical or mutual information maximization task, the authors approach it geometrically: they introduce a new loss function inspired by the short-range repulsive forces in particle physics, encouraging the learned representations to compactly fill space with minimal overlap. The paper offers an elegant and theoretically grounded framework for understanding representation learning through the lens of neural manifold packing. The fact that it achieves the best brain score for V2 and IT suggests a potential computational-level convergence between this framework and the primate visual systems.", "questions": "1. Does manifold packing scale to hierarchical or compositional concepts?\n2. What practical benefits does this new perspective offer for self-supervised learning?  Improvement on the loss function? Does it lead to the design of new objective?", "limitations": "1. While similar geometric structuring of representations (low-dimensional manifolds, separation of class-specific activity) has been observed in neural population codes in primate cortex, it is not certain how this can be implemented using biological mechanisms. Is this primarily a computational theory level description of the principle?\n2. It is not certain how manifold the packing framework scales to hierarchical or compositional concepts, or large dataset.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. A major contribution is the theoretical and empirical demonstration that this formulation leads to the emergence of low-dimensional, well-separated manifolds, each corresponding to different semantic image categories—even in the absence of labels. This structure supports linear decodability and provides a geometric rationale for the effectiveness of self-supervised representations.\n2. Experiments on image datasets (CIFAR-10, STL-10, etc.) confirm that learned representations exhibit this packing behavior. Even without labels, the manifolds align with semantic classes.\n3. One of the most striking results is that the learned representations achieve state-of-the-art brain scores in both area V2 and IT cortex when evaluated against neural data, suggesting a degree of alignment between the structure of artificial representations and those found in the primate visual system. Similar geometric structuring of representations (low-dimensional manifolds, separation of class-specific activity) has been observed in neural population codes in primate cortex. While the biological plausibility of the learning mechanisms remains uncertain, the computational-level similarity (in Marr's terms) offers intriguing evidence that efficient manifold packing may be a shared organizational principle.\n4. The paper brings much-needed theoretical clarity to the success of contrastive learning. It contributes not only a practical improvement in self-supervised learning but also a conceptual bridge between deep learning and neuroscience. It provides a unifying geometric perspective that connects contrastive objectives, biological representations, and manifold theory. \n\nWeaknesses:\n1. While the geometric structure aligns well with cortical representations, the underlying mechanisms—such as particle-inspired repulsion—are not themselves biologically grounded, and more work is needed to connect this theory to known neurophysiological learning rules.\n2. Evaluation: The evaluation presumes alignment between data structure and downstream labels. It's less clear how CLAMP behaves when class structure is ambiguous or hierarchical.\n3. While packing helps linear separability, how does it generalize to unseen concepts or transfer across tasks? \n4. The practical benefit is modest. The experiments are solid but limited to standard benchmarks; it would be valuable to see if the same geometric structure emerges in larger-scale vision-language or multimodal settings.", "was_revised": true, "final_justification": "Thank you for the responses, clarification, and the new results concerning the practical utility and application values. I do think it is an interesting paper. I guess extending to a compositional hierarchical system might be too hard a question for you to answer at this point, and not critical for this paper.  I will keep the current favorable score.", "rebuttal": "We thank the reviewer for the time and efforts on reviewing our work, which have helped improve our manuscript. Below, please find our detailed responses.\n+ \" While the geometric structure...\" :  CLAMP’s repulsive packing loss isn’t directly grounded on a specific synaptic learning rule. However, its main idea, encouraging competition and decorrelation among sub‑manifolds may be analogous to anti‑Hebbian plasticity. Bridging CLAMP’s high‑level objective with a concrete, biologically‑plausible learning model remains a very important direction for future work. We have mentioned this point in Section 8, line 343-346.\n\n+ \"Evaluation: The evaluation ...\" : Thanks for raising this interesting point. Our t‑SNE visualization reveals that categorical labels naturally segregate suggesting an emergent structure in the embedding space. By analogy, we hypothesize that CLAMP’s representations will likewise organize into hierarchical clusters (subcategorical clusters within parent categories). We will discuss this hypothesis and seek supporting evidence for it on ImageNet in the revised version of the manuscript.\n\n\n+ \" While packing helps ...\" : The representation learned by CLAMP demonstrates its ability to distinguish between different categories, indicating strong classification capabilities. This property is beneficial for a variety of downstream tasks. Accordingly, fine-tuning self-supervised models on object detection and segmentation tasks often yields high performance. Here, we present preliminary results on object detection using a Faster R-CNN head with a C4 backbone (pretrained on ImageNet-1K with 4 views for 100 epochs), fine-tuned on VOC2007+2012 training dataset and tested on VOC2007 test dataset. Comparative metrics for other methods are sourced from the MMCR paper (arXiv:2303.03307).\n| VOC2007+12| mAP | AP50 | AP75|\n| --------------- | :-------------: | ---------------:|---------------:|\n| Barlow Twins   | 53.1  | 80.9   | 57.7   |\n| SimCLR           | 54.4  | 81.6   | 61.0   |\n| MoCo v2          | 54.7  | 81.7   | 60.2   |\n| BYOL               | 55.6  | 82.3   | 62.0   |\n| MMCR             | 54.6  | 81.9   | 60.0   |\n| CLAMP            | 55.6  | 82.2   | 61.5   |\n*Higher mAP/ AP50/AP75 shows better performance. \n Overall our results show that CLAMP is competitive with the state of the art. We will further improve object detection by training on the larger COCO dataset and through hyperparameter tuning. We will include the new results in the revised version of the manuscript.\n\n+ \"The practical benefit ... \" : It is interesting to apply this type of self-supervised loss function to other tasks including multimodal tasks. We will mention this possibility in the discussion and reserve a detailed benchmark on these large-scale datasets for future work, commensurably with the available computational resources.  \n\n+ \"What practical benefits ...\" : CLAMP uses a single repulsive potential whose parameters  have clear geometric meaning, making the training dynamics tractable. The manifold‑packing view immediately suggests richer objectives such as adaptive radius schedules analogous to the Lubachevsky-Stilliger packing algorithm. We will include this discussion on the possibility of deriving new physically-motivated objectives in Section 8."}, {"review_id": "UPp4oo3do5", "reviewer": "Reviewer_4tLf", "summary": "The paper presents Contrastive Learning As Manifold Packing (CLAMP), a new self-supervised learning method for visual representations. The key idea is to model the embeddings from augmented views of an image as a \"sub-manifold,\" which is then approximated by a high-dimensional ellipsoid. Learning is driven by a single loss function inspired by short-range repulsive potentials in physical systems. This loss function simultaneously encourages the separation of sub-manifolds from different images (negative pairs) and the contraction of individual sub-manifold volumes (positive pairs). The authors argue this unified approach is elegant, interpretable, and effective at preventing representational collapse. The method is evaluated on CIFAR-10, ImageNet-100, and ImageNet-1K using linear probing and semi-supervised learning protocols , where it demonstrates promising, and in one case, state-of-the-art results.", "questions": "1.\tGiven that the central claim is about learning better-structured representations through geometric packing, wouldn't evaluation on tasks that heavily rely on rich spatial features (e.g., object detection, segmentation) be the most convincing way to validate this? \n2.\tThe paper's claims hinge on the ellipsoid approximation being a valid model for augmentation sub-manifolds. What empirical evidence can the authors provide to show this is a robust choice? For instance, how do the results change if a more complex geometric model is used, or when faced with augmentations that might produce a non-convex or multi-modal distribution of embeddings?\n3.\tIn the training dynamics (Figure 3b), the sub-manifold size decreases as expected. What mechanism in the framework prevents a trivial solution where the radii collapse to near-zero values for all sub-manifolds, thus minimizing the loss? \n4.\tThe performance of many contrastive methods is highly dependent on batch size. Since CLAMP's loss is computed over all pairs in a batch, it likely shares this dependency. How does CLAMP's performance scale with varying batch sizes?", "limitations": "1.\tThe primary limitation is the scope of the evaluation. As detailed in the weaknesses, the lack of experiments on diverse downstream tasks makes it difficult to fully assess the quality of the representations learned by CLAMP. This should be explicitly stated as a key limitation of the current work.\n2.\tThe simplification of manifold geometry by ignoring orientation and using an isotropic radius approximation is a significant constraint, as the authors themselves note. This limits the \"packing\" from achieving its full theoretical potential and makes the physics analogy less direct than it could be.\n3.\tThe framework's effectiveness is intrinsically tied to the choice of data augmentations, as these define the very geometry of the sub-manifolds being packed. The paper would benefit from a more thorough discussion of this dependency and its implications.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.\tThe main contribution is the novel framing of SSL as a manifold packing problem. This is a refreshing and intuitive departure from point-wise contrastive methods. \n2.\tThe physics-based motivation provides a clear mental model, and the resulting loss function (Eq. 1) is elegant. Its hyperparameters, like the scaling factor have a tangible physical meaning, which is a commendable feature.\nWeaknesses\n1.\tThis is the most significant weakness. The paper's empirical support is not comprehensive enough to demonstrate the superiority or generalizability of the proposed method.\n2.\tThe evaluation is restricted to linear and semi-supervised classification. Modern SSL methods are expected to be benchmarked on a wider array of downstream tasks, such as object detection and semantic segmentation on datasets like COCO, to prove the robustness and richness of the learned features. Without this, it is difficult to ascertain if CLAMP's representations are truly general-purpose.\n3.\tWhile ImageNet and CIFAR-10 are foundational, the SSL field is moving towards more complex and varied datasets. Demonstrating strong performance exclusively on these benchmarks is no longer a sufficient condition for claiming SOTA status for a general vision backbone.\n4.\tThe ablation study for key hyperparameters (Appendix K) was performed on a 30% subset of the ImageNet-1K training data. While the authors cite computational constraints, this reduces the reliability of the findings and makes it difficult to assess the true sensitivity of the model to parameters like and the number of views m.\n5.\tThe framework is built on strong simplifying assumptions whose validity is not sufficiently proven by the limited experiments. The approximation of a complex augmentation distribution as a single ellipsoid is a major simplification. The paper does not provide empirical analysis to justify that this geometric approximation is robust across different augmentation strategies or that it captures the essential structure needed for learning. The limited experiments do not provide enough evidence to bridge the gap between the elegant theory and the practical implementation.", "was_revised": true, "final_justification": "The paper presents a novel self-supervised learning method of Contrastive Learning As Manifold Packing (CLAMP) for visual representations. In the rebuttal, the new object detection results have effectively addressed my main concerns regarding the evaluation on downstream tasks.", "rebuttal": "We thank the reviewer for the constructive feedback. Below, we provide our detailed, point‑by‑point responses.\n\n+ \"Given that the central ...\" :  we present preliminary results on object detection using a Faster R-CNN head with a C4 backbone (pretrained on ImageNet-1K with 4 views for 100 epochs), fine-tuned on VOC2007+2012 training dataset and tested on VOC2007 test dataset. Comparative metrics for other methods are sourced from the MMCR paper (arXiv:2303.03307).\n| VOC2007+12| mAP | AP50 | AP75|\n| --------------- | :-------------: | ---------------:|---------------:|\n| Barlow Twins   | 53.1  | 80.9   | 57.7   |\n| SimCLR           | 54.4  | 81.6   | 61.0   |\n| MoCo v2          | 54.7  | 81.7   | 60.2   |\n| BYOL               | 55.6  | 82.3   | 62.0   |\n| MMCR             | 54.6  | 81.9   | 60.0   |\n| CLAMP            | 55.6  | 82.2   | 61.5   |\n*Higher mAP/ AP50/AP75 shows better performance. \n  Overall our results show that CLAMP is competitive with the state of the art. We will further improve object detection by training on the larger COCO dataset and through hyperparameter tuning. We will include the new results in the revised version of the manuscript.\n\n+ \"The paper's claims hinge ...\" : For perturbative Gaussian noise as the augmentation transformation where the input is x and the transformation is $x+\\zeta$  with $\\zeta \\sim \\mathcal{N}(0, \\Sigma)$, the augmentation submanifold is provably elliptical. The resulting augmented embedding follows a Gaussian distribution:\n$ \\mathcal{N}\\left(g(f(x)), J_x(g(f(x))) \\  \\Sigma \\ J_x(g(f(x)))^\\top \\right) + O(\\zeta ^3)$,\nwhere $J_x( \\cdot)$​ denotes the Jacobian matix. $f(\\cdot)$ and $g(\\cdot)$ denote the encoder and projector network.\nFor more general augmentations, the resulting submanifold is not necessarily elliptical. In such cases, we can approximate the augmentation submanifold with a sufficiently large enclosing sphere, as done in CLAMP. However, this approximation may lose fine geometric details and underutilize the space’s packing capability. Conversely, modeling the submanifold with more accurate geometries  demands significantly more augmentations to estimate, on the order of $O(D)∼100-1000 $ in high-dimensional spaces, which is computationally costly untenable at scale. Thus, there is a trade-off between augmentation cost and packing efficiency.We have mentioned this point in Section 8, line 351.\n\n+ \"In the training dynamics (Figure 3b) ...\" :  A zero-volume submanifold occurs when an image and all its augmentations collapse to the same representation, reflecting perfect transformation invariance. While this invariance can aid image classification, it may discard the fine details needed for other downstream tasks. CLAMP’s packing loss naturally prevents such collapse: radii only contract while two submanifolds overlap, and once they become merely tangent, the shrinkage gradient disappears. As a result, each radius settles at a nonzero “margin” value instead of vanishing.\n\n+ \"The performance of many ... \" : Thank you for raising this point. Here we show the linear evaluation results of pretrained models on ImageNet-1K for 100 epochs with n_views=4 and rs=8.5.\n| Batch sizes | 256 | 512 | 1024 |\n| --------------- | :-------------: | ---------------:|---------------:|\n| Linear evaluation   | 69.0  | 69.4   | 69.1   |\n\nThe training performance depends weakly on the batch size. Such weak dependency   has also been observed for MMCR.We will further improve the ablation study by including this table and additional results that we are currently generating for different numbers of views."}, {"review_id": "mZFRI4ktgf", "reviewer": "Reviewer_bg49", "summary": "A new approach for self-supervised learning of manifolds, based on the idea of packing problem - the idea that different \"objects\" cannot occupy the same space, so that if \"overlapping\" objects repel each other, we may find a packing of the objects in space in which they do not overlap. Here, \"objects\" are actually \"object manifolds\", the collections of neural responses to the same objects. If objects overlap, they cannot be classified correctly; thus, a physical-simulation-like learning where object manifolds repel each other leads to representations where objects can be reliably classified from one another.", "questions": "1. Do CLAMP have meta-parameters which control the balance between alignment and uniformity (lines 119-121)? i.e., control the resulting manifold uniformity despite the locality of all interactions?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "## Strengths \n 1. Great idea and clear presentation of the physics intuition, with our understanding of jammed objects really helping to visualise the process.\n 2. The results are interesting, showing comparable performance to other methods (what the authors call \"competitive\").\n 3. Provide a very simple framing which pushes the representation toward both uniformity (large distances between manifolds) and small size (small distances within manifolds).\n\n## Weaknesses\n 1. The \"with only local operations\" argument is weird and misleading: while only \"near-by\" objects get pushed, this push would change the representation of ALL objects (because of changes to the encoder). \n 2. It is not clear which changes happen in the encoder and which happen in the embedding. As the embedding is not used for classification problems, some of the learned information is actually lost!\n 3. The difference between MMCR and CLAMP is that the former consider all binary linear classifications, which CLAMP consider all binary non-linear classifications. The benefit of \"local interactions\" (lines 11-113) is silly as noted above. \n 4. The results on brain-score are basically null-results (except for V2 maybe), showing SOME correspondence between the learned representations and the brains' representations, without a clear rule. Section 6 could have been omitted.\n 5. The argument on a similar exponent like V1 is extremely weak, as there are probably 4-5 controllable parameters which affect the exponent, making the results meaningless. Section 7 could have been omitted.", "was_revised": true, "final_justification": "I am satisfied with the clarifications and have adjusted my score upwards. I hope the improvements will make it to the published manuscript.", "rebuttal": "We thank the reviewer for time and efforts. Below, we provide our detailed, point‑by‑point responses.\n\n+ \" The \"with only local operations\" argument ... \" : Thanks for pointing this out. The reviewer is correct that the optimization of any active pair of overlapped manifolds changes the encoder and, as a result, all representations are modified. We will clarify what we mean by \"with only local operations\" and will test explicitly the effect on the centroids of inactive manifolds due to an operation on the centroids of two active manifolds.\n\n+ \"It is not clear which ... \" : Thank you for raising this point. The projection head implemented as an MLP (whose outputs we refer to as embeddings) acts as an effective scaffold during contrastive training. In both SimCLR (arXiv:2002.05709) and MoCo v2 (arXiv:2003.04297), inserting this projection head during training and removing it at inference time consistently improves downstream classification accuracy compared to using no projection head at all. Following their success, subsequent methods such as BYOL and Barlow Twins have adopted the same architecture. Here, we also incorporate an MLP projection head to enhance our model’s training performance. In our revised ablation study, we will test the effect of not including the projection head.\n\n+ \"The difference between MMCR ...\" : MMCR seeks to maximize the “manifold capacity” of a set of N low dimensional manifolds. This should be understood in the sense of “perceptron capacity”, i.e. MMCR seeks to improve the relative arrangement of the manifolds so that a linear classifier can shatter as many random assignments of binary labels (1 or 0) to the N manifolds, in certain technical limits. So MMCR ignores the fact that the N manifolds belong to multiple distinct classes, and instead probes the problem as a linear binary classification of low-dimensional manifolds carrying one of two labels. So, MMCR is not explicitly trying to pack manifolds, but it achieves manifold separation because it tends to favor higher perceptron capacity (note however that this is quite indirect and less interpretable than the clear physical picture provided by CLAMP). CLAMP instead directly addresses the N-ary problem of packing and nonlinearly classifying N distinct manifolds in a simple and physically interpretable framework. We will clarify what we mean by local interactions as stated above, and will test explicitly the effect on the centroids of inactive manifolds due to an operation on the centroids of two active manifolds.\n\n+ \" The results on brain-score ...\" :  We appreciate the reviewer’s suggestions about our Brain‑Score (Section 6). Our intent in these sections was not to claim that CLAMP perfectly mirrors every stage of visual cortex, but rather to probe whether a repulsive objective, with no explicit biological constraints, nonetheless gives rise to any of the statistical hallmarks we observe in real neural populations and shows the potential of implementing CLAMP by biologically plausible learning rules.  The limitations of Brain-Score have been discussed in section 8.  We will modify section 8 to show more detailed limitations about the Brain-Score.\n\n_Brain-Score’s linear mapping between deep nets output and experimental activity may overlook non-linear relationships between model and neural representation. It also does not  guarantee that CLAMP implements the same computations or dynamics as cortex._\n\n+ \"The argument on a similar ...\" : We appreciate the reviewer’s feedback on Section 7. As noted by Stringer et al. (Nature, 571(7765):361–365, 2019), smooth neural manifolds are expected to exhibit a power-law exponent greater than 1 across a range of experiments involving different input types and varying fractions of sampled neurons. In this context, our analysis demonstrates that the representations learned by CLAMP exhibit a similar power-law behavior. While we agree that multiple controllable parameters can influence the exponent, we view our findings as a meaningful extension of Stringer et al.’s test, providing an additional perspective on the structure of learned representations.\n\n+ \"Do CLAMP have meta-parameters ...\" :  Thanks for the question. CLAMP’s packing loss is self-balancing: by minimizing the objective it either shrinks submanifold sizes (improving alignment) or pushes them farther apart (improving uniformity), with no extra meta-parameter required to trade off these effects. In other words, the alignment–uniformity balance emerges implicitly from the packing loss itself. As we demonstrate in Section 4.1:Linear Evaluation, the CLAMP packing loss not only simplifies the formulation but also outperforms the original alignment-uniformity method."}, {"review_id": "KbHt7lVCeE", "reviewer": "Reviewer_qXva", "summary": "The paper proposes a new objective criterion for contrastive self-supervised learning. The key idea is based on: (i) a novel representation that treats different augmentations of the same image as submanifolds parameterized by ellipsoids, and (ii) a criterion that quantifies the overlap between these ellipsoids to select negative pairs. The selected ellipsoids are then used in the objective function, encouraging the network to produce similar representations within each ellipsoid while repelling the centroids of different ellipsoids. Overall, the proposed strategy can be seen as a method to effectively harness negative sample pairs.\nThe experimental analysis is conducted by first evaluating linear probing performance on ImageNet-100/1K, comparing the proposed method against several self-supervised baselines, and then by analyzing the properties of the learned representations along with their neuroscientific interpretations. The results demonstrate that the proposed approach performs comparably to state-of-the-art self-supervised learning methods and shows strong alignment with neuroscientific findings.", "questions": "Please refer to the above mentioned weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strenghts**\n- The paper is well written and easy to follow **Clarity**\n- The idea of using ellipsoids and quantifying their overlap is intuitive, simple, yet original **Originality**\n- The experimental methodology on linear probing evaluation is sound **Quality**.\n- The experiments to evaluate the alignment with neuro-scientific findings are highly appreciated **Quality**\n- Code is provided along with the submission **Code availability**\n\n**Weaknesses**\n- The main weakness lies in the significance of the proposed solution **Significance**. Since the method achieves performance comparable to the baselines in linear probing, it remains unclear whether the observations reported in Sections 5, 6, and 7 also hold for the baselines. The claims and analysis would be much stronger and more convincing if comparable results were provided for those cases.\n- There is extensive literature on harnessing negative samples in contrastive learning, e.g. [1], as well as approaches that enforce clustered representations without explicit clustering, e.g. [2] **Quality**. A discussion of related work is necessary to better contextualize and highlight the novelty of the work.\n\n**References** \\\n[1] Contrastive Learning with Hard Negative Samples. ICLR 2021 \\\n[2] Collapse-Proof Non-Contrastive Self-Supervised Learning. ICML 2025", "was_revised": true, "final_justification": "Authors have clarified all weaknesses about significance and positioning with respect to existing literature. Therefore, I update my score to account for the additional information provided in the rebuttal\n\nThe work provides experimental benefits especially on the aspect regarding neuroscientific alignment and is of interest to both researchers working in the area of self-supervised learning as well as computational neuroscience.\n\nWhile this is good work, I believe it is not of groundbreaking nature and therefore I wouldn't recommend it for an oral.", "rebuttal": "We thank the reviewer for the careful review. Below, we provide our detailed, point‑by‑point responses.\n\n+ \"The main weakness lies ...\"  : In Section 5 we deliberately track sub‑manifold sizes and nearest‑neighbour counts because these metrics are intrinsic to CLAMP’s augmented submanifold loss; contrastive baselines like SimCLR, BYOL, and MoCo do not explicitly form or optimize over sub‑manifolds, so these measurements simply don’t map onto their  training dynamics.\nFor Sections 6 and 7, we instead draw on the published Brain‑Score benchmarks and eigenspectrum exponents reported in the MMCR paper (NeurIPS 2023: 24103–24128). We have extracted the relevant numbers from MMCR and included them alongside our own results to enable a fair comparison, showing that CLAMP’s manifold‑packing framework yields distinct gains in neuroscientific alignment and geometric uniformity over these established methods.\n\n+ \"There is extensive literature ...\" :  Thanks for referring to these two works. We will add the sentences below to the related work section in the finalized version.\n\n   _Hard‑negative methods draw the negative samples  from the tailored Boltzmann-like distributions to sharpen sample‑wise contrasts._\n\n  _Collapse‑proof non‑contrastive methods preserve feature diversity and avoid collapse by implicitly clustering representations and enforcing augmentation invariance without negative samples._"}]

Author rebuttal:

Meta-review: The paper explores self-supervised contrastive learning through the lens of manifold geometry, drawing inspiration from neuroscience and physics. Building on the observation that neuronal responses in the cortex form structured manifolds, the authors propose Contrastive Learning As Manifold Packing (CLAMP), a framework that reformulates representation learning as a packing problem in embedding space. The key idea is to treat augmented views of an image as sub-manifolds, whose positions and sizes are optimized using a loss function inspired by the repulsive interactions in particle systems studied in condensed matter physics. This “packing loss” provides geometrically interpretable dynamics and introduces hyperparameters tied to manifold separation. The approach offers an interpretable physical analogy to jamming and packing phenomena. The authors also show the practical effectiveness, achieving competitive results under the linear evaluation protocol relative to state-of-the-art contrastive methods. 

Reviewers consistently praised the conceptual novelty of linking representation learning with packing dynamics, the clarity of the geometric intuition, and the interpretability of the loss function, whose hyperparameters have clear meaning. Empirical strengths include competitive linear evaluation results on ImageNet and CIFAR benchmarks, neuroscientific alignment evidenced by strong Brain-Score performance, and new results on object detection that help validate downstream utility. The reviewers also appreciated that the work bridges machine learning with insights from neuroscience and physics. Some weaknesses that the reviewers noted was the method’s biological grounding is more metaphorical than mechanistic, the evaluation was initially limited to standard classification tasks, and the ellipsoid approximation of augmentation sub-manifolds may oversimplify real geometric structure. Some reviewers also highlighted that while the framework is elegant, the practical gains are modest compared to existing SOTA approaches, and its robustness on larger-scale or multimodal tasks remains to be demonstrated. The rebuttal seems to have addressed these concerns with new experimental results, clarifications on collapse avoidance and batch-size sensitivity, and a discussion of broader applicability. The reviewers all recommend acceptance and I concur.
Decision: Accept (poster)