Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations
Authors: Thomas Edward Yerxa
Yilun Kuang
Eero P Simoncelli
SueYeon Chung
Venue: NeurIPS 2023 poster (2023)
Keywords: computational neuroscience
theoretical neuroscience
efficient coding
representation geometry
neural manifolds
self-supervised learning
statistical physics of learning

Abstract: The efficient coding hypothesis proposes that the response properties of sensory systems are adapted to the statistics of their inputs such that they capture maximal information about the environment, subject to biological constraints. While elegant, information theoretic properties are notoriously difficult to measure in practical settings or to employ as objective functions in optimization. This difficulty has necessitated that computational models designed to test the hypothesis employ several different information metrics ranging from approximations and lower bounds to proxy measures like reconstruction error. Recent theoretical advances have characterized a novel and ecologically relevant efficiency metric, the ``manifold capacity,” which is the number of object categories that may be represented in a linearly separable fashion. However, calculating manifold capacity is a computationally intensive iterative procedure that until now has precluded its use as an objective. Here we outline the simplifying assumptions that allow manifold capacity to be optimized directly, yielding Maximum Manifold Capacity Representations (MMCR). The resulting method is closely related to and inspired by advances in the field of self supervised learning (SSL), and we demonstrate that MMCRs are competitive with state of the art results on standard SSL benchmarks. Empirical analyses reveal differences between MMCRs and representations learned by other SSL frameworks, and suggest a mechanism by which manifold compression gives rise to class separability.  Finally we evaluate a set of SSL methods on a suite of neural predicitivity benchmarks, and find MMCRs are higly competitive as models of the ventral stream.

Efficient Coding of Natural Images using Maximum Manifold Capacity Representations
Thomas Yerxa
Yilun Kuang 2.3 Eero Simoncelli 1,2.3 Yuo Chung 1.2
Center for Neural Science, New York University
2 Center for Computational Neuroscience, Flatiron Institute Courant Inst. of Mathematical Sciences, tey214@nyu.c edu edu
Abstract
The efficient coding hypothesis posits that sensory systems are adapted to the statis- ties their inputs, maximizing mutual information between environmental signals and their representations, subject to biological constraints. While elegant, informa- tion theoretic quantities are notoriously difficult to measure or optimize, and most research the hypothesis employs approximations, bounds, substitutes (e.g., reconstruction error). A recently developed measure of coding efficiency, the "man- ifold capacity", quantifies the number of object categories that may be represented in a linearly separable fashion, but its calculation relies on a computationally inten- sive iterative procedure that precludes its this measure to a form that facilitates direct optimization, use to learn Maximum Manifold Capacity Representations (MMCRs), and demonstrate that these are competitive with state-of-the-an results on current self-supervised learning (SSL. recogmition benchmarks. Empirical analyses reveal important differences between MMCRs and the representations learned by other SSL frameworks, and suggest a mechanism by which manifold compression gives rise class separability. Finally, we evaluate a set of SSL methods on a suite of neural predictivity benchmarks, and find MMCRs are highly competitive as models of the primate ventral stream.
objective. Here, simplify
Introduction
Biological visual systems learn complex representations of the world that support wide range of cognitive behaviors, without relying on large number labelled examples. The efficient coding hypothesis (8) (65) suggests that this is accomplished by adapting the sensory representation to the statistics the input signal, so as to reduce redundancy or dimensionality. Visual signals have several clear sources of redundancy. They evolve slowly in time, since temporally adjacent inputs typically correspond to different views of the same scene, which in turn are usually more similar than views of distinct scenes. Moreover, the variations within individual scenes often correspond variations small number of parameters, such as those controlling viewing and lighting conditions, and are thus inherently low dimensional. Many previous results have demonstrated how the computations of neural circuits can be seen as matched to such structures in naturalistic environments (48) 29 65 [7)
Studies in various modalities have identified geometric structures in neural data that are associated with behavioral tasks (10 25 33 56), and explored metrics for quantifying these structures.
The recent development of "manifold capacity theory" provides more explicit connection between the geometry (size and dimensionality) of neural representations and their coding capacity (18). This theory has been used evaluate efficiency of biological and artificial neural networks across modalities (17 24 66 52). However, usage as a design principle for building model representations has not been explored.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Motivated by these observations, we seek to learn representations in which manifolds containing dif- ferent views of the same scene are both compact and low-dimensional, while manifolds corresponding to distinct scene are maximally separated. Specifically:
We develop form of Manifold Capacity that can be used as an objective function for learning-
We demonstrate that a Maximum Manifold Capacity Representation (MMCR) supports high-quality object recognition (matching the SoTA for self- eelf-suee learning), when evaluated using the standard linear evaluation paradigm (i.e., applying an optimized linear classifier to the output of the self-supervised network) (14).
Through a analysis of internal representations and learning signals, we analyze the un- derlying mechanism responsible for the emergence of semantically relevant features from unsupervised objective functions.
We validate the effectiveness of MMCR as brain model by comparing its learned represen- tations against neural data obtained from Macaque visual cortex.
Our work thus leverages normative goals of representational efficiency to obtain novel model for visual representation that is both effective for recognition and consistent with neural responses the primate ventral stream.
1.1 Related Work
Geometry of Neural Representations. Previous work has sought to characterize how representa- tional geometry, often measured through spectral quantities like the participation ratio (the squared ratio of the I and l2 norms of the eigenvalues of the covariance matrix), shapes different aspects of performance on downstream tasks (32). Elmoznino and Bonner (27) found that high dimensionality in ANN representations was associated with ability to both predict neural data and generalize to unseen categories. Stringer et al. (67) observed that the spectrum of the representation of natural images in mouse cortex follows a power law with a decay coefficient near 1, and Agrawal et al. (2) report that (in artifical representations) the proximity of the spectral decay coefficient to one is an effective predictor of how well a representation will generalize to downstream tasks.
Self-Supervised Learning. Our methodology is related to (and inspired by) recent advances in contrastive self-supervised representation learning (SSL), but has a distinctly different motivation and formulation. Many recent frameworks craft objectives that are designed to maximize the mutual information between representations of different views of the same object (58) 14 58 68
However, estimating mutual information in high dimensional feature spaces is difficult (), and furthermore it is not clear that closer approximation of mutual information in the objective yields improved representations (1) By contrast, capacity measures developed in spin glass theory [D5] are derived in the "large N (thermodynamic) limit"and and thus are intended to operate in the regime of large ambient dimension (88 5. We examine whether one such measure, which until now had been used only to evaluate representation quality, is also useful as an objective function for SSL.
Many SSL methods minimize the distance between representations of different augmented views of the same image while employing constraints to prevent collapse trivial solutions (e.g., repulsion of negative pairs (14), or feature space whitening (77, (8)). The limitations of using a single pairwise distance comparison have been demonstrated, notably in the development of the "multi-crop" strategy implemented in SWAV (13) and in the contrastive multiview coding approach Tian et al. (68). Our approach is based on the assumption that different views of an image form a continuous manifold that we aim to compress. We characterize each set of image views with the spectrum of singular values of their representations, using the nuclear norm as a combined measure of the manifold size and dimensionality.
The nuclear norm has been previously used to infer or induce low rank structure in the representation of data (42) 72 49). In particular, Wang et al. (72) use it as regularizer to supplement an InfoNCE loss. Our approach represents a more radical departure from the traditional InfoNCE loss, as we detail below. Rather than pair a low-rank prior with a logistic aggesssion-base likelihood, we make the more symmetric choice of employing high rank likelihood. This allows the objective to explicitly discourage dimensional collapse, well known issue in SSL (45).

---

2 Maximum manifold capacity representations
Low Capacity
Reduce Dimensionality
Before Training
After Training
Augroerpation Sprane
Radius
Figure I: Two dimensional illustrations of high and low capacity representations. Left: the capacity (linear separability) of random set of spherical regions can be improved, either by reducing their radii (while maintaining their dimensionalities), or by reducing their dimensionalities (while maintaining their average radii). Right: the objective proposed in this paper aims to minimize the nuclear norm (equal to the product of radius and sqrt dimensionality) of normalized data vectors (ie., lying the unit sphere). Before training, the manifolds have a large extent and thus the matrix of their corresponding centroid vectors has low nuclear norm. After training, the capacity is increased: The manifolds are compressed and repelled from each other, resulting in centroid matrix with larger nuclear norm and lower similarity.
2.1 Manifold Capacity Theory
Consider a set of P manifolds embedded in feature space of dimensionality D, each assigned a class label. Manifold capacity theory is concerned with the question: what the largest value of 7y such that there exists (with high probability) a hyperplane separating a random dichotomy (3)) Recent theoretical work has demonstrated that there exists critical value, dubbed the manifold capacity ac, such that when f < ag the probability of finding a separating hyperplane is approximately 1.0, and when P > ac the probability is approximately 0.0 (18). The capacity oc can be accurately 5
predicted from three key quantities: (1) the manifold radius Ry which measures the size of the manifold relative to the distance of its centroid from the origin, (2) the manifold dimensionality DM which quanitifies the number dimensions along which a manifold has significant extent, and (3) the correlation of the manifold centroids. When the centroid correlation is low the manifold capacity can be approximated by o( ((RMVDDi where ((.) is a monotonically decreasing function.
For manifolds of arbitrary geometry. the radius and dimensionality may be computed using an iterative process that alternates between determining the set of "anchor points" on each manifold that are relevant for the classification problem, and computing the statistics of random projections of these anchor points (00). This process both computationally costly and non-differentiable and therefore unsuitable for use as objective function. For more detail on the general theory see Appendix C However, the manifolds are assumed be elliptical shape, then both radius and dimensionality may be expressed analytically:
RM Σ DM = Σ,A, Σ
(I)
where the X? are the eigenvalues of the covariance matrix of the manifold. For comparison, when computing these values for a set of 100 128-D manifolds with 100 points sampled from each, the use of the analytical expression is approximately 500 times faster (in "wall-clock time") than the general iterative procedure.
Using these definitions for manifold radius and dimensionality we can write the capacity ac s 6(E a)) where d; are the singular values of matrix containing points on the manifold (equivalently, the square roots of the eigenvalues of the covariance matrix). In this form, the sum is the L, norm of singular values, known as the Nuclear Norm of the matrix. When used as an objective function, this measure favors sparse solutions (i.e., those with a small number of non-zero singular values) corresponding low dimensionality. It is worth comparing this objective to another natural candidate for quantifying size: the determinant of the covariance matrix. The determinant is equal to the product

---

of the eigenvalues (which captures the squared volume of the corresponding ellipsoid), but lacks the preference for lower dimensionality that comes with the Nuclear Norm. Specifically, since the determinant is zero whenever one (or more) eigenvalue is zero, it cannot distinguish voro-volume manifolds of different dimensionality. Lossy coding rate (entropy) has also been used as a measure of compactness (76), which simplifies to the log determinant of the covariance matrix under Gaussian model (50). In this case, the identity matrix is added to a multiple of the feature covariance matrix before evaluating the determinant, which solves the dimensionality issue described above.
2.2 Optimizing Manifold Capacity
Now we construct an SSL objective function based on manifold capacity. For each input image (notated as a vector X₁ RR)) we generate & samples from the corresponding manifold by applying a set of random augmentations (each drawn from the same distribution), yielding manifold sample matrix XB= E Dxx Each augmented image is transformed by a Deep Neural Network, which computes nonlinear function [xx;; 0) parameterized by 0, and the d-dimensional responses are projected onto the unit sphere yielding manifold response matrix Zo Rd The centroid c₁ is approximated by averaging across the columns (response vectors). For set of images {x1, .... XB) we compute normalized response matrices { IZ..... Z B) and assemble their corresponding centroids into matrix C' E d
Given the responses and their centroids, the MMCR loss function can be written simply:
C= -=-|C\||++ ++
(2)
where ||- indicates the nuclear norm. The loss explicitly maximize the extent of the "centroid manifold", which interestingly is sufficient to learn useful representation. Concretely, maximizing
||C||| implicitly minimizes the extent of each individual object manifold measured by ||Zo||.. We build intuition for this effect below.
Compression by Maximizing Centroid Nuclear Norm Alone. Each centroid vector is mean of unit vectors, and thus has a norm that is linearly related to the average cosine similarity of those unit vectors. Specifically,
K
k
||.
T b,b Zb1
K
K
k=1 1=1
Here zb,i denotes the representation of the 7th augmentation of æs. We can gain further insight by considering how the distribution singular vectors of a matrix depends on the norms and pairwise similarities of the constituent column vectors. While no closed form solution exists for the singular values of an arbitrary matrix, the case where the matrix is composed of two column vectors can provide useful intuition. If C [c,, C2). Z [21,1, 112 Ze
[22,1 22,2], the singular values of
C' and Z; are:
|c 2 + | 2 + ((1
(1|c111 | 12)2 1(c (cT
OCC)
V
(4)
((.) = / 1 2T Z1.2 Z1.2-
(5)
So. [[][ =|C||| is maximized when the centroid vectors have maximal norms (hounded above by since they are the centroids of unit vectors), and are orthogonal each other. As we saw above
the centroid norm is a linear function of within-manifold similarity. Similarly, |oo [Z.)ll || |- is minimized when the within-manifold similarity is maximal. Thus the single term |C]|. encapsulates both of the key ingredients of a contrastive learning framework, and we will demonstrate below that simply maximizing ||C||= is sufficient to learn a useful representation. This is because the compressive role of "positives" in contrastive learning is carried out by forming the centroid vectors, so the objective not posetive-free. For example, if only a single view used the objective lacks a compressive component and fails to produce a useful representation. In Appendix F we demonstrate empirically that this implicit form effectively reduces |Zall= by comparing to the case where IZbll is minimized explicitly. So, all three factors which determine the manifold capacity (radius,

---

dimensionality, and centroid correlations) can be succinctly expressed in an objective function with a single term, term,-//C'|...
Computational Complexity. Evaluating the loss for our method involves computing a singular value decomposition of C Rdx which has complexity O O(Bd Bd x min( d)), where B is the batch size and d is the dimensionality of the output. By comparison, contrastive methods that compute all pairwise distances in batch have complexity O d) and non-contrastive methods that involve regularizing the covariance structure have complexity O ( Additionally, the complexity of our method is constant with respect to the number of views used (though the feature extraction phase is linear in the number of views), while pairwise similarity metrics have complexity that is quadratic in the number of views. It is also worth noting that doing implicit compression by maximizing ||C*|| offers an advantage in computational complexity relative to explicit compression. This is because evaluating term such Di=1
Z
has computational complexity of O(B²d mi)(B.d)).
2.3 Conditions for Optimal Embeddings
Recently HaoChen et al. (39) developed a framework based on spectral decomposition of the "pop- ulation augmentation graph", which provides theoretical guarantees for the performance of self- supervised learning on downstream tasks under linear probing. This work was extended to provide insights into various other SSL objectives by Balestriero and LeCun (6), and we show below that leveraging this approach can lead to explicit conditions for the optimality of representation under our proposed objective as well.
Given a dataset X" = [m1,,.., xN | N we construct new dataset by creating k randomly augmented views of the original data, X = (x'), view (X') NNx The advantage of doing so is that we can now leverage the knowledge that different views of the same underlying datapoint are semantically related. We can express this notion of similarity in the symmetric matrix G E (0, 1} Nk*N with G.j = augmented datapoints i and j are semantically related (and G. 1 as any datapoint is related to itself). We can normalize G such that its rows and columns sum to (so rows of G are k-sparse with nonzero entries equal to 1/k).
Now let Z E Nexd be embedding of the augmented dataset. Then we have GZ [C,,... "C."....T"" where C' is the matrix of centroid vectors introduced above, and the number of repetitions of C k. Then because ((C....,C)) = V/k((C) we can write MMCR loss function as,
== -=
(6)
This connection allows us to make the following statements about the optimal embeddings Z under our loss, which we prove in Appendix A
Theorem:
Under the proposed loss, the left singular vectors of optimal embedding, Z", are the eigenvectors of G, and the singular values of Z" are proportional to the top eigenvalues of G.
3 Results
Architecture. For all experiments we use ResNet-50 (40) as a backbone architecture (for variants trained on CIFAR we removed max pooling layers). Following Chen et al. (14), we append small perceptron to the output of the average pooling layer of the ResNet so that z; = ()())))) where h is the ResNet and g is the MLP. For ImageNet-1k/110 we used an MLP with dimensions [8192, 8192, 512] and for smaller datascts we used [512, 128).
Optimization. We employ the set of augmentations proposed in (38). For ImageNet we used the LARS optimizer with a learning rate of 4.8, linear warmup during the first 10 epochs and cosine decay thereafter with batchsize of 2048, and pre-train for 100 epochs. Note that though we report results using a default batch size of 2048, batch size as small as 256 can be used obtain reasonable results (1.2% reduction compared to batch size 2048 - see Appendix pooer sweep over batch size). We additionally employ a momentum encoder for ImageNet pre-training (all views are fed through an online network and an additional network whose parameters are a slowly moving average of the online network and all embeddings of each image are averaged to form centroids). We found the use of momentum encoder provided small advantage in terms of downstream classification
5

---

Table 1: Evaluation learned features on downstream classification tasks. The leftmost columns show results for the standard frozen-linear evaluation procedure on ImageNet (IN). Results for most methods in this setting are taken from Ozsoy al. (59) except for SwAV which is taken from the original paper (13). Columns 2 and 3 show semi-supervised evaluation ImageNet (fine-tuning on 1% and 10% of labels). The results for this setting for VICReg and CorInfo Max are copied from Ozsoy et al. (59). The final three columns show frozen-linear evaluation We evaluated models for which pretrained weights in the 100 epoch setting were available online; MoCo. Barlow Twins and BYL were taken from solo-learn da Costa et al. (23) (https://githe) http:/turiisiisoolleerraa while SwAV and SimCLR were taken from VISSL Goyal et al. (37) bhttps:/
other datasets.
200 md). For all
evaluations we performed we report the mean and standard deviation over 3 evaluation runs).
Method
IN
1 1%
10%
Food-101
Flowers- 102
DTD
W-MSE 28
69.4
NNCLR
69.4
SwAV (3)
64.6
SimSiam 16
68.1
CorInfoMax 59
69.1
44.9
64.3
VICReg
68.7
44.8
62.2
Barlow Twins 77
68.7
45.1
61.7 + .03
69.8 + .03
86.2 - .22
67.7 = 20
SimCLR
66.5
42.6
61.6 .09
67.2
84.0 .19
64.8
BYOL (88
69.3
49.8 +
15.0+.00
70.6 +11
84.8 .33
67.6 14
MoCo-Va V2 (16)
67.4
43.4
63.2
68.6
82.4
66.6 = 16
SwAV [13)
72.1
49.8 .09
66.9 .05
72.1 .08
89.3 .11
68.2 .18
MMCR (2 views)
69.5 +: .02
46.6 + .02
63.9 + .02
72.0 .02
90.0 24
68.5 = .07
MMCR (4 views)
71.5 1 .04
49.4
.05
66.0 1 .05
73.2 1 .07
91.0 1 .04
70.4 .46
MMCR (8 views)
72.1 1 .04
51.0 1 .2
67.7 1 .11
73.6 1 .04
91.4 1 .07
70.0 .24
performance, Appendix for this ablation. For smaller CIFAR-10 used smaller batch size, many more views (40), and the Adam optimizer with fixed learning rate. See Appendix D for exact details.
3.1 Transfer Downstream Tasks
We used standard linear evaluation technique, freezing the parameters of the encoding network and training linear classifier with supervision (14), to verify that our method extracts semantically relevant features from the data. We also perform semi-supervised evaluation, where all model parameters are fine tuned using small number of labelled examples, and also check whether the learned features generalize three -nitt--ddittiitti datasets: Flowers- Food-101 and the Describable Textures Dataset (57 11 19). The results are summarized in Table and details of the training of the linear classifier are provided in Appendix G Finally we also evaluate our best model in and each of the baselines on object detection on the VOCO7 dataset. We followed (7)), fine tuning the representation networks for detection with a Faster R-CNN head and C-4 backbone using the Ix schedule. MMCR achieved mean average precision (mAP) of 54.6 and the baseline method performance ranged from 53.1 to 56.0, demonstrating that though our framework was inspired by theory of classification the learned features do generalize to other vision tasks. See Appendix M for detailed results.
Note that all included models were trained using the same backbone architecture (ResNet-50), dataset (ImageNet-1kk and number of pretraining epochs (100; we briefly explore the impact of longer pretraining in Appendix [P). The results for networks pretrained on smaller datasets can be found in the Appendix
3.2 Analyses of Learned Representations
next conduct series experiment to elucidate the differences between representations learned with different SSL procedures, and suggest a mechanism by which augmentation manifold compres-

---

sion gives rise to class separability. To reduce the computational requirements, these analyses are carried out on models trained on the CIFAR- 10 dataset.
Augmentation Manifolds
Class Manifolds
10.6
Dimension
56.0
46.0
00.5
26.3
Radius
1.2
1.0
Barlow Twins
Capacity
0.08
SimCLR MMCR
8.04
0.00
Representation
Reprosmtttin
Plaes
ResNet Hierarchy
Puels
DeNNethieerchh
Figure 2: Mean field Manifold Capacity analysis. Manifold dimensionality (top row), radius (middle row), and capacity (bottom row), as function of stage in the representational hierarchy (from pixel inputs to output of the encoder/learned representation). Shaded regions indicate 95% confidence intervals around the mean (analysis was conducted with 5 different random samples from the dataset, see Appendix
Mean Field Theory Manifold Capacity. In Fig. we show that our representation, which is optimized using an objective that assumes elliptical manifold geometry, nevertheless yields repre- sentations with high values of the more general mean field manifold capacity (relative baseline methods). For completeness we also analyzed the geometries of class manifolds, whose points are the representations of different examples from the same class. This analysis provided further evidence that learning to maximize augmentation manifold capacity compresses and separates class manifolds, leading to a useful representation. Interestingly MMCRs seem to use a different strategy than the baseline methods to increase the capacity, geedingclas/augmentatioo manifolds with larger radii, but lower dimensionality (Fig. 2 These geometrical differences do not emerge until the tail end of the hierarchy, suggesting early layers of each network are carrying out stereotyped transformations and loss function induced specialization does not emerge until later layers.
Emergence of neural manifolds via gradient coherence. We hypothesize that class separability in MMCRs arises because augmentation manifolds corresponding examples from the same class are optimally compressed by more similar transformations than those stemming from distinct classes. To investigate this empirically, we evaluate the gradient of the objective function for inputs belonging to the same class. We can then check whether gradients obtained from (distinct) batches of the same class are more similar each other than those obtained from different classes, which would suggest that the strategy for compressing augmentation manifolds from the same class are relatively similar to each other. Fig. demonstratees that this is the case: within class gradient coherence, as measured by cosine similarity, is consistently higher than across class coherence across both training epochs and model hierarchy.
Manifold subspace alignment. Within-class gradient coherence constitutes a plausible mechanistic explanation for the emergence of class separability, but it does not explain why members of the same class share similar compression strategies. To explore this question we examine the geometric proper- ties of augmentation manifolds in the pixel domain. Here we observe small but measurable differences between the distributions of within-class similarity and across-class similarity, as demonstrated in the top row of Fig. The subtle difference the geometric properties of augmentation manifolds in the pixel domain in turn leads to the increased gradient coherence observed above, which leads to representation that rearranges and reshapes augmentation manifolds from the same class in a similar fashion (bottom row of Fig. thus allowing better linear separation of classes. Not only are centroids
7

---

All Parameters
First Layer
Last Layer
Gradient Similartt
Distinct Closs
DW
0.1
CL
8W
50
100 120
Epochhs 200
100
120
330
Epocihs
Epochs
Figure 3: Cosine similarity of gradients for pairs of single-class batches. We plot the mean pairwise similarity for pairs of gradients for different subsets of the model parameters (all parameters, and the first and last linear operators) obtained from single-class-aatee batches coming from the same or distinct classes the course of training. Because large number of bathces intervals about these means are too small to be visible. To the left is visualization of the fact that single-class gradients flow backward through the model in more similar directions.
used 95% confidence
of came-class-manifole in more similar regions of the representation space than those coming from distinct classes (Fig. third column bottom row) but additionally same-class-maniool have more similar shapes to each other (Fig. bottom row columns I and 2 show saaee-class-manifokkee occupy subspaces with lower relative angles and share more variance).
We next ask how the representation learned with the MMCR objective differs from those optimized for other self-supervised loss functions. While MMCR encourages centroids be orthogonal, the InfoNCE loss (14) encourages negative pairs to be as dissimilar as possible, which is achieved when they lie opposite regions of the same subspace The Barlow Twins (77) loss not an explicit function of feature vector similarities, but instead encourages individual features to be correlated and distinct features to be uncorrelated, across batch dimension. Fig. 5 shows that these intuitions are borne out empirically: the MMCR representation produces augmentation manifold centroids that are significantly more orthogonal to each other than the two baseline methods.
Subspace Angle
Shared Variance
Centroid Simiarity
1Dgg Augiamation
Singla Cal Roemphrr
Class Menbolld
Aeg mentatte
Girgle Dog
Regi eeteetttttionnnnn
Figure 4: The distributions various similarity metrics for augmentation manifolds from the same input images (top row), and the (orange) and distinct (blue) classes. These shown for both the
learned representation (bottom row). Left: schematic illustration of the exemplar-augment augmentation manifold-class manifold structure of the learned representation.
3.3 Biological relevance
Neuroscience has provided motivation for many of the developments in artifical neural networks, and is of interest to ask whether SSL networks can characterize the measured behaviors of neurons in biological visual systems. As simple test, Table 2 shows performance of our model compared with five other SSL models on the BrainScore repository (64, 30 53 51). We find that MMCR achieves the highest performance in explaining neural data from primate visual areas V2 and V4, second-highest for and IT, and the most or second most predictive for 8 out of the 11 individual datasets (see Appendix K.
In addition, we examine general response spectral characteristics that have been recently described for neural populations. In particular, Stringer et al. (67) reported that the eigenspectrum of the

---

Same Class
Distinct Classes
Barlow Twins SimCLR MMCR
0.08
0.12
Probabiltty
0.03
0.04
0.01
0.2
0.4 04 0.8
0.2
0.6 0.8
Centroid Similarity
Centroid Similarity
Figure 5: Cosine similarities of centroids for models trained according different SSL objectives. The left panel shows the distribution of cosine similarities for centroids of augmentation manifolds for examples of the same class, while the right shows the same distribution for examples from distinct classes. Note that because we are analyzing the outputs of the ResNet backbone (which are rectified), the minimum possible cosine similarity is O.
Table 2: Neural prediction and spectral properties of self-supervised models. First four columns provide average BrainScore correlation values (64) for neurons recorded in four areas of primate ventral stream. Last provide participation ratio (PR) and spectral decay coefficients (a), estimated for 10 bootstrapped samples of the features for the ImageNet validation set. When estimating the spectral decay coefficient we omitted the tails (where the power decays more rapidly), see L for exact details. Entries indicate mean and standard eπrоr of the mean, and boldface indicates best performance within standard error. MMCR results are for a model trained with views.
Model
VI
V2
V4
IT
PR 279.2
Cx
MMCR
.494 + .006
.311 + .005
.481
+ .005
.416 + .003
+ .3
1.04 - le-4
SimCLR
.500 + .007
.288 .007
.475 - .004
.420 .003
124.6 +
1.34 = 3e-4
BYOL
.500 .006
.291 .007
.477 .005
.404 + .003
248.0+
1.35 + le-4
MoCo
.499 + .007
.293 + .006
.477 .005
.417 + .003
147.7
1.44 + 3e-4
Barlow
.498 .007
.933 + .008
.477 + .005
.404 + .003
252.2
1,21 + 3e-4
SwAV
.488 .007
.26 + .009
.463 .004
.398 + .003
163.9 1.2
1.15 + 3e-4
covariance matrix of population activity in visual area V1 follows a power law decay with a decay coefficient of approximately 1 (A,, X n with studies in artifical networks have found that such a decay spectrum is associated with increased robustness to adversarial perturbations and favorable generalization properties (55) 2). Additionally, several recent works have investigated the connection between representational dimensionality and particular, Elmoznino and Bonner (27) report that high intrinsic dimensionality (as measured with the participation ratio of the representation covariance) is correlated with stronger ability predict neural activity. Table provides values for the participation ratio of each representation over the ImageNet validation set, as well the decay coefficient of the covariance spectrum (see Appendix L for more details on each experiment). We see that the MMCR has highest participation ratio (dimensionality) note that this differs from the quantity optimized in the objective function, which lies the embedding space. addition, MMCR also also yields features with decay coefficient that is nearest to one.
where X, is the th cigenvalues). Subsequent
neural predictivity 77 63)
We find that in this controlled setting each model explains very similar fraction of neural variance through linear regression. This is consistent with recent and concurrent works (21 62) which have identified this lack of ability discriminate between alternative models as weakness of the dominant paradigm used for model-to brain comparisons. However, our results demonstrate that different SSL algorithms produce representations with meaningfully different geometries (as evidenced by the large spread in the spectral properties such as the participation ratio and decay coefficient). This suggests the need for the development of new metries for comparing models data, such as geometrical measures, that capture these important differences between candidate models.

---

4
Discussion
We have presented a novel self-supervised learning algorithm inspired by manifold capacity theory. Many existing SSL methods can be categorized as either "contrastive" or "non-contrastin depending on whether they avoid collapse by imposing constraints on the embedding gram or covariance matrix, respectively. Our framework strikes a compromise, optimizing the singular values of the embedding matrix itself. By directly optimizing this population level feature (the spectrum), we are able to encourage alignment and uniformity (7I) simultaneously with single-term objective. Additionally, this formulation circumvents the need for making a large number of pairwise comparisons, either between instances dimensions. As result learning MMCRs is efficient, requiring neither large batch size nor large embedding dimension. Finally, our method extends naturally to the multi-view case, offering improved performance with minimal increases in computational cost.
Our formulation approximates manifold geometries as elliptical, reducing computational requirements while still yielding useful learning signal and networks with high manifold capacity. Specifically, we were able to leverage manifold capacity analysis in its full generality to gain insight into the geometry of MMCR networks after training. Further research could explore objectives based on more modest reductions of mean field manifold capacity that capture non-elliptical structure. Intriguingly, our method produces augmentation and class manifolds with lower dimensionality but larger radius than either Barlow Twins or SimCLR (Fig. 2 We do not understand why this is the case, but the differences indicate that capacity analysis encoding strategies encouraged by various SSL paradigms.
provide useful tool for elucidating the different
Finally we investigated two recently proposed theories on ideal spectral properties for neural repre- sentations. For our considered set of models a spectral decay coefficient near was associated with better performance on the dithin-iistrbbution task and generalization to unseen datasets (MMCR, SwAV, and Barlow were the top three models for each of the cattof-dditributiin classification tasks), finding which is broadly aligned with both the empirical and theoretical findings of Agrawal et al. (2) However, we also found that high dimensionality did not always correspond to strong neural predictivity: Despite having the lowest dimensionality, SimCLR performed strongly in terms of neural predictivity. This implies, perhaps unsurprisingly, that global dimensionality alone is not sufficient to explain the response properties of neurons in the primate ventral stream. In fact our experiments add to a growing body of work on the need for complementary approaches to linear predictivity for discriminating between candidate models of the visual system (21). Happily the field is already moving to address this limitation, for instance concurrent work (12) finds that decomposing neural predictivity error into distinct modes can yield insights into how different models fit different aspects of the neural data (even if they yield similar overall predictivity).
One promising direction for improving the quality of artificial networks as models of neural computa- tions to incorporate the constraints associated with biologically plausible learning (i.e. the need for local learning rules, Illing et al. (44)). A complementary direction is to better align training data diets with ecological inputs. For example, temporal invariance rather than augmentation invariance seems more plausible objective for a neural system to optimize 77 3 6T. We speculate that variant of the MMCR objective that operates over time may be well-suited to a neural circuit implementation as its computation would only require mechanisms for tracking (1) short timescale temporal means (to form centroids) and (2) the singular values of the population activity over longer timescales.
Neuroscience is brimming with newly collected datasets recorded from ever larger populations of neurons, and there a long history of methods that aim to make sense of these measurements through normative lens. Here we have demonstrated that one such technique that has proven useful for gleaning insights from neural data (60 75 3D can be reformulated for use as an objective function to learn useful abstract representation of images. Future work should aim to close the loop between modelling and analysis by using these learned models to generate experimentally testable predictions and constrain new experimental designs.
10

---

References
[1] Alia Abbaras, Benjamin Aubin, Florent Krzakala, and Lenka Zdeborová. Rademacher com- plexity and spin glasses: A link between the replica and statistical theories of learning. In Mathematical and Scientific Machine Learning, pages 27-54. PMLR, 2020.
[2] Kumar K Agrawal, Amab Kumar Mondal, Arna Ghosh, and Blake Richards. Assessing representation quality in self-supervised learning by measuring cigenspectrum decay. Advances in Neural Information Processing Systems, 15:17666-16666 2022.
[3] Arthur Aubret, Markus R Ernst, Céline Teulière, and Jochen Triesch. Time to augment self- supervised visual representation learning. In The Eleventh International Conference Learning Representations, 2022.
[4] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
[5] Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl- Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11(1), 2020.
[6] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205. 2022.
[7] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varimace-inarrincce-cvvaaiiee regular- ization for self-supervised learning. arXiv preprint arXiv:2105.04900 2021.
[8] Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication, 1(01), 1961.
[9] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR, 2018
[10] Silvia Bernardi, Marcus K. Benna, Mattia Rigotti, Jérôme Munuera. Stefano Fusi, and C Daniel Salzman. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell, 183(4): 954-967, 2020.
[11] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101 mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[12] Abdulkadir Canatar, Jenelle Feather, Albert Wakhloo, and Sue Yeon Chung. A spectral theory of neural prediction and alignment. arXiv preprint arXiv:2309, 12821, 2023.
[13] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-999 99244, 2020.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1599711077. PMLR, 2020.
[15] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Patlern Recognition, pages 15750-15758, 2021.
[16] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.04297 2020.
[17] Yeo Chung and LF Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion in neurobiology, 70:137-144, 2021.
[18] Sue Chung, Daniel D Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Physical Review X, 8(3):031003, 2018.

---

[19] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In Proceedings the IEEE conference on computer vision and paltern recognition, pages 3606-3613, 2014.
[20] Uri Cohen, Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):1-13, 2020.
[21] Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv, 2023, doi: 10.1101/2022.00 03 28, 28.485868.
URL https
//www. biw.xiv...ii..ooo wwwpxiii.rggeoooooo wwg/
03. 28. 485868
[22] Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers, (3):326-334, 1965.
[23] Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of self-supervised methods for visual representation learning. J. Mach. Learn. Res,, 23:56-1, 2022.
[24] Joel Dapello, Jenelle Feather, Hang Le, Tiago Marques, David Cox, Josh McDermott, James J DiCarlo, and SueYeon Chung. Neural population geometry reveals the role of stochasticity in robust perception. Advances in Neural Information Processing Systems, 34: 15695- 2021.
[25] James DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 11 ((()):333-41, 2007.
[26] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representa- tions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588 9597, 2021.
[27] Eric Elmoznino and Michael F Bonner. High- gigh-performing neural network models of visual cortex benefit from high latent dimensionality. bioRxiv, 2022.
[28] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International Conference on Machine Learning, pages 3015-3024. PMLR, 2021.
[29] Adrienne L Fairhall, Geoffrey D Lewen, William Bialek, and Robert de Ruyter van Steveninck. Efficiency and ambiguity in an adaptive neural code. Nature, 412(6849):7877792. 2001.
[30] Jeremy Freeman, Corey M Ziemba, David Heeger, Eero P Simoncelli, and J Anthony Movshon. functional and perceptual signature of the second visual area in primates. Nature neuroscience, 16(7):974-981, 2013.
[31] Emmanouil Froudarakis, Uri Cohen, Maria Diamantaki, Edgar Y Walker, Jacob Reimer, Philipp Berens, Haim Sompolinsky, and Andreas S Tolias. Object manifold geometry across the mouse cortical visual hierarchy. BioRxiv, pages 2020-08, 2020.
[32] Stefano Fusi, Earl K Miller, and Mattia Rigotti. Why neurons mix: high dimensionality for higher cognition. Current opinion in neurobiology, 37:66-74, 2016.
[33] Juan A Gallego, Matthew G Perich, Lee E Miller, and Sara A Solla. Neural manifolds for the control of movement. Neuron, 94(5):978-99) 98448888884048 2017.
[34] Elizabeth Gardner. The space of interactions in neural network models. Journal of physics A: Mathematical and general, 21(1):257, 1988.
[35] Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathematical and general, 21(1): 1988.
[36] Gerl and U Krey. Storage capacity and optimal learning of potts-mode perceptrons by a cavity method. Journal of Physics A: Mathematical and General, 27(22):7353, 1994.
12

---

[37] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl. http:////gttuub..ccooaaaeeooooee
viss/ 2021.
[38] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning, Advances in neural information processing systems, 33:21271-21284, 2020.
[39] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-5011, 2021.
[40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[41] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVE conference on computer vision and puttern recognition, pages 9729-9738, 2020.
[42] O J Hénaff, N Rabinowitz. Ballé, and E P Simoncelli. The local low-dimensionality of natural images. In Int'l Conf on Learning Representations (ICLR), San Diego, CA, May 2015. URL abs/ 14412.6622
http:///rxivv
[43] Olivier J Hénaff, Yoon Bai, Julie A Charlton, Ian Nauhaus, Eero P Simoncelli, and Robbe LT Goris. Primary visual cortex straightens natural video trajectories. Nature communications, 12 (1):1-12, 2021.
[44] Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-super contrastive predictions. Advances in Neural Information Processing Systems, 11:30365-307799 2021.
[45] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:21 2021.
[46] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[47] Andrew Knyazev and Merico E Argentati. Principal angles between subspaces in an a-based scalar product: algorithms and perturbation estimates. SIAM Journal on Scientific Computing, 23(6)2008-2004 2040, 2002.
[48] Simon Laughlin. A simple coding procedure enhances neuron's information capacity. Zeitschrift für Naturforschung c, 36(9-10):910 912, 1981.
[49] José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Ole: Orthogonal low-rank embedding-a plug and play geometric loss for deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8109-8118, 2018
[50] Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE transactions on pattern analysis and machine intelligence, 29(9): 154--1562. 2007.
[51] Najib J Majaj. Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 35(39): 3402-13418, 2015.
[52] Jonathan Mamou, Hang Le, Miguel Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and Sucyeon Chung. Emergence of separable manifolds in deep language representations. In International Conference Machine Learning, pages 6713-6723. PMLR, 2020.
[53] Tiago Marques, Martin Schrimpf, and James J DiCarlo. Multi-scale hierarchical neural network models that bridge from single neurons in the primate primary visual cortex to object recognition behavior. bioRxiv, pages 2021-03, 2021.
13

---

[54] Albert W Marshall, Ingram Olkin, and Barry C Amold. Inequalities: theory of majorization and its applications, volume 143. Springer, 1979.
[55] Josue Nassar, Piotr Sokol, Sue Yeon Chung, Kenneth Harris, and Memming Park. On 1/n neural representation and robustness. Advances in Neural Information Processing Systems, 33: 6211-6222, 2020.
[56] Edward H Nieh, Manuel Schottdorf, Nicolas W Freeman, Ryan J Low, Sam Lewallen, Sue Ann Koay, Lucas Pinto, Jeffrey L Gauthier, Carlos D Brody, and David Tank. Geometry of abstract learned knowledge in the hippocampus. Nature, 595(7865):80 2021.
[57] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
[58] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1007::800.0377 00740,3 2018.
[59] Serdar Ozsoy, Shadi Hamdan, Sercan Ö Arik, Deniz Yuret, and Alper T Erdogan. Self-supervised learning with information maximization criterion. arXiv preprint arXiv:2209. 07099, 2022.
[60] Nihaad Paraouty, Justin D Yao, Léo Varnet, Chi-Ning Chou, SueYeon Chung, and Dan H Sanes. Sensory cortex plasticity supports auditory social learning. Nature Communications, 14(1): 5828, 2023.
[61] Nikhil Parthasarathy, SM Eslami, João Carreira, and Olivier Hénaff. Self-supervised video pretraining yields strong image representations. arXiv preprint 12XXi:2210006433, 2022.
[62] Andrew Saxe, Stephanie Nelli, and Christopher Summerfield. If deep learning is the answer, what the question? Nature Reviews Neuroscience, 22(1):55-677 2021.
[63] Rylan Schaeffer, Mikail Khona, and Ila Fiete. No free lunch from deep learning in neuroscience: A case study through models of the antorhinal-hippocampall circuit. Advances in Neural Information Processing Systems, 35: 16052-16067, 2022.
[64] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv preprint, 2018. URL http:::////// //www/www.... biorxiv org/ content/10. 1070/407077v
[65] Eero Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual review of neuroscience, 24(1):1193-1216 2001.
[66] Cory Stephenson, Jenelle Feather, Suchismita Padhy, Oguz Elibol, Hanlin Tang, Josh McDer- mott, and Sue Yeon Chung. Untangling in invariant speech recognition. Advances in neural information processing systems, 32, 2019.
[67] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High-dimensional geometry of population responses in visual cortex. Nature,571(7766)) 361-365, 2019.
gcomett
[68] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pages 776-794. Springer. 2020.
[69] Nenad Tomasev, Ioana Bica, Brian McWilliass, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05/ 2022.
[70] Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H McDermott. Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions. bioRxiv, pages 2022-09, 2022.
14

---

[71] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through aligmment and uniformity on the hypersphere. International Conference Machine Learning. pages 9939.9939. PMLR, M, 2020.
[72] Yu Wang, Jingyang Lin, Qi Cai, Yingwei Pan, Ting Yao, Hongyang Chao, and Tao Mei. A low rank promoting prior for unsupervised contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[73] Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
[74] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018.
[75] Justin D Yao, Klavdia O Zemlianova, David I. Hocker, Cristina Savin, Christine M Constantino- ple, Sue Yeon Chung, and Dan Sanes. Transformation of acoustic information to sensory decision variables in the parietal cortex. Proceedings of the National Academy of Sciences, 120 (2)=e2112120120, 2023.
[76] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-9434 9434,, 2020.
[77] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, pages 10-12220. PMLR, 2021.
15

---

A Optimal Embeddings
Recall the setting of self-supervised learning described in Balestriero and LeCun ( given dataset X' [ (
...., x xx|² RNNDD we construct new dataset by creating k randomly augmented views of the original data, X [view] view (X'] NNX The advantage of doing so is that we can now leverage the knowledge that different views of the same underlying datapoint are semantically related. We can express this notion of similarity in the symmetric matrix G E {0, 1 JN with Gij = augmented datapoints i and j are semantically related (and G
any datapoint is related to itself). We normalize G' such that its rows and columns sum to (so rows of G are k-sparse with nonzero entries equal to 1/k).
Now let Z E Nexd be an embedding of the augmented dataset. Then we have GZ GZ= = -C,,., CIT
where C' is the matrix of centroid vectors introduced above, and the number of repetitions of C is k. Then because o(C,..., CI) = Vko (C) we can write MMCR loss function as,
C 3= -IIGZ
=-IQQAQ USV"U
= =-IIAQ"US USI..
Where we have taken the cigendecomposition of G which is real and symmetric and the SVD of Z, and then used the fact that the singular value spectrum is invariant under left or right orthogonal transformations. We now show that global optima of this objective is achieved when the left singular vectors of Z are the eigenvectors of G and the singular values of Z are proportional to the eigenvalues of G. Throughout we will assume that the size of the dataset is greater than the dimensionality of the embedddings, N d, as the case in practical applications. First we prove simple lemma about the spectrum of matrices who are extended by zeros (i.e. embedded in higher dimensional space).
Lemma A.1: For A E R NNN B E Nxd with d < N, |AB - RNNN
AB|, where B =B,0] E
Proof: First note that AB = |AB, 0) so suffices to show that for arbitrary X that o
o[X,)) O)). Taking the SVD of X,
Σ [
x x Ú
V'
1
USV
0
Then valid singular value decomposition for X is
x x= Ū
Σ
0
VT
0
0
Clearly then, || |
|| ||.
Theorem: The proposed loss achieves a global minimum when the left singular vectors of Z are the eigenvectors of G, and the singular values of Z are proportional to the top d eigenvalues of G.
Proof: Let Ż = IZ. 0] E NNN, By Lemma A.I we have ||ZZ|| = ||GŽ]|. Von Neumann's 's
trace inequality can be used to show ||GŽ|| < Ei= (G) ( (see Marshall et al. (54) for proof). Examining (4) it clear that this bound is achieved when = Q. The problem can therefore be reduced to the constrained optimization problem,
Nk
N min Σ (G)o, (.
Nkk
subject to
(0) Nkk
where the constraint comes from the fact that columns of Z are unit vectors. Intuitively, we are maximizing the inner product between fixed vector o (G) and vector with fixed L2 norm. The
16

---

solution of course is to align the two vectors as closely possible, i.c. when oz x (G, for
i = L, I,...,d d. It is worth noting that by construction o, (Z) 0 for i > d and the columns of U associated with these zero valued singular values are unconstrained.
B Pytorch Style Pseudocode for MMCR
encoder
g: projection head
T: momentum temperature
B: batch size
K: number of augmentations projector output dimensionality
Imbda: trade-off parameter
f_0, g-o ResNet500), ()P() online networks
11
12 initialize momentum network with identical params
13 f f_m, 8.m = cop, s.o.copp ()
15 momentum networks not updated via gradient descent
16 f_m. requires...e
False
17 g_m. m.m. rrquires-grad False
oor in loader:
20
randomly augmented views x multi_ augment (x) # B
21
22
push through encoder and projector
z (x)) D
25
z_ g_m(h_[(x)) x
26
z concatenate z_m, dim=1) # append outputs
27
project onto unit sphere normalize dim=-1)
29
30
31
calculate centroids (mean over augmentation axis) mean (dim=1) 3
34
calculate singular values U_z, $_2, V_z svd(z) batch svd U_c, S_c, V_c svd(c)
35
36
calculate loss loss -1.0 (8_c) Imbda sum ($_2) B
39
40
backward pass and optinization step loss (ackward spepm
43
44
45
# perform momentum update
with torch. torch ()
f_m. f.m.mtrtmte (1 T) f_o. parameteer () f_m.
(arameters()
48
( = (1 T) g_o parametera () + * g_m. parameters
C Mean Field Theory Manifold Capacity Background Information
For completeness we summarize some of the central arguments from Chung et al. which develops the general form of manifold capactiy theory.
17

---

0. Mean Field Theory Recall the problem setting for manifold capacity analysis: given a set of P manifolds embedded in a feature space of dimensionality D, each assigned random binary class label (88). Manifold capacity theory is concerned with the question: what is the largest value of F such that there exists (with high probability) hyperplane separating the two classes? In the thermodynamic limit, where P, D + xc but D remains finite, the inverse capacity can be written exactly,
a = Ex(F(T)
(8)
where, F(T) ming { {IIV - 212 gs gs(V) 2 =0. S is the set defining the manifold geometry (i.e. the set of vectors 5 that are points individual manifold), T are random vectors drawn from white multivariate Gaussian distribution, and gs (V) ming(V 5 1.5 S), J the concave support function.
The KKT equations for this convex optimization problem are:
- T - 0
A 2 0
(VV - K 2 0
(9)
A gss (V)
where S(T) is subgradient of the support function. When the support function is differentiable, the subgradient is unique and equal to the gradient,
5(T) Vgs arg min V S
(10)
SES
((T) is the unique point in the convex hull of S that satisfies the first KKT equation, and is called the "anchor point" for S induced by the random vector T.
Equivalent Interpretation of Anchor Points For given dichotomy (random binary class labelling) the weight vector of the maximum margin separating hyperplane can be decomposed into a sum of at most P vectors, with each manifold contributing single vector, which lies within the convex hull of the manifold. The position of said point point is function of the manifolds position relative to all of the other manifolds the space and depends on the particular set of random labels. Thus there exists a distribution of the cavity method it can be shown that these points are none other than the anchor points that are involved in solving the optimization problem described above (36).
poiiits for each individual manifold. Using
samples several random Gaussian vectors T, and then for each T, V and S determined by solving the quadratic program- ming program given above. The capacity is then estimated as the mean value of F or the samples T.
Numerical Solution To solve the mean field equations numerically,
Manifold Geometries The way the capacity varies in terms of the statistics of the anchor points can be simplified by introducing two key quantities, the manifold radius R&s and manifold dimensionality Ry:
R2, = (7)) DM E, S(T))
(11)
where Š(T) is a unit-vector the direction of the anchor point S. In particular as discussed in the main text, the manifold capacity can be approximated by di (RM)DM)) where & is monotonically decreasing function.
18

---

Elliptical Geometries In the case where the manifolds exhibit elliptical symmetries, the manifold radius and dimensionality can be written in terms of the eigenvalues of the covariance matrix of the anchor points:
Ris Σ X²
(12)
E 2=) Σ3 1
DM
So, in this case RM is the total variability of the anchor points, and DM is generalized participation ratio of the anchor point covariance, well known soft measure of dimensionality.
D Additional Pre-training information
Settings for CIFAR/STL-10 We take the parameters of each augmentation directly from Zbontar et al. (77), but for these lower resolution images we omitted Gaussian blurring and solarization augmentations. All models were trained for 500 epochs using the Adam optimizer (46) with learning rate of le - 3 and weight decay of le 6. For all three methods we used a one hidden layer MLP with hidden dimension of 512 and output dimension of 128 for the projector head g. We swept batch size for each method and chose the one that resulted in the highest downstream task performance. For both SimCLR and Barlow Twins we found that batch size of 128 was optimal (among 32, 64, 128, 256, and 512) for all 3 datasets. For MMCR there is a trade-off between batch size and the number of augmentations used, and the optimal value of that trade-off is highly dataset dependent. For CIFAR-10 and CIFAR- 100 we used batch size of 32 and 40 views, and for STL-10 we used a batch of 64 with 20 views For Barlow Twins we used À 128 which normalizes for the number of elements in the on-diagonal and off-diagonal terms in the loss. For SimCLR we used the recommended setting of 0.5. The overall performance of both baseline methods (and likely MMCR as well) could be increased with a more thorough hyperparameter search and by employing methodology that more closely matches the original works. For example, both methods would likely benefit from the combination of larger batch size, the use of the LARS optimizer (which is designed for large batch optimization),, a learning rate scheduler consisting of linear warm-up followed by cosine annealing. longer training, and the use of more diverse augmentations (i.e. including solarization and gaussian blur). Additionally Barlow Twins reports that the representation can benefit from using a much larger projector network than we use. Because our goal was primarily to demonstrate that MMCR can produce representations that are comparable to these baselines rather than to produce state-of-ther of-the-rrt results on small scale datasets we opted for simplifications wherever possible (using off the shelf Adam for optimization with a fixed learning rate, and fixing architectural hyperparameters like the projector dimensionalityy.
Settings for ImageNet-100 For ImageNet we more closely match the pre-training procedures of previous works. We use a batch size of 2048 and a smaller number of views for MMCR (4), and also use the full suite of augmentations from Zbontar et al. (77). For the sake of efficiency we train for a reduced number of epochs (200). For MMCR and SimCLR we modified the projector hidden dimensionality to be 4096 for the projector head, following the original work (14). For Barlow Twins we used the recommended 2-layer MLP with hidden and output dimensions of 8192, and set 5e 3, 3. however these hyperparameters were optimal for the full ImageNet dataset, and not neccesarrily for 100 We were unable to achieve better downstream performance using Re backbone than what has previously been reported in the literature for this dataset with a ResNet- 18 backhone, therefore we report the ResNet- 18 performance reported in (33). For SimCLR we use T = 0.1 which is the recommended setting for larger batch sizes.
Settings for ImageNet-1k: For ImageNet-Ir Ik we use mostly identical settings to ImageNet- 100, but we increased the capacity of the projector network (using 2 hidden layer MLP with hidden dimenisons of 8192 and output dimension of 512). We scaled the learning rate linearly with batch size: Ir= 0.6 x batch
256 Additionally we reduce the number of pretraining epochs 100. Finally for ImageNet-1k 1k we found that employing a momentum encoder slightly boosted downstream performance (around + +1% on ImageNet frozen-linear evaluation). Specifically, an identical encoding network and projector architecture is initialized with the same parameters as the initial "online" network, and during training the weights of this "momentum" network track a slowly moving average of the online network parameters (only the online network parameters are updated via gradient
19

---

descent). Each augmented view is passed through both the online and momentum networks, and the resultant embeddings are all averaged form the centroid vector for a particular image in the batch. We used momentum coefficient of 0.99.
Pre-training 16 A 100 GPUs using 8 views (our most compute intensive setting) takes approximately 32 hours.
E Details of Representational Analyses
E.1 Manifold Capacity Analysis
For each pre-trained model, we extract layer activations across the ResNet hicrarchy after a forward pass of set of images. For class manifold analysis, the set of images contain 10 classes, where each class has 100 examples. Augmentation manifolds instead have 100 exemplars with 100 examples each. Following (20), we take activations from all convolutional layers in ResNet-50 after ReLU non-linearity. The specific extracted layers highlighted in bold fonts are given by Table B The final analysis results averaged over five data samplings with different random seeds and random projections of intermediate features to lower-dimension spaces (default 5000 dimensions).
E.2 Gradient Coherence Analysis
In Fig. for each of the classes of CIFAR-10, we generate 100 batches of 32 augmentation manifolds of samples from a specific class (with 40 augmentations each). We then measure the gradient of the loss function for each batch during different stages of training. and compute the cosine similarity between every pair of gradients. Across all stages of training the mean cosine similarity between gradients generated from batches of the same class is larger than those from distinct classes (left column). This observation remains true when isolating the gradients of parameters from different stages in the resnet-50 hierarchy (center and right columns, respectively).
E.3 Manifold Subspace Alignment
For Fig. 4 we generated 100 samples from the augmentation manfiolds of 500 images in the CIFAR-10 dataset. We then measure the mean subspsace angle (left column), fraction of shared variance (middle column) and centroid cosine similarity between each pair of manifolds. The same procedure was used for generating the data for Fig. 5
Subspace Angle. Besides measuring the size and dimensionality of individual object manifolds we also wish characterize the degree of overlap between pairs of manifolds. For this, we measure the angle between their subspaces (47). which is generalization of the notion of angles that applies to subspaces arbitrary dimension.
Object manifolds will generally have a lower intrinsic dimensionality then the space in which they are embedded. Therefore, the data will have low variance along several of the principal vectors used to calculate the set of subspace angles, and so many of the principal angles will have little meaning. To address this limitation we also compute the shared variance between the linear subspaces that contain object manifolds.
Shared Variance.
F Implicit MMCR Effectively Reduces Augmentation Manifold Nuclear Norm
To test whether or not implicit manifold compression actually reduces the mean augmentation manifold nuclear norm, we can vary the value of X. Below we see the evolution both terms the loss for several different values of lambda during training on CIFAR-10. For these experiments the batch size was 64 and the number of augmentations per image was 4.0. As shown Fig. [ the level of compression of individual manifolds is nearly the same across all values of the parameter.
20

---

Table A Total of Extracted ResNet-50 Layers (in Bold) for MFTMA Analysis
Layer
Type
Conv2d Size (H W x C)
pixel
Input
None
Co
conv]
BatchNorm
x
[|] x 64
ReLU
[d]onv2d
[[]] x x 64
[1]
x 33
conv2_x
BatchNorm
x 3
3 64
x
ReLU
[1x 1 256
[d]Convdd
[[]] 1 x 128
[1]
x 3
conv3_x
BatchNorm
X 4
3 x x 128
x 4
ReLU
1 1 512
[d]Con22
[[]] 1 2567
[4]
x 3
conv4 x
BatchNorm
6
3 x 3 x 256
6
ReLU
[1x 1024
[]]oov2d
[[]] 1 x 5127
[
3
conv5_x x
BatchNorm
3
3 x 3 x 512
x 3
ReLU
x 2048
Centrold Nuclear
Manifold Nuclear
2,8
48.0
|C|
Ex| Zn
Lambdo 0.0
Lambds
40.0
Lambdo 0.5
1.5
Lambda 1.0
40.0
80.0
0.0
80.0
80.0
Epochs
Epochs
Figure 6: Validation loss values for different values of A
G Classification Evaluation Procedure
During pre-training all models were monitored with a k-nearest neighbor classifier (k=200) and checkpointed every 5 epochs. After pre-training. we trained linear classifiers on all checkpoints whose monitor accuracy was within 1% of the highest observed accuracy, and select the model that achieves the highest linear classification accuracy. Linear classifiers were trained using the Adam optimizer with batch size of 1024 and an initial learning rate 0. which decayed according to cosine scheduler over the course of 50 epochs. For the linear classifier training, at train time we use the same set of augmentations as during unsupervised pretraining, at test time we only center cropping and random horizontal flipping.
CIFAR and STL-10:
21

---

ImageNet-1kk100: For ImageNet datasets we closely followed the most widely adopted evaluation procedure. Following pre-training we freeze the encoder weights and train linear layer in a supervised fashion using SGD with batch size of 2048, learning rate of 1 1.6, and weight decay of le-6 for 50 epochs. During linear classifier training the only data augmentations are random cropping and random horizontal flips, and during evaluation inputs are center cropped.
Semi-Supervised: For semi-supervised evaluation we mostly follow the procedure outlined in Bardes et al. (7). We use the SGD optimizer with momentum of 0.9 and weight decay of le-6 and the standard cross entropy loss. The augmentation procedure was the same as described above. Because the linear classifier is being trained from scratch and the representation is being fine tuned the learning rate for the parameters of the backbone is scaled down by a factor of 10, and both learning rates (backbone and classifier) followed a cosine decay schedule for 20 epochs. We used batch size of 256 and swept the learning rate over [0.1, 0.3, 1.0] for each model.
Other Downstream Classification Tasks: Classifiers on these datasets were trained in a similar fashion to those trained on the CIFAR and STL-10 datasets. The only differences being that for each method and datasets we swept the batch size over [128, 256, 512, 1024] and the the initial learning rate over [3e-2, 3e-3, 3e-4], and the augmentation procedure matched the standard setup for ImageNet training (only random cropping and horizontal flipping for training, resizing and center cropping for evaluation).
H Training Metrics
In the Fig. 7 below we monitor the evolution of both the objective (second panel), the mean augmentation manifold nuclear norm, the centroid norm, and the mean centroid similarity evaluated on the test set over the course of training.
IZDll-
||C||.
Centroid Similarity
Centroid Norm
11.0
330.0
0.18
0.45
300.0
0.1
6.9-
9.2
0.35
0.05
6.8
7.0
0.02
100
150
200
120.0-
50
100
150
209
100
350
280
30
170
150
200
Epochs
Epochs
Epochs
Epochs
Figure 7: Evolution of various metries during training. Geometric measures are evaluated on a set of 200 manifolds, each defined by an image drawn from the CIFAR-I0 dataset, along with 16 augmentations. Shaded regions indicate a 95% confidence interval around the mean.
I Classification Performance on Smaller Datasets
In Table below we report the performance of both our method as well Barlow Twins and SimCLR when trained using ResNet-50 backbone on smaller datasets.
J Batch Size Dependence
One of the most cited drawbacks of contrastive SSL methods has been that strong performance on downstream tasks requires training with large batch sizes, while non-contrastive methods (c.g.. VICReg or Barlow Twins ()7 that place constraints on the cooss-corelaaiio/coovaaana of the embeddings are much more amenable to smaller batch training. It is also worth noting that the need for large batch sizes in contrastive methods can be alleviated in various ways, such as maintaining a memory bank (74) or employing a slowly updating momentum encoder (41). Given that our method is neither wholly contrastive nor non-contrastive (since it acts on the spectrum of the embedding matrix directly), we wondered howwould depend on training batch size. We pretrained on ImageNet-Il Ik using batch sizes of 256, 512, 1024, 2048, 4096 and evaluate the linear classification
matrices
22

---

Table Top-1 classification accuracies of linear classifiers for representations trained with various datasets and objective functions. Note: for Barlow Twins on ImageNet-100 100 we report the result from da Costa et (23) which uses a ResNet-18 backbone, as we were unable obtain better performance. For MMCR on ImageNet-100 we tested both 2 views (matched to baselines) and views, results are formatted (2-vie))((-view)
Method
CIFAR-10
CIFAR- 100
STL-10
ImageNet-
Barlow Twins (our repro.)
90.91
67.91
89.96
80.38*
SimCLR (our repro.)
92.22
70.04
91.11
79.64
MMCR (A 0.0)
93.53
69.87
90.62
81.52/82.88
MMCR (A 0.01)
93.39
70.94
90.77
81.28/82.56
accuracy for each. Encouragingly we observed only a modest decrease in performance for the smallest batch size tested. The results this sweep, comparison Barlow Twins and SimCLR, is shown in in Fig. 8 Note that for these runs used two views and the linear learning rate scaling as described in Appendix D Future work should endeavor to better understand the impact of various hyperparameters on the quality of learned representations.
important detail is that for this experiment did not employ momentum encoder when training MMCR. It is argued in He et al. (41) that the momentum encoder increases the effective minibatch size the slowly moving weights encode information from preceding batches. However here are interested explicitly interested in batch size dependence so we ablate this architectural confound (neither Zbontar al. (77); Chen et
(14) employ momentum encoders).
Top-1 1 Accuracy
-1.0
Barlow Twins
-3.0
MMCR
-..0
2048
Batch Size
Figure 8: Drop in top-1 performance relative to that of the best setting for three methods. Data for both Barlow Twins and SimCLR are copied from Zbontar et al. (77).
K Additional Details on BrainScore
Brain-Score evaluates model in terms of its ability to predict the measured responses of neurons images. We provide a brief introduction to the metric here (see (64) for a complete description). Let y IR denote the average response of single (biological) neuron to set of V training images and X E R be the response of K model neurons to the same images. Brain Score first solves the linear regression problem y X w for weights w. The model then predicts responses y set of held out images. Next the Pearson correlation coefficient between y and is calculated, and the score for a particular dataset is the median of the individual neuron predicitivities in said dataset. The mean of these scores is taken over different train-test splits of each dataset. The score for a total brain area (as shown in Table 2 is the mean over train-test splits and distinct datasets. The standard error of the means for each dataset (which are typically between le-3 and le-2) within an area are summed quadrature and divided by the number of datasets to produce the errors reported in table 2 In table 5 we split the brain area scores into individual datascts.
23

---

Model
V1.0
V1.1
V2.0
V4.0
V4.1
V4.2
MMCR
0.270
0.718
0.311
0.577
0.627
0,492
SimCLR
0.224
0.776
0.288
0.576
0.626
0.48
BYOL
0.274
0.727
0.291
0.585
0.626
0.48
MoCo
0.273
.726
0.293
0.57
0.629
0.492
Barlow
0.276
.721
0.293
0.568
0.626
0.493
SwAV
0.252
.723
0.296
0.568
0.614
0.469
Model
V4.3
IT.0
IT.1
IT.2
IT.3
MMCR
0.226
0.554
0.558
0.545
0.424
SimCLR
0.224
0.552
0.545
0.518
0.456
BYOL
0.216
0.55
0.545
0,516
0.41
MoCo
0.215
0.54
0.560
0.550
0.437
Barlow 0.221
0.545
0.547
0.518
0.412
SwAV
0.202
0.533
0.537
0.518
0.405
Table 5: Brain-score comparison of six self-supervised models, over 11 different electrophysiological
data sets recorded from macaque monkeys (64). Datasets V1.0 and V2.0 are from Freeman al. (30). V1.1 is from Marques et al. (53), and V4.0, V4.1, IT.0, and IT. are from Majaj et al. (51). (V4/IT) are the Sanghavilozwik20220 and SanghaviMurty datasets as denoted by brainscore.
L Additional Details on Spectral Properties
Participation Ratio We first extracted the 2048 dimensional feature vectors for each model in response to the images in the ImageNet validation set. Images were resized to 256 x 256 and then center cropped 224 224 following the setting in which the classifiers are tested. We resampled the resultant feature matrices with replacement 10 times independently for each model For each resampled dataset (of features) calculate the empirical covariance matrix, associated eigenspectra, and associated participation ratios (squared ratio of L to L2 norm of the eigenvcetors).
Decay Coefficient The spectra obtained using the procedure outlined above all decayed rapidly near tails (the least significant eigenvalues). To avoid undo bias from these tails when estimating the decay coefficients only considered the top 2000 eigenvalues. To estimate the decay coefficient fit regression line to the logarithm of the eigenvalues as a function the logarithm of their indexes. This fitting procedure was repeated across the bootstrapped spectra for each model to obtain standard errors of the mean. For all models the linear regression produced a strong fit to the data, with the minimum observed R² value being 0.95.
M Object Detection
To ensure that the representations obtained with MMCR are not hyperspecialized to classification tasks we also evaluated our highest performing model on object detection. We follow (7)) fine tuning the representation network with a Faster R-CNN head and C-4 backbone the VOCO7+12 dataset (training with the vrain+val split and evaluating on the VOCO7 test split). All settings except for the initial learning rate (which we we similarly used the detectron2 library for this evaluation. MMCR with 8 views and 100 epochs of pretraining produced an AP50 of 81.9 (this the mean over three independent fine tunes, the standard deviation was 0.2), demonstrating that the representeations generated by our method are not limited object recognition: Table ?? gives full results. For reference, representation trained using MoCo v2 for 200 epochs achieves an AP50 (the most common evaluation metric for this dataset) of 82.4 (in the table below we report the result we obtained for MoCo pretrained for 100 epochs for consistency).
0.12)
identical those from He al. (41), and
N Limitations
Time and compute limitations prevented from conducting exhaustive optimization of all design choices. For example we do not explore effects of varying the projector network width and depth. We additionally restrict the model to ResNet-50 encoding network pretrained on the ImageNet-Ik

---

Method
mAP
AP50
AP75
Barlow
53.1
80.9
57.7
SWAV
54.4
81.6
61.0
MMCR
54.6
81.9
60.6
SimCLR
54.7
81.7
60.2
MoCo v2
55.6
82.3
61.7
BYOL
56.0
82.3
62.0
Table 6: We follow 77), fine tuning the representation network for detection with a Faster R-CNN head and C-4 backbone on the VOCO7+12 dataset.
dataset for 100 epochs. This choice allows to make fair comparisons to several recently developed SSL methods with a modest compute budget, but precludes answering questions about how the method scales up to larger problems.
We also that although our objective function has favorable computational complexity compared to existing methods, the evaluation of our objective is not straightforward to distribute across machines. This is because we need compute the SVD over the centroid matrix, which requires gathering the outputs of distributed forward pass onto single machine. Efficient methods for the distributed computation of the SVD could help alleviate this issue the future.
Impact of Momentum Encoder
With all other settings fixed, we find that the use of momentum encoder confers small advantage in terms of classification performance. Below we report frozen linear evaluation accuracy for MMCR training and 8 views with and without momentum encoder.
Views
4 Views
8 Views
With Momentum Encoder 69.5
71.4
72.1
Without Momentum Encoder 68.4
70.2
71.5
Table 7: Shown are Top-1 accuracies under the linear evaluation protocol for representations trained with without a momentum encoder using the settings described above. appears that as the number of views increases the added diversity of positive sample representations conferred by the momentum encoder diminishes.
P MMCR Benefits from Longer Pretraining
To verify that MMCR benefits from longer pretraining we modified our setup to pretrain with two views and reduced our base learning rate to 0.4 and train for 1000 epochs. We additionally changed the hyperparameters for pretraining the linear classifier when evaluating this model (the classifier was trained with an learning rate of 0.3 and batch size of 2048 for 100 epochs). We did not perform any hyperparameter tuning for this setting, these results represent performance floor for 2-view MMCR.
25

---

Method
Top-1 1 Accuracy
SimCLR
69.3
MoCov2
71.1
SimSiam
71.3
SwAV (without multicrop)
71.3
MMCR (2 views)
73.0
Barlow Twins
73.2
BYOL
74.3
SwAV (with multicrop)
75.3
NNCLR
75.6
ReLICv2 69
77.1
Table 8: Shown are Top-1 accuracies under the linear evaluation protocol for representations trained for 1000 epochs using various self supervised learning frameworks.
26


Reviews:
[{"review_id": "WKEYfKSOUR", "reviewer": "Reviewer_moiT", "summary": "The authors convert recent results in manifold perceptron capacity calculations to a practical self-supervised learning algorithm. They then (1) provide one theoretical result, (2) show their method is predictive of primate visual cortex recordings, and (3) characterize some empirical properties of their method versus other methods.", "questions": "- Figure 5: The authors state \"the InfoNCE loss employed in Chen et al. (12) benefits when negative pairs are as dissimilar as possible, which is achieved when the two points lie in opposite regions of the same subspace\". If this is correct (and I believe it is), we should expect to see negative cosine similarities between distinct classes. But then the right subplot shows that SimCLR has positive cosine similarity (~0.3) between distinct classes. Could the authors please clarify?\n\n", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The conversion of manifold perceptron capacity to a practically implementable SSL algorithm is novel\n\n- This paper is quite thorough. It contains a little bit of everything: a new algorithm, one theoretical result for the algorithm, comparison with biological data, empirical study of how the algorithm differs from other SSL algorithms, eigenspectrum decay. Its experimental results are particularly thorough.", "weaknesses": "Ordered by location in the text, rather than prioritization:\n\n- Section 2.1 either (1) makes a subtle but important leap that I'm not sure I followed or (2) uses confusing pluralization. Lines 93 to 103 state we consider P manifolds (plural) and then discuss three quantities: (1) the manifold radius (singular), (2) the manifold dimension (singular), (3) the correlation between the manifold centroids (plural). Additionally, a subscript $M$ appears and I do not know what this $M$ refers to. I'm not trying to nitpick pluralization; rather, I'm confused whether each of the P manifolds has its own radius and its own dimension, or whether all P manifolds collectively have 1 radius and 1 dimension. If the former, how are each of the P manifolds' radii and dimensions combined to determine the threshold capacity of the manifold population? If the latter, what does linear separability mean for a monolithic grouping of the P manifolds? Or must all $P$ manifolds have the same radius and dimension, but are permitted to possess their own centroids? I would appreciate if the authors could clarify this section.\n\n- The move from Section 2.1 to Section 2.2 needs better motivation. Section 2.1 introduces the manifold capacity as $\\phi(R_M \\sqrt{D_M})$ with differentiable, closed-form expressions for both $R_M$ and $D_M$. I am then expecting that we'll be maximizing capacity by minimizing $R_M \\sqrt{D_M}$. But instead, we jump to Equation (2)'s, with a small inline connection to $\\phi(R_M \\sqrt{D_M})$, and are then told that the second term of Equation 2 doesn't matter and so we'll set its prefactor $\\lambda$ to 0. I think this section could be greatly improved if the authors emphasize & explain the inline approximation , and better motivate Equation 2. I would recommend either leaving out the second term of Eqn 2 altogether, or alternatively, clarifying why the second term _should_ be mentioned if it is to be immediately eliminated.\n\n- I'm unsure whether I should be persuaded by the 2D example of 2 centroids (Equations 3 and 4) . I do love attempting to build intuition, and I understand that no closed form expressions exists for singular values of an arbitrary matrix, so I commend the authors, but I'm unsure whether the intuition will hold in (a) higher dimensions and (2) with significantly more data i.e. more centroids. Even staying in 2D, what will MMR do if we have 3, 4, 5 etc. centroids? I might've guess something like the Thomson problem (https://en.wikipedia.org/wiki/Thomson_problem) but that can't be the case because 2 electrons choose antipodal points where MMCR chooses orthogonal points.\n\n- Moreover, upon another read, I realized the paragraph \"Compression by Maximizing Centroid Nuclear Norm Alone\" seems odd compared with Equation 2. The paragraph argues that the nuclear norm of $C$ alone is sufficient because it both (1) incentivizes making each manifold as tightly compressed as possible (i.e. each mean should have maximum norm, bounded above by 1) and (2) incentivizes making centroids pairwise orthogonal. I agree with this - the 2D case is clear and convincing. But if those two incentives are what we care about, why not directly use them as our loss? I'm specifically suggesting something like $-\\sum_b ||c_b||^2 + \\sum_{b, b'} (c_b \\cdot c_{b'})^2$? And if so, isn't that almost exactly the loss of TiCo (Zhu et al 2021/2022 https://arxiv.org/abs/2206.10698 ), specifically equation 6?  \n\n- With all SSL papers, there are many implementation decisions and hyperparameters that complicate interpreting the results. Line 205-208 caught my eye (\"We additionally employ a momentum encoder for ImageNet pre-training...\". While there is nothing wrong with using a momentum encoder, I would like to know what performance MMCR achieves with vs. without the momentum encoder. Could the authors please include this ablation? If I missed it, I apologize.\n\n- Please add some measure of uncertainty (e.g. 95% confidence intervals) to Figure 3 (unless such intervals already exist and are too small to be seen, in which case, please note that in the figure caption).\n\n- The authors write \"Elmoznino and Bonner (23) found that high dimensionality in ANN representations was associated with ability to both predict neural data.\" and later \"Additionally, Elmoznino and Bonner (23) report that high intrinsic dimensionality (as measured with the participation ratio of the representation covariance) is correlated with a representation’s ability to predict neural activity.\" The claim that participation ratio is highly correlated with neural predictivity was noted concurrently by Schaeffer et al. 2022 \"No Free Lunch from Deep Learning in Neuroscience\" at an ICML 2022 workshop (albeit in mouse MEC on navigation tasks, rather than primate visual cortex on vision tasks) and published at NeurIPS 2022. Then, soon after, Tuckute et al. 2022 \"Many but not all deep neural network audio models capture brain responses and exhibit hierarchical region correspondence\" released a preprint showing the same correlation between participation ratio and neural predictivity in human fMRI recordings of audio (Figure S6 in biorxiv v1; I'm too lazy to find the same plot in the more recent biorxiv v4). From an attribution perspective, I believe all three should be cited since they were all within ~1-3 months of one another. I also think citing all three will strengthen your paper since it points out the same phenomenon appears in multiple modalities (navigation, vision, audition), multiple species (rodent - can't remember mouse or rate, monkey, human) , multiple recording technologies.\n\nI realize that many of these \"weaknesses\" might be my own misunderstandings. If the authors could clarify, or lightly revise the paper to address these above points in a satisfactory manner, I would be happy to increase my score :)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very thorough review of our work! Below we address each point listed in the weaknesses:\n\n- *\"Section 2.1 ...\"*:   Apologies that this was not expressed more clearly: each of the manifolds is assumed to have the same dimensionality and radius, but each has its own centroid. We will clarify in the revision.\n\n- *\"The move from section 2.1 to section 2.2...\"*: The jump to Equation (2) is a bit abrupt (we left these steps out due to space constraints). The capacity is a monotonic function of the nuclear norm in lines 114-115.  We neglected to mention that its functional form is not well suited to gradient descent (evaluation requires numerical integration) but the monotonicity means it is largely irrelevant for the purpose of gradient-based optimization. We included the second term in Eq (2) because we were aiming to formulate a contrastive objective: an invariance term (the second) and a term to prevent representational collapse (the first). It was not initially obvious to us that the collapse prevention term (in tandem with the unit sphere constraint) naturally encourages invariance. We will revise the presentation to communicate the above sentiment “up-front,” and then modify Eq (2) to only include the first term. \n\n- *\"I'm unsure whether I should be persuaded by the 2D example... \"*: Thank you for carefully considering the implications of our proposed objective. We included the 2-D example to build intuition for two properties of the objective: it maximizes the angle between the centroids, and it also encourages augmentation manifold compression (indirectly, by maximizing the norm of centroids). When there are more centroids than dimensions, maximizing the nuclear norm encourages them to form a simplex equiangular tight frame (sETF) - note that this can only be achieved exactly for certain combinations of dimensionality and number of points.  For the 2D case, they would be equally distributed around  the unit circle.\n\n- \"Moreover, upon another read...\"*: The most important property of our objective is that it incentivizes global high dimensionality without relying on expectations over large numbers of pairwise comparisons (as required by TiCo and many other SSL methods). We will clarify this in the revised text.\n\n- *\"With all SSL papers...\"*: Thanks for the suggestion - we agree this is worth reporting. The momentum encoder conferred a small advantage in terms of downstream classification performance:\n\n|                             | 2 Views | 4 Views | 8 Views |\n|---------------------|---------|---------|-------------|\n| With Momentum Encoder |   69.5  |   71.4  |   72.1  |\n| Without Momentum Encoder | 68.4 |   70.2  |   71.5  |\n\n- *\"Please add some measure of uncertainty...\"*: Thank you for pointing out this omission. The figure does indeed have confidence intervals which are too small to be seen (due to the large sample size). We will mention this information to the figure caption. \n\n- *\"The authors write...\"*: Thank you for the pointers - we agree that these articles should be cited as well!\n\n- *\"Figure 5: The authors state\"*: We are visualizing the cosine similarity at the output of the encoder network, which is the learned representation used for downstream tasks. Because the encoder is a ResNet-50, the outputs are rectified and the minimum possible similarity is zero. If we instead look at the outputs of the projector network (where the loss is applied), your intuition is correct: the distribution of centroid similarities for SimCLR for distinct classes peaks below zero (see the figure included in the pdf uploaded along with our general response, we match the setting of Figure 5 except for we instead examine the outputs of the projector for SimCLR). \n\n\n "}, {"review_id": "njEsCZTRPq", "reviewer": "Reviewer_1Hrd", "summary": "The efficient coding hypothesis suggests sensory systems maximize mutual information between their inputs and the environment. A recent adaptation, \"manifold capacity\", calculates the number of linearly separable object categories, but is computationally intensive. The authors simplify this measure to directly optimize Maximum Manifold Capacity Representations (MMCRs). They show MMCRs are competitive with top results on self-supervised learning (SSL) recognition benchmarks. MMCRs differ from other SSL frameworks and may enhance class separability through manifold compression. On neural predictivity benchmarks, MMCRs prove competitive as models of the ventral stream, a part of the brain involved in object recognition.", "questions": "- An illustrative figure that intuitively demonstrate the intuition of singularity of the covariance matrix in terms of the objective function would be beneficial to the reader's understanding. \n- In the introduction, it might be better to highlight the contribution to the SSL more clearly. ", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe paper presents a novel method on SSL leveraging Manifold Capacity theory. The contribution is clear and precise. \n-\tThe paper connect SSL with Manifold Capacity theory with a effective theoretical formulation for the SSL objective. \n-\tThe paper empirically validates its claim by running large scale experiments and demonstrates competitive performance. \n-\tThe method mitigate the large batch / large dimension training bottleneck of the previous SSL method. \n-\tThe paper also evaluates on the neural data to show that its method can explain the neural data better. \n", "weaknesses": "-\tAlthough the initial results look very promising, the author didn’t include an error bar in Table-1. It would be beneficial to see the error bar to further eliminate the effect of chance. \n-\tThe paper posits that sensory systems maximize mutual information between their representations and the environment, based on the efficient coding hypothesis. However, it might lack a discussion on the biological plausibility of MMCRs. How feasible is it for biological sensory systems to implement MMCRs? How does this model account for the biological constraints mentioned in the efficient coding hypothesis? \n-\t[Minor] In the legend of Figure 2, the description indicates the top row showing the radius but the actual graph shows dimensionality.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of our paper. We respond to each of the listed weaknesses below:\n\n- *\"Although the initial results look very promising...\"*: We agree including a measure of uncertainty would help strengthen the paper. For ImageNet experiments repeated trainings of the linear classifier yields a standard deviation of \\~0.05, and for the smaller datasets we have observed slightly more variability (~0.1). We will update the table with these confidence intervals in the revised paper.\n\n- *\"The paper posits that sensory systems...\"*: Thank you for this thoughtful question. One of the key biological constraints is that not all of the information carried by a sensory stream can be faithfully encoded with limited resources. This constraint is applied in MMCR by encouraging invariance to augmentation variability. The learned representations are efficient in that they compress this source of variability while maximizing the discriminability over the images in the dataset. \n\n    Moreover, we speculate that the MMCR objective is better suited to biological implementation because it does not require a large number of pairwise comparisons to evaluate the objective. Instead the MMCR objective is a function of the summary statistics (singular values) of a global representation of the image manifold. Whether and how this could be optimized using neural circuits is of interest, but well beyond the scope of the current paper.\n\n- *\"In the legend of Figure 2...\"*: Thank you for bringing this to our attention, we will revise the caption in the updated paper.\n"}, {"review_id": "9eVqiiAZMl", "reviewer": "Reviewer_eLMY", "summary": "This paper proposes an alternative way (i.e., maximizing manifold capacity) to learn useful representation in a self-supervised manner. In particular, it maximizes the separability of the centroid manifold (i.e., negative samples), while the second term aims to minimize the nuclear norm among positive samples.\ntends to figure out crucial properties of self-supervised learning (SSL) methods, which promote good performance on downstream tasks. To reach this target, the authors propose a unifying conceptual ISSL framework. In particular, their main contributions contains: i) increasing the dimensionality of presentation and using asymmetric projection heads; ii) presenting a new non-contrastive ISSL objective; iii) applying non-linear probes.", "questions": "The questions are corresponding to the above main concerns:\n- What is the relation or difference between the proposed objective and [1]?\n- Please add more theoretical or empirical analyses on the necessity of implicitly minimizing object manifold compression.\n- Please add more theoretical or empirical comparisons between MMCR and the nuclear norm-based methods [2,3,4].\n- Please add more discussion about the results in Figure 2.", "limitations": "Yes, the authors have addressed the societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper introduces manifold capacity in self-supervised learning (SSL). This idea is somewhat novel and provides an alternative way to avoid the collapse issue in SSL. Extensive experiments conducted on nearly 7 datasets demonstrate that the proposed method achieves comparable performance to existing state-of-the-art (SOTA) methods. Furthermore, empirical analysis of Maximum Manifold Capacity Representations (MMCRs) reveals distinct characteristics compared to existing approaches.", "weaknesses": "However, my main doubts/concerns regarding the paper are the following:\n\n- As shown in Line 138-141, the first term aims to maximize the separability of negative samples, while the second term aims to minimize the nuclear norm among positive samples. The motivation behind the proposed loss is similar to [1], which pursues \"the alignment of features from positive pairs and the uniformity of the induced distribution of the normalized features on the hypersphere.\" It would be better to provide further discussion in relation or difference to [1].\n- The observation in Figure 6 is interesting. However, this result raises the question of whether implicitly minimizing object manifold compression is truly necessary, as the second term with nuclear norm does not impact the mean manifold nuclear norm. Therefore, conducting an ablation study between these two terms on several downstream tasks is essential to further investigate their effects.\n- The proposed MMCR method shares strong connections with nuclear norm-based approaches [2,3,4], especially in relation to [2]. However, the current study lacks either theoretical or empirical comparisons between MMCR and these nuclear norm-based methods.\n- In Figure 2, the Mean Filed Manifold Capacity Analysis shows that existing SOTA methods with different objectives all explicitly learn representations with large radii but low dimensionality. These results may question the necessity of the proposed method.\n- In Figure 5, the centroid similarity distributions of MMCR for both same class and distinct classes appear to be generally smaller than those of existing methods (Barlow Twins and SimCLR). Therefore, this comparison may be somewhat unfair. To ensure a fair comparison, it would be better to normalize the centroid similarities (e.g., by computing z-scores) for each method and then compare the normalized distribution across different methods.\n\nMinor:\n- The experimental setting described in Section 3.2 is unclear, which could potentially result in unfair comparisons between MMCR and other state-of-the-art (SOTA) methods. If MMCR employs multi-crop augmentation while Barlow Twins and SimCLR only use two views, the comparison would be unfair in Figure 2 and 5.\n- The authors might unintentionally modify the default LaTeX template, which results in incorrect formatting of the citations. In particular, (1) -> [1] or (1;2) -> [1,2].\n- Incorrect format of theorems, lemmas, and proofs is detrimental to this paper since it makes the paper appears informal.\n- Almost all equations lack commas or full stops at the end.\n- Please standardize the usage of either \"Fig. x\" or \"Figure x\" for consistency.\n- Figure 2: \"see E\" -> \"see appendix E\". Table 2: \"see L\" -> \"see appendix L\".\n\n\n[1] T. Wang, et al., \"Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere\", ICML 2020.\n\n[2] Y. Wang, et al., \"A Low Rank Promoting Prior for Unsupervised Contrastive Learning\", TPAMI 2022.\n\n[3] O. Hénaff, et al., \"The Local Low-dimensionality of Natural Images\", ICLR 2015.\n\n[4] J. Lezama, et al., \"OLÉ: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learning\", CVPR 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our submission. Below we respond to each of the listed weaknesses:\n\n- *\"As shown in Line 138-141...\"*: Many self-supervised learning methods share these core motivations, and [1] in particular demonstrates that the logarithm of the average pairwise Gaussian potential is optimized by a uniform distribution on the hypersphere. The unique aspect of our method is that it doesn’t rely on pairwise comparisons in order to achieve the desired properties in the global representation, but rather optimizes a population level metric directly. We will make this point more explicit in the revision.  \n\n- *\"The observation in Figure 6 is interesting...\"*: We feel that Figure 6 does not imply the necessity of implicit manifold compression, but rather demonstrates the effectiveness of doing so. This is an important interpretational difference, as using explicit compression substantially increases the computational complexity of evaluating the objective. In practice we observe that including this term in the loss significantly increases the run time, and on CIFAR experiments we did not observe any benefit in terms of the quality of the learned representation. We will emphasize this practical advantage of implicit compression in the revised paper.\n\n- *\"The proposed MMCR method shares strong connections\"*: Our formulation is related to [2, 3, 4] in the use of nuclear norm (and we’ve cited all of them for this). However, each of them differ significantly enough from our approach that a more detailed comparison did not seem warranted.  Specifically:\n    - [3] is primarily focused on learning a set of linear filters in an autoencoder architecture. The substantial difference in experimental setting prevents a useful empirical comparison, but we will include a more thorough discussion of this paper in our revised related works section.\n    - [2, 4] employ low rank (nuclear norm) regularizers to supplement their objective functions. In contrast, our objective directly maximizes rank. In addition, [4] differs in that it employs supervised training. We do think a more direct comparison with [2] could be interesting, i.e. we could compare the spectra of augmentation manifolds and the global dimensionality of the learned representations. A fair comparison will require matching the pre-training settings (specifically the amount of pretraining, LORAC trains for 200 epochs), and compute limitations have prevented us from completing these experiments during the rebuttal period. We will however report preliminary results during the upcoming discussion period and in include such analyses in the final paper.  \n\n- *\"In Figure 2, the Mean Filed Manifold Capacity Analysis shows...\"*: For methods that perform strongly in terms of linear classification, capacity analysis provides further insight as to whether this is accomplished with class manifolds with low dimensionality or low radius. Figure 2 demonstrates that MMCR yields high capacity by reducing dimensionality at the expense of increasing radius, relative to SOTA methods.  \n    \n    Your comment helped us to realize that the formatting of Figure 2 obscures this point, since the differences only emerge at the tail end of the ResNet hierarchy. In the revision, we’ll include a table of values taken from the final representation layer, and move the figure to the appendix.\n\n- *\"In Figure 5, the centroid similarity distributions...\"*: We would appreciate it if the reviewer could clarify this concern. The distribution of similarities has a smaller mean than the alternatives, which is the message we were trying to communicate. If the distributions for each method were z-scored, they would differ only in higher order (shape related) properties, which could be interesting but is not central to the argument being made.\n\n- *\"The experimental setting described in Section 3.2...\"*: All the model evaluations use the same number of augmentations - we will clarify this point in the revised paper. It is true that during *pre-training*, SimCLR and Barlow Twins use 2-view augmentations.  We think this is fair, since these were the procedures used in their respective original implementations (in addition, it is not clear how Barlow Twins would be adapted to the multi-view setting).  We’ll clarify these points in the revised text.\n\n- *[Minor]*: Thank you for bringing these formatting issues to our attention, these will be corrected in the revised text.\n\n\n\n"}, {"review_id": "H0iNsRRHvm", "reviewer": "Reviewer_g5Gb", "summary": "This paper proposes a new SSL objective to directly optimize manifold capacity (num of categories represented in a linearly separable fashion). It is found that the resulting representations can support linear classification well. It is also shown that the representations are somewhat sample efficient (reasonable performance in 1% and 10% settings, Food, Flowers, etc). It is also found that that the representations are slightly better predictors of neuronal representations in V2 compared to baselines.", "questions": "1. Can the authors please make their discussion on relevance to neuroscience more specific? \n2. Can the authors please expand on their evaluations along any of the directions listed in weaknesses?\n", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The description of the objective function and simplifying assumptions are clear\n2. The evaluations (e.g. Tables 1, 2) and analyses (e.g. Figs, 4, 5) are clear.\n3. The eigenspectrum alpha being close to 1 is an interesting finding. \n\n\n\n", "weaknesses": "[Limited evaluations, objective too specific?]\n\nThere is some novelty in the specific objective function used, but evaluation is limited. For example, by more directly optimizing for linear separability has there been a loss in generality of representations, e.g. on segmentation or depth estimation? The detection results in the appendix appear to be consistent with MMCR being more specialized for linear classification. Also unclear how well does the objective work when the data are more complex (e.g. COCO)? Imagenet normally contains few objects and the correct category is very often an object in the center. Moreover, the architecture explored is also limited to ResNet-50 and is not competitive with more recent SSL methods (e.g. RELICv2, NNCLR).\n\nAnother important limitation is that evaluations are for SSL models only trained for 100 epochs. Different SSL methods converge to good representations at different speeds, so limiting evaluation to 100 epochs is does not give us an adequate understanding of various methods - this limitation applies to comparisons on ml benchmarks as well as neuronal data.\n\n[Purpose of comparing to neural data unclear.]\n\nMMCR has slightly better predictive power in V2, and the eigenspectrum alpha is close to 1. These are interesting empirical observations, but the discussion does not appear to really engage with these. In the discussion (l316) it is claimed that future work should go beyond accounting for current data. It is not clear to me how the current work provides an account of current data? What are the precise claims/hypotheses being made and how do the experiments support it? What does it entail for our understanding of the brain that was not previously known?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our work, our responses to the listed weaknesses are below:\n\n- *[Limited evaluations, objective too specific?]*: We agree that evaluating on more tasks can strengthen the assessment of a learned representation. We have run additional evaluations of baseline models trained in the same setting (100 epochs), which show that MMCR performs comparably to other SOTA methods on object detection and outperforms the supervised baseline. These evaluations are on the VOC dataset using the same procedure described in the appendix of our original submission:\n\n|             |  mAP  |  AP50  |  AP75  |\n|-------------|-------|--------|--------|\n| Supervised  |  53.5 |  81.3  |  58.8  |\n| MMCR        |  54.6 |  81.9  |  60.6  |\n| SimCLR      |  54.7 |  81.3  |  60.2  |\n| SwAV+multicrop | 54.4 |  81.6  |  61.0  |\n| BYOL        |  56.0 |  82.3  |  62.0  |\n| Barlow Twins |  53.1 |  80.9  |  57.7  |\n| MoCo v2* (200 epochs) | 57.0 |  82.4  |  64.0  |\n    \nResults for MoCo are reported from the official paper repository.\n\n- *\"Another important limitation is that evaluations are for SSL models only trained for 100 epochs...\"*: We chose this setting because implementations and models are readily available using ResNet-50 with 100 epochs of pretraining, allowing us to compare to many existing methods directly while operating with a limited compute budget. We do agree that it would be valuable to demonstrate that our method’s strong performance is not limited to 100 epoch pretraining, and have now launched experiments using 1000 epochs and 2 views. This experiment is still running at the time of posting but we will include results in the updated paper.\n\n   \n    It is worth noting that NNCLR does report results for linear evaluation using 100 epochs of pretraining, where they achieve 69.4%    accuracy, slightly underperforming MMCR with two views.\n\n\n- *[Purpose of comparing to neural data unclear.]*: The results show that MMCR is competitive in explaining physiological data from area V2, but the BrainScore evaluation does not provide a means of assessing/interpreting the detailed nature of the fits. Our table indicates that, although different objective functions produce representations with very different spectral properties (as evidenced by the spread in participation ratio and alpha decay coefficients), there is much less spread in their neural predictivity. \n\n\n\n  "}, {"review_id": "w4u1KTLhMg", "reviewer": "Reviewer_9aX7", "summary": "This paper develops a form that facilitates direct optimization, use it to learn Maximum Manifold Capacity Representations (MMCRs), and demonstrate that these are competitive with state-of-the-art results on standard self-supervised learning (SSL) recognition benchmarks. Empirical analyses reveal important differences between MMCRs and the representations learned by other SSL frameworks.", "questions": "NA", "limitations": "NA", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Inspired by manifold capacity theory, the proposed self-supervised learning algorithm is efficient to learn and requires neither large batch size nor large embedding dimension. \n\nThis work also provides very interesting empirical results which are well aligned with the proposed theorem. \n\nBesides, the paper is well-presented.", "weaknesses": "Since I am not in this area, the things mentioned in this paper are very convincing to me. I cannot point out very specific weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive evaluation of our work. Please let us know if any questions arise during the discussion period and we will be happy to clarify. "}]

Author rebuttal:
We would like to thank the reviewers for their careful consideration of our submission. Their feedback has helped to identify several simple ways to improve our paper. Below we summarize some key revisions to be made in the case our work is accepted:

- The framing/introduction of our objective will be clarified. In particular we will change Equation (2) to only include the first term, as we immediately drop the second term in the following experiments.

- Relatedly, we will better motivate the use of “implicit” compression by noting the practical advantage (in terms of runtime).

- We will include a more thorough evaluation on object detection performance on the Pascal VOC dataset in the main text, to demonstrate that though our method is inspired by a theoretical analysis of linear classification the learned representations perform well on more complex computer vision tasks.

- We will better characterize the contribution of our neural data experiments, which show that different SSL methods trained in matched settings produce representations with diverse spectral/geometric properties but surprisingly similar neural predictivity.



Meta-review: The paper presents a practical SSL algorithm which optimizes manifold capacity. The resulting model can perform well on linear-separable classification and are competitive with recognition and neural predictivity benchmarks. The approach is interesting and novel, well presented and justified, and well validated through evaluation and analysis. Furthermore, the approach to representation learning is interesting and stands out in comparison to popular SSL methods. I therefore recommend the paper to be accepted as a poster to NeurIPS.
Decision: Accept (poster)