Title: Learning Efficient Coding of Natural Images with Maximum Manifold Capacity Representations
Venue: NeurIPS 2023
Accepted: yes
Keywords: computational neuroscience

theoretical neuroscience

efficient coding

representation geometry

neural manifolds

self-supervised learning

statistical physics of learning
Abstract: The efficient coding hypothesis proposes that the response properties of sensory systems are adapted to the statistics of their inputs such that they capture maximal information about the environment, subject to biological constraints. While elegant, information theoretic properties are notoriously difficult to measure in practical settings or to employ as objective functions in optimization. This difficulty has necessitated that computational models designed to test the hypothesis employ several different information metrics ranging from approximations and lower bounds to proxy measures like reconstruction error. Recent theoretical advances have characterized a novel and ecologically relevant efficiency metric, the ``manifold capacity,” which is the number of object categories that may be represented in a linearly separable fashion. However, calculating manifold capacity is a computationally intensive iterative procedure that until now has precluded its use as an objective. Here we outline the simplifying assumptions that allow manifold capacity to be optimized directly, yielding Maximum Manifold Capacity Representations (MMCR). The resulting method is closely related to and inspired by advances in the field of self supervised learning (SSL), and we demonstrate that MMCRs are competitive with state of the art results on standard SSL benchmarks. Empirical analyses reveal differences between MMCRs and representations learned by other SSL frameworks, and suggest a mechanism by which manifold compression gives rise to class separability.  Finally we evaluate a set of SSL methods on a suite of neural predicitivity benchmarks, and find MMCRs are higly competitive as models of the ventral stream.

Efficient Coding of Natural Images using Maximum Manifold Capacity Representations

Abstract

The efficient coding hypothesis posits that sensory systems are adapted to the statistics of their inputs, maximizing mutual information between environmental signals and their representations, subject to biological constraints. While elegant, information theoretic quantities are notoriously difficult to measure or optimize, and most research on the hypothesis employs approximations, bounds, or substitutes (e.g., reconstruction error). A recently developed measure of coding efficiency, the 'manifold capacity', quantifies the number of object categories that may be represented in a linearly separable fashion, but its calculation relies on a computationally intensive iterative procedure that precludes its use as an objective. Here, we simplify this measure to a form that facilitates direct optimization, use it to learn Maximum Manifold Capacity Representations (MMCRs), and demonstrate that these are competitive with state-of-the-art results on current self-supervised learning (SSL) recognition benchmarks. Empirical analyses reveal important differences between MMCRs and the representations learned by other SSL frameworks, and suggest a mechanism by which manifold compression gives rise to class separability. Finally, we evaluate a set of SSL methods on a suite of neural predictivity benchmarks, and find MMCRs are highly competitive as models of the primate ventral stream.

1 Introduction

Biological visual systems learn complex representations of the world that support a wide range of cognitive behaviors, without relying on a large number of labelled examples. The efficient coding hypothesis (8; 65) suggests that this is accomplished by adapting the sensory representation to the statistics of the input signal, so as to reduce redundancy or dimensionality. Visual signals have several clear sources of redundancy. They evolve slowly in time, since temporally adjacent inputs typically correspond to different views of the same scene, which in turn are usually more similar than views of distinct scenes. Moreover, the variations within individual scenes often correspond to variations in a small number of parameters, such as those controlling viewing and lighting conditions, and are thus inherently low dimensional. Many previous results have demonstrated how the computations of neural circuits can be seen as matched to such structures in naturalistic environments (48; 29; 65; 17) Studies in various modalities have identified geometric structures in neural data that are associated with behavioral tasks (10; 25; 43; 33; 56), and explored metrics for quantifying these structures.

The recent development of 'manifold capacity theory' provides a more explicit connection between the geometry (size and dimensionality) of neural representations and their coding capacity (18). This theory has been used to evaluate efficiency of biological and artificial neural networks across modalities (17; 24; 66; 52). However, usage as a design principle for building model representations has not been explored.

Motivated by these observations, we seek to learn representations in which manifolds containing different views of the same scene are both compact and low-dimensional, while manifolds corresponding to distinct scene are maximally separated. Specifically:

We develop a form of Manifold Capacity that can be used as an objective function for learning.

We demonstrate that a Maximum Manifold Capacity Representation (MMCR) supports high-quality object recognition (matching the SoTA for self-supervised learning), when evaluated using the standard linear evaluation paradigm (i.e., applying an optimized linear classifier to the output of the self-supervised network) (14).

Through a analysis of internal representations and learning signals, we analyze the underlying mechanism responsible for the emergence of semantically relevant features from unsupervised objective functions.

We validate the effectiveness of MMCR as a brain model by comparing its learned representations against neural data obtained from Macaque visual cortex.

Our work thus leverages normative goals of representational efficiency to obtain a novel model for visual representation that is both effective for recognition and consistent with neural responses in the primate ventral stream.

1.1 Related Work

Geometry of Neural Representations. Previous work has sought to characterize how representational geometry, often measured through spectral quantities like the participation ratio (the squared ratio of the l 1 and l 2 norms of the eigenvalues of the covariance matrix), shapes different aspects of performance on downstream tasks (32). Elmoznino and Bonner (27) found that high dimensionality in ANN representations was associated with ability to both predict neural data and generalize to unseen categories. Stringer et al. (67) observed that the spectrum of the representation of natural images in mouse cortex follows a power law with a decay coefficient near 1, and Agrawal et al. (2) report that (in artifical representations) the proximity of the spectral decay coefficient to one is an effective predictor of how well a representation will generalize to downstream tasks.

Self-Supervised Learning. Our methodology is related to (and inspired by) recent advances in contrastive self-supervised representation learning (SSL), but has a distinctly different motivation and formulation. Many recent frameworks craft objectives that are designed to maximize the mutual information between representations of different views of the same object (58; 14; 58; 68; 4)). However, estimating mutual information in high dimensional feature spaces is difficult (9), and furthermore it is not clear that closer approximation of mutual information in the objective yields improved representations (71) By contrast, capacity measures developed in spin glass theory (35; 1) are derived in the 'large N (thermodynamic) limit'and thus are intended to operate in the regime of large ambient dimension (18; 5). We examine whether one such measure, which until now had been used only to evaluate representation quality, is also useful as an objective function for SSL.

Many SSL methods minimize the distance between representations of different augmented views of the same image while employing constraints to prevent collapse to trivial solutions (e.g., repulsion of negative pairs (14), or feature space whitening (77; 7; 28)). The limitations of using a single pairwise distance comparison have been demonstrated, notably in the development of the 'multi-crop' strategy implemented in SwAV (13) and in the contrastive multiview coding approach Tian et al. (68). Our approach is based on the assumption that different views of an image form a continuous manifold that we aim to compress. We characterize each set of image views with the spectrum of singular values of their representations, using the nuclear norm as a combined measure of the manifold size and dimensionality.

The nuclear norm has been previously used to infer or induce low rank structure in the representation of data (42; 72; 49). In particular, Wang et al. (72) use it as a regularizer to supplement an InfoNCE loss. Our approach represents a more radical departure from the traditional InfoNCE loss, as we detail below. Rather than pair a low-rank prior with a logistic regression-based likelihood, we make the more symmetric choice of employing a high rank likelihood. This allows the objective to explicitly discourage dimensional collapse, a well known issue in SSL (45).

2 Maximum manifold capacity representations

Low Capacity

ReduceDimensionality

BeforeTraining

After Training

Augmentation

Manifold on

Sphere

ReduceRadius

2.1 Manifold Capacity Theory

Consider a set of P manifolds embedded in a feature space of dimensionality D , each assigned a class label. Manifold capacity theory is concerned with the question: what is the largest value of P D such that there exists (with high probability) a hyperplane separating a random dichotomy (22; 34)? Recent theoretical work has demonstrated that there exists a critical value, dubbed the manifold capacity α C , such that when P D < α C the probability of finding a separating hyperplane is approximately 1 . 0 , and when P D > α C the probability is approximately 0 . 0 (18). The capacity α C can be accurately predicted from three key quantities: (1) the manifold radius R M , which measures the size of the manifold relative to the distance of its centroid from the origin, (2) the manifold dimensionality D M which quanitifies the number of dimensions along which a manifold has significant extent, and (3) the correlation of the manifold centroids. When the centroid correlation is low the manifold capacity can be approximated by ϕ ( R M √ D M ) where ϕ ( · ) is a monotonically decreasing function.

For manifolds of arbitrary geometry, the radius and dimensionality may be computed using an iterative process that alternates between determining the set of 'anchor points' on each manifold that are relevant for the classification problem, and computing the statistics of random projections of these anchor points (20). This process is both computationally costly and non-differentiable, and therefore unsuitable for use as an objective function. For more detail on the general theory see Appendix C. However, if the manifolds are assumed to be elliptical in shape, then both radius and dimensionality may be expressed analytically:

where the λ 2 i are the eigenvalues of the covariance matrix of the manifold. For comparison, when computing these values for a set of 100 128-D manifolds with 100 points sampled from each, the use of the analytical expression is approximately 500 times faster (in 'wall-clock time') than the general iterative procedure.

Using these definitions for manifold radius and dimensionality we can write the capacity as α C ≈ ϕ ( ∑ i σ i ) where σ i are the singular values of a matrix containing points on the manifold (equivalently, the square roots of the eigenvalues of the covariance matrix). In this form, the sum is the L 1 norm of the singular values, known as the Nuclear Norm of the matrix. When used as an objective function, this measure favors sparse solutions (i.e., those with a small number of non-zero singular values) corresponding to low dimensionality. It is worth comparing this objective to another natural candidate for quantifying size: the determinant of the covariance matrix. The determinant is equal to the product

of the eigenvalues (which captures the squared volume of the corresponding ellipsoid), but lacks the preference for lower dimensionality that comes with the Nuclear Norm. Specifically, since the determinant is zero whenever one (or more) eigenvalue is zero, it cannot distinguish zero-volume manifolds of different dimensionality. Lossy coding rate (entropy) has also been used as a measure of compactness (76), which simplifies to the log determinant of the covariance matrix under a Gaussian model (50). In this case, the identity matrix is added to a multiple of the feature covariance matrix before evaluating the determinant, which solves the dimensionality issue described above.

2.2 Optimizing Manifold Capacity

Now we construct an SSL objective function based on manifold capacity. For each input image (notated as a vector x b ∈ R D ) we generate k samples from the corresponding manifold by applying a set of random augmentations (each drawn from the same distribution), yielding manifold sample matrix ˜ X b ∈ R D × k . Each augmented image is transformed by a Deep Neural Network, which computes nonlinear function f ( x b ; θ ) parameterized by θ , and the d -dimensional responses are projected onto the unit sphere yielding manifold response matrix Z b ∈ R d × k . The centroid c b is approximated by averaging across the columns (response vectors). For a set of images { x 1 , ..., x B } we compute normalized response matrices { Z 1 , ..., Z B } and assemble their corresponding centroids into matrix C ∈ R d × B .

Given the responses and their centroids, the MMCR loss function can be written simply:

where || · || ∗ indicates the nuclear norm. The loss explicitly maximize the extent of the 'centroid manifold', which interestingly is sufficient to learn a useful representation. Concretely, maximizing || C || ∗ implicitly minimizes the extent of each individual object manifold as measured by || Z b || ∗ . We build intuition for this effect below.

Compression by Maximizing Centroid Nuclear Norm Alone. Each centroid vector is a mean of unit vectors, and thus has a norm that is linearly related to the average cosine similarity of those unit vectors. Specifically,

Here z b,i denotes the representation of the i th augmentation of x b . We can gain further insight by considering how the distribution of singular vectors of a matrix depends on the norms and pairwise similarities of the constituent column vectors. While no closed form solution exists for the singular values of an arbitrary matrix, the case where the matrix is composed of two column vectors can provide useful intuition. If C = [ c 1 , c 2 ] , Z 1 = [ z 1 , 1 , z 1 , 2 ] , Z 2 = [ z 2 , 1 , z 2 , 2 ] , the singular values of C and Z i are:

So, || σ ( C ) || 1 = || C || ∗ is maximized when the centroid vectors have maximal norms (bounded above by 1, since they are the centroids of unit vectors), and are orthogonal to each other. As we saw above the centroid norm is a linear function of within-manifold similarity. Similarly, || σ ( Z i ) || 1 = || Z i || ∗ is minimized when the within-manifold similarity is maximal. Thus the single term || C || ∗ encapsulates both of the key ingredients of a contrastive learning framework, and we will demonstrate below that simply maximizing || C || ∗ is sufficient to learn a useful representation. This is because the compressive role of 'positives' in contrastive learning is carried out by forming the centroid vectors, so the objective is not positive-free. For example, if only a single view is used the objective lacks a compressive component and fails to produce a useful representation. In Appendix F we demonstrate empirically that this implicit form effectively reduces || Z b || ∗ by comparing to the case where || Z b || ∗ is minimized explicitly. So, all three factors which determine the manifold capacity (radius,

dimensionality, and centroid correlations) can be succinctly expressed in an objective function with a single term, -|| C || ∗ .

Computational Complexity. Evaluating the loss for our method involves computing a singular value decomposition of C ∈ R d × B which has complexity O ( Bd × min ( B,d )) , where B is the batch size and d is the dimensionality of the output. By comparison, contrastive methods that compute all pairwise distances in a batch have complexity O ( B 2 d ) and non-contrastive methods that involve regularizing the covariance structure have complexity O ( Bd 2 ) . Additionally, the complexity of our method is constant with respect to the number of views used (though the feature extraction phase is linear in the number of views), while pairwise similarity metrics have complexity that is quadratic in the number of views. It is also worth noting that doing implicit compression by maximizing || C || ∗ offers an advantage in computational complexity relative to explicit compression. This is because evaluating a term such as ∑ B b =1 || Z b || ∗ has computational complexity of O ( B 2 d × min ( B,d )) .

2.3 Conditions for Optimal Embeddings

Recently HaoChen et al. (39) developed a framework based on spectral decomposition of the 'population augmentation graph', which provides theoretical guarantees for the performance of selfsupervised learning on downstream tasks under linear probing. This work was extended to provide insights into various other SSL objectives by Balestriero and LeCun (6), and we show below that leveraging this approach can lead to explicit conditions for the optimality of representation under our proposed objective as well.

Given a dataset X ′ = [ x 1 , ..., x N ] T ∈ R N × D ′ we construct a new dataset by creating k randomly augmented views of the original data, X = [ view 1 ( X ′ ) , ..., view k ( X ′ )] ∈ R Nk × D . The advantage of doing so is that we can now leverage the knowledge that different views of the same underlying datapoint are semantically related . We can express this notion of similarity in the symmetric matrix G ∈ { 0 , 1 } Nk × Nk with G ij = 1 if augmented datapoints i and j are semantically related (and G ii = 1 as any datapoint is related to itself). We can normalize G such that its rows and columns sum to 1 (so rows of G are k -sparse with nonzero entries equal to 1 /k ).

Now let Z ∈ R Nk × d be an embedding of the augmented dataset. Then we have GZ = [ C , ..., C ] T where C is the matrix of centroid vectors introduced above, and the number of repetitions of C is k . Then because σ ([ C , ..., C ]) = √ k σ ( C ) we can write MMCR loss function as,

This connection allows us to make the following statements about the optimal embeddings Z under our loss, which we prove in Appendix A:

Theorem: Under the proposed loss, the left singular vectors of an optimal embedding, Z ∗ , are the eigenvectors of G , and the singular values of Z ∗ are proportional to the top d eigenvalues of G .

3 Results

Architecture. For all experiments we use ResNet-50 (40) as a backbone architecture (for variants trained on CIFAR we removed max pooling layers). Following Chen et al. (14), we append a small perceptron to the output of the average pooling layer of the ResNet so that z i = g ( h ( x i )) , where h is the ResNet and g is the MLP. For ImageNet-1k/100 we used an MLP with dimensions [8192 , 8192 , 512] and for smaller datasets we used [512 , 128] .

Optimization. We employ the set of augmentations proposed in (38). For ImageNet we used the LARS optimizer with a learning rate of 4.8, linear warmup during the first 10 epochs and cosine decay thereafter with a batchsize of 2048, and pre-train for 100 epochs. Note that though we report results using a default batch size of 2048, a batch size as small as 256 can be used to obtain reasonable results (1.2% reduction compared to batch size 2048 - see Appendix J for a sweep over batch size). We additionally employ a momentum encoder for ImageNet pre-training (all views are fed through an online network and an additional network whose parameters are a slowly moving average of the online network and all embeddings of each image are averaged to form centroids). We found the use of a momentum encoder provided a small advantage in terms of downstream classification

performance, see Appendix O for this ablation. For smaller CIFAR-10 we used a smaller batch size, many more views (40), and the Adam optimizer with fixed learning rate. See Appendix D for exact details.

3.1 Transfer to Downstream Tasks

We used a standard linear evaluation technique, freezing the parameters of the encoding network and training a linear classifier with supervision (14), to verify that our method extracts semantically relevant features from the data. We also perform semi-supervised evaluation, where all model parameters are fine tuned using a small number of labelled examples, and also check whether the learned features generalize to three out-of-distribution datasets: Flowers-102, Food-101, and the Describable Textures Dataset (57; 11; 19) . The results are summarized in Table 1 and details of the training of the linear classifier are provided in Appendix G. Finally we also evaluate our best model in and each of the baselines on object detection on the VOC07 dataset. We followed (41; 77), fine tuning the representation networks for detection with a Faster R-CNN head and C-4 backbone using the 1x schedule. MMCR achieved a mean average precision (mAP) of 54.6 and the baseline method performance ranged from 53.1 to 56.0, demonstrating that though our framework was inspired by a theory of classification the learned features do generalize to other vision tasks. See Appendix M for detailed results.

Note that all included models were trained using the same backbone architecture (ResNet-50), dataset (ImageNet-1k), and number of pretraining epochs (100; we briefly explore the impact of longer pretraining in Appendix P ). The results for networks pretrained on smaller datasets can be found in the Appendix I.

3.2 Analyses of Learned Representations

We next conduct a series of experiment to elucidate the differences between representations learned with different SSL procedures, and suggest a mechanism by which augmentation manifold compres-

sion gives rise to class separability. To reduce the computational requirements, these analyses are carried out on models trained on the CIFAR-10 dataset.

AugmentationManifolds

Class Manifolds

Dimension

50.0

70.0

60.0

40.0

50.0

30.0

40.0

30.0

20.0

20.0

1.4

Radius

1.2

1.3

1.1

1.2

1.1

1.0

1.0

0.9

0.9

0.1

Capacity

0.12

三

BarlowTwins

0.1

0.08

SimCLR

0.08

0.06

MMCR

0.06

0.04

0.04

0.02

ResNetHierarchy

ResNetHierarchy

Pixels

Representation

Pixels

Representation

Mean Field Theory Manifold Capacity. In Fig. 2 we show that our representation, which is optimized using an objective that assumes elliptical manifold geometry, nevertheless yields representations with high values of the more general mean field manifold capacity (relative to baseline methods). For completeness we also analyzed the geometries of class manifolds, whose points are the representations of different examples from the same class. This analysis provided further evidence that learning to maximize augmentation manifold capacity compresses and separates class manifolds, leading to a useful representation. Interestingly MMCRs seem to use a different strategy than the baseline methods to increase the capacity, yieldingclass/augmentation manifolds with larger radii, but lower dimensionality (Fig. 2) These geometrical differences do not emerge until the tail end of the hierarchy, suggesting early layers of each network are carrying out stereotyped transformations and loss function induced specialization does not emerge until later layers.

Emergence of neural manifolds via gradient coherence. We hypothesize that class separability in MMCRs arises because augmentation manifolds corresponding to examples from the same class are optimally compressed by more similar transformations than those stemming from distinct classes. To investigate this empirically, we evaluate the gradient of the objective function for inputs belonging to the same class. We can then check whether gradients obtained from (distinct) batches of the same class are more similar to each other than those obtained from different classes, which would suggest that the strategy for compressing augmentation manifolds from the same class are relatively similar to each other. Fig. 3 demonstrates that this is the case: within class gradient coherence, as measured by cosine similarity, is consistently higher than across class coherence across both training epochs and model hierarchy.

Manifold subspace alignment. Within-class gradient coherence constitutes a plausible mechanistic explanation for the emergence of class separability, but it does not explain why members of the same class share similar compression strategies. To explore this question we examine the geometric properties of augmentation manifolds in the pixel domain. Here we observe small but measurable differences between the distributions of within-class similarity and across-class similarity, as demonstrated in the top row of Fig. 4. The subtle difference in the geometric properties of augmentation manifolds in the pixel domain in turn leads to the increased gradient coherence observed above, which leads to a representation that rearranges and reshapes augmentation manifolds from the same class in a similar fashion (bottom row of Fig. 4), thus allowing better linear separation of classes. Not only are centroids

AllParameters

First Layer

0.3

0.31

L

Gradient Similarity

Distinct Classes

Same Class

0.4

6oa

5W

0.2

0.2

Dog

E'0

&L

0.1

0.1

5W

Cat

Cat

0.2

0.0

0.0

0

50

100

150

200

0

50

100

150

200

0

50

100

150

200

Epochs

Epochs

Epochs

of same-class-manifolds in more similar regions of the representation space than those coming from distinct classes (Fig. 4 third column bottom row) but additionally same-class-manifolds have more similar shapes to each other (Fig. 4 bottom row columns 1 and 2 show same-class-manifolds occupy subspaces with lower relative angles and share more variance).

We next ask how the representation learned with the MMCR objective differs from those optimized for other self-supervised loss functions. While MMCR encourages centroids to be orthogonal, the InfoNCE loss (14) encourages negative pairs to be as dissimilar as possible, which is achieved when they lie in opposite regions of the same subspace . The Barlow Twins (77) loss is not an explicit function of feature vector similarities, but instead encourages individual features to be correlated and distinct features to be uncorrelated, across the batch dimension. Fig. 5 shows that these intuitions are borne out empirically: the MMCR representation produces augmentation manifold centroids that are significantly more orthogonal to each other than the two baseline methods.

SubspaceAngle

SharedVariance

CentroidSimilarity

0.14

0.1

0.05

Pixels

Probability

0.1

1 Dog Augmentation

Single Cat Exemplar

0.06

0.03

Manifold

/Cat Class Manifold

0.06

Dog Class Manifold

0.02

0.02

0.01

1CatAugmentation

Single Dog Exemplar

Manifold

0.8

1.0

1.2

0.1

0.3

0.5

0.7

6°0

-1.0

-0.5

0.0

0.5

1.0

0.22

0.7

0.18

Representation

0.18

0.14

Probability

0.5

0.12

0.1

0.08

0.06

0.02

0.1

0.02

1.0

1.2

1.4

0°0

Fraction of Shared Variance

0.4

0.8

0.0

Centroid Cosine Similarity

0.4

0.8

1.2

3.3 Biological relevance

Neuroscience has provided motivation for many of the developments in artifical neural networks, and it is of interest to ask whether SSL networks can characterize the measured behaviors of neurons in biological visual systems. As a simple test, Table 2 shows performance of our model compared with five other SSL models on the BrainScore repository (64; 30; 53; 51). We find that MMCR achieves the highest performance in explaining neural data from primate visual areas V2 and V4, second-highest for V1 and IT, and is the most or second most predictive for 8 out of the 11 individual datasets (see Appendix K).

In addition, we examine general response spectral characteristics that have been recently described for neural populations. In particular, Stringer et al. (67) reported that the eigenspectrum of the

SameClass

DistinctClasses

0.16

BarlowTwins

0.08

SimCLR

0.12

MMCR

Probability

0.06

0.08

0.04

0.02

0.04

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

Centroid Similarity

Centroid Similarity

covariance matrix of population activity in visual area V1 follows a power law decay with a decay coefficient of approximately 1 ( λ n ∝ n -α with α ≈ 1 where λ n is the n th eigenvalues). Subsequent studies in artifical networks have found that such a decay spectrum is associated with increased robustness to adversarial perturbations and favorable generalization properties (55; 2). Additionally, several recent works have investigated the connection between representational dimensionality and neural predictivity (70; 63). In particular, Elmoznino and Bonner (27) report that high intrinsic dimensionality (as measured with the participation ratio of the representation covariance) is correlated with stronger ability to predict neural activity. Table 2 provides values for the participation ratio of each representation over the ImageNet validation set, as well as the decay coefficient of the covariance spectrum (see Appendix L for more details on each experiment). We see that the MMCR has the highest participation ratio (dimensionality) - note that this differs from the quantity optimized in the objective function, which lies in the embedding space. In addition, MMCR also also yields features with a decay coefficient that is nearest to one.

We find that in this controlled setting each model explains a very similar fraction of neural variance through linear regression. This is consistent with recent and concurrent works (21; 62) which have identified this lack of ability to discriminate between alternative models as a weakness of the dominant paradigm used for model-to-brain comparisons. However, our results demonstrate that different SSL algorithms produce representations with meaningfully different geometries (as evidenced by the large spread in the spectral properties such as the participation ratio and decay coefficient). This suggests the need for the development of new metrics for comparing models to data, such as geometrical measures, that capture these important differences between candidate models.

4 Discussion

We have presented a novel self-supervised learning algorithm inspired by manifold capacity theory. Many existing SSL methods can be categorized as either 'contrastive' or 'non-contrastive' depending on whether they avoid collapse by imposing constraints on the embedding gram or covariance matrix, respectively. Our framework strikes a compromise, optimizing the singular values of the embedding matrix itself. By directly optimizing this population level feature (the spectrum), we are able to encourage alignment and uniformity (71) simultaneously with a single-term objective. Additionally, this formulation circumvents the need for making a large number of pairwise comparisons, either between instances or dimensions. As a result learning MMCRs is efficient, requiring neither large batch size nor large embedding dimension. Finally, our method extends naturally to the multi-view case, offering improved performance with minimal increases in computational cost.

Our formulation approximates manifold geometries as elliptical, reducing computational requirements while still yielding a useful learning signal and networks with high manifold capacity. Specifically, we were able to leverage manifold capacity analysis in its full generality to gain insight into the geometry of MMCR networks after training. Further research could explore objectives based on more modest reductions of mean field manifold capacity that capture non-elliptical structure. Intriguingly, our method produces augmentation and class manifolds with lower dimensionality but larger radius than either Barlow Twins or SimCLR (Fig. 2). We do not understand why this is the case, but the differences indicate that capacity analysis can provide a useful tool for elucidating the different encoding strategies encouraged by various SSL paradigms.

Finally we investigated two recently proposed theories on ideal spectral properties for neural representations. For our considered set of models a spectral decay coefficient near 1 was associated with better performance on the within-distribution task and generalization to unseen datasets (MMCR, SwAV, and Barlow were the top three models for each of the out-of-distribution classification tasks), a finding which is broadly aligned with both the empirical and theoretical findings of Agrawal et al. (2). However, we also found that high dimensionality did not always correspond to strong neural predictivity: Despite having the lowest dimensionality, SimCLR performed strongly in terms of neural predictivity. This implies, perhaps unsurprisingly, that global dimensionality alone is not sufficient to explain the response properties of neurons in the primate ventral stream. In fact our experiments add to a growing body of work on the need for complementary approaches to linear predictivity for discriminating between candidate models of the visual system (21). Happily the field is already moving to address this limitation, for instance concurrent work (12) finds that decomposing neural predictivity error into distinct modes can yield insights into how different models fit different aspects of the neural data (even if they yield similar overall predictivity).

One promising direction for improving the quality of artificial networks as models of neural computations is to incorporate the constraints associated with biologically plausible learning (i.e. the need for local learning rules, Illing et al. (44)). A complementary direction is to better align training data diets with ecological inputs. For example, temporal invariance rather than augmentation invariance seems a more plausible objective for a neural system to optimize (73; 3; 61). We speculate that a variant of the MMCR objective that operates over time may be well-suited to a neural circuit implementation as its computation would only require mechanisms for tracking (1) short timescale temporal means (to form centroids) and (2) the singular values of the population activity over longer timescales.

Neuroscience is brimming with newly collected datasets recorded from ever larger populations of neurons, and there is a long history of methods that aim to make sense of these measurements through a normative lens. Here we have demonstrated that one such technique that has proven useful for gleaning insights from neural data (60; 75; 31) can be reformulated for use as an objective function to learn a useful abstract representation of images. Future work should aim to close the loop between modelling and analysis by using these learned models to generate experimentally testable predictions and constrain new experimental designs.

References

Alia Abbaras, Benjamin Aubin, Florent Krzakala, and Lenka Zdeborová. Rademacher complexity and spin glasses: A link between the replica and statistical theories of learning. In Mathematical and Scientific Machine Learning , pages 27-54. PMLR, 2020.

Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Richards. Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. Advances in Neural Information Processing Systems , 35:17626-17638, 2022.

Arthur Aubret, Markus R Ernst, Céline Teulière, and Jochen Triesch. Time to augment selfsupervised visual representation learning. In The Eleventh International Conference on Learning Representations , 2022.

Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems , 32, 2019.

Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha SohlDickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics , 11(1), 2020.

Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205.11508 , 2022.

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021.

Horace B Barlow et al. Possible principles underlying the transformation of sensory messages. Sensory communication , 1(01), 1961.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning , pages 531-540. PMLR, 2018.

Silvia Bernardi, Marcus K Benna, Mattia Rigotti, Jérôme Munuera, Stefano Fusi, and C Daniel Salzman. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell , 183(4): 954-967, 2020.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision , 2014.

Abdulkadir Canatar, Jenelle Feather, Albert Wakhloo, and SueYeon Chung. A spectral theory of neural prediction and alignment. arXiv preprint arXiv:2309.12821 , 2023.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems , 33:9912-9924, 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15750-15758, 2021.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

SueYeon Chung and LF Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion in neurobiology , 70:137-144, 2021.

SueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Physical Review X , 8(3):031003, 2018.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

Uri Cohen, SueYeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications , 11(1):1-13, 2020.

Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv , 2023. doi: 10.1101/2022.03.28.485868. URL https: //www.biorxiv.org/content/early/2023/07/01/2022.03.28.485868 .

Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition. IEEE transactions on electronic computers , (3):326-334, 1965.

Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. sololearn: A library of self-supervised methods for visual representation learning. J. Mach. Learn. Res. , 23:56-1, 2022.

Joel Dapello, Jenelle Feather, Hang Le, Tiago Marques, David Cox, Josh McDermott, James J DiCarlo, and SueYeon Chung. Neural population geometry reveals the role of stochasticity in robust perception. Advances in Neural Information Processing Systems , 34:15595-15607, 2021.

James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences , 11(8):333-341, 2007.

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9588-9597, 2021.

Eric Elmoznino and Michael F Bonner. High-performing neural network models of visual cortex benefit from high latent dimensionality. bioRxiv , 2022.

Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International Conference on Machine Learning , pages 3015-3024. PMLR, 2021.

Adrienne L Fairhall, Geoffrey D Lewen, William Bialek, and Robert R de Ruyter van Steveninck. Efficiency and ambiguity in an adaptive neural code. Nature , 412(6849):787-792, 2001.

Jeremy Freeman, Corey M Ziemba, David J Heeger, Eero P Simoncelli, and J Anthony Movshon. Afunctional and perceptual signature of the second visual area in primates. Nature neuroscience , 16(7):974-981, 2013.

Emmanouil Froudarakis, Uri Cohen, Maria Diamantaki, Edgar Y Walker, Jacob Reimer, Philipp Berens, Haim Sompolinsky, and Andreas S Tolias. Object manifold geometry across the mouse cortical visual hierarchy. BioRxiv , pages 2020-08, 2020.

Stefano Fusi, Earl K Miller, and Mattia Rigotti. Why neurons mix: high dimensionality for higher cognition. Current opinion in neurobiology , 37:66-74, 2016.

Juan A Gallego, Matthew G Perich, Lee E Miller, and Sara A Solla. Neural manifolds for the control of movement. Neuron , 94(5):978-984, 2017.

Elizabeth Gardner. The space of interactions in neural network models. Journal of physics A: Mathematical and general , 21(1):257, 1988.

Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathematical and general , 21(1):271, 1988.

F Gerl and U Krey. Storage capacity and optimal learning of potts-model perceptrons by a cavity method. Journal of Physics A: Mathematical and General , 27(22):7353, 1994.

Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl. https://github.com/facebookresearch/vissl , 2021.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems , 34:5000-5011, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

O J Hénaff, N Rabinowitz, J Ballé, and E P Simoncelli. The local low-dimensionality of natural images. In Int'l Conf on Learning Representations (ICLR) , San Diego, CA, May 2015. URL http://arxiv.org/abs/1412.6626 .

Olivier J Hénaff, Yoon Bai, Julie A Charlton, Ian Nauhaus, Eero P Simoncelli, and Robbe LT Goris. Primary visual cortex straightens natural video trajectories. Nature communications , 12 (1):1-12, 2021.

Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive predictions. Advances in Neural Information Processing Systems , 34:30365-30379, 2021.

Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348 , 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Andrew V Knyazev and Merico E Argentati. Principal angles between subspaces in an a-based scalar product: algorithms and perturbation estimates. SIAM Journal on Scientific Computing , 23(6):2008-2040, 2002.

Simon Laughlin. A simple coding procedure enhances a neuron's information capacity. Zeitschrift für Naturforschung c , 36(9-10):910-912, 1981.

José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Ole: Orthogonal low-rank embedding-a plug and play geometric loss for deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8109-8118, 2018.

Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE transactions on pattern analysis and machine intelligence , 29(9):1546-1562, 2007.

Najib J Majaj, Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience , 35(39):13402-13418, 2015.

Jonathan Mamou, Hang Le, Miguel Del Rio, Cory Stephenson, Hanlin Tang, Yoon Kim, and Sueyeon Chung. Emergence of separable manifolds in deep language representations. In International Conference on Machine Learning , pages 6713-6723. PMLR, 2020.

Tiago Marques, Martin Schrimpf, and James J DiCarlo. Multi-scale hierarchical neural network models that bridge from single neurons in the primate primary visual cortex to object recognition behavior. bioRxiv , pages 2021-03, 2021.

Albert W Marshall, Ingram Olkin, and Barry C Arnold. Inequalities: theory of majorization and its applications , volume 143. Springer, 1979.

Josue Nassar, Piotr Sokol, SueYeon Chung, Kenneth D Harris, and Il Memming Park. On 1/n neural representation and robustness. Advances in Neural Information Processing Systems , 33: 6211-6222, 2020.

Edward H Nieh, Manuel Schottdorf, Nicolas W Freeman, Ryan J Low, Sam Lewallen, Sue Ann Koay, Lucas Pinto, Jeffrey L Gauthier, Carlos D Brody, and David W Tank. Geometry of abstract learned knowledge in the hippocampus. Nature , 595(7865):80-84, 2021.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729. IEEE, 2008.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Serdar Ozsoy, Shadi Hamdan, Sercan Ö Arik, Deniz Yuret, and Alper T Erdogan. Self-supervised learning with an information maximization criterion. arXiv preprint arXiv:2209.07999 , 2022.

Nihaad Paraouty, Justin D Yao, Léo Varnet, Chi-Ning Chou, SueYeon Chung, and Dan H Sanes. Sensory cortex plasticity supports auditory social learning. Nature Communications , 14(1): 5828, 2023.

Nikhil Parthasarathy, SM Eslami, João Carreira, and Olivier J Hénaff. Self-supervised video pretraining yields strong image representations. arXiv preprint arXiv:2210.06433 , 2022.

Andrew Saxe, Stephanie Nelli, and Christopher Summerfield. If deep learning is the answer, what is the question? Nature Reviews Neuroscience , 22(1):55-67, 2021.

Rylan Schaeffer, Mikail Khona, and Ila Fiete. No free lunch from deep learning in neuroscience: A case study through models of the entorhinal-hippocampal circuit. Advances in Neural Information Processing Systems , 35:16052-16067, 2022.

Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv preprint , 2018. URL https://www.biorxiv.org/ content/10.1101/407007v2 .

Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual review of neuroscience , 24(1):1193-1216, 2001.

Cory Stephenson, Jenelle Feather, Suchismita Padhy, Oguz Elibol, Hanlin Tang, Josh McDermott, and SueYeon Chung. Untangling in invariant speech recognition. Advances in neural information processing systems , 32, 2019.

Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High-dimensional geometry of population responses in visual cortex. Nature , 571(7765): 361-365, 2019.

Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision , pages 776-794. Springer, 2020.

Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learning without labels on imagenet? arXiv preprint arXiv:2201.05119 , 2022.

Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H McDermott. Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions. bioRxiv , pages 2022-09, 2022.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning , pages 9929-9939. PMLR, 2020.

Yu Wang, Jingyang Lin, Qi Cai, Yingwei Pan, Ting Yao, Hongyang Chao, and Tao Mei. A low rank promoting prior for unsupervised contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation , 14(4):715-770, 2002.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3733-3742, 2018.

Justin D Yao, Klavdia O Zemlianova, David L Hocker, Cristina Savin, Christine M Constantinople, SueYeon Chung, and Dan H Sanes. Transformation of acoustic information to sensory decision variables in the parietal cortex. Proceedings of the National Academy of Sciences , 120 (2):e2212120120, 2023.

Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems , 33:9422-9434, 2020.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

A Optimal Embeddings

Recall the setting of self-supervised learning as described in Balestriero and LeCun (6): given a dataset X ′ = [ x 1 , ..., x N ] T ∈ R N × D ′ we construct a new dataset by creating k randomly augmented views of the original data, X = [ view 1 ( X ′ ) , ..., view k ( X ′ )] ∈ R Nk × D . The advantage of doing so is that we can now leverage the knowledge that different views of the same underlying datapoint are semantically related . We can express this notion of similarity in the symmetric matrix G ∈ { 0 , 1 } Nk × Nk with G ij = 1 if augmented datapoints i and j are semantically related (and G ii = 1 as any datapoint is related to itself). We can normalize G such that its rows and columns sum to 1 (so rows of G are k -sparse with nonzero entries equal to 1 /k ).

Now let Z ∈ R Nk × d be an embedding of the augmented dataset. Then we have GZ = [ C , ..., C ] T where C is the matrix of centroid vectors introduced above, and the number of repetitions of C is k . Then because σ ([ C , ..., C ]) = √ k σ ( C ) we can write MMCR loss function as,

Where we have taken the eigendecomposition of G which is real and symmetric and the SVD of Z , and then used the fact that the singular value spectrum is invariant under left or right orthogonal transformations. We now show that a global optima of this objective is achieved when the left singular vectors of Z are the eigenvectors of G and the singular values of Z are proportional to the eigenvalues of G . Throughout we will assume that the size of the dataset is greater than the dimensionality of the embedddings, N > d , as is the case in practical applications. First we prove a simple lemma about the spectrum of matrices who are extended by zeros (i.e. embedded in a higher dimensional space).

Lemma A.1 : For A ∈ R N × N , B ∈ R N × d with d < N , || AB || ∗ = || A ˜ B || ∗ where ˜ B = [ B , 0 ] ∈ R N × N .

Proof : First note that A ˜ B = [ AB , 0 ] so it suffices to show that for arbitrary X that σ ( X ) = σ ([ X , 0 ]) . Taking the SVD of X ,

Then a valid singular value decomposition for ˜ X is

Clearly then, || X || ∗ = || ˜ X || ∗

Theorem: The proposed loss achieves a global minimum when the left singular vectors of Z are the eigenvectors of G , and the singular values of Z are proportional to the top d eigenvalues of G .

Proof: Let ˜ Z = [ Z , 0 ] ∈ R N × N . By Lemma A.1 we have || GZ || ∗ = || G ˜ Z || ∗ . Von Neumann's trace inequality can be used to show || G ˜ Z || ∗ ≤ ∑ Nk i =1 σ i ( G ) σ i ( ˜ Z ) (see Marshall et al. (54) for proof). Examining (4) it is clear that this bound is achieved when U = Q . The problem can therefore be reduced to the constrained optimization problem,

where the constraint comes from the fact that columns of Z are unit vectors. Intuitively, we are maximizing the inner product between a fixed vector σ ( G ) and a vector with fixed L2 norm. The

solution of course is to align the two vectors as closely as possible, i.e. when σ i ( ˜ Z ) ∝ σ i ( G ) for i = 1 , ..., d . It is worth noting that by construction σ i ( ˜ Z ) = 0 for i > d and the columns of U associated with these zero valued singular values are unconstrained.

B Pytorch Style Pseudocode for MMCR

C Mean Field Theory Manifold Capacity Background Information

For completeness we summarize some of the central arguments from Chung et al. (18), which develops the general form of manifold capactiy theory.

Mean Field Theory Recall the problem setting for manifold capacity analysis: given a set of P manifolds embedded in a feature space of dimensionality D , each assigned a random binary class label (18). Manifold capacity theory is concerned with the question: what is the largest value of P D such that there exists (with high probability) a hyperplane separating the two classes? In the thermodynamic limit, where P, D →∞ but P D remains finite, the inverse capacity can be written exactly,

where, F ( ⃗ T ) = min ⃗ V { ∥ ⃗ V -⃗ T ∥ 2 | g S ( ⃗ V ) ≥ 0 } , S is the set defining the manifold geometry (i.e. the set of vectors ⃗ S that are points on an individual manifold), ⃗ T are random vectors drawn from a white multivariate Gaussian distribution, and g S ( ⃗ V ) = min ⃗ S { ⃗ V · ⃗ S | ⃗ S ∈ S} , is the concave support function.

The KKT equations for this convex optimization problem are:

, where ˜ S ( ⃗ T ) is a subgradient of the support function. When the support function is differentiable, the subgradient is unique and equal to the gradient,

˜ S ( ⃗ T ) is the unique point in the convex hull of S that satisfies the first KKT equation, and is called the 'anchor point' for S induced by the random vector ⃗ T .

Equivalent Interpretation of Anchor Points For a given dichotomy (random binary class labelling) the weight vector of the maximum margin separating hyperplane can be decomposed into a sum of at most P vectors, with each manifold contributing a single vector, which lies within the convex hull of the manifold. The position of said point point is a function of the manifolds position relative to all of the other manifolds in the space and depends on the particular set of random labels. Thus there exists a distribution of separating-hyperplane-determining-points for each individual manifold. Using the cavity method it can be shown that these points are none other than the anchor points that are involved in solving the optimization problem described above (36).

Numerical Solution To solve the mean field equations numerically, one samples several random Gaussian vectors ⃗ T , and then for each ⃗ T , ⃗ V and ⃗ S are determined by solving the quadratic programming program given above. The capacity is then estimated as the mean value of F or the samples ⃗ T .

Manifold Geometries The way the capacity varies in terms of the statistics of the anchor points can be simplified by introducing two key quantities, the manifold radius R M and manifold dimensionality R M :

where ˆ S ( ⃗ T ) is a unit-vector in the direction of the anchor point ˜ S . In particular as discussed in the main text, the manifold capacity can be approximated by ϕ ( R M √ D M ) where ϕ is a monotonically decreasing function.

Elliptical Geometries In the case where the manifolds exhibit elliptical symmetries, the manifold radius and dimensionality can be written in terms of the eigenvalues of the covariance matrix of the anchor points:

So, in this case R M is the total variability of the anchor points, and D M is a generalized participation ratio of the anchor point covariance, a well known soft measure of dimensionality.

D Additional Pre-training information

Settings for CIFAR/STL-10 We take the parameters of each augmentation directly from Zbontar et al. (77), but for these lower resolution images we omitted Gaussian blurring and solarization augmentations. All models were trained for 500 epochs using the Adam optimizer (46) with a learning rate of 1 e -3 and weight decay of 1 e -6 . For all three methods we used a one hidden layer MLP with hidden dimension of 512 and output dimension of 128 for the projector head g . We swept batch size for each method and chose the one that resulted in the highest downstream task performance. For both SimCLR and Barlow Twins we found that a batch size of 128 was optimal (among 32, 64, 128, 256, and 512) for all 3 datasets. For MMCR there is a trade-off between batch size and the number of augmentations used, and the optimal value of that trade-off is highly dataset dependent. For CIFAR-10 and CIFAR-100 we used batch size of 32 and 40 views, and for STL-10 we used a batch of 64 with 20 views For Barlow Twins we used λ = 1 128 which normalizes for the number of elements in the on-diagonal and off-diagonal terms in the loss. For SimCLR we used the recommended setting of τ = 0 . 5 . The overall performance of both baseline methods (and likely MMCR as well) could be increased with a more thorough hyperparameter search and by employing methodology that more closely matches the original works. For example, both methods would likely benefit from the combination of larger batch size, the use of the LARS optimizer (which is designed for large batch optimization), a learning rate scheduler consisting of linear warm-up followed by cosine annealing, longer training, and the use of more diverse augmentations (i.e. including solarization and gaussian blur). Additionally Barlow Twins reports that the representation can benefit from using a much larger projector network than we use. Because our goal was primarily to demonstrate that MMCR can produce representations that are comparable to these baselines rather than to produce state-of-the-art results on small scale datasets we opted for simplifications wherever possible (using off the shelf Adam for optimization with a fixed learning rate, and fixing architectural hyperparameters like the projector dimensionality).

Settings for ImageNet-100 For ImageNet we more closely match the pre-training procedures of previous works. We use a batch size of 2048 and a smaller number of views for MMCR (4), and also use the full suite of augmentations from Zbontar et al. (77). For the sake of efficiency we train for a reduced number of epochs (200). For MMCR and SimCLR we modified the projector hidden dimensionality to be 4096 for the projector head, following the original work (14). For Barlow Twins we used the recommended 2-layer MLP with hidden and output dimensions of 8192, and set λ = 5 e -3 , however these hyperparameters were optimal for the full ImageNet dataset, and not neccesarrily for ImageNet-100. We were unable to achieve better downstream performance using a ResNet-50 backbone than what has previously been reported in the literature for this dataset with a ResNet-18 backbone, therefore we report the ResNet-18 performance reported in (23). For SimCLR we use τ = 0 . 1 which is the recommended setting for larger batch sizes.

Settings for ImageNet-1k : For ImageNet-1k we use mostly identical settings to ImageNet-100, but we increased the capacity of the projector network (using a 2 hidden layer MLP with hidden dimenisons of 8192 and output dimension of 512). We scaled the learning rate linearly with batch size: lr = 0 . 6 × batch size 256 . Additionally we reduce the number of pretraining epochs to 100. Finally for ImageNet-1k we found that employing a momentum encoder slightly boosted downstream performance (around +1% on ImageNet frozen-linear evaluation). Specifically, an identical encoding network and projector architecture is initialized with the same parameters as the initial 'online' network, and during training the weights of this 'momentum' network track a slowly moving average of the online network parameters (only the online network parameters are updated via gradient

descent). Each augmented view is passed through both the online and momentum networks, and the resultant embeddings are all averaged to form the centroid vector for a particular image in the batch. We used a momentum coefficient of 0.99.

Pre-training on 16 A100 GPUs using 8 views (our most compute intensive setting) takes approximately 32 hours.

E Details of Representational Analyses

E.1 Manifold Capacity Analysis

For each pre-trained model, we extract layer activations across the ResNet hierarchy after a forward pass of a set of images. For class manifold analysis, the set of images contain 10 classes, where each class has 100 examples. Augmentation manifolds instead have 100 exemplars with 100 examples each. Following (20), we take activations from all convolutional layers in ResNet-50 after a ReLU non-linearity. The specific extracted layers highlighted in bold fonts are given by Table 3. The final analysis results are averaged over five data samplings with different random seeds and random projections of intermediate features to lower-dimension spaces (default 5000 dimensions).

E.2 Gradient Coherence Analysis

In Fig. 3, for each of the classes of CIFAR-10, we generate 100 batches of 32 augmentation manifolds of samples from a specific class (with 40 augmentations each). We then measure the gradient of the loss function for each batch during different stages of training, and compute the cosine similarity between every pair of gradients. Across all stages of training the mean cosine similarity between gradients generated from batches of the same class is larger than those from distinct classes (left column). This observation remains true when isolating the gradients of parameters from different stages of in the resnet-50 hierarchy (center and right columns, respectively).

E.3 Manifold Subspace Alignment

For Fig. 4 we generated 100 samples from the augmentation manfiolds of 500 images in the CIFAR-10 dataset. We then measure the mean subspsace angle (left column), fraction of shared variance (middle column) and centroid cosine similarity between each pair of manifolds. The same procedure was used for generating the data for Fig. 5.

Subspace Angle. Besides measuring the size and dimensionality of individual object manifolds we also wish to characterize the degree of overlap between pairs of manifolds. For this, we measure the angle between their subspaces (47), which is a generalization of the notion of angles that applies to subspaces of arbitrary dimension.

Shared Variance. Object manifolds will generally have a lower intrinsic dimensionality then the space in which they are embedded. Therefore, the data will have low variance along several of the principal vectors used to calculate the set of subspace angles, and so many of the principal angles will have little meaning. To address this limitation we also compute the shared variance between the linear subspaces that contain object manifolds.

F Implicit MMCR Effectively Reduces Augmentation Manifold Nuclear Norm

To test whether or not implicit manifold compression actually reduces the mean augmentation manifold nuclear norm, we can vary the value of λ . Below we see the evolution of both terms of the loss for several different values of lambda during training on CIFAR-10. For these experiments the batch size was 64 and the number of augmentations per image was 4.0. As shown in Fig. 6, the level of compression of individual manifolds is nearly the same across all values of the parameter.

Layer

Type

Conv2d Size (H

×

W

×

C)

pixel

Input

None

conv1

[

[

l

]

Conv2d

BatchNorm

ReLU

]

×

1

[[

l

]7

×

7

×

64]

×

1

conv2_x





[

l

]

[

[

l

]

Conv2d

BatchNorm

]

×

3

ReLU





×

3

[

[

l

]1

×

1

×

64

3

×

3

×

64

1

×

1

×

256

]

×

3

conv3_x





[

l

]

[

[

l

]

Conv2d

BatchNorm

]

×

3

ReLU





×

4

[

[

l

]1

×

1

×

128

3

×

3

×

128

1

×

1

×

512

]

×

4

conv4_x





[

l

]

[

[

l

]

Conv2d

BatchNorm

]

×

3

ReLU





×

6

[

[

l

]1

×

1

×

256

3

×

3

×

256

1

×

1

×

1024

]

×

6

conv5_x





[

l

]

[

[

l

]

Conv2d

BatchNorm

]

×

3

ReLU





×

3

[

[

l

]1

×

1

×

512

3

×

3

×

512

1

×

1

×

2048

]

×

3

CentroidNuclearNorm

MeanManifoldNuclearNorm

52.0

2.8

48.0

2.4

44.0

2.0

Lambda = 0.0

Lambda = 0.01

Lambda = 0.05

Lambda = 0.1

40.0

Lambda = 0.5

1.6

Lambda = 1.0

0.0

40.0

80.0

0.0

40.0

80.0

Epochs

Epochs

G Classification Evaluation Procedure

CIFAR and STL-10 : During pre-training all models were monitored with a k-nearest neighbor classifier (k=200) and checkpointed every 5 epochs. After pre-training, we trained linear classifiers on all checkpoints whose monitor accuracy was within 1% of the highest observed accuracy, and select the model that achieves the highest linear classification accuracy. Linear classifiers were trained using the Adam optimizer with batch size of 1024 and an initial learning rate of 0.1, which decayed according to a cosine scheduler over the course of 50 epochs. For the linear classifier training, at train time we use the same set of augmentations as during unsupervised pretraining, at test time we only use center cropping and random horizontal flipping.

ImageNet-1k/100 : For ImageNet datasets we closely followed the most widely adopted evaluation procedure. Following pre-training we freeze the encoder weights and train a linear layer in a supervised fashion using SGD with a batch size of 2048, learning rate of 1.6, and weight decay of 1e-6 for 50 epochs. During linear classifier training the only data augmentations are random cropping and random horizontal flips, and during evaluation inputs are center cropped.

Semi-Supervised : For semi-supervised evaluation we mostly follow the procedure outlined in Bardes et al. (7). We use the SGD optimizer with momentum of 0 . 9 and weight decay of 1e-6 and the standard cross entropy loss. The augmentation procedure was the same as described above. Because the linear classifier is being trained from scratch and the representation is being fine tuned the learning rate for the parameters of the backbone is scaled down by a factor of 10, and both learning rates (backbone and classifier) followed a cosine decay schedule for 20 epochs. We used a batch size of 256 and swept the learning rate over [0.1, 0.3, 1.0] for each model.

Other Downstream Classification Tasks : Classifiers on these datasets were trained in a similar fashion to those trained on the CIFAR and STL-10 datasets. The only differences being that for each method and datasets we swept the batch size over [128, 256, 512, 1024] and the the initial learning rate over [3e-2, 3e-3, 3e-4], and the augmentation procedure matched the standard setup for ImageNet training (only random cropping and horizontal flipping for training, resizing and center cropping for evaluation).

H Training Metrics

In the Fig. 7 below we monitor the evolution of both the objective (second panel), the mean augmentation manifold nuclear norm, the centroid norm, and the mean centroid similarity evaluated on the test set over the course of training.

I|Zbll*

IICI*

Centroid Similarity

Centroid Norm

11.0

180.0

0.14

0.95

160.0

0.1

0.9

9.0

0.85

0.06

140.0

0.8

7.01

0.02

0

50

100

150

200

120.0

0

50

100

150

200

0

50

100

150

200

0.75

0

50

100

150

200

Epochs

Epochs

Epochs

Epochs

I Classification Performance on Smaller Datasets

In Table 4 below we report the performance of both our method as well as Barlow Twins and SimCLR when trained using a ResNet-50 backbone on smaller datasets.

J Batch Size Dependence

One of the most cited drawbacks of contrastive SSL methods has been that strong performance on downstream tasks requires training with large batch sizes, while non-contrastive methods (e.g., VICReg or Barlow Twins (77; 7)) that place constraints on the cross-correlation/covariance matrices of the embeddings are much more amenable to smaller batch training. It is also worth noting that the need for large batch sizes in contrastive methods can be alleviated in various ways, such as maintaining a memory bank (74) or employing a slowly updating momentum encoder (41). Given that our method is neither wholly contrastive nor non-contrastive (since it acts on the spectrum of the embedding matrix directly), we wondered howwould depend on training batch size. We pretrained on ImageNet-1k using batch sizes of 256 , 512 , 1024 , 2048 , 4096 and evaluate the linear classification

accuracy for each. Encouragingly we observed only a modest decrease in performance for the smallest batch size tested. The results of this sweep, in comparison to Barlow Twins and SimCLR, is shown in in Fig. 8 Note that for these runs we used two views and the linear learning rate scaling as described in Appendix D. Future work should endeavor to better understand the impact of various hyperparameters on the quality of learned representations.

An important detail is that for this experiment we did not employ a momentum encoder when training MMCR. It is argued in He et al. (41) that the momentum encoder increases the effective minibatch size as the slowly moving weights encode information from preceding batches. However here we are interested explicitly interested in batch size dependence so we ablate this architectural confound (neither Zbontar et al. (77); Chen et al. (14) employ momentum encoders).

Top-1 Accuracy

0.0

-1.0

-2.0

BarlowTwins

SimCLR

-3.0

△

MMCR

-4.0

256

512

1024

2048

4096

K Additional Details on BrainScore

Brain-Score evaluates a model in terms of its ability to predict the measured responses of neurons to images. We provide a brief introduction to the metric here (see (64) for a complete description). Let y ∈ R N denote the average response of a single (biological) neuron to a set of N training images and X ∈ R N × K be the response of K model neurons to the same images. Brain Score first solves the linear regression problem y = Xw for weights w . The model then predicts responses y ′ to a set of held out images. Next the Pearson correlation coefficient between y ′ and y is calculated, and the score for a particular dataset is the median of the individual neuron predicitivities in said dataset. The mean of these scores is taken over different train-test splits of each dataset. The score for a total brain area (as shown in Table 2 is the mean over train-test splits and distinct datasets. The standard error of the means for each dataset (which are typically between 1e-3 and 1e-2) within an area are summed quadrature and divided by the number of datasets to produce the errors reported in table 2. In table 5 we split the brain area scores into individual datasets.

L Additional Details on Spectral Properties

Participation Ratio We first extracted the 2048 dimensional feature vectors for each model in response to the images in the ImageNet validation set. Images were resized to 256 × 256 and then center cropped to 224 × 224 following the setting in which the classifiers are tested. We resampled the resultant feature matrices with replacement 10 times independently for each model For each resampled dataset (of features) we calculate the empirical covariance matrix, associated eigenspectra, and associated participation ratios (squared ratio of L 1 to L 2 norm of the eigenvcetors).

Decay Coefficient The spectra obtained using the procedure outlined above all decayed rapidly near the tails (the least significant eigenvalues). To avoid undo bias from these tails when estimating the decay coefficients we only considered the top 2000 eigenvalues. To estimate the decay coefficient we fit a regression line to the logarithm of the eigenvalues as a function of the logarithm of their indexes. This fitting procedure was repeated across the bootstrapped spectra for each model to obtain standard errors of the mean. For all models the linear regression produced a strong fit to the data, with the minimum observed R 2 value being 0 . 95 .

M Object Detection

To ensure that the representations obtained with MMCR are not hyperspecialized to classification tasks we also evaluated our highest performing model on object detection. We follow (41; 77), fine tuning the representation network with a Faster R-CNN head and C-4 backbone on the VOC07+12 dataset (training with the train+val split and evaluating on the VOC07 test split). All settings except for the initial learning rate (which we set to 0.12) were identical to those from He et al. (41), and we similarly used the detectron2 library for this evaluation. MMCR with 8 views and 100 epochs of pretraining produced an AP50 of 81.9 (this is the mean over three independent fine tunes, the standard deviation was 0.2), demonstrating that the representeations generated by our method are not limited to object recognition; Table ?? gives full results. For reference, a representation trained using MoCo v2 for 200 epochs achieves an AP50 (the most common evaluation metric for this dataset) of 82.4 (in the table below we report the result we obtained for MoCo pretrained for 100 epochs for consistency).

N Limitations

Time and compute limitations prevented us from conducting exhaustive optimization of all design choices. For example we do not explore effects of varying the projector network width and depth. We additionally restrict the model to a ResNet-50 encoding network pretrained on the ImageNet-1k

dataset for 100 epochs. This choice allows us to make fair comparisons to several recently developed SSL methods with a modest compute budget, but precludes answering questions about how the method scales up to larger problems.

Wealso note that although our objective function has favorable computational complexity compared to existing methods, the evaluation of our objective is not straightforward to distribute across machines. This is because we need to compute the SVD over the centroid matrix, which requires gathering the outputs of a distributed forward pass onto a single machine. Efficient methods for the distributed computation of the SVD could help alleviate this issue in the future.

O Impact of Momentum Encoder

With all other settings fixed, we find that the use of a momentum encoder confers a small advantage in terms of classification performance. Below we report frozen linear evaluation accuracy for MMCR training 2, 4, and 8 views with and without a momentum encoder.

P MMCRBenefits from Longer Pretraining

To verify that MMCR benefits from longer pretraining we modified our setup to pretrain with two views and reduced our base learning rate to 0.4 and train for 1000 epochs. We additionally changed the hyperparameters for pretraining the linear classifier when evaluating this model (the classifier was trained with an learning rate of 0.3 and batch size of 2048 for 100 epochs). We did not perform any hyperparameter tuning for this setting, so these results represent a performance floor for 2-view MMCR.

Peer reviews:
comments: SUMMARY:
This paper develops a form that facilitates direct optimization, use it to learn Maximum Manifold Capacity Representations (MMCRs), and demonstrate that these are competitive with state-of-the-art results on standard self-supervised learning (SSL) recognition benchmarks. Empirical analyses reveal important differences between MMCRs and the representations learned by other SSL frameworks.

STRENGTHS:
Inspired by manifold capacity theory, the proposed self-supervised learning algorithm is efficient to learn and requires neither large batch size nor large embedding dimension. 

This work also provides very interesting empirical results which are well aligned with the proposed theorem. 

Besides, the paper is well-presented.

WEAKNESSES:
Since I am not in this area, the things mentioned in this paper are very convincing to me. I cannot point out very specific weakness.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new SSL objective to directly optimize manifold capacity (num of categories represented in a linearly separable fashion). It is found that the resulting representations can support linear classification well. It is also shown that the representations are somewhat sample efficient (reasonable performance in 1% and 10% settings, Food, Flowers, etc). It is also found that that the representations are slightly better predictors of neuronal representations in V2 compared to baselines.\n\nSTRENGTHS:\n1. The description of the objective function and simplifying assumptions are clear\n2. The evaluations (e.g. Tables 1, 2) and analyses (e.g. Figs, 4, 5) are clear.\n3. The eigenspectrum alpha being close to 1 is an interesting finding.\n\nWEAKNESSES:\n[Limited evaluations, objective too specific?]\n\nThere is some novelty in the specific objective function used, but evaluation is limited. For example, by more directly optimizing for linear separability has there been a loss in generality of representations, e.g. on segmentation or depth estimation? The detection results in the appendix appear to be consistent with MMCR being more specialized for linear classification. Also unclear how well does the objective work when the data are more complex (e.g. COCO)? Imagenet normally contains few objects and the correct category is very often an object in the center. Moreover, the architecture explored is also limited to ResNet-50 and is not competitive with more recent SSL methods (e.g. RELICv2, NNCLR).\n\nAnother important limitation is that evaluations are for SSL models only trained for 100 epochs. Different SSL methods converge to good representations at different speeds, so limiting evaluation to 100 epochs is does not give us an adequate understanding of various methods - this limitation applies to comparisons on ml benchmarks as well as neuronal data.\n\n[Purpose of comparing to neural data unclear.]\n\nMMCR has slightly better predictive power in V2, and the eigenspectrum alpha is close to 1. These are interesting empirical observations, but the discussion does not appear to really engage with these. In the discussion (l316) it is claimed that future work should go beyond accounting for current data. It is not clear to me how the current work provides an account of current data? What are the precise claims/hypotheses being made and how do the experiments support it? What does it entail for our understanding of the brain that was not previously known?\n\nQUESTIONS:\n1. Can the authors please make their discussion on relevance to neuroscience more specific? \n2. Can the authors please expand on their evaluations along any of the directions listed in weaknesses?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an alternative way (i.e., maximizing manifold capacity) to learn useful representation in a self-supervised manner. In particular, it maximizes the separability of the centroid manifold (i.e., negative samples), while the second term aims to minimize the nuclear norm among positive samples.\ntends to figure out crucial properties of self-supervised learning (SSL) methods, which promote good performance on downstream tasks. To reach this target, the authors propose a unifying conceptual ISSL framework. In particular, their main contributions contains: i) increasing the dimensionality of presentation and using asymmetric projection heads; ii) presenting a new non-contrastive ISSL objective; iii) applying non-linear probes.\n\nSTRENGTHS:\nThis paper introduces manifold capacity in self-supervised learning (SSL). This idea is somewhat novel and provides an alternative way to avoid the collapse issue in SSL. Extensive experiments conducted on nearly 7 datasets demonstrate that the proposed method achieves comparable performance to existing state-of-the-art (SOTA) methods. Furthermore, empirical analysis of Maximum Manifold Capacity Representations (MMCRs) reveals distinct characteristics compared to existing approaches.\n\nWEAKNESSES:\nHowever, my main doubts/concerns regarding the paper are the following:\n\n- As shown in Line 138-141, the first term aims to maximize the separability of negative samples, while the second term aims to minimize the nuclear norm among positive samples. The motivation behind the proposed loss is similar to [1], which pursues \"the alignment of features from positive pairs and the uniformity of the induced distribution of the normalized features on the hypersphere.\" It would be better to provide further discussion in relation or difference to [1].\n- The observation in Figure 6 is interesting. However, this result raises the question of whether implicitly minimizing object manifold compression is truly necessary, as the second term with nuclear norm does not impact the mean manifold nuclear norm. Therefore, conducting an ablation study between these two terms on several downstream tasks is essential to further investigate their effects.\n- The proposed MMCR method shares strong connections with nuclear norm-based approaches [2,3,4], especially in relation to [2]. However, the current study lacks either theoretical or empirical comparisons between MMCR and these nuclear norm-based methods.\n- In Figure 2, the Mean Filed Manifold Capacity Analysis shows that existing SOTA methods with different objectives all explicitly learn representations with large radii but low dimensionality. These results may question the necessity of the proposed method.\n- In Figure 5, the centroid similarity distributions of MMCR for both same class and distinct classes appear to be generally smaller than those of existing methods (Barlow Twins and SimCLR). Therefore, this comparison may be somewhat unfair. To ensure a fair comparison, it would be better to normalize the centroid similarities (e.g., by computing z-scores) for each method and then compare the normalized distribution across different methods.\n\nMinor:\n- The experimental setting described in Section 3.2 is unclear, which could potentially result in unfair comparisons between MMCR and other state-of-the-art (SOTA) methods. If MMCR employs multi-crop augmentation while Barlow Twins and SimCLR only use two views, the comparison would be unfair in Figure 2 and 5.\n- The authors might unintentionally modify the default LaTeX template, which results in incorrect formatting of the citations. In particular, (1) -> [1] or (1;2) -> [1,2].\n- Incorrect format of theorems, lemmas, and proofs is detrimental to this paper since it makes the paper appears informal.\n- Almost all equations lack commas or full stops at the end.\n- Please standardize the usage of either \"Fig. x\" or \"Figure x\" for consistency.\n- Figure 2: \"see E\" -> \"see appendix E\". Table 2: \"see L\" -> \"see appendix L\".\n\n\n[1] T. Wang, et al., \"Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere\", ICML 2020.\n\n[2] Y. Wang, et al., \"A Low Rank Promoting Prior for Unsupervised Contrastive Learning\", TPAMI 2022.\n\n[3] O. Hénaff, et al., \"The Local Low-dimensionality of Natural Images\", ICLR 2015.\n\n[4] J. Lezama, et al., \"OLÉ: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learning\", CVPR 2018.\n\nQUESTIONS:\nThe questions are corresponding to the above main concerns:\n- What is the relation or difference between the proposed objective and [1]?\n- Please add more theoretical or empirical analyses on the necessity of implicitly minimizing object manifold compression.\n- Please add more theoretical or empirical comparisons between MMCR and the nuclear norm-based methods [2,3,4].\n- Please add more discussion about the results in Figure 2.\n\nLIMITATIONS:\nYes, the authors have addressed the societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe efficient coding hypothesis suggests sensory systems maximize mutual information between their inputs and the environment. A recent adaptation, \"manifold capacity\", calculates the number of linearly separable object categories, but is computationally intensive. The authors simplify this measure to directly optimize Maximum Manifold Capacity Representations (MMCRs). They show MMCRs are competitive with top results on self-supervised learning (SSL) recognition benchmarks. MMCRs differ from other SSL frameworks and may enhance class separability through manifold compression. On neural predictivity benchmarks, MMCRs prove competitive as models of the ventral stream, a part of the brain involved in object recognition.\n\nSTRENGTHS:\n-\tThe paper presents a novel method on SSL leveraging Manifold Capacity theory. The contribution is clear and precise. \n-\tThe paper connect SSL with Manifold Capacity theory with a effective theoretical formulation for the SSL objective. \n-\tThe paper empirically validates its claim by running large scale experiments and demonstrates competitive performance. \n-\tThe method mitigate the large batch / large dimension training bottleneck of the previous SSL method. \n-\tThe paper also evaluates on the neural data to show that its method can explain the neural data better.\n\nWEAKNESSES:\n-\tAlthough the initial results look very promising, the author didn’t include an error bar in Table-1. It would be beneficial to see the error bar to further eliminate the effect of chance. \n-\tThe paper posits that sensory systems maximize mutual information between their representations and the environment, based on the efficient coding hypothesis. However, it might lack a discussion on the biological plausibility of MMCRs. How feasible is it for biological sensory systems to implement MMCRs? How does this model account for the biological constraints mentioned in the efficient coding hypothesis? \n-\t[Minor] In the legend of Figure 2, the description indicates the top row showing the radius but the actual graph shows dimensionality.\n\nQUESTIONS:\n- An illustrative figure that intuitively demonstrate the intuition of singularity of the covariance matrix in terms of the objective function would be beneficial to the reader's understanding. \n- In the introduction, it might be better to highlight the contribution to the SSL more clearly.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors convert recent results in manifold perceptron capacity calculations to a practical self-supervised learning algorithm. They then (1) provide one theoretical result, (2) show their method is predictive of primate visual cortex recordings, and (3) characterize some empirical properties of their method versus other methods.\n\nSTRENGTHS:\n- The conversion of manifold perceptron capacity to a practically implementable SSL algorithm is novel\n\n- This paper is quite thorough. It contains a little bit of everything: a new algorithm, one theoretical result for the algorithm, comparison with biological data, empirical study of how the algorithm differs from other SSL algorithms, eigenspectrum decay. Its experimental results are particularly thorough.\n\nWEAKNESSES:\nOrdered by location in the text, rather than prioritization:\n\n- Section 2.1 either (1) makes a subtle but important leap that I'm not sure I followed or (2) uses confusing pluralization. Lines 93 to 103 state we consider P manifolds (plural) and then discuss three quantities: (1) the manifold radius (singular), (2) the manifold dimension (singular), (3) the correlation between the manifold centroids (plural). Additionally, a subscript $M$ appears and I do not know what this $M$ refers to. I'm not trying to nitpick pluralization; rather, I'm confused whether each of the P manifolds has its own radius and its own dimension, or whether all P manifolds collectively have 1 radius and 1 dimension. If the former, how are each of the P manifolds' radii and dimensions combined to determine the threshold capacity of the manifold population? If the latter, what does linear separability mean for a monolithic grouping of the P manifolds? Or must all $P$ manifolds have the same radius and dimension, but are permitted to possess their own centroids? I would appreciate if the authors could clarify this section.\n\n- The move from Section 2.1 to Section 2.2 needs better motivation. Section 2.1 introduces the manifold capacity as $\\phi(R_M \\sqrt{D_M})$ with differentiable, closed-form expressions for both $R_M$ and $D_M$. I am then expecting that we'll be maximizing capacity by minimizing $R_M \\sqrt{D_M}$. But instead, we jump to Equation (2)'s, with a small inline connection to $\\phi(R_M \\sqrt{D_M})$, and are then told that the second term of Equation 2 doesn't matter and so we'll set its prefactor $\\lambda$ to 0. I think this section could be greatly improved if the authors emphasize & explain the inline approximation , and better motivate Equation 2. I would recommend either leaving out the second term of Eqn 2 altogether, or alternatively, clarifying why the second term _should_ be mentioned if it is to be immediately eliminated.\n\n- I'm unsure whether I should be persuaded by the 2D example of 2 centroids (Equations 3 and 4) . I do love attempting to build intuition, and I understand that no closed form expressions exists for singular values of an arbitrary matrix, so I commend the authors, but I'm unsure whether the intuition will hold in (a) higher dimensions and (2) with significantly more data i.e. more centroids. Even staying in 2D, what will MMR do if we have 3, 4, 5 etc. centroids? I might've guess something like the Thomson problem (https://en.wikipedia.org/wiki/Thomson_problem) but that can't be the case because 2 electrons choose antipodal points where MMCR chooses orthogonal points.\n\n- Moreover, upon another read, I realized the paragraph \"Compression by Maximizing Centroid Nuclear Norm Alone\" seems odd compared with Equation 2. The paragraph argues that the nuclear norm of $C$ alone is sufficient because it both (1) incentivizes making each manifold as tightly compressed as possible (i.e. each mean should have maximum norm, bounded above by 1) and (2) incentivizes making centroids pairwise orthogonal. I agree with this - the 2D case is clear and convincing. But if those two incentives are what we care about, why not directly use them as our loss? I'm specifically suggesting something like $-\\sum_b ||c_b||^2 + \\sum_{b, b'} (c_b \\cdot c_{b'})^2$? And if so, isn't that almost exactly the loss of TiCo (Zhu et al 2021/2022 https://arxiv.org/abs/2206.10698 ), specifically equation 6?  \n\n- With all SSL papers, there are many implementation decisions and hyperparameters that complicate interpreting the results. Line 205-208 caught my eye (\"We additionally employ a momentum encoder for ImageNet pre-training...\". While there is nothing wrong with using a momentum encoder, I would like to know what performance MMCR achieves with vs. without the momentum encoder. Could the authors please include this ablation? If I missed it, I apologize.\n\n- Please add some measure of uncertainty (e.g. 95% confidence intervals) to Figure 3 (unless such intervals already exist and are too small to be seen, in which case, please note that in the figure caption).\n\n- The authors write \"Elmoznino and Bonner (23) found that high dimensionality in ANN representations was associated with ability to both predict neural data.\" and later \"Additionally, Elmoznino and Bonner (23) report that high intrinsic dimensionality (as measured with the participation ratio of the representation covariance) is correlated with a representation’s ability to predict neural activity.\" The claim that participation ratio is highly correlated with neural predictivity was noted concurrently by Schaeffer et al. 2022 \"No Free Lunch from Deep Learning in Neuroscience\" at an ICML 2022 workshop (albeit in mouse MEC on navigation tasks, rather than primate visual cortex on vision tasks) and published at NeurIPS 2022. Then, soon after, Tuckute et al. 2022 \"Many but not all deep neural network audio models capture brain responses and exhibit hierarchical region correspondence\" released a preprint showing the same correlation between participation ratio and neural predictivity in human fMRI recordings of audio (Figure S6 in biorxiv v1; I'm too lazy to find the same plot in the more recent biorxiv v4). From an attribution perspective, I believe all three should be cited since they were all within ~1-3 months of one another. I also think citing all three will strengthen your paper since it points out the same phenomenon appears in multiple modalities (navigation, vision, audition), multiple species (rodent - can't remember mouse or rate, monkey, human) , multiple recording technologies.\n\nI realize that many of these \"weaknesses\" might be my own misunderstandings. If the authors could clarify, or lightly revise the paper to address these above points in a satisfactory manner, I would be happy to increase my score :)\n\nQUESTIONS:\n- Figure 5: The authors state \"the InfoNCE loss employed in Chen et al. (12) benefits when negative pairs are as dissimilar as possible, which is achieved when the two points lie in opposite regions of the same subspace\". If this is correct (and I believe it is), we should expect to see negative cosine similarities between distinct classes. But then the right subplot shows that SimCLR has positive cosine similarity (~0.3) between distinct classes. Could the authors please clarify?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}