Self-Supervised Learning of Maximum Manifold Capacity Representations
Authors: Thomas Edward Yerxa
Yilun Kuang
Eero P Simoncelli
SueYeon Chung
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised learning
representation geometry
neural manifolds
statistical physics of learning
theoretical neuroscience

Abstract: Self-supervised Learning (SSL) has recently emerged as a successful strategy for learning useful representations of images without relying on hand-assigned labels. Many such methods aim to learn a function that maps distinct views of the same scene or object to nearby points in the representation space. These methods are often justified by showing that they optimize an objective that is an approximation of (or correlated with) the mutual information between representations of different views. Here, we recast the problem from the perspective of manifold capacity, a recently developed measure for evaluating the quality of a representation. Specifically, we develop a contrastive learning framework that aims to maximize the number of linearly separable object manifolds, yielding a Maximum Manifold Capacity Representation (MMCR). We apply this method to unlabeled images, each augmented by a set of basic transformations, and find that it learns meaningful features using the standard linear evaluation protocol. Specifically, we find that MMCRs support performance on object recognition comparable to recently developed SSL frameworks, while providing more robustness to adversarial attacks. Finally, empirical analysis reveals the means by which compression of object manifolds gives rise to class separability. 

Under review as a conference paper at ICLR 2023
SELF-SUPERVISED LEARNING OF MAXIMUM MANIFOLD CAPACITY REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervised Learning (SSL) provides strategy for constructing useful repre- sentations of images without relying on assiined labels. Many such meth- ods aim to learn a function that maps distinct views of the same scene or object to nearby points in the representation space. These methods are often justified by showing that they optimize objective that is approximation of (or cor- related with) the mutual information between representations of different views. Here, recast the problem from the perspective of manifold capacity, measure that has been used to evaluate the classification capabilities of a representation. Specifically, develop contrastive learning framework that aims to maximize the number of linearly separable object manifolds, yielding a Maximum Manifold Capacity Representation (MMCR). We apply this method to unlabeled images, each augmented by a set of basic transformations, and find that it learns mean- ingful features using the standard linear evaluation protocol. Specifically, we find that MMCRs support performance on object recognition comparable or better than recently developed SSL. frameworks, while providing more robustness adver- sarial attacks. Finally, empirical analysis reveals the means by which compression object manifolds gives rise to class separability.
INTRODUCTION
Natural images lic, at least locally, within manifolds whose intrinsic dimensionality is low relative to that of their embedding space (the set of pixel intensities). Nevertheless, these manifolds are enormously complex, as evidenced by the variety of natural scenes. fundamental goal of ma- chine learning is to extract these structures from observations, and use them to perform inference tasks. In the context of recognition, consider the object submanifold, .Mj. which consists of all images of object (for example, those taken from different camera locations, lighting conditions). Object recognition networks act to map images within a submanifold to nearby representations, relative to images from other submanifolds, and this concept has been effictively exploited in recent self-supervised learning (SSL) methods (Zbontar et al. 2021 Chen et al.) 2020) Caron al. 2020 Bachman al., 2019 Wang & Isola 2020; Wang I (002).. Most these operate by minimizing pairwise distances between images within submanifolds, while contrastively maximizing pairwise distances between images in different submanifolds.
under different
A parallel effort in computational neuroscience has aimed to characterize manifolds in neural repre- sentations, and their relationship underlying neural circuits (Kriegeskorte & Kievit 2013 Chung & Abbott 2021). Studies in various modalities have identified geometric structures in neural data that are associated with behavioral tasks (Bernardi al., 2020; DiCarlo Cox] 2007 HHénfff al.)
2021 Gallego et al., 2017 Nieh et al. 2021). and explored metrics for quantifying these represen- tation geometries.
Here, we make use of recently developed measure of manifold capacity, rooted in statistical physics (Chung et al., 2018), which has been used to evaluate how many manifolds can be lin- early separated within the representation space of various models. We develop simplified form of this meausre, and incorporate into a novel contrastive objective, that maximizes the extent of the global image manifold while minimizing that of constituent object manifolds. We apply this to an unlabeled set of images, each augmented to form a small set of samples from their corresponding manifold. We show that the learned representations

---

Under review as a conference paper at ICLR 2023
support high-quulity object recognition, when evaluated using the standard linear evalua- tion (Chen et al. 2020) paradigm (i.e., training linear classifer to operate on the output of the unsupervised network). In particular, performance is approximately matched to that of other recently proposed SSL methods.
extract semantically relevant features from the data, that can be revealed by examining the learning signal derived from the unsupervised task
have interpretable geometric properties
are more robust to adversarial attack than those of other recently proposed SSL methods.
1.1
RELATED WORK
Our methodology is closely related to and inspired by recent advances in contrastive self-supervised representation learning (SSL), but has a distinctly different motivation. Many recent frameworks craft objectives that are designed to maximize the mutual information between representations of
different views of the same object (Oord et al,, 2018; Chen et al,, 2020, Oord et al.] 2018: Tian ett al., 2020; Bachman et al., 2019)). However, estimating mutual information in high dimensional feature spaces (which is the regime of modern deep learning models models) has been difficult to compute historically (Belghazi et al. 2018), and furthermore it is not clear that more closely approximating mutual information in the objective produces improved representations (Wang & Isola 2020). ! By contrast, capacity estimation theories operate in the regime of large ambient dimension as they are derived in the "large N (thermodynamic) limit" (Chung et al. [018 Bahri 2020). Therefore we test whether one such measure, which until now had been used (o evaluate the quality of representations, might be useful as objective function in SSL.
Operationally, many existing methods are optimized minimize some notion of distance between the representations of different augmented views of the same image, while maximizing the distance between representations of (augmented views of) distinct images (these are thought of as encour- aging alignment and uniformity in the framework of [Wann & Isola (2020)). When taking the view that different views of an image form a continuous manifold that we aim to compress, the distance between two randomly sampled points from said manifold seems strange choice for the size met- ric to optimize for. Perhaps unsurprisingly it has been demonstrated on multiple occasions, notably by the success of the "multi-crop." strategy implemented SWAV (Caron et al.) (2200) and earlier in the contrastive multiview coding work by Tian et al. (2020). However most commonly the use of multiple views is such that the objective effectively becomes Monte Carlo estimate with more than one sample of the same pairwise distance function.
Rather than using the mean distance or cosine similarity between pairs of points, norm as a combined measure of size and dimensionality of groups of points, an idea that is strongly motivated by learning theory. The nuclear norm has been previously used induce or infer low rank structure in the representation of data, for example, in Hénaff et al. (013);; Wang et al. (2022); Lezama al.] (2018). In particular, |Wang et al. (2022) employ the nuclear norm regularizer to supplement an InfoNCE loss. Our approach represents a more radical departure from the traditional InfoNCE loss, as we will detail below. Rather than pair a low-rank prior with a logistic regression- based likelihood, we make the more symmetric choice of employing a high rank likelihood. This allows the objective to explicitly discourage dimensional collapse, a well known issue in SSL (Jing al. 202T)
nuclear
Another consequence of encouraging maximal rank over the dataset that the objective encourages the representation to form simplex equiangular tight frame (sETF). sETFs have been shown to be optimal in terms of cross-entropy loss when features lie on the unit hyperspherr (Lu & Steinerberger, 2020), and such representations can be obtained in the supervised setting when optimizing either the traditional cross-entropy loss or supervised contrastive loss (Papyan et al.; 2020 Graf et al.) 2021) Recent work has shown that many popular objectives in SSL can be understood as different methods of approximating a loss function whose minima form sETFs (Dubois et al. (022). Our approach is novel, in that it encourages sETF representations by directly optimizing the distribution of singular values, rather than minimizing a cross cross-entropy loss.
Barlow Twins Zbontar et al. 2021 notably avoids the curse of dimensionality because their objective effectively estimates information under a Gaussian parameterization rather than doing pon-parametrically as in the InfoNCE loss. Our method also makes use of Guasian/sccond order parameterizations, as detailed below.
2

---

Under review as a conference paper at ICLR 2023
2
MAXIMUM MANIFOLD CAPACITY REPRESENTATIONS
Low Capacity
Reduce Dimensionality
Before Training
Training
Maveltidiio Spe
Reduce Radius
Figure 1: Two dimensional illustrations of high and low capacity representations. Left: the capac- ity (linear separability) of random set of elliptical regions be improved, either by reducing their sizes (while maintaining their dimensionalities), or by reducing their dimensionalities (while maintaining their sizes). Right: the objective proposed in this paper aims minimize the nuclear norm (product of size and sqrt dimensionality) of normalized data vectors (ie., lying on the unit sphere). Before training the manifolds have large extent and thus the matrix of their corresponding centroid vectors has low nuclear norm. After training the capacity is increased. The manifolds are compressed and repelled from each other, resulting in centroid matrix with larger nuclear and lower similarity.
2.1 MANIFOLD CAPACITY THEORY
Consider set of P manifolds embedded in feature space of dimensionality D, each assigned random binary class label. Manifold capacity theory is concerned with the question: what is the largest value of oJ such that there exists (with high probability) hyperplane separating the two classes? Recent theoretical work has demonstrated that there exists a critical value, dubbed the manifold capacity ac, such that when D oc the probability of finding separating hyperplane
D approximately 1.0, and when P. > ac the probability is approximately 0.0. Furthermore, ac can be accurately predicted from three key quantities: (1) the manifold radius Ras, which measures the size of the manifold relative to its distance from the origin, (2) the manifold dimensionality DM which estimates the number of dimensions along which manifold has significant extent, and (3) the centroid correlation (if the positions of manifolds are correlated with each other they will be more difficult to separate). In particular, when the centroid correlation is low the manifold capacity can be approximated by di (DM)DM) where s(-) is a monotonically decreasing function.
For manifolds of arbitrary geometry calculating the manifold radii and dimensionalities involves iterative process that alternates between determining the set of "anchor points" on each manifold that relevant for the classification problem, and computing the statistics of random projections of these anchor points (Cohen et al., 2020). This process is both computationally costly and non- differentiable, and therefore not suitable for use as an objective function. For more detail on the general theory see A.2 However, if the submanifolds are assumed to be elliptical in shape there is an analytical expression for each of these,
E, Σ
RM = VE DM
(1)
where the X3 are the eigenvalues of the covariance matrix of points the manifold. For reference, for batch of 100 128-D manifolds with 100 points sampled from each, computing these elliptical- assuming measures is approximately 500 times faster in terms of wall-clock time.
Using these definitions for manifold radius and dimensionality we can write the capacity as ac <<E.,, o,) where d; are the singular values of matrix containing points on the manifold (which are the square roots of the eigenvalues of the covariance matrix). In this form, the sum the ob- jective function, this measure will prefer sparse solutions (i.c., a small number non zero singular values) corresponding to low dimensionality. is worth comparing this objective to another natural
L1 norm of the singular values, known as the Nuclear Norm of the matrix. When used
3

---

Under review as a conference paper at ICLR 2023
candidate for quantifying size: the determinant of the covariance matrix. The determinant is equal to the product of the eigenvalues (which captures the squared volume of the corresponding ellipse), but lacks the preference for lower dimensionality that comes with the Nuclear Norm. Specifically, since the determinant is zero whenever one (or more) eigenvalue is zero, it cannot distinguish zero- volume manifolds of different dimensionality. In Yu et (2020), lossy coding rate (entropy) is used as a measure of compactness, which simplifies to the log determinant under a Gaussian model Ma al. (2007). In that work, the identity matrix added to multiple of the feature covariance matrix before evaluating the determinant, which solves the dimensionality issue described above.
2.2 OPTIMIZING MANIFOLD CAPACITY
Manifold Capacity has been previously used to evaluate and compare network representations (Chung al., 2018 Cohen et al. 2020). Here, we explore its use as an objective function for self-supervised learning. For each input image (notated as a vector x E RP) we generate K sam- ples from the corresponding manifold by applying set of random augmentations (each drawn from the same distribution), yielding manifold sample matrix X, DxD Fach augmented image is transformed by Deep Neural Network, which computes nonlinear function f(x;; 0) parameterized by and the d-dimensional responses are projected onto the unit sphere yielding manifold response matrix Zi E Rd The centroid es is approximated by averaging across the columns. For a set of images {x1, XB} we compute normalized response matrices {Z1, .... Zz} } and assemble their corresponding centroids into matrix C' E Rdx
Given the responses and their centroids, the loss function is expressed as:
L= - + [
(2)
where -|- |l+ indicates the nuclear norm and À is a tradeoff parameter. The first term maximizes the extent the "centroid manifold" to encourage separability while the second term encourages object manifold compression.
Compression by Maximizing Centroid Nuclear Norm Alone Interestingly, the first term also has a compressive effect. This is because each centroid vector, as a mean of unit vectors, has norm that is linearly related to the average cosine similarity of vectors said unit vectors. Specifically,
2
K
k
||cb1|2
+
Zb,1
(3)
K
K2
k=
Here Zb.; denotes the representation of the 7th augmentation of æs. Then because the nuclear norm
is bounded below by the Frobenius norm (Recht et al. 2010), ||C]|+ 2 ||C|| = / ll maximizing the centroid nuclear norm optimizes an upper bound on the norms of centroid vectors, thus encouraging intra-object manifold similarity. We can gain further insight by considering how the distribution of singular vectors of a matrix depends on the norms and pairwise similarities of the constituent column vectors. While no closed form solution exists for the singular values of an arbitrary matrix, the case where the matrix is composed of two column vectors can provide useful intuition. If = C2], Zi [2 21. Z₂- = 2,, 22.2], the singular values of C and Z, are:
||.112 + ||2 12 = ((1ce 12 |c 12)2 4(cfF222)) 22)1/22
(CC) =
2
(4)
a (2,) =V1 + 2[, zi,2
So, [Io(C)) = ||C||. is maximized when the centroid vectors have maximal norms (bounded above by 1, since they are the centroids of unit vectors), and are orthogonal to each other. As we saw above the centroid norms is linear function of within-manifold similarity. Similarly, Ilo (Z.))l = IZ ||- is minimized when the within-manifold similarity maximal. So, both terms in the objective encourage object manifold compression (in the simple case described above the ef- fect is nearly mathematically equivalent). Surprisingly, this implies the first term alone encapsulates both of the key ingredients of contrastive learning framework, and we do observe that simply max- imizing || Cl|l is sufficient to learn a useful representation. This is because the compressive role of "positives" in contrastive learning is carried out by forming the centroid vectors, the objective is
4

---

Under review as a conference paper at ICLR 2023
not positive-free with X O. For example, if only single view is used the objective lacks compressive component and fails to produce a useful representation. We will refer to this version of the objective (with À = 0) "implicit manifold compression." In A.5 we demonstrate empir- ically that this implicit form effectively reduces || Z So, all three factors which determine the manifold capacity (radius, dimensionality, and centroid correlations) be elegantly expressed in an objective function with single term, - -ICC',..
Computational Complexity: The implicit form of our method involves computing a singular value decomposition of C' & Rdx which has complexity O(Bd x min [, d)), where B is the batch size and d is the dimensionality of the output. By comparison, many popular methods in SSL involve computing the pairwise cosine similarity between members of the batch, which has complexity ( Additionally, the complexity of our method (in implicit form) constant with respect to the number of views used (though the feature extraction phase is linear in the number of views), while pairwise similarity metrics will have quadratic complexity with the number of views.
3 METHODS
3.1 IMPLEMENTATION DETAILS
Architecture. For all experiments we use ResNet-50 (He et al. (for variants trained on CIFAR we removed max pooling layers). Following Chen et al. (2020). we append small perceptron with one hidden layer to the output the average pooling layer of the ResNet so that = g(h(zz)), where h is the ResNet and is the MLP.
2016 as a backbone architecture
Optimization We employ a standard set of augmenetations, taking them directly from Zhontar et al. 2021). For CIFAR and STL-10 we used relatively small batch sizes and the Adam optimizer with a fixed learning rate, for ImageNet- 100 we used batch size of 2048 for all three methods and the LARS optimizer with linear warmup and cosine decay. We used large number of views for small datasets with MMCR (40 for CIFAR and 20 for STL), and a much smaller number of views for ImageNet- 100 (we report results for 4 views and 2 views). For more details on specific hyperpa- rameter settings see A.3
3.2 EVALUATION METHODS
Classification Accuracy. We follow a standard linear evaluation protocol: the parameters of the encoder, f are frozen and a linear function is trained classify images using the standard supervised cross entropy loss function, see A.6 details. We pre-train and evaluate classification performance on CIFAR-10/100, STL-10, and ImageNet-100 100 (a 100 class subset of the full ImageNet Dataset) Krizhevsky et al. 2009 Coates et al. 2011 Deng et al. 2009).
Mean Field Theory Manifold Analysis. In manifold capacity theory et al. (2018), the radius and dimensionality of manifolds with arbitrary geometry (i.e., not restricted to elliptical form) are determined by the statistics of particular anchor points within the convex hull of the manifold. The anchor points of each manifold are those that uniquely specify the maximum margin separating hyperplanes between said manifold and other manifolds. Though this measure is not well suited for use as an objective, we use it to analyze the capacity of our self-supervised representation (see Fig. 33)
Subspace Angle. Besides measuring the size and dimensionality of individual object manifolds we also wish to characterize the degree of overlap between pairs of manifolds. For this, we measure the angle between their subspaces (Knyazev & Argentati] 2002).
Shared Variance. Object manifolds will generally have a lower intrinsic dimensionality then the space they embedded in. Therefore, the data will have low variance along several of the principal vectors used to calculate the set of subspace angles, and so many of the principal angles will have little meaning. To address this limitation we also compute the shared variance between the linear subspaces that contain object manifolds.
Adversaiial Robustness. Once linear classifier has been trained top of the representation, the end-to-end system can be subjected to standard adversarial attacks. We evaluate adversarial robustness under lo PGD attacks and AutoAttack (Madry et al. 20188 Croce & Hein 2020).
5

---

Under review as a conference paper at ICLR 2023
Table 1: Top-1 classification accuracies of linear classifiers for representations trained with various datasets and objective functions. Note: for Barlow Twins on ImageNet-100 100 we report the result from da Costa et al. (2022) which uses a ResNet-18 backbone, as we were unable to obtain better performance. For MMCR on IaageNet-100 we tested both 2 views (matched to baselines) and views, results are formatted (2-vie))(44vvew)
Method
CIFAR-10
CIFAR-
STL-10
ImageNet-100
Barlow Twins (our repro.)
90.91
67.91
89.96
80.38*
SimCLR (our repro.)
92.22
70.04
91.11
79.64
Implicit MMCR (ours)
93.53
69.87
90.62
81.52/82.88
MMCR (A 0.01)
93.39
70.94
90.77
81.28/82.56
Baseline Models We chose SimCLR (Chen et al. 2020) and Barlow Twins (Zbontar et al., 2021) for primary comparisons with our model. These methods are recent, simple to implement, and achieve performance comparable to state of the art for self-supervised learning on the Imagenet- Ik dataset.
3.3 PERFORMANCE
details the evolution of the representation during the course of training. The centroid nuclear norm (Fig. 2b) increases steadily as the centroids become increasingly orthogonal to each other (Fig. 2c) and grow in norm (Fig. 2d). The compression of individual augmentation manifolds is reflected in Fig. 2a. The downstream classification accuracies are reported in Table [ Subsequent analyses are carried on the CIFAR-I0 dataset using the implicit MMCR model it achieved the highest performance.
Figure 2
/Zp/--
|||||
Centroid Similarity
Centroid Norm
100.0
0.05
6.8
0.02
0
50
100
150
200
0.75
100
350
Epochs
Epochs
Epochs
Epochs
Figure 2: Evolution of various metrics during training. Geometric measures are evaluated set of 200 manifolds, each defined by an image drawn from the CIFAR-10 dataset, along with 16 augmentations. Shaded regions indicate 95% confidence interval around the mean.
3.4 REPRESENTATION GEOMETRIC ANALYSIS
In figure 3 we show that our representation, which is optimized using an objective that assumes ellip- tical manifold geometry, nevertheless yields representations with high mean field manifold capacity (relative to baseline methods). For completeness we also analyzed the geometries of class mani- folds, whose points are the representations of different examples from the same class. This analysis provided further evidence that learning to maximize augmentation manifold capacity compresses and separates class manifolds, leading use different strategy than the baseline methods to increase the capacity, namely MMCRs product class/augmentattinn manifolds with larger radii, but lower dimensionality (Fig.
useful representation. Interestingly MMCRs seem to
3.5 EMERGENCE OF NEURAL MANIFOLDS VIA GRADIENT COHERENCE
We hypothesize the class separability in MMCRs arises because augmentation manifolds corre- sponding to examples from the same class are optimally compressed by more similar transforma- tions than those stemming from distinct classes. To investigate this empirically, evaluate the

---

Under review as a conference paper at ICLR 2023
Augmentation Manifolds
Class Manifolds
Dimension
00.0
46.0
50.0
16.0
20.3
20.0
Radius
1.3
Capacity
8.52
Barlow Twins
SimCLR MMCR
2:08
0.0
8.4
Representation
0.0
Represmtation
Picels
Hierarchy
Phads
ResNet Hilearchhy
Figure 3: Mean Field Manifold Capacity Analysis. The shared x-axis of all plots is the representa- tional hierarchy, the leftmost entries represent the inputs (pixels) and the rightmost the output of the encoder/learned representation. The top row shows the manifold radius, the middle the dimensional- ity, and the bottom the resultant capacity. Shaded regions indicate a 95% confidence interval around the mean (analysis was conducted with different random samples from the dataset, see A.4).
gradient of the objective function for inputs belonging to the same class. We can then check whether gradients obtained from (distinct) batches of the same class obtained from different classes, which would suggest that the strategy for compressing augmenta- tion manifolds from the same class are relatively similar to each other. Figure demonstrates that this is the case: within class gradient coherence, as measured by cosine similarity, is consistently higher than across class coherence across both training epochs and model hierarchy.
similar each other than those
All Parameters
First Layer
Last Layer
BL
Gradient Simllatty
Distinct
Same Clas
BWV
0.2
6.2
BWV
100
1.0
92
140
200
60
100
152
Epochts
Epochh
Epochs
Figure 4: Gradient cosine similarity for pairs of single-class batches. We plot the mean pairwise similarity for pairs of gradients for for different subsets of the model parameters (all parameters, and the first and last linear operators) obtained from single-class-atthees coming from the same or distinct classes over the course of training. To the left is visualization of the fact that single-class gradients flow backward through the model more similar directions.
3.6 MANIFOLD SURSPACE ALIGNMENT
Within-class gradient coherence constitutes a plausible mechanistic explanation for the emergence of class separability, but does not explain why members the same class share similar compression strategies. To begin answering this question we examine the geometric properties of augmentation manifolds in the pixel domain. Here we observe small but measurable differences between the distributions of within-class similarity and across-class similarity, as demonstrated in the top row of figure 5 The subtle difference in the geometric properties of augmentation manifolds in the pixel domain in turn leads to the increased gradient coherence observed above, which over training leads to a representation that rearranges and reshapes augmentation manifolds from the same class similar fashion (bottom row of figure 5), thus allowing for the linear separation of classes. Not only

---

Under review as a conference paper at ICLR 2023
centroids of same-class manifolds in more similar regions of the representation space than those coming from distinct classes (Fig third column bottom row) but additionally same-class manifolds mlsmmmlls-ma have more similar shapes to each other (Fig (Figtoootomm bottom row columns 1 and 2).
same-clas-maniiools
Subspace Angle
Shared Variance
Centroid Simlonity
Pacls
Augreaaionn Vaaded
Cleon literied
Dog Clasa Mandbad
Ress eeireseeinn Resiessettation rtation
Figure 5: The distributions of various similarity metries for augmentation manifolds from either the same and distinct classes. In the top row we consider augmentation manifolds in the pixel domain, and in the bottom row we observe how these distributions are transformed by the learned representation. To the left schematic shows details the exemplat-auggmenttioon manifold-class manifold structure of the learned representation.
We next ask how the representation learned according to the MMCR objective differs from those optimized for other self supervised loss functions. While MMCR encourages centroids to be as close orthogonal each other, the InfoNCE loss employed Chen al. (2020) benefits when negative pairs are as dissimilar as possible, which is achieved when the two points lie opposite regions of the same subspace rather than in distinct subspaces. The Barlow Twins (Zbontar al. 202 loss is not an explicit function of feature vector similarities, but instead encourages individual features to be correlated across the batch dimension and distinct features to be uncorrelated. In we demonstrate that these intuitions are borne out empirically: the MMCR representation produces augmentation manifold centroids that are significantly less similar to each other than the two baseline methods.
Same Class
Distinct Classes
0.16
Barlow Twins SimCLR
MMCR
Probability
0.04
0.2
0.4
0.6
0.8
0.2
0.4
0.8
Centroid Similarity
Centroid Similarity
Figure 6: Centroid similarities for models trained according to different SSL objectives. The left panel shows the distribution of centroid cosine similarities for augmentation manifolds for examples of the same class, while the right shows the same distribution for examples from distinct classes.
3.7 ADVERSARIAL ROBUSTNESS
Previous works that used similar geometrically motivated loss functions such the orthogonal low-rank embedding (Lezama et al. 2018) and maximal coding rate reduction Yu et al.) (2020) have reported the resulting representations have increased inter-class margins and are more robust to label noise. We therefore tested whether the increased tendency to orthogonalize in MMCR models leads to any benefit in terms of adversarial robustness. In we show that the MMCR model (and attached classifier) is indeed more robust than either Barlow Twins or SimCLR trained models against PGD attacks with range of strengths (Madry al. 2018). We found similar results using the stronger
8

---

Under review as a conference paper at ICLR 2023
Infinity Norm
Figure Adversarial Robustness of SSL Models under PGD Attack. For each of the three SSL models with trained classifiers, we apply Projected Gradient Descent (PGD) with -nom perturbation under 50 attack iterations. Inputs were scaled such that their stan- dard deviation was 1.0, raw attack strengths on the x-axis. Ad- ditional details can be found in A.7
Robust Accuracy
Barlow Twins SimCLR MMCR
10"
10'4
report the
10a
0.0
0.4
0.6
Attack Strength
AutoAttack protocol (Croce & Hein 2020), see A.7 Note that Barlow Twins models seem to be more robust than those arising from SimCLR. We speculate that this is result the decorrelation (rather than anti-correlation) encouraged by the Barlow Twins objective.
4 DISCUSSION
Previous work has proposed manifold capacity theory means of understanding the performance of deep neural networks trained using traditional supervised methods (Cohen et al. 2020), Given that manifold capacity theory is particularly well suited for analyzing high dimensional feature spaces, while many SSL methods are motivated by information criterion, we wondered whether manifold capacity could serve as a useful objective function. By approximating the manifold geometries as elliptical, which significantly reduces the computation required to calculate the geometric properties that dictate capacity, we able achieve exactly this. Although representational manifold ge- ometries are generally not elliptical, we have demonstrated that this approximation can nonetheless produce useful learning signal, and leads to networks with high manifold capacity (Fig. (3. Nev- ertheless, would be interesting to consider other reductions of the mean field manifold capacity theory that can capture non-elliptical structure of individual manifolds, perhaps by computing higher order statistics of constituent points.
We demonstrate the utility of our approach by applying the method three small datasets of un- labeled images and demonstrating that the resulting representations can yield similar downstream task performance to two baselines. Furthermore we conducted gradient based analysis to begin to understand why the self-supervised learning signal is capable of producing useful representations. Finally, motivated by an empirical exploration of the geometrical differences between the represen- tations produced by the three considered methods, we demonstrate that MMCRs can offer improved robustness to adversarial attacks.
Additionally, leveraged manifold capacity analysis in its full generality to gain insight into the geometry of the MMCR. Intriguingly, our method produces augmentation and class manifolds with lower dimensionality but larger radius than either Barlow Twins SimCLR (Fig. 3) Future work will seek to understand why this is the case, but more generally this suggests that capacity anal- ysis can be fruitful way to understand the different encoding strategies encouraged by various SSL paradigms. Another factor that distinguishs MMCRs from other models is a tendency to or- thogonalize augmentation manifold centroids and thus form a representation that is globally high dimensional. Given that the recent observations that the representations in visual cortex and high performing models of visual cortex are surprisingly high dimensional (Stringer et al.) 2019 E1- moznino & Bonner 2022), it may be interesting to test how well MMCRs predict neural activity.
In this study, we introduced one specific model of learning using metrics based on a specific theory of representations: self-supervised learning via maximizing neural manifold capacity. More broadly, this work demonstrates the efficacy of a learning objective solely based on representations (rather than relying on classifiers), and motivates new directions of research where representation--evell phenomena in neurobiology inform the design choice in neural networks. With recent trends in neuroscience focused representation geometric observations in neural data, we hope that this work lays foundations for future studies learning based on representation geometries informed by new discoveries in neuroscience.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam Schoenholz, Jascha Sohl-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11(1), 2020.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learming, pp. 531-540. PMLR, 2018.
Silvia Bernardi, Marcus K Benna, Matria Rigotti, Jérôme Munuera, Stefano Fusi, and C Daniel Salzman. The geometry of abstraction in the hippocampus and prefrontal cortex. Cell, 183(4): 954-967, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-99244 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Yeon Chung and LF Abbott. Neural population geometry: An approach for understanding bio- logical and artificial neural networks. Current opinion neurobiology. 70:137-144, 2021.
Yue Chung, Daniel Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds, Physical Review X, 8(3):03 1003, 2018.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Uri Cohen, Sue Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):1-13, 2020.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with ensem- ble of diverse parameter-free attacks. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self-supervised methods for visual representation learning. J. Mach. Learn. Res., 23: 56-1, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 11(8):333-344 2007.
Yann Dubois, Tatsunori Hashimoto, Stefano Ermon, and Percy Liang. Improving self-supervised
learning by characterizing idealized representations. arXiv preprint arXhv:2209.06235. 2022.
Eric Elmoznino and Michael F Bonner. High- high-performing neural network models of visual cortex benefit from high latent dimensionality. bioRxiv, 2022.
Juan A Gallego, Matthew G Perich, Lee E. Miller, and Sara A Solla. Neural manifolds for the control of movement. Neuron, 94(5):978-98) 2017.
Gerl and Krey. Storage capacity and optimal learning of potts-model perceptrons by a cavity method. Journal of Physics A: Mathematical and General, 27(22):7353, 1994.
10

---

Under review as a conference paper at ICLR 2023
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised con- strastive learning. In International Conference on Machine Learning. pp. 3821-3830. PMLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hénaff, N Rabinowitz, Ballé, and E Simoncelli. The local low-dimensionality of natural images. In Inr'l Conf on Learning Representations (ICLR), San Diego, CA, May 2015. URL
wwt.:o/gggabbbaooooo.o abt///bss///// 6626
Olivier Hénaff, Yoon Bai, Julie A Charlton, Ian Nauhaus, Eero Simoncelli, and Robbe LT Goris. Primary visual cortex straightens natural video trajectories. Nature communications, 12(1):1-12, 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv:/4/2.6990 2014.
Andrew V Knyazev and Merico E Argentati. Principal angles between subspaces in an a-based scalar product: algorithms and perturbation estimates. SIAM Journal on Scientific Computing, 23 (6):2008-200 2040, 2002.
Nikolaus Kriegeskorte and Rogier A Kievit. Representational geometry: integrating cognition, com- putation, and the brain. Trends in cognitive sciences, 17(8):401-41,, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Ole: Orthogonal low-rank embedding- plug and play geometric loss for deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8109-8118, 2018
Jianfeng Lu and Stefan Steinerberger. Neural collapse with cross-entropy loss. arXiv preprint arXiv:2012.08445. 2020.
Yi Ma. Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE Iransactions on pattern analysis and machine intelligence, 2949):154611562, 2007.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations, 2018. (tRLtt/tt://// /ttpa/////peeeeeeeeeeeme net.ee.wweeeeeeeeeeee..eee net/ foo.
Edward H Nieh, Manuel Schottdorf, Nicolas W Freeman, Ryan Low, Sam Lewallen, Sue Ann Koay, Lucas Pinto, Jeffrey L Gauthier, Carlos D Brody, and David W Tank. Geometry of abstract learned knowledge in the hippocampus. Nature, 595(7865):80-88 84, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint 18Xiv:1807.03744. 2018
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Benjamin Recht, Maryam Fazel, and Pablo Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3):471-5011 2010.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High-dimensional geometry of population responses in visual cortex. Nature, 571(7765):361- 365, 2019.
11

---

Under review as a conference paper at ICLR 2023
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 1929-99399 PMLR, 2020.
Yu Wang, Jingyang Lin, Qi Cai, Yingwei Pan, Ting Yao. Hongyang Chao, and Tao Mei. A low rank promoting prior for unsupervised contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-9434 2020.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference Machine Learning, pp. 12310- 12320. PMLR, 2021.
A APPENDIX
A.I PYTORCH STYLE PSEUDOCODE FOR MMCR
h: encoder
g: projection head
B: batch size
K: number of augmentations
D: projecter output dimensi
onal dimensionalits
Imbda: trade-off parameter
for x in Loader:
# К randomly augmented views
10
X multi_augment (x) $ X
12
push through encoder and projector
13
z (x()x) F
14
# project onto unitsphere normalize (z, dim=-1)
16
17
18
calculate controids (mean over augmentation axis) = Amean/dim=1] B D
calculate singular values U_2, $_,, V_z (2) batch svd U_c, S_c, V_c = svd(c)
22
23
calculate loss loss -1.0 * (S Imbda sum
26
27
28
backward pass and optimization step loss. .assbbacwwrd)))
30
optim.step step())
A.2 MEAN FIELD THEORY MANIFOLD CAPACITY BACKGROUND INFORMATION
Mean Field Theory Recall the problem setting for manifold capacity analysis: given a set manifolds embedded in feature space of dimensionality D. each assigned random binary class label Chung et al. (2018). Manifold capacity theory is concerned with the question: what is the largest value of 5 such that there exists (with high probability) hyperplane separating the two
12

---

Under review as a conference paper at ICLR 2023
classes? In the thermodynamic limit, where P.D. D oo but f remains finite, the inverse capacity can be written exactly.
aM = =B][F(T)
(5)
where, F(T) - ming { {1V 11 (s)sss 2 0). S the set defining the manifold geometry (i.e. the set of vectors S that are points individual manifold), T are random vectors drawn from white multivariate Gaussian distribution, and gs gs ming(V
S), is the concave
support function.
The KKT equations for this convex optimization problem are:
V- --T-15(() A5(T)) 0
x 2 0
(6)
gs k 2 0
O. A us V)
where S(7) is subgradient of the support function. When the support function is differentiable, the subgradient is unique and equal to the gradient,
5(T) = Vgs = arg min V S
SEs
S(T) is the unique point in the convex hull of S that satisfies the first KKT equation, and is called the "anchor point" for S induced by the random vector T.
Equivalent Interpretation of Anehor Points For given dichotomy (random binary class labelling) the weight vector of the maximum margin separating hyperplane be decomposed into sum of at most P vectors, with each manifold contributing a single vector, which lies within the convex hull of the manifold. The position of said point point is function of the manifolds position relative to all of the other manifolds in the space and depends on the particular set of random labels. Thus there exists distribution Using the "cavity" method it can be shown that these points are none other than the anchor points that are involved in solving the optimization problem described above Gerl & Krey (1994).
for each individual manifold.
Numerical Solution To solve the mean field equations numerically, one samples several random Gaussian vectors T, and then for each T, V and Š are determined by solving the quadratic program- ming program given above. The capacity is then estimated as the mean value of or the samples T.
Manifold Geometries The way the capacity varies in terms of the statistics of the anchor points can be simplified by introducing two key quantities, the manifold radius Ras and manifold dimension- ality Ry:
R2, (T)) DM = [T Š(7)
(8)
where Š(7) is a unit-vector in the direction of the anchor point S. In particular as discussed in the main text, the manifold capacity can be approximated by ø Ras /DM) where & is a monotonically decreasing function.
Elliptical Geometries In the case where the manifolds exhibit elliptical symmetries, the manifold radius and dimensionality can be written in terms of the eigenvalues of the covariance matrix of the
13

---

Under review as a conference paper at ICLR 2023
anchor points:
12 = Σ X2
(9)
DM = EE 2
Σ
So, in this case RM is the total variability of the anchor points, and DM is generalized participation ratio of the anchor point covariance, a well known soft measure of dimensionality.
A.3 ADDITIONAL PRE-TRAINING INFORMATION
Settings for CIFAR/STL-10 We take the parameters of each augmentation directly from Zbontar jt al. (202T), but for these lower resolution images we omitted Gaussian blurring and solarization augmentations. All models were trained for 500 epochs using the Adam optimizer (Kingma & Baj 2014) with a learning rate of 1e 3 and weight decay of 6. For all three methods we used a one hidden layer MLP with hidden dimension of 512 and output dimension of 128 for the projector head g. We swept batch size for each method and chose the one that resulted the highest downstream task performance. For both SimCLR and Barlow Twins we found that batch size of 128 was optimal (among 32, 64, 128, 256, and 512) for all datasets. For MMCR there is trade-off between batch size and the number of augmentations used, and the optimal value of that trade-off is highly dataset dependent. For CIFAR-10 10 and CIFAR-100 we used batch size of 32 and 40 views, and for STL-10 we used batch of 64 with 20 views For Barlow Twins we used À 128 which normalizes for the number of elements in the on-diagonal and off-diagonal terms in the loss. For SimCLR we used the recommended setting of 0.5. The overall performance of both baseline methods (and likely MMCR as well) could be increased with more thorough hyperparameter search and by employing methodology that more closely matches the original works. For example, both methods would likely benefit from the combination of larger batch size, the use of the LARS optimizer (which is designed for large batch optimization), a learning rate scheduler consisting of linear warm- up followed by cosine annealing, longer training, and the use of more diverse augmentations (i.c. including solarization and gaussian blur). Additionally Barlow Twins reports that the representation can benefit from using much larger projector network than use. Because our goal was primarily to demonstrate that MMCR can produce representations that are comparable to these baselines rather than to produce sate-offthe-an -hrt results small scale datasets we opted for simplifications wherever possible (using off the shelf Adam for optimization with a fixed learning rate, and fixing architectural hyperparameters like the projector dimensionality).
1
Settings for ImageNet-100 For ImageNet we more closely match the pre-training procedures of previous works. We use a batch size of 2048 and a smaller number of views for MMCR (4), and also use the full suite of augmentations from Zbontar et al. (202I). For the sake of efficiency we train for a reduced number of epochs (200). For MMCR and SimCLR we modified the projector hidden dimensionality to be 4096 for the projector head, following the original work Chen et al. (2020). For Barlow Twins we used the recommended 2-layer MLP with hidden and output dimensions of 8192, and set À = 5e 3, however these hyperparameters were optimal for the full ImageNet dataset, and not neccesarrily for ImageNet- 100. We were unable to achieve better downstream performance using a ResNet-50 backbone than what has previously been reported in the literature for this dataset with a ResNet-18 backbone, therefore we report the ResNet- 18 performance reported in (da Costa For SimCLR we use 0.1 which is the recommended setting for larger batch sizes.
et al. 2022
A.4 DETAILS OF REPRESENTATIONALL ANALYSES
Manifold Capacity Analysis For each pre-trained model, we extract layer activations across the ResNet hierarchy after a forward pass of a set of images. For class manifold analysis, the set of images contain 10 classes, where each class has 100 examples. Augmentation manifolds instead have 100 exemplars with 100 examples each. Following Cohen al. (2020), we take activations from all convolutional layers in ResNet-50 after ReLU non-linearity. The specific extracted layers highlighted in bold fonts are given by Table 2 The final analysis results are averaged over five data samplings with different random seeds and random projections of intermediate features to lower- dimension spaces (default 5000 dimensions).
14

---

Under review as a conference paper at ICLR 2023
Table 2: A Total of Extracted ResNet-50 Layers (in Bold) for MFTMA Analysis
Layer
Type
Conv2d Size (H W X C)
pixel
Input
None
Conv2d BatchNorm
conv1
x 1
[ 7 64 x
ReLUU
Conv2d BatchNorm
x 64
x 3
conv2.x
x 3
x
x 3
ReLU
X 1 X 256
Conv2d BatchNorm
[1 1 x 128
x
conv3_x
x 4
3 x 128
x
ReLU
1 512
Conv2d BatchNorm
[
x 256
x 3
conv4_x
X 6
3 3 X 256
x
ReLU
1024
Conv2d
x 512
x
conv5_x
BatchNorm
3
3 3 X 512
x
ReLU
X 2048
Gradient Coherence Analysis In Fig for each of the classes of CIFAR-10, we generate 100 batches of 32 augmentation manifolds of samples from specific class (with 40 augmentations each). We then measure the gradient of the loss function for each batch during different stages of training, and compute the cosine similarity between every pair of gradients. Across all stages of training the mean cosine similarity between gradients generated from batches of the same class is larger than those from distinct classes (left column). This observation remains true when isolating the gradients of parameters from different stages of in the resnet-50 hierarchy (center and right columns, respectively).
Manifold Subspace Alignment For Fig. 5 we generated 100 samples from the augmentation man- fiolds of 500 images in the CIFAR-I0 dataset. We then measure the mean subspsace angle (left column), fraction of shared variance (middle column) and centroid cosine similarity between each pair manifolds. The same procedure was used for generating the data for 6
A.5 IMPLICIT MMCR EFFECTIVELY REDUCES AUGMENTATION MANIFOLD NUCLEAR NORM
To test whether or not implicit manifold compression actually reduces the mean augmentation man- ifold nuclear norm, we can vary the value of X. Below see the evolution of bother loss for several different values of lambda during training on CIFAR-10. 10. For these experiments the batch size was and the number of augmentations per image was 4.0. As shown in [8] the level of compression of individual manifolds is nearly the same across all values of the tradeoff parameter tested.
of the
A.6 CLASSIFICATION EVALUATION PROCEDURE
During pre-training all models monitored with k-nearest neighbor classifier (k=200) and checkpointed every epochs. After pre-training, we trained linear classifiers on all checkpoints whose monitor accuracy within 1% of the highest observed accuracy, and select the model that
15

---

Under review as a conference paper at ICLR 2023
Centrold Nudear Norm
Mean Manifold Nuclear Norm
52.0
2.4
[C||
[y Zn
46.0
2.0-
Lambds 0.01
Lambda 0.05
Lambdo 0.1
Lambds
40.0
80.0
00.0
Epochs
Epochs
Figure Validation loss values for different values
achieves the highest linear classification accuracy. Linear classifiers were trained using the Adam optimizer with batch size 1024 and an initial learning rate of 0.1, which decayed according to cosine scheduler over the course of 50 epochs. For the linear classifier training, at train time we use the same set of augmentations during unsupervised pretraining, at test time we only use center cropping and random horizontal flipping.
A.7 ADDITIONAL DETAILS FOR ADVERSARIAL ROBUSTNESS ANALYSES
In Figure 7, we choose 50 iterations for the PGD -nom since it guarantees a robust accuracy value far away from asymptotically larger PGD attack iterations (Madry et 2018 Croce & Hein 2020). In our experiment, we have shown that the PGD attack indeed converges in a similar fashion (See Figure However, the robust accuracy for MMCR tends converge larger PGD attack iterations.
Inf No 23/225 Epsionl
Norm 01000225 Epolion)
Norm 11607225 Epsiion
int Norm 1160/225 Epstoni
Accnny
SinLUl
tttii sttitiin
Altack teeations
Attack Iherstions
Atlock Breaaiions
Attacic steemtton
Figure 9: Convergence for different settings of adversarial attack strengths
We therefore also analyzed the robust accuracies for the three SSL methods with varying iterations across all epsilon attack strength. Figure [0 shows MMCR exhibits a significantly higher robust accuracy compared Barlow-Twins and SimCLR in the low iterations regime.
Aside from the standard PGD adversarial attack, we also tested three SSL methods under the Au- toAttack protocol. The lo -orm AutoAttack accuracy given by Table
16

---

Under review as a conference paper at ICLR 2023
Ind (nnaations)
Inf (40 tonations)
Int Norm (70 Iterat ong)
Nnf (100 Itorations)
Sencun
Robust
Attadi Stranggh'
Attack Strength
Attack Strangth
Attack Strongth
Figure 10: PGD -nom attack with varying iterations.
Table AutoAttack -orm Robust Accuracy
Method
Clean Accuracy
Eps = 40/255
Eps = 160/255
Barlow Twins (our repro.)
90.91
74.55
31.53
SimCLR (our repro.)
92.22
72.48
26.37
Implicit MMCR (ours)
93.53
75.88
32.47
17


Reviews:
[{"review_id": "bxCXHzAbtM", "reviewer": "Reviewer_Syek", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the self-supervised learning via maximizing manifold capacity. Specifically, it incorporates the manifold capacity into a contrastive objective. It maximizes the extent of the global image manifold and minimizes every object manifold. The paper approximates the manifold geometries as elliptical to reduces the computation. Experiments on object recognition and robustness to adversarial attacks test the effectiveness of the proposed approach.", "summary_of_the_review": "Overall, this paper could be an interesting algorithmic contribution. However, my main concern is the novelty of the paper. Currently, I am leaning toward rejecting the paper. ", "strength_and_weaknesses": "Strength:\nMany self-supervised learning methods try to optimize an approximation of the mutual information between representations of different views. The paper uses the manifold capacity to maximize the number of linearly separable object manifolds. It is a new approach that prevents the collapse in the representation space in contrastive learning.\n\nWeakness:\n1.\tIt is better to discuss the relation with dimensional collapse in contrastive learning. Also, it does not compare with the methods addressed the limitation. For example, in the following paper, it uses the subspaces to represent global image manifold and develops new method to prevent the collapse in the representation space.\nLi Jing, Pascal Vincent, Yann LeCun, Yuandong Tian. Understanding Dimensional Collapse in Contrastive Self-supervised Learning. ICLR 2022.\n2.\tIt is better to detail the solution process of (2). For large image sets, how to solve problem (2) efficiently?\n3.\tIt is better to discuss the possibility to use sophisticated manifold geometries instead of elliptical.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UHPWofOCpE", "reviewer": "Reviewer_Xa59", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is an empirical paper that proposes a new self-supervised learning objective, which consists in minimizing of nuclear norm of the representation of positive examples (~low rank => low variability) while maximizing that of negative examples (more precisely their centroids). This is inspired by recent work on manifold capacity, ie, the number of linear classifiers that can separate point clouds/manifolds. The authors show on small datasets (CIFAR10/100 and STL10) that such objective performs well, and analyze how the geometry (eg distance between positives and negatives) changes during training and across layers. ", "summary_of_the_review": "I enjoyed reviewing the paper in that it brought to my attention novel work on manifold capacity that was developed outside of ML. Unfortunately, the paper is not self-contained, and provides no theory, and no convincing experiments. Although I do not think that the current version should be accepted, I think that future versions will be of interest to the community. If the main contribution is shedding light on the theory of manifold capacity (which would be fine), the authors should discuss those tools more formally (definitions, main statements). If the contributions go beyond that then the authors should scale up their experiments or provide some theoretical guarantees using the manifold capacity theory.\n\nI am happy to increase my score if actions take action on those suggestions or provide convincing arguments as to why those are not necessary.", "strength_and_weaknesses": "**Update**: I've updated my score since he authors added experiments on a non toy ImageNet-100 dataset, but I still find the contributions of the work minor.\n\n**Strengths**\n- **Shedding light on recent theory on manifold capacity** this paper sheds light on recent advancements on manifold capacity that can be of interest to the representation learning community.  These findings are the starting point of the paper (not a contribution) but disseminating these tools to the ML community has its own value given that those works were not published in ML venues. Unfortunately, the paper only provides some high-level informal discussion about those tools.\n- **problem is of interest to the community** representation learning is definitely of interest (even more so if it is easier to analyse theoretically**\n- **well written** the paper is generally well written and clear (besides the fact that it is not self contained as discussed below)\n\n**Weaknesses**\n- **Not self-contained** I had to partially read Chung 2018 and Cohen 2020 to understand the really understand section 2 (what is manifold capacity and why one could care). From the current paper, it is not even clear which definition they use for important terms that are used throughout (eg manifold, manifold radius, manifold effective dimensionality, and capacity).\n- **Unconvincing/small experiments** the method performs similarly to standard SSL on small datasets, it is unclear what to make out of that. Why is that important?\n- **Unclear or minor contributions**\n    - methodology: the paper essentially pretrains using a loss that was previously proposed for evaluation\n    - theory: no theory, not even proper discussion of the theory that inspired them\n    - experiments: the method performs essentially as well as standard SSL on small datasets. It is unclear what are the benefits of the method.\n- **Lack of discussion of previous work** there are many previous works that provides a similar intuition. Eg the following papers and reference therein:\n    - [[1]](https://arxiv.org/abs/2209.06235) this paper characterizes the optimal geometry of representations for maximizing linear probing capacity and generalization (and investigates them in practice). One possible such optimal representation is the sETF, which I believe is also what you are suggesting (ie minima to your loss.\n    - neural collapse literature: there's an entire line of work that analysis the geometry of representations and in particular sETF. Although this is mostly about supervised learning I think it is very related, one example is [[2]](https://arxiv.org/abs/2102.08817) which analysis contrastive learning\n   - [[3]](https://arxiv.org/abs/2106.04156) their loss (eq 6) is pretty related to your loss although their derivation comes from the decomposition of the augmentation graph. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SZr-Io8h114", "reviewer": "Reviewer_HB7z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Maximum Manifold Capacity Representation (MMCR) is proposed as a novel self-supervised learning framework by maximize the number of linearly separable object manifolds, which is interesting to see, and different from many SSL methods are inspired by information & entropy criterions. \n\nExperimental results on several computer vision data (e.g., CIFAR-10, STL-100, CIFAR-100) are included for a number of applications, e.g., classification accuracy, manifold capacity analysis, subspace alignment, adversarial robustness, etc. ", "summary_of_the_review": "This is an interesting work in the SSL domain, the introducing of the concept of manifold capacity is informative, and overall technical contribution seems to be marginal but solid. Perhaps the empirical result is relative limited as no good results from large data set included yet, and my recommendation for this paper is 6 \"marginally above the acceptance threshold\".", "strength_and_weaknesses": "Strengths:\n\nIntroduction the concept of manifold capacity to SSL is quite interesting and seems to be the first work in this direction (unless I missed some prior works). By utilizing the nuclear norm with solid theoretical support, this proposed loss function in eq. 2 is well presented as the combination of \"encourage separability\" plus \"manifold compression\". \n\nThis \"implicit manifold compression\" version of MMCR with only the first term , is also encourage object manifold compression and can learn useful representation. Also, the connection to previous work MCR2 is discussed, good to see the similarities and difference from this line of work (probably can have more). \n\nBoth Figure 1 and the analysis in Section 2.2 are good to have, to help understand intuition behind the proposed objective function. \n\nExperimental results on image data across a number of tasks from classification accuracy to manifold structure related analysis, this is helpful to support the contributions. \n\nWeaknesses:\n\nSynthetic data is often helpful to give us insights, and for this paper seems there is no results from simulated data experiments. It should be good to see how exactly \"manifold capacity\" been numerically optimized under simple toy examples.  \n\nGiven all experiments are performed under ResNet-50 as the backbone architecture, should be nice to have some computational complexity discussion for the proposed MMCR method, as this is less clear overall. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LJKaTTPbIV", "reviewer": "Reviewer_kyPr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a contrastive learning approach by maximizing manifold capacity via a nuclear norm for self-supervised representation learning. Experiments show that the proposed approach is able to yield better linear evaluation performance, extract sementically relevant features, and be more robust to adversarial attack.", "summary_of_the_review": "While introducing the nuclear norm from the perspective of measuring manifold capacity is new to contrastive learning, overall the novelty is marginal due to that the nuclear norm as a loss function has been used in contrastive learning.  The empirical evaluation is okay but not that strong. ", "strength_and_weaknesses": "The strengths of the paper: \n\n+ It is interesting and somewhat novel to introduce the measure of manifold capacity into contrastive self-supervised representation learning.\n \n\nThe weaknesses of the paper: \n\n- The novelty of introducing the manifold capacity into contrastive learning is somewhat weak. In this paper, the manifold capacity concept at the end of the day again turns out to be optimizing with a loss function of the nuclear norm on the centroids.  Nevertheless, the similar idea of using nuclear norm into contrastive learning has been explored in (Wang et al. TPAMI 2022). The differences from (Wang et al. TPAMI 2022) is not clearly discussed. \n\n- It is correct to relate the mean to the average consine similarity as Eq. (3). However, it seems problematic to interprete the rationale to keep only the first term in Eq. (2) by introducing a norm inequality, i.e., the nuclear norm is bounded from below by the Frobenius norm. Note that the optimization is to maximize the nuclear norm term, it might not guarantee to maximize its lower bound. \n\nMoreover, the reviewer is confused by the mixed expressions in Eq. (4). \n\n- In experiments, the computation time is not mentioned. Compared to the existing method, is there some advantage in computation time?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject