SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Efficient Coding of Natural Images using Maximum Manifold Capacity Representations
Thomas Yerxa  $^{1}$  Yilun Kuang  $^{2,3}$  Eero Simoncelli  $^{1,2,3}$  SueYeon Chung  $^{1,2}$
<sup>1</sup>Center for Neural Science, New York University
$^{2}$ Center for Computational Neuroscience, Flatiron Institute
$^{3}$ Courant Inst. of Mathematical Sciences,
tey214@nyu.edu
The efficient coding hypothesis posits that sensory systems are adapted to the statistics of their inputs, maximizing mutual information between environmental signals and their representations, subject to biological constraints. While elegant, information theoretic quantities are notoriously difficult to measure or optimize, and most research on the hypothesis employs approximations, bounds, or substitutes (e.g., reconstruction error). A recently developed measure of coding efficiency, the "manifold capacity", quantifies the number of object categories that may be represented in a linearly separable fashion, but its calculation relies on a computationally intensive iterative procedure that precludes its use as an objective. Here, we simplify this measure to a form that facilitates direct optimization, use it to learn Maximum Manifold Capacity Representations (MMCRs), and demonstrate that these are competitive with state-of-the-art results on current self-supervised learning (SSL) recognition benchmarks. Empirical analyses reveal important differences between MMCRs and the representations learned by other SSL frameworks, and suggest a mechanism by which manifold compression gives rise to class separability. Finally, we evaluate a set of SSL methods on a suite of neural predictivity benchmarks, and find MMCRs are highly competitive as models of the primate ventral stream.
Biological visual systems learn complex representations of the world that support a wide range of cognitive behaviors, without relying on a large number of labelled examples. The efficient coding hypothesis (8; 65) suggests that this is accomplished by adapting the sensory representation to the statistics of the input signal, so as to reduce redundancy or dimensionality. Visual signals have several clear sources of redundancy. They evolve slowly in time, since temporally adjacent inputs typically correspond to different views of the same scene, which in turn are usually more similar than views of distinct scenes. Moreover, the variations within individual scenes often correspond to variations in a small number of parameters, such as those controlling viewing and lighting conditions, and are thus inherently low dimensional. Many previous results have demonstrated how the computations of neural circuits can be seen as matched to such structures in naturalistic environments (48; 29; 65; 17) Studies in various modalities have identified geometric structures in neural data that are associated with behavioral tasks (10; 25; 43; 33; 56), and explored metrics for quantifying these structures.
The recent development of "manifold capacity theory" provides a more explicit connection between the geometry (size and dimensionality) of neural representations and their coding capacity (18). This theory has been used to evaluate efficiency of biological and artificial neural networks across modalities (17; 24; 66; 52). However, usage as a design principle for building model representations has not been explored.
Motivated by these observations, we seek to learn representations in which manifolds containing different views of the same scene are both compact and low-dimensional, while manifolds corresponding to distinct scene are maximally separated. Specifically:
- We develop a form of Manifold Capacity that can be used as an objective function for learning.
- We demonstrate that a Maximum Manifold Capacity Representation (MMCR) supports high-quality object recognition (matching the SoTA for self-supervised learning), when evaluated using the standard linear evaluation paradigm (i.e., applying an optimized linear classifier to the output of the self-supervised network) (14).
- Through a analysis of internal representations and learning signals, we analyze the underlying mechanism responsible for the emergence of semantically relevant features from unsupervised objective functions.
- We validate the effectiveness of MMCR as a brain model by comparing its learned representations against neural data obtained from Macaque visual cortex.
Our work thus leverages normative goals of representational efficiency to obtain a novel model for visual representation that is both effective for recognition and consistent with neural responses in the primate ventral stream.
Geometry of Neural Representations. Previous work has sought to characterize how representational geometry, often measured through spectral quantities like the participation ratio (the squared ratio of the  $l_{1}$  and  $l_{2}$  norms of the eigenvalues of the covariance matrix), shapes different aspects of performance on downstream tasks (32). Elmoznino and Bonner (27) found that high dimensionality in ANN representations was associated with ability to both predict neural data and generalize to unseen categories. Stringer et al. (67) observed that the spectrum of the representation of natural images in mouse cortex follows a power law with a decay coefficient near 1, and Agrawal et al. (2) report that (in artificial representations) the proximity of the spectral decay coefficient to one is an effective predictor of how well a representation will generalize to downstream tasks.
Self-Supervised Learning. Our methodology is related to (and inspired by) recent advances in contrastive self-supervised representation learning (SSL), but has a distinctly different motivation and formulation. Many recent frameworks craft objectives that are designed to maximize the mutual information between representations of different views of the same object (58; 14; 58; 68; 4)). However, estimating mutual information in high dimensional feature spaces is difficult (9), and furthermore it is not clear that closer approximation of mutual information in the objective yields improved representations (71). By contrast, capacity measures developed in spin glass theory (35; 1) are derived in the "large N (thermodynamic) limit" and thus are intended to operate in the regime of large ambient dimension (18; 5). We examine whether one such measure, which until now had been used only to evaluate representation quality, is also useful as an objective function for SSL.
Many SSL methods minimize the distance between representations of different augmented views of the same image while employing constraints to prevent collapse to trivial solutions (e.g., repulsion of negative pairs (14), or feature space whitening (77; 7; 28)). The limitations of using a single pairwise distance comparison have been demonstrated, notably in the development of the "multi-crop" strategy implemented in SwAV (13) and in the contrastive multiview coding approach Tian et al. (68). Our approach is based on the assumption that different views of an image form a continuous manifold that we aim to compress. We characterize each set of image views with the spectrum of singular values of their representations, using the nuclear norm as a combined measure of the manifold size and dimensionality.
The nuclear norm has been previously used to infer or induce low rank structure in the representation of data (42; 72; 49). In particular, Wang et al. (72) use it as a regularizer to supplement an InfoNCE loss. Our approach represents a more radical departure from the traditional InfoNCE loss, as we detail below. Rather than pair a low-rank prior with a logistic regression-based likelihood, we make the more symmetric choice of employing a high rank likelihood. This allows the objective to explicitly discourage dimensional collapse, a well known issue in SSL (45).
2 Maximum manifold capacity representations
Reduce Dimensionality
Reduce Radius
Figure 1: Two dimensional illustrations of high and low capacity representations. Left: the capacity (linear separability) of a random set of spherical regions can be improved, either by reducing their radii (while maintaining their dimensionalities), or by reducing their dimensionalities (while maintaining their average radii). Right: the objective proposed in this paper aims to minimize the nuclear norm (equal to the product of radius and sqrt dimensionality) of normalized data vectors (ie., lying on the unit sphere). Before training, the manifolds have a large extent and thus the matrix of their corresponding centroid vectors has low nuclear norm. After training, the capacity is increased: The manifolds are compressed and repelled from each other, resulting in centroid matrix with larger nuclear norm and lower similarity.
Before Training
After Training
2.1 Manifold Capacity Theory
Consider a set of  $P$  manifolds embedded in a feature space of dimensionality  $D$ , each assigned a class label. Manifold capacity theory is concerned with the question: what is the largest value of  $\frac{P}{D}$  such that there exists (with high probability) a hyperplane separating a random dichotomy (22; 34)? Recent theoretical work has demonstrated that there exists a critical value, dubbed the manifold capacity  $\alpha_{C}$ , such that when  $\frac{P}{D} < \alpha_{C}$  the probability of finding a separating hyperplane is approximately 1.0, and when  $\frac{P}{D} > \alpha_{C}$  the probability is approximately 0.0 (18). The capacity  $\alpha_{C}$  can be accurately predicted from three key quantities: (1) the manifold radius  $R_{M}$ , which measures the size of the manifold relative to the distance of its centroid from the origin, (2) the manifold dimensionality  $D_{M}$  which quantifies the number of dimensions along which a manifold has significant extent, and (3) the correlation of the manifold centroids. When the centroid correlation is low the manifold capacity can be approximated by  $\phi(R_{M}\sqrt{D_{M}})$  where  $\phi(\cdot)$  is a monotonically decreasing function.
For manifolds of arbitrary geometry, the radius and dimensionality may be computed using an iterative process that alternates between determining the set of "anchor points" on each manifold that are relevant for the classification problem, and computing the statistics of random projections of these anchor points (20). This process is both computationally costly and non-differentiable, and therefore unsuitable for use as an objective function. For more detail on the general theory see Appendix C. However, if the manifolds are assumed to be elliptical in shape, then both radius and dimensionality may be expressed analytically:
$$
R _ {M} = \sqrt {\sum_ {i} \lambda_ {i} ^ {2}}, \quad D _ {M} = \frac {\left(\sum_ {i} \lambda_ {i}\right) ^ {2}}{\sum_ {i} \lambda_ {i} ^ {2}}, \tag {1}
$$
where the  $\lambda_{i}^{2}$  are the eigenvalues of the covariance matrix of the manifold. For comparison, when computing these values for a set of 100 128-D manifolds with 100 points sampled from each, the use of the analytical expression is approximately 500 times faster (in "wall-clock time") than the general iterative procedure.
Using these definitions for manifold radius and dimensionality we can write the capacity as  $\alpha_{C} \approx \phi\left(\sum_{i} \sigma_{i}\right)$  where  $\sigma_{i}$  are the singular values of a matrix containing points on the manifold (equivalently, the square roots of the eigenvalues of the covariance matrix). In this form, the sum is the  $L_{1}$  norm of the singular values, known as the Nuclear Norm of the matrix. When used as an objective function, this measure favors sparse solutions (i.e., those with a small number of non-zero singular values) corresponding to low dimensionality. It is worth comparing this objective to another natural candidate for quantifying size: the determinant of the covariance matrix. The determinant is equal to the product
of the eigenvalues (which captures the squared volume of the corresponding ellipsoid), but lacks the preference for lower dimensionality that comes with the Nuclear Norm. Specifically, since the determinant is zero whenever one (or more) eigenvalue is zero, it cannot distinguish zero-volume manifolds of different dimensionality. Lossy coding rate (entropy) has also been used as a measure of compactness (76), which simplifies to the log determinant of the covariance matrix under a Gaussian model (50). In this case, the identity matrix is added to a multiple of the feature covariance matrix before evaluating the determinant, which solves the dimensionality issue described above.
2.2 Optimizing Manifold Capacity
Now we construct an SSL objective function based on manifold capacity. For each input image (notated as a vector  $\mathbf{x}_b\in \mathbb{R}^{\mathbb{D}}$ ) we generate  $k$  samples from the corresponding manifold by applying a set of random augmentations (each drawn from the same distribution), yielding manifold sample matrix  $\tilde{\pmb{X}}_b\in \mathbb{R}^{D\times k}$ . Each augmented image is transformed by a Deep Neural Network, which computes nonlinear function  $f(\mathbf{x}_b;\theta)$  parameterized by  $\theta$ , and the  $d$ -dimensional responses are projected onto the unit sphere yielding manifold response matrix  $\pmb{Z}_b\in \mathbb{R}^{d\times k}$ . The centroid  $\pmb{c}_b$  is approximated by averaging across the columns (response vectors). For a set of images  $\{\mathbf{x}_1,\dots,\mathbf{x}_B\}$  we compute normalized response matrices  $\{Z_{1},\ldots,Z_{B}\}$  and assemble their corresponding centroids into matrix  $\pmb{C}\in \mathbb{R}^{d\times B}$ .
Given the responses and their centroids, the MMCR loss function can be written simply:
$$
\mathcal {L} = - \left\| \boldsymbol {C} \right\| _ {*}, \tag {2}
$$
where  $||\cdot ||_{*}$  indicates the nuclear norm. The loss explicitly maximizes the extent of the "centroid manifold", which interestingly is sufficient to learn a useful representation. Concretely, maximizing  $||C||_{*}$  implicitly minimizes the extent of each individual object manifold as measured by  $||Z_{b}||_{*}$ . We build intuition for this effect below.
Compression by Maximizing Centroid Nuclear Norm Alone. Each centroid vector is a mean of unit vectors, and thus has a norm that is linearly related to the average cosine similarity of those unit vectors. Specifically,
$$
\left\| \boldsymbol {c} _ {b} \right\| ^ {2} = \frac {1}{K} + \frac {2}{K ^ {2}} \sum_ {k = 1} ^ {K} \sum_ {l = 1} ^ {k - 1} \boldsymbol {z} _ {b, k} ^ {T} \boldsymbol {z} _ {b, l} \tag {3}
$$
Here  $\boldsymbol{z}_{b,i}$  denotes the representation of the  $i^{th}$  augmentation of  $\boldsymbol{x}_b$ . We can gain further insight by considering how the distribution of singular vectors of a matrix depends on the norms and pairwise similarities of the constituent column vectors. While no closed form solution exists for the singular values of an arbitrary matrix, the case where the matrix is composed of two column vectors can provide useful intuition. If  $C = [c_1,c_2]$ ,  $Z_{1} = [z_{1,1},z_{1,2}]$ ,  $Z_{2} = [z_{2,1},z_{2,2}]$ , the singular values of  $C$  and  $Z_{i}$  are:
$$
\boldsymbol {\sigma} (\boldsymbol {C}) = \sqrt {\frac {\left\| \boldsymbol {c} _ {1} \right\| ^ {2} + \left\| \boldsymbol {c} _ {2} \right\| ^ {2} \pm \left(\left(\left\| \boldsymbol {c} _ {1} \right\| ^ {2} - \left\| \boldsymbol {c} _ {2} \right\| ^ {2}\right) ^ {2} + 4 \left(\boldsymbol {c} _ {1} ^ {T} \boldsymbol {c} _ {2}\right) ^ {2}\right) ^ {1 / 2}}{2}}, \tag {4}
$$
$$
\boldsymbol {\sigma} \left(\boldsymbol {Z} _ {i}\right) = \sqrt {1 \pm \boldsymbol {z} _ {i , 1} ^ {T} \boldsymbol {z} _ {i , 2}}. \tag {5}
$$
So,  $||\sigma(C)||_1 = ||C||_*$  is maximized when the centroid vectors have maximal norms (bounded above by 1, since they are the centroids of unit vectors), and are orthogonal to each other. As we saw above the centroid norm is a linear function of within-manifold similarity. Similarly,  $||\sigma(Z_i)||_1 = ||Z_i||_*$  is minimized when the within-manifold similarity is maximal. Thus the single term  $||C||_*$  encapsulates both of the key ingredients of a contrastive learning framework, and we will demonstrate below that simply maximizing  $||C||_*$  is sufficient to learn a useful representation. This is because the compressive role of "positives" in contrastive learning is carried out by forming the centroid vectors, so the objective is not positive-free. For example, if only a single view is used the objective lacks a compressive component and fails to produce a useful representation. In Appendix F we demonstrate empirically that this implicit form effectively reduces  $||Z_b||_*$  by comparing to the case where  $||Z_b||_*$  is minimized explicitly. So, all three factors which determine the manifold capacity (radius,
dimensionality, and centroid correlations) can be succinctly expressed in an objective function with a single term,  $-||\boldsymbol{C}||_{*}$ .
Computational Complexity. Evaluating the loss for our method involves computing a singular value decomposition of  $C \in \mathbb{R}^{d \times B}$  which has complexity  $\mathcal{O}(Bd \times \min(B, d))$ , where  $B$  is the batch size and  $d$  is the dimensionality of the output. By comparison, contrastive methods that compute all pairwise distances in a batch have complexity  $\mathcal{O}(B^2 d)$  and non-contrastive methods that involve regularizing the covariance structure have complexity  $\mathcal{O}(Bd^2)$ . Additionally, the complexity of our method is constant with respect to the number of views used (though the feature extraction phase is linear in the number of views), while pairwise similarity metrics have complexity that is quadratic in the number of views. It is also worth noting that doing implicit compression by maximizing  $||C||_*$  offers an advantage in computational complexity relative to explicit compression. This is because evaluating a term such as  $\sum_{b=1}^{B} ||Z_b||_*$  has computational complexity of  $\mathcal{O}(B^2 d \times \min(B, d))$ .
2.3 Conditions for Optimal Embeddings
Recently HaoChen et al. (39) developed a framework based on spectral decomposition of the "population augmentation graph", which provides theoretical guarantees for the performance of self-supervised learning on downstream tasks under linear probing. This work was extended to provide insights into various other SSL objectives by Balestriero and LeCun (6), and we show below that leveraging this approach can lead to explicit conditions for the optimality of representation under our proposed objective as well.
Given a dataset  $\mathbf{X}' = [\pmb{x}_1, \dots, \pmb{x}_N]^T \in \mathbb{R}^{N \times D'}$  we construct a new dataset by creating  $k$  randomly augmented views of the original data,  $\mathbf{X} = [\mathrm{view}_1(\mathbf{X}')', \dots, \mathrm{view}_k(\mathbf{X}')'] \in \mathbb{R}^{Nk \times D}$ . The advantage of doing so is that we can now leverage the knowledge that different views of the same underlying datapoint are semantically related. We can express this notion of similarity in the symmetric matrix  $\mathbf{G} \in \{0, 1\}^{Nk \times Nk}$  with  $G_{ij} = 1$  if augmented datapoints  $i$  and  $j$  are semantically related (and  $G_{ii} = 1$  as any datapoint is related to itself). We can normalize  $\mathbf{G}$  such that its rows and columns sum to 1 (so rows of  $\mathbf{G}$  are  $k$ -sparse with nonzero entries equal to  $1 / k$ ).
Now let  $\mathbf{Z} \in \mathbb{R}^{Nk \times d}$  be an embedding of the augmented dataset. Then we have  $\mathbf{G}\mathbf{Z} = [\mathbf{C},\dots,\mathbf{C}]^T$  where  $\mathbf{C}$  is the matrix of centroid vectors introduced above, and the number of repetitions of  $\mathbf{C}$  is  $k$ . Then because  $\sigma([C,\dots,C]) = \sqrt{k}\sigma(\mathbf{C})$  we can write MMCR loss function as,
$$
\mathcal {L} = - \left\| \boldsymbol {G} \boldsymbol {Z} \right\| _ {*} \tag {6}
$$
This connection allows us to make the following statements about the optimal embeddings  $Z$  under our loss, which we prove in Appendix A:
Theorem: Under the proposed loss, the left singular vectors of an optimal embedding,  $Z^{*}$ , are the eigenvectors of  $G$ , and the singular values of  $Z^{*}$  are proportional to the top  $d$  eigenvalues of  $G$ .
Architecture. For all experiments we use ResNet-50 (40) as a backbone architecture (for variants trained on CIFAR we removed max pooling layers). Following Chen et al. (14), we append a small perceptron to the output of the average pooling layer of the ResNet so that  $z_{i} = g(h(x_{i}))$ , where  $h$  is the ResNet and  $g$  is the MLP. For ImageNet-1k/100 we used an MLP with dimensions [8192, 8192, 512] and for smaller datasets we used [512, 128].
**Optimization.** We employ the set of augmentations proposed in (38). For ImageNet we used the LARS optimizer with a learning rate of 4.8, linear warmup during the first 10 epochs and cosine decay thereafter with a batchsize of 2048, and pre-train for 100 epochs. Note that though we report results using a default batch size of 2048, a batch size as small as 256 can be used to obtain reasonable results (1.2% reduction compared to batch size 2048 - see Appendix J for a sweep over batch size). We additionally employ a momentum encoder for ImageNet pre-training (all views are fed through an online network and an additional network whose parameters are a slowly moving average of the online network and all embeddings of each image are averaged to form centroids). We found the use of a momentum encoder provided a small advantage in terms of downstream classification
Table 1: Evaluation of learned features on downstream classification tasks. The leftmost columns show results for the standard frozen-linear evaluation procedure on ImageNet (IN). Results for most methods in this setting are taken from Ozsoy et al. (59) except for SwAV which is taken from the original paper (13). Columns 2 and 3 show semi-supervised evaluation on ImageNet (fine-tuning on  $1\%$  and  $10\%$  of labels). The results for this setting for VICReg and CorInfo Max are copied from Ozsoy et al. (59). The final three columns show frozen-linear evaluation on other datasets. We evaluated models for which pretrained weights in the 100 epoch setting were available online; MoCo, Barlow Twins and BYOL were taken from solo-learn da Costa et al. (23) (https://github.com/vturrisi/solo-learn), while SwAV and SimCLR were taken from VISSL Goyal et al. (37) (https://github.com/facebookresearch/vissl/blob/main/MODEL_Z00.md). For all evaluations we performed we report the mean and standard deviation over 3 evaluation runs).
Method	IN	1%	10%	Food-101	Flowers-102	DTD
W-MSE (28)	69.4	-	-	-	-	-
NNCLR (26)	69.4	-	-	-	-	-
SwAV (13)	64.6	-	-	-	-	-
SimSiam (15)	68.1	-	-	-	-	-
CorInfoMax (59)	69.1	44.9	64.3	-	-	-
VICReg (7)	68.7	44.8	62.2	-	-	-
BarlowTwins (77)	68.7	45.1 ± .12	61.7 ± .03	69.8 ± .03	86.2 ± .22	67.7 ± .20
SimCLR (14)	66.5	42.6 ± .02	61.6 ± .09	67.2 ± .24	84.0 ± .19	64.8 ± .07
BYOL (38)	69.3	49.8 ± .05	65.0 ± .05	70.6 ± .1	84.8 ± .43	67.6 ± .14
MoCo-V2 (16)	67.4	43.4 ± .07	63.2 ± .07	68.6 ± .03	82.4 ± .27	66.6 ± .16
SwAV (13)	72.1	49.8 ± .09	66.9 ± .05	72.1 ± .08	89.3 ± .1	68.2 ± .18
MMCR (2 views)	69.5 ± .02	46.6 ± .02	63.9 ± .02	72.0 ± .02	90.0 ± .24	68.5 ± .07
MMCR (4 views)	71.5 ± .04	49.4 ± .05	66.0 ± .05	73.2 ± .07	91.0 ± .04	70.4 ± .46
MMCR (8 views)	72.1 ± .04	51.0 ± .02	67.7 ± .11	73.6 ± .04	91.4 ± .07	70.0 ± .24
performance, see Appendix O for this ablation. For smaller CIFAR-10 we used a smaller batch size, many more views (40), and the Adam optimizer with fixed learning rate. See Appendix D for exact details.
3.1 Transfer to Downstream Tasks