SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The subspace is given by linear equations  $\mathcal{A}(\mathbf{X}) = \mathbf{b}$  and linear operator  $\mathcal{A}:\mathbb{R}^{n\times n}\to \mathbb{R}^p$  can be a random sampling operator. The dual for the nuclear norm  $|\cdot |_{*}$  is the operator norm  $|\cdot |$ . The problem in (9) has been initially introduced as a heuristic method for seeking minimum rank solutions to linear matrix equations, but has later been shown to have theoretical guarantees under certain conditions on the linear operator  $\mathcal{A}$  [28, 27, 32, 17]. The Lagrangian dual of (9) is given by (10), where operator  $\mathcal{A}^{*}:\mathbb{R}^{p}\rightarrow \mathbb{R}^{n\times n}$  is the adjoint of  $\mathcal{A}$ .
We write down an instance of (9) in (11) below to better reflect the specifics of our setting. Since the true underlying heat kernel matrix  $\mathbf{H}_t$  is observed only partially with known entries indicated by a collection of index pairs  $\Omega$  induced by augmentation process, we can form a sampling symmetric matrix  $\mathbf{W}$ :  $\mathbf{W}_{ij} = 1$  if  $(i,j)\in \Omega$ , and 0 otherwise, indicating observed entries. Now, the incomplete kernel matrix can be written down explicitly as  $\widehat{\mathbf{H}} = \mathbf{W}\odot \mathbf{H}$ , where  $\odot$  denotes Hadamard (element-wise) product. The constraint in (9) instantiates as  $\mathcal{A}(\mathbf{X}) = \mathrm{vec}(\mathbf{W}\odot \mathbf{X})$ .
$$
\min  _ {\mathbf {X} \in \mathcal {S} _ {+}} \| \mathbf {X} \| _ {*}
$$
$$
\max _ {\mathbf {Z} \in \mathbb {R} ^ {n \times d}} \operatorname {T r} \mathbf {Z} ^ {\top} \widehat {\mathbf {H}} \mathbf {Z}
$$
subject to  $\operatorname{vec}(\mathbf{W} \odot \mathbf{X}) = \operatorname{vec}(\hat{\mathbf{H}})$
subject to  $\mathbf{Z}^{\top}\mathbf{Z} = \mathbf{I}$
We proceed by showing that maximisation of the trace formulation in (12) embraces reconstruction of the incomplete underlying kernel matrix with entries known to us only from the augmentation process and specified by the matrix  $\mathbf{W}$ .
Proposition 4.1. The trace maximization problem given in (12) is a Lagrangian dual of low-rank matrix completion problem in (11).
Proof. First, we show that (12) is an instance of (10). Let linear operator  $\mathcal{A}(\mathbf{X}) : \mathbb{R}^{n \times n} \to \mathbb{R}^{n^2}$  be a sampling vectorization of  $\operatorname{vec}(\mathbf{W} \odot \mathbf{X})$ . By trace of a matrix product, we can rewrite the objective as  $\mathrm{Tr}[\mathbf{Z}^\top \widehat{\mathbf{H}} \mathbf{Z}] = \mathrm{Tr}[\widehat{\mathbf{H}} \mathbf{Z} \mathbf{Z}^\top] = (\operatorname{vec} \widehat{\mathbf{H}})^\top \operatorname{vec}(\mathbf{Z} \mathbf{Z}^\top) = \mathbf{b}^\top \mathbf{q}$ .
The adjoint operator  $\mathcal{A}^{*}(\mathbf{q}) = \mathrm{mat}(\mathbf{D}_{\mathbf{W}}\mathbf{q})$  acts as sampling matricization, i.e. it samples and maps  $\mathbf{q}$  back to  $\mathbb{R}^{n\times n}$ :  $\mathcal{A}^{*}(\mathbf{q}) = \mathbf{W}\odot \mathbf{Z}\mathbf{Z}^{\top}$ , and the constraint of the dual (10) becomes  $\| \mathbf{W}\odot \mathbf{Z}\mathbf{Z}^{\top}\| \leq 1$ . As  $\mathbf{Z}$  admits singular value decomposition  $\mathbf{Z} = \mathbf{U}\Sigma \mathbf{V}^{\top}$ , the constraint in (12)  $\mathbf{Z}^{\top}\mathbf{Z} = \mathbf{I}$  implies  $\sigma_{i}(\mathbf{Z}) = 1$  for all  $i$ .
To establish equivalence of the constraints in (10) and (12), i.e.  $\| \mathbf{W} \odot \mathbf{Z}\mathbf{Z}^{\top} \| = \sigma_{1}(\mathbf{W} \odot \mathbf{Z}\mathbf{Z}^{\top}) \leq 1$  given  $\mathbf{W}$  and  $\sigma_{1}(\mathbf{Z}) = 1$ , we can use a result for singular values of Hadamard product [, specifically, for  $k = 1, 2, \ldots, n$ :  $\sum_{i=1}^{k} \sigma_{i}(\mathbf{A} \odot \mathbf{B}) \leq \sum_{i=1}^{k} \min(c_{i}(\mathbf{A}), r_{i}(\mathbf{A})) \sigma_{i}(\mathbf{B})$ , where  $c_{i}(\mathbf{A})$  and  $r_{i}(\mathbf{A})$  are column and row lengths of  $\mathbf{A}$ , and  $\sigma_{1} \geq \sigma_{2} \geq \dots \geq \sigma_{n}$ . Let  $\mathbf{A} = \mathbf{W}$  and  $\mathbf{B} = \mathbf{Z}\mathbf{Z}^{\top}$ , then  $r_{i} = c_{i} = \sqrt{K}$ , yielding  $\sigma_{1}(\mathbf{W} \odot \mathbf{Z}\mathbf{Z}^{\top}) \leq \sqrt{K}$ , where  $K = n_{\text{epochs}} \times a$  is a constant, consequently, it can be accounted for by rescaling  $\mathbf{W}$  and not affecting the maximization. Finally, since (11) is an instance of (9), we can conclude that (12) is dual to (11).
This result allows us to make use of theoretical guarantees obtained in matrix completion in what follows.
4.2 Convergence
Incoherence Standard incoherence [7, 23, 20, 17, 27] is the key notion in the analysis of matrix completion problem. Intuitively, incoherence characterises the ability to extract information from a small random subsample of columns in the matrix. More formally, it is defined as an extent to which the singular vectors are aligned with the standard basis.
Definition 4.2. Given matrix  $\mathbf{M} \in \mathbb{R}^{n_1 \times n_2}$  with rank  $r$  and SVD  $\mathbf{M} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^{\top}$ ,  $\mathbf{M}$  is said to satisfy the standard incoherence condition with coherence parameter  $\mu$  if
$$
\max  _ {1 \leq i \leq n _ {1}} \| \mathbf {U} ^ {\top} \mathbf {e} _ {i} \| _ {2} \leq \sqrt {\frac {\mu r}{n _ {1}}} ， \quad \max  _ {1 \leq j \leq n _ {2}} \| \mathbf {V} ^ {\top} \mathbf {e} _ {j} \| _ {2} \leq \sqrt {\frac {\mu r}{n _ {2}}} ， \tag {13}
$$
where  $\mathbf{e}_i$  is the  $i$ -th standard basis vector of a respective dimension.
Since the matrix we aim to recover is symmetric,  $n_1 = n_2 = n$  and its left, right singular and eigenvectors are identical. The coherence parameter  $\mu = \frac{n}{r} \max_{i} \| \mathbf{U} \mathbf{e}_i \|_2^2$  range from 1 (incoherent) to  $\frac{n}{r}$  (coherent).
To the best of our knowledge, optimal sample complexity bounds for matrix recovery via nuclear norm minimisation were obtained in [10]. Specifically, given  $\mathbf{M}$  satisfies standard incoherence condition (13) with parameter  $\mu$ , if uniform sampling probability  $p^* \geq c_0 \mu r \log^2(n)/n$  for some  $c_0 > 0$ , then  $\mathbf{M}$  is the unique solution to (1) with high probability.
Matrix recovery is easy when it is highly incoherent — when information is spread more uniformly among its columns/rows, loosing a random subset of its entries is not as big of a deal as when information is concentrated in certain important columns/rows. On the other hand, high incoherence intuitively makes matrices harder when used as feature matrices in downstream tasks. This might explain why typical SSL methods rely on the output of backbone network (representations) rather than the output of projection head (embeddings).
It is easy to see that downstream performance depends on the alignment of the target matrix  $\mathbf{Y}$  with the left eigenvectors of the feature matrix  $\mathbf{X}$ . The target matrix  $\mathbf{Y}$  in downstream classification task is typically a very simple binary matrix, and can be shown to have low incoherence. However, whenever  $\mathbf{X}$  is obtained as the projection head output of a network learned via self-supervised learning method with a spectral embedding type objective, the incoherence of  $\mathbf{X}$  is inherently tied to the incoherence of the kernel matrix. The latter needs to have high incoherence to be recoverable. Consequently, we put forward the following proposition and find its empirical support in Section 5.2.
Proposition 4.3. Projection head outputs (embeddings) yield lower performance on the downstream task due to their high incoherence. The complexity of the projection head correlates with coherence of the backbone output (representations).
Only a fraction of total entries in  $\mathbf{H}_t$  is required for matrix recovery with high probability. We can further narrow down the bound on  $p^*$  if we consider structured matrix completion problem in the form of some side information which has a direct connection to self-supervised setup. Let the column/row space of  $\mathbf{M}$  lie in some known  $r'$ -dimensional subspace of  $\mathbb{R}^n$  spanned by the columns of  $\bar{\mathbf{U}}$ ,  $n > r' \geq r$ . Then the nuclear norm minimization problem transforms into:
$$
\min  _ {\mathbf {X}} \| \mathbf {X} \| _ {*} \quad \text {s u b j e c t t o} \quad (\bar {\mathbf {U}} \mathbf {X} \bar {\mathbf {U}} ^ {\top}) _ {i j} = \mathbf {M} _ {i j}, \quad (i, j) \in \Omega . \tag {14}
$$
In practice, we use neural network parameterisation to learn the heat kernel map, this choice inadvertently restricts the reconstructed kernel to be aligned with the column space of the network outputs, bringing the inductive bias of the architecture into picture.
The optimal sampling complexity bound for (14) extends as  $p^* \gtrsim \mu \bar{\mu} r\bar{r}\log (\bar{\mu}\bar{r})\log n / n^2$ , where  $\bar{\mu}$  and  $\bar{r}$  are the coherence and the rank of  $\bar{\mathbf{U}}$ , respectively. Suppose we wanted to recover some binary adjacency matrix  $\mathbf{A}$ , such that  $\mathbf{A}_{ij} = 1$  if  $i,j$  belong to the same cluster, 0 otherwise. Because  $\mathbf{A}$  can be rearranged to be block-diagonal with  $r$  blocks, its coherence  $\mu (\mathbf{A}) = n / rn_{min}$ , and exact reconstruction is possible provided
$$
p ^ {*} \gtrsim \bar {\mu} \bar {r} \log (\bar {\mu} \bar {r}) \log n / n n _ {\min }, \tag {15}
$$
where  $n_{min}$  is the minimal cluster size. Heat kernel matrix constructed from such  $\mathbf{A}$  will have its eigenspectrum closely resembling that of  $\mathbf{A}$ , albeit smooth, yet still having same pattern in eigengaps. So we may safely adopt this bound for  $\mathbf{H}_t$ . For balanced class datasets  $n_{min} = n / c$ , and we can immediately see that the number of required observations  $m = p^{*}n^{2} = c\bar{\mu}\bar{r}\log (\bar{\mu}\bar{r})\log n$  grows linearly with the number of classes  $c$  in the dataset.
For illustrative purposes we plot the theoretical bound  $p^*$  on the fraction of the observed entries for a successful matrix completion from (15) in Figure 1 along with the actual fraction  $p$  of observed
Table 1: Performance comparison for the trace maximization formulation (RQMIN) and VICREG. Mean and standard deviation for validation set accuracy across 5-10 runs for CIFAR-10, CIFAR-100 and ImageNet-100.
CIFAR-10	CIFAR-100	ImageNet-100
top-1	top-5	top-1	top-5	top-1	top-5
VICREG	91.15 ± 0.16	99.64 ± 0.05	67.57 ± 0.20	89.90 ± 0.13	78.89 ± 0.38	93.94 ± 0.17
RQMIN	91.19 ± 0.13	99.67 ± 0.04	68.12 ± 0.26	90.12 ± 0.11	78.98 ± 0.33	94.45 ± 0.23
entries under self-supervised learning augmentation protocol to demonstrate that the latter intercepts the former given enough training epochs. To be specific, we set the size of the training dataset  $N = 50\mathrm{k}$  (CIFAR-10 size), the cluster size (number of same class examples)  $n_{min} = 5000$  ( $c = 10$ ), the number of views  $a = 2$ , the number of epochs  $n_{epochs}$  range from 1 to 1000, and  $r = 512$  (embedding size), assume  $\mu = 20$  (which seems to be a fair estimate in light of the experiments in Section 5) and  $c_0 = 5$ , a constant used to control the probability of exact matrix recovery.
Based on this bound, we highlight the following factors that play important role in the success of any SSL method with spectral embedding type objective. The choice of the similarity function affects the incoherence parameter in the bound. The number of samples per class (alternatively the minimal cluster size  $n_{min}$ ) should also be high enough for  $p^*$  to decrease rapidly. Finally, though potentially in contrast to the empirical observations (higher  $d$  on ImageNet yields higher downstream performance), the rank  $r$ , effectively the dimension of embedding  $d$ , should not be too large.
First, we verify that the performance of the proposed formulation in (4) and the corresponding loss function, denoted RQMIN, is at least on par with the state-of-the-art methods. We then study the effect of the complexity of the projection head on the incoherence and its connection with the downstream performance of the backbone against projection head outputs.
Here we report the training hyperparameters for all of the experiments. As VICReg is extremely sensitive to the choice of hyperparameters (e.g. increasing learning rate with increased batch size negatively affects training - learning diverges), we adopt the same hyperparameters for training RQMIN for a fair comparison. We follow the standard VICReg protocol adopted and finetuned for CIFAR-10/100 and ImageNet-100 in the library for self-supervised learning methods for visual representation learning solo-learn [12].
We train ResNet-18 backbone architecture with 3-layer MLP projection head (respective hidden dimensions: 2048-2048-2048). The batch size is 256 for CIFAR datasets and 512 for ImageNet-100. For pretraining, the learning rate schedule is linear warm-up for 10 epochs and cosine annealing, the optimizer is LARS with learning rate 0.3. For linear probe training, SGD with step learning rate schedule with steps at 60 and 80 epochs. The number of pre-training epochs is 1000 for CIFAR and 400 for ImageNet-100, downstream training - 100 epochs.
5.1 Comparable performance
To demonstrate that our trace maximization objective is on par with existing SSL methods, we test our objective in (4) on a standard ResNet-18 backbone neural network with 3-layer MLP projection head and obtain comparable results on CIFAR-10, CIFAR-100, and ImageNet-100 to state-of-art methods. As most of the latter yield almost identical performance given enough hyperparameter tuning, we pick VICREG as a representative to compare against. Following the standard protocol with 1000 and 400 pre-training epochs for both CIFAR datasets and for ImageNet-100, respectively, linear probe for downstream evaluation is trained for 100 epochs. We do not tune hyperparameters and use default values. Mean and standard deviation for downstream accuracy across 5-10 trials with different seed values are reported in Table 1.
Figure 2: (left) Coherence of the backbone outputs (representations) grows with the increasing number of layers in the projection head. (right) Downstream accuracy versus incoherence. Green marks show downstream performance and incoherence of embeddings, blue marks - representations. Embeddings and representations of the same type of model are almost always separable in these coordinates: embeddings exhibit lower performance and lower coherence (small  $\mu$ ) while representations perform better and have higher coherence (large  $\mu$ ).
5.2 Incoherence effect on downstream performance
To test whether incoherence could explain the performance disparity between the outputs of the backbone, termed representations, and the projection head, known as embeddings, we train several models with various configurations of projection heads against SSL objectives on the ImageNet-100 dataset and calculate incoherences of representations and embeddings along with respective downstream accuracies. The performance of the representations and embeddings of the pre-trained model is evaluated in a downstream classification task, while incoherence  $\mu$  for both candidates is estimated on the training set.
Since the dimensionality of representations and embeddings differ, i.e. 512 for backbone output and 2048 for projection head output, we need to pick the rank to compute  $\mu$  accordingly. The common way to pick rank in the numerical methods is through tolerance, i.e. thresholding based on the values of the normalized singular values, i.e.  $\sigma_i' = \sigma_i / \sum_j \sigma_j$ . Otherwise, one could use the notion of effective rank [30], given by  $r_e(\mathbf{A}) = \exp(H(\pmb{\sigma}))$ , where the entropy  $H(\pmb{\sigma}) = -\sum_i \pmb{\sigma}_i \log(\pmb{\sigma}_i)$  with vector of singular values  $\pmb{\sigma}$ . Overall, we compute coherence as the following expression:
$$
\mu (\mathbf{A}) = \frac{n}{r_{e}(\mathbf{A})}\max_{1\leq i\leq \lceil r_{e}\rceil}\| \mathbf{U}_{\mathbf{A}}^{\top}\mathbf{e}_{i}\|_{2}^{2}.
$$
Incoherence and projection head complexity. To estimate coherence of representations in Figure 2 (left), we embed the training set of ImageNet-100 to get representations matrix  $\mathbf{A} \in \mathbb{R}^{125952 \times 512}$  and compute incoherence  $\mu(\mathbf{A})$  using effective rank  $r_e(\mathbf{A})$ . We embed the training set using three distinct pre-trained models for each of the projection head configurations characterized by the number of layers  $l, l \in \{1, 2, 3, 4\}$ , and average the values across ten embedding runs. The resulting mean and standard deviation plot suggests that incoherence (low  $\mu$ ) is higher for more shallow projection heads and decreases ( $\mu$  grows) as the number of layers in the head increase, a result we anticipated.
Embeddings and representations disparity. Figure 2 (right) plots distinct models representations and embeddings in the Accuracy-Coherence plane. It encodes different type of loss with a shape: diamonds for VICREG and squares for RQMIN. While blue colour signifies the position of each model's representations, the green colour reflects the corresponding embeddings. Both objectives demonstrate a separation of the embedding (low coherence, low accuracy) and representation (high coherence, high accuracy) points, which is more distinctive in the case of RQMIN (squares).
The resulting plots support our hypothesis in Proposition 4.3 that incoherence indeed plays a crucial role in explaining the use of the backbone outputs. For successful matrix completion, high incoherence of the partially observed affinity matrix is essential. However, for the downstream performance of the representations, the opposite is preferred. The projection head functions as a disentangling buffer, enabling the representations to maintain low incoherence. Conversely, the embeddings inherit the incoherence of the affinity matrix.
While incoherence seems to be an attractive candidate for unsupervised embedding evaluation metric in scenarios with little to no test data, one should use caution as the coherence value  $\mu$  does not reflect
the amount of relevant information stored in the matrix of embeddings due to normalization with reference to its rank. This idea has been explored in [35] on embeddings of supervised models.
Recent success of self-supervised methods [9, 15, 2, 38, 39], especially in the domain of computer vision received great amount of attention since the learned representations ended up respecting the semantics of the data. There has been a great interest in trying to understand the inner workings of the seemingly heuristic objectives since.
While there are many lenses one may take up to study the problem [21, 37, 22, 34, 33, 18], a particularly related to this work concurrent body of literature has adopted the view of the kernel or laplacian-based spectral representation learning [13, 1], which we also share in this work. We highlight our main difference to the results provided in these very recent papers. [1] does a brilliant job connecting and characterizing modern SSL methods into classical existing counterparts. However, it does not answer an important question whether an incomplete a priori knowledge about the data manifold stemming from augmentations can provide a good approximation to essentially nonlinear dimensionality reduction methods such as LLE [29], MDS [24], and kernel PCA [31].
We not only show SSL methods to have an objective function similar to the objectives of classical spectral decomposition methods, e.g. LaplacianEigenmaps [3], but also try to address the problem of incomplete and noisy measurements that we get as an inductive bias during the augmentation process. We hope that this perspective via the low-matrix completion problem will yield further theoretical results on the success of self-supervised learning and practical benefits when applying these methods in the wild, e.g. in domains such as medical imaging [16] and hard sciences [25].
In this work, we make an attempt to bridge modern self-supervised methods with classical Laplacian-based dimensionality reduction methods and low-rank matrix completion in hopes to provide theoretical insights on the recent successes of SSL methods.
We show that these methods are not only doing Laplacian-based nonlinear reduction but are able to approximate and recover the truncated version of the underlying Laplace operator given only noisy and incomplete information from augmentation protocol by adopting low-rank matrix completion extensive literature and results. However, when working with datasets with potentially large number of classes, it might be a good idea to consider whether the size of the sample is large enough so that the minimal cluster size allows the full data matrix to be considered low-rank, otherwise the SSL methods would possibly fail to converge.
We also spot a direct influence of the inductive bias in the parameterization of the learned map on the column space of the recovered matrix. We also hypothesize that the disparity in downstream performance between backbone and projection head outputs can be explained by the high incoherence if the latter which is tied to the incoherence of the kernel one is recovering during training. The kernel should have high incoherence to be recoverable.
One of the possible avenues for future work stems from the notion of incoherence. We see it in exploring incoherence property of different types of the similarity or weighting functions one may use to instantiate the adjacency matrix with.
We hope that this work paves the way for a deeper study of the connection between self-supervised methods and classical problems such as matrix completion to yield better practical and theoretical understanding of various applications of SSL in different domains, not only computer vision.
[1] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205.11508, 2022. 1, 10