SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marina Munkhoeva†
Ivan Oseledets
Self-supervised methods received tremendous attention thanks to their seemingly heuristic approach to learning representations that respect the semantics of the data without any apparent supervision in the form of labels. A growing body of literature is already being published in an attempt to build a coherent and theoretically grounded understanding of the workings of a zoo of losses used in modern self-supervised representation learning methods. In this paper, we attempt to provide an understanding from the perspective of a Laplace operator and connect the inductive bias stemming from the augmentation process to a low-rank matrix completion problem. To this end, we leverage the results from low-rank matrix completion to provide theoretical analysis on the convergence of modern SSL methods and a key property that affects their downstream performance.
Self-supervised methods have garnered significant interest due to their heuristic approach to learning representations that capture the semantic information of data without requiring explicit supervision in the form of labels. Contrastive learning based methods among the former would use the repulsion among arbitrary pair of points in the batch, while non-contrastive would rely on the consistency among different views of the same image. While self-supervised representation learning becomes more ubiquitous in the wild, especially in the important domain such as medical imaging [16], the theoretical grounding of these methods would potentially help avoid the pitfalls in applications. Unsurprisingly, researchers are already trying to build theoretically sound understanding of modern self-supervised representation learning methods.
The overall goal of this work is to understand self-supervised representation learning through the lens of nonlinear dimensionality reduction methods (e.g. Laplacian Eigenmaps [3]) and low-rank matrix completion problem [28]. To this end, we take on a Laplace operator perspective on learning the optimal representations in the manifold assumption. We then derive a trace maximization formulation to learn eigenfunctions of the Laplace operator of the underlying data manifold. We adopt the heat kernel based embedding map that in theory under certain conditions is an almost isometric embedding of the low-dimensional manifold into the Euclidean space. As a result, we discuss how existing several SSL methods (e.g. SIMCLR [9], BARLOWTWINS [1], VICREG [2]) can be comprehended under this view.
It is important to note that our current understanding of the topic lacks one crucial aspect. Traditional spectral methods commonly operate with complete kernel matrices, whereas our approach deals with incomplete and potentially noisy ones. The only available similarity information among examples is derived from the data augmentation process, which generates a positive pair. Meanwhile the
†Correspondence to marina.munkhoeva@tuebingen.mpg.de
$^{\ddagger}$ Max Planck Institute for Intelligent Systems, Tübingen, Germany
$^{\S}$  Artificial Intelligence Research Institute (AIRI), Skolkovo Institute of Science and Technology (Skoltech), Moscow, Russian Federation
remaining examples in the batch are either seen as negative ones (contrastive) or are not considered at all (non-contrastive).
A pertinent and often overlooked question emerges: how can self-supervised learning methods effectively leverage such limited signals to converge towards meaningful representations? In response, we shed light on this matter by establishing a connection between SSL and a matrix completion problem. We demonstrate that these optimization problems are Lagrangian dual of each other, implying that optimizing the SSL objective simultaneously entails reconstructing the kernel matrix. We can summarize the contributions of this paper as follows:
- We propose an eigen-problem objective for spectral embeddings from graphs induced by augmentations and use it to interpret modern SSL methods.
- We show that SSL methods do Laplacian-based nonlinear dimensionality reduction and low-rank matrix completion simultaneously. We leverage theory behind matrix completion problem to provide insights on the success of SSL methods and their use in practice.
- While the number of observed entries required by theory decreases with epochs, we find that the actual number is a constant and the former eventually intersects the latter.
- We find a possible explanation for disparity in downstream performance of the backbone and projection outputs.
Figure 1: Illustrative graph of the theoretical fraction  $p^*$  of the observed entries required for matrix completion to succeed with high probability. As training proceeds, unmaterialized kernel matrix size  $n$  increases and  $p^*$ , which is roughly  $\sim \log n / n$ , decreases. Eventually, the actual (constant) fraction  $p$  of observed entries under self-supervised learning augmentation protocol intersects the theoretical bound.
This work relies on the manifold hypothesis, a hypothesis that many naturally occurring high-dimensional data lie along a low-dimensional latent manifold inside the high-dimensional space. Since we can only observe a sample from the manifold in the ambient space, neither the true manifold nor its metric are available to us. Although we never explicitly work with Laplace-Beltrami operator, we still give a brief definition below to provide some grounding for the reasoning later.
Laplace operator Let  $\mathcal{M}$  be a Riemannian manifold and  $g$  be the Riemannian metric on  $\mathcal{M}$ . For any smooth function  $u$  on  $\mathcal{M}$ , the gradient  $\nabla u$  is a vector field on  $\mathcal{M}$ . Let  $\nu$  be the Riemannian volume on  $\mathcal{M}$ ,  $d\nu = \sqrt{\operatorname*{det}g} dx^{1}\ldots dx^{D}$ . By the divergence theorem, for any smooth functions  $u$  and  $v$  (plus smooth and compact support assumptions)  $\int_{\mathcal{M}}u\mathrm{div}\nabla v  d\nu = -\int_{\mathcal{M}}\langle \nabla u,\nabla v\rangle d\nu$ , where  $\langle \cdot ,\cdot \rangle = g(\cdot ,\cdot)$ . The operator  $\Delta = \mathrm{div}\nabla$  is called the Laplace-Beltrami operator of the Riemannian manifold  $\mathcal{M}$ .
In practice, we usually work with finite samples, and Laplace operator is typically approximated with graphs or meshes. While the latter are typically used in computational mathematics, the former find a widespread use in machine learning. Below is a brief overview of the relevant graph notions, for a detailed exposition please see [36].
Graph Laplacian and Spectral Embedding Given a graph  $\Gamma = (V,E)$  with  $|V| = n$  vertices and a set of edges  $e_{ij} = (v_i,v_j)$  that form a weighted adjacency matrix  $\mathbf{A}_{ij} = w_{ij}$  with non-negative weight  $w_{ij}\geq 0$ , whenever there is  $e_{ij}\in E$ , otherwise 0. With a degree matrix  $\mathbf{D} = \mathrm{diag}(\mathbf{A1})$ , the graph Laplacian is given by  $\mathbf{L} = \mathbf{D} - \mathbf{A}$ , the corresponding random walk Laplacian is a normalization  $\mathbf{L}_{rw} = \mathbf{I} - \mathbf{D}^{-1}\mathbf{A}$ . All graph Laplacians admit an eigenvalue decomposition, i.e.  $\mathbf{L} = \mathbf{U}\Lambda \mathbf{U}^{\top}$  where  $\mathbf{U}\in \mathbb{R}^{n\times n}$  contains eigenvectors in columns and a diagonal matrix  $\Lambda \in \mathbb{R}^{n\times n}$  has eigenvalues on the diagonal. Note that there is a trivial eigenpair  $(0,\mathbf{1})$ . [5, 4] show that the eigenvectors of the graph Laplacian of a point-cloud dataset converge to the eigenfunctions of the Laplace-Beltrami operator under uniform sampling assumption.
Whenever one has an affinity matrix, a positive semidefinite pairwise similarity relations among points, the classical way to obtain embeddings for the points is to perform spectral embedding. A typical algorithm would include constructing a graph based on the affinity matrix, computing  $k$  first eigenvectors of its Laplacian and setting the embeddings to be the rows of the matrix  $\mathbf{U} \in \mathbb{R}^{n \times k}$  that contains the eigenvectors as columns.
Matrix completion Let  $\mathbf{M} \in \mathbb{R}^{n \times n}$  is partially observed matrix with rank  $r$ . Under Bernoulli model, each entry  $\mathbf{M}_{ij}$  is observed independently of all others with probability  $p$ . Let  $\Omega$  be the set of observed indices. The matrix completion problem aims to recover  $\mathbf{M}$  from  $m = |\Omega|$  observations. The standard way to solve this problem is via nuclear norm minimization:
$$
\min  _ {\mathbf {X} \in \mathbb {R} ^ {n \times n}} \| \mathbf {X} \| _ {*} \quad \text {s u b j e c t t o} \quad \mathbf {X} _ {i j} = \mathbf {M} _ {i j} \text {f o r} (i, j) \in \Omega , \tag {1}
$$
where  $\| \mathbf{X} \|_*$  is the nuclear norm of  $\mathbf{X}$ , i.e. the sum of its singular values. A large body of work [8, 28, 27, 10] has succeeded in providing and enhancing of the conditions that guarantee the optimal solution  $\mathbf{X}^*$  to be both unique and equal to  $\mathbf{M}$  with high probability.
Notation Any matrix  $\mathbf{X} \in \mathbb{R}^{n_1 \times n_2}$  has a singular value decomposition (SVD)  $\mathbf{X} = \mathbf{U} \Sigma \mathbf{V}^{\top}$ , where the columns of the matrix  $\mathbf{U} \in \mathbb{R}^{n_1 \times n_1}$  are left singular vectors, the singular values  $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_{\min(n_1, n_2)}$  lie on the diagonal of a diagonal matrix  $\Sigma \in \mathbb{R}^{n_1 \times n_2}$ , and right singular vectors are the columns of  $\mathbf{V} \in \mathbb{R}^{n_2 \times n_2}$ .  $||\mathbf{X}||_F$ ,  $||\mathbf{X}|| = \sigma_1$ ,  $||\mathbf{X}|_* = \sum_i \sigma_i$  denote Frobenius, spectral and nuclear norms of  $\mathbf{X}$ , respectively.  $||\mathbf{x}||_p$  denotes  $p$ -th vector norm.
3 SSL and Spectral Embedding
In this section, we will provide a construction that covers most of the self-supervised learning methods and gives SSL a novel interpretation. First, we formalize the setup from the perspective of the manifold hypothesis. Then, we make some modelling choices to yield an SSL formulation as a trace maximization problem, a form of eigenvalue problem. Finally, we describe how three well known representatives SSL methods fall under this formulation.
Let  $\mathbf{X} \in \mathbb{R}^{n \times d'}$  be a set of  $n$  points in  $\mathbb{R}^{d'}$  sampled from a low-dimensional data manifold observed in a  $d'$ -dimensional ambient Euclidean space. However, data is rarely given in the space where each dimension is meaningful, in other words  $d' \gg d^*$ , where  $d^*$  is unknown true dimensionality of the manifold. The goal of nonlinear dimensionality reduction is to find a useful embedding map into  $d$ -dimensional Euclidean space with  $d \ll d'$ . Two of the classical approaches, namely Eigenmaps [3] and Diffusion maps [11], use eigenfunctions of the graph Laplacian, an approximation of the Laplace operator associated with the data manifold. Both can be loosely described as a variant of map
$$
\Phi (\mathbf {x}) = \left[ \begin{array}{l l l l} \phi_ {1} (\mathbf {x}) & \phi_ {2} (\mathbf {x}) & \dots & \phi_ {d} (\mathbf {x}) \end{array} \right], \tag {2}
$$
where  $\phi_{k}$  is  $k$ -th eigenfunction of the negative Laplace operator on the underlying manifold.
This type of map bears a direct connection to theoretical question how to find an embedding of certain types of manifolds into Euclidean spaces, studied in differential geometry. For instance, [6] construct embedding for a given smooth manifold by using its heat kernel. This approach has been subsequently enhanced through the application of a truncated heat kernel expansion. Furthering this trajectory, [26] explore whether and when such embedding is close to being isometric, which is desirable as isometric embedding preserves distances.
Motivated by the heat kernel embedding literature, we provide a general construction for self-supervised methods in what follows. The heat kernel on a manifold can be represented as an expansion  $H(p,q,t) = \sum_{k=0}^{\infty} e^{-\lambda_k t} \phi_k(p) \phi_k(q)$ , where  $p$  and  $q$  are some points on the manifold,  $t > 0$  and  $\phi_k$  are normalized eigenfunctions of the Laplace operator, i.e.  $-\Delta \phi_k = \lambda_k \phi_k$  and  $|\phi_k|_2 = 1$ . However, working with finite data, we can only operate with a graph approximation of the manifold, and will make use of the heat kernel construction for graphs. The latter is given by a matrix exponential  $\mathbf{H}_t = e^{-t\mathbf{L}}$ , alternatively represented through an expansion  $\mathbf{H}_t = \sum_{i=0}^{n} e^{-\lambda_k t} \phi_k \phi_k^T$ , where  $\mathbf{L}$  is a graph Laplacian and  $(\lambda_k, \phi_k)$  is  $k$ -th eigenpair of  $\mathbf{L}$ .
Consider Laplacian EigenMaps method, it first constructs a graph and weighs the edges with a heat kernel of the Euclidean space which takes the form of a Gaussian kernel
$h_{E}(\mathbf{x},\mathbf{y},t) = \exp (-||\mathbf{x} - \mathbf{y}||_{2}^{2} / t)$  whenever the distance is less than some hyperparameter, i.e.  $||\mathbf{x} - \mathbf{y}||_2 < \epsilon$ , or  $\mathbf{y}$  is among  $k$  nearest neighbours of  $\mathbf{x}$ . Next, the method proceeds with finding the eigenvectors of the graph Laplacian constructed from the thus weighted adjacency matrix  $\mathbf{W}$ .
Contrarily, the proposed construction acknowledges the cardinality and complexity typically associated with the modern day datasets, where it is challenging to select the cutoff distance or the number of neighbours, and impractical to compute all pairwise distances. Our ideal graph has edges reflecting the semantic similarity between points; e.g. there is an edge whenever the pair of points belong to the same class. The drawback is that this matrix is unknown. This is where augmentation comes into play as it provides a peek into only a fraction of the entries in the unmaterialized heat kernel matrix. Consequently, we claim that SSL implicitly solves a low-rank matrix completion problem by instantiating some of the pairwise similarity entries in  $\mathbf{H}_t$  via the augmentation process. However, prior to this, we need to demonstrate the generality of the perspective we have formulated. To accomplish this, we articulate a trace maximization problem.
3.1 Trace Minimization
To start, we construct an unmaterialized heat kernel matrix in accordance with the standard self-supervised learning protocol for augmentations. Given a dataset with  $N$  points, one training epoch generates  $a$  views of each point. Typically,  $a = 2$  and training runs for  $n_{\text{epochs}}$  number of epochs. The whole training generates exactly  $n_{\text{epochs}} \times a$  views per original instance. As a result the total number of processed examples is  $n = N \times a \times n_{\text{epochs}}$ , thereby we have that  $\mathbf{H}_t \in \mathbb{R}^{n \times n}$ .
As a rule, SSL utilises architectures with a heavy inductive bias, e.g. ResNet [19] / Vision Transformers [14] for image data. Thus, we may safely assume that the same instance views are close in the embedding space and are connected. This results in a block diagonal adjacency matrix  $\mathbf{W}$ , where  $i$ -th block of ones accounts for all the views of  $i$ -th image among  $N$  original images. The fraction of observed entries equal to  $p = N \times (n_{\text{epochs}} \times a)^2 / n^2 = 1 / N$ , a constant fraction for a given dataset.
Note that in the ideal scenario, we would know the cluster/class affiliation for each point and would have connected same-class views with edges. However, in reality, we only know instance affiliation of each view. Let us denote the ideal scenario heat kernel matrix as  $\mathbf{H}_t$  and its partially observed counterpart used in reality —  $\widehat{\mathbf{H}}_t$ .
To obtain the heat kernel matrix, we use the normalized random walk Laplacian  $\mathbf{L}_{rw}$ , for which the heat kernel matrix is as before:  $\mathbf{H}_t = \exp (-t\mathbf{L}_{rw}) = \sum_{k = 0}^n\exp (-\lambda_kt)\mathbf{u}_k\mathbf{u}_k^\top$ , where  $\mathbf{u}_k$  is  $k$ -th eigenvector and  $\lambda_{k}$  a corresponding eigenvalue of  $\mathbf{L}_{rw}$ . To obtain spectral embeddings from an ideal  $\mathbf{H}_t$ , one would then proceed with solving a trace minimization form of eigenvalue problem in (3), where the optimal solution is exactly the eigenvectors of  $\mathbf{L}_{rw}$ , i.e.  $\mathbf{Z}^{*} = \mathbf{U}$ .
$$
\max  _ {\mathbf {Z}} \quad \operatorname {T r} \left(\mathbf {Z} ^ {\top} \mathbf {H} _ {t} \mathbf {Z}\right) \tag {3}
$$
$$
\begin{array}{l} \text {s . t .} \quad \mathbf {Z} ^ {\top} \mathbf {Z} = \mathbf {I}, \end{array}
$$
$$
\max  _ {\theta} \operatorname {T r} \left(\mathbf {Z} _ {\theta} ^ {\top} \widehat {\mathbf {H}} _ {t} \mathbf {Z} _ {\theta}\right) \tag {4}
$$
$$
\begin{array}{l l} \text {s . t .} & \mathbf {Z} _ {\theta} ^ {\top} \mathbf {Z} _ {\theta} = \mathbf {I} _ {d}, \mathbf {Z} _ {\theta} ^ {\top} \mathbf {1} = \mathbf {0}, \end{array}
$$
However, for lack of a better alternative we resort to the incomplete  $\widehat{\mathbf{H}}_t$  and need to learn a parameterized map  $\mathcal{F}_{\theta}(\mathbf{X}) = \mathbf{Z}_{\theta}$  in (4). Apart from the specified eigenfunction pairwise orthogonality constraint, the resulting loss is comprised of implicit constraints on the trivial eigenfunction (identity) and function norm. But before we treat (3) with incomplete  $\mathbf{H}_t$  as matrix completion problem, we show that this formulation can be seen as a generalization for a number of existing self-supervised learning methods. Specifically, several modelling choices differentiate the resultant methods.
3.2 Self-Supervised Methods Learn Eigenfunctions of the Laplace Operator
SIMCLR Consider the following contrastive loss, variants of which are widely adopted in self-supervised learning methods:  $L_{CL} = \sum_{i,j\in +pairs}l_{ij} + l_{ji}$ , where
$$
l _ {i j} = \log \frac {e ^ {\kappa \left\langle \mathbf {z} _ {i} , \mathbf {z} _ {j} \right\rangle}}{\sum_ {k \neq i} e ^ {\kappa \left\langle \mathbf {z} _ {i} , \mathbf {z} _ {k} \right\rangle}} = \kappa \left\langle \mathbf {z} _ {i}, \mathbf {z} _ {j} \right\rangle - \log \sum_ {k \neq i} e ^ {\kappa \left\langle \mathbf {z} _ {i}, \mathbf {z} _ {k} \right\rangle}, \tag {5}
$$
with  $\mathbf{z}_i = f_\theta(\mathbf{x}_i)$  is an embedding of input  $\mathbf{x}_i$  given by function  $f$  parameterized by a neural network with learnable parameters  $\theta$ , which produces unit-norm embeddings, i.e.  $\| \mathbf{z}_i \|_2 = 1$ . One iteration of SIMCLR takes a batch of  $N$  data points and creates  $2N$  views, augmenting each sample to obtain a positive pair  $(\mathbf{z}_i, \mathbf{z}_j)$ , while treating all the other samples and their augmentations as contrastive samples  $\mathbf{z}_k$ .
Let us include  $k = i$  in the sum in denominator of (5) for a moment. A goal of SIMCLR is to obtain optimal representations  $\mathbf{Z} \in \mathbb{R}^{2N \times d}$  such that for each positive pair  $(i,j)$ , their representations will be aligned as much as possible  $\langle \mathbf{z}_i, \mathbf{z}_j \rangle \to 1$ . Let  $\mathbf{A}_{ij} = \exp(\kappa[\mathbf{Z}\mathbf{Z}^\top]_{ij})$  and  $\mathbf{D} = \mathrm{diag}(\mathbf{A}\mathbf{1})$ , then we can rewrite (5) as
$$
l _ {i j} = \log \mathbf {A} _ {i j} - \log \sum_ {k = 1} ^ {2 N} \mathbf {A} _ {i k} = \log \mathbf {A} _ {i j} - \log \mathbf {D} _ {i i} = \log \frac {\mathbf {A} _ {i j}}{\mathbf {D} _ {i i}} = \log [ \mathbf {D} ^ {- 1} \mathbf {A} ] _ {i j} \tag {6}
$$
where we are interested in the right hand side  $\log [\mathbf{D}^{-1}\mathbf{A}]_{ij}$ . Let us note that  $[\mathbf{D}^{-1}\mathbf{A}]_{ij}$  from (6) is a normalized adjacency of a graph  $G = (V,E)$ , where node set  $V$  is the set of views. Since representations live on a unit  $(d - 1)$ -sphere, we have a choice of naturally arising distributions (von Mises-Fisher, spherical normal, etc) to instantiate the weighting function  $\mu (\mathbf{x}_i,\mathbf{x}_j)$ . The model choice of SIMCLR is to instantiate the weighting function with the density of the von Mises-Fisher distribution with  $\kappa >0$  (without the partition function):
$$
\mathbf {A} _ {i j} = \exp \left(\kappa \mathbf {z} _ {i} ^ {\top} \mathbf {z} _ {j}\right), \tag {7}
$$
where  $\mathbf{A}_{ij}$  equals  $e^{\kappa}$ , with a typical value  $\kappa = 2$ , whenever  $i,j$  is a positive pair, and  $e^0 = 1$  otherwise. Note that  $\log (\mathbf{D}^{-1}\mathbf{A}) = \mathbf{D}^{-1}\mathbf{A} - \mathbf{I} + o((\mathbf{D}^{-1}\mathbf{A})^2)\approx -\mathbf{L}_{rw}$ , thus in loose terms the objective in (5) may be seen as a minimization of a trace of a negative of the graph Laplacian by learning  $\mathbf{z}$ 's that shape  $\mathbf{A}$ .
Decoupled Contrastive Learning [38] can be seen as reweighting the adjacency matrix by setting  $\mathbf{A}_{ij} = w(z_i,z_j)$ , where  $w(z_{i},z_{j})$  is a reweighting function to emphasize hard examples.
For non-contrastive methods (BARLOWTWINS, VICREG, etc) SSL methods (where no negative examples are used, e.g. BARLOWTWINS, SIMSIAM), the respective losses could also be interpreted as learning a diffusion map. The key distinction with contrastive methods expresses itself in setting  $A_{ij} = 0$  for view pairs  $(i,j)$  that do not have shared original, thus eliminating any signal from a possibly negative pair.
BARLOWTWINS method normalizes the columns of representation matrices  $\mathbf{Z}_a, \mathbf{Z}_b \in \mathbb{R}^{N \times d}$ , which is the same as the function norm constraint in (4). The overall objective is as follows:
$$
J = \sum_ {i i} \left(\mathbf {Z} _ {a} ^ {\top} \mathbf {Z} _ {b} - \mathbf {I}\right) _ {i i} ^ {2} + \alpha \sum_ {i \neq j} \left(\mathbf {Z} _ {a} ^ {\top} \mathbf {Z} _ {b} - \mathbf {I}\right) _ {i j} ^ {2}, \tag {8}
$$
and simultaneously aims at three things: (i) enforce the closeness of the rows, i.e. positive pair embeddings, (ii) restrict the norm of the columns, i.e. the eigenfunctions, (iii) orthogonalize columns, again the eigenfunctions. The kernel matrix choice in BARLOWTWINS is a simple bi-diagonal adjacency matrix  $\mathbf{A}_{ij} = 1$  as long as  $(i,j)$  is a positive pair, and 0 otherwise.
VICREG objective is a weighted sum of variance, covariance and invariance terms:
$$
J _ {v a r} = \sum_ {k = 1} ^ {d} \max (0, 1 - \sqrt {[ {\bf Z} ^ {\top} {\bf Z} ] _ {k k}}), \quad J _ {c o v} = \sum_ {k \neq l} [ {\bf Z} ^ {\top} {\bf Z} ] _ {k l} ^ {2}, \quad J _ {i n v} = \sum_ {i j} {\bf A} _ {i j} \| {\bf Z} _ {i} - {\bf Z} _ {j} \| ^ {2},
$$
a similar formulation to the previous method, however, the choice for the adjacency matrix here is little bit different. Individual terms of this loss have separate coefficients. The choice of these hyperparameters defines the implicit adjacency matrix entries, controls the maximum function norm allowed and the trade-off between orthogonality and the trace terms in (4).
4 SSL and Low-Rank Matrix Completion
In this section, we use low-rank matrix completion theory to show that the limited information from augmentations might be quite enough under certain conditions. First, we establish the connection between our self-supervised learning objective (4) and the low-rank matrix completion problem. We then draw practical insights from the existing convergence theory for matrix completion problem.
4.1 Low-Rank Matrix Completion Dual
We argue that the objective in (3) with a substitute incomplete kernel matrix  $\widehat{\mathbf{H}}_t$  implicitly contains an objective for the low-rank matrix completion problem. To show this, we first introduce the general form of the nuclear norm minimization problem with affine subspace constraint in (9).
$$
\min _ {\mathbf {X}} \quad \| \mathbf {X} \| _ {*}
$$
$$
\text {s u b j e c t} \quad \mathcal {A} (\mathbf {X}) = \mathbf {b},
$$
$$
\max _ {\mathbf {q}} \mathbf {b} ^ {\top} \mathbf {q}
$$
$$
\text {s u b j e c t} \quad \| \mathcal {A} ^ {*} (\mathbf {q}) \| \leq 1,
$$