SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xing Gao
xgao53@uic.edu
Yu Cheng
yu_cheng@brown.edu
Low-rank matrix recovery is a fundamental problem in machine learning with numerous applications. In practice, the problem can be solved by convex optimization namely nuclear norm minimization, or by non-convex optimization as it is well-known that for low-rank matrix problems like matrix sensing and matrix completion, all local optima of the natural non-convex objectives are also globally optimal under certain ideal assumptions.
In this paper, we relax the assumptions and study new approaches for matrix sensing in a semi-random model where an adversary can add any number of arbitrary sensing matrices. More precisely, the problem is to recover a low-rank matrix  $X^{*}$  from linear measurements  $b_{i} = \langle A_{i},X^{*}\rangle$ , where an unknown subset of the sensing matrices satisfies the Restricted Isometry Property (RIP) and the rest of the  $A_{i}$ 's are chosen adversarially.
It is known that in the semi-random model, existing non-convex objectives can have bad local optima. To fix this, we present a descent-style algorithm that provably recovers the ground-truth matrix  $X^{*}$ . For the closely-related problem of semi-random matrix completion, prior work [CG18] showed that all bad local optima can be eliminated by reweighting the input data. However, the analogous approach for matrix sensing requires reweighting a set of matrices to satisfy RIP, which is a condition that is NP-hard to check. Instead, we build on the framework proposed in  $\mathrm{[KLL^{+}23]}$  for semi-random sparse linear regression, where the algorithm in each iteration reweights the input based on the current solution, and then takes a weighted gradient step that is guaranteed to work well locally. Our analysis crucially exploits the connection between sparsity in vector problems and low-rankness in matrix problems, which may have other applications in obtaining robust algorithms for sparse and low-rank problems.
Low-rank matrix recovery is a popular inverse problem with many applications in machine learning such as collaborative filtering, image compression, and robust principal component analysis (PCA) [RS05, FCRP08, CLMW11]. In this paper, we study one of the most basic low-rank matrix recovery problems namely matrix sensing [CRT06, RFP10]. In the matrix sensing problem, we want to reconstruct a low-rank ground-truth matrix  $X^{*} \in \mathbb{R}^{d_{1} \times d_{2}}$  from a collection of sensing matrices  $\{A_{i}\}_{i=1}^{n}$  and the corresponding linear measurements  $b_{i} = \langle A_{i}, X \rangle$ .
For notational convenience, we define a sensing operator  $\mathcal{A}[\cdot ]:\mathbb{R}^{d_1\times d_2}\to \mathbb{R}^n$  such that  $\mathcal{A}[X] = b$  with  $b_{i} = \langle A_{i},X\rangle$  for  $i = 1\dots n$ . The goal is to solve the following rank-constrained optimization problem:
$$
\min  _ {X \in \mathbb {R} ^ {d _ {1} \times d _ {2}}} \| \mathcal {A} [ X ] - b \| _ {2} ^ {2} \text {s u b j e c t t o} \operatorname {r a n k} (X) \leq r.
$$
As optimizing over low-rank matrices are often computationally hard, one common approach is to replace the non-convex low-rank constraint with its convex-relaxation, which results in the following
nuclear norm minimization problem [RFP10]:
$$
\min  _ {X \in \mathbb {R} ^ {d _ {1} \times d _ {2}}} \| X \| _ {*} \text {s u b j e c t} A [ X ] = b. \tag {1}
$$
Another widely-used approach in practice is to consider the unconstrained non-convex factorized parametrization [RFP10, GJZ17, BNS16]:
$$
\min  _ {U \in \mathbb {R} ^ {d _ {1} \times r}, V \in \mathbb {R} ^ {d _ {2} \times r}} \| \mathcal {A} [ U V ^ {\top} ] - b \| _ {2} ^ {2}. \tag {2}
$$
and solve it with some form of gradient descent or alternating minimization.
Existing convex and non-convex approaches all rely on certain assumptions. A standard assumption in the literature is that the sensing matrices satisfy the Restricted Isometry Property (RIP), which means that the sensing matrices approximately preserve the norm of a low-rank matrix. (Formally,  $\frac{1}{L}\cdot \| X\| _F^2\leq \frac{1}{n}\sum_{i = 1}^{n}\langle A_i,X\rangle^2\leq L\cdot \| X\| _F^2$  given  $\mathrm{rank}(X)\leq r$  for some parameters  $r$  and  $L$ .)
In this paper, we relax the RIP condition on the sensing matrices and study a robust version of the problem, which is often referred to as the semi-random model. More specifically, an adversary "corrupts" the input data by providing any number of additional sensing matrices  $A_{i}$  that are adversarially chosen, but the corresponding measurements  $b_{i} = \langle A_{i}, X^{*} \rangle$  remain consistent with the ground truth matrix  $X^{*}$ . Consequently, only a subset of the sensing matrices satisfy the RIP condition and the rest of them are arbitrary. This is an intermediate scenario between the average case and the worst case, which arises more frequently in practice.
To the best of our knowledge, we are the first to study the matrix sensing problem in this semi-random model. Formally, we consider the following adversary: suppose that originally there was a collection of RIP sensing matrices  $\{A_i\}_{i=1}^m$  ("good" matrices), then the adversary augmented some arbitrary  $\{A_i\}_{i=m+1}^n$  ("bad" matrices) and then shuffled all the sensing matrices. The algorithm is then given the measurements based on the "good" and "bad" matrices together. The combined sensing matrices are no longer guaranteed to satisfy the RIP condition, but there exists a subset (indicated by an indicator vector  $w^*$ ) that does, i.e.,  $\frac{1}{L} \cdot \|X\|_F^2 \leq \sum_{i=1}^n w_i^* \langle A_i, X \rangle^2 \leq L \cdot \|X\|_F^2$ , where  $w_i^* = \frac{1}{m}$  on the original "good" matrices and  $w_i^* = 0$  on the "bad" matrices added by the adversary. In general, the subset may be replaced by a convex combination and the indicator vector by a simplex. Inspired by the adversary for semi-random vector regression in  $[\mathrm{KLL}^{+}23]$ , we refer to this condition as wRIP (weighted RIP) and formally define it in Definition 2.2.
Since the wRIP condition is a more general assumption than RIP, existing solutions that rely on RIP might fail under the semi-random model with wRIP condition. As stated in  $\mathrm{[KLL^{+}23]}$ , this type of adversary does not break the problem from an "information-theoretic perspective", but affects the problem computationally. In particular, existing non-convex approaches for matrix sensing (e.g., 2) may get stuck at bad local minima as the RIP condition is necessary for proving landscape results regarding the non-convex objective (see, e.g., the counter-examples provided in [BNS16]. The convex relaxation approach (1) does continue to work in the semi-random model, because the augmented linear measurements are consistent with the ground-truth matrix  $X^{*}$  which simply provides additional optimization constraints. However, convex approaches are often less desirable in practice and can become computationally prohibitive when  $d_{1}, d_{2} > 100$  as pointed out in [RFP10].
The limitations of existing algorithms motivate us to pose and study the problem of semi-random matrix sensing in this paper. We summarize our main contributions below:
- Pose and study matrix sensing in the semi-random model. We introduce the more general wRIP condition on matrix sensing as a relaxation of the typical RIP assumption, and provide a solution that is more robust to input contamination. Our work will serve as a starting point for the design of more efficient robust algorithms for matrix sensing, as well as other low-rank matrix problems, in the semi-random model.
- Design an efficient robust algorithm for semi-random matrix sensing. Our algorithm is guaranteed to converge to a global optimum which improves on the existing non-convex solution [BNS16] that can get stuck in bad local optima in the semi-random model, while achieving a comparable running time as existing convex solution [RFP10], informally stated in Theorem 1.1 below. A formal statement can be found in Theorem 3.1.
- Adapt a reweighting scheme for semi-random matrix sensing. In contrast to the non-convex approach that failed and the convex approach that avoided the challenge posed by the adversary altogether, we study a new approach that directly targets the semi-random adversary instead. We develop an algorithm using an iterative reweighting approach inspired by  $\left[\mathrm{KLL}^{+}23\right]$ : in each iteration, the algorithm reweights the sensing matrices to combat the effect of the adversary and then takes a weighted gradient step that works well based on the current solution.
- Exploit the connection between sparsity and low-rankness. Observing a duality between sparse vectors and low-rank matrices, we draw a parallel between linear regression and matrix sensing problems. By exploring the structural similarities and differences between vector and matrix problems, we are able to extend and generalize the work of  $\left[\mathrm{KLL}^{+}23\right]$  on semi-random sparse vector recovery to the higher dimensional problem of semi-random matrix sensing. We emphasize that even though the generalization from vector to matrix problems is natural, the analysis behind the intuition is often nontrivial and involves different mathematical tools.
We state a simplified version of our main algorithmic result assuming Gaussian design. The more general result is stated as Theorem 3.1 in Section 3.
Theorem 1.1 (Semi-Random Matrix Sensing). Suppose the ground-truth matrix  $X^{*} \in \mathbb{R}^{d_{1} \times d_{2}}$  satisfies  $\mathrm{rank}(X^{*}) \leq r$  and  $\|X^{*}\|_{F} \leq \mathrm{poly}(d)$ . Let  $A_{1}, \ldots, A_{n}$  be the sensing matrices and let  $b_{i} = \langle A_{i}, X^{*} \rangle$  be the corresponding measurements. Suppose there exists a (hidden) set of  $\Omega(dr)$  sensing matrices with i.i.d. standard Gaussian entries, and the remaining sensing matrices are chosen adversarially, where  $d = \max(d_{1}, d_{2})$ .
There exists an algorithm that can output  $X \in \mathbb{R}^{d_1 \times d_2}$  such that  $\| X - X^* \|_F \leq \epsilon$  with high probability in time  $\widetilde{O}(nd^{\omega + 1}r\log(1 / \epsilon))^1$  where  $\omega < 2.373$  is the matrix multiplication exponent.
1.2 Overview of Our Techniques
Since there exists a subset (or a convex combination in general) of the sensing matrices that satisfy the RIP condition, a natural strategy is to reverse the effect from the adversary by reweighting the sensing matrices so that they satisfy the RIP condition. However, it is NP-hard to verify RIP condition on all low-rank inputs, so it is unclear how to preprocess and "fix" the input in the beginning and then apply existing solutions to matrix sensing. Instead, we make a trade-off between the frequency of reweighting and the requirement on the weights by adopting an iterative reweighting approach: in each iteration, we only aim to find a set of weights so that the weighted matrices satisfy some desirable properties (not necessarily RIP) with respect to the current estimate  $X$  (as opposed to all low-rank matrices).
Inspired by the workflow in  $\left[\mathrm{KLL}^{+}23\right]$ , our semi-random matrix sensing algorithm (Algorithm 1) repeatedly calls a halving algorithm to reduce the error of our estimate arbitrarily small. The halving algorithm (Algorithm 2) contracts the upper bound on  $\|X - X^{*}\|_{F}$ , which is the error between our current estimate  $X$  and the ground truth  $X^{*}$ , each time it is run. Inside this algorithm is a gradient descent style loop, where in each iteration we try to minimize a weighted objective function, which is essentially the weighted  $\ell_2$ -norm of  $\mathcal{A}[X_t] - b$  (the distance to  $X^{*}$  "measured" by the sensing matrices), where the weights are provided by an oracle implemented in Algorithm 3. The algorithm proceeds by taking a step opposite to the gradient direction, and the step is then projected onto a nuclear-norm-bounded ball which is necessary for the weight oracle to continue working in the next step. As we mentioned before, the weights from the oracle need to satisfy some nice properties with respect to the current iteration estimate  $X_{t}$ . Ideally, the property should: firstly, ensure the gradient step makes enough progress towards  $X^{*}$ ; secondly, can be derived from the wRIP condition so that we know such a requirement is feasible; and lastly, be easily verifiable as opposed to the NP-hard RIP condition.
With the first requirement in mind, we define the weight oracle as in Definition 2.5. The oracle output should satisfy two properties, namely the progress and decomposition guarantees, and together they ensure the gradient step makes good enough progress toward  $X^{*}$ . Intuitively speaking, the progress guarantee ensures the gradient step is large in the direction parallel to the "actual" deviation  $X - X^{*}$  (as opposed to only reducing the "measured" deviation  $\mathcal{A}[X] - b$ ) and thus will make significant progress, while the decomposition guarantee ensures the gradient step has small contribution and
1 Throughout the paper, we write  $\widetilde{O}(f(n))$  for  $O(f(n)$  polylog  $f(n))$  and similarly for  $\widetilde{\Omega}(\cdot)$ .
effect in other directions thus will not cancel the progress after the projection. While the progress guarantee is quantified as an inner product, we introduce a concept called "norm-decomposition" (Definition 2.4) to capture the decomposition guarantee, and we will provide more details later. For the second requirement, we can loosely relate the two oracle guarantees to the wRIP condition: the (large) progress guarantee makes use of the lower bound in wRIP condition  $\sum_{i=1}^{n}w_{i}^{*}\langle A_{i},\frac{X}{\|X\|_{F}}\rangle^{2}\geq\frac{1}{L}$ , and the (small) decomposition guarantee makes use of the upper bound  $\sum_{i=1}^{n}w_{i}^{*}\langle A_{i},\frac{X}{\|X\|_{F}}\rangle^{2}\leq L$ . We introduce a condition called dRIP (decomposable wRIP defined in Definition 2.3) to formally capture this relation, and we will show that it follows from the wRIP condition thus we can achieve such an oracle. Lastly, we will show that the oracle properties can be easily verified, meeting our third requirement.
A formal statement and a road map that leads to our main result can be found in Section 3.
Matrix sensing (RIP): There are two main types of existing solutions. The convex-relaxation formulation 1 of the problem can be posed as a semidefinite program via the standard form primal-dual pair [RFP10], where the primal problem has a  $(d_{1} + d_{2})^{2}$  semidefinite constraint and  $n$  linear constraints. State of the art SDP solver  $\mathrm{[JKL^{+}20]}$  requires running time of  $\tilde{O}(nd^{2.5})$  where  $d = \max(d_{1}, d_{2})$ . The other approach uses non-convex formulation 2 to reduce the size of the decision variable from  $d^{2}$  to  $dr$ , improving computational efficiency. It is shown in [BNS16] that there are no spurious local minima given RIP sensing matrices and incoherent linear measurements in the non-convex approach, however, it is no longer applicable in the semi-random model.
Semi-random model: First introduced by [BS95], the semi-random model has been studied for various graph problems [FK01, PW17, MS10, MMV12]. Previously the work of [CG18] applied the semi-random model to the matrix completion problem, and recently  $\mathrm{[KLL^{+}23]}$  studied sparse vector recovery in this model.
Semi-random matrix completion: Low-rank matrix problems such as matrix completion and matrix sensing have similar optimization landscapes [GJZ17], thus development in one often lends insight to another. Prior work [CG18] on the closely-related problem of matrix completion under the semirandom model showed that all bad local optima can be eliminated by reweighting the input data via a preprocessing step. It exploits the connection between the observation data matrix and the Laplacian matrix of a complete bipartite graph, and gives a reweighting algorithm to preprocess the data in a black-box manner. However, the analogous approach for matrix sensing requires reweighting a set of matrices to satisfy RIP, which is a condition that is NP-hard to check, thus is not practical in the matrix sensing problem.
Semi-random vector regression: In order to overcome the barrier of the reweighting or preprocessing approach mentioned earlier, we take inspiration from the work of  $\mathrm{[KLL^{+}23]}$  on sparse vector recovery under the semi-random model. One of their main contributions is the "short-flat decomposition", which is a property that can be efficiently verified for a given vector (locally), instead of verifying the RIP condition for all sparse vectors (globally). They provide a projected gradient descent style algorithm, where the rows of the sensing matrix are reweighted differently in each iteration to ensure a "short-flat decomposition" exists for the gradient. We draw a parallel between the problem of sparse vector regression and low-rank matrix sensing, and extend their work on linear regression of sparse vectors to the more generalized problem of sensing low-rank matrices.
Throughout the paper, we denote the ground-truth low-rank matrix as  $X^{*}$ . We assume  $X^{*} \in \mathbb{R}^{d_{1} \times d_{2}}$ ,  $\mathrm{rank}(X^{*}) = r$ , and  $d_{1}, d_{2}$  have the same order of magnitude. Let  $d = \max(d_{1}, d_{2})$ .
We write  $[n]$  for the set of integers  $\{1,\dots,n\}$ . We use  $\Delta^n$  for the nonnegative probability simplex in dimension  $n$ , and  $\mathbb{R}_{\geq 0}^n$  for the set of vectors with nonnegative coordinates in  $\mathbb{R}^n$ . For a vector  $x$ , we denote its  $\ell_1, \ell_2$ , and  $\ell_{\infty}$ -norms as  $\| x\| _1$ ,  $\| x\| _2$  and  $\| x\|_{\infty}$  respectively, and write the  $i^{\mathrm{th}}$  coordinate in  $x$  as  $x_{i}$ . For a matrix  $A$ , we use  $\| A\|_{*}$ ,  $\| A\| _2$ , and  $\| A\| _F$  for the nuclear, spectral (operator), and
Frobenius norms of  $A$  respectively. For a matrix  $A$ , we use  $A_{(k)} = \mathrm{argmin}_{\mathrm{rank}(A') \leq k} \| A - A' \|_F$  to denote the best rank- $k$  approximation of  $A$ ; or equivalently, given the SVD of  $A = \sum_{i=1}^{r} \sigma_i u_i v_i^\top$ , we have  $A_{(k)} = \sum_{i=1}^{k} \sigma_i u_i v_i^\top$  where  $\sigma_1, \ldots, \sigma_k$  are the top  $k$  singular values of  $A$ .
We write  $\operatorname{tr}(A)$  for the trace of a square matrix  $A$ . For matrices  $A, B \in \mathbb{R}^{d_1 \times d_2}$ , we write  $\langle A, B \rangle$  for their entrywise inner product  $\langle A, B \rangle = \langle A, B \rangle = \operatorname{tr}(A^\top B) = \sum_{j,k} A_{jk} B_{jk}$ . A symmetric matrix  $A \in \mathbb{R}^{d \times d}$  is positive semidefinite (PSD) if and only if  $A = U^\top U$  for some matrix  $U$ , and we write  $A \preceq B$  if  $A$  and  $B$  have the same dimension and  $B - A$  is positive semidefinite. We write  $\exp(A)$  as the matrix exponential of  $A$ ; if  $A$  is diagonalizable as  $A = U D U^{-1}$  then  $\exp(A) = U \exp(D) U^{-1}$ .
2.2 Definitions
We formally define the matrix sensing operator and observation vector below.
Definition 2.1 (Matrix Sensing Operator). Given a collection of sensing matrices  $\mathcal{A} = \{A_i\}_{i=1}^n \subset \mathbb{R}^{d_1 \times d_2}$ , we define the sensing operator  $\mathcal{A}[\cdot] : \mathbb{R}^{d_1 \times d_2} \to \mathbb{R}^n$  such that  $\mathcal{A}[X] = b$  where  $b_i = \langle A_i, X \rangle$  for  $X \in \mathbb{R}^{d_1 \times d_2}$ .
In other words, we have  $b \coloneqq \sum_{i=1}^{n} \langle A_i, X \rangle e_i$  where  $e_i$  is the  $i^{\text{th}}$  standard basis vector in  $\mathbb{R}^n$ . Throughout the paper, we use either  $\mathcal{A}$  or  $\{A_i\}_{i=1}^n$  to represent the sensing matrices.
To consistently recover a rank- $r$  matrix in general, the number of measurements  $n$  should be at least  $(d_{1} + d_{2} - r)r$  [CP11], hence we assume  $n = \widetilde{\Omega}(dr)$  where  $\widetilde{\Omega}$  suppresses log factors. In most matrix sensing literature, it is standard to impose the Restricted Isometry Property (RIP) condition on the sensing matrices. The RIP condition states that  $\mathcal{A}[\cdot]$  is approximately an isometry on low-rank matrices, which means the  $\ell_{2}$ -norm of the observation vector is close to the Frobenius norm of  $X^{*}$ .
In this paper, we consider a semi-random model and relax the RIP condition as follows: we require that there exist weights  $\{w_i^*\}_{i=1}^n$  (or  $w^* \in \Delta^n$ ) so that the weighted sensing matrices  $\{\sqrt{w_i^*} A_i\}_{i=1}^n$  satisfy the RIP condition. We call this relaxed assumption the wRIP (weighted RIP) condition.
We formally define RIP and wRIP conditions below.
Definition 2.2 (RIP and wRIP Conditions). We say a collection of sensing matrices  $\mathcal{A} = \{A_i\}_{i=1}^n \subset \mathbb{R}^{d_1 \times d_2}$  satisfies the RIP (Restricted Isometry Property) condition with parameters  $r, L$ , and  $\rho$  if the following conditions hold for all  $X \in \mathbb{R}^{d_1 \times d_2}$  with  $\mathrm{rank}(X) \leq r$ :
1. Boundedness:  $\| A_i\| _2\leq \rho$
2. Isometry:  $\frac{1}{L} \cdot \|X\|_F^2 \leq \frac{1}{n} \sum_{i=1}^{n} \langle A_i, X \rangle^2 \leq L \cdot \|X\|_F^2$ .
Further, we say  $\mathcal{A} = \{A_i\}_{i=1}^n$  satisfies the wRIP (weighted RIP) condition with parameters  $r, L, \rho$ , if  $\exists w^* \in \Delta^n$  such that the following conditions hold for all  $X \in \mathbb{R}^{d_1 \times d_2}$  with  $\mathrm{rank}(X) \leq r$ :
1. Boundedness:  $\| A_i\| _2\leq \rho$
2. Isometry:  $\frac{1}{L} \cdot \|X\|_F^2 \leq \sum_{i=1}^{n} w_i^* \langle A_i, X \rangle^2 \leq L \cdot \|X\|_F^2$ .
Notice that wRIP is a relaxation of the RIP condition, because we can choose  $w_{i}^{*} = 1 / n$  for all  $i$  in the standard RIP setting. More importantly, wRIP is strictly weaker. For example, wRIP allows a (possibly majority) fraction of the sensing matrices to be chosen adversarially. We want to emphasize that the algorithm does not know  $w^{*}$  — one of the main challenges of semi-random matrix sensing is that finding  $w^{*}$  seems computationally hard, because it is NP-Hard to check the RIP condition.
For presenting our algorithm and analysis, we introduce a variant of the wRIP condition called dRIP (decomposable-wRIP).
Definition 2.3 (dRIP Condition). We say a collection of sensing matrices  $\mathcal{A} = \{A_i\}_{i=1}^n \subset \mathbb{R}^{d_1 \times d_2}$  satisfies the dRIP (decomposable wRIP) condition if  $\exists w^* \in \Delta^n$  and constants  $L, K, r, \rho \geq 1$ , such that for all  $V \in \mathbb{R}^{d_1 \times d_2}$  satisfying  $\|V\|_F \in [\frac{1}{4}, 1]$ ,  $\|V\|_* \leq 2\sqrt{2r}$ :
1. Boundedness:  $\| A_i\| _2\leq \rho$
2. Isometry:  $\frac{1}{L} \leq \sum_{i=1}^{n} w_i^* \langle A_i, V \rangle^2 \leq L$ ;
3. Decomposability:  $\exists (L, \frac{1}{K\sqrt{r}})$ -norm-decomposition of  $G^{*} = \sum_{i=1}^{n} w_{i}^{*}\langle A_{i}, V \rangle A_{i} = \sum_{i=1}^{n} w_{i}^{*}u_{i}A_{i}$ .
Definition 2.4 (Norm Decomposition). We say a matrix  $G$  has a  $(C_F, C_2)$ -norm-decomposition if  $\exists S$  and  $E$  s.t.  $G = S + E$ , and  $\| S \|_F \leq C_F$ ,  $\| E \|_2 \leq C_2$ .
The main difference with wRIP is that dRIP requires the additional "decomposition" property. Observe that  $G^{*}$  is the (weighted) gradient at the point  $V$ . At a high level, we would like to decompose the gradient into two matrices, one with small Frobenius norm and the other one with small operator norm. Our matrix norm-decomposition is inspired by the "short-flat-decomposition" for vectors in  $\mathrm{[KLL^{+}23]}$ .
In Section 4, we will explain the motivation behind the norm decomposition as well as how to efficiently verify such a decomposition exists. We will also show that the dRIP condition is closely related to wRIP (by choosing parameters within a constant factor of each other) in Appendix C.