SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 1. The loss function  $f$  is twice continuously differentiable.
Assumption 2. The ground truth matrix  $M^{*}$  is PSD and rank-  $r$
Assumption 3. The ground truth matrix  $M^{*}$  is a global minimizer of  $f[(\cdot)_{\Omega};M_{\Omega}^{*}]$ .
We note that these assumptions are standard in the low-rank optimization literature. The first assumption is a mild regularity assumption on the loss function and is satisfied by a wide range of loss functions, including the  $\ell_2$ -loss and the negative log-likelihood function of various probability distributions. The second assumption is based on the prior knowledge about the specific application. The third assumption is necessary for the exact recovery of the ground truth  $M^{*}$ .
We give an informal statement of the main result in the following theorem. Here, we consider the  $(\delta ,\Omega)$  -RIP  $2r,2r$  condition, which is an extension of the classic  $\delta$  -RIP  $2r,2r$  condition. Intuitively, the constant  $\delta \in [0,1)$  measures the similarity between  $f$  and the  $\ell_2$  -loss function. The rigorous definition is provided in Definition 4.
Theorem 1 (Informal). Suppose that the loss function  $f$  satisfies the  $(1/16, \Omega)$ - $RIP_{2r,2r}$  condition with a high probability over  $\Omega$  and the sampling rate  $p$  satisfies
$$
n p \geq \Theta \left[ \mu^ {2} r ^ {3} \left(\kappa^ {*}\right) ^ {2} \log n \right],
$$
where  $\kappa^{*}$  is the condition number of  $M^{*}$ . Then, with a suitable choice of the parameters  $\alpha$  and  $\lambda$ , problem (12) satisfies the strict saddle property with high probability. Furthermore, there exist algorithms that can find a solution  $U_{0}$  such that  $\| U_0U_0^T - M^*\|_F \leq \epsilon$  in polynomial time with the same probability.
The above theorem provides the first theoretical result for the MSoG problem. Basically, Theorem 1 says that a combination of the incoherence condition and the  $\Omega$ -RIP condition is sufficient to guarantee that problem (12) can be solved in polynomial time with high probability. In addition, the lower bound on the sampling rate is better than the bound in [17] (i.e., the bound in (9)). This is a result of our improved regularizer. Finally, the upper bound on the  $\Omega$ -RIP constant is an absolute constant (namely,  $1/16$ ).
Remark 1. We note that since the result of Theorem 1 only holds with high probability, it is not necessary for the  $\Omega$ -RIP condition to hold for all subsets  $\Omega$ . Instead, we only require the  $\Omega$ -RIP condition to hold for a set of  $\Omega$  that correspond to a high probability over the distribution of  $\Omega$ . In the special case when  $\Omega$  is a deterministic subset that satisfies similar "benign" properties as in the random graph case, we only need the  $\Omega$ -RIP condition for this deterministic subset  $\Omega$ .
The next informal theorem shows that our upper bound is optimal up to a constant.
Theorem 2 (Informal). There exists a loss function  $f$  such that: (i)  $f$  satisfies the  $(1/2, \Omega)$ - $RIP_{2r,2r}$  condition for all  $\Omega \subset [n] \times [n]$ , (ii) for all  $p \in [0,1]$ , problem (12) has a spurious second-order critical point with probability at least  $(3 - \sqrt{5}) / 2 \approx 0.38$ .
Intuitively, Theorem 2 provides a negative result saying that the tightest upper bound on the RIP constant cannot be better than  $1/2$  if the problem (12) has a benign landscape with high probability. This is also the first negative result on the problem (12).
1.5 Notation
For every natural number  $n$ , we denote  $[n] := \{1, \ldots, n\}$ . The operator 2-norm and the Frobenius norm of a matrix  $M$  are denoted as  $\|M\|_2$  and  $\|M\|_F$ , respectively. The trace of matrix  $M$  is denoted as  $\operatorname{tr}(M)$ . The inner product between two matrices is defined as  $\langle M, N \rangle := \operatorname{tr}(M^T N)$ . For any matrix  $M \in \mathbb{R}^{n \times n}$ , we denote its singular values by  $\sigma_1(M) \geq \dots \geq \sigma_k(M)$ . Let  $\sigma_i^*$  be the singular value  $\sigma_i(M^*)$ . The condition number of  $M^*$  is  $\kappa^* = \sigma_1^* / \sigma_r^*$ . The minimum eigenvalue of matrix  $M$  is denoted as  $\lambda_{min}(M)$ . For any two matrices  $A, B \in \mathbb{R}^{n \times m}$ , we use  $A \otimes B$  to denote the fourth-order tensor whose  $(i, j, k, \ell)$  element is  $A_{i,j}B_{k,\ell}$ . The identity tensor is denoted as  $\mathcal{I}$ . The notation  $M \succeq 0$  means that the matrix  $M$  is PSD. The sub-matrix  $R_{i:j,k:\ell}$  consists of the  $i$ -th to the  $j$ -th rows and the  $k$ -th to the  $\ell$ -th columns of matrix  $R$ . The action of the Hessian  $\nabla^2 f(M)$  on any two matrices  $K$  and  $L$  is given by  $[\nabla^2 f(M)](K, L) := \sum_{i,j,k,\ell} [\nabla^2 f(M)]_{i,j,k,\ell} K_{ij}L_{k,\ell}$ . The notation  $f = O(g)$  means that there exists an absolute constant  $C > 0$  such that  $f \leq C \cdot g$ . The notation  $f = \Theta(g)$  means that there exist absolute constants  $C_1, C_2, > 0$  such that  $C_1 \cdot g \leq f \leq C_2 \cdot g$ .
2  $\Omega$ -RIP Condition
Since the objective function  $f(M_{\Omega};M_{\Omega}^{*})$  does not satisfy the classic RIP condition unless all elements of  $M^{*}$  are observed (i.e., when  $\Omega = [n]\times [n]$ ), it is necessary to consider a generalization of the RIP condition to the partial observation case.
Definition 4 ( $\Omega$ -RIP Condition). Given a subset  $\Omega \subset [n] \times [n]$  and natural numbers  $r, s$ , the function  $f(\cdot; M^*)$  is said to satisfy the  $\Omega$ -Restricted Isometry Property ( $\Omega$ -RIP) of rank  $(2r, 2s)$  for a constant  $\delta \in [0,1)$ , denoted as  $(\delta, \Omega)$ - $RIP_{2r,2s}$ , if
$$
(1 - \delta) \| K _ {\Omega} \| _ {F} ^ {2} \leq \left[ \nabla^ {2} f \left[ M _ {\Omega}; \left(M ^ {*}\right) _ {\Omega} \right] \right] \left(K _ {\Omega}, K _ {\Omega}\right) \leq (1 + \delta) \| K _ {\Omega} \| _ {F} ^ {2} \tag {13}
$$
holds for all matrices  $M, K \in \mathbb{R}^{n \times n}$  such that  $\mathrm{rank}(M) \leq 2r$ ,  $\mathrm{rank}(K) \leq 2s$ .
Note that the matrix completion problem satisfies the  $(0,\Omega)$ - $\mathrm{RIP}_{2r,2s}$  condition. In the following, we demonstrate another example for which the  $\Omega$ -RIP condition holds with a non-zero constant  $\delta$ .
Example 1 (Linear matrix sensing over graphs). In the linear matrix sensing over graphs problem, the loss function is defined as
$$
f \left(M _ {\Omega}; M _ {\Omega} ^ {*}\right) := \left\| \mathcal {A} \left(M _ {\Omega}\right) - \mathcal {A} \left(M _ {\Omega} ^ {*}\right) \right\| _ {F} ^ {2},
$$
where the linear operator  $\mathcal{A}:\mathbb{R}^{n\times n}\mapsto \mathbb{R}^m$  (defined in (3)) is generated by Gaussian measurements and  $m$  is the number of measurements. For all subsets  $\Omega \subset [n]\times [n]$ , a similar proof to that of Theorem 2.3 of [9] implies that the  $(\delta ,\Omega)$ - $RIP_{2r,2r}$  condition holds with high probability when  $m\geq cnr / \delta^2$  for some constant  $c > 0$ . The intuition behind the proof is that the constructed  $\epsilon$ -net for the linear matrix sensing problem is also an  $\epsilon$ -net for the linear MSoG problem since  $\| M_{\Omega} - M_{\Omega}^{\prime}\|_{F}\leq \| M - M^{\prime}\|_{F}$  for all  $M,M^{\prime}\in \mathbb{R}^{n\times n}$ .
Remark 2. In the above example and other examples in practice, the  $\Omega$ -RIP condition holds with high probability for a fixed subset  $\Omega$ . Therefore, we can focus on the event when the  $\Omega$ -RIP condition holds since otherwise the results will only differ by a sufficiently small probability.
In Section 4 and Appendix C, we numerically show that the optimization complexity of problem (11) is closely related to the  $\Omega$ -RIP constant  $\delta$  and the sampling rate  $p$ .
2A point  $U_{0}$  is called a spurious second-order critical point of problem (12) if  $U_{0}$  satisfies the first-order and the second-order necessary optimality conditions but  $U_{0}U_{0}^{T} \neq M^{*}$ .
In this section, we provide strong theoretical results on the MSoG problem (12). We first develop a sufficient condition on the benign landscape of problem (12) and then study the tightness of our developed condition.
3.1 Global landscape: Strict saddle property
First, we develop conditions under which problem (12) does not have any spurious second-order critical points and therefore saddle-escaping methods (e.g., [22, 23]) can find an approximate global minimum in polynomial time. To guarantee the global convergence, the strict saddle property is commonly considered in the literature:
Definition 5 (Strict Saddle Property [33]). Consider an optimization problem  $\min_{x\in \mathcal{X}\subset \mathbb{R}^d}F(x)$  and let  $\mathcal{X}^*$  denote the set of its global minima. We say that the problem satisfies the  $(\theta ,\beta ,\gamma)$ -strict saddle property for  $\theta ,\beta ,\gamma >0$  if at least one of the following conditions is satisfied for every  $x\in \mathcal{X}$ :
$$
\operatorname {d i s t} (x, \mathcal {X} ^ {*}) \leq \theta ; \quad \| \nabla F (x) \| _ {F} \geq \beta ; \quad \lambda_ {\min } [ \nabla^ {2} F (x) ] \leq - \gamma ,
$$
where  $\mathrm{dist}(x,\mathcal{X}^*)\coloneqq \inf_{x^*\in \mathcal{X}^*}\| x - x^*\| _F$  is the distance between  $x$  and  $\mathcal{X}^*$
For the low-rank optimization problem, the distance in the factorization space is equivalent to the distance in the matrix space in the sense that there exist constants  $c_{1}(\mathcal{X}^{*}), c_{2}(\mathcal{X}^{*}) > 0$  such that
$$
c _ {1} \left(\mathcal {X} ^ {*}\right) \cdot \| U - U ^ {*} \| _ {F} \leq \| U U ^ {T} - U ^ {*} \left(U ^ {*}\right) ^ {T} \| _ {F} \leq c _ {2} \left(\mathcal {X} ^ {*}\right) \cdot \| U - U ^ {*} \| _ {F}
$$
holds for all  $U \in \mathcal{X}$  as long as  $\| U - U^{*}\|_{F}$  is small and  $\mathcal{X}^*$  is bounded [34]. Denote the objective function of problem (12) as  $\ell(U)$ . As an example of saddle-avoiding methods, the accelerated perturbed gradient descent algorithm [23] can find a point  $U_0$  such that
$$
\| \nabla \ell (U _ {0}) \| _ {F} = O (\epsilon), \quad \lambda_ {\min } [ \nabla^ {2} \ell (U _ {0}) ] = - O (\sqrt {\epsilon})
$$
in  $O(\epsilon^{-1.75})$  iterations with high probability. If we choose  $\epsilon > 0$  to be small enough, the strict saddle property ensures that the point  $U_0$  satisfies  $\| U_0U_0^T - M^* \|_F = O(\theta)$ . We note that the Lipschitz continuity of the Hessian of  $\ell$  can be guaranteed by the regularity assumption 1 and the boundedness of trajectories of the algorithm, which can be proved in a similar way as Theorem 8 in [22]. In summary, if the strict saddle property holds, we can apply saddle-avoiding methods to achieve the polynomial-time global convergence for problem (12).
Now, we prove that the problem (12) satisfies the strict saddle property with high probability under the  $\Omega$ - $\mathrm{RIP}_{2r,2r}$  condition and the incoherence condition. Compared with Theorem 1, we assume, for the simplicity of the statement of the theorem, that the  $\Omega$ -RIP condition holds for all subset  $\Omega$  since otherwise, the result will only differ by a small probability.
Theorem 3. Suppose that the loss function  $f$  satisfies the  $(\delta, \Omega)$ - $RIP_{2r,2r}$  condition for all  $\Omega \subset [n] \times [n]$  and
$$
\alpha^ {2} = \Theta \left(\frac {\mu r \sigma_ {1} ^ {*}}{n}\right), \quad \lambda = \Theta \left[ \frac {(\sqrt {\mu} + \sqrt {n} \delta) n}{\sqrt {\mu} r} \right], \quad n p \geq \Theta [ \mu^ {2} r ^ {3} (\kappa^ {*}) ^ {2} \log n ], \quad \delta <   \frac {1}{1 6}.
$$
Then, there exists a small constant  $\epsilon > 0$  such that with probability at least  $1 - 1 / \mathrm{poly}(n)$ , the MSoG problem (12) satisfies the  $(\theta, \beta, \gamma)$ -strict saddle property with
$$
\theta = \Theta (\epsilon / \sigma_ {r} ^ {*}), \quad \beta = \epsilon , \quad \gamma = \Theta (\epsilon^ {2} / \sigma_ {r} ^ {*}).
$$
In Theorem 3, we provide the first theoretical results on the MSoG problem (12). Basically, the theorem shows that if the  $\Omega$ -RIP $_{2r,2r}$  constant is smaller than an absolute constant, then the same sampling rate as for the matrix completion problem is sufficient to guarantee the benign landscape of problem (12). Therefore, the result is a generalization of the results in [17], which considered the case when the  $\Omega$ -RIP constant  $\delta$  is zero and the sampling rate  $p$  satisfies a stricter condition (9). On the other hand, if the sampling rate  $p$  is equal to 1, Theorem 3 guarantees that the strict saddle property holds when the (regular) RIP condition holds with  $\delta < 1/16$ . Compared with the state-of-the-art results in [38, 5], the upper bound on  $\delta$  in our work may not be optimal but is only worse by an absolute constant. We leave the improvement of the upper bound as future work.
3.2 Tightness of the  $\Omega$ -RIP condition
To study the tightness of our  $\Omega$ -RIP condition for problem (12), we construct an instance of problem (12) that satisfies the  $(1/2, \Omega)$ -RIP condition but has spurious second-order critical points. Note that the existence of spurious second-order critical points negates the strict saddle property. The counterexample is based on a similar idea as those in [38]. More specifically, we assume that  $n \geq 2r$  and consider the tensor:
$$
\begin{array}{l} \mathcal {H} := \sum_ {i \in [ r ]} \left\{- \frac {1}{2} \left[ \left(e _ {2 i - 1} e _ {2 i - 1} ^ {T}\right) \otimes \left(e _ {2 i - 1} e _ {2 i - 1} ^ {T}\right) + \left(e _ {2 i} e _ {2 i} ^ {T}\right) \otimes \left(e _ {2 i} e _ {2 i} ^ {T}\right) \right] \right. \\ + \frac {1}{2} \left[ \left(e _ {2 i - 1} e _ {2 i - 1} ^ {T}\right) \otimes \left(e _ {2 i} e _ {2 i} ^ {T}\right) + \left(e _ {2 i} e _ {2 i} ^ {T}\right) \otimes \left(e _ {2 i - 1} e _ {2 i - 1} ^ {T}\right) \right] \\ - \frac {1}{4} \left[ \left(e _ {2 i - 1} e _ {2 i} ^ {T}\right) \otimes \left(e _ {2 i - 1} e _ {2 i} ^ {T}\right) + \left(e _ {2 i} e _ {2 i - 1} ^ {T}\right) \otimes \left(e _ {2 i} e _ {2 i - 1} ^ {T}\right) \right] \\ \left. + \frac {1}{4} \left[ \left(e _ {2 i - 1} e _ {2 i} ^ {T}\right) \otimes \left(e _ {2 i} e _ {2 i - 1} ^ {T}\right) + \left(e _ {2 i} e _ {2 i - 1} ^ {T}\right) \otimes \left(e _ {2 i - 1} e _ {2 i} ^ {T}\right) \right] \right\}, \\ \end{array}
$$
where  $e_i \in \mathbb{R}^n$  is the  $i$ -th standard basis of  $\mathbb{R}^n$ . The rank-  $r$  ground truth matrix  $M^{*}$  is constructed as
$$
U ^ {*} := \left[ \begin{array}{c c c c} e _ {1} & e _ {3} & \dots & e _ {2 r - 1} \end{array} \right], \quad M ^ {*} := U ^ {*} (U ^ {*}) ^ {T} = \sum_ {i \in [ r ]} e _ {2 i - 1} e _ {2 i - 1} ^ {T}.
$$
Then, the loss function is given by
$$
f _ {3 / 2} \left(M _ {\Omega}; M _ {\Omega} ^ {*}\right) := \frac {1}{2} \left(M _ {\Omega} - M _ {\Omega} ^ {*}\right): \left(\frac {3}{2} \cdot \mathcal {I} + \mathcal {H}\right): \left(M _ {\Omega} - M _ {\Omega} ^ {*}\right), \quad \forall M \in \mathbb {R} ^ {n \times n}.
$$
It is proved in [38] that if  $\Omega = [n]\times [n]$ , the function  $f_{3 / 2}$  satisfies the  $1 / 2\text{-RIP}_{2r,2r}$  condition and has a spurious second-order critical point. In this work, we generalize the results to the case when the set  $\Omega$  is random and the objective function contains a regularizer.
Theorem 4. Suppose that the loss function in problem (12) is chosen to be  $f_{3/2}$ . Then, it holds that:
1. The loss function  $f_{3/2}$  satisfies the  $(1/2, \Omega)$ - $RIP_{2r,2r}$  condition for all  $\Omega \subset [n] \times [n]$ ;
2. For all  $p \in [0,1]$ , problem (12) has a spurious second-order critical point with probability at least  $\max \{p^{r(r + 1)}, 1 - p^{r(r + 1)/2}\} \geq (3 - \sqrt{5})/2$ .
We note that the results in Theorem 4 hold for all  $p \in [0,1]$ . From Theorem 4, we cannot improve the upper bound of  $\delta$  in Theorem 3 to be better than  $1/2$ . In addition, this result shows that our bound in Theorem 3 is optimal up to a constant. This tightness result is consistent with that of the matrix sensing problem in [40, 38].
4 Numerical Illustrations
In this section, we show how the optimization complexity is related to the  $\Omega$ -RIP constant  $\delta$  and the sampling rate  $p$  via a numerical example. Here, the optimization complexity refers to the probability that the randomly initialized gradient descent algorithm can find the ground truth matrix  $M^{*}$ . In this example, we choose a random orthogonal matrix  $V \in \mathbb{R}^{n \times n}$  and define the loss function to be
$$
f _ {c} \left[ M _ {\Omega}; \left(V M ^ {*} V ^ {T}\right) _ {\Omega} \right] := \frac {1}{2} \left[ M - \left(V M ^ {*} V ^ {T}\right) \right] _ {\Omega}: (c \cdot \mathcal {I} + \mathcal {H}): \left[ M - \left(V M ^ {*} V ^ {T}\right) \right] _ {\Omega}, \forall M \in \mathbb {R} ^ {n \times n},
$$
where  $c \in \mathbb{R}$  is a hyper-parameter and the tensor  $\mathcal{H}$  and the ground truth  $M^{*}$  are defined in Section 3.2. In addition, by a similar analysis as in Section 3.2, we can prove that the function  $f_{c}$  satisfies the  $(1/2, \Omega)$ - $\mathrm{RIP}_{2r,2r}$  condition if we choose  $c = 3/2$ . We also numerically verify this conclusion by checking the curvature  $[\nabla^2 f_{3/2}(M_{\Omega}; M_{\Omega}^*)](K, K)$  along  $10^4$  random directions  $K \in \mathbb{R}^{n \times n}$ . Since  $f_{3/2}$  is a quadratic function, the  $(1/2, \Omega)$ - $\mathrm{RIP}_{2r,2r}$  condition is given by
$$
\frac {1}{2} \| K _ {\Omega} \| _ {F} ^ {2} \leq K _ {\Omega}: \left(\frac {3}{2} \cdot \mathcal {I} + \mathcal {H}\right): K _ {\Omega} \leq \frac {3}{2} \| K _ {\Omega} \| _ {F} ^ {2}, \quad \forall K \in \mathbb {R} ^ {n \times n},
$$
Figure 2: Success rate of the algorithm with different sampling rate  $p$  and RIP constant  $\delta$ .
where we define the tensor multiplication  $K: \mathcal{H}': K = \sum_{i,j,k,\ell \in [n]} \mathcal{H}_{ijk\ell}' K_{ij} K_{k\ell}$  for all  $K \in \mathbb{R}^{n \times n}$  and fourth-order tensor  $\mathcal{H}' \in \mathbb{R}^{n \times n \times n \times n}$ . In addition, since the identity tensor  $\mathcal{I}$  satisfies the  $(0, \Omega)$ - $\mathrm{RIP}_{2r,2r}$  condition, it is straightforward to prove that the function  $f_c$  satisfies the  $(\delta, \Omega)$ - $\mathrm{RIP}_{2r,2r}$  condition with  $\delta = 1/(2c - 1)$  for all  $c \geq 1$ .
We choose the problem size to be  $n = 10$  and  $r = 5$ . The regularization parameters are  $\alpha = 10$  and  $\lambda = 100$ . The set of sampling rates and  $\Omega$ -RIP $_{2r,2r}$  constants are
$$
p \in \{0. 7, 0. 7 5, \dots , 0. 9 5, 1. 0 \}, \quad \delta \in \{0. 2, 0. 2 5, \dots , 0. 7 5, 0. 8 \}.
$$
We solve each problem instance by the Burer-Monterio factorization and the perturbed accelerated gradient descent algorithm [23], where the constant step size is  $0.007 / c$ . We generate 100 independent problem instances and compute the success rate of the gradient descent algorithm with random initialization. We say that the algorithm successfully solves the instance if the generalization error  $\| UU^T - M^* \|_F$  is less than  $10^{-3}$ . If this condition fails, it means that the algorithm is stuck at a spurious local solution.
The results are plotted in Figure 2. We can see from the figure that the optimization complexity grows when  $\delta$  becomes smaller and when  $p$  becomes larger. This result shows that the  $\Omega$ -RIP condition plays an important role in characterizing the optimization complexity of problem (12). To be more concrete, we expect that an upper bound on the  $\Omega$ -RIP constant will be able to guarantee the benign optimization landscape of problem (12). Moreover, the result is consistent with the results for the matrix sensing problem and the matrix completion problem. Furthermore, we can see that when the  $\Omega$ -RIP constant is smaller than  $1/2$ , the success rate has a stronger correlation with  $p$  than  $\delta$ . This observation is also reflected in Theorem 1, where the lower bound on the sampling rate is on the same order as those in the existing works [18, 17] if  $\delta$  is upper bounded. For the cases when  $p$  is close to 1 and  $\delta$  is larger than 0.5, the success rate is better than the case when  $p = 1$ . One possible explanation for this phenomenon is that the loss function  $f_{c}$  has multiple global minima when the sampling rate is smaller than 1. Instead of converging to a spurious local solution, the implicit regularization [26] makes the perturbed gradient descent algorithm more likely to converge to the global solution with the "minimal complexity", which is likely the ground truth  $VM^{*}V^{T}$ . If the sampling rate is much smaller than 1, the number of spurious local minima will increase and the perturbed gradient descent algorithm may stuck at spurious local minima and fail to converge to the ground truth.
In this work, we provide the first theoretical analysis of the MSoG problem. A new notion, dubbed as the  $\Omega$ -RIP condition, is proposed and shown to be useful in characterizing the optimization complexity of the MSoG problem. Using an improved incoherence regularizer, we proved the polynomial-time global convergence of saddle-avoiding methods under the  $\Omega$ -RIP condition and the incoherence condition. The bounds on the sampling rate and the  $\Omega$ -RIP constant are state-of-the-art up to a constant. Moreover, we showed that our bound on the  $\Omega$ -RIP condition is tight (up to a constant). Future works include improving the upper bound on the  $\Omega$ -RIP constant and the lower bound on the sampling rate.
This work was supported by grants from AFOSR, ARO, ONR, and NSF. Haixiang Zhang was supported by the Two Sigma Ph.D. Fellowship.