SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haixiang Zhang
haixiang_zhang@berkeley.edu
Ying Chen
ying-chen@berkeley.edu
In this work, we consider the problem of matrix sensing over graphs (MSoG). As a general case of matrix completion and matrix sensing problems, the MSoG problem has not been analyzed in the literature and the existing results cannot be directly applied to the MSoG problem. This work provides the first theoretical results on the optimization landscape of the MSoG problem. More specifically, we propose a new condition, named the  $\Omega$ -RIP condition, to characterize the optimization complexity of the problem. In addition, with an improved regularizer of the incoherence, we prove that the strict saddle property holds for the MSoG problem with high probability under the incoherence condition and the  $\Omega$ -RIP condition, which guarantees the polynomial-time global convergence of saddle-avoiding methods. Compared with state-of-the-art results, the bounds in this work are tight up to a constant. Besides the theoretical guarantees, we numerically illustrate the close relation between the  $\Omega$ -RIP condition and the optimization complexity.
In a wide range of problems in the fields of machine learning, signal processing and power systems, an unknown low-rank matrix parameter should be estimated from a few measurements of the matrix. To be more specific, given some measurements of the unknown symmetric and positive semi-definite (PSD) matrix  $M^{*} \in \mathbb{R}^{n \times n}$  of rank  $r \ll n$ , the low-rank matrix optimization problem can be formulated as
$$
\min  _ {M \in \mathbb {R} ^ {n \times n}} f (M; M ^ {*}) \quad \text {s . t .} \quad M \succeq 0, \quad \operatorname {r a n k} (M) \leq r, \tag {1}
$$
where  $f(\cdot; M^*)$  is a loss function that penalizes the mismatch between  $M$  and  $M^*$ . The goal is to recover the matrix  $M^*$  via finding a global minimizer of problem (1). Applications of this problem include matrix sensing [31, 40, 38], matrix completion [10, 11, 17], phase retrieval [8, 33, 14], and power systems [39, 24]; see the review papers [13, 15] for more applications. Early attempts to deal with the nonconvex rank constraint of the problem focused on solving a convex relaxation of (1); see [10, 31, 11, 7]. However, the convex relaxation approach usually updates the matrix variable via the Singular Value Decomposition (SVD) in each iteration. This will lead to an  $O(n^{3})$  computational complexity in each iteration and an  $O(n^{2})$  space complexity, which are prohibitively
high for large-scale problems; see the numerical comparison in [41]. Similar issues are observed for algorithms based on the Singular Value Projection [20] and Riemannian optimization algorithms [35, 36, 19, 1, 27].
To improve the computation and memory efficiency, the Burer-Monteiro factorization approach was proposed in [6], which is based on the fact that the mapping  $U \mapsto U U^T$  is surjective onto the manifold of PSD matrices of rank at most  $r$ , where  $U \in \mathbb{R}^{n \times r}$ . More concretely, problem (1) is equivalent to
$$
\min  _ {U \in \mathbb {R} ^ {n \times r}} f \left(U U ^ {T}; M ^ {*}\right). \tag {2}
$$
Due to the non-convexity of the mapping  $U \mapsto U U^T$ , problem (2) is an unconstrained non-convex problem and may have spurious second-order critical points (i.e., second-order critical points that do not correspond to the ground truth matrix  $M^{*}$ ). In general, saddle-avoiding local search methods are only able to find  $\epsilon$ -approximate second-order critical points<sup>1</sup>. As a result, local search methods with a random initialization will likely be stuck at spurious second-order critical points and unable to converge to the ground truth solution. However, in a variety of real-world applications, simple algorithms such as perturbed gradient descent methods and alternating minimization methods have achieved empirical success on problem (2). Recently, substantial progress has been made on the theoretical explanation of the benign behavior of these algorithms. For example, the alternating minimization algorithm was studied in [21, 28, 29]. The (stochastic) gradient descent algorithm, which is in general easier to implement than the alternating minimization algorithm, was analyzed in [8, 34, 37, 14, 13]. Moreover, the gradient descent algorithm is proved to have the implicit regularization phenomenon in the over-parameterization case [26, 16, 32].
Besides the algorithmic analysis, a large amount of literature [17, 33, 42, 38] focused on the geometric analysis of the landscape of problem (2), which usually depends on the strict-saddle property [33]. Intuitively, the strict-saddle property states that at any feasible point of problem (2), at least one of the three properties will hold: (i) the point is close to a global solution; (ii) the norm of the gradient is large; (iii) the Hessian matrix has a negative eigenvalue. In the later two cases, saddle-escaping algorithms [12, 23, 2] are able to find a descent direction and thus, these algorithms will converge globally in polynomial time. The formal definition of the strict-saddle property is provided in Section 3.1. In the following, we review the state-of-the-art conditions for two special classes of problem (2) that guarantee the strict-saddle property; see the survey [15] for other problem classes.
1.1 Matrix sensing and the Restricted Isometry Property (RIP) condition
In the matrix sensing problem, the information of the ground truth matrix  $M^{*}$  is gathered via the measurement operator  $\mathcal{A}$ . The loss function  $f$  is usually chosen to be the negative log-likelihood function. For example, in the classic linear matrix sensing problem, the operator  $\mathcal{A}$  is a linear operator defined as
$$
\mathcal {A} (M) := \left[ \langle A _ {1}, M \rangle , \dots , \langle A _ {m}, M \rangle \right], \quad \forall M \in \mathbb {R} ^ {n \times n}, \tag {3}
$$
where  $m$  is the number of measurements and  $A_{i}\in \mathbb{R}^{n\times n}$  contains independently identically distributed (i.i.d) Gaussian random entries and  $A_{i}$ 's are independent of each other. If the measurement noise is Gaussian, the maximum likelihood estimation is equivalent to the following minimization problem
$$
\min  _ {U \in \mathbb {R} ^ {n \times r}} \| \mathcal {A} (U U ^ {T}) - \mathcal {A} (M ^ {*}) \| _ {F} ^ {2}. \tag {4}
$$
More examples of the (non-linear) matrix sensing problem are discussed in [42]. One of the most important conditions that guarantee the benign landscapes is the RIP condition:
Definition 1 (RIP Condition[31, 42]). Given natural numbers  $r$  and  $s$ , the function  $f(\cdot; M^*)$  is said to satisfy the Restricted Isometry Property (RIP) of rank  $(2r, 2s)$  for a constant  $\delta \in [0, 1)$ , denoted as  $\delta - RIP_{2r,2s}$ , if
$$
(1 - \delta) \| K \| _ {F} ^ {2} \leq \left[ \nabla^ {2} f (M; M ^ {*}) \right] (K, K) \leq (1 + \delta) \| K \| _ {F} ^ {2}
$$
holds for all matrices  $M, K \in \mathbb{R}^{n \times n}$  such that  $\mathrm{rank}(M) \leq 2r$  and  $\mathrm{rank}(K) \leq 2s$ , where  $\left[\nabla^2 f(M; M^*)\right](K, K)$  is the curvature of the Hessian at point  $M$  along direction  $K$ .
<sup>1</sup>A point  $x_0$  is called an  $\epsilon$ -approximate second-order critical point to the optimization problem  $\min_x F(x)$  if  $\| \nabla F(x_0) \|_F \leq \epsilon$  and  $\lambda_{min}[\nabla^2 F(x_0)] \geq -\epsilon$ .
For the linear matrix sensing problem (4), it is proved in [9] that the  $\delta$ - $\mathrm{RIP}_{2r,2s}$  condition holds with high probability if  $m = \Theta (nr\delta^{-2})$ . The RIP condition is also established in many other applications of the matrix sensing problem [42, 4] and is of independent research interest. The constant  $\delta$  plays a critical role in characterizing the optimization landscape of problem (2). More specifically, in [5], the authors showed that the strict-saddle property holds for problem (2) if the  $\delta$ - $\mathrm{RIP}_{2r,2r}$  condition holds with  $\delta < 1/2$ . Counterexamples have been constructed in [40, 38] to illustrate that the strict-saddle property may fail under the  $\delta$ - $\mathrm{RIP}_{2r,2r}$  condition with  $\delta \geq 1/2$ .
1.2 Matrix completion and the incoherence condition
In spite of the strong theoretical results under the RIP condition, there exist a large number of applications that do not satisfy the RIP condition. A well-studied class of problems that does not satisfy the RIP condition is the matrix completion problem. For the matrix completion problem, a subset  $\Omega \subset [n] \times [n]$  of entries of  $M^{*}$  are observed and the goal is to recover the low-rank ground truth matrix from the observed entries. For every matrix  $M \in \mathbb{R}^{n \times n}$ , we denote the projection of  $M$  onto  $\Omega$  as  $M_{\Omega}$ , namely,
$$
(M _ {\Omega}) _ {i j} = \left\{ \begin{array}{l l} M _ {i j} & \text {i f} (i, j) \in \Omega , \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
Using the  $\ell_2$ -loss function, the matrix completion problem is defined as
$$
\min  _ {U \in \mathbb {R} ^ {n \times r}} \| M _ {\Omega} - M _ {\Omega} ^ {*} \| _ {F} ^ {2}. \tag {6}
$$
The matrix completion problem (6) is a special case of the linear matrix sensing problem (4), where the sample size  $m$  is equal to  $|\Omega|$  and each measurement matrix  $A_{i}$  contains exactly one nonzero entry. We note that the  $\delta$ -RIP $_{2r,2r}$  condition does not hold for problem (6) unless we observe all entries of  $M^{*}$ , i.e., when  $\Omega = [n] \times [n]$ . As an alternative to the RIP condition, the incoherence of  $M^{*}$  is useful in characterizing the complexity of problem (6).
Definition 2 (Incoherence Condition [10]). Given  $\mu \in [1,n]$ , the ground truth matrix  $M^{*}$  is said to be  $\mu$ -incoherent if
$$
\left\| e _ {i} ^ {T} V ^ {*} \right\| _ {F} \leq \sqrt {\mu r / n}, \quad \forall i \in [ n ], \tag {7}
$$
where  $V^{*}\Lambda^{*}(V^{*})^{T}$  is the truncated SVD of  $M^{*}$  and  $e_i$  is the  $i$ -th standard basis of  $\mathbb{R}^n$ .
Intuitively, the incoherence of  $M^{*}$  measures the sparsity of the low-rank ground truth matrix. If the incoherence is large, the matrix  $M^{*}$  is highly sparse and it is necessary to measure considerably many entries of  $M^{*}$  to observe nonzero entries. On the other hand, a relatively small incoherence of  $M^{*}$  is able to avoid the extreme case. Except the limited literature on the deterministic matrix completion problem [3, 25, 30], the majority of the matrix completion literature considered the following Bernoulli sampling model:
Definition 3 (Bernoulli Sampling Model). Given a sampling rate  $p \in (0,1]$ , each index  $(i,j) \in [n] \times [n]$  belongs to the set  $\Omega$  independently with probability  $p$ .
Under the Bernoulli sampling model, the objective function of problem (6) is well-behaved over the set of matrices with a small incoherence. Therefore, a regularizer that penalizes the incoherence of  $UU^T$  is included and we instead solve the following regularized matrix completion problem:
$$
\min  _ {U \in \mathbb {R} ^ {n \times r}} \frac {1}{p} \left\| \left(U U ^ {T}\right) _ {\Omega} - M _ {\Omega} ^ {*} \right\| _ {F} ^ {2} + \lambda \sum_ {i \in [ n ]} \left(\left\| e _ {i} ^ {T} U \right\| _ {F} - \alpha\right) _ {+} ^ {4}, \tag {8}
$$
where  $(x)_{+} := \max \{x, 0\}$  for all  $x \in \mathbb{R}$  and  $\alpha, \lambda > 0$  are constants. Intuitively, the coefficient  $1 / p$  is used to "normalize" the  $\ell_2$ -norm. We note that there exists an algorithm that can solve problem (6) without the incoherence regularizer [13]. However, the algorithm relies on the spectral initialization, which requires more computational effort than the factorization approach (2). It is proved in [18, 17] that if the sampling rate satisfies
$$
n p \geq \Theta \left[ \mu^ {4} r ^ {6} \left(\kappa^ {*}\right) ^ {6} \log n \right], \tag {9}
$$
the problem (8) satisfies the strict-saddle property with high probability, where  $\kappa^{*}$  is the condition number of  $M^{*}$ . On the other hand, it is proved in [11] that the information-theoretical lower bound  $np\geq \Theta (\mu r\log n)$  is necessary for the exact completion with high probability.
Figure 1: Example of the problem of matrix sensing over graphs. The vertex set is  $\mathcal{V} = [6]$  and the edge set is  $\mathcal{E} = \{(1,1),(1,2),(1,3),(1,4),(2,2),(3,4),(3,5),(4,6),(5,6),(6,6)\}$ .
1.3 Motivating example: Matrix sensing over graphs
Although the matrix sensing problem and the matrix completion problem are well-studied in literature, we show that there exist important applications of (2) that belong to a much broader class of problems than the special classes previously studied by the existing works. Hence, the current theoretical results cannot be directly applied to these applications and it remains unknown whether saddle-avoiding algorithms can find the ground truth matrix  $M^{*}$  in polynomial time.
Consider an undirected graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , where  $\mathcal{V}$  and  $\mathcal{E}$  are the set of vertices and the set of edges, respectively. For the notational simplicity, we assume that  $\mathcal{V} = [n]$ . Then, the edge set  $\mathcal{E}$  is a subset of  $[n]\times [n]$ . The goal of this problem is to recover the ground truth matrix  $M^{*}$  via measurements of its entries  $\{M_{ij}^{*}\mid (i,j)\in \mathcal{E}\}$ . Instead of directly observing each entry  $M_{ij}^{*}$ , for each node  $i\in \mathcal{V}$ , we observe a "mixture" (i.e., the output of a function) of the entries in the set
$$
\mathcal {E} _ {i} := \left\{\left(i, j\right) \mid j \text {i s i n c i d e n t t o} i \right\}.
$$
Denote the loss function for node  $i$  as  $f_{i}\left(M_{\mathcal{E}_{i}};M_{\mathcal{E}_{i}}^{*}\right)$ . The total loss function is the sum of the loss function for all nodes; namely, we consider the problem
$$
\min  _ {U \in \mathbb {R} ^ {n \times r}} \sum_ {i \in [ n ]} f _ {i} \left[ (U U ^ {T}) _ {\varepsilon_ {i}}; M _ {\varepsilon_ {i}} ^ {*} \right]. \tag {10}
$$
Since  $\mathcal{E} = \cup_{i\in [n]}\mathcal{E}_i$  , the above problem is a special case of the general problem
$$
\min  _ {U \in \mathbb {R} ^ {n \times r}} f \left[ (U U ^ {T}) _ {\mathcal {E}}; M _ {\mathcal {E}} ^ {*} \right], \tag {11}
$$
where we define  $f\left[(UU^T)_{\mathcal{E}};M_{\mathcal{E}}^*\right] \coloneqq \sum_{i\in [n]}f_i\left[(UU^T)_{\mathcal{E}_i};M_{\mathcal{E}_i}^*\right]$ . We name the problem (11) as the matrix sensing over graphs (MSoG) problem. The MSoG problem has a number of applications, including the state estimation problem in power systems [39, 24]. We note that in those applications, the loss function  $f_{i}$  is a quadratic function of entries of  $M_{\mathcal{E}_i}$  and  $M_{\mathcal{E}_i}^*$ ; see the example in the end of this section and the graph-structured quadratic sensing problem in [24].
To better understand (10) as a special type of MSoG, we consider a toy example where the undirected graph  $\mathcal{G}$  has  $n = 6$  vertices and is plotted in Figure 1. In this example, the objective function of the MSoG problem is
$$
\begin{array}{l} f \left(M _ {\mathcal {E}}; M _ {\mathcal {E}} ^ {*}\right) = f _ {1} \left(M _ {1 1}, M _ {1 2}, M _ {1 3}; M _ {1 1} ^ {*}, M _ {1 2} ^ {*}, M _ {1 3} ^ {*}\right) + f _ {2} \left(M _ {2 1}, M _ {2 2}; M _ {2 1} ^ {*}, M _ {2 2} ^ {*}\right) \\ + f _ {3} \left(M _ {3 1}, M _ {3 4}, M _ {3 5}; M _ {3 1} ^ {*}, M _ {3 4} ^ {*}, M _ {3 5} ^ {*}\right) + f _ {4} \left(M _ {4 1}, M _ {4 3}, M _ {4 6}; M _ {4 1} ^ {*}, M _ {4 3} ^ {*}, M _ {4 6} ^ {*}\right) \\ + f _ {5} \left(M _ {5 3}, M _ {5 6}; M _ {5 3} ^ {*}, M _ {5 6} ^ {*}\right) + f _ {6} \left(M _ {6 4}, M _ {6 5}, M _ {6 6}; M _ {6 4} ^ {*}, M _ {6 5} ^ {*}, M _ {6 6} ^ {*}\right), \\ \end{array}
$$
where we recall that we factorize  $M$  into  $UU^T$  in the Burer-Monteiro factorization approach. Hence, only 13 entries of the matrix  $M^{*}$  with  $n^2 = 36$  entries appear in the measurements. For example, the entry  $M_{16}^{*}$  does not appear in any measurements. If we directly observe these 13 entries, we can define the loss function  $f_{1}$  by the  $\ell_2$ -loss function:
$$
f _ {1} (M _ {1 1}, M _ {1 2}, M _ {1 3}; M _ {1 1} ^ {*}, M _ {1 2} ^ {*}, M _ {1 3} ^ {*}) = (M _ {1 1} - M _ {1 1} ^ {*}) ^ {2} + (M _ {1 2} - M _ {1 2} ^ {*}) ^ {2} + (M _ {1 3} - M _ {1 3} ^ {*}) ^ {2}.
$$
We can define other loss functions  $f_{2}, \ldots, f_{6}$  in a similar way. Then, the MSoG problem reduces to the matrix completion problem with  $\Omega = \mathcal{E}$ , namely, the objective function becomes
$$
f \left(M _ {\mathcal {E}}; M _ {\mathcal {E}} ^ {*}\right) = \left\| M _ {\mathcal {E}} - M _ {\mathcal {E}} ^ {*} \right\| _ {F} ^ {2}.
$$
Therefore, the matrix completion problem (6) is a special case of the MSoG problem (11) and the existing results for the matrix completion problem cannot be directly applied to problem (11). Similarly, if  $\Omega$  is a complete graph, the matrix sensing problem (10) becomes a special case of the MSoG problem. Moreover, since some entries of  $M^{*}$  do not appear in any of the measurements in general, it is easy to verify that the RIP condition (5) does not hold for problem (11). In summary, the existing results based on the RIP condition [38] and the incoherence condition [17, 13], as well as the results for other applications of the low-rank matrix optimization problem, cannot be directly applied to the MSoG problem.
Next, we show that the power state estimation problem [39, 24] can be formulated as the MSoG problem (11). This is the most important data-analysis problem for power systems and is solved every 5-15 minutes in practice using heuristic methods with little theoretical guarantees. Several major blackouts were attributed to the failure of solving this problem. An  $N$ -bus power system can be represented by an undirected graph  $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ , where  $\mathcal{V} = [N]$  and  $\mathcal{E} \subset \mathcal{V} \times \mathcal{V}$  are the set of buses and the set of lines, respectively. The state of the power system is characterized by the voltage phasors  $\mathbf{v} \in \mathbb{C}^N$ , where  $\mathbb{C}^N$  is the  $N$ -dimensional complex space. Our goal is to recover the voltage phasors via some measurements, without loss of generality, we assume that we measure the voltage magnitude at each bus and the active power flows over each line. Suppose that  $\mathbf{v} = \mathbf{x} + \mathbf{i}\mathbf{y}$  for  $\mathbf{x}, \mathbf{y} \in \mathbb{R}^N$ , where  $\mathbf{i}$  is the imaginary unit. Define
$$
\mathbf {z} := \left[ \begin{array}{c} \mathbf {x} \\ \mathbf {y} \end{array} \right] \in \mathbb {R} ^ {2 N}.
$$
The following two types of measurements are available:
1. For each bus  $k \in \mathcal{V}$ , the voltage magnitude is defined as  $|v_k|^2$ , where the magnitude for a complex number  $v_k = x_k + \mathrm{i}y_k$  is defined as  $|v_k|^2 \coloneqq x_k^2 + y_k^2$ . The voltage magnitude can be written in the matrix form as
$$
\left| v _ {k} \right| ^ {2} = \mathbf {z} ^ {T} \left(\mathbf {e} _ {k} \mathbf {e} _ {k} ^ {T} + \mathbf {e} _ {k + N} \mathbf {e} _ {k + N} ^ {T}\right) \mathbf {z},
$$
where  $\mathbf{e}_k$  is the  $k$ -th standard basis of  $\mathbb{R}^N$ .
2. For each line  $(k,\ell)\in \mathcal{E}$ , let  $Y_{k\ell}\in \mathbb{C}$  be the admittance of the line. The active power flow from bus  $k$  to bus  $\ell$  is  $p_{k\ell} = \operatorname {Re}(v_k(v_k - v_\ell)^* Y_{k\ell}^*)$ , where  $z^{*}$  is the conjugate of complex number  $z$ . We observe the active power flows  $p_{k\ell}$  and  $p_{\ell k}$  for all lines  $(k,\ell)\in \mathcal{E}$ . Suppose that  $Y_{k\ell} = P_{k\ell} + \mathbf{i}Q_{k\ell}$  for  $P_{k\ell},Q_{k\ell}\in \mathbb{R}$  for all  $(k,\ell)\in \mathcal{E}$ . Then, the active power flow can be written in the matrix form as
$$
p _ {k \ell} = \mathbf {z} ^ {T} \left[ P _ {k \ell} (\mathbf {e} _ {k} \mathbf {e} _ {k} ^ {T} - \mathbf {e} _ {k} \mathbf {e} _ {\ell} ^ {T} + \mathbf {e} _ {k + N} \mathbf {e} _ {k + N} ^ {T} - \mathbf {e} _ {N + k} \mathbf {e} _ {N + k} ^ {T}) + Q _ {k \ell} (\mathbf {e} _ {k} \mathbf {e} _ {N + \ell} ^ {T} - \mathbf {e} _ {\ell} \mathbf {e} _ {N + k} ^ {T}) \right] \mathbf {z}.
$$
Therefore, these observations can be written as the quadratic form  $\mathbf{z}^T M_j\mathbf{z}$  for some matrices  $M_1,\ldots ,M_m\in \mathbb{R}^{2N\times 2N}$ , where  $m = 2N + |\mathcal{E}|$  is the number of observations. For each sensing matrix  $M_{j}$ , only the diagonal entries and  $(k,\ell)$ ,  $(N + k,\ell)$ ,  $(k,N + \ell)$ ,  $(N + k,N + \ell)$  entries for  $(k,\ell)\in \mathcal{E}$  may be nonzero. In addition, we fix  $y_{1} = z_{N + 1} = 0$  since a unitary transformation on  $\mathbf{v}$  will not change the loss function. As a result, the problem can be written as a  $(2N - 1)$ -dimensional rank-1 MSoG problem:
$$
\min  _ {\mathbf {w} \in \mathbb {R} ^ {2 N}} \sum_ {j \in [ m ]} \left\langle M _ {j}, \mathbf {w w} ^ {T} - \mathbf {z z} ^ {T} \right\rangle^ {2}, \quad \text {s . t .} w _ {N + 1} = 0.
$$
1.4 Problem formulation and contributions
In this work, we propose the first sufficient condition that guarantees the benign landscape of the MSoG problem. More specifically, we consider the case when the graph  $\mathcal{G}$  is a random graph obeying the Erdős–Rényi model. Namely, each pair  $(i,j) \in \mathcal{V} \times \mathcal{V}$  belongs to the edge set independently with probability  $p$ . Under this random graph model and the assumption that the vertex set  $\mathcal{V}$  is  $[n]$ , each entry  $M_{ij}^{*}$  is indirectly observed (i.e., is involved in some measurements) independently with probability  $p$ . For comparison with the existing results of the matrix completion problem (6), we denote the edge set, which is equivalent to the set of indices of observed entries, as  $\Omega \subset [n] \times [n]$ . Then, the set  $\Omega$  follows the Bernoulli sampling model (Definition 3). Similar to the matrix completion problem, we aim at recovering the ground truth matrix  $M^{*}$  by solving the following problem with an
improved regularizer:
$$
\min  _ {U \in \mathbb {R} ^ {n \times r}} \frac {1}{p} f \left[ (U U ^ {T}) _ {\Omega}; M _ {\Omega} ^ {*} \right] + \sum_ {i \in [ n ]} r \left(\left\| e _ {i} ^ {T} U \right\| _ {F}\right), \tag {12}
$$
where we define the regularizer
$$
r (x) := \lambda \int_ {- 1} ^ {1} \left[ (x + \alpha y - 1 0 \alpha) _ {+} + 9 \alpha \right] ^ {4} (1 - | y |) d y, \quad \forall x \in \mathbb {R}.
$$
Note that in problem (12), we use a novel incoherence regularizer that is different from those in the prior literature. The regularizer is the convolution between  $\left[(x + \alpha y - 10\alpha)_{+} + 9\alpha \right]^{4}$  and the probability density function  $1 - |y|$  on  $[-1,1]$ , which is twice continuously differentiable in  $x$  (the set of discontinuous points of the derivatives is a zero measure set). Hence, the regularizer  $r\left(\| e_i^T U\| _F\right)$  is twice continuously differentiable in  $U$ . In addition, we can exchange the convolution and the differentiation with respect to  $x$ . Since the integrand is a quartic polynomial, the objective value and the derivatives of the regularizer can be exactly evaluated by numerical integration schemes with  $O(1)$  computations. Let  $\mathcal{R}\subset \mathbb{R}^{n\times r}$  be the set of  $U$  such that  $\| \mathbf{e}_i^T U\| _F = O(\alpha)$  for all  $i\in [n]$ . The intuition behind the design of the incoherence regularizer is to ensure that every approximate first-order critical point of problem (11) must lie in set  $\mathcal{R}$ . More specifically, the regularizer ensures that (i) outside  $\mathcal{R}$ , the gradient of the regularizer is large enough such that any approximate first-order critical point must lie in  $\mathcal{R}$ , (ii) the regularizer and its Hessian matrix have a sufficiently small contribution to the objective function in  $\mathcal{R}$ .
Besides the randomness model of  $\Omega$ , we make the following three assumptions about problem (12).