SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Proving that a point on the GD trajectory  $\mathbf{w}_t$  admits a certain breakdown in the form  $\mathbf{w}_t = \langle \mathbf{Z}_t, \mathbf{w}_0 \rangle - \mathbf{E}_t$  for some  $\mathbf{Z}_t$  and  $\mathbf{E}_t$ .
2. Proving that the spectral norm (equivalence of largest singular value) of  $\mathbf{E}_t$  is small (scales with initialization scale  $\epsilon$ )
3. Proving that  $\langle \mathbf{Z}_t, \mathbf{w}_0 \rangle$  has a large separation between its largest and second largest eigenvalues using a tensor version of Weyl's inequality.
4. Showing that, with the above holding true,  $\mathbf{w}_t$  is predominantly rank-1 after some step  $t_*$ .
Lemmas 12, 13, 2 and Theorem correspond to the above four steps, respectively. The reader is referred to the lemmas and theorem for more details.
4.1 A Primer on Tensor Algebra and Maintaining Symmetric Property
We start with the spectral norm of tensors, which resembles the operator norm of matrices [38].
Definition 3. Given a cubic tensor  $\mathbf{w} \in \mathbb{R}^{n\mathcal{O}l}$ , its spectral norm  $\| \cdot \|_S$  is defined respectively as:
$$
\left\| \mathbf {w} \right\| _ {S} = \sup  \left\{\left| \langle \mathbf {w}, u ^ {\otimes l} \rangle \right| \left\| u \right\| _ {2} = 1, u \in \mathbb {R} ^ {n} \right\}
$$
There are many definitions for tensor eigenvalues [39], and in this paper we introduce a novel variational characterization of eigenvalues that resembles the Courant-Fisher minimax definition for eigenvalues of matrices, called the v-Eigenvalue. We denote the  $i^{th}$  v-Eigenvalue of  $\mathbf{w}$  as  $\lambda_i^v (\mathbf{w})$ . Note this is a new definition that is first introduced in this paper and might be of independent interest outside of the current scope.
Definition 4 (Variational Eigenvalue of Tensors). For a given tensor  $\mathbf{w} \in \mathbb{R}^{n\mathcal{o}l}$ , we define its  $k^{th}$  variational eigenvalue (v-Eigenvalue)  $\lambda_k^v (\mathbf{w})$  as
$$
\lambda_{k}^{v}(\mathbf{w}):= \max_{\substack{S\\ \dim (S) = k}}\min_{\mathbf{u}\in S}\frac{|\langle\mathbf{w},\mathbf{u}\rangle|}{\|\mathbf{u}\|_{F}^{2}},\quad k\in [n]
$$
where  $S$  is a subspace of  $\mathbb{R}^{n\bullet l}$  that is spanned by a set of orthogonal, symmetric, rank-1 tensors. Its dimension denotes the number of orthogonal tensors that span this space. It is apparent from the definition that  $\| \mathbf{w}\| _S = \lambda_1^v (\mathbf{w})$ .
Next, since most of our analysis relies on the symmetry of the underlying tensor, it is desirable to show that every tensor along the optimization trajectory of GD on  $[5]$  remains symmetric if started from a symmetric tensor. Please find its proof in Appendix C.2
Lemma 1. If the  $GD$  trajectory of (5)  $\{\mathbf{w}_t\}_{t=0}^{\infty}$  is initialized at a symmetric rank-1 tensor  $\mathbf{w}_0$ , then  $\{\mathbf{w}_t\}_{t=0}^{\infty}$  will all be symmetric.
4.2 Main Ideas and Proof Sketch
In this subsection, we highlight the main ideas behind implicit bias in GD. Lemma [12] and [13] details the first and second step, and are deferred to Appendix C.2. The proofs to the results of this section can also be found in that appendix. The lemmas alongside with their proofs are highly technical and not particularly enlightening, therefore omitted here for simplicity. However, the most important takeaway is that for the  $t^{th}$  iterate along the GD trajectory of (5), we have the decomposition
$$
\mathbf {w} _ {t + 1} = \left\langle \mathbf {Z} _ {t}, \mathbf {w} _ {0} \right\rangle - \mathbf {E} _ {t} := \tilde {\mathbf {w}} _ {t} - \mathbf {E} _ {t}
$$
for some  $\mathbf{Z}_t$  and  $\mathbf{E}_t$  such that  $\| \mathbf{E}_t\| _S = \mathcal{O}(\epsilon^3)$ . This essentially means that by scaling the initialization  $\mathbf{w}_0$  to be small in scale, the error term  $\mathbf{E}_t$  can be ignored from a spectral standpoint, and scales with  $\epsilon$  at a cubic rate. This will soon be proven to be useful next.
Lemma 2. Given  $\mathbf{w}_t$  along the GD trajectory of (5), its first two  $v$ -eigenvalues, as defined in definition 4 satisfy the relation
$$
\frac {\lambda_ {2} ^ {v} (\mathbf {w} _ {t})}{\lambda_ {1} ^ {v} (\mathbf {w} _ {t})} \leq \frac {\| x _ {0} \| _ {2} ^ {l} (1 + \eta \sigma_ {2} ^ {l} (U)) ^ {t} + \| \mathbf {E} _ {t} \| _ {S} / \epsilon}{\left| v _ {1} ^ {\top} x _ {0} \right| ^ {l} (1 + \eta \sigma_ {1} ^ {l} (U)) ^ {t} - \| \mathbf {E} _ {t} \| _ {S} / \epsilon} = \frac {\| x _ {0} \| _ {2} ^ {l} (1 + \eta \sigma_ {2} ^ {l} (U)) ^ {t} + \mathcal {O} (\epsilon^ {2})}{\left| v _ {1} ^ {\top} x _ {0} \right| ^ {l} (1 + \eta \sigma_ {1} ^ {l} (U)) ^ {t} - \mathcal {O} (\epsilon^ {2})} \tag {6}
$$
where  $\sigma_1(U)$  and  $\sigma_2(U)$  denote the first and second singular values of  $U = \langle \mathbf{A}_r^*\mathbf{A},M^*\rangle \in \mathbb{R}^{nr\times nr}$ , and  $v_{1},v_{2}$  are the associated singular vectors.
Lemma 2 showcases that when  $\epsilon$  is small, the ratio between the largest and second largest v-eigenvalues of  $\mathbf{w}$  is dominated by  $(\| x_0\| _2^l (1 + \eta \sigma_2^l (U))^t) / (|v_1^\top x_0|^l (1 + \eta \sigma_1^l (U))^t)$ .
Now, if either  $\| x_0\| _2^l$  is large or  $|v_{1}^{\top}x_{0}|^{l}$  approaches 0 in value, then the ratio may be relatively large, contradicting our claim. However, this issue can be easily addressed by letting  $x_0 = v_1 + g\in \mathbb{R}^{nr}$ , where  $g$  is a vector with each entry being i.i.d sampled from the Gaussian distribution  $\mathcal{N}(0,\rho)$ . Note that since  $U = \langle \mathbf{A}_r,b\rangle_3$ , we can calculate  $U$  and  $v_{1}$  directly. Lemma [14] in Appendix [C.2] shows that with this initialization,  $|v_{1}^{\top}x_{0}|^{l} = \mathcal{O}(1)$  and  $\| x_0\| _2^l = \mathcal{O}(1)$  with high probability if we select  $\rho = \mathcal{O}(1 / nr)$ . Therefore, the  $t^{th}$  iterate along the GD trajectory of (5) satisfies
$$
\frac {\lambda_ {2} ^ {v} \left(\mathbf {w} _ {t}\right)}{\lambda_ {1} ^ {v} \left(\mathbf {w} _ {t}\right)} \asymp \frac {\left(1 + \eta \sigma_ {2} ^ {l} (U)\right) ^ {t}}{\left(1 + \eta \sigma_ {1} ^ {l} (U)\right) ^ {t}} \tag {7}
$$
with hight probability if  $\rho$  is small. This implies that "the level of parametrization helps with separation of eigenvalues", since increasing  $l$  will decrease ratio  $\lambda_2^v (\mathbf{w}_t) / \lambda_1^v (\mathbf{w}_t)$ . Furthermore, regardless of the value of  $\sigma_{1}(U)$ , a larger  $t$  will make this ratio exponentially smaller, proving the efficacy of algorithmic regularization of GD in tensor space.
By combining the above facts, we arrive at a major result showing how a small initialization could make the points along the GD trajectory penalize towards rank-1 as  $t$  increases
Theorem 1. Given the optimization problem (5) and its  $GD$  trajectory over some finite horizon  $T$ , i.e.,  $\{\mathbf{w}_t\}_{t=0}^T$  with  $\mathbf{w}_{t+1} = \mathbf{w}_t - \eta \nabla h^l(\mathbf{w}_t)$ , where  $\eta$  is the stepsize, then there exist  $t(\kappa, l) \geq 1$  and  $\kappa < 1$  such that
$$
\frac {\lambda_ {2} ^ {v} \left(\mathbf {w} _ {t}\right)}{\lambda_ {1} ^ {v} \left(\mathbf {w} _ {t}\right)} \leq \kappa , \quad \forall t \in [ t (\kappa , l), t _ {T} ] \tag {8}
$$
if  $\mathbf{w}_0$  is initialized as  $\mathbf{w}_0 = \epsilon x_0^{\otimes l}$  with a sufficiently small  $\epsilon$ , where  $t(\kappa, l)$  is expressed as
$$
t (\kappa , l) = \left[ \ln \left(\frac {\| x _ {0} \| _ {2} ^ {l}}{\kappa \left| v _ {1} ^ {\top} x _ {0} \right| ^ {l}}\right) \ln \left(\frac {1 + \eta \sigma_ {1} ^ {l} (U)}{1 + \eta \sigma_ {2} ^ {l} (U)}\right) ^ {- 1} \right] \tag {9}
$$
By using the initialization introduced in Lemma [14], we can improve the result of Theorem [1], which does not need  $\epsilon$  to be arbitrarily small. The full details are presented in Corollary [1] in Appendix C.2, stating that as along as  $t \asymp \ln(1/\kappa) \ln\left((1 + \eta \sigma_1^l(U)) / (1 + \eta \sigma_2^l(U))\right)^{-1}$ ,  $\mathbf{w}_t$  will be  $\kappa$ -rank-1, as long as  $\epsilon$  is chosen as a function of  $U, r, n, L_s$ , and  $\kappa$ . Note that we say a tensor  $\mathbf{w}$  is "  $\kappa$ -rank-1" if  $\lambda_2^v(\mathbf{w}) / \lambda_1^v(\mathbf{w}) \leq \kappa$ .
5 Approximate Rank-1 Tensors are Benign
Now that we have established the fact that performing gradient descent on (5) will penalize the tensor towards rank-1, it begs the question whether approximate rank-1 tensors can also escape from saddle points, which is the most important question under study in this paper. Please find the proofs to the results in this section in Appendix D.
To do so, we first introduce a major spectral decomposition of symmetric tensors that is helpful.
Proposition 1. Given a symmetric tensor  $\mathbf{w} \in \mathbb{R}^{nrol}$ , it can be decomposed into two terms, namely a term consisting of its dominant component and another term that is orthogonal to this direction:
$$
\mathbf {w} = \pm \lambda_ {1} ^ {v} (\mathbf {w}) w _ {s} ^ {\otimes l} + \mathbf {w} ^ {\dagger} := \mathbf {w} _ {\sigma} + \mathbf {w} ^ {\dagger}, \quad w _ {s} \in \mathbb {R} ^ {n}, \| w _ {s} \| _ {2} = 1 \tag {10}
$$
where  $\langle \mathbf{w}, w_s^{\otimes l} \rangle = \lambda_1^v(\mathbf{w})$  and  $\langle \mathbf{w}^\dagger, w_s^{\otimes l} \rangle = 0$ . Furthermore, if  $\mathbf{w}$  is a  $\kappa$ -rank-1 tensor, then  $\|\mathbf{w}^\dagger\|_S \leq \kappa \lambda_1^v(\mathbf{w}_t)$ .
Next, we characterize the first-order points of (5) with approximate rank-1 tensors in mind. Previously, we showed that if a given FOP of (5) is symmetric and rank-1, it has a one-to-one correspondence with FOPs of (2). However, if the FOPs of (5) are not exactly rank-1, but instead  $\kappa$ -rank-1, it is essential to understand whether they maintain the previous properties. This will be addressed below.
Proposition 2. Assume that a symmetric tensor  $\mathbf{w} \in \mathbb{R}^{nrol}$  is an FOP of (5), meaning that (17a) holds. If it is a  $\kappa$ -rank-1 tensor with  $\kappa \leq \mathcal{O}(1 / \| M^{*}\|_{F}^{2})$ , then it admits a decomposition as
$$
\mathbf {w} = \pm \lambda_ {1} ^ {v} (\mathbf {w}) \hat {w} ^ {\otimes l} + \mathbf {w} ^ {\dagger}
$$
with  $\mathrm{mat}(\hat{w})\in \mathbb{R}^{n\times r}$  being an FOP of (2) and  $\| \mathbf{w}^{\dagger}\|_{S}\leq \kappa \lambda_{1}^{v}(\mathbf{w})$  by definition.
The proposition above asserts that for any given FOP of (5), if it is  $\kappa$ -rank-1 rather than being truly rank-1, it will consist of a rank-1 term representing a lifted version of an unlifted FOP, as well as a term with a small spectral norm. Referring to (58), it is possible to achieve a significantly low  $\kappa$  through a moderate number of iterations. This result, considered the cornerstone of this paper, demonstrates that the use of gradient descent with small initialization will find critical points that are lifted FOPs of (2) with added noise, maintaining a robust association between FOPs of (5) and (2). This finding also facilitates this subsequent theorem:
Theorem 2. Assume that a symmetric tensor  $\hat{\mathbf{w}}\in \mathbb{R}^{nrol}$  is an FOP of (5) that is  $\kappa$  -rank-1 with  $\kappa \leq \mathcal{O}(1 / \| M^{*}\|_{F}^{2})$ . Consider its major spectral decomposition  $\hat{\mathbf{w}} = \lambda_S\hat{x}^{\otimes l} + \hat{\mathbf{w}}^\dagger$  with  $\hat{x}\in \mathbb{R}^{nr}$ , then it has a rank-1 escape direction if  $\hat{X} = \mathrm{mat}(\hat{x})$  satisfies the inequality
$$
\left\| M ^ {*} - \hat {X} \hat {X} ^ {\top} \right\| _ {F} ^ {2} \geq \frac {L _ {s}}{\alpha_ {s}} \lambda_ {r} \left(\hat {X} \hat {X} ^ {\top}\right) \operatorname {t r} \left(M ^ {*}\right) + \mathcal {O} \left(r \kappa^ {1 / l}\right) \tag {11}
$$
where  $l$  is odd and large enough so that  $l > 1 / (1 - \log_2(2\beta))$  and  $\beta$  is defined as
$$
\beta = \frac {L _ {s} \operatorname {t r} (M ^ {*}) \lambda_ {r} (\hat {X} \hat {X} ^ {\top})}{\alpha_ {s} \| M ^ {*} - \hat {X} \hat {X} ^ {\top} \| _ {F} ^ {2} - \mathcal {O} (r \kappa^ {1 / l})}.
$$
This theorem conveys the message that by running GD on (5), all critical points have escape directions as long as the point is not close to the ground truth solution. In Appendix B, we present Theorem 7 to provide sufficient conditions for the conversion to hold globally when (11) is hard to hold.
In this section, after we run a given algorithm on (5) to completion and obtain a final tensor  $\mathbf{w}_T$ , we then apply tensor PCA (detailed in Appendix F) on  $\mathbf{w}_T$  to extract its dominant rank-1 component and recover  $X_T \in \mathbb{R}^{n \times r}$  such that  $(\mathbf{w}_T)_s = \lambda_s \operatorname{vec}(X_T)^{\otimes l}$ . Since  $\mathbf{w}_T$  will be approximately rank-1, the success of this operation is expected [23, 24]. We consider a trial to be successful if the recovered  $X_T$  satisfies  $\| X_T X_T^\top - M^*\|_F \leq 0.05$ . We also initialize our algorithm as per Lemma 14.
6.1 Perturbed Matrix Completion
The perturbed matrix completion problem is introduced in [20], which is a noisy version of classic matrix completion problems. The  $\mathcal{A}$  operator is introduced as
$$
\mathcal {A} _ {\rho} (\mathbf {M}) _ {i j} := \left\{ \begin{array}{l l} \mathbf {M} _ {i j}, & \text {i f} (i, j) \in \Omega \\ \rho \mathbf {M} _ {i j}, & \text {o t h e r w i s e} \end{array} , \right. \tag {12}
$$
where  $\Omega$  is a measurement set such that  $\Omega = \{(i,i),(i,2k),(2k,i)|\forall i\in [n],k\in [\lfloor n / 2\rfloor ]\}$ . [20] has proved that each such instance has  $\mathcal{O}(2^{\lceil n / 2\rceil} - 2)$  spurious local minima, while it satisfies the RIP property with  $\delta_{2r} = (1 - \rho) / (1 + \rho)$  for some sufficiently small  $\rho$ . This implies that common first-order methods fail with high probability for this class of problems. In our experiment, we apply both lifted and unlifted formulations to [12] with  $\rho = 0.01$ , yielding  $\delta_{2r}\approx 1$ . We test different values of  $n$  and  $\epsilon$ , using a lifted level of  $l = 3$ . We ran 10 trials each to calculate success rate. If unspecified in the plot, we default  $n = 10$ ,  $\epsilon = 10^{-7}$ . Figure 1 reveals a higher success rate for the lifted formulation across different problem sizes, with smaller problems performing better as expected (since larger problems require a higher lifting level). Success rates improve with smaller  $\epsilon$ , emphasizing the importance of small initialization. We employed customGD, a modified gradient descent algorithm with heuristic saddle escaping. This algorithm will deterministically escape from critical points utilizing knowledge from the proof of Theorem 4. For details please refer to Appendix F. Furthermore, to showcase the implicit penalization affects of GD, we obtained approximate measures for  $\lambda_2^v (\mathbf{w}_t) / \lambda_1^v (\mathbf{w}_t)$  (since exactly solving for them is NP-hard) along the trajectory, and presented the results and methods in Appendix E.
Additionally, we examine different algorithms for (5), including customGD, vanilla GD, perturbed GD ([40], for its ability to escape saddles), and ADAM [41]. Figure 2 suggest that ADAM is an effective optimizer with a high success rate and rapid convergence, indicating that momentum acceleration may not hinder implicit regularization and warrants further research. Perturbed GD performed poorly, possibly due to random noise disrupting rank-1 penalization.
6.2 Shallow Neural Network Training with Quadratic Activation
It has long been known that the matrix sensing problem (2) includes the training of two-layer neural networks (NN) with quadratic activation as a special case [12]. In summary, the output of the neural
$^{2}$ https://github.com/anonpapersbm/implicit.bias_tensor',run on 2021 Macbook Pro
Figure 1: Success rate of the lifted formulation versus the unlifted formulation against varying  $n$  and  $\epsilon$ . The algorithm of choice is CustomGD (details in Appendix F).
Figure 2: Performance of different algorithms applied to the lifted formulation (5).
network  $y \in \mathbb{R}^m$  with respect to  $m$  inputs  $\{d_i\}_{i=1}^m \in \mathbb{R}^n$  can be expressed as  $y_i = 1^\top q(X^\top d_i)$ , which implies  $y_i = \langle d_i d_i^\top, XX^\top \rangle$ , where  $q(\cdot)$  is the element-wise quadratic function and  $X \in \mathbb{R}^{n \times r}$  in (2) represents the weights of the neural network. Thus  $r$  represents the number of hidden neurons. In our experiment, we demonstrate that when  $m$  is small, the lifted framework (5) outperforms standard neural network training in success rate, yielding improved recovery of the true weights. We set the hidden neurons number to be  $n$  for the standard network training, thereby comparing the existing over-parametrization framework (recall Section 1.1, with  $r_{\mathrm{search}} = n$ ) with the lifted one. We employ the ADAM optimizer for both methods. Table 2 showcases the success rate under various problem and sample sizes. Sampling both data and true weights  $Z \in \mathbb{R}^{n \times r}$  from an i.i.d Gaussian distribution, we calculate the observations  $y$  and attempt to recover  $Z$  using both approaches. As the number of samples increases, so does the success rate, with the lifted approach offering significantly better accuracy overall, even when the standard training has a  $0\%$  success rate.
Success Rate	m = 20	m=30	m=40
n=8	0.9(0)	1(0.3)	0.9(0.5)
n=10	0.2(0)	0.6(0)	0.8(0)
n=12	0.1(0)	0.4(0)	0.8(0)
(a) Ground truth weight with  $r = 1$
Success Rate	m = 30	m=40	m=50
n=8	0.3(0)	0.3(0)	0.8(0)
n=10	0.3(0)	0.4(0)	0.2(0)
n=12	0(0)	0(0)	0.2(0)
(b) Ground truth weight with  $r = 2$
Table 2: Success rate of NN training using (5) and original formulation. The number inside the parentheses denotes the success rate of the original formulations.  $\epsilon = 10^{-5}$  and  $l = 3$ .
Our study highlights the pivotal role of gradient descent in inducing implicit regularization within tensor optimization, specifically in the context of the lifted matrix sensing framework. We reveal that GD can lead to approximate rank-1 tensors and critical points with escape directions when initialized at an adequately small scale. This work also contributes to the usage of tensors in machine learning models, as we introduce novel concepts and techniques to cope with the intrinsic complexities of tensors.
8 Acknowledgement
This work was supported by grants from ARO, ONR, AFOSR, NSF, and the UC Noyce Initiative.
[1] D. L. Donoho, "Compressed sensing," IEEE Transactions on information theory, vol. 52, no. 4, pp. 1289-1306, 2006.
[2] E. J. Candès and B. Recht, "Exact matrix completion via convex optimization," Foundations of Computational Mathematics, vol. 9, no. 6, pp. 717-772, 2009.
[3] E. J. Candès and T. Tao, "The power of convex relaxation: Near-optimal matrix completion," IEEE Transactions on Information Theory, vol. 56, no. 5, pp. 2053-2080, 2010.
[4] L. T. Nguyen, J. Kim, and B. Shim, “Low-rank matrix completion: A contemporary survey,” IEEE Access, vol. 7, pp. 94215–94237, 2019.
[5] A. Singer, "Angular synchronization by eigenvectors and semidefinite programming," Applied and Computational Harmonic Analysis, vol. 30, no. 1, pp. 20-36, 2011.
[6] N. Boumal, “Nonconvex phase synchronization,” SIAM Journal on Optimization, vol. 26, no. 4, pp. 2355–2377, 2016.
[7] Y. Shechtman, Y. C. Eldar, O. Cohen, H. N. Chapman, J. Miao, and M. Segev, "Phase retrieval with application to optical imaging: A contemporary overview," IEEE Signal Processing Magazine, vol. 32, no. 3, pp. 87-109, 2015.
[8] S. Fattahi and S. Sojoudi, "Exact guarantees on the absence of spurious local minima for nonnegative rank-1 robust principal component analysis," Journal of Machine Learning Research, vol. 21, pp. 1-51, 2020.
[9] Y. Zhang, R. Madani, and J. Lavaei, "Conic relaxations for power system state estimation with line measurements," IEEE Transactions on Control of Network Systems, vol. 5, no. 3, pp. 1193-1205, 2017.
[10] M. Jin, I. Molybog, R. Mohammadi-Ghazi, and J. Lavaei, "Towards robust and scalable power system state estimation," in 2019 IEEE 58th Conference on Decision and Control (CDC), pp. 3245-3252, IEEE, 2019.
[11] I. Molybog, R. Madani, and J. Lavaei, "Conic optimization for quadratic regression under sparse noise," The Journal of Machine Learning Research, vol. 21, no. 1, pp. 7994-8029, 2020.
[12] Y. Li, T. Ma, and H. Zhang, "Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations," in Conference On Learning Theory, pp. 2-47, PMLR, 2018.
[13] S. Burer and R. D. Monteiro, “A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization,” Mathematical Programming, vol. 95, no. 2, pp. 329–357, 2003.
[14] R. Y. Zhang, S. Sojoudi, and J. Lavaei, "Sharp restricted isometry bounds for the inexistence of spurious local minima in nonconvex matrix recovery," Journal of Machine Learning Research, vol. 20, no. 114, pp. 1-34, 2019.
[15] Z. Ma, Y. Bi, J. Lavaei, and S. Sojoudi, "Sharp restricted isometry property bounds for low-rank matrix recovery problems with corrupted measurements," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 7672-7681, 2022.
[16] W. Ha, H. Liu, and R. F. Barber, "An equivalence between critical points for rank constraints versus low-rank factorizations," SIAM Journal on Optimization, vol. 30, no. 4, pp. 2927-2955, 2020.
[36] R. Ge, Y. Ren, X. Wang, and M. Zhou, "Understanding deflation process in over-parametrized tensor decomposition," Advances in Neural Information Processing Systems, vol. 34, pp. 1299-1311, 2021.
[37] E. J. Candes and Y. Plan, "Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements," 2011.
[38] L. Qi, S. Hu, X. Zhang, and Y. Chen, "Tensor norm, cubic power and gelfand limit," arXiv preprint arXiv:1909.10942, 2019.
[39] L. Qi, “The spectral theory of tensors (rough version),” arXiv preprint arXiv:1201.3424, 2012.
[40] C. Jin, P. Netrapalli, R. Ge, S. M. Kakade, and M. I. Jordan, "On nonconvex optimization for machine learning: Gradients, stochasticity, and saddle points," Journal of the ACM (JACM), vol. 68, no. 2, pp. 1-29, 2021.
[41] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
[42] Q. Li, Z. Zhu, and G. Tang, “The non-convex geometry of low-rank matrix optimization,” Information and Inference: A Journal of the IMA, vol. 8, no. 1, pp. 51–96, 2019.
[43] T. G. Kolda, “Numerical optimization for symmetric tensor decomposition,” Mathematical Programming, vol. 151, no. 1, pp. 225–248, 2015.
[44] K. B. Petersen, M. S. Pedersen, et al., “The matrix cookbook,” Technical University of Denmark, vol. 7, no. 15, p. 510, 2008.
[45] Z. Ma and S. Sojoudi, "Noisy low-rank matrix optimization: Geometry of local minima and convergence rate," in International Conference on Artificial Intelligence and Statistics, pp. 3125-3150, PMLR, 2023.
[46] G. Zhang and R. Y. Zhang, "How many samples is a good initial point worth in low-rank matrix recovery?", in Advances in Neural Information Processing Systems, vol. 33, pp. 12583-12592, 2020.
[47] Y. Bi and J. Lavaei, “Global and local analyses of nonlinear low-rank matrix recovery problems,” 2020. arXiv:2010.04349.
[48] L.-H. Lim and P. Comon, "Blind multilinear identification," IEEE Transactions on Information Theory, vol. 60, no. 2, pp. 1260-1280, 2013.
[49] P. Comon, G. Golub, L.-H. Lim, and B. Mourrain, "Symmetric tensors and symmetric tensor rank," SIAM Journal on Matrix Analysis and Applications, vol. 30, no. 3, pp. 1254-1279, 2008.
[50] G. Ni, "Hermitian tensor and quantum mixed state," arXiv preprint arXiv:1902.02640, 2019.
[51] S. Y. Chang, “Hanson-wright inequality for random tensors under einstein product,” arXiv preprint arXiv:2111.12169, 2021.
[52] R. Vershynin, High-dimensional probability: An introduction with applications in data science, vol. 47. Cambridge university press, 2018.