Title: Rank-1 Matrix Completion with Gradient Descent and Small Random Initialization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Matrix completion

gradient descent

random initialization
Abstract: The nonconvex formulation of the matrix completion problem has received significant attention in recent years due to its affordable complexity compared to the convex formulation. Gradient Descent (GD) is a simple yet efficient baseline algorithm for solving nonconvex optimization problems. The success of GD has been witnessed in many different problems in both theory and practice when it is combined with random initialization. However, previous works on matrix completion require either careful initialization or regularizers to prove the convergence of GD. In this paper, we study the rank-1 symmetric matrix completion and prove that GD converges to the ground truth when small random initialization is used. We show that in a logarithmic number of iterations, the trajectory enters the region where local convergence occurs. We provide an upper bound on the initialization size that is sufficient to guarantee the convergence, and show that a larger initialization can be used as more samples are available. We observe that the implicit regularization effect of GD plays a critical role in the analysis, and for the entire trajectory, it prevents each entry from becoming much larger than the others.

Rank-1 Matrix Completion with Gradient Descent and Small Random Initialization

Abstract

The nonconvex formulation of the matrix completion problem has received significant attention in recent years due to its affordable complexity compared to the convex formulation. Gradient Descent (GD) is a simple yet efficient baseline algorithm for solving nonconvex optimization problems. The success of GD has been witnessed in many different problems in both theory and practice when it is combined with random initialization. However, previous works on matrix completion require either careful initialization or regularizers to prove the convergence of GD. In this paper, we study the rank-1 symmetric matrix completion and prove that GD converges to the ground truth when small random initialization is used. We show that in a logarithmic number of iterations, the trajectory enters the region where local convergence occurs. We provide an upper bound on the initialization size that is sufficient to guarantee the convergence, and show that a larger initialization can be used as more samples are available. We observe that the implicit regularization effect of GD plays a critical role in the analysis, and for the entire trajectory, it prevents each entry from becoming much larger than the others.

1 Introduction

Recovering a low-rank matrix from a set of linear measurements is at the heart of many statistical learning problems. Depending on the structure of the matrix and the linear measurements, it reduces to various problems such as phase retrieval [1], blind deconvolution [2], and matrix sensing [3]. Matrix completion [4] is also one such type of problem where each measurement provides an entry of the matrix, and the goal is to recover the low-rank matrix from a partial, usually very sparse, observation of the entries. One of the most notable applications of matrix completion is collaborative filtering [5], which aims to predict user preferences for items based on a highly incomplete observation of user-item ratings. There are also a number of different applications, such as principal component analysis [6] and image reconstruction [7], just to name a few.

Extensive amount of work has been dedicated to provide an efficient recovery algorithm for matrix completion with theoretical guarantees [8]. The convex relaxation based nuclear norm minimization [4, 9] was the first algorithm proven to recover the matrix with near optimal sample complexity. Despite its theoretical success, the convex algorithm was found hard to be used in practical scenarios due to its unaffordable computational complexity and memory size. Therefore, the nonconvex formulation of matrix completion with quadratic loss has received significant attention in recent years. Many different algorithms have been proposed for the nonconvex problem, and their convergence toward the ground truth has been analyzed. Examples include optimization on Grassmann manifolds [10], alternating minimization [11], projected gradient descent [12], gradient descent with regularizer [13], and (vanilla) gradient descent [14, 15].

Gradient descent (GD) has served as a baseline algorithm for solving nonconvex optimization problems. However, the convergence of GD to global minimizers is not guaranteed, and it can take exponential time to escape saddle points [16]. Nevertheless, GD with random initialization has been shown to successfully recover the global minimum in many different problems such as phase retrieval [1], matrix sensing [17], matrix factorization [18], and neural network training [19]. Previous work on matrix completion [14, 15] proved the convergence of GD under the spectral initialization, which locates the initial point in the local region of the minima. However, the role of random initialization in solving matrix completion with GD is not fully understood yet, although its success is observed in practice. Therefore, we aim to answer the following question:

Can GD with random initialization solve the nonconvex matrix completion problem?

We answer this question affirmatively and show that GD with small random initialization successfully converges to the ground truth for rank-1 symmetric matrix completion. In the analysis, we use vanilla GD, which does not incorporate any modifications, such as regularization or truncation, into the GD algorithm. We also characterize the entire trajectory that GD follows by showing that the trajectory is well approximated by the fully observed case. The small initialization plays a critical role in analyzing the trajectory of the early stages, where the randomly initialized vector is nearly orthogonal to the first eigenvector of the ground truth matrix. We provide a bound on the required initialization size for the algorithm to converge, and our bound suggests that one can use a larger initialization to improve the convergence speed as more samples are provided. However, in any case, GD with a small random initialization takes only logarithmic amount of time (with respect to the matrix dimension) to reach the point where local convergence can begin. To the best of our knowledge, this is the first result on matrix completion that proves the convergence of vanilla GD without a carefully designed initialization.

Although our result is restricted to the rank-1 case, we believe that this work provides an important evidence for understanding the more general rankr case. At the end of this paper, we will discuss some technical difficulties that the rankr case naturally has, and provide some empirical results related to them. However, studying the rank-1 matrix completion problem is not only motivated by theoretical interest, but the problem itself also appears in some practical problems such as crowdsourcing [20, 21].

Related Works This work is motivated by the recent success of small initialization in matrix factorization and matrix sensing. It was first conjectured in [22] that sufficiently small step sizes and initialization lead GD to converge to the minimum nuclear norm solution of a full-dimensional matrix sensing problem. The conjecture was proved in [17] for the fully overparameterized matrix sensing under the standard restricted isometry property (RIP). A recent study by [23] provided more general results by showing that the early iterations of GD with small initialization have spectral bias. Many other works such as [24, 25, 26] have also studied how GD or gradient flow with small initialization implicitly forces the recovered matrix to be low-rank. However, the recovery guarantee for matrix completion has not been provided by any work.

For the matrix sensing where RIP holds, the loss function has global benign geometry in that it does not contain any spurious local minima or non-strict saddle points [27]. In the case of matrix completion, a similar result was obtained but with a regularizer that penalizes the matrices with large rows [28]. Controlling the norm of each row (absolute value of each entry in the case of rank-1) is the biggest hurdle in the analysis of matrix completion. In the local convergence analysis of [14], it was proved that GD implicitly regularizes the largest ℓ 2 -norm of the rows of error matrices, showing that explicit regularization is unnecessary. In this paper, we also prove that such an implicit regularization is induced by GD when it starts from a point of small size. We show that the trajectory is close to the fully observed case in both ℓ 2 and ℓ ∞ norms. Thus, the trajectory is confined to the region where it has benign geometry, and GD can converge without an explicit regularizer.

Notations We denote vectors with lowercase bold letters and matrices with uppercase bold letters. The components or entries of them are written without bold. We use ∥·∥ 2 and ∥·∥ ∞ to denote ℓ 2 and ℓ ∞ -norm of vectors, respectively, and ∥·∥ F is used for Frobenius norm of matrices. For any norm ∥·∥ and two vectors x , y , we let ∥ x ± y ∥ = min {∥ x + y ∥ , ∥ x -y ∥} . Asymptotic dependencies with respect to the matrix dimension are denoted with the standard big O notations, or with the symbols, ≲ , ≍ and ≳ .

2 Problem Formulation

The matrix completion problem aims to reconstruct a low-rank matrix from partially observed entries. In this paper, we focus on the case where the ground truth matrix, denoted by M ⋆ ∈ R n × n , is a rank-1 positive semidefinite matrix. Thus, the ground truth matrix is decomposed as M ⋆ = λ ⋆ u ⋆ u ⋆ ⊤ with λ ⋆ > 0 and a unit vector u ⋆ . We define x ⋆ = √ λ ⋆ u ⋆ so that M ⋆ = x ⋆ x ⋆ ⊤ . To follow the standard incoherence assumption, we let ∥ u ⋆ ∥ ∞ = √ µ n and allow µ to be as large as poly(log n ) . Weconsider a random sampling model that is also symmetric as M ⋆ . Each entry in the diagonal and the upper (or lower) triangular part of M ⋆ is independently revealed with probability 0 < p ≤ 1 . We consider the noisy case where Gaussian noise is added to each observation. Formally, we get as an observation the matrix M ◦ whose ( i, j ) th entry is 1 p δ ij ( M ⋆ ij + E ij ) , where [ δ ij ] 1 ≤ i ≤ j ≤ n are independent Bernoulli random variables with expectation p and [ E ij ] 1 ≤ i ≤ j ≤ n are independent Gaussian random variables with the distribution N (0 , σ 2 ) . They are both symmetric in the sense that δ ij = δ ji and E ij = E ji for all 1 ≤ i ≤ j ≤ n . We use E to denote the symmetric matrix whose entries are E ij . We denote the set of observed entries as Ω := { ( i, j ) | δ ij = 1 } , and define an operator P Ω on matrices that sets the entries not contained in Ω to zero. (e.g. M ◦ = 1 p P Ω ( M ⋆ + E ) )

To recover the matrix M ⋆ , we find x ∈ R n that minimizes the nonconvex loss function f ( x ) , which is the sum of the squared differences on the observed entries. It is explicitly written as f ( x ) := 1 4 p ∑ ( i,j ) ∈ Ω ( x i x j -x ⋆ i x ⋆ j -E ij ) 2 . We apply vanilla GD to solve the optimization problem starting from a small randomly initialized vector x (0) . Each entry of x (0) is sampled independently from the Gaussian distribution N ( 0 , 1 n β 2 0 ) , so that the squared norm of x (0) is expected to be β 2 0 . The update rule of GD is written as

where η > 0 is the step size.

We define F as the loss function f when all entries of M ⋆ are observed without noise, i.e., F ( x ) := 1 4 ∥ ∥ xx ⊤ -M ⋆ ∥ ∥ 2 F . We also define ˜ x ( t ) as the trajectory of GD when it is applied to F with the same initial point x (0) , i.e., ˜ x ( t ) is the trajectory of the fully observed case. Specificially, it evolves with

from the same starting point ˜ x (0) = x (0) .

Lastly, we introduce the so-called leave-one-out sequences. These were the main ingredient in controlling the ℓ ∞ -norm of trajectory in [14]. We use them for a similar purpose. For each l ∈ [ n ] , we define an operator P ( l ) Ω such that P ( l ) Ω ( X ) is equal to X on the l th row and column, and equal to 1 p P Ω ( X ) otherwise. The l th leave-one-out sequence, x ( t,l ) , evolves with

for x (0 ,l ) = x (0) , where M ( l ) = P ( l ) Ω ( M ⋆ ) + E ( l ) , and E ( l ) is obtained by zeroing out the l th row and column of 1 p P Ω ( E ) .

3 Main Results

In this section, we present our main results. The first main result concerns the global convergence of GD with small random initialization.

Theorem 3.1. Let us consider a rank-1 matrix completion problem that recovers the matrix M ⋆ = x ⋆ x ⋆ ⊤ ∈ R n × n such that ∥ x ⋆ ∥ 2 = √ λ ⋆ and ∥ x ⋆ ∥ ∞ = √ µ n ∥ x ⋆ ∥ 2 , where µ = O (poly(log n )) . Let the initial point x (0) ∈ R n be sampled from the Gaussian distribution N ( 0 , 1 n β 2 0 I ) and x ( t ) be updated with (1) . Suppose that a small step size with ηλ ⋆ < 0 . 1 is used and the sample complexity satisfies n 2 p ≳ µ 5 n log 22 n . Then, there exists T ⋆ = (1 + o (1)) 1 ηλ ⋆ log √ λ ⋆ n β 0 such that

hold at t = T ⋆ with probability at least 1 -o (1 / √ log n ) , if a sufficiently small initialization with

is used and the noise satisfies σ ≲ λ ⋆ µ n √ log n .

Theorem 3.1 proves that, starting from a small random initialization, the trajectory of GD eventually enters the local region of the global minimizers ± x ⋆ in terms of both ℓ 2 and ℓ ∞ norms. Combined with the result of [14], GD starts to converge linearly to either x ⋆ or -x ⋆ after t = T ⋆ , as stated in the corollary below.

Corollary 3.2. Suppose that the conditions in Theorem 3.1 are satisfied, and let ρ be a constant such that 1 -η 10 ≤ ρ < 1 . Then, with probability at least 1 -o (1 / √ log n ) , we have

The desired global convergence result is provided by Corollary 3.2. Several remarks about Theorem 3.1 and Corollary 3.2 are in order.

Matrix Recovery Suppose x ( t ) converges to a global minimum y ⋆ of the function f , which is different from ± x ⋆ . In such a case, despite achieving global convergence, the reconstructed matrix y ⋆ y ⋆ ⊤ deviates from the ground truth matrix M ⋆ . However, Theorem 3.1 establishes that x ( t ) converges exclusively to the correct global minima ± x ⋆ , so that the matrix M ⋆ is recovered with high probability.

Leave-one-out Sequence To apply the local convergence result of [14], in addition to (4) and (5), the existence of leave-one-out sequences { x ( t,l ) } l ∈ [ n ] satisfying (6) and (7) is required. Leave-one-out sequences also play a critical role and appear naturally in the proof of Theorem 3.1.

Sample Complexity The required sample complexity for Theorem 3.1 to hold is optimal up to a logarithmic factor compared to the statistical lower bound of Ω( n log n ) . We have not done our best to optimize the log factors, and about half of them can be reduced with more delicate analysis. We will discuss this briefly in Section 6.

Convergence Time Considering that β -1 0 is at most polynomial in n (due to the lower bound of (8)), only O (log n ) iterations are required for GD to enter the local region. It takes O (log( 1 ϵ )) more iterations to achieve ϵ -accuracy in the local region, so the total iteration complexity is given by O (log n ) + O (log( 1 ϵ )) .

Initialization Size Although small initialization provides a good geometry to GD, a larger initialization is preferred because the convergence time, T ⋆ , is inversely proportional to β 0 . When the sample complexity is optimal, i.e., n 2 p ≍ n poly(log n ) , an upper bound on the initialization size given by Theorem 3.1 is n -1 4 , ignoring the log factors. However, as more samples are provided, we are allowed to use a larger initialization to reduce the convergence time. When the sample complexity satisfies n 2 p ≍ n 1+ a , the bound is n -1 4 (1 -a ) ignoring the log factors. The bound becomes nearly constant as a approaches 1 , namely the fully observed case, and this is consistent with the previous result that small initialization is unnecessary for the fully observed case [18]. We also note that the lower bound of (8) is necessary in the proof of Theorem 3.1, since we derive probabilistic bounds for all iterations, and the lower bound limits the maximum number of iterations. However, we can further reduce the lower bound n -10 to n -c for any constant c > 10 by tuning some constant factors during the proof.

Noise Size From the incoherence assumption, the maximum absolute value of entries of M ⋆ is bounded by λ ⋆ µ n . The condition σ ≲ λ ⋆ µ n √ log n in Theorem 3.1 allows the standard deviation of the Gaussian noise to be much larger than the maximum entry. It also implies σ λ ⋆ √ n p ≲ µ √ log n np , so that the upper bounds in Corollary 3.2 are dominated by the first terms at t = T ⋆ and they eventually converge to the second terms as t increases.

Estimation Error The current estimation bounds (4) to (7) are all proportional to 1 √ log n times the norms of x ⋆ . However, if we do not allow the initialization size to grow with the sample complexity, we are able to obtain tighter bounds; if we use the fixed initialization size n -1 4 regardless of the sample complexity, in Theorem 3.1, the factor 1 √ log n is improved to 1 √ np + σ λ ⋆ √ n p , and the upper bound on noise size is also improved to σ ≲ λ ⋆ µ n √ np (not being precise on the factors of µ and log n here). Then, the estimation error in Corollary 3.2 is improved to 1 √ np ρ t + σ λ ⋆ √ n p to match the result of [14] which uses spectral initialization. Thus, we have a tradeoff between estimation error and initialization size.

The next main result concerns the trajectory of GD before it enters the local region. The theorem states that for all t ≤ T ⋆ , x ( t ) stays close to the fully observed case ˜ x ( t ) in both ℓ 2 and ℓ ∞ -norm.

Theorem 3.3. Suppose that the conditions of Theorem 3.1 hold, and T ⋆ is defined as in Theorem 3.1. Then, for all t ≤ T ⋆ , we have

with probability at least 1 -o (1 / √ log n ) .

Trajectory of GD The sequence ˜ x ( t ) is a linear combination of x (0) and u ⋆ (see (C.1) in the appendix), and it is easy to analyze how ˜ x ( t ) evolves. By showing that x ( t ) stays close to ˜ x ( t ) for all iterations, we not only show the convergence of GD with small initialization as in Theorem 3.1, but also characterize the exact trajectory that GD follows by Theorem 3.3.

Implicit Regularization One can prove that ˜ x ( t ) is incoherent up to some log factors over all iterations, and from (11) and (12), the incoherence of x ( t ) is bounded by that of ˜ x ( t ) . Thus, Theorem 3.3 shows that the incoherence of x ( t ) is implicitly controlled by GD without any regularizer. This is an improvement over the previous result on the global convergence of GD for matrix completion [28], where an explicit regularizer was used to control the ℓ ∞ -norm of x ( t ) , although no small initialization was used in that work.

4 Fully Observed Case and Proof Sketch

Before we explain the proof of Theorems 3.1 and 3.3, we describe the trajectory of the fully observed case. We characterize ˜ x ( t ) with three variables: ˜ α t = ∣ ∣ u ⋆ ⊤ ˜ x ( t ) ∣ ∣ , ˜ β t = ∥ ˜ x ( t ) ∥ 2 , and ˜ γ t = ∥ ˜ x ( t ) ⊥ ∥ 2 , where ˜ x ( t ) ⊥ = ˜ x ( t ) -u ⋆ u ⋆ ⊤ ˜ x ( t ) . According to (2), the three variables are updated with

At t = 0 , due to random initialization, the initial vector is nearly orthogonal to u ⋆ , and we have ˜ α 0 ≈ 1 √ n β 0 and ˜ γ 0 ≈ ˜ β 0 = β 0 . Also, due to the small initialization, the term η ˜ β 2 t is ignorable until ˜ β t becomes sufficiently large, so ˜ α t grows exponentially at the rate of 1 + ηλ ⋆ , while ˜ γ t remains still. Thus, in the early iterations where (1+ ηλ ⋆ ) t is still much less than √ n , ˜ β t is kept close to its initial

10 -1

10 -2

10 -3

10 -4

10 -5

10 0

0

50

100

150

Phase I

Phase II

Local

Region

Iteration

value β 0 while the trajectory becomes more parallel to u ⋆ as ˜ α t increases. When (1 + ηλ ⋆ ) t becomes much larger than √ n , the trajectory becomes almost parallel to u ⋆ in that ˜ β t ≈ ˜ α t ≫ ˜ γ t . Until ˜ β t (asymptotically) reaches √ λ ⋆ √ log n , we can consider ˜ α t as increasing at a rate of (1 + ηλ ⋆ ) , and it takes about 1 log(1+ ηλ ⋆ ) log √ λ ⋆ n β 0 steps to reach this point. After that, we can no longer ignore the term η ˜ β 2 t , and ˜ α t increases at a slower rate as ˜ β t increases. We can show that ˜ β 2 t becomes sufficiently close to λ ⋆ within O (log log n ) additional iterations, as stated in the following lemma.

Finally, local convergence to u ⋆ occurs in that ˜ α t approaches λ ⋆ and ˜ γ t decreases exponentially with the rate (1 -ηλ ⋆ ) . The actual behavior of quantities ˜ α t , ˜ β t , ˜ γ t are plotted in Figure 1.

We define the iterates before (1 + ηλ ⋆ ) t reaches 1 √ np √ n , within some logarithmic factors, as Phase I, and the next iterates before ˜ β 2 t reaches λ ⋆ ( 1 -1 log n ) as Phase II. Different techniques are used for each phase to prove that x ( t ) stays close to ˜ x ( t ) . At the end of Phase I, ˜ α t is increased to 1 √ np β 0 from its initial scale 1 √ n β 0 , but it is still not dominant over β 0 . Therefore, the magnitudes of both x ( t ) and ˜ x ( t ) are kept close to β 0 throughout Phase I, and we take advantage of the small random initialization to show that the deviation of x ( t ) from ˜ x ( t ) does not increase much, and is kept at √ 1 np times the norms of x ( t ) . In Phase II, we show that x ( t ) -˜ x ( t ) expands at a rate of at most (1 + ηλ ⋆ ) . Since the norms of x ( t ) also grows at a rate of (1 + ηλ ⋆ ) during most of Phase II, the norms of x ( t ) -˜ x ( t ) remain negligible compared to those of x ( t ) . The next two sections give the main lemmas of Phase I and II, respectively, which are used to prove Theorems 3.1 and 3.3. For a visual representation of the results in the following two sections, please refer to Figure 2.

5 Phase I: Finding Direction

We provide detailed results and proof ideas for Phase I. Our main goal is to analyze the deviation of x ( t ) from ˜ x ( t ) . First, if we look at the update equations (1) and (2), the second term is proportional to the third power of ∥ ∥ x ( t ) ∥ ∥ 2 , while the other terms depend linearly on ∥ ∥ x ( t ) ∥ ∥ 2 . Thus, the second term is almost negligible due to the small initialization. Without the second terms, the difference between x ( t ) and ˜ x ( t ) at t = 1 is η ( M ◦ -M ⋆ ) x (0) . From concentration inequalities, one can see that the ℓ 2 and ℓ ∞ norms of η ( M ◦ -M ⋆ ) x (0) are about 1 √ np times smaller than those of ˜ x (1) .

Due to the third terms of (1) and (2), the norms of x ( t ) -˜ x ( t ) can grow exponentially at a rate of (1+ ηλ ⋆ ) in the worst case where x ( t ) -˜ x ( t ) is parallel to u ⋆ . In such a case, the norms of x ( t ) -˜ x ( t ) would be larger than those of ˜ x ( t ) at the end of Phase I, since those of ˜ x ( t ) remain still in Phase I. However, we overcome this problem by proving that the bounds grow at most polynomially with respect to t , and since t is at most O (log n ) , the bounds remain 1 √ np times smaller than the norms of ˜ x ( t ) up to logarithmic factors throughout Phase I.

Lemma 5.1. Let T 1 be the largest t such that (1 + ηλ ⋆ ) t ≤ √ µ 4 log 21 n np √ n . Under the conditions of Theorem 3.1, with probability at least 1 -o (1 / √ log n ) , for all t ≤ T 1 , we have

T 1 is defined to be the end of Phase I. Lemma 5.1 proves Theorem 3.3 for Phase I.

Proof of (13) We will first demonstrate how to obtain the ℓ 2 -norm bound of Lemma 5.1. Let us define a sequence ̂ x ( t ) that is updated as

(a)

(b)

(c)

Phase I

Iteration

part1

part2

Phase II

part3

local

Iteration

Phase I

Iteration

Note that the norm of ˜ x ( t ) is used in the second term of (15). The update equation of ̂ x ( t ) differs from ˜ x ( t ) in the third term and from x ( t ) in the second term. We use ̂ x ( t ) as a proxy for bounding ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ 2 . We first show that ∥ ∥ ̂ x ( t ) -˜ x ( t ) ∥ ∥ 2 grows at most linearly with respect to t . √

Lemma 5.2. With probability at least 1 -o (1 / log n ) , for all t ≤ T 1 , we have

The proof of this lemma is based on the fact that ̂ x ( t ) is a product of x (0) and a matrix polynomial of I and M ◦ , while ˜ x ( t ) is a product between x (0) and a matrix polynomial of I and M ⋆ . We prove the lemma by comparing the two matrix polynomials. We remark that Lemma 5.2 holds regardless of the small initialization, but it relies on the randomness of x (0) .

Since x ( t ) and ̂ x ( t ) differ only in the second term, their initial difference is proportional to β 3 0 . More precisely, it is 1 √ np β 3 0 . We show that the difference grows exponentially at a rate of (1 + ηλ ⋆ ) .

Lemma 5.3. If (14) holds for all t ≤ T 1 , we have

for all t ≤ T 1 with probability at least 1 -o (1 / √ log n ) .

The upper bound in (17) becomes smaller than that of (16) if (1 + ηλ ⋆ ) t β 2 0 ≤ λ ⋆ √ 1 µ log 2 n . One can check that this condition is satisfied from the definition of T 1 given in Lemma 5.1 and the bound on the initialization size (8). Thus, (13) is proved by (16) and (17).

Proof of (14) We control the l th component of x ( t ) -˜ x ( t ) using the l th leave-one-out sequence. Leave-one-out sequences have two important properties. First, because they are defined without only one row/column, they are extremely close to x ( t ) , and at t = 1 , ∥ ∥ x ( t ) -x ( t,l ) ∥ ∥ 2 is about 1 √ np β 0 √ n . Second, the l th component of the l th leave-one-out sequence evolves similarly to that of ˜ x ( t ) and is easy to analyze. With these two properties, we bound the l th component of x ( t ) -˜ x ( t ) as

We claim that both ∥ ∥ x ( t ) -x ( t,l ) ∥ ∥ 2 and ∣ ∣ ( x ( t,l ) -˜ x ( t ) ) l ∣ ∣ increase at most polynomially with respect to t from the initial scale 1 √ np β 0 √ n .

Lemma 5.4. With probability at least 1 -o (1 / √ log n ) , for all t ≤ T 1 , we have

As explained for x ( t ) -˜ x ( t ) , due to the third terms of (1) and (3), x ( t ) -x ( t,l ) can also grow exponentially at the rate of (1 + ηλ ⋆ ) in the worst case where x ( t ) -x ( t,l ) is parallel to u ⋆ . This contradicts our result (19) that ∥ ∥ x ( t ) -x ( t,l ) ∥ ∥ 2 grows only linearly. We show that x ( t ) -x ( t,l ) remains nearly orthogonal to u ⋆ in Phase I, and thus the worst case does not occur.

Lemma 5.5. For all l ∈ [ n ] and t ≤ T 1 , we have

with probability at least 1 -o (1 / √ log n ) , where u ( l ) is the first eigenvector of M ( l ) .

Note that u ( l ) is almost parallel to u ⋆ (see Lemma A.5 in the appendix). The u ( l ) component of x ( t ) -x ( t,l ) is initialized to the order of 1 √ np β 0 n , which is 1 √ n times smaller than ∥ ∥ x ( t ) -x ( t,l ) ∥ ∥ 2 . Although it is increased exponentially, from the definition of T 1 , the u ( l ) component remains much smaller than ∥ ∥ x ( t ) -x ( t,l ) ∥ ∥ 2 in Phase I.

One can see that ∣ ∣ ( x ( t,l ) -˜ x ( t ) ) l ∣ ∣ increases by ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ 2 ∥ u ⋆ ∥ ∞ at each step, and summing the bound (13) up to t gives (20). Finally, (14) is obtained by putting (19) and (20) into (18).

6 Phase II: Expansion

In the next phase, we show that the bounds obtained in Phase I are increased at a rate of (1 + ηλ ⋆ ) .

Lemma 6.1. Let T 2 be the largest t such that ˜ β 2 t ≤ λ ⋆ ( 1 -1 log n ) . Then, for all T 1 < t ≤ T 2 , we have

with probability at least 1 -o (1 / √ log n ) .

T 2 is defined as the end of Phase II. We will explain how Lemma 6.1 leads to Theorem 3.3 in Phase II. Let us first focus on (21) and (11). We can divide Phase II into three parts according to the behavior of ∥ ∥ ˜ x ( t ) ∥ ∥ 2 . First, ∥ ∥ ˜ x ( t ) ∥ ∥ 2 is kept close to β 0 until (1 + ηλ ⋆ ) t becomes √ n , or (1 + ηλ ⋆ ) t -T 1 becomes √ np . In this part, although the bounds increase exponentially with the rate of (1 + ηλ ⋆ ) , the factor 1 √ np , which was already present in (13) of Phase I, compensates for this increase. At the end of the first part, ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ 2 is smaller than ∥ ∥ ˜ x ( t ) ∥ ∥ 2 by some log factors. Next, ∥ ∥ ˜ x ( t ) ∥ ∥ 2 grows at the rate of (1+ ηλ ⋆ ) until it reaches √ λ ⋆ 8 √ log n . Since both ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ 2 and ∥ ∥ ˜ x ( t ) ∥ ∥ 2 increase with (1+ ηλ ⋆ ) ,

Iteration

0

50

100

150

200

250

10 -1

10 -2

10 -3

10 -4

10 -5

10 0

10 -6

10 -7

1.4

10 -1

10 -2

10 -3

10 -4

10 -5

10 0

10 -6

10 -7

10 -8

10 -9

0

0.2

0.4

0.6

0.8

1.0

1.2

Initialization Size

Iteration

0

50

100

150

200

250

10 -1

10 -2

10 -3

10 -4

10 -5

10 0

10 -6

10 -7

(a)

(b)

(c)

the ratio between them is maintained in the second part. Finally, in the remaining iterations, ∥ ∥ ˜ x ( t ) ∥ ∥ 2 increases with (1 -η ˜ β 2 t + ηλ ⋆ ) at each step, and the increment becomes smaller as it converges to √ λ ⋆ . Thus, as in the first part, ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ 2 increases faster than ∥ ∥ ˜ x ( t ) ∥ ∥ 2 . However, from Lemma 4.1, the length of this part is O (log log n ) , and the ratio between ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ 2 and ∥ ∥ ˜ x ( t ) ∥ ∥ 2 increases only by log 6 n . We prove that the log factors already present at the end of the second part compensate this, and finally (11) holds for all t in Phase II. A more delicate analysis may prove that ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ 2 grows at the same rate as ∥ ∥ ˜ x ( t ) ∥ ∥ 2 in the third part, and this will reduce the required sample complexity by at most log 12 n . A similar argument can be used to prove that the bounds for ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ ∞ , ∥ ∥ x ( t ) -x ( t,l ) ∥ ∥ 2 , and ∣ ∣ ( x ( t,l ) -˜ x ( t ) ) l ∣ ∣ are smaller than ∥ ∥ ˜ x ( t ) ∥ ∥ ∞ by some log factors throughout Phase II.

At the end of Phase II, ˜ x ( t ) is very close to ± x ⋆ in both ℓ 2 and ℓ ∞ norms (see Corollary C.3 in the appendix), so one can replace ˜ x ( t ) of Lemma 6.1 with ± x ⋆ to prove (4) to (7) of Theorem 3.1. Hence, we can let T ⋆ = T 2 , and as explained in Section 4, T 2 is approximately given by 1 log(1+ ηλ ⋆ ) log √ λ ⋆ n β 0 + O (log log n ) .

7 Simulation

In this section, we present some simulation results that support our theoretical findings.

Trajectory of GD With the dimension n = 5000 , we constructed the ground truth vector u ⋆ by sampling it from the Gaussian distribution N ( 0 , 1 n I ) and normalizing it to have unit norm. We let λ ⋆ = 1 so that the matrix M ⋆ is given by u ⋆ u ⋆ ⊤ , and we randomly sampled the matrix symmetrically with a sampling rate of p = 0 . 1 and Gaussian noise of σ = 0 . 1 n . The initialization size was set to β 0 = 1 n and a step size of 0 . 1 was used for GD. Figure 3 (a) and (b) represent one trial of the experiment, but similar graphs were obtained in each repetition of the experiment. The evolution of some important quantities such as ∥ ∥ x ( t ) ∥ ∥ 2 and ∣ ∣ u ⋆ ⊤ x ( t ) ∣ ∣ is shown in Figure 3(a). As in the fully observed case, the signal component ∣ ∣ u ⋆ ⊤ x ( t ) ∣ ∣ increases at the the rate of (1 + ηλ ⋆ ) until it approaches √ λ ⋆ , and a local convergence to x ⋆ occurs, where ∥ ∥ x ( t ) -x ⋆ ∥ ∥ 2 decreases exponentially and saturates at the level determined by the noise size σ . In Figure 3(b), we describe the deviation of x ( t ) from ˜ x ( t ) in both ℓ 2 and ℓ ∞ norms. The solid lines represent the norms of x ( t ) and the dotted lines represent those of x ( t ) -˜ x ( t ) . We can see that there is a gap between the solid and the dotted lines during the whole iterations. Thus, x ( t ) stays close to the trajectory of the fully observed case, as we proved in Theorem 3.3.

Small Initialization In the next experiment, we investigated the importance of a small initialization for the convergence of GD. We used the same conditions as in the previous experiment except

n = 500 . We measured ∥ ∥ x ( t ) ± x ⋆ ∥ ∥ 2 at t = 1 log(1+ ηλ ⋆ ) log √ λ ⋆ n β 0 +100 and averaged it over 1000 trials. We repeated the experiment while changing the initialization size from 10 0 to 10 -9 and the sampling probability from 0 . 01 to 0 . 04 . The result is summarized in Figure 3(c). For all sampling probabilities, the small initialization improves the convergence of GD. Also, the performance starts to saturate at much larger initialization sizes as the sampling probability increases, and this is consistent with our finding (8) that a larger initialization is possible as more samples are available.

8 Discussion

In this paper, we showed that for rank-1 symmetric matrix completion with ℓ 2 loss, GD can converge to the ground truth starting from a small random initialization. Ignoring log factors, the bound on the initialization size is n -1 4 when the optimal n poly(log n ) samples are provided , and the bound becomes larger as more samples are provided. The result is interesting because the loss function does not have global benign geometry if no regularizer is applied. Our result does not use any explicit regularizer and relies only on the implicit regularizing effect of GD.

The most important future work is an extension to the rankr case. Suppose that M ⋆ is a rankr matrix and its eigendecomposition is given by U ⋆ Σ ⋆ U ⋆ ⊤ = X ⋆ X ⋆ ⊤ , where Σ ⋆ = diag( λ ⋆ 1 , · · · , λ ⋆ r ) and X ⋆ = U ⋆ Σ ⋆ 1 2 . Then, the trajectory of GD becomes an n × r matrix X ( t ) , which is updated as

10 -1

10 -2

10 -3

10 -4

10 -5

10 0

10 -6

10 1

0

50

100

150

200

250

300

350

400

Iteration

Each entry of X (0) is sampled independently from the Gaussian distribution N ( 0 , 1 n β 2 0 ) as in the rank-1 case.

An instance of X ( t ) is shown in Figure 4. The same conditions as in Figure 3 are used, except that the ground truth matrix is a rank-3 matrix with non-zero eigenvalues 1 , 0 . 75 , 0 . 5 . The singular values of X ( t ) behave similarly to ∥ ∥ x ( t ) ∥ ∥ 2 in the rank-1 case. In the early iterations, where the orthogonal components dominate, the singular values stay close to their initial scale β 0 . After that, each singular value σ i ( X ( t ) ) increases at a rate of (1 + ηλ ⋆ i ) and saturates at √ λ ⋆ i . We use ∥ X -Y ∥ R to denote the Frobenius norm between X and Y under best rotational alignment. ∥ ∥ X ( t ) -X ⋆ ∥ ∥ R decreases exponentially and saturates at the level determined by the noise size σ , after all singular values have saturated, as local convergence begins.

To extend the results of the rank-1 case, we need to show that ∥ ∥ X ( t ) -˜ X ( t ) ∥ ∥ R remains much smaller than ∥ ∥ X ( t ) ∥ ∥ F throughout the iterations, where ˜ X ( t ) is the trajectory of the fully observed case. Before σ 1 ( X ( t ) ) saturates around √ λ ⋆ 1 , it behaves similarly to ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ 2 of the rank-1 case, i.e., it expands at a rate of (1 + ηλ ⋆ 1 ) along with ∥ ∥ X ( t ) ∥ ∥ F after the early iterations. However, because each singular value grows at a different rate, a different phenomenon is observed for the rankr case. During the iterations before σ i +1 ( X ( t ) ) saturates after σ i ( X ( t ) ) does, both ∥ ∥ X ( t ) -˜ X ( t ) ∥ ∥ R and ∥ ∥ X ( t ) ∥ ∥ F do not increase much. Our current theory can only show that ∥ ∥ X ( t ) -˜ X ( t ) ∥ ∥ R increases at a rate less than (1+ ηλ ⋆ i +1 ) , and in order for ∥ ∥ X ( t ) -˜ X ( t ) ∥ ∥ R to remain much smaller than ∥ ∥ X ( t ) ∥ ∥ F , additional sample complexity is required to compensate for the exponential increases. Therefore, we expect that the convergence of GD for the case of rankr can be proved with the techniques developed in this paper if n 1+Θ( κ -1) poly( κ, r, log n ) samples are provided, where κ = λ ⋆ 1 λ ⋆ r is the condition number. Nevertheless, whether GD can converge with the optimal n poly( κ, r, log n ) samples for the rankr matrix completion problem remains an open problem.

Acknowledgments and Disclosure of Funding

This research was supported by the National Research Foundation of Korea under grant 2021R1C1C11008539.

References

Y. Chen, Y. Chi, J. Fan, and C. Ma, 'Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval,' Mathematical Programming , vol. 176, no. 1-2, pp. 5-37, 2019.

A. Ahmed, B. Recht, and J. Romberg, 'Blind deconvolution using convex programming,' IEEE Transactions on Information Theory , vol. 60, no. 3, pp. 1711-1732, 2014.

S. Tu, R. Boczar, M. Simchowitz, M. Soltanolkotabi, and B. Recht, 'Low-rank solutions of linear matrix equations via procrustes flow,' in International Conference on Machine Learning , pp. 964-973, 2016.

E. J. Candès and B. Recht, 'Exact matrix completion via convex optimization,' Foundations of Computational Mathematics , vol. 9, no. 6, pp. 717-772, 2009.

D. F. Gleich and L.-h. Lim, 'Rank aggregation via nuclear norm minimization,' in Proceedings of the 17th ACM international conference on Knowledge discovery and data mining , pp. 60-68, ACM, 2011.

E. J. Candès, X. Li, Y. Ma, and J. Wright, 'Robust principal component analysis?,' Journal of the ACM , vol. 58, no. 3, pp. 11:1-11:37, 2011.

Y. Hu, X. Liu, and M. Jacob, 'A generalized structured low-rank matrix completion algorithm for mr image recovery,' IEEE Transactions on Medical Imaging , vol. 38, no. 8, pp. 1841-1851, 2019.

Y. Chi, Y. M. Lu, and Y. Chen, 'Nonconvex optimization meets low-rank matrix factorization: An overview,' IEEE Transactions on Signal Processing , vol. 67, no. 20, pp. 5239-5269, 2019.

E. J. Candès and T. Tao, 'The power of convex relaxation: Near-optimal matrix completion,' IEEE Transactions on Information Theory , vol. 56, no. 5, pp. 2053-2080, 2010.

R. H. Keshavan, A. Montanari, and S. Oh, 'Matrix completion from a few entries,' IEEE Transactions on Information Theory , vol. 56, no. 6, pp. 2980-2998, 2010.

P. Jain, P. Netrapalli, and S. Sanghavi, 'Low-rank matrix completion using alternating minimization,' in Proceedings of the forty-fifth annual ACM symposium on Theory of computing , pp. 665-674, ACM, 2013.

Y. Chen and M. J. Wainwright, 'Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees,' arXiv preprint arXiv:1509.03025 , 2015.

R. Sun and Z.-Q. Luo, 'Guaranteed matrix completion via non-convex factorization,' IEEE Transactions on Information Theory , vol. 62, no. 11, pp. 6535-6579, 2016.

C. Ma, K. Wang, Y. Chi, and Y. Chen, 'Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval, matrix completion, and blind deconvolution,' Foundations of Computational Mathematics , vol. 20, no. 3, pp. 451-632, 2020.

J. Chen, D. Liu, and X. Li, 'Nonconvex rectangular matrix completion via gradient descent without ℓ 2 , ∞ regularization,' IEEE Transactions on Information Theory , vol. 66, no. 9, pp. 58065841, 2020.

S. S. Du, C. Jin, J. D. Lee, M. I. Jordan, A. Singh, and B. Poczos, 'Gradient descent can take exponential time to escape saddle points,' in Advances in Neural Information Processing Systems , 2017.

Y. Li, T. Ma, and H. Zhang, 'Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations,' in Conference on Learning Theory , pp. 2-47, 2018.

T. Ye and S. S. Du, 'Global convergence of gradient descent for asymmetric low-rank matrix factorization,' in Advances in Neural Information Processing Systems , pp. 1429-1439, 2021.

S. Du, J. Lee, H. Li, L. Wang, and X. Zhai, 'Gradient descent finds global minima of deep neural networks,' in International Conference on Machine Learning , pp. 1675-1685, 2019.

Y. Ma, A. Olshevsky, C. Szepesvari, and V. Saligrama, 'Gradient descent for sparse rankone matrix completion for crowd-sourced aggregation of sparsely interacting workers,' in International Conference on Machine Learning , pp. 3335-3344, 2018.

Q. Ma and A. Olshevsky, 'Adversarial crowdsourcing through robust rank-one matrix completion,' in Advances in Neural Information Processing Systems , pp. 21841-21852, 2020.

S. Gunasekar, B. E. Woodworth, S. Bhojanapalli, B. Neyshabur, and N. Srebro, 'Implicit regularization in matrix factorization,' in Advances in Neural Information Processing Systems , 2017.

D. Stöger and M. Soltanolkotabi, 'Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction,' in Advances in Neural Information Processing Systems , pp. 23831-23843, 2021.

S. Arora, N. Cohen, W. Hu, and Y. Luo, 'Implicit regularization in deep matrix factorization,' in Advances in Neural Information Processing Systems , pp. 7413-7424, 2019.

N. Razin and N. Cohen, 'Implicit regularization in deep learning may not be explainable by norms,' in Advances in Neural Information Processing Systems , pp. 21174-21187, 2020.

Z. Li, Y. Luo, and K. L. Lyu, 'Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning,' in In International Conference on Learning Representations , 2021.

S. Bhojanapalli, B. Neyshabur, and N. Srebro, 'Global optimality of local search for low rank matrix recovery,' in Advances in Neural Information Processing Systems , 2016.

R. Ge, J. D. Lee, and T. Ma, 'Matrix completion has no spurious local minimum,' in Advances in Neural Information Processing Systems , 2016.

Detailed proofs for the results explained in the main text are provided in this appendix. We say that an event happens with high probability if it happens with probability at least 1 -1 n C for a constant C > 0 and C can be made arbitrary large by controlling constant factors. A union of poly( n ) number of events that happens with high probability still happens with high probability. For a matrix A , we denote the spectral norm by ∥ A ∥ and the maximum absolute value of entries by ∥ A ∥ ∞ . Also, the largest ℓ 2 -norm of rows of A is denoted as ∥ A ∥ 2 , ∞ .

A Spectral Analysis

We introduce some spectral bounds related to random sampling and Gaussian noise.

Lemma A.1. If n 2 p ≳ n log n , we have

with high probability.

Lemma A.2. If n 2 p ≳ µn log n , for all l ∈ [ n ] , we have

with high probability.

Lemma A.3. If n 2 p ≳ n log 2 n , we have

with high probability.

Note that Lemma A.3 also implies that ∥ ∥ E ( l ) ∥ ∥ ≲ σ √ n p for all l ∈ [ n ] with high probability. Combined with the condition σ ≲ λ ⋆ µ √ log n , we have

n

for all l ∈ [ n ] . Proofs for Lemmas A.1 and A.2 are provided in Appendix G. Check Lemma 11 of [12] for the proof of Lemma A.3.

Next, we state bounds on the eigenvalues of M ◦ and M ( l ) . The first eigenvalues of M ◦ and M ( l ) are denoted as λ ◦ and λ ( l ) , respectively. The following lemma is derived from Lemmas A.1 and A.2 with Weyl's Theorem.

Lemma A.4. If n 2 p ≳ µn log n , we have

for all l ∈ [ n ] with high probability.

Lastly, Lemmas A.1 and A.2 with Davis-Kahan Theorem give the following lemma.

Lemma A.5. If n 2 p ≳ µn log n , we have

for all l ∈ [ n ] with high probability.

B Initialization

In this section, we introduce some properties that the initialization vector x (0) satisfies. Recall that each entry of x (0) is sampled from N (0 , 1 n β 2 0 ) independently. We use H to denote the perturbation M ◦ -M ⋆ .

Lemma B.1. The initialization vector x (0) satisfies

with probability at least 1 -e -n/ 32 , and

with probability at least 1 -1 n -30 log n n 2 . It also satisfies

with probability at least 1 -1 2 √ log n .

Proof. To bound ∥ ∥ x (0) ∥ ∥ 2 , we use the following basic concentration inequality that holds for i.i.d. standard normal variables { X i } i ∈ [ n ] .

If we put t = 1 2 , with probability at least 1 -e -n/ 32 , we have

and this implies (B.1).

For a centered Gaussian random variable with standard deviation σ , we have

Hence, an entry of x (0) is less than 2 √ log n β 0 √ n with probability at least 1 -1 n 2 , and all entries of x (0) are less than 2 √ log n β 0 √ n with probability at least 1 -1 n . u ⋆ ⊤ H s x (0) follows a centered Gaussian distribution with standard deviation ∥ H s u ⋆ ∥ 2 ≤ ∥ H ∥ s ∥ u ⋆ ∥ 2 for all s , and (B.3) holds with probability at least 1 -10 log n n 2 .

For a random variable X that is sampled from N (0 , σ 2 ) , we have

Hence, we have with probability at least 1 -1 2 √ log n

Lemma B.1 implies that the u ⋆ component of x (0) is in the range

Lemma B.2. We have because

if ηλ ⋆ 64 log n ≤ 1 2 . Note that the upper bounds in (C.2) hold even if t > T 2 .

From (C.2), we have the approximation ˜ x ( t ) ≈ x (0) +(1 + ηλ ⋆ ) t ( u ⋆ ⊤ x (0) ) u ⋆ for all t ≤ T ′ 2 and the ℓ 2 -norm of ˜ x ( t ) is also approximately given by ( 1 + (1+ ηλ ⋆ ) t √ n ) β 0 . The ℓ ∞ -norm is about 1 √ n times smaller than the ℓ 2 -norm. We make this observation rigorous with the following lemma.

Lemma C.1. For all t ≤ T ′ 2 , we have

Proof. For brevity, les us drop the superscript ( t ) and write ˜ x = A x (0) + B u ⋆ . For the upper bounds, we may use the triangle inequality

for all a, b , with probability at least 1 -exp ( -n 2 µ )

Proof. The probability that an entry of x (0) is less than β 0 √ n is bounded by 1 √ 2 π . Without loss of generality, let us assume that all entries of u ⋆ are not negative and a, b ≥ 0 . There are at least n µ entries of u ⋆ that are larger than 1 √ n . For such entries, the probability that all entries of x (0) is less than β 0 √ n is bounded by ( 1 √ 2 π ) n µ ≤ exp ( -n 2 µ ) . Hence, for at least one position, both entries of x (0) and u ⋆ are larger than β 0 √ n and 1 √ n , respectively, with probability at least 1 -exp ( -n 2 µ ) .

In the following sections, we assume that we are given an initialization vector x (0) that satisfies (B.1) to (B.6).

C Fully Observed Case

We provide some lemmas related to ˜ x ( t ) in this section. We first note that ˜ x ( t ) is explicitly written as

Let us define T ′ 2 as the last t such that ˜ β 2 t ≤ λ ⋆ 64 log n . We claim that T ′ 2 ≤ 64 log n ηλ ⋆ and prove this later. Then, for all t ≤ T ′ 2 , we have

If we use (B.5) and (C.2), we get the upper bounds for A and B . We have ∥ ∥ x (0) ∥ ∥ 2 ≤ 2 β 0 by (B.1), and the ℓ ∞ -norm of x (0) is controlled through (B.2). These finish the proof for the upper bounds.

From the definition of B , we have B ( u ⋆ ⊤ x (0) ) ≥ 0 , and

(B.1) and (B.4) together with the lower bound in (C.2) give the desired lower bound for ∥ ˜ x ∥ 2 . The lower bound for ℓ ∞ -norm is directly implied from Lemma B.2 together with (B.4) and (C.2).

and thus

For t ≤ T 1 where (1 + ηλ ⋆ ) t is not big, the bounds in Lemma C.1 are simplified to

After ˜ x ( t ) becomes almost parallel to u ⋆ and before T ′ 2 , we could approximate ˜ β t as increasing with the rate (1 + ηλ ⋆ ) . However, after T ′ 2 , this approximation is invalid, and ˜ β t grows at a slower rate as it increases and it eventually converges to √ λ ⋆ . How much iterations will be required for it to reach √ λ ⋆ √ 1 -1 log n after T ′ 2 ? With Lemma C.2, we will prove that O (log log n ) iterations are required after T ′ 2 .

Proof. From the decomposition (C.1), we have and thus

holds for all t . Because ˜ β t ≳ 1 log n for all t ≥ T ′ 2 , (C.5) implies that ˜ β t is well approximated by ˜ α t . Hence, we will focus on ˜ α t , which is an increasing sequence that evolves with

For all i ≥ 1 , let N i be the last t such that λ ⋆ -˜ α 2 t ≥ λ ⋆ e i . Then, we have

Let i ≥ 2 . For all N i -1 < t ≤ N i ,

We used (C.5), (C.6), and the fact that ˜ α t , ˜ β t ≤ √ λ ⋆ for all t . This implies

From the lower and upper bounds provided by (C.6), we have

Taking log on both sides and using the inequality 1 2 x < log(1 + x ) < x that holds for 0 < x < 1 , we get

For t ≤ N 1 , we have and thus

Taking log on both sides we get

Hence, we have

but at t = N log log n +1 +1 , it holds that

and we have as desired. Note that ˜ β t is also an increasing sequence as ˜ α t .

It is implied from Lemma C.2 that T 2 ≤ 1 log(1+ ηλ ⋆ ) log √ λ ⋆ n β 0 + 6 log log n log(1+ ηλ ⋆ ) = (1+ o (1)) 1 ηλ ⋆ log √ λ ⋆ n β 0 . The following corollary shows that ˜ x ( t ) is sufficiently close to x ⋆ at t = T 2 . Corollary C.3. At t = T 2 , we have

Proof. When B ( t ) > 0 , from the decomposition

we have

For the cases B < 0 and ℓ

( t ) ∞ -norm, we may use similar technique.

D Phase I

D.1 Proof of Lemma 5.2

In this subsection, we provide a proof to the following lemma, which is a formal statement of Lemma 5.2.

Lemma D.1. With high probability, there exists a universal constant c 0 > 0 such that

for all t ≤ T 1 , if n 2 p ≳ µ 4 n log 21 n and the initialization point x (0) satisfies (B.1) to (B.6).

Proof. Let us rewrite the update equations (2) and (15) as

where ˜ β t = ∥ ∥ ˜ x ( t ) ∥ ∥ 2 2 . Then, ̂ x ( t ) -˜ x ( t ) is a product between x (0) and P ( t ) ( I , M ⋆ , H ) , which is a matrix polynomial of I , M ⋆ , H , where H = M ◦ -M ⋆ .

We classify the terms that appear after expanding the matrix polynomial P ( t ) ( I , M ⋆ , H ) into two types; 1) the terms that contain H but not M ⋆ , 2) the terms that contain both H and M ⋆ . We define P ( t ) 1 ( I , H ) to be a matrix polynomial of I and H , which is equal to summation of the first type, and it is explicitly written as

We correspondingly define P ( t ) 2 ( I , M ⋆ , H ) to be summation of the second type, and it is equal to

For x, y ∈ R , we define P ( t ) 1 ( x, y ) as the value that is obtained by substituting x, y instead of I , H , respectively. For example, P ( t ) 1 (1 , 2) = ∏ t s =1 (1 -η ˜ β 2 s +2 η ) -∏ t s =1 (1 -η ˜ β 2 s ) . For x, y, z ∈ R , P ( t ) 2 ( x, y, z ) is defined in a similar manner.

We bound the contribution of each type separately because the triangle inequality gives

Every term in P ( t ) 1 ( I , H ) is H s times a constant. We have ∥ ∥ H s x (0) ∥ ∥ 2 ≤ ∥ H ∥ s ∥ ∥ x (0) ∥ ∥ 2 , and hence with triangle inequality

If n 2 p ≳ µ 2 n log 3 n , we can further bound P ( t ) 1 (1 , ∥ H ∥ ) as

The third line uses the fact that ˜ β 2 t ≤ λ ⋆ for all t ≤ T 1 . The fourth and fifth lines are derived from an elementary inequality 1 + x ≤ e x ≤ 1 + 2 x , which holds for small x > 0 . Note that η ∥ H ∥ t ≲ ηλ ⋆ µ √ log n np ≪ 1 from Lemmas A.1 and A.3, and the fact that T 1 ≲ log n .

We can decompose every term of second type as a product of η , λ ⋆ , u ⋆ , H s u ⋆ , u ⋆ ⊤ H s x (0) , u ⋆ ⊤ H s u ⋆ , and u ⋆ ⊤ x (0) . We describe this with some examples.

The terms H s u ⋆ and u ⋆ ⊤ H s u ⋆ are bounded with

and the terms that contain x (0) are bounded with (B.3). For every term of second type that includes s 1 times of η M ⋆ and s 2 times of η H , the bounds (D.4) and (B.3) imply that ℓ 2 -norm of the term multiplied by x (0) is at most

Hence, similar to the first type, we have

If n 2 p ≳ µ 2 n log 3 n , we can further bound P ( t ) 2 (1 , λ ⋆ , ∥ H ∥ ) as

Combining all, we have

for all t ≤ T 1 for some constant c 0 > 0 if n 2 p ≳ µ 4 n log 22 n .

D.2 Proof of Lemmas 5.3 to 5.5

We prove Lemmas 5.3 to 5.5 all together in an inductive manner.

Lemma D.2. Suppose that the initialization point x (0) satisfies (B.1) to (B.6). If n 2 p ≳ µ 5 n log 22 n , for all t ≤ T 1 , we have

with high probability, where c 2 , c 5 , c 6 are positive constants.

Before we start the proof, we introduce some notations. For x ∈ R n , let us define

∥ x ∥ 2 ,i is the ℓ 2 -norm of x estimated with sampling of the i th row. With this notation, we can write the gradient of f as

The function g is defined as and its gradient satisfies

The Hessian of g ( x ) is equal to

The base case t = 0 for induction hypotheses (D.5) to (D.10) trivially hold because all three sequences x ( t ) , ̂ x ( t ) , ˜ x ( t ) start from the same point. Now, we assume that the hypotheses hold up to the t th iteration and show that they hold at the ( t +1) st iteration. For brevity, we drop the superscript ( t ) from x ( t ) , x ( t,l ) , ̂ x ( t ) , ˜ x ( t ) and denote them as x , x ( l ) , ̂ x , ˜ x , respectively. Also, recall that T 1 is defined to be the last t such that (1 + ηλ ⋆ ) t ≤ √ µ 4 log 21 n np √ n , and the magnitude of initialization satisfies β 2 0 ≲ λ ⋆ √ np µ 5 log 26 n 1 √ n so that there exists a constant c 1 > 0 such that (1 + ηλ ⋆ ) t β 2 0 ≤ c 1 λ ⋆ √ µ log 5 n .

(D.7) at ( t +1) x ( t +1) -x ( t +1)

With the help of Lemma G.8, we bound the maximum entry of a diagonal matrix ∥ x ∥ 2 2 I x -∥ ˜ x ∥ 2 2 I . We have

if n 2 p ≳ n log 2 n . Hence, there exists a universal constant c 2 > 0 that is independent of t such that

for all t ≤ T 1 . With the decomposition (D.11), we have

From (A.1), there exists a universal constant c 3 > 0 such that ηλ ◦ ≤ ηλ ⋆ + c 3 log 2 n if n 2 p ≳ µ 2 n log 5 n . Combining all, for all s ≤ t , we have

An analysis on the recursive equation

proves that

where x ( l ) ( τ ) = x ( l ) + τ ( x -x ( l ) ) . 1 is easily bounded by

From Lemma G.10 and (C.4), for all 0 ≤ τ ≤ 1 , we have

From the definition of x ( τ ) , we have

where the last inequality is from the induction hypotheses (D.6), (D.8), and the fact that t ≤ T 1 ≲ log n . Inserting this bound back to (D.12), we get

which also implies

It is implied from Lemma G.11 that

A bound on 4 follows from the induction hypothesis (D.9) and the spectral bound (A.2).

The second largest eigenvalue of M ( l ) is at most ∥ ∥ M ( l ) -M ⋆ ∥ ∥ by Weyl's Theorem, and from Lemmas A.1 and A.2, we have

Hence, we get

Lastly, we apply Lemmas G.11 and G.13 to get

There exists a universal constant c 4 > 0 such that

if n 2 p ≳ µ 2 n log 5 n , and there exists a universal constant c 5 > 0 such that

if n 2 p ≳ µ 5 n log 20 n . Combining all, we have

for all s ≤ t . An analysis on the recursive equation

gives the desired bound

where we used basic inequalities (1 + x ) a ≤ e ax and e ax -1 ≤ 2 ax which hold if x is small and ax is small, respectively.

The definition of Phase I was used to bound (1 + ηλ ⋆ ) t t in deriving the last line. We use (D.13) to get

We divide 4 into two terms that are related to sampling and noise, respectively.

Then, Cauchy-Schwartz inequality is applied to yield

∣

∣

∥

(

∥

∥

∥

∥

(

∥

∥

∥

1

p

1

p

∥

∥

∥

∥

2

∥

∥

∥

∥

∥

2

Applying Lemmas G.11 and G.13 to the two terms, respectively, we get

For the term 5 , we decompose it into two terms as for 4 .

Then, we apply Lemmas G.12 and G.14 to each term to obtain

Combining all, there exists a universal constant c 6 > 0 such that

if n 2 p ≳ µ 2 n log 3 n , and there exists a universal constant c 7 > 0 such that

by (A.2) if n 2 p ≳ µ 2 n log 5 n . Finally, we have

An analysis on the recursive equation

gives the bound

⋆

(

l

Ω

)

⋆

)

(

l

)

(

l

∣

4

∣

P

Ω

(

M

)

-P

(

M

)

u

∥

x

-

x

)

+

P

Ω

(

E

)

-

E

(

l

)

)

u

(

l

)

∥

∥

∥

∥

2

∥

∥

∥

x

-

x

(

l

)

∥

∥

∥

2

.

(D.10) at ( t +1) We decompose ( x ( t +1 ,l ) -˜ x ( t +1) ) l as

and this implies

From (D.5) and (D.8), we have

If n is sufficiently large, we have

for all t ≤ T 1 because

Hence, we have

for all s ≤ t . Finally, we have

(D.5) at ( t +1) We can obtain this through the combination of (D.7) and Lemma 5.2.

(D.6) at ( t +1) The l th component of x ( t +1) -˜ x ( t +1) is bounded by

At t = T 1 Because T 1 ≲ log n , it is implied from Lemma D.2 that at t = T 1 , there exists a constant c 7 > 0 such that

These bounds serve as a base case for the induction of the next part.

E Phase II

This section is mostly devoted to the proof of Lemma E.1 which is a formal version of Lemma 6.1.

Lemma E.1. Suppose that (D.14) holds at t = T 1 and the initialization point x (0) satisfies (B.1) to (B.6). Then, for all T 1 < t ≤ T 2 , we have

with high probability, where T 2 is the largest t such that ˜ β 2 t ≤ λ ⋆ ( 1 -1 log n ) , and c 13 > 0 is a constant.

Proof of Theorems 3.1 and 3.3 We first explain how Theorems 3.1 and 3.3 are derived from Lemmas D.2 and E.1. We first focus on ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ 2 . For t ≤ T 1 , from Lemma D.2 and (C.3), we have

provided that n 2 p ≳ µ 2 n log 7 n . For T 1 < t ≤ T 2 , from the definition of T 1 and Lemma E.1, we have

From the lower bound of Lemma C.1, for all t ≤ T ′ 2 , we have

Now, for T ′ 2 < t ≤ T 2 , we have

For any T ′ 2 < t ≤ T 2 , it is Lemma C.2 implied from Lemma C.2 that (1 + ηλ ⋆ ) t -T ′ 2 ≤ log 6 n , and we have ∥ ∥ ∥˜ x ( T ′ 2 ) ∥ ∥ ∥ 2 ≤ ∥ ˜ x ∥ 2 . Hence, we get

and the proof for (11) of Theorem 3.3 is completed. If we combine this with (C.7), we are able to prove (4) of Theorem 3.1.

We move on to ∥ ∥ x ( t ) -˜ x ( t ) ∥ ∥ ∞ . For t ≤ T 1 , from Lemma D.2 and (C.4), we have

provided that n 2 p ≳ µ 3 n log 10 n . For T 1 < t ≤ T ′ 2 , from the definition of T 1 and Lemma E.1, we have

From the lower bound of Lemma C.1, for all t ≤ T ′ 2 , we have

Now, for T ′ 2 < t ≤ T 2 , if we do the same as before, we get

and the proof for (12) of Theorem 3.3 is completed. If we combine this with (C.8), we are able to prove (5) of Theorem 3.1.

Going through a similar way with (22) and (24), we can complete the proof of Theorems 3.1 and 3.3.

Proof of Lemma E.1 Before we start the proof, we define a function G as

The gradient of G satisfies

Now, we assume that the hypotheses hold up to the t th iteration and show that they hold at the ( t +1) st iteration. For brevity, we drop the superscript ( t ) from x ( t ) , x ( t,l ) , ˜ x ( t ) and denote them as x , x ( l ) , ˜ x , respectively.

where x ( τ ) = ˜ x + τ ( x -˜ x ) . For the term 1 , we require a bound on

If we write ˜ x as ˜ α t u ⋆ + ˜ x ⊥ , we have ˜ α 2 t ≤ λ ⋆ and ∥ ˜ x ⊥ ∥ 2 ≲ β 0 . Then, we have

for some universal constant c 8 > 0 . This implies the desired bound

For all 0 ≤ τ ≤ 1 , we have ∥ x ( τ ) -˜ x ∥ ∞ ≤ ∥ x -˜ x ∥ ∞ , and the induction hypothesis (E.2) gives

if n 2 p ≳ µ 2 n log 15 n because ∥ ˜ x ∥ ∞ ≲ √ µ log n (1 + ηλ ⋆ ) T 2 β 0 n and (1 + ηλ ⋆ ) T 2 ≲ √ λ ⋆ √ n β 0 . This gives

For the term 3 , we use Lemma G.8 to obtain

Lastly, the term 4 is bounded with

Combining all, there exists a universal constant c 9 > 0 such that

Because ∥ ∥ x ( T 1 ) ∥ ∥ 2 ≲ √ log nβ 0 by (C.3) and ∥ ∥ ˜ x ( t ) ∥ ∥ 2 can grow at a rate at most (1 + ηλ ⋆ ) , there exists a universal constant c 10 > 0 such that

Hence, for all T 1 ≤ s ≤ t , we have

An analysis on the recursive equation

proves that

(E.3) at ( t +1) Similar to the proof of (D.8), we have the decomposition

where x ( l ) ( τ ) = x ( l ) + τ ( x -x ( l ) ) . Both of the terms 1 and 2 can be bounded similar to 1 and 2 of x ( t +1) -˜ x ( t +1) as

for some universal constant c 11 > 0 . For the terms 3 and 5 , we use Lemma G.11 to obtain

From Lemmas A.1 and A.3, the term 4 is bounded as

Combining all with (E.7), there exists a universal constant c 12 > 0 such that

Hence, we have

for all T 1 ≤ s ≤ t . An analysis on the recursive equation

proves that

holds for some universal constant c 13 > 0 because T 2 -T 1 ≲ log n .

(E.4) at ( t +1) We use the same bound

that was used in the proof of (D.10). From (E.1) and (E.3), we have

Combined with the fact that λ ⋆ ∥ u ⋆ ∥ ∞ + ∥ ˜ x ∥ 2 ∥ ˜ x ∥ ∞ ≤ 3 λ ⋆ √ µ n , there exists a universal constant c 14 > 0 such that

Hence, for all T 1 ≤ s ≤ t , we have

and this implies

(E.2) at ( t +1) The l th component of x ( t +1) -˜ x ( t +1) is bounded by

F Fixed Initialization Size

In Section 3, we claimed that the estimation error is improved to 1 √ np + σ λ ⋆ √ n p if the initialization size is fixed to n -1 / 4 regardless of the sample complexity. We briefly discuss how the proofs should change in such a case. For clear presentation, µ and log n factors are ignored in this section.

For every bound of Phase I (Lemmas 5.1 to 5.5), 1 √ np is changed to 1 √ np + σ λ ⋆ √ n p , while allowing σ to be as large as λ ⋆ µ n √ np . More importantly, the definition of Phase I is changed to be the largest t such that (1 + ηλ ⋆ ) t ≤ √ n , so it is lengthened by √ np times than before. In the original proof, the estimation error of 1 √ np obtained at the end of Phase I was increased to 1 poly(log n ) during the first part of Phase II. However, if (1 + ηλ ⋆ ) t equals √ n at the end of Phase I, we do not have such a part in Phase II, and the estimation error obtained at the end of Phase I is maintained through Phase II.

G Technical Lemmas

We introduce some technical lemmas in this section. Most of them are the results of classical concentration inequalities.

Theorem G.1 (Matrix Bernstein Inequality) . Let { X i } be n × n independent symmetric random matrices. Assume that each random matrix satisfies E X i = 0 and ∥ X i ∥ ≤ L almost surely. Then, for all τ ≥ 0 , we have

where V = ∥ ∥ ∑ i E ( X 2 i ) ∥ ∥ .

Corollary G.2 (Matrix Bernstein Inequality) . Let { X i } be n × n independent symmetric random matrices. Assume that each random matrix satisfies E X i = 0 and ∥ X i ∥ ≤ L almost surely. Then, with high probability, we have

where V = ∥ ∥ ∑ i E ( X 2 i ) ∥ ∥ .

Lemma G.3. For any fixed matrix M ∈ R n × n , we have

with high probability.

Proof. We decompose the matrix into the sum of independent symmetric matrices.

We calculate L and V of Corollary G.2. We have L ≤ 1 p ∥ M ∥ ∞ because

We also have the following bound on V .

Hence, Corollary G.2 implies the desired result.

We can prove Lemma A.1 by applying Lemma G.3 to M ⋆ and using ∥ M ⋆ ∥ ∞ = λ ⋆ µ n .

We introduce classical Bernstein inequality and the results obtained from it.

Theorem G.4 (Bernstein Inequality) . Let { X i } be independent random variables. Assume that each random variable satisfies E X i = 0 and | X i | ≤ L almost surely. Then, for all τ ≥ 0 , we have

where V = ∑ i E [ X 2 i ] .

Corollary G.5 (Bernstein Inequality) . Let { X i } be independent random variables. Assume that each random variable satisfies E X i = 0 and | X i | ≤ L almost surely. Then, with high probability, we have

where V = ∑ i E [ X 2 i ] .

Lemma G.6. Let and { X i } be independent Bernoulli random variables with expectation p . Then, for any fixed vector a , we have

with high probability.

Lemma G.7. If n 2 p ≳ µn log n , we have

with high probability.

Proof. Let us consider ℓ 2 -norm of the i th row of M ◦ .

The third line follows from Lemma G.6.

Proof of Lemma A.2. The spectral norm of a symmetric matrix that has nonzero entries only on the l th row/column is bounded by twice of the norm of its l th row. Hence,

where the last inequality follows from Lemma G.7.

Lemma G.8. Let y be a vector that is independent from the sampling. Then, if n 2 p ≳ n log n , we have

with very high probability.

Proof. Let us fix i and decompose the difference as

The first term is bounded as

and the second term is bounded as

by Lemma G.6.

Lemma G.9. Let y be a vector that is independent from the sampling. Then, if n 2 p ≳ n log n , we have

with very high probability.

Proof. We have the following sequence of inequalities

where the second line is derived from a basic inequality ∥ A ∥ ≤ ∥| A |∥ that holds for any matrix A , and the last line follows by applying Lemma G.3 to 11 ⊤ and yy ⊤ .

Lemma G.10. Let y be a vector that is independent from the sampling. Then, if n 2 p ≳ n log n , we have

Proof. This follows directly from Lemmas G.8 and G.9.

Let us define an operator P Ω l such that an entry of P Ω l ( X ) is equal to that of X if it is contained both in the l th row/column and Ω , and otherwise 0 . We also define an operator P l that makes the entries outside the l th row/column zero. Then, we have

The following lemma was also introduced in [14], but we include the proof for completeness.

Lemma G.11. Suppose that a matrix M and a vector v are independent from sampling of the l th row/column. If n 2 p ≳ n log n , we have

with high probability.

Also, note that

Proof. If we consider the contribution of l th term and the other terms separately, we have

From Lemma G.6, we have

with high probability. Regarding the second term, notice that

Lemma G.6 implies that ∑ n i =1 δ il p ≍ n with high probability if n 2 p ≳ n log n . Hence, we have

and this finishes the proof.

Lemma G.12. Let M be a matrix and v , w be vectors that are independent from sampling of the l th row/column. Then, if n 2 p ≳ n log n , we have

Proof. We can consider the l th row and column separately by

If we apply Lemma G.6 to the summations, we get the desired result.

Lemma G.13. Let E be a symmetric matrix whose upper and on diagonal entries are drawn from Gaussian distribution N (0 , σ 2 ) independently. Let v be a vector that is independent from sampling of the l th row and column. Then, if n 2 p ≳ n log 2 n , we have

Proof. If we consider the contribution of l th term and the other terms separately, we have

For the first term, we will calculate V and L of Corollary G.5. V is calculated as

To find L , we first note that ∥ E l ∗ ∥ ∞ ≲ σ √ log n with high probability, where E l ∗ is the l th row of E . Thus, for all j ∈ [ n ] , we have

Corollary G.5 implies that the first term is bounded as

For the second term, it suffices to bound

As before, we obtain V and L through

Corollary G.5 implies that

Because ∑ n i =1 δ il ≍ np , we have

if n 2 p ≳ n log 2 n . Combining (G.1) and (G.2), we get the desired bound.

Lemma G.14. Let E be a symmetric matrix whose upper and on diagonal entries are drawn from Gaussian distribution N (0 , σ 2 ) independently. Let v , w be vectors that are independent from sampling of the l th row and column. Then, if n 2 p ≳ n log n , we have

Proof. We can consider the l th row and column separately by

We bound the two summations similar to (G.1) and for the last term, we note that | E ll | ≲ σ √ log n with high probability.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors proved the global convergence of the gradient descent algorithm with a small initialization for the rank-1 matrix completion problem.\n\nSTRENGTHS:\nThe results in this work is novel and should be interesting to audiences in optimization and machine learning fields. This work shows that the incoherence regularizer may be unnecessary for the matrix completion problem.\n\nWEAKNESSES:\nA more detailed comparison with \n\nChen, J., Liu, D., & Li, X. (2020). Nonconvex Rectangular Matrix Completion via Gradient Descent Without ℓ₂,∞ Regularization. IEEE Transactions on Information Theory, 66(9), 5806-5841.\n\nshould be included. In addition, the importance of analyzing the rank-1 case should be discussed in more detail.\n\nQUESTIONS:\n(1) Line 55: I would suggest the authors to be more specific on what parameters the convergence time has a logarithmic dependence on.\n\n(2) Line 78: it would be better to be consistent in using \"GD\" as the abbreviation of \"gradient descent\".\n\n(3) Line 86: it would be better to explicitly mention that u^* is a vector.\n\n(4) Line 102: I think the squared norm of x^0 is expected to be \\beta_0^2. But I am not sure if the norm of x^0 is expected to be \\beta_0.\n\n(5) Line 128: I think it may be better to include the global convergence result in a formal theorem (instead of a remark).\n\n(6) Line 143: besides the relation between T^* and \\beta_0, I wonder if there is a reason why \\beta_0 is lower bounded. It seems that the initialization size is not necessarily lower bounded in [17, 23]. It may be better to explain the reason why a lower bound is necessary.\n\n(7) Line 154: the remark on the estimation error could also be formalized as a theorem.\n\n(8) in Line 88, the authors mentioned that the incoherence at be at most poly(log n). But this condition is not included in Theorem 3.1. I wonder if this condition is necessary for the results.\n\n(9) Line 170: please be more specific on the meaning of \"incoherent up to a logarithmic factor\".\n\n(10) In my opinion, the discussion of proof ideas in Sections 4-5 is a little too long. It would be ideal if the length can be reduced by ~2 pages. With that said, I am okay with the current structure.\n\n(11) Line 319: \"an optimal number of samples\" is confusing. Please consider using a different word.\n\n(12) Another interesting open problem will be whether the results can be extended to the over-parameterized case, where a small initialization is also required.\n\n(13) Besides the aforementioned problem, it would also be interesting to consider the asymmetric matrix completion problem; see the follow-up work:\n\nSoltanolkotabi, M., Stöger, D., & Xie, C. (2023). Implicit balancing and regularization: Generalization and convergence guarantees for overparameterized asymmetric matrix sensing. arXiv preprint arXiv:2303.14244.\n\nLIMITATIONS:\nSee my comments in the previous section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies global convergence of GD (with a fixed step-size) for the rank-1 matrix completion problem (with symmetric and i.i.d. Bernoulli(p) observation and Gaussian noise on the entries) started from \"small\" random initialization. The authors prove that such vanilla GD, without any explicit regularization as commonly used in the literature, converges to the ground truth matrix in a polynomial time with near-optimal sample complexity. The result is interesting and the program is well-motivated. The analysis is mostly based on dynamical systems rather than optimization techniques, and a similar approach could be applied for related problems. \n\nThe proof idea seems to be novel (although some of the ideas seems to be motivated by recent literatures on GD with small initialization & stepwise for matrix factorization and matrix sensing problem). First, the authors first show that GD for the fully observed case with small initialization converges to the ground truth matrix (Corollary 1 in the appendix). This is possible due to an explicit formula for the GD iterates (a linear combination of the initialization and the ground truth with dynamic coefficients). The analysis is not very difficult, but also not so trivial. \n\nHaving established the desired convergence result for the fully observed GD dynamics, the main idea is to couple that with the partially observed GD dynamics by starting at the same initialization and use an interpolated dynamics eq. (13). The main novelty is in showing that these two trajectories remain close, and much closer than the norm of the iterates from the fully observed case. This allows one to show that the partially observed iterates converges to a local region near the ground truth after a polynomial number of iterations, and then one can use existing local convergence result [14].\n\nSTRENGTHS:\nThe main text is exceptionally well-written (with minor comments/suggestions) in that it gives the structure of the proof of the main result in a very clear manner (which was a pleasure to read for the most part). The argument in the main text is well-complemented with various simulation results. The general rank-r case was also discussed at the end, and the main difficulty of having the r singular values possibly growing at different rates is well-pointed out. It seems that the appendix gives a rigorous justification of most of the claims in the main text (although some references and pointers are missing). I've only read the appendix until Section C and cannot assess that the remainder (the coupling analysis, which is the most substantial part) is correct, but the sketch in the main text is convincing.\n\nWEAKNESSES:\nMinor comments: \n\nL98: \"To recover the matrix,\"  --> \"To recover the matrix $M^{\\star}$,\"\n\nL110: \"controlling the $\\ell_{\\infty}$-norm in [14]\" --> $\\ell_{\\infty}$-norm of what?\n\nNotation $\\lesssim$ is not defined \n\nL166: \".. linear combination of $x^{(0)}$ and $u^{\\star}$, ..\" --> \".. linear combination of $x^{(0)}$ and $u^{\\star}$ (see eq. (C.1) in the appendix)\" \n\nL214: \"becomes\" --> \"is\"\n\nL233 and L234: \"between\" --> \"of\"\n\nL259: \".. parallel to $u^{\\star}$.\" --> \".. parallel to $u^{\\star}$ (see Lem. A.5 in the appendix).\"\n\nL286: \".. in both $\\ell_{2}$ and $\\ell_{\\infty}$ norms, ..\" --> \".. in both $\\ell_{2}$ and $\\ell_{\\infty}$ norms (see Cor. 1 in the appendix), ..\"\n\nQUESTIONS:\neq. (8): Is it intentional to not to cancel out $n^{1/4}$ factor in the upper bound?\n\nL216: Shouldn't $\\tilde{x}^{(1)}$ here be $x^{(0)}$?\n\nLemma 5.5: $u^{(l)}$ is not defined. Same as $u^{\\star}$ but 0 at the $\\ell$th coordinate?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study a random initialization scheme for gradient descent applied to the problem of rank-one matrices, assuming a known ground truth. Their results concern global convergence properties of the with respect to a particular random model of partially observed matrices. Specifically, starting from a $n\\times n$ symmetric positive definite, fully-revealed ground truth matrix $\\bf{M}^\\star = \\lambda^\\star \\bf{u}^\\star (\\bf{u}^\\star)^T= \\bf{x}^\\star (\\bf{x}^\\star)^T$, entries above or on the diagonal are perturbed by noise drawn i.i.d. from a zero-mean Gaussian, and revealed independently with probability $p.$ The main result, Theorem 3.1, requires a coherence assumption: namely, for $\\lVert \\mathbf{u}^\\star \\rVert = \\sqrt{\\frac{\\mu}{n}},$ the quantity $\\mu$ is polynomially-bounded in $n.$ With this assumption, the main result claims that, if the Gaussian noise is sufficiently small and an initial iterate $\\mathbf{x}_0$ is provided whose magnitude is not too large or small according to quantities depending on $n, \\mu , \\lambda^\\star ,$ and $p,$ that gradient descent will converge to the given ground truth with probability tending to $1$ as $n\\to \\infty .$ In addition to giving proofs, the authors explain the qualitative behavior of convergence as determined by several phases, which appear in both the analysis and a simulation study.\n\nSTRENGTHS:\nIt seems that the combination of leave-one-out sequences developed in [14] and the random initialization used in areas like phase retrieval are a novel aspect of this work, although as mentioned in Sec. 6 there is difficulty in extending this combination to the case of arbitrary rank. Theorems and definitions are, for the most part, stated unambiguously, and I was unable to find any errors. The topic is clearly a good fit for NeuRIPS.\n\nWEAKNESSES:\nOne issue I have with this paper is that rank-1 symmetric matrix completion (as well as rank-1 general matrix completion) is simply a much easier problem than general matrix completion. \n\nIndeed, I would like to point out the reference \"Uniqueness of Low-Rank Matrix Completion by Rigidity theory\", by Singer and Cucuringu, which is not cited in this work. Section 5 of this paper shows that the existence of an exact completion is guaranteed (with \"probability one\") based purely on combinatorial conditions of the graph of revealed entries. By contrast, the authors make high-probability statements about matrices whose entries are drawn and hidden according to specific distributional assumptions as the sample size goes to infinity, which seem to be much stronger. In general, the bibliography could be more extensive.\n\nAdditionally, equation (8) shows that there are _lower_ bounds in addition to upper bounds on the magnitude of the initialization that is needed. Thus, merely a \"small\" initialization is not enough to ensure convergence, contrary to the title. Finally, there is a nontrivial coherence assumption. This may be standard in compressed sensing, but it is still a nontrivial assumption.\n\nIn summary, the assumptions are overly restrictive, and, unlike the arbitrarily matrix completion problem, rank-1 matrix completion is a fairly simple problem. So, I think the paper's claimed results are not very interesting overall.\n\nQUESTIONS:\nline 3: Do you mean \"simplest yet _most_ efficient?\"\n\nline 121: It would helpful to clarify that asymptotic notation like $o(1)$ refers to the regime $n\\to \\infty ,$ as opposed to other parameters tending towards infinity or zero.\n\nTheorem 3.1: It seems that a corollary of this theorem would be that $\\pm \\bf{x}^\\star$ are the only ground-truth solutions to the matrix completion problem. Is that something that is already assumed in the proof of your result? I don't see anywhere an explanation of why the cases with multiple ground truth solutions would be asymptotically rare.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the global convergence of vanilla GD for the rank-1 matrix completion problem. It is shown that with small random initialization and after a logarithmic number of steps, GD enters a region around the global minimizers in which linear convergence happens. The paper provides sufficient conditions on the initialization scale to ensure this phenomenon happens and shows a tradeoff between the initialization scale and the number of available samples. Illustrative simulations are provided.\n\nSTRENGTHS:\nThe paper studies a topic of interest for the Neurips community, which it presents in a clear manner and for which it provides results of both practical and theoretical importance. The derivations in the main paper are cleanly carried out, and the reasoning behind them is well-presented.\n\nWEAKNESSES:\n- The sample complexity seems to be quite large (for example, compared to that in [1]). In this light, can the authors elaborate further on this aspect (my question is despite the further commentary in section 6)? \n\n[1] C. Ma, K. Wang, Y. Chi, and Y. Chen, “Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval, matrix completion, and blind deconvolution,” Foundations of Computational Mathematics, vol. 20, no. 3, pp. 451–632, 2020.\n\nQUESTIONS:\n- The nature of the presented proofs lies very much in the detail. While the authors did a good job providing a higher-level view of the proof strategy, and the writing is clear, the text is still difficult to follow at times. A possible improvement would be to include some pictorial description of the phases -- similar to the one in section 4, for the analysis carried out in sections 5, 6. This can significantly aid understanding, in my opinion. \n\n- Figure 2 a: the green line is labelled as $\\\\| x^{(t)} - x^{\\star} \\\\|$, but in the figure commentary it is written $\\\\|x^{(t)} \\pm x^{\\star}\\\\|$. Which one is the correct one?\n\n- Figure 2 c: Perhaps instead the label \"Distance\" can be replaced with $\\\\|x^{(t)} \\pm x^{\\star}\\\\|$ for clarity.\n\nLIMITATIONS:\nThe limitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows some convergence properties of gradient descent with small random initialization for rank-1 noisy matrix completion.\n\nSTRENGTHS:\nThis paper uses a new approach (gradient descent with small random initialization) to solve the nonconvex formulation for rank-1 noisy matrix completion. It is shown that the GD trajectory will arrive at a local neighborhood (in both $\\ell_2$ and $\\ell_{\\infty$ norm) of the ground truth within a number of iteration.\n\nWEAKNESSES:\n1. My major concern of this paper is a lack of theoretical novelty. After looking at the results and quickly go through the proof, I believe the proof idea of this paper is similar to that in [1], which focus on a general low-rank matrix sensing problem (we know that the low-rank matrix sensing problem with certain RIP condition has the same population-level loss function as the noisy matrix completion problem). For example, the analysis idea that the GD dynamic is close to a simplified linear evolution system in the initial phase thanks to the small initialization already appeared in [1]. Compared with [1] which works for general low-rank setting, this paper only works for the rank-1 case, which is more restricted. On the other hand, matrix completion problems are known to be more difficult than the matrix sensing problem in the sense that it requires incoherence and $\\ell_{2,\\infty}$ error analysis to show that the empirical loss concentrates around the population counterpart. This difficulty was not encountered in [1]. This paper uses leave-one-out analysis that has been widely used in low-rank estimation literature like [2,3] to track the $\\ell_{\\infty}$ error of the GD trajectory. The analysis does not seems to be challenging, if one is familiar with the above-mentioned literature. If I underestimate the technical novelty of the paper, I hope the authors could clarify and please highlight their technical novelty.\n\n2. I am not satisfied with the convergence guarantees. The estimation error provided by Theorem 3.1 only shows that the output of the proposed algorithm is consistent, namely is $o(1)$. However state-of-the-art results for matrix completion already shows that GD with spectral initialization (which I believe is equivalent to small random initialization in some sense, since the initial phase of the latter algorithm is similar to some form of power method) achieves minimax-optimal estimation error [2]. Could the authors please explain why their analysis leads to looser error bound?\n\n3. In addition, the noise condition in Theorem 3.1 is $\\sqrt{np}$ times more stringent than that in [2]. Could the authors please explain why their analysis requires stronger noise conditions?\n\n[1] Li, Yuanzhi, Tengyu Ma, and Hongyang Zhang. \"Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations.\" Conference On Learning Theory. PMLR, 2018.\n\n[2] Ma, Cong, et al. \"Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution.\" Foundations of Computational Mathematics 20 (2020): 451-632.\n\n[3] Chen, Yuxin, et al. \"Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval.\" Mathematical Programming 176 (2019): 5-37.\n\nQUESTIONS:\nI have no question at this moment.\n\nLIMITATIONS:\nThis paper does not have potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the convergence analyses of gradient descent method for rank-one matrix completion problem. In particular, this work assumes small random initialization, which is relatively relaxed condition compared to existing work. With such assumption, the logarithmic convergence of the gradient descent method has been proved in this work. Also, the impact of the regularization for gradient descent method has been analyzed.\n\nSTRENGTHS:\nThe motivation of this work is clear and valid, also this work is well-organized. Meanwhile, this work is technically sound, where the proof and related analyses are provided, also some simulations are provided to further support the major results.\n\nWEAKNESSES:\n- I have concern for the novelty or the contribution of this work. Rank-one matrix completion problem is the simplest problem for matrix completion problems. Also there have constraints for the noise assumption of the revealed entries in this paper. For such cases, there are many efficient methods to deal with, like alternating minimization or projected gradient descent method. For gradient descent method, there also have existing works which have proved its convergence. Though this work provided with relaxed conditions, such contribution may be limited for the optimization and machine learning community, let alone the real problems in industry.\n- The presentation for the proof of the major results can be further improved. For instance, the proof of Lemma 5.1 is separated into two parts at different places, it may be better to combine them together after finishing other proofs.\n\nQUESTIONS:\n- Can you compare the major results about the rank-one matrix completion problem with the following works?\nhttps://arxiv.org/pdf/2008.04988.pdf\nhttps://proceedings.neurips.cc/paper/2020/file/f86890095c957e9b949d11d15f0d0cd5-Paper.pdf\n\nLIMITATIONS:\nPlease see comments above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}