Title: Lower Bounds on Adaptive Sensing for Matrix Recovery
Venue: NeurIPS 2023
Accepted: yes
Keywords: Compressed Sensing

Matrix Recovery

Low rank approximation
Abstract: We study lower bounds on adaptive sensing algorithms for recovering low rank matrices using linear measurements. Given an $n \times n$ matrix $A$, a general linear measurement $S(A)$, for an $n \times n$ matrix $S$, is just the inner product of $S$ and $A$, each treated as $n^2$-dimensional vectors. By performing as few linear measurements as possible on a rank-$r$ matrix $A$, we hope to construct a matrix $\hat{A}$ that satisfies $|A - \hat{A}|\_F^2 \le c |A|\_F^2$, for a small constant $c$. Here $|A|\_F$ denotes the Frobenius norm $(\sum_{i,j} A_{i,j}^2)^{1/2}$. It is commonly assumed that when measuring $A$ with $S$, the response is corrupted with an independent Gaussian random variable of mean $0$ and variance $\sigma^2$. Candès and Plan (IEEE Trans. Inform. Theory 2011) study non-adaptive algorithms for low rank matrix recovery using random linear measurements. They use the restricted isometry property (RIP) of Random Gaussian Matrices to give tractable algorithms to estimate $A$ from the measurements.

At the edge of the noise level where recovery is information-theoretically feasible, it is known that their non-adaptive algorithms need to perform $\Omega(n^2)$ measurements, which amounts to reading the entire matrix. An important question is whether adaptivity helps in decreasing the overall number of measurements. While for the related problem of sparse recovery, adaptive algorithms have been extensively studied, as far as we are aware adaptive algorithms and lower bounds on them seem largely unexplored for matrix recovery. We show that any adaptive algorithm that uses $k$ linear measurements in each round and outputs an approximation as in (1) with probability $\ge 9/10$ must run for $t = \Omega(\log(n^2/k)/\log\log n)$ rounds. Our lower bound shows that any adaptive algorithm which uses $n^{2-\beta}$ ($\beta > 0$ is arbitrary constant) linear measurements in each round must run for $\Omega(\log n/\log\log n)$ rounds. Our techniques also readily extend to obtain lower bounds on adaptive algorithms for tensor recovery. 

Our hard distribution also allows us to give a measurement-vs-rounds trade-off for many sensing problems in numerical linear algebra, such as spectral norm low rank approximation, Frobenius norm low rank approximation, singular vector approximation, and more.

Lower Bounds on Adaptive Sensing for Matrix Recovery

Abstract

We study lower bounds on adaptive sensing algorithms for recovering low rank matrices using linear measurements. Given an n × n matrix A , a general linear measurement S ( A ) , for an n × n matrix S , is just the inner product of S and A , each treated as n 2 -dimensional vectors. By performing as few linear measurements as possible on a rankr matrix A , we hope to construct a matrix ˆ A that satisfies

for a small constant c . Here ∥ A ∥ F denotes the Frobenius norm ( ∑ i,j A 2 i,j ) 1 / 2 . It is commonly assumed that when measuring A with S , the response is corrupted with an independent Gaussian random variable of mean 0 and variance σ 2 . Cand´ es and Plan (IEEE Trans. Inform. Theory 2011) study non-adaptive algorithms for low rank matrix recovery using random linear measurements. They use the restricted isometry property (RIP) of Random Gaussian Matrices to give tractable algorithms to estimate A from the measurements.

At the edge of the noise level where recovery is information-theoretically feasible, it is known that their non-adaptive algorithms need to perform Ω( n 2 ) measurements, which amounts to reading the entire matrix. An important question is whether adaptivity helps in decreasing the overall number of measurements. While for the related problem of sparse recovery, adaptive algorithms have been extensively studied, as far as we are aware adaptive algorithms and lower bounds on them seem largely unexplored for matrix recovery. We show that any adaptive algorithm that uses k linear measurements in each round and outputs an approximation as in (1) with probability ≥ 9 / 10 must run for t = Ω(log( n 2 /k ) / log log n ) rounds. Our lower bound shows that any adaptive algorithm which uses n 2 -β (for any constant β > 0 ) linear measurements in each round must run for Ω(log n/ log log n ) rounds to compute a good reconstruction with probability ≥ 9 / 10 . Hence any adaptive algorithm that has o (log n/ log log n ) rounds must use an overall Ω( n 2 ) linear measurements. Our techniques also readily extend to obtain lower bounds on adaptive algorithms for tensor recovery.

Our hard distribution also allows us to give a measurement-vs-rounds trade-off for many sensing problems in numerical linear algebra, such as spectral norm low rank approximation, Frobenius norm low rank approximation, singular vector approximation, and more.

1 Introduction

Sparse recovery, also known as compressed sensing, is the study of under-determined systems of equations subject to a sparsity constraint. Suppose we know that an unknown vector x ∈ R n

has at most r ≪ n non-zero entries. We would like to use a measurement matrix M ∈ R k × n to recover the vector x given measurements y = Mx . The number k of measurements is an important parameter we would like to optimize as it models the equipment cost in physical settings and running times in computational settings. Ideally one would like for the number k of measurements to scale proportionally to the sparsity of the unknown vector x .

One way of modeling sparse recovery is as 'stable sparse recovery'. Here we want a distribution M over k × n matrices such that for any x ∈ R n , with probability ≥ 1 -δ over M ∼ M , we can construct a vector ˆ x from M x such that

Note that the above formulation is more robust as it does not require the underlying vector x to be exactly r -sparse and instead asks to recover the top r coordinates of x .

For p = 2 , it is known that k = Θ( ε -1 r log( n/r )) measurements is both necessary and sufficient [6, 9, 22, 4]. See [23] and references therein for upper and lower bounds for other values of p .

In the same vein, the problem of low rank matrix recovery has been studied. See [5, 35] and references therein for earlier work and numerous applications. In this problem, the aim is to recover a low rank matrix A using linear measurements. Here we want a distribution M over linear operators M : R n × n → R k , such that for any low rank matrix A , with probability ≥ 1 -δ over M ∼ M , we can construct a matrix ˆ A from M ( A ) such that the reconstruction error ∥ A -ˆ A ∥ 2 F is small. Note that a linear operator M : R n × n → R k can be equivalently represented as a matrix M ∈ R k × n 2 such that M · vec ( A ) = M ( A ) where vec ( A ) denotes the appropriate flattening of the matrix A into a vector. We call the rows of M linear measurements . Without loss of generality, we can assume that the rows of the matrix M are orthonormal, as the responses for non-orthonormal queries can be obtained via a simple change of basis.

In addition, to model the measurement error that occurs in practice, it is a standard assumption that when querying with M , we receive M · vec( A ) + g , where g is a k -dimensional vector with independent Gaussian random variables of mean 0 and variance σ 2 , and we hope to reconstruct A with small error from M · vec( A ) + g . Clearly, when A has rank r , we need to perform Ω( nr ) linear measurements, as the matrix A has Θ( nr ) independent parameters. Hence, the aim is to perform not many more linear measurements than nr while being able to obtain an estimate ˆ A for A with low estimation error.

Given a rankr matrix A , [5] show that if the k × n 2 matrix M is constructed with independent standard Gaussian entries, then with probability ≥ 1 -exp( -cn ) , an estimate ˆ A can be constructed from M · vec( A ) + g such that ∥ A -ˆ A ∥ 2 F ≤ O ( nrσ 2 /k ) . They use the restricted isometry property (RIP) of the Gaussian matrix M to obtain algorithms that give an estimate ˆ A . The error bound is intuitive since the reconstruction error increases with increasing noise and proportionally goes down when the number of measurements k increases.

While we formulated the sparse recovery and matrix recovery problems in a non-adaptive way, there have been works which study adaptive algorithms for sparse recovery. Here we can produce matrices M ( i ) adaptively based on the responses received M (1) x, M (2) x, . . . , M ( i -1) x and the hope is that allowing for adaptive algorithms with a small number of adaptive rounds, we obtain algorithms that overall perform fewer linear measurements than non-adaptive algorithms with the same reconstruction error. It is additionally assumed that the noise across different rounds is independent. For sparse recovery, in the case of p = 2 , it is known that over O (log log n ) rounds adaptivity, a total of O ( ε -1 r log log( n log( n/r ))) linear measurements suffices [14, 21], improving over the requirement of Θ( ε -1 r log( n/r )) linear measurements for non-adaptive algorithms. For a constant ε , for any number of rounds, it is known that Ω( r +loglog n ) linear measurements are required [23]. Recently, [16] gave a lower bound on the total number of measurements if the number of adaptive rounds is constant.

While there has been a lot of interest in adaptive sparse recovery, both from the algorithms and the lower bounds perspective, the adaptive matrix recovery problem surprisingly does not seem to have any lower bounds. In adaptive matrix recovery, similar to adaptive sparse recovery, one is allowed to query a matrix M ( i ) in round i based on the responses received in the previous rounds, and again the hope is that with more rounds of adaptivity, the total number of linear measurements that is to

be performed decreases. There is some work that studies adaptive matrix recovery with 2 rounds of adaptivity (see [34] and references therein) but the full landscape of adaptive algorithms for matrix recovery seems unexplored.

̸

We address this from the lower bounds side in this work. We show lower bounds on adaptive algorithms that recover a rankr matrix of the form ( α/ √ n ) ∑ r i =1 u i v T i , where the coordinates of u i and v i are sampled independently from the standard Gaussian distribution. Without loss of generality 1 , we assume that the measurement matrix M ( i ) in each round i has orthonormal rows. We also assume that for i = j , the measurement matrices M ( i ) ( M ( j ) ) T = 0 , i.e., measurements across adaptive rounds are orthonormal, since non-orthonormal measurements can be reconstructed from orthonormal measurement matrices by a change of basis.

We now give an alternate way of looking at the adaptive sparse recovery problem: Fix a set of vectors u 1 , . . . , u r and v 1 , . . . , v r and let the underlying matrix we want to reconstruct be ( α/ √ n ) ∑ r i =1 u i v T i for a large enough constant α . In round 1 , we query a matrix M (1) ∈ R k × n 2 drawn from an appropriate distribution and receive the response vector r (1) = M (1) vec(( α/ √ n ) ∑ r i =1 u i v T i ) + g (1) where g (1) is a vector of independent Gaussian random variables with mean 0 and variance σ 2 . In round 2 , as a function of M (1) and r (1) and our randomness, we query a random matrix M (2) [ r (1) ] ∈ R k × n 2 and receive a response vector r (2) = ( M (2) [ r (1) ]) vec(( α/ √ n ) ∑ r i =1 u i v T i ) + g (2) where g (2) is again a vector with independent Gaussian entries and independent of g (1) , and so on. Crucially, using the rotational invariance of Gaussian random vectors, if G is an n × n matrix with independent Gaussian random variables with mean 0 and variance σ 2 , the response r (1) has the same distribution as ( M (1) ) · vec(( α/ √ n ) ∑ r i =1 u i v T i + G ) and as M (2) [ r (1) ] is chosen to be orthonormal to M (1) , the distribution of r (2) conditioned on r (1) is the same as that of ( M (2) [ r (1) ]) · (vec(( α/ √ n ) ∑ r i =1 u i v T i + G )) .

Thus, any adaptive matrix recovery algorithm can be seen as performing non-noisy adaptive measurements on the matrix ( α/ √ n ) ∑ r i =1 u i v T i + G where the Gaussian matrix G is sampled independently of the measurement algorithm. From the responses the algorithm receives, it then tries to reconstruct the matrix ( α/ √ n ) ∑ r i =1 u i v T i . This way of looking at adaptive sparse recovery immediately yields an adaptive algorithm: when the smallest singular value of ( α/ √ n ) ∑ r i =1 u i v T i is a constant times larger than ∥ G ∥ 2 , then the power method with a block size of r outputs an approximation of ( α/ √ n ) ∑ r i =1 u i v T i in O (log n ) rounds. Note that r matrix-vector products can be implemented using nr linear measurements. More generally, Gu [12] showed that using power iteration with a block size of k/n (for k ≥ nr ), we can obtain an approximation in O (log( n 2 /k )) adaptive rounds. Thus the already existing algorithms exhibit a no. of measurements vs no. of rounds trade-off.

From results in random matrix theory, we have that ∥ G ∥ 2 ≈ σ √ n with high probability. And as we are interested in reconstruction when the vectors u 1 , . . . , u r and v 1 , . . . , v r follow the Gaussian distribution we also have that ∥ u i ∥ 2 ≈ ∥ v i ∥ 2 ≈ √ n simultaneously with large probability. Thus, to make the extraction of ( α/ √ n ) ∑ r i =1 u i v T i information-theoretically possible, we need to assume that α ≳ σ . Hence, in this work we assume that σ = 1 and that α is a large enough constant, and we study lower bounds on adaptive matrix recovery algorithms.

If the vectors u 1 , . . . , u r and v 1 , . . . , v r are sampled from the standard n dimensional Gaussian distribution and r ≤ n/ 2 , we also have that with high probability ∥ ( α/ √ n ) ∑ r i =1 u i v T i ∥ 2 F ≈ α 2 nr. Now the algorithms of [5], which use a uniform random Gaussian matrix M with m rows as the measurement matrix, for ( α/ √ n ) ∑ i =1 u i v T i reconstruct a matrix ˆ A such that ∥ ˆ A -( α/ √ n ) ∑ r i =1 u i v T i ∥ 2 F ≤ C nrσ 2 m m where σ m is the measurement error. As we assumed above that σ = 1 when measuring with a matrix with orthonormal rows, we assume that the measurement error σ m when measuring with a Gaussian matrix is n , as each row of M has n 2 independent Gaussian coordinates and therefore has a norm ≈ n . Thus, using reconstruction algorithms from [5], we obtain a matrix ˆ A satisfying ∥ ˆ A -( α/ √ n ) ∑ r i =1 u i v T i ∥ 2 F ≤ C n 3 r m . Now, to make ∥ ˆ A -( α/ √ n ) ∑ r i =1 u i v T i ∥ 2 F ≤ (1 / 10) ∥ ( α/ √ n ) ∑ r i =1 u i v T i ∥ 2 F , we need to set m = Θ( n 2 /α 2 ) . Hence, in the parameter regimes we

study, the algorithms of [5] need to perform Ω( n 2 ) non-adaptive queries, which essentially means that they have to read a constant fraction of the n 2 entries in the matrix. While the power method performs O ( nr ) linear measurements in each round over O (log n ) adaptive rounds to output an approximation ˆ A . The question we ask is:

'Is the power method optimal? Are there algorithms that have o (log n ) adaptive rounds and use n 2 -β measurements in each round?'

We answer this question by showing that any algorithm that has o (log n/ log log n ) adaptive rounds must use ≥ n 2 -β linear measurements, for any constant β > 0 , in each round. The lower bound shows that power method is essentially optimal if we want to use n 2 -β linear measurements in each round and that any algorithm with o (log n/ log log n ) adaptive rounds essentially reads the whole matrix.

We further obtain a rounds vs measurements trade-off for many numerical linear algebra problems in the sensing model. In this model, there is an n × n matrix A with which we can interact only using general linear measurements and we want to solve problems such as spectral norm low rank approximation of A , Frobenius norm low rank approximation of A , etc. In general, numerical linear algebra algorithms assume that they either have access to the entire matrix or that the matrix is accessible in the matrix-vector product model where one can query a vector v and obtain the result A · v . Recently, the vector-matrix-vector product model has received significant attention as well. Linear measurements are more powerful than both the matrix-vector product model and the vector-matrix-vector product model. Any matrix vector product A · v can be computed using n linear measurements of A and any vector-matrix-vector product u T Av can be computed using a single linear measurement of A . Thus, the model of general linear measurements may lead to faster algorithms.

For certain problems in numerical linear algebra, general linear measurements are significantly more powerful than the matrix-vector product model. Indeed, for computing the trace of an n × n matrix A , one can do this exactly with a single deterministic general linear measurement, just by adding up the diagonal entries of A . However, in the matrix-vector product model, it is known that Ω( n ) matrix-vector products are needed to compute the trace exactly [29], even if one uses randomization. A number of problems were studied in the vector-matrix-vector product model in [24], and in [32] it was shown that to approximate the trace of A up to a (1 + ε ) -factor with probability 1 -δ , one needs Ω( ε -2 log(1 /δ )) queries. This contrasts sharply with the single deterministic general linear measurement for computing the trace exactly. Thus, there are good reasons to conjecture that general linear measurements may lead to algorithms requiring fewer rounds of adaptivity compared to algorithms in the matrix-vector product query model. Surprisingly, our lower bounds show that for many numerical linear algebra problems, linear measurements do not give much advantage over matrix-vector products.

1.1 Our Results

We assume that there is an unknown rankr matrix A ∈ R n × n to be recovered. Given any linear measurement q ∈ R n 2 , we receive a response ⟨ q, vec( A ) ⟩ + g , where g ∼ N (0 , ∥ q ∥ 2 2 ) . We further assume that the noise for two different measurements is independent. Without loss of generality, we also assume throughout that all the queries an algorithm makes across all rounds form a matrix with orthonormal rows. Our main result for adaptive matrix sensing is as follows:

Theorem 1.1. There exists a constant c such that any randomized algorithm which makes k ≥ nr noisy linear measurements of an arbitrary rankr matrix A with ∥ A ∥ 2 F = Θ( nr ) in each of t rounds, and outputs an estimate ˆ A satisfying ∥ A -ˆ A ∥ 2 F ≤ c ∥ A ∥ 2 F with probability ≥ 9 / 10 over the randomness of the algorithm and the Gaussian noise, requires t = Ω(log( n 2 /k ) / (log log n )) .

Dependence on noise In our results, we assumed that given a linear measurement q ∈ R n 2 , the response is distributed as N ( ⟨ q, vec( A ) ⟩ , ∥ q ∥ 2 2 ) . Our lower bounds also hold when the response is distributed as N ( ⟨ q, vec( A ) ⟩ , σ 2 ∥ q ∥ 2 2 ) for any parameter σ such that c ′ ≤ σ ≤ 1 , where c ′ is a constant. This can be seen by simply scaling the matrix A in the theorem above and adjusting the constants while proving the theorem.

Tensor Recovery The problem of tensor recovery with linear measurements has also been studied (see [25, 11] and references therein) where given a low rank tensor, the task is to recover an approximation to the tensor with few linear measurements. Our techniques can also be used to obtain lower bounds on adaptive algorithms for tensor recovery. Our main tool, Lemma 3.1, readily extends to tensors of higher orders by using the corresponding tail bounds from [31].

Numerical Linear Algebra We also derive lower bounds for many numerical linear algebra problems in the linear measurements model. Table 1 shows our lower bounds on the number of adaptive rounds required for any randomized algorithm using k general linear measurements in each round. See Appendix E for precise statements and proofs.

1.2 Notation

Throughout the paper, we use uppercase letters such as G,M,Q to denote matrices and lowercase letters such as u, v to denote vectors. For vectors u, v ∈ R n , u ⊗ v ∈ R n 2 denotes the tensor product of u and v . For an arbitrary matrix M ∈ R m × n , the vector vec( M ) ∈ R mn denotes a flattening of the matrix M with the convention that vec( uv T ) = u ⊗ v . We use boldface symbols such as M , G , u , v to denote that these objects are explicitly sampled from an appropriate distribution. We use g k to denote a multivariate Gaussian in k dimensions where each coordinate is independently sampled from N (0 , 1) . We also use G ( j ) to denote a collection of j independent multivariate Gaussian random variables of appropriate dimensions.

1.3 Our Techniques

Using the rotational invariance of the Gaussian distribution, we argued that any adaptive randomized low rank matrix recovery algorithm with access to a hidden matrix ( α/ √ n ) ∑ r i =1 u i v T i using noisy linear measurements can be seen as a randomized algorithm that has access to a random matrix ( α/ √ n ) ∑ r i =1 u i v T i + G using perfect linear measurements where each coordinate of G is independently sampled from a Gaussian distribution.

Let A be any algorithm that satisfies the matrix recovery guarantees with, say, a success probability ≥ 9 / 10 . Let A ( X, σ , γ ) be the output of the randomized algorithm A , where the hidden matrix is X , the random seed of A is denoted by σ , and γ denotes the measurement randomness. We have that if X has rank r and satisfies ∥ X ∥ 2 F = Θ( nr ) , then

We say A ( X, σ , γ ) is correct when the output ˆ X satisfies ∥ ˆ X -X ∥ 2 F ≤ (1 / 100) ∥ X ∥ 2 F . By the above reduction, from A we have a randomized algorithm A ′ that runs on the random matrix X + G with access to exact linear measurements and outputs a correct reconstruction ˆ X with probability ≥ 9 / 10 if X has rank r and ∥ X ∥ 2 F = Θ( nr ) . Thus, for all such X ,

Here σ denotes the randomness used by the algorithm A ′ . Now, if X is a random matrix constructed as ( α/ √ n ) ∑ r i =1 u i v T i with u i , v i being random vectors with independent Gaussian entries of mean 0 and variance 1 , then with probability ≥ 99 / 100 , ∥ X ∥ 2 F = Θ( nr ) . Thus, Pr X , G ,σ [ A ′ ( X + G , σ ) is correct ] ≥ 8 / 10 . Hence, there exists some fixed σ such that

Thus, the existence of a randomized algorithm that solves low rank matrix recovery as in Theorem 1.1 implies the existence of a deterministic algorithm which given access to perfect linear measurements of random matrix ( α/ √ n ) ∑ r i =1 u i v T i + G outputs a reconstruction of ( α/ √ n ) ∑ r i =1 u i v T i with probability ≥ 8 / 10 . This is essentially a reduction from randomized algorithms to deterministic algorithms using Yao's lemma. From here on, we prove lower bounds on such deterministic algorithms and conclude the lower bounds in Theorem 1.1. For simplicity, we explain the proof of our lower bound for r = 1 here and extend to general r later. We consider the random matrix G +( α/ √ n ) uv T and show how the lower bound proof proceeds.

Note that the matrix A := G + ( α/ √ n ) uv T ∈ R n × n can be flattened to the vector a = g + ( α/ √ n )( u ⊗ v ) ∈ R n 2 . Also, a general linear measurement, which we call a query Q ∈ R n × n , can be vectorized to q = vec( Q ) ∈ R n 2 with Q ( A ) = ⟨ q, a ⟩ . Fix any deterministic algorithm. In the first round, the algorithm starts with a fixed matrix Q (1) ∈ R k × n 2 that corresponds to the k queries and receives the response r (1) := Q (1) a . Then, as a function of the response r (1) in the first round, the algorithm picks a matrix Q (2) [ r (1) ] in the second round and receives the response r (2) := Q (2) [ r (1) ] · a . Similarly, in the i -th round, the deterministic algorithm picks a matrix Q ( i ) [ r (1) , . . . , r ( i -1) ] ∈ R k × n 2 as a function of r (1) , . . . , r ( i -1) and receives the response r ( i ) := Q ( i ) [ r (1) , . . . , r ( i -1) ] · a . Note that we assumed that the query matrices Q ( i ) chosen by the algorithm have orthonormal rows and also that Q ( i ) ( Q ( j ) ) T = 0 , i.e., the queries across rounds are also orthonormal.

For a fixed u, v ∈ R n , we see that the response r (1) = Q (1) g +( α/ √ n ) Q (1) ( u ⊗ v ) . As the matrix Q (1) has orthonormal rows, the random variable Q (1) g ≡ g k , where g k ∼ N (0 , I k ) is drawn from a mean-zero normal distribution with identity covariance. Thus, for fixed u, v , the distribution of the first round responses to the algorithm is N (( α/ √ n ) Q (1) ( u ⊗ v ) , I k ) . Now the key observation is that ∥ ( α/ √ n ) Q (1) ( u ⊗ v ) ∥ 2 2 = Θ( α 2 k/n ) with high probability over the inputs ( u , v ) . This uses a recent concentration result for random tensors due to [31], and critically uses the fact that Q (1) has operator norm 1 and Frobenius norm √ k . This means that for a large fraction of ( u, v ) pairs, the distribution of the responses seen by the algorithm in the first round is close to N (0 , I k ) , and therefore just looking at the response r (1) , the algorithm cannot have a lot of 'information' about which ( u, v ) pair is involved in the matrix that is unknown to the algorithm. So the query chosen in the second round Q (2) [ r (1) ] cannot have a 'large' value of Q (2) [ r (1) ]( u ⊗ v ) for most inputs ( u, v ) , with a high probability over the Gaussian component of the matrix.

Suppose the value of Q (2) [ r (1) ]( u ⊗ v ) is small. Again, r (2) = Q (2) [ r (1) ] g +( α/ √ n ) Q (2) [ r (1) ]( u ⊗ v ) . Crucially, as the queries in round 2 are orthogonal to the queries in round 1 , we have by the rotational invariance of the Gaussian distribution that Q (2) [ r (1) ] g is independent of Q (1) g , and that Q (2) [ r (1) ] g is distributed as N (0 , I k ) . So, for a fixed u, v , conditioned on the first round response r (1) , the distribution of the second round response r (2) is given by N (( α/ √ n ) Q (2) [ r (1) ]( u ⊗ v ) , I k ) ≈ N (0 , I k ) using the assumption that Q (2) [ r (1) ]( u ⊗ v ) is small. We again have that for a large fraction of pairs ( u, v ) for which Q (2) [ r (1) ]( u ⊗ v ) is small, the distribution of the second round response is also close to N (0 , I k ) . Therefore, the algorithm again does not gain a lot of information about which ( u, v ) pair is involved in the matrix, and the third round query Q (3) [ r (1) , r (2) ] cannot have a large value of ∥ Q (3) [ r (1) , r (2) ]( u ⊗ v ) ∥ 2 . The proof proceeds similarly for further rounds.

To formalize the above intuitive idea, we use Bayes risk lower bounds [7]. We show that with a large probability over the input matrix, the squared projection of u ⊗ v onto the query space of the algorithm is upper bounded and we use an iterative argument to show that an upper bound on the information up until round i can in turn be used to upper bound the information up until round i +1 . Bayes risk bounds are very general and let us obtain upper bounds on the information learned by a deterministic learner. Concretely, let Θ be a parameter space and P = { P θ : θ ∈ Θ } be a set

of distributions, one for each θ ∈ Θ . Let w be a distribution over Θ . We sample θ ∼ w and then x ∼ P θ and provide the learner with x . Given an action space A , the learner uses a deterministic decision rule d : X → A to minimize a 0 -1 loss function L : Θ × A → { 0 , 1 } in expectation, i.e., E θ ∼ w [E x ∼ P θ L ( θ , d ( x ))] . Let R Bayes ( L, Θ , w ) = inf d E θ ∼ w [ E x ∼ P θ L ( θ , d ( x ))] be the loss achievable by the best deterministic decision rule d . Bayes risk lower bounds let us obtain lower bounds on R Bayes .

In our setting after round 1 , we have Θ = { ( u, v ) : u, v ∈ R n } , w is the joint distribution of two independent standard Gaussian random variables in R n and for each ( u, v ) ∈ Θ we let P uv be the distribution of r (1) = g k + ( α/ √ n ) Q (1) ( u ⊗ v ) , an action space A of all k × n 2 matrices with orthonormal rows (corresponding to the queries in the next round), and define a 0 -1 loss function

for an appropriate threshold parameter T . By setting T appropriately as a function of t , we obtain a Bayes risk lower bound of R Bayes ≥ 1 -1 / (100 t 2 ) . Thus, we obtain that for any deterministic decision rule d , with probability ≥ 1 -1 / 10 t over ( u , v ) ∼ w , we have

and in particular, we have that with probability ≥ 1 -1 / 10 t over ( u , v ) ∼ w ,

The above statement essentially says that with probability ≥ 1 -1 / 10 t over the inputs ( u , v ) , the second query Q (2) [ r (1) ] chosen by the deterministic algorithm has the property that ∥ Q (2) [ r (1) ]( u ⊗ v ) ∥ 2 2 < T with probability ≥ 1 -1 / 10 t over the Gaussian G . In the second round, we restrict our analysis of the algorithm to only those ( u, v ) which satisfy (2). We again define a distribution w ′ over the inputs and for each such ( u, v ) we define a distribution P uv over the round 1 and round 2 responses received by the algorithm. We define a new loss function with parameter T ′ = ∆ · T for a multiplicative factor ∆ and again obtain a statement similar to (2) for a large fraction of inputs ( u, v ) and repeat a similar argument for t rounds and show that there is an Ω(1) fraction of the inputs for which the squared norm of the projection of u ⊗ v onto the query space after t rounds is bounded by T ( t ) with high probability over the Gaussian part of the input. This gives the result in Theorem 3.2. Note that ∥ u ⊗ v ∥ 2 2 = Ω( n 2 ) with high probability and for any fixed matrix Q with k orthonormal rows, E[ ∥ Q ( u ⊗ v ) ∥ 2 2 ] = k which corresponds to the amount of 'information' the algorithm starts with. As we show that the 'information' in each round grows by some multiplicative factor ∆ , a number Ω(log ∆ ( n 2 /k )) of rounds is required to obtain an 'information' of Θ( n 2 ) , which is how we obtain our lower bounds. Here information is measured as the squared projection of u ⊗ v onto the query space of the algorithm.

We also extend our results to identifying a rank r spike (sum of r random outer products) corrupted by Gaussian noise. Specifically, we consider the random matrix M = G + ( α/ √ n ) ∑ r i =1 u i v T i where all the coordinates of G , u i , v i are independently sampled from N (0 , 1) . We show that if there is an algorithm that uses k iterations in each round and identifies the spike with probability ≥ 9 / 10 , then it must run for Ω(log( n 2 /k ) / log log n ) rounds by appealing to the lower bound we described above for the case of r = 1 . We show that if there is an algorithm for r > 1 that requires t < O (log( n 2 /k ) / log log n ) adaptive rounds, then it can be used to solve the rank 1 spike estimation problem in t rounds as well which contradicts the lower bound.

We then provide lower bounds on approximate algorithms for a host of problems such as spectral norm low rank approximation (LRA), Schatten norm LRA, Ky-Fan norm LRA and reduced rank regression, by showing that algorithms to solve these problems can be used to estimate the spike uv T in the random matrix G + ( α/ √ n ) uv T , and then use the aforementioned lower bounds on algorithms that can estimate the spike. Although our hard distribution is supported on non-symmetric matrices, we are still able to obtain lower bounds for algorithms for spectral norm LRA (rank r ≥ 2 ) for symmetric matrices as well by considering a suitably defined symmetric random matrix using our hard distribution. Let A := G +( α/ √ n ) uv T be the hard distribution in the case of rank r = 1 .

We symmetrize the matrix by considering, A sym = [ 0 A A T 0 ] . This symmetrization, as opposed

to AA T or A T A , has the advantage that a linear measurement of A sym can be obtained from an appropriately transformed linear measurement of A , thereby letting us obtain lower bounds for symmetric instances as well in the linear measurements model. However, we cannot obtain lower bounds for rank 1 spectral norm LRA for symmetric matrices using this symmetrization as the top two singular values of A sym are equal and hence even a zero matrix is a perfect rank 1 spectral norm LRA for A sym which does not give any information about the plant u ⊗ v .

1.4 Related Work

As discussed, low rank matrix recovery has been extensively studied (see [5] and references therein for earlier work). Relatedly, [13] study the Robust PCA problem where the aim is to estimate the sum of a low rank matrix and a sparse matrix from noisy vector products with the hidden matrix. [30] study the Robust PCA problem when given access to linear measurements with the hidden matrix.

For non-adaptive algorithms for low rank matrix recovery with Gaussian errors, [5] show that their selectors based on the restricted isometry property of measurement matrices are optimal up to constant factors in the minimax error sense when the noise follows a Gaussian distribution. Our Theorem 1.1 extends their lower bounds and shows that if there is any randomized measurement matrix 2 M with k rows coupled with a recovery algorithm that outputs a reconstruction for any rank r matrix with ∥ A ∥ 2 F = Θ( nr ) , then it must have k = Ω( n 2 -o (1) ) . We again note that we give lower bounds even for algorithms with multiple adaptive rounds.

Our technique to show lower bounds is to plant a low rank matrix ( α/ √ n ) ∑ r i =1 u i v T i in an n × n Gaussian matrix so that any 'orthonormal' access to the plant is corrupted by independent Gaussian noise. Notably this technique has been employed to obtain lower bounds on adaptive algorithms for sparse recovery in [23]. Even in the non-adaptive setting, Li and Woodruff [18] use the same hard distribution as we do to obtain sketching lower bounds for approximating Schatten p norms, the operator norm, and the Ky Fan norms. The technique to show their lower bounds is that if a sketching matrix has k ≤ c/ ( r 2 s 4 ) rows 3 , it cannot distinguish between the random matrix G and the random matrix G + s ∑ r i =1 u i v T i where all the coordinates of G , u i , v i are drawn uniformly at random. They prove this by showing that d TV ( L 1 , L 2 ) is small if L 1 is the distribution of M · vec( G ) and L 2 is the distribution of M vec( G + s ∑ n i =1 u i v T i ) for any fixed measurement matrix M with k ≤ c/ ( r 2 s 4 ) rows. Their techniques do not extend to the plant estimation in the distribution G + s ∑ r i =1 u i v T i as the statement they prove only says that over the randomness of G , u i , v i , the response distribution for G + s ∑ r i =1 u i v T i is close to the response distribution for G , but in our case, we want the distributions L u,v and L u ′ ,v ′ to be indistinguishable, where L u,v is the response distribution for G +( α/ √ n ) ∑ i =1 u i v T i .

Later, [28] considered the distribution of the symmetric random matrix W + λ UU T , where W is the n × n symmetric Wigner matrix ( G + G T ) / √ 2 and U is a uniformly random n × r matrix with orthonormal columns. They focus on obtaining lower bounds on adaptive algorithms that estimate the spike U in the matrix-vector product model. In particular, they show that if Q is a basis for the query space spanned by any deterministic algorithm after querying k adaptive matrix-vector queries, then λ r ( Q T UU T Q ) grows as ∼ λ k/r . Using this, they show that any algorithm which, given access to an arbitrary symmetric matrix A in the matrix-vector product query model, must use Ω( r log( n ) / √ gap ) adaptive queries to output an n × r orthonormal matrix V satisfying, for a small enough constant c ,

where gap = ( λ r ( A ) -λ r +1 ( A )) /λ r ( A ) . We note the above guarantee is non-standard in the low rank approximation literature, which instead focuses more on approximation algorithms for Frobenius norm and spectral norm LRA. While in the matrix-vector product model, their hard distribution helps in getting lower bounds for numerical linear algebra problems on symmetric matrices, it seems that our hard distribution G + ( α/ √ n ) ∑ r i =1 u i v T i is easier to understand in the linear measurement model and gives the important property that the noise between rounds is independent, which is

what lets us reduce the matrix-recovery problem with noisy measurements to spike estimation in G +( α/ √ n ) ∑ r i =1 u i v T i .

Recently, Bakshi and Narayanan [1] obtained a tight lower bound for rank1 spectral norm low rank approximation problem in the matrix-vector product model. They show that Ω(log n/ √ ε ) matrix vector products are required to obtain a 1 + ε spectral norm low rank approximation. We stress that while our results are not for 1 + ε approximations, they hold in the stronger linear measurements model.

2 Notation and Preliminaries

Given an integer n ≥ 1 , we use [ n ] to denote the set { 1 , . . . , n } . For a vector v ∈ R n , ∥ v ∥ 2 := ( ∑ i ∈ [ n ] v 2 i ) 1 / 2 denotes the Euclidean norm. For a matrix A ∈ R n × d , ∥ A ∥ F denotes the Frobenius norm ( ∑ i,j A 2 ij ) 1 / 2 and ∥ A ∥ 2 denotes the spectral norm sup x =0 ∥ Ax ∥ 2 / ∥ x ∥ 2 . For p ≥ 2 , we use ∥ A ∥ S p to denote the Schattenp norm ( ∑ i σ p i ) 1 /p and for p ∈ [min( n, d )] , we use ∥ A ∥ F p to denote the Ky-Fan p norm ∑ p i =1 σ i , where σ 1 , . . . , σ min( n,d ) denote the singular values of the matrix A . Given a parameter k , the matrix [ A ] k denotes the best rank k approximation for matrix A in Frobenius norm. By the Eckart-Young-Mirsky theorem, the best rank k approximation under any unitarily invariant norm, which includes the spectral norm, Frobenius norm, Schatten norms and Ky-Fan norms, is given by truncating the Singular Value Decomposition of the matrix A to top k singular values. We describe the Bayes Risk Lower Bounds that we use to prove Lemma 3.1 in the Appendix. Our presentation is based on [7]. Due to space constraints, we have placed all proofs in the appendix.

̸

3 Number of Linear Measurements Versus Number of Adaptive Rounds

We now state the main theorem which shows a lower bound on the number of adaptive rounds required for any deterministic algorithm to estimate the plant when the input is a random matrix G +( α/ √ n ) ∑ r i =1 u i v T i . We use this theorem to prove Theorem 1.1 and lower bounds for other numerical linear algebra problems.

We prove the lower bound for the rankr plant estimation by first proving a lower bound on the rank1 plant estimation and then reducing the rank1 recovery problem to rankr recovery problem. For the rank1 recovery problem, we prove the following lemma:

Lemma 3.1. Given an integer n , and parameters α ≥ 10 and γ ≥ 1 , define the n × n random matrix M = G + s uv T for s := α/ √ n , where the entries of G , u , v are drawn independently from the distribution N (0 , 1) . Let Alg be any t -round deterministic algorithm that makes k ≥ n adaptive linear measurements in each round. Let Q ( j ) ∈ R k × n 2 denote the matrix of general linear queries made by the algorithm in round j and Q be a matrix with orthonormal rows such that rowspan ( Q ) = rowspan ( Q (1) ) + . . . + rowspan ( Q ( t ) ) . Then for all t such that O ( k log( n )) ≤ ( Kα 2 γ 2 ) t k ≤ O ( n 2 ) for a universal constant K ,

Setting γ = O (log( n )) , the theorem shows that if t ≤ c log( n 2 /k ) / (log log( n ) + log( α )) for a small enough constant c , then for any t -round deterministic algorithm,

Setting γ = O (1) , we obtain that if t ≤ c log( n 2 /k ) / (log( α )) for a small enough constant c , then

The proof of the above lemma is in Appendix B. The above lemma directly shows lower bounds on any deterministic algorithm that can approximate the rank1 planted matrix. We show that the lower bounds can be extended to algorithms that estimate the rankr plant as well.

Theorem 3.2. Let n and r ≤ n/ 2 be input parameters and α be a large enough constant. Let the random matrix G + ( α/ √ n ) ∑ r i =1 u i v T i be the input which can be accessed using linear measurements. If Alg is a t -round adaptive algorithm that uses k linear measurements and at the end of t -rounds outputs a matrix ˆ A such that with probability ≥ 9 / 10 over the randomness of the input and the internal randomness of the algorithm,

for a small enough constant c , then t = Ω(log( n 2 /k ) / (log( α ) + log log n )) .

Using the reduction from matrix recovery with noisy measurements to plant estimation with exact measurements that we described in the previous sections, we can now prove Theorem 1.1. The proof of Theorem 1.1 is in Appendix D.

References

Ainesh Bakshi and Shyam Narayanan. Krylov methods are (nearly) optimal for low-rank approximation. arXiv preprint arXiv:2304.03191 , 2023. 9

Ainesh Bakshi, Kenneth L. Clarkson, and David P. Woodruff. Low-rank approximation with 1 /ε 1 / 3 matrix-vector products. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , STOC 2022, 2022. 25

Mark Braverman, Elad Hazan, Max Simchowitz, and Blake Woodworth. The gradient complexity of linear regression. In Conference on Learning Theory , pages 627-647. PMLR, 2020. 24

Emmanuel J. Cand` es and Mark A. Davenport. How well can we estimate a sparse vector? Appl. Comput. Harmon. Anal. , 34(2):317-323, 2013. ISSN 1063-5203. doi: 10.1016/j.acha.2012.08. 010. 2

Emmanuel J Cand` es and Yaniv Plan. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory , 57(4):2342-2359, 2011. 2, 3, 4, 8

Emmanuel J. Cand` es, Justin K. Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Comm. Pure Appl. Math. , 59(8):1207-1223, 2006. ISSN 0010-3640. doi: 10.1002/cpa.20124. 2

Xi Chen, Adityanand Guntuboyina, and Yuchen Zhang. On bayes risk lower bounds. Journal of Machine Learning Research , 17(218):1-58, 2016. 6, 9, 13

Simon Foucart and Holger Rauhut. A mathematical introduction to compressive sensing . Applied and Numerical Harmonic Analysis. Birkh¨ auser/Springer, New York, 2013. ISBN 978-0-8176-4947-0; 978-0-8176-4948-7. doi: 10.1007/978-0-8176-4948-7. 29

Anna C. Gilbert, Yi Li, Ely Porat, and Martin J. Strauss. Approximate sparse recovery: optimizing time and measurements. SIAM J. Comput. , 41(2):436-453, 2012. ISSN 0097-5397. doi: 10.1137/100816705. 2

Gene H Golub, Franklin T Luk, and Michael L Overton. A block lanczos method for computing the singular values and corresponding singular vectors of a matrix. ACM Transactions on Mathematical Software (TOMS) , 7(2):149-169, 1981. 26

Rachel Grotheer, Shuang Li, Anna Ma, Deanna Needell, and Jing Qin. Iterative hard thresholding for low cp-rank tensor models. Linear and Multilinear Algebra , pages 1-17, 2021. 5

Ming Gu. Subspace iteration randomization and singular value problems. SIAM Journal on Scientific Computing , 37(3):A1139-A1173, 2015. 3, 23, 26

Wooseok Ha and Rina Foygel Barber. Robust pca with compressed data. Advances in Neural Information Processing Systems , 28, 2015. 8

Piotr Indyk, Eric Price, and David P. Woodruff. On the power of adaptivity in sparse recovery. In IEEE 52nd Annual Symposium on Foundations of Computer Science, FOCS 2011 . IEEE Computer Society, 2011. 2

Praneeth Kacham and David Woodruff. Reduced-rank regression with operator norm error. In Conference on Learning Theory , pages 2679-2716. PMLR, 2021. 27

Akshay Kamath and Eric Price. Adaptive sparse recovery with limited adaptivity. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2729-2744. SIAM, Philadelphia, PA, 2019. doi: 10.1137/1.9781611975482.169. 2

Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics , pages 1302-1338, 2000. 15, 20, 22

Yi Li and David P. Woodruff. Tight bounds for sketching the operator norm, Schatten norms, and subspace embeddings. In Approximation, randomization, and combinatorial optimization. Algorithms and techniques , volume 60 of LIPIcs. Leibniz Int. Proc. Inform. , pages Art. No. 39, 11. Schloss Dagstuhl. Leibniz-Zent. Inform., Wadern, 2016. 8

Yi Li, Huy L. Nguyen, and David P. Woodruff. On sketching matrix norms and the top singular vector. In Proceedings of the Twenty-Fifth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2014, Portland, Oregon, USA, January 5-7, 2014 , pages 1562-1581. SIAM, 2014. 24

Cameron Musco and Christopher Musco. Randomized block krylov methods for stronger and faster approximate singular value decomposition. Advances in neural information processing systems , 28, 2015. 26, 27

Vasileios Nakos, Xiaofei Shi, David P. Woodruff, and Hongyang Zhang. Improved algorithms for adaptive compressed sensing. In 45th International Colloquium on Automata, Languages, and Programming , volume 107 of LIPIcs. Leibniz Int. Proc. Inform. , pages Art. No. 90, 14. Schloss Dagstuhl. Leibniz-Zent. Inform., Wadern, 2018. 2

Eric Price and David P. Woodruff. (1 + ε ) -approximate sparse recovery. In 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science-FOCS 2011 , pages 295-304. IEEE Computer Soc., Los Alamitos, CA, 2011. doi: 10.1109/FOCS.2011.92. 2

Eric Price and David P. Woodruff. Lower bounds for adaptive sparse recovery. In Proceedings of the Twenty-Fourth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 652-663. SIAM, Philadelphia, PA, 2012. 2, 8

Cyrus Rashtchian, David P. Woodruff, and Hanlin Zhu. Vector-matrix-vector queries for solving linear algebra, statistics, and graph problems. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM 2020 , volume 176 of LIPIcs , pages 26:1-26:20. Schloss Dagstuhl - Leibniz-Zentrum f¨ ur Informatik, 2020. 4

Holger Rauhut, Reinhold Schneider, and ˇ Zeljka Stojanac. Low rank tensor recovery via iterative hard thresholding. Linear Algebra Appl. , 523:220-262, 2017. ISSN 0024-3795. doi: 10.1016/j.laa.2017.02.028. 5

Mark Rudelson and Roman Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences , 62(12):1707-1739, 2009. 15, 20, 22

Arvind K Saibaba. Randomized subspace iteration: Analysis of canonical angles and unitarily invariant norms. SIAM Journal on Matrix Analysis and Applications , 40(1):23-48, 2019. 25

Max Simchowitz, Ahmed El Alaoui, and Benjamin Recht. Tight query complexity lower bounds for pca via finite sample deformed wigner law. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 1249-1259, 2018. 8, 24

Xiaoming Sun, David P. Woodruff, Guang Yang, and Jialin Zhang. Querying a matrix through matrix-vector products. ACM Trans. Algorithms , 17(4):31:1-31:19, 2021. 4

Jared Tanner and Simon Vary. Compressed sensing of low-rank plus sparse matrices. Applied and Computational Harmonic Analysis , 2023. 8

Roman Vershynin. Concentration inequalities for random tensors. Bernoulli , 26(4):3139 - 3162, 2020. 5, 6, 15

Karl Wimmer, Yi Wu, and Peng Zhang. Optimal query complexity for estimating the trace of a matrix. In International Colloquium on Automata, Languages, and Programming , pages 1051-1062. Springer, 2014. 4

David P Woodruff. Sketching as a tool for numerical linear algebra. arXiv preprint arXiv:1411.4357 , 2014. 27

Wei Zhang, Taejoon Kim, Guojun Xiong, and Shu-Hung Leung. Leveraging subspace information for low-rank matrix reconstruction. Signal processing , 163:123-131, 2019. 3

Kai Zhong, Prateek Jain, and Inderjit S Dhillon. Efficient matrix sensing using rank-1 gaussian measurements. In International conference on algorithmic learning theory , pages 3-18. Springer, 2015. 2

A Preliminaries

A.1 Bayes Risk Lower Bounds

Let Θ be a set of parameters and P = { P θ : θ ∈ Θ } be a set of distributions over X . Let w be the prior distribution on Θ known to a learner. Suppose θ ∼ w , and x ∼ P θ and that the learner is given x . Now the learner wants to learn some information about θ . Let d : X → A be a mapping from sample x to action space A and L : Θ ×A→{ 0 , 1 } be a zero-one loss function. We define

and

We drop Θ from the notation when it is clear.

f -divergence Let C denote the set of all convex functions f with f (1) = 0 . Let P and Q be two distributions with densities p and q over a common measure µ . Then we have the f -divergence D f ( P ∥ Q ) defined as

using the convention 0 · ∞ = ∞ . For f = x log( x ) , the f -divergence D f ( P ∥ Q ) = d KL ( P ∥ Q ) , the Kullback-Leibler (KL) divergence. We frequently use d KL ( X ∥ Y ) for some random variables X and Y , which means the KL divergence between the distributions of X and Y .

Now we can define the f -informativity of a set of distributions P with respect to a distribution w as follows:

We use I ( P , w ) to denote the f -informativity for f = x log x . We finally have the following theorem from [7].

Theorem A.1. For any 0-1 loss function L ,

where for 0 ≤ a, b ≤ 1 , ϕ f ( a, b ) denotes the f -divergence between distributions P, Q over { 0 , 1 } with P { 1 } = a and Q { 1 } = b .

As a corollary of the above theorem, [7] prove the following result which is the generalized Fano inequality.

Theorem A.2. For any prior w and 0-1 loss function L ,

A.2 KL Divergence

We state some properties of the KL-divergence that we use throughout the paper.

Lemma A.3. Let P and Q be two random variables with probability measures p and q such that p is absolutely continuous with respect to q . Let ˜ P , with probability measure ˜ p , be the restriction of P to an event E , i.e., for all E ′ , Pr[ ˜ P ∈ E ′ ] = Pr[ P ∈ E ∩ E ′ ] / Pr( P ∈ E ) . Then

Proof. For x ∈ E , we have ˜ p ( x ) = p ( x ) / Pr[ P ∈ E ] and for x ̸∈ E , ˜ p ( x ) = 0 . By definition

using the convention 0 · log(0 / 1) = 0 . Now,

We also have,

which implies

Thus, it is enough to lower bound ∫ ¯ E p ( x ) log ( p ( x ) /q ( x )) d x to obtain an upper bound on the left hand side. So,

where the last inequality follows from -log( x ) being convex. Finally,

where we used p log( q/p ) = p log( q ) + p log(1 /p ) ≤ 0 + 1 = 1 for 0 ≤ p, q ≤ 1 .

The following lemma states the chain rule for KL-divergence.

Lemma A.4 (Chain Rule) . Let ( X , Y ) be jointly distributed random variables and Z , W be independent random variables. Then

The following lemma states the KL-divergence between two multivariate Gaussians.

Lemma A.5 (KL-divergence between Gaussians, folklore) .

where n is the dimension of the Gaussian.

In this article, we use the above lemma only for Σ 1 = Σ 2 = I in which case the above lemma implies that d KL ( N ( µ 1 , I ) ∥ N ( µ 2 , I )) = (1 / 2) ∥ µ 2 -µ 1 ∥ 2 2 .

A.3 Properties of Gaussian random matrices and vectors

We use the following bounds on the norms of Gaussian matrices and vectors throughout the paper: (i) If g is an n -dimensional Gaussian with all its entries independently drawn from N (0 , 1) , then with probability ≥ 1 -exp( -Θ( n )) , ( n/ 2) ≤ ∥ g ∥ 2 2 ≤ (3 n/ 2) , and (ii) If G is an m × n ( m ≥ n ) Gaussian matrix with i.i.d. entries from N (0 , 1) , then Pr[ ∥ G ∥ 2 ∈ [ √ m - √ n -t, √ m + √ n + t ]] ≥ 1 -exp( -Θ( t 2 )) . See [26, 17] and references therein for proofs.

Rotational Invariance We frequently use the rotational invariance property of multivariate Gaussian vectors g ∼ N (0 , I n ) , which implies that for any n × n orthogonal matrix Q independent of g , the vector Q g is also distributed as N (0 , I n ) . Additionally, if q 1 , . . . , q n denote rows of the matrix Q with q 1 chosen arbitrarily independent of g and the subsequent vectors q i := f i ( q 1 , . . . , q i -1 , ⟨ q 1 , g ⟩ , . . . , ⟨ q i -1 , g ⟩ ) for arbitrary functions f i satisfying q i ⊥ q 1 , . . . , q i -1 , then ⟨ q i , g ⟩ is distributed as N (0 , 1) and is independent of ⟨ q 1 , g ⟩ , . . . , ⟨ q i -1 , g ⟩ . We crucially use this property in the proof of Theorem 3.2.

B Proof of Lemma 3.1

First we discuss some notation, distributions and random variables that we use throughout the proof. Fix an arbitrary deterministic algorithm Alg . Let A = G + suv T be a fixed realization of the random matrix A . Recall s = α/ √ n . The algorithm Alg queries a fixed orthonormal matrix Q (1) and receives the response r (1) := Q (1) vec( A ) = Q (1) g + sQ (1) ( u ⊗ v ) where we use g := vec( G ) . As Alg is deterministic , in the second round it queries the matrix Q (2) [ r (1) ] . We use Q (2) [ r (1) ] to emphasize that Alg picks Q (2) purely as a function of r (1) . It receives the response r (2) = Q (2) [ r (1) ] g + sQ (2) [ r (1) ]( u ⊗ v ) and picks queries Q (3) [ r (1) , r (2) ] for the third round and so on. Using h ( j ) := ( r (1) , . . . , r ( j ) ) to denote the history of the algorithm until the end of round j , we have for j = 0 , . . . , t -1

For each j = 1 , . . . , t , the randomness of A induces a distribution H ( j ) over histories until round j . Then for u and v , the distribution H ( j ) uv ≡ H ( j ) | { u = u, v = v } is the distribution of the history of Alg after j rounds conditioned on u = u and v = v where the randomness in the histories is purely from the randomness of G . Recall that without loss of generality, we assume all the general linear queries made by the algorithm are orthogonal to each other. We first prove a tail bound on the amount of information that a non-adaptive query matrix can obtain.

B.1 A Tail Bound

Let u and v be independent Gaussian random vectors. Let Q ∈ R k × n 2 be an arbitrary matrix with k orthonormal rows. We have the following lemma.

Lemma B.1. There is a small enough universal constant β such that for all k ≥ n and C satisfying 4 k ≤ Ck ≤ 16 n 2 , we have for any orthonormal matrix Q k × n 2 that

Proof. By Theorem 1.4 and Remark 1.5 of [31], we have that

for all 0 ≤ t ≤ 2 n . For t = √ Ck/ 4 , for 4 k ≤ Ck ≤ 16 n 2 , we have that

which implies, taking β = c/ 8 , that Pr ˜ u , ˜ v [ ∥ Q (˜ u ⊗ v ) ∥ 2 2 ≥ Ck ] ≤ exp( -βCk/n ) .

B.2 First Round

Let w (1) be the distribution of ( u , v ) where u and v are random variables that are independently sampled from N (0 , I n ) . As already defined, the distribution of the history of the algorithm after the first round for a fixed u, v is given by H (1) uv . We have that H (1) uv is the distribution of the random variable

Let P (1) uv be the distribution of the above random variable (although P (1) uv ≡ H (1) uv , we distinguish P ( j ) uv and H ( j ) uv in later rounds) and let P (1) = { P (1) uv } be the set of distributions for all u, v . Then we have

We define the loss function

for Q ∈ R k × n 2 with orthonormal rows and some function f 1 ( α, γ ) satisfying 4 ≤ f 1 ( α, γ ) ≤ 16 n 2 /k for a parameter γ ≥ 1 . From Lemma B.1, we have

Picking f 1 ( α, γ ) = Kα 2 γ 2 , we have that R Bayes ≥ 1 -1 / (100 γ 2 ) . Thus with probability 1 -1 / (10 γ ) over ( u , v ) ∼ w (1) , we have that

Let Good (1) be the set of all ( u, v ) that satisfy the above property. Let w (2) be the distribution of ( u , v ) ∼ w (1) conditioned on ( u , v ) ∈ Good (1) . For each ( u, v ) ∈ Good (1) , let

We have for all ( u, v ) ∈ Good (1) that Pr h (1) ∼ P (1) uv [ h (1) ∈ GoodH (1) uv ] ≥ 1 -1 / (10 γ ) . The overall conclusion of this is that with a large probability over ( u , v ) ∼ w (1) , the squared projection of u ⊗ v on the query asked by Alg in round 2 is small with large probability over G .

B.3 Further Rounds

We first define the following objects inductively.

For j ≥ 2 , let w ( j ) be the distribution of ( u , v ) ∼ w ( j -1) conditioned on ( u , v ) ∈ Good ( j -1) . So w ( j ) is the distribution over only those inputs for which the squared projection of u ⊗ v on the query space of Alg until round j is small with high probability over G .

( u, v ) ∈ ( j -1)

P ( j ) uv denotes the distribution over histories after round j , conditioned on the queries used until round j not having a lot of 'information' about u ⊗ v .

Let P ( j ) := { P ( j ) uv : ( u, v ) ∈ Good ( j -1) } .

which implies

For j ≥ 1 ,

The set Good ( j ) denotes those values of ( u, v ) for which with large probability over G , the queries used by the algorithm until round j +1 do not have a lot of 'information' about u ⊗ v .

For ( u, v ) ∈ Good ( j ) ,

GoodH ( j ) uv denotes those histories, for which the queries used by the algorithm until round j +1 do not have a lot of 'information' about u ⊗ v .

Let f 0 ( α, γ ) = K and for j ≥ 1 , let f j ( α, γ ) = Kα 2 γ 2 f j -1 ( α, γ ) for a large enough universal constant K .

Lemma B.2. For all 1 ≤ j satisfying f j ( α, γ ) k ≤ 16 n 2 ,

1.

For all ( u, v ) ∈ Good ( j ) ,

where G ( j ) is the joint distribution of j independent k -dimensional Gaussian random variables.

Proof. From the previous section, all the above statements hold for j = 1 . Now assume that all the above statements hold for rounds 1 , . . . , j -1 . We prove the statements inductively for round j . Recall w ( j ) is defined to be the distribution of ( u , v ) ∼ w ( j -1) conditioned on ( u , v ) ∈ Good ( j -1) . For each ( u, v ) ∈ Good ( j -1) , we now bound d ( P ( j ) uv ∥ G ( j ) ) . By definition,

where h ( j ) = ( h ( j -1) , r ( j ) ) ∼ P ( j ) uv . Note that the marginal distribution of h ( j -1) is given by conditioning the distribution P ( j -1) uv on the event GoodH ( j -1) uv . By Lemma A.4, we have

Now, using Lemma A.3, we have

3.

Here we used the inductive assumption that Pr h ( j -1) ∼ P ( j -1) uv [ h ( j -1) ∈ GoodH ( j -1) ] ≥ 1 -1 / (10 γ ) ≥ 9 / 10 where the last inequality follows as the parameter γ ≥ 1 . Next, we upper bound the second term in (6). We have that r ( j ) | h ( j -1) = Q ( j ) [ h ( j -1) ] g + s ( Q ( j ) [ h ( j -1) ])( u ⊗ v ) is distributed as N ( s ( Q ( j ) [ h ( j -1) ])( u ⊗ v ) , I k ) by rotational invariance of the Gaussian distribution. Therefore,

In the last inequality, we used the fact that h ( j -1) ∈ GoodH ( j -1) uv . Finally,

as the distribution w ( j ) is obtained by conditioning w ( j -1) on an event with probability ≥ 1 -1 / (10 γ ) ≥ 1 / 2 and d KL ( ·∥· ) ≥ 0 . Now, using the inductive assumption, we have

where we use α ≥ 15 . This proves the third statement in the lemma for round j . We also have, I ( w ( j ) , P ( j ) ) ≤ E ( u , v ) ∼ w ( j ) [ d ( P ( j ) uv ∥ G ( j ) )] ≤ 2 α 2 f j -1 ( α, γ )( k/n ) . We now define a loss function L ( j ) and use Bayes risk lower bounds to prove the remaining statements. Let

where Q ∈ R k × n 2 is an orthonormal matrix with k ≥ n rows. We have for j such that f j ( α, γ ) k ≤ 16 n 2 ,

where we use Lemma B.1 and the fact that the distribution w ( j ) is obtained by conditioning w (1) on an event with probability ≥ (1 -1 / (10 γ )) j which we prove in section B.4. By the generalized Fano inequality, we obtain

As f j ( α, γ ) = Kα 2 γ 2 f j -1 ( α, γ ) and f 0 ( α, γ ) ≥ K for a large enough constant K , we have βf j ( α, γ ) ≥ -10 j log(1 -1 / (10 γ ))

and therefore for K large enough,

By definition of R Bayes, we conclude that

By Markov's inequality, we have that with probability ≥ 1 -1 / (10 γ ) over ( u , v ) ∼ w ( j ) , it holds that

which is equivalent to

Thus, we conclude that Pr ( u , v ) ∼ w ( j ) [( u , v ) ∈ Good ( j ) ] ≥ 1 -1 / (10 γ ) and that for ( u, v ) ∈ Good ( j ) , we have Pr h ( j ) ∼ P ( j ) uv [ h ( j ) ∈ GoodH ( j ) uv ] ≥ 1 -1 / (10 γ ) .

B.4 Wrap-up

Let j ≥ 1 satisfy f j ( α, γ ) k ≤ 16 n 2 . By definition, Good ( j ) ⊆ Good ( j -1) ⊆ . . . Good (1) . We also have

Now, using the fact that Good ( j ) ⊆ Good ( j -1) ,

As w ( j ) = w (1) | Good ( j -1) , we have

where the last inequality is from Lemma B.2. Thus, Pr ( u , v ) ∼ w (1) [( u , v ) ∈ Good ( j ) ] ≥ (1 -1 / (10 γ )) j .

Similarly, for ( u, v ) ∈ Good ( j ) , we have that

where again, the last inequality follows from Lemma B.2. Thus, Pr h ( j ) ∼ H ( j ) uv [ h ( j ) ∈ GoodH ( j ) uv ] ≥ (1 -1 / (10 γ )) j . Now, for ( u , v ) ∼ w (1) and h ( j ) ∼ H ( j ) uv , we have that with probability ≥ (1 -1 / (10 γ )) 2 j it holds that ( u , v ) ∈ Good ( j ) and h ( j ) ∈ GoodH ( j ) uv . Thus, with probability ≥ (1 -1 / (10 γ )) 2 j over the input matrix G + α ∑ r i =1 u i v T i , we have that

With probability ≥ 1 -1 / poly( n ) , using Lemma B.1, ∥ Q (0) ( u ⊗ v ) ∥ 2 2 ≤ k log( n ) . Using a union bound, we obtain that with probability ≥ (1 -1 / (10 γ )) 2 j -1 / poly( n ) ,

where K is a large enough absolute constant which proves the theorem for j such that ( Kα 2 γ 2 ) j = Ω(log( n )) .

C Proof of Theorem 3.2

Suppose Alg is a t round algorithm that uses k liner measurements in each round such that when run on the matrix G +( α/ √ n ) uv T , it produces a matrix ˆ A such that with probability 9 / 10 over the input matrix,

for a small enough constant c . By making the algorithm to query the vector vec( ˆ A ) in round t +1 , we can therefore ensure that if Q is the overall query space of the algorithm, then

By Lemma 3.1, we obtain that t + 1 = Ω(log( n 2 /k ) / (log( α ) + log log n )) and hence t = Ω(log( n 2 /k ) / (log( α ) + log log n )) .

Now suppose Alg r is a t round algorithm that uses k linear measurements of the input random matrix G +( α/ √ n ) ∑ r i =1 u i v T i and outputs a matrix ˆ A such that with probability ≥ 9 / 10 over the input,

for a small enough constant c . We obtain a lower bound on t by reducing the rank1 plant estimation problem to the rankr plant estimation problem. Suppose the input is the random matrix G + ( α/ √ n ) u 1 v T 1 . We sample u 2 , . . . , u r and v 2 , . . . , v r so that the coordinates of these vectors are independent standard Gaussian random variables. Now we note that we can perform arbitrary linear measurements of the matrix G +( α/ √ n ) ∑ r i =1 u i v T i since we have access to linear measurements of G +( α/ √ n ) u 1 v T 1 and we know the vectors u 2 , . . . , u r and v 2 , . . . , v r . We can then use Algorithm Alg r to obtain a matrix ˆ A such that with probability ≥ 9 / 10 over the input and our sampled vectors,

Condition on this event. We have ∥ ˆ A -[ ˆ A ] r ∥ 2 F ≤ ∥ ∑ r i =1 u i v T i -ˆ A ∥ 2 F ≤ cn 2 r which then implies ∥ ∑ r i =1 u i v T i -[ ˆ A ] r ∥ 2 F ≤ 2 cn 2 r . Now, let P be the at most rank r projection matrix onto the rowspace of the matrix [ ˆ A ] r and assume that U and V are matrices with columns given by u 1 , . . . , u r and v 1 , . . . , v r respectively. From above, we get

which then implies that

Now, U is an n × r matrix with coordinates being independent standard Gaussian random variables. We have from [26] that if r ≤ n/ 2 , then with probability 1 -exp( -n ) , σ min ( U ) ≥ c 1 √ n . From [17] we also have that with probability ≥ 1 -exp( -n ) , c 2 n ≤ ∥ u 1 ∥ 2 2 , . . . , ∥ u 2 ∥ 2 2 ≤ Cn for a small enough c 2 and large enough C . Conditioned on all these events, we have

Hence, at least 9 r/ 10 indices i have the property that ∥ u i v T i ( I -P ) ∥ 2 F ≤ (20 cC/c 1 ) n 2 . Since the marginal distributions of u 1 v T 1 , . . . , u r v T r are identical, we obtain that with probability ≥ 8 / 10 ,

Note that rank of P is at most r and therefore P = QQ T for an orthonormal matrix Q with at most r columns. In the ( t +1) -th round, we make nr linear measurements of the input matrix and obtain the matrix

and therefore, we can obtain the matrix M = G P +( α/ √ n ) u 1 v T 1 P . Let [ M ] 1 be the best rank1 approximation of M in operator norm. We have

Since, ∥ G ∥ 2 ≤ 2 √ n with probability 1 -exp( -Θ( n )) , we get

and

For α large enough and c small enough, by querying the vector vec([ M ] 1 ) , we can again ensure that if Q is the query space of the algorithm after t +2 rounds, we have with probability ≥ 1 / 2 over the input random matrix that

We again have from Lemma 3.1 that t +2 = Ω(log( n 2 /k ) / (log log n +log α )) and therefore that t = Ω(log( n 2 /k ) / (log log n +log α )) .

D Proof of Theorem 1.1 using Theorem 3.2

Let A be any any randomized algorithm that queries orthonormal measurements of the underlying matrix A and outputs a reconstruction ˆ A of A satisfying ∥ A -ˆ A ∥ 2 F ≤ c ∥ A ∥ 2 F with probability ≥ p , over both the randomness of the algorithm and randomness of the measurements. Let ˆ A = A ( A, σ , γ ) where σ captures the randomness of the algorithm and γ captures the randomness in measurements. First we have the following lemma.

Lemma D.1. If there is a randomized algorithm A ( A, σ , γ ) such that for all rank ≤ r matrices A with ∥ A ∥ 2 F = Θ( nr ) ,

then there is a randomized algorithm A ′ such that for all A with ∥ A ∥ 2 F = Θ( nr ) ,

Proof. Fix a particular σ . The algorithm first queries an orthonormal matrix Q (1) ∈ R k × n 2 and receives a random response r (1) = Q (1) · vec( A ) + g (1) where g (1) is a vector with independent Gaussian components of mean 0 and variance 1 . As a function of r (1) , the algorithm queries an orthonormal matrix Q (1) [ r (1) ] and receives the random response r (2) = Q (2) [ r (1) ] · vec( A ) + g (2) where g (2) is again a random Gaussian vector independent of g (1) . Importantly, we also have that Q (2) [ r (1) ] is orthogonal to the query matrix Q (1) in the first round. The algorithm proceeds in further rounds accordingly where it picks query matrices as a function of the responses in all the previous rounds.

Let G be an n × n Gaussian matrix with independent entries of mean 0 and variance 1 . We now observe that r (1) = Q (1) · vec( A )+ g (1) has the same distribution as Q (1) · vec( A + G ) by rotational invariance of Gaussian distribution. Now conditioning on r (1) , we obtain that r (2) = Q (2) [ r (1) ] · vec( A ) + g (2) has the same distribution as Q (2) [ r (1) ] · vec( A + G ) as Q (2) [ r (1) ] is orthogonal to Q (1) and hence Q (2) [ r (1) ] · g is independent of Q (1) g , again by rotational invariance.

Thus, we can consider a deterministic algorithm A ′ parameterized by the same σ which exactly simulates the behavior of A performing matrix recovery with noisy measurements. Thus,

Taking expectation over σ , we obtain the result.

Let u 1 , . . . , u r and v 1 , . . . , v r be 2 r random vectors with coordinates being independent standard normal random variables. Let U be an n × r matrix with columns given by u 1 , . . . , u r and V be an n × r matrix with columns given by v 1 , . . . , v r .

Claim D.2. If r ≤ n/ 2 , With high probability,

Proof. Clearly, UV T = ∑ r i =1 u i v T i . We now have 2 nr ≥ ∥ V ∥ 2 F ≥ nr/ 2 with probability 1 -exp( -Θ( nr )) from [17]. From [26], with probability ≥ 1 -exp( -Θ( n )) , σ min ( U ) ≥ c √ n for a constant c and as seen in preliminaries, ∥ U ∥ 2 ≤ 2 √ n . Thus, conditioned on both these events, and

Hence ∥ ( α/ √ n ) ∑ r i =1 u i v T i ∥ 2 F = Θ( nr ) with a large probability. Now, we obtain that

Thus, there is some σ such that

Thus with probability ≥ p (1 -exp( -Θ( n ))) ,

By picking c small enough, we obtain that t = Ω(log( n 2 /k ) / log log n ) for algorithms with success probability p ≥ 9 / 10 .

E Applications to Other Problems

E.1 Spectral Low Rank Approximation

Theorem E.1. Given n, r ∈ Z , if a t -round adaptive algorithm that performs k ≥ nr general linear measurements in each round is such that for every n × n matrix A , the algorithm outputs a rank r matrix B such that with probability ≥ 99 / 100 , ∥ A -B ∥ 2 ≤ 2 σ r +1 ( A ) , then t ≥ c log( n 2 /k ) log log( n ) .

We prove the following helper lemma that we use in our proof.

Lemma E.2. If A is a rankr matrix and B is an arbitrary matrix satisfying ∥ A -B ∥ 2 ≤ t , then ∥ A -[ B ] r ∥ 2 ≤ 2 t , where [ B ] r denotes the best rankr approximation of the matrix B in operator norm.

Proof of Theorem E.1. By a standard reduction, it suffices to show that there is a distribution on n × n matrices for which any t -round deterministic algorithm which makes k general linear measurements in each round and outputs a 2 -approximate spectral rank approximation satisfies t ≥ c log( n 2 /k ) / log log( n ) .

Consider the n × n random matrix M = G + s uv T for s = 500 / √ n . Suppose there is a deterministic t -round algorithm Alg that makes k linear measurements and outputs rankr matrix K = Alg ( M ) such that with probability ≥ 99 / 100 ,

This implies that ∥ s uv T -K ∥ 2 ≤ 3 ∥ G ∥ 2 and ∥ uv T -K/s ∥ 2 ≤ 3 ∥ G ∥ 2 /s .

With probability ≥ 1 -exp( -Θ( n )) , we have 2 √ n ≥ ∥ u ∥ 2 , ∥ v ∥ 2 ≥ √ n/ 2 and ∥ G ∥ 2 ≤ 3 √ n simultaneously. By a union bound, we have that with probability ≥ 0 . 98 , ∥ uv T -K/s ∥ 2 ≤ (9 / 500) n .

By Lemma E.1, we have ∥ uv T -[ K/s ] 1 ∥ 2 ≤ (9 / 250) n which implies that ∥ u ⊗ v -vec([ K/s ] 1 ) ∥ 2 2 = ∥ uv T -[ K/s ] 1 ∥ 2 F ≤ 2 ∥ uv T -[ K/s ] 1 ∥ 2 2 ≤ 2(9 / 250) 2 n 2 where we used the fact that the matrix uv T -[ K/s ] 1 has rank at most 2 . Let q ∈ R n 2 = vec([ K/s ] 1 ) / ∥ [ K/s ] 1 ∥ F be a unit vector. We can show that

by a simple application of the triangle inequality. Thus, using Alg we can construct a deterministic t + 1 round algorithm such that the squared projection of u ⊗ v onto the query space is at least ≥ n 2 / 64 with probability ≥ 98 / 100 . By (4), we have that t + 1 ≥ c ′ log( n 2 /k ) / log log( n ) and therefore we have t ≥ c log( n 2 /k ) / log log( n ) for a small enough constant c .

The following theorem states our lower bound for algorithms which output a spectral LRA for each matrix with high probability.

Theorem E.3. Given n, r ∈ Z , if a t -round adaptive algorithm that performs k ≥ nr general linear measurements in each round is such that for every n × n matrix A , the algorithm outputs a rank r matrix B such that with probability ≥ 1 -1 / poly( n ) , ∥ A -B ∥ 2 ≤ 2 σ r +1 ( A ) , then t ≥ c log( n 2 /k ) .

Proof of Theorem E.3. The proof of this lemma is similar to that of Theorem E.1. The existence of a randomized t round algorithm that outputs a 2-approximate spectral norm LRA for every instance with probability ≥ 1 -1 / poly( n ) implies the existence of a deterministic algorithm that outputs a 2-approximate spectral norm LRA with probability ≥ 1 -1 / poly( n ) over the distribution of G +( α/ √ n ) uv T . As in the proof of the Theorem E.1, this algorithm can be used to construct a t +1 round algorithm that with probability ≥ 1 -1 / poly( n ) over the matrix G +( α/ √ n ) uv T , computes a unit vector q satisfying

Now, (5) implies that t +1 ≥ c log( n 2 /k ) for a small enough constant c .

For the random matrix M = G + ( α/ √ n ) uv T for a large enough constant α considered in the above theorem, we have that ( σ 1 ( M ) /σ 2 ( M )) ≥ 2 with high probability. Now consider the random matrix

With high probability, we have σ r ( M ′ ) = σ 1 ( M ) ≥ ( α/ 2) √ n and σ r +1 ( M ′ ) = σ 2 ( M ) ≤ 2 √ n implying σ r +1 ( M ′ ) /σ r ( M ′ ) ≤ 1 / 2 . A proof similar to that of the above theorem now shows that algorithms using k ≥ nr general linear measurements in each round and outputting a 2 -approximate rankr LRA with probability ≥ 1 -1 / poly( n ) for matrices M with σ r +1 ( M ) /σ r ( M ) ≤ 1 / 2 have a lower bound of Ω(log( n 2 /k )) rounds. Moreover for k ≥ Cnr for a large enough constant C and r = O (1) , the randomized subspace iteration algorithm starting with a subspace of k/n -dimensions, after O (log( n 2 /k )) rounds outputs, with probability ≥ 1 -1 / poly( n ) , a 2 -approximate rankr spectral norm LRA for all matrices satisfying σ r +1 ( M ) /σ r ( M ) ≤ 1 / 2 . See [12, Theorem 5.8] for a proof.

Note that the subspace iteration algorithm starting with a subspace of k/n -dimensions performs ( k/n ) · n = k general linear measurements in each round. Thus for r = O (1) and k ≥ Cnr for a large enough constant C , the lower bound of Ω(log( n 2 /k )) rounds for high probability spectral norm LRA algorithms for 'well-conditioned' ( σ r +1 /σ r ≤ 1 / 2 ) instances is tight up to constant factors and shows that general linear measurements offer no improvement over matrix-vector products for well-conditioned problems. Thus we have the following theorem.

Theorem E.4. Any randomized t -round algorithm that with probability ≥ 1 -1 / poly( n ) outputs a 2-approximate rankr spectral norm LRA for any arbitrary matrix A satisfying σ r ( A ) /σ r +1 ( A ) ≥ 2 , must have t = Ω(log( n 2 /k )) , where k ≥ nr is the number of linear measurements the algorithm makes in each round.

Moreover, for r = O (1) , the subspace iteration algorithm [12] matches the lower bound up to constant factors and outputs a 2 -approximate spectral norm LRA for all such instances with probability ≥ 1 -1 / poly( n ) in O (log( n 2 /k )) rounds.

E.2 Symmetric Spectral Norm Low Rank Approximation

An interesting property of the hard distributions from previous works [28, 3] is that those distributions are supported on symmetric matrices and they hence obtain lower bounds for algorithms that work even only on symmetric instances. Although our hard distribution is non-symmetric, we can construct a distribution supported only on symmetric matrices and show lower bounds on algorithms using generalized linear queries for symmetric matrices.

Theorem E.5. Given n, r ∈ Z , if a t -round adaptive algorithm that performs k ≥ nr general linear measurements in each round is such that for every n × n matrix A , the algorithm outputs a rank r ≥ 2 matrix B such that with probability ≥ 99 / 100 , ∥ A -B ∥ 2 ≤ 2 σ r +1 ( A ) , then t ≥ c log( n 2 /k ) log log n .

Proof. Consider the 2 n × 2 n random matrix M ′ defined as

where M = G + s uv T for s = 500 / √ n and all coordinates of G , u , v are independent standard Gaussian random variables. We have that for i ∈ [ n ] , σ 2 i -1 ( M ′ ) = σ 2 i ( M ′ ) = σ i ( M ) and that any arbitrary k generalized linear measurements of the matrix M ′ can be simulated using 2 k general linear measurements of M . Now suppose an algorithm that uses k general linear measurements in each round outputs a rank r ≥ 2 matrix K satisfying

Let the matrix K be of the form then

As any sub-matrix of K has rank at most that of K , we obtain that K 2 is a rankr matrix satisfying

Thus, the existence of a t -round deterministic algorithm that uses k generalized queries in each round and outputs a constant factor approximation of rank r , spectral norm LRA, for the random matrix M ′ with probability ≥ 99 / 100 implies the existence of a t -round algorithm that uses 2 k generalized queries in each round and outputs a constant factor approximation of rankr spectral norm LRA for the random matrix M with probability ≥ 99 / 100 . Now, as in the proof of Theorem E.1, we obtain that

We obtain the proof by appropriately scaling n in the statement.

The above theorem proves lower bounds for algorithms that solve constant factor rankr spectral norm Low Rank Approximation (LRA) for all r ≥ 2 even for symmetric instances. This leaves open just a lower bound on algorithms solving rank 1 spectral norm LRA for symmetric instances.

E.3 Schatten Norm Low Rank Approximation

We first note the following lemma which bounds the Schattenp norm of an n × n Gaussian matrix. Lemma E.6 (Equation 3.3 in [19]) . If G is an n × n matrix with independent entries sampled from N (0 , 1) and p ≥ 2 , then with probability ≥ 9 / 10 , ∥ G ∥ S p ≤ 30 n 1 / 2+1 /p .

We now state the theorem that shows lower bounds for Schatten norm low rank approximation.

Theorem E.7. Given n, r ∈ Z , if a t -round adaptive algorithm that performs k ≥ nr general linear measurements in each round is such that for every n × n matrix A , the algorithm outputs a rank r matrix B such that with probability ≥ 99 / 100 we have

Proof. The proof is very similar to the proof of Theorem E.1. Let M = G + s uv T for s = α/ √ n for α to be chosen later. Let Alg be a t -round deterministic algorithm that outputs a 2-approximate Schatten p -norm low rank approximation for M with probability ≥ 99 / 100 over M . Then we have

as min rankr X ∥ G + s uv T -X ∥ S p ≤ ∥ G ∥ S p . Using Lemma E.6, with probability ≥ 9 / 10 over M , we have that

By picking α = Bn 1 /p for a large enough constant B , we obtain that

Similar to the proof of Theorem E.1, we can construct a unit vector q ∈ R n 2 such that with probability ≥ 8 / 10 over M , we have

Using (4), we obtain that

In particular, for p = O (log( n ) / log log( n )) , this gives a lower bound of Ω( p (2 -log( k ) / log( n ))) on the number of adaptive rounds required if an algorithm can query k general linear measurements in each round. Recently, Bakshi, Clarkson and Woodruff [2, Algorithm 2] gave an algorithm for Schattenp low rank approximation for arbitrary matrices that runs in O ( p 1 / 2 log( n )) 4 iterations to output a constant factor approximation. For instances with σ r ( A ) /σ r +1 ( A ) = 1 + Ω(1) , Saibaba [27, Theorem 8] showed that randomized subspace iteration gives a constant factor approximation to rankr Schattenp norm LRA in O (log( n )) iterations using nr linear measurements in each round.

E.4 Ky-Fan Norm Low Rank Approximation

Theorem E.8. Given n, r ∈ Z , if a t -round adaptive algorithm that performs k ≥ nr general linear measurements in each round is such that, for every n × n matrix A , the algorithm outputs a rankr matrix B such that with probability ≥ 99 / 100 we have

Proof. Again consider the same instance M = G + s uv T with s = α/ √ n for α chosen later. If K is a 2 -approximate rankr low rank approximation of M in Ky-Fan p norm, then

which by the triangle inequality implies that ∥ s ∑ r i =1 u i v T i -K ∥ F p ≤ 3 ∥ G ∥ F p . As ∥ G ∥ 2 ≤ 2 √ n with high probability, we have that ∥ G ∥ F p ≤ 2 p √ n with high probability. Thus,

then

Picking α = Bp for a large enough constant B , we obtain that using the matrix K , we can construct a unit vector q such that ⟨ q, u ⊗ v ⟩ 2 ≥ n 2 / 64 , thus showing a

round lower bound on any algorithm that performs k general linear measurements to output a 2 -approximate rankr approximation in Ky-Fan p norm.

For p = O (1) , the Block Krylov iteration algorithm [20] gives an O (1) approximate solution to the rankr Ky-Fan norm low rank approximation problem in O (log( n )) rounds while querying nr general linear measurements in each round. Thus our lower bound on constant approximate algorithms for Ky-Fan norm LRA is optimal up to a log log( n ) factor for r, p = O (1) .

E.5 Frobenius Norm Low Rank Approximation

Theorem E.9. Given an n × n matrix A with σ 1 ( A ) /σ 2 ( A ) ≥ 2 , if a t -round algorithm outputs a rank1 matrix B satisfying ∥ A -B ∥ 2 F ≤ (1 + 1 /n ) ∥ A -[ A ] 1 ∥ 2 F with probability ≥ 99 / 100 , then we have t ≥ c log( n 2 /k ) / log log( n ) .

First, we prove the following helper lemma.

Lemma E.10. Given A ∈ R n × n , if a rank r matrix B is a 1 + 1 / ( n -r ) approximate rank r Frobenius norm LRA, then

Proof. We use the fact [12, Theoem 3.4] that if a rank r matrix satisfies ∥ A -B ∥ 2 F ≤ ∥ A -[ A ] r ∥ 2 F + η , then ∥ A -B ∥ 2 2 ≤ ∥ A -[ A ] r ∥ 2 2 + η . If B is a 1+1 / ( n -r ) approximate solution for rank r Frobenius norm LRA, we have

which implies ∥ A -B ∥ 2 2 ≤ ∥ A -[ A ] r ∥ 2 2 + σ r +1 ( A ) 2 = 2 σ r +1 ( A ) 2 .

Proof of Theorem E.9. Using the above lemma, we have that whenever ∥ A -B ∥ 2 F ≤ (1+1 /n ) ∥ A -[ A ] 1 ∥ 2 F for a rank 1 matrix B , then ∥ A -B ∥ 2 2 ≤ 2 ∥ A -B ∥ 2 2 . Consider the random matrix M = G + ( α/ √ n ) uv T considered in Theorem E.1. With probability ≥ 99 / 100 , we have that σ 1 ( M ) /σ 2 ( M ) ≥ 2 by picking α to be a large enough constant.

A t -round randomized algorithm for (1+1 /n ) -approximate rank1 Frobenius norm LRA thus implies the existence of a ( t +1) round deterministic algorithm that with probability ≥ 98 / 100 over the random matrix G +( α/ √ n ) uv T makes general queries such that the squared projection of u ⊗ v onto the query space of the algorithm exceeds n 2 / 64 using similar arguments as in proof of Theorem E.1. Now, using (4), we obtain that

for a small enough constant c .

For matrices A with σ 1 ( A ) /σ 2 ( A ) ≥ 2 , subspace iteration algorithm gives a 1 + 1 /n -approximate solution in O (log( n )) rounds using n -linear measurements [10, 20]. The above lower bound shows that if an algorithm is allowed o (log( n ) / log log( n )) adaptive rounds, then the algorithm has to make n 2 -o (1) linear measurements in each round, which is almost as many measurements as is required to read the entire matrix.

E.6 Lower Bound for Reduced-Rank Regression in Spectral Norm

Given matrices A , B , and a rank parameter r , the reduced-rank regression problem in spectral norm is defined as

Taking A = I , we have that the spectral norm low rank approximation is a special case of the reducedrank regression problem. Thus we have the following lower bound on the number of rounds of an adaptive algorithm that outputs a 2-approximate solution for the reduced-rank regression problem.

Theorem E.11. If a t -round adaptive algorithm which given arbitrary n × n matrices A,B and a parameter r outputs a 2 -approximate solution for the reduced-rank regression problem with probability ≥ 9 / 10 , then t = Ω(log( n 2 /k ) / log log( n )) where k is the number of linear measurements of the matrices A and B that the adaptive algorithm performs in each of the t rounds.

With nr adaptive linear measurements in each round, the above theorem gives a lower bound of Ω(log( n/r ) / log log( n )) on the number of rounds required to obtain a factor 2 approximation. Kacham and Woodruff [15] give an algorithm for reduced-rank regression that outputs constant factor approximations to the reduced-rank problem in O (poly(log n )) (ignoring polylogarithmic factors from condition numbers) adaptive rounds using nr linear measurements in each round.

If the matrix B = M + P has the structure of a planted matrix studied in this paper with P being a rank r matrix and ∥ P ∥ 2 > 10 ∥ B -[ B ] r ∥ 2 and if the matrix A is well conditioned, then we can find an O (1) approximate solution to reduced-rank regression problem in O (log( n ) + log( ∥ P ∥ 2 / OPT )) rounds, where OPT is the optimal value of the reduced-rank regression problem.

First, we find a rank r matrix P ′ such that ∥ B -P ′ ∥ 2 ≤ 2 ∥ B -[ B ] r ∥ 2 in O (log( n )) adaptive rounds using the Block Krylov iteration algorithm of [20], which queries nr linear measurements in each round. Now, for any matrix X ,

Let P ′ = U Σ V T be the singular value decomposition with matrix U Σ having r columns. Then using high precision regression routines (see [33] and references therein), we find a matrix ˜ X such that

in O (log(1 /ε )) iterations, where each iteration queries nr linear measurements. Again, using the triangle inequality, we have

If X ∗ is the optimal solution for the reduced rank regression problem, we have

We also have ∥ AA + U Σ -A ˜ X ∥ 2 ≤ ∥ AA + U Σ -A ˜ X ∥ F ≤ O ( √ rε ∥ P ∥ 2 ) . We set ε = ( OPT / ∥ P ∥ 2 ) 2 /Kr for a large enough constant K and obtain that ∥ AA + U Σ -A ˜ X ∥ 2 ≤ OPT. Overall, we have ∥ A ˜ XV T -B ∥ 2 ≤ ∥ A ˜ XV T -P ′ ∥ 2 +2 · OPT ≤ 6 · OPT. Thus we obtain a 6 -approximate solution in O (log( n ) +log( ∥ P ∥ 2 / OPT )) iterations given that A is well-conditioned and B is of the form M + P with ∥ P ∥ 2 ≥ 10 ∥ M ∥ 2 for a rank r matrix P . Thus the lower bound is near optimal for algorithms that output O (1) approximate solution for planted models and well-conditioned coefficient matrices.

E.7 Lower Bound for Approximating the i -th Singular Vectors

Theorem E.12. If a t -round algorithm, given an n × n matrix A and i ∈ [ n ] , outputs unit vectors u ′ i and v ′ i such that with probability ≥ 9 / 10 ,

where u i and v i are respectively the i -th left and right singular vectors of the matrix A , then t = Ω(log( n 2 /k ) / log log( n )) , where k is the number of linear measurements the algorithm performs on the matrix A in each of the t rounds.

We also have a t = Ω(log( n 2 /k )) lower bound on the number of rounds required for an adaptive algorithm to compute approximations to the i -th left and right singular vectors with probability ≥ 1 -1 / poly( n ) .

Proof. Consider the n × n random matrix M = G +( α/ √ n ) uv T for a large enough constant α . The existence of a t -round algorithm as in the statement implies that there is a deterministic t -round algorithm that outputs approximations to singular vectors of the matrix M with probability ≥ 9 / 10 . We then have that 10 ∥ G ∥ 2 ≤ ∥ ( α/ √ n ) uv T ∥ 2 and ∥ u ∥ 2 , ∥ v ∥ 2 ≥ √ n/ 2 with large probability. Condition on this event. Let v 1 ∈ R n be the top right singular vector of the matrix M . We have

(9 / 10) ∥ ( α/ √ n ) uv T ∥ 2 ≤ ∥ ( α/ √ n ) uv T ∥ 2 -∥ G ∥ 2 ≤ ∥ M ∥ 2 = ∥ ( G +( α/ √ n ) uv T ) v 1 ∥ 2

By the triangle inequality, we have

Thus, we obtain that | v T v 1 | ≥ 4 / 5 ∥ v ∥ 2 . Similarly, if u 1 is the top left singular vector, we have that | u T u 1 | ≥ 4 / 5 ∥ u ∥ 2 . Suppose v ′ 1 is a unit vector such that ∥ v 1 -v ′ 1 ∥ 2 ≤ 1 / 10 . Then, | v T v ′ 1 | ≥ | v T v 1 | -(1 / 10) ∥ v ∥ 2 ≥ (7 / 10) ∥ v ∥ 2 . Similarly, | u T u ′ 1 | ≥ (7 / 10) which implies

Thus, with a large probability over the random matrix M , approximations to the top left and right singular vectors of the matrix M can be used to construct an n 2 -dimensional unit vector q such that ⟨ q, u ⊗ v ⟩ 2 ≥ n 2 / 10 . Thus, we have a t +1 round deterministic algorithm such that the squared projection of u ⊗ v onto the query space of the algorithm is ≥ n 2 / 10 with probability ≥ 8 / 10 over the random matrix M . Now, (4) implies that t +1 ≥ c log( n 2 /k ) / log log( n ) for a small enough constant c .

To extend the above lower bound to approximating i -th singular vector for all i ≤ n/ 2 , we can use the following instance:

where M is the ( n/ 2) × ( n/ 2) random matrix G +( α/ √ n/ 2) uv T and S is a deterministic diagonal matrix chosen such that the top singular vectors of M correspond to i -th singular vectors of the matrix M ′ . If S is chosen to be the diagonal matrix with i -1 values equal to 10 √ nα and the rest of the entries are set to 0 , we have that with high probability that the i -th left and right singular vectors of M ′ correspond to the top left and right singular vectors of M as ∥ M ∥ 2 ≤ 10 √ nα with high probability. Now the existence of a t -round randomized algorithm to approximate i -th left and right singular vectors of an n × n matrix for i ≤ n/ 2 with probability ≥ 9 / 10 implies that there is a deterministic t -round algorithm that approximates the top left and right singular vectors of the ( n/ 2) × ( n/ 2) matrix M with probability ≥ 8 / 10 . From the above argument we have t +1 = Ω(log(( n/ 2) 2 /k ) / log log( n/ 2)) = Ω(log( n 2 /k ) / log log( n )) .

For i ≥ n/ 2 , we can do a similar reduction. We have that the top left singular vector u 1 of the ( n/ 2) × ( n/ 2) matrix M is same as the top singular (eigen-) vector of the matrix MM T and with high probability is equal to the last singular (eigen-) vector of the matrix (100 nα 2 ) I -MM T . Using the same trick as above, we can create an n × n matrix such that the i -th singular vector for i ≥ n/ 2 corresponds to u 1 . Instead of MM T , if we use the matrix M T M , then we can make the top right singular vector v 1 of the matrix M correspond to the i -th singular vector of an n × n matrix. Thus, for all i ∈ [ n ] , we have an Ω(log( n 2 /k ) / log log( n )) lower bound for any randomized algorithm that computes the i -th left and right singular vectors of an arbitrary n × n matrix A using k linear measurements of A in each round of the algorithm.

The lower bounds for algorithms that output approximations to singular vectors with probability ≥ 1 -1 / poly( n ) follow similarly using the high probability lower bounds from (5).

E.8 Lower Bounds for Sparse Matrices

The following theorem gives a lower bound of Ω(log( m/k )) rounds on adaptive algorithms that compute 2 -approximate spectral norm LRA for matrices with at most m nonzero entries.

Theorem E.13. Any t round randomized algorithm that computes, given an arbitrary n × n matrix A with at most m nonzero entries, a 2 -approximate rank r spectral norm LRA of A with probability ≥ 1 -1 / poly( m ) , must have t = Ω(log( m/k )) , where k is the number of general linear measurements queried by the algorithm in each of the t rounds.

Proof. We consider the distribution of the √ m × √ m random matrix G +( α/m 1 / 4 ) uv T for a large enough constant α and embed this instance into an n × n matrix. Clearly, all the matrices drawn from this distribution have at most m nonzero entries. And from Theorem E.3, any deterministic algorithm using k linear measurements in each round and computing a 2 -approximate spectral norm approximation for a matrix drawn from this distribution, with probability ≥ 1 -1 / poly( m ) , must use Ω(log(( √ m ) 2 /k )) = Ω(log( m/k )) rounds.

Thus by Yao's minimax lemma, any randomized algorithm that outputs a 2 -approximate spectral norm approximation with probability ≥ 1 -1 / poly( m ) for an arbitrary matrix with m nonzero entries must use t = Ω(log( m/k )) rounds.

The above theorem implies that any algorithm with only O (1) adaptive rounds must query Ω( m ) linear measurements. This lower bound is tight up to constant factors as with 2 m linear measurements, since all of the m non-zero entries of any arbitrary matrix can be computed in just 1 round using a Vandermonde matrix [8, Theorem 2.14].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSparse recovery for vectors has been studied for a long time. It has been shown that allowing adaptive queries will give extra power to reduce the number of queries. We also have upper and lower bounds in this setting. More recently, sparse recovery for low-rank matrices has also been extensively studied. With non-adaptive queries, $\\Omega(n^2)$ queries are necessary. With adaptive queries, based on the power method, we get an algorithm with $O(nr)$ linear measurements in each round over $O(\\log n)$ adaptive rounds. It is an interesting question to see if the power method is optimal in this setting.\n\nThis paper gives an affirmative answer to this question by showing a measurement-vs-rounds trade-off for recovering low-rank matrices using linear measurements. More specifically, let $A$ be an $n$-by-$n$ matrix with rank $r$. In each round, the algorithm can make $k$ queries. Each query is an $n$-by-$n$ matrix $S$, and the answer is $\\langle S, A\\rangle$ with some Gaussian noise. The goal is to reconstruct a matrix $\\hat{A}$ such that $\\|\\hat{A}-A\\|_F\\leq c\\|A\\|_F$. The main result of this paper is that any adaptive algorithm which uses $n^{2-\\beta}$ linear measurements in each round must run for $o(\\log n/ \\log \\log n)$ rounds to compute a good reconstruction with high probability. Their techniques also apply to obtain measurements-vs-rounds trade-offs for other numerical linear algebra problems, including low-rank approximation in several different norms, singular vector approximation, etc.\n\nTechnically, the construction of the hard instance is $A=\\frac{\\alpha}{\\sqrt{n}}\\sum_{i=1}^r u_iv_i^\\top$, where $u_i,v_i$ are independent, Gaussian random vectors. In the proof of their main result, they first reduce to show the lower bound for a deterministic algorithm with perfect linear measurements of the random matrix $\\frac{\\alpha}{\\sqrt{n}}\\sum_{i=1}^r u_iv_i^\\top+G$, where $G$ is a Gaussian matrix, outputting a reconstruction of $\\frac{\\alpha}{\\sqrt{n}}\\sum_{i=1}^r u_iv_i^\\top$. The key observation is that the distribution of the responses in the first round is close to $N(0, I_k)$, and therefore the algorithm cannot have a lot of “information” about the target matrix. The proof relies on a random tensor concentration result and Bayes risk lower bounds.\n\nSTRENGTHS:\nThis paper fills a gap in the field of sparse recovery and makes significant progress in understanding the limitations of adaptive queries in solving numerical linear algebra problems. Their main result is the lower bound for the sparse recovery of low-rank matrices, nearly matching the upper bound via the power method. Even if they use some techniques in prior works, the proofs are still non-trivial. This paper is well-motivated, and the idea of their techniques and proofs are clearly presented. Most of the claims are sound to me.\n\nWEAKNESSES:\nFor the applications, it is difficult to judge the significance since there is not enough comparison between the results in this paper and prior works. It seems that some applications are for the regimes incomparable to the previous literature.\n\nQUESTIONS:\n1. Does the lower bound require that all the iterations take $k$ queries uniformly? What if the algorithm can adaptively decide the number of queries in each iteration? \n\n2. It would be better to show some known upper bounds for the problems in Table 1.\n\n3. Line 52: $M\\in \\mathbb{R}^{n\\times n} \\rightarrow \\mathbb{R}^t$. $t$ should be $k$.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on investigating the lower bound of the adaptive low-rank matrix sensing problem. Specifically, the authors demonstrate that when the noise level significantly exceeds the signal, any adaptive algorithm requiring fewer than $o(\\log(n)/\\log\\log n)$ rounds must utilize at least $\\Omega(n^2)$ linear measurements in total. This finding highlights an intriguing trade-off between the number of measurements and the number of rounds in various matrix sensing problems within numerical linear algebra.\n\nThe paper presents a clear message, and the theoretical results are robust and captivating. However, one aspect worth considering is the general interest in the noise level examined in this study.\n\nSTRENGTHS:\n- The theoretical results are very sound\n- The presentation is very clean and easy to follow\n- The adaptive settings of matrix recovery is of general interest\n\nWEAKNESSES:\nThis paper examines the case where the noise level is assumed to be $O(1)$, while the signal of each entry is considered to be $O(1/\\sqrt{N})$. It is worth noting that in a more typical scenario where both signals and noises are of magnitude $O(1)$, a single round with $\\Omega(nr)$ measurements, as presented in [5], would suffice. This raises questions about the significance of studying this extreme case and the role that $\\sigma_l$ plays in the lower-bound tradeoff.\n\nTo provide more justifications for studying this extreme case, it would be beneficial for the authors to elaborate on the motivations and implications behind their choice of noise and signal levels. By offering insights into why this specific scenario is relevant and shedding light on the insights gained from this extreme case analysis, the authors can strengthen the value and message of their paper.\n\nQUESTIONS:\nThe authors make an assumption that $M^{(i)}$ represents an orthonormal basis and argue that this assumption does not affect the generality of their approach due to the possibility of a change of basis. However, it is important to consider whether this change of basis has any impact on the homogeneous noise level, i.e., $Ag$ for a Gaussian vector $g$ is not a homogeneous Gaussian vector anymore if $A$ is not unitary. It would be valuable for the authors to provide further elaboration on this point.\n\nLIMITATIONS:\nI do not forsee any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
In this paper, the authors discuss the power of adaptive algorithms in low-rank approximation. This is a setting where one observes general linear measurements of a matrix and wants to produce the best r-rank approximation of it.

It is known that non-adaptive algorithms need order n^2 measurements and that with log n rounds a spectral algo works with nearly linear measurements.

The authors discuss whether something can be done with o(log n) rounds.
## Contribution

The authors prove that for an algo that works with o(log n/log log n) rounds, we need order n^{2-o(1)} measurements.

STRENGTHS:
The authors point to the interesting importance of having access to approx log n rounds, as they prove that having access to o(log n/log log n) rounds is like having one round in terms of measurement complexity. I find the contribution clean, correct, and interesting.

WEAKNESSES:
It is a little bit disappointing that the authors cannot only assume o(log n) rounds, and need to assume o(log n/log log n) rounds. Could they comment more on that weakness?

QUESTIONS:
It seems that for the lower bound the authors only assume that the target matrix is r-rank Gaussian spike plus Gaussian noise. Is that true? 

If so, please highlight it as this is a rather simple "bad" example to build a lower bound, which makes the result even more appealing.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

comments: SUMMARY:
This paper studies the problem of low-rank matrix reconstruction from linear measurements, which is a matrix generalization of the well known sparse reconstruction setting. The provide new lower bounds for this problem under different error metrics, such as the Frobenius norm, essentially showing that non-trivial reconstruction error is impossible unless $\Omega(\log n)$ adaptive rounds of measurements are made. The hard instance is a rank-1 matrix planted into i.i.d Gaussian noise.

STRENGTHS:
- The problem solved is fundamental and to the best of my knowledge the contributions are new and generally applicable.
- The technical steps are clearly explained and well written, and look sound.

WEAKNESSES:
- The presentation in Section 1 and 1.3 could be improved. While I like that the explanation goes into technical detail, it could be significantly improved by having an outline of the 2-3 most significant contributions and a more modular presentation.

QUESTIONS:
Is there any potential connection between this work and the hardness results used in [1] for matrix completion based on planted clique (or the references within)? Are there any implications for matrix completion?

[1] Yudong Chen, Incoherence-Optimal Matrix Completion, IEEE Transactions on Information Theory, 2015

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07