Rate-Optimal Subspace Estimation on Random Graphs

1 Introduction . There have been many fruitful results on subspace estimation principal component analysis . Researchers have been considering random matrix with Gaussian and sub-Gaussian noise . In the present work , we focus on random bipartite graph , which have been a popular model in machine learning and data mining [ 26 , 11 , 19 ] . A bipartite graph is a graph whose vertices can be divided into two disjoint sets , U and V , such that every edge connects a vertex in U to one in V . In a random bipartite graph , an edge connecting node i ∈ U to node j ∈ V exists with probability Mij independently . Suppose |U| = n1 and |V| = n2 , a bipartite graph can be represented by an adjacency matrix A , whose elements Aij , ( i , j ) ∈ [ n1 ] × [ n2 ] are independent Bernoulli random variables with latent parameters Mij . In other words , we can focus on the random adjacency matrix A with entries Aij ∼ Bernoulli ( Mij ) for ( i , j ) ∈ [ n1 ] × [ n2 ] independently . ( 1 ) M with elements Mij is called the connectivity matrix . The goal of this paper is to find the rate optimal estimation of the latent connectivity matrix M and its singular space , under the assumption that M follows certain low-rank structure . Formally speaking , for the task of estimating matrix M , we consider the following parameter space : Θ1 ( n1 , n2 , p , r ) = { M ∈ [ 0 , p ] n1×n2 : rank ( M ) ≤ r } for positive integers n1 , n2 , r. ( 2 ) In other words , Θ1 ( n1 , n2 , p , r ) consists of all n1 × n2 matrices with entries between 0 and p and rank at most r. To avoid empty parameter space , we always assume r ≤ min { n1 , n2 } . The parameter p controls the sparsity of the bipartite graph . The work of Zhixin Zhou was conducted as a postdoctoral researcher at Baidu Research - Bellevue WA . The work of Cun-Hui Zhang was conducted as a consulting researcher at Baidu Research - Bellevue WA . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Matrix estimation has been studied under different assumptions . In [ 7 ] , the author assumes the random matrix has bounded entries . The universal singular value thresholding ( USVT ) algorithm is proved to be optimal in the parameter space with bounded nuclear norm . In [ 15 ] , the authors consider a more general parameter space with subexponential random matrices with bounded means . However , their approach is not optimal under our random graph assumption . One-bit matrix completion is considered in [ 5 , 9 ] , but parameters therein are not necessarily of low rank matrices , so the minimax error is not comparable with that of the family ( 2 ) in the present paper . In network analysis problems , the density of network is expected to be sparse . The upper bound of connection probability p in our model ( 2 ) plays a key role in error rates of the estimation . When we consider the singular space estimation problem , an extra assumption about the smallest non-zero singular value is required . We define the corresponding parameter space as follows : Θ2 ( n1 , n2 , p , r , σ∗ ) = { M ∈ [ 0 , p ] n1×n2 : rank ( M ) = r , σr ( M ) ≥ σ∗ } , ( 3 ) where σr ( M ) is the rth largest singular value of M. It is clear that Θ2 ( n1 , n2 , p , r , σ∗ ) ⊂ Θ1 ( n1 , n2 , p , r ) . In other words , in the singular space estimation problem , we consider parameters in Θ1 ( n1 , n2 , p , r ) whose smallest nonzero singular value is bounded below by σ∗ . Singular space estimation in the Gaussian case has been considered in [ 4 ] . To show the optimality of their estimation , they use packing number of Grassmanian manifold provided in [ 21 ] . In the random bipartite graph model that we consider in this paper , the connectivity matrix M in ( 3 ) is a merely proper subset of the Grassmanian manifold because the entries of M are contained in the interval [ 0 , p ] . As a consequence , the previous approach fails to calculate the lower bound of the packing number of the parameter space Θ2 ( n1 , n2 , p , r , σ∗ ) . In the literature of community detection , spectral methods have been applied in clustering tasks on random graphs [ 17 , 8 , 13 , 28 , 29 , 30 ] . The authors in the paper [ 27 ] consider this problem under information diffusion . However , these works have not focused on the optimality of singular space estimation . With the help of recent development in non-asymptotic analysis of random matrices , we propose a stochastic approach to compute the minimax lower bound of subspace estimation over the parameter space ( 3 ) . Here are some notations we will use in this paper . For real numbers a and b , a ∨ b = max { a , b } and a ∧ b = min { a , b } . σr ( M ) is the rth largest singular value of matrix M. ‖M‖op = σ1 ( M ) , i.e. , the largest singular value , is the spectral norm of M. ‖M‖F = ( ∑n1 i=1 ∑n2 j=1 M 2 ij ) 1/2 is the Frobenius norm of M. We note that ‖M‖2F = ∑n1∧n2 i=1 σi ( M ) 2 . Col ( M ) is the column space of M. Suppose a = ( a1 , a2 , . . . , an ) ∈ Rn , then diag ( a ) is an n × n diagonal matrix with entries a1 , a2 , . . . , an on the diagonal . For positive integer n , [ n ] : = { 1 , 2 , . . . , n } . For positive sequence an and bn , if there exists a constant C such that an ≤ Cbn for all n , then we write an = O ( bn ) or an . bn . If an = O ( bn ) and bn = O ( an ) , then we write an = Θ ( bn ) . If an/bn → 0 as n→∞ , then we write an = o ( bn ) or an bn . When we discuss asymptotic properties , we consider min { n1 , n2 } → ∞ , i.e. , the size the bipartite graph tends to infinity . The rest of this paper is organized as follows . In Section 2 , we introduce the algorithm for matrix estimation problem and also provide theoretical analysis for the error rate and minimax lower bound of this problem . Applying the result of matrix estimation , we propose the algorithm for column and row space estimation in Section 3 . Then we show that the algorithm outputs estimator with minimax optimal error rate . Numerical experiments on the algorithms appear in Section 4 . We discuss some possible future works in Section 5 . The proofs of the theorems can be found in the supplementary material . 2 Low Rank Matrix Estimation of Random Graph . Given a random bipartite graph , we aim to estimate true connecting probability of the adjacency matrix assuming that the connectivity matrix has low rank . To be more specific , in this section , we consider the task of low rank matrix estimation parameters belong to the space Θ1 ( n1 , n2 , p , r ) = { M ∈ [ 0 , p ] n1×n2 : rank ( M ) ≤ r } from ( 2 ) . Without loss of generality , we assume n1 ≥ n2 in this section . We recall that a random bipartite graph with adjacency matrix A is observed , and Aij ∼ Bernoulli ( Mij ) independently for ( i , j ) ∈ [ n1 ] × [ n2 ] . We will first proposed the algorithm to estimate M , then we will show the optimality of this estimator in the minimax sense . 2.1 Hard Singular Value Thresholding . We first introduce the algorithm to compute the estimator M̂ . The hard thresholding procedure is similar as universal singular value thresholding proposed in [ 7 ] . However , our algorithm is specifically designed for random graph , which leads to tighter error rate . The algorithm first applies degree truncation on bipartite adjacency matrix A , then compute the best low rank approximation on the resulting matrix by singular value decomposition ( SVD ) . For n1 × n2 matrix X , and recall that we assume n1 ≥ n2 , the SVD of X is defined by X = n2∑ i=1 σiUiV > i = UΛV > where U = ( U1 , U2 , . . . , Un2 ) , Λ = diag ( σ1 , σ2 , . . . , σn2 ) , and V = ( V1 , V2 , . . . , Vn2 ) with singular values σ1 ≥ σ2 ≥ · · · ≥ σn2 . Now we are ready to present Algorithm 1 for estimating M by hard singular value thresholding . Algorithm 1 Hard Singular Value Thresholding 1 : Input : adjacency matrix A , sparsity parameter p , rank r. 2 : Output : the estimate of M , M̂ . 3 : Set r′ = r ∧ bn2pc . 4 : If the degree of any row or column of A is greater than 2n1p , replace a subset of 1 ’ s in such rows and columns by 0 so that the column-degrees and the row degrees of the resulting matrix are bounded by 2n1p . We call this regularized matrix Are . 5 : Let M̂ be the best rank-r′ approximation of Are composed of the first r′ SVD components with the largest singular values . ( More details of this step will be discussed later in this section . ) Remark 1 . Suppose the rank of the connectivity matrix M is as large as the maximal expected row degree n2p , then it is not sufficient to propose a low rank approximation . In this case , the rate of the estimation error will be the same as a trivial estimation with M̂ = A . Step 3 in this algorithm plays role as checking if r is sufficiently small . Remark 2 . We note that the regularization in step 4 is necessary to achieve the optimal error rate . See experimental and theoretical result in [ 16 ] . It is worth noting that the regularization method presented in the algorithm is not the unique one . Besides replacing the entries by zeroes , we can multiply the columns and rows by a small constant so that the degree is bounded by 2n1p . Applying this method , the regularized adjacency matrix loses less information , so the estimation is more accurate . The threshold of regularization can be replaced by cn1p rather than 2n1p . In most cases , the best c can be smaller than 2 , though the theoretical estimation error rate can not be improved by choosing the optimal c. We will discuss the choice of c by simulation in Section 4 . We will describe more details in finding best rank-r′ approximation in step 5 . Suppose the regularized adjacency matrix Are from step 4 has SVD Are = n2∑ i=1 σiUiV > i , with singular values σ1 ≥ σ2 ≥ · · · ≥ σn2 , then we define the estimator M̂ to be the best rank-r′ approximation Are : M̂ = r′∑ i=1 σiUiV > i . We note that M̂ does not necessarily belong to the parameter space Θ1 ( n1 , n2 , p , r ) , but there is no further operation that can improve the error rate in the next theorem . Now we are ready to present the high probability bounds for estimation of M. The error will be measured by both operator norm ‖M̂−M‖op and Frobenius norm ‖M̂−M‖F . The upper bounds of the error rate are shown in the following theorem . Theorem 1 . Given the adjacency matrix A generated by parameter M from space Θ1 ( n1 , n2 , p , r ) and assuming n1 ≥ n2 , the estimator M̂ obtained from Algorithm 1 satisfies sup M∈Θ1 ( n1 , n2 , p , r ) P ( ‖M̂−M‖2op & ( n1p ) ∧ ( n1n2p2 ) ) ≤ n−11 ; ( 4 ) sup M∈Θ1 ( n1 , n2 , p , r ) P ( ‖M̂−M‖2F & ( n1pr ) ∧ ( n1n2p2 ) ) ≤ n−11 . ( 5 ) This theorem shows that if we consider the average loss of estimating entries of M , 1 n1n2 ‖M̂−M‖2F . pr n2 ∧ p2 holds with high probability as long as the dimension of the matrix tends to infinity . p2 is a trivial upper bound since this can be obtained by letting M̂ = 0 . As long as r = o ( n2p ) , the low rank structure of M leads to a better bound in entrywise loss . This coincides with the discussion in Remark 1 . This upper bound has also been studied in [ 13 , 28 ] . The authors applied their result for community detection when the number of communities are sufficiently smaller than the average degree of the graph . However , the term n1n2p2 is missing in previous works . Remark 3 . Without the Step 4 in Algorithm 1 , ( 4 ) can still achieve the error rate [ ( n1p ) ∨ ( log n1 ) ] ∧ ( n1n2p 2 ) with high probability . One can check the technical reason from the concentration of random graph without regularization in [ 10 , 17 , 16 ] . However , this error rate can not achieve the minimax lower bound in Theorem 3 .

Summary: The authors discuss the estimation of a connectivity matrix M \in [0,p]^{n_1 \times n_2} based on one instance of a generated random bipartite graph where node i of the one part is connected with node j of the other part with probability M_{i,j}. The authors discuss estimation of M in terms of operator and Frobenius norm, as we all estimation of the singular space of M.  The authors provide tight minimax results for all three notions of recovery. Their upper bounds are through a relatively (regularized) simple singular value thresholding (SVT) algorithm, not far from the well-known in the field universal singular value thresholding algorithm (USVT) by Chaterjee.