Range-Net: A High Precision Neural SVD

For Big Data applications , computing a rank-r Singular Value Decomposition ( SVD ) is restrictive due to the main memory requirements . Recently introduced streaming Randomized SVD schemes work under the restrictive assumption that the singular value spectrum of the data has an exponential decay . This is seldom true for any practical data . Further , the approximation errors in the singular vectors and values are high due to the randomized projection . We present Range-Net as a low memory alternative to rank-r SVD that satisfies the lower bound on tail-energy given by Eckart-Young-Mirsky ( EYM ) theorem at machine precision . Range-Net is a deterministic two-stage neural optimization approach with random initialization , where the memory requirement depends explicitly on the feature dimension and desired rank , independent of the sample dimension . The data samples are read in a streaming manner with the network minimization problem converging to the desired rank-r approximation . Range-Net is fully interpretable where all the network outputs and weights have a specific meaning . We provide theoretical guarantees that Range-Net extracted SVD factors satisfy EYM tail-energy lower bound with numerical experiments on real datasets at various scales that confirm these bounds . A comparison against the state-of-the-art streaming Randomized SVD shows that Range-Net is six orders of magnitude more accurate in terms of tail energy while correctly extracting the singular values and vectors . 1 INTRODUCTION . Singular Value Decomposition ( SVD ) is pivotal to exploratory data analysis in identifying an invariant structure under a minimalistic representation ( assumptions on the structure ) containing the span of resolvable information in the dataset . Finding a low rank structure is a fundamental task in applications including Image Compression ( de Souza et al. , 2015 ) , Image Recovery ( Brand , 2002 ) , Background Removal ( Wang et al. , 2018 ) , Recommendation Systems ( Zhang et al. , 2005 ) and as a pre-processing step for Clustering ( Drineas et al. , 2004 ) and Classification ( Jing et al. , 2017 ) . With the advent of digital sensors and modern day data acquisition technologies , the sheer amount of data now requires that we revisit the solution scheme with reduced memory consumption as the target . In this work , we reformulate SVD with special emphasis on the main memory requirement , with no loss in accuracy , that precludes it ’ s use for big data applications . It is well known that natural data matrices have a decaying spectrum wherein saving the data in memory in its original form is either redundant or not required from an application point of view . However , any assumption on the decay rate can only be validated if the singular value decomposition is known a priori , which is seldom the case in exploratory data analysis ( see Fig . 3 ) . Visually assessing a rank-r approximation for image processing applications might seem correct qualitatively ( see Fig . 4 ) but are still prone to large errors due to limited human vision acuity ( see Fig . 6 ) . This is further exacerbated when the application at hand is associated with scientific computations wherein the anomalies or unaccounted phenomena are still being explored from large scale datasets . The reader is preemptively referred to Fig . 19 where the high frequency features related to turbulence can not be disregarded . Furthermore , for classification and clustering problems where feature dimension reduction is desirable it is imperative that a low-rank approximation of a dataset contains most ( ≥ 90 % ) of the original information content without altering the subspace information . In this case , an over-sampled rank can exceed the feature dimension of the data itself ( see Section 4.2 ) . 1.1 PROBLEM STATEMENT . Let us denote the raw data matrix as X ∈ Rm×n of rank f ≤ min ( m , n ) and its approximation as Xr ∈ Rm×n . The SVD of X is X = UΣV T , where U ∈ Rm×f = [ u1 , · · · , uf ] and V ∈ Rn×f = [ v1 , · · · , vf ] are its left and right singular vectors respectively , and Σ ∈ Rf×f = diag ( σ1 , · · · , σf ) are its largest non-zero singular values . The rank r ( r ≤ f ) truncation of X is then Xr = UrΣrV Tr , where Σr = Σ [ 1 : r ] are the top r singular values , and Ur = U [ 1 : r ] and Vr = V [ 1 : r ] are the left and right singular vectors . In other words , X = UΣV T = UrΣrV Tr + Uf\rΣf\rV T f\r = Xr + Xf\r . Here , Uf\r , Vf\r are the trailing f − r left and right singular vectors . Theorem 1 . Eckart-Young-Mirsky Theorem ( Eckart & Young , 1936 ; Mirsky , 1960 ) : Let X ∈ Rm×n be a real , rank-f , matrix with m ≥ n with the singular value decomposition as X = UΣV T , where the orthonormal matrices U , V contain the left and right singular vectors of X and Σ is a diagonal matrix of singular values . Then for an arbitrary rank-r , r ≤ f matrix Br ∈ Rm×n , ‖X −Br‖F ≥ ‖X −Xr‖F where Xr = UrΣrVr with Σr is the diagonal matrix of the largest r singular values and Ur , Vr are the corresponding left and right singular vector matrices . The problem statement is then : Given X ∈ Rm×n find X̂ such that , arg min X̂∈Rm×n , rank ( X̂ ) ≤r ‖X − X̂‖F ( 1 ) In effect , the minimizer X̂∗ of the above problem gives us the rank-r approximation of X such that Xr = X̂∗ . In this work we utilize the minimizer of the above problem to extract the top rank-r SVD factors of X without loading the entire data matrix into the main memory . Note that the minimizer naturally gives the lower bound on this tail energy in addition to being a rank-r approximation . 1.2 MAIN CONTRIBUTIONS . Data and Representation Driven Neural SVD : The representation driven network loss terms ensures that the data matrix X is decomposed into the desired SVD factors such that X = UΣV T . In the absence of the representation enforcing loss term , the minimizer of Eq . 1 results in an arbitrary decomposition such that X = ABC different from SVD factors . A Deterministic Approach with GPU Bit-precision Results : The network can be initialized with weights drawn from a random distribution where the iterative minimization is deterministic . The streaming order of the samples is of no consequence and the user is free to choose the order in which the samples are processed in a batch-wise manner ( indexed or randomized ) . First Streaming Architecture with Exact Low Memory Cost : Range-net requires an exact memory specification based upon the desired rank-r and data dimensions X ∈ Rm×n given by r ( n+ r ) , independent of the sample dimension m. This is the first streaming algorithm that does require the user to wait until the streaming step is complete , contrary to randomized streaming algorithms . Layer-wise Fully Interpretable : Range-Net is a low-weight , fully interpretable , dense neural network where all the network weights and outputs have a precise definition . The network weights are placeholders for the right ( or left ) orthonormal vectors upon convergence of the network minimization problems ( see Appendix D ) . The user can explicitly plug a ground truth solution to verify the network design and directly arrive at the tail energy bound . 2 RELATED WORKS . The core idea behind randomized matrix decomposition is to make one or more passes over the data and compute efficient sketches . They can be broadly categorized into four branches : 1 ) Sampling based methods ( Subset Selection ( Boutsidis et al. , 2014 ) and Randomized CUR ( Drineas et al. , 2006 ) ) ; 2 ) Random Projection based QR ( Halko et al. , 2011 ) ; 3 ) Randomized SVD ( Halko et al. , 2011 ) ; and 4 ) Power iteration methods ( Musco & Musco , 2015 ) . The sketches can represent any combination of row space , column space or the space generated by the intersection of rows and columns ( core space ) . However , all of these methods require loading the entire data in memory . Readers are referred to Kishore Kumar & Schneider ( 2017 ) ; Ye et al . ( 2019 ) for an expanded survey . Conventional SVD although deterministic and accurate , becomes expensive when the data size increases and requires r passes over the data ( see Table 1 ) . The two branches of interest to us are the randomized SVD and Power iteration Methods for extracting SVD factors . Randomized SVD algorithms ( Halko et al. , 2011 ) are generally a two stage process : 1 ) Randomized Sketching uses random sampling to obtain a reduced matri ( x/ces ) which covers any combination of the row , column and core space of the data ; and 2 ) Deterministic Post-processing performs conventional SVD on the reduced system from Randomized Sketching stage . These approaches make only one pass over the data assuming that the singular value spectrum decays rapidly . Power iteration based approach ( Musco & Musco , 2015 ) requires multiple passes over the data and are used when the singular value spectrum decays slowly . This class of algorithm constructs a Krylov matrix inspired by Block Lanczos ( Golub & Underwood , 1977 ) to obtain a polynomial series expansion of the sketch . Although these algorithms achieve lower tail-energy errors , they can not be used in big-data applications when X itself is too large to be retained in the main memory . Here , constructing a Krylov matrix with higher order terms such as ( AAT or ATA ) is not feasible1 . Due to main-memory restrictions on remote compute machines , streaming ( Clarkson & Woodruff , 2009 ; Liberty , 2013 ) algorithms became popular . For low-rank SVD approximations these involve streaming the data and updating low-memory sketches covering the row , column and core spaces . Existing randomized SVD capable of streaming include Halko et al . ( 2011 ) ; Upadhyay ( 2016 ) ; Tropp et al . ( 2017a ; 2019 ) , each with different sketch sizes and upper bounds on approximation errors ( Table 1 ) . SketchySVD ( Tropp et al. , 2019 ) is the state of the art streaming randomized SVD , with sketch sizes comparable to it ’ s predecessors and tighter upper bounds on the tail energy and lower errors . As a two stage approach , SketchySVD ( Alg . 1 ) constructs an overestimated rank- ( k , s ) sketch of the data based on row , column and core projections . A QR decomposition on the row and column sketches gives an estimate of the rank-k subspace . This is followed by a conventional SVD on the core matrix to extract it ’ s singular values and vectors . Finally , the singular vectors are returned after projecting them back to the original row and column space . The time cost of SketchySVD is O ( k2 ( m + n ) ) with memory cost O ( k ( m+ n ) + s2 ) with oversamling parameters k = 4r + 1 and s = 2k + 1 .

Summary: The paper presents a multipass streaming algorithm for rank-r SVD.  Given an input matrix X in R^{mxn} the algorithm identifies two matrices V* in R^{nxr} and H in R^{rxr}. V* has orthonormal columns that span the top r right singular vectors.  H rotates V* so that V*H = Vr, where Vr in R^{nxr} is the matrix of the top-r right singular vectors.  V* and H are computed by minibatch gradient descent, trained until convergence with custom loss functions.  V* is computed in the first stage and then H in the second stage.   The algorithm is accurate and uses very little storage.