Range-Net: A High Precision Neural SVD

For Big Data applications , computing a rank-r Singular Value Decomposition ( SVD ) is restrictive due to the main memory requirements . Recently introduced streaming Randomized SVD schemes work under the restrictive assumption that the singular value spectrum of the data has an exponential decay . This is seldom true for any practical data . Further , the approximation errors in the singular vectors and values are high due to the randomized projection . We present Range-Net as a low memory alternative to rank-r SVD that satisfies the lower bound on tail-energy given by Eckart-Young-Mirsky ( EYM ) theorem at machine precision . Range-Net is a deterministic two-stage neural optimization approach with random initialization , where the memory requirement depends explicitly on the feature dimension and desired rank , independent of the sample dimension . The data samples are read in a streaming manner with the network minimization problem converging to the desired rank-r approximation . Range-Net is fully interpretable where all the network outputs and weights have a specific meaning . We provide theoretical guarantees that Range-Net extracted SVD factors satisfy EYM tail-energy lower bound with numerical experiments on real datasets at various scales that confirm these bounds . A comparison against the state-of-the-art streaming Randomized SVD shows that Range-Net is six orders of magnitude more accurate in terms of tail energy while correctly extracting the singular values and vectors . 1 INTRODUCTION . Singular Value Decomposition ( SVD ) is pivotal to exploratory data analysis in identifying an invariant structure under a minimalistic representation ( assumptions on the structure ) containing the span of resolvable information in the dataset . Finding a low rank structure is a fundamental task in applications including Image Compression ( de Souza et al. , 2015 ) , Image Recovery ( Brand , 2002 ) , Background Removal ( Wang et al. , 2018 ) , Recommendation Systems ( Zhang et al. , 2005 ) and as a pre-processing step for Clustering ( Drineas et al. , 2004 ) and Classification ( Jing et al. , 2017 ) . With the advent of digital sensors and modern day data acquisition technologies , the sheer amount of data now requires that we revisit the solution scheme with reduced memory consumption as the target . In this work , we reformulate SVD with special emphasis on the main memory requirement , with no loss in accuracy , that precludes it ’ s use for big data applications . It is well known that natural data matrices have a decaying spectrum wherein saving the data in memory in its original form is either redundant or not required from an application point of view . However , any assumption on the decay rate can only be validated if the singular value decomposition is known a priori , which is seldom the case in exploratory data analysis ( see Fig . 3 ) . Visually assessing a rank-r approximation for image processing applications might seem correct qualitatively ( see Fig . 4 ) but are still prone to large errors due to limited human vision acuity ( see Fig . 6 ) . This is further exacerbated when the application at hand is associated with scientific computations wherein the anomalies or unaccounted phenomena are still being explored from large scale datasets . The reader is preemptively referred to Fig . 19 where the high frequency features related to turbulence can not be disregarded . Furthermore , for classification and clustering problems where feature dimension reduction is desirable it is imperative that a low-rank approximation of a dataset contains most ( ≥ 90 % ) of the original information content without altering the subspace information . In this case , an over-sampled rank can exceed the feature dimension of the data itself ( see Section 4.2 ) . 1.1 PROBLEM STATEMENT . Let us denote the raw data matrix as X ∈ Rm×n of rank f ≤ min ( m , n ) and its approximation as Xr ∈ Rm×n . The SVD of X is X = UΣV T , where U ∈ Rm×f = [ u1 , · · · , uf ] and V ∈ Rn×f = [ v1 , · · · , vf ] are its left and right singular vectors respectively , and Σ ∈ Rf×f = diag ( σ1 , · · · , σf ) are its largest non-zero singular values . The rank r ( r ≤ f ) truncation of X is then Xr = UrΣrV Tr , where Σr = Σ [ 1 : r ] are the top r singular values , and Ur = U [ 1 : r ] and Vr = V [ 1 : r ] are the left and right singular vectors . In other words , X = UΣV T = UrΣrV Tr + Uf\rΣf\rV T f\r = Xr + Xf\r . Here , Uf\r , Vf\r are the trailing f − r left and right singular vectors . Theorem 1 . Eckart-Young-Mirsky Theorem ( Eckart & Young , 1936 ; Mirsky , 1960 ) : Let X ∈ Rm×n be a real , rank-f , matrix with m ≥ n with the singular value decomposition as X = UΣV T , where the orthonormal matrices U , V contain the left and right singular vectors of X and Σ is a diagonal matrix of singular values . Then for an arbitrary rank-r , r ≤ f matrix Br ∈ Rm×n , ‖X −Br‖F ≥ ‖X −Xr‖F where Xr = UrΣrVr with Σr is the diagonal matrix of the largest r singular values and Ur , Vr are the corresponding left and right singular vector matrices . The problem statement is then : Given X ∈ Rm×n find X̂ such that , arg min X̂∈Rm×n , rank ( X̂ ) ≤r ‖X − X̂‖F ( 1 ) In effect , the minimizer X̂∗ of the above problem gives us the rank-r approximation of X such that Xr = X̂∗ . In this work we utilize the minimizer of the above problem to extract the top rank-r SVD factors of X without loading the entire data matrix into the main memory . Note that the minimizer naturally gives the lower bound on this tail energy in addition to being a rank-r approximation . 1.2 MAIN CONTRIBUTIONS . Data and Representation Driven Neural SVD : The representation driven network loss terms ensures that the data matrix X is decomposed into the desired SVD factors such that X = UΣV T . In the absence of the representation enforcing loss term , the minimizer of Eq . 1 results in an arbitrary decomposition such that X = ABC different from SVD factors . A Deterministic Approach with GPU Bit-precision Results : The network can be initialized with weights drawn from a random distribution where the iterative minimization is deterministic . The streaming order of the samples is of no consequence and the user is free to choose the order in which the samples are processed in a batch-wise manner ( indexed or randomized ) . First Streaming Architecture with Exact Low Memory Cost : Range-net requires an exact memory specification based upon the desired rank-r and data dimensions X ∈ Rm×n given by r ( n+ r ) , independent of the sample dimension m. This is the first streaming algorithm that does require the user to wait until the streaming step is complete , contrary to randomized streaming algorithms . Layer-wise Fully Interpretable : Range-Net is a low-weight , fully interpretable , dense neural network where all the network weights and outputs have a precise definition . The network weights are placeholders for the right ( or left ) orthonormal vectors upon convergence of the network minimization problems ( see Appendix D ) . The user can explicitly plug a ground truth solution to verify the network design and directly arrive at the tail energy bound . 2 RELATED WORKS . The core idea behind randomized matrix decomposition is to make one or more passes over the data and compute efficient sketches . They can be broadly categorized into four branches : 1 ) Sampling based methods ( Subset Selection ( Boutsidis et al. , 2014 ) and Randomized CUR ( Drineas et al. , 2006 ) ) ; 2 ) Random Projection based QR ( Halko et al. , 2011 ) ; 3 ) Randomized SVD ( Halko et al. , 2011 ) ; and 4 ) Power iteration methods ( Musco & Musco , 2015 ) . The sketches can represent any combination of row space , column space or the space generated by the intersection of rows and columns ( core space ) . However , all of these methods require loading the entire data in memory . Readers are referred to Kishore Kumar & Schneider ( 2017 ) ; Ye et al . ( 2019 ) for an expanded survey . Conventional SVD although deterministic and accurate , becomes expensive when the data size increases and requires r passes over the data ( see Table 1 ) . The two branches of interest to us are the randomized SVD and Power iteration Methods for extracting SVD factors . Randomized SVD algorithms ( Halko et al. , 2011 ) are generally a two stage process : 1 ) Randomized Sketching uses random sampling to obtain a reduced matri ( x/ces ) which covers any combination of the row , column and core space of the data ; and 2 ) Deterministic Post-processing performs conventional SVD on the reduced system from Randomized Sketching stage . These approaches make only one pass over the data assuming that the singular value spectrum decays rapidly . Power iteration based approach ( Musco & Musco , 2015 ) requires multiple passes over the data and are used when the singular value spectrum decays slowly . This class of algorithm constructs a Krylov matrix inspired by Block Lanczos ( Golub & Underwood , 1977 ) to obtain a polynomial series expansion of the sketch . Although these algorithms achieve lower tail-energy errors , they can not be used in big-data applications when X itself is too large to be retained in the main memory . Here , constructing a Krylov matrix with higher order terms such as ( AAT or ATA ) is not feasible1 . Due to main-memory restrictions on remote compute machines , streaming ( Clarkson & Woodruff , 2009 ; Liberty , 2013 ) algorithms became popular . For low-rank SVD approximations these involve streaming the data and updating low-memory sketches covering the row , column and core spaces . Existing randomized SVD capable of streaming include Halko et al . ( 2011 ) ; Upadhyay ( 2016 ) ; Tropp et al . ( 2017a ; 2019 ) , each with different sketch sizes and upper bounds on approximation errors ( Table 1 ) . SketchySVD ( Tropp et al. , 2019 ) is the state of the art streaming randomized SVD , with sketch sizes comparable to it ’ s predecessors and tighter upper bounds on the tail energy and lower errors . As a two stage approach , SketchySVD ( Alg . 1 ) constructs an overestimated rank- ( k , s ) sketch of the data based on row , column and core projections . A QR decomposition on the row and column sketches gives an estimate of the rank-k subspace . This is followed by a conventional SVD on the core matrix to extract it ’ s singular values and vectors . Finally , the singular vectors are returned after projecting them back to the original row and column space . The time cost of SketchySVD is O ( k2 ( m + n ) ) with memory cost O ( k ( m+ n ) + s2 ) with oversamling parameters k = 4r + 1 and s = 2k + 1 .

Summary: The paper presents a streaming method to compute an approximate Singular Value Decomposition, without requiring to load the entire data matrix into the main memory. The first stage of the method identifies a basis optimizing for the constraint of orthonormality, as well as minimizing the error between the input and the reconstruction of the input based on the learnt basis. The second stage corresponds to a rotation step, which aligns the low-rank approximation extracted with the SVD factors.