Hyperparameter Tuning is All You Need for LISTA

1 Introduction . In this paper , we study the sparse linear inverse problem , where we strive to recover an unknown sparse vector x∗ ∈ Rn from its noisy linear measurement b generated from b = Ax∗ + ε , ( 1 ) where b ∈ Rm is the measurement that we observe , A ∈ Rm×n is the dictionary , x∗ ∈ Rn is the unknown ground truth that we aim to recover , and ε ∈ Rm is additive Gaussian white noise . For simplicity , each column of A , is normalized to have unit ` 2 norm . Typically , we have much fewer rows than columns in the dictionary A , i.e. , m n. Therefore , Equation ( 1 ) is an under-determined system . The sparse inverse problem , also known as sparse coding , plays essential roles in a wide range of applications including feature selection , signal reconstruction and pattern recognition . Sparse linear inverse problems are well studied in the literature of optimization . For example , it can be formulated into LASSO [ 29 ] and solved by many optimization algorithms [ 9 , 3 ] . These solutions explicitly consider and incorporate the sparsity prior and usually exploit iterative routines . Deep-learning-based approaches are proposed for empirically solving inverse problems recently [ 25 ] , which produce black-box models that are trained with data in an end-to-end way , while somehow ignoring the sparsity prior . Comparing the two streams , the former type , i.e. , the classic optimization methods , takes hundreds to thousands of iterations to generate accurate solutions , while black-box deep learning models , if properly trained , can achieve similar accuracy in tens of layers . However , classic methods come with data-agnostic convergence ( rate ) guarantees under suitable conditions , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ∗The first two authors made equal contributions . whereas deep learning methods only empirically work for instances similar to the training data , lacking theoretical guarantees and interpretability . More discussions are found in [ 6 ] . Unrolling is an uprising approach that bridges the two streams [ 22 , 21 ] . By converting each iteration of a classic iterative algorithm into one layer of a neural network , one can unroll and truncate a classic optimization method into a feed-forward neural network , with a finite number of layers . Relevant parameters ( e.g. , the dictionary , step sizes and thresholds ) in the original algorithm are transformed into learnable weights , that are trained on data . [ 15 ] pioneered the unrolling scheme for solving sparse coding and achieved great empirical success . By unrolling the Iterative Shrinkage-Thresholding Algorithm ( ISTA ) , the authors empirically demonstrated up to two magnitudes of acceleration of the learned ISTA ( LISTA ) compared to the original ISTA . The similar unrolling idea was later extended to numerous optimization problems and algorithms [ 28 , 30 , 26 , 13 , 36 , 2 , 16 , 8 , 5 ] . 1.1 Related Works . The balance between empirical performance and interpretability of unrolling motivates a line of theoretical efforts to explain their acceleration success . [ 23 ] attempted to understand LISTA by re-factorizing the Gram matrix of the dictionary A in ( 1 ) and proved that such re-parameterization converges sublinearly and empirically showed that it achieved similar acceleration gain to LISTA . [ 33 ] investigated unrolling iterative hard thresholding ( IHT ) and proved the existence of a matrix transformation that can improve the restricted isometry property constant of the original dictionary . Although it is difficult to search for the optimal transformation using classic optimization techniques , the authors of [ 33 ] argued that the data-driven training process is able to learn that transformation from data , and hence achieving the acceleration effect in practice . The work [ 7 ] extended the weight coupling necessary condition in [ 33 ] to LISTA . The authors for the first time proved the existence of a learnable parameter set that guarantees linear convergence of LISTA . Later in [ 18 ] , they further showed that the weight matrix in [ 7 ] can be analytically derived from the dictionary A and free from learning , reducing the learnable parameter set to only tens of scalars ( layer-wise step sizes and thresholds ) . The resulting algorithm , called Analytic LISTA ( ALISTA ) , sets one milestone in simplifying LISTA . [ 31 ] extends ALISTA by introducing gain and overshoot gating mechanisms and integrate them into unrolling . [ 37 ] proposed error-based thresholding ( EBT ) , which leverages a function of the layer-wise reconstruction error to suggest an appropriate threshold value for each observation on each layer . Another interesting work [ 4 ] uses a black-box LSTM model that takes the historical residual and update quantity as input and generates layerwise step sizes and thresholds in ALISTA . Their proposed model , called Neurally Augmented ALISTA ( NA-ALISTA ) , performed well in large-scale problems . 1.2 Our Contributions . Our research question is along the line of [ 18 , 37 , 4 ] : can we improve the LISTA parameterization , by further disentangling the learnable parameters from the observable terms ( e.g. , reconstruction errors , intermediate outputs ) ? Our aim is to create more light-weight , albeit more robust LISTA models with “ baked in '' adaptivity to testing samples from unseen distributions . The goal should be achieved without sacrificing our previously attained benefits : empirical performance , and convergence ( rate ) guarantees – if not improving them further . We present an affirmative answer to the above question by learning instance-optimal parameters with a new ALISTA parameterization . First , we prove that by augmenting ALISTA with momentum , the new unrolled network can achieve a better convergence rate over ALISTA . Second , we find that once the above LISTA network is further enabled with instance-optimal parameters , then a superlinear convergence can be attained . Lastly , we show that those instance-optimal parameters of each layer can be generated in closed forms , consequently reducing ALISTA to only learning three instanceand layer-invariant hyperparameters . Our new ultra-light weight network is termed as HyperLISTA . Compared to ALISTA , the new parameterization of HyperLISTA facilitates new backpropagation-free “ training '' options ( e.g. , bayesian optimization ) , and can unroll to more iterations in testing than training thanks to its layer-wise decomposed form . When comparing to state-of-the-art LISTA variants in experiments , HyperLISTA achieves the same strong performance on seen data distributions , and performs better when tested on unseen distributions ( e.g. , with different sparsity levels and nonzero magnitudes ) . In the supple- mentary materials , we also show that the superlinear convergence can be observed in HyperLISTA experiments , when our automatic parameter tuning is performed per instance . 2 Assumptions and Preliminaries . Mathematical Notations : We represent a matrix as a capital bold letter W , diag ( W ) means the diagonal elements of matrix W and the pseudoinverse of a matrix is denoted by W+ . A vector is represented as a lowercase bold letter x and its entries are represented as lowercase regular letters with lower indices x = [ x1 , x2 , · · · , x3 ] T . Upper indices represent iterations or layers . Moreover , 0,1 represent a vector with all zeros or all ones , respectively . “ supp ( x ) '' denotes the support , the indices of all nonzeros in vector x. ISTA Algorithm : With an ` 1 penalty term , the minimizer of LASSO , argminx 12‖Ax−b‖ 2 2+λ‖x‖1 , provides an estimate of the solution of the sparse recovery problem ( 1 ) . A popular algorithm to solve LASSO is Iterative Shrinkage-Thresholding Algorithm ( ISTA ) : x ( k+1 ) = ηλ/L ( x ( k ) + 1 L AT ( b−Ax ( k ) ) ) , k = 0 , 1 , 2 , . . . where ηλ/L is the element-wise soft-thresholding function ηθ ( x ) = sign ( x ) max ( 0 , |x| − θ ) with θ = λ/L and L is usually taken as the largest eigenvalue of ATA . LISTA Model : The original LISTA work [ 15 ] parameterizes ISTA as x ( k+1 ) = ηθ ( k ) ( W ( k ) 1 x ( k ) + W ( k ) 2 b ) , k = 0 , 1 , 2 , . . . ( 2 ) where Θ = { W ( k ) 1 , W ( k ) 2 , θ ( k ) } are learnable parameters to train from data . In the training process , iterations formulated in ( 2 ) are unrolled and truncated to K steps and the parameters are trained by minimizing the following loss function defined over the distribution of training samples Dtr : minimize Θ E ( x∗ , b ) ∼Dtr ∥∥∥x ( K ) ( Θ , b , x ( 0 ) ) − x∗∥∥∥2 2 . ( 3 ) Here x ( K ) ( Θ , b , x ( 0 ) ) , the output of the LISTA model , is a function with input Θ , b and x ( 0 ) . In practice , we usually set x ( 0 ) = 0 . We will use x ( K ) to refer to x ( K ) ( Θ , b , x ( 0 ) ) for brevity in the remainder of this paper . x∗ denotes the underlying ground truth defined in ( 1 ) . Although it can not be accessed in iterative algorithm ( 2 ) , it can be used as the label in supervised training ( 3 ) . ALISTA Model : ALISTA [ 18 ] advances LISTA with a lighter and more interpretable parameterization , where the learnable parameters are Θ = { γ ( k ) , θ ( k ) } and the iterations are formulated as x ( k+1 ) = ηp ( k ) θ ( k ) ( x ( k ) + γ ( k ) W T ( b−Ax ( k ) ) ) , ( 4 ) where W is calculated by solving the following optimization problem : W ∈ argmin W∈Rm×n ∥∥W TA∥∥2 F , s.t . diag ( W TA ) = 1 . ( 5 ) The thresholding operator is improved with support selection . At layer k , a certain percentage of entries with largest magnitudes are trusted as “ true support ” and will not be passed through thresholding . Specifically , the i-th element of ηp ( k ) θ ( k ) ( v ) is defined as ( ηp ( k ) θ ( k ) ( v ) ) i = vi : vi > θ ( k ) , i ∈ Sp ( k ) ( v ) , vi − θ ( k ) : vi > θ ( k ) , i /∈ Sp ( k ) ( v ) , 0 : −θ ( k ) ≤ vi ≤ θ ( k ) vi + θ ( k ) : vi < −θ ( k ) , i /∈ Sp ( k ) ( v ) , vi : vi < −θ ( k ) , i ∈ Sp ( k ) ( v ) , ( 6 ) where Sp ( k ) ( v ) includes the elements with the largest p ( k ) magnitudes in vector v ∈ Rn : Sp ( k ) ( v ) = { i1 , i2 , · · · , ip ( k ) ∣∣∣|vi1 | ≥ |vi2 | ≥ · · · |vip ( k ) | · · · ≥ |vin | } . ( 7 ) With p ( k ) = 0 , the operator reduces to the soft-thresholding and , with p ( k ) = n , the operator reduces to the hard-thresholding . Thus , operator ηp ( k ) θ ( k ) ( v ) is actually a balance between soft- and hard-thresholding . When k is small , p ( k ) is usually chosen as a small fraction of n and the operator tends to not trust the signal , and vice versa . In ALISTA , p ( k ) is proportional to the index k , capped by a maximal value , i.e. , p ( k ) = min ( p · k , pmax ) , where p and pmax are two hyperparameters that will be tuned manually . Later in this work , we will consider p ( k ) as a parameter that could be adaptively calculated by the information from previous iterations ( layers ) . More Notations and Assumptions : Throughout this paper , we refer to “ parameters ” as the learnable weights in the models , such as the thresholds θ ( k ) and the step sizes γ ( k ) that are defined in the ALISTA model , which are usually trained by back-propagation . In contrast , we refer “ hyperparameters ” as those constants that are often ad-hoc pre-defined to calculate parameters . For example , θ ( k ) = c1‖A+ ( Ax ( k ) − b ) ‖1 , where θ ( k ) is a parameter and c1 is a hyperparameter . Assumption 1 ( Basic assumptions ) . The signal x∗ and noise ε are sampled from the following set : X ( B , B , s , σ ) , { ( x∗ , ε ) ∣∣∣‖ε‖1 ≤ σ , ‖x∗‖0 ≤ s , 0 < B ≤ |x∗i | ≤ B , ∀i ∈ supp ( x∗ ) } . ( 8 )

Summary: The authors propose an extension to the ALISTA algorithm. Momentum is added to the update rule to speed up convergence, and parameters are adaptively determined per layer, rather than learned over a training distribution. Such adaptivity was recently proposed in the NA-ALISTA algorithm as well, but this solution is more light-weight. HyperLISTA was shown to converge faster and to lower optimum than ALISTA (and variants thereof) in a synthetic data setup.