Hyperparameter Tuning is All You Need for LISTA

1 Introduction . In this paper , we study the sparse linear inverse problem , where we strive to recover an unknown sparse vector x∗ ∈ Rn from its noisy linear measurement b generated from b = Ax∗ + ε , ( 1 ) where b ∈ Rm is the measurement that we observe , A ∈ Rm×n is the dictionary , x∗ ∈ Rn is the unknown ground truth that we aim to recover , and ε ∈ Rm is additive Gaussian white noise . For simplicity , each column of A , is normalized to have unit ` 2 norm . Typically , we have much fewer rows than columns in the dictionary A , i.e. , m n. Therefore , Equation ( 1 ) is an under-determined system . The sparse inverse problem , also known as sparse coding , plays essential roles in a wide range of applications including feature selection , signal reconstruction and pattern recognition . Sparse linear inverse problems are well studied in the literature of optimization . For example , it can be formulated into LASSO [ 29 ] and solved by many optimization algorithms [ 9 , 3 ] . These solutions explicitly consider and incorporate the sparsity prior and usually exploit iterative routines . Deep-learning-based approaches are proposed for empirically solving inverse problems recently [ 25 ] , which produce black-box models that are trained with data in an end-to-end way , while somehow ignoring the sparsity prior . Comparing the two streams , the former type , i.e. , the classic optimization methods , takes hundreds to thousands of iterations to generate accurate solutions , while black-box deep learning models , if properly trained , can achieve similar accuracy in tens of layers . However , classic methods come with data-agnostic convergence ( rate ) guarantees under suitable conditions , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ∗The first two authors made equal contributions . whereas deep learning methods only empirically work for instances similar to the training data , lacking theoretical guarantees and interpretability . More discussions are found in [ 6 ] . Unrolling is an uprising approach that bridges the two streams [ 22 , 21 ] . By converting each iteration of a classic iterative algorithm into one layer of a neural network , one can unroll and truncate a classic optimization method into a feed-forward neural network , with a finite number of layers . Relevant parameters ( e.g. , the dictionary , step sizes and thresholds ) in the original algorithm are transformed into learnable weights , that are trained on data . [ 15 ] pioneered the unrolling scheme for solving sparse coding and achieved great empirical success . By unrolling the Iterative Shrinkage-Thresholding Algorithm ( ISTA ) , the authors empirically demonstrated up to two magnitudes of acceleration of the learned ISTA ( LISTA ) compared to the original ISTA . The similar unrolling idea was later extended to numerous optimization problems and algorithms [ 28 , 30 , 26 , 13 , 36 , 2 , 16 , 8 , 5 ] . 1.1 Related Works . The balance between empirical performance and interpretability of unrolling motivates a line of theoretical efforts to explain their acceleration success . [ 23 ] attempted to understand LISTA by re-factorizing the Gram matrix of the dictionary A in ( 1 ) and proved that such re-parameterization converges sublinearly and empirically showed that it achieved similar acceleration gain to LISTA . [ 33 ] investigated unrolling iterative hard thresholding ( IHT ) and proved the existence of a matrix transformation that can improve the restricted isometry property constant of the original dictionary . Although it is difficult to search for the optimal transformation using classic optimization techniques , the authors of [ 33 ] argued that the data-driven training process is able to learn that transformation from data , and hence achieving the acceleration effect in practice . The work [ 7 ] extended the weight coupling necessary condition in [ 33 ] to LISTA . The authors for the first time proved the existence of a learnable parameter set that guarantees linear convergence of LISTA . Later in [ 18 ] , they further showed that the weight matrix in [ 7 ] can be analytically derived from the dictionary A and free from learning , reducing the learnable parameter set to only tens of scalars ( layer-wise step sizes and thresholds ) . The resulting algorithm , called Analytic LISTA ( ALISTA ) , sets one milestone in simplifying LISTA . [ 31 ] extends ALISTA by introducing gain and overshoot gating mechanisms and integrate them into unrolling . [ 37 ] proposed error-based thresholding ( EBT ) , which leverages a function of the layer-wise reconstruction error to suggest an appropriate threshold value for each observation on each layer . Another interesting work [ 4 ] uses a black-box LSTM model that takes the historical residual and update quantity as input and generates layerwise step sizes and thresholds in ALISTA . Their proposed model , called Neurally Augmented ALISTA ( NA-ALISTA ) , performed well in large-scale problems . 1.2 Our Contributions . Our research question is along the line of [ 18 , 37 , 4 ] : can we improve the LISTA parameterization , by further disentangling the learnable parameters from the observable terms ( e.g. , reconstruction errors , intermediate outputs ) ? Our aim is to create more light-weight , albeit more robust LISTA models with “ baked in '' adaptivity to testing samples from unseen distributions . The goal should be achieved without sacrificing our previously attained benefits : empirical performance , and convergence ( rate ) guarantees – if not improving them further . We present an affirmative answer to the above question by learning instance-optimal parameters with a new ALISTA parameterization . First , we prove that by augmenting ALISTA with momentum , the new unrolled network can achieve a better convergence rate over ALISTA . Second , we find that once the above LISTA network is further enabled with instance-optimal parameters , then a superlinear convergence can be attained . Lastly , we show that those instance-optimal parameters of each layer can be generated in closed forms , consequently reducing ALISTA to only learning three instanceand layer-invariant hyperparameters . Our new ultra-light weight network is termed as HyperLISTA . Compared to ALISTA , the new parameterization of HyperLISTA facilitates new backpropagation-free “ training '' options ( e.g. , bayesian optimization ) , and can unroll to more iterations in testing than training thanks to its layer-wise decomposed form . When comparing to state-of-the-art LISTA variants in experiments , HyperLISTA achieves the same strong performance on seen data distributions , and performs better when tested on unseen distributions ( e.g. , with different sparsity levels and nonzero magnitudes ) . In the supple- mentary materials , we also show that the superlinear convergence can be observed in HyperLISTA experiments , when our automatic parameter tuning is performed per instance . 2 Assumptions and Preliminaries . Mathematical Notations : We represent a matrix as a capital bold letter W , diag ( W ) means the diagonal elements of matrix W and the pseudoinverse of a matrix is denoted by W+ . A vector is represented as a lowercase bold letter x and its entries are represented as lowercase regular letters with lower indices x = [ x1 , x2 , · · · , x3 ] T . Upper indices represent iterations or layers . Moreover , 0,1 represent a vector with all zeros or all ones , respectively . “ supp ( x ) '' denotes the support , the indices of all nonzeros in vector x. ISTA Algorithm : With an ` 1 penalty term , the minimizer of LASSO , argminx 12‖Ax−b‖ 2 2+λ‖x‖1 , provides an estimate of the solution of the sparse recovery problem ( 1 ) . A popular algorithm to solve LASSO is Iterative Shrinkage-Thresholding Algorithm ( ISTA ) : x ( k+1 ) = ηλ/L ( x ( k ) + 1 L AT ( b−Ax ( k ) ) ) , k = 0 , 1 , 2 , . . . where ηλ/L is the element-wise soft-thresholding function ηθ ( x ) = sign ( x ) max ( 0 , |x| − θ ) with θ = λ/L and L is usually taken as the largest eigenvalue of ATA . LISTA Model : The original LISTA work [ 15 ] parameterizes ISTA as x ( k+1 ) = ηθ ( k ) ( W ( k ) 1 x ( k ) + W ( k ) 2 b ) , k = 0 , 1 , 2 , . . . ( 2 ) where Θ = { W ( k ) 1 , W ( k ) 2 , θ ( k ) } are learnable parameters to train from data . In the training process , iterations formulated in ( 2 ) are unrolled and truncated to K steps and the parameters are trained by minimizing the following loss function defined over the distribution of training samples Dtr : minimize Θ E ( x∗ , b ) ∼Dtr ∥∥∥x ( K ) ( Θ , b , x ( 0 ) ) − x∗∥∥∥2 2 . ( 3 ) Here x ( K ) ( Θ , b , x ( 0 ) ) , the output of the LISTA model , is a function with input Θ , b and x ( 0 ) . In practice , we usually set x ( 0 ) = 0 . We will use x ( K ) to refer to x ( K ) ( Θ , b , x ( 0 ) ) for brevity in the remainder of this paper . x∗ denotes the underlying ground truth defined in ( 1 ) . Although it can not be accessed in iterative algorithm ( 2 ) , it can be used as the label in supervised training ( 3 ) . ALISTA Model : ALISTA [ 18 ] advances LISTA with a lighter and more interpretable parameterization , where the learnable parameters are Θ = { γ ( k ) , θ ( k ) } and the iterations are formulated as x ( k+1 ) = ηp ( k ) θ ( k ) ( x ( k ) + γ ( k ) W T ( b−Ax ( k ) ) ) , ( 4 ) where W is calculated by solving the following optimization problem : W ∈ argmin W∈Rm×n ∥∥W TA∥∥2 F , s.t . diag ( W TA ) = 1 . ( 5 ) The thresholding operator is improved with support selection . At layer k , a certain percentage of entries with largest magnitudes are trusted as “ true support ” and will not be passed through thresholding . Specifically , the i-th element of ηp ( k ) θ ( k ) ( v ) is defined as ( ηp ( k ) θ ( k ) ( v ) ) i = vi : vi > θ ( k ) , i ∈ Sp ( k ) ( v ) , vi − θ ( k ) : vi > θ ( k ) , i /∈ Sp ( k ) ( v ) , 0 : −θ ( k ) ≤ vi ≤ θ ( k ) vi + θ ( k ) : vi < −θ ( k ) , i /∈ Sp ( k ) ( v ) , vi : vi < −θ ( k ) , i ∈ Sp ( k ) ( v ) , ( 6 ) where Sp ( k ) ( v ) includes the elements with the largest p ( k ) magnitudes in vector v ∈ Rn : Sp ( k ) ( v ) = { i1 , i2 , · · · , ip ( k ) ∣∣∣|vi1 | ≥ |vi2 | ≥ · · · |vip ( k ) | · · · ≥ |vin | } . ( 7 ) With p ( k ) = 0 , the operator reduces to the soft-thresholding and , with p ( k ) = n , the operator reduces to the hard-thresholding . Thus , operator ηp ( k ) θ ( k ) ( v ) is actually a balance between soft- and hard-thresholding . When k is small , p ( k ) is usually chosen as a small fraction of n and the operator tends to not trust the signal , and vice versa . In ALISTA , p ( k ) is proportional to the index k , capped by a maximal value , i.e. , p ( k ) = min ( p · k , pmax ) , where p and pmax are two hyperparameters that will be tuned manually . Later in this work , we will consider p ( k ) as a parameter that could be adaptively calculated by the information from previous iterations ( layers ) . More Notations and Assumptions : Throughout this paper , we refer to “ parameters ” as the learnable weights in the models , such as the thresholds θ ( k ) and the step sizes γ ( k ) that are defined in the ALISTA model , which are usually trained by back-propagation . In contrast , we refer “ hyperparameters ” as those constants that are often ad-hoc pre-defined to calculate parameters . For example , θ ( k ) = c1‖A+ ( Ax ( k ) − b ) ‖1 , where θ ( k ) is a parameter and c1 is a hyperparameter . Assumption 1 ( Basic assumptions ) . The signal x∗ and noise ε are sampled from the following set : X ( B , B , s , σ ) , { ( x∗ , ε ) ∣∣∣‖ε‖1 ≤ σ , ‖x∗‖0 ≤ s , 0 < B ≤ |x∗i | ≤ B , ∀i ∈ supp ( x∗ ) } . ( 8 )

Summary: This paper proposes an ultra-light variant of LISTA, called HyperLISTA, for solving sparse linear inverse problems. The authors’ adaptive parameterization reduces the training of HyperLISTA to only tuning three instance- and layer-invariant hyperparameters. HyperLISTA is theoretically proved and empirically observed to have super-linear convergence rates: the first time ever so in LISTA literature. It also leads to superior test-time adaptivity.