ALISTA: Analytic Weights Are As Good As Learned Weights in LISTA

1 INTRODUCTION . Sparse vector recovery , or sparse coding , is a classical problem in source coding , signal reconstruction , pattern recognition and feature selection . There is an unknown sparse vector x∗ = [ x∗1 , · · · , x∗M ] T ∈ RM . We observe its noisy linear measurements : b = M∑ m=1 dmx ∗ m + ε = Dx ∗ + ε , ( 1 ) where b ∈ RN , D = [ d1 , · · · , dM ] ∈ RN×M is the dictionary , and ε ∈ RN is additive Gaussian white noise . For simplicity , each column of D , named as a dictionary kernel , is normalized , that is , ‖dm‖2 = ‖D : ,m‖2 = 1 , m = 1 , 2 , · · · , M . Typically , we have N M , so Equation ( 1 ) is an under-determined system . However , when x∗ is sufficiently sparse , it can be recovered faithfully . A popular approach is to solve the LASSO problem below ( where λ is a scalar ) : minimize x 1 2 ‖b−Dx‖22 + λ‖x‖1 ( 2 ) using iterative algorithms such as the iterative shrinkage thresholding algorithm ( ISTA ) : x ( k+1 ) = ηλ/L ( x ( k ) + 1 L DT ( b−Dx ( k ) ) ) , k = 0 , 1 , 2 , . . . ( 3 ) ∗These authors contributed equally and are listed alphabetically . where ηθ is the soft-thresholding function1 and L is usually taken as the largest eigenvalue of DTD . Inspired by ISTA , the authors of ( Gregor & LeCun , 2010 ) proposed to learn the weights in the matrices in ISTA rather than fixing them . Their methods is called Learned ISTA ( LISTA ) and resembles a recurrent neural network ( RNN ) . If the iteration is truncated to K iterations , LISTA becomes a K-layer feed-forward neural network with side connections . Specifically , LISTA is : x ( k+1 ) = ηθ ( k ) ( W ( k ) 1 b + W ( k ) 2 x ( k ) ) , k = 0 , 1 , · · · , K − 1 . ( 4 ) If we set W ( k ) 1 ≡ 1LD T , W ( k ) 2 ≡ I − 1LD TD , θ ( k ) ≡ 1Lλ , then LISTA recovers ISTA . Given each pair of sparse vector and its noisy measurements ( x∗ , b ) , applying ( 4 ) from some initial point x ( 0 ) and using b as the input yields x ( k ) . Our goal is to choose the parameters Θ = { W ( k ) 1 , W ( k ) w , θ ( k ) } k=0,1 , ... , K−1 such that x ( k ) is close to x∗ for all sparse x∗ following some distribution P . Therefore , given the distribution P , all parameters in Θ are subject to learning : minimize Θ Ex∗ , b∼P ∥∥∥x ( K ) ( Θ , b , x ( 0 ) ) − x∗∥∥∥2 2 . ( 5 ) This problem is approximately solved over a training dataset { ( x∗i , bi ) } Ni=1 sampled from P . Many empirical results , e.g. , ( Gregor & LeCun , 2010 ; Sprechmann et al. , 2015 ; Wang et al. , 2016b ) , show that a trained K-layer LISTA ( with K usually set to 10 ∼ 20 ) or its variants can generalize more than well to unseen samples ( x′ , b′ ) from the same distribution and recover x′ from b′ to the same accuracy within one or two order-of-magnitude fewer iterations than the original ISTA . Additionally , the accuracies of the outputs { x ( k ) } of the layers k = 1 , .. , K gradually improve . However , such networks will generalize worse when the input deviates from the training distribution ( e.g. , when D varies ) , in contrast to the classical iterative algorithms such as ISTA that are trainingfree and thus agnostic to the input distribution . The Analysis-Synthesis model ( Rubinstein & Elad , 2014 ; Yang et al. , 2016 ) could also be viewed as a special LISTA model with only one layer ( K = 1 ) . More recently , the convolutional sparse coding ( CSC ) , an extension of the sparse coding ( 1 ) , gains increasingly attention in the machine learning area . ( Sreter & Giryes , 2018 ) showed that the CSC could be similarly approximated and accelerated by a LISTA-type feed-forward network . ( Tolooshams et al. , 2018 ) designed a structure of sparse auto-encoder inspired by multi-layer CSC . ( Papyan et al. , 2016 ; Sulam et al. , 2017 ) also revealed CSC as a potentially useful tool for understanding general convolutional neural networks ( CNNs ) . 1.1 RELATED WORK . Despite the empirical success ( Sprechmann et al. , 2015 ; Wang et al. , 2016a ; b ; c ; d ; Zhang & Ghanem , 2018 ; Zhou et al. , 2018 ; Ito et al. , 2018 ) in constructing fast trainable regressors for approximating iterative sparse solvers , the theoretical understanding of such approximations remains limited . A handful of recent works have been investigating the theory of LISTA . ( Moreau & Bruna , 2017 ) re-factorized the Gram matrix of dictionary , by trying to nearly diagonalize the Gram matrix with a basis , subject to a small ` 1 perturbation . They thus re-parameterized LISTA a new factorized architecture that achieved similar acceleration gain to LISTA , hence ending up with an “ indirect ” proof . They concluded that LISTA can converge faster than ISTA , but still sublinearly . ( Giryes et al. , 2018 ) interpreted LISTA as a projected gradient descent descent ( PGD ) where the projection step was inaccurate , which enables a trade-off between approximation error and convergence speed . The latest work ( Chen et al. , 2018 ) presented the more related results to ours : they introduced necessary conditions for the LISTA weight structure in order to achieve asymptotic linear convergence of LISTA , which also proved to be a theoretical convergence rate upper bound . They also introduced a thresholding scheme for practically improving the convergence speed . Note that , none of the above works extended their discussions to CSC and its similar LISTA-type architectures . Several other works examined the theoretical properties of some sibling architectures to LISTA . ( Xin et al. , 2016 ) studied the model proposed by ( Wang et al. , 2016b ) , which unfolded/truncated the iterative hard thresholding ( IHT ) algorithm instead of ISTA , for approximating the solution to ` 0- minimization . They showed that the learnable fast regressor can be obtained by using a transformed dictionary with improved restricted isometry property ( RIP ) . However , their discussions are not 1Soft- thresholding function is defined in a component-wise way : ηθ ( x ) = sign ( x ) max ( 0 , |x| − θ ) applicable to LISTA directly , although IHT is linearly convergent ( Blumensath & Davies , 2009 ) under rather strong assumptions . Their discussions were also limited to linear sparse coding and resulting fully-connected networks only . ( Borgerding et al. , 2017 ; Metzler et al. , 2017 ) studied a similar learning-based model inspired from another LASSO solver , called approximated message passing ( AMP ) . ( Borgerding et al. , 2017 ) showed the MMSE-optimality of an AMP-inspired model , but not accompanied with any convergence rate result . Also , the popular assumption in analyzing AMP algorithms ( called “ state evolution ” ) does not hold when analyzing ISTA . 1.2 MOTIVATION AND CONTRIBUTIONS . This paper presents multi-fold contributions in advancing the theoretical understanding of LISTA , beyond state-of-the-art results . Firstly , we show that the layer-wise weights in LISTA need not being learned from data . That is based on decoupling LISTA training into a data-free analytic optimization stage followed by a lighter-weight data-driven learning stage without compromising the optimal linear convergence rate proved in ( Chen et al. , 2018 ) . We establish a minimum-coherence criterion between the desired LISTA weights and the dictionary D , which leads to an efficient algorithm that can analytically solve the former from the latter , independent of the distribution of x . The data-driven training is then reduced to learning layer-wise step sizes and thresholds only , which will fit the distribution of x . The new scheme , called Analytic LISTA ( ALISTA ) , provides important insights into the working mechanism of LISTA . Experiments shows ALISTA to perform comparably with previous LISTA models ( Gregor & LeCun , 2010 ; Chen et al. , 2018 ) with much lighter-weight training . Then , we extend the above discussions and conclusions to CSC , and introduce an efficient algorithm to solve the convolutional version of coherence minimization . Further , we introduce a new robust LISTA learning scheme benefiting from the decoupled structure , by adding perturbations to D during training . The resulting model is shown to possess much stronger robustness when the input distribution varies , even when D changes to some extent , compared to classical LISTA models that learn to ( over- ) fit one specific D . 2 ANALYTIC LISTA : CALCULATING WEIGHTS WITHOUT TRAINING . We theoretically analyze the LISTA-CPSS model defined in ( Chen et al. , 2018 ) : x ( k+1 ) = ηθ ( k ) ( x ( k ) − ( W ( k ) ) T ( Dx ( k ) − b ) ) , ( 6 ) where W ( k ) = [ w ( k ) 1 , · · · , w ( k ) M ] ∈ RN×M is a linear operator with the same dimensionality with D , x ( k ) = [ x ( k ) 1 , · · · , x ( k ) M ] is the kth layer node . In ( 6 ) , Θ = { W ( k ) , θ ( k ) } k are parameters to train . Model ( 6 ) can be derived from ( 4 ) with W ( k ) 1 = ( W ( k ) ) T , W ( k ) 2 = I−W ( k ) 1 D. ( Chen et al. , 2018 ) showed that ( 6 ) has the same representation capability with ( 4 ) on the sparse recovery problem , with a specifically light weight structure . Our theoretical analysis will further define and establish properties of “ good ” parameters Θ in ( 6 ) , and then discuss how to analytically compute those good parameters rather than relying solely on black-box training . In this way , the LISTA model could be further significantly simplified , with little performance loss . The proofs of all the theorems in this paper are provided in the appendix . 2.1 RECOVERY ERROR UPPER BOUND . We start with an assumption on the “ ground truth ” signal x∗ and the noise ε . Assumption 1 ( Basic assumptions ) . Signal x∗ is sampled from the following set : x∗ ∈ X ( B , s ) , { x∗ ∣∣∣|x∗i | ≤ B , ∀i , ‖x∗‖0 ≤ s } . ( 7 ) In other words , x∗ is bounded and s-sparse2 ( s ≥ 2 ) . Furthermore , we assume ε = 0 . The zero-noise assumption is for simplicity of the proofs . Our experiments will show that our models are robust to noisy cases . The mutual coherence of the dictionary D is a significant concept in compressive sensing ( Donoho & Elad , 2003 ; Elad , 2007 ; Lu et al. , 2018 ) . A dictionary with small coherence possesses better sparse recovery performance . Motivated by this point , we introduce the following definition . 2A signal is s-sparse if it has no more than s non-zero entries . Definition 1 . Given D ∈ RN×M with each of its column normalized , we define the generalized mutual coherence : µ̃ ( D ) = inf W∈RN×M ( W : ,i ) TD : ,i=1,1≤i≤M { max i 6=j 1≤i , j≤M ( W : ,i ) TD : ,j } . ( 8 ) Additionally , We define W ( D ) = { W ∈ RN×M : W attains the infimum given ( 8 ) } . A weight matrix W is “ good ” if W ∈ W ( D ) . In the above definition , problem ( 8 ) is feasible and attainable , i.e. , W ( D ) 6= ∅ , which was proven in Lemma 1 of ( Chen et al. , 2018 ) . Theorem 1 ( Recovery error upper bound ) . Take any x∗ ∈ X ( B , s ) , any W ∈ W ( D ) , and any sequence γ ( k ) ∈ ( 0 , 22µ̃s−µ̃+1 ) . Using them , define the parameters { W ( k ) , θ ( k ) } : W ( k ) = γ ( k ) W , θ ( k ) = γ ( k ) µ̃ ( D ) sup x∗∈X ( B , s ) { ‖x ( k ) ( x∗ ) − x∗‖1 } , ( 9 ) while the sequence { x ( k ) ( x∗ ) } ∞k=1 is generated by ( 6 ) using the above parameters and x ( 0 ) = 0 ( Note that each x ( k ) ( x∗ ) depends only on θ ( k−1 ) , θ ( k−2 ) , . . . and defines θ ( k ) ) . Let Assumption 1 hold with any B > 0 and s < ( 1 + 1/µ̃ ) /2 . Then , we have support ( x ( k ) ( x∗ ) ) ⊂ S , ‖x ( k ) ( x∗ ) − x∗‖2 ≤ sB exp ( − k−1∑ τ=0 c ( τ ) ) , k = 1 , 2 , . . . ( 10 ) where S is the support of x∗ and c ( k ) = − log ( ( 2µ̃s− µ̃ ) γ ( k ) + |1− γ ( k ) | ) is a positive constant . In Theorem 1 , Eqn . ( 9 ) defines the properties of “ good ” parameters : • The weights W ( k ) can be separated as the product of a scalar γ ( k ) and a matrix W independent of layer index k , where W has small coherence with D. • γ ( k ) is bounded in an interval . • θ ( k ) /γ ( k ) is proportional to the ` 1 error of the output of the kth layer . The factor c ( k ) takes the maximum at γ ( k ) = 1 . If γ ( k ) ≡ 1 , the recovery error converges to zero in a linear rate ( Chen et al. , 2018 ) : ‖x ( k ) ( x∗ ) − x∗‖2 ≤ sB exp ( − ck ) , where c = − log ( 2µ̃s− µ̃ ) ≥ c ( k ) . Although γ ( k ) ≡ 1 gives the optimal theoretical upper bound if there are infinitely many layers k = 0 , 1 , 2 , · · · , it is not the optimal choice for finite k. Practically , there are finitely many layers and γ ( k ) obtained by learning is bounded in an interval .

Summary: The paper raises many important questions about unrolled iterative optimization algorithms, and answers many questions for the case of iterative soft thresholding algorithm (ISTA, and learned variant LISTA). The authors demonstrate that a major simplification is available for the learned network: instead of learning a matrix for each layer, or even a single (potentially large) matrix, one may obtain the matrix analytically and learn only a series of scalars. These simplifications are not only practically useful but allow for theoretical analysis in the context of optimization theory. On top of this seminal contribution, the results are extended to the convolutional-LISTA setting. Finally, yet another fascinating result is presented, namely that the analytic weights can  be determined from a Gaussian-perturbed version of the dictionary. Experimental validation of all results is presented.