Sparse Coding with Gated Learned ISTA

1 INTRODUCTION . Sparse coding serves as the foundation of many machine learning applications , e.g. , the direction-ofarrival estimation ( Xu et al. , 2012 ) , signal denoising ( Elad & Aharon , 2006 ) , and super resolution imaging ( Yang et al. , 2010 ) . In general , it aims to recover an inherently sparse vector xs ∈ Rn from an observation y ∈ Rm corrupted by a noise vector ε ∈ Rm . That is , y = Axs + ε , ( 1 ) in which A ∈ Rm×n is an over-complete basis matrix . The problem of recovering xs , however , is a challenging task , in which the main difficulties are to incorporate the sparse constraint which is nonconvex and to further determine the indices of its non-zero elements , i.e. , the support of the vector . A reasonable solution to the problem is to use convex functions as surrogates to relax the constraint of sparsity , among which the most classical one probably is the l1-norm penalty . Such a problem is carefully studied in Lasso ( Tibshirani , 1996 ) , and it can be solved via least angle regression ( Efron et al. , 2004 ) , the iterative shrinkage and thresholding algorithm ( ISTA ) ( Daubechies et al. , 2004 ) , etc . Despite the simplicity , these conventional solvers suffer from critical shortcomings . Taking ISTA as an example , we know that 1 ) it converges very slowly with only a sublinear rate ( Beck & Teboulle , 2009 ) , 2 ) the correlation between each of the two columns of A should be relatively low . In recent years , deep learning ( LeCun et al. , 2015 ) methods have achieved remarkable successes . Deep neural ∗Kailun Wu and Yiwen Guo contribute equally to the paper . †Kailun Wu , Ziang Li , and Changshui Zhang are with the Institute for Artificial Intelligence ( THUAI ) , the State Key Lab of Intelligent Technologies and Systems , the Beijing National Research Center for Information Science and Technology ( BNRist ) , and the Department of Automation , Tsinghua University . Kailun Wu did his work when he was a Ph.D. student at Tsinghua Unversity , and now he works at Alibaba Group . This work is ( jointly or partly ) funded by the NSFC ( Grant No . 61876095 ) and Beijing Academy of Artificial Intelligence ( BAAI ) . networks ( DNNs ) have been proven both effective and efficient in dealing with many tasks , including image classification ( He et al. , 2016 ) , object detection ( Girshick , 2015 ) , speech recognition ( Hinton et al. , 2012 ) , and also sparse coding ( Gregor & LeCun , 2010 ; Wang et al. , 2016 ; Borgerding et al. , 2017 ; He et al. , 2017 ; Zhang & Ghanem , 2018 ; Chen et al. , 2018 ; Liu et al. , 2019 ; Sulam et al. , 2019 ) . The core idea behind deep learning-based sparse coding is to train DNNs to approximate the optimal sparse code . For instance , an initial work of Gregor and LeCun ’ s ( 2010 ) takes the inspiration from ISTA and develops an approximator named learned ISTA ( LISTA ) , which is structurally similar to a recurrent neural network ( RNN ) . It has been demonstrated both empirically and theoretically that LISTA is superior to ISTA ( Wang et al. , 2016 ; Moreau & Bruna , 2017 ; Giryes et al. , 2018 ; Chen et al. , 2018 ) . Nevertheless , it is also uncontroversial that there exists much room for further enhancing it . In this paper , we delve deeply into the foundation of ( L ) ISTA and discover possible weaknesses of LISTA . First and foremost , we know from prior arts ( Chen et al. , 2018 ; Liu et al. , 2019 ) that LISTA tends to learn large enough biases to achieve no “ false positive ” in the support of generated codes and further ensure linear convergence , and we prove that this tendency , however , also makes the magnitude of the code components being lower than that of the ground-truth . That said , there probably exists a requirement of gains in the code estimations . Second , regarding the optimization procedure of ISTA as to minimize an upper bound of its objective function at each step , we conjecture that the element-wise update of ( L ) ISTA normally “ lags behind ” the optimal solution , which suggests that it requires overshoots to reach the optimum , just like what has been suggested in fast ISTA ( FISTA ) ( Beck & Teboulle , 2009 ) and learned FISTA ( LFISTA ) ( Moreau & Bruna , 2017 ) . In this paper , our main contributions are summarized as follows : • We discover weaknesses of LISTA by theoretically analyzing its optimization procedure , for mitigating which we introduce gain gates and overshoot gates , akin to update gate and reset gate mechanisms in the gated recurrent unit ( GRU ) Cho et al . ( 2014 ) . • We provide convergence analyses for LISTA ( with or without gates ) , which further give rise to conditions on which the performance of our method with gain gates can be guaranteed . A practical case is considered , where the assumption of no “ false positive ” is relaxed . • Insightful expressions for the gates are presented . In comparison with state-of-the-art sparse coding networks ( not limited to previous extensions to LISTA ) , our method achieves superior performance . It also applies to variants of LISTA , e.g. , LFSITA ( Moreau & Bruna , 2017 ) and ALISTA ( Liu et al. , 2019 ) . Notations : In this paper , unless otherwise clarified , vectors and matrices are denoted by lowercase and uppercase characters , respectively . For vectors/matrices originally introduced without any subscript , adding a subscript ( e.g. , i ) indicates its element/column at the corresponding position . For instance , for x ∈ Rn , xi represents the i-th element of the vector , and W : ,i and Wi , : denote the i-th column and row of a matrix W respectively . While for vectors introduced with subscripts already , e.g. , xs , we use ( xs ) i to denote its i-th element . The operator is used to indicate element-wise multiplication of two vectors . The support of a vector is denoted as supp ( x ) : = { i|xi 6= 0 } . We use supxs as the simplified form of supxs∈X ( B , s,0 ) , see Assumption 1 for the definition of X ( B , s , 0 ) . 2 BACKGROUND . In general , sparse coding solves the problem that can be formulated as min x f ( x , y ) + λr ( x ) , ( 2 ) in which f ( x , y ) calculates the residual of approximating y using a linear combination of column-wise features in A . The function f ( x , y ) is convex with respect to x in general . In particular , if ε is a Gaussian vector , then it should be f ( x , y ) = ‖Ax− y‖22 . The term λr ( x ) serves as a regularizer for sparsity and we have r ( x ) = ‖x‖1 in Lasso . As mentioned , a variety of algorithms can be applied to solve the problem and our focus in the paper is ( L ) ISTA . We first revisit the optimization procedure of ISTA , which is the foundation of LISTA as well . Given y , let us introduce a scalar γ > 0 that fulfills γI −∇2xf ( x , y ) 0 , ∀x , then it can be considered as optimizing an upper bound of the objective function obtained via Taylor expansion . To be more specific , for any presumed x ( t ) , we have f ( x , y ) + λr ( x ) ≤ f ( x ( t ) , y ) + ( x− x ( t ) ) ∇xf ( x ( t ) ) + γ 2 ‖x− x ( t ) ‖2 + λr ( x ) . ( 3 ) By substituting r ( x ) with ‖x‖1 and optimizing the bound in an element-wise manner , we can easily get the one-step update rule that zeros the gradient based on x ( t ) . It is , x ( 0 ) = 0 and x ( t+1 ) = sλ/γ ( x ( t ) −∇xf ( x ( t ) ) /γ ) , ∀t ≥ 0 , ( 4 ) in which sb ( x ) : = sign ( x ) ( |x| − b ) + is a shrinking function and ( · ) + is a rectified linear unit ( ReLU ) calculating max { 0 , · } . For Gaussian noises , the formulation reduces to x ( t+1 ) = sλ/γ ( ( I − A TA γ ) x ( t ) + AT γ y ) . ( 5 ) The update as shown in Eq . ( 4 ) and ( 5 ) can be performed iteratively until convergence . However , the convergence of ISTA ( along with some other conventional solvers ) is known to be slow , and it has been shown that DNNs can be utilized to accelerate the procedure . Many researchers have explored the idea since the initial work of Gregor and LeCun ’ s ( i.e. , LISTA ) . For LISTA , they design deep architectures following the main procedure of ISTA yet to learn parameters in an end-to-end manner from data ( Gregor & LeCun , 2010 ; Hershey et al. , 2014 ) . The inference process of LISTA is similar to that of an RNN and can be formulated as x ( 0 ) = 0 and x ( t+1 ) = sb ( t ) ( W ( t ) x ( t ) + U ( t ) y ) , t = 0 , · · · , d− 1 , ( 6 ) where Θ = { U ( t ) , W ( t ) , b ( t ) } t=0,1 , ... , d−1 , is learnable parameters set . Some works ( Xin et al. , 2016 ; Chen et al. , 2018 ) have proved that W ( t ) and U ( t ) should satisfy the constraint W ( t ) = I − U ( t ) A , such that x ( t+1 ) = sb ( t ) ( x ( t ) + U ( t ) ( Ax ( t ) − y ) ) , t = 0 , · · · , d− 1 . ( 7 ) The parameters in Θ are normally learned from a set of training samples by minimizing the difference between the final code estimations and ground-truth . In this paper , our main assumption for theoretical analyses follows those of prior works ( Chen et al. , 2018 ; Liu et al. , 2019 ) in a noiseless case , and noisy cases will be considered in the experiments . Assumption 1 . The sparse vector xs and noise vector ε are sampled from a set X ( B , s , 0 ) fulfilling : X ( B , s , 0 ) : = { x ∣∣‖x‖∞ ≤ B , ε = 0 , ‖x‖0 ≤ s } .

Summary: The authors propose extensions to LISTA with the goal of addressing underestimation (by introducing “gain gates”) and including momentum (by introducing “overshoot gates”). The authors provide theoretical analysis for each step of their LISTA augmentations, showing that it improves convergence rate. Their theoretical statements are empirically validated and then a numerical comparison is performed between the most interesting LISTA variants. Their proposed GLISTA performs favorably, especially for networks of depth greater than 10.