Learned ISTA with Error-based Thresholding for Adaptive Sparse Coding

1 INTRODUCTION . Sparse coding is widely used in many machine learning applications ( Xu et al. , 2012 ; Dabov et al. , 2007 ; Yang et al. , 2010 ; Ikehata et al. , 2012 ) , and its core problem is to deduce the high-dimensional sparse code from the obtained low-dimensional observation , for example , under the assumption of y = Axs + ε , where y ∈ Rm is the observation corrupted by the inevitable noise ε ∈ Rm , xs ∈ Rn is the sparse code to be estimated , andA ∈ Rm×n is an over-complete dictionary matrix . To recover xs purely from y is called sparse linear inverse problem ( SLIP ) . The main challenge for solving SLIP is its ill-posed nature because of the over-complete modeling , i.e. , m < n. A possible solution to SLIP can be obtained via solving a LASSO problem using the l1 regularization : min x ‖y −Ax‖2 + λ‖x‖1 . ( 1 ) Possible solutions for Eq . ( 1 ) are iterative shrinking thresholding algorithm ( ISTA ) ( Daubechies et al. , 2004 ) and its variants , e.g. , fast ISTA ( FISTA ) ( Beck & Teboulle , 2009 ) . Despite their simplicity , these traditional optimization algorithm suffer from slow convergence speed in large scale problems . Therefore , Gregor & LeCun ( 2010 ) proposed the learned ISTA ( LISTA ) which was a deep neural network ( DNN ) whose architecture followed the iterative process of ISTA . The thresholding mechanism was modified into shrinkage functions in the DNNs together with learnable thresholds . LISTA achieved superior performance in sparse coding , and many theoretical analyses have been proposed to modify LISTA to further improve its performance ( Chen et al. , 2018 ; Liu et al. , 2019 ; Zhou et al. , 2018 ; Ablin et al. , 2019 ; Wu et al. , 2020 ) . Yet , LISTA and many other deep networks based on it suffer from two issues . ( a ) Though the thresholds of the shrinkage functions in LISTA were learnable , their values were shared among all training samples and thus lack adaptability to the variety of training samples and robustness to outliers . According to prior work ( Chen et al. , 2018 ; Liu et al. , 2019 ) , the thresholds should be proportional to the upper bound of the norm of the current estimation error to guarantee fast convergence in LISTA . However , outliers with drastically higher estimation errors will affect the thresholds more , making the learned thresholds less suitable to other ( training ) samples . ( b ) For the same reason , it may also lead to poor generalization to test data with different distribution ( or sparsity ( Chen et al. , 2018 ) ) from the training data . For instance , in practice , we may only be given some synthetic sparse codes but not the real ones for training , and current LISTA models may fail to generalize under such circumstances . In this paper , we propose an error-based thresholding ( EBT ) mechanism to address the aforementioned issues of LISTA-based models to improve their performance . Drawing on theoretical insights , EBT introduces a function of the evolving estimation error to provide each threshold in the shrinkage functions . It has no extra parameter to learn compared with original LISTA-based models yet shows significantly better performance . The main contributions of our paper are listed as follows : • The EBT mechanism can be readily incorporated into popular sparse coding DNNs ( e.g. , LISTA ( Gregor & LeCun , 2010 ) and LISTA with support selection ( Chen et al. , 2018 ) ) to speed up the convergence with no extra parameters . • We give a rigorous analysis to prove that the estimation error of EBT-LISTA ( i.e. , a combination of our EBT and LISTA ) and EBT-LISTA with support selection ( i.e. , a combination of our EBT and LISTA with support selection ) is theoretically lower than the original LISTA ( Gregor & LeCun , 2010 ) and LISTA with support selection ( Chen et al. , 2018 ) , respectively . In addition , the introduced parameters in our EBT are well-disentangled from the reconstruction errors and need only to be correlated with the dictionary matrix to ensure convergence . These results guarantee the superiority of our EBT in theory . • We demonstrate the effectiveness of our EBT in the original LISTA and several of its variants in simulation experiments . We also show that it can be applied to practical applications ( e.g. , photometric stereo analysis ) and achieve superior performance as well . The organization of this paper is structured as follows . In Section 2 , we will review some preliminary knowledge of our study . In Section 3 , we will introduce a basic form of our EBT and several of its improved versions . Section 4 provides a theoretical study of the convergence of EBT-LISTA . Experimental results in Section 5 valid the effectiveness of our method in practice . Section 6 summarizes this paper . 2 BACKGROUND AND PRELIMINARY KNOWLEDGE . As mentioned in Section 1 , ISTA is an iterative algorithm for solving LASSO in Eq . ( 1 ) . Its update rule is : x ( 0 ) = 0 and x ( t+1 ) = shλ/γ ( ( I −ATA/γ ) x ( t ) +AT y/γ ) , ∀t ≥ 0 , ( 2 ) where shb ( x ) = sign ( x ) ( |x| − b ) + is a shrinkage function with a threshold b ≥ 0 and ( · ) + = max { 0 , · } , γ is a positive constant scalar greater than or equal to the maximal eigenvalue of the symmetric matrix ATA . LISTA kept the update rule of ISTA but learned parameters via end-to-end training . Its inference process can be formulated as x ( 0 ) = 0 and x ( t+1 ) = shb ( t ) ( W ( t ) x ( t ) + U ( t ) y ) , t = 0 , . . . , d , ( 3 ) where Θ = { W ( t ) , U ( t ) , b ( t ) } t=0 , ... , d is a set of learnable parameters , and , specifically , b ( t ) is the layer-wise threshold which is learnable but shared among all samples . LISTA achieved lower reconstruction error between its output and the ground-truth xs compared with ISTA , and it is proved to convergence linearly ( Chen et al. , 2018 ) with W ( t ) = I−U ( t ) A holds for any layer t. Thus , Eq . ( 3 ) can be written as . x ( t+1 ) = shb ( t ) ( ( I − U ( t ) A ) x ( t ) + U ( t ) y ) , t = 0 , . . . , d. ( 4 ) Chen et al . ( 2018 ) further proposed support selection for LISTA , which introduced shp ( b ( t ) , p ) ( x ) whose elements are defined as ( shp ( b ( t ) , p ) ( x ) ) i = sign ( xi ) ( |xi| − b ) , if |xi| > b , i /∈ Sp xi , if |xi| > b , i ∈ Sp 0 , otherwise , ( 5 ) to substitute the original shrinking function shb ( t ) ( x ) , where Sp is the set of the index of the largest p % elements ( in absolute value ) in vector x . Formally , the update rule of LISTA with support selection is formulated as x ( 0 ) = 0 and x ( t+1 ) = shp ( b ( t ) , p ( t ) ) ( ( I − UA ) x ( t ) + U ( t ) y ) , t = 0 , . . . , d , ( 6 ) where p ( t ) is a hyper-parameter and it increases from early layers to later layers . LISTA with support selection can achieve faster convergence compared with LISTA ( Chen et al. , 2018 ) . Theoretical studies ( Chen et al. , 2018 ; Liu et al. , 2019 ) also demonstrate that the threshold of LISTA and its variants should satisfy the equality b ( t ) ← µ ( A ) sup xs∈S ‖x ( t ) − xs‖p ( 7 ) to ensure fast convergence in the noiseless case ( i.e. , ε = 0 ) , where S is the training set and µ ( A ) is the general mutual coherence coefficient of the dictionary matrixA . Note that µ ( A ) is a crucial term in this paper , here we formally give its definition together with the definition ofW ( A ) as follows . Definition 1 For A ∈ Rm×n , its generalized coherence coefficient is defined as µ ( A ) = infW∈Rn×m , Wi , :A : ,i=1 maxi 6=j | ( Wi , :A : ,j ) | , and we say W ∈ W ( A ) if maxi 6=j ( Wi , :A : ,j ) = µ ( A ) . 3 METHODS . In LISTA and its variants , the threshold b ( t ) is commonly treated as a learnable parameter . As demonstrated in Eq . ( 7 ) , b ( t ) should be proportional to the upper bound of the estimation error of the t-th layer in the noiseless case to ensure fast convergence . Thus , some outliers or extreme training samples largely influence the value of b ( t ) , making the obtained threshold not fit the majority of the data . To be specific , we know that the suggested value of b ( t ) is b ( t ) = µ ( A ) supi=0,1 , ... , n‖x ( t ) i − xsi‖p for ‖x‖1a training set { xsi } i=0,1 , ... , n , and normal training of LISTA leads to it in theory ( Chen et al. , 2018 ) . Yet , if a new training sample xsn+1 with higher reconstruction error is introduced , the expected b ( t ) shall change to µ ( A ) ‖x ( t ) n+1 − xsn+1‖p , which is probably undesirable for the other samples . Similar problems occur if there exists a large variety in the value of reconstruction errors . In order to solve this problem , we propose to disentangle the reconstruction error term from the learnable part of the threshold and introduce adaptive thresholds for LISTA and related networks . We attempt to rewrite the threshold at the t-th layer as something like b ( t ) = ρ ( t ) ‖x ( t ) − xs‖p , ( 8 ) where ρ ( t ) is a layer-specific learnable parameter . However , the ground-truth xs is actually unknown for the inference process in SLIP . Therefore , we need to find an alternative formulation . Notice that in the noiseless case , it holds that Ax ( t ) − y = A ( x ( t ) − xs ) , thus we further rewrite Eq . ( 8 ) into b ( t ) = ρ ( t ) ‖Q ( Ax ( t ) − y ) ‖p , ( 9 ) whereQ ∈ Rn×m is a compensation matrix introduced to let Eq . ( 9 ) approximate Eq . ( 8 ) better , i.e. , a matrix that makesQA approaches the identity matrix is more desired . However , note that although QA is a low-rank matrix and can never be an identity matrix , we can encourage its diagonal elements to be 1 and the non-diagonal elements to be nearly zero . This can be directly achieved by letting Q ∈ W ( A ) , where W ( A ) is defined in Definition 1 . According to some prior works ( Liu et al. , 2019 ; Wu et al. , 2020 ; Chen et al. , 2018 ) , we also know that U ( t ) ∈ W ( A ) to guarantee linear convergence , thus U ( t ) can probably be a reasonable option for the matrix Q in our method , making it layer-specific as well . Therefore , our EBT-LISTA is formulated as x ( 0 ) = 0 and x ( t+1 ) = shb ( t ) ( ( I − U ( t ) A ) x ( t ) + U ( t ) y ) , t = 0 , . . . , d , b ( t ) = ρ ( t ) ‖U ( t ) ( Ax ( t ) − y ) ‖p . ( 10 ) Note that only ρ ( t ) and U ( t ) are learnable parameters in the above formulation , thus our EBT-LISTA actually introduces no extra parameters compared with the original LISTA . The architecture of LISTA and EBT-LISTA are shown in Figure 1 . We can also apply our EBT mechanism on LISTA with support selection ( Chen et al. , 2018 ) . It is straightforward to keep the support set selection operation and replace the fixed threshold with our EBT , and such a combination can be similarly formulated as x ( 0 ) = 0 and x ( t+1 ) = shp ( b ( t ) , p ( t ) ) ( ( I − U ( t ) A ) x ( t ) + U ( t ) y ) , t = 0 , . . . , d , b ( t ) = ρ ( t ) ‖U ( t ) ( Ax ( t ) − y ) ‖p . ( 11 ) Our former analysis is based on the noiseless case . For noise case , there is A ( x ( t ) − xs ) = Ax ( t ) − y + ε . Since the noise is generally unknown in practical problems , we may add an extra learnable parameter on the threshold to compensate the noise , i.e. , b ( t ) = ρ ( t ) ‖U ( t ) ( Ax ( t ) − y ) ‖p + α ( t ) , ( 12 ) where α ( t ) is the learnable parameter for the observation noise .

Summary: In the paper, authors propose a new error-based thresholding mechanism for LISTA which introduces a function of the evolving estimation error to provide each threshold in the shrinkage functions. They provided the theoretical analysis for EBT-LISTA and EBT-LISTA with support selection and proved that the  estimation error of the proposed algorithm is theoretically lower than compared methods. The authors also evaluated the proposed method on multiple synthetic or real tasks. Experimental results show that the proposed method achieves a better estimation error and  higher adaptivity to different observations with a variety of sparsity.