Learning Cluster Structured Sparsity by Reweighting

1 INTRODUCTION . Sparsity is an important inherent property that describes the low-dimensionality of signals . Learning the sparse representation of signals or data plays an important role in many applications , such as image restoration ( Liu et al. , 2018 ; 2019b ) , classification ( Wright et al. , 2009 ) , radar detection ( Ahmad & Amin , 2013 ) , medical imaging ( Lustig et al. , 2007 ) , or black hole image in astronomy ( Honma et al. , 2016 ) . As a fundamental , sparse recovery or sparse representation ( SR ) has been substantially investigated in the last two decades due to the emergence of Compressive Sensing ( CS ) ( Zhang & Rao , 2011 ; Yu et al. , 2012 ; Needell & Tropp , 2009 ; Yu et al. , 2015 ; Daubechies et al. , 2004 ) . Particularly , the objective of SR is to find a concise representation of a signal using a few atoms from some specified ( over-complete ) dictionary y = Ax + ( 1 ) with y ∈ RM the observed measurements corrupted by some noises , x ∈ RN the sparse representation with no more than S nonzero entries ( S-sparsity ) and A ∈ RM×N the dictionary ( normally M N ) . Consequently , the ill-posedness of ( 1 ) prohibits a direct solution of x by inverting A . In the last two decades , such ill-posed inversion problem has been exhaustively investigated in the community of signal processing ( Becker et al. , 2011 ; Daubechies et al. , 2004 ; Needell & Tropp , 2009 ) . Many iterative algorithms such as ISTA ( Daubechies et al. , 2004 ) , FISTA ( Beck & Teboulle , 2009 ; Becker et al. , 2011 ) , ADMM ( Chartrand & Wohlberg , 2013 ) and AMP ( Donoho et al. , 2009 ) have been proposed to solve it , which can be considered as unlearnable approaches since all the parameters are fixed instead of being learned from data . Recently , inspired by the deep learning techniques , SR problem is instead turn to the data-driven approaches . The seminal work LISTA ( Gregor & LeCun , 2010 ; Sprechmann et al. , 2013 ; Zhang & Ghanem , 2018 ) and its consecutive variations ( Zhou et al. , 2018 ; Moreau & Bruna , 2016 ; Ito et al. , 2019 ; Sun et al. , 2016 ) have largely improved the SR performance by simply unfolding iterations of existing SR algorithms into limited number of neural network layers , and then training the network through back propagation with huge amount of data . Theoretically ( Moreau & Bruna , 2016 ) and empirically ( Gregor & LeCun , 2010 ; Sprechmann et al. , 2013 ; Zhang & Ghanem , 2018 ) , the resulted algorithms are with improved accuracy and efficiency comparing to their original counterparts ( Giryes et al. , 2018 ) . On the other hand , the structure behind sparsity , imposing constraint on the support of sparse signals , is an important prior knowledge that can be used to enhance the recovery performance ( Wang et al. , 2014 ; Prasad et al. , 2015 ; Liu et al. , 2018 ; Yu et al. , 2012 ) . Specifically , structures often exhibit as a sharing of zero/nonzero pattern for grouped entries ( dependent on each other ) of sparse signals . And the cluster structured sparsity ( CSS ) is one of the most important cases , where the zero/nonzero entries appear in clusters . Since many natural sparse signals tend to have clustered sparse structure , this pattern has been widely exploited in many practical applications such as gene expression analysis ( Tibshirani et al. , 2005 ) , and inverse synthetic aperture radar imaging ( Lv et al. , 2014 ) . However , to the best of the authors ’ knowledge , few of the data-driven SR algorithms have been proposed for structured SR problems . Thus , there is an urgent need for the study of a learnable approach to utilize the structure behind sparse signals . In this paper , we try to bridge this gap with special attention on cluster structured sparse recovery . We aim at tackling this problem with neural networks which are explainable and fully trainable . As discussed above , LISTA provides an exemplar approach to bridge the connection between SR and NN in an explainable manner . To further consider the clustered structure , we exploit the reweighted iterative soft-thresholding algorithm ( RwISTA ) ( Fosson , 2018 ) as the prototype of our proposed network . Essentially , the weights in RwISTA act as a latent variable that controls the sparsity pattern of corresponding coefficients . Based on this observation , in addition to unfolding RwISTA as a feedforward network , we recast the reweighting process into a parameterized reweighting block that can inference the inherent dependencies between coefficients and thus promote structures . To this end , a one-layer fully connected reweighting block ( RW-LISTA-fc ) that exploits the global dependencies of all elements is first proposed . And empirically , the adjacency matrix of the FC layer after training with CSS signals reveals local dependence of elements for CSS signals . Consequently , a two-layer convolutional reweighting block ( RW-LISTA-conv ) is then proposed to capture such dependencies , namely the connection of neighboring coefficients . Through exhaustive numerical experiments , the resulted network RW-LISTA is proven on the superiority over existing methods in solving CSS recovery . Our high-level contribution can be concluded as follows : • We propose a novel supervised approach for cluster structured sparse recovery . Leveraging the iterative reweighting algorithm , we demonstrate that , we can learn the structured sparsity by reweighting adaptively . Furthermore , a deep architecture called RW-LISTA is proposed by unrolling the existing model into deep networks . • Instead of element-wise manner , we introduce a fully connected ( FC ) and a convolutional reweighting block in RW-LISTA to promote the structures . Excitedly , the learned adjacency matrix in FC layer reveals local dependency for coefficients of CSS signals , which is suitable to be captured by convolution . Although this work specially focuses on CSS , it gives further insights to other structured problems beyond SR. • We achieved state-of-the-art performance on several tasks based on both synthesized and real-world data . In these cases , we compare our algorithm with others in different settings . And the result further verifies the effectiveness and superiority of our proposed methods . 2 LEARNING CSS BY REWEIGHTING . 2.1 REWEIGHTING THE ` 1 NORM ITERATIVELY . Sparse recovery is often cast into a LASSO formulation , namely a minimization problem with ` 1 norm constraint used to promote sparsity . To alleviate the penalty on nonzero coefficients , Candes et al . ( 2008 ) propose to reweight the ` 1 norm . Formally , consider the following problem : min x∈RN ||y −Ax||22 + λ N∑ i=1 wi|xi| ( 2 ) where wi corresponds to the weight associated to ith entry of x . To reach a more “ democratic ” penalization among coefficients , Candes et al . ( 2008 ) assign the weight wi = 1|xi|+ with > 0 a small value . And thus , ∑N i=1 wi|xi| can be seen as a smooth approximation of ` 0 norm . Through reweighting , different entries are penalized differently . Large weights will discourage nonzero entries in the recovered signal while small weights tend to encourage nonzero entries . Consequently , to improve the estimation , the weights should be determined properly according to the true location of nonzero entries . In the original paper , Candes et al . ( 2008 ) adopts an iterative algorithm ( IRL1 ) that alternates between reconstruction x̂ and redefining the weights , which falls into a general form of majorizationminimization algorithm ( a generalization of EM algorithm ( Dempster et al. , 1977 ) ) . Later , Fosson ( 2018 ) proposed a simple variant algorithm called Reweighted Iterative Soft-Threshold Algorithm ( RwISTA ) , which can be formulated as : w ( k ) ←− ϕ ( ∣∣∣x ( k ) ∣∣∣ ) , x ( k+1 ) ←− Γ ( x ( k ) + 1 L AT ( y −A ) x ( k ) , λ L w ( k ) ) ( 3 ) where Γ ( · , · ) is an element-wise soft-thresholding function defined as [ Γ ( x , θ ) ] i = max ( |xi|−θi , 0 ) · sign ( xi ) for i = 1 , 2 , ... , N . L is often taken as the largest eigenvalue of ATA , | · | applies elementwisely and ϕ ( · ) is the reweighting function that takes the magnitude of signal as input . Generally , the reweighting function is usually defined element-wisely as : [ ϕ ( x ) ] i = g ′ ( xi ) , i = 1 , 2 , ... , N ( 4 ) where g ( · ) is a concave , non-decreasing function called merit function . Obviously , if we define g ( x ) = x , then ϕ ( x ) = 1 , ( 3 ) will be reduced to normal ISTA . However , a popular choice is g ( x ) = log ( x + ) with > 0 a pre-defined parameter . In this case , the weights are determined inversely proportional to the magnitudes of the corresponding coefficients as wi = 1|xi|+ . Consequently , if any coefficient x ( k+1 ) i becomes larger , the corresponding weight becomes smaller , and in the next iteration a weaker penalty will be applied . Since nonzero entries are more likely to be identified with greater magnitude , iterative reweighting algorithm introduces a positive feedback that prevents the overshrinkage of nonzero entries while suppressing the magnitude of zero coefficients . 2.2 LEARNED ITERATIVE SOFT-THESHOLDING ALGORITHM WITH REWEIGHTING BLOCK . Although quite successful , there are some reasons that make RwISTA unsuitable for our problem , namely learning to recover cluster structured sparsity ( CSS ) . First and most importantly , the reweighting function is often defined element-wisely , which ignores the structures behind sparsity . In CSS signals , for example , images of MNIST digits , nonzero entries tend to appear in clusters so that the sparsity pattern of the neighboring coefficients are statistically dependent . But such relationship can not be captured by the element-wise reweighting manner . Besides , RwISTA can be viewed as a recurrent structure with pre-fixed parameters . However , according to recent successful applications of deep learning in sparse representation , learnable parameters that benefit from huge amount of training data will behave significantly better with respect to both convergence speed and recovery accuracy . To address the issues mentioned above , we propose RW-LISTA as shown in Figure 1 , which unfolds RwISTA ( 3 ) into a finite-length deep neural network . Formally , the forward pass of RW-LISTA can be formulated as : w ( k ) ←− ϕ ( k ) ( ∣∣∣x ( k ) ∣∣∣ ) , x ( k+1 ) ←− Γ ( B ( k ) x ( k ) + C ( k ) y , θ ( k ) w ( k ) ) ( 5 ) where B ( k ) ∈ RN×N , C ( k ) ∈ RN×M and ϕ ( k ) ( · ) denotes the paratermized reweighting block in the kth layer . The model ’ s architecture is illustrated at Fig . 1 . In RwISTA we fix B ( k ) = I − 1LA TA , C ( k ) = 1LA T and θ ( k ) = λL for all layers . However , now all the parameters in RW-LISTA ( including the parameters of the reweighting block ϕ ( k ) ) are optimized by minimizing the reconstruction error ||x̂− x∗||22 through back-propagation to fix the distribution of the data . Since weights are able to control the sparsity pattern of the coefficients , what we need to do is to determine the weight properly so that they have small values in nonzero locations and significantly greater values elsewhere . It turns out that for structured sparsity , the magnitude of the entry is not the only clue that can be utilized . The dependencies between coefficients should also be carefully considered . Existing works including Yu et al . ( 2015 ; 2012 ) and Fang et al . ( 2014 ; 2016 ) exploit the neighboring dependencies to tackle the problem of CSS recovery , where if the neighbors of one coefficient appears to be nonzero with great magnitude , then the current coefficient is more likely to be nonzero and thus should be reweighted less . It inspires us to build the reweighting block ϕ with respect to the amplitude of coefficients x . And the objective is to output weights with small values if the corresponding coefficients belong to some clustered blocks , while with large values if the corresponding coefficients are isolated or small in magnitude . The details of reweighting block construction will be discussed in the next section . And Figure 2 illustrates an example of the output of the proposed reweighting block ϕ ( after training ) .

Summary: The paper combines deep learning and compressed sensing. Specifically the RW-LISTA algorithm is proposed for cluster-structured sparse recovery, building upon two existing methods: the Reweighted Iterative Shrinkage Algorithm and the LISTA algorithm. The reweighing process is employed to infer the dependencies between coefficients and encourage cluster structure. Strategies for local and global dependence are presented. The approach is evaluated on synthetic and real datasets.