Neurally Augmented ALISTA

1 INTRODUCTION AND RELATED WORK . Compressed sensing deals with the problem of recovering a sparse vector from very few compressive linear observations , far less than its ambient dimension . Fundamental works of Candes et al . ( Candès et al. , 2006 ) and Donoho ( Donoho , 2006 ) show that this can be achieved in a robust and stable manner with computationally tractable algorithms given that the observation matrix fulfills certain conditions , for an overview see Foucart & Rauhut ( 2017 ) . Formally , consider the set of s-sparse vectors in RN , i.e . ΣNs : = { x ∈ RN ∣∣‖x‖0 ≤ s } where the size of the support of x is denoted by ‖x‖0 : = |supp ( x ) | = | { i : xi 6= 0 } | . Furthermore , let Φ ∈ RM×N be the measurement matrix , with typically M N . For a given noiseless observation y = Φx∗ of an unknown but s-sparse x∗ ∈ ΣNs we therefore wish to solve : argmin x ‖x‖0 s.t . y = Φx ( 1 ) In ( Candès et al. , 2006 ) it has been shown , that under certain assumptions on Φ , the solution to the combinatorial problem in ( 1 ) can be also obtained by a convex relaxation where one instead minimizes the ` 1–norm of x . The Lagrangian formalism yields then an unconstrained optimization problem also known as LASSO ( Tibshirani , 1996 ) , which penalizes the ` 1-norm via the hyperparameter λ ∈ R : x̂ = argmin x 1 2 ‖y − Φx‖22 + λ‖x‖1 ( 2 ) A very popular approach for solving this problem is the iterative shrinkage thresholding algorithm ( ISTA ) ( Daubechies et al. , 2003 ) , in which a reconstruction x ( k ) is obtained after k iterations from initial x ( 0 ) = 0 via the iteration : x ( k+1 ) = ηλ/L ( x ( k ) + 1 L ΦT ( y − Φx ( k ) ) ) ( 3 ) ∗equal contribution † The work is partially funded by DFG grant JU 2795/3 and the German Federal Ministry of Education and Research ( BMBF ) in the framework of the international future AI lab `` AI4EO – Artificial Intelligence for Earth Observation : Reasoning , Uncertainties , Ethics and Beyond '' ( Grant number : 01DD20001 ) . where ηθ is the soft thresholding function given by ηθ ( x ) = sign ( x ) max ( 0 , |x| − θ ) ( applied coordinate-wise ) and L is the Lipschitz constant ( i.e . the largest eigenvalue ) of ΦTΦ . Famously , the computational graph of ISTA with K iterations can be unrolled to yield Learned ISTA ( LISTA ) ( Gregor & LeCun , 2010 ) , a K-layer neural network in which all parameters involved can be trained ( each layer k has an individual threshold parameter and individual or shared matrix weights ) using backpropagation and gradient descent . LISTA achieves impressive empirical reconstruction performance for many sparse datasets but loses the theoretical guarantees of ISTA . Bridging the gap between LISTA ’ s strong reconstruction quality and the theoretical guarantees for ISTA , ALISTA ( Liu et al. , 2019 ) was introduced . ALISTA , introduces a matrix WT , related to the measurement matrix ΦT in ( 3 ) , which is computed by optimizing the generalized coherence : µ ( W , Φ ) = inf W∈RM×N max i 6=j WT : ,iΦ : ,j s.t . ∀i ∈ { 1 , . . . , N } : WT : ,iΦ : ,i = 1 ( 4 ) Then , contrary to LISTA , all matrices are excluded from learning in order to retain desirable properties such as low coherence . For each layer of ALISTA , only a scalar step size parameter γ ( k ) and a scalar threshold θ ( k ) is learned from the data , yielding the iteration : x ( k+1 ) = ηθ ( k ) ( x ( k ) − γ ( k ) WT ( Φx ( k ) − y ) ) ( 5 ) As in LISTA , the parameters for ALISTA are learned end-to-end using backpropagation and stochastic gradient descent by empirically minimizing the reconstruction error : min θ ( 1 ) , ... , θ ( K ) , γ ( 1 ) , ... , γ ( K ) Ex∗ [ ‖x ( K ) − x∗‖22 ] ( 6 ) The authors rigorously upper-bound the reconstruction error of ALISTA in the noiseless case and demonstrate strong empirical reconstruction quality even in the noisy case . The empirical performance similar to LISTA , the retained theoretical guarantees , and the reduction of number of parameters to train from either O ( KM2 + NM ) in vanilla LISTA or O ( MNK ) in the variant of LISTA-CPSS ( Chen et al. , 2018 ) to just O ( K ) , make ALISTA an appealing algorithm to study and extend . In ( Ablin et al. , 2019 ) , instead of directly focusing on the reconstruction problem , where λ is not known a priori , analytical conditions for optimal step sizes in ISTA are derived for LASSO , yielding Stepsize-ISTA . Stepsize-ISTA is a variant of LISTA in which the measurement matrices are exempt from training like in ALISTA , outperforming existing approaches to directly solving LASSO . Thresholds that are adaptive to the current target vector have been explored in ALISTA-AT ( Kim & Park , 2020 ) . Following the majorization-minimization method , component-wise thresholds are computed from previous iterations . In a particular case this yields θ ( k ) i = 1/ ( 1 + |x ( k−1 ) i |/ ) for some > 0 , known as iterative reweighted ` 1-minimization . By unrolling this algorithm , the authors demonstrate superior recovery over ALISTA for a specific setting of M , N and s. In a related approach ( Wu et al. , 2020 ) identify undershooting , meaning that reconstructed components are smaller than target components , as a shortcoming of LISTA and propose Gated-LISTA to address these issues . The authors introduce gain and overshoot gates to LISTA , which can amplify the reconstruction after each iteration before and after thresholding , yielding an architecture resembling GRU cells ( Cho et al. , 2014 ) . The authors demonstrate better sparse reconstruction than previous LISTA-variants and also show that adding their proposed gates to ALISTA , named AGLISTA , it is possible to improve its performance in the same setting of M , N and s as ALISTA-AT . In this paper , motivated by essential proof steps of ALISTA ’ s recovery guarantee , we propose an alternative method for adaptively choosing thresholds and step sizes during reconstruction . Our method directly extends ALISTA by using a recurrent neural network to predict thresholds and step sizes depending on an estimate of the ` 1-error between the reconstruction and the unknown target vector after each iteration . We refer to our method as Neurally Augmented ALISTA ( NA-ALISTA ) , as the method falls into the general framework of neural augmentation of unrolled algorithms ( Welling , 2020 ; Monga et al. , 2019 ; Diamond et al. , 2017 ) . The rest of the paper is structured as follows : we provide theoretical motivation for NA-ALISTA in Section 2 , before describing our method in detail in Section 3 . In Section 4 , we demonstrate experimentally that NA-ALISTA achieves state-of-the-art performance in all evaluated settings . To summarize , our main contributions are : 1 . We introduce Neurally Augmented ALISTA ( NA-ALISTA ) , an algorithm which learns to adaptively compute thresholds and step-sizes for individual target vectors during recovery . The number of parameters added does not scale with the problem size . 2 . We provide theoretical motivation inspired by guarantees for sparse reconstruction which show that NA-ALISTA can achieve arrive tighter error bounds depending on the target x∗ . 3 . We find that NA-ALISTA empirically outperforms ALISTA and other state-of-the-art algorithms in a synthetic setting as well as in a real-world application from wireless communications and that the gains increase with decreasing M/N . 2 THEORETICAL MOTIVATION . The thresholds θ ( k ) in ( 5 ) play an important role in the analysis of ALISTA . While the authors of ( Liu et al. , 2019 ) prove that θ ( k ) must be larger than a certain value in order to guarantee no false positives in the support of the reconstruction x ( k ) , the thresholds θ ( k ) also appear as an additive term in the reconstruction error upper bound . Thus , to guarantee good reconstruction θ ( k ) should be just slightly larger than the value it must surpass in order to both minimize the error and verify the assumption . In this section , we repeat key insights from ALISTA and motivate the choice of adaptive thresholds - the key improvement in our proposed NA-ALISTA . More specifically , we repeat the conditions under which ALISTA guarantees no false positives and highlight an intermediate step in the error bound from ( Liu et al. , 2019 ) , which tightens when the thresholds can adapt to specific instances of x∗ . Assumption ( adapted from Assumption 1 from ( Liu et al. , 2019 ) 1 ) Let x∗ ∈ ΣNs be a fixed s–sparse target vector . Let W be such that it attains the infimum of the generalized coherence with Φ ( as in ( 4 ) ) and denote this generalized coherence as µ̃ = µ ( W , Φ ) . Let s < ( 1 + 1/µ̃ ) /2 . Let γ ( 1 ) , . . . , γ ( K ) be any sequence of scalars taking values in ( 0 , 22µ̃s−µ̃+1 ) and θ ( 1 ) , . . . , θ ( K ) with : θ ( k ) ≥ γ ( k ) µ̃‖x ( k ) − x∗‖1 ( 7 ) Because in ALISTA , the thresholds γ ( 1 ) , . . . , γ ( K ) and stepsizes θ ( 1 ) , . . . , θ ( K ) are optimized in expectation over the training data , the inequality in ( 7 ) holds only in the general case if the thresholds are larger than the worst case ` 1-error committed by the algorithm over all training vectors x∗ i.e . : θ ( k ) ≥ γ̃ ( k ) µ̃ sup x∗ ‖x ( k ) − x∗‖1 ( 8 ) This is needed in order to fulfill the Assumption . Under these conditions it is guaranteed that no false positives are in the support of the reconstruction : No false positives ( Lemma 1 from ( Liu et al. , 2019 ) ) Under the settings of the Assumption , it holds that : supp ( x ( k ) ) ⊆ supp ( x∗ ) ( 9 ) However , the threshold θ ( k ) also reappears in the error upper bound . Here we employ an intermediate step of the error upper bound from ( Liu et al. , 2019 ) : Reconstruction error upper bound ( Theorem 1 from ( Liu et al. , 2019 ) ) Under the settings of the Assumption , it holds that : ‖x ( k+1 ) −x∗‖2 ≤ ‖x ( k+1 ) −x∗‖1 ≤ µ̃γ ( k ) ( s−1 ) ‖x ( k ) −x∗‖1+θ ( k ) s+|1−γ ( k ) |‖x ( k ) −x∗‖1 ( 10 ) Where the first inequality holds for all real vectors and the second inequality is derived in detail in Appendix A of ( Liu et al. , 2019 ) . According to ( 10 ) it is therefore desirable that θ ( k ) is as small as 1Note that in this work and in Liu et al . ( 2019 ) the noiseless case is considered to simplify the theorems and proofs . For similar statements in the noisy case , we refer the reader to Chen et al . ( 2018 ) . possible , but such that it still satisfies ( 7 ) . This means that ALISTA has to learn thresholds θ̃ ( k ) at least proportional to the largest possible committed ` 1-error over all possible x∗ in order to guarantee good reconstruction , for which it is in turn penalized in the error bound . However , if an algorithm would have access to ‖x ( k ) − x∗‖1 and were allowed to choose thresholds adaptively based on x∗ , the more relaxed inequality ( 7 ) could be employed directly , without taking the supremum over all possible x∗ as in ( 8 ) . Then , this algorithm could obtain a tighter error bound for some individual targets x∗ than ALISTA since θ̃ ( k ) ≥ θ ( k ) . Finding such an algorithm is the aim of this paper .

Summary: This paper extends the framework of ALISTA, a variant of learned ISTA called Neurally Augmented ALISTA (AG-ALISTA), which significantly reduces the number parameters in the model (down to 2 scalars per layer, one for step size and the other for the threshold in soft-thresholding function). Specifically, the authors use a LSTM to generate these two parameters in each layer along iterations, taking reconstruction error related signals as input. This method is based on (1) the previous previous finding of the relation of the step size and threshold with the $\ell_1$ signal recovery error; and (2) the empirical observation of the correlation between the $\ell_1$ signal recovery error and reconstruction error. Experiments in synthetic setting show the superiority of AG-ALISTA over ALISTA and other variants that follow it, especially in settings where the compression ratios are challenging, which is claimed to be more realistic in real-world settings.