user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Sparse Coding with Gated Learned ISTA

Paper content:
In this paper, we study the learned iterative shrinkage thresholding algorithm (LISTA) for solving sparse coding problems.
Following assumptions made by prior works, we first discover that the code components in its estimations may be lower than expected, i.e., require gains, and to address this problem, a gated mechanism amenable to theoretical analysis is then introduced.
Specific design of the gates is inspired by convergence analyses of the mechanism and hence its effectiveness can be formally guaranteed.
In addition to the gain gates, we further introduce overshoot gates for compensating insufficient step size in LISTA.
Extensive empirical results confirm our theoretical findings and verify the effectiveness of our method.
Sparse coding serves as the foundation of many machine learning applications, e.g., the direction-ofarrival estimation (Xu et al., 2012) , signal denoising (Elad & Aharon, 2006) , and super resolution imaging (Yang et al., 2010) .
In general, it aims to recover an inherently sparse vector x s ∈ R n from an observation y ∈ R m corrupted by a noise vector ε ∈ R m .
That is,
in which A ∈ R m×n is an over-complete basis matrix.
The problem of recovering x s , however, is a challenging task, in which the main difficulties are to incorporate the sparse constraint which is nonconvex and to further determine the indices of its non-zero elements, i.e., the support of the vector.
A reasonable solution to the problem is to use convex functions as surrogates to relax the constraint of sparsity, among which the most classical one probably is the l 1 -norm penalty.
Such a problem is carefully studied in Lasso (Tibshirani, 1996) , and it can be solved via least angle regression (Efron et al., 2004) , the iterative shrinkage and thresholding algorithm (ISTA) (Daubechies et al., 2004) , etc.
Despite the simplicity, these conventional solvers suffer from critical shortcomings.
Taking ISTA as an example, we know that 1) it converges very slowly with only a sublinear rate (Beck & Teboulle, 2009) , 2) the correlation between each of the two columns of A should be relatively low.
In recent years, deep learning (LeCun et al., 2015) methods have achieved remarkable successes.
Deep neural networks (DNNs) have been proven both effective and efficient in dealing with many tasks, including image classification (He et al., 2016) , object detection (Girshick, 2015) , speech recognition (Hinton et al., 2012) , and also sparse coding (Gregor & LeCun, 2010; Borgerding et al., 2017; He et al., 2017; Zhang & Ghanem, 2018; Chen et al., 2018; Liu et al., 2019; Sulam et al., 2019) .
The core idea behind deep learning-based sparse coding is to train DNNs to approximate the optimal sparse code.
For instance, an initial work of Gregor and LeCun's (2010) takes the inspiration from ISTA and develops an approximator named learned ISTA (LISTA), which is structurally similar to a recurrent neural network (RNN).
It has been demonstrated both empirically and theoretically that LISTA is superior to ISTA Moreau & Bruna, 2017; Giryes et al., 2018; Chen et al., 2018) .
Nevertheless, it is also uncontroversial that there exists much room for further enhancing it.
In this paper, we delve deeply into the foundation of (L)ISTA and discover possible weaknesses of LISTA.
First and foremost, we know from prior arts (Chen et al., 2018; Liu et al., 2019) that LISTA tends to learn large enough biases to achieve no "false positive" in the support of generated codes and further ensure linear convergence, and we prove that this tendency, however, also makes the magnitude of the code components being lower than that of the ground-truth.
That said, there probably exists a requirement of gains in the code estimations.
Second, regarding the optimization procedure of ISTA as to minimize an upper bound of its objective function at each step, we conjecture that the element-wise update of (L)ISTA normally "lags behind" the optimal solution, which suggests that it requires overshoots to reach the optimum, just like what has been suggested in fast ISTA (FISTA) (Beck & Teboulle, 2009 ) and learned FISTA (LFISTA) (Moreau & Bruna, 2017) .
In this paper, our main contributions are summarized as follows:
• We discover weaknesses of LISTA by theoretically analyzing its optimization procedure, for mitigating which we introduce gain gates and overshoot gates, akin to update gate and reset gate mechanisms in the gated recurrent unit (GRU) Cho et al. (2014) .
• We provide convergence analyses for LISTA (with or without gates), which further give rise to conditions on which the performance of our method with gain gates can be guaranteed.
A practical case is considered, where the assumption of no "false positive" is relaxed.
• Insightful expressions for the gates are presented.
In comparison with state-of-the-art sparse coding networks (not limited to previous extensions to LISTA), our method achieves superior performance.
It also applies to variants of LISTA, e.g., LFSITA (Moreau & Bruna, 2017) and ALISTA (Liu et al., 2019) .
Notations: In this paper, unless otherwise clarified, vectors and matrices are denoted by lowercase and uppercase characters, respectively.
For vectors/matrices originally introduced without any subscript, adding a subscript (e.g., i) indicates its element/column at the corresponding position.
For instance, for x ∈ R n , x i represents the i-th element of the vector, and W :,i and W i,: denote the i-th column and row of a matrix W respectively.
While for vectors introduced with subscripts already, e.g., x s , we use (x s ) i to denote its i-th element.
The operator is used to indicate element-wise multiplication of two vectors.
The support of a vector is denoted as supp(x) := {i|x i = 0}.
We use sup xs as the simplified form of sup xs∈X (B,s,0) , see Assumption 1 for the definition of X (B, s, 0).
In this paper, we study LISTA for solving sparse coding problems.
We discover its potential weaknesses and introduce gated mechanisms to address them accordingly.
In particular, we theoretically prove that LISTA with gain gates can achieve faster convergence than the standard LISTA.
We also discover that LISTA (with or without gates) can obtain lower reconstruction errors under a weaker assumption of "false positive" in its code estimations.
It helps us improve the convergence analyses to achieve more solid theoretical results, which have been perfectly confirmed in simulation experiments.
The effectiveness of our introduced gates is verified in a variety of sparse coding experiments and the state-of-the-art performance is achieved.
In the future, we aim to extend the method to convolutional neural networks to deal with more complex tasks.
Before we delve deeply into the proof, we first give some importance notations.
We define S as the support of the vector x s , i.e. S = supp(x s ), and let |S| denote the number of elements in the set S. For a vector that shares the same size with x s , say z, we denote by z S ∈ R |S| a vector that keeps the elements with indices of z in S and removes the others.
If the vectors have been introduced with subscripts already, e.g. x s , we use (x s ) S to denote vectors obtained in such a manner.
For a square matrix with the same number of row and column as the size of x s , say M , M (S, S) is its principal minor with the index set formed by removing rows and columns whose indices are not in S. Assume a vector x with no zero elements, sign(·) is defined as (sign(x)) i = x i /|x i |, i.e. (sign(x)) i = 1 when x i > 0, and (sign(x)) i = −1 when x i < 0.

assistant: text: Proposes extensions to LISTA which address underestimation by introducing "gain gates" and including momentum with "overshoot gates", showing improved convergence rates.