Learned ISTA with Error-based Thresholding for Adaptive Sparse Coding

Abstract: The learned iterative shrinkage thresholding algorithm (LISTA) introduces deep unfolding models with learnable thresholds in the shrinkage function for sparse coding. Drawing on some theoretical insights, we advocate an error-based thresholding (EBT) mechanism for LISTA, which leverages a function of the layer-wise reconstruction error to suggest an appropriate threshold value for each observation on each layer. We show that the EBT mechanism well-disentangles the learnable parameters in the shrinkage functions from the reconstruction errors, making them more adaptive to the various observations. With rigorous theoretical analyses, we show that the proposed EBT can lead to faster convergence on the basis of LISTA and its variants, in addition to its higher adaptivity. Extensive experimental results confirm our theoretical analyses and verify the effectiveness of our methods.

Reviews and author responses:

### Strengh

- The idea makes sense to adapt the thresholding mechanism to an input distribution with various reconstruction error. It might bring a much better empirical performance compared to thresholds fixed globally and it seems to be adapted in a denoising setting.


### Weakness

- The motivation for this work is not sufficiently furnished. The authors claim that it is useful when there is a discrepancy between train/test distribution but do not provide reference to realistic situation where such a problem arise.
- Also, this method cannot really be used for real sparse coding problems as the network needs to be trained with the ground truth which is often not known in practice.
- The theoretical contribution is marginal as it is almost straightforwardly adapted from Chen et al. (2018) and Liu et al. (2019).
- The theoretical results are not precise enough (`s small enough`) while this assumption might very well render all the results only applicable to toyish case. Moreover, these assumptions are stronger than the one in `Chen et al. (2018)` and `Liu et al. (2019)` for $x_s$ realizing the sup in the expression of $b^t$ (Eq.7).
- Almost all the experiments use the same setting as previous work, failing to highlight the advantage of the proposed method.
- The performance advantage over LISTA seems to be minor from Figure.2.


## Extra remarks


- The proposed goal is to adapt LISTA in a setting where the input training distribution is *different* from the testing one. However, in this setting, using an algorithm like LISTA does not make sense as the learned weights have no reason to be adapted to the new distribution if the distribution do not overlap at all. There might be some degree to which it is possible to adapt but this should be made more explicit and better discussed. In particular, what type of distribution shift are considered and would make sense -- sparsity is mentionned but it is unclear.

- p.3: `normal training of LISTA leads to it.` I don't think there is any results showing that SGD over LISTA achieves such threshold in theory and I haven't seen any proper empirical validation. If it exists, a proper citation is needed. Else, the statement should be updated.

- p.3: `According to some prior works, we also know that U(t)∈ W(A)`: in the three cited papers, there seems to be no results showing that the learned `U(t)` verifies this. The statement is once again too strong.

- Eq.(8): Would it be interesting to evaluate the usage of $\rho^{(t)} = \mu(A)$?

- p.4: `the main results are obtained under a mild assumption of the ground-truth sparse code` -> The assumption is not mild. For instance, it seems to never be verified in any of the experiments. The statement $\mu(A)s \ll 1$ seems not backed by any experimental evidence and I don't think this is true.

- p.4: `the above assumption gives a more detailed description of the distribution for $x_s$` while it is true that it sets a distribution on the space $\mathcal X$, it is not more precise as in the assumptions by `Chen et al. (2018)`, I believe no distribution is mentionned. so overall it constrains the type of distribution while it is not required in `Chen et al. (2018)`


## Minor comments, nitpicks and typos

- citations in () could use `citealt` to remove the extra parenthesis.
- p.1: Lasso can also be solved using CD algorithms, which are typically state of the art.
- p.1: In Gregor&LeCun (2010), the thresholding is not modified compared to ISTA.
- p.2: `with W(t)=I−U(t)A holds for any layer` -> `in the case where W(t)=I−U(t)A holds for any layer`.
- Eq.(7):
- `Liu et al. (2018)`: The proper citation is `Liu, J., Chen, X., Wang, Z. & Yin, W. ALISTA: Analytic Weights are as good as Learned weigths in LISTA. in International Conference on Learning Representation (ICLR) 1113–1117 (2019).`
- p.4: `a truncated distribution` -> I assume the authors mean `truncated gaussian distribution`?


To Q1: Indeed, LISTA performs worse when the sparsity level of the test data is different from that of the training data, and our experimental results validate this. However, introducing our EBT mechanism largely mitigates the performance degradation (see Figure 5 in the updated paper). We have also added experimental results in Figure 5(a) to show how the models perform under larger distribution shifts, and it can be seen that the superiority of our method is more significant under such circumstances.

To Q3: We have actually added an explanation at the very beginning of Section 4 for the assumption that $s$ being sufficiently small (highlighted in blue in the current version of the paper). As explained in our previous response to your comments, we aimed to say $\mu(A)s\ll 1$ when mentioning $s$ being sufficiently small. We have added more discussions in Section 5.1 about such an assumption and how it would affect the empirical sparse coding results when our EBT is used. Comparing experimental results on $p_b=0.8$, $p_b=0.9$, and $p_b=0.95$ in Figure 7 and Figure 9, we can see that the performance gain of our EBT mechanism on the basis of LISTA and its variants is larger with $p_b=0.95$, for which the assumption of a sufficiently small $s$ is more likely to hold. 

To Q4: We appreciate the suggestion and we have now added experimental results in a setting where a larger distribution gap exists (see Figure 5(b) in the updated paper). Specifically, for Figure 5(a), the ratio of non-zero elements in the test sparse codes (with $p_b=0.99$) is generally 20x smaller than in the training sparse codes (with $p_b=0.8$) and our method still performs favourably well.

To Q6 and Q7: Indeed, the assumption of our theory mostly follows that of prior theoretical work, and we would like to study them empirically and report results in a future version of the paper, due to the time limit of the rebuttal period.


We would like to thank the reviewer again for recognizing our technical contributions and providing further suggestions. We have done an extra experiment to test the model performance in dealing with larger sparsity shifts (e.g., from 0.8 to 0.99). The results show that the superiority of our method is more significant under larger sparsity shifts (see Figure 5(a) in the updated paper). Following the suggestions from AnonReviewer1, we have revised the paper to add more explanations and remove imprecise statements (e.g., we have pointed out that for a sufficiently small $s$, we meant $\mu(A)s \ll 1$ at the beginning of Section 4). We will definitely keep revising the paper to improve the organization and highlight the theoretical contributions.

We thank the reviewer for the constructive comments. Our responses are given as below:

Q1 & Q2: Motivation of the work when there is a discrepancy between train/test distribution, and the usage in real sparse coding problems.

A: Similar to LISTA and its variants, our work considers a setting for model training where a set of observations paired with the corresponding ground-truth sparse codes are used. We believe that the discrepancy between train/test distribution arises in many practical applications. For instance, in the task of photometric stereo analysis, if the training and test observations are obtained from two different devices, it is highly likely that the sparsity of the training and test noise vectors are different. In addition, in many practical applications, as has been mentioned by the reviewer, we may not be given the ground-truth sparse representations of the observations, yet, with the dictionary matrix $A$, we can still simulate a set of sparse codes together with the observations $\{(y,x_s)\}$ for training, which leads to a discrepancy between the train and test distribution. Our method can be used to handle such scenarios.

Q3: Theory results are not precise and the assumptions are not mild.

A3: We appreciate the pointer to imprecise statements. We aimed to say $\mu(A)s \ll 1$ when mentioning $s$ being sufficiently small. Note that in the prior work, $s$ is also assumed to be sufficiently small to satisfy $\mu(A)(2s-1)<1$ (see Theorem 2 in [2] and Theorem 1 in [3]). Our assumption is given in a similar manner. Experimental results in Figure 6(a) ($p_b=0.95$), Figure 9(a) ($p_b=0.9$), and Figure 9(b) ($p_b=0.8$) also show that our EBT mechanism leads to better performance when $p_b$ is larger, i.e., when $s$ is smaller, which well validates our theory. In addition, even without such an additional assumption, we still have some theoretical results that our EBT-LISTA and EBT-LISTA-SS converge faster than LISTA and LISTA-SS with probability (see our proof). 
Overall, we argue that the theoretical contribution of our work is significant. We not only demonstrate how much performance gain our EBT can achieve in theory but also shed light on the two convergence phases of the variants of LISTA.

Q4: The experiments use the same setting as previous work, failing to highlight the advantage of the proposed method.

A4: We followed some common practice to design experiments for fair comparisons with prior work, and, in order to show the advantages of our method more clearly in some concerned settings, we have shown more results in the two paragraphs of “Adaptivity to unknown sparsity” (page 8) and “Random sparsity” (page 11) in our paper. Experimental results demonstrate that our EBT mechanism leads to superior performance than that of all competitors. We encourage the reviewer to take a closer look at the two paragraphs for more details.

Q5: Comment on Figure 2 for our empirical advantage.

A5: Figure 2 is shown to validate our theoretical results that many variants of LISTA have two convergence phases and our EBT accelerates, in particular, the first phase. We have revised the caption of Figure 2 to make our statement more concrete. 

Q6: Comment on “Normal training of LISTA leads to it”.

A6: We thank the reviewer for pointing out the confusing statement. We aimed to say that linear convergence can be obtained if $b^{(t)} = \mu(A)\mathrm{sup}_{i=0,1,\ldots,n}\Vert x_i^{(t)}-x_{s_i}\Vert_p$ and some other conditions (including $U^{(t)}\in \mathcal{W}(A)$ as mentioned) hold. This is a theoretical result of prior work (see Eq. (30) in [2]), and we have added proper reference there in the updated version of our paper.

Q7: Comment on $U^{(t)}\in \mathcal{W}(A)$.

A7: $U^{(t)}\in \mathcal{W}(A)$ is also one of the assumptions for guaranteeing linear convergence of LISTA [2][3][4] (see Theorem 2 in [2], Theorem 1 in [3], and Proposition 1 in [4] for more details). We have revised the statement in the updated version of our paper.

Thank you for the clarifications and the updated figure. These help to make this paper more clear.

After reading other reviews, especially comments from AnonReviewer1, I will still maintain my current score, although I do share the same concern about the innovative contributions of this paper, especially the theory part. I am open to more discussion about his paper.

However, I buy the motivation and the technical contributions of this paper much. Different from AnonReviewer1, in my humble opinion, I think the proposed method can also work in the existence of sparsity adaptation. As AnonReviewer1 mentioned, "So in case there is a large shift of sparsity,", according to equation (12), when the residual is large due to the large step sizes learned in the small sparsity situation, the threshold $b^{(t)}$ will also be large, thresholding more aggressively so that divergence could possibly be alleviated. In contrast, the classic LISTA models where only one single threshold parameter is learned in each layer can not adapt to this sparsity shift. But I also agree with AnonReviewer1's point that experiments with larger sparsity shift can further strengthen the solidness of the proposed methodology. I encourage the authors to add such experiments and reflect them in the final version, at least in the appendix, although it might not be ready before the discussion period closes.

I would also recommend the authors to address the ambiguity and imprecision of some statements in the paper, many of which were pointed out by AnonReviewer1. For example, as the adaptation experiments are the core empirical results of this paper, it might be put somewhere earlier in the paper, and leave the sanity check experiments using classic LISTA settings later.

I would like to thanks the authors for the detailed response.

I would like to clarify that my main issue is not really with the results of the paper. I also find them a bit incremental as AnonReviewer2 but with proper presentation and experiments, I would be fine with this. The main issue is that the presentation of the paper is misleading. The motivation on the distribution shift should be more precise on what would work (and is validated) and what remains to be shown (Q1). The theoretical results should be made more precise to highlight the fact that they don't apply in the proposed experiments but that it seems to be working fine in practice (Q3). The experiments are not totally convincing (Q4) and other statements are still incorrect (Q6). These points should be corrected, else the readers might draw incorrect conclusion from this paper. I detail more these points bellow. Overall, I still think this paper is not ready for publication and can be largely improved. But I am ready to discuss with the other reviewers if they think the paper is good enough, I will increase my rating.

### Answer to the rebuttal

Q1: To me, it seems that the proposed method can mainly be used when there is a discrepancy in noise level. Indeed in this case, adapting the thresholding will automatically adjust the denoising level to the noise level and you can gain something. However, LISTA is all about learning a better algorithm to solve Lasso on a given input distribution. If the input distribution change -- like the sparsity level, the distribution of the non zero coefficients or more importantly the type of noise -- then, there is little hope to still reach convergence. It can easily be seen for instance with the sparsity level.
For very low sparsity, one can use large step sizes but if in the test, there is a sample which is non sparse, the algorithm can easily diverge toward infinity. So in case there is a large shift of sparsity, the proposed method will also fail (shift in Fig6 are quite limited, see Q4).
For different noise model -- which is the situation the authors refer to when saying training on generated samples where one don't know a good noise model -- it is unclear that this would really work.
Thus, I still think the motivation is too vague in the intro. Saying that the method can adapt to different noise level would be a better motivation, coherent with what is shown in the paper.

Q3: I still don't understand why the statement `s small enough`   is not made clear in the paper. This is quite important as this determine whether or not the result apply in practice and make it possible to compare with the literature. In particular, reading the proof show that this condition is stronger for some $x_s$ in $\gamma(B, s)$ than the statement in the literature. Also, the theoretical results show that there is a faster convergence with probability $1-\mu(A)s$. But in most practical cases, this will be a probability 0 -- I might be wrong but I am almost sure $\mu(A)s$ is 0 in most the proposed experiments. This means that  the results don't applies in the proposed experiments. This is not fundamentally a flaw but this should be highlighted and properly discussed.

Q4: I agree that it is common practice and that it is good to show for the first few plot. But still, the experiments could be improved to better prove the point. For instance in Fig6, the authors could show how far sparsity can be changed before the different networks fail instead of picking 3 small magnitude changes. Also, adding error bars on the curve and larger font would improve the plots and make the results more robust.

Q5: Thanks for updating this, it is clearer this way.

Q6: There is nothing in Chen et al. (2018) showing that SGD on LISTA learns a threshold like this. What Eq (30) says is "we propose a choice of parameters". I don't think there is any results showing convergence of the weights of LISTA to given values.

Q7: This is also an assumption and there is no results showing convergence of learned weights to such set.

Q8/9/10: Thanks for taking these remarks into account. The extra experiment is interesting as it shows that the network learns something a bit different.

Thanks for the positive feedback and comments. Our responses are given as below:

Q1: Whether the new parameterization used in EBT changes the training dynamics or not.

A1: Following the suggestion, we have redrawn Figure 3 in the paper for better illustration of the training dynamics. In the figure, we can see that the learned thresholds in our EBT-based methods and the original LISTA and LISTA-SS are similar, which indicates that the introduced EBT mechanism does not modify the training dynamics of the original methods, and it works by disentangling the reconstruction error and learnable parameters.

Q2: How many samples are used for training?

A2: Following prior work [1] [2] [3], we synthesized in-stream data for training all models in comparison, thus the number of training samples grows as the training proceeds. Since all models were trained for the same number of iterations, they all share the same training complexity.

Q3: Mistakes in eqn (15).

A3: Thanks for pointing out. In our paper, $L^{\dagger}$ is the orthogonal complement of $L$ rather than the pseudo-inverse, and thus Eq.(14) can be rewritten as $$ L^{\dagger}o = \rho L^{\dagger}Ln + L^{\dagger}e = L^{\dagger}e.$$ We have revised Eq.(15) in the latest version of the paper for better clarity.

[1]Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9061–9071, 2018.

[2] Jialin Liu, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. ALISTA: Analytic weights are as good as learned weights in LISTA. In Proceedings of the International Conference on Learning Representations (ICLR), pp. 1113–1117, 2019. 

[3] Kailun Wu, Yiwen Guo, Ziang Li, and Changshui Zhang. Sparse coding with gated learned ISTA. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.


Q8: Evaluate the usage of $\rho^{(t)}=\mu(A)$.

A8: We appreciate the suggestion of testing $\rho^{(t)}=\mu(A)$. We have run the experiment based on LISTA and LISTA-SS (the new methods are codenamed EBT-LISTA-fixed and EBT-LISTA-SS-fixed). The below table compares the result (shown in dB) of EBT-LISTA, EBT-LISTA-SS, EBT-LISTA-fixed, and EBT-LISTA-SS-fixed. It can be seen that using $\rho^{(t)}=\mu(A)$ does not lead to superior performance that that of our original EBT mechanism.

|  Method   | Step 10  | Step 16 |
|  :----  | :----:  | :----:|
| EBT-LISTA-fixed  | -20.78 | -22.47|
| EBT-LISTA  | -22.14 | -37.48|
|EBT-LISTA-SS-fixed| -40.83 | -64.39|
|EBT-LISTA-SS| -45.24 | -68.73 |


Q9: Comment on “more detailed description” in our discussions about the assumption.

A9: Indeed, we further introduce an assumption about the distribution of the sparse codes, and we believe that it is necessary for studying LISTA in scenarios where there exists a discrepancy between training and test data. We have rephrased the paper to state that “a more detailed yet also stricter description” is provided.

Q10: Minor comments, nitpicks and typos.

A10: Thanks for pointing out the typos and making suggestions on our writing. We have revised the paper accordingly. Besides, we would like to clarify that in LISTA [1], the thresholds are actually learnable parameters, which is different from ISTA. The related descriptions can be found in [1] (its Section 3.3, page 6).

[1] Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 399–406. Omnipress, 2010

[2] Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9061–9071, 2018.

[3] Jialin Liu, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. ALISTA: Analytic weights are as good as learned weights in LISTA. In Proceedings of the International Conference on Learning Representations (ICLR), pp. 1113–1117, 2019. 

[4] Kailun Wu, Yiwen Guo, Ziang Li, and Changshui Zhang. Sparse coding with gated learned ISTA. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.


Thanks for the positive feedback and constructive suggestions. We have revised the paper accordingly. The architecture of LISTA and our proposed EBT-LISTA has been shown in Section 3, and Table 1 has been modified. The results of EBT-ISTA and EBT-FISTA have been shown in Appendix A.1 (Figure 11). Specifically, introducing our EBT mechanism in the classical ISTA and FISTA leads to faster initial convergences but worse final performance. This is because ISTA and FISTA converge in a sub-linear manner, but our EBT is mostly proposed for methods that converge linearly, which might cause a mismatch to our assumptions.

We would like to thank the reviewer for the positive feedback.

In the paper, authors propose a new error-based thresholding mechanism for LISTA which introduces a function of the evolving estimation error to provide each threshold in the shrinkage functions. They provided the theoretical analysis for EBT-LISTA and EBT-LISTA with support selection and proved that the  estimation error of the proposed algorithm is theoretically lower than compared methods. The authors also evaluated the proposed method on multiple synthetic or real tasks. Experimental results show that the proposed method achieves a better estimation error and  higher adaptivity to different observations with a variety of sparsity.



This paper disentangles the threshold parameters in LISTA-type models from the reconstruction errors, proposing the Error-Based Threholding (EBT) mechanism which mainly follows a theoretical results in (Chen et al., 2019; Liu et al., 2018), where the threshold at one layer is proportional to the recovery error of current iterate. The benefits brought by the proposed EBT method are faster convergence and better adaptivity to a wider range of samples. To bypass the requirement of ground truth sparse signals, EBT uses the reconstruction error following a learned linear transform, which in theory has good coherence property with the dictionary and therefore can approximate the recovery error well. The authors theoretically show that the proposed EBT mechanism enjoys faster convergence in both cases with and without the support selection technique. Emprirical experiments on standard synthetic setting and cross-sparsity setting are shown to support the efficacy of EBT. The authors also do real-world photometric stereo analysis to show the superiority of EBT.

Pros:
- This paper is a successful extention of basic LISTA-type models by disentangling the learnable threshold parameters.
- Theoretical analysis of the benefits of EBT is provided and looks correct to me.
- The empirical experiments are solid enough to show the superiority of EBT.

Cons:
- My main concern is about that this paper is a little bit incremental, as it seems to be a very direct extension based on previous theoretical results.

Other comments:
- In Figure 1, it can be observed that the thresholds learned LISTA-SS have a bumpy curve, which is also observed in previous works. In contrast, the $\rho^{(t)}$ parameters look much stabilized. It might be better to also show the curves of reconstruction error term, i.e. the $\ell_p$ term in eqn (12) to see how they look like. This would help to understand if the new parameterization used in EBT changes the training dynamics or not.
- In the basic settings part, the numbers of validation and testing samples are provided. How many samples are used for training?
- In eqn (15), is the term about $\rho n$ omitted on purpose, or mistakenly?

In summary, I think this is a good extension paper but I have a little concern about its incremental contribution. Overall I think it is slightly above the threshold. I am open to more opinions from other reviewers.


Summary:
The authors propose an automatic threshold selection for LISTA-style neural nets. The threshold introduces negligible number of parameters (either 0, or one per layer). This choice is shown theoretically and empirically to have faster convergence than methods without it and popular alternatives.

Clarity:
The clarity of your idea would be improved using a graphic: for example showing the LISTA architecture, and what sort of computations are performed in a "feed in" direction toward the shrinkage operator. Just a thought, not a criticism. The information in Table 1 might be more succinctly reported using exponential notation, or speedup as a percentage (I count 39 extraneous 0's)

Quality:
The quality would be improved if you applied this adaptive thresholding idea to classic (F)ISTA, i.e. updating (F)ISTA's thresholds via the parameterless form of EBT. How would this impact standard (F)ISTA convergence, i.e., compared to using a fixed threshold run on a whole test set? (is ebt-threshold selection better than say, doing a parameter search for fixed threshold "b", over a training set on (F)ISTA?)

The experiment variety is of good quality: experiments across condition numbers and sparsities, validation of theorems, etc.

Originality: 
I am not familiar with other works like this, and I like the idea. I wish we were given insight to how it affects nonlearned (F)ISTA algorithms, which are still widely used in practice.

Significance:
The result is significant in the specific application of LISTA.

Meta-review: The paper received mixed reviews, with one review voting for acceptance, one strongly opposed, and two borderline ones. The discussion essentially involved R1 and R2, who gave the most informative reviews. After discussion, they did not update their score, even though they appreciated the work and effort done by the authors during the rebuttal. 

In short, the paper has some merit, but several concerns were raised, which the area chair agrees with, leading to a rejection recommendation. The innovation was found to be limited and the discussion between practice and theory (meaning assumptions made in this work) are not discussed in a convincing manner, and these concerns remained after the rebuttal. The experiments were also subject to improvements.

It is however likely that with a major revision, this work may become publishable to a another venue.

Decision: withdrawn-rejected-submissions