Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
The authors propose an automatic threshold selection for LISTA-style neural nets. The threshold introduces negligible number of parameters (either 0, or one per layer). This choice is shown theoretically and empirically to have faster convergence than methods without it and popular alternatives.

Clarity:
The clarity of your idea would be improved using a graphic: for example showing the LISTA architecture, and what sort of computations are performed in a "feed in" direction toward the shrinkage operator. Just a thought, not a criticism. The information in Table 1 might be more succinctly reported using exponential notation, or speedup as a percentage (I count 39 extraneous 0's)

Quality:
The quality would be improved if you applied this adaptive thresholding idea to classic (F)ISTA, i.e. updating (F)ISTA's thresholds via the parameterless form of EBT. How would this impact standard (F)ISTA convergence, i.e., compared to using a fixed threshold run on a whole test set? (is ebt-threshold selection better than say, doing a parameter search for fixed threshold "b", over a training set on (F)ISTA?)

The experiment variety is of good quality: experiments across condition numbers and sparsities, validation of theorems, etc.

Originality: 
I am not familiar with other works like this, and I like the idea. I wish we were given insight to how it affects nonlearned (F)ISTA algorithms, which are still widely used in practice.

Significance:
The result is significant in the specific application of LISTA.
review_title: automatic threshold selection for LISTA-like nets

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper disentangles the threshold parameters in LISTA-type models from the reconstruction errors, proposing the Error-Based Threholding (EBT) mechanism which mainly follows a theoretical results in (Chen et al., 2019; Liu et al., 2018), where the threshold at one layer is proportional to the recovery error of current iterate. The benefits brought by the proposed EBT method are faster convergence and better adaptivity to a wider range of samples. To bypass the requirement of ground truth sparse signals, EBT uses the reconstruction error following a learned linear transform, which in theory has good coherence property with the dictionary and therefore can approximate the recovery error well. The authors theoretically show that the proposed EBT mechanism enjoys faster convergence in both cases with and without the support selection technique. Emprirical experiments on standard synthetic setting and cross-sparsity setting are shown to support the efficacy of EBT. The authors also do real-world photometric stereo analysis to show the superiority of EBT.

Pros:
- This paper is a successful extention of basic LISTA-type models by disentangling the learnable threshold parameters.
- Theoretical analysis of the benefits of EBT is provided and looks correct to me.
- The empirical experiments are solid enough to show the superiority of EBT.

Cons:
- My main concern is about that this paper is a little bit incremental, as it seems to be a very direct extension based on previous theoretical results.

Other comments:
- In Figure 1, it can be observed that the thresholds learned LISTA-SS have a bumpy curve, which is also observed in previous works. In contrast, the $\rho^{(t)}$ parameters look much stabilized. It might be better to also show the curves of reconstruction error term, i.e. the $\ell_p$ term in eqn (12) to see how they look like. This would help to understand if the new parameterization used in EBT changes the training dynamics or not.
- In the basic settings part, the numbers of validation and testing samples are provided. How many samples are used for training?
- In eqn (15), is the term about $\rho n$ omitted on purpose, or mistakenly?

In summary, I think this is a good extension paper but I have a little concern about its incremental contribution. Overall I think it is slightly above the threshold. I am open to more opinions from other reviewers.

review_title: A very good extention to LISTA-type models but looks a little bit incremental

final_score: rating: 6: Marginally above acceptance threshold
review_content: In the paper, authors propose a new error-based thresholding mechanism for LISTA which introduces a function of the evolving estimation error to provide each threshold in the shrinkage functions. They provided the theoretical analysis for EBT-LISTA and EBT-LISTA with support selection and proved that the  estimation error of the proposed algorithm is theoretically lower than compared methods. The authors also evaluated the proposed method on multiple synthetic or real tasks. Experimental results show that the proposed method achieves a better estimation error and  higher adaptivity to different observations with a variety of sparsity.


review_title:  A new error-based thresholding mechanism for LISTA with theoretical guarantee

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Strengh

- The idea makes sense to adapt the thresholding mechanism to an input distribution with various reconstruction error. It might bring a much better empirical performance compared to thresholds fixed globally and it seems to be adapted in a denoising setting.


### Weakness

- The motivation for this work is not sufficiently furnished. The authors claim that it is useful when there is a discrepancy between train/test distribution but do not provide reference to realistic situation where such a problem arise.
- Also, this method cannot really be used for real sparse coding problems as the network needs to be trained with the ground truth which is often not known in practice.
- The theoretical contribution is marginal as it is almost straightforwardly adapted from Chen et al. (2018) and Liu et al. (2019).
- The theoretical results are not precise enough (`s small enough`) while this assumption might very well render all the results only applicable to toyish case. Moreover, these assumptions are stronger than the one in `Chen et al. (2018)` and `Liu et al. (2019)` for $x_s$ realizing the sup in the expression of $b^t$ (Eq.7).
- Almost all the experiments use the same setting as previous work, failing to highlight the advantage of the proposed method.
- The performance advantage over LISTA seems to be minor from Figure.2.


## Extra remarks


- The proposed goal is to adapt LISTA in a setting where the input training distribution is *different* from the testing one. However, in this setting, using an algorithm like LISTA does not make sense as the learned weights have no reason to be adapted to the new distribution if the distribution do not overlap at all. There might be some degree to which it is possible to adapt but this should be made more explicit and better discussed. In particular, what type of distribution shift are considered and would make sense -- sparsity is mentionned but it is unclear.

- p.3: `normal training of LISTA leads to it.` I don't think there is any results showing that SGD over LISTA achieves such threshold in theory and I haven't seen any proper empirical validation. If it exists, a proper citation is needed. Else, the statement should be updated.

- p.3: `According to some prior works, we also know that U(t)∈ W(A)`: in the three cited papers, there seems to be no results showing that the learned `U(t)` verifies this. The statement is once again too strong.

- Eq.(8): Would it be interesting to evaluate the usage of $\rho^{(t)} = \mu(A)$?

- p.4: `the main results are obtained under a mild assumption of the ground-truth sparse code` -> The assumption is not mild. For instance, it seems to never be verified in any of the experiments. The statement $\mu(A)s \ll 1$ seems not backed by any experimental evidence and I don't think this is true.

- p.4: `the above assumption gives a more detailed description of the distribution for $x_s$` while it is true that it sets a distribution on the space $\mathcal X$, it is not more precise as in the assumptions by `Chen et al. (2018)`, I believe no distribution is mentionned. so overall it constrains the type of distribution while it is not required in `Chen et al. (2018)`


## Minor comments, nitpicks and typos

- citations in () could use `citealt` to remove the extra parenthesis.
- p.1: Lasso can also be solved using CD algorithms, which are typically state of the art.
- p.1: In Gregor&LeCun (2010), the thresholding is not modified compared to ISTA.
- p.2: `with W(t)=I−U(t)A holds for any layer` -> `in the case where W(t)=I−U(t)A holds for any layer`.
- Eq.(7):
- `Liu et al. (2018)`: The proper citation is `Liu, J., Chen, X., Wang, Z. & Yin, W. ALISTA: Analytic Weights are as good as Learned weigths in LISTA. in International Conference on Learning Representation (ICLR) 1113–1117 (2019).`
- p.4: `a truncated distribution` -> I assume the authors mean `truncated gaussian distribution`?

review_title: Interesting idea but theoretical results and evaluation  are not satisfactory

Meta-review: The paper received mixed reviews, with one review voting for acceptance, one strongly opposed, and two borderline ones. The discussion essentially involved R1 and R2, who gave the most informative reviews. After discussion, they did not update their score, even though they appreciated the work and effort done by the authors during the rebuttal. 

In short, the paper has some merit, but several concerns were raised, which the area chair agrees with, leading to a rejection recommendation. The innovation was found to be limited and the discussion between practice and theory (meaning assumptions made in this work) are not discussed in a convincing manner, and these concerns remained after the rebuttal. The experiments were also subject to improvements.

It is however likely that with a major revision, this work may become publishable to a another venue.
Decision: Reject