Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposed two novel gates to improve the convergence speed of the learned ISTA (LISTA) algorithm for sparse coding. The first gate is designed to address the problem that the output of the LISTA algorithm usually has a lower magnitude compared against the ground-truth. To address this, the paper proposed a gain gate to increase the magnitude of a layer in the LISTA algorithm. The second gate is designed to further improve the convergence with a technique similar to the momentum but with a time-varying coefficient. 

This paper provides rigorous theoretical justifications for the observations that motivate the two gates in two propositions. It also provides theoretical guarantees for the first gain gate under practical assumptions. 

Thorough synthetic experiments are conducted to empirically validate the proposals as well as the theorems. The effectiveness of the gates is also proved in a real-world computer vision task, photometric stereo. 

Given the thoroughness of the theoretical and experimental justifications, I strongly recommend the acceptance of the paper.

Despite the strength of the paper, I have the following suggestions and questions regarding the two gates proposed by the paper:

1. The second proposed gate, the overshoot gate, lacks theoretical justification and is not as well studied as the gain gate. The experiments for this gate are not included in the main text but got placed in the appendix. I suggest squeezing some of them into the main text to show a complete picture about the overshoot gate. If there are more theoretical results about the behavior of the overshoot gate, the author should include it in the paper; otherwise, the author should clearly states this limitation in the main text of the paper.

2. The author proposed two gates to address two different issues of the original LISTA algorithm. However, the two gates are only applied seperately to modify the LISTA algorithm; they have not been integrated into a single concise algorithm. It would strengthen the paper if the author could provide a further discussion on how the two gates can be combined to address the two issues in one algorithm.

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Summary:

This paper is focused on solving sparse coding problems using LISTA-type
networks. It discusses the weakness of the ``no false positive'' assumption in
previous works and the weakness results in underestimated code components. The
authors propose a ``gain gating function'' to mitigate the weakness. Moreover,
the paper incorporates another ``overshoot'' gating function inspired by
momentum-based methods. Both contributions are supported with theoretical and
empirical results. Numerical experiments show that the proposed model is
superior to previous works especially in cases with high measurement noises or
ill-composed basis matrix. The paper is well written and easy to follow, and the
empirical results are impressive.

I really like the relaxation of ``no false positive'' assumption as in real world
application, learning-based algorithms may find it difficult to keep satisfying
the assumption while maintaining good empirical performance. And this relaxation
contributes to guarantee the convergence in a more real scenario.

I think in (Chen et al., 2018), the support selection technique serves to
mitigate the problem of underestimated code components as it bypasses the
thresholding function for codes with large magnitudes. The empirical results
also show that in many cases LISTA with support selection has comparable
performance. I don't know if LISTA with support selection also suffers from
this underestimation problem severely.

Questions:

1. The theorems in Section 3.1 hold without overshooting mechanism, i.e. eta==1.
Prop. 2 says that for one step of iteration, the optimal updates requires
overshoot. When the gain and overshoot mechanisms are combined, however, will
the theoretical convergence still hold?

2. I don't have a very good understanding of why GLISTA performs so well given
ill-composed basis matrix compared to previous works. Could there be some
(intuitive) explanation as to this based on the theoretical results?

3. More details about the training process should be included. For example, I
can guess the loss function used for the training. But it is confusing not
mentioning it at all in the paper. Some might think the objective funciton in
eqn (2) but the convergence analysis is with respect to the ground truth sparse
vector. Also, the training scheme and some hyperparameter selection should also
be stated at least in Appendix.


Overall I hold very positive attitude towards this paper due to its theoretical
contributions and good empirical results.

================================
Update: the authors addressed my questions well and I will keep my positive decision on this paper.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: 1. Summary
The authors propose extensions to LISTA with the goal of addressing underestimation (by introducing “gain gates”) and including momentum (by introducing “overshoot gates”). The authors provide theoretical analysis for each step of their LISTA augmentations, showing that it improves convergence rate. Their theoretical statements are empirically validated and then a numerical comparison is performed between the most interesting LISTA variants. Their proposed GLISTA performs favorably, especially for networks of depth greater than 10.
2. Decision and arguments
The theory given is quite comprehensive and seems solid. Moreover it’s motivated by and helps real problems, namely L1’s well-known underestimation and LISTA’s lack of momentum. On top of that, the numerical results not only show your GLISTA outperforming others, but that by adding gates to other LISTA variants you can get better results. On the other hand (see details in the Questions section), there is some critical information missing from the empirical section which makes the results non-repeatable. 
Training is not described, i.e. hyperparameter searches and stopping criteria. A couple of the plots validating theory are difficult to understand. The datasets used are synthesized and tiny, only 1000 samples (training/testing/validation sets are not described) and 250x500 dimensions. That means your network has many more parameters than data samples unless I am mistaken.
So I have given weak reject because the paper has strong theory but weak experiments making it hard to trust the conclusions. I really want to hear back about the questions raised below.
3. Questions 
a) Just before eqn 12 you say that gains greater than 1 can be more appropriate. Am I understanding this correctly: *before* shrinkage you want to apply a gain on code elements that are “truly” nonzero, in order to cancel out the imminent L1 penalization? And you will learn to predict those code elements via parameters Lambda?
b) Just before Section 4, you describe the difference between yours and Moreau & Bruna’s momentum taps. If I understand correctly they have a matrix called W_m^(k) which is multiplied onto the previous iterate—but for each layer / time unit (k), the matrix may vary by learning (note that in the main body of their paper they don’t explicitly use ^(k) notation, but it is explained to be the same as the other LISTA parameter matrices which vary with time, and is made explicit in the appendix). So I think their momentum is in fact time-varying. Moreover it is certainly dependent on the previous iterate, using the function f(z^(k-1)) = W_m^(k) * z^(k-1). Did you mean something else by that? In what sense do you have a higher capacity—do you have more parameters?
c) If I understand correctly, in Figure 5a, 5b, for each point you have trained independently an N-layer GLISTA. Then you observe properties of your learned parameters.
For 5a: how do you compute the “output” of the gain gates? What is the input that gives this output?
For 5b: which ‘t’ is used to calculate ||W(t)D-I+U(t)A||? Or did you just train a single 15-layer GLISTA, and the x-axis is just ‘t’?
d) Could you tell us more about the hyperparameter tests for every method? How do we know it was a fair comparison? There are no error bars, but from experience I know that training LISTA type networks can be a pain. What algorithms did you use? Stopping criteria? Some plots start at zero layers (4c, 5b, 6a-c) and some start at 1 layer… 
e) Interesting that in your real-data example, the sparse vector e has more non-zeros than zeros, is that really sparse? What dictionary A did you use, then? By your own description this task does not fit the sparse coding model you have analyzed…. Am I misunderstanding?
4. Additional feedback/ minor comments
a) Add reference for DOA estimation application
b) You should put something like “we prove this in the supplement” for props and theorems. Otherwise to readers less familiar with the literature it will seem like you forgot to put a reference.
c) It seems unnecessary to put ||epsilon||<= 0 , instead of epsilon=0.
d) Basically I think you should explain what your gate is before providing proofs about it. It would be useful to see a plot of the function g and/or kappa so provide some intuition about what you are doing to the architecture (nonlinear? Linear? Threshold? Etc.). Capital “Lambda” is not defined until after theorems are provided about the functions g, kappa, f. In fact it seems like kappa is only useful for the sake of proofs. The first time through, I over-read the statement “All the learnable parameters are thus collected as Lambda…” just before Section 3.1.1 because I was thinking “Why isn’t Lambda in the definitions of f/g/kappa??”. Anyway it makes perfect sense when you define it clearly as after Eqn 18.
e) At the beginning of Section 3.2 you have a clause “the over-shoot gates act more like on the output”, which doesn’t make sense.
f) Figures should be approximately self-explanatory—but “Ratio of lower components” is not explained in the caption of Fig 4b. Although I greatly appreciate that you have made the actual plot lines/markers very easy to see! Should these results be averaged over many training attempts with error bars?

review_title: Official Blind Review #2

Meta-review: The paper extends LISTA by introducing gain gates and overshoot gates, which respectively address underestimation of code components and compensation of small step size of LISTA. The authors theoretically analyze these extensions and backup the effectiveness of their proposed algorithm with encouraging empirical results. All reviewers are highly positive on the contributions of this paper, and appreciate the rigorous theory which is further supported by convincing experiments. All three reviewers recommended accept.

Decision: Accept (Spotlight)