Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes a novel method to solve the sequential signal reconstruction problem. The method is based on the deep unfolding methods and incorporates the reweighting mechanism. Additionally, they derive the generalization error bound and show how their over-parameterized reweighting RNNs ensure good generalization. Lastly, the experiments on the task of video sequence reconstruction suggest the superior performance of the proposed method.

I recommend the paper to be accepted for mainly two reasons. First, they derive a tighter generalization bound for deep RNNs; Second, the experiment results align with the theory and show the continuous improvements when increasing the depth of RNNs.

Questions:
1. How is the computation complexity of the proposed method when compared with other methods? Will the reweighting l1-l1 norm significantly increase the computation time?
2. The experiments show that increasing the depth and/or width of the networks yields better performance, however, is there a boundary for such performance gain? For example, if the depth continues increasing, will the proposed method suffer the similar problem as other methods (performance does not improve or even degrade)?
3. As the MOVING MNIST dataset is from a relatively simple and special domain, is it possible to reproduce the similar performance gain on other more realistic datasets?
4. Are there any known limitations of the proposed method?
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Authors proposed a deep RNN via unfolding reweighted l1-l1 minimization, where reweighted l1-l1 minimization algorithms are applied to a video task. 
Overall, the paper is well explained in a theoretical part and exhibits a good result compared with other conventional RNN methods in the experiment. In Section 3, authors formulate Rademacher complexities for both conventional and proposed method, which shows the generalization performance of the proposed method when d increases. And this is empirically highlighted in Table 3 in Section 4.

Major points:
Section 1:
-	First part of the introduction can be confusing because Eq. (1) sounds like representing dictionary learning framework (plus DNN is immediately described after Eq. (1) instead of RNN) and RNN is not explicitly written. It should be clearly written and flow should be considered.
Section 2:
-	It is hard to get how parameter g in Eq. (3) derives. 
Section 3:
-	How to build network depth d for the network? A figure should be required.
Section 4:
-	Even though previous papers (e.g., Wisdom et al. and Le et al.) just focus on single dataset like moving MNIST, I believe testing on language data is also quite important (this is a full paper and exhaustive experiments should be mandatory). For example, it may be good to use Penn TreeBank dataset to make a comparison. 
-	In Table 3, how did you set LSTM deeper? Is it a stacked LSTM?
-	Existing RNN methods should include other variations of LSTM (in particular, SOTA methods are welcomed) such as bidirectional LSTM and LSTM with attention mechanism. It should be better to compare with these methods.

Appendix:
-	It would be helpful for readers to show interpretabilities of the model additionally. For example, visualizing features from each RNN model would be beneficial.

Minor points:
-	After introduction of unfolding reweighted l1-l1 minimization, how did the computational cost increase compared to previous l1-l1 minimization?
-	In Section3, for easiness to readers, it may be good to briefly summarize how does the predictor’s generalizability and Rademacher complexities relate.
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: 
Strength:
This paper proposes a new reweighted-RNN by unfolding a reweighted L1-L1 minimization problem. It develops an iterative algorithm to solve the reweighted L1-L1 minimization problem, where the soft-thresholding functions can be adaptively learned. This paper provides the generalization error bound for deep RNNs and shows that the proposed reweighted-RNN has a lower generalization error bound. In addition, the paper shows that the proposed algorithm can be applied to video-frame reconstruction and achieves favorable results against state-of-the-art methods. The paper is well organized, and the motivation is clear. 

Weakness:
The effectiveness of the reweighted L1-L1 minimization method should be better explained and evaluated. It is not clear why the reweighted L1-L1 regularization is better than the L1-L1 regularization. In addition, the experimental evaluation does not support this claim well. The authors should compare the baseline method which uses the  L1-L1 regularization in their framework instead of directly comparing the proposed algorithm with [Le et al., 2019] as there exist differences in the algorithm design. This is an important baseline. 

As claimed by the authors, the proposed reweighted-RNN has different sets of {W_l;U_l} for each hidden layer. This will definitively increase the model size when the depth increases. The authors should clarify whether the performance gains due to the only use of large model parameters. 

Overall, this paper proposes an effective reweighted-RNN model based on the solver of a reweighted L1-L1 minimization. Theoretical analysis and experimental results are provided. I would be willing to increase the score if these problems are solved in the authors’ response. 

review_title: Official Blind Review #1

Meta-review: This paper presents a novel RNN algorithm based on unfolding a reweighted L1-L1 minimization problem. Authors derive the generalization error bound which is tighter than existing methods. 
All reviewers appreciate the theoretical contributions of the paper, particularly the derivation of generalization error bounds. However, at a higher-level, the overall idea is incremental because RNN by unfolding L1-L1 minimization problem (Le+,2019) and reweighted L1 minimization (Candes+,2008) are both known techniques. The proposed method is essentially a simple combination of them and therefore the result seems somewhat obvious. Also, I agree with reviewers that some experiments are not deep enough to support the theory. For example, for over-parameterization (large model parameters) issue, one can compare the models with the same number of parameters and observe how they generalize. 
Overall, this is the very borderline paper that provides a good theoretical contribution with limited conceptual novelty and empirical evidences. As a conclusion, I decided to recommend rejection but could be accepted if there is a room.

Decision: Reject