Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper the neural tangent kernel for RNNs is derived. It is emphasized how this results in a proper kernel that can handle samples of different lengths.

The  theoretical derivations are correct to the best of my knowledge and help to complete the picture of deep architectures for which the NTK has been derived. This is a valuable contribution.

My main concern is on the empirical side. Since the sinusoid regression is a toy problem I'll focus on the other two:

- Google stock price regression: Appendix A2 doesn't provide much insight into the experiment setup, but it seems that the objective is to predict the stock price of the next day based on the previous ones. I would assume that a strong baseline for this task will be to simply predict the stock price to be the same as the previous day. This is a trivial predictor that requires a training set size of zero. So if you were to plot its performance on Figs 4c and 4d, which (constant) SNR would it yield? I'd be surprised if it is beaten (and that would be a very interesting result), but also I'd be surprised if it's not matched (seems a simple enough predictor to discover if there is no overfitting). However, those graphs show a big spread between different predictors, which is not intuitive given the previous sentences. Can you pleaser report the SNR of "my" trivial predictor and explain the apparent discrepancy?

- The 53 UCR classification datasets: Why those 53? The database contains 128, and your paper states that you took "data sets with fewer than 1000 training samples and fewer than 1000 time steps". But datasets "Adiac" and "Beef" fulfill those requirements and were not taken. On the other hand, "StarLightCurves" doesn't seem to fulfill them (1024 steps) and yet you took it. Can you clearly state how you chose those 53 datasets out of the 128 available?
review_title: Theoretical results are valuable,  empirical results are insufficient.

final_score: rating: 7: Good paper, accept
review_content: This paper studies the NTK of RNNs in the infinite-width limit, and shows a number of interesting features of such networks, that are somehow surprising knowing the problems of exploding gradients, or knowing the need for independence of parameters in proofs involving the NTK (for the tied weights giving the same as the untied case). While the techniques are not particularly new (they are relying on the applciation of techniques appearing in earlier papers), and the idea to look at RNNs is fairly straightforward, I think this is an interesting and useful paper, with nontrivial estimates (that look correct, although I may need a bit more time to check). The quality of the writing is extremely high, the notations are optimal, and it is overall a pleasure to read, and this paper will serve as an excellent basis for future investigations.
review_title: Limited in scope, but interesting, and excellent in the presentation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper extends NTK to RNN to explain behavior of RNNs in overparametrized case. It’s a good extension study and interesting to see RNN with infinite-width limit converges to a kernel. The paper proves the same RNTK formula when the weights are shared and not shared. The proposed sensitivity for computationally friendly RNTK hyperparameter tuning is also insightful.

Weakness:

In the experimental part, the paper claims they restrict the data with shorter and fewer samples. This may be a downside of the proposed method as the goal of RNNs is to handle various length of data samples. Also, it seems that the proposed RNTK cannot outperform other SOA methods. Any reason? Why are GRU and identity RNNs chosen? Please highlight the best number for each experiment for an easier comparison. 


review_title: overall this is a good submission

Meta-review: Reviewers agreed on the value of theoretical contribution, especially the surprising conclusion that the weight-tied and untied RNTK are identical. The empirical results were updated in response to reviewer's suggestion. I believe this would be of interest to ICLR audience.
Decision: Accept (Poster)