Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The presented MIST architecture certainly has got its merits, but in my opinion is not very novel, given the fact that NARX RNNs have been described 20 years ago, and Clockwork RNNs (which, as the authors point out in section 2, have a similar structure) have also been in use for several years. Still, the presented results are good, with standard LSTMs being substantially outperformed in three out of five standard RNN/LSTM benchmark tasks. The analysis in section 3 is decent (see however the minor comments below), but does not offer revolutionary new insights - it's perhaps more like a corollary of previous work (Pascanu et al., 2013).

Regarding the concrete results, I would have wished for a more detailed analysis of the more surprising results, in particular, for the copy task (section 5.2): Is it really true that Clockwork RNNs fail because they make it "difficult to learn long-term behavior that must be detected at high frequency" [section 2]? How relevant are the results in figure 2 (yes, the gradient properties are very different, but is this an issue for accuracy)? In the sequential pMNIST classification, what about increasing the LSTM number of hidden units? If this brings the error rate further down, one could ask why exactly the LSTM captures long-term structure so differently with different number of units?

In summary, for me this paper is solid, and although the architecture is not that new, it is worth bringing it again into the focus of attention.


Minor comments:
- In several places, the formulas are rather strange and/or occasionally incorrect. In particular,
* on the right-hand sind of the inline formula in section 3.1, the symbol v is missing completely, which cannot be right;
* in formula 16, the primes seem to be misplaced, and the symbols t', t''', etc. should be defined;
* the \theta_l in the beginning of section 3.3 (formula 13) is completely superfluous.
- The position of the tables and figures is rather weird, making the paper less readable than necessary. The authors should consider moving floating parts around (one could also move figure three to the bottom of a suitable page, for example).
- It is a matter of taste, but since all experimental results except the ones on the copy task are tabulated, one could think of adding a table with the results now contained in figure 3.

Relation to prior work: the authors are aware of most relevant work. 

On p2 they write: "Many other approaches have also been proposed to capture long-term dependencies." There is one that seems close to what the authors do: 

J. Schmidhuber. Learning complex, extended sequences using the principle of history compression. Neural Computation, 4(2):234-242, 1992

It is related to clockwork RNNs, about which the authors write:

"A recent architecture that is similar in spirit to our work is that of Clockwork RNNs (Koutnik et al., 2014), which split weights and hidden units into partitions, each with a distinct period. When it’s not a partition’s time to tick, its hidden units are passed through unchanged, thus in some ways mimicking the behavior of NARX RNNs. However Clockwork RNNs differ in two key ways. First, Clockwork RNNs sever high-frequency-to-low-frequency paths, thus making it difficult to learn long-term behavior that must be detected at high frequency (for example, learning to depend on quick motions from the past for activity recognition). Second, Clockwork RNNs require hidden units to be partitioned a priori, which in practice is difficult to do in any meaningful way. NARX RNNs suffer from neither of these drawbacks."

The neural history compressor, however, adapts to the frequency of unexpected events, by ticking only when there is an unpredictable event, thus overcoming some of the issues above. Perhaps this trick could further improve the system of the authors, as well as the Clockwork RNNs, at least for certain tasks?

General recommendation: Accept, provided the comments are taken into account.

review_title: The paper introduces a variant of the well-known (but as of today not very frequently used) NARX architecture for Recurrent Neural Networks. It is demonstrated that with the proposed method (MIST RNNs), good performance is achieved on several common RNN problems.

final_score: rating: 3: Clear rejection
review_content: The followings are my main critics of the paper: 
1. Analysis does not provide any new insights. 
2. Similar work (recurrent skip coefficient and the corresponding architecture in [1]) has been done, but has not been mentioned. 
3. The experimental results are not convincing. This includes 1. the choices of tasks are limited -- very small in size, 2. the performance in pMNIST is worse than [1], under the same settings.

Hence I think the novelty of the paper is very little, and the experiments are not convincing.

[1] Architectural Complexity Measures of Recurrent Neural Networks. Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan Salakhutdinov, Yoshua Bengio. NIPS, 2016. 
review_title: little novelty and unconvincing

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The authors introduce a variant of NARX RNNs, which has an additional attention mechanism and a reset mechanism. The attention is only applied on subsets of hidden states, referred as delays. The delays are aggregated into a vector using the attention coefficients as weights, and then this vector is multiplied by the reset gates. 

The model sounds a bit incremental, however, the performance improvements over pMNIST, copy and MobiAct tasks are interesting.

A similar kind of architecture has been already proposed:
[1] Soltani et al. “Higher Order Recurrent Neural Networks”, arXiv 1605.00064

review_title: Review

Meta-review: I think the model itself is not very novel, as pointed by the reviewers and the analysis is not very insightful either. However, the results themselves are interesting and quite good (on the copy task and pMnist, but not so much the other datasets presented (timit etc) where it not clear that long term dependencies would lead to better results). Since the method itself is not very novel, the onus is upon the authors to make a strong case for the merits of the paper --  It would be worth exploring these architectures further to see if there are useful elements for real world tasks -- more so than is demonstrated in the paper --  for example showing it on tasks such as machine translation or language modelling tasks requiring long term propagation of information or even real speech recognition, not just basic TIMIT phone frame classification rate.

As a result, while I think the paper could make for an interesting contribution, in its present form, I have settled on recommending the paper for the workshop track.


As a side note, paper is related to paper 874 in that an attention model is used to look at the past. The difference is in how the past is connected to the current model. 
Decision: Invite to Workshop Track