Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This submission first proposes a new variant of LSTM by introducing a set of independent weights for each time step, to learn longer dependency from the input. Then the submission propose a dependent bidirectional structure by using the output as input to the RNN cell to introduce the dependency of the outputs.

While LSTM do have problem to learn very long term dependency, the model proposed in this paper is very inefficient, the number of parameters are depend on the the length of sequences. Also, there is no analysis about why adding these additional weights could help the model learn better long-term dependency. In the other word, why this approach is better than attention/ self-attention? How to handle very long sequence and also, how to deal with different length? Just ignore the additional weights?

In the second part, the author argued a standard seq2seq model is vulnerable to previous erroneous predictions. But I don't understand why the DBRNN can handle it. It essentially just a multitask learning function: L = L_f + L_b + L_fb where error signal backprop to different layer directly which is not new.

The experimental results are weak. It compare with Seq2Seq model without attention. The other baseline for POS tag is from 1997. 
review_title: Review

final_score: rating: 3: Clear rejection
review_content: The paper proposes a new recurrent cell and a new way to make predictions for sequence tagging. It starts with a theoretical analysis of memory capabilities in different RNN cells and goes on with experiments on POS tagging and dependency parsing. There are serious presentation issues in the paper, which make it hard to understand the ideas and claims.

First, I was not able to understand the message of the theoretical analysis from Section 2 and could not see how it is different from similar derivations (i.e. using a linearized version of an RNN and eigenvalue decomposition) that can be found in many other papers, including (Bengio et al, 1994) and (Pascanu et al, 2013). Novelty aside, the analysis has presentation issues. SRN is introduced without a nonlinearity from the beginning, although normally it should have one. From the classical upper bound with a power of the largest singular value the paper concludes that “Clearly, the memory will explode if \lambda_{max} > 1”, which is not true: the memory *may* explode, having an exponentially growing upper bound does not mean that it *will* explode. The notation chosen from LSTM is  different from the standard in deep learning community and was very hard to understand (Y_t is used instead of h_t, and h_t is used instead of c_t). This notation also does not seem consistent with the rest of the paper, for example Equations 28 and 29 suggest that Y_t are discrete outputs and not vectors. 

The novel cell SLSTM-I is meant to be different from LSTM by addition of “input weight vector c_i”, but is not explained where c_i come from. Are they trainable vectors, one for each time step? If yes, then how could such a cell be applied to sequence which are longer than the training ones?

Equations 28, 29, 30 describe a very unusual kind of a Bidirectional Recurrent Network. To the best of my knowledge it is much more common to make one prediction based on future and past information, whereas the paper describes an approach in which first predictions are made separately based on the past and on the future. It is also very common to use several BiRNN layers, whereas the paper only uses one. As for the proposed DBRNN method, unfortunately, I was not able to understand it.

I also have concerns regarding the experiments. Why is seq2seq without attention is used? On such small datasets attention is likely to make a big difference. What’s the point of reporting results of an LSTM without output nonlinearity (Table 5)?

To sum up, the paper needs a lot work on many fronts, but most importantly, presentation should be improved.

review_title: serious presentation issues

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper introduces a different form of Memory cell for RNN which has more capabilities of long-term memorizing. Furthermore, it presents and efficient architecture for sequence-to-sequence mapping.

While the claim of the paper sounds very ambitious and good, the paper has several flaws. First of all, the mathematical analysis is a bit problematic. Of course, it is known that Simple Recurrent Networks (SRN) have a vanishing gradient problem. However, the way you proof it is not correct, as you ignore the application of f() for calculating the output (which is routed to the input) and you use an upper bound to show a general behaviour.
The analysis of the Memory capabilities of LSTM is a bit simplified, however, it is okay. Note, that various experiments by Schmidhuber's group, as well as Otte & al have shown that LSTM can generalize and memorize to sequences of more than a million time steps, if the learning rates is small enough.

The extended memory which the authors call SLSTM-I has similar memory capabilities as LSTM. The other one (SLSTM-II) looses the capability of forgetting as it seems. An analysis would be crucial in this paper to show the benefits mathematically. 

The authors should have a look at "Evolving memory cell structures for sequence learning" by Justin Bayer, Daan Wierstra, Julian Togelius and J¨urgen Schmidhuber, published in 2009. Note that the SLSTM belongs to the family of networks which could be generated by that paper as well.

Also "Neural Architecture Search with Reinforcement Learning" by Barret Zoph and Quoc V. Le would be interesting.

In your experiments it would be fair to compare to Cheng et al. 2016

I suggest the authors being more modest with the name of the memory cell as well as with the abstract (especially in the POS experiment, SLSTM is not superior)
review_title: Generally a nice approach, but the paper has several issues, especially, recent SotA should be taken into account

Meta-review: The reviewers of the paper are not very enthusiastic of the new model proposed, nor are they very happy with the experiments presented.   It is unclear from both the POS tagging and dependency parsing results where they stand with respect to state of the art methods that do not use RNNs.  We understand that the idea is to compare various RNN architectures, but it is surprising that the authors do not show any comparisons with other methods in the literature.  The idea of truncating sequences beyond a certain length is also a really strange choice.  Addressing the concerns of the reviewers will lead to a much stronger paper in the future.
Decision: Reject