Task: Language Identification
Input (Spanish, English): This paper proposes an idea of looking n-steps backward when modelling sequences with RNNs. The proposed RNN does not only use the previous hidden state (t-1) but also looks further back ( (t - k) steps, where k=1,2,3,4 ). The paper also proposes a few different ways to aggregate multiple hidden states from the past.   The reviewer can see few issues with this paper.  Firstly, the writing of this paper requires improvement. The introduction and abstract are wasting too much space just to explain unrelated facts or to describe already well-known things in the literature. Some of the statements written in the paper are misleading. For instance, it explains, “Among various neural network models, recurrent neural networks (RNNs) are appealing for modeling sequential data because they can capture long term dependency in sequential data using a simple mechanism of recurrent feedback” and then it says RNNs cannot actually capture long-term dependencies that well. RNNs are appealing in the first place because they can handle variable length sequences and can model temporal relationships between each symbol in a sequence. The criticism against LSTMs is hard to accept when it says: LSTMs are slow and because of the slowness, they are hard to scale at larger tasks. But we all know that some companies are already using gigantic seq2seq models for their production (LSTMs are used as building blocks in their systems). This indicates that the LSTMs can be practically used in a very large-scale setting.   Secondly, the idea proposed in the paper is incremental and not new to the field. There are other previous works that propose to use direct connections to the previous hidden states [1]. However, the previous works do not use aggregation of multiple number of previous hidden states. Most importantly, the paper fails to deliver a proper analysis on whether its main contribution is actually helpful to improve the problem posed in the paper. The new architecture is said that it handles the long-term dependencies better, however, there is no rigorous proof or intuitive design in the architecture that help us to understand why it should work better. By the design of the architecture, and speaking in very high-level, it seems like the model maybe helpful to mitigate the vanishing gradients issue by a linear factor. It is always a good practice to have at least one page to analyze the empirical findings in the paper.   Thirdly, the baseline models used in this paper are very weak. Their are plenty of other models that are trained and tested on word-level language modelling task using Penn Treebank corpus, but the paper only contains a few of outdated models. I cannot fully agree on the statement “To the best of our knowledge, this is the best performance on PTB under the same training condition”, these days, RNN-based methods usually score below 80 in terms of the test perplexity, which are far lower than 100 achieved in this paper.   [1] Zhang et al., “Architectural Complexity Measures of Recurrent Neural Networks”, NIPS’16
Output (English): en