Venue: ICLR 2018 conference

Reviews:

final_score: rating: null
review_content: Although I welcome this work, I'm afraid that the authors jumped to the conclusion "that the content and output layers are redundant, and that the space of element-wise weighted sums is sufficiently powerful to compete with fully parameterized LSTMs" too quickly.

All comparison against LSTMs (and of course, any other architectures) should be done carefully, especially when there is evidence that LSTMs weren't properly tuned. For instance, Melis et al. (https://arxiv.org/pdf/1707.05589.pdf) point out that LSTMs if tuned properly can outperform most state-of-the-art models. In Melis et al. LSTMs achieve 59.6 on PTB, whereas the authors' LSTMs reach 78.8, which is much much far behind. 

For dependency parsing, I was wondering why the authors didn't do the comparison on Penn Treebank, which is more standard and reported by Dozat & Manning, 2016.
review_title: proper evaluation? 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an analysis of LSTMS showing that they have a from where the memory cell contents at each step is a weighted combination of the “content update” values computed at each time step. The weightings are defined in terms of an exponential decay on each dimension at each time step (given by the forget gate), which lets the cell be computed sequentially in linear time rather than in the exhaustive quadratic time that would apparently be necessary for this definition. Second, the paper offers a simplification of LSTMs that compute the value by which the memory cell at each time step in terms of a deterministic function of the input rather than a function of the input and the current context. This reduced form of the LSTM is shown to perform comparably to “full” LSTMs.

The decomposition of the LSTM in terms of these weights is useful, and suggests new strategies for comparing existing quadratic time attention-based extensions to RNNs. The proposed model variations (which replaces the “content update” that has a recurrent network in terms of context-independent update) and their evaluations seem rather more arbitrary. First, there are two RNNs present in the LSTM- one controls the gates, one controls the content update. You get rid of one, not the other. You can make an argument for why the one that was ablated was “more interesting”, but really this is an obvious empirical question that should be addressed. The second problem of what tasks to evaluate on is a general problem with comparing RNNs. One non-language task (e.g., some RL agent with an LSTM, or learning to execute or something) and one synthetic task (copying or something) might be sensible. Although I don’t think this is the responsibility of this paper (although something that should be considered).

Finally, there are many further simplifications of LSTMs that could have been explored in the literature: coupled input-forget gates (Greff et al, 2015), diagonal matrices for gates, GRUs. When proposing yet another simplification, some sense for how these different reductions is useful, so I would recommend comparison to those.

Notes on clarity:
Before Eq 1 it’s hard to know what the antecedent of “which” is without reading ahead.

For componentwise multiplication, you have been using \circ, but then for the iterated component wise product, \prod is used. To be consistent, notation like \odot and \bigodot might be a bit clearer.

The discussion of dynamic programming: the dynamic program is also only available because the attention pattern is limited in a way that self attention is not. This might be worth mentioning.

When presenting Eq 11, the definition of w_j^t elides a lot of complexity. Indeed, w_j^t is only ever implicitly defined in Eq 8, whereas things like the input and forget gates are defined multiple times in the text. Since w_j^t can be defined iteratively and recursively (as a dynamic program), it’s probably worth writing both out, for expository clarity.

Eq 11 might be clearer if you show that Eq 8 can also be rewritten in the same wheat, provided, you make h_{t-1} an argument to output and content.

Table 4 is unclear. In a language model, the figure looks like it is attending to the word that is being generated, which is clearly not what you want to convey since language models don’t condition on the word they are predicting. Presumably the strong diagonal attention is attending to the previous word when computing the representation to generate the subsequent word? In any case, this figure should be corrected to reflect this. This objection also concerns the right hand figure, and the semantics of the meaning of the upper vs lower triangles should be clarified in the caption (rather than just in the text).
review_title: Useful anaylsis of LSTMs

final_score: rating: 7: Good paper, accept
review_content: Summary: the paper proposes a new insight to LSTM in which the core is an element-wise weighted sum. The paper then argues that LSTM is redundant by keeping only input and forget gates to compute the weights. Experimental results show that the simplified versions work as well as the full LSTM. 


Comment: I kinda like the idea and welcome this line of research. The paper is very well written and has nice visualisation of demonstrating weights. I have only one question:

in the simplified versions, content(x_t) = Wx_t , which works very well (outperforming full LSTM). I was wondering if the problem is from the tanh activation function (eq 2). What if content(x_t) = W_1 . h_{t-1} + W_2 . x_t? 
review_title: review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a simplified LSTM variants by removing the non-linearity of content item and output gate. It shows comparable results with standard LSTM.

I believe this is a updated version of https://arxiv.org/abs/1705.07393 (Recurrent Additive Networks) with stronger experimental results. 

However, the formulation is very similar to "[1] Semi-supervised Question Retrieval with Gated Convolutions" 2016 by Lei, and "Deriving Neural Architectures from Sequence and Graph Kernels" which give theoretical view from string kernel about why this type of networks works. Both of the two paper don't have output gate and non-linearity of "Wx_t" and results on PTB also stronger than this paper. It also have some visualization about how the model decay the weights. Other AnonReviewer also point out some similar work. I won't repeat it here. In the paper, the author argued "we propose and evaluate the minimal changes..." but I think the these type of analysis also been covered by [1], Figure 5. 

On the experimental side, to draw the conclusion, "weighted sum" is enough for LSTM. I think at least Machine Translation and other classification results should be added. I'm not very familiar with SQuAD dataset, but the results seems worse than "Reading Wikipedia to answer open-domain questions" Table 4 which seems use a vanilla LSTM setup. 

Update: the revised version of the paper addresses all my concerns about experiments. So I increased my score. 

review_title: Review

Meta-review: The paper performs an ablation analysis on LSTM, showing that the gating component is the most important. There is little novelty in the analysis, and in its current form, its impact is rather limited.
Decision: Reject