Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work proposes Sparse Attentive Backtracking, an attention-based approach to incorporating long-range dependencies into RNNs. Through time, a “macrostate” of previous hidden states is accumulated. An attention mechanism is used to select the states within the macro-state most relevant to the current timestep. A weighted combination of these previous states is then added to the hidden state as computed in the ordinary way. This construction allows gradients to flow backwards quickly across longer time scales via the macrostate. The proposed architecture is compared against LSTMs trained with both BPTT and truncated BPTT.

Pros:
- Novel combination of recurrent skip connections with attention.
- The paper is overall written clearly and structured well.
  

Cons:
- The proposed algorithm is compared against TBPTT but it is unclear the extent to which it is solving the same computational issues TBPTT is designed to solve.
- Design decisions, particularly regarding the attention computation, are not fully explained.

SAB, like TBPTT, allows for more frequent updates to the parameters. However, unlike TBPTT, activations for previous timesteps (even those far in the past) need to be maintained since gradients could flow backwards to them via the macrostate. Thus SAB seems to have higher memory requirements than TBPTT. The empirical results demonstrate that SAB performs slightly better than TBPTT for most tasks in terms of accuracy/CE, but there is no mention of comparing the memory requirements of each. Results demonstrating also whether SAB trains more quickly than the LSTM baselines would be helpful.

The proposed affine form of attention does not appear to actually represent the salience of a microstate and a given time. The second term of the RHS of equation 1 (w_2^T \hat{h}^{(t)}) is canceled out in the subtraction in equation 2, since this term is constant for all i. Thus the attention weights for a given microstate are constant throughout time, which seems undesirable.

The related work discusses skip connections in the context of convolutional nets, but doesn’t mention previous works incorporating skip connections into RNN architectures, such as [1], [2], or [3].

Overall, the combination of recurrent skip connections and attention appears to be novel, but experimental comparisons to other skip connection RNN architectures are missing and thus it is not clear how this work is positioned relative to previous related work. 

[1] Lin, Tsungnan, et al. "Learning long-term dependencies in NARX recurrent neural networks." IEEE Transactions on Neural Networks 7.6 (1996): 1329-1338.
[2] Koutnik, Jan, et al. "A clockwork rnn." International Conference on Machine Learning. 2014.
[3] Chang, Shiyu, et al. "Dilated recurrent neural networks." Advances in Neural Information Processing Systems. 2017.

EDIT: I have read the updated paper and the author's rebuttal. I am satisfied with the update to the attention weight formulation. Overall, I still feel that the proposed SAB approach represents a change to the model structure via skip connections. Therefore SAB should also be compared against other approaches that use skip connections, and not just BPTT / TBPTT, which operate on the standard LSTM. Thus to me the experiments are still lacking. However, I think the approach is quite interesting and as such I am revising my rating from 4 to 5.
review_title: SAB combines skip connections with attention

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes sparse attentive backtracking, essentially an attention mechanism that performs truncated BPTT around a subset of the selected states.

The early claims regarding biological plausibility seem stretched, at least when applying to this work. The "waiting for life to end to learn" and student study / test analogies were not helpful from an understanding point of view and indeed raised more questions than insight. The latter hippocampal discussion was at least more grounded.

While a strong motivator for this work would be in allowing for higher efficiency on longer BPTT sequences, potentially capturing longer term dependencies, this aspect was not explored to this reviewer's understanding. To ensure clarity, in the character level PTB or Text8 examples, SAB's previous attention was limited to sequences of T = 100 or 180 respectively?
Limiting the truncation to values below the sequence length for the LSTM baselines also appears strange given the standard within the literature is setting sequence length equal to BPTT length. I presume this was done to keep the number of optimizer updates equal?
Another broader question is whether longer term dependencies could be caught at all given the model doesn't feature "exploration" in the reinforcement learning sense, especially for non-trivial longer term dependencies.

When noting the speed of generating a sparsely sourced summary vector (equation 3), it is worth pointing out that weighted summation over vectors in traditional attention is not a limiting factor as it's a very rapid element-wise only operation over already computed states.

For the experiments, I was looking for comparisons to attention over the "LSTM (full BPTT)" window. This experiment would provide an upper bound and an understanding of how much of SAB's improvement may be as a result of simply adding attention to the underlying LSTM models. Even a simpler and fast (cuDNN compatible) attention mechanism such as [a single cuDNN LSTM layer over the input, an attentional mechanism over the results of the first layer (masked to avoid observing timesteps from the future), summed, and then passed into a softmax] would be informative.

Finally, whilst not a deal breaker for introducing new techniques, stronger LSTM baselines help to further underline the efficacy of the technique. For sequential MNIST, a relatively small dataset, previous papers have LSTM models that achieve 98.2% test accuracy (Arjovsky et al, https://arxiv.org/abs/1511.06464) and the IRNN example included as part of the Keras framework achieves 93% out of the box.

Noting similarities to the Transformer architecture and other similar architectures would also be useful. Both are using attention to minimize the length of a gradient's path, though in Transformers it eliminates the RNN entirely. If a Transformer network performed a k=5 convolution or limited RNN run to produce the initial inputs to the Transformer, it would share many similarities to SAB, though without the sparsity.
review_title: SAB Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: re. Introduction, page 2: Briefly explain here how SAB is different from regular Attention?

Good paper. There's not that much discussion of the proposed SAB compared to regular Attention, perhaps that could be expanded. Also, I suggest summarizing the experimental findings in the Conclusion.
review_title: Sparse attention backtracking, an alternative to (T)BPTT

Meta-review: The authors propose to use attention over past time steps to try and solve the gradient flow problem in learning recurrent neural networks. Attention is performed over a subset of past states by a hueristic that boils to selecting best time-steps.

I agree with the authors that they offer a lot of comparisons, but like the reviewers, I am inclined to find the experiments not very convincing of the arguments they are attempting to make.  The model that they propose has similarities to seq2seq in that they use attention to pass more information in the forward pass; in a sense this is a seq2seq model with the same encoder and decoder, and there are parallels to self-attention. The model also has similarities to clockwork RNNs and other skip connection methods.. However, the experiments offered to not tease out these effects. It is unsurprising that a fixed size neural network is unable to do a long copy task perfectly, but an attention model can. What would have been more interesting would have been to explore if other RNN models could have done so. The experiments on pMNIST aren't really compelling as the baselines are far from SOTA (example: https://arxiv.org/pdf/1606.01305.pdf report 0.041 error rate (95.9% test acc) with LSTMs and regularization).  Text8 also shows worse results in full BPTT on LSTM.  If BPTT is consistently better than this method, it defeats the argument that gradient explosion and forgetting over long sequences is a problem for RNNs (one of the motivations offered for this attention model).

Decision: Reject