[33] Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 1994.
[34] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, 2013.
[35] Guy E. Blelloch. Prefix sums and their applications, 1990.
[36] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[37] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference on Machine Learning, 2017.
[38] Michael C Mozer. A focused backpropagation algorithm for temporal pattern recognition. Complex Systems, 3, 1989.
[39] Marco Gori, Yoshua Bengio, and Renato De Mori. BPS: a learning algorithm for capturing the dynamic nature of speech. In International Joint Conference on Neural Networks, 1989.
[40] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9 (8):1735-1780, 1997.
[41] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
[42] Nikita Nangia and Samuel R Bowman. ListOps: A diagnostic dataset for latent tree learning. arXiv preprint arXiv:1804.06028, 2018.
[43] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011.
[44] Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, 2014.
[45] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
[46] Friedemann Zenke and Emre O. Neftci. Brain-inspired learning on neuromorphic substrates. Proceedings of the IEEE, 109(5):935-950, May 2021.
[47] Charlotte Frenkel and Giacomo Indiveri. ReckOn: a 28nm sub-mm2 task-agnostic spiking recurrent neural network processor enabling on-chip learning over second-long timescales. In 2022 IEEE International Solid-State Circuits Conference (ISSCC), volume 65, February 2022.
[48] Yigit Demirag, Charlotte Frenkel, Melika Payvand, and Giacomo Indiveri. Online training of spiking recurrent neural networks with phase-change memory synapses. arXiv preprint arXiv:2108.01804, 2021.
[49] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing machines. arXiv preprint arXiv:1410.5401, 2014.
[50] Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: towards language modeling with state space models. arXiv preprint arXiv:2212.14052, 2023.
[51] Vernon B. Mountcastle. Modality and topographic properties of single neurons of cat's somatic sensory cortex. Journal of Neurophysiology, 20(4):408-434, 1957.
[52] Rodney J. Douglas, Kevan A.C. Martin, and David Whitteridge. A canonical microcircuit for neocortex. Neural Computation, 1(4):480-488, 1989.