[GB22] Ankit Gupta and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. ARXIV.ORG, 2022.
$\left[\mathrm{GDE}^{+}20\right]$  Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1474-1487. Curran Associates, Inc., 2020.
[GGGR22] Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. ARXIV.ORG, 2022.
[GGR21] Albert Gu, Karan Goel, and Christopher R'e. Efficiently modeling long sequences with structured state spaces. International Conference On Learning Representations, 2021.
$\left[\mathrm{GLH}^{+}19\right]$  Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, S. Levine, Yoshua Bengio, and B. Scholkopf. Recurrent independent mechanisms. International Conference on Learning Representations, 2019.
$\left[\mathrm{GLL}^{+}22\right]$  Yue Guan, Zhengyi Li, Jingwen Leng, Zhouhan Lin, and Minyi Guo. Transkimmer: Transformer learns to layer-wise skim. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7275-7286, Dublin, Ireland, may 2022. Association for Computational Linguistics.
[Gra16] A. Graves. Adaptive computation time for recurrent neural networks. ARXIV.org, 2016.
[GZYE20] Trevor Gale, M. Zaharia, C. Young, and Erich Elsen. Sparse gpu kernels for deep learning. International Conference For High Performance Computing, Networking, Storage And Analysis, 2020.
[HDLL22] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc V. Le. Transformer quality in linear time. International Conference On Machine Learning, 2022.
$\left[\mathrm{HLW}^{+}22\right]$  Ramin Hasani, Mathias Lechner, Tsun-Hsuan Wang, Makram Chahine, Alexander Amini, and Daniela Rus. Liquid structural state-space models. arXiv preprint arXiv:2209.12951, 2022.
[Hut06] Marcus Hutter. The human knowledge compression contest. http://prize.hutter1.net/, 2006.
$\left[\mathrm{JGB}^{+}21\right]$  Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver: General perception with iterative attention. International Conference On Machine Learning, 2021.
[JGP17] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[JJNH91] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. *Neural Computation*, 3:79–87, 1991.
$\left[\mathrm{KH}^{+}09\right]$  Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[KKL20] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451, 2020.
[LH18] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
$\left[\mathrm{LJH}^{+}19\right]$  Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. International Conference on Learning Representations, 2019.