Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are RNNs: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML), pp. 5294-5303, 2020. URL https://fleuret.org/papers/katharopoulos-et-al-icml2020.pdf.
Kim, S., Shen, S., Thorsley, D., Gholami, A., Kwon, W., Hassoun, J., and Keutzer, K. Learned token pruning for transformers. In KDD, pp. 784-794. ACM, 2022.
Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. In *ICLR*. OpenReview.net, 2020.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278-2324, 1998.
Li, J., Cotterell, R., and Sachan, M. Differentiable subset pruning of transformer heads. Trans. Assoc. Comput. Linguistics, 9:1442-1459, 2021.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In  $ICLR$  (Poster). OpenReview.net, 2019.
Michel, P., Levy, O., and Neubig, G. Are sixteen heads really better than one? In NeurIPS, pp. 14014-14024, 2019.
Peng, H., Schwartz, R., Li, D., and Smith, N. A. A mixture of h - 1 heads is better than h heads. In ACL, pp. 6566-6577. Association for Computational Linguistics, 2020.
Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., and Kong, L. Random feature attention. In ICLR. OpenReview.net, 2021.
Qin, Z., Sun, W., Deng, H., Li, D., Wei, Y., Lv, B., Yan, J., Kong, L., and Zhong, Y. cosformer: Rethinking softmax in attention. In ICLR. OpenReview.net, 2022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners, 2019.
Raganato, A., Scherrer, Y., and Tiedemann, J. Fixed encoder self-attention patterns in transformer-based machine translation. In EMNLP (Findings), volume EMNLP 2020 of Findings of ACL, pp. 556-568. Association for Computational Linguistics, 2020.
Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. Efficient transformers: A survey. CoRR, abs/2009.06732, 2020. URL https://arxiv.org/abs/2009.06732.
Tay, Y., Bahri, D., Metzler, D., Juan, D., Zhao, Z., and Zheng, C. Synthesizer: Rethinking self-attention for transformer models. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 10183-10192. PMLR, 2021a.
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena: A benchmark for efficient transformers. In ICLR. OpenReview.net, 2021b.
Tillet, P., Kung, H. T., and Cox, D. Triton: An intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, MAPL 2019, pp. 10-19, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367196. doi: 10.1145/3315508.3329973. URL https://doi.org/10.1145/3315508.3329973.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762.
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL (1), pp. 5797-5808. Association for Computational Linguistics, 2019.