SlowGuess/NAACL_2022_OCR huggingface dataset:
Interestingly, the gap between BERT and FNet shrinks to just  $3\%$  for Large models; this is likely due to FNet-Large being more stable during training than BERT-Large. The Linear-Large model severely underperforms its Base counterpart on GLUE benchmark due to training instabilities. We generally found that the Linear model and BERT were less stable than the models with no parameters in their mixing sublayers, namely the FNet, Random and FF-only models.
The speed vs MLM accuracy curve for GPU (8 V100 chips) pre-training is shown in Figure 2 (see Appendix A.2 for TPU results). Both TPU and GPU models are trained for 1 million steps as in
<sup>9</sup>Devlin et al. (2019) obtain a roughly 2.5 average point boost on the Test split going from BERT-Base to BERT-Large. We only see a roughly 1.5 boost on the Validation split, which may be due to reduced headroom.
Figure 2: Speed-accuracy trade-offs for GPU pre-training. The dashed line shows the Pareto efficiency frontier, indicating the best trade-offs. For smaller models (faster training speeds; left-hand side of figure), the FNet (yellow squares) and Linear (red triangles) models define the frontier, while for larger models (slower training speeds; right-hand side of figure), BERT (blue circles) and FNet-Hybrid (green stars) define the frontier.
Devlin et al. (2019). Motivated by the models considered in Turc et al. (2019), we evaluated several model sizes; see Table 6 in Appendix A.1. We found that the smaller model architectures benefited from larger learning rates, so we select the best result using  $10^{-3}$  and  $10^{-4}$  for all models. $^{10}$
The GPU (Figure 2), and TPU (Figure 3 in Appendix A.2) results display the same trends. For larger, slower models, BERT and FNet-Hybrid define the Pareto speed-accuracy efficiency frontier. For smaller, faster models, FNet and the Linear model define the efficiency frontier.
4.2 Long-Range Arena (LRA) benchmark
Of the efficient Transformers evaluated on LRA benchmark by Tay et al. (2021a), their results suggest that (1) the vanilla Transformer is (by a small margin) the second most accurate model, and (2) the Performer (Choromanski et al., 2021) is the fastest model. We benchmark FNet's accuracy against both of these models using Tay et al. (2021a)'s codebase and running on the same hardware  $(4\times 4$  TPU v3 chips); the results are shown in Table 4a. To ensure a fair comparison, we also report the results of our own experiments for the
vanilla Transformer (see Appendix A.6 for details).
Table 4a suggests that, in aggregate, the (vanilla) Transformer and FNet obtain comparable results. Given that the Transformer is the second most accurate model evaluated by Tay et al. (2021a) and that the relative differences in the average accuracy scores within Table 4a are small, our results suggest that FNet is competitive with the most accurate of the efficient Transformers on LRA.
Turning to efficiency, in Table 4b, we provide training speed and memory usage statistics from our experiments on GPUs (8 V100 chips); see Appendix A.2 for results on TPUs. We perform a sweep over sequence lengths \{512, 1024, 2048, 4096, 8192\}. On GPUs, FNet is much faster than all other models across all sequence lengths, due to the highly efficient FFT implementation on GPUs. Table 4b also indicates that FNet has a lighter memory footprint (this holds for both GPUs and TPUs; see extended results in Appendix A.2). This is partly because FNet has no learnable parameters in its mixing sublayer, but also due to the FFT's efficiency, especially at longer sequence lengths. Lastly, Table 4c shows that training speed gains generally carry over to inference gains (see Appendix A.2 for detailed TPU results).
In this work, we studied simplified token mixing modules for Transformer-like encoder architectures, making several contributions. First, we showed that simple, linear mixing transformations,
<sup>10</sup>We have opted to compare FNet with Transformer models as the latter are the most commonly used models in NLP transfer learning settings. It would also be interesting to compare FNet with convolutional-based models, although, to our knowledge, such models have only recently found limited success in pre-training NLP setups (Tay et al., 2021b); and even there, the authors did not consider the small model regime.
11The "Linear" model in Table 4 is the baseline model introduced in Section 4.1.
Table 4: Accuracy, inference speed and memory usage results on the Long-Range Arena (LRA) benchmark.
Model	ListOps	Text	Retrieval	Image	Pathfinder	Path-X	Avg.
Transformer (ours)	36.06	61.54	59.67	41.51	80.38	OOM	55.83
Linear (ours)	33.75	53.35	58.95	41.04	83.69	FAIL	54.16
FNet (ours)	35.33	65.11	59.61	38.67	77.80	FAIL	55.30
Transformer (*)	36.37	64.27	57.46	42.44	71.40	FAIL	54.39
Local Attention (*)	15.82	52.98	53.39	41.46	66.63	FAIL	46.06
Sparse Trans. (*)	17.07	63.58	59.59	44.24	71.71	FAIL	51.24
Longformer (*)	35.63	62.85	56.89	42.22	69.71	FAIL	53.46
Linformer (*)	35.70	53.94	52.27	38.56	76.34	FAIL	51.36
Reformer (*)	37.27	56.10	53.40	38.07	68.50	FAIL	50.67
Sinkhorn Trans. (*)	33.67	61.20	53.83	41.23	67.45	FAIL	51.39
Synthesizer (*)	36.99	61.68	64.67	41.61	69.45	FAIL	52.88
BigBird (*)	36.05	64.02	59.29	40.83	74.87	FAIL	55.01
Linear Trans. (*)	16.13	65.90	53.09	42.34	75.30	FAIL	50.55
Performer (*)	18.01	65.40	53.82	42.77	77.05	FAIL	51.41
(a) Accuracy results obtained on TPUs as in Tay et al. (2021a). Asterisked results quoted from Tay et al. (2021a). Average does not include the Path-X task, which all models fail (Transformer due to memory limits; others perform no better than chance).
Seq. length	Training Speed (steps/s)	Peak Memory Usage (GB)
512	1024	2048	4096	8192	512	1024	2048	4096	8192
Transformer	21	10	4	OOM	OOM	1.6	4.0	12.2	OOM	OOM
Linear	34 (1.6x)	19 (1.8x)	9 (2.0x)	4	OOM	0.9	1.6	2.8	6.9	OOM
FNet (FFT)	43 (2.0x)	24 (2.3x)	14 (3.2x)	7	4	0.8	1.3	2.2	3.9	7.4
Performer	28 (1.3x)	15 (1.5x)	9 (1.9x)	4	2	1.1	1.9	3.1	5.5	10.4
(b) GPU training for sequence lengths up to 8192. Only the fastest efficient Transformer, namely Performer, from Tay et al. (2021a) is shown. Left: training speeds (in steps per second; larger is better), with speed-up multipliers relative to the Transformer given in parentheses. Right: peak memory usage (in GB; smaller is better).
Seq. length	512	1024	2048	4096	8192	16384
Transformer	12	28	76	244	OOM	OOM
Linear	9 (1.4x)	14 (2.0x)	30 (2.6x)	72 (3.4x)	208	OOM
FNet (FFT)	8 (1.5x)	12 (2.3x)	23 (3.4x)	43 (5.7x)	83	164
Performer	11 (1.2x)	17 (1.6x)	32 (2.4x)	60 (4.0x)	116	238
(c) GPU inference speeds on the LRA Text classification task (in milliseconds per batch; smaller is better). Only the fastest efficient Transformer, Performer, from Tay et al. (2021a) is shown. Speed up relative to the Transformer is given in parentheses.
along with the nonlinearities in feed-forward layers, can competently model diverse semantic relationships in text. Second, we introduced FNet, a Transformer-like model wherein the self-attention sublayer is replaced by an unparameterized Fourier Transform. FNets achieve 92 and  $97\%$  of their respective BERT-Base and BERT-Large counterparts' accuracy on the GLUE benchmark, but train  $70 - 80\%$  faster on GPUs/TPUs. Third, because of its favorable scaling properties, FNet is very competitive with the "efficient Transformers" evaluated on the Long-Range Arena benchmark, matching the accuracy of the most accurate models while being much faster and lighter on memory.
Our work highlights the potential of linear units
as a drop-in replacement for the attention mechanism in text classification tasks. We found the Fourier Transform to be a particularly efficient and effective mixing mechanism, due to the speed of the FFT. However, we only performed a cursory survey of other linear transformations (see also Appendix A.3), and additional fast alternatives are worth exploring.
Given the speed and accuracy advantages of smaller FNet models relative to Transformers, we suspect that FNet will be effective as a lightweight, distilled student model deployed in resource-constrained settings such as production services or on edge devices. The need for such lightweight serving models is only forecast to grow given the
interest in giant models (Raffel et al., 2020; Brown et al., 2020; Lepikhin et al., 2021). A natural avenue to explore in this regard is knowledge distillation of small FNet models from larger Transformer teacher models, following, for example, Sanh et al. (2019); Jiao et al. (2020); Turc et al. (2019).
Another aspect of interest and worthy of further study is hybrid FNet-attention models. We found that adding only a few self-attention sublayers to FNet offers a simple way to trade speed for accuracy. Specifically, replacing the final two Fourier sublayers with self-attention provided  $97 - 99\%$  of BERT's accuracy with limited speed penalties.
Throughout this work we have restricted our focus to encoders. FNet decoders can be designed by "causally" masking the Vandermonde matrix, but a lower level implementation is required to introduce causal masking to FFTs. How to adapt Fourier mixing for encoder-decoder cross-attention is an open question as evidence suggests that cross-attention may be crucial to performance (You et al., 2020). We have focused on tasks which do not require generation so we leave FNet decoders and encoder-decoder setups to future work; although we do remark that the FNet encoder could be used as a drop in replacement in a Transformer as other works have successfully demonstrated; see, for example, (Zaheer et al., 2020; Guo et al., 2021).
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Va-clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. Etc: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 268-284.
Andrew R Barron. 1993. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930-945.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish,
Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Yu Cheng, Felix X. Yu, Rogerio S. Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. 2015. An exploration of parameter redundancy in deep networks with circulant projections. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2857-2865.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509.
Kamran Chitsaz, Mohsen Hajabdollahi, Nader Karimi, Shadrokh Samavi, and Shahram Shirani. 2020. Acceleration of convolutional neural network using fft-based split convolutions. arXiv preprint arXiv:2003.12621.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamas Sarlos, David Belanger, Lucy Colwell, and Adrian Weller. 2020. Masked language modeling for proteins via linearly scalable long-context transformers. arXiv preprint arXiv:2006.03555.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. 2021. Rethinking attention with performers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
James W Cooley and John W Tukey. 1965. An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 19(90):297-301.
George Cybenko. 1989. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Hazem M El-Bakry and Qiangfu Zhao. 2004. Fast object/face detection using neural networks and fast fourier transform. International Journal of Signal Processing, 1(3):182-187.
Matteo Frigo and Steven G Johnson. 2005. The design and implementation of fftw3. Proceedings of the IEEE, 93(2):216-231.
Kfir Goldberg, Stav Shapiro, Elad Richardson, and Shai Avidan. 2020. Rethinking fun: Frequency-domain utilization networks. arXiv preprint arXiv:2012.03357.
Himanshu Gothwal, Silky Kedawat, Rajesh Kumar, et al. 2011. Cardiac arrhythmias detection in an ecg beat signal using fast fourier transform and artificial neural network. Journal of Biomedical Science and Engineering, 4(04):289.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontonon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2021. Longt5: Efficient text-to-text transformer for long sequences. arXiv preprint arXiv:2112.07916.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
Tyler Highlander and Andres Rodriguez. 2015. Very efficient training of convolutional neural networks using fast fourier transform and overlap-and-add. In Proceedings of the British Machine Vision Conference 2015, BMVC 2015, Swansea, UK, September 7-10, 2015, pages 160.1-160.9. BMVA Press.
Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. 2020. Data movement is all you need: A case study of transformer networks. arXiv preprint arXiv:2007.00072.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174. Association for Computational Linguistics.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156-5165. PMLR.
Young Jin Kim and Hany Hassan. 2020. FastFormers: Highly efficient transformer models for natural language understanding. In Proceedings of SustainNLP: Workshop on Simple and Efficient Natural Language Processing, pages 149-158. Association for Computational Linguistics.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Renée Koplon and Eduardo D Sontag. 1997. Using fourier-neural recurrent networks to fit sequential input/output data. Neurocomputing, 15(3-4):225-248.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Henry O. Kunz. 1979. On the equivalence between one-dimensional discrete walsh-hadamard and multidimensional discrete fourier transforms. IEEE Computer Architecture Letters, 28(03):267-268.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Az-izzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew M. Stuart, and Anima Anandkumar. 2021. Fourier neural operator for parametric partial differential equations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Sheng Lin, Ning Liu, Mahdi Nazemi, Hongjia Li, Caiwen Ding, Yanzhi Wang, and Massoud Pedram. 2018. Fft-based deep learning deployment in embedded systems. In 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 1045-1050. IEEE.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Michael Mathieu, Mikael Henaff, and Yann LeCun. 2014. Fast training of convolutional networks through ffts: International conference on learning representations (iclr2014), cbls, april 2014. In 2nd International Conference on Learning Representations, ICLR 2014.
Kei-ichiro Minami, Hiroshi Nakajima, and Takeshi Toyoshima. 1999. Real-time discrimination of ventricular tachyarrhythmia with fourier-transform neural network. IEEE transactions on Biomedical Engineering, 46(2):179-185.
Martina Mironovova and Jiri Bila. 2015. Fast fourier transform for feature extraction and neural network for classification of electrocardiogram signals. In 2015 Fourth International Conference on Future Generation Communication Technology (FGCT), pages 1-6. IEEE.
Marcin Moczulski, Misha Denil, Jeremy Appleyard, and Nando de Freitas. 2016. ACDC: A structured efficient linear layer. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Sharan Narang, Hyung Won Chung, Yi Tay, Liam Fedus, Thibault Fevry, Michael Matena, Karishma Malkan, Noah Fiedel, Noam Shazeer, Zhenzhong Lan, Yanqi Zhou, Wei Li, Nan Ding, Jake Marcus, Adam Roberts, and Colin Raffel. 2021. Do transformer modifications transfer across implementations and applications? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5758-5773. Association for Computational Linguistics.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021. Random feature attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Harry Pratt, Bryan Williams, Frans Coenen, and Yalin Zheng. 2017. Fcnn: Fourier convolutional neural networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 786-798. Springer.
Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. 2020. Blockwise selfattention for long document understanding. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2555-2565. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou,
Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2020. Fixed encoder self-attention patterns in transformer-based machine translation. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 556–568. Association for Computational Linguistics.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David P. Kreil, Michael K. Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. 2021. Hopfield networks is all you need. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150.
Sam Shleifer and Alexander M Rush. 2020. Pretrained summarization distillation. arXiv preprint arXiv:2010.13002.
Vikas Sindhwani, Tara N Sainath, and Sanjiv Kumar. 2015. Structured transforms for small-footprint deep learning. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2, pages 3088-3096.
Alex Tamkin, Dan Jurafsky, and Noah Goodman. 2020. Language through a prism: A spectral approach for multiscale language representations. In Advances in Neural Information Processing Systems, volume 33, pages 5492-5504. Curran Associates, Inc.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. 2020a. Synthesizer: Rethinking self-attention in transformer models. arXiv preprint arXiv:2005.00743.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020b. Sparse sinkhorn attention. In International Conference on Machine Learning, pages 9438-9447. PMLR.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021a.