Before revision:  Perhaps most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92 \% of the accuracy of BERT  on the GLUE benchmark, but pre-trains and runs up to seven times faster on GPUs and twice as fast on TPUs.
After revision:  Most surprisingly, we find that replacing the self-attention sublayer in a Transformer encoder with a standard, unparameterized Fourier Transform achieves 92 \% of the accuracy of BERT on the GLUE benchmark, but pre-trains and runs up to seven times faster on GPUs and twice as fast on TPUs.