Replacing softmax with ReLU in Vision Transformers
Authors: name: Mitchell Wortsman, name: Jaehoon Lee, name: Justin Gilmer, name: Simon Kornblith

Previous research observed accuracy degradation when replacing the attention
softmax with a point-wise activation such as ReLU. In the context of vision
transformers, we find that this degradation is mitigated when dividing by
sequence length. Our experiments training small to large vision transformers on
ImageNet-21k indicate that ReLU-attention can approach or match the performance
of softmax-attention in terms of scaling behavior as a function of compute.