Before revision:  Here we propose a simple attention-free network architecture, gMLP, based solely on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications.
After revision:  Here we propose a simple attention-free network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications.