Before revision:  We invalidate the seemingly plausible hypothesis by which widening is as effective as deepening for self-attention , and show that in fact stacking self-attention layers is so effective that it quickly saturates a capacity of the network width.
After revision:  We invalidate the seemingly plausible hypothesis by which widening is as effective as deepening for self-attention network size. The race towards beyond 1-Trillion parameter language models renders informed guidelines for increasing self-attention layers is so effective that it quickly saturates a capacity of the network width.