Before revision:  In this paper, we theoretically study the interplay between depth and width in self-attention , and shed light on the root of the above phenomenon. We invalidate the seemingly plausible hypothesis by which widening is as effective as deepening for self-attention , and show that in fact stacking self-attention layers is so effective that it quickly saturates a capacity of the network width.
After revision:  In this paper, we theoretically study the interplay between depth and width in self-attention . We conduct systematic empirical ablations on networks of depths 6 to 48 that clearly reveal the theoretically predicted behaviors, and provide explicit quantitative suggestions regarding the optimal depth-to-width allocation for a given self-attention , and show that in fact stacking self-attention layers is so effective that it quickly saturates a capacity of the network width.