Before revision:  We invalidate the seemingly plausible hypothesis by which widening is as effective as deepening for self-attention , and show that in fact stacking self-attention layers is so effective that it quickly saturates a capacity of the network width. Specifically, we pinpoint a "depth threshold" that is logarithmic in d_x, the network width: L_{\textrm{th self-attention operation, while for depths over the threshold we show that depth-inefficiency kicks in . Our predictions strongly accord with extensive empirical ablations in Kaplan et al. (2020), accounting for the different behaviors in the two depth-(in)efficiency regimes.
After revision:  We invalidate the seemingly plausible hypothesis by which widening is as effective as deepening for self-attention , and show that in fact stacking self-attention depth and width in tandem an essential ingredient. Our guidelines elucidate the depth-to-width trade-off in self-attention operation, while for depths over the threshold we show that depth-inefficiency kicks in . Our predictions strongly accord with extensive empirical ablations in Kaplan et al. (2020), accounting for the different behaviors in the two depth-(in)efficiency regimes.