Before revision:  Specifically, we pinpoint a "depth threshold" that is logarithmic in d_x, the network width: L_{\textrm{th self-attention operation, while for depths over the threshold we show that depth-inefficiency kicks in . Our predictions strongly accord with extensive empirical ablations in Kaplan et al. (2020), accounting for the different behaviors in the two depth-(in)efficiency regimes. By identifying network width as a limiting factor, our analysis indicates that solutions for dramatically increasing the width can facilitate the next leap in self-attention expressivity .
After revision:  Specifically, we pinpoint a "depth threshold" that is logarithmic in d_x, the network width: L_{\textrm{th self-attention networks of sizes up to the scale of GPT3 (which is too deep for its size), and beyond, marking an unprecedented width of 30K as optimal for a 1-Trillion parameter self-attention expressivity .