Transformer Taking the single-head attention mechanism in Transformer as an example, in the calculation of the attention mechanism, the each linear transformations of  $Query(Q)$ ,  $Key(K)$  and  $Value(V)$  require matrix multiplication operation of size  $d_{\mathrm{model}} \times d_{\mathrm{model}}$ . In the calculation of the attention weights, the  $Q$  and transpose of  $K$  are multiplied together, and since the size of  $Q$  and  $K$  are  $L \times d_{\mathrm{model}}$ , the complexity of computing the attention weights is  $L \times d_{\mathrm{model}} \times L$ , and the size of attention weights is  $L \times L$ . Similarly, when we multiply the attention weights with  $V$  in the computation, the complexity is  $L \times L \times d_{\mathrm{model}}$ . The output from the attention matrix multiplication is further linearly transformed, and thus requires an additional  $d_{\mathrm{model}} \times d_{\mathrm{model}}$  complexity. To summarize, the overall complexity of the single-headed attention computation in Transformer can be expressed as: 1) Linear transformations of  $Q$ ,  $K$  and  $V$ :  $3 \times L \times d_{\mathrm{model}} \times d_{\mathrm{model}}$ . 2) Calculation of attention weights:  $L \times d_{\mathrm{model}} \times L$ . 3) Matrix multiplication of attention weights and  $V$ :  $L \times L \times d_{\mathrm{model}}$ . 4) Linear transformation of the output:  $L \times d_{\mathrm{model}} \times d_{\mathrm{model}}$ . The total complexity of the single-headed attention computation in Transformer is  $3 \times L \times d_{\mathrm{model}} \times d_{\mathrm{model}} + L \times d_{\mathrm{model}} \times L + L \times L \times d_{\mathrm{model}} + L \times d_{\mathrm{model}} \times d_{\mathrm{model}} = 4 \times L \times d_{\mathrm{model}}^2 + 2 \times L^2 \times d_{\mathrm{model}}$ . It should be noted that the complexity improvements in Transformer-based methods only apply to the  $L^2$  regularization term, as the linear transformations are necessary for any Transformer-based model. In addition, assuming the dimension of the Feed-Forward Networks (FFN) in Transformer is  $d_{\mathrm{ff}}$ , the first layer of the fully connected network has a complexity of  $L \times d_{\mathrm{model}} \times d_{\mathrm{ff}}$ , while the second layer has a complexity of  $L \times d_{\mathrm{ff}} \times d_{\mathrm{ff}}$ , and the total complexity of FFN is  $L \times d_{\mathrm{model}} \times d_{\mathrm{ff}} + L \times d_{\mathrm{ff}}^2$ . It should be noted that the input and output transformations in Transformer still require a certain level of complexity. On the hand, as it needs to transform the input data from dimension  $d_{\mathrm{in}}$  to dimension  $d_{\mathrm{model}}$ , the input transformation in Transformer requires at least a  $L \times d_{\mathrm{in}} \times d_{\mathrm{model}}$  complexity. On the other hand, as it needs to transform the input data from dimension  $d_{\mathrm{ff}}$  to dimension  $d_{\mathrm{model}}$ , the output transformation requires a  $L \times d_{\mathrm{ff}} \times d_{\mathrm{model}}$  complexity. Therefore, the overall complexity of Transformer can be summarized as:  $L \times d_{\mathrm{in}} \times d_{\mathrm{model}} + 4 \times L \times d_{\mathrm{model}}^2 + 2 \times L^2 \times d_{\mathrm{model}} + L \times d_{\mathrm{model}} \times d_{\mathrm{ff}} + L \times d_{\mathrm{ff}}^2 + L \times d_{\mathrm{ff}} \times d_{\mathrm{model}}$ . It should be noted that in most cases for Transformer,  $d_{\mathrm{ff}}$  is larger than  $d_{\mathrm{model}}$ .