# Video Description Model Based on Temporal-Spatial and Channel Multi-Attention Mechanisms
## 3 Multi-Attention Video Description Model
### Attention Calculation
#### 3.2.3 Temporal Attention  
After the feature processing of the spatial attention and channel attention, the video feature is \(X_{att-c}^{(t)}=\{v_{1-c},v_{2-c}\ldots v_{n-c}\}\) with a dimension of \(K*K*C\). Then, we adopt a pooling operation on it. Finally, we get \(V^{\prime}=\big{\{}v_{1}^{\prime},\ v_{2}^{\prime},\ldots,v_{n}^{\prime}\big{\}}\) with a dimension of \(C\). After obtaining the features of the video sequence, we can weigh the features by temporal attention to obtain the whole features expression of the whole video. Suppose \(\gamma^{(t)}=\Big{\{}\gamma_{1}^{(t)},\ \gamma_{2}^{(t)},\ldots,\gamma_{n}^{(t)}\Big{\}}\) and \(\gamma_{i}^{(t)}\) is the weight of the video feature of the frame \(i\) when the model predicts words at time \(t\), which satisfies \(\sum_{i=1}^{n}\gamma_{i}^{(t)}=1\). \(\gamma_{i}^{(t)}\) can be computed from the output of the previous time \(h_{t-1}\) and the video feature \(V\):  
\[e_{i}^{t}=w_{att-t}^{T}\tan\ln\Big{(}W_{att-t}h_{t-1}+U_{att-t}v_{i}^{\prime}+ b_{att-t}\Big{)} \tag{17}\]  
\[\gamma_{i}^{(t)}=\exp\Big{\{}e_{i}^{(t)}\Big{\}}/\sum_{s=1}^{n}\exp\Big{\{}e_ {g}^{(t)}\Big{\}} \tag{18}\]  
Then, for the \(i\) frame, the result is:  
\[v_{i-t}=v_{i-c}*\gamma_{i}^{(t)} \tag{19}\]  
\[X_{att-t}^{(t)}=f_{att-t}\Big{(}X_{att-c^{\prime}}^{(t)}\gamma^{(t)}\Big{)} \tag{20}\]  
\(f_{att-t}\) means the operation of temporal attention for all frames as in Equations (17)-(19) and we obtain the video features after the temporal attention operation: \(X_{att-t}^{(t)}=\{v_{1-t},v_{2-t}\ldots v_{n-t}\}\).  
After the operations in spatial, channel, and temporal domain, we get the feature \(X_{att-t}^{(t)}\). After weighting the \(X_{att-t^{\prime}}^{(t)}\), the feature \(z\) that needs to be sent to the decoding network is obtained, where the pool indicates the space pooling operations:  
\[z=\frac{1}{n}\sum_{i}^{n}\text{pool}(v_{i-t}) \tag{21}\]  
Furthermore, the dimensions of \(X_{att-t^{\prime}}^{(t)}\), \(X_{att-c^{\prime}}^{(t)}\), \(X_{att-s}^{(t)}\), and \(V\) are the same. \(X_{att-t^{\prime}}^{(t)}\), \(X_{att-c^{\prime}}^{(t)}\) and \(X_{att-s}^{(t)}\) represent the processing results of temporal, channel, and spatial attention to \(V\), respectively. Here, we calculate the spatial (\(S\)) first and then the channel (\(C\)), and finally the temporal (\(T\)). Obviously, the logical order of the calculations can be replaced; that is, the calculation order of the formula 12/16/20 can be changed and the corresponding video features can be used. Equation (3) expresses the calculation of the entire attention feature. Here we mainly calculate the spatial (\(S\)), then calculate the channel (\(C\)), and finally calculate the temporal (\(T\)), which is the S-C-T model. Similarly, in order to calculate the attention characteristics of the T-S-C model, we just need to calculate the above three characteristics of \(X_{att-t^{\prime}}^{(t)}\), \(X_{att-c^{\prime}}^{(t)}\), and \(X_{att-s}^{(t)}\) according to the corresponding calculation order. For exploring the influence of the different logical orders of attention operation on the model, we propose eight models of T-S-C, T-C-S, S-T-C, S-C-T, C-T-S, C-S-T, ST-C, and C-ST. The first six models are combinations of different logical orders of the three attention mechanisms. In the video features, the temporal-spatial features are usually regarded as a holistic feature, so the ST-C and C-ST models represent the temporal-spatial features as a whole for modeling, and the calculation principle is the same with the first six models. We will analyze the results of these models in the experiment.