# Video Description Model Based on Temporal-Spatial and Channel Multi-Attention Mechanisms
## 3 Multi-Attention Video Description Model  
### Network Architecture  
In the previous section, we introduced temporal, spatial, and channel attention mechanisms and their corresponding extraction abilities for different features in a video. To make full use of capabilities of attention mechanisms, we propose a video description model based on the temporal-spatial and channel attention mechanisms in this section. The fundamental network architecture is shown in Figure 2. This model utilizes different attention mechanisms. For example, when predicting the words in need, this model will combine the features effectively in the video by the attention mechanism  
Figure 1: Visualization result of low-level and high-level feature maps. (**a**) Original image; (**b**) low-level feature maps; (**c**) high-level feature maps. automatically, then generate the appropriate corresponding words; thus, it is a solution to the problem related to the consistency of visual content features and sentence descriptions. As shown in Figure 2, the multi-attention model proposed in this paper consists of three parts: a video encoding network, a multi-attention network, and a decoding network. The video encoding network mainly uses the CNN to extract features of video frames. The multi-attention network comprises two parts: an attention network and a feature fusion. Therefore, the attention network can utilize the video features and the previous output to calculate the attention weight of the video feature in time, space, and channel respectively, then recalculate the fused features by the obtained weights and the video features. In this way, we can get more useful features. Finally, the fused features will be exploited by the decoding network for encoding output to obtain more consistent description with video content. By extracting visual features effectively in time, space, and channel, the multi-attention description model proposed in this paper enhances the representation ability of the model. The three attention models we propose are relatively independent. We can combine three models randomly to research the different effects of various combinations. An attention model named S-C-T (spatial-channel-temporal) is shown in Figure 2.  
First, from Video I, we can extract the network features after operating it in CNN:  
\[V=f_{CNN}(I) \tag{1}\]  
\(V=\{v_{1},\ v_{2},\ \ldots,v_{n}\}\) indicates that the video has \(n\) frames with a dimension of \(K*K*C\), and the video feature of every frame is \(v_{i}\in R^{K*K*C}\). At time \(t\), we use the decode unit, LSTM (Long Short-Term Memory), to get the output \(h_{t-1}\) at the previous time. With known values of \(h_{t-1}\) and \(V\), the unknown attention weight factors \((\alpha,\beta,\gamma)\) of time, space, and channel can be calculated by Equation (2):  
\[(\alpha,\beta,\gamma)=\Phi(h_{t-1},V) \tag{2}\]  
\(\Phi\) covers a series of equations in the Section 3.2. We can obtain \((\alpha,\beta,\gamma)\) from \(h_{t-1}\) and \(V\), where \(\alpha\in R^{K*K}\), \(\beta\in R^{C}\), \(\gamma\in R^{N}\). The corresponding formula is presented in detail in the next section. Then, we apply the three weight factors to \(V\) and obtain the features needed to be input into the LSTM:  
\[z=f(V,\alpha,\beta,\gamma) \tag{3}\]  
Figure 2: Video description model based on temporal-spatial and channel attention. where \(f\) represents the output of three attentional operations on \(V\). Section 3.2 is the detailed introduction to Equation (3). Then, we can start to predict the next word:  
\[h_{t}=f_{LSTM}(h_{t-1},z,w_{t-1}) \tag{4}\]  
\[y_{t}\sim p_{t}=\text{softmax}(W_{e}h_{t}+b) \tag{5}\]  
where \(w_{t-1}\) represents the word vector of the corresponding word \(y_{t-1}\) and \(p_{t}\in R^{|D|}\) is the probability distribution of words. In this model, there is a word vector space \(W_{D}\in R^{D\times S}\), where \(D\) is the number of words and \(S\) is the dimension of the word vector.