# Video Description Model Based on Temporal-Spatial and Channel Multi-Attention Mechanisms
###### Abstract  
Video description plays an important role in the field of intelligent imaging technology. Attention perception mechanisms are extensively applied in video description models based on deep learning. Most existing models use a temporal-spatial attention mechanism to enhance the accuracy of models. Temporal attention mechanisms can obtain the global features of a video, whereas spatial attention mechanisms obtain local features. Nevertheless, because each channel of the convolutional neural network (CNN) feature maps has certain spatial semantic information, it is insufficient to merely divide the CNN features into regions and then apply a spatial attention mechanism. In this paper, we propose a temporal-spatial and channel attention mechanism that enables the model to take advantage of various video features and ensures the consistency of visual features between sentence descriptions to enhance the effect of the model. Meanwhile, in order to prove the effectiveness of the attention mechanism, this paper proposes a video visualization model based on the video description. Experimental results show that, our model has achieved good performance on the Microsoft Video Description (MSVD) dataset and a certain improvement on the Microsoft Research-Video to Text (MSR-VTT) dataset. intelligent imaging technology; deep learning; video description; multi-attention perception mechanism; consistency of visual features; visualization model +
Footnote †: journal: Applied Sciences  
Article  
Video description is widely used in advanced intelligent technology, including smart city, smart transportation and smart home [1; 2; 3; 4; 5]. Video description technology is a part of computer vision and natural language processing, which has attracted much attention in recent years [6; 7; 8; 9]. In 2014, [PERSON] [10; 11] proposed a video description model based on the framework of \"encoding-decoding.\" The encoding method in his model extracted features from a single frame of video by using CNN, then adopted the mean pooling and time series encoding models, respectively. Although the [PERSON] model proposed has been applied successfully in video description, there are also some problems with this model. The first problem is that video features are not utilized effectively. Video features are only used in the first decoding and not used subsequently, thus reducing the ability of video features to predict words when time-series increase [10; 11]. Therefore, the capability of sentence generation decreases. The second problem is the consistency of visual content features and sentence descriptions. In the first problem, application of the temporal attention mechanism increases the utilization of videofeatures. However, this approach does not model the relationship between video features and sentence descriptions [12; 13]. Therefore, it brings about the second problem that is how to ensure the consistency of visual content features and sentence descriptions. For these problems, one solution is to add the video feature each time. However, the video feature consists of multiple images. If we still use the pooling encoding method to send the video feature into the decoding model each time, then the video feature will not be utilized effectively. [PERSON] [12] proposed an image description model based on attention mechanism. The model weighted every region of each image by using the attention mechanism before each word-predicting process, making the feature used in each prediction different. Based on this idea, [PERSON] [13] proposed a video description model based on a temporal attention mechanism. Their model weighted the features of all video frames and summed them whenever making word prediction. Experimental results showed that the video feature was utilized effectively. In this paper, we propose a video description model based on temporal-spatial and channel attention to solve the abovementioned problems. At present, the most effective way to extract image features is the convolutional neural network (CNN) [14; 15].