# Video Captioning Based on Channel Soft Attention and Semantic Reconstruct
## 3 Method
### Channel Soff Attention  
The input of SeqVLAD at each time step is a 3-dimensional feature map, which is the output of effective models pretrained on ImageNet [19]. If SeqVLAD receives the feature maps of one frame at each time step, it would lose the ability to exploit the global temporal structure of the video. The features and information in the later frames of a video are associated with the previous frames. Inspired by soft attention applied in natural-language processing successfully, we propose a channel soft attention to exploit the temporal structure for 3-dimensional image feature maps. As shown in Figure 2, in a sequence of video features, the shape of feature maps at each time step is \(C\times H\times W\). Each block in Figure 2 with different colors denotes one feature map with shape \(H\times W\). The feature map at \(i\)-th channel at each time step is generated by the same convolution kernel. Our approach is to exploit the temporal structure for each channel. Instead of taking each feature map of one frame at each time step, we take the dynamic weighted sum of the temporal feature maps for each channel. The input of SeqVLAD at each time step is defined as:  
\[\varphi_{t}(V_{c})=\sum_{i=1}^{N}A^{t}_{c,i}v_{c,i} \tag{9}\]  
where \(\sum_{i=1}^{N}A^{t}_{c,i}=1\) and \(A^{t}_{c,i}\) is computed at each time step \(t\) inside the SGRU-RCN encoder, it denotes the attention weight of \(i\)-th frame's channel \(c\) at time step \(t\). The value of attention weight reflects the relevance of the \(i\)-th temporal feature of each channel in the input video given all the previously generated assignments. \(v_{c,i}\) denotes channel \(c\)'s feature map of \(i\)-th frame. Concatenating all the \(\varphi_{t}(V_{c})\) can get the feature maps with shape \(C\times H\times W\), which is the same as the original shape of feature maps of one frame. We obtain the attention weight \(A^{t}_{c,i}\) by normalizing the relevance scores \(e^{t}_{c,i}\):  
\[A^{t}_{c,i}=\frac{e^{t^{c}_{c,i}}}{\sum_{j=1}^{N}e^{t^{c}_{c,i}}} \tag{10}\]  
We design a function that takes the previous assignment \(a_{t-1}\) of the SGRU-RCN as input and summarizes all the previously generated assignments and the feature maps of the \(i\)-th temporal feature. The function gives the unnormalized relevance score \(e^{t}_{i}\):  
\[e^{t}_{i}=CA(relu(W_{a}*a_{t-1}+U_{a}*v+b_{a})), \tag{11}\]  
\[CA(f)=\tanh{(MLP(MaxPool(f))+MLP(AvgPool(f)))} \tag{12}\]  
where \(W_{a}\), \(U_{a}\) are the convolutional kernels and \(b_{a}\) are the parameters, those are trained together in the encoder-decoder networks, \(v\) is the input feature maps of the entire video sequence. The relevance score \(e^{t}_{i}\) with shape \(N\times C\) involves the score of each channel at  
Figure 2: An illustration of the weighted feature maps for each channel at each time step.  
each time step, and \(CA(\cdot)\) is a channel attention module. To compute the 2-dimensional relevance score with shape \(N\times C\times 1\times 1\), we use the channel attention module in CBAM [7], and make a small change as follows. Consider that \(sigmoid\) ranges from 0 to 1 and \(tanh\) ranges from \(-1\) to 1. To expand the differentiation of each weight in a controllable range, the activate function \(sigmoid\) is replaced by \(tanh\). \(e_{i}^{t}\) with \(C\) channels is the relevance score of \(i\)-th frame at \(t\)-th time step. It can be viewed as \(e_{c,i}^{t}\) to represent the score for each channel.