# Video Captioning Based on Channel Soft Attention and Semantic Reconstruct
## 3 Method
### VLAD and SeqVLAD  
The VLAD can be seen as a kind of simplified Fisher Vector. The Fisher vector encoding uses GMM (Gaussian Mixture Model) to construct visual descriptors. In addition, the VLAD does not store second-order information about the features and typically use K-Means instead of GMMs to generate descriptors. The VLAD calculates the differences between the local descriptor and its nearest cluster center, then aggregates the local descriptors into a fixed dimension vector. Suppose we employ K-means to generate \(K\) centers \(C=\{c_{1},c_{2},\,\ldots,\,c_{K}\}\). In addition, the local descriptors with dimension \(D\) are \(X=\{x_{1},x_{2},\,\ldots,\,x_{N}\}\), the VLAD encoding can be expressed by the following formulation:  
\[\
u_{k}=\sum_{n=1}^{N}a_{k}(x_{n})(x_{n}-c_{k}) \tag{1}\]  
where \(N\) is the number of descriptors, \(x_{n}\) is the \(n\)-th descriptor, \(c_{k}\) is the \(k\)-th cluster center, and the assignment \(a_{k}(x_{n})\) represents the relationship between current descriptor with \(k\)-th cluster center. If \(c_{k}\) is the nearest cluster center to current descriptor, \(a_{k}(x_{n})=1\), and otherwise, \(a_{k}(x_{n})=0\). VLAD calculates \(\
u_{k}\) over all cluster centers as \(V=\{\
u_{1},\
u_{2},\,\ldots,\,\
u_{k}\}\). With the dimension \(D\) of each center, we can compute the final representation with dimension  
Figure 1: The overall structure of the model based on channel soft attention and semantic reconstructor. \(K\times D\). The VLAD performs well in image retrieval. However, because assignment \(a_{k}(x_{n})\) is discontinuous and non-differentiable, how to train VLAD in network is a difficult problem. The purpose of SeqVLAD is to plug VLAD into a deep neural network to aggregate the video representation locally and temporally. SeqVLAD tries to leverage the benefits of both convolutional units and the VLAD encoding method in RNNs architecture, using parameter shared gated recurrent convolutional units (SGRU-RCN) as the assignment of VLAD encoding. Recurrent convolutional networks (RCNs) successfully combine CNNs and RNNs which could obtain the video features in both spatial and temporal for video processing tasks. It has been shown that RNN focuses more on global feature changes, discards the fine motion information between successive video frames. To get more detailed information, the recurrent unit named gated recurrent convolutional units (GRU-RCN) which replace the fully connected operation with convolution operation, inputs a 3-dimensional feature map and outputs learned feature map with the same dimension. To reduce the number of parameters and the chance of overfitting, SGRU-RCN shared the input-to-hidden convolutional kernels. An SGRU-RCN unit for VLAD encoding is defined as follows:  
\[q_{t}=W_{q}*x_{t}, \tag{2}\]  
\[z_{t}=\sigma(q_{t}+U_{z}*a_{t-1}), \tag{3}\]  
\[r_{t}=\sigma(q_{t}+U_{r}*a_{t-1}), \tag{4}\]  
\[\tilde{a_{t}}=tanh(q_{t}+U*(r_{\odot}a_{t-1})), \tag{5}\]  
\[a_{t}=(1-z_{t})a_{t-1}+z_{t}\tilde{a_{t}} \tag{6}\]  
where \(*\) denotes a convolution operation, \(\sigma\) is a _sigmoid_ function, and \(\odot\) is an element-wise multiplication. The input \(x_{t}\) at time step \(t\) is a 3-dimensional feature map with shape \(C\times H\times W\), where \(C\), \(H\) and \(W\) denote the number of channels, height, and width of the feature maps, respectively. The shape of \(W_{q}\) is \(K\times C\times 1\times 1\). \(W_{q}\) is the shared convolutional kernel for \(z_{t}\), \(r_{t}\), and \(\tilde{a_{t}}\), where \(z_{t}\) is an update gate indicating whether previous information needs to be updated, and \(r_{t}\) is a reset gate indicating whether previous information needs to be reset. Parameters \(U_{Z}\), \(U_{r}\), \(U\) are 2-dimensional convolutional kernels. The output of SGRU-RCN \(a_{t}\) with shape \(K\times H\times W\) can be taken as the weights of spatial and temporal assignment for VLAD encoding.