# Video Captioning Based on Channel Soft Attention and Semantic Reconstruct
## 3 Method
### Semantic Reconstruction Network  
A convolutional GRU can get rich temporal and spatial information of a video. However, if we want to generate semantically accurate sentences, semantic information is needed to enrich the features that get from the SeqVLAD encoder. In our approach, the semantic detection network (SDN) is used for generating the semantic vectors and semantic compositional network (SCN, a variant of LSTM) is took as the decoder. Moreover, a dual learning mechanism is adopted to maintain the information of the semantic vector during training, and the semantic information in forward flow can be better used. For semantic detection, the candidate semantic representation should be obtained first, there are some state-of-art methods such as keyword-based representation [29] and semantic-based representation [30]. Our ground-truth labels are some short sentences, and those sentences of a video might be composed of different words. Moreover, the vocabulary does not contain huge amounts of words, some videos cannot be contained by semantic representation if using semantic represent methods. To avoid the influence of semantic represents methods, we built the semantic representation tags by the manual selection which are \(K\) most common words in the training set. Manual selection is not feasible if the dataset is extremely large, then the semantic representation tags should build by semantic represent methods such as TF-IDF [29]. The task of semantic detection can be seen as a multi-label classification task. For more accurate classification, the fully connected features of the 3D convolutional network and 2D convolutional network are extracted, then concatenate these two features as the input of SDN. Suppose the representation of \(i\)-th video is \(F_{i}\), then, the ground-truth is \(y_{i}=\{y_{i,1},y_{i,2},...,y_{i,K}\}\in\{0,1\}^{K}\). If the \(j\)-th tag exists in the \(i\)-th video's annotations, then \(y_{i,j}=1\), otherwise \(y_{i,j}=0\). A semantic detection network can be implemented by a deep multi-layer perceptron (MLP), which can be defined as:  
\[s_{i}=\sigma(MLP(F_{i})) \tag{13}\]  
where \(\sigma\) is the \(sigmoid\) activation function, and \(s_{i}\) is the semantic vector of \(i\)-th video. The loss function of SDN is defined as:  
\[Loss_{SDN}(s,y)=\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{K}(y_{ij}\log(s_{ij}+(1-y _{ij})\log(1-s_{ij}))) \tag{14}\]  
As for decoder, SCN-LSTM could avoid the problem of long-term dependency, and embeds the semantic attributes of the input video. Suppose there are encoded feature \(x_{t}\) at time step \(t\), the hidden state \(h_{t-1}\) at time step \(t-1\) and semantic feature \(\mathbf{s}\). \[\widetilde{x}_{z,t}=W_{z,a}(W_{z,b}\mathbf{s}\odot W_{z,c}x_{t}),z\in\{c,i,f,o\}, \tag{15}\]  
\[\widetilde{h}_{z,t-1}=U_{z,a}(U_{z,b}\mathbf{s}\odot U_{z,c}h_{t-1}),z\in\{c,i,f,o\} \tag{16}\]  
where \(c\) denotes the cell state, \(i\) denotes the input gate, \(f\) denotes the forget gate, and \(o\) denotes the output gate. \(w\) and \(U\) are learnable weight matrices. The SCN-LSTM can be described as:  
\[i_{t}=\sigma(\widetilde{x}_{i,t}+\widetilde{h}_{i,t-1}+b_{i}), \tag{17}\]  
\[f_{t}=\sigma(\widetilde{x}_{f,t}+\widetilde{h}_{f,t-1}+b_{f}), \tag{18}\]\[o_{t}=\sigma(\widetilde{x}_{o,t}+\widetilde{h}_{o,t-1}+b_{o}), \tag{19}\] \[\widetilde{c_{t}}=\sigma(\widetilde{x}_{c,t}+\widetilde{h}_{c,t-1}+b _{c}),\] (20) \[c_{t}=i_{t}\odot\widetilde{c_{t}}+f_{t}\odot c_{t-1},\] (21) \[h_{t}=o_{t}\odot\tanh{(c_{t})} \tag{22}\]  
where \(\sigma\) demotes the _sigmoid_ function and \(b\) is a bias for each gate. The semantic vector for each video is a 1-dimensional vector with shape 300 (as in paper [11]). It is a smaller vector than video features, so there are fewer parameters calculate with semantic vector, semantic vector might lose its impacts on the result because of a few bad trained parameters.