# Video Captioning Based on Channel Soft Attention and Semantic Reconstruct
## 4 Experimental Evaluation
### Implementation Details  
Before training, each dataset contains a small number of captions with too few or too many words, which is bad for training. We removed captions where the number of words is more than 30 or less than 4. In addition, build a semantic vocabulary by manual selection of the most common words in the dataset. It contains the keywords in a sentence such as \"man\", \"boy\", \"playing\", \"riding\". The vocabulary is all composed of \"subjects\", \"verbs\" and \"objects\", which can represent the main idea of the video. Our experiments are based on MSVD and MSR-VTT datasets, because of the low complexity of datasets, the size of semantic vocabulary is set to 300 (as in paper [11]).  
For semantic detection, 2D and 3D video features are needed to be extracted for training a semantic detection network (SDN). The image representation we used is the output of 1536-way last \(AvgPool2d\) layer from InceptionResNetV2 [37] pretrained on the ImageNet dataset [19]. In addition, the video representation is extract by a 3D CNN (C3D) [38] which pretrained on the Sports-1M video dataset [39]. The video representation is a 4096-way vector obtained from the \(fc7\) layer of C3D. In addition, uniformly extract 40 frames from each video and take the RGB video frames as input. Video frames are resized to \(112\times 112\) for C3D and resized to \(299\times 299\) for InceptionResNetV2. The input of C3D is video clips with 16 frames (as in paper [39]) and has 8 frames overlap. We train the semantic detector using the procedure described in model architecture. For sequence VLAD encoding, we take the output of _Conv2d_7b_ layer from InceptionResNetV2 which with shape \(1536\times 8\times 8\).  
For model training, there are some well-known word embedding methods, such as GloVe [40], FastText [41] and word2 vec [42]. The end-to-end video caption model proposed in this paper can be learned online, but GloVe is a count-based method that needs overall statistics. FastText uses n-gram and subword to consider the order of words, so it would build a large corpus. The last linear layer in our model which output the vectors with the size of the corpus takes up nearly half of the parameters. Using FastText will double the number of parameters, so we initialize the word embedding vectors with the publicly available word2 vec vectors. In the decoder, the size of word embedding, feature embedding, and the hidden state of semantic LSTM is set to 512. Because of various lengths of sentences, every sentence added <BOS> and <EOS> to indicate the begin of a sentence and the end of a sentence. To avoid overfitting, the dropout value is set to 0.5. The batch size during training is 64. The model is optimized by [PERSON] [43] with learning rate \(2\times 10^{-4}\). We empirically train our model with 50 epochs.  
For testing, we employ a beam search algorithm to generate sentences. Beam search selects \(k\) highest probability words at each time step, then input these \(k\) words into the next time step to generate new \(k\) highest probability words. The beam size is set to 5 in our experiments.