## 4 Experiments
### Results and Analysis
#### 4.3.1 Quantitative Analysis  
**The Effect of Different CNN Encoders**  
In the famous imageNet competition, ResNet have get great performance due to deep structure. Since action recognition is not the same as image classification, we verify the result of different CNN encoders. To date, there are four widely used CNN encoders, namely GoogLeNet, VGG, ResNet-101 and ResNet-152, to extract visual features. In this sub-experiment, we study the influence of different versions of CNN encoders on our framework. The experiments are conducted on RGB data on the UCF11 and first split of HMDB51 and UCF101 datasets. The results are shown in Table 1. These above networks are all pre-trained on imageNet dataset. We can easily find that, by taking ResNet-152 as the visual decoder, our method perform best with 91.2% on UCF11, 54.4% on HMDB51 and 87.7% on UCF101.  
**The Effect of Every Component**  
The baseline of these methods is a simple \"encode-decode\" model with CNN encoding the input videos and LSTM decoding features into categories. In [48], they compare baseline performance of LSTM, Attention-LSTM (ALSTM), ConvLSTM [49] and ConvALSTM. In Table 2, we list the performance of all these LSTM variants and our main component for action recognition. To conduct a fair comparison, we use VGG net as our encode network and conduct experiment on split 1 of UCF-101 and HMDB51. Our proposed method significantly outperforms the variants on each dataset. From the component analysis, we find that the three LSTM layers would perform worse than two LSTM layers, thus we use two LSTM layers as our architecture. Moreover, attention mechanism greatly improves the performance, and \"attention-again\" model gains the result of recognizing actions.  
Furthermore, in decode network, we only use the output of the last LSTM unit. To verify the performance of this small change, we compare with conventional methods which make softmax operation among the outputs of every LSTM unit. Then, we conduct experiments on UCF11 and use the same encode network (VGG). The results are shown in Figure 3. The loss is shown in Figure 3a, while Figure 3b represents accuracy. The red and blue lines in Figure 3  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **UCF11** & **HMDB51** & **UCF101** \\ \hline GoogLeNet & 89.7 & 52.3 & 85.4 \\ VGG & 90.1 & 52.6 & 85.8 \\ ResNet-101 & 90.9 & 53.8 & 87.2 \\ ResNet-152 & **91.2** & **54.4** & **87.7** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Convolutional neural networks (CNNs) encoder analysis on UCF11 and first split of HMDB51 and UCF101. Classification accuracy is used as evaluation measure which represented as recognition accuracies (%). The **bold** text represents the best result.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **HMDB51** & **UCF101** \\ \hline LSTM & 41.3 & 77.5 \\ ALSTM & 40.9 & 77.0 \\ ConvLSTM & 41.8 & 77.6 \\ ConvALSTM & 43.3 & 79.6 \\ ConvLSTM + hierarchical LSTM (Three layers) & 45.2 & 81.7 \\ ConvLSTM + hierarchical LSTM (Two layers) & 46.6 & 82.4 \\ + attention mechanism & 50.9 & 84.1 \\ + “attention-again” model & 52.6 & 85.8 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of LSTM variants and different components on first split of HMDB51 and UCF101. Classification accuracy is used as evaluation measure which represented as recognition accuracies (%).  
softmax method, respectively. From Figure 3, it is easily concluded that our method is more fluent than softmax. More importantly, the accuracy of our method is higher than softmax. Therefore, this structure of decode network can be proven.