## 2 Related Works
### Action Recognition Methods  
In recent years, deep learning received considerable attention in computer vision applications. Many deep learning-based approaches have been proposed to tackle the problem of human action recognition. In the following we present some works on action recognition mostly based on RGB inputs and exploiting different deep learning strategies. Table 1 summarizes the performance of some of the relevant methods in the state-of-the-art. Results are reported for the most common datasets. The works in the literature are mainly based on one of the following three deep learning strategies: fusing different pieces of information about the video stream (i.e., two-stream networks [20]); including spatio-temporal structure (i.e., 3D convolutional networks [21]); including temporal analysis of video contents (i.e., long short-term memory networks [22]). In two-stream networks, the spatial RGB information, is usually combined with temporal information in the form of motion vectors or optical flows. The two sources of information are used for training two separate networks and the outputs are fused in late layers. In [20], a spatial network is trained on single RGB frames while the temporal network is trained on a stack of optical flow frames. The two networks perform classification and the fusion is applied to the class scores using a Support Vector Machine (SVM). In [38], computed optical flows are substituted with motion vectors that are readily available in video streams, thus improving the efficiency of the two-stream networks making the approach usable in real-time applications. In [29], instead of performing late fusion, the two streams are fused in middle layers using convolution and pooling layers. Long-range temporal structure modeling and warped flows are exploited in [30] in a temporal segment network (TSN) improving the original two-stream network results. Fusing multiple information is also exploited in [42] where a multitask deep architecture is used to perform 2D and 3D pose estimation jointly with action recognition. The model first predicts location of body joints and then, using this information, it predicts the action performed in the video. The joint pose/action learning and recognition is shown to be more robust than using the information separately. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Method** & **Year** & \begin{tabular}{c} **HMDB-51** \\ [23] \\ \end{tabular} & \begin{tabular}{c} **UCF-101** \\ [24] \\ \end{tabular} & \begin{tabular}{c} **Kinetics** \\ [25] \\ \end{tabular} & \
\begin{tabular}{c} **Charades** \\ \end{tabular} &
\begin{tabular}{c} **NTU** \\ [27] \\ \end{tabular} \\ \hline Two streams (RGB+OF) [20] & 2014 & 59.4\% & 88.0\% & & & \\ C3D+Linear SVM [21] & 2015 & & 85.2\% & & & \\ LSTM30+OF+RGB [28] & 2015 & & 88.6\% & & & \\ S:VGG-16, T:VGG-16 [29] & 2016 & 65.4\% & 92.5\% & & & \\ TSN (3 modalities) [30] & 2016 & 69.4\% & 94.2\% & & & \\ ST-LSTM+Trust Gate [31] & 2016 & & & & 69.2\% \\ LTC [32] & 2017 & 67.2\% & 92.7\% & & & \\ I3D [33] & 2017 & 80.9\% & 98.0\% & 74.2\% & & \\ T3D(+TSN) [34] & 2017 & 63.5\% & 93.2\% & 71.5\% & & \\ P3D ResNet [35] & 2017 & & 88.6\% & & & \\ L2 STM [36] & 2017 & 66.2\% & 93.6\% & & & \\ STA-LSTM [37] & 2018 & & & & 73.4\% \\ DTMV+RGB-CNN [38] & 2018 & 55.3\% & 87.5\% & & & \\ R(2+1)D-Two [39] & 2018 & 78.7\% & 97.3\% & 75.4\% & & \\ NL I3D [40] & 2018 & & & 77.7\% & 39.5\% & \\ VideoLSTM [41] & 2018 & 56.4\% & 88.9\% & & & \\ DeepHAR (RGB only) [42] & 2018 & & & & 84.6\% \\ R(2+1)D-152 [43] & 2019 & & & & 81.3\% & \\ P3D+13D [44] & 2019 & 82.1\% & & & 41.0\% & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: List of major action recognition methods and their performance on used datasets. Temporal information can be also incorporated into the network architectures by considering stack of frames and 3D convolutions. The work introducing this rationale is [21]. It is shown that an architecture with small \(3\times 3\times 3\) convolution kernels in all layers can improve recognition performance.