# Activity Recognition with Combination of Deeply Learned Visual Attention and Pose Estimation
## 4 Experiments
### Experiment Result  
Table 2 shows the experimental results using the Penn Action dataset. Each experiment was performed using different input data. Basically, appearance-based data was used as input, and the accuracy of the final activity recognition was calculated when the pose estimation result was added, the visual attention result was added, and the pose estimation result and the visual attention result were added. When only the pose estimation result  
Figure 10: Activity recognition results using visual attention. Soccer and Golf are paying attention correctly (**Top**). But Jump and Hug are wrong (**Bottom**).  
was added, it was 97.9%, when only the visual attention resultwas added, it was 97.7%, and when both were added, it was 98.9%, which was 0.8% higher than theresult suggested by [PERSON] et al. (2020).  
Table 3 shows the experimental results using the NTU dataset. Since NTU's skeletal data is often noisy, we trained with 10% of the NTU's data and 90% of the pose estimation data using video clips of 20 body joints and frames. As shown in Table 2, the method in this paper improves accuracy by adding RGB frames, pose estimation, and visual attention results. It is 87.7% when only RGB frame and pose estimation are added, 87.5% when only visual attention is added, and 87.9% when both are added, which is 25% better than the method proposed by [PERSON] et al. (2020). Many of the previous methods use the posture provided by Kinect-v2. This pose estimation result is known to be noisy. As shown in Table 3, the method of this paper improves accuracy by 12.3% compared to the method proposed by [PERSON] et al. (2020) using Kinect when pose estimation is added. This means that the performance is better when visual attention is used than when visual attention is not used. And the performance is better when the pose estimation is added than when it is not added. In addition, there may be limitations with only one method of visual attention, pose estimation, and appearance-based activity recognition, but it can be said that the accuracy was improved by complementing the limitations.  
Table 4 shows the results of experiments on the NW-UCLA dataset. First, the accuracy improved to 79.9% when only the extension and LSTM were used, and 83.4% when only the extension and LSTM were used. In addition, when VNect is used, it is 87.2%, which is improved by 3.8%, and when the method proposed in this paper is used, it shows the highest performance at 88.6%. Figure 11 shows the results of loss function value during training.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Methods** & **RGB** & **Optical Flow** & **Estimated Poses** & **Visual Attention** & **Accuracy** \\ \hline [PERSON] et al. (2019) & X & O & X & X & 85.5\% \\ [PERSON] et al. (2019) & O & O & X & X & 79.0\% \\ [PERSON] et al. (2019) & X & X & X & X & 92.9\% \\ [PERSON] et al. (2020) & X & O & O & X & 98.1\% \\ [PERSON] et al. (2020) & X & O & X & X & 95.3\% \\ Based on Visual attention & O & O & X & O & 97.7\% \\ Based on Estimated poses & O & O & O & X & 97.9\% \\ Proposed & O & O & O & O & **98.9\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Experimental results with the Penn Action dataset.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Methods** & **Kinect Poses** & **RGB** & **Estimated Poses** & **Visual Attention** & **Accuracy** \\ \hline [PERSON] et al. (2020) & X & O & O & X & 62.9\% \\ [PERSON] et al. (2020) & X & O & O & X & 69.2\% \\ [PERSON] et al. (2020) & X & O & O & X & 73.4\% \\ [PERSON] et al. (2020) & X & O & O & X & 74.4\% \\ [PERSON] et al. (2020) & X & X & O & X & 74.9\% \\ [PERSON] et al. (2020) & X & O & O & X & 77.1\% \\ [PERSON] et al. (2020) & O & X & O & X & 75.6\% \\ [PERSON] et al. (2020) & X & X & O & X & 84.8\% \\ Based on Visual attention & X & O & X & O & 87.5\% \\ Based on Estimated poses & X & O & O & X & 87.7\% \\ Proposed & X & O & O & O & **87.9\%** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Experimental results with the NTU dataset.