# Comparative Study of Movie Shot Classification Based on Semantic Segmentation
## 3 Comparison of Shot Type Classification Methods Based on Semantic Segmentation
### Experimental Results  
In the experimental data, 11,066 frames were extracted randomly from 89 movies, such as _Inception_, and which were classified by shot type through the ground truth operation. Of the 89 movies, 75 were used for training and validation, and the remaining 14 were used for testing. As shown in Table 1, 8679 frames were used for training and 1787 frames were used for validation, with 600 frames used to measure the accuracy of the shot type classification. The ratios of training data and validation  
Figure 8: Structure of the suggested semantic segmentation.  
Figure 9: Regular frame examples of semantic segmentation applications: (**a**) Non-mask; (**b**) Mask R-CNN; (**c**) Yolact.  
data were set to 82% and 15% for shot, and 85% and 15% for medium shot for supporting enough training data.  
In order to evaluate the accuracy of the shot type classifications, a general frame and frame to which semantic segmentation was applied using Mask R-CNN and Yolact were classified using VGG-16 and ResNet-50 Afterward, the accuracies were evaluated and compared.  
As a result, we confirmed that the average accuracy of the shot type classification preprocessed with semantic segmentation was 1.9% higher than shot type classifications using a general frame, as seen in the results presented in Table 2. Also, as shown in Table 3, shot type classification using ResNet-50 was more accurate than shot type classification using VGG-16.  
A conventional shot type classification study using a CNN classified shot types with 94% accuracy for close-ups, medium shots, and long shots using VGG-16 [1]. In the present study, when classifying close-ups, medium shots, and long shots using VGG-16 on general frames, the shot types were classified with 92.5% accuracy, as seen in Table 3. This was expected to occur due to the differences in the data. In this study, the method to classify the shot types with ResNet-50 after preprocessing for semantic segmentation using Yolact is illustrated as a confusion matrix in Figure 10 and showed the highest accuracy among the experimental results (96%). This classified shots with performance better than classifying shot types with only VGG-16 and ResNet-50, as done in the previous study.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Shot Type** & **Training** & **Validation** & **Testing** \\ \hline Close-up & 3586 & 782 & 200 \\ Medium & 2392 & 413 & 200 \\ Long & 2701 & 592 & 200 \\ Total & 8679 & 1787 & 600 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Number of Frames.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Classification Type** & **Without Semantic Segmentation** & \multicolumn{2}{c}{**With Semantic Segmentation**} \\ \cline{2-3}  & & **Mask R-CNN Added** & **Yolact Added** \\ \hline VGG-16 & 92.5\% & 94.8\% & 93.8\% \\ ResNet-50 & 93.5\% & 95.3\% & 96\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Average Accuracy of Shot Type Classification.  
Figure 10: Confusion Matrix of ResNet-50 with Yolact.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Classification Type** & **Without Semantic Segmentation** & \multicolumn{2}{c}{**With Semantic Segmentation**} \\ \cline{3-4}  & & **Mask R-CNN Added** & **Yolact Added** \\ \hline VGG-16 & 92.5\% & 94.8\% & 93.8\% \\ ResNet-50 & 93.5\% & 95.3\% & 96\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy of Shot Type Classification.  
The training time of ResNet-50 and Yolact with the highest accuracy was 1654 s, and this model took 66s to assort 600 test frames. So, it took approximately 0.11s per frame. Therefore, this model is difficult to apply in real time to process 30 frames.