# Comparative Study of Movie Shot Classification Based on Semantic Segmentation
## 1 Introduction  
In films, movie shot types are classified based on the distance between the camera and the subject, and the general types of shots are the close-up shot, the medium shot, and the long shot [1, 2]. Among them, close-up shots are used for expressing the emotions and psychology of the characters, with the subject occupying most of the screen. As shown in Figure 0(a), emotion or psychology is expressed with the character's eyes, mouth, and facial muscles by making the character's face occupy most of the screen [3]. In medium shots, a portion of the character's body below the waist or elbow is located at the bottom of the screen. Medium shots are also used to express a character's gaze direction and movement, since the character's body above the waist appears on the screen [3]. Figure 0(b) is an example of the medium shot, which shows the character's gaze direction, motion, and conversation partner. In long shots, the subject occupies about one-sixth of the screen, giving the audience information about the place (inside or outside, in an apartment, a shop, a forest, etc.) and time (day, night, season) [3].  
Additionally, the director uses the close-up, medium shot, and the long shot alternately to appropriately place the flow of emotion in the film and the importance of the scene [4]. As such, the shots contain a lot of information, such as the arrangement and flow of emotion, the psychology of the characters, and the important scenes, so classifying the shot type is a very important pre-task in movie analysis.  
Two representative studies of shot type classification include classifying shot types based on the proportion of the face occupying the screen and classifying shot types using a convolutional neural network (CNN). In one study that classified shot types from the proportion of the face on-screen, the accuracy of shot type classification was high for close-ups and medium shots (with faces in the frame), but the accuracy in shot type classification for long shots (without faces) was low. Also, shot type classification cannot be performed if there is no face on-screen. Shot type classification using the CNN is able to classify the shot type even if no one is on-screen, but a certain portion of the shots cannot be classified, owing to the absence of a semantic approach that people can grasp (that is, the image cannot be semantically analyzed).  
Therefore, in the present study, the performance of shot type classification was improved by pre-processing semantic segmentation of the frames extracted from the movie. Semantic segmentation approaches the images in order to distinguish boundary relationships among objects. The representative technologies of semantic segmentation include the Mask R-CNN and Yolact. We conducted a study for comparative evaluation of the performance attained from using these as preprocesses for shot type classification.  
The proposed approach automatically classifies close-up shots, medium shots, and long shots using a CNN and semantic segmentation, and the accuracy of shot type classification is enhanced by applying the boundary relationships of objects to the frames extracted from films using Mask R-CNN and Yolact. For the data, a total of 11,066 frames extracted from 89 films, such as _Inception_, were used.  
Section 2 of this article explains the works related to the background knowledge of shot type classification technology. Section 3 explains the structure suggested for shot type classification, and contemplates the experiment conducted, and its results. Lastly, the results are summarized in Section 4.