# Lester: Rotoscope Animation through Video Object Segmentation and Tracking
## 3 Methodology
### Contours Simplification
Finally, a pixelation effect is achieved by resizing the image down using bilinear interpolation and scaling it back up with nearest neighbor interpolation. Figure 6 shows how rim light, facial features and pixelation are progressively applied to an example frame. ## 4 Experiments and Results

In order to evaluate the performance of the method, we collected a set of 25 short videos ranging from 5 to 20 s in duration. The videos showcase a lone human subject displaying various clothing styles and engaging in diverse actions. The videos were obtained from the UCF101 Human Actions dataset [33], the Fashion Video Dataset [34], the Kinetics dataset [35] and YouTube. The assembled dataset includes both low- and high-resolution videos and both in-the-wild and controlled-setting videos. The dataset and the results of the experiments can be found in [36]. Figures 7 and 8 show some of example results. To assess the perceptual quality of the generated animations, a user study was conducted. A survey was designed comprising questions about three visual aspects: shape quality, color consistency and temporal consistency, plus and overall score. Scoring options were based on the Category Rating (ACR) described in ITU-T P.910 recommendation [37] scale from 1 = Bad to 5 = Excellent). The subjective quality assessment of the results is summarized in Table 1. We report the MOS (mean opinion score described in ITU-T P.800.1 [38]). The videos are grouped into three categories, in

Figure 5: Coloring process: (**a**) input frame; (**b**) segmentation; (**c**) visual guide with unique identifiers for each color segment; (**d**) final frame. Figure 6: Optional finishing details: (**a**) Flat method result (**b**) Rim light (**c**) Facial features (**d**) Pixelation. (ITW-LR), in-the-wild high resolution (ITW-HR) and controlled setting high resolution (C-HR). **Strengths:** The results show that the method is able to synthesize quality animations from diverse input videos (an average MOS 4.02 over 5). The performance significantly improves (4.67) on high-resolution videos taken in controlled conditions (videos 2, 11, 12, 13, 14 and 15). The method exhibits a very good temporal consistency and can correctly process videos with different poses and appearances, dynamic shots (e.g., video 5), videos with partial shots (e.g., the medium close-up shot in video 7) and diverse backgrounds. It is worth noting that, although it is not the primary focus of this work, the method can process videos containing multiple people. Objects associated with different individuals are tracked independently, each with a unique identifier, even if the objects are of the same type. This enables mapping these objects to different colors, allowing the entire pipeline to be applied to multi-person shots. Figure 9 shows an example result from a multi-person frame. **Limitations:** The segmentation and tracking performance decreases in videos with low resolution (all videos from the UCF101 Human Actions dataset), complex backgrounds (videos 17 and 20), low-contrast colors (videos 8, 10, 22 and 23), inappropriate lighting (videos 14 and 25) and occlusions (video 23). Figure 10 shows four example frames with errors from four different videos (videos 22, 23, 25 and 8 from the test dataset). **Computational cost:** The method has been implemented utilizing the Python programming language. The experiments have been run on a high-end server with a quadcore Intel i7-3820 at 3.6 GHz with 64 GB of DDR3 RAM, and 4 NVIDIA Tesla K40 GPU cards with 12 GB of GDDR5 each. The peak RAM usage of the method was recorded at 3.1 GB during the execution of the segmentation task, which also accounts for a peak GPU RAM usage of 2.1 GB. For a single frame, step 1 (segmentation and tracking) takes 0.56 s, step 2 (contours simplification) takes 0.43 s and step 3 (finishing details, mainly facial features) takes 1.32 s. Overall, the method operates at 0.43 FPS (1 FPS without facial features).