# Lester: Rotoscope Animation through Video Object Segmentation and Tracking
## 3 Methodology
### Contours Simplification
**Comparison with image-to-image generative models:** Although we are not aware of other methods that have attempted to solve the problem for this specific visual style, an alternative could involve using conditional generative models. One potential pipeline, described in [13], adapts Stable Diffusion [7] for image-to-image translation using ControlNet [10], and employs tiled inputs along with video-guided style interpolation (EbSynth [14]) to maintain visual consistency across frames. We tested this processing pipeline with some videos from the test dataset to evaluate the feasibility of this approach. To achieve the desired style, we fine-tuned Stable Diffusion with Dreambooth [39]. Figure 11 shows four example frames obtained with this approach, but to assess the temporal consistency the results can be found in our results repository in [36]. The results reveal several drawbacks compared to the proposed method. Firstly, they exhibit poor temporal consistency across frames. Secondly, the method requires specifying the level of fidelity to the desired style, where higher fidelity compromises the original frame structure and details, leading to an undesirable trade-off. Thirdly, the performance varies significantly depending on various aspects of the input video, such as scene and resolution. Lastly, the method operates significantly slower than the proposed method, with a processing speed of only 0.3 FPS. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Subset** & \# & **Shape** & **Color** & **Temporal** & **Overall** \\ \hline ITW-LR a & 5 & 3.20 & 3.24 & 3.84 & 3.36 \\ ITW-HR b & 14 & 3.81 & 4.03 & 4.33 & 3.99 \\ C-HR c & 6 & 4.70 & 4.60 & 4.67 & 4.67 \\ all & & 3.90 & 4.01 & 4.31 & 4.02 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Subjective quality assessment MOS values of the ACR (1 = Bad to 5 = Excellent) of shape quality, color consistency, temporal consistency and overall quality. a ITW-LR \(\rightarrow\) in-the-wild low resolution; b ITW-HR \(\rightarrow\) in-the-wild high resolution; c C-HR \(\rightarrow\) controlled setting high resolution. Figure 7: Example results. Odd rows: input videos (faces have been anonymized). Even rows: method results. Figure 8: Example results. Odd rows: input videos (faces have been anonymized). Even rows: method results. Figure 11: Comparison with a generative pipeline: (**a**) input frame; (**b**) generative pipeline described in [13]; (**c**) Lester (proposed method). Figure 10: Example failure cases. Top row: input videos (videos 22, 23, 25 and 8 from the test dataset, faces have been anonymized); bottom row: results. Figure 9: Example synthesized animation frame (right) and its related original frame from an example input multi-person video. Discussion: The results obtained are very satisfactory and confirm that the latest advances in segmentation and tracking make new applications possible, also in the field of image synthesis. The proposed method has proven to be an effective method for generating rotoscopic animations from videos. The limitations detected do not represent a handicap since it is expected that the method will be used in videos selected or created expressly. Nor does its computational cost since the method is expected to be used in editing tasks that do not require real-time execution. The method is primarily designed for videos featuring a single person, but could be adapted to videos with multiple people by using multiple parameterizations for segmentation prompts and color palettes. ## 5 Conclusions

In this work we present Lester, a method for automatically synthesizing 2D retro-style animated characters from videos. Rather than addressing the challenge with an image-to-image generative model, or with human pose estimation tools, it is addressed mainly as an object segmentation and tracking problem. Subjective quality assessment shows that the approach is feasible and that the method can generate quality animations from videos with different poses and appearances, dynamic shots, partial shots and diverse backgrounds.