Title: SEDiT: Mask-Free Video Subtitle Erasure with Prompt Instruction
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: video inpainting, subtitle removal

Abstract: Recent breakthroughs in video diffusion models have significantly accelerated the development of video editing techniques. However, existing methods often rely on inpainting video frames based on masked input, which requires extracting the target video mask in advance. The precision of the segmentation directly affects the quality of the completion. In this paper, we present $\textbf{SEDiT}$, a novel end-to-end video 
$\textbf{S}$ubtitle $\textbf{E}$rasure approach via $\textbf{Di}$ffusion $\textbf{T}$ransformer. We introduce a mask-free training approach, driven by prompt instructions, that enables direct erasure of targeted sutitle. The proposed end-to-end framework mitigates the suboptimality inherent in the two-stage processing of prior models. To address the challenge of long-term temporal consistency, we adopt a hybrid training strategy by occasionally conditioning the model with clean first-frame latents. This facilitates temporal continuity, allowing each segment during inference to leverage the output of its predecessor. To avoid visible seams caused by cropping and reinserting processed targets, particularly in scenarios involving substantial motion, we feed the original video directly into SEDiT. Thanks to the highly compressed Variational Autoencoder (VAE) in the base model and chunk-wise streaming inference, our method can efficiently handle naive 1080p video with infinite length.

TL;DR: end-to-end mask-free video subtitle removal