ProPainter: Improving Propagation and Transformer for Video Inpainting
Authors: name: Shangchen Zhou; name: Chongyi Li; name: Kelvin C. K. Chan; name: Chen Change Loy
Published: 2023-09-07 17:57:29

Abstract: Flow-based propagation and spatiotemporal Transformer are two mainstream
mechanisms in video inpainting (VI). Despite the effectiveness of these
components, they still suffer from some limitations that affect their
performance. Previous propagation-based approaches are performed separately
either in the image or feature domain. Global image propagation isolated from
learning may cause spatial misalignment due to inaccurate optical flow.
Moreover, memory or computational constraints limit the temporal range of
feature propagation and video Transformer, preventing exploration of
correspondence information from distant frames. To address these issues, we
propose an improved framework, called ProPainter, which involves enhanced
ProPagation and an efficient Transformer. Specifically, we introduce
dual-domain propagation that combines the advantages of image and feature
warping, exploiting global correspondences reliably. We also propose a
mask-guided sparse video Transformer, which achieves high efficiency by
discarding unnecessary and redundant tokens. With these components, ProPainter
outperforms prior arts by a large margin of 1.46 dB in PSNR while maintaining
appealing efficiency.