SMITE: Segment Me In TimE
Authors: name: Amirhossein Alimohammadi; name: Sauradip Nag; name: Saeid Asgari Taghanaki; name: Andrea Tagliasacchi; name: Ghassan Hamarneh; name: Ali Mahdavi Amiri
Published: 2024-10-24 08:38:20

Abstract: Segmenting an object in a video presents significant challenges. Each pixel
must be accurately labelled, and these labels must remain consistent across
frames. The difficulty increases when the segmentation is with arbitrary
granularity, meaning the number of segments can vary arbitrarily, and masks are
defined based on only one or a few sample images. In this paper, we address
this issue by employing a pre-trained text to image diffusion model
supplemented with an additional tracking mechanism. We demonstrate that our
approach can effectively manage various segmentation scenarios and outperforms
state-of-the-art alternatives.