Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes
Authors: name: Yaoting Wang; name: Peiwen Sun; name: Dongzhan Zhou; name: Guangyao Li; name: Honggang Zhang; name: Di Hu
Published: 2024-07-15 17:54:45

Abstract: Traditional reference segmentation tasks have predominantly focused on silent
visual scenes, neglecting the integral role of multimodal perception and
interaction in human experiences. In this work, we introduce a novel task
called Reference Audio-Visual Segmentation (Ref-AVS), which seeks to segment
objects within the visual domain based on expressions containing multimodal
cues. Such expressions are articulated in natural language forms but are
enriched with multimodal cues, including audio and visual descriptions. To
facilitate this research, we construct the first Ref-AVS benchmark, which
provides pixel-level annotations for objects described in corresponding
multimodal-cue expressions. To tackle the Ref-AVS task, we propose a new method
that adequately utilizes multimodal cues to offer precise segmentation
guidance. Finally, we conduct quantitative and qualitative experiments on three
test subsets to compare our approach with existing methods from related tasks.
The results demonstrate the effectiveness of our method, highlighting its
capability to precisely segment objects using multimodal-cue expressions.
Dataset is available at
https://gewu-lab.github.io/Ref-AVS{https://gewu-lab.github.io/Ref-AVS}.