MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image
  Segmentation
Authors: name: Minhyun Lee; name: Seungho Lee; name: Song Park; name: Dongyoon Han; name: Byeongho Heo; name: Hyunjung Shim
Published: 2024-11-28 11:27:56

Abstract: Referring Image Segmentation (RIS) is an advanced vision-language task that
involves identifying and segmenting objects within an image as described by
free-form text descriptions. While previous studies focused on aligning visual
and language features, exploring training techniques, such as data
augmentation, remains underexplored. In this work, we explore effective data
augmentation for RIS and propose a novel training framework called Masked
Referring Image Segmentation (MaskRIS). We observe that the conventional image
augmentations fall short of RIS, leading to performance degradation, while
simple random masking significantly enhances the performance of RIS. MaskRIS
uses both image and text masking, followed by Distortion-aware Contextual
Learning (DCL) to fully exploit the benefits of the masking strategy. This
approach can improve the model's robustness to occlusions, incomplete
information, and various linguistic complexities, resulting in a significant
performance improvement. Experiments demonstrate that MaskRIS can easily be
applied to various RIS models, outperforming existing methods in both fully
supervised and weakly supervised settings. Finally, MaskRIS achieves new
state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg datasets. Code
is available at https://github.com/naver-ai/maskris.