Title: MOSAIC: Multimodal Object and Semantic Segmentation with Adapter Integration and Contextual Fusion
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: multimodal RGB-IR, object detection, semantic segmentation, computer vision, modality misalignment

Abstract: We introduce MOSAIC, a novel framework for enhancing multimodal RGB-IR object detection and semantic segmentation. MOSAIC utilizes Vision Transformers and introduces modules like the Deformable Feature Sampling, Feature Attention Fusion Block, and Contextual Feature Enhancer. These components dynamically align and integrate RGB-IR features, capturing multi-scale contextual information to enhance object detection and segmentation tasks. Extensive evaluations demonstrate that MOSAIC achieves state-of-the-art results on FLIR, LLVIP, MFNet and VT-series benchmark datasets, significantly improving robustness and accuracy in RGB-IR downstream tasks.

TL;DR: This paper presents MOSAIC, a framework enhancing multimodal RGB-IR object detection and semantic segmentation by using Vision Transformers with novel modules, achieving state-of-the-art results on benchmark datasets.