Unlocking the Potential of MLLMs in Referring Expression Segmentation
  via a Light-weight Mask Decode
Authors: name: Jingchao Wang, name: Zhijian Wu, name: Dingjiang Huang, name: Yefeng Zheng, name: Hong Wang

Reference Expression Segmentation (RES) aims to segment image regions
specified by referring expressions and has become popular with the rise of
multimodal large models (MLLMs). While MLLMs excel in semantic understanding,
their token-generation paradigm struggles with pixel-level dense prediction.
Existing RES methods either couple MLLMs with the parameter-heavy Segment
Anything Model (SAM) with 632M network parameters or adopt SAM-free lightweight
pipelines that sacrifice accuracy. To address the trade-off between performance
and cost, we specifically propose MLLMSeg, a novel framework that fully
exploits the inherent visual detail features encoded in the MLLM vision encoder
without introducing an extra visual encoder. Besides, we propose a
detail-enhanced and semantic-consistent feature fusion module (DSFF) that fully
integrates the detail-related visual feature with the semantic-related feature
output by the large language model (LLM) of MLLM. Finally, we establish a
light-weight mask decoder with only 34M network parameters that optimally
leverages detailed spatial features from the visual encoder and semantic
features from the LLM to achieve precise mask prediction. Extensive experiments
demonstrate that our method generally surpasses both SAM-based and SAM-free
competitors, striking a better balance between performance and cost. Code is
available at https://github.com/jcwang0602/MLLMSeg.