FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary
  Segmentation
Authors: name: Xi Chen; name: Haosen Yang; name: Sheng Jin; name: Xiatian Zhu; name: Hongxun Yao
Published: 2024-09-05 13:36:50

Abstract: Open-vocabulary segmentation poses significant challenges, as it requires
segmenting and recognizing objects across an open set of categories in
unconstrained environments. Building on the success of powerful vision-language
(ViL) foundation models, such as CLIP, recent efforts sought to harness their
zero-short capabilities to recognize unseen categories. Despite notable
performance improvements, these models still encounter the critical issue of
generating precise mask proposals for unseen categories and scenarios,
resulting in inferior segmentation performance eventually. To address this
challenge, we introduce a novel approach, FrozenSeg, designed to integrate
spatial knowledge from a localization foundation model (e.g., SAM) and semantic
knowledge extracted from a ViL model (e.g., CLIP), in a synergistic framework.
Taking the ViL model's visual encoder as the feature backbone, we inject the
space-aware feature into the learnable queries and CLIP features within the
transformer decoder. In addition, we devise a mask proposal ensemble strategy
for further improving the recall rate and mask quality. To fully exploit
pre-trained knowledge while minimizing training overhead, we freeze both
foundation models, focusing optimization efforts solely on a lightweight
transformer decoder for mask proposal generation-the performance bottleneck.
Extensive experiments demonstrate that FrozenSeg advances state-of-the-art
results across various segmentation benchmarks, trained exclusively on COCO
panoptic data, and tested in a zero-shot manner. Code is available at
https://github.com/chenxi52/FrozenSeg.