Title: FusionFormer: Multi-Window Fusion for Efficient Real-Time Segmentation with Vision Foundation Models
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Real-Time Semantic Segmentation, Vision Foundation Models, Efficient Transformer, FusionFormer, Multi-Window Fusion Attention

Abstract: Recent advances in real-time segmentation has been driven by lightweight Transformer variants. To address the limitations of such designs, state-of-the-art (SOTA) methods employ bidirectional architectures with training-only Transformer branch for long-range contextual guidance. However, these approaches typically depend on task-specific pre-trained models with limited scalability, potentially limiting their maximum performance.
In this paper, we introduce the \textit{Multi-Window Fusion Transformer} (\textbf{\textit{FusionFormer}}) to effectively leverage vision foundation models (VFMs). Specifically, we constrain self-attention computation within different window sizes and aggregate tokens within each window using varying fusion ratios. Our attention mechanism approximates attended fields from local to global while maintaining linear computational complexity, enabling better utilization of global attention guidance in Vision Transformer (ViT)-based VFMs. We first evaluate FusionFormer on the ImageNet-1k classification task, demonstrating its potential as a versatile efficient backbone. Further,extensive experiments on the ADE20K and Cityscapes datasets, coupling FusionFormer with a simple light head and DINOv2-B/14, demonstrate its excellent trade-off between segmentation accuracy and computational cost. Compared to previous methods, FusionFormer achieves a more efficient utilization of global guidance from VFMs. Our code will be released soon after the paper is accepted.

TL;DR: FusionFormer introduces Multi-Window Fusion Attention and a novel guidance mechanism to efficiently leverage Vision Foundation Models for state-of-the-art real-time semantic segmentation.