Title: Vision Transformers Need Zoomer: Efficient ViT with Visual Intent-Guided Zoom Adapter
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Efficient Vision Transformer, Adaptive Tokenization, Image Classification

Abstract: Vision Transformers (ViTs) have made significant strides recently, but vanilla ViT models struggle with complex scenes, particularly multi-label images and occluded objects. Humans can extract specific visual intents from complex images to guide effective classification. Inspired by human visual attention mechanisms that selectively focus on regions of interest while ignoring class-irrelevant areas, we propose ZoomViT, a novel approach that introduces visual intent-guided zoom adaptation for efficient vision transformers. ZoomViT is based on two key observations: (1) Humans and advanced models can intelligently ignore class-irrelevant areas and focus on semantically important regions through visual intent. (2) Standard ViTs can achieve superior classification accuracy when guided by adaptive zooming into regions that align with visual intent. Our approach introduces the Zoomer, a lightweight adapter with only 0.8M parameters that generates visual intent-guided score maps for image regions and dynamically adjusts patch sizes accordingly. This bio-inspired component simulates human-like visual attention by increasing patch density in regions deemed important by the visual intent guidance, while using larger patches for less critical areas. The visual intent-guided adaptation enhances both efficiency and accuracy, especially in complex images. Experiments show ZoomViT, based on the DeiT-S framework, achieves 83.8%(+4.0%) top-1 accuracy on ImageNet-1k, surpassing existing efficient state-of-the-art (SOTA) ViTs in accuracy and efficiency. The code will be publicly available.

TL;DR: This paper proposes a Zoomable Vision Transformer (ZoomViT) that uses visual intent-guided zoom adaptation to focus on class-decisive regions, achieving superior accuracy while maintaining computational efficiency.