Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoning
Authors: name: Yuyao Ge; name: Shenghua Liu; name: Yiwei Wang; name: Lingrui Mei; name: Baolong Bi; name: Xuanshan Zhou; name: Jiayu Yao; name: Jiafeng Guo; name: Xueqi Cheng
Published: 2025-09-08 09:20:04

Abstract: Vision-Language Models (VLMs) have demonstrated remarkable success across
diverse visual tasks, yet their performance degrades in complex visual
environments. While existing enhancement approaches require additional
training, rely on external segmentation tools, or operate at coarse-grained
levels, they overlook the innate ability within VLMs. To bridge this gap, we
investigate VLMs' attention patterns and discover that: (1) visual complexity
strongly correlates with attention entropy, negatively impacting reasoning
performance; (2) attention progressively refines from global scanning in
shallow layers to focused convergence in deeper layers, with convergence degree
determined by visual complexity. (3) Theoretically, we prove that the contrast
of attention maps between general queries and task-specific queries enables the
decomposition of visual signal into semantic signals and visual noise
components. Building on these insights, we propose Contrastive Attention
Refinement for Visual Enhancement (CARVE), a training-free method that extracts
task-relevant visual signals through attention contrasting at the pixel level.
Extensive experiments demonstrate that CARVE consistently enhances performance,
achieving up to 75% improvement on open-source models. Our work provides
critical insights into the interplay between visual complexity and attention
mechanisms, offering an efficient pathway for improving visual reasoning with
contrasting attention.