Inherently Faithful Attention Maps for Vision Transformers
Authors: name: Ananthu Aniraj; name: Cassio F. Dantas; name: Dino Ienco; name: Diego Marcos
Published: 2025-06-10 15:41:22

Abstract: We introduce an attention-based method that uses learned binary attention
masks to ensure that only attended image regions influence the prediction.
Context can strongly affect object perception, sometimes leading to biased
representations, particularly when objects appear in out-of-distribution
backgrounds. At the same time, many image-level object-centric tasks require
identifying relevant regions, often requiring context. To address this
conundrum, we propose a two-stage framework: stage 1 processes the full image
to discover object parts and identify task-relevant regions, while stage 2
leverages input attention masking to restrict its receptive field to these
regions, enabling a focused analysis while filtering out potentially spurious
information. Both stages are trained jointly, allowing stage 2 to refine stage
1. Extensive experiments across diverse benchmarks demonstrate that our
approach significantly improves robustness against spurious correlations and
out-of-distribution backgrounds.