ICLR 2025

Published as a conference paper at ICLR 2025
LOCALITY ALIGNMENT IMPROVES VISION-LANGUAGE
MODELS
Ian Covert, Tony Sun, James Zou∗, Tatsunori Hashimoto∗
Stanford University
{icovert, suntony, jamesz, thashim}@stanford.edu
ABSTRACT
Vision language models (VLMs) have seen growing adoption in recent years,
but many still struggle with basic spatial reasoning errors. We hypothesize that
this is due to VLMs adopting pre-trained vision backbones, specifically vision
transformers (ViTs) trained with image-level supervision and minimal inductive
biases. Such models may fail to encode the class contents at each position in
the image, and our goal is to resolve this with a vision backbone that effectively
captures both local and global image semantics. Our main insight is that we do
not require new supervision to learn this capability – pre-trained models contain
significant knowledge of local semantics that we can extract and use for scalable
self-supervision. We propose a new efficient post-training stage for ViTs called
locality alignment and a novel fine-tuning procedure called MaskEmbed that uses
a masked reconstruction loss to learn semantic contributions for each image patch.
We first evaluate locality alignment with a vision-only benchmark, finding that it
improves a model’s performance at patch-level semantic segmentation, especially
for strong backbones trained with image-caption pairs (e.g., CLIP and SigLIP). We
then train a series of VLMs with and without locality alignment, and show that
locality-aligned backbones improve performance across a range of benchmarks,
particularly ones that involve spatial understanding (e.g., RefCOCO, OCID-Ref,
TallyQA, VSR, AI2D). Overall, we demonstrate that we can efficiently learn local
semantic extraction via a locality alignment stage, and that this procedure benefits
VLM training recipes that use off-the-shelf vision backbones.
1
INTRODUCTION
Auto-regressive VLMs are an exciting new type of model that emerged in the last couple ...