Visual Representation Alignment for Multimodal Large Language Models
Authors: name: Heeji Yoon; name: Jaewoo Jung; name: Junwan Kim; name: Hyungyu Choi; name: Heeseong Shin; name: Sangbeom Lim; name: Honggyu An; name: Chaehyun Kim; name: Jisang Han; name: Donghyun Kim; name: Chanho Eom; name: Sunghwan Hong; name: Seungryong Kim
Published: 2025-09-09 17:59:14

Abstract: Multimodal large language models (MLLMs) trained with visual instruction
tuning have achieved strong performance across diverse tasks, yet they remain
limited in vision-centric tasks such as object counting or spatial reasoning.
We attribute this gap to the prevailing text-only supervision paradigm, which
provides only indirect guidance for the visual pathway and often leads MLLMs to
discard fine-grained visual details during training. In this paper, we present
VIsual Representation ALignment (VIRAL), a simple yet effective regularization
strategy that aligns the internal visual representations of MLLMs with those of
pre-trained vision foundation models (VFMs). By explicitly enforcing this
alignment, VIRAL enables the model not only to retain critical visual details
from the input vision encoder but also to complement additional visual
knowledge from VFMs, thereby enhancing its ability to reason over complex
visual inputs. Our experiments demonstrate consistent improvements across all
tasks on widely adopted multimodal benchmarks. Furthermore, we conduct
comprehensive ablation studies to validate the key design choices underlying
our framework. We believe this simple finding opens up an important direction
for the effective integration of visual information in training MLLMs.