Law of Vision Representation in MLLMs
Authors: name: Shijia Yang; name: Bohan Zhai; name: Quanzeng You; name: Jianbo Yuan; name: Hongxia Yang; name: Chenfeng Xu
Published: 2024-08-29 08:56:48

Abstract: We present the "Law of Vision Representation" in multimodal large language
models (MLLMs). It reveals a strong correlation between the combination of
cross-modal alignment, correspondence in vision representation, and MLLM
performance. We quantify the two factors using the cross-modal Alignment and
Correspondence score (AC score). Through extensive experiments involving
thirteen different vision representation settings and evaluations across eight
benchmarks, we find that the AC score is linearly correlated to model
performance. By leveraging this relationship, we are able to identify and train
the optimal vision representation only, which does not require finetuning the
language model every time, resulting in a 99.7% reduction in computational
cost.