Efficient Multi-modal Large Language Models via Progressive Consistency
  Distillation
Authors: name: Zichen Wen, name: Shaobo Wang, name: Yufa Zhou, name: Junyuan Zhang, name: Qintong Zhang, name: Yifeng Gao, name: Zhaorun Chen, name: Bin Wang, name: Weijia Li, name: Conghui He, name: Linfeng Zhang
Organization: Shanghai Jiao Tong University

Visual tokens consume substantial computational resources in multi-modal
large models (MLLMs), significantly compromising their efficiency. Recent works
have attempted to improve efficiency by compressing visual tokens during
training, either through modifications to model components or by introducing
additional parameters. However, they often overlook the increased learning
difficulty caused by such compression, as the model's parameter space struggles
to quickly adapt to the substantial perturbations in the feature space induced
by token compression. In this work, we propose to develop Efficient MLLMs via
Progressive Consistency Distillation (EPIC), a progressive learning framework.
Specifically, by decomposing the feature space perturbations introduced by
token compression along the token-wise and layer-wise dimensions, we introduce
token consistency distillation and layer consistency distillation,
respectively, aiming to reduce the training difficulty by leveraging guidance
from a teacher model and following a progressive learning trajectory. Extensive
experiments demonstrate the superior effectiveness, robustness, and
generalization capabilities of our proposed framework.