Learning to Inference Adaptively for Multimodal Large Language Models
Authors: name: Zhuoyan Xu, name: Khoi Duc Nguyen, name: Preeti Mukherjee, name: Saurabh Bagchi, name: Somali Chaterji, name: Yingyu Liang, name: Yin Li

Multimodal Large Language Models (MLLMs) have shown impressive capabilities
in reasoning, yet come with substantial computational cost, limiting their
deployment in resource-constrained settings. Despite recent efforts on
improving the efficiency of MLLMs, prior solutions fall short in responding to
varying runtime conditions, in particular changing resource availability (e.g.,
contention due to the execution of other programs on the device). To bridge
this gap, we introduce AdaLLaVA, an adaptive inference framework that learns to
dynamically reconfigure operations in an MLLM during inference, accounting for
the input data and a latency budget. We conduct extensive experiments across
benchmarks involving question-answering, reasoning, and hallucination. Our
results show that AdaLLaVA effectively adheres to input latency budget,
achieving varying accuracy and latency tradeoffs at runtime. Further, we
demonstrate that AdaLLaVA adapts to both input latency and content, can be
integrated with token selection for enhanced efficiency, and generalizes across
MLLMs. Our project webpage with code release is at
https://zhuoyan-xu.github.io/ada-llava/.