Visual Question Decomposition on Multimodal Large Language Models
Authors: name: Haowei Zhang; name: Jianzhe Liu; name: Zhen Han; name: Shuo Chen; name: Bailan He; name: Volker Tresp; name: Zhiqiang Xu; name: Jindong Gu
Published: 2024-09-28 12:49:16

Abstract: Question decomposition has emerged as an effective strategy for prompting
Large Language Models (LLMs) to answer complex questions. However, while
existing methods primarily focus on unimodal language models, the question
decomposition capability of Multimodal Large Language Models (MLLMs) has yet to
be explored. To this end, this paper explores visual question decomposition on
MLLMs. Specifically, we introduce a systematic evaluation framework including a
dataset and several evaluation criteria to assess the quality of the decomposed
sub-questions, revealing that existing MLLMs struggle to produce high-quality
sub-questions. To address this limitation, we propose a specific finetuning
dataset, DecoVQA+, for enhancing the model's question decomposition capability.
Aiming at enabling models to perform appropriate selective decomposition, we
propose an efficient finetuning pipeline. The finetuning pipeline consists of
our proposed dataset and a training objective for selective decomposition.
Finetuned MLLMs demonstrate significant improvements in the quality of
sub-questions and the policy of selective question decomposition. Additionally,
the models also achieve higher accuracy with selective decomposition on VQA
benchmark datasets.