AVG-LLaVA: A Large Multimodal Model with Adaptive Visual Granularity

Authors: Zhibin Lan, Liqiang Niu, Fandong Meng, Wenbo Li, Jie Zhou, Jinsong Su
Primary area: foundation or frontier models, including LLMs
Keywords: Large multimodal model, multi-stage training, adaptive visual granularity

Abstract:
Recently, when dealing with high-resolution images, dominant large multimodal models (LMMs) usually divide them into multiple local images and one global image, which will lead to a large number of visual tokens. In this work, we introduce AVG-LLaVA, an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction. This approach not only reduces the number of visual tokens and speeds up inference, but also improves the overall model performance. Specifically, we introduce the following modules based on LLaVA-NeXT: (a) a visual granularity scaler that includes multiple pooling layers to obtain visual tokens with different granularities; (b) a visual granularity router, which includes a Transformer layer, an MLP layer, and a voter layer, used to select the appropriate visual granularity based on the image and instruction. Furthermore, we propose RGLF, a novel training paradigm that aims at aligning the granularity predicted by the router with the preferences of the LMM, without the need for additional manually annotated data. Extensive experiments and analysis show that AVG-LLaVA achieves superior performance across 11 benchmarks, as well as significantly reduces the number of visual tokens and speeds up inference (e.g., an 85.3\% reduction in visual tokens and a 2.53$\times$ increase in inference speed on the AI2D benchmark).

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730205997001

thread_type:
official_review

<thread_object>
creator: reviewer_k4vy

<content>
summary:
The authors propose an adaptive visual granularity mechanism dubbed AVG-LLaVA.  Based on this assumption, they employ the visual granularity scaler to generate visual tokens with various granularities, and the visual granularity router to select the appropriate visual granularity. Besides, the paper introduces a training paradigm RGLF to enhance the router.  Comprehensive experiments are performed on various visual benchmarks to validate the effectiveness of the method.

strengths:
1. The motivation is novel, and different prompts require information at different visual granularities. And the manuscript is explicit and well-organized.
2. The authors solve the problem of training the router in VLM directly and utilize the ranking loss to supervise, which is impressive.
3. Experimental validation is sufficient. The authors conduct comprehensive experiments on various tasks and show improvements, to validate the effectiveness of the method.

weaknesses:
1. The method lacks novelty. (1) the multiple pooling operation in visual granularity scaler is very common, like the most classic SPPNet [1]. (2) the router operation has been proposed for many years.
2. Although the method sounds simple, the overall pipeline is complex. The stage 2 and 3 cost more training resources and time, where the vision encoder and LLM both are trained.

[1] He, K., Zhang, X., Ren, S. and Sun, J., 2015. Spatial pyramid pooling in deep convolutional networks for visual recognition. *IEEE transactions on pattern analysis and machine intelligence*, *37*(9), pp.1904-1916.

questions:
1. Is it convenient to list their accuracy in Table 3 for further comparison? Besides, I want to know the absolute value of its actual speed.
2. I would like to see a visualization of actual token clipping, such as the image in Figure 1, and what the router results would be for different prompts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730434298748

thread_type:
official_review

<thread_object>
creator: reviewer_uyrw

<content>
summary:
This work aims to enhance the LMM LLAVA-NeXT through improved visual granularity selection. 
To achieve this, we introduce AVG-LLAVA, which consists of a visual granularity scaler, 
a visual granularity router, and the RGLF training paradigm. 
Experiments have been conducted to validate the effectiveness of the proposed method.

strengths:
1. The research focus  is intriguing, particularly the aspects of visual granularity selection and the Ranking Granularity to Align LMM Feedback.

2. Experimental results demonstrate its effectiveness.

weaknesses:
1. The training pipeline has become more complicated, moving from original two stages to four, which increases the training overhead despite the performance improvements.

2. I think the description of the main contributions is not well-articulated; it should better to include an algorithm, especially the Visual Granularity Router.

3. It would be beneficial to provide direct, rigorous evidence for the selection of granularity to illustrate the proposed method.

4. Providing visual examples that highlight the need for granularity, such as attention maps of visual tokens in the LLM, would be advantageous.

5. In Table 3, for ChartQA, the token per grid is 99.1%, while the speed is 0.97x without any increment.

questions:
It should better to provide total token numbers of each method in main performance comparsion for each method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730520841823

thread_type:
official_review

<thread_object>
creator: reviewer_je8e

<content>
summary:
This paper introduces a model that dynamically adjusts the granularity of visual tokens based on input images and instructions. This adaptive mechanism improves both efficiency and performance in multimodal tasks, reducing token usage and speeding up inference. The authors propose a novel training method, Ranking Granularity to Align LMM Feedback (RGLF), and test the model across 11 benchmarks. While the approach optimizes efficiency, concerns remain regarding scalability and performance trade-offs on certain tasks. The work offers promising advancements in multimodal learning.

strengths:
The paper introduces a visual granularity scaler and router, which adaptively selects the appropriate granularity for visual tokens based on the input image and instructions. This adaptive selection mechanism is a significant advancement over static high-resolution LMMs, potentially improving both efficiency and accuracy in multimodal tasks.

weaknesses:
1.	Lack of novelty: The motivation of this paper is highly similar to Matryoshka model, which also employs hierarchical token merging for visual token reduction, akin to token pruning in this paper. It seems that the difference is that the authors design an router to allocate weights to several granularities, which is incremental in terms of novelty.
2.	Insufficient experiments: This paper does not fully explore alternative approaches for granularity selection, such as task-specific fine-tuning or manual selection for certain tasks that might further improve performance.
3.	While the model's adaptive granularity selection is a strength, the architecture of the visual granularity router (involving multiple pooling layers, Transformer layers, and a voter layer) adds significant complexity and a substantial computational cost.
4.	The performance improvement is not superior across all benchmarks. For example, in GQA and ScienceQA, the proposed method underperforms slightly compared to some baselines, raising concerns about whether token reduction is always beneficial.
5.	Repeated Training Data: The training data for Stages 2, 3, and 4 are identical. Therefore, it is unclear whether the performance improvement is due to repeated training, akin to training for three epochs.
6.	Performance on OCR Tasks: As shown in Table 5, the visual tokens for OCR tasks are almost entirely retained, rendering the filter ineffective. The improvement in OCR tasks may primarily stem from repeated training.

questions:
1.	The ablation study in Section 4.5 suggests a strong reliance on instruction tokens for granularity selection. Could the model's robustness be affected in situations where instructions are ambiguous or noisy? This is more important to the industry from my perspective.
2.	The benchmarks used are well-known public datasets. However, has the model been evaluated in real-world scenarios with less curated, noisier data? This would test its robustness in a more practical context.
3.	Training Cost: Provide details of the training costs associated with each of the four training stages.
4.	Comparative Experiment: Conduct a comparative experiment by training LLaVA-Next with repeated SFT data two or three times and present the detailed results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730788050233

thread_type:
official_review

<thread_object>
creator: reviewer_icsi

<content>
summary:
The paper presents AVG-LLaVA, a large multimodal model capable of adaptively selecting the appropriate visual granularity based on input images and instructions, aiming to enhance model performance and reduce the number of visual tokens to expedite inference. AVG-LLaVA extends LLaVA-NeXT with the addition of a visual granularity scaler and a visual granularity router, along with a novel training paradigm called RGLF, which aligns the router's predicted probabilities of multiple granularities with the preferences of the LMM through a ranking loss.

strengths:
1. The paper introduces a novel approach to handle high-res images by adaptively selecting the appropriate granularity based on the input image and instruction. Also, it conducted experiments to develop the appropriate tuning practice (the training state 3 & 4) to unlock the potential of the new paradigm.
2. On multiple benchmarks, AVG-LLaVA demonstrates its efficacy. It can achieve better results compared to LLaVA-NeXT while consumes much less computations.

weaknesses:
1. The training paradigm is complex. It incorporates two additional training stages, each requires extensive computation costs. The additional training cost may hinder this approach from being widely adopted.
2. The framework is not thoroughly investigates and the ablation study is not sufficient (see Questions).

questions:
1. It's well known than finetuning VLMs on instruction tuning corpora with multiple epochs will typically improve the performance on benchmarks. The authors need to prove that the improvement cannot be simply attributed to 3x tuning epochs (corresponding to stage 2 to 4).
2. Achieving better performance with fewer visual tokens is not a usual case. Would you please include more qualitative & quantitative examples & analysis and discuss under which circumstances the VLM can achieve this?
3. The AVG-LLaVA framework can be easily extended to perform patch-wise granularity selection (for example, select different granularity for different patches). Would that be helpful to save more visual tokens under text-rich scenarios (the current AVG-LLaVA did not save much visual tokens for TextVQA and ChartQA). 
4. Recently, Qwen2-VL proposed to use native dynamic resolution visual encoders (no patchify) to generate visual embeddings. It would be beneficial to show that AVG-LLaVA also works for that kind of visual encoders.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732337109831

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking Forward to Further Engagement with Reviewers

comment:
We deeply appreciate the reviewers' efforts and valuable feedback on our work. While we have not received responses during the rebuttal period, we remain eager to address any remaining concerns or questions and welcome further discussions.

Over the past few days, we have worked diligently to address the reviewers' concerns and questions through additional experiments and detailed explanations. Therefore, we kindly hope that these clarifications and additional experiments will be considered in reevaluating our work. 

We would like to express our heartfelt gratitude to all the reviewers for their time, effort, and invaluable contributions.

Sincerely,

Authors of Paper #4331
</content>
</thread_object>

</thread>
<thread>
creation date:
1732474287195

thread_type:
official_comment

<thread_object>
creator: area_chair_dqo7

<content>
comment:
Dear Reviewers,

This is a friendly reminder that the discussion period will end on Nov 26th (Anywhere on Earth). If you have not already, please take a careful look at the other reviews and author responses, and comment on whether your original rating stands. Thank you.

Best, AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732809999916

thread_type:
official_comment

<thread_object>
creator: area_chair_dqo7

<content>
comment:
Dear reviewers,

This is a friendly reminder that the discussion period has been extended until December 2nd. If you haven’t yet, we kindly encourage you to review the authors' rebuttal and messages at your earliest convenience and confirm whether your comments have been adequately addressed.

We greatly appreciate your service to this process.

Best, AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces AVG-LLaVA, a large multimodal model designed to adaptively determine the appropriate level of visual granularity based on the input image and instruction. It builds upon LLaVA-NeXT and incorporates a visual granularity scaler and a visual granularity router, which work together to extract multi-granularity visual features and select the optimal granularity for a given image and instruction.  The paper received scores of 5, 5, 5, 6.  Mentioned positives include good motivation, intriguing approach, and promising results.  Mentioned negatives include incremental novelty, complex training paradigm, minor performance improvements, and insufficient experiments and analyses.  Only one of the reviewers engaged in the rebuttal, but felt their concerns were not adequately addressed.  The AC carefully considered the paper, rebuttal, and author messages.  The rebuttal and author messages address some concerns, particularly regarding experiments and analyses, but challenges related to incremental novelty and complex training persist.  The AC agrees with the reviewers that the paper does not meet the bar for acceptance to ICLR.