Understanding Alignment in Multimodal LLMs: A Comprehensive Study

Authors: Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, Peter Grasch
Primary area: foundation or frontier models, including LLMs
Keywords: foundation models, multimodal llm, alignment, image understanding

Abstract:
Preference alignment has become a crucial component in enhancing the performance of Large Language Models (LLMs), yet its impact in Multimodal Large Language Models (MLLMs) remains comparatively underexplored. Similar to language models, MLLMs for image understanding tasks encounter challenges like hallucination. In MLLMs, hallucination can occur not only by stating incorrect facts but also by producing responses that are inconsistent with the image content. A primary objective of alignment for MLLMs is to encourage these models to align responses more closely with image information. Recently, multiple works have introduced preference datasets for MLLMs and examined different alignment methods, including Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO). However, due to variations in datasets, base model types, and alignment methods, it remains unclear which specific elements contribute most significantly to the reported improvements in these works. In this paper, we independently analyze each aspect of preference alignment in MLLMs. We start by categorizing the alignment algorithms into two groups, offline (such as DPO), and online (such as online-DPO), and show that combining offline and online methods can improve the performance of the model in certain scenarios. 
We review a variety of published multimodal preference datasets and discuss how the details of their construction impact model performance. Based on these insights, we introduce a novel way of creating multimodal preference data called Bias-Driven Hallucination Sampling (BDHS) that needs neither additional annotation nor external models, and show that it can achieve competitive performance to previously published alignment work for multimodal models across a range of benchmarks.

Decision: reject
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730483328130

thread_type:
official_review

<thread_object>
creator: reviewer_z5vu

<content>
summary:
The paper explores preference alignment for improving Multimodal Large Language Models (MLLMs), specifically focusing on reducing hallucinations and increasing alignment between model outputs and image content. It provides a thorough analysis of various alignment methods and introduces a novel approach, Bias-Driven Hallucination Sampling (BDHS), which effectively generates preference data without human annotation or external models.

strengths:
1. The study systematically compares offline and online alignment methods, examining their impact on model performance across various metrics like hallucination reduction and response quality.
2. BDHS presents a low-cost, innovative solution to generate preference data, showing competitive results against other data-heavy methods.

weaknesses:
1. While the paper examines alignment techniques and datasets, it does not clearly articulate the primary findings from these investigations, which can make it challenging for readers to grasp the significance and implications of the study

2. BDHS demonstrates promising results; however, its effectiveness may differ across various MLLMs and visual tasks. Conducting additional experiments with diverse model architectures would bolster claims regarding its generalizability.

questions:
1. What are the effects of scaling up BDHS in terms of data size or complexity on model performance?
2. What specific modifications could be made to BDHS to achieve state-of-the-art results on key benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684624420

thread_type:
official_review

<thread_object>
creator: reviewer_jkxp

<content>
summary:
This paper investigates preference alignment techniques for Multimodal Large Language Models (MLLMs), focusing on how they address hallucinations, which occur when models produce responses not grounded in visual inputs. The study categorizes alignment methods into offline and online approaches and examines various multimodal preference datasets. The authors propose a novel data generation method called Bias-Driven Hallucination Sampling (BDHS), which does not require human annotations or external models. Experimental results demonstrate BDHS’s effectiveness compared to more resource-intensive methods.

strengths:
1、Comprehensive Analysis: The paper provides a detailed comparison of alignment methods, including offline and online strategies, and evaluates their effectiveness using diverse datasets.


2、Novel Data Generation Method: The introduction of BDHS offers a cost-effective alternative to traditional alignment approaches, reducing the need for human annotation or external supervision while maintaining competitive performance.

weaknesses:
1、Clarification of Methodological Choices: It would be helpful to better understand why specific thresholds and parameters were chosen for BDHS, such as the similarity score threshold and masking strategy.


2、Generalizability of BDHS: It remains unclear whether BDHS can be effectively applied to models beyond the specific ones studied. Further discussion on its applicability to other MLLMs or domains would strengthen the paper.

questions:
1、The DHS method induces hallucinations by performing attentional masking in the latent space. Is it possible that this strategy could affect the sensitivity of the model to critical details in the image? Have ablation experiments been performed to quantify the effect of this attentional masking in scenes of varying visual complexity? In addition, how to select the range of attention masking, and whether the alignment effect can be optimized by dynamic adjustment?

2、The paper mentions filtering out different non-preferred responses by semantic similarity score. For this filtering mechanism, is it possible that there is a bias that makes the model perform better or worse on specific types of semantic content? Have comparative experiments with different similarity scoring models been conducted to confirm the robustness of the selection mechanism? Furthermore, could this similarity score lead to a tendency for models to oversimplify when faced with less common or more complex visual scenes?

3、Does the performance of the BDHS method on the LLaVA 1.6-7B model generalize to larger or smaller model sizes? Have any experiments been conducted on models with different parameter numbers to explore whether this approach exhibits different advantages or disadvantages depending on the model size? Especially on small-scale models, is it possible that the method effect is not significant due to parameter limitations?

4、To what extent do current hallucina-evaluation benchmarks such as POPE and MMHALBench-V truly reflect model performance in real-world applications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731048609969

thread_type:
official_review

<thread_object>
creator: reviewer_a6hu

<content>
summary:
The paper addresses challenges in aligning MLLMs with human preferences to improve response accuracy and reduce hallucinations. It reviews various offline and online alignment strategies, including DPO  and RLHF, and introduces BDHS. BDHS generates preference data without human annotation, leveraging model-inherent biases to enhance performance cost-effectively. Results indicate BDHS is competitive with established preference datasets, demonstrating its potential as a lightweight alternative to traditional alignment approaches for MLLMs, especially in tasks requiring high fidelity between visual inputs and textual responses.

strengths:
1. The paper introduces a unique approach to generate preference data for MLLMs by utilizing model biases without human or external model annotations.
2. The paper provides empirical analysis, comparing BDHS with other alignment methods across multiple benchmarks, highlighting its effectiveness and resource efficiency in aligning MLLMs.

weaknesses:
1. The proposed data sampling approach partially mitigates hallucination issues in MLLMs but does not completely resolve them.
2. The BDHS method's dependency on hyperparameters, such as mask thresholds, could affect reproducibility across different model implementations.

questions:
1. Will the code be open-sourced to facilitate further research in this area?
2. How does the proposed approach ensure that the distribution of generated hallucination data aligns with real-world hallucination data distributions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732043054944

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to the reviewers

comment:
We sincerely thank all the reviewers for their time and effort to review our work. We are also grateful to our reviewers for recognizing the novelty and contribution of our research and providing thoughtful feedback. We have updated the submitted PDF to address the feedback and comments of the reviewers. Please find our responses to the questions and clarifications of key points below.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Paper analyzes off-line and on-line MLLM / VLM human preference alignment approaches, including DPO and PPO and introduces its own variant -- BDHS. BDHS generates preference data automatically, by leveraging VQA pairs and generating negative pairings through LLM inherent biases using masking. 

The paper was reviewed by three expert reviewers and received: 1 x accept, good paper and 2 x marginally above the acceptance threshold ratings. Overall reviewers agree that the approach is sound and presentation is good, but differ in the view of overarching contributions (with two reviewers rating it as "Fair"). Additional concerns with the work revolved around a few core issues: (1) proposed data sampling approach seems to only have limited effectiveness [a6hU], (2) stability with respect to hyper parameters (e.g., mask threshold) [a6hU] and selection [JkXP], (3) limited evaluation of BDHS method, mainly on the LLaVA 1.6-7B [JkXP], and (4) lacking clear articulation of findings which makes it more difficult to ascertain significance of the work [Z5VU]. Authors have provided a rebuttal and while [a6hU] appears to be convinced by the arguments, he/she did not further update the score. Unfortunately, the remaining two reviewers [JkXP] and [Z5VU] did not engage in discussion. 

AC has read the reviews, the rebuttal and the discussion; also looked and read the paper itself. With respect to concerns raised by [JkXP] and [Z5VU], AC didn't find arguments from the rebuttal completely convincing. Specifically, it would be good to see that BDHS approach's utility expands beyond LLaVA family of models and models of different sizes. In other words, while the findings are interesting and suggestive, the full impact on the breadth of MLLM / VLM models is not clear. 

Given the borderline nature of the paper, it was also discussed at length between AC and Senior AC. Given that none of the reviewers strongly championed the paper, and considering that some concerns still remain, AC and Senior AC, collectively, are recommending Rejection at this time. Authors are encouraged to address the raised concerns and to resubmit the paper to the next venue. 

Also, while AC wants to be clear that this has NOT been taken into account in evaluation of the work, given that this is concurrent, authors are encouraged to look at the "Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback" paper from NeurIPS 2024, which is closely related in terms of their analysis.