OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference
Authors: Xiangyu Zhao, Shengyuan Ding, Zicheng Zhang, Haian Huang, Maosongcao, Jiaqi Wang, Weiyun Wang, Xinyu Fang, Wenhai Wang, Guangtao Zhai, Hua Yang, Haodong Duan, Kai Chen
Accepted at: acl25

OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

Xiangyu Zhao1,2* Shengyuan Ding2,3* Zicheng Zhang1,2
Haian Huang2 Maosong Cao2 Weiyun Wang2,4 Jiaqi Wang2 Xinyu Fang2,5
Wenhai Wang2 Guangtao Zhai1,2 Haodong Duan2† Hua Yang1,2‡ Kai Chen2†
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University1
Shanghai AI Laboratory2 Nanjing University3 Fudan University4 Zhejiang University5

Figure 1: OmniAlign-V consists of curated images paired with open-ended, comprehensive question-answer pairs, significantly improving the alignment of MLLMs with human preferences. Additionally, we introduce MM-AlignBench, a human-annotated, high-quality benchmark specifically designed to evaluate the ability of MLLMs to align with human values.

Abstract

Recent advancements in open-source multi-modal large language models (MLLMs) have primarily focused on enhancing foundational capabilities, leaving a significant gap in human preference alignment. This paper introduces OmniAlign-V, a comprehensive dataset of 200K high-quality training samples featuring diverse images, complex questions, and varied response formats to improve MLLMs’ alignment with human preferences. We also present MM-AlignBench, a human-annotated benchmark specifically designed to evaluate MLLMs’ alignment with human values. Experimental results show that finetuning MLLMs with OmniAlign-V, using Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO), significantly enhances human preference alignment while maintaining or enhancing performance on standard VQA benchmarks, preserving their fundamental capabilities.

(MLLMs) (OpenAI, 2023b; Team et al., 2023) have also seen significant improvements. Most open-source MLLMs (Liu et al., 2023b; Chen et al., 2023b) are developed by connecting a vision encoder (Dosovitskiy, 2020) to a pretrained LLM, followed by vision instruction tuning. Existing vision instruction tuning datasets (Liu et al., 2023b; Chen et al., 2023a, 2024a) and multi-modal evaluation benchmarks (Liu et al., 2023d; Yue et al., 2023; Lu et al., 2023) primarily focus on assessing fundamental capabilities (object recognition, OCR, etc.) of MLLMs, while paying little attention to human preference alignment. Consequently, while open-source MLLMs achieve comparable performance to proprietary counterparts on objective metrics of these foundational skills, they display a significant gap in the alignment with human preferences, which detrimentally impacts user experience in multi-modal conversational interactions, as demonstrated in Appx. A.

In this work, a preliminary investigation was conducted to quantitatively assess the degradation of human preference alignment in MLLMs(Sec. 3). Experimental results on text-only subjective evaluation benchmarks (Dubois et al., 2024; Li et al., 2024b) revealed that MLLMs exhibited a sub-

1 Introduction

With the rapid advancement of large language models (LLMs) (OpenAI, 2023a; Touvron et al., 2023), multi-modal large language models
stantial performance drop compared to their corresponding LLMs. A plausible hypothesis suggests that MLLMs suffer from catastrophic forgetting during vision instruction tuning. Following this hypothesis, a straightforward solution would be incorporating both multi-modal and text-only SFT data (Xu et al., 2024; Cao et al., 2025; Ding et al., 2023) for joint learning. However, our experiments (Tab. 2) indicate that this approach not only failed to yield improvements in human preference alignment under multi-modal contexts but also demonstrated negative impacts on the foundational multi-modal skills of MLLMs. These observations suggest that enhancing human alignment in multi-modal scenarios necessitates the development of specialized multi-modal instruction tuning datasets.

Existing multi-modal instruction tuning datasets predominantly focus on foundational capabilities, featuring simple language patterns and uniform response styles (Fig. 5 in appendix). This study argues that effective multi-modal datasets for enhancing human preference alignment should incorporate several critical characteristics: open-ended questions, broad topic coverage, diverse response formats (varying in length and style), and strict adherence to instructions. Based on these principles, we constructed OmniAlign-V. In terms of image sources, OmniAlign-V encompasses natural images and infographics such as posters and charts. Furthermore, a novel solution was developed to filter out semantically rich images from natural image collections. With respect to tasks, complex knowledge-based question answering, creative tasks, and reasoning tasks were designed for different image types. Each task category incorporates diverse subtasks, and state-of-the-art MLLMs were leveraged to obtain diverse, high-quality responses. Ultimately, OmniAlign-V comprises ~200K multi-modal instruction tuning samples, exhibiting a significantly different overall data distribution compared to existing multi-modal SFT datasets.

Leveraging OmniAlign-V, comprehensive experiments were conducted to explore its full potential for enhancing human alignment in MLLMs. Integrating OmniAlign-V into the SFT stage of the LLaVA-NeXT (Liu et al., 2024a) structure with InternLM2.5-7B and Qwen2.5-32B yielded significant improvements in human preference alignment across various powerful LLM encoders, including InternLM2.5 (Cai et al., 2024) and Qwen2.5 (Bai et al., 2023a). Furthermore, on ground-truth-based VQA benchmarks like MMMU (Yue et al., 2023) or OCRBench (Liu et al., 2023e), MLLMs fine-tuned with OmniAlign-V displayed comparable or superior performance. Beyond its application in SFT, OmniAlign-V also demonstrated additional value when applied to Direct Preference Optimization (DPO) (Rafailov et al., 2024). Experimental results indicate that implementing OmniAlign-V for DPO yields further improvements in preference alignment, surpassing baseline models finetuned on the same dataset. After the SFT and DPO stage, our LLaVA-Next baseline with Qwen2.5-32B surpasses the state-of-the-art model Qwen2VL-72B (Chen et al., 2024b) finetuned on extensive proprietary datasets.

Throughout the exploration, we also observed that existing multi-modal human preference benchmarks (Lu et al., 2024; Qian et al., 2024) lack diversity in image sources, contain repetitive questions, and lack clarity. To address these limitations, we introduce MM-AlignBench, a high-quality benchmark comprising 252 carefully curated samples with diverse image sources and meticulously crafted questions by human annotators, enabling comprehensive evaluation of MLLMs’ alignment with human preferences. MM-AlignBench, along with other MLLM human preference benchmarks, was employed throughout this study for evaluation.

The contributions of this study are as follows:
1. An in-depth investigation into the degradation of MLLM human alignment, analyzing the impact of both text-only and multi-modal tuning data.
2. The introduction of OmniAlign-V, a comprehensive open-ended multi-modal SFT dataset, complemented by OmniAlign-V-DPO. Extensive experiments demonstrate the effectiveness of these datasets in improving human preference alignment.
3. The development of MM-AlignBench, a carefully curated benchmark comprising high-quality, human-annotated samples specifically designed to evaluate MLLMs’ human preference alignment.

2 Related Work

Alignment of LLMs. Alignment, encompassing the ability to follow human instructions and provide helpful assistance (Liu et al., 2024b), has long been a critical aspect of LLMs. Recent works (Wang et al., 2023; Xu et al., 2024; Cao et al., 2025) focused on generating high-quality SFT training data to enhance the alignment of LLMs. Besides, recent benchmarks (Liu et al., 2023c; Li et al., 2023,
2024b) have introduced open-ended and challenging questions to assess the alignment performance of LLMs. However, there is a notable lack of benchmarks designed to evaluate the alignment of MLLMs.

Visual Question Answering Datasets. In the early stages, Visual Question Answering (VQA) datasets were primarily used for basic semantic alignment between images and text (Radford et al., 2021; Li et al., 2022; Zhao et al., 2024). These datasets were relatively simple in structure. Popular VQA datasets (Hudson and Manning, 2019; Antol et al., 2015) predominantly featured straightforward questions that elicited single-word or short-sentence answers. Similarly, visual classification and detection datasets (Lin et al., 2014; Schuhmann et al., 2021; Sharma et al., 2018) typically consisted of brief descriptions that focused on the main object within the image.

Instruction Tuning data. With the rapid development of MLLMs (Chen et al., 2024b; Bai et al., 2023b), the instruction tuning data has gained more attention. LLaVA (Liu et al., 2023b) leveraged advanced LLMs to generate instruction-following formatted data from traditional VQA datasets. Recent works have further expanded this approach by employing MLLM to generate captions (Chen et al., 2023a), complex question-answer pairs (Chen et al., 2024a; Gu et al., 2024), OCR data (Carter, 2024) and math-related data (Shi et al., 2024). Other efforts (Tong et al., 2024; Li et al., 2024a) aggregates multiple publicly available document image datasets into unified resources. However, these datasets primarily focus on fundamental visual capabilities with short dialogues and factual questions, resulting in suboptimal alignment with human preferences.

3 Human Alignment of MLLMs: The Preliminary Study

When handling open-ended and flexible questions involving images, state-of-the-art open-source MLLMs – despite excelling in certain recognition tasks – exhibit significantly weaker alignment with human preferences compared to GPT-4o (Fig. 6 in appendix). We hypothesize that this decline in multi-modal preference alignment stems from a reduction in the language model’s proficiency after the multi-modal SFT stage. To test this hypothesis, we evaluated state-of-the-art MLLMs on text-only human preference alignment benchmark (Liu et al., 2023c; Li et al., 2023, 2024b). Additionally, we constructed a LLaVA (Liu et al., 2023b) baseline using InternLM-2.5-7B, fine-tuned on the LLaVA-Next-778k (Liu et al., 2023a) SFT dataset. As shown in Tab. 1, the LLM’s ability to handle text-only open-ended questions degrades significantly after multi-modal SFT. This degradation may be due to (1) insufficient quantity or quality of text-only samples during multi-modal SFT, or (2) the overly simplistic style of multi-modal fine-tuning data derived from traditional VQA datasets (Goyal et al., 2017; Hudson and Manning, 2019), often consisting of simple questions and short, factual answers.

To assess the impact of the first factor on multi-modal alignment, we examined the role of text-only data in multi-modal SFT. The LLaVA-Next-778K dataset includes approximately 40K text-only samples from ShareGPT (Chiang et al., 2023), which are relatively outdated and of lower quality. To improve alignment, we experimented with replacing these samples with better-quality alternatives and increasing their proportion in the multi-modal fine-tuning data. Specifically, we sampled 40K / 80K instances from two high-quality language SFT datasets: Magpie-Llama3.3 (Xu et al., 2024) and Condor (Cao et al., 2025), using these to replace the original language data in LLaVA-Next. We then evaluated models fine-tuned on these different data mixtures using both text-only benchmarks and multi-modal benchmarks, including WildVision (Lu et al., 2024) for human preference alignment and various benchmarks from OpenVLM Leaderboard (Duan et al., 2024) for assessing various fundamental multi-modal capabilities.*

Results in Tab. 2 reveal several key findings. While MLLMs tuned with higher-quality text-only samples show significant improvements on text-

<table>
  <tr>
    <th>Method</th>
    <th>Type</th>
    <th>AlignBench</th>
    <th>AlpacaEval-V2</th>
    <th>ArenaHard</th>
  </tr>
  <tr>
    <td>InternLM-2.5-7B</td>
    <td>LLM</td>
    <td>6.36</td>
    <td>27.58</td>
    <td>10.06</td>
  </tr>
  <tr>
    <td>InternLM-2.5B</td>
    <td>MLLM</td>
    <td>4.00(26.4%)</td>
    <td>3.35(26.9%)</td>
    <td>4.65(42.8%)</td>
  </tr>
  <tr>
    <td>LLaVA-Internlm</td>
    <td>MLLM</td>
    <td>4.66(-26.7%)</td>
    <td>4.22(-8.47%)</td>
    <td>4.93(-8.18%)</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>LLM</td>
    <td>6.02</td>
    <td>24.47</td>
    <td>3.84</td>
  </tr>
  <tr>
    <td>Qwen2V1-7B</td>
    <td>MLLM</td>
    <td>4.92(-18.3%)</td>
    <td>3.85(-8.34%)</td>
    <td>6.46(-80.3%)</td>
  </tr>
  <tr>
    <td>LLaMA-8B</td>
    <td>LLM</td>
    <td>4.88</td>
    <td>30.19</td>
    <td>3.196</td>
  </tr>
  <tr>
    <td>MiniCPM-V2.5</td>
    <td>MLLM</td>
    <td>3.84(-21.5%)</td>
    <td>7.33(-75.7%)</td>
    <td>8.05(-74.8%)</td>
  </tr>
  <tr>
    <td>InternLM-2.0B</td>
    <td>LLM</td>
    <td>5.49</td>
    <td>43.35</td>
    <td>33.75</td>
  </tr>
  <tr>
    <td>InternLM-2.5B</td>
    <td>MLLM</td>
    <td>4.39(-20.0%)</td>
    <td>5.34(-87.7%)</td>
    <td>11.25(-66.7%)</td>
  </tr>
  <tr>
    <td>Hermes2-Llama3-70b</td>
    <td>LLM</td>
    <td>5.72</td>
    <td>46.09</td>
    <td>57.40</td>
  </tr>
  <tr>
    <td>Hermes2-Llama3-70b</td>
    <td>MLLM</td>
    <td>4.33(-24.3%)</td>
    <td>8.32(-81.9%)</td>
    <td>16.17(-72.3%)</td>
  </tr>
</table>

Table 1: Language Alignment Benchmark Results. After multi-modal SFT, MLLMs demonstrate significant decline in human preference alignment compared to their corresponding LLMs.

*MMVet is open-ended, while others are closed-ended.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Language Benchmarks</th>
    <th colspan="5">Multi-modal Benchmarks</th>
  </tr>
  <tr>
    <th>AlignGen</th>
    <th>AlpacaEval-V2</th>
    <th>ArenaHard</th>
    <th>WildVision</th>
    <th>MMVet</th>
    <th>MMBench-V1.1</th>
    <th>AI2D</th>
    <th>OCRbench</th>
  </tr>
  <tr>
    <td>LLaVA<sup>1</sup>-LLaVANext778k</td>
    <td>4.7</td>
    <td></td>
    <td>21.6</td>
    <td>18.4/55.1</td>
    <td>47.2</td>
    <td>73.7</td>
    <td>74.2</td>
    <td>39.7</td>
  </tr>
  <tr>
    <td>LLaVA<sup>1</sup>-LLaVANext<sub>mm</sub>, 738k-Magpie40k</td>
    <td>4.5 ↑</td>
    <td>65.5 ↑</td>
    <td>44.6 ↑</td>
    <td>16.8/58.9 ↓</td>
    <td>37.2 ↓</td>
    <td>73.1 ↓</td>
    <td>73.4 ↓</td>
    <td>38.7 ↓</td>
  </tr>
  <tr>
    <td>LLaVA<sup>1</sup>-LLaVANext<sub>mm</sub>, 738k-Condor40k</td>
    <td>5.6 ↑</td>
    <td>72.9 ↑</td>
    <td>55.7 ↑</td>
    <td>16.8/57.4 ↓</td>
    <td>38.3 ↓</td>
    <td>72.6 ↓</td>
    <td>73.6 ↓</td>
    <td>38.5 ↓</td>
  </tr>
  <tr>
    <td>LLaVA<sup>1</sup>-LLaVANext<sub>mm</sub>, 738k-Condor80k</td>
    <td>5.7 ↑</td>
    <td>75.2 ↑</td>
    <td>55.7 ↑</td>
    <td>16.8/57.0 ↓</td>
    <td>38.3 ↓</td>
    <td>72.6 ↓</td>
    <td>74.0 ↓</td>
    <td>37.6 ↓</td>
  </tr>
</table>

Table 2: Performance of Incorporating High-Quality Language Data. LLaVA<sup>1</sup> denotes LLaVA structure with InternLM2.5-7B as language model, and LLaVANext<sub>mm</sub> 738k refers to the multi-modal data in LLaVANext-778K. Integrating high-quality language data significantly improves alignment performance on language benchmarks. However, it leads to a decline in multi-modal alignment performance on benchmarks such as WildVision and MMVet. For AlpacaEval and ArenaHard, we present the winning rate against GPT-3.5.

only alignment benchmarks, they unexpectedly demonstrate degraded performance on both multi-modal alignment and common VQA benchmarks. This counter-intuitive phenomenon suggests that language alignment capability does not directly translate to multi-modal alignment. We therefore argue that high-quality multi-modal human-aligned data is crucial for improving MLLMs’ human preference alignment in multi-modal contexts.

4 OmniAlign-V

Current MLLM instruction tuning datasets primarily focus on enhancing basic capabilities like perception, OCR, and mathematical reasoning. These datasets typically contain simple, brief question-answer pairs that inadequately capture human preferences and real-world interaction complexity, as shown in Fig. 5. We propose that multi-modal training data should also incorporate: (1) Open-ended, Diverse, and Creative Questions requiring interdisciplinary knowledge, (2) Comprehensive and Knowledge-rich Responses. To address these requirements, we develop a novel data synthesis pipeline to generate high-quality human-aligned multi-modal training data, resulting in the creation of OmniAlign-V.

4.1 Task Taxonomy: A Big Picture

Image content plays a crucial role in constructing multi-modal training data. To ensure comprehensive coverage, we classify images into two major categories: natural images and infographic images, as shown in Fig. 3(a). Our data synthesis pipeline first determines whether an input image belongs to natural images (captured from real-world scenes) or infographic images (human-created to convey information). Based on this classification, different vision-language tasks are assigned.

For natural images, we define three primary tasks: Knowledge, Inferential, and Creation, each requiring diverse and complex question formats with comprehensive, reasoned responses. These tasks enhance the model’s ability to interpret real-world scenes effectively.

For infographic images, given their diverse content, we identify four key types that elicit intricate and challenging questions: Arts, Charts, Diagrams, and Posters. These categories necessitate a deep understanding of human-designed visuals, including both abstract and detailed elements.

4.2 Image Selection Strategy

For natural images, rich semantic content leads to more comprehensive and insightful QA pairs. To enhance training data quality, we developed a pipeline to select semantically rich images from diverse sources, as shown in Fig. 2(a). This pipeline consists of two key steps.

First, we use the Image Complexity (IC) recognition model IC9600 (Feng et al., 2022) to assign IC scores to images. Images with low semantic content — characterized by few objects or simple, uniform backgrounds — receive low IC scores and are then excluded. While IC9600 effectively filters out low-complexity images, high IC scores alone do not guarantee semantic richness. For instance, as shown in Fig. 7, an image filled with tents may have a high IC score but lacks sufficient semantic information for multi-modal training.

To refine selection, we employ the Recognize Anything Model (Zhang et al., 2024) to identify objects within images, filtering out those with high complexity but minimal meaningful content. This two-step approach ensures our pipeline accurately selects images that are both complex and semantically rich. Experiment results have validated the effectiveness of our proposed image selection strategy, as shown in Appx. E.

4.3 Data Generation Pipeline

SFT QA Generation. In Fig. 2(b), we outline the generation process of SFT QA pairs. For vision-language tasks involving natural images
Figure 2: Overall pipeline of OmniAlign-V. By utilizing an image filter and employing a customized pipeline for distinct tasks, we curate semantically rich images paired with high-quality open-ended question-answer sets. Post-refinement further enhances both the variety and quality of our dataset.

(knowledge, creative, inferential), we first apply our image selection strategy to filter semantically rich images from CC-3M(Sharma et al., 2018), Flickr30k (Plummer et al., 2015), and GQA (Hudson and Manning, 2019). For infographic image tasks, images are collected from various existing sources (Masry et al., 2022; Li et al., 2018; Kemhavi et al., 2016). More details are provided in Appx. B.

Knowledge & Inferential Tasks. In preliminary experiments, we found that GPT-4o can generate diverse, content-relevant questions for the two tasks when provided with well-designed few-shot prompts. Consequently, we carefully designed a single prompt for each task category, incorporating comprehensive instructions and selected few-shot examples. These prompts were then used with GPT-4o across diverse images to generate QA pairs.

Creative Tasks. We noticed that a single prompt cannot generate sufficiently diverse, content-relevant creative questions. Therefore, we developed a more sophisticated pipeline inspired by Condor (Cao et al., 2025). First, we created a set of seed creative questions:

\[
Q_s = \{Q_1, Q_2, ... Q_N\}
\]

where each seed question corresponds to a distinct creative task. Since directly using all seed questions as few-shot examples leads to repetition and lack of diversity, we employ a light-weight MLLM (Luo et al., 2024) to generate detailed captions \( C \) for each image. An LLM \( \mathcal{M} \) then selects a relevant subset of seed questions according to the caption:

\[
Q'_s = \mathcal{M}(C, Q_s), \|Q'_s\| \ll \|Q_s\|
\]

Finally, we randomly select three question types from \( Q'_s \) as few-shot examples for GPT-4o, preserving both quality and diversity of synthesized data.

Infographic Tasks. For infographic image tasks (Charts, Diagrams, etc.), questions and answers are closely tied to specific image content. Unlike natural images, these visuals convey information primarily through text, colors, lines, and symbolic elements, making evaluation based on image complexity or object categories unsuitable. Therefore, instead of applying image selection strategies, we carefully select image sources containing rich, detailed information. We then design specialized prompts for GPT-4o to generate questions that require comprehensive background knowledge understanding. The difference is shown below:

InfographicVQA: What is the respiratory disease death rate for individuals aged 70+?
OmniAlign-V: How does the respiratory disease death rate for individuals aged 70+ compare to the other age groups, and what might this suggest?
Figure 3: Data distribution of OmniAlign-V. Our dataset includes a diverse range of tasks, characterized by a more balanced distribution of answer lengths compared to those observed in ALLaVA and ShareGPT4V.

Post Refinement. To further improve the quality of the synthesized data, we implemented a series of post-processing methods for refinement, as illustrated in Fig. 2(c).

Instruction Augmented Knowledge QAs. Instruction following is a crucial capability significantly impacting human preference. To enhance this, we incorporate complex instructions and restrictions into our knowledge QA pairs and reformulate responses accordingly. As shown in Fig. 2(c1), for each knowledge QA, we use a powerful LLM to select an appropriate instruction type that can be integrated into the question without depending on visual content. The instruction is then incorporated into the existing question, and an LLM adjusts the corresponding answer to ensure alignment with both the modified question and original context, resulting in instruction-augmented knowledge QAs.

Enriched Inferential QAs. For many inferential QAs, the answers lack sufficient detail to fully explain underlying logic and background knowledge. To address this, we employ a knowledge-rich LLM to enrich responses with detailed explanations, relevant background information, and logical reasoning (Fig. 2(c2)). This refinement enhances alignment with user preferences.

Quality Improved Infographic QAs. For infographic tasks, particularly those involving Chart data, we observed that even SOTA MLLMs struggle with complex charts and detailed questions. While GPT-4o excels at explaining background knowledge but often produces inaccurate OCR results, SOTA open-source MLLMs (Chen et al., 2024b; Bai et al., 2023b) show superior OCR accuracy but lack detailed explanations. Therefore, we developed a refinement pipeline to generate responses combining rich background knowledge and accurate OCR results. We filter out questions where GPT-4o and SOTA open-source MLLM responses show significant discrepancies in trends. For remaining questions, the responses from different MLLMs are merged to produce a final response that is both precise and comprehensive. The merged answers are further reviewed by human experts to ensure their quality and consistency. More details are demonstrated in Appx. C.

OmniAlign-V comprises 39K Knowledge QAs, 37K Inferential QAs, 10K Creative QAs, 38K Instruction-Following (Knowledge) QAs, and 44K Infographic QAs (2K Art, 8K Diagram, 11K Chart, 23K Poster). Additionally, we prompt GPT-4o to generate 35K QAs focusing on image details, resulting in a total of 205K high-quality SFT training samples. Examples are shown in Figs. 9 to 12.

DPO Data Generation. OmniAlign-V’s high-quality, human-aligned QA pairs can serve as positive samples for DPO training. Inspired by Reject Sampling (Casella et al., 2004), we generate negative samples by prompting a LLava-Next baseline (generator \( G \)) trained on LLava-Next-778k. For each question \( Q_i \), the generator produces \( N \) responses with high temperature, \( \mathbf{R} = \{ r_1^i, r_2^i, ... r_N^i \} \). An LLM Judger \( J \) then evaluates these responses to select the one that most deviates from the original question’s intent and context as the negative sample \( r_{N,eq}^i \), ensuring clear contrast between positive and negative samples.

Figure 4: Samples in MM-AlignBench.

5 MM-AlignBench

Current benchmarks for assessing multi-modal alignment capabilities are limited. While WildVision (Lu et al., 2024) aims to evaluate human preferences in real-world interactions, it employs repetitive and simplistic question formats that inadequately assess response quality, as shown in Fig. 8.
<table>
  <tr>
    <th>Model</th>
    <th>Data</th>
    <th>LLM</th>
    <th>MM-AlignBench</th>
    <th>WildVision</th>
    <th>MIA-Bench</th>
    <th>MMVet</th>
    <th>MMMU</th>
    <th>MMBenchV1.1</th>
    <th>AID2</th>
    <th>OCRbench</th>
  </tr>
  <tr>
    <td rowspan="2">LLaVA</td>
    <td rowspan="2">LLaVANext-778k<br>OmniAlign-V<sub>mix</sub></td>
    <td>InternLM2.5-7B</td>
    <td>3.67 / -8.21</td>
    <td>18.47 / -55.1</td>
    <td>75.4</td>
    <td>41.2</td>
    <td>42.6</td>
    <td>73.6</td>
    <td>74.1</td>
    <td>39.7</td>
  </tr>
  <tr>
    <td>InternLM2.5-7B</td>
    <td>50.0 / +38.8</td>
    <td>28.2 / +24.6</td>
    <td>85.4</td>
    <td>43.5</td>
    <td>43.3</td>
    <td>73.7</td>
    <td>74.7</td>
    <td>41.3</td>
  </tr>
  <tr>
    <td rowspan="2">LLaVA</td>
    <td rowspan="2">LLaVANext-778k<br>OmniAlign-V<sub>mix</sub></td>
    <td>InternLM2.5-7B</td>
    <td>20.6 / +42.7</td>
    <td>23.4 / +45.0</td>
    <td>76.9</td>
    <td>41.8</td>
    <td>44.1</td>
    <td>75.1</td>
    <td>74.7</td>
    <td>56.2</td>
  </tr>
  <tr>
    <td>InternLM2.5-7B</td>
    <td>57.1 / +111.1</td>
    <td>29.6 / +31.3</td>
    <td>86.7</td>
    <td>47.7</td>
    <td>46.8</td>
    <td>74.9</td>
    <td>77.5</td>
    <td>58.9</td>
  </tr>
  <tr>
    <td rowspan="2">LLaVA</td>
    <td rowspan="2">LLaVANext-778k<br>OmniAlign-V<sub>mix</sub></td>
    <td>Qwen2.5-32B</td>
    <td>+ 36.5 / +53.8</td>
    <td>+ 6.2 / +13.7</td>
    <td>+ 9.8</td>
    <td>+ 5.9</td>
    <td>+ 2.7</td>
    <td>- 0.2</td>
    <td>+ 2.8</td>
    <td>+ 2.7</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>26.6 / -29.0</td>
    <td>25.2 / -41.3</td>
    <td>86.0</td>
    <td>47.7</td>
    <td>55.2</td>
    <td>79.3</td>
    <td>79.6</td>
    <td>55.9</td>
  </tr>
  <tr>
    <td rowspan="2">LLaVA</td>
    <td rowspan="2">LLaVANext-778k<br>OmniAlign-V<sub>mix</sub></td>
    <td>Qwen2.5-32B</td>
    <td>62.3 / +19.4</td>
    <td>40.2 / +14.9</td>
    <td>89.6</td>
    <td>56.9</td>
    <td>60.7</td>
    <td>80.6</td>
    <td>81.7</td>
    <td>55.9</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>+ 35.7 / +48.4</td>
    <td>+ 15.0/26.4</td>
    <td>+ 3.6</td>
    <td>+ 9.2</td>
    <td>+ 5.5</td>
    <td>+ 1.3</td>
    <td>+ 2.1</td>
    <td>+ 0.0</td>
  </tr>
</table>

Table 3: Evaluation Results of Integrating OmniAlign-V into the SFT stage. By integrating OmniAlign-V, the multi-modal preference alignment of MLLMs significantly improved. Additionally, we also observe comparable or better performance on common VQA benchmarks. In ‘Model’ column, LLaVA and LLaVANext denote the model structure and training strategy. For MM-AlignBench and WildVision, notation A/B denotes Winning Rate/Rewards.

<table>
  <tr>
    <th>Model</th>
    <th>Type</th>
    <th>AlpacaEvalv2</th>
    <th>ArenaHard v.s. GPT-3.5/GPT-4</th>
  </tr>
  <tr>
    <td>LLaVA</td>
    <td>MLLM</td>
    <td>29.8 / 3.8</td>
    <td>21.4 / 4.93</td>
  </tr>
  <tr>
    <td>LLaVA</td>
    <td>LLM</td>
    <td>50.1 / 7.8</td>
    <td>30.4 / 9.33</td>
  </tr>
  <tr>
    <td>InternLM2.5-7B</td>
    <td>LLM</td>
    <td>78.3 / 26.2</td>
    <td>47.5 / 19.1</td>
  </tr>
  <tr>
    <td>LLaVA</td>
    <td>MLLM</td>
    <td>50.6 / 7.0</td>
    <td>54.5 / 18.0</td>
  </tr>
  <tr>
    <td>LLaVA</td>
    <td>MLLM</td>
    <td>77.6 / 18.0</td>
    <td>87.2 / 58.1</td>
  </tr>
  <tr>
    <td>Qwen2.5-32B</td>
    <td>LLM</td>
    <td>92.1 / 37.1</td>
    <td>94.8 / 75.9</td>
  </tr>
</table>

Table 4: Performance on text-only alignment benchmarks. OA denotes models trained with OmniAlign-V. We present the winning rate against GPT-3.5 and GPT-4 (original setting). OmniAlign-V can also enhance MLLM’s performance on text-only alignment benchmarks.

To address this, we developed MM-AlignBench, featuring human-curated images and questions for more nuanced evaluation.

We selected high-quality images from SAM-1B (Kirillov et al., 2023), CC-3M-Test (Sharma et al., 2018), AID2 (Kembhavi et al., 2016), ChartQA (Masry et al., 2022), and InfographicVQA (Mathew et al., 2022). For natural images, we applied our image selection strategy from Sec. 4.2 to identify 2,000 semantically rich images and combine them with 1,000 carefully selected infographics images. GPT-4o was then used to generate diverse questions for these images. Human experts reviewed and refined the image-question pairs, filtering out low-quality, repetitive, or contextually weak samples. This process resulted in 252 high-quality question-image pairs, featuring diverse question types and semantically rich images. Several examples are shown in Fig. 4.

For evaluation, we follow WildVision’s approach, using GPT-4o as the judge model to compare model responses with reference responses generated by Claude3V-Sonnet (Anthropic, 2024), reporting winning rates and reward scores.

6 Evaluation Results

6.1 SFT with OmniAlign-V

We conduct extensive experiments to demonstrate OmniAlign-V’s effectiveness. We combine OmniAlign-V with LLaVA-Next-778k (excluding text-only samples), creating OmniAlign-V<sub>mix</sub> with 946K training samples. We evaluate various MLLMs tuned on OmniAlign-V against their counterparts tuned on LLaVA-Next-778k.

Our evaluation spans multiple multi-modal benchmarks, including standard VQA benchmarks (Yu et al., 2023; Liu et al., 2024c, 2023e; Yue et al., 2023; Kembhavi et al., 2016) and human-preference alignment benchmarks: MM-AlignBench, WildVision (Lu et al., 2024), and MIA-Bench (Qian et al., 2024). Results in Tab. 3 show that OmniAlign-V significantly improves human alignment across all benchmarks. Moreover, our training data improves general multi-modal capabilities, particularly on MMVet and MMMU, demonstrating a trend distinct from text-only data.

Notably, despite excluding language samples from training data, models maintain stronger language alignment than those trained on LLaVA-Next-778k, as shown in Table 4. This suggests that while high-quality language data alone may not significantly impact multi-modal capabilities, enhancing multi-modal data quality can improve both language and multi-modal performance, highlighting the crucial role of high-quality, human-aligned multi-modal training data.

6.2 DPO with OmniAlign-V-DPO

We conduct DPO post-training on three models: LLaVA-Next trained with LLaVA-Next-778k, LLaVA-Next trained with OmniAlign-V<sub>mix</sub>, and InternVL2-8B. Results in Tab. 5 show that DPO tuning significantly improves performance on real-world questions (WildVision) across all models.
<table>
  <tr>
    <th>Model</th>
    <th>Stage</th>
    <th>MM-AlignBench</th>
    <th>WildVision</th>
  </tr>
  <tr>
    <td>LLaVANext<sup>†</sup></td>
    <td>SFT</td>
    <td>9.5 / -69.2</td>
    <td>30.4 / -34.2</td>
  </tr>
  <tr>
    <td>LLaVANext<sup>†</sup></td>
    <td>SFT+DPO</td>
    <td>11.1 / -64.5</td>
    <td>35.5 / -23.4</td>
  </tr>
  <tr>
    <td>LLaVANext<sup>†</sup>-OA</td>
    <td>SFT</td>
    <td>57.1 / +11.1</td>
    <td>29.6 / -31.3</td>
  </tr>
  <tr>
    <td>LLaVANext<sup>†</sup>-OA</td>
    <td>SFT+DPO</td>
    <td>64.3 / +22.4</td>
    <td>41.8 / -10.1</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>SFT</td>
    <td>31.4 / -21.8</td>
    <td>48.6 / +1.4</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>SFT+DPO</td>
    <td>64.7 / +19.4</td>
    <td>51.4 / +1.9</td>
  </tr>
</table>

Table 5: Performance of applying DPO with OmniAlign-V-DPO. For models finetuned with human-aligned data, by employing DPO training, the model’s alignment with human preference further improved.

<table>
  <tr>
    <th>Model</th>
    <th>Win Rate</th>
    <th>Reward</th>
    <th>B+</th>
    <th>B-</th>
    <th>T</th>
    <th>W</th>
    <th>W+</th>
    <th>W-</th>
  </tr>
  <tr>
    <td>Claude3.5V-Sonnet</td>
    <td>8.7</td>
    <td>+51.7</td>
    <td>10</td>
    <td>13</td>
    <td>13</td>
    <td>25</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>81.3</td>
    <td>+49.0</td>
    <td>81</td>
    <td>124</td>
    <td>12</td>
    <td>31</td>
    <td>4</td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4v</td>
    <td>82.5</td>
    <td>+46.0</td>
    <td>57</td>
    <td>151</td>
    <td>12</td>
    <td>31</td>
    <td>1</td>
    <td></td>
  </tr>
  <tr>
    <td>GeminiFlash1.5-002</td>
    <td>77.0</td>
    <td>+39.1</td>
    <td>56</td>
    <td>138</td>
    <td>14</td>
    <td>35</td>
    <td>9</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaVANext-OA-32B-DPO</td>
    <td>74.2</td>
    <td>+36.9</td>
    <td>49</td>
    <td>138</td>
    <td>20</td>
    <td>40</td>
    <td>5</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2VL-72B</td>
    <td>61.5</td>
    <td>+21.6</td>
    <td>43</td>
    <td>112</td>
    <td>15</td>
    <td>75</td>
    <td>7</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaVANext-OA-32B</td>
    <td>62.3</td>
    <td>+19.4</td>
    <td>31</td>
    <td>126</td>
    <td>19</td>
    <td>62</td>
    <td>14</td>
    <td></td>
  </tr>
  <tr>
    <td>Claude3.5V-Sonnet</td>
    <td>50</td>
    <td>-</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2VL-7B</td>
    <td>44.4</td>
    <td>-5.8</td>
    <td>28</td>
    <td>84</td>
    <td>5</td>
    <td>101</td>
    <td>34</td>
    <td></td>
  </tr>
  <tr>
    <td>InternVL2-72B</td>
    <td>44.4</td>
    <td>-6.9</td>
    <td>19</td>
    <td>93</td>
    <td>8</td>
    <td>98</td>
    <td>34</td>
    <td></td>
  </tr>
  <tr>
    <td>InternVL2-8B-MPO</td>
    <td>40.1</td>
    <td>-10.9</td>
    <td>26</td>
    <td>75</td>
    <td>10</td>
    <td>100</td>
    <td>41</td>
    <td></td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>31.3</td>
    <td>-21.8</td>
    <td>18</td>
    <td>61</td>
    <td>15</td>
    <td>109</td>
    <td>49</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaMA2-Vision-11B</td>
    <td>27.8</td>
    <td>-33.7</td>
    <td>18</td>
    <td>52</td>
    <td>4</td>
    <td>98</td>
    <td>86</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaVANext-Qwen32B</td>
    <td>26.6</td>
    <td>-29.0</td>
    <td>16</td>
    <td>51</td>
    <td>10</td>
    <td>101</td>
    <td>54</td>
    <td></td>
  </tr>
  <tr>
    <td>LLaVA-OneVision-7B</td>
    <td>23.8</td>
    <td>-46.2</td>
    <td>14</td>
    <td>71</td>
    <td>7</td>
    <td>75</td>
    <td>16</td>
    <td></td>
  </tr>
  <tr>
    <td>MegaCPM-V2-5</td>
    <td>12.7</td>
    <td>-53.0</td>
    <td>9</td>
    <td>23</td>
    <td>8</td>
    <td>116</td>
    <td>96</td>
    <td></td>
  </tr>
  <tr>
    <td>Xcompos2.5-7B</td>
    <td>7.5</td>
    <td>-74.0</td>
    <td>5</td>
    <td>14</td>
    <td>3</td>
    <td>63</td>
    <td>167</td>
    <td></td>
  </tr>
  <tr>
    <td>Idefics3-8B</td>
    <td>2.7</td>
    <td>-92.3</td>
    <td>3</td>
    <td>4</td>
    <td>0</td>
    <td>15</td>
    <td>230</td>
    <td></td>
  </tr>
</table>

Table 6: Performance of existing MLLMs on MM-AlignBench. B+/B/T/W/W+/W- denotes MuchBetter/Better/Tie/Worse/MuchWorse. Our LLaVA-Next-OmniAlign(OA)-32B-DPO, trained with OmniAlign-V and applied DPO with OmniAlign-V-DPO, demonstrates outstanding performance, surpassing a wide range of strong MLLMs, even Qwen2VL-72B.

While the baseline trained solely on LLaVA-Next-778k shows minimal improvement on MM-AlignBench, models incorporating OmniAlign-V during SFT demonstrate substantial gains after DPO. Similarly, InternVL2-8B, a state-of-the-art MLLM partially trained on proprietary multimodal corpora, shows significant improvement on MM-AlignBench post-DPO. This indicates that if a model has been trained on data aligned with human preferences, such as open-ended or long-context data during SFT phase, the subsequent DPO training on high-quality human-aligned data can significantly activate the model’s ability, leading to a considerable improvement in alignment performance. In contrast, if the model has not been exposed to such alignment-focused datasets during SFT, training with open-ended data alone via DPO will not significantly improve its capabilities of alignment. These findings demonstrate the value of OmniAlign-V in both SFT and DPO stages for enhancing human preference alignment.

<table>
  <tr>
    <th>Model</th>
    <th>MM-AlignBench</th>
    <th>WildVision</th>
    <th>MMVet</th>
  </tr>
  <tr>
    <td>LLaVANext-778k</td>
    <td>20.6 / -42.7</td>
    <td>23.4 / -45.0</td>
    <td>41.7</td>
  </tr>
  <tr>
    <td>+Know / Infer / Detail</td>
    <td>23.4 / -42.1</td>
    <td>23.2 / -43.2</td>
    <td>42.8</td>
  </tr>
  <tr>
    <td>+Instruction Following</td>
    <td>36.5 / -17.3</td>
    <td>26.2 / -37.5</td>
    <td>44.6</td>
  </tr>
  <tr>
    <td>+Creation</td>
    <td>43.7 / -5.0</td>
    <td>26.6 / -37.7</td>
    <td>44.4</td>
  </tr>
  <tr>
    <td>+Chart / Diagram / Poster</td>
    <td>57.1 / +11.1</td>
    <td>29.6 / -31.3</td>
    <td>47.7</td>
  </tr>
</table>

Table 7: Ablation of Subsets in OmniAlign-V. By progressively incorporating tasks within OmniAlign-V, the model’s alignment performance gradually improves.

6.3 MM-AlignBench

We evaluate various state-of-the-art MLLMs (Ope-nAI, 2023a; Team et al., 2023; Anthropic, 2024; Bai et al., 2023b; Chen et al., 2024b; Li et al., 2024a; OpenBMB, 2024; Dong et al., 2024; Meta, 2024; Laurencon et al., 2024; Wang et al., 2024) on MM-Alignbench, with results shown in Tab. 6. Closed-source models like GPT, Claude, and Gemini demonstrate strong alignment in responding to open-ended questions. In contrast, while Qwen2VL and InternVL2 excel at common VQA tasks, they show relatively lower human preference alignment. This highlights the importance of prioritizing MLLM alignment for improved daily human interactions. Our LLaVA-OA-32B, trained with OmniAlign-V, achieves exceptional performance, outperforming numerous strong MLLMs and nearly matching Qwen2VL-72B. After applying DPO with OmniAlign-V-DPO, LLaVA-OA-32B-DPO achieves winning rate of 72.6 with an average reward of +33.5, surpassing the performance of Qwen2VL-72B. These results highlight the high quality and effectiveness of the OmniAlign-V dataset in improving model alignment with human preferences.

6.4 Ablation Study

We conduct an ablation study to evaluate each subset of OmniAlign-V, reporting results on MM-Alignbench, WildVision, and MMVet in Tab. 7. Performance improves progressively as different tasks from OmniAlign-V are incorporated. Notably, Instruction-Following data significantly enhances performance across all three benchmarks, demonstrating its crucial role. The creation data subset uniquely improves performance on MM-Alignbench but not on WildVision and MMVet. This discrepancy can be attributed to the absence of multi-modal creative question types in these two benchmarks, suggesting their incompleteness in capturing full spectrum of alignment challenges.
7 Conclusion

In this paper, we introduce OmniAlign-V, a dataset designed to enhance the alignment of MLLMs with human preferences, as well as MM-AlignBench, a high-quality, specific-purpose benchmark for evaluating such alignment. We investigate the impact of both language and multi-modal training data, emphasizing the critical role of multi-modal open-ended training data. By incorporating OmniAlign-V into SFT and DPO stages, we achieve significant improvements in the alignment of MLLMs.

8 Limitation

Although the OmniAlign-V pipeline can be easily scaled to support larger datasets, the scale of the dataset used in this paper may be insufficient for large-scale training due to cost limitations. Deeper exploration into the alignment of MLLMs is still needed to address these limitations and further advance the field.

9 Acknowledgements

This research was partly supported by grants of National Natural Science Foundation of China (NSFC, Grant No. 62171281), Science and Technology Commission of Shanghai Municipality (STCSM, Grant Nos. 20DZ1200203, 2021SHZDX0102), National Key R&D Program of China (No.2022ZD0161600), the Shanghai Postdoctoral Excellence Program (No.2023023), China Postdoctoral Science Fund (No.2024M751559), and Shanghai Artificial intelligence Laboratory.

References

Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Lui Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023a. Qwen technical report. arXiv preprint arXiv:2309.16609.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023b. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoan Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zhang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. 2024. Intermlm2 technical report. Preprint, arXiv:2403.17297.

Maosong Cao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Haodong Duan, Songyang Zhang, and Kai Chen. 2025. Condor: Enhance llm alignment with knowledge-driven data synthesis and refinement. arXiv preprint arXiv:2501.12273.

Jimmy Carter. 2024. Textocr-gpt4v. https://huggingface.co/datasets/jimmycartert/textocr-gpt4v.

George Casella, Christian P Robert, and Martin T Wells. 2004. Generalized accept-reject sampling schemes. Lecture notes-monograph series, pages 342–347.

Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. 2024a. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. Preprint, arXiv:2402.11684.

Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023a. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi
Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. 2024b. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Preprint, arXiv:2404.16821.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233.

Xiaoyi Dong, Pan Zhang, Yuhang Zhang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yiming Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. 2024. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420.

Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zhang, Pan Zhang, Jiaqi Wang, et al. 2024. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM international conference on multimedia, pages 11198–11201.

Yann Dubois, Balazs Galambosi, Percy Liang, and Tat-sunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475.

Tinglei Feng, Yingjie Zhai, Jufeng Yang, Jie Liang, Deng-Ping Fan, Jing Zhang, Ling Shao, and Dacheng Tao. 2022. Ic9600: a benchmark dataset for automatic image complexity assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):8577–8593.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.

Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. 2024. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410.18558.

Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026.

Hugo Laurencón, Andrés Marafioti, Victor Sanh, and Léo Tronchon. 2024. Building and better understanding vision-language models: insights and future directions. Preprint, arXiv:2408.12637.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. 2024a. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR.

Juzheng Li, Hang Su, Jun Zhu, Siyu Wang, and Bo Zhang. 2018. Textbook question answering under instructor guidance with memory networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3655–3663.

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024b. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939.

Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and
Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024a. Llava-next: Improved reasoning, ocr, and world knowledge.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. arXiv preprint arXiv:2304.08485.

Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. 2023c. Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18743.

Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klocchkov, Muhammad Faizal Taufiq, and Hang Li. 2024b. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment. Preprint, arXiv:2308.05374.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2023d. Mm-bench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2024c. Mm-bench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer.

Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. 2023e. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.

Yujiie Lu, Dongfu Jiang, Wenhui Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. 2024. Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint arXiv:2406.11069.

Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jifeng Dai, Yu Qiao, and Xizhou Zhu. 2024. Mono-intervl: Pushing the boundaries of monolithic multi-modal large language models with endogenous visual pre-training. arXiv preprint arXiv:2410.08202.

Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.

Minesh Mathew, Viraj Bagal, Rubén Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706.

Meta. 2024. Llama3.2-vision.

OpenAI. 2023a. Chatgpt. https://openai.com/blog/chatgpt.

OpenAI. 2023b. Gpt-4 technical report. ArXiv, abs/2303.08774.

OpenBMB. 2024. Minicpm: Unveiling the potential of end-side large language models.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641–2649.

Yusu Qian, Hanrong Ye, Jean-Philippe Fauconnier, Peter Grasch, Yinfei Yang, and Zhe Gan. 2024. Mia-bench: Towards better instruction following evaluation of multimodal llms. arXiv preprint arXiv:2407.01509.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Rafael Raffailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Itsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL.

Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. 2024. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai; Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

WikiArt Volunteer Team. 2025. Wikiart.

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. 2024. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235.

Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. 2024. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442.

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2023. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502.

Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. 2024. Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1724–1732.

Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, and Haian Huang. 2024. An open and comprehensive pipeline for unified object grounding and detection. arXiv preprint arXiv:2401.02361.
A Alignment of MLLMs with Human Preference

Although open-source MLLMs have already matched or even surpassed proprietary models like the GPT and Claude series in common VQA tasks like OCR and Visual Perception, a significant gap remains in their alignment with human preferences. When posed with open-ended questions that require knowledge-rich responses, even the most advanced open-source MLLM, InternVL2-76B, struggles to provide comprehensive answers with high readability, as illustrated in Fig. 6. In contrast, GPT-4o not only accurately identifies the main objects relevant to the question but also provides well-structured responses enriched with comprehensive contextual knowledge, achieving a high level of alignment with human preferences.

B Image sources

We carefully select image sources for Arts, Charts, Diagrams, and Posters tasks. For Arts, WikiArt (Team, 2025) is selected as the image source, offering a diverse range of painting styles. We uniformly sample 2000 images across all painting styles to ensure diversity in the dataset. For Charts, we select ChartQA (Masry et al., 2022), a dataset featuring charts that contain substantial statistical information. ChartQA includes several subcategories, from which we filter out simplistic charts with only two columns and retain those that contain charts with rich contextual information and diverse types. For Diagrams, we choose images from TextbookVQA (Team, 2025), which provides diagrams rich in natural content and detailed explanations. We exclusively utilize the question images and teaching images from the image sources, as they meet our specific requirements. For Posters, we utilize InfographicsVQA (Mathew et al., 2022), a dataset containing high-quality posters with intricate designs and informative content.

C Chart Post-Refine Pipeline

After curating images along with high-quality question, we first employ the current most powerful models(GPT-4o, InternVL2-72B, Qwen2VL-72B) to generate answers separately. A powerful LLM(Qwen2.5-72B) is then utilized to extract objective facts from the chart within each generated answer. The facts extracted from the answers of multiple models are compared for consistency. If the facts differ significantly and lead to entirely different conclusions, such responses are flagged for further review or discarded to avoid misinformation. For cases where the facts exhibit only minor differences, we merge the detailed factual content from Qwen2VL-72B into the comprehensive explanations provided by GPT-4o. The merged answers are further reviewed and supervised by human experts to ensure their quality and consistency.

D Training Details

Our training strategy largely follows the approaches adopted by LLaVA and LLaVANext. CLIP-Large-336-Patch14 is employed as the visual encoder. In line with the LLaVA training strategy, we first conduct a pretraining stage where both the visual encoder and the LLM are frozen. We utilize the LLaVA-pretrain-558k and ALLaVA-pretrain-728k datasets for pretraining. The batch size is uniformly set to 256 and learning rate is set to 1e-3. During this phase, images are resized to 336x336, and no image-splitting method is applied.

For SFT stage, we unfreeze the LLM for LLaVA and further unfreeze the visual encoder for LLaVANext. In the case of LLaVANext, we apply the image-splitting method, setting the maximum split size to 3x3. The batch size is uniformly set to 128 and learning rate is set to 2e-5.. LLaVA-InternLM2.5-7B is trained using 8xA800 GPUs for 12 hours. LLaVANext-InternLM2.5-7B is trained using 16xA800 GPUs for 13 hours. LLaVANext-Qwen2.5-32B is trained using 32xA800 GPUs for 24 hours.

![Examples of limitation with current multi-modal instruction tuning dataset.](page_1012_120_393_393.png)

Figure 5: Examples of limitation with current multi-modal instruction tuning dataset.
<table>
  <tr>
    <th>Model</th>
    <th>MM-AlignBench</th>
    <th>WildVision</th>
    <th>MMVet</th>
  </tr>
  <tr>
    <td>LLaVANext-77k</td>
    <td>15.1/-52.6</td>
    <td>13.6 / -63.1</td>
    <td>37.7</td>
  </tr>
  <tr>
    <td>+ 33k w.o. Imager Filter</td>
    <td>31.4 / -42.3</td>
    <td>22.0 / -42.3</td>
    <td>42.0</td>
  </tr>
  <tr>
    <td>+ 33k w. Imager Filter</td>
    <td>35.3 / -41.0</td>
    <td>22.6 / -37.5</td>
    <td>44.4</td>
  </tr>
</table>

Table 8: Ablation study on the impact of utilizing image filter.

E  Ablation on Image Filter

We conduct ablation studies to evaluate the impact of using our image filter. We randomly sample 77k images from the LLaVA-Next-778K SFT dataset as a baseline and separately select images from CC3M both with and without applying our image filter. Subsequently, the selected images are used to generate Knowledge and Inferential Question-Answer pairs following the pipeline described in Sec. 4.3. We then assess the effect of these datasets on the performance of LLaVA-Next-InternLM2.5-7B, with the results presented in Tab. 8. It can be observed that by utilizing the image filter, the selected images contain richer semantic information, leading to the generation of higher-quality image-question pairs. This, in turn, enhances the model’s performance in terms of alignment with human preferences.

F  Human Experts

In this study, two authors (both PhD students in computer science) serve as human experts. They are responsible for reviewing and refining the questions in MM-AlignBench, as well as evaluating and filtering incorrect merged cases following the Chart Post-Processing Refinement stage.

G  License

The InternLM and Qwen models are licensed under the Apache-2.0 license. The ChartQA dataset is distributed under the GNU General Public License v3.0. The remaining datasets are licensed under CC BY-NC 4.0, which permits only non-commercial use.
Text Prompt
How does the loading and unloading process depicted in the image contribute to international shipping and trade?
What are the potential logistical challenges faced at such ports?

InternVL2-76B:
Ambiguous | Ignorance | Simplistic

The loading and unloading process shown in the image is a critical component of international shipping and trade, as it facilitates the movement of goods between different countries and regions. The use of cranes and trucks allows for the efficient transfer of cargo from ships to land transport, and vice versa. This process is essential for the global supply chain, enabling the distribution of goods to various markets.

Potential logistical challenges at such ports include coordinating the arrival and departure of ships, managing the flow of trucks and other vehicles, ensuring the safety and security of cargo, and dealing with weather conditions that can impact operations. Additionally, there may be challenges related to customs clearance, documentation, and compliance with international trade regulations. The image suggests a well-organized operation, but these challenges are inherent to the complex and dynamic environment of international shipping and trade.

GPT-4o:
Clarity | Knowledge-Rich | Comprehensive

The loading and unloading process depicted in the image is a critical component of international shipping and trade. This process involves transferring goods between ships and other forms of transportation, such as trucks, which facilitates the movement of goods across different regions and countries. Here’s how it contributes to international shipping and trade, along with potential logistical challenges:

Contribution to International Shipping and Trade:
1. Efficient Movement of Goods:
The process ensures that goods are efficiently transferred from ships to trucks or vice versa, enabling the seamless flow of products from manufacturers to consumers across the globe.
2. Intermodal Transportation:
Ports facilitate hub-and-spoke intermodal transportation, where goods are transferred between different modes of transport (e.g., sea to land). This interconnectivity is essential for global supply chains.
3. Economic Growth:
Ports facilitate international trade, which is a significant driver of economic growth. Efficient port operations can enhance a country’s trade capabilities and competitiveness.
4. Supply Chain Management:
Effective loading and unloading processes are crucial for maintaining the integrity and timing of supply chains, ensuring that goods reach their destinations on schedule.

Potential Logistical Challenges:
1. Congestion:
Ports can become congested, especially during peak times, leading to delays in loading and unloading. This can disrupt supply chains and increase costs.
2. Infrastructure Limitations:
Inadequate infrastructure, such as insufficient cranes, storage facilities, or transportation links, can hinder the efficiency of port operations.
3. Customs and Regulatory Compliance:
Navigating customs and regulatory requirements can be complex and time-consuming, potentially causing delays in the movement of goods.
4. Labor Issues:
Labor shortages, strikes, or disputes can impact port operations, leading to delays and increased costs.
5. Weather Conditions:
Adverse weather conditions can disrupt port activities, affecting the loading and unloading process and causing delays.
6. Security Concerns:
Ensuring the security of goods and preventing theft or damage is a significant challenge. Ports must implement robust security measures to protect cargo.
7. Technological Integration:
Integrating advanced technologies for tracking, managing, and automating port operations can be challenging but is necessary for improving efficiency and reducing errors.
8. Environmental Regulations:
Ports must comply with environmental regulations, which can impact operations and require investments in cleaner technologies and practices. Addressing these challenges requires coordinated efforts between port authorities, shipping companies, logistics providers, and government agencies to ensure smooth and efficient port operations, which are vital for sustaining international trade and economic growth.

Figure 6: GPT-4o shows superior alignment with human preference than InternVL2-76B.
IC Score=0.10    Low IC
IC Score=0.89    High IC
RAM Count=6     Low RAM Count
IC Score=0.69    High IC
RAM Count=12    High RAM Count

Meaningless   ✗
High Content Redundancy   ✗
Rich Semantic Information   ✔

Figure 7: Demonstration examples of our image filter.

Text Prompt:
How many chairs are in this pic?
Reference Answer:
The image shows 4 chairs in what appears to be a waiting or reception area outside an "H&R Block" office or service location.

Simplistic & Factual Question

Text Prompt:
What is this?
Reference Answer:
The image depicts a dense forest of coniferous trees, likely pines or firs, characterized by their tall, slender trunks and conical shapes. The trees are uniformly green, suggesting they are healthy and possibly located in a temperate climate where such trees thrive. The forest appears lush and undisturbed, indicating it may be a protected natural area or a well-maintained park. The perspective shows ground level looking upwards into the canopy, which emphasizes the height and density of the trees. There are no visible signs of human activity or structures, reinforcing the impression of a pristine, natural environment.

Meaningless Image & Question

Text Prompt:
a chef standing in a kitchen, pizza's recepie such as flower and tomatoes are on top of the shelves, a pizza on the background being cooked inside an oven
Reference Answer:
The image does not depict a chef standing in a kitchen with pizza ingredients or an oven cooking pizza as you described. Instead, the image shows a close-up portrait of an older man with gray hair and wrinkled features. He is wearing a white chef's uniform or jacket and has a serious, slightly stern expression on his face. The background is a solid dark gray or black color, putting the focus entirely on the man's face and upper body.

Confusing Prompt

Figure 8: Examples of limitation within current multi-modal benchmark for alignment.
Knowledge

Please introduce the history of the landmark in the image, and suggest a culturally enriching leisure activity plan for visitors.

### History of the Landmark
The landmark in the image is the Colosseum, an iconic symbol of ancient Rome located in the heart of the city. Constructed between 70-80 AD under the reign of Emperor Vespasian and completed by his successor Titus, the Colosseum was the largest amphitheater ever built at the time and could hold between 50,000 to 80,000 spectators.

It was primarily used for gladiatorial contests and public spectacles such as animal hunts, mock sea battles, and executions. Despite suffering damage from natural disasters and stone robbers, the Colosseum remains a testament to Roman engineering and architectural prowess.

### Culturally Enriching Leisure Activity Plan
#### Morning: Guided Historical Tour
Start your day with a guided tour of the Colosseum. A knowledgeable guide will provide insights into the architectural techniques used in its construction and recount tales of the gladiators who fought within its walls. This tour will help visitors appreciate the historical significance and the cultural heritage of the Colosseum.

#### Afternoon: Roman Forum Exploration
After the Colosseum, head to the nearby Roman Forum, once the center of Roman public life. Walk through the ruins and imagine the political, religious, and economic activities that took place here. Consider using an audio guide or a historian-led tour for deeper understanding.

#### Evening: Traditional Italian Dining Experience
Conclude your day with a dinner at a nearby trattoria, indulging in traditional Italian cuisine. Opt for dishes such as pasta alla carbonara or cacio e pepe, paired with a glass of local wine. End your meal with a classic tiramisu or gelato, savoring the flavors of Rome.

This plan not only immerses visitors in the rich history of the Colosseum and its surroundings but also offers a taste of authentic Roman culture.

Inferential

What occasion are the people in the image likely celebrating?

The people in the image are likely celebrating a holiday, most probably Christmas. This conclusion is drawn from several key elements visible in the scene:
- Wrapped Gifts: The presence of wrapped gifts suggests a gift-giving occasion, which is a common tradition during Christmas.
- Stockings on the Fireplace: Stockings hung by the fireplace are a traditional Christmas decoration, often used for Santa Claus to fill with small gifts and treats.
- Festive Decorations: The overall festive atmosphere, including other possible decorations like a Christmas tree or lights, further supports the idea that this is a Christmas celebration.

These elements together create a clear picture of a family or group gathered to enjoy the warmth and joy of the holiday season.

Creation

Imagine you are the athlete in this image. Describe your thoughts and emotions as you prepare for an important game. What motivates you to give your best performance on the ice?

Standing by the rink, I feel the electric buzz of anticipation coursing through my veins. The cool air of the arena sharpens my focus, while the roar of the crowd fuels the fire within me. Wearing the Vancouver jersey, I am more than just another athlete on the ice; I am part of something greater, a legacy I am determined to uphold.

As I grip my stick, memories of early mornings on frozen ponds flash through my mind. Each moment, each sacrifice, has led me to this pivotal game. My heart beats in sync with the rhythm of skates slicing through ice, a reminder that this is where I belong. The motivation to perform doesn’t just come from a desire to win, but from the faces of my teammates who have become family, the unwavering support from fans, and the pride in representing my city. I visualize each play, every pass, every shot, letting the anticipation transform into confidence.

Tonight, as the whistle blows and the puck drops, I am ready to give everything I have. This is my stage, my moment, and I intend to seize it with every ounce of passion in me.
Chart

Coal reserves, 2019
Proved reserves is generally taken to be those quantities that geological and engineering information indicates with reasonable certainty can be recovered in the future from known reservoirs under existing economic and operating conditions.

<table>
  <tr>
    <th></th>
    <th>2019</th>
    <th>2018</th>
  </tr>
  <tr>
    <td>Brazil</td>
    <td>6.6 billion t</td>
    <td>6.6 billion t</td>
  </tr>
  <tr>
    <td>Czechia</td>
    <td>2.93 billion t</td>
    <td>2.93 billion t</td>
  </tr>
  <tr>
    <td>Uzbekistan</td>
    <td>1.38 billion t</td>
    <td>1.38 billion t</td>
  </tr>
</table>

What is the relationship between the coal reserves of Brazil, Czechia, and Uzbekistan, and how might this affect their energy policies?

The coal reserves of Brazil, Czechia, and Uzbekistan differ significantly, which may influence their energy policies and strategies:

### 1. Coal Reserves Overview
- Brazil: 6.6 billion tons (largest reserves among the three).
- Czechia: 2.93 billion tons (moderate reserves).
- Uzbekistan: 1.38 billion tons (smallest reserves).

### 2. Implications for Energy Policies
- Brazil:
  - Its large coal reserves might provide a reliable domestic energy source for power generation and industry.
  - However, Brazil has a strong focus on renewable energy (e.g., hydropower) and may prefer to limit coal use to reduce environmental impacts.
  - Coal could be used as a backup or exported for economic gains.
- Czechia:
  - With moderate reserves, coal has traditionally played a key role in Czechia’s energy mix, particularly for electricity generation.
  - Czechia faces pressure to transition away from coal due to European Union climate targets, potentially accelerating the shift toward renewables and nuclear energy.
- Uzbekistan:
  - Limited coal reserves may constrain its reliance on coal for energy.
  - Uzbekistan has been investing in natural gas and renewable energy, and its smaller coal reserves may push it to focus more on diversifying its energy sources.

### 3. Global Transition Trends
- All three countries may face international pressure to reduce coal use in favor of cleaner energy sources due to global climate change goals.
- The extent of their reliance on coal will depend on domestic policies, economic priorities, and access to alternative energy resources.

### 4. Export Potential
- Brazil, with the largest reserves, has more potential for coal exports, which could shape its energy trade policies.
- Czechia and Uzbekistan are less likely to focus on coal exports due to smaller reserves and domestic needs.

In summary, Brazil’s larger reserves provide more flexibility for domestic use and export, while Czechia and Uzbekistan’s smaller reserves may push them to prioritize energy diversification and transition to cleaner energy.

Diagram

THE ROCK CYCLE

Weathering
Transportation
Deposition
Sediments
Lithification
Sedimentary rocks
Metamorphism
Metamorphic rocks
Igneous rocks (intrusive)
Melting
Magma
Solidification
Igneous rocks (extrusive)
Uplift and Exposure

How does the relationship between weathering and sediment formation affect the rock cycle?

The relationship between weathering and sediment formation is a vital component of the rock cycle, as it provides the materials necessary for the formation of sedimentary rocks. Here’s how they interact and affect the rock cycle:

1. Weathering Breaks Down Rocks:
- Weathering is the process in which rocks are broken down into smaller particles, such as sediments, through physical, chemical, or biological means.
- Rocks exposed at the Earth’s surface, including igneous, metamorphic, or even sedimentary rocks, are subjected to weathering.

2. Sediment Formation and Transportation:
- The weathered material becomes sediments, which are transported by wind, water, or ice to new locations. This transportation step is essential for distributing sediments across different environments.

3. Deposition and Lithification:
- Sediments are eventually deposited in layers, often in water bodies like rivers, lakes, or oceans. Over time, these sediments are compacted and cemented through lithification to form sedimentary rocks.

4. Sedimentary Rocks and the Rock Cycle:
- Sedimentary rocks can undergo further changes in the rock cycle:
  - They may be subjected to heat and pressure to become metamorphic rocks.
  - They can also be subducted into Earth’s mantle, where they melt and form magma, which solidifies into igneous rocks.

5. Continuous Cycling:
- Weathering and sediment formation ensure that materials from older rocks re-enter the rock cycle, highlighting the interconnectedness of Earth’s processes.

In summary, weathering and sediment formation are essential steps in recycling materials within the rock cycle, connecting the processes that form sedimentary, metamorphic, and igneous rocks.
Poster

SMALL BUSINESSES HAVE A GOLDEN OPPORTUNITY TO SEIZE NEW GROWTH OPPORTUNITIES

DESPISE
2/3 of UK SMEs are not seizing opportunities to grow their business
54% cite lack of opportunity to grow their business

BACKED BY INITIATIVES SUCH AS ...
STARTUP GRANT SCHEME
STAY UP BUSINESS

BUT HOW?
45% want support in managing their business

TO ENABLE GROWTH ...
21% cite changing customer needs and expectations
1/5 cite changing market conditions or challenges from larger competitors
37% cite changing technology and digital trends

69% are looking to win new customers and partners
58% are looking to strengthen relationships with existing customers and partners

TECHNOLOGY IS THE KEY...
2/5 have a low reliable internet connection with the small businesses they serve
63% find advice and support from other small businesses helpful, emphasizing the importance of collaboration and shared insights
58% see social tools as essential for operations
24% are prioritizing investment in technology (e.g., mobile hardware and security systems)
65% of SMEs recognize that technology improves communication with customers

In what ways does the infographic suggest small businesses can differentiate themselves from larger competitors, and what strategies are highlighted as crucial?

The infographic highlights several ways small businesses can differentiate themselves from larger competitors and underscores specific strategies as crucial for growth:

### Ways to Differentiate:
1. Agility in Business Strategy:
   - 21% of small businesses are changing their business strategies to effectively compete with larger companies.
   - The focus is on flexibility and responsiveness to market changes.
2. Expanding Product and Service Offerings:
   - 1 in 5 small businesses now offer additional products or services to attract and retain clients.
3. Geographical Expansion:
   - 37% of SMEs plan to expand into new areas over the coming year to tap into untapped markets.
4. Customer Relationships:
   - 69% prioritize winning new customers and partners.
   - 58% focus on strengthening relationships with existing customers and partners.

### Crucial Strategies:
1. Leveraging Technology:
   - 65% of SMEs recognize that technology improves communication with customers.
   - 24% are prioritizing investment in technology (e.g., mobile hardware and security systems).
2. Social Media and Networking:
   - 58% see social tools as essential for operations.
   - 2 in 5 SMEs plan to use mobile social networking to connect with other small businesses.
3. Community and Support:
   - 63% find advice and support from other small businesses helpful, emphasizing the importance of collaboration and shared insights.

### Key Takeaway:
The infographic emphasizes that investing in technology, adapting strategies, and building strong customer relationships are critical for small businesses to differentiate themselves and thrive in competitive markets.
What are the prominent features of the setting and figures depicted in the painting, and how do they reflect the themes and techniques commonly used by the artist?

The painting 'The Charms of Life' by Antoine Watteau showcases a vivid and lively scene set in a pastoral landscape, typical of Watteau's work. Watteau is known for his 'fêtes galantes' style, which depicts scenes of aristocratic outdoor entertainment.

### Features and Themes:
- **Setting:** The painting features a lush, picturesque countryside, suggesting a leisurely and idyllic lifestyle. The use of soft brushwork and a subtle color palette creates an ethereal, dreamlike quality.
- **Figures:** The elegantly dressed figures are engaged in music and conversation, portraying a sense of enjoyment and harmony. The attire and gestures reflect the social elegance of the Rococo period.
- **Architectural Elements:** The grand columns and structured terrace highlight the blend of nature and refined culture, a common theme in Watteau's works.

### Techniques:
- **Color and Light:** Watteau's use of light captures the luminous quality of the outdoors, enhancing the serene and carefree atmosphere.
- **Composition:** The arrangement of figures in a relaxed, informal manner typifies the fluidity and spontaneity characteristic of Rococo art.
Overall, the painting reflects Watteau's mastery in capturing the fleeting pleasures of life, combining sophisticated elegance with a deep appreciation for nature and human interaction.

What elements in the image reflect urban fashion trends, and how do these elements align with current fashion industry predictions for seasonal attire? Please describe each element using a metaphor and ensure your response includes at least three distinct metaphors.

The image showcases several elements reflective of urban fashion trends, each of which can be described using vivid metaphors to highlight their significance and alignment with current fashion industry predictions for seasonal attire.
1. **Floral Dress:** The central figure's floral dress is like a garden in full bloom, with its bright, bold patterns that capture the essence of spring and summer. This aligns with fashion industry predictions that emphasize vibrant colors and nature-inspired designs as key trends for these seasons. Just as a garden bursts with life and color, the dress brings a sense of freshness and vitality to the urban landscape.
2. **Bright Pink Shoes:** The bright pink shoes are like a neon sign in a bustling city, drawing attention and adding a pop of color to the outfit. This technique of using bold, eye-catching colors is a common strategy in urban fashion to make outfits more dynamic and memorable. The shoes serve as a beacon of style, guiding the eye and adding a modern, urban flair to the ensemble.
3. **Bright Orange Handbag:** The bright orange handbag carried by the person on the left is like a vibrant sunset, blending practicality with style. It complements the idea of combining functional fashion items with bold, statement pieces, a growing trend in urban fashion. Just as a sunset transforms the sky with its warm, radiant hues, the handbag adds a touch of warmth and energy to the overall look, making it both practical and visually striking.
These elements collectively reflect the broader industry move towards integrating bold, statement pieces with functional fashion items that cater to an urban lifestyle, ensuring that the wearer stands out while remaining practical and stylish.

Figure 12: Examples of each task in OmniAlign-V.
Prompt for Knowledge Task

Examine the image provided and generate a knowledge-based and exploratory question based on the content of the image and supply corresponding detailed answers.

Question Guidelines:
- Your question should invite insightful discussion on the types of elements in the image, such as:
  - **Objects**: For example, animals, plants, food, or products.
    - E.g., "What breed of dog is in the picture, and what are their characteristics?", "Can you give me a recipe for the food in image?", "Write a Product Description for the product in the image.".
  - **Locations and Features**: Relevant to countries, landmarks, famous people, or scenic spots.
    - E.g., "Please introduce the history of the landmark in the picture.", "How did the states in image get their names?", "Who is the person in the image and what is him famous for?".
  - **Activities and Technologies**: Related to sports, machines, technology, and environmental details.
    - E.g., "Can you explain how the game in image is played?", "How is the machine shown in the image operated?"
  - **Events, Literature, and Media**: Concerning books, movies, or series in the image.
    - E.g., "Write a short description about the movie or series in the image.", "Think of books that would be enjoyable for someone who liked the books in the image."

Answer Guidelines:
- Ensure your answers are factual and comprehensive.
- Please use Markdown formatting in your text to enhance the content, making it visually appealing and easy to read. Include appropriate headings, subheadings, lists, code blocks, and other Markdown elements to optimize your answers.
Output Format:
Your response should strictly follow this format:

```json
{
  "question": "Question text",
  "answer": "Answer text"
}
```

Figure 13: An Example of the prompt for Knowledge task generation.
Prompt for Creative tasks

You are a skilled writer with a talent for crafting insightful and engaging questions based on the content of a given image.
**Task Guidelines**:
- Analyze the content of the provided image and select one of the question types listed below. Use it to create engaging questions that lead the viewer to explore and interpret the image in different level of complexity.
- Be closely tied to the content of the image, emphasizing its primary visual or thematic elements.
- Your questions should **avoid directly referencing specific details** in the image. Instead, they should encourage deeper reflection, ensuring the question cannot be answered without seeing the image.
- Avoid overly rigid or direct phrasing, focusing instead on open-ended exploration.
**Question Types**
Your questions can be from the following types, each followed by an example with a different level of complexity.
- Simple (basic observation or initial reflection)
- Moderate (more thought-provoking, requiring a deeper understanding and more structured response)
- Difficult (complex or abstract, requiring analysis and strict formatting)
{Match Types}
**Output Format**:
Your response should strictly follow this format:
“json
{
"question": "Question text",
"type": "Question type",
"level": "Question level"
}””

Figure 14: An Example of the prompt for Creative task generation.
Prompt for Inferential tasks

You are an image analysis expert skilled in posing high-quality **inferential questions**. Please provide 2-5 of the most insightful questions you can think of, following these guidelines:
For Questions:
- **Focus on image-based questions:** Ensure that your question cannot be answered without analyzing the image. **You should not directly provide image’s data or details in your generated questions.** For example, "What might be the impact on the radio industry due to 34 stations having their licenses revoked?" includes the data in the image and can be answered without analyzing the image, so it is a bad question.
- Ensure that questions are natural, not overly rigid. **You should be quite certain and confident about the questions you pose and their answers, and avoid using words like "possibly", "maybe" or "might be" in both questions and answers.**
- Your questions must require reasoning beyond the direct content of the image, making reasonable inferences based on the information presented. - The scope of the questions should not be overly broad or delve into political, philosophical, speculative, sensitive, or controversial topics. Stay within the context of the scene and elements inferred from it.
For Answers:
- You should provide a clear and concise answer to the question.
Good Examples:
- What precautions are the people on the boat taking to stay comfortable during the trip? - Is there anything else on the table other than the pizza? - Why do these people choose to dress in this style?
- What decorative element is present in this public restroom that is not typical? Bad Examples:
- What might indicate

Figure 15: An Example of the prompt for Inferential task generation.
Prompt for Chart tasks

You're a great image analyst. You need to analyze the image provided and generate some insightful questions based on the content of the image.
Question generation guidelines:
- Ensure that your questions require the image to be answered and do not include explicit information from the image. Instead, pose questions that prompt the respondent to analyze the image to find the answer.
- Your question should be explainable and require some reasoning to answer.
- Your question could contain different analytical perspectives, such as trends, comparisons, causal inference, etc.
- - Your questions should be insightful but also clear and straightforward. Avoid overly complex or niche questions.
Bad question examples:
- "What might be some factors contributing to the significantly higher private health expenditure per person in Argentina compared to Fiji and Benin?" (This includes specific details from the chart.) Its correct clarification should be "Is there any difference in private health expenditure per person between Argentina, Fiji, and Benin? If so, what might cause the difference?"
- "What trends can be observed in private health expenditures per person among the three countries shown?" (This question is unclear because 'trends between countries' is not a standard analytical concept. Trends typically refer to patterns over time or categories, not direct cross-entity comparisons.)
Output format:
Your response should strictly follow this format:
{
"questions": [
{
"question": "Question text"
},
{
"question": "Another question text"
}
]
}
Prompt for Poster tasks

You’re an excellent image analyst and good at generating insightful questions about the image. You need to analyze the image provided and generate some insightful questions based on the content of the image, and you should answer the questions you generated.

Possible Categories Reference:
1. **Cultural and Social Context**
2. **Analysis and Inference**.
3. **Visual Elements and Design Techniques**.

Question Guidelines:
- **Focus on image-based questions:** Ensure that your question cannot be answered without analyzing the image. **You should not directly provide image’s data or details in your generated questions.** For example, "What might be the impact on the radio industry due to 34 stations having their licenses revoked?" includes the data in the image and can be answered without analyzing the image, so it is a bad question.
- **Encourage thoughtful, structured responses:** Your question should be explainable and need some reasoning to answer. You should not generate questions that are just extracting information from the image. For example, "What percentage of organizations verify the past employment records of new employees according to the image?" is a bad question.
- **Ensure diversity in the questions:** Cover different aspects of the image, encouraging multiple perspectives. You can choose some appropriate categories from the possible categories reference. **For the same category, you can generate multiple questions.**
- **Generate high-quality questions:** You can choose to generate challenging questions, but their answers should be able to clearly explain.

Output Format:
Your response should strictly follow this format:

"questions": [
{
"question": "Question text"
}
]
]

Figure 17: An Example of the prompt for Diagram task generation.
Prompt for Diagram tasks

You’re a great diagram analyst. You need to analyze the diagram provided and generate 2-4 insightful questions based on the content of the diagram.
Question Guidelines:
- Your questions should be guiding and should not directly point to the content. For example: "How does acid rain affect water bodies, soil, and plant life?" should be changed to "How does the process in image affect water bodies, soil, and plant life?"
- Your question should invite insightful discussion, such as:
  - **Interpretation**: Symbol Interpretation, Data Extraction
  - **Examples**:
    - "What is the role of the cytokine-producing cell in the process shown?"
    - "Enumerate the steps outlined in the flowchart."
  - **Relation Analysis**:
    - **Examples**:
      - "How does variable A affect variable B in the diagram?"
      - "How many ways can A to B be achieved in the diagram?"
  - **Inference**:
    - **Examples**:
      - "What can be inferred about the system’s stability from the diagram?"
      - "What does the bacterium do once it has the hybrid plasmid?"
Output Format:
Your response should strictly follow this format:
```
{
  "question":["Question text 1", "Question text 2", ...],
}
```

Figure 18: An Example of the prompt for Diagram task generation.

Reviewer paper summaries: The authors introduce OmniAlign-V, a large-scale dataset comprising 200K diverse images paired with complex, open-ended questions, specifically collected to enhance human-value alignment. Additionally, they propose MM-AlignBench, a new benchmark designed explicitly to assess this alignment in MLLMs. Experimental results demonstrate that fine-tuning MLLMs using OmniAlign-V through Supervised Fine-Tuning (SFT) or Direct Preference Optimization (DPO) can improve their alignment with human preferences.
Summary of strengths: - The authors introduce a comprehensive, diverse dataset (200K samples) explicitly designed to enhance the alignment of MLLMs with human preferences. It consists of semantically rich natural and infographic images with complex, open-ended, knowledge-rich questions.

- The authors introduce a human-annotated, high-quality benchmark targeting human preference alignment evaluation in MLLMs.

- The work is well-motivated, with an insightful preliminary study exploring the current limits of human alignment in existing MLLMs.

- Experimental results convincingly demonstrate that models fine-tuned on OmniAlign-V outperform baseline methods on alignment benchmarks while simultaneously preserving or enhancing their performance on standard VQA benchmarks.
Summary of weaknesses: - A critical concern I have is the method used to generate answers in the dataset. For example, the authors used GPT-4o as the judge model in the proposed benchmark. The authors claim improved alignment with human preferences, but the answers in OmniAlign-V are machine-generated rather than directly collected from human annotators. This raises questions about whether these automatically generated answers genuinely reflect real human values or preferences. The paper does not sufficiently address how generated answers can reliably serve as ground truth for evaluating human preference alignment. 

- As shown in Table 4, even when leveraging the proposed multimodal data, the performance of MLLMs on language benchmarks remains lower than that of language-only LLMs. Can we design improved methods or strategies that better utilize existing language-only datasets, rather than relying primarily on the collection of additional multimodal data?
Comments, suggestions and typos: Please address the key concerns in Weaknesses.
Overall assessment scores: 3.5