Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (2024)

Abstract: Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production. However, the current MLLMs trained with visual-question-answering (VQA) datasets could suffer from degradation, as VQA datasets lack the diversity and complexity of the original text instruction datasets with which the underlying language model was trained. To address this degradation, we first collect a lightweight, 5k-sample VQA preference dataset where answers were annotated by Gemini for five quality metrics in a granular fashion and investigate standard Supervised Fine-tuning, rejection sampling, Direct Preference Optimization (DPO) and SteerLM algorithms. Our findings indicate that with DPO, we can surpass the instruction-following capabilities of the language model, achieving a 6.73 score on MT-Bench, compared to Vicuna’s 6.57 and LLaVA’s 5.99. This enhancement in textual instruction-following capability correlates with boosted visual instruction performance (+4.9% on MM-Vet, +6% on LLaVABench), with minimal alignment tax on visual knowledge benchmarks compared to the previous RLHF approach. In conclusion, we propose a distillation-based multi-modal alignment model with fine-grained annotations on a small dataset that restores and boosts MLLM’s language capability after visual instruction tuning.

heading: 1 Introduction
text: Recent advancements in artificial intelligence have led to the rise of multi-modal large language models (MLLMs), which combine textual and visual interpretation capabilities in a single model (Shen et al., 2023). However, effectively blending multimodality in one system has proven non-trivial. Integrating diverse data forms often creates internal
*Corresponding author.
representation conflicts, giving rise to the issue known as "catastrophic forgetting" (Kirkpatrick et al., 2017). The diversity constraint in visual question answering (VQA) datasets could be attributed as a source of the issue. VQA tasks typically focus on descriptive queries about image contents, whereas textual datasets encompass a broader range of complex cognitive tasks, including reasoning, writing, summarization, and coding. This discrepancy in dataset complexity is a key factor contributing to the observed performance degradation in MLLMs. Our evaluation of models such as BLIP-2, InstructBLIP, and LLaVA against language instruction-following benchmarks like MTBench (Zheng et al., 2023) and AlpacaEval (Li et al., 2023b) revealed diminished language capabilities in comparison to their linguistic backbones. For instance, LLaVA, built on the Vicuna13b LLM, demonstrated a decline in MT-Bench performance from 6.57 to 5.92, even underperforming the Vicuna-7B model.
Driven by the limitations observed in distillationbased instruction tuning, particularly its constrained generalizability and the narrow performance improvements on tasks outside the training distribution, this study investigates the efficacy of distillation-based preference alignment in addressing modality conflict in MLLMs. The decision to explore this avenue is predicated on the hypothesis that integrating AI-generated preference data can provide a more granular and nuanced alignment with human expectations, potentially mitigating the adverse effects of modality conflict.
This study rigorously evaluates three baseline methodologies—Direct Preference Optimization (DPO), SteerLM, and Rejection Sampling—as potential solutions to utilize the distilled preference data to enhance the instruction-following capabilities and address the modality conflict inherent in MLLMs. Each of these methods offers a unique approach to model alignment, from the direct opti-
14188
mization of preferences in DPO to the conditional supervision in SteerLM and the selective acceptance in Rejection Sampling. Our empirical analysis reveals that DPO, in particular, demonstrates a pronounced efficacy in reconciling the performance disparities observed between textual and visual modalities. By leveraging a refined preference dataset, fine-tuned with the DPO objective and supplemented with comprehensive annotations from advanced AI models, DPO not only addresses the modality conflict but also significantly enhances the model’s performance across a spectrum of benchmarks. The results indicate that, through the application of DPO, MLLMs can achieve a more robust alignment with human-like preferences, thereby mitigating the adverse effects of catastrophic forgetting and modality conflict and elevating the models’ capabilities to a level that surpasses traditional instruction tuning methods.
Our main contributions are: Exploration of Modality Degradation: This work is the first to identify and address modality degradation in MLLMs, a phenomenon where visual instruction tuning detrimentally impacts language instruction capabilities. Our systematic investigation into this issue contributes novel insights to the field, laying the groundwork for further research in mitigating such degradation.
Efficient and scalable preference alignment pipeline as remedy Our data collection strategy employs a granular quality metric annotation format, leveraging cost-effective commercial APIs. This scalable approach enables the efficient production of high-quality datasets. We are able to surpass LLaVA and Vicuna’s language instructionfollowing capability with DPO on a 6k dataset.

heading: 2 Related Work

heading: 2.1 MLLMs and Visual Instruction Tuning
text: Incorporating another modality into large language models represents a natural evolution for these systems. Modality expansion can be achieved through system-level enhancements at inference time, with approaches such as Mm-react (Yang et al., 2023), Visual ChatGPT (Wu et al., 2023), and HuggingGPT (Shen et al., 2023) enabling the LLM to invoke off-the-shelf vision models and APIs. An alternative strand of research involves the training of end-to-end MLLMs. To avoid the prohibitive costs associated with pre-training from scratch, these models often integrate pre-trained vision models
with large language models, applying various degrees of modality adaptation. Mini-GPT4 (Gong et al., 2023) focuses solely on training a linear projection matrix to connect CLIP-based (Radford et al., 2021) vision representations with the LLaMA model (Touvron et al., 2023a); BLIP-2 introduces a cross-attention module to extract vision tokens relevant to the query. Both LLaVA (Liu et al., 2023b) and mPlug-OWL (Ye et al., 2023a) feature cross-modality connectors between the vision and language domains, but they also fine-tune the LLM and vision encoder, respectively. Flamingo (Alayrac et al., 2022), in contrast, incorporates new cross-attention layers directly into the LLM.
In the language domain, Wei et al. (2022) discovered that fine-tuning a base LLM with instructions described in natural language enhances the model’s ability to follow those instructions. In a similar vein, MLLMs are typically fine-tuned with visual instructions; Mini-GPT4 (Gong et al., 2023) utilized template instructions based on image-text pairs, while InstructBLIP (Dai et al., 2023), Otter (Li et al., 2023a), and LLaVA (Liu et al., 2023b) employed human-written visual question-answers or synthetically generated question-answer pairs by prompting GPT-4 with COCO captions and bounding boxes (Liu et al., 2023b). However, considering that both LLaVA and Instruct-BLIP utilize Vicuna (Chiang et al., 2023), it remains a topic of debate whether their steps of visual instruction tuning genuinely add to the model’s instructionfollowing capabilities or merely conform to the instruction-following format.

heading: 2.2 Mitigating Modality Conflict in MLLMs
text: In response to modality conflicts, LLaVA (Liu et al., 2023b) and mPLUG-OWL (Ye et al., 2023b) reincorporate language instruction data from ShareGPT into their training datasets. Despite this, LLaVA 1.5’s language instruction-following degrades significantly. As measured by MT-Bench, LLaVA1.5-13B lags behind its Vicuna-V1.5-13b (5.99 vs 6.57) or even the smaller Vicuna-V1.5-7b (5.99 vs 6.17). While mPLUG-OWL-2 shows promise on text benchmarks and addresses modality conflicts with new architecture features, it does so at the cost of increased complexity and a higher parameter count, rising from 7.2 billion to 8.2 billion. These modifications necessitate greater computational resources and complicate the inference process due to separate processing pathways for different modalities.

heading: 2.3 Distillation-based Instruction Tuning
text: Smaller open-source models, including Vicuna (Chiang et al., 2023), Alpaca, and more recently ShareGPT4V (Chen et al., 2023) have been finetuned using the outputs of large proprietary models, yet this approach often falls short in terms of broadening the models’ generalization capabilities. Gudibande et al. (2023) criticizes the reliance on imitation for instruction tuning, noting that it fails to substantially bridge the performance gap for tasks beyond the training data’s scope. Our findings suggest that such strategies do not effectively extend to out-of-distribution scenarios. In fact, small-scale distillation efforts tend to overly specialize the model, diminishing its performance on general benchmarks. This issue was highlighted in an experiment where fine-tuning LLaVA with a 5k VQA dataset from Gemini Pro-generated answers led to notable declines across textual and visual benchmarks. Our research takes a novel approach by exploring alignment-based methods such as Direct Preference Optimization (DPO), which not only preserves but enhances the model’s capabilities across diverse benchmarks, thereby offering a more effective solution to the limitations of traditional distillation-based instruction tuning.

heading: 2.4 Preference Alignment
text: The Instruct-GPT series (Ouyang et al., 2022) has shown that merely employing supervised finetuning (SFT) on Large Language Models (LLMs) is insufficient for aligning them with human preferences. The technique of Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) addresses this by constructing a reward model that encapsulates human preferences and then applying reinforcement learning to maximize this reward. The Direct Preference Optimization (DPO) approach posits that directly tuning the preference dataset can serve as an effective substitute for reward modeling, offering the added benefit of reduced computational complexity. SteerLM has recently been identified to achieve performance akin to RLHF by incorporating humanannotated quality metrics before generation, serving as a conditional SFT-based strategy for alignment (Dong et al., 2023). Rejection sampling selects the highest-rated self-generated sampling and applied SFT and has been used in combination with RLHF in works such as LLaMA-2 (Touvron et al., 2023b). Our experiments with DPO, SteerLM, and
rejection sampling are compared to the prior work applying RLHF to LLaVA models as a baseline.

heading: 2.5 Distilling AI Feedback for Preference Alignment
text: In the realm of alignment methods, reliance on human-annotated preference annotations is common. While effective on a large scale, this approach incurs substantial costs and operational complexities (Touvron et al., 2023b). The effectiveness of reward models based on pairwise ranking is constrained by the inherent subjectivity of human preferences, with LLaMA’s reward model achieving an accuracy range of 64.3-70.6% and the LLaVARLHF model reaching 67%. In response to these limitations, Zephyr (Tunstall et al., 2023) and UltraFeedback (Cui et al., 2023) have utilized preference annotations distilled from GPT-4 to train models with 7B parameters and achieve promising performance. Motivated to adapt this distillationpreference alignment approach for MLLMs, our work introduces a chain-of-thought prompting coupled with a granular annotation guide spanning five assessment metrics.

heading: 3 Method
text: The focus of this study is to investigate improving the text instruction following of MLLMs while retaining or potentially enhancing their multi-modal capabilities. For this purpose, we propose to harness alignment methods that utilize self-sampled responses and preference annotations in addition to Supervised Fine-Tuning (SFT) as a baseline.

heading: 3.1 Data Collection
text: Sources of Multi-modal Prompts. We have curated 5,000 samples from SciGraphQA (Li and Tajbakhsh, 2023) and LRV-Instruct (Liu et al., 2023a) to assemble the image-text prompts for our multi-modal preference dataset. SciGraphQA is a multi-turn question-answering dataset regarding scientific graphs. The LRV-Instruct dataset is a visual instruction dataset aimed at mitigating hallucinations by incorporating both positive prompts (inquiring about objects present in the image) and negative prompts (requesting information about absent objects) (Liu et al., 2023a). Including negative examples encourages the multi-modal LLM to critically evaluate the prompts and identify instances where the requested objects are absent in the image. The LRV-Instruct dataset enhanced its performance
in reducing hallucinations, as evidenced by its application in the Pope framework (Liu et al., 2023a).
Granular Annotation: We generate 4 completions for the dataset above using LLaVA-1.5-13B with a temperature of 0.7 and then prompt GeminiVision (Team et al., 2023) with the labeling guide of HelpSteer gave to Scale AI workers (Wang et al., 2023), images, questions, and the 4 completions, obtaining granular annotations for various quality
metrics, including helpfulness, correctness, and coherence, providing multifaceted insights into the model’s outputs. We leveraged Zero-Shot Chain-of Thought prompting (Kojima et al., 2022) such that the Gemini gives the reasoning for rating each response, an inner calibration monologue, and ratings as shown in 1. The appendix provides two annotation examples from the LLaVA-RLHF dataset (Sun et al., 2023) to visualize how Gemini reasons and rates each metric compared to binary preference annotations provided by crowd-sourced workers.
We selected Gemini Pro (dated 01/01/2024) for its performance, cost efficiency, and bias mitigation (Team et al., 2023). Leading the OpenCompass multi-modal leaderboard with an average rank of 1.89, outperforming GPT-4V’s 2.89 (ope, 2023), its generous free tier supports extensive data collection. Crucially, as GPT-4 is used as a judge in multiple instruction-following benchmarks, using GPT-4(V) for both benchmark judging and data labeling could introduce bias, potentially skewing our models towards GPT-4’s preferences. This consideration led us to opt for Gemini to ensure a more
objective evaluation of our fine-tuning efforts.

heading: 3.2 Alignment Methods
text: Self-sampled SteerLM: SteerLM, a conditional Supervised Fine-Tuning (SFT) technique, aligns LLMs by augmenting prompts with a description of the desired response quality, as introduced by (Dong et al., 2023). This method conditions SFT on granular annotation generated by Gemini and surpasses traditional SFT and alignment strategies. We construct a conditional prompt template for this technique by incorporating a conditional prompt following HelpSteer guidelines. For instance, Rejection Sampling: Following Constitutional AI (Bai et al., 2022) and LLaMA-2 (Touvron et al., 2023b), we adopt a simplified rejection sampling approach. Specifically, we select the top-scoring response from the four completions described above based on Gemini’s aggregated scores for helpfulness, correctness, and coherence. We apply standard SFT based on the selected responses without additional prompt conditions, following a straightforward format: (image, prompt, best response). Direct Preference Optimization (DPO): In the DPO approach, we first convert our dataset of granular annotations into a preference set. This conversion involves evaluating four responses per sample, each scored by Gemini on metrics including Helpfulness and Correctness. To form the preference set, we aggregate these scores to determine the overall quality of each response. The response with the highest aggregated score is designated as the ’preferred’ response. The rejected response is selected from the remaining options. To qualify as a ’rejected’ response, it must have a summed score that is at least 2 points lower than that of the preferred response. This selection process filters out pairs of preferred and rejected responses that could not be clearly distinguished. This method allows us to harness robust preference signals that enhance the training of the DPO model, ensuring that it aligns more closely with granular quality signals. We follow the method outlined in (Rafailov et al., 2023) for implementing the loss function, which effectively utilizes these preference pairs to optimize the model’s performance. SFT from Gemini Responses This is an important baseline not using self-sampling but using answers from Gemini directly for pure SFT. Gemini was prompted with the questions and images from our dataset. We gather the answer directly from Gemini answering the question and use the same hyper-
parameters in SteerLM and Rejection Sampling.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: For training, we utilize Azure Cloud (NC-A100 series) with 4 A100-80G GPUs. In all experiments, LoRA is employed for parameter-efficient tuning. For the SFT experiments, including standard SFT with Gemini responses, rejection sampling, and self-sampled SteerLM, we adhered to the same hyperparameters used in LLaVA-v1.5’s instruction tuning.
We conducted a hyperparameter search to optimize the performance of the DPO model on the LLaVA bench, an open-ended instruction tuning benchmark. Our experiment involved 1,000 samples to evaluate the influence of the beta term, which we hypothesized to significantly determine the divergence of the finetuned DPO model from the original model, potentially leading to knowledge catastrophic forgetting. The results are in the Appendix 9
Based on these results, we selected the optimal hyperparameter set: Beta of 0.1, not using average log probability and a learning rate of 5e-5. This configuration led to the highest LLaVA-bench score of 77.4, indicating superior performance in instruction tuning. The LLaVA bench, while primarily an instruction tuning benchmark, does not exclusively focus on knowledge. Future work will need to consider whether to fit a knowledge benchmark exclusively or a mix of knowledge and instruction tuning benchmarks to further validate the robustness and applicability of the DPO model.
In our DPO experiments, we performed a hyperparameter search based on 1000 samples on LLaVA Bench, exploring various values for beta (0.1, 0.2, and 0.3, averaged and non-averaged log probabilities, and learning rates of 5e-5 and 5e6. We pre-computed the log probabilities of the reference model (LLaVA-v1.5-13b) for our preference dataset before training. The complete list is provided in the Appendix.

heading: 4.2 Benchmarks
text: We have the following three sets of benchmarks.
Visual Instruction Benchmarks
• MM-Vet: MM-Vet measures six core visuallanguage capabilities across 128 tasks, providing a comprehensive evaluation of multi-
Table 3: Model parameters and hyperparameters setup for reproduction. The base model is LLaVA-v1.5-13b, derived from Vicuna-1.5-13b (LLama-2-13b). The training was conducted using Deepspeed Stage-3 on a 4x A100 80GB GPU machine with LoRA for parameter-efficient fine-tuning. DPO and SFT (including SteerLM and rejection sampling) employed distinct hyperparameter sets optimized for performance.
Parameter Settings Name Value Model architecture Pre-trained model LLaVA-v1.5-13b
Language Backbone Vicuna-v1.5-13B Vision Tower clip-vit-large Multi-modal Adapter 2-layer Multi-layer-Perception
Lora Setting Lora Rank 128 Lora Alpha 256 DPO Setting Average Log Probability False Fine-grained DPO False Beta 0.1 Learning Rate 5e-5 Gradient Accumulation Steps 4 Per device train batch size 8 Model Max length 3000 SFT Setting Learning Rate 4e-4 Per device train batch size 16 Model Max Length 2048 Common settings Hardware 4X A100 80G Distributed Learning Zero-3 Use BF-16 True Learning Rate Scheduler Cosine Learning Rate Warm up 0.003 Weight Decay False
modal understanding. It blends math, reasoning, and visual knowledge. Each task is scored by GPT-4 based on the correctness of the model’s response relative to GPT-4-generated ground truth, with scores ranging from 0 to 1, aggregated across 187 samples (Yu et al., 2023).
• LLaVA-Bench: LLaVA-Bench (in the wild) comprises 60 tasks designed to test visual instruction-following and question-answering capabilities in natural settings. Scoring follows the same methodology as MM-Vet, where GPT-4 generates the ground truth answers and scores the model’s responses from 0 to 1 (Liu et al., 2023b).
Visual Multi-Choice Benchmarks • PoPE: An object hallucination benchmark
with 8,440 samples aimed at evaluating the model’s ability to accurately discern and describe visual content. This benchmark uses yes-no binary classification evaluations, reporting F-1 scores, and questions are structured to probe the model on the presence of objects within images, such as ‘Is there a car in the image’ (Li et al., 2023c).
• MM-Bench: A comprehensive multi-modal benchmark with 4.7K samples featuring
multiple-choice visual knowledge and reasoning tasks. To prevent bias towards any particular answer choice, questions are assessed through rotational augmentations (e.g., ABC, BAC), and a question is considered correctly answered only if all rotational formats are correct. Overall success is reported as a total accuracy percentage (Liu et al., 2023c).
Language Instruction-Following Benchmarks • MT-Bench: Utilizes LLMs to approximate
human preferences with over 80% agreement, focusing on measuring the helpfulness of responses across 160 samples in single-turn and multi-turn settings. The benchmark evaluates responses based on their alignment with GPT4-generated preferred responses (Zheng et al., 2023).
• AlpacaEval: Leverages GPT-4 to assess the percentage of cases where the candidate LLM outperforms the GPT-3 API (text-davinci-003) across 160 evaluations. This benchmark highlights advancements in LLM capabilities over previous model iterations (Li et al., 2023b).
We run benchmark code once on the target model, with greedy decoding by default. In case a benchmark uses a different decoding setting officially, we follow the benchmark’s settings.
Table 4: Performance comparison among alignment strategies. The results demonstrate DPO-13B’s superior performance, particularly in reconciling language capabilities while enhancing visual task performance, validating the DPO methodology’s efficacy in multi-modal alignment.
Model Name Visual Instruction Benchmark Visual Multi-Choice Benchmark Text Instruction Benchmark MM-Vet LLaVA-bench PoPe MM-Bench MT-bench AlpacaEval Vicuna-1.5-13b (Chiang et al., 2023) - - - - 6.57 81.4 LLaVA-1.5-13b (Liu et al., 2023b) 36.3 73.1 0.859 67.4 5.99 79.3 LLaVA-RLHF-13b (Sun et al., 2023) 37.2 76.8 0.869 60.1 6.18 81.0 Alignment method we benchmarked, finetuning LLaVA-1.5-13b Standard SFT 36.5 63.7 0.850 65.4 5.01 50.2 SteerLM 35.2 67.0 0.878 65.1 5.70 68.8 Rejection-sampling 38.0 70.6 0.883 67.6 6.22 74.9 DPO 41.2 79.1 0.870 66.8 6.73 86.4

heading: 4.3 Results
text: Table 4 illustrates a comparative analysis of various alignment methods, including RLHF, Standard SFT, SteerLM, Rejection Sampling, and DPO, aimed at enhancing the language capabilities of the LLaVA model that were compromised postvisual instruction tuning. Vicuna, the base language model of LLaVA, is tested on language tasks. After visual instruction tuning, LLaVA experiences a decline in language benchmark scores from Vicuna (from 6.57 to 5.99 on MT-Bench and from 81.4 to 79.3 on AlpacaEval). However, DPO, among the alignment strategies on the 5k multi-modal dataset, not only mitigates the degradation problem but also surpasses Vicuna’s performance on both benchmarks.
In open-ended visual instruction tasks, DPO significantly outperforms the baseline LLaVA and LLaVA-RLHF models on both the MM-Vet and LLaVA-bench. These open-ended benchmarks, structured similarly to MT-Bench where GPT-4 assesses the responses against a gold standard, revealed a relatively high 0.73 Pearson Correlation between MM-Vet and MT-Bench (n=7, p=0.1).
In visual multi-choice benchmarks, PoPe, which evaluates for hallucination, and MM-Bench, which assesses world knowledge and reasoning, rejection sampling emerged as the most effective method, whereas DPO showed lesser efficacy. While DPO improved open-ended benchmarks, it slightly decreased LLaVA’s MM-Bench score from 67.4 to 66.8, contrasting with LLaVA-RLHF’s notable drop from 67.4 to 60.1, indicating a less significant alignment tax. Figure 2 illustrates the effect of scaling the DPO dataset on the MT-Bench scores, signifying the efficacy of DPO in advancing the
performance of the LLaVA-v1.5-13b model after visual instruction tuning. The baseline at 0% DPO data, marked at a score of 5.99, represents the initial performance of LLaVA-v1.5-13b. As the DPO dataset’s size increases, a notable rise in MT-Bench scores is observed, peaking at 6.73 with 100% data utilization. Remarkably, the performance surpasses that of the Vicuna-v1.5-13b benchmark using less than 75% or 4.2K of the DPO data, underlining DPO’s data efficiency. This data scaling trend emphasizes DPO’s potential as an effective alignment strategy for MLLMs, addressing the challenge of performance degradation due to visual instruction tuning.

heading: 5 Discussion

heading: 5.1 Limitation of collecting multi-modal preference data manually
text: The scarcity, high cost, and inconsistency of existing multi-modal preference datasets present significant challenges to model alignment. Our manual labeling of a selected subset from the LLaVA-RLHF
human preference dataset underscored the difficulty in achieving consensus or identifying clear preferences, shedding light on the inherent subjectivity of these datasets. To quantify these observations, we embarked on a targeted experiment involving 500 samples. Gemini annotated two distinct responses from the dataset employed in the LLaVaRLHF reward model’s data collection (Sun et al., 2023). This facilitated a direct comparison between Gemini’s annotations and human preferences. The correlation heatmap depicted in Figure 3 reveals a notably weak correlation between human preferences from LLaVA-RLHF and Gemini scores, with correlation coefficients falling below 0.1. This observation was corroborated by further manual inspections, which frequently resulted in an inability to definitively determine clear preferences. This finding underscores the high subjectivity and individual bias within the LLaVA-RLHF preference data.

heading: 5.2 Empirical validation of data quality
text: To further validate the quality and reliability of our annotations, we conducted a data noising experiment by randomly flipping the polarity of (preferred, rejected) pairs in the annotations and measured the trained model’s MT-Bench score 5. We hypothesized that if our annotations contained significant noise, flipping the polarity would not substantially affect the model’s performance. However, the results of the experiment indicate otherwise, with a clear trend showing performance degradation as the level of data noising increased. These findings suggest that the original annotations possess a strong and consistent signal. The model’s performance degradation with increased noise levels substantiates the reliability of our annotation process with Gemini, affirming that the annotations effectively capture meaningful distinctions between preferred and rejected responses.
Table 6: Performance improvements with multi-modal preference data application.
Model Name MT-Bench Score MM-Vet Score
Vicuna-7B-v1.5 6.17 N/A LLaVA-V1.5-7b 5.87 30.5 DPO-7b (ours) 6.228 39.8

heading: 5.3 Robustness of the model with noisy context
text: We introduced a variant of the language instructionfollowing capability benchmark MT-Bench by sending a random image sampled from the CoCo2017 train set to simulate real-world use cases. we also note that as 7 showed, our DPO model is more robust than other baselines in real-world conditions where the user may have an irrelevant image in the context.

heading: 5.4 Cross-model transfer ability: Can preference dataset generated by one model be transferred to other models?
text: To investigate if the preference alignment dataset needs to be sampled and fine-tuned on the same model, we investigated applying preferences sampled from LLaVA-13b to fine-tune a smaller model, LLaVA-7b. Notably, this approach yielded significant enhancements, as demonstrated by the performance improvements on key benchmarks. The following table encapsulates the performance metrics observed in these experiments: This cross-model transfer ability not only corroborates the versatility of our proposed approach but also implies a scalable pathway for improving multi-modal language models without necessitating the construction of new preference data for each model iteration.

heading: 5.5 Multi-modal preference alignment as a data-efficient remedy to instruction tuning capabilities
text: The superficial alignment hypothesis states that a model’s knowledge capabilities are largely acquired during its initial pre-training stage (Zhou et al., 2023). A corollary of this hypothesis is that alignment tuning refines the model output generation with a preferred response format rather than knowledge acquisition. As a result, models can be effectively realigned post-visual instruction using a relatively small set of examples (Kirstain et al., 2022). This principle applies to MLLMs as well, which acquire multi-modal knowledge representation via visual instruction tuning (Liu et al.,
2023b). However, existing work mixed large-scale text instruction data (518K out of 1.23 million in the case of mPlug-OWL 2 and 40K in the case of LLaVA-1.5). We hypothesize that the data inefficiency above is attributed to the underlying alignment strategy and demonstrate that one would need only a small alignment dataset so long as a proper alignment strategy such as DPO is utilized.
As suggested by Table 4, Direct Preference Optimization (DPO) emerges as a computationally efficient solution for enhancing model performance in the mixed-modal alignment space. Unlike the mixing text instruction as described above or LLaVARLHF, which used a large 82K dataset and complex training pipeline involving reward modeling and PPO, DPO achieves significant improvements in language capabilities with a smaller dataset and one-stop training setup. A notable advantage of DPO is its minimal alignment tax, which curtails the degradation of existing knowledge, as evidenced by its performance on benchmarks like MM-Bench, where DPO shows minimal impact. This method not only enables effective alignment of multi-modal models post-visual instruction tuning but also ensures the preservation of model performance. Our methodology exhibits notable proficiency in value alignment and data efficiency, yet it is imperative to acknowledge certain limitations and potential risks. One key consideration is the scalability of our approach. While our data scaling analysis suggests significant improvements up to a 6K preference dataset, the full extent of scalability beyond this threshold remains unexplored. As the foundational open-sourced models like LLaVA evolve in complexity and size, the effectiveness of our distillation-based approach might encounter diminishing returns.

heading: 6 Limitations
text: This research, while pioneering advancements in Direct Preference Optimization (DPO) and multimodal large language models (MLLMs), confronts several limitations. The preference dataset, derived from artificial annotations, may not fully capture the nuanced spectrum of human values, potentially affecting the model’s alignment with true human preferences and exacerbating existing biases in automated decision-making. Additionally, the generalizability of these findings across different multimodal contexts and languages is yet to be thoroughly tested, and the deployment of such models in critical domains raises significant safety risks due to the potential unreliability of responses under varied real-world conditions. These limitations highlight the need for cautious application, further empirical validation, and ongoing research to mitigate ethical concerns and enhance model robustness and fairness.

heading: 7 Conclusion
text: In this paper, we investigated and addressed the performance decline of the widely-used multi-modal LLM, LLaVA, on language instruction-following benchmarks. We investigated various alignment strategies to enhance the model’s language instruction capabilities, including standard SFT, SteerLM, and DPO. Our comprehensive benchmark evaluations revealed that DPO stands out as the most effective method, not only restoring but also surpassing the language capabilities of LLaVA’s language backbone (Vicuna) on both MT-Bench and AlpacaEval. Specifically, LLaVA’s performance on MT-Bench had dipped from Vicuna’s 6.57 to 5.99, whereas our DPO model advanced to 6.73. Beyond language proficiency, DPO also bolstered performance on multi-modal benchmarks, enhancing accuracy by 4.9% on MM-Vet and 6% on LLaVABench. Remarkably, DPO achieved this with just 5k preference examples, proving more dataefficient than other strategies like RLHF, which typically demand over 80k training examples. Our findings highlight DPO’s minimal alignment tax effectively boosting one performance area without substantially detracting from another. This aspect is particularly valuable for practitioners aiming to maintain the pre-training knowledge and avoid catastrophic forgetting.

heading: Future Work
text: Future work will need to consider whether to fit a knowledge benchmark exclusively or a mix of knowledge and instruction tuning benchmarks to further validate the robustness and applicability of the DPO model. In our DPO experiments, we performed a hyperparameter search based on 1000 samples on LLaVA Bench, exploring various values for beta (0. 1, 0. 2, and 0. 3, averaged and non-averaged log probabilities, and learning rates of 5e-5 and 5e6. We pre-computed the log probabilities of the reference model (LLaVA-v1. 5-13b) for our preference dataset before training. The complete list is provided in the Appendix