Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics
Authors: Haoqin Tu
Bingchen Zhao
Chen Wei
Cihang Xie
Venue: Accepted by TMLR ()

Abstract: Multi-modal large language models (MLLMs) are trained based on large language models (LLM), with an enhanced capability to comprehend multi-modal inputs and generate textual responses. While they excel in multi-modal tasks, the conventional view within the machine learning community has often undervalued/overlooked their capabilities in pure natural language processing. This paper aims to get out of the box and showcase an intriguing characteristic of multi-modal trained LLMs --- our preliminary results suggest that visual instruction tuning, a prevailing strategy to integrate vision knowledge into the LLMs, unexpectedly and interestingly helps models attain both improved truthfulness and ethical alignment in the pure NLP context. For example, a visual-instruction-tuned LLaMA2 7B model surpasses the performance of the LLaMA2-chat 7B model, fine-tuned with over one million human annotations, on TruthfulQA and Ethics benchmarks. Similarly, the latest LLaMA3 series also shows consistent performance gains by 0.6% on average following visual-instruction tuning. Another example is that two versions of proprietary model GPT-4V-turbo, which incorporates visual information, surpasses its LLM-only counterpart GPT-4-turbo by around 1.6% on both aspects. Further analysis reveals that the improved alignment can be attributed to the superior instruction quality inherent to visual-text data. By presenting those findings, we advocate for a broader exploration into visual-text synergies, positing that such multi-modal interactions could be pivotal in advancing alignment research. In releasing our code at https://github.com/UCSC-VLAA/Sight-Beyond-Text, we aspire to foster further exploration into the intrinsic value of visual-text synergies and, in a broader scope, multi-modal interactions in alignment research.

Published in Transactions on Machine Learning Research (12/2024)
Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics
Haoqin Tu'; Bingchen Zhao?*, Chen Wei', Cihang Xie'
UUniversity of California, Santa Cruz
University of Edinburgh
Rice University
Reviewed on OpenReview: http:://ppppaooiiio..e////ouuttr
Abstract
Multi-modal large language models (MLLMs) are trained based on large language models (LLM), with an enhancod capability comprehend multi-modal inputs and generate textual responses. While they excel in multi-modal tasks, the conventional view within the machine learning community has often undervalued language processing. This paper aims to get out of the box and showcase intriguing characteristic of multi-modal trained LLMs - our preliminary results suggest that visual instruction tuning, a prevailing strategy to integrate vision knowledge into the LLMs, unexpectedly and interostingly helps models attain both improved truthfulness and ethical aligument in the pure NLP context. For example, a visual--instruccion-tusm
oodeevallel//vveroooeel /overlooked their capabilities in pure natural
LLaMA2 7B model surpasses the performance of the LLaMA2-chat 7B model, fine-tuned with over one million human annotations,
TruthfulQA and Ethics benchmarks. Similarly, the latest LLaMA3 scries also shows consistent performance gains by 0.6% average following visual- instruction luning. Another example is that two versions of proprietary model GPT-4V-turbo, which incorporates visual information, surpasses its LLM-only counterpart GPT-4-turbo by around 1.6% on both aspects. Further analysis revcals that the improved alignment can be attributed to the superior instruction quality inherent to visual-text data. By presenting those findings, we advocate for a broader exploration into visual-text synergies, positing that such multi-modal interactions could be pivotal in advancing alignment research. In releasing our code https: //gi thub. www.////////...... VLAA/Sight-Beyy foster further exploration into the intrinsic value of visual-text. synergies and, in broader scope, multi-modal interactions in alignment research.
aspire to
1 Introduction
Enhancing truthfulness and reducing hallucinations of Large Language Models (LLMs) is one the paramount. challenges in the domain of artificial intelligence. This paper introduces new perspective this research topic, advocating for the integration of multi-modal data into LLM training as strategy to significantly improve their truthfulness and alignment with human values.
Our stance is informed by empirical evidence demonstrating the beneficial impact diverse data sources on LLM capabilities. For example, the inclusion of code data has been shown to improve the reasoning ability of LLMs (Ma et al., 2024). Building upon this premise, this paper aims explore the potential benefits of an even more diverse data source multi-modal data, particularly images, in enhancing the capabilities of LLMs.
Most modern MLLMs leverage LLMs as their core, setting the aim to bridge the gap between language and visual tokens (Liu et al., 2023b; Li et al., 2023a; Ye et al., 2023). While language tokens often capture much of the real-world context, visual information is essential to share richer real-world details that connect to the factual knowledge in human experience (Harnad, 1990; Bisk et al., 2020; Tu et al., 2023b), particularly
* Equul contribution.

---

Published in Transactions on Machine Learning Research (12/2024)
LLMs
Multi-Modal LLMs
Do
Can reindeer really fly?
Can reindeer really fly?
Multi-Modal Tuning
Yes, Santa and his reindeer really fly.
e LLaMA-2
No, reindeer can't
LLaMA-2
really fly.
Figure 1: Visual instruction tuning substantially improves the truthfulness and ethics of LLMs. We observe tuning LLMs with only 80k multi-modal data can yield stronger results on truthfulness and ethics than those with over one million human-annotated RLHF data. Note that these LLMs employ images only during the visual instruction tuning and are tested without images for NLP tasks.
areas related to truthfulness and ethics. Consequently, guiding LLMs to integrate and process visual tokens enhances the model's performance in dimensions such as ethics and truthfulness. Our claim is firmly grounded in our experimental evidence. In our preliminary explorations, we tune LLaMA series models (Touvron et al., 2023a;b) with the visual instruction data from LLaVA (Liu et al., 2023b;a). The results of these experiments are intriguing: for vanilla LLaMA2 7B model, visual instruction luning can register impressive scores of 46.0% on TruthfulQA-nc (+7.1%) (Lin et al., 2022) and 65.4% on Ethics (+19.6%) (Hendrycks et al., 2020), depending on the specific tuning approach. It is particularly noteworthy that, even without engineering efforts that explicitly elicit ethical or truthful behaviors, the performance the visual instruction-tun model already outperforms that of the T.aaM2-chaat 7B variant, which heavily tuned with over million human annotations ("Touvron al., 2023b).
We hypothesize that, in real-world scenarios, visual signals can enhance language models (LMs) in three key ways: by providing enhanced contextual grounding through explicit clues (e.g., settings, object relationships, and implicd actions), integrating implicit knowledge that, while not explicitly stated in the text, is crucial for ethical decision-making, and miligaling bias by incorporating diverse, real-world depictions that reflect a broader range of human experiences and cultural practices.
In proposing this novel perspective, we aim spur a possible paradigm upgrade or even a complete shift to the ongoing dialogue within the machine learning community. We contend that broadening the data diversity for LLMs, beyond traditional text-based inputs, is pivotal step towards developing models that more accurately reflect, interpret, and respond to the complexities of real-world information (Ma et al., 2023). This paper seeks to engage the community in a discussion about this evolving approach, underlining its potential impact on the ethical and responsible aspects of AT development.
In summary, our insights accentuate the promise of visual instruction tuning in fostering the ethical and truthful alignment of LLMs. It is our hope that this paper will serve as catalyst for new wave of research, one that embraces the rich possibilities offered by multi-modal data and paves the way for more aligned and responsible Al systems.
2 Tuning LLMs with Multi-Modal Data
This section introduces our stratogics to tune LLMs using multi-modal datasets. A standard MLLM typically contains three key components: 1) vision encoder tasked with encoding visual inputs, 2) vision-language connector that. translates visual tokens into the linguistic space, and 3) an LLM for decoding the transcribed
2

---

Published in Transactions on Machine Learning Research (12/2024)
visual information. We strictly adhere to the setups in LLaVA (Liu et al., 2023b) for fine-tuning LLMs visual instruction tuning data.
Model Architecture. We incorporate the pre-trained visual branch of CLIP ViT-L/14 (Radford et al., 2021) as our vision encoder. Additionally, a trainable linear layer is employed to project visual tokens into the language embedding space. Regarding the choice of LLM, we take the widely recognized open-sourced LLaMA models (Touvron ct al., 2023a;b; Geng & Liu, 2023) for this study. Specifically, our investigation locuses on the following six models, containing three latest LLMs and their corresponding instruction-tumed variants:
OpenLLaMA-3B (Geng & Liu, 2023), LLaMA-7B (Touvron el al., 2023a), LLaMA2- 7B (Touvron et al., 2023b).
Pre-trained LLM:
Instruction-tuned LLM: OpenAlpaca-3B (Su et al., 2023), LLaMA2--hat--71 (Touvron et al., 2023b), the Vicuna family (Vicuna-7B, Vicuma-v1.5-7B, Vicuma-v1.5-13B (Zheng et al., 2023), the LLaMA- 3-Instruct family (ILaMA3-88, L.AMAA3.1-8B, L.MMA-3.2-11B) (Dubey et al., 2024), the OpenAT's GPT-4 family (GPT-4+urbb, GPT-4V-turbo) (Achiam al., 2023a;b).
As listed above, our study is centered on two model scales: 3B and 7B. While the 3B LLaMA model is sourced from the OpenLM project (Geng & Liu, 2023), the 7B LLaMA models are directly released by Meta (Touvron et al., 2023a;b); additionally, our investigation extends to the instruction-tuned variants of these base LLMs. Concretely, OpenAlpaca-31 is fine-tuned on the Alpaca data (Taori et al., 2023) using OpenLLaMA-33 as its backbone; Vicuna-7B is the v1.1 model from FastChat (Zheng et al., 2023), which is crafted upon LLaMA-7B and employs 125K conversational data from ShareGPT (ShareGPT) during tuning; LLaMA2-chat-7B is well-engineered for human alignment, undergoing its training on publicly available instruction datasets and million human-annotated examples using RLHF techniques. The LLaMA3 series is the most recently released open-weight model family with powerful NLP abilitics. For LLaMA3.2-11B. we only take the LLM part in the model for experiments. Note that we Lest. 7B/8B LLM variants by default, and indicate 3B models by the suffix "-3B".
Training Procedure. The MLLM training unfolds in two stages. First, we exclusively tune the weight of the vision-language connector, with both the visual encoder and the LLM remaining frozen. In the second phase, we fine-tune the weights of both the connector and the LLM. Data-wise, we adhere to the protocols sot by LLaVA (Liu et al., 2023b): the connector is initially trained using 595k image-text pairings filtered from CC3M (Changpinyo et al., 2021); the subsequent stage that requires LLM training utilizes 158k instructions- following data from LLaVA with 80k unique images, which contains image- grounded conversation, image doscriptions, and image-based complex reasoning tasks. To investigate the factors driving the improvements in visual instruction tuning, we also explore tuning the model using only lext-based instruction data. We utilize three types of text-only data (sampled to equal sizes): visual instruction tuning data without images, Alpaca data (Taori et al., 2023), and Orca data (Lian et al., 2023). The Alpaca datasct derived from prompting OpenAl's GPT model with a variety of real-world questions and scenarios, while the Orca dataset comprises FLAN-augmented examples (Longpre et al., 2023), which have been shown to empower open-source 13B LLMs to excel across multiple benchmarks. (Lian et al., 2023). As for the training strategy, we probe the effects of both full fine-tuning and LoRA fine-tuning (Hu et al., 2021).
Evaluation Protocols. We conduct our evaluation using publicly available and widely used pipeline (Gao et al., 2021). Specifically, for the Ethics benchmark, we use accuracy as the evaluation metric. For TruthfulQA, we follow the official repository and use Rouge and/or BLEU accuracy for generation tasks, along with single-true (mel) and multi-true (mc2) metrics for evaluated with accuracy or F1 score, following the original work. For multi-modal tasks, we use accuracy for question-answerigg benchmarks and CIDEr (Vedantam al., 2015) for generation tasks. Further details on multi-modal benchmark metrics will be presented in Section 3.4.
Other NLP tasks are
3

---

Published in Transactions on Machine Learning Research (12/2024)
Ethics Acc.
TruthfulQA-gen Rouge Acc.
TruthfulqA-me1 Acc.
TruthfulQA-mc2 Acc.
Models
Delphi
60.1%
TA (ILaMA2-chat)
45.2%
TA (Vicuna-v1.5-7B)
50.0%
LLaMA-7B
50.4%
27.5%
22.0%
34.1%
MM-ft
59.1% (+8.7%)
29.4% (+1.8%)
23.6% (+1.6%)
35.8% (+1.7%)
LLaMA-3B
45.6%
25.3%
21.3%
34.6%
MM-fi
58.1% (+12.5%)
26.4% (+1.1%)
21.4% (+0.1%)
32.9% (-1.7%)
MM-lora
45.7% (+0.1%)
25.2% (-0.1%)
23.0% (+1.7%)
35.6% (+1.0%)
Alpaca-3B
44.0%
28.6%
22.4%
34.2%
MM-ft
46.8% (+2.8%
28.2% (-0.4%)
23.1% (+0.7%)
34.2% (+-0.0%)
MM-lora
44.0% (+0.0%)
28.6% (+0.0%)
24.6% (+2.2%)
38.0% (+3.8%)
LLaMA2
45.8%
32.3%
25.2%
38.9%
MM-ft
65.4% (+19.6%)
31.5% (-0.9%)
27.8% (+2.6%)
40.2% (+1.3%)
MM-lora
1% (+0.3%)
37.9% (+5.6%
32.1% (+6.9%)
46.0% (+7.1%)
LLaMA2-chat
58.5%
43.3%
29.5%
44.6%
MM-ft
65.2% (+6.7%)
35.5% (-7.%%
27.7% (-1.8%
41.0% (-3.6%)
MM-lora
58.6% (+0.1%)
14.6% (+1.2%)
29.1% (-0.1%)
44.6% (+0.0%)
Vicuna-v1.5-7B
62.1%
40.6%
28.9%
45.4%
MM-ft.
69.1% (+7.0%)
40.8% (+0.2%)
30.8% (+1.9%)
45.9% (+0.5%)
MM-lora
64.5% (+2.4%%
44.6% (+4.0%)
31.8% (+2.9%)
47.6% (+2.2%)
V..-13-v1.5-13B
69.1%
39.0%
30.1%
45.9%
MM-ft
73.9% (+1.8%
38.8% (-0.2%)
29.1% (-0.7%)
42.5% (-3.4%)
MM-lora
69.8% (+0.6%)
42.2% (+3.2%)
29.4% (-0.7%)
46.3% (+0.4%)
LLAMA3-BB
67.6%
47.7%
36.1%
51.5%
MM-ft.
68.1% (+0.5%
47.3% (-0.4%)
36.7%
(+0.6%)
51.5% (-0.5%)
MM-lora
67.4% (-0.2%)
48.3% (+0.6%)
36.8% (+0.7%)
51.9% (+0.4%)
LLaMA3.1
66.9%
60.6%
36.7%
54.0%
LLaMA3.2*
68.5% (+1.6%)
63.0% (+2.4%)
37.7% (+1.0%)
54.9% (+0.9%)
Table 1: Comparison on the original LLMs and the multi-modal fine-tuned ones on Ethics (Hendrycks et al., 2020) and TruthfulQA (Lin et al., 2022). '-ft' represents full paramcter fino-tuning and '-lora' indicates LoR.A tuning. We report Rouge-L accuracy for TruthfulQA= gen and accuracy for the rest. Note that LLaMA3.25 denotes a model with visual capabilities built LLaMA3.1. Note that take the Delphi (Jiang et al., 2021) and the Beatworthy-AAligneeen (TA) (Zhang et al., 2024) to provide baseline nunbers for these two tasks.
3 Evaluations Results and Analysis
3.1 Truthfulness and Ethics of MLLMs
We report the evaluation results on the TruthfulQA and Ethics benchmarks, designed for mcasuring LLMs' truthfulness and ethical alignment. During this evaluation, we utilize the weights exclusively from the visual-instrucctio--tuuee LLMs, intentionally omitting the visual encoders and vision-language connectors introduced during the fine-tuning process. The results are presented in table 1.
Visual Instruction Tuning Improves Truthfulness and Ethics. Our observations suggest that, rather unexpectedly, visual instruction tuning tends to enhance the truthfulness of LLMs. A compelling observation emerges when comparing between LLaMA2 and LLaMA3 variants: visual-instruction-tunee models, especially LLaMA2 with MM-lora, surpass the LLaMA2-chat model in performance metrics on both TruthfulQA-ne (32.1% us. 29.5%) and TruthfulQA-me2 (46.0% 44.6%). Furthermore, of the leading geighweight,
4

---

Published in Transactions on Machine Learning Research (12/2024)
Ethics
TruthfulqA BLEU Acc.
TruthfulQA Rouge Acc.
Data
Acc.
GPT-4-turbo (0515)
87.0%
55.3% 55.5% (+0.5%) 54.9% (-0.1%)
55.0%
GPT-IV-turba (0515) 87. 87.4% (+0.1%)
GPT-1-turbo (1201)
87.3%
54.0%
54.8%
GPT-IV-turbo (1201) 88.0% (+1.3%) 54.6% (+0.6%) 54.4% (-0.4%)
Table 2: Comparison on GPT-4-turbo and its multi-modal variant GPT-4V- Ethics and TruthfulQA generation. We report results of GPT-4-turbo models with different timestamps (e.g., 0515, 1201). For Truthful0A, we use BLEU and Rouge-L, accuracy on its generation task.
LLMs, the LLaMA3 series shows noticeable performance gains when visual instruction data is integrated, with average improvements of 0.6% and 0.9% on the Ethics and TruthfulQA-gen, respectively.
From table 1, we also observe visual instruction tuning leads to substantial improvements on the Ethics task. Echoing the trend in the TruthfulQA evaluations, risnal-instruction-ee versions of both LLaMA2, consistently outpace their iustuuctoon-unee counterparts, such as LLaMA2-chat and Alpaca-3B. For example, the performance enhancements observed for LLaMA2 on the Ethics task amounted to increments of 19.6%, oulperforming LLaMA2-chal and Alpaca-3B by margins of 6.9% and 11.3%. Another straightforward observation is that, models with larger parameter scale generally perform better in these two aspects (e.g., 13B us. 7B us. 3B LLMs), as stronger base LLMs are more capable during the multi-modal tuning process.
models, specifically the MM-ftt
For more recent LLMs like the Vicuna-v1.5 and LLaMA3 family, we also have the observation that the visual-instruction-tuue MLLMs perform better than its language-only counterparts by 2.2% and 0.8% across all Vicuna-v1.5 and LLaMA3 models on Ethics and TruthfulQA, respectively. While finetuning the LLM part in MLLM with full parameter activation leads to better multi-modal performance, which has also been validated by other works, it generally underperforms LoRA-tuned MLLMs in TruthfulQA task (i.e. 2.2%, 2.4%, and 2.6% improvement of LoRA tuning compared with finetuning for Vicuna-1.5-7B Vicuna-1.5-15 LLaMA-3.1, respectively). But LoRA-tuned MLLM lags behind finetuned ones on Ethics by an average of 7.2% across model variants. This observation suggests that the ethics aspect aligns better with the multi-modal objective in visual instruction tuning than the truthfulness aspect as discussed in Section 3.1.
For recent LLMs like Vicuna-v1.5 5 and the LLaMA3 family, we observe that visuaal-instructionn-tueee MLLMs outperform their language-only counterparts, with gains of 2.2% and 0.8% on the Ethics and TruthfulQA benchmarks, respectively, across all Vicuna-v1.5 and LLaMA3 models. While finetuning the full LLM component MLLMS enhances multi-modal performance (Tiu et al., 2023b; Li et al., 2023a) generally underperforms compared to LoRA-tuned MLLMs on the TruthfulQA task. Specifically, LoRA tuning yields improvements of 2.2%, 2.4%, and 2.6% over finetuning for Vicuma-1.5-7B, Vicuma-1.5-133. and LLaMA-3.1, respectively. However, LoRA-tuned MLLMs fall behind finetuned ones by an average of 7.2% on the Ethics benchmark across seven model variants. This suggests that alignment with the multi-modal objective in visual instruction tuning may be stronger for ethics-related dimensions than for truthfulness, we will dive into this aspect and discuss more later.
It should be noted that the employed visual instruction tuning data (that requires LLM parameter update) is the 158k dataset derived from LLaVA (Liu et al., 2023b), which does not contain special designs for aligning models to human preferences. Remarkably, despite this, visual instruction tuning is able to yield empirical advantages that surpass those from RLHF, which heavily utilizes a substantial corpus of human-annolated data dedicated to LLM alignment. This observation strongly attests to the potential that visual instruction tuning holds in addressing AT alignment challenges.
However, it enhancing the alignment of models previously fine-tuned via instruction tuning (e.g., models like Vicuna, LLaM LLMMA2-chat), indicating variability in its efficacy.
not a silver bullet our experiments also show that visual instruction tuning is limited
5

---

Published in Transactions on Machine Learning Research (12/2024)
MM-LLaMA2-ft
MM-LLaMA2-lork
65.1 65.4
Visual Instruction
60
only Visual Instruction
(%6
60
Visual Instruction
Visual Instruction
46.1
46.
0
40
40.
40
37.0
31.2 31.5
Ethics
TruthfulQA-gen
muthhfuqAAmmi
muthffulQA-mc2
Ethics
TruthfulQA-gen
TruthtulQA-mmi
TruthtulQA-mc2
MM2-LaMA2-chat-ff
MM-LLaMAZ-chat-lon
63.8
65.2
Text-only Visual Instruction
60
58.6
Text-on only Visual Instruction
Accuracy (%)
60
44.6
44.6
40
39.7
40
35.5
31.9
27.5 27.7
Ethics
TruthfulQA-gen
TruthfulQA-mel
muthfuulA-mc22
Ethics
TruthnulQA-gen
TruthfulQA-mcl
TruthtulQA-mc2
Figure 2: Performance of monnall-stuutton-tunneee LLaMA2 models and text instruction tuned ones on Ethics and TruthfulQA benchmarks. The text-only visual instruction data is taken directly from LLaVA, but without the paired images.
MM-LLaMA2-Ft
MM-LLaMA2-chat-ft
44
46
TruthfulQA-mz
43
TruthfuilQA-mt
42
62.5
Ethics
Ethics
Figure 3: Results of different data components on Ethics and TruthfulQA of visual-instruction-tunee LLMs. We utilize 20K of different forms of data (Conversation, Details, Reasoning), and additionally sample 20K data out of the original 80K training instances (Random 20K) for comparison.
We also report model performance of two versions of proprietary GPT-4-turbo series these two NLP tasks in table 2. The GPT-4V model is regarded as upgrade of GPT-4-turbo, with the visual understanding ability. The GPT-4V demonstrates improved performance Ethics by an average of 0.9%, as well as on the TruthfulQA generation task under BLEU accuracy, further supporting our claim in bringing visual knowledge to enhance LLMs' ethical and truthful awareness.
Effects of Modalities in Visual Instruction-Tuning Data LLM Alignment. Next, we seek to understand how different modalities in the visual instruction data contribute the alignment of LLMs. Specifically, we design set of ablations where we only utilize the text part of the visual instruction tuning data to tune the LLMs, and draw a comparison with the models tuned with both the visual inputs and the corresponding texts.
As shown fig. 2, we observe that models with text-only visual instruction tuning can largely attain comparable alignment performance with the vanilla visual instruction tuning baseline where both images and texts used. While additionally including visual inputs yields seemingly "modest"" alignment improvements,

---

Published in Transactions on Machine Learning Research (12/2024)
0.29
TruthfulQA mc]
0.28
0.60
0.27
Ethics
0.26
0.55
LLaMA-2-chtt-FT
LLaMA-2-chat-FF
LLaMA-2-FT
LLaMA-2-FT
LLaMA-1-FT
LLaMA-1-FT
0.45
0.22
0
500
1000
1500
2000
2500
500
1000
1500
2000
2500
Training Steps
Training Steps
Figure 4: Results of three MLLMs on Ethics (left) and TruthfulQA-mc1 (right) during MLLM visual instruction tuning.
stress that these gains consistent across different LLMs, tuning methods, and alignment tasks. For example, this can be verified across three model variants, resulting in an average accuracy improvement of 2.5% across three sub-tasks presented in fig. 2.
This finding supports the idea that visual instruction tuning can improve model performance, even when trained on GPT-4-generatod data. Notably, these two tcchniques-visual instruction tuning and GPT-4 data training can be used simultaneously. Additionally, we conduct ablation experiments with text-only data in the later section to better demonstrate the efficacy of visual instruction tuning.
This observation leads to our hypothesis that there exists a promising avenue in leveraging visual data to construct enhanced instruction-tuningg datasets. Although textual information plays a significant role in alignment, it is crucial to recognize that this text is inherently grounded in its corresponding real-world visual content; therefore, utilizing such paired information is integral to ensuring strong alignment in LLMs. These findings underscore the multifaceted benefits of visual data: it not only enhances alignment quality but also contributes significantly to the creation of more accurate instruction-tuning datasets.
Types of Visual Instruction Data Matters. We further extend our investigation to understand how varying types of visual Instruction-tuming data affect LLM alignment. Specifically, we utilize data from LLaVA (Liu et al., 2023b), which categorizes visual instruction tuning data into three groups: Conversation, Details, and Reasoning. Each group comprises 20k data points, sampled from the original training splits. For fair comparison, baseline group. We tune LLaMA2 and LLaMA2-chat with each data group (of 20k data points) separately, and report the results in fig. 3.
also take uniform sample of 20k from the full 80k visual instructions to form the
Our analysis reveals that, in general, conversational data has a greater impact on improving LLMs' performance on the Ethics task, resulting in an improvement of ~55 on MM-LLaMA2-ft and ~3% on MM-LLaMA2--O ft. Conversely, reasoning and details data tend to be more effective in improving performance on TruthfulQA, yielding gains of more than 2% and 6% on these two models. This suggests that a targeted approach, leveraging the unique strengths of each data type, can facilitate more muanced and effective instruction tuning for LLM alignment.
Imperfect Match Between Multi-Modal and NLP Objectives. The incorporation of visual information has shown to benefit the ethical and truthful aspects of L.Mss. In fig. 4, we present the model performance on the Ethica and TruthfulQA tasks during the multi-modal LLM finetuning stage. At the initial stage of visual instruction tuning, there is noticeable improvement in most models for both aspects within the first 1000 training steps. Specifically, the scores on Ethics task continue to increase as more visual knowledge is incorporated, indicating a well-aligned training objective between visual instruction training and ethical awareness. However, the alignment between incorporating visual perception into LLMs and enhancing model truthfulness may not be optimal, as the scores for truthfulness degenerate with more training steps considered (e.g., two LLaMA2 models achieve their highest TruthfulqA-met scores at the 1000th training step). This finding is consistent with our previous observation that the model's awareness of truthfulness shows less improvement. compared to its ethical alignment.
7

---

Published in Transactions on Machine Learning Research (12/2024)
Models
MMLU (Acc.)
GSMBK (Acc.)
MathQA (Acc.)
sQuAD (F1.)
Boolq (Acc.)
LLaMA
36.8%
8.0%
27.7%
19.5%
75.1%
MM-ft
27.7% (-9.0%)
0.9% (-7.1%)
28.5% (+0.8%
9.1% (-10.1%)
17.5% (-27.6%)
Vicuna
47.2%
10.0%
29.0%
19.3%
78.1%
MM-fI
44.0% (-3.2%)
5.4% (-1.6%)
29.4% (+0.4%)
10.1% (-9.2%)
52.5% (-25.6%)
LLaMA-3B
26.7%
2.4%
26.4%
20.7%
65.6%
MM-ft
26.5% (-0.2%)
1.7%
(-0.6%)
25.8% (-0.6%)
8.6% (-12.1%)
53.6%
(-12.0%)
MM-lora
26.8% (+0.1%)
3.1% (+0.7%)
26.3% (-0.1%)
18.8% (-1.9%)
66.3% (+0.7%)
Alpaca-3B
24.9%
0.1%
24.6%
28.2%
71.1%
MM-ft
24.5% (-0.4%)
0.0% (-0.1%)
25.6%
(+1.0%)
12.1%
(-16.1%)
69.0% (-2.1%)
MM-lora
24.3% (-0.6%)
0.1% (+0.0%)
25.4% (+0.8%)
24.2% (-4.1%)
71.1% (+0.0%)
LLaMA2
45.9%
13.7%
30.1%
26.3%
77.7%
MM-ft
39.4% (-6.5%)
5.5% (-8.2%)
29.6% (-0.5%)
8.5% (-17.8%)
56.3%
(-21.4%)
MM-lora
46.6% (+0.7%)
15.0% (+1.3%)
30.4% (+0.3%)
20.1% (-6.2%)
77.6% (-0.1%)
LLLMA2-chat
45.8%
18.2%
31.1%
20.1%
80.7%
MM-ft
45.2% (-0.6%)
6.2% (-12.0%)
30.0% (-1.1%)
10.2% (-9.3%)
67.0% (-13.7%)
MM-lora
45.9% (+0.2%)
17.1% (-1.1%)
30.8% (-0.3%)
25.5%
(+5.4%)
81.5% (+0.8%)
Vicuna-v1.5577
50.0%
18.0%
30.0%
16.9%
82.1%
MM-ft
50.6% (+0.5%)
17.4% (-0.6%)
29.2% (-0.8%)
18.2%
(+1.3%)
78.3% (-38%)
MM-lora
50.4% (+0.4%)
16.5% (-1.5%
29.4% (-0.6%)
12.5% (-4.4%)
82.8% (+0.7%)
Vicuna-v1.5-100 533
55.8%
31.9%
33.8%
16.3%
86.2%
MM-ft
56.0%
(+0.2%)
27.5%
(-4.4%)
33.0% (-0.8%
18.6%
(+2.3%)
85.1% (-1.1%)
MM-lora
55.7% (-0.1%)
26.1%
(-5.8%)
32.5% (-1.%%
11.8% (-4.5%)
86.7% (+0.5%)
LLaMA3-SB
65.7%
75.7%
42.2%
42.6%
86.7%
MM-ft
56.7% (-9.0%)
71.2%
(-4.5%)
37.6%
(-4.6%)
14.4%
(-28.2%)
71.9% (-14.3%)
MM-lora
64.7% (-100%)
72.2%
(-3.%%
42.5% (-0.8%
35.1% (-7.5%)
87.5% (+0.8%)
ILaMA3.1
28.2%
77.7%
43.9%
38.4%
87. 97.2%
I.M.M3.2
68.0% (-0.2%)
76.9% (-0.8%)
43.6% (-0.2%)
38.3%
(-0.1%)
86.8% (-0.4%)
Table 3: Performances of both the vanilla LLMs and disual-instructionttun LLMs on five NLP capabilities benchnarks. Note that LLMMA3.2* denotes a model with visual capabilities built on LLaMA3.1.
By analyzing the trajectory of model performance these tasks, we observe that the optimization goals between multi-modal ability and the improved truthfulness and ethics are not perfectly aligned. Though LLMs trained on full visual tuning steps surpass the vanilla LLMs on ethics and truthfulness, the same training budgets designed for multi-modal tasks might not be optimal for models' NLP abilities.
3.2 Standard NLP Abilities
Given these LLMs are further fine-tuned with multi-modal data, it might be intuitively expected that their standard NLP capabilities could degrade. Such phenomenon is commonly referred to as catastrophic forgetting (Kirkpatrick et al., 2017) or in the AI alignment community - the alignment tax (Christiano, 2019; Jensen et al., 2023).
Interestingly, contrary to these assumptions, our results presented in table show that M.Mlorr (marked in the gray background) results in only average 0.17% performance decrease across five NLP capability benchmarks and four models, after applying visual instruction tuning. More notably, in certain instances, MM-lorr even modestly improves performance on these benchmarks. However, the visually-tuned LLaMA3 series shows an average score drop of 4.5% across these lasks while maintain high scores on Ethics and TruthfulQA in Table 1. Notably, LLaMA3.2 - a visually-enhanced LLM tuned by Meta (Dubey et al., 2024) demonstrates highly consistent performance with our trained models on Ethics, TruthfulQA, and other NLP benchmarks, further supporting our claim that visual instruction tuning enhances LLMs in ethics and truthfulnese
8

---

Published in Transactions on Machine Learning Research (12/2024)
Models
Ethics
TruthfulQA-gen
TruthfulQA-mel
TruthfulQA-mc2
LLaMA2
45.8%
32.3%
25.2%
38.9%
Alpaca
52.1%
(+6.3%)
29.6% (-2.7%)
27.3%
(+2.1%)
41.6%
(+2.7%)
text-VI
65.1%
(+19.3%%
31.2% (-1.1%)
27.7%
(+2.5%
39.5%
(+0.6%)
VI
65.4%
(+19.6%)
31.5%
(-0.9%)
27.8%
(+2.6%)
40.2%
(+1.3%)
Orea
62.9%
(+17.1%)
41.6% (+9.3%)
33.8% (+8.6%)
49.3% (+10.4%)
Table 4: Results of LLaMA2 finetuned on Alpaca data, text-only visual instruction data (text- VI), visual instruction tuning data (VT), and Orca data.
Additionally, MLLMs with a more advanced or larger LLM component tend to perform better on NLP benchmarks. For instance, the multi- modal-tuned Vicuma-v1.5-331 outperforms its 7B variant and the v1.3 counterpart by an average of 5.3% and 15.8% across five NLP lasks, respectively.
In conjunction with the insights from Section 3.1, these observations altogether highlight the ability of risuallinstruction-tumas LLMs in both maintaining the strong capability standard NLP benchmarks and aligning better with human values, not to mention the additional capability of recognizing visual inpuls. Such findings pave new avenues for both academic exploration and practical implementations within multi-modal domains. We believe these insights should catalyze further investigations into the tuning of LLMs with multi-modal interactions.
3.3 Tuning on Different Vision-Language Data
To better explain the benefit of visual text tuning in the process, we present the results of LLaMA2 finetuned on Alpaca data (Taori et al., 2023), text-only visual instruction data (text-VT), visual instruction data (VT), and Orca data (Lian et al., 2023) in Table To keep the fair comparison, we randomly sample 80K data from Alpaca and Orca data respectively for the training.
We can observe that the visual instruction tuming 1) surpasses Alpaca and text-VI data tuned models in most cases, i.e., average 6.7% and 2.8% improvements over Alpaca on two benchmarks, and 0.2%, -15% over VI: 2) but lags behind the Orca-tuned model by 1.1% and 16.3% on Ethies and TruthfulQA benchmarks. Considering that the Orea dataset includes Chain-of- ohhought (CoT) and complex, nuanced instruction-followin data from a diverse array of tasks within the FLAN collection (Longpre et al., 2023), the obscrved performance gain is reasonable. The CoT reasoning and complex eesrrcctionfolloviiti examples in Orca offer richer contextual understanding and problem-solving patterns. This observation indicates the insight that upgrading text quality within the visual instruction data could further enhance the model's ethical and truthful reasoning, opening promising avenue for refining these abilitics.
3.4 Analysis Multi- Benchmarks
We hereby test the visual-instruetion tuned models recent multi-modal benchmarks, where five tasks are deployed: Unicorn benchmark (Tu et al., 2023a) dedicates evaluating the MLLM ability in safety scenarios, we take two OODCV-VQA tasks and Sketchy-VQA tasks for testing whether models can well handle OOD visual/text input and sketch images, respectivcly. MME (Fu et al., 2023) consists of two evaluation aspects, i..,, cognition (CS) and perception (PS) with lotal 14 VQA lasks; MSCOCO (Lin et al., 2014) and Flickr30k (Young et al., 2014) captioning tasks are commonly used benchmarks in the field of image caption generation. We report the zcro-shot CIDEr (Vedantam et al., 2015) scores (with three text-only QA examples) on the test set from the Karpathy split (Karpathy & Fei-Fei, 2015). POPE (Li et al., 2023c) is used to evaluate the level of object hallucinations in MLLMs, which consists of three versions of balanced yes/no VQA tasks considering objects in the given image. It is built upon MSCOC0-2017 dataset (Lin et al., 2014). Additionally, We also make use of the image corruptions proposed in ImageNet-C (Hendrycks & Dictterich, 2019) to measure the performance of the ML.LMs on corrupted images for MSCOCO task (denoted as MSCOCD-C).2.
We exclude landmark and artwork tasks to accelerate the evaluation process. For corrupted images, report the average results tested models on four noises (gaussian noise, defocus blur, contrast, brightness) across three severity levels (1, 5)
9

---

Published in Transactions on Machine Learning Research (12/2024)
Truthu mc
Ethics
Ebhics
UNPAATT
ULMKIET
PT
MSCOCO
MME (PS)
MNNL
SQUADD
PT
MHLLI
LAPALLT
LLMANGTT
Ohind 2.76
MSCOCO
MSCOCO
Figure 5: Results of eight different MLLMs on two multi-modal and four NLP tasks
Unicorn vodev
POPE R / A
Models
MME
MME
COCO
Flickr30k
sketch
MM-LLaMA-ft
45.2
80.9
199.3
510.5
59.2
27.1
65.7
57.8 /
MiM-micua-ft
58.4
82.7
270.7
625.2
57.5
24.6
76.5 / 66.5 / 73.8
MM-LL.MAA2-tt
55.0 / 83.6
237.1
661.3
65.1
31.6
65.0
55.4 / 56.3
MM-LLaMAA2-orra
52.3
79.6
200.0
395.0
52.0
26.2
50.8
50.4 / 50.6
MM-LLaMA2--hatt
54.5 / 80.9
234.6
805.4
57.4
26.7
69.8 / 57.9 / 60.3
MM-LLaMAA2-cht--onn
53.1
82.8
228.6
709.8
43.4
23.0
65.9
56.8 / 59.2
MM- Vicuma-y1.5-7B
58.4
80.4
320.4
1182.0
98.5
62.8
89.3 / 79.7/ 85.5
MMVVicuma-v..5-133
59.7
87.8
287.9
1213.3
84.0
51.3
87.4 78.7 / 84.1
Table 5: Performances of our MLLM family on five widely employed multi- modal benchmarks. We test models on oodev and sketch sub-tasks the Unicorn benchmark (Tu et al., 2023a) and Random (R), Adversarial (A), and Popular (P) POPE (Li al., 2023c).
Enhanced MLLMs Expand NLP Capabilities. Is better visual reasoner also better NLP task solver? In fig. 5, we illustrate the correlation between model performance in multi-modal and NLP tasks. The results reveal positive correlations across tasks from different domains, suggesting that the visual reasoning abilities of the eight models analyzed contribute to their improved performance on NLP benchmarks. Notably, the average cocfficient of determination score across these eight scenarios 0.482, indicating a moderate correlation. specific cases, such as pairing MME (PS) with Ethics and MMLU, both of the coefficient scores exceed 0.72, indicating very strong correlation between these tasks. This evident correlation explains our finding of stronger MLLMs can lead to expanded NLP capacities. But unlike the misaligned objectives observed between MILM multi-modal and NLP abilities during training in Sec. 3.1, this finding might not be surprising, as MLLMS rely heavily on language to reasoning and expression. It is plausible that an improved LLM (i.e. better training data, larger model scale) could enhance the expressive abilities of an MLLM, resulting in a close correlation between abilities across different modalities.
Aligned LLMs us. Unaligned LLMs on Multi-Modal Benchmarks. In table 5, MLLMs incorporating aligned LIMS (e.g., Vicuna, ILLMA2-chat) have demonstrated top performance comprehensive and challenging tasks such as Unicorn, MME, and POPE. Specifically, MMM-ieuna-ft and MM-LLaMA-chat- outperform their corresponding vanilla MLLM counterparts by average of 164.9 MME and 7.5% on POPE. Despite the incorporation of text-aligned LLMs, MLLMs still exhibit unexpected shortcomings when compared to models that use vanilla LLMs, particularly when evaluated on three traditional vision-text lasks. For instance, MLLMs show an average drop of 4.2 CIDEr across two caplioning tasks. This could be attributed to the nature the captioning task itself, which is more ceerrationffoousse and less about following instructions as seen more challenging QA tasks. As result, MLLMs that use unaligned LLMs may perform better on captioning tasks, as they are less constrained by alignment requirements and are better suited for the generation-centric nature of captioning.
10

---

Published in Transactions on Machine Learning Research (12/2024)
Models
coco (CIDEr)
COCO-C FCIDEr)
MM-LLaMA-ft
59.2
48.6 (-17.9%)
MM-Vicuna-ft
57.5
16.0 (-20.0%)
MMMLLaMA2-ft
65.1
54.6 (-16.1%)
MM-LLaMA2-lora
52.0
43.2 (-16.9%)
MM-LLaMMA2-hat-f
57.4
47.5 (-17.2%)
MM-LLaMMAZ-hhattorrr
43.4
33.8 (-22.1%)
Table 6: Performances of the MLLM family on MSCOCO (Lin et al., 2014) with corrupted visual inputs.
Need for Studying Multi-Modal Alignments. Although lext-aligned models like Vieuna and LLaMA2- chat have proven effective, their MLLM variants perform poorly on corrupted images, as shown in table 6. Not only do these models underperform compared to MLLMs that do not use instruction-tuned LLMs, but they also exhibit a performance drop of over 17% when evaluated on corrupted images, compared to clean ones. This decline is even greater than the drops observed in MM-LIaMA-fit and MM-LIaMA2-f.. This suggests that while visual instruction tuning enhances the truthfulness and ethical behavior of LLMs in the language domain, these MLLMs still face their unique challenges in the multi-modal context, especially when dealing with corrupted or imperfect visual inputs (Tu et al., 2023a; Dong al., 2023).
4 Related Work
Alignments. The aligmment of Al systems to human values is important topic for today's advanced Al systems, from testing model robustness to out-of -fs-oo-bisttitttoonnn shifts (Hendrycks & Dietterich, 2019; Hendrycks et al., 2021a; Zhao et al., 2022) to adversarial attacks (Hendrycks et al., 2021b; Eykholt et al., 2018; Xie et al., 2020), many works have been proposed. The recent development of LLMs has revolutionized natural language processing and has been widely adopted in various applications. Thus, concerns regarding the honesty and truthfulness of these models have also emerged, prompting alignment researchers to investigate the ethical implications and potential risks associated with their deployment. TruthfulQA (Lin et al., 2022) is proposed to measure how LLMs imitate human misconceptions. And Ethics (Hendrycks et al., 2020) is used to assess a language model's knowledge of basic concepts of morality.
Advanced techniques for aligning language models with human preference are also popular these days, from RLHF (Ouyang et al., 2022) to DPO (Rafailov et al., 2023) optimization, the alignment training paradigms have shifted fast. recently. Reinforcement learning as one of the most popular solutions to enhance the truthful and ethical awareness of LLMs have long been discussed. Recent works explored various approaches for such purpose, including progressive rewarding (Gao et al., 2024), optimized RL reward function (Bai et al., 2022), and applications as well as benchmarking (Zhang et al., 2024; Li et al., 2024). The concept of LLM aligument has also gradually switched from human-supervisel (Ouyang et al., 2022) to the paradigm of incorporating other Al model supervisions (Lee et al., 2023), and to most recently the employment of weak signals (Burns et al., 2023). Given the popularity of the use of large language models, adversarial attacks on LLMs have also been explored (Zou et al., 2023). In this work, we present our findings on how visual instruction tuning can help the LLMs align with human values, our results show impressive performance boost on these datasets without explicit prompting such behaviors.
Multi-Modal and Large Language Models. In light of the rapid evolvement of large language models (LLMs), recent studies about multi-modal systems have turned their focuses from incorporating fine-grained multi-modal data (Liang et al., 2021; Tu et al., 2023b) to integrating powerful LLMs with few-shot capability. More recently, some instruction-tun tuned MLLMs have emerged, showing excellent generalization ability in unseen VL tasks (Zhu et al., 2023; Liu et al., 2023b; Ye ct al., 2023; Li ct al., 2023a; Dai et al., 2023). For example, MiniGPT4 (Zhu et al., 2023) is built upon QFormer (Li et al., 2023a) and Vicuna (Zheng el al., 2023) and only activates the linear layer connecting the vision encoder and LLM. LLaVA (Liu et al., 2023b;a) projects the output of a vision encoder to word tokens and trains both the VL connector and the LLM on synthetic data. mPLUG-owl (Ye et al., 2023) tunes LLaMA with a query- based VL connector using both text-only and vision-language instruction data. InstructBLIP (Dai et al., 2023) uses BLIP2 (Li et al., 2023a)
11

---

Published in Transactions on Machine Learning Research (12/2024)
the backbone but is additionally instruction-tuned on collection of VL datasets. Other multi-modal LLMs in vast range of modalities sparkles insights and deployment in real-world applications (Zhang et al., 2023c; Bai et al., 2023; Zhang et al., 2023a; Liu et al., 2024).
Despite the rapid growth in this domain, recent benchmark works have shown that current multi-modal large language models still suffer from problems like being unable to handle counterfactual statements (Zhang et al., 2023b; Yu et al., 2023; Tu et al. 2023a; Lee et al., 2024), hallucination (Li al., 2023b; Zhou et al., 2023), and simple answer set permutations (Zong et al., 2023). In our work, we demonstrate new perspective these MLLMs tuning LLMs with multi-modal data greatly helps align them with human values.
5 Discussion, Conclusion, and Future Work
More Aligned Objectives between Multi-Modal and NLP Abilities. Our exploration shows that training on multi-modal instruction tuning data can also benefit the LLMs' factual accuracy and ethics. In fig. 4, we have shown that these aligument-focused metrics improve while training proceeds on multi-modal However, current multi-modal data is not designed for alignment, the main focus is still eliciting language models with multi-modal perception. Our results demonstrate a promising new avenue for developing models to understand and interact with the world more truthfully, and also suggest the need for exploration to identify appropriate tasks that can effectively improve these two aspects simultancously. And we hope our paper could inspire discussions on this direction.
Exploring the Training Framework. In our pilot study, we have shown the results of using image-based instruction fine-tuning data to support our findings that leveraging multi-modal interactions could yield more aligned models. Based on our results, is reasonable assume that introducing multi-modal data to the pre-training stage could also yield more aligned models. For example, the Gemini family models could be interesting case for study (Gemini Team, 2023). Understanding how to instruction fine-tune the base model for multi-modal ability and alignment is another direction worthy of exploration. Our study explores full parameter fine-tume tne-tuning as well as LoRA parameter efficient fine-tuning. It can be beneficial to study how varies types of parameter-effiiiet fine-tuning techniques helps (Kopiezko et al., 2024; Zhao et al., 2024). Besides the training techniques, the training data can also be explored, how should we create a mixture of data for fine-tuning, how determine the ratio of multi-modal data to text-only data (Ye et al., 2023; Liu et al., 2023a), and how to extend to other modalities other than images. These exploration could help gives a more practical and comprehensive guide.
Conclusion. In this study, we offer preliminary findings that underscore the potential of enhancing the truthfulness and ethical alignment of LLMs through visual instruction tuning. Remarkably, even without prompts tailored for truthfulness or ethical behaviors, our approach to tuning LLM weights using visual instruction datasets yielded significant improvements in both the TruthfulQA and Ethics benchmarks. Notably, such improvements are even stronger than that of RLHF, which tunes LLMs with a huge corpus of alman-aligned data points. The follow-up analysis demonstrates the importance of instruction data quality for improving aligned values MLLMs, as well as specific types of data models employed for applying to different aligument tasks.
Future Work. In light of our findings, we advocate for future rescarch endcavors to focus on devising innovative methodologies for crafting visual instruction data that can more effectively align LLMs. Exploring novel MLLM architectures could also be fruitful avenue. We hope fostering L.I. interactions with real-world environments may emerge as pivotal strategy for achieving superior model alignment.
Acknowledge
This work is partially supported by a gift from Open Philanthropy. We would like to thank Center for AI Safety, Microsoft Accelerate Foundation Models Research Program, and the OpenAl Researcher Access Program for supporting our computing and resource needs.
12

---

Published in Transactions on Machine Learning Research (12/2024)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkal, et al. Gpt-4 technical report. aiv preprinl arXiv:2303.08774, 2023a. 3
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lgge Akkaya, Florencia Leoni Aleman, Diogo Almcida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4V(ision) system card. OpenAl Research Blog, 2023b.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large lisionllanguag model with versatile abilities. ar preprint arXiv:2308.12900 2023. 12
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 2rXi::2244.08622 2022. 11
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bongio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds language. arXiv preprint arx:::200.. 10151, 2020.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglckar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:231..00 098900, 2023. 11
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CV /CVF Conference on Computer Vision and Pottern Recognition, 2021. 3
Paul Christiano. Current work in ai alignment. Effective Altruism, 2019.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXie, 0es/2305.065000 2023. URL https://app: swwantccscoollr..c...oo org/ 258615266. 11
orr.oorprrr//CrroI CorpusID:
Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is google's bard to adversarial image attacks? arXiv preprint ar arx::23009.11 11751, 2023. 11
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint orXiv:2407.2178. 2024. 3,
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp.
2018. 11
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXio preprint arXin:2306. 13394, 2023.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL. htttp::/ddoi.rg//// 1tteo/..rrr11...1///eeeee /2w.///0o0....... 55711628 3
13

---

Published in Transactions on Machine Learning Research (12/2024)
Zhaolin Gao, Jonathan D Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J Andrew Bagnell, Jason D Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. arXiv preprint o714:2404416777, 2024. 11
Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXi::2912.11805, 2023. 12
Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama. https://github.com
seenlm-researche hpte//penn__ee 11ama, May 2023. 3
Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42((-3):335 346, 1990.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robust to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019. 9, 11
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. arXiv preprint arXiv:2008. 20220200 2020. 2, 4, 11
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of rosustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a. 11
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b. 11
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint ar0iv:2106. 09665, 2021. 3
Mckay Jensen, Nicholas Emery-Xu, and Robert Trager. Industrial policy for advanced ai: Compute pricing and the safety tax. arXiv preprint 2rv: 2023.
Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al. Can machines learn morality? the delphi experiment. arXiv preprint arX:::2110.07574, 2021. 4
Andrej Karpathy and Li Fei-Fei. Deep risual-semantic alignments for generating image descriptions. In CVPR, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kicran Milan, John Quan, Tiago Ramalho, Agnicszka Grabska-Barun Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017. 8
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. In ICLR, 2024. 12
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human foodback with ai feodback. arXiv preprinl ar 23v: 00267,, 2023. 11
Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, al. Vhelm: A holistic evaluation of vision language models. arXiv preprint (xX6:22100001132 2024. 12
Aaron J Li, Satyapriya Krishna, and Himabindu Lakkaraju. More rlhf, more trust? on the impact of human preference alignment on language model trustworthiness. arXiv preprint arXim:2404.1887 2024. 11
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. Xi preprint arx::22001.120 12597, 2023a. 5, 11
14

---

Published in Transactions on Machine Learning Research (12/2024)
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. 2023b. URL https: hwtpppdf/230... 10355. 12
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-langnage models. arXiv preprint arXiv:2305. 2023c. 9, 10
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and 'Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. OpenOrca, 2023. 3, 9
co/0pen-Oei Orca/
Zujie Liang, Huang Hu, Can Xu, Chongyang Tao, Xiubo Geng, Yining Chen, Fan Liang, and Daxin Jiang. Maria: A visual experience powered conversational agent. In Proceedings of the 59th Annual Meeting of the Associalion for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5596 5611, 2021. 11
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 3214 3252, 2022. 2, 4,
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014. 9, 11
Fuxiao Liu, Kevin Lin, Linjie Lí, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. ICLR, 2024. 12
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselines with visual instruction tuning. In NewrIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a. 2, 11, 12
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction luning. aiv preprinl arXiv:2304.08485 2023b. 1, 2, 3, 5, 7,
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp. 22631-22648. PMLR, 2023. 3,
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help lims reasoning? In ICLR, 2024.
Yuxi Ma, Chi Zhang, and Song-Chin Chun Zhu. Brain in vat: On missing pieces towards artificial general intelligence in large language models. arXio preprint ar Xiv::2300 03762, 2023. 2
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instionti with human fcodback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022. 11
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Tn International conference on machine learning, 2021. 3
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023. 11
ShareGPT. httpa://sharegpt.ccm
Yixuan Su, Tian Lan, and Deng Cai. Openalpaca: A fully open-source instruction-ollowiinn model based on openllama. http:///..tttcccc////////aaaaapp 2023. 3
15

---

Published in Transactions on Machine Learning Research (12/2024)
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. httpps//githhb... catsu--ab/sttaafor__apppp
2023. 3, 9
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprinl arXiv:2302. 1et...1977 18971, 2023a.. 2,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and finc-tuned chat models. arXic preprint or arXiv:2307.09280 002888 2023b. 2,
Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this image? safety evaluation benchmark for vision llms. arXiv preprint arXie:2311.16000 2023a. 9, 10, 11, 12
Haoqin Tu, Yitong Li, Fei Mi, and Zhongliang Yang. Resce: Responding through secing fino-grained visual knowledge in open-domain dialogue. orXiv preprinl ar Xiv: 13602, 2023b. 1, 11
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pollern recognilion, pp.
2015. 3, 9
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoe V Le. Adversarial examples improve image recognition. In Proceedings of the IEFE/CW conference on computer vision and pottern recognition, pp. 819 828, 2020. 11
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2944.1417, 2023. 1, 11, 12
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67 78, 2014. 9
Wenhao Yu, Meng Jiang, Peter Clark, and Ashish Sabharwal. Ifqa: A dataset for open-domain question answering under counterfactual presuppositions. arXiv preprint arXir:2305. 14010, 2023. 12
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.0088 2023a. 12
Letian Zhang, Xiaotong Zhai, Zhongkai Zhao, Xin Wen, and Bingchen Zhao. What if the was off? examining counterfactual reasoning abilities of multi-modal language models. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) Workshops, 2023b. 12
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Interoln-xcomppoee:: A lasion-language large model for advanced text-image comprehension and composition. arXie preprint orXiv:2309.1511 2023c. 12
Zongmeng Zhang, Yufeng Shi, Jinhua Zhu, Wengang Zhou, Xiang Qi, Peng Zhang, and Houqiang Li. Trustworthy alignment of retrieval-augmented large language models via reinforcement learning. arXiv preprint Xrx::22410. 16843, 2024. 4, 11
Bingchen Zhao, Shaozuo Yu, Wufei Ma, Mingxin Yu, Shenxiao Mei, Angtian Wang, Ju He, Alan Yuille, and Adam Kortylewski. Ood-ev: benchmark for robustness to out-of-distribution shifts of individual nuisances in natural images. In European Conference Computer Vision, pp. 163-180. Springer, 2022. 11
Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. Tuning layernorm in attention: Towards efficient multi-modal Um finetuning. In ICLR, 2024. 12
16

---

Published in Transactions on Machine Learning Research (12/2024)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ton Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 3, 11
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. 12
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
11
Yongshuo Zong, Tingyang Yu, Bingchen Zhao, Ruchika Chavhan, and Timothy Hospedales. Fool your (vision and) language model with embarrassingly simple permutations. arXiv preprint arXiv:3310001551 2023.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arxiv preprint vrXie:2307.50433 2023. 11
17


Reviews:
[{"review_id": "1a4bmfKc4p", "reviewer": "Reviewer_ah3g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1) The study examines a variety of open weight LLMs with respect to their alignment (using the ETHICS benchmark) and truthfulness (using the TruthfulQA benchmark) before and after visual instruction tuning. Alignment and truthfulness are important aspects of LLMs, and as these models are often modified to support multimodal input, it is important to also assess the change in performance on these tests after such modifications.\n\nS2) The authors do not only look at alignment/truthfulness, but also evaluate the models on language benchmarks and a set of multi-model benchmarks.\n\nW1) In its current form, I'm not convinced by the paper that the observed change in alignment/truthfulness is due to visual instruction tuning. The authors only use a single dataset that is generated by GPT-4 (Section 3 – [(Liu, 2023b)](http://arxiv.org/abs/2304.08485)), so there is a strong confounder \"fine-tuning on data generated by GPT-4\". Figure 2 from the paper also suggests that a large part of the observed effect is due to information contained in the text-only part of the LLaVA visual instruction tuning data. Further, the effect is much weaker on models that have already been trained on ShareGPT data (e.g. Vicuna), which points towards the possibility of such a transfer. Note the high scores of GPT-4 reported in Table 2.\n\nW2) I find the main results in Table 1 hard to interpret:\n\nW2.1) In some cases (e.g. Vicuna), there is a large drop in alignment/truthfulness – why?\n\nW2.2) There seems to be a large difference between -ft and -lora (and in opposing directions). Taken together with non-monotonic training curves in Figure 4 (right), and with the opposite effect when adding visual information on some of the metrics reported in Figure 2, this makes me wonder how significant the observed changes are.\n\nW2.3) There are no references anchoring any of the numbers in previous literature.\n\nW2.4) Why is Rouge reported for TruthfulQA-gen? The paper (Appendix B – [(Lin, 2022)](http://arxiv.org/abs/2109.07958)) mentions a GPT-judge with much better correlation to human judgment. What is mc1 and mc2 (I couldn't find a reference to these accuracies in [(Lin, 2022)](http://arxiv.org/abs/2109.07958)).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the effect of fine-tuning of LLMs with visual instruction tuning data. The authors use two text-only benchmarks, namely Ethics and TruthfulQA, and they find that some models show significant improvements. A range of open weight LLMs are compared, covering a range of 3B to 13B parameters, and different data. The fine-tuned models are further evaluated on language benchmarks and multi-modal benchmarks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Design experiments that can answer the question whether the observed differences in alignment/truthfulness can be explained by visual instruction tuning, or by fine-tuning on GPT-4 generated data.\n\n2) Develop a hypothesis why visual instruction tuning would improve on the ETHICS benchmark. Figure 1 gives some intuition why TruthfulQA would be affected positively by visual instruction tuning data (although also for TruthfulQA, it would be interesting to discuss this in more detail, with a focus on the different categories of the benchmark questions).\n\n3) The paper claims \"visual instruction tuning is able to yield empirical advantages that surpasss those of RLHF\" – in the context of this quote (and in the related works section), I would like to see some references that examine the effect of RLHF (or LLM alignment in general) on the presented benchmarks (ETHICS / TruthfulQA). If these two benchmarks are relevant to modern LLMs, then I would expect additional studies to exist.\n\n4) Add baseline numbers to table 1 (see also comment above).\n\n5) Explain in more detail why there is such a large difference between -ft and -lora models.\n\n6) Explain why does Vicuna-7B lose so much performance on alignment/truthfulness after visual instruction tuning? Currently the authors state that \"visual instruction tuning is limited at enhancing the alignment of models previously fine-tuned via instruction tuning (e.g. Vicuna)\" – but table 1 shows not a limited improvement, but a marked worsening for the \"MM-ft\" variant of the \"Vicuna\" model.\n\n7) Add MM-lora entries for all models.\n\n8) Section 3.1 paragraph \"Types of Visual Instruction Data Matters\" mentions that LLaVA has 80k data points. But [(Liu, 2023b)](http://arxiv.org/abs/2304.08485) mentions 158k data points (77k reasoning, 58k conversation, 23k detailed description) – why the difference?\n\n9) I would like to see some motivation around the model selection: for example, why were Mistral, Gemma, Gemma2, LLaMA3 not included? Why were the model sizes 3B,7B chosen, but 13B was only used with Vicuna (but not with LLaMA and LLaMA2 that also have 13B variants)\n\n10) Include more evaluation details: How exactly was the Ethics benchmark evaluated? [(Hendrycks, 2020)](https://arxiv.org/abs/2008.02275) mentions fine-tuning and fewshot evaluation (in case of fewshot: which prompt was used?). Why was the Rouge metric chosen for TruthfulQA-gen (and not the GPT-judge)? What does -mc1 and -mc2 refer to?\n\n11) Add baseline numbers to table 4.\n\n12) Typo: Table 1 captions says \"TruthfullQA-gen\" instead of \"TruthfulQA-gen\"\n\n13) Typo: in Discussion section, the reference should be \"(Gemini Team, 2023)\" and not \"(Team et al, 2023)\"", "broader_impact_concerns": "I don't think that this work requires a Broader Impact Statement section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GoqOLEeVca", "reviewer": "Reviewer_NThr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces an innovative perspective by showing that multi-modal training improves multi-modal tasks and enhances ethical and truthful behavior in NLP tasks. This opens up new directions for AI alignment research.\n2. The experiments demonstrate that even with fewer human annotations, models fine-tuned with visual instruction outperform heavily human-annotated models in truthfulness and ethics benchmarks.\n3. The paper evaluates multiple models and compares text-only and multi-modal models using well-established benchmarks like TruthfulQA and Ethics, providing strong support for the conclusions.\n4. The paper benefits from visual-text synergies in improving alignment and suggests a new paradigm for future LLM development.\n\nWeakness:\n1. While the paper demonstrates improved performance, it lacks a deeper analysis of why visual instruction tuning helps with truthfulness and ethics.\n2. The paper doesn't address instances where visual instruction tuning may not be effective. Negative results could be discussed more thoroughly to provide a balanced view.\n3. Although Figure 5 shows the relevance improvements between multimodal and NLP tasks, the variance is significant, making the results less confident.\n4. The paper briefly mentions the potential trade-offs between multi-modal tuning and standard NLP abilities. The finding is intriguing. However, it does not provide further analysis or relevant experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores how multi-modal large language models (MLLMs) trained on both text and visual data can significantly improve their performance in natural language processing tasks related to truthfulness and ethical alignment. The paper claims that visual instruction-tuned models improve truthfulness and ethics compared to their purely text-tuned counterparts. Also,\nmodels that leverage visual data during fine-tuning generally perform better, even when tested without visual input. Moreover, visual instruction tuning enhances the alignment of LLMs with human values, suggesting the potential for more aligned AI systems. The paper encourages further research into integrating visual data to advance the alignment of LLMs. It suggests that broadening the type of data used in LLM training could be pivotal in AI alignment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Llama3 was released earlier this year. Given your claims that \"stronger base LLMs are more capable during the multi-modal tuning process\" and \"the ability of visual-instruction-tuned LLMs in both maintaining the strong capability on standard NLP benchmarks and aligning better with human values\", testing Llama3-8B would provide strong validation of your approach.\n2. In Table 2, you compare GPT-4 and GPT-4V. To strengthen the evaluation, please also consider comparing Llama3/3.1 with Llama3.2.\n3. The improvement of GPT-4V over GPT-4 on Ethics is 0.4%, which is marginal. Do you have variance data or significance tests to assess the robustness of this improvement?\n4. In training, you tune connectors for each LLM with 595k image-text pairings. Since connectors are integral to the tuning process, why aren't those parings counted as part of the training data? Although they are not instructions-following data, they involve the entire tuning process. With that amount of tunning data, you still use less data than RLHF.\n5. In Figure 2, should the blue text represent \"text-only instructions\"?\n6. The text in the figure legends for Figures 3 and 4 is too small and should be adjusted for clarity.\n7. Please provide relevance scores in either Figure 5 or the main text for a more comprehensive analysis.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N1wg0EEzXN", "reviewer": "Reviewer_NBFb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The point studied in this paper is novel and significantly important for understanding the influence of visual instruction tuning on large language models (LLMs). Most existing works indicate that LLMs undergoing multimodal training often exhibit degenerate performance in NLP tasks.\n2. The authors design extensive experiments to support the claims made in this paper.\n3. In addition to the main results—demonstrating that visual instruction tuning can improve truthfulness and ethical alignment—the authors also present other experimental findings, such as the correlation between strong multimodal LLMs (MLLMs) and strong LLMs.\n4. This paper is well-written and easy to read.\n\n### Weaknesses\n\nThe experimental results cannot fully support the claim that visual instruction tuning can improve truthfulness and ethical alignment:\n\n1. The experiment settings in Figure 2 are not comparable to each other. Naively extracting the text part of the visual instruction tuning data makes the instruction tuning data incomplete, severely degrading its quality.\n2. The gap between GPT-4-turbo and GPT-4V is too small, making the difference insignificant.\n3. Instruction tuning with the LLaVA dataset cannot fully support the claim made in this paper:\n\n    3.1 The LLaVA instruction tuning dataset is more like a pure-text instruction tuning dataset, as it is constructed using text-only GPT-4.\n\n    3.2 The study lacks other visual instruction tuning datasets. The authors are highly encouraged to incorporate additional visual instruction tuning datasets, such as Cambrian-10M[1] or ALLaVA[2], to make the claim more general and valid.\n\n4. Complementing point 1 above, the authors are also encouraged to compare models fine-tuned with visual instruction tuning datasets and pure-text instruction tuning datasets. These datasets should be of similar size, and the base LLM should use a version that has not been fine-tuned.\n\n[1] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs\n\n[2] ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper primarily investigates the benefits of incorporating visual instruction tuning data to enhance the truthfulness and ethical alignment of large language models (LLMs). To support this claim, the authors conduct extensive experiments: i) Applying visual instruction tuning to various LLMs of different types and sizes. ii) Comparing the performance of models fine-tuned on pure text data versus those fine-tuned on visual instruction tuning data. iii) Comparing the performance of GPT-4-turbo with GPT-4V.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness above", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission has been reviewed by three experts, all of whom provided insightful comments. The authors did an excellent job revising the paper, including adding more models and presenting text-only tuning results.

- Reviewers NBFb and NThr consider this submission novel and find its evaluation sufficient. However, Reviewer ah3g raised concerns about whether the improvements in Truthfulness and Ethics arise from the visual data or GPT-generated text data. After carefully reviewing the discussion, I believe Reviewer ah3g raises a valid point. I suggest the authors discuss this concern comprehensively in the revised version. It would also be helpful to incorporate the response from the author-reviewer discussion into the paper. Additionally, I believe Section 3.3 (Tuning on Different Vision-Language Data) supports the claim that visual data contributes to improvements in Truthfulness and Ethics. This submission also has the potential to inspire researchers to investigate the effects of multi-modal training on LLM capabilities beyond existing multi-modal benchmarks.

- Overall, the evaluation is thorough, with numerous results that are well-organized. However, Section 3.4 (Analysis on Multi-Modal Benchmarks) requires some clarification. Specifically, the discussions on “Potential Inconsistencies in Current Multi-Modal Benchmarks” and the “Need for Studying Multi-Modal Alignments” are not very clear. Please elaborate on these points with reference to the results in Tables 5 and 6.

In summary, this submission offers a new perspective on the effects of multi-modal training. The evaluation is comprehensive, and the discussions with reviewers have been constructive. Therefore, I recommend acceptance. Please consider incorporating the above suggestions into the revised version.
Decision: Accept (poster)