Venue: NAACL 2024

# What Matters in Training a GPT4-Style Language Model

with Multimodal Inputs?

Yan Zeng\({}^{*12}\) Hanbo Zhang\({}^{*2}\) Jiani Zheng\({}^{*2}\)

Jiangnan Xia\({}^{2}\) Guoqiang Wei\({}^{2}\) Yang Wei\({}^{2}\) Yuchen Zhang\({}^{2}\) Tao Kong\({}^{2}\) Ruihua Song\({}^{\dagger}\)\({}^{1}\)Renmin University of China \({}^{2}\)ByteDance Research

{yanne,rsong}@ruc.edu.cn, {zhb,zjn}@bytedance.com

Equal contribution.Corresponding author.

###### Abstract

Recent advancements in GPT-4V have displayed remarkable multi-modal capabilities in processing image inputs and following open-ended instructions. Despite these advancements, there is considerable scope for enhancing open-source multi-modal LLMs, especially in terms of multi-modal understanding accuracy and instruction-following proficiency. In this paper, we conduct a comprehensive study on training GPT4-style models. We introduce **Lynx**, a multi-modal LLM developed through a series of controlled experiments comparing various model variants. This process allowed us to identify and implement an optimal training strategy tailored for multi-modal LLMs. In addition to our model development, we propose a plug-and-play technique designed to augment the instruction-following capabilities of multimodal LLMs. We have validated the performance of Lynx on multiple benchmarks. Results demonstrate that Lynx not only achieves strong image understanding accuracy but also excels in instruction-following tasks, paving the path for ongoing enhancements in multi-modal LLMs.

## 1 Introduction

Large Language Models (LLMs) Brown et al. (2020); Hoffmann et al. (2022); Chung et al. (2022); Ouyang et al. (2022); Scao et al. (2022); Iyer et al. (2022); Touvron et al. (2023) have progressed rapidly in recent years and achieved impressive performance in language understanding and generalization. With instruction fine-tuning Ouyang et al. (2022); Chung et al. (2022); Wang et al. (2022); Chiang et al. (2023); Xu et al. (2023); Peng et al. (2023), LLMs can be further improved to follow open-ended instructions from non-expert users and serve as dialog-based assistants in our daily lives. Leveraging powerful LLMs, recent studies have examined methods for adapting LLMs to multimodal inputs (e.g., images Alayrac et al. (2022); Zhu et al. (2023); Dai et al. (2023), videos Li et al. (2023); Chen et al. (2023); Zhao et al. (2023), and audio Huang et al. (2023)) and outputs (e.g., vision tasks Wang et al. (2023), and robotic manipulation skills Jin et al. (2023); Driess et al. (2023); Jiang et al. (2022)). Notably, GPT-4V OpenAI (2023b) has astounded the world with its impressively versatile yet practical capabilities, such as generating descriptions, stories, poetry, advertisements, and codes given images, which were rarely observed in previous vision language models Li et al. (2021); Bao et al. (2022); Zeng et al. (2021, 2022).

However, it still remains a mystery that: _How does GPT-4V obtain its impressive smartness?_ GPT-4V can analyze and generate descriptions for various types of images, including diagrams, text in

Figure 1: Illustration of hallucination and diminished instruction-following capabilities generated from open-source multi-modal LLMs.



images, maps, screenshots of software interfaces, illustrations, comics, and medical imagery. Such capabilities likely depend on extensive annotated data and perhaps the integration of tools like OCR. This presents a substantial challenge for open-source multi-modal LLMs which are end-to-end models trained on open-source datasets with lower-quality annotations.

Despite these challenges, we have observed that current multi-modal language models usually suffer from hallucinations, such as generation of facts unrelated to the image inputs, and degraded instruction-following abilities compared to their text-only counterparts, as shown in Figure 1. Though actively investigated recently, the open-source multi-modal LLMs are usually different in training data, training recipes, prompts, and evaluation benchmarks, which makes it challenging to identify which factors are crucial in achieving a strong baseline model.

In this paper, we conduct a comprehensive study on training GPT4-style models. By implementing multiple model variants under controlled settings and conducting extensive experiments to draw reliable conclusions both quantitatively and qualitatively, our findings can be summarized as follows:

* Data quality is more important than quantity. Our experiments with COY0700M (Byeon et al., 2022), DataComp1B (Gadre et al., 2023), and BlipCapFilt (Li et al., 2022) demonstrate that pre-training on COY0700M and DataComp1B does not yield improved model performance. Furthermore, it is crucial to avoid utilizing low-quality annotations during the instruction fine-tuning phase.
* Diversified prompts are essential for enhancing the model's ability to follow instructions. We incorporate a range of open-source multimodal datasets, transforming them into an instruction-following format using manually crafted prompts, supplemented by additional prompts generated by GPT-4, producing 500 prompts for over 50 tasks in total.
* Language instruction fine-tuning plays a significant role in boosting the instruction-following capabilities of multi-modal models. Integrating NLP instruction data during training enables the model to handle a broader range of tasks, compensating for the typical limitations of open-source multi-modal datasets, which are predominantly focused on image captioning and visual reasoning. Furthermore, our results also indicate that multi-modal models based on Vicuna-7B, which have undergone instruction fine-tuning, generally outperform those based on LLaMA-7B.

Through our study, we present **Lynx**, a multi-modal LLM with a three-stage training recipe. The first stage focuses on aligning vision and language, the second stage enhances the resolution of image inputs, and the final stage concentrates on instruction fine-tuning. At each stage, a different set of model parameters is trained using a varied mixture of image-text pairs, multi-modal datasets, and NLP instruction data, in order to fully exploit training data.

Furthermore, we propose a plug-and-play method aimed at boosting the instruction-following capabilities of multi-modal LLMs. Our observations indicate that a more extended dialogue history can effectively unlock and amplify the instruction-following abilities inherent in LLMs. We recognize that most multi-modal tasks can be broken down into two steps: describing the image in details, and subsequently utilizing the capabilities of LLMs. Consequently, during evaluation, we initiate the task by prompting the model to describe the image. This strategy consistently results in improved performance across a variety of multi-modal instruction-following tasks, demonstrating the efficacy of our proposed method.

Experimental results show that Lynx is a strong baseline for multi-modal LLMs, achieving state-of-art performance. 1 Specifically, our model excels in MME's perception (Fu et al., 2023), demonstrating superior image understanding accuracy, and shows impressive results in VisIT-bench (Bitton et al., 2023), which evaluates multi-modal instruction-following abilities. We also adopt the OwlEval test set proposed by mPLUG-owl (Ye et al., 2023) for human evaluation, with Lynx achieving state-of-art multi-modal instruction-following results.

Footnote 1: We compared with open-sourced multi-modal LLMs of comparable model sizes available as of August 31, 2023, for all evaluations.

## 2 Lynnx

Lynx is a large language model that can take images and videos as inputs. In this section, we will introduce our Lynx in detail, including the architecture (2.1) and three-stage training recipe(2.2).

 



### Model Architecture

OverviewOur model takes vision and language as inputs to generate text responses following the input instructions. The overall structure of our model is shown in Figure 2. Concretely, vision inputs are first processed by a vision encoder to get a sequence of vision tokens \({\textbf{w}_{v}}\). After that, \({\textbf{w}_{v}}\) are concatenated with instruction tokens \({\textbf{w}_{l}}\) as the input of LLMs for multi-modal tasks. To generate responses, the left-to-right causal decoder auto-regressively predicts the next token by taking all previous tokens as inputs until encountering the <EOS>.

AdapterThe trainable adapters are inserted into the LLMs after every \(M\) blocks. In our experiments, \(M=2\). As shown in Figure 3(b), the adapter linearly projects each token into a lower-dimensional space and then re-projects it back. Concretely, in Lynx, the hidden state for each token is 4096-d. The adapter first imposes layer normalization Ba et al. (2016) onto the hidden states. Then a linear layer is used to downsample the dimension of each token state from 4096 to 2048, based on which SiLU Elfwing et al. (2018) is set as the non-linear activation function, which keeps consistent with LLaMA Touvron et al. (2023). Finally, the other linear layer is used to re-map the 2048-d hidden state back to 4096-d.

Vision EncoderTo extract vision features of images and video frames, we apply EVA-1B Fang et al. (2023); Sun et al. (2023) as our vision encoder \(\phi_{v}(x)\). It maps an image to a sequence of visual tokens. The downsample rate is 14, meaning that an image with resolution \(H\times W\) will be represented by a sequence of \(\frac{H}{14}\times\frac{W}{14}\) tokens. To improve the efficiency of training and inference, we adapt the resampler \(\Phi\) mechanism Jaegle et al. (2021); Alayrac et al. (2022) that reduces the dimensions of vision inputs by injecting the long vision token sequence into a short and learnable query sequence \({\textbf{w}^{q}_{v}}\):

\[{\textbf{w}_{v}}=\Phi(\phi_{v}(x),{\textbf{w}^{q}_{v}}) \tag{1}\]

where \(x\) is the input image, \(\phi_{v}(x)\) is the raw tokens directly given by the vision encoder, \({\textbf{w}_{v}}\) is the condensed token sequence consisting of 32 tokens regardless of the number of raw tokens from the vision encoder.

### Training Recipe

We propose a three-stage training recipe to enhance image understanding accuracy and instruction-following abilities. Specifically, we first train the model to align vision inputs to the pre-trained LLM, and then we enhance the model by increasing the resolution of visual inputs. Last, we conduct instruction fine-tuning with carefully selected training datasets. Our model is trained on a total of \(\sim\)14B tokens during the pretraining and resolution enhancement stage, \(\sim\)3B tokens during the instruction-finetuning stage, all phases employing causal prediction loss.

PretrainingDuring pretraining, we freeze the vision encoder and language model and train the resampler and the inserted adapters for 100k steps

Figure 3: Architecture of Lynx. (a) Overall; (b) Adapter.

Figure 2: Architecture of Lynx. The vision tokens are concatenated with the text tokens to generate text outputs auto-regressively.



using a batch size of 1536. The learning rate is set to 0.0001 (details about hyper-parameters can be found in Appendix Table D.2). To accelerate pretraining, the image resolution is set to 224x224. To establish alignment between visual features and the language model, we utilize more than 120M image-text pairs, mainly consist of BlipCapFilt 115M Li et al. (2022), CC12M Changpinyo et al. (2021), CC3M Sharma et al. (2018), and SBU Ordonez et al. (2011). Additionally, we also utilize public multi-modal downstream tasks during this phase. We include three predominant tasks, image captioning, visual question answering, and image classification. Details of the datasets we adopted and their mixing percentage are listed in Appendix Table 12.

Resolution EnhancementTraining on low-resolution images is insufficient for certain downstream tasks, such as table reading and OCR. Therefore, after 100k steps of pretraining at low resolution, we increase the input resolution to 420\(\times\)420 and continue training for an additional 10k steps. During this stage, the batch size is reduced to 448, and the learning rate is adjusted to 0.00001. In this phrase, we train the vision encoder together with the resampler and the inserted adapters, adapting the model to higher resolution inputs. Moreover, we adjust the training datasets by excluding large-scale noise image-text pairs and including new multi-modal downstream tasks that requires high-resolution image inputs, such as table reading and OCR.

Instruction FintuningIn this phase, we only train the resampler and the inserted adapters, same as in the pre-training stage. The instruction fine-tuning process consists of 20k training iterations with a batch size of 480 and a learning rate of 0.00002. Our finetuning datasets consists of text-only, image-text, and video-text tasks for complex multi-modal reasoning and instruction following, which mainly belongs to 5 categories: text-only instruction-following task, image/video visual question answering, image/video captioning, classification, and image-conditioned dialog. To fine-tune our model with diversified instructions, we provide appropriate instructions for each of these public datasets (see Appendix Table 13 for details). Specifically, we manually labeled at least 3 different prompts for each of these tasks, and then invoke GPT4 to automatically generate more based on the following "meta prompt", i.e., the prompt used to generate prompts for different tasks:

_Here are some instructions that define a visual-language task. Continue to write 15 instructions with the same meaning: 1) PROMPT1; 2) PROMPT2; 3) PROMPT3;_

Besides, we also collect some available public (visual-)text instruction data (also listed in Appendix Table 12) to further improve the ability of our model to follow open-ended instructions, including the instruction data used in FlanT5 Chung et al. (2022), Alpaca Wang et al. (2022), MiniGPT4 Zhu et al. (2023), LLAVA Liu et al. (2023a), and Baize Xu et al. (2023).

We observe that different combinations of the instruction data have a crucial influence on the final performance. Empirically, we finally impose the weighting strategy presented in Appendix Table 12.

### Inference

During the inference, we employ the nucleus sampling decoding method together with the beam search strategy. Detailed hyper-parameters for the generation process are presented in Appendix Table 9.

Moreover, we propose a plug-and-play method to improve instruction-following capabilities of multi-modal LLMs. We observe that most multimodal tasks can be broken down into two steps: describing the image in details, and subsequently utilizing the inherent capabilities of LLMs. Therefore, we propose to _initiate a task by prompting the model to describe the image first during evaluation_. By doing so, we effectively unlock and amplify the instruction-following abilities inherent in multi-modal LLMs with a more extended dialogue history.

## 3 Experiment

In this section, we aim to answer the following questions by empirical studies:

a) What advantages does our Lynx offer in comparison to existing models? (Section 3.1)

b) What are the key factors in training a high-performance GPT4-style model? (Section 3.2)

c) How does our proposed plug-and-play method enhance the instruction-following capabilities of the model? (Section 3.2)

### Quantitative Experiments

The evaluation of multi-modal LLMs is essentially different from typical visual-language methods. The primary challenge when evaluating a GPT4 

style model is balancing multi-modal understanding accuracy and instruction-following abilities. For a thorough evaluation of both aspects, we adopt four protocols for quantitative evaluation. _The Open-VQA and MME benchmarks primarily assess the precision of visual comprehension, whereas the VisIT-Bench and OwlEval benchmarks evaluate instruction-following capabilities._

Open-VQA benchmarkWe collect an Open-ended Visual Question Answering (Open-VQA) test set, including questions on objects, OCR, counting, reasoning, action recognition, chronological ordering, and more. Different from standard VQA Antol et al. (2015); Zhang et al. (2016), the ground-truth answer in Open-VQA is open-ended. To evaluate the performance on Open-VQA, we prompt GPT4 to make it a discriminator, yielding a 95% agreement with human evaluation, see Appendix B for details. From the results of Table 1 and 2, we can conclude that our model has achieved the best performance both in the image and video understanding tasks. Notably, InstructBLIP Dai et al. (2023) also achieves high performance in most cases, even better than our model in OCR, color recognition, and action recognition tasks. However, we observe that it always outputs one word answer as shown in Appendix Figure 6 and 7, which is less preferred by most of the users (see Table 4). We also showcase some of the examples in Appendix Figure 10. More cases including video VQA examples can be found Figure 10 and 11 in the Appendix. We can see that our model can give the correct answer in most cases as well as a concise reason that supports the answer.

Mme benchmarkWe further compare Lynx with existing open-source models on the MME benchmark Fu et al. (2023). Instruction-answer pairs for the MME benchmark are constructed manually, covering the examination of perception and cognition abilities. These instructions are deliberately succinct, facilitating intuitive and convenient quantitative analysis, as opposed to employing GPT models or manual scoring methods. Results are shown in Figure 4 and Appendix D.4. We can see that our model is a state-of-the-art model in 7 out of 14 subtasks, especially for the perception tasks including color, celebrity, scene, landmark, position, count, and existence. Yet, from the figure, we can also see that our model seems not to perform well on code reasoning, text translation, and numerical. This deficiency may be attributed to the absence of text translation, coding reasoning, and numerical tasks in our training datasets. Moreover, each of these three tasks only contains 20 examples, which may lead to high variance in the evaluation of different checkpoints.

VisIT benchmarkWe further conduct evaluation on the VisIT-Bench Bitton et al. (2023) Single Images benchmark, a benchmark for evaluation of the instruction-following multi-modal LLMs. This benchmark comprises 592 test queries across varied domains, including art, object recognition, spatial understanding, and chemical analysis, etc. Each query is paired with a human-generated, instruction-based caption, enabling automated evaluation against text-only LLMs, and providing a standard answer for reference. The ELO-based results are presented in Table 3, this approach uses an GPT-4 evaluator to compare two models with an instruction and an instruction-conditioned caption. Our model ranks third, underperforming only LLava(13B) and LLava-a1(13B), surpassing counterparts including mPlug-Owl Ye et al. (2023), LlamAadapter-v2 Gao et al. (2023), InstructBLIP Dai et al. (2023), Otter Li et al. (2023), etc, demonstrating the best performance among models with a comparable parameter scale (i.e. 7B).

 

given images. From the human evaluation results in Table 4, we can see that our model has the best instruction-following performance while keeping high performance on the Open-VQA benchmark. BLIP2 Li et al. (2023) and InstructBLIP Dai et al. (2023), though achieved high performance on the Open-VQA benchmark, are not preferred by human users due to their extremely short outputs, i.e., in most cases, they only output one word or phrase as the answer without any explanation. In contrast, MiniGPT4 Zhu et al. (2023) and mPLUG-Owl Ye et al. (2023) keep more instruction-following abilities. Hence, they are preferred over the BLIP models, though they may make more factual errors. We show more results on the OwlEval in Appendix Figure 7.

Overall, if a model has lower accuracy on the Open-VQA and MME benchmark, it tends to make factual errors. However, previous methods with higher performance on these two benchmarks usually tend to lose instruction-following abilities, e.g., always generating short answers, leading to inferior performance on OwlEval and VisIT-Bench. We attribute it to the under-training or over-training on visual-language tasks. Specifically, existing training data from visual-language tasks predominantly have short outputs. Though, by training on these data the model learns vision language alignments, it loses the instruction-following abilities inherited from the large language model. According to the experimental results, we can see that the multi-stage training recipe of Lynx contribute to accurate image understanding and enhanced instruction-following abilities.

### Ablation Study

We conduct an in-depth ablation study to investigate the impact of different components or training recipes on multi-modal understanding and instruction-following performances.

Impact of Training DataWe investigate the impact of data quantity and quality by training our model with or without the large-scale yet noisy image-text pairs COYO700M Byeon et al. (2022)

\begin{table}
\begin{tabular}{l|c c c c c c c|c|c} \hline  & OCR & Counting & Reasoning & Place & Color & Spatial & Action & Others & Overall \\ \hline Open-Flamingo-0 & 20/53 & 5/37 & 15/31 & **18/22** & 5/30 & 7/15 & 11/20 & 53/94 & 44.37 \\ Open-Flamingo-4 & 14/53 & 6/37 & 15/31 & 17/22 & 9/30 & 7/15 & 11/20 & 51/94 & 43.05 \\ Multimodal GPT & 19/53 & 8/37 & 21/31 & 12/22 & 8/30 & 6/15 & 12/20 & 56/94 & 47.02 \\ MiniGPT-4 & 32/53 & 13/37 & 13/31 & 17/22 & 16/30 & **9/15** & 16/20 & 63/94 & 59.27 \\ LLaVA & 21/53 & 8/37 & 13/31 & 11/22 & 12/30 & 4/15 & 16/20 & 49/94 & 44.37 \\ mPLUG-owl & 34/53 & 8/37 & 16/31 & 16/22 & 14/30 & **9/15** & 13/20 & 62/94 & 56.95 \\ BLIP2 & 29/53 & 15/37 & 21/31 & 12/22 & 17/30 & 8/15 & 16/20 & 67/94 & 61.26 \\ InstructBLIP & **41/53** & 20/37 & **26/31** & 14/22 & **23/30** & 6/15 & **18/20** & 77/94 & 74.50 \\ \hline Ours & 36/53 & **25/37** & **26/31** & 17/22 & 21/30 & **9/15** & 17/20 & **79/94** & **76.16** \\ \hline \end{tabular}
\end{table}
Table 1: Quantitative evaluation results (accuracy) on Open-VQA image test set. For all models, we apply the same hyper-parameters defined in Appendix D.3.

\begin{table}
\begin{tabular}{l|r r l|r l|r l|r l} \hline  & \multicolumn{2}{c|}{Action (Y/N)} & \multicolumn{2}{c|}{Others} & \multicolumn{2}{c|}{Overall} & \multicolumn{2}{c}{plug-and-play} \\ \hline InstructBLIP & 62/108 & 21/40 & 56.08 & \multicolumn{2}{c|}{✓} \\ mPLUG-owl & 65/108 & 19/40 & 56.76 & \multicolumn{2}{c|}{✓} \\ MiniGPT-4 & 56/108 & 18/40 & 50.00 & \multicolumn{2}{c|}{✓} \\ \hline Ours & 59/108 & 26/40 & 57.43 & \multicolumn{2}{c|}{} \\ Ours & **69/108** & **29/40** & **66.22** & \multicolumn{2}{c|}{✓} \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of existing open-source multimodal LLMs on the Open-VQA video benchmark.

\begin{table}
\begin{tabular}{l|r l l} \hline  & Elo & matches & Win vs. Reference(w/# ratings) \\ \hline human verified reference & 1361 & 6030 & – & – \\ Lisar(13b)+al & 1206 & 724 & 30.159 & (m=136) \\ Lisar(13b) & 1091 & 5474 & 18.535 & (mt=475) \\ Lyra(TWB*) & 1078 & 708 & 15.159 & (mm=132) \\ nPLUG-owl & 1076 & 5465 & 16.048 & (m=480) \\ lLamaDadre-v2 & 1055 & 5485 & 14.144 & (m=480) \\ idefic?gb & 1030 & 842 & 9.725 & (m=144) \\ Ljra(TWB) & 1012 & 827 & 11.433 & (m=140) \\ InvnetRLIP & 995 & 5505 & 14.12 & (n=503) \\ Outer & 970 & 5495 & 7.015 & (m=499) \\ visual gpt divisi003 & 937 & 5486 & 1.575 & (m=510) \\ Octopus-V2 & 936 & 820 & 8.905 & (m=146) \\ MinigJT-4 & 899 & 5473 & 3.363 & (m=506) \\ OpenHandings & 831 & 5490 & 2.595 & (m=509) \\ PandaGPT(13bb) & 767 & 5480 & 2.706 & (m=519) \\ mmgt & 757 & 5504 & 0.195 & (m=527) \\ \hline \end{tabular}
\end{table}
Table 3: Reference-free Elo rankings on VisIT-Bench (Single Image). Lynnx(7B)* indicates the results applied the proposed plug-and-play method.

 

and DataComp1B (Gadre et al., 2023)). During our experiments, we find training data in both pretraining and finetuning largely influence the model performance. Different from traditional visual-language pretraining (Radford et al., 2021), we find that multi-modal LLMs do not benefit from large-scale but noisy image-text pairs because many of the texts in such datasets are not fluent or natural language expressions. For the generative pretraining in our model, they largely damage the language generation ability as shown in Appendix Figure 9(b). As a result, pretraining on such large-scale datasets achieves no better results than only training on a much smaller but cleaner dataset as evaluated by the human users as shown in Table 5.

Impact of Diversified PromptsIt has been proved to be important to train LLMs on instruction data so as to make them follow instructions correctly (Chung et al., 2022; Ouyang et al., 2022). Therefore, we ablate our model with diversified prompts written by both users and GPT4. The results in Table 5 show that our prompts help to balance different abilities. Moreover, we also find that by using diversified prompts, our model can follow the open-ended instructions better than the ones trained without these prompts (Appendix Table 13). This observation accords with the text-only models. The human evaluation results in Table 5 also accord with our observations.

LLaMA vs. VicunaAs shown in Table 5, in the aspect of multi-modal understanding accuracy, Lynx with instruction-finetuned backbone, namely Vicuna, substantially outperforms the counterpart with LLaMA on the Open-VQA benchmark in both image understanding and video understanding. The human evaluation on OwlEval, however, shows the two model variants have comparable performances. We further test the two variants on a simple yet practical instruction, "give a short answer", and the results show that Vicuna-based model follows the instruction better, producing answers with an average length of 15.81, compared to 20.15 from the LLaMA-based model. One can also refer to Appendix Figure 9(a) for examples of the comparison in terms of their instruction-following ability.

Impact of Larger Image ResolutionOur research includes an ablation study to assess the effect of image resolution on model efficacy. The results presented in Table 5 demonstrate enhanced performance for the model trained with 420\(\times\)420 resolution images relative to those trained with 224\(\times\)224 resolution, with notable improvements observed in OCR and counting tasks within the Open-VQA benchmark, where OCR accuracy increases from 55.6% to 67.9%, and counting accuracy rises from 54.1% to 67.6%. More details can be found in the Appendix Table 7.

Plug-And-Play MethodWe introduce a plug-and-play method to improve the instruction

Figure 5: Illustration of the effectiveness of the proposed plug-and-play method.

\begin{table}
\begin{tabular}{p{71.1pt}|c c|c} \hline  & Open-VQA & Open-VQA & OurL Eval \\  & image & video & (win/all) \\ \hline w/ LLaMA & 70.86 & 60.81 & 42/82 \\ w/o diverse prompts & 75.50 & 59.46 & 18/82 \\ w/ large-scale noisy data & 72.85 & 60.14 & 38/82 \\ w/o high-resolution & 73.51 & 62.16 & - \\ \hline Ours & **76.16** & **66.22** & - \\ \hline \end{tabular}
\end{table}
Table 5: Comparison of ablation models on the Open-VQA benchmark(see Appendix Table 7 for details) for evaluating multi-modal understanding accuracy, and on the OwlEval benchmark for assessing instruction-following abilities. The ’win/all’ metric indicates that, out of all 82 cases in the OwlEval benchmark, this ablation model outperforms in these instances.

 

following capabilities of multi-modal LLMs by first prompting the model with "describe the image in detail". As indicated in Table 3 on VisITBench, our model's rank improves from seventh to third with the proposed method. A similar improvement is observed on the Open-VQA video benchmark, where our model's score increases significantly from 57.43 to 66.22, detailed in Table 2. Experiments on the MME benchmark reveals that the most pronounced gains are in tasks requiring numerical calculation, code reasoning, and text translation, as showed in Table 6. Despite the notable lack of these types of data in our training process, with the proposed plug-and-play method, we unlock the inherent abilities of LLMs for tasks such as code writing or text translation. One showcase of this method is illustrated in Figure 5, where the model provides accurate code based on the information obtained from the dialogue history.

## 4 Related Work

**Centralized Multi-modal Interactive System.** Recent works investigate actively to design of such multi-modal interactive models. One of the most intuitive ideas, such as Visual ChatGPT Wu et al. (2023), MM-REACT Yang et al. (2023), Hugging GPT Shen et al. (2023), InternGPT Liu et al. (2023b), SayCan Ahn et al. (2022), InnerMonologue Huang et al. (2022), integrates various existing individual models or tools. In such a system, the LLM works as a "manager" that directly accepts instructions from users and selects the most appropriate tools to respond to requests while the integrated individual models are "workers" responsible for a specific kind of task. Typically, such models are powerful to address problems that are already well-defined. Yet, they, to some extent, lack zero-shot ability when encountering open-ended instructions which cannot be handled by any of their workers.

**End-to-end Multi-modal Large Language Models.** By contrast, inspired by the recent advances of LLMs, it has also been shown feasible and promising to directly train the neural networks that directly accept multi-modal inputs and output responses end-to-end. To achieve so, one intuitive idea is to adapt the LLMs to multi-modal inputs by adding some additional trainable parameters and finetuning them on multi-modal data. For example, Flamings Alayrac et al. (2022) is one of the early works to explore this idea. Firstly, it takes a vision encoder (like NFNet Brock et al. (2021) in their original version, or recent CLIP ViT Radford et al. (2021)) to extract visual embeddings. Then, it applies multi-layer cross-attention to fuse the multi-modal inputs for the final prediction. Recent works directly concatenate vision embeddings to the inputs of LLMs and finetune LLMs end-to-end. To do so, they usually add an additional projection layer to map the vision embeddings to the same dimension as the language embeddings, and then directly feed them into LLMs for further training. Different methods may take different training strategies. See A for more.

## 5 Conclusions

In this paper, we present Lynx, a multi-modal large language model that can take as input images/videos and responses with open-ended natural languages. Through extensive empirical study, we show that our model outperforms other existing open-source models both in multi-modal understanding accuracy and instruction-following capabilities. We also explore different factors that can affect the performance of a multi-modal large language model and conclude that: 1) the generative pretraining is much more sensitive to the quality of training data than previous methods such as contrastive training; 2) the abilities of instruction following are closely related to the number of different tasks and prompts used for training; 3) improving both the multi-modal understanding accuracy and instruction-following capabilities is important for multi-modal large language models; 4) by first prompting the model to describe the image, the instruction-following abilities can be improved. For future work, it is promising to scale up the model to a larger size (e.g. 30B and 65B LLaMA Touvron et al. (2023)). Moreover, a high-quality multi-modal dataset with diverse instructions is also needed to train such models.

\begin{table}
\begin{tabular}{l|c c c c|c} \hline  & Reasoning & Calculation & Translating & Code & Overall \\ \hline Lynx & 110.71 & 17.50 & 42.50 & 45.00 & 215.71 \\ Lynx* & 103.57 & 55.00† & 75.00† & 77.50† & 311.07† \\ \hline \end{tabular}
\end{table}
Table 6: Comparison of Lynx and Lynx* applied the plug-and-play method on MME cognition benchmark, including commonsense reasoning, numerical calculation, text translation, and code reasoning tasks.

 



## Acknowledgements

This work is supported by the Beijing Natural Science Foundation (L233008) and the National Natural Science Foundation of China (No. 62276268). Moreover, we would like to acknowledge Hang Li at ByteDance for his generous assistance in insightful comments in technical discussions.

## References

* Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022. Do as i can, not as is say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691.
* Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736.
* Antol et al. [2015] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433.
* Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
* Bao et al. [2022] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems, 35:32897-32912.
* Bigham et al. [2010] Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. 2010. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333-342.
* Biten et al. [2019] Ali Furkan Biten, Ruben Tito, Andres Mafla, Iluis Gomez, Marcal Rusinol, Ernest Valveny, CV Javahat, and Dimosthenis Karatzas. 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291-4301.
* Bitton et al. [2023] Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schimdt. 2023. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. arXiv preprint arXiv:2308.06595.
* Brock et al. [2021] Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. 2021. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pp. 1059-1071. PMLR.
* Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Pfaffulla Dhariwal, Arvind Neelakantran, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
* Bulbul et al. [2018] Erhan Bulbul, Aydin Cetin, and Ibrahim Alper Dogru. 2018. Human activity recognition using smartphones. In _2018 2nd international symposium on multidisciplinary studies and innovative technologies (ismsit)_, pages 1-6. IEEE.
* Breeon et al. [2022] Minwoo Breeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. 2022. Cyoo-700m: Image-text pair dataset. [https://github.com/kakaobrain/coyo-dataset](https://github.com/kakaobrain/coyo-dataset).
* Changpinyo et al. [2021] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558-3568.
* Chen and Dolan [2011] David Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190-200.
* Chen et al. [2023] Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jian Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. 2023. Videoollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305.13292.
* Chen et al. [2022] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycer, Basil Mustafa, Lucas Beyert, et al. 2022. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794.
* Chen et al. [2021] Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabo Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021. Webster: A dataset for web-based structural reading comprehension. arXiv preprint arXiv:2101.09465.
* Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325.
* Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Liannin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpulity

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. _arXiv preprint arXiv:2204.02311_.
* [65] Hyung Won Chung, Le Hou, Shayne Longpre, Barrett Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Debghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* [66] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructhlip: Towards general-purpose vision-language models with instruction tuning. _arXiv preprint arXiv:2305.06500_.
* [67] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_, pages 248-255. Ieee.
* [68] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. 2023. Palm-e: An embodied multimodal language model. _arXiv preprint arXiv:2303.03378_.
* [69] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2018. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. _Neural Networks_, 107:3-11.
* [70] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. Eva: Exploring the limits of masked visual representation learning at scale. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 19358-19369.
* [71] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. _arXiv preprint arXiv:2306.13394_.
* [72] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. 2023. Datacomp: In search of the next generation of multimodal datasets. _arXiv preprint arXiv:2304.14108_.
* [73] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. _arXiv preprint arXiv:2101.00027_.
* [74] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shije Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. 2023a. Llama-adapter v2: Parameter-efficient visual instruction model. _arXiv preprint arXiv:2304.15010_.
* [75] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. 2023b. Llama-adapter v2: Parameter-efficient visual instruction model. _arXiv preprint arXiv:2304.15010_.
* [76] Raghav Goyal, Samira Erbraimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yinalios, Moritz Mueller-Freitag, et al. 2017. The" something something smething" video database for learning and evaluating visual common sense. In _Proceedings of the IEEE international conference on computer vision_, pages 5842-5850.
* [77] Michael Grubinger, Paul Clough, Henning Muller, and Thomas Deselaers. 2006. The iaprt tc-12 benchmark: A new evaluation resource for visual information systems. In _International workshop ontoImage_, volume 2.
* [78] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. An empirical analysis of compute-optimal large language model training. _Advances in Neural Information Processing Systems_, 35:30016-30030.
* [79] Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning language models with (almost) no human labor.
* [80] Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2023a. Audiogpt: Understanding and generating speech, music, sound, and talking head. _arXiv preprint arXiv:2304.12995_.
* [81]Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023b. Language is not all you need: Aligning perception with language models. _arXiv preprint arXiv:2302.14045_.
* [82] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Petc Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022. Inner monologue: Embodied reasoning through planning with language models. _arXiv preprint arXiv:2207.05608_.
* [83]Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 6700-6709 

Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shustter, Tianlu Wang, Qing Liu, Punit Singh Korura, et al. 2022. Opt-impl: Scaling language model instruction meta learning through the lens of generalization. _arXiv preprint arXiv:2212.12017_.
* Jaegle et al. (2021) Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. 2021. Perciver: General perception with iterative attention. In _International conference on machine learning_, pages 4651-4664. PMLR.
* Jiang et al. (2022) Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. 2022. Vima: General robot manipulation with multimodal prompts. _arXiv preprint arXiv:2210.03094_.
* Jin et al. (2023) Chuhao Jin, Wenhui Tan, Jiaage Yang, Bei Liu, Ruihua Song, Limin Wang, and Jianlong Fu. 2023. Alphablock: Embodied finetuning for vision-language reasoning in robot manipulation. _arXiv preprint arXiv:2305.18898_.
* Kafle and Kanan (2017) Kushal Kafle and Christopher Kanan. 2017. An analysis of visual question answering algorithms. In _ICCV_.
* Kiela et al. (2020) Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amampreet Singh, Pratik Ringsha, and Davide Testugine. 2020. The hateful memes challenge: Detecting hat speech in multimodal memes. _Advances in Neural Information Processing Systems_, 33:2611-2624.
* Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. _International journal of computer vision_, 123:32-73.
* Li et al. (2023a) Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. 2023a. Otter: A multi-modal model with in-context instruction tuning. _arXiv preprint arXiv:2305.03726_.
* Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. _arXiv preprint arXiv:2301.12597_.
* Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _International Conference on Machine Learning_, pages 12888-12900. PMLR.
* Li et al. (2021) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. _Advances in neural information processing systems_, 34:9694-9705.
* Li et al. (2017) Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. 2017. Person search with natural language description. _arXiv preprint arXiv:1702.05729_.
* Liu et al. (2023a) Haotiian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. _arXiv preprint arXiv:2304.08485_.
* Liu et al. (2023b) Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Yang Yang, Qingyun Li, Jiashuo Yu, et al. 2023b. Internchat: Solving vision-centric tasks by interacting with chatbots beyond language. _arXiv preprint arXiv:2305.05662_.
* Marino et al. (2019) Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In _Proceedings of the IEEE/cvf conference on computer vision and pattern recognition_, pages 3195-3204.
* Mishra et al. (2019) Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. 2019. Ocr-vqa: Visual question answering by reading text in images. In _2019 international conference on document analysis and recognition (ICDAR)_, pages 947-952. IEEE.
* Nguyen et al. (2023) Huu Nguyen, Sameer Suri, Ken Tsui, Shahules786, Together.xyz, and Christoph Schuhmann. 2023. The oig small.
* OpenAI (2023a) OpenAI. 2023a. Gpt-4 technical report. _arXiv_, page 2303.08774.
* OpenAI (2023b) OpenAI. 2023b. Gpt-4v(ision system card).
* Ordonez et al. (2011) Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2text: Describing images using 1 million captioned photographs. _Advances in neural information processing systems_, 24.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. _arXiv preprint arXiv:2304.03277_.
* Pont-Tuset et al. (2020) Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020. Connecting vision and language with localized narratives. In _Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16_, pages 647-664. Springer.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In _International 

conference on machine learning, pp. 8748-8763. Cited by: SS2.
* C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier (2010)Collecting image annotations using amazon's mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's Mechanical Turk, pp. 139-147. External Links: Link Cited by: SS5.
* J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He (2020)Deepspeed: system optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506. Cited by: SS2.
* T. Le Scao, A. Fan, D. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagne, A. Sasha Luccioni, F. Yvon, M. Galle, et al. (2022)Bloom: a 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. Cited by: SS2.
* C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, Theo Coombes, A. Katta, Clayton Mullis, Mitchell Worstman, et al. (2022)Laion-5b: an open large-scale dataset for training next generation imagetext models. arXiv preprint arXiv:2210.08402. Cited by: SS2.
* P. Sharma, N. Ding, S. Goodman, and R. Soricut (2018)Conceptual captions: a cleaned, hypernymed, image alt-text dataset for automatic image captioning.. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565. Cited by: SS2.
* Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang (2023)Hugging gpt: solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580. Cited by: SS2.
* O. Sidorov, R. Hu, M. Rohrbach, and A. Singh (2020)Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 742-758. Cited by: SS2.
* A. Singh, V. Natarjan, M. Shah, Yu Jiang, X. Chen, D. Parikh, and M. Rohrbach (2019)Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8317-8326. Cited by: SS2.
* A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi (2018)A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491. Cited by: SS2.
* Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao (2023)Eva-clip: improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389. Cited by: SS2.
* W. Ren Tan, C. S. Chan, H. Aguirre, and K. Tanaka (2019)Improved artgan for conditional synthesis of natural image and artwork. IEEE Transactions on Image Processing28 (1), pp. 394-409. Cited by: SS2.
* H. Touvron, Thibaut Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Roziere, N. Goyal, E. Hambro, F. Azhar, et al. (2023)LIMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: SS2.
* W. Wang, Z. Chen, X. Chen, J. Wu, X. Zhu, G. Zeng, P. Luo, T. Lu, J. Zhou, Y. Qiao, et al. (2023)VisionIlm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.11175. Cited by: SS2.
* Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi (2022)Self-instruct: aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560. Cited by: SS2.
* C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan (2023)Visual chatgpt: talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671. Cited by: SS2.
* Junbin Xiao, Xindi Shang, Angela Yao, and T. Chua (2021)Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9777-9786. Cited by: SS2.
* Ning Xie, Farley Lai, Derek Doran, and Asim Kadav (2019)Visual entailment: a novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706. Cited by: SS2.
* Canwen Xu, D. Guo, N. Duan, and J. McAuley (2023)Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196. Cited by: SS2.
* Dejing Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang (2017)Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, Cited by: SS2.
* J. Xu, T. Mei, T. Yao, and Y. Rui (2016)Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296. Cited by: SS2.
* Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang (2023)Mmreact: prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381.

 

Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mplug-owl: Modularization empowers large language models with multimodality. _arXiv preprint arXiv:2304.14178_.
* Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. _Transactions of the Association for Computational Linguistics_, 2:67-78.
* Zeng et al. (2021) Yan Zeng, Xinsong Zhang, and Hang Li. 2021. Multi-grained vision language pre-training: Aligning texts with visual concepts. _arXiv preprint arXiv:2111.08276_.
* Zeng et al. (2022) Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, and Wangchunshu Zhou. 2022. x\({}^{2}\)vlm: All-in-one pre-trained model for vision-language tasks. _arXiv preprint arXiv:2211.12402_.
* Zhang et al. (2023) Hanbo Zhang, Yuchen Mo, Jie Xu, Qingyi Si, and Tao Kong. 2023. Invig: Interactive visual-language disambiguation with 21k human-to-human dialogues. [https://github.com/ZhangHanbo/invig-dataset](https://github.com/ZhangHanbo/invig-dataset).
* Zhang et al. (2016) Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 5014-5022.
* Zhao et al. (2023) Yue Zhao, Ishan Misra, Philipp Krahenbuhl, and Rohit Girdhar. 2023. Learning video representations from large language models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 6586-6597.
* Zhou et al. (2017) Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. 2017. Places: A 10 million image database for scene recognition. _IEEE transactions on pattern analysis and machine intelligence_, 40(6):1452-1464.
* Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minight-4: Enhancing vision-language understanding with advanced large language models. _arXiv preprint arXiv:2304.10592_.
* Zhu et al. (2016) Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2016. Visual7w: Grounded question answering in images. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 4995-5004.

## Appendix A Background

BLIP2Li et al. (2022) designs a Q-Former, which is the only trainable part, to align the dimensions of vision and language tokens. PaLM-E Driess et al. (2023), which is built upon PaLM Chowdhrery et al. (2022), is trained totally end-to-end with no fixed layers using a mix of multi-modal datasets including WebLI 10B dataset Chen et al. (2022). Mini-GPT4Zhu et al. (2023) freezes all weights of the vision encoder and the LLM while only finentuning the weights of the projection layer. LLAVA Liu et al. (2023) fixes the vision encoder while keeping the LLMs trainable during the instruction fine-tuning stage. mPLUG-owl Ye et al. (2023) tunes the vision encoder and keeps LLMs fixed to align the vision and language embeddings in the first stage while further tuning the LLMs and keeping the vision encoder fixed in the second instruction-finetuning stage. KOSMOS-1 Huang et al. (2023) does not rely on any pretrained LLMs and is trained from scratch on large amounts of mixed data including image-text pairs COYO700M Byeon et al. (2022), LAION2B Schuhmann et al. (2022), etc.), text corpora Common Crawl, the Pile Gao et al. (2020), etc.), and interleaved image-text data. These models are all powerful and show promising results to develop multi-modal large language models.

## Appendix B Evaluation Protocols

The evaluation of GPT4-style generative language models is challenging because the quality of natural languages is inherently subjective and highly depends on specific cases. Existing models like PaLM-E Driess et al. (2023), PaLI Chen et al. (2022), BLIP2 Li et al. (2023), or InstructBLIP Dai et al. (2023) turn to the evaluation on visual-language benchmarks like image caption Chen et al. (2015) or visual question answering Antol et al. (2015), i.e., fine-tuning multi-modal LLMs on a single downstream task on which the evaluation is conducted. Nevertheless, though it may achieve better performance, over-finetuning on such benchmarks will damage the generation ability of large language models, which conflicts with the primary motivation to use large language models. Moreover, such benchmarks, especially the (semi-)automatically generated ones like TDIUC Kafle and Kanan (2017), always contain a high ratio of easy or noisy examples, making them less suitable. On the contrary, other methods like MiniGPT4 Zhu et al. (2023) or LLaVA Liu et al. (2023) only 

showcase their performance in some challenging yet practical scenarios without quantitative results due to the lack of quantitative benchmarks for such generative multi-modal language models. Therefore, in this section, we propose to evaluate the GPT4-style models in the following two aspects:

* A cleaned subset of visual-language benchmark, which should be challenging and compatible with generative models, with prompted GPT4 to get the quantitative results.
* An open-world challenging yet practical test set to evaluate the performance on realistic scenarios where GPT4-style models are needed, with humans to evaluate the user experience.

To do so, we manually collect an Open-VQA test set consisting of 450 samples with image or video input, which contains diverse questions on objects, OCR, counting, reasoning, action recognition, chronological ordering, etc., from VQA 2.0 Antol et al. (2015), OCRVOA Mishra et al. (2019), Place365 Zhou et al. (2017), MSVD Chen and Dolan (2011), MSRVT Xu et al. (2016), and Something-Something-V2 SthV2) Goyal et al. (2017). Though Place365 is a classification task and SthV2 is a video captioning task, we write proper prompts to make them both VQA tasks. Besides, we carefully examine the data and modify the questions and ground-truth answers if necessary to make them reliably correct and challenging enough to be a benchmark for GPT4-style models. Randomly sampled examples are given in Fig. 8(a). Different from the traditional VQA benchmark, Open-VQA supports open-ended answers. To achieve so, we prompt GPT4 to make it the referee, which achieves a consistency of more than 95% compared with humans2. The prompt for GPT4 used in this phase is as follows:

Footnote 2: We evaluate the consistency on 100 samples from a randomly selected subset with our model.

_Given the question "QUESTION", does the answer "PREDICATION" imply the answer "GROUNDTRUTH"? Answer with Yes or No._

Moreover, general-purpose language generation with image inputs is also important to multi-modal LLMs. Therefore, we also adopt the OwlEval test set proposed by mPLUG-owl Ye et al. (2023), which contains 82 questions based on 50 images, where 21 from MiniGPT-4 Zhu et al. (2023), 13 from MM-REACT Yang et al. (2023), 9 from BLIP2 Li et al. (2023b), 3 from GPT4 OpenAI (2023a), and 4 collected by mPLUG-owl itself. The test set includes diversified and practical cases such as dense image captioning, dialogue writing, story writing, poem writing, teaching, programming, etc.

We give some examples in Fig.8(b). However, OwlEval is proposed together with mPLUG-owl. Hence, directly using it as the benchmark is possibly unfair to other models. To make the comparison fair, we pad each image in the OwlEval with 8 pixels as shown in Fig.8(b) before feeding them into the models. We recruit human annotators to evaluate the performance. Scores range from 1 to 5. If two models are considered to be equally good or bad, they will have the same score. For each data, the annotator will assign a score for each model. We only allow at most 2 models that are equally good or bad, and for each annotator, the total number of ties should be no more than 10 for the whole set. During the evaluation, the correctness has the highest priority, then should be the richness of the generated content.

Figure 6: Qualitative results on our Open-VQA benchmark of different models. We choose InstructBLIP and mPLUG-Owl because they perform best on the Open-VQA benchmark and OwlEval benchmark in all baseline algorithms.

 

* [16] M. C. Bader, A. D. Krizhevsky, and R. G. Hinton. Learning to be done by using a deep neural network. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 259-270, Dec. 2014.
* [17] J. L. Friedman, S. F. Friedman, and P. W. Friedman. The MIT Press, Cambridge University Press, Cambridge, USA, 2008.
* [18] T. Friedman, S. F. Friedman, and P. W. Friedman. The MIT Press, Cambridge University Press, Cambridge, UK, 2008.
* [19] H. Friedman, S. F. Friedman, and P. W. Friedman. The MIT Press, Cambridge University Press, Cambridge, UK, 2008.
* [20] Y. Bengio, I. Guyon, and V. Lux Luxembourg: An introduction to boosting machine learning. _Journal of Machine Learning Research_**39**, 1055-1068, 2002.
* [21] N. Srivastava, E. Sutskever, and I. Sutskever. Dropout: A simple way for reducing the dimensionality of data. In _Advances in Neural Information Processing Systems (NIPS'02)_, volume 1, pages 1-10, Springer, New York, 2002.
* [22] X. Chen, Z. Li, and Q. Weinberger. Deep residual learning with convolutional networks. In _Advances in Neural Information Processing Systems (ICASSP)_, volume 1, pages 1-10, Springer, New York, 2002.
* [23] Y. Deng, L. Fei-Fei, and G. LeCun. ImageNet: Accelerating image recognition via deep learning. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2002.
* [24] Y. Dong, Y. Jia, and J. Sun. Faster rgbg: A review. arXiv preprint arXiv:1406.03015v1 [cs.LG], 2014.
* [25] Y. Dong, Y. Jia, and J. Sun. Faster rgbg: Towards efficient object detection. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [26] Y. Dong, Y. Jia, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [27] Y. Dong, Y. Jia, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [28] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [29] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [30] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [31] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [32] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [33] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [34] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [35] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [36] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [37] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [38] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [39] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [40] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [41] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [42] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [43] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [44] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [45] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [46] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [47] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [48] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [49] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [50] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [51] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [52] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [53] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [54] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [55] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [56] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [57] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [58] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [59] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [60] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [61] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [62] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [63] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [64] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks. In _Advances in Neural Information Processing Systems (ICIP)_, volume 1, pages 1-10, Springer, New York, 2014.
* [65] Y. Jia, J. Sun, and J. Sun. Faster rgbg: Towards efficient object detection via deep convolutional networks.

 

Finally, we also compare our method with others on the newly proposed MME benchmark (Fu et al., 2023) and VisIT-Bench(Bitton et al., 2023).

## Appendix C Discussions and Limitations

### Findings and Takeaways

Multi-modal LLMs are not as instruction-following as LLMs.In our experiments, we find that current multi-modal LLMs are not as good at the instruction following as language models. For example, InstructBLIP (Dai et al., 2023) tends to generate short responses regardless of the input instructions, while other models tend to generate long sentences without considering the instruction like "Give a short answer" or "Answer in one word". We assume that this is from the lacking of high-quality and diversified multi-modal instruction data.

The quality of training data is critical to model performance.As concluded in Section 3.2, based on the experimentation on different pretraining data, we find that a small number of high-quality data with fluent texts can perform even slightly better than the large-scale noisy datasets. We attribute this to the difference between generative pretraining and contrastive pretraining, since generative pretraining is directly learning the conditional distribution of words but not the similarity between texts and images. Therefore, to train a high-performance multi-modal LLM, despite the quantity of data, it is crucial to prepare a high-quality dataset that satisfies: 1) it includes high-quality and fluent texts; 2) it aligns the texts and images well.

Tasks and prompts are crucial for zero-shot abilities.As shown in Section 3.2, diversified prompts have a great impact on the final performance. The essential observation behind this is that the zero-shot generality of multi-modal language models depends on the diversity of tasks involved during training. The model can generalize to more and more unseen instructions as it sees more and more types of tasks. This accords with the observation in text-only models (Radford et al., 2021).

Balancing the correctness and language generation ability is important.In our experiments, we find that if the model is under-trained on downstream tasks such as VQA, it will suffer from the problem of hallucination and keep making mistakes. While if the model is over-trained on downstream tasks, it will not be able to follow the user's instructions to generate long answers. Therefore, it

Figure 8: Examples of our test set. (a) Open-VQA benchmark to validate the accuracy of visual understanding; (b) OwIEval to evaluate the quality of language generation.

 

would be important to carefully balance the training data to train it so as to correctly read images and videos while keeping its generation ability.

### Limitations

EvaluationIt is hard to evaluate a multi-modal large language model since its evaluation is essentially different from traditional visual-language models. Though we take the first step to quantitatively evaluate both the multi-modal understanding accuracy and language generation ability, it is still an open problem: _how can we establish a comprehensive and automatic benchmark to evaluate existing multi-modal large language models?_

Training DataThough we have successfully collected and cleaned a mixed dataset to train our Lynx, we still put a lot of effort to balance different abilities (e.g. correctness and language generation, long and short answers). Moreover, there are still no available image-text datasets that contain long texts which are ideal for pretraining. Besides, restricted by the computational resources that we can use, we do not conduct extensive experiments to find the optimal data combination strategy (e.g. sampling ratios, tasks, and prompts), which has been left for future work.

Multi-lingualOur model is built upon LLaMA Touvron et al. (2023), which is mainly trained on English corpus. Therefore, our model is not that good at multi-lingual responses. Though it can understand and sometimes output other languages (like shown in Figure 15), it is still unexplored how to build a high-performance multi-lingual and multi-modal large language model.

SafetyCurrently, we do not conduct safety checks and restrict the outputs of our model. Therefore, the model may output contents that are not appropriate and even toxic, depending on and restricted by the data used for training. The authors do not support the use of harmful language generation using our codes and models, like any usage on ethical, political, and racism issues.

## Appendix D Experimental Details

### Ablation Study

#### Training Details

The model comprises approximately 8B parameters, of which around 1B are trainable. We use the DeepSpeed Rasley et al. (2020) to accelerate training, and set the BFloat16 as the default model precision. We report the detailed model training hyperparameters in Table 8.

#### Hyper-parameters for Generation

For MiniGPT4 Zhu et al. (2023), we generated the response with its default settings. Similarly, for mPLUG-owl Ye et al. (2023), we follow the default parameters presented at [http://vlarena.opengylab.com/](http://vlarena.opengylab.com/). Detailed settings can be found in 9 for different tasks.

### MME Performance

### Training Data
 



\begin{table}
\begin{tabular}{l|c c c c c c c|c c c} \hline  & \multicolumn{8}{c}{**images**} & \multicolumn{4}{c|}{**video**} \\ \hline  & OCR & Counting & Reasoning & Place & Color & Spatial & Action & Others & Overall & Action (Y/N) & Others & Overall \\ \hline w/ LLaMA & 33/53 & 18/37 & 19/31 & 17/22 & 22/30 & 10/15 & 17/20 & 78/94 & 70.86 & 65/109 & 25/40 & 60.81 \\ w/o diverse prompts & 33/53 & 22/37 & 23/31 & 20/22 & 21/30 & 12/15 & 17/20 & 80/94 & 75.50 & 62/109 & 26/40 & 59.46 \\ w/ large-scale noisy data & 33/53 & 20/37 & 28/31 & 17/22 & 17/30 & 10/15 & 16/20 & 79/94 & 72.85 & 63/109 & 26/40 & 60.14 \\ w/o high-resolution & 30/53 & 20/37 & 26/31 & 15/22 & 25/30 & 8/15 & 19/20 & 79/94 & 73.51 & 66/109 & 26/40 & 62.16 \\ \hline Ours & 36/53 & 25/37 & 26/31 & 17/22 & 21/30 & 9/15 & 17/20 & 79/94 & 76.16 & 69/109 & 29/40 & 66.22 \\ \hline \end{tabular}
\end{table}
Table 7: Quantitative evaluation of different ablation models on the Open-VQA benchmark.

\begin{table}
\begin{tabular}{l|c c c c} \hline Hyperparameters & Pretraining & Resolution Enhancement & Instruction Finetuning \\ \hline Env & A100*32 & A100*32 & A100*24 \\ Training steps & 100,000 & 10,000 & 20,000 \\ Warup steps rate & 0.05 & 0.05 & 0.05 \\ WarupUp read & le-5 & le-6 & 2-6 \\ Optimizer & AdamW & AdamW & AdamW \\ Learning rate decay & linear & linear & linear \\ Adam \(\epsilon\) & le-8 & le-8 & le-8 \\ Adam \(\beta\) & 03, 09, 099 & (03, 0999) & (03, 0999) \\ Weight decay & 0.01 & 0.01 & 0.01 \\ Training Time & 3 days & 8 hours & 16 hours \\ \hline \end{tabular}

* The hyperparameters to generate the 0th-round detailed description, if applicable.

\end{table}
Table 8: Training hyperparameters. Some parameters not use learning rate decay schedule.

\begin{table}
\begin{tabular}{l|c c c c c} \hline  & max new tokens & beam size & top-p & top-k & length penalty & no repeat ngram & do sample \\ \hline Image Description & 64 & 5 & 1.0 & 1 & -2.0 & 2 & False \\ Open-VQA image & 64 & 5 & 1.0 & 1 & -2.0 & 2 & False \\ Video Description\({}^{*}\) & 128 & 1 & 0.9 & 3 & 1.0 & 3 & True \\ Open-VQA video & 128 & 3 & 1.0 & 1 & -1.0 & 3 & False \\ OwElval Description\({}^{*}\) & 128 & 1 & 0.9 & 3 & 1.0 & 3 & True \\ OwElval & 256 & 3 & 0.9 & 3 & 1.0 & 3 & True \\ VisIT-Bench Description\({}^{*}\) & 128 & 1 & 0.9 & 3 & 1.0 & 3 & True \\ VisIT-Bench & 256 & 3 & 0.9 & 3 & 1.0 & 3 & True \\ MME Benchmark & 64 & 5 & 1.0 & 1 & -2.0 & 2 & False \\ demo(ours) & 256 & 3 & 0.9 & 3 & 1.0 & 3 & True \\ \hline \end{tabular}

* The hyperparameters to generate the 0th-round detailed description, if applicable.

\end{table}
Table 9: Hyper-parameters for visual question answering evaluation and general-purpose natural language generation with vision inputs respectively. We set hyper-parameters to encourage short response generation for the Open-VQA benchmark and MME Benchmark.



* [15] M. C. B. K. Wilson, and R. J. S. Thorne, "The role of the _Journal of Machine Learning Research_, vol. 30, no. 2, pp. 47-68, 1970.
* [16] A. D. Martin, "A survey on machine learning in Python: a large-scale hierarchical approach to boosting neural networks," _arXiv preprint arXiv:1905.05815v1_, 2019.
* [17] G. P. Kingma and Ba Ba, "Adam: A method for stochastic optimization," in _Advances in Neural Information Processing Systems (NIPS)_, volume 1, pages 1-10, 2014. PMLR, 2014.
* [18] L. Bottou, Y. Bengio, and T. Salakhutdinov, "Dropout normalization with deep residual learning," in _Proceedings of the Thirtieth International Conference on Artificial Intelligence_, 2015.
* [19] V. Balcanathan, I. Goodfellow, and N. Mirza, "Generative adversarial nets for image recognition," _IEEE Transactions on Pattern Analysis and Machine Intelligence_, vol. 28, no. 1, pp. 1-10, 2015.
* [20] H. Bahdan, F. Bachrach, and E. Rabinovich, "Deep learning for image recognition," _IEEE transactions on pattern analysis and machine intelligence_, vol. 10, no. 1, pp. 1-10, 2015.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l|c c c c c c c c}  & BLP2 (Li et al., 2023b) & Internet- & LLAMA- & mPLUG & MiniOPT4 (Zhu et al., 2023a) & LLAMA (Liu et al., 2023b) & Ours- \\  & RLIP (Chi et al., 2019) & Adapter Y2 (Guo et al., 2023b) & Out (Yoc et al., 2023) & MiniOPT4 (Zhu et al., 2023) & LSAVA (Lin et al., 2023b) & Ours- \\ \hline Existence & 3 & 2 & 5 & 5 & 6 & 7 & 1 \\ Count & 3 & 2 & 5 & 4 & 7 & 1 \\ Position & 3 & 4 & 7 & 5 & 2 & 6 & 1 \\ Color & 3 & 2 & 5 & 6 & 4 & 7 & 1 \\ Poster & 1 & 4 & 5 & 2 & 6 & 7 & 3 \\ Coleley & 2 & 3 & 5 & 4 & 6 & 7 & 1 \\ Scene & 4 & 2 & 3 & 5 & 6 & 7 & 1 \\ Landmark & 4 & 5 & 3 & 2 & 6 & 7 & 1 \\ Artwork & 1 & 2 & 5 & 4 & 6 & 7 & 3 \\ OCR & 2 & 5 & 1 & 6 & 3 & 7 & 4 \\ \hline Perception & 2 & 3 & 4 & 5 & 6 & 7 & 1 \\ \hline Communisee & 3 & 1 & 4 & 4 & 6 & 7 & 2 \\ Numerical & 5 & 6 & 1 & 2 & 3 & 4 & ? \\ Text Translation & 2 & 3 & 6 & 1 & 5 & 4 & ? \\ Code Reasoning & 2 & 3 & 5 & 4 & 1 & 6 & ? \\ \hline Cognition & 3 & 2 & 5 & 4 & 1 & 7 & 6 \\ \end{tabular}
\end{table}
Table 11: Rankings of existing open-sourced multi-modal LLMs on MME benchmark (Fu et al., 2023).

Figure 10: More cases on our Open-VQA image benchmark.





\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} Dataset & Total size & Type & Pretrain & Pretain Ratio & Finetune & Finetune Ratio \\ \hline BlipCapFilt (Li et al., 2022) & 102.8M & Image-text Pair & ✓ & 30.525\% & ✗ & - \\ CC12M (Changpinyo et al., 2021) & 8.3M & Image-text Pair & ✓ & 2.465\% & ✗ & - \\ CC3M (Sharma et al., 2018) & 2.9M & Image-text Pair & ✓ & 10.076\% & ✗ & - \\ \hline SBU (Ordonez et al., 2011) & 859.7K & Image Caption & ✓ & 2.987\% & ✗ & - \\ TextCaps (Sidorov et al., 2020) & 109.8K & Image Caption & ✓ & 0.381\% & ✗ & - \\ COCO Caption (Chen et al., 2015) & 82.7K & Image Caption & ✓ & 0.287\% & ✗ & - \\ CUHR-PEDES (Li et al., 2017) & 34.1K & Image Caption & ✓ & 0.118\% & ✗ & - \\ FlickrJok (Young et al., 2014) & 29.8K & Image Caption & ✓ & 0.104\% & ✗ & - \\ Peeks 110k & 26.2X & Image Caption & ✓ & 0.091\% & ✗ & - \\ LLaVA Caption (Liu et al., 2023a) & 23.2K & Image Caption & ✗ & - & ✓ & 0.945\% \\ IAPR TC-12 (Grubinger et al., 2006) & 20.0K & Image Caption & ✓ & 0.069\% & ✗ & - \\ Visual Genome Caption (Krishna et al., 2017) & 19.6K & Image Caption & ✗ & - & ✓ & 0.798\% \\ MiniGPT4 IFT (Zhu et al., 2023) & 3.4K & Image Caption & ✗ & - & ✓ & 0.138\% \\ Pascal Sentences (Rashtchian et al., 2010) & 1.0K & Image Caption & ✓ & 0.003\% & ✗ & - \\ \hline VGQA (Krishna et al., 2017) & 1.4M & VQA & ✓ & 8.711\% & ✓ & 10.880\% \\ GQA (Hudson and Manning, 2019) & 943.0K & VQA & ✓ & 5.868\% & ✓ & 3.999\% \\ OCRVQA (Mikrabarti et al., 2019) & 894.0K & VQA & ✓ & 5.364\% & ✓ & 12.349\% \\ VQAV2 (Antol et al., 2015) & 443.8K & VQA & ✓ & 2.761\% & ✓ & 3.449\% \\ Visual7W (Zhu et al., 2016) & 139.9K & VQA & ✓ & 0.870\% & ✓ & 0.593\% \\ ViZWiz (Bigham et al., 2010) & 20.5K & VQA & ✓ & 0.128\% & ✓ & 0.087\% \\ OMVQA (Marino et al., 2019) & 9.0K & VQA & ✓ & 0.056\% & ✓ & 0.038\% \\ TDIUC (Kafle and Kannan, 2017) & 705.4K & VQA & ✓ & 4.389\% & ✗ & - \\ WebSRC (Chen et al., 2021) & 131.3K & VQA & ✗ & - & ✓ & 1.814\% \\ LLaVA Reasoning (Liu et al., 2023a) & 76.6K & VQA & ✗ & - & ✓ & 3.119\% \\ TextVQA (Singh et al., 2019) & 34.6K & VQA & ✗ & - & ✓ & 0.478\% \\ STVQA (Biten et al., 2019) & 26.0K & VQA & ✗ & - & ✓ & 0.359\% \\ Places365 (Zhou et al., 2017) & 1.8M & Classification & ✓ & 10.921\% & ✓ & 5.000\% \\ ImageNetIK Deng et al., 2009) & 1.3M & Classification & ✓ & 7.887\% & ✗ & - \\ SNLI-VE Xie et al., 2019) & 529.5K & Classification & ✓ & 3.213\% & ✗ & - \\ Visual7W Multi-choice Zhu et al., 2016) & 139.9K & Classification & ✓ & 0.849\% & ✗ & - \\ AiCrrowdGoDai et al., 2018) & 100.3K & Classification & ✓ & 0.609\% & ✗ & - \\ NLVR2 (Suhr et al., 2018) & 86.4K & Classification & ✓ & 0.518\% & ✓ & 0.671\% \\ WikArt (Tan et al., 2019) & 42.5K & Classification & ✓ & 0.264\% & ✓ & 0.180\% \\ HAR (Bulbul et al., 2018) & 12.6K & Classification & ✓ & 0.078\% & ✓ & 0.053\% \\ TimeClassification & 11.5K & Classification & ✓ & 0.072\% & ✓ & 0.049\% \\ HatefulMemes (Kiela et al., 2020) & 8.5K & Classification & ✓ & 0.026\% & ✗ & - \\ \hline MSR-VTT-QA (Xu et al., 2016, 2017) & 158.6K & Video VQA & ✗ & - & ✓ & 3.137\% \\ VLN VQA (Pont-Tuset et al., 2020) & 31.8K & Video VQA & ✗ & - & ✓ & 0.629\% \\ NeXT-QA Xiao et al., 2021) & 31.5K & Video VQA & ✗ & - & ✓ & 0.623\% \\ MSVD-QA (Chen and Dolan, 2011; Xu et al., 2017) & 30.9K & Video VQA & ✗ & - & ✓ & 0.611\% \\ \hline SthV2 (Goyal et al., 2017) & 168.9K & Video Caption & ✗ & - & ✓ & 5.000\% \\ VLN Caption (Pont-Tuset et al., 2020) & 17.6K & Video Caption & ✗ & - & ✓ & 5.000\% \\ \hline LLaVA Instruction (Liu et al., 2023a) & 361.4K & Dialog & ✗ & - & ✓ & 5.845\% \\ LLaVA Dialog Liu et al., 2023a) & 256.9K & Dialog & ✗ & - & ✓ & 4.155\% \\ InViG (Zhang et al., 2023) & 49.9K & Dialog & ✓ & 0.310\% & ✗ & - \\ \hline Flan V2 (Chung et al., 2022) & & Text Instructions & ✗ & - & ✓ & 15.000\% \\ LAION OIG Small (Nguyen et al., 2023) & 210.3 & Text Instructions & ✗ & - & ✓ & 3.884\% \\ Alpaca GPT4 (Wang et al., 2022) & 51.7 & Text Instructions & ✗ & - & ✓ & 0.955\% \\ Unnatural Instruction (Honovich et al., 2022) & 8.7 & Text Instructions & ✗ & - & ✓ & 0.161\% \\ Baize (Xu et al., 2023) & 601.1 & Text Instructions & ✗ & - & ✓ & 10.000\% \\ \end{tabular}
\end{table}
Table 12: Training Data.





\begin{table}
\begin{tabular}{l|c|l} \hline Dataset & Type & Prompt Example \\ \hline BlipCapFilt & Image-text Pair & Describe the image briefly. \\ CC12M & Image-text Pair & Write a relevant description to pair with the image. \\ CC3M & Image-text Pair & Write a relevant description to pair with the image. \\ \hline SBU & Image Caption & Describe the image. \\ TextCaps & Image Caption & Describe the image shortly by reading the texts. \\ COCO Caption & Image Caption & Describe the image briefly. \\ CUHK-PEDES & Image Caption & Describe the person in the image. \\ Flickr30k & Image Caption & Describe the image briefly. \\ Pexels 110k & Image Caption & Describe the image briefly. \\ LLaVA Caption & Image Caption & INSTRUCTION1  \\ LAPR TC-12 & Image Caption & Describe the key elements in the image. \\ Visual Genome Caption & Image Caption & Describe the image in detail. \\ MinGPT4 IFT & Image Caption & Describe the image in detail. \\ Pascal Sentences & Image Caption & Describe the image briefly. \\ \hline VGQA & VQA & [QUESTION]2 Give a short answer. \\ GOA & VQA & [QUESTION] Give a short answer. \\ OCRVQA & VQA & [QUESTION] Give a short answer. \\ VQav2 & VQA & [QUESTION] Give a short answer. \\ Visuat7W & VQA & [QUESTION] Give a short answer. \\ VizWiz & VQA & [QUESTION] Give a short answer. \\ OKVQA & VQA & [QUESTION] Give a short answer. \\ TDIUC & VQA & [QUESTION] Give a short answer. \\ WebSRC & VQA & Answer the question briefly by reading the webpage. [QUESTION] \\ ILaAv Reasoning & VQA & [QUESTION] \\ TextVQA & VQA & Answer the question shortly by reading the texts. [QUESTION] \\ STVQA & VQA & [QUESTION] Give a short answer. \\ \hline Places365 & Classification & Where is this? Answer with a place name. \\ ImageNet1K & Classification & What is in the image? Answer with its name. \\ SNLI-VE & Classification & Does the image semantically entail the following text? Text: [HYPOTHESI]3 Options: 1. neutral 2. entailment 3. contradiction \\ Visual7W Multi-choice & Classification & Choose the correct answer. Question: [QUESTION] Options: [OPITONS]4 \\ AirCrowdFood & Classification & What food is it? \\ NLVR2 & Classification & Given the claim ”[HYPOTHESIS]”, is it True or False? \\ WikiArt & Classification & What artistic movement or style dose this art picture belong to? Answer with a style name. \\ HAR & Classification & What is the person doing? Answer shortly. \\ TimeClassification & Classification & What is the time now? Give a short answer. \\ HatefulMemes & Classification & Is “[MEME]” a hateful meme? Answer with Yes or No. \\ \hline MSR-VTT-QA & Video VQA & [QUESTION] Give a short answer. \\ VLN VQA & Video VQA & [QUESTION] Give a short answer. \\ NeXT-QA & Video VQA & [QUESTION] Give a short answer. \\ MSVD-QA & Video VQA & [QUESTION] Give a short answer. \\ \hline SthV2 & Video Caption & Describe the video shortly. \\ VLN Caption & Video Caption & Describe the video in detail. \\ \hline LLaVA Instruction & Dialog & [DIALOG]6 \\ ILaAv Dialog & Dialog & [DIALOG] \\ InViG & Dialog & [DIALOG] \\ \hline Flan V2 & Text Instructions & [INSTRUCTION] \\ LAION OIG Small & Text Instructions & [INSTRUCTION] \\ Alpaca GPT4 & Text Instructions & [INSTRUCTION] \\ Unnatural Instruction & Text Instructions & [INSTRUCTION] \\ Baize & Text Instructions & [INSTRUCTION] \\ \hline \end{tabular}

* We directly use the original instructions if it is applicable.
* 2. [QUESTION] means the original question from the dataset. We do some post process, e.g., add the “-” if it is missing.
* 3. [HYPOTHESIST] means the statement in the SRVLIVE dataset.
* 4. (OPTIONS) means the options list provided by the dataset.
* 5. [MEME] is the meme included in the Hateful Memes dataset.
* 6. We split each dialog into rounds, and prompt the model to speak for the next round based on the dialog history.

\end{table}
Table 13: Prompt examples of different datasets.



* [16] M. C. Bader, R. D. Kiefer, and A. J. Brox, "A new approach to the design of a multi-layer architecture for deep learning," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 2354-2358, 2017.
* [17] S. G. Hinton, I. Goodfellow, and Y. Bengio, "Deep residual networks: A simple way to prevent neural networks from overfitting," in _Advances in Neural Information Processing Systems (NIPS'98)_, volume 1, pp. 1398-1403, 2009.
* [18] L. Bottou, E. Rabinovich, and V. Vanhoucke, "Batch normalization by reducing internal covariate effects," _arXiv preprint arXiv:1412.6098_, 2014.
* [19] P. Baud, N. Srivastava, and A. Zisserman, "Dropout classification with dropout layers," in _Neural Networks (NIPS'98)_, volume 1, pp. 1-10, 2009.
* [20] T. Chua, H. Li, and X. Zhang, "Dropout detection using recurrent neural network models," in _IEEE International Conference on Learning Representations (ICLRW & Teholinguistic Association)_, pages 1-10, 2015.

[MISSING_PAGE_POST]

 


Figure 12: More cases on our OwlEval benchmark.

 

the image is many because it is a cell designed up in a console monitor software, sitting on a couch with a plate of cold nodules in front of the red to holding a plate tip. In games and there are no cookies for the blue tape clip of the table with the white gate filter tube has a rather high number of cells (both on top and here) a whole subcubic on the white gate filter tube. The cookies are in the shape of string tension, which adds to the hammer of the image. The captioner will suddenly inject its auto aming, as it implies that the carls behavior is unusual or at one of the ordinary.

The image was a sense of whitney and jayfulnesses, as it sounds similar as typical stimuli exhibit videos and imaginative play. The fact that the safe cat is wearing plasma burst further emphasizes how much images pass through the same area moving. The presence of the two people asks another way of thumb: the outside arm/is-like arrows emporter of the code or simply engaging in a casual conversation while observing the scene. Overall, the image captures the essence of a fan and lightheaded moment, making it an empirple and memorative movement for viewers.
 


Figure 14: Multi-turn dialog demonstrations.

 

Ours:

voiler de probleme dans formee d'un soleul couver de l'espace leursque la mati descendent que isole se retire, un solell sur la mer. le voilver et obobt avur le port de lennification or legage vers function. il y a une nature translation à farriere de la volleuse, a men est came ont elaboration à farther de la volleuse, a men est came et tel le celt eeu rouge et orange, la soleli est couchte dearles la montagne en amie-glam.

Figure 15: Multi-lingual demonstrations.

 

* [15] M. Cacciari, G. P. Salam, and R. S. Thorne, "A new approach to boosting the model for multi-channel channels," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2048-2069, 2013.
* [16] A. D. Krizhevsky, I. Sutskever, and E. Rabinovich, "Improving deep neural networks with applications to image recognition," in _IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 779-783, 2014.
* [17] J. L. Friedman, "The maximum likelihood method for combining multiple images," _Journal of Machine Learning Research_, vol. 10, no. 1, pp. 1-10, 1994.
* [18] T. F. Hastieres, "An introduction to machine learning," _Machine Learning_, vol. 1, no. 1, pp. 1-10, 1994.
* [19] H. B. Nielsen, "Learning to improve the quality of a single image by exploiting the information from tiny images," _arXiv preprint arXiv:1405.03015v1_, 2014.
* [20] Y. Bengio, "Deep residual learning for large data sets," _Advances in Neural Information Processing Systems_, vol. 1, no. 1, pp. 1-10, 2006.
* [21] W. Dong, X. Li, Z. Zhang, and Q. Weinberger, "Deep residual learning for large scale visual recognition," _CoRR_, abs/1412.03015v1, 2014.
* [22] V. Kalicka, "Image classification via deep convolutional nets," in _NIPS Workshop on Deep Representation Learning (ICLR)_, p. 1-1, 2015.

[MISSING_PAGE_POST]