MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
Authors: Deyao Zhu
Jun Chen
Xiaoqian Shen
Xiang Li
Mohamed Elhoseiny
Venue: ICLR 2024 poster (2024)
Keywords: large language models
vision language models
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed.
We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). 
To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. 
Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, 
such as detailed image description generation and website creation from hand-drawn drafts.
Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. 
In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability.

Published as a conference paper at ICLR 2024
MINIGPT-4:
ENHANCING VISION-LANGUAGE UNDERSTANDING WITH ADVANCED LARGE LANGUAGE MODELS
Deyao Zhu", Jun Chen", Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny King Abdullah University of Science and Technology fdeyao. zhu, jun. chen, xiaoqian, shen, x sia... 1 mohamed e wwwhhseiinggn @kuust edu.
ABSTRACT
The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision- language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LL.M.. To examine this phenomenon, we present MiniGPT-4, which aligns frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation drafts. Furthermore, we also observe other
and website creation from emerging capabilities MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g.. repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available https: / //mmmiiiiit gi hub. io/.
INTRODUCTION
In recent years, large language models (LLMs) have experienced rapid advancements (Ouyang et al., 2022; OpenAI, 2022; Brown et al., 2020; Scao et al., 2022a; Touvron et al., 2023; Chowdhery et al., 2022; Hoffmann et al., 2022). With exceptional language understanding capabilities, these models can perform a variety of intricate linguistic tasks in a zero-shot manner. Notably, GPT-4, a large-scale multimodal model, has been recently introduced and demonstrated several impressive capabilities of vision-languag understanding and generation (OpenAI, 2023). For example, GPT-4 can produce detailed and accurate image descriptions, explain unusual visual phenomena, and even construct websites based on handwritten text instructions.
Although GPT-4 has exhibited remarkable vision language capabilities, the methods behind its exceptional abilities are still a mystery (OpenAI, 2023). We believe that these impressive skills may stem from the utilization of more advanced large language model (LLM). LLMs have demonstrated various emergent abilities, evidenced in GPT-3's few-shot prompting setup (Brown et al., 2020) and the findings of Wei et al. (2022) (Wei et al., 2022). Such emergent properties are hard to find in smaller-scale models. It is conjectured that these emergent abilities also applicable to multi-modal models, which could be the foundation of GPT-4's impressive visual description capabilities.
To substantiate our hypothesis, present novel vasion-language model named MiniGPT-4. It utilizes an advanced large language model (LLM), Vicuna (Chiang et al., 2023), which is built upon
cqual contribution

---

Published as a conference paper at ICLR 2024
The logo design is simple minimalistic, with a pink drawing of flamingo standing on one leg in the water. The design is clean and easy to recognite, making it flamingo-
such logo for beach themed event. The use of flamingo as a symbol adds touch of whimsy and fun to the design, making memorable and eye-catching. Overall, well-designed logo that effectively communicates the brand's message.
suitable for use in various
Vicuna
BER Human:
Linear Layer
What you logo design? eut Assistant:
Q-Former
VIT
The architecture of MiniGPT-4. It consists of a vision encoder with pretrained ViT and Q-Former, a single linear projection layer, and an advanced Vicuna large language model. MiniGPT-4 only requires training the linear projection layer to align the visual features with the Vicuna.
Figure I:
LLaMA (Touvron et al., 2023) and reported to achieve 90% of ChatGPT's quality per GPT-4's evaluation, as the language decoder. In terms of visual perception, we employ the same pretrained vision components of BLIP-2 (Li et al., 2023c) that consists of ViT-G/14 from EVA-CLIP (Fang et al., 2022) and a Q-Former network. MiniGPT-4 adds a single projection layer to align the encoded visual features with the Vieuna language model and freezes all the other vision and language components. MiniGPT-4 is initially trained for 20k steps using a batch size of 256 on 4 A 100 GPUs, leveraging a combined image captioning dataset that includes images from LAION (Schuhmann et al., 2021), Conceptual Captions (Changpinyo et al., 2021; Sharma et al., 2018), and SBU (Ordonez et al., 2011) to align visual features with the Vicuna language model. Nevertheless, merely aligning visual features with the language model (LLM) is inadequate to ensure robust visual conversation capabilities, resembling that of a chatbot. The presence of underlying noise in raw image-text pairs can lead to subpar language outputs. Therefore, we collect another 3,500 detailed image description pairs to further fine-tune the model with designed conversational template in order to improve the naturalness of the generated language and its usability.
In our experiments, we discovered that MiniGPT-4 possesses numerous capabilities similar to those demonstrated by GPT-4. For instance, MiniGPT-4 can generate intricate image descriptions, create websites based on handwritten text instructions, and explain unusual visual phenomena. Furthermore, our findings revealed that MiniGPT-4 also has variety of other intriguing abilities not showcased in the GPT-4 demonstrations. For example, MiniGPT-4 can directly generate detailed cooking recipes from food photos, write stories or poems inspired by images, write advertisements for products in images, identify problems shown in photos and provide corresponding solutions, and retrieve rich facts about people, movies, or art directly from images, among other capabilities. These abilities are absent in previous vision-language models like Kosmos-1 I (Huang et al., 2023) and BLIP-2 (Li et al., 2023c) that use less powerful language models. This further validates that integrating visual features with an advanced language model is one of the keys to enhancing vision-language models. We present a summary of our key findings:
Our research reveals with compelling evidence that by aligning visual features with advanced large language models like Vicuna, MiniGPT-4 can achieve advanced vasion-language capabilities comparable to those exhibited in the GPT-4 demonstrations.
Our findings suggest that training merely one projection layer can effectively align a pre- trained vision encoder with the large language model. Our MiniGPT-4 only requires training approximately 10 hours on 4 A100 GPUs.
We discovered that simply aligning visual features with large language models using short image caption pairs is not sufficient for developing a well cell-perfonmin model and leads to unnatural language generation. Further finetuning with a small but detailed image description pairs can address this limitation and significantly improves its usability.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORKS
Large language models have experienced tremendous success in recent years due to the scaling up of training data and an increase in the number of parameters. Early models, such as BERT (Devlin et al., 2018), GPT-2 (Radford et al., 2019), and T5 (Raffel et al., 2020), laid the foundation for this progress. Subsequently, GPT-3 (Brown et al., 2020), with a massive scale of 175 billion parameters, was introduced, demonstrating significant breakthroughs across numerous language benchmarks. This development inspired the creation of various other large language models, including Megatron- Turing NLG (Smith et al., 2022), Chinchilla (Hoffmmann et al., 2022), PaLM (Chowdhery et al., 2022), OPT (Zhang et al., 2022), BLOOM (Scao et al., 2022b), and LLaMA (Touvron et al., 2023), among others. Wei et al. (Wei et al., 2022) further discovered several emergent abilities, which appear exclusively in large models. The emergence of these abilities underscores the importance of scaling up in the development of large language models. Moreover, by aligning the pre-trained large language model GPT-3 with human intent, instructions and human feedback, InstructGPT (Ouyang al., 2022) and ChatGPT (OpenAI, 2022) enable conversational interactions with humans and can answer a wide range of diverse and complex questions. More recently, several open-soureed models, such as Alpaca (Taori al., 2023) and Vicuna (Chiang et al., 2023), have been developed based on LLaMA (Touvron et al., 2023) and also exhibit similar performance.
Leveraging Pre-trained LLMs in Vision-Language Tasks. The use of autoregressive language models as decoders in asion-language tasks has become increasingly popular (Chen et al., 2022; Huang et al., 2023; Yang et al., 2022: Tiong et al., 2022; Alayrac et al., 2022; Li et al., 2023e; 2022; Driess et al., 2023), facilitating cross-modal knowledge transfer. Notable examples include VisualGPT (Chen et al., 2022) and Frozen (Tsimpoukelli ct al., 2021), which integrate pre-trained language models for decoding. Flamingo (Alayrac et al., 2022) aligns vision encoder and language model, excelling in few-shot learning. BLIP-2 (Li et al., 2023c) combines a Flan-T5 (Chung et al., 2022) with Q-Former for efficient alignment. PaLM-E (Driess et al., 2023), with its 562 billion parameters, merges real-world sensor data into an LLM, linking perceptions and languages. GPT- 4 (OpenAI, 2023) further advances visual understanding and reasoning after extensive image-text data pre-training. Contemporary works such as LLaVa (Liu et al., 2023a). InstructBLIP (Dai et al., 2023), mPLUG-Owl (Ye et al., 2023), Multimodal-Gii GPP (Gong al., 2023), and Otter (Li et al., 2023b) align language models with visual encoders using multimodal instruction following datasets. Compared to these methods, MiniGPT-4 demonstrates both data efficiency and parameter efficiency, where only a single linear layer is learnable and the training time is just 10 hours with 4 A100 GPUs. In addition, LLaVa (Liu et al., 2023a), MIMIC-IT (Li et al., 2023a), and M3IT (Li et al., 2023c) collect visual instruction datasets by either generating from ChatGPT or from the human annotators. Such methods require access to image datasets with ground truth image information in text format. Compared to these methods, the visual instruction dataset used in MiniGPT-4 is generated by MiniGPT-4 itself, making data collection model-infommn
LLMs like ChatGPT can enhance vision- languaee tasks by collaborating with specialized models. Visual ChatGPT (Wu et al., 2023) and MM-REACT (Yang* et al., 2023) show ChatGPT integrating various visual models for complex challenges. ChatCaptioner (Zhu et al., 2023) uses ChatGPT to generate questions for BLIP-2, summarizing image content through dialogue. Video ChatCap- tioner (Chen et al., 2023) extends this video understanding. ViperGPT (Surís et al., 2023) combines an LLM with vision models for visual queries. MiniGPT-4 aligns visual information with the language model directly, avoiding external models.
3 METHOD
MiniGPT-4 aims to align visual information from a pretrained vision encoder with an advanced large language model (LLM). Specifically, we utilize the Vicuna (Chiang et al., 2023) as our language decoder, which is constructed upon LLaMA (Touvron et al., 2023) and can perform wide range of complex linguistic tasks. For visual perception, we employ the same visual encoder as used in BLIP-2 (Li et al., 2023c), ViT backbone (Fang et al., 2022) coupled with their pre-trained Q-Former. Both language and vision models are open-sourced. We target to bridge the gap between the visual encoder and I.I.M using a linear projection layer, with overview of our model displayed in Fig.1.
3

---

Published as a conference paper at ICLR 2024
We use two-stage training method. First, we pretrain it vast set of image- pairs to learn lasounlanguuae skills. Then, we finetune the model using a smaller, high-quality image-text dataset and conversational template, improving generation reliability and usability.
3.1 FIRST PRETRAINING STAGE
In the initial pretraining stage, our model uses large collection of aligned image-text pairs to gain vision-language knowledge. The output from the projection layer serves as soft prompt for the LLM, leading it to generate corresponding ground-truth texts. Throughout pretraining, the pretrained vision encoder and LLM remain frozen, with only the linear projection layer undergoing training. We utilize datasets from Conceptual Caption (Changpinyo et al., 2021; Sharma et al., 2018), SBU (Ordonez et al., 201 I), and LAION (Schuhmann et al., 2021) for this process. The model undergoes 20,000 training steps with a batch size of 256, covering about 5 million image-text pairs, and completes in around 10 hours on 4 A100 (80GB) GPUs.
Issues of the first pretraining stage After its initial pretraining, MiniGPT-4 shows the ability to hold a wealth of knowledge and respond reasonably to human queries. Yet, it sometimes generates incoherent outputs like repetitive words sentences, fragmented phrases, irrelevant content, which impairs its capacity for fluent visual conversation with humans.
GPT-3, despite its extensive language dataset pretraining, faced challenges in aligning outputs with user intentions. Instruction finetuning and reinforcement learning from human feedback transformed it into GPT-3.5 (Ouyang et al., 2022; OpenAI, 2022), enhancing its ability to produce human- human-friendly outputs. This mirrors MiniGPT-4's state after pretraining, explaining its current difficulties in generating fluent, natural human language outputs.
3.2 CURATING HIGH--QULIITY ALIGNMENT DATASET FOR VISION-LANGUAGE DOMAIN.
To achieve greater naturalness in the generated language and enhance the model's 's usability, second- stage alignment process is essential. While in the realm of NLP, instruction fine-tuning datasets (Taori et al. 2023) and conversations (sha, 2023) are easily accessible, no equivalent datasets exist for the vision-language domain the time of this project. To address this deficiency, we curated a detailed image description dataset, specifically tailored for vision-language alignment purposes. This dataset is subsequently utilized to fine-tune our MiniGPT-4 during the second-stage alignment process.
Initial aligned image-text generation In the initial phase, we employ the model derived from the first pretraining stage to generate comprehensive descriptions of input images. To enable our model to produce more detailed image descriptions, we designed a prompt that adheres to the conversational format of the Vicuna (Chiang al., 2023) language model, as shown below. this prompt, <ImageFeaturez represents the visual features produced by the linear projection layer.
</meatrrreeemmmeeeeeeeeeeeee this image in detail. Give as many details as possible. Say everything you see. ###Assistant:
###Human:
To identify incomplete sentences, we examine whether the generated sentence exceeds 80 tokens. it does not, we incorporate additional prompt, ###Iluman: Continue ###Assistan: prompting our MiniGPT-4 to extend the generation process. By concatenating the outputs from both steps, we create more comprehensive image description. This approach enables us to generate image-text pairs with detailed and informative image descriptions. We randomly select 5,000 images from the Conceptual Caption dataset (Changpinyo al., 2021; Sharma et al., 2018) 1) and use the pretrained model to generate corresponding language descriptions for each image.
Data post-processing The generated image descriptions marred by issues like repetitive words or sentences, fragmented sentences, and irrelevant content. To rectify these, we use ChaiGPT with a specific prompt improve the descriptions.
Fix the error in the given paragraph. Remove any repeating sentences, meaningless characters, not English sentences, and so Remove unnecessary repetition. Rewrite any incomplete sentences. Return directly the results without explanation. Return directly the input paragraph if it is already correct without explanation.
4

---

Published as a conference paper at ICLR 2024
Input Image
mage
Human
Help me write an advertisement this.
BUP-2
people walking riding motorcycles.
MiniGP74 The image shows busy city street with clock tower the boekgmundd The street lined with shops recsuments both sides, There
MiniGPT-4
Are you looking for a unique and styloh lighting solution your home or ortice// Look no further than our bmos toucan lampl This beautffully crattdd bird lamp sure add touch of whimsy and charm to any room. The toucan's boak designed hale light bulb, pronding a warm and Inviting The
of large clock with rumerals have spire top. building:
walling doon
dock
touch wood, addingg addition cvenall design The touran lamp whether lookingg brighton personality space. yours today bring tropics your home
omnate facades basconies. The strect pawod cobolestones and there are streediights on either side of road. The sky is clear and blue, and there are flw dloucs in the dieance.
Figure Detailed description
Figure Advertisement promotion
Upon completing the post-processing stage, we manually verify the correctness of each image description to guarantee its high quality. Specifically, we first identified several frequently shown errors ("I'm sorry made a mistake...", or "I apologize for that ..."" and then hard-coded rules to automatically filter them out. We also manually refine the generated captions by climinating redundant words or sentences that ChatGPT fails to detect. Finally, only approximately 3,500 out of 5,000 image-text pairs satisfy requirement, and these pairs subsequently utilized for the second-stage alignment process.
3.3 SECOND-STAGE STAGE FINETUNING
During the second stage, we finetune our pretrained model with the curated high-quality image-text pairs. During the finetuning, we use the predefined prompts in the following template:
###Hummn:
In this prompt, <Instruction>> represents a randomly sampled instruction from our predefined instruction set containing variant forms of instructions such as "Describe this image in detail" or "Could you describe the contents of this image for me". It is important to note that we do not calculate the regression loss for this specitic text-image prompt.
As a result, MiniGPT-4 is now capable of producing more natural and reliable language outputs. Furthermore, we observed that this tnnettnnig process is remarkably efficient, only requiring mere 400 training steps with a batch size of 12, which takes around 7 minutes with a single A100 GPU.
4 EXPERIMENTS
In the experiment, we aim to showcase the diverse and emergent capabilities of our MiniGPT-4 model through various qualitative examples. These abilities include generating detailed image deseriptions, identifying amusing aspects within memes, providing food recipes from photos, writing poems for images, etc. Additionally, we present quantitative results on the task image captioning.
4.1 UNCOVERING EMERGENT ABILITIES WITH MINIGPT-4 THROUGH QUALITATIVE EXAMPLES
MiniGPT-4 demonstrates many advanced abilities compared to traditional Dision-langual models. For example, it can describe images in detail and interpret the humorous aspects of given meme. Here, we qualitatively compared our model to one of the leading vision- vision-language models, BLIP-2 (Li al., 2023c), with eight distinct examples, each highlighting different ability.
5

---

Published as a conference paper at ICLR 2024
Fig.2 shows MiniGPT-4's ability to identify multiple elements in an image, like busy streets, clock towers, shops, streetlights, and restaurants, whereas BLIP-2 only notes streets, people, and motor- eycles. In another instance, Fig.4a, MiniGPT-4 aptly explains the humor in meme by relating the dog's expression to common Monday blues, a concept BIJI-2 misses, merely describing the image without grasping its humorous aspect.
MiniGPT-4 has many other capabilities, including creating ads from images (Fig.3), extracting facts from movie photos (Fig.8), generating recipes from food images (Fig.11), diagnosing and suggesting treatments for plant diseases (Fig. 12), designing websites from hand-written drafts (Fig.4b), and writing poems inspired by images (Fig.10). These abilities surpass those of traditional models like BLIP-2, which uses Flan-T5 XXL (Chung et al., 2022) as a language model. This difference highlights the importance of aligning visual features with an advanced L.L.M like Vieuna (Chiang et al., 2023) to unlock advanced vision-language capabilities.
"oke WebRae
Gortem of 1.
MONDAY.
Puncliie 1)
Human
We be HTMM /5 turn this moolhng letto colerful replaced
WIS__MONDAYY
BLIP2
Welcome to the Jake Website
Explain why floor with caption
Monday.
BLIP-2 FT
dog is dog.
MiviPPT O-former
Walcome the Joire boc
MiniGPT-4
This funny because shows dog steeping floor:
LocNa
This mome funny because shows a small dog laying the floor with Its logs stretched out, seemingly onjoving day. caption, suagesting
MiniEPT-4
No O.Former
just monday," dog relaxed
doesn't even the energy to get up. The image also humorous because depicts common scenario that many people can re ate to the feeling being tired that day. Overall,
Joke Website
Hame About Contect
lightheorted
depiction the joys of relaxatiio.
Today's Joke
MisGGPT4
This funny because with head resting on its pows, as it tired or sleepy. The caption, "monday just monday," adds to the humor that the
dog laying on
diessingt
people Mondays, most dreaded day the week, The meme relotable and humorous, making it populer among internet users.
considered
2009 Juko webclte
(a) Meme explaining
(b) Website Creating
Figure 4: Model generations from BLIP-2, BLIP-2 finetuned our second stage data (BLIP-2 FT), MiniGPT-4 finetuned with Local Narrative data in the second stage (MiniGPT-4 LocNa), MiniGPT-4 model without Q- Q-Former (MiniGPT-4 No Q-Former), and MiniGPT-4.
Table 1:
Quantitative results on advanced vision-language tasks. MiniGPT-4 shows strong perfor- mance and successfully responses to 65% of the requests.
Meme
Recipes
Ads
Poem
Avg.
BLIP-2
0/25
4/25
1/25
0/25
5/100
MiniGPT-4
8/25
18/25
19/25
20/25
65/100

---

Published as a conference paper at ICLR 2024
4.2 QUANTITATIVE ANALYSIS
Advanced Abilities Our evaluation dataset for vision-language tasks included 100 images divided across four tasks: meme interpretation, recipe generation, advertisement creation, and poem com- position, each with 25 images. Human evaluators assessed the model's responses. We compared MiniGPT-4 with BLIP-2, detailed in Tab.1. MiniGPT-- outperformed BLIP-2 (Li et al., 2023c), especially in recipe, advertisement, and poem tasks, successfully handling 80% of these. It also interpreted humor memes correctly in 8 out of 25 cases, a challenging aspect for BLIP-2.
Image Captioning We evaluate the performance of MiniGPT-4 on the COCO caption benchmark and compare with BLIP-2 (Li et al., 2023c). Our model's generated captions typically contain rich visual details. As such, conventional similarity-basee image-caption evaluation metrics struggle to provide an accurate evaluation. To evaluate, we check how many of COCO's ground truth captions per image are covered by MiniGPT-4's captions, using GPT- turbo. Evaluation details can be found in A.3..A.3.. Results in Tab.2 show MiniGPT-4 averaged 2.22 ground truth captions, better than BLIP-2's 1.96, proving its captions to be more informative. Additional evaluations on traditional VQA tasks are detailed in Appx.A.2.
Video Understanding Here, we evaluate MiniGPT-4 for video understanding. We finetuned MiniGPT- 4 on 1.2k videos from the V00e (Maaz et al., 2023), using 50 frames and subtitles per video. Experimental results on the rideo-based generative performance benchmark (Maaz al., 2023) in Tab. 4 show that MiniGPT-4 outperformed the strongest baseline Video-ChatGPT (Maaz et al., 2023) in correctness, detail, context, and time comprehension, while also showing strong consistency, demonstrating MiniGPT-4's potential in processing videos.
Other Benchmarks MinGPT-4 has been densely evaluated and compared with contemporary base- lines like LLaVa (Liu et al., 2023a) and MaPlu--Owl (Ye et al., 2023) by many popular benchmarks like MMBench (Liu et al., 2023b) quantitatively. detailed discussion of MiniGPT-4's performance on these benchmarks can be found in Appx.A.5.
4.3 ANALYSIS ON THE SECOND-STAGE FINETUNING
Effectiveness of the second-stage finetuning Utilizing MiniGPT-4 solely after the first pretraining stage leads to issues like repetitive fragmented sentences. These largely resolved after the second-stage finetuning, as shown in Fig.5, where MiniGPT-4 evolves from generating incomplete to fluent captions. This section assesses the second-stage finetuning's importance and effectiveness.
To measure its impact, we sampled 100 images from the COCO test set for the detailed description and poem writing tasks, using the prompts "Describe the image in detail." " and "Can you write beautiful poem about this image?". Both pre- and post-second-staaee finetuned models attempted these tasks. Results in Tab.3 show a significant drop failures post- post-finetuning, with less than two failures
Table 2: COCO caption evaluation. We use GPT-4 turbo to count the number of ground truth captions the model output can cover. MITGPPT-4GPPT-x)) denotes variant trained using GPT-4V generated data in the second stage.
Table Failure rates of detailed caption and poem generation tasks before and after second-stage finetuning. The finetuning stage significantly reduces generation failures.
Failure rate
Detailed cuption
Poem
BLIP-2
MiniGPT4
MiniGPT-4 (GPT-v)
Before stage-2
35%
32%
#GT Cover
1.96
2.22
2.26
After stage-2
2
1%
Correctness
Detail
Contextual
Temporal
Consistency
Video Chat (Li et al., 2023d)
2.23
2.50
2.53
1.94
2.24
Llama Adapter (Zhang et al., 2023b)
2.03
2.32
2.30
1.98
2.15
Video LLama (Zhang al., 2023a)
1.96
2.18
2.16
1.82
1.79
Mideo-ChatGPT (Maaz al,, 2023)
2.40
2.52
2.62
1.98
2.37
MiniGPT-4
2.68
2.76
3,20
2.26
2.18
Table 4: Video understanding the video-based generative performance benchmark.
7

---

Published as a conference paper at ICLR 2024
Image
Imput Image
Human
Descrise this image detailed as possille.
Human
Deser be this image in detail.
MiniGPT-4
image desicts a wooden bench in the snow on sunny day, The bench
MiniGPT4
dining, set diners. The tables are covered in white tablec othe and have silvarwore, glasses, and bottles liquor on them. walls are painted light blue and have lange windows that look out citycape. There is chandeller hanging from the collngg
The image
sevural
before Stage-2 2
After Stage-2
This iage close-up up shot of wooden bench ditting
snswe-covered hill The bunch is made wood
with ground plled around bench, giiigg the impression winter scars, background, there are trees and mountains vattle, with slow the sky indicating sunset. The image besutifll with the snow adding serene otmosphere to the
failly
windowsil.
domage. snow
Human
Are the windows on the left side the photo?
MiniGPT-4
Yes, the windows are the left side of the photo.
Figure 5: MiniGPT-4 before second-stage fine- Figure 6: An example of MiniGPT-4's limitations. tuning fails to output completed texts. The gener- MiniGPT-4 hallucinates unexisting tablecloths ation is improved after the finetuning.
and can't locate the windows correctly.
in 100 images for each task, indicating a notable improvement in output quality. Fig.5 provides qualitative examples of this enhancement.
Can the original BLIP-2 benefit from the second-stage data? In this study, we finetune 3LIP-2 et al., 2023c) with our second-stage data the same way as MiniGPT-4, and check it can obtain similar advanced abilities MiniGPT-4, The finetuned BLIP-2 2 is denoted as BLIP-2 FT. Note that MiniGPT-4 uses the same visual module as BI.I-2;; while BLIP--2 uses FlanT55 XXL. (Chung et al., 2022) as the language model, which is not strong the Vicuna (Chiang et al., 2023) model used in our MiniGPT-4 model. We rely on the same prompts to assess the advanced capabilities of our model. Qualitative results shown in Fig.4, 13, and 14. We discover that BLIP-2 FT still generates short responses and fails to generalize to advanced tasks like meme explaining and website coding (Fig.4). Our finding suggests that BLIP-2's relatively weaker language model FlanT5 XXL benefits less from such a small dataset, and highlights the effectiveness of a more advanced LLM in a VLM system.
Second stage with Localized Narratives We tested MiniGPT-4's performance by substituting our self-collected dataset with the Localized Narratives dataset (Pont-Tuset et al., 2020) in the second training stage. We name this variant MiniGPT-4 LocNa. The Localized Narratives dataset features detailed image descriptions with corresponding regional localizations. Qualitative results shown in Fig.4, 13, and 14 reveal that MiniGPT-4 LocNa can produce lengthy image descriptions (as seen in 14). However, these outputs of lower quality, often with monotonous expressions. MiniGPT4 LocNa also shows weaker generalization in complex tasks, like explaining meme humor (Fig.4a), compared to the original MiniGPT-4. This performance difference may stem from the repetitive and monotonous nature of the Localized Narratives dataset.
Second stage with GPT-4V generated data. We conduct further ablation experiments using 2,000 GPT-4V generated image-text pairs collected by LAION (LAION, 2023) in the second stage. Results Tab.2 shows performance improvements from this fine-tuning.
Table 6: Hallucination Evaluation
Table 5: Ablation on architecture designs
CHAIR.
Avg. Length
Model
AOK-VQA
GQA
Blip-2
1.3
6.5
MiniGPT-4
58.2
32.2
mPLUG-OW!
30,2
98.5
(a) MiniGPT-4 w/o Q-Former
56.9
33.4
LLaVa
90.7
(b) MiniGPT4 Layers
49.7
31.0
(c) MiniGPT-4 Finetune Q-Former
52.1
28.0
MiniGPT-4 (short)
7.2
28.8
MiniGPT-4 (long)
9.6
175
8

---

Published as a conference paper at ICLR 2024
Amount of traing data in the first stage This ablation study can be found in Appx.A.4.
4.4 ABLATION ON THE ARCHITECTURE DESIGNS
To further demonstrate the effectiveness of using one single linear layer to align visual features with LLM, we conduct experiments with different architecture designs, including (a) removing the Q- Former and directly mapping the VIT's output to Vicuna's 's embedding space (i.e., without Q-former), (b) using three linear layers instead of one layer, and (c) additionally finetuning the Q-Former in the vision module. All the variants are trained in the same way as the original design. Results on AOK-VQA (Schwenk et al., 2022) and GQA (Hudson & Manning, 2019) datasets in Tab.5 show that the variant (a) MiniGPT-4 w/o Q-Former has a similar performance to the original design. Qualitative results of this variant in Fig.4, 13, and 14 also show similar advanced skills. This reveals that the Q-Former from BLIP-2 doesn't I plays a critical roles for advanced skills. Besides, both variants (b) MiniGPT-4+ Layers and (c) MiniGPT-4 + finetuning Q-Former, perform slightly worse than the original MiniGPT-44 This indicates a single projection layer is sufficient to align the vision encoder and the large language model in our limited training data setting.
4.5 LIMITATION ANALYSIS
Hallucination As MiniGPT-4 is built upon LLMs, it inherits LLM's limitations like hallucinating nonexistent knowledge. An example in Fig. shows that MiniGPT-4 incorrectly identifies the presence of white tablecloths in the image, despite their absence. Here, we use the metric CHAIR; (Rohrbach et al., 2018) to gauge the hallucination rate of the generation, with the two distinct prompts to control the model generation length: MiniGPT-4 (long): Please describe this image detailed as possible. MiniGPT-4 (short): Please describe the image shortly and precisely, in less than 20 words.
Results in Tab.6 show that longer captions tend to have higher hallucination rates. For example, MiniGPT-4 (long) generates captions averaging 175 words with a higher hallucination rate, while MiniGPT-4 (short) averages 28.8 words with a lower rate. BLIP-2, averaging 6.5 words, hallucinates less but covers fewer objects as seen in Tab.2. Compared to contemporary methods like LLaVa mPlug-Owl, MiniGPT--4 generates longer descriptions with fewer hallucination. Hallucination in detailed image descriptions still an unresolved issue. Using Reinforcement Learning with AI feadback with hallucination detection modules may be a potential solution.
Spatial Information Understanding MiniGPT-4's visual perception remains limited. It may struggle to differentiate spatial localization. For example, MiniGPT-4 in Fig. 6 fails to identify the location of the windows. This limitation may stem from lack of aligned image-text data designed for spatial information understanding. Training on such datasets like RefCOCO (Kazemzadeh et al., 2014) or Visual Genome (Krishna et al., 2017) could potentially alleviate this issue.
5 DISCUSSION
How does MiniGPT-4 obtain these advanced abilities? Many of the advanced vision-language capa- bilities demonstrated by GPT-4 can be understood as compositional skills rooted in two foundational skills: image understanding and language generation. Take the task of image-based poem writing as example. Advanced LLMs like ChatGPT and Vicuna can already craft poems based on users' instructions. If they acquire the ability to understand images, compositionally generalizing to the task of image-based poem writing even without having image-poem pairs in their training data is possible.
In its first pretraining stage, MiniGPT-4 learns image understanding by correlating images with short descriptions from caption datasets. However, the language style in these datasets differs from that of modern LLMs, leading to distorted language generation and impeding compositional generalization. To address this, second-stage stage finetuning is introduced to improve language generation. Post two-stage training, MiniGPT-4 successfully demonstrates advanced compositional vision- langunel abilities. such draft-to webbstt interpreting memes, confirming our approach. Future research could explore the mechanisms of compositional generalization further. Our work, as a preliminary exploration of vision-based L.LM capabilities, aims encourage more studies in this area.

---

Published as a conference paper at ICLR 2024
REFERENCES
Sharegpt. http://www.....oooooococc//a/aaa 2023
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr. Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive Vision-language models. arXiv preprint arXiv:2308.01330 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 202...........
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 1 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny, Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18030-18040, 2022.
Jun Chen, Deyao Zhu, Kilichbek Haydarov, Xiang Li, and Mohamed Elhoseiny, Video chatcaptioner: Towards the enriched spatiotemporal descriptions. arXiv preprint wwXiv:2304.0222) 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: //vicuna. Imsys. org.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023: 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finttuee language models. arXiv preprint arXiv:2210.1144 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purga purpose vision-language models with instruction tuning, 2023.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 088000 2018.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 2023.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeling of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 920-335,70 2022.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. arXiv preprint arXiv:2211.076. 2022.
Tao Gong, Chenggi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao. Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpi A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.0470 2023.
10

---

Published as a conference paper at ICLR 2024
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv: 15556, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprini arXiv:2106.099: 09685, 2021.
Shaohan Huang. Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
LAION. Laion gpt4v dataset. apt4v-datase,, 2023.
http:://bcggiggfacccc//aaaaa
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023a.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888 -29800 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12555 2023c.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023d.
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M it: A large-scale dataset towards multi-modal multilingual instruction tuning. arXiv preprint 2XXi:23366.4387 2023c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv
preprint arXiv:2304. 08485, 2023a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.062. 2023b.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424 2023.
OpenAI. Introducing chatgpt.
http:://openiiccomm/ooolllgggggp 2022.
OpenAl. Gpt-4 technical report, 2023.

---

Published as a conference paper at ICLR 2024
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt V 16, pp. 647-664. Springer, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1(5):5485-5551, 2020.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint arXiv: TrXiv:/809.02/56, 2018.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvn,, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 2022a.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221..05100, 2022b.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: benchmark for visual question answering using world knowledge. In European Conference Computer Vision, pp. 146-162. Springer, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, 2018.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron train megatron- turing nlg 530b, a large-scale generative language model. arXiv preprint 1XXx:2201.1/9900 2022.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:23033088 /28, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follott followwig Ilama model. https: // /i thub. com tat lab/ stanford_as pacc,, 2023.
Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. Plug-and- play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. arXiv preprint arXiv:2210.087733 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ana Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
12

---

Published as a conference paper at ICLR 2024
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 14:200-212, 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large lan- guage models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL Survey Certification.
https:/ https////ppeeeeeeeeeeee net/ Forum?id=yzksuui
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. arXiv preprint arXiv:2206.08155, 2022.
Zhengyuan Yang", Linjie Li*, Jianfeng Wang*, Kevin Lin", Ehsan Azarnasab Faisal Ahmed*, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multi- modal reasoning and action. 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprini arXiv:2304.14178 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306. 02858, 2023a.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning language models with zero-init attention. arXiv preprint arXiv:230..66199 2023b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint rxXi:2205.01066 2022.
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elho- seiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.0663 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
MORE QUALITATIVE RESULTS
MONDAY.
Input
JUST...MONDAY
Human
Can you give mo short Introduction to this movie?
Human
Explain why this mome funny.
BLIP-2
The godfather is 1969 american gangster fim directed by
BUPP2
dag loying down on the floor with the caption monday Monday.
MiniGPT-4
directed by Francis Ford Coppola and released 1972. based on the novel of the same name by Mario Puzo and follows the story City. The stars Marlon Brando as the patriarch of the family, Vito Corfeone, and Al Padino his son, Michael performances, memorable dialogue, and its portrayal of the American mafia liestylee it considered one of the greatest films of all time and had lasting
Godfither
funny because chows dog laying
with its head resting on its paws, as it is tired or sleepy. The humor way many people do on Mondays, which are often considered to the most dreaded day of the week. The humorous, making popular among users.
powerful
"monday just monday." adds suggesting
dog foeling
relatable and
popular culture.
Figure 7: Advertisement promotion
Figure 8: Factual retrieval
A Boke Websitte
Input Image
Input Image
Content of dolie 1
Punchling m 2)
Human
brief HTML/IS this colorful website, where the joles replaced joke.
you beautiful poom image?
page punchiine
BLIP-2
man and his dog sit on a hill overlooking the city.
MiniGPT-4
HTMI:
pho
MiniGPT-4
Sure, here
about the
provided:
chtml>
man sits on hill, with his dog by his side
<head>
The behind them, casting golden light city below g, they Enjoying the moment, each other's ambrace
Websites/ttle>>
<link rel= "stylesheet' type=" "text/es" tref="style.css"
</heads
wrapped
friend
As they watch the world go by, with a gentle bend The dog's head rests on the man's shoulder their unbreakable
fonover
A Joke Website About
The sky is ablsze with colos, beautiful sight moment pure joy, memory to keep Forever etched in their hearts, as they sit and weep
Home
Contact
bright
Today's 's Joke
Why did the tomaso turn red?
us cherish moments, our four-legged friends For they bring much happiness, until the very end And when gone, will live memorias
Because saw the saled dressing!
strong.
2023 A Joke Website
Figure 9; Website creating
Figure 10:
Poem writing
14

---

Published as a conference paper at ICLR 2024
Inpurt Image
Input Image
Human
How should make something like this?
BLIP-2
Prehest the 400 degrees f and line baking sheet with parchment paper
Human
What is the soue with my plant? What should do now?
MiniGPT-4
To make dish, you will need the following ingredients:
BLIP-2
A coffee leaf with brown spots
4 lobsters
MiniGPT4
The image shows a leaf with brown spots on it. The spots are likely caused by fungal infection, which can treated with fungleide: important Identify the type fungus causing the spots in order to choose the appropriate treatment.
2 tablespoons alive ail
2 cloves garlic, minced
1 dried thyme
teaspoon dried oregano
1/2 teaspoon paprika
you
Salt pepper,
Identily the type fungus causing the spots on the leaves.
wedges, serving
2. Use fangizide specifically labeled for the type of furgus you have dentified.
1. Prehest the to 400'F (200'C).
product's instructions, making sure to cover all affected leaves.
Apply fungicide according
2. In baking dish,
lobsters
garlic, thyme, oregano, paprika, salt, and pesger.
4, Repeat the treatment every 7 14 days, directed by the product's instr actronttons, until fungus compl etaly Water the plant well treatment to help the fungicide penetrate the leaves.
3. Roest the lobsters the prehested oven for 20-25 minutes,
or until they cooked through.
4. Remove the lobsters from the oven and let them cool for
few minutes.
5. Serve lobsters the lemon wedges side.
also Important to keep plant healthy provicing
with proper care, such adequate watering, fertiliaatio, and pruning This will help the plan to recover and prevent future fungal infections.
hope this helpsi Let me know have other quessions.
Figure 11: Food recipe generation
Figure 12: Plant cultivating
A.2 EVALUATION IN TRADITIONAL VQA BENCHMARKS
The aim of this study is to replicate the remarkable multí-modal capabilities demonstrated in GPT-4, such as generating detailed image descriptions and creating websites from hand-drawn drafts. To emphasize the most crucial component of advanced vision-language skills, the methodology of MiniGPT-4 is intentionally kept minimal. For instance, the learnable model capacity is limited (only one linear layer), and MiniGPT-4 is trained with just 5 million pairs, in contrast to BLIP-2 with 129 million image-text pairs. Such pared-down approach is anticipated to yield suboptimal results on traditional benchmarks. While this isn't I our primary goal, we offer a quantitative analysis of the VQA datasets A-OKVQA (multi-choicee (Schwenk et al., 2022) and GQA (Hudson & Manning, 2019). Additionally, to showease the potential of MiniGPT-4 with traditional benchmarks, straightforward ablation study. Here, we simply unfreeze the LLM using LoRA (Hu et al., 2021) and incorporate more training data from the VQAv2, OK VKQ and A-OK datasets during the second finetuning stage. Results in Tab. 7 indicate that the original MiniGPT-4 lags behind BL.I--2 by a reasonable margin, and merely augmenting the learning capacity and the training data results in a substantial performance improvement, which confirms our expectations. We believe our model's performance on conventional vision benchmarks can be enhanced with carefully designed training strategy (e.g., dataset sample ratios, learning rate schedule, etc.), more training data/datasets, and additional learnable parameters. Since enhancing performance on traditional vision benchmarks isn't t this project's objective, we reserve this aspect for future research.
conduct
Model
Training data
BOK-VQA
GQA
Blip-2
129M image-text text pairs
80.2
42.4
MinGGPT4
SM image- text pairs
58.2
32.2
MiniGPT (Pinetune Vicuna)
(Finntuue
5M image-text pairs
67.2
43.5
Table 7: Performance Comparison between BLIP-2 and MiniGPT-4
15

---

Published as a conference paper at ICLR 2024
A.3 DETAILS OF CAPTION EVALUATION
We utilize GPT-4 turbo (gpt-4- 106-preview) to assess whether the generated descriptions capture the content of each ground truth caption individually. In the COCO dataset, each image accompanied by ground truth captions. For every image, we calculate the number of captions covered by the generated descriptions and then average this count across 5000 random sampled images from the verify whether the information gr caption included in the test description.
validation set to derive the final score.
Here the prompt we use in GPT-4 turbo
Given test image description and a list of gt image caption,
The answer should be yes no.
Input is in this format:
Test: (test sentence)
1: (gtl)
2: (gt2)
3: (gt3)
you need to answer yes for each gt in the following format:
1: (yes/no)
2: (yes/no)
3: (yes/no)
A.4 AMOUNT OF TRAINING DATA IN THE FIRST STAGE.
We evaluate the impact of training data volume in the first stage by using checkpoints 10%. 30%, and 50% of stage duration, subsequently finetuned in stage 2. Results in Tab. show significant performance drop with only 10% of stage I data. However, utilizing 30% of stage 1 data, equivalent to 1 1.5M image-text pairs can achieve similar performance with the original MiniGPT-4. No gains were seen beyond 50% of stage data, indicating potential saturation of the model's learnable capacity this juncture.
Metric
10%
30%
100%
#GT Cover
1.62
2.15
2.26
2.22
Table 8: Captioning performance with different amount of training data in stage-1,
A.5 MINIGPT-4 ON MMBENCH
MMBench (Liu et al., 2023b) is new multi-modality benchmark with diverse evaluation ques- tions to evaluate different abilities of vision language model. MMBench evaluated MiniGPT-4 together with other contemporary vision language models like OpenFlamingo (Awadalla et al., 2023), VisualGLM (Du et al., 2022), LLaVa (Liu et al., 2023a), and InstructBlip Dai et al. (2023). Here, show the performance of MiniGPT-4 and other baseline models in Tab. 9, Results show that MiniGPT-4 demonstrates competitive performance compared to contemporary methods, e.g.. InstructBlip. It surpasses InstructBlip in several key areas: logical reasoning (LR), fine-grained per- ception for single instance (FP-S), and fine- me-grrined perception across instances (FP-C). Additionally, MiniGPT-4 achieves competitive results in relation reasoning (RR), attribute reasoning (AR), and coarse perception (CP).
16

---

Published as a conference paper at ICLR 2024
Model
Overall
LR
AR
RR
FP-S
FP-C
CP
OpenFlamingo
4.6
6.7
8.0
0.0
6.7
2.8
2.0
VisualGLM
38.1
10.8
44.3
35.7
43.8
23.4
47.3
LLaVa
38.7
16.7
48.3
30.4
45.5
32.4
40.6
InstructBlip
44.0
19.1
54.2
47.8
24.8
56.4
MiniGPT-4
42.3
50.7
26.2
50.7
Table 9: Perforance on MMBench benchmark. Numbers are from Liu et al. (2023b).
A.6 MORE QUALITATIVE ABLATION RESULTS
RU1-2
Nechol degress lovo bakking
pard
garlic
Locko
Tore the dist will nead the owiing mot
To tai dult naad tal rigeedeerta
mopi dhan
coou girlic ininced
nestburtnnss
Prekust your car to asd's pastii
On the le oa oh rmuge. cun as chock lowes
bebsters under colil waterr
poper aewets
rignt
image,, can mony micto roches parked road.
highgh
tcsaters murrnn with sas and peppon and conkrer memutis per tee and
On the of the
shops nestsurents
Trander the libotiiss tur lok rigiic and Buke for 10-12 mirate, or will the coutedt tough the shate
On the right adde the imaga, we can see many mictorycdes cles parked on the roed.
middle image, many people on roud.
lood,
innige
Serve the totaao in the perio--eman butee on top and femoo wedge
hose tha helpil
you how casations
MixiGPT44 No Q-Fomee
image shoow streat nights. There are more people walking
The
the Image The is clar and there are sturs walcis thealyy The mage sakan from High angis. looking down stredt
and vin There clock
Lurga lobstors
oive gorkic entroed
street with shope sertaurnnt
didee
tevepcen omageno
motorcycle partad the idde of the sand, and peop wollingg down the streel dack tower has lange dlack with Htor nu nuseaull and smnul
tuaepic
"44temmn redggn. serving
etter did the roso. The cear slue and thene a fow coudd the distace.
Prethest oweov 4007 pD0C.
ar birgr diri
lrattes with gadli,
Bout the locater in prchessed owen min saas on they
hemove Neve te-ticeee da mecioon widgen di-idde
fowr
Figure Ablation Study on Detailed Description
thozu tha habo mu know new my other cuartonn.
Figure 13: Ablation Study Recipe Genera- tion
17


Reviews:
[{"review_id": "ygupOwUBFq", "reviewer": "Reviewer_7oDs", "summary": "This work introduces MiniGPT-4, which combines a frozen visual encoder with a frozen advanced LLM, Vicuna, through a single projection layer. Remarkably, MiniGPT-4 exhibits advanced multi-modal abilities similar to GPT-4, extending to generating detailed image descriptions, creating websites from hand-drawn drafts, writing stories and poems inspired by images, and providing cooking instructions based on food photos. However, initial training on short image captions led to issues like repetition and fragmentation in the generated text. To remedy this, the authors curated a detailed image description dataset for a second stage of fine-tuning, significantly enhancing the model’s reliability and overall performance. They have made their code available for further research and validation.", "questions": "(1) What is the maximum input and output length for the training and inference stage? \n\n(2) In Section 3.2, the authors mention they would examine if the generated sentence exceeds 80 tokens or not – how is the length of 80 determined? Is it through empirical observations?\n\n(3) What’s the length of the learned visual queries from the Q-Former in Mini-GPT4?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This submission is one of the earliest attempts at open-source reproduction of large vision-and-language models. It provides a parameter-efficient solution to this difficult problem, and makes use of the existing open-source models as much as possible, which inspires many follow-up works.", "weaknesses": "There are two main weaknesses with regard to the model structure and the evaluation setup. My evaluation of this submission will be greatly improved if the authors properly address these two main issues. \n\n\n**(1) Model Structure:**\nThe introduction of MiniGPT-4 in the submission presents a novel approach in the domain of vision-and-language models. However, a noticeable limitation lies in its capacity to process only a single visual input per run. This design contrasts with other large-scale models such as Flamingo and GPT-4, which have demonstrated capabilities in handling multiple visual inputs simultaneously, as acknowledged in the related work section of the submission.\n\nAdditionally, the literature review appears to omit significant works [1,2,3,4] in the realm of large vision-and-language models that facilitate in-context learning with multiple visual inputs. The inclusion of these works could provide a more comprehensive backdrop for the MiniGPT-4, situating it within the broader context of ongoing research in the field.\n\nThe MiniGPT-4’s visual encoder is founded on the Q-Former architecture, converting each visual input into a fixed-length learned visual query. With reference to the BLIP-2 paper, this length is specified as 32 dimensions. Considering that this is considerably shorter than the maximum input length supported by the LLM backbone, there seems to be an opportunity to extend the model’s capacity to accommodate multiple visual inputs. An exploration into why the MiniGPT-4 is constrained to single image inputs, alongside potential avenues for extending its flexibility and application range, would strengthen the submission.\n\n**(2) Evaluation Setup:**\nThe submission primarily focuses on showcasing qualitative results derived from the MiniGPT-4, complemented by various ablation studies. However, there is an apparent gap in benchmark comparisons with other established large vision-and-language models. A number of benchmarks [5,6,7], encompassing diverse aspects and task setups related to vision-and-language evaluation, have included results from MiniGPT-4 alongside comparisons with other models. The incorporation of such direct comparisons would offer a more definitive assessment of MiniGPT-4’s performance, enhancing the overall credibility of the evaluation presented.\n\n\n\n**Missing references**:\n\n[1] OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models\n\n[2] Otter: A Multi-Modal Model with In-Context Instruction Tuning\n\n[3] mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality\n\n[4] MIMIC-IT: Multi-Modal In-Context Instruction Tuning\n\n[5] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models\n\n[6] MMBench: Is Your Multi-modal Model an All-around Player?\n\n[7] VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BnhmXKLIro", "reviewer": "Reviewer_TFFX", "summary": "This paper attempts to create Multimodal LLMs that can understand the visual inputs. To do this, MiniGPT-4 uses a frozen visual encoder along with a frozen vicuna model and only trains the single projection layer. Various emerging properties of the M-LLMs are discussed and a novel detailed image description-based fine-tuning stage is proposed to further improve the image understanding.\n\nOverall this paper has contributed significantly. However, it’s not yet ready for publication. Other rounds of revisions are needed with extensive benchmarking.", "questions": "* Instead of having the data post-processing steps for stage 2 training, why not use GPT-4 itself to get the image captions and perform knowledge distillation as stage 2? Getting ~3000 such image-caption pairs should not be that costly.\n* Does MiniGPT-4 apply a unique projection layer to all output vectors corresponding to the learned queries? Or some other processing steps are involved?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* One of the first works attempts to introduce visual modalities in pre-trained LLMs in a parameter-efficient manner.\n* MiniGPT-4 requires 40 A100 GPU hours for training and is able to outperform the BLIP-2. \n* Various qualitative examples are shown along with ablation studies to measure the effectiveness of the two-stage training procedure. \n* Limitations of the MiniGPT-4 (in terms of hallucinations and spatial relations) are discussed.", "weaknesses": "* Benchmark is very limited. Rigorous benchmarking on different downstream tasks is needed. For reference, MMBench, TextVQA, etc.\n* Only one baseline is reported. However, for a holistic understanding of the approach, more baselines are needed. \n\nSome of the missing References:\n* Liu, Haotian, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. \"Visual instruction tuning.\" arXiv preprint arXiv:2304.08485 (2023).\n* Mou, Chong, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. \"T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.\" arXiv preprint arXiv:2302.08453 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ygbOM4ceck", "reviewer": "Reviewer_A8a5", "summary": "The paper introduces MiniGPT-4, which combines a frozen visual encoder with a frozen large language model, Vicuna, using a single projection layer. The research revealed that aligning visual features with a sophisticated large language model can replicate many of GPT-4's advanced multi-modal functions. A two-step training process is proposed, and the second step plays a vital role in generating long and meaningful responses. Additionally, MiniGPT-4 showcased unique capabilities like writing stories or poems inspired by images and teaching users cooking techniques based on food photos.", "questions": "* Is it possible to use less data from the combined dataset to train the projection layer in the first pertaining stage? It is better to provide insights on the size data necessary to align the vision and text space.\n\n* Surprisingly, the performance drops a lot when three projection layers are used instead of one. Could you try different designs of the projection module, e.g., two or more than three projection layers, or adding activation layers between projection layers to provide insights on why one projection layer is better and why the drop is so huge?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Easy Approach: The paper presents an easy two-step approach to combining a visual encoder with a large language model, providing insights into GPT-4's enhanced multi-modal capabilities.\n\nDiverse Capabilities: MiniGPT-4 replicates many of GPT-4's features, such as writing stories based on images and culinary guidance from food photos.\n\nInteresting Experiments: The authors provide qualitative and quantitative experiments to show the various abilities of the proposed method, including the human study of advanced abilities.", "weaknesses": "* The necessity of using a large number of image and text pairs to train a projection layer is not addressed sufficiently. \n\n* The comparisons between BLIP-2 and MiniGPT are unfair because BLIP-2 uses a weaker LLM and is not designed for the advanced abilities mentioned in the paper. Furthermore, the designed new evaluation method using ChatGPT to check whether the generation covers all the objects and visual relations is not compelling. It is possible because many hallucinations are generated, as Table 5 illustrates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2mOyEQoWo", "reviewer": "Reviewer_auSX", "summary": "This paper presents a recipe for reproducing the GPT-4 vision model. The approach involves aligning the pre-trained vision encoder of BLIP-2 (ViT + Qformer) with the LLM Vicuna using a noisy image-caption dataset. The model is then fine-tuned with a curated detailed image description dataset, which is generated by refining the initial model predictions using ChatGPT and manual refinement. Experimental results demonstrate the capabilities of the proposed MiniGPT-4 model in challenging vision-language tasks like Poem Generation and Meme understanding. While qualitative results are mostly presented, quantitative findings indicate that MiniGPT-4 outperforms BLIP-2 on image captioning when evaluated with ChatGPT, although it tends to generate more hallucinations than BLIP-2.", "questions": "Q1: What criteria did you use to select the Advanced Abilities tasks? Additionally, can you clarify if the images used for evaluation in these tasks overlap with the dataset used in stage 1 pre-training and stage 2 fine-tuning?\n\nQ2: Could you specify the version of ChatGPT that you adopted for answer revision in your experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The exploration to achieve a GPT-4 level VLMs by combining LLMs & vision encoders is greatly appreciated. Additionally, the provision of open-sourced model checkpoints and dataset enhances the reproducibility and transparency of the research.\n- The qualitative results presented in the study demonstrate the potential effectiveness of the proposed recipe. The process of aligning the model through training on a noisy dataset and subsequently fine-tuning it with a high-quality dataset shows promise in improving the model's performance and response quality.", "weaknesses": "- The quantitative evaluation setup lacks convincing elements:\n  - The paper only compares the proposed model with BLIP-2 as the baseline, which utilizes a FLAN-T5 model as the base LLM. To enhance the credibility of the results, it would be beneficial to compare the proposed model with more recent open-sourced components that utilize similar base LLMs, such as LLaVA[1], InstructBLIP[2], and mPlug-Owl[3].\n  - The selection of the four Advanced Abilities tasks appears arbitrary, and the evaluation sets for these tasks are relatively small (25 samples for each task). To provide a more comprehensive evaluation, it is recommended to include evaluation on recent multimedia benchmarks such as TouchStone[4] and MMBench[5].\n  - In Table 2, the coco caption evaluation with ChatGPT seems questionable. The approach of randomly selecting a ground-truth caption and asking ChatGPT if the generated text covers the main visual concepts in the label text may lead to instances where the selected \"ground-truth\" caption and the generated text depict the image in different aspects, yet both are considered correct. Reporting standard metrics such as BLEU and Rouge scores, which consider multiple candidate ground-truth captions, could mitigate the issues.\n  - Table 5 lacks the CHAIR_S score for different methods. Additionally, it indicates that MiniGPT-4 fails to follow the instruction's requirement of 20 words, as evidenced by the average length of 27 words for MiniGPT-4 (short). To me, instead of showcasing the promising results on so-called adavanced tasks, satisfying the user requirement of a basical instruction should be of higher priority.\n- Some claims made in the paper are misleading:\n  - In Section 3.2, the paper claims that no equivalent datasets exist (for multi-modal instruction tuning) in the vision-language domain. However, there have been recent studies addressing this problem, such as LLaVA[1], InstructBLIP[2], M3IT[6], and MultiInstruct[7]. To accurately reflect the current state of the field, the author should properly cite these studies and consider exploring further fine-tuning with these datasets.\n  - In Section 4.4, the paper claims that MiniGPT-4 achieves similar results with Qformer, implying that Qformer does not play a critical role in advanced skills. However, this claim may not be fully supported by the AOK-VQA and GQA results, along with additional case studies. Additionally, if the advanced skills require inference between multiple images, the downsampling effect of Qformer (i.e., downsampling visual patches from 256 to 32) could be crucial and should be considered.\n\n## References\n\nIt is not a curated list, and most of them appeared  more than 3 months before the submission deadline.\n\n[1] Visual Instruction Tuning, https://arxiv.org/abs/2304.08485\n\n[2] InstructBLIP: Towards general-purpose vision-language models with instruction tuning, https://arxiv.org/abs/2305.06500 \n\n[3] mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality, https://arxiv.org/abs/2304.14178\n\n[4] TorchStone: https://github.com/QwenLM/Qwen-VL/blob/master/touchstone/README.md\n\n[5] MMBench: https://opencompass.org.cn/leaderboard-multimodal\n\n[6] M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning, https://arxiv.org/abs/2306.04387\n\n[7] MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning, https://arxiv.org/abs/2212.10773", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)