TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-06-02 18:52:28

Abstract: In recent years, multimodal large language models (MLLMs) such as GPT-4V have demonstrated remarkable advancements, excelling in a variety of vision-language tasks. Despite their prowess, the closed-source nature and computational demands of such models limit their accessibility and applicability. This study introduces TinyGPT-V, a novel open-source MLLM, designed for efficient training and inference across various vision-language tasks, including image captioning (IC) and visual question answering (VQA). Leveraging a compact yet powerful architecture, TinyGPT-V integrates the Phi-2 language model with pre-trained vision encoders, utilizing a unique mapping module for visual and linguistic information fusion. With a training regimen optimized for small backbones and employing a diverse dataset amalgam, TinyGPT-V requires significantly lower computational resources—24GB for training and as little as 8GB for inference—without compromising on performance. Our experiments demonstrate that TinyGPT-V, with its language model 2.8 billion parameters, achieves comparable results in VQA and image inference tasks to its larger counterparts while being uniquely suited for deployment on resource-constrained devices through innovative quantization techniques. This work not only paves the way for more accessible and efficient MLLMs but also underscores the potential of smaller, optimized models in bridging the gap between high performance and computational efficiency in real-world applications. Additionally, this paper introduces a new approach to multimodal large language models using smaller backbones. Our code and training weights are available in the supplementary material.

Main paper: 
 introduction: 
 In recent years, the field of artificial intelligence has seen significant advancements through the development of multimodal large language models (MLLMs), such as GPT-4V, which have shown exceptional performance across a range of vision-language tasks (Yang et al., 2023) . Despite GPT-4V's impressive capabilities, its closed-source nature limits its widespread application and adaptability. In contrast, the open-source landscape for MLLMs is rapidly evolving, presenting models like LLaVA and MiniGPT-4 that excel in image captioning (IC), visual question answering (VQA) often comparable GPT-4V in these areas (Dai et al., 2023; Liu et al., 2023a; b; Zhu et al., 2023) . Notably, MiniGPT-v2 (Chen et al., 2023) has demonstrated superior performance in various visual grounding and question-answering tasks. However, its training code remains proprietary, which poses challenges for community-driven advancements and adaptability.Although the impressive vision-language capabilities demonstrated by some open-source MLLMs, they frequently necessitate significant computational resources for training and inference. For example, training LLaVA-v1.5-13B (Liu et al., 2023a) derlying performance of large language models, which are integral to MLLMs, is pivotal. Models such as LLaVA-v1.5-13B and MiniGPT-v2 depend on high-capacity backbones which are Vicuna-13b-v1.5 (Zheng et al., 2023) and LLaMA2-7B-Chat (Touvron et al., 2023) , respectively, necessitating a substantial number of parameters to effectively tackle complex tasks including IC and VQA.We introduce TinyGPT-V, a novel model designed for efficient training and inference, requiring only 24GB of GPU memory for training and as little as 8GB of GPU or CPU memory for inference. This model makes use of the advanced large language model Phi-2 (Javaheripi et al., 2023) and incorporates pre-trained vision modules from (Li et al., 2023a) and CLIP (Radford et al., 2021) as its vision encoder, coupled with a mapping module to facilitate the integration of visual information. During training, TinyGPT-V adopts a novel training methodology focused on small pre-trained backbones, unlike any other MLLMs, utilizing the unique mapping module between the visual encoder and the language model as well as novelty normalization methods, while keeping all other components frozen. For its training dataset, TinyGPT-V employs the multi-tasks datasets, including LAION (Schuhmann et al., 2021) , Conceptual Captions (Changpinyo et al., 2021; Sharma et al., 2018) , SBU (Ordonez et al., 2011) , and others (Lin et al., 2015; Schwenk et al., 2022; Hudson & Manning, 2019; Kiela et al., 2020; Lu et al., 2021; Gurari et al., 2018; Mao et al., 2016; Kazemzadeh et al., 2014; Yu et al., 2016) .In our study, we found that TinyGPT-V exhibits similar traits with GPT-4, especially when doing some VQA and image inference. With only 2.8 billion parameters of its language model, TinyGPT-V employs a unique quantization process, like using 8-bit quantization, making it well-suited for local deployment and inference on 8GB mobile devices. This model represents a significant advancement in achieving a balance between exceptional performance and efficiency in MLLMs, as shown in Figure 1 . Our work not only aims to enable the community to develop more cost-effective, efficient, and high-performing MLLMs for widespread realworld applications but also introduces a training framework optimized for small pre-trained backbones. 
related work: 
 Advanced language model. The evolution of language models has been marked by significant milestones, starting with early successes like GPT2 (Radford et al., 2019) and BERT (Devlin et al., 2018) in natural language processing (NLP). These foundational models set the stage for the subsequent development of vastly larger language models, encompassing hundreds of billions of parameters. This dramatic increase in scale has led to the emergence of advanced capabilities as seen in models like GPT-3 (Brown et al., 2020) , Chinchilla (Hoffmann et al., 2022) , OPT (Zhang et al., 2022) , and BLOOM (Workshop et al., 2022) . These large language models (LLMs) have been instrumental in further advancements in the field. For instance, Chat-GPT (OpenAI, 2022) and InstructGPT (Ouyang et al., 2022) leverage these powerful models to answer diverse questions and perform complex tasks such as coding. The introduction of open-source LLMs like LLaMA (Touvron et al., 2023) has further propelled research in this area, inspiring subsequent developments like Alpaca (Taori et al., 2023) , Vicuna (Chiang et al., 2023) . These models fine-tune the LLaMA model with additional high-quality instruction datasets, showcasing the versatility and adaptability of LLM frameworks.Among the most notable recent advancements are Phi (Li et al., 2023b) and its successor, Phi-2 (Javaheripi et al., 2023) . These models have demonstrated exceptional performance, rivaling or even surpassing models up to 25 times larger in scale. This indicates a significant shift in the landscape of language modeling, emphasizing efficiency and effectiveness without necessarily relying on sheer size.Multimodal language model. In recent years, the trend of aligning visual input to large language models for visionlanguage tasks has gained significant attention (Chen et al., 2022; Tsimpoukelli et al., 2021; Alayrac et al., 2022; Li et al., 2023a; Liu et al., 2023b; a; Zhu et al., 2023; Chen et al., 2023) . Seminal works like VisualGPT (Chen et al., 2022) and Frozen (Tsimpoukelli et al., 2021) , which utilized pre-trained language models for image captioning and visual question answering. This approach was further advanced by models such as Flamingo (Alayrac et al., 2022) , which incorporated gated cross-attention mechanisms to align pre-trained vision encoders and language models, training on vast image-text pairs. BLIP-2 (Li et al., 2023a) introduced an efficient Q-Former for aligning visual and language modalities. These groundbreaking studies have paved the way for further innovations in the field, leading to the development of models like LLaVA (Liu et al., 2023b) and MiniGPT4 (Zhu et al., 2023) , and their subsequent iterations, LLaVA-v1.5 (Liu et al., 2023a) , MiniGPT-v2 (Chen et al., 2023) , ArtGPT-4 (Yuan et al., 2023) , instruction GPT-4 (Wei et al., 2023) and Instruction Mining (Cao et al., 2023) . These models have demonstrated advanced multimodal capabilities through instruction tuning, showcasing remarkable generalization abilities. 
method: 
 We briefly introduce our vision-language model, TinyGPT-V, followed by an analysis of its structure, culminating in a detailed description of the training process for each stage.In this subsection, we present the architecture of TinyGPT-V, which consists of a visual encoder, projection layers, and a large language model, as shown in Figure 4 .Visual encoder backbone. In the TinyGPT-V, it utilizes EVA (Fang et al., 2022) of the ViT serves as the visual foundation model, which remains inactive during the entire training process. Our model operates at an image resolution of 224x224 for Stages 1, 2, and 3, and at 448x448 for Stage 4. The positional encoding is enhanced to accommodate the increased image resolution which is known as the Relative Position Bias (Dufter et al., 2021) . It enhances the model's understanding of the spatial relationships between elements in an image.Projection layers. The Projection layers embed visual features extracted by the visual encoder into the language model, enhancing the model's ability to process imagebased information. We adopt the Q-Former layers from the BLIP-2 architecture (Li et al., 2023a) as the initial pro- This is an image of an alpaca.Alpacas are domesticated species of South American camelids, known for their soft fluffy coats which are used for making wool.[vqa]What's this? jection layer, aiming to leverage the full potential of the pre-trained BLIP system within visual language models. This strategy significantly reduces the number of parameters needing training. The second and third layers are linear projection layers, designed to bridge the dimensionality gap between the Q-Former output and the language model's embedding layer, thereby aligning visual tokens more effectively with the language model's hidden space. As shown in Figure 6 , to expedite TinyGPT-V's training, we initially use a pre-trained Linear Projection from MiniGPT-4 (Vicuna 7B) as the second layer. We then introduce an additional linear projection layer, initialized with a Gaussian distribution, as the third layer to seamlessly integrate into the hidden space of the Phi-2 model.Large lanuguage model backbone. Our TinyGPT-V large language model is built upon the Phi-2 model (Javaheripi et al., 2023) as its backbone. Phi-2, a 2.7 billion-parameter language model, exhibits exceptional reasoning and language comprehension abilities, achieving state-of-the-art performance among language models with fewer than 13 billion parameters. In complex benchmarks, Phi-2 either matches or exceeds the performance of models up to 25 times its size. We primarily use Phi-2's linguistic abilities to do various vision-language tasks. Specifically, for vision reasoning tasks that involve spatial location identification, we instruct the linguistic model to generate textual descriptions of what will happen in the next scenario, representing their objects' coordinates, as shown in Table 8 .Normalization and LoRA for TinyGPT-V. In Section 4.4, we conclude that training smaller-scale large language models for transfer learning, especially across different modalities (e.g., from text to image), poses significant challenges.Our studies indicate that these smaller models are prone to encountering NaN or INF values during multimodal data computations. This issue often leads to a computational loss value of NaN, thereby causing failure in the initial batch forward propagation. Moreover, the limited number of trainable parameters in these models may lead to gradient vanishing during training. To mitigate these problems, as depicted in Figure 5 (c), we incorporate the post-norm and input norm mechanisms from LLaMA-2, applying RMS Norm after each Multi-Head Attention Layer (MHA) to normalize data for downstream processing. In addition, we have to update all layer norms in the Phi-2 model to improve training stability, as detailed in the subsequent equation.LayerNorm input (x hidden ) = γ x hidden -µ √ σ 2 + ϵ + β (1)Where, x hidden is the input of this layer, µ and σ 2 are the mean and variance of the inputs to the layer, respectively, ϵ is a small number to prevent division by zero, γ and β are trainable parameters.RMSNorm(x post ) = x post 1 N N i=1 x 2 i + ϵ (2)where x post is the input after MHA, N is the dimension of x post .Furthermore, (Henry et al., 2020) have underscored the vital role of Query-Key Normalization in low-resource learning scenarios. Hence, as show in Figure 5 (d), we have incorporated Query-Key Normalization into the Phi-2 model, as detailed in the following equation.Attention(Q, K, V ) = softmax LayerNorm(Q)LayerNorm(K) T √ d k V (3)where d k denotes the dimension of Q or K.The structure of the LoRA mechanism (Hu et al., 2021) is show in Figure 5 (a), which is an efficient fine-tuning method in parallel to the frozen pre-training weights as shown in Figure 5 (c), which does not increase the inference time consuming for large language models and is easier to optimize.In this subsection, the four-stage training process of TinyGPT-V will be described.  
experiments: 
 In this section, we describe the training and evaluation methods in detail.Experimental setting. The experimental environment for this study was established with a single NVIDIA RTX 3090 GPU, equipped with a substantial 24GB of VRAM. The central processing was handled by an AMD EPYC 7552 48-core Processor, offering 15 virtual CPUs. Memory allocation was set at 80GB, ensuring sufficient capacity for handling large datasets. The software environment was standardized on PyTorch version 2.0.0, with CUDA 11.8 support, facilitating optimized tensor operations on the GPU.Training process. In our experimental process, we meticulously orchestrated the training of our model through four distinct stages, each characterized by specific learning rate strategies and loss profiles, as shown in Figure 7 .Stage 1: Spanning 17 epochs, with each epoch consisting of 1000 iterations, we employed a dynamic learning rate approach. The learning rate commenced at 1e-5 at the beginning of each epoch and gradually ascended to 1e-4 by the epoch's end. This pattern was consistently applied across all 17 epochs. The training loss exhibited a steady decline, starting from 7.152 and progressively tapering down to 2.620, reflecting the model's increasing proficiency in learning from the data. The purpose of this stage is to be able to make the Phi-2 model in TinyGPT-V react in some way to the input of the imaging modality. The alignment of text and image in the semantic space is done.Stage 2: Comprising 4 epochs, each with 5000 iterations, this stage introduced the "linear_warmup_cosine_lr" (He et al., 2018; Goyal et al., 2018) learning rate schedule. We initiated a warmup phase of 5000 steps, where the learning rate linearly increased from 1e-6 (warmup_lr) to 1e-4 (init_lr), followed by a cosine decay down to a minimum learning rate of 8e-5. This phase saw a consistent reduction in loss, starting at 2.726 and culminating at 2.343. The purpose of this stage is to enable the LoRA module to play a role in multimodal data, further reducing the model's loss on image-text pairs and improving the model's ability to learn from the data.Stage 3: This stage lasted for 5 epochs, each with 200 iterations. We maintained the "linear_warmup_cosine_lr" schedule, with a warmup phase of 200 steps. The learning rate began at 1e-6, ascending to 3e-5 (init_lr), before decaying to 1e-5 (min_lr). The loss values reflected significant improvements, starting at 1.992 and reducing to 1.125. The purpose of this stage is to allow TinyGPT-V to accept both verbal and image modal inputs and produce responses to them. After this stage of training TinyGPT-V has been able to perform most of the image answering tasks.Stage 4: The final stage stretched over 50 epochs, each comprising 1000 iterations. We adhered to the "lin-ear_warmup_cosine_lr" schedule with a 1000-step warmup phase. The learning rate was initiated at 1e-6, reaching up to 1e-5 (init_lr), and then experiencing a cosine decay to a minimum of 8e-5. The training loss values displayed a consistent downward trajectory, beginning at 2.720 and ultimately reaching as low as 1.399. The purpose of this stage is to allow TinyGPT-V to perform various tasks such as VQA or VSR tasks at the same time, increasing the generalization performance of TinyGPT-V on multimodal tasks.Evaluation datasets. GQA (Hudson & Manning, 2019) is a dataset for real-world visual reasoning and compositional question answering, featuring a powerful question engine that generates 22 million diverse reasoning questions. VSR (Liu et al., 2023b) comprises over 10k natural textimage pairs in English, encompassing 66 types of spatial relations. IconQA (Lu et al., 2021) with 107,439 questions aimed at challenging visual understanding and reasoning in the context of icon images, encompassing three sub-tasks (multi-image-choice, multi-text-choice, and filling-in-theblank). VizWiz (Gurari et al., 2018 ) is a collection of more than 31,000 visual queries, each derived from a photo taken by a visually impaired individual using a smartphone, accompanied by a vocalized question regarding the image, and supplemented with 10 answers sourced from a crowd for each query. The Hateful Memes dataset (HM) (Kiela et al., 2021) , developed by Facebook AI, is a comprehensive multimodal collection specifically designed for the detection of hateful content in memes, combining both image and text elements, and comprises over 10,000 newly created multimodal examples.Visual question answering results. As shown in Table 2 , it becomes evident that TinyGPT-V, a language model with only 2.8 billion parameters, exhibits notably competitive performance across multiple benchmarks, closely rivaling models with nearly 13 billion parameters. Specifically, in the VSR (Visual-Spatial Reasoning) zero-shot task, TinyGPT-V outshines its counterparts by securing the highest score of 54.7%. This is particularly impressive considering its parameter size is approximately 4.6 times smaller than other leading models such as BLIP-2, LLaVA, and InstructBLIP.In the GQA benchmark, while TinyGPT-V scores are 38.9%, it lags behind the highest score achieved by InstructBLIP, which is 49.5%. However, TinyGPT-V shows robust perfor- mance in the IconVQ challenge, attaining a score of 44.7%, just 0.1% short of InstructBLIP's leading score of 44.8%. Similarly, in the VizWiz task, TinyGPT-V demonstrates commendable capabilities with a score of 37.8%, which, is not only the highest but is notable given its reduced parameter count. In the context of the Hateful Memes (HM) dataset, TinyGPT-V matches InstructBLIP's top score of 57.5% with its own score of 54.0%, again underscoring its efficiency and capacity to compete with models of larger scales. Overall, TinyGPT-V's performance across these diverse and challenging benchmarks is striking, especially when considering its parameter efficiencyThe comparative analysis revealed TinyGPT-V's distinct advantage in delivering concise and accurate visual interpretations. In the reasoning task to find a hiding spot during a game of hide and seek, TinyGPT-V demonstrated its superior capability by providing a singular, viable suggestion: 'under couch'. This contrasts with other models that either offered multiple options, some of which were incorrect as indicated by the text in red (e.g., GPT-4V suggesting 'Inside the Bookshelf'), or specified less practical hiding spots. When asked about potential activities in an image with an alligator, TinyGPT-V suggested a cautious response without speculating beyond what was visible. In contrast, other models, like LLaVA-1.5, provided extended narratives that introduced assumptions not directly inferred from the image. Similarly, in describing a soccer match scene, TinyGPT-V's response was succinct and focused on the key elements, avoiding the inaccuracies noted in MiniGPT-v2's account, which incorrectly identified multiple soccer balls on the Furthermore, our reveal a notable trend: the smaller the large language model used for transfer learning (particularly in transitioning from text-to-image modality), the more challenging the training process becomes. We observed a pronounced need for additional normalization layers to stabilize the training, especially when scaling down from larger models like Vicuna-13B to smaller ones like Phi-2 (2.7B), Phi-1.5 (1.3B), and other small backbones as detailed in the Appendix B. 
conclusion: 
 In this study, we introduce TinyGPT-V, a parameter-efficient MLLMs tailored for a range of real-world vision-language applications. Our model innovatively builds on the compact yet powerful Phi-2 small language model framework. This approach results in TinyGPT-V delivering exceptional outcomes in diverse benchmarks like visual question-answering and referring expression comprehension while keeping computational demands manageable. Remarkably, TinyGPT-V can be trained on a 24G GPU and deployed on an 8G device, demonstrating a significant advancement in creating costeffective, efficient, and potent MLLMs. This paper marks a contribution towards crafting smaller, yet robust multimodal language models for practical, real-world use cases. We envision that our work will catalyze further explorations into developing compact MLLMs for diverse applications.of training. Moreover, the sequential progression in training losses across stages for models without these modifications demonstrates a clear degradation in training efficiency and effectiveness. For example, the removal of Input Layer Norm and RMS Norm not only heightened Stage 1 loss across Phi-1.5 and TinyLLaMA but also precipitated gradient vanishing in later stages, showcasing the compound impact of these modules on model stability and learning capability. This analysis incontrovertibly highlights a fundamental challenge in training smaller LLMs for migration to MLLMs: the absence of key architectural and normalization modules severely impedes their training process, making them more prone to early training halts and efficiency losses. The results underscore the necessity of these components in supporting the stability and gradual learning progression of smaller LLMs, thus illuminating a pivotal consideration for developers aiming to optimize the training framework for seamless model.  
 
 Appendix:  


Decision: **Strengths**
- TinyGPT-V reduces the size and costs of MLLMs thus increasing the availability of this important technology. 
- The paper is well written, and methods well described fostering reproducibility. 
- TinyGPT-V is open sourced and thus useful for the broad community. 

**Weaknesses**
- It would be useful to discuss TinyGPT-V limitations, where it is inferior to larger MLLMs. 
- Evaluation should include a wider range of benchmarks and tasks. 

**Summary**
This is a timely and important work for the community.