Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction (2024)

Abstract: In this paper, we introduce EVLGen, a streamlined framework designed for the pre-training of visually conditioned language generation models with high computational demands, utilizing frozen pre-trained large language models (LLMs). The conventional approach in visionlanguage pre-training (VLP) typically involves a two-stage optimization process: an initial resource-intensive phase dedicated to generalpurpose vision-language representation learning, focused on extracting and consolidating relevant visual features. This is followed by a subsequent phase that emphasizes end-to-end alignment between visual and linguistic modalities. Our novel one-stage, single-loss framework bypasses the computationally demanding first training stage by gradually merging similar visual tokens during training, while avoiding model collapse caused by single-stage training of BLIP-2 type models. The gradual merging process effectively condenses visual information while preserving semantic richness, resulting in rapid convergence without compromising performance. Our experimental findings demonstrate that our approach accelerates the training of vision-language models by a factor of 5 without a noticeable impact on overall performance. Furthermore, we illustrate that our models significantly narrow the performance gap to current vision-language models using only 1/10 of the data. Finally, we showcase how our image-text models can seamlessly adapt to video-conditioned language generation tasks through novel soft attentive temporal token contextualizing modules. Code is available at https://github.com/yiren-jian/EVLGen.

heading: 1 Introduction
text: The landscape of vision-language modeling has undergone significant transformations in recent years, with CLIP (Radford et al., 2021) serving as a landmark development. It distinguished itself through
*This work was done during an internship at ByteDance Inc.
unparalleled zero-shot classification capabilities and efficiency in image-text retrieval tasks from prior works (Zhang et al., 2022a). Successive models like ALBEF (Li et al., 2021a), X-VLM (Zeng et al., 2022), and VLMo (Bao et al., 2022) further broadened the scope, addressing a myriad of tasks such as retrieval, visual entailment, and closed-set Visual Question Answering (VQA), among others.
Recently, the field has been enriched by the advent of generative models designed for complex image-to-language tasks. Notable contributions include CoCa (Yu et al., 2022), SimVLM (Wang et al., 2022c), Frozen (Tsimpoukelli et al., 2021), and Flamingo (Alayrac et al., 2022), targeting tasks like image and video captioning and open-set VQA. These models all rely on billion-scale datasets for training from scratch to bridge the substantial modality gap between vision and language.
As a result, the resource-intensive requirements (i.e., thousands of TPUs) of these training-fromscratch Vision-Language Models (VLMs) led to the conceptualization of BLIP-2 (Li et al., 2023a): this model alleviates computational costs (e.g., only requiring 16× fewer GPUs) by integrating existing well-pretrained vision encoders (ViT) with language decoders (LLM), and then tuning their joint operation. A central innovation in aligning vision and language modules in BLIP-2 is Q-former, a multimodal connector equipped with learnable queries for enhancing cross-attention mechanisms. This architectural choice, however, prevents the full model from end-to-end training and therefore still demands an additional pre-training regimen for Qformer, referred to as BLIP-2’s Stage 1. The stage involves three learning objectives—image-text contrastive, image-text matching, and language generation—and necessitates multiple forward passes for facilitating the Q-former’s optimization.
Despite its efficiency gains over CoCa, BLIP2’s training still imposes considerable computational costs. This poses challenges for research en-
300
vironments with limited computational resources, such as university labs. Our experiments indicate that the Stage-1 training of BLIP-2 took approximately eight days on eight A100-80G GPUs (See Appendix B for training configurations). This computational burden has consequently restricted research to using the pre-trained Q-former, hindering the exploration of alternative ViTs in VLMs. This limitation is evident in subsequent works such as InstructBLIP (Dai et al., 2023b), VideoChat (Li et al., 2023b), Video-LLaMA (Zhang et al., 2023b), X-LLM (Chen et al., 2023a).
The prospect of reducing BLIP-2’s computational cost through end-to-end, single-stage training is compelling. Such an approach would remove the complexities associated with resource allocation and hyper-parameter tuning inherent in multi-stage training. Yet, direct end-to-end training with BLIP-2 poses substantial challenges, corroborated by both original findings from BLIP-2 and our own empirical analyses. We hypothesize that these challenges emanate from the intrinsic design of the Q-former. Specifically, the inclusion of randomly initialized learnable queries and crossattention mechanisms complicates the optimization landscape, especially when the aim is to minimize the representational disparity between visual and linguistic modalities.
In this paper, we propose a token merging Transformer (TomeFormer) as an efficient visionlanguage connector. TomeFormer employs a systematic token-merging (Bolya et al., 2023) strategy that is both intuitive and effective. By connecting a pre-trained ViT as the visual encoder and a frozen LLM as the language decoder, we intro-
duce a new VLM “Expedited Visual Language Generation model” (EVLGen), facilitates a streamlined, single-stage training process. It requires only a singular learning objective and a single forward pass per optimization step. This stands in contrast to BLIP-2’s multi-stage training, laden with multiple objectives and several forward passes.
Further, we introduce a soft attentive temporal contextualization mechanism within the ViT for effective video-language modeling. This uncovers more shared semantic features across temporal frames, thereby improving the efficiency of the spatial token merging process. It eliminates the need for modality realignment, contrasting approaches such as the temporal Q-former (Zhang et al., 2023b), or the addition of new learnable temporal queries (Li et al., 2023b). Our strategy simplifies the optimization challenges tied to working with relatively smaller video-text datasets, compared to their image-text counterparts. Remarkably, we demonstrate that even without video pre-training, our temporal token contextualize approach can effectively train robust video-language models. This differs from recent work in videolanguage models (Yan et al., 2022; Chen et al., 2023b) that depend on pre-training models using vast million-scale video-text datasets. In summary, our contributions are:
• For reducing vision redundancy within the vision language connector, we adopt Token Merging, initially designed to enhance ViT inference speed without training. Concurrently, we present a novel temporal token contextualization scheme for video modeling. • Our proposed VLM featuring TomeFormer com-
petes effectively with BLIP-2, while requiring just a fraction of the computational resources. Given the reliance on BLIP-2’s pre-trained model in contemporary studies, our approach widens the exploratory scope for various ViTs. • We introduce a straightforward spatial attentive temporal modeling technique that allows for the seamless adaptation of pre-trained image-text models to video tasks. This approach eliminates the need for complex modality re-alignment, a common requirement in alternative methods.

heading: 2 Related Work
text: Image-Language Models CoCa (Yu et al., 2022), trained on billions of image-text pairs, represents a state-of-the-art approach in generative tasks like open VQA and visual captioning. To mitigate the computational demands of pre-training, BLIP2 (Li et al., 2023a) employs frozen pre-trained ViT and LLM components, focusing on training a specialized connector between visual and linguistic modalities called the Q-former. Due to the computationally intensive nature of training BLIP-2, subsequent models in visual instruction (Dai et al., 2023b; Zhu et al., 2023; Li et al., 2023b) have predominantly utilized the pre-trained Q-former, which is aligned with the eva-vit-g model supplied by BLIP-2. Additional related works on image-language modeling are further discussed in Appendix A.
Video-Language Models While many imagetext models can be adapted for video-text tasks through simple feature pooling (e.g., VideoCoCa (Yan et al., 2022)), the field has seen specialized models that incorporate temporal dynamics. Building on the foundation of BLIP-2, VideoLLaMA (Zhang et al., 2023b) enhances its architecture by introducing additional temporal Q-former layers between the spatial Q-former and the LLM components of BLIP-2. Inspired by BLIP-2, most recent works such as VideoChat (Li et al., 2023b), PandaGPT (Su et al., 2023), Valley (Luo et al., 2023), and Video-ChatGPT (Muhammad Maaz and Khan, 2023) leverage frozen LLMs in their videolanguage models.
Token Merging Token Merging (ToMe) (Bolya et al., 2023) aims to improve the inference speed of pre-trained ViTs without requiring re-training. At each Transformer layer, tokens are divided into two sets and subsequently merged based on similarity,
effectively reducing the token count and thereby accelerating inference. This method maintains classification and generation quality.
In our work, we repurpose ToMe to condense the visual features used as language prompts in the LLM. We integrate a standard Transformer with ToMe capabilities, resulting in a model we term TomeFormer. This model serves as an effective connector between visual and language domains, preserving semantic richness while reducing token count. Importantly, this integration of ToMe does not introduce any additional parameters. Inspired by spatial ToMe, we introduce a novel soft temporal ToMe variant within the vision encoder, thereby adding temporal modeling capabilities to our image-text models. Additional related works on token redundancy are further discussed in Appendix A.

heading: 3 Methods
text: We begin by presenting our image-text model and then describe the adaptations made to this pretrained model for video-related tasks.

heading: 3.1 Preliminary
text: We follow BLIP-2’s efficient training paradigm, i.e., utilizing frozen but well-pretrained ViTs and LLMs while solely training a vision-to-language connector. However, BLIP-2 still remains a challenge, since it necessitates an extra Stage-1 as a pre-training phase for the unstable Q-former (i.e., the vision-to-language connector), before the final end-to-end fine-tuning.
Our observation underscores the pivot role of BLIP-2 Stage-1 pre-training (which takes approximately 8 days on eight A100 GPUs): without it, the BLIP-2 model collapses, as evidenced in Table 1. To avoid this extra stage-1, we replace Q-former with a novel vision-to-language connector, which is designed to discover vision redundancy and then significantly accelerate visual-language alignment, often resulting in enhanced performance.

heading: 3.2 EVLGen-Image
text: We introduce EVLGen-Image (abbreviated as EVLGen, shown in Figure 1), an optimized visionlanguage generative pre-training model. EVLGen utilizes a ViT for visual encoding and an LLM for linguistic decoding. The key innovation is the incorporation of a standard Transformer, augmented with spatial Token Merging, to act as the connector between the visual and linguistic modalities.
Formally, our framework includes a vision encoder Evision, which ingests an input image I and encodes it into a fixed set of visual tokens: [v1, v2, ...vL] = Evision(I). Here, L denotes the number of image patches. Subsequently, we employ a Transformer equipped with token-merging modules (further technical details are provided in Appendix C), termed as TomeFormer (Tv→l) as the vision-to-language connector. This module effectively compresses the token count:
[v′1, v ′ 2, ...v ′ L′ ] = Tv→l(fproj1([v1, v2, ...vL])). (1)
In this equation, L′ is considerably smaller than the initial token count L1. The LLM decoder then employs these compressed tokens as soft prompts for text generation:
output = DLLM(fproj2([v ′ 1, v ′ 2, ...v ′ L′ ])). (2)
Projection functions fproj1 and fproj2 are used to ensure dimension compatibility. The training objective is to minimize the cross-entropy between the output and ground truth caption:
L = CrossEntropyLoss(output, capgt). (3)
Three main advantages of using TomeFormer are: 1We merge a fixed number of tokens at each layer of the TomeFormer. Finally, 256 visual tokens are reduced to 28 tokens. Ablation on merged tokens at each layer is studied in Section 5.
• Efficient token reduction, facilitating the transformation of loosely structured visual data into a more concise yet informative representation. • Computational efficiency, as the uncompressed ViT output consists of 256 tokens, plus a [CLS] token. Without compression, the subsequent vision-to-language connector would be computationally expensive in terms of both memory and processing power. • Semantic richness of the compressed tokens. Unlike BLIP-2, which requires an extensive pretraining phase for Q-former, TomeFormer naturally merges semantically similar tokens. Our empirical evidence confirms that TomeFormerequipped models train more efficiently compared to alternatives like BLIP-2.

heading: 3.3 EVLGen-Video
text: Although many image-text models can be adapted for video-text tasks with minor modifications, such adaptations either result in inadequate temporal modeling (as in VideoCoCa or InstructBLIP) or require re-alignment with substantial video-text pairs due to additional learnable Q-formers (as in VideoChat and Video-LLaMA).
In this paper, we propose a novel module called Temporal Attentive Soft Token Contextualizing to enhance the ViT backbone with temporal modeling capabilities (depicted in Figure 2). A key feature
of temporal soft contextualizing is that it is equivalent to the identity operator when the input is static images instead of videos. Thus, our approach maintains the integration of pre-existing, well-trained image-text models, thus avoiding the additional need for modality realignment, a requirement in alternative designs that insert a temporal Q-former between the visual encoder and LLM.
Formally, let v be a video feature tensor with dimensions [B × N × L × D], where B is the batch size, N is the number of frames, L is the sequence length (i.e., the number of patches in a single video frame), and D is the hidden dimension. Initially, we reshape v into [(B × N) × L × D] which is subsequently fed into the self-attention layer of the ViT for spatial modeling as:
v′ = self-attn(v.reshape(B ×N,L,D)). (4)
For temporal modeling, v′ is reshaped to [(B × L), N,D]. We then project this into key and query matrices k and q and compute v′′ using our Temporal Attentive Soft Token Contextualizing as follows:
k = Wkey(v ′.reshape(B × L,N,D)), (5) q = Wquery(v ′.reshape(B × L,N,D)), (6) v′′ = v′ + softmax(matmal(q, k)) · v′. (7)
The softmax operation models temporal weights and softly fuses tokens among multiple frames. This is distinct from spatial token merging, which employs average pooling and reduces the token count. Here, the weighted average pooling is applied to multiple frames for contextualization. It preserves the original count of tokens while enhancing the shared semantic content that is spread across various frames. Therefore, it allows a higher rate of token merging in the subsequent spatial merging processes.

heading: 4 Experiments
text: Our experimental setup is as follows: • Pre-training Data Our model is pre-trained
using the MSCOCO (Lin et al., 2014) and CapFilt (Li et al., 2022) datasets, which include BLIP’s pseudo-labeled Conceptual Captioning (Sharma et al., 2018), SBU (Ordonez et al., 2011), and LAION (Schuhmann et al., 2022) datasets—similar to the data sources utilized in BLIP-2. Note that we intentionally exclude the VG (Krishna et al., 2017) dataset from our pre-training procedure, as it mainly consists of localized captions.
• Models To facilitate a direct and fair comparison with BLIP-2, we employ the same ViT, textttevavit-g (Fang et al., 2023). For the language model decoders, we explore both opt-2.7b (Zhang et al., 2022b) and vicuna-7b (Chiang et al., 2023). Our TomeFormer is initialized using bert-base-uncased, ensuring parameter count parity with BLIP-2’s Q-former. • Pre-training Details Our pre-training setup closely mirrors the configurations of BLIP-2. We utilize a maximum learning rate of 1e−4 and a minimum learning rate of 1e−5. The learning rate follows a schedule that begins with a linear warm-up phase of 5000 steps starting from 1e−6
and then transitions to a cosine decay schedule. Weight decay is set at 0.05. The training is conducted with a batch size of 1600, distributed over either 8× A100-80G or 32× V100-32G. • Downstream Tasks EVLGen-Image is evaluated without additional fine-tuning on a variety of tasks, including MSCOCO captioning, VQAv2 (Goyal et al., 2017), GQA (Hudson and Manning, 2019), and OK-VQA (Marino et al., 2019). For video tasks, EVLGen-Video is evaluated on fine-tuned MSR-VTT (Xu et al., 2016) and MSVD (Chen and Dolan, 2011) captioning. We use the standard train/val/test splits.

heading: 4.1 Evaluation on Image-Text Benchmarks
text: We conducted comparative evaluations between EVLGen and BLIP-2 on multiple image-text benchmarks, including zero-shot VQAv2, GQA, OKVQA, and MSCOCO captioning. It is essential to note that BLIP-2 demands an extensive Stage-1 pre-training phase involving 250,000 optimization steps. This phase incorporates three distinct loss functions and necessitates multiple forward passes through the model, a process crucial for BLIP-2 to prevent model divergence.
Table 1 summarizes the results of our experiments. Our primary insights can be distilled into the following key points: • Utilizing the same training set of 104M image-
text pairs and an equal number of optimization steps (250K), EVLGen consistently outperforms BLIP-2 across nearly all evaluated tasks. • Remarkably, EVLGen maintains competitive performance even when its training budget is trimmed to approximately one-third of BLIP-2’s, specifically 150K optimization steps. • Our experiments show that EVLGen can produce satisfactory results with a significantly reduced
training dataset of 11 million image-text pairs, while still undergoing 150K optimization steps. • EVLGen retains its efficacy even when the training budget is restricted to as few as 90K steps, showing the model’s efficiency and robustness.
We further evaluate BLIP-2 and EVLGen on zeroshot NoCaps and Flickr30K datasets. Shown in Table 2, EVLGen consistently outperforms BLIP-2 in both datasets using different LLMs.
Training Time In the Stage-1 pre-training phase, BLIP-2 requires considerable time, necessitating multiple forward passes to optimize three separate loss functions. We document the training durations for both BLIP-2 and EVLGen when utilizing eight A100-80G GPUs in the last column of Table 1.
Although BLIP-2 significantly reduces training time relative to predecessors like CoCa, it still mandates an extended training duration, approximately ten days (8 days for stage 1 and 2 days for stage 2). This extensive time commitment limits the feasibility of researchers to investigate various ViT configurations. Most subsequent works based on BLIP-2 continue to use the pre-trained Q-former in conjunction with the eva-vit-g model, thereby narrowing the scope of ViT exploration. In contrast, EVLGen significantly trims the training time while maintaining satisfactory performances, thus providing researchers with the latitude to explore a wider array of advanced ViTs in future investigations.
Furthermore, MACs (FLOPs) in Q-Former and TomeFormer is discussed in Section 5.5.

heading: 4.2 Evaluation of EVLGen-Video
text: We proceed to evaluate the performance of finetuned EVLGen-Video models in video captioning tasks, utilizing OPT-2.7b as the language model decoder. Our investigation includes two specific variants of EVLGen-Video: the first is exclusively pre-trained on image data, while the second is further enhanced by pre-training on a corpus of 2 million video-text pairs sourced from the WebVid (Bain et al., 2021) dataset. To provide a comprehensive evaluation, we benchmark EVLGen-Video against five distinct models, described as follows: • Baseline (concat): This model processes each
frame of a video individually and concatenates their visual features to generate a single prompt for the LLM. This method is analogous to the strategy employed in InstructBLIP. • Baseline (mean): Similar to the concat baseline, this model processes each video frame individually but averages the visual features to create a
single prompt for the LLM. • Video-LLaMA: This variant incorporates the
BLIP-2 framework and enhances it with an additional temporal Q-former layer. For this evaluation, we focus solely on the vision-language component of Video-LLaMA. • VideoChat: This model extends BLIP-2 by integrating additional Uniformer modules within the ViT architecture and also incorporates learnable temporal queries in its Q-former component. • VideoCoCa: In this model, we adapt the OpenCoCa framework by mlfoundations and augment the existing CoCa architecture with a learnable attentional pooler, resulting in VideoCoCa.
Evaluation on MSR-VTT As detailed in Table 3, EVLGen-Video demonstrates superior performance relative to the baseline models, even without the aid of video-text pre-training. This result highlights the effectiveness of our proposed Temporal Attentive Soft Token Contextualizing in capturing temporal dynamics. Additionally, we observe an enhancement in performance when incorporating video-text pre-training along with Self-Critical Sequence Training (SCST) (Rennie et al., 2017).
Temporal Attentive Soft Token Contextualizing has the distinct advantage of maintaining the integration of the well-pretrained image-text model (i.e., EVLGen-Image). This contrasts with models such as Video-LLaMA and VideoChat, where the original BLIP-2 architecture is altered, necessitating a complex re-alignment process using videotext pairs. Our empirical analysis indicates that such re-alignment is a non-trivial endeavor (as shown in Table 3, Video-LLaMA and VideoChat struggle to re-align with 2M WebVid video-text pairs). It is worth noting that our VideoCoCa model is at a disadvantage when benchmarked against
Google’s reported results, which benefit from extensive training on a much larger billion-scale dataset.
Evaluation on MSVD Similarly, we evaluate EVLGen’s performance against Video-LLaMA, VideoChat, and VideoCoCa using the MSVD caption dataset (presented in Table 4). Our results corroborate that EVLGen consistently surpasses these competing models, further attesting to its robust performance across different video caption tasks.

heading: 5 Ablations and Analysis

heading: 5.1 Ablations on TomeFormer
text: Within the TomeFormer, the vision-to-language connector in EVLGen, we introduce a hyperparameter r that regulates the number of spatial tokens merged at each layer. Increasing r substantially reduces the token count, but runs the risk of eliminating important visual details. On the other hand, a smaller r produces two main effects: (1) a more diffuse representation of visual features, complicating the optimization landscape, and (2) elongated soft prompts for the LLM, leading to increased computational cost during training, such as memory overflow and extended training duration.
To study the effects of r, we conduct an ablation experiment using 8× RTX-A6000 and the CCS-14M dataset for pre-training. The models are trained for 60,000 steps, and their performance is evaluated using CIDEr scores on MSCOCO cap-
tioning. In Figure 3, we observe that a smaller r (e.g., 10) places a higher computational load on both TomeFormer and the LLM, extending training time and compromising optimization, as evidenced by lower CIDEr scores. In contrast, a larger r value (e.g., 25) expedites training but at the expense of model performance, likely due to excessive feature compression and consequent information loss. Additional ablation results on VQAv2, GQA and OKVQA are provided in Appendix G.

heading: 5.2 Ablations on ViT
text: Experimental results on EVLGen with different visual encoders (ViT) are provided in Appendix F. EVLGen is robust to different visual encoders, and the stronger ViT generally leads to better results. This implies that while EVLGen also requires retraining for different ViTs, but the single-stage training and quick convergence allow it to benefit from a future release of the latest ViTs, given its capability of fast adaptation.

heading: 5.3 Vicuna-7b as the LLM
text: In Table 5, we provide experimental results of BLIP-2 and ours using Vicuna-7b as the LLM decoder, on zero-shot VQAv2, GQA, OKVQA and MSCOCO captioning (CIDEr) tasks without additional fine-tuning.
EVLGen achieves better performance in GQA, OKVQA, and MSCOCO captioning using considerably less computing, though our model under-
performs in VQAv2. As we discussed in Limitations, the inability of EVLGen to extract questionconditioned visual features may lead to inferior results on VQAv2.

heading: 5.4 Token Merging Visualization in EVLGen
text: One notable advantage of EVLGen over BLIP-2 is the absence of a requisite Stage-1 pre-training for the vision-to-language connector. This simplifies the training pipeline by removing the need to train the model to extract text-informative visual features. We posit that the token merging process in TomeFormer naturally aggregates tokens associated with visually similar elements, thereby yielding concise yet semantically rich visual features from the onset of training. This inherent capability allows EVLGen to benefit from a more streamlined, single-stage training regimen with just one learning objective.
Essentially, our token merging strategy serves as an efficient approximation of Q-Former’s functionality, compressing visual features in a semantically meaningful manner. Figure 4 illustrates this, displaying the visual tokens before and after training with our TomeFormer. The figure shows that the compressed visual features obtained via token merging are semantically informative and offer basic object segmentation within the image. Furthermore, the semantic coherence of these merged tokens improves as training advances. Additional visualization examples are shown in Appendix E.

heading: 5.5 MACs (FLOPs) in Q-Former and TomeFormer
text: In this section, we compute multiply–accumulate operations (MACs) in Q-Former and TomeFormer. MACs performs a← a+(b×c). Whereas, FLOPs is floating operations which includes × / + / ÷ ... etc. One MACs has one × and one +. And thus, roughly speaking, FLOPs is two times as MACs.
In our experiments, BLIP-2 and EVLGen have identical ViTs and LM decoders. Thus, we only
compare the MACs in VL Connector in BLIP-2 and EVLGen (i.e., Q-Former and TomeFormer).
There’s a large MACs in BLIP-2 stage-1 due to three forward passes using Q-Former, where the last forward-pass used for caption loss dominates (27.0G). In contrast, EVLGen does not require such a representation training stage (stage-1) at all.
Another reason why BLIP-2 stage-1 is slow is that the computation of Image-Text Contrasive and Image-Text Matching losses needs concat_all_gather operations that require GPU communications. Further Image-Text Matching requires binomial sampling of hard negatives. In comparison, our EVLGen circumvents such computations/communications.

heading: 6 Discussion and Conclusion
text: This paper introduces EVLGen, an efficient and streamlined pre-training framework for visionlanguage generative models. Like BLIP-2, EVLGen employs frozen ViT and LLM. It further leverages a conventional Transformer architecture with tokenmerging capabilities, known as TomeFormer, to act as the vision-to-language connector. Compared to BLIP-2, EVLGen offers the distinct advantage of one-stage training. This reduces computational overhead and maintains competitive performance even with only 1/3 to 1/6 of the computational budget required by BLIP-2.
We have also extended EVLGen’s applicability to video captioning tasks by incorporating the Temporal Attentive Soft Token Contextualizing into its ViT. This enhances the model’s temporal modeling capabilities, culminating in the creation of EVLGenVideo. This extension has proven efficacious, delivering commendable performance even without specialized video-text pre-training. Our investigation underscores that a temporal module, which does not disrupt the integration of the well-pretrained image-text model (e.g., BLIP-2 and EVLGen), is a key factor contributing to this success.
EVLGen demonstrates the possibility of achieving state-of-the-art performance in vision-language tasks without the need for complex training regimens or high computational budgets. This work thus makes a significant contribution to the ongoing efforts to develop more accessible, efficient, and powerful models for understanding and generating visual and textual information.
Limitations
While E2VLGen has showcased its capacity for rapid convergence in VLM pre-training and has demonstrated notable proficiency in zero-shot image/video captioning, certain limitations warrant consideration. • Our approach is guided by a straightforward de-
sign aimed at facilitating the efficient and effective training of VLMs. To maintain simplicity in our methodology, we adopt a fixed value of r (19) within TomeFormer to compress visual information (i.e., a fixed length of visual softprompt). However, it is worth acknowledging that various images or videos might benefit from distinct optimal compression rates (r). Consequently, the incorporation of a variable r (i.e., variable length of soft-prompts for language models) may be deemed more desirable (A similar concern is present in BLIP-2, where the length of soft-prompts is consistently set to 32.). • One trade-off associated with the simplistic design of TomeFormer is its inability to enable text-specific selection of visual features. In applications like VQA, extracting visual features conditioned on the accompanying questions is considered beneficial. However, the current configuration of TomeFormer lacks the provision for this text-conditioned property within VLMs. A prospective redesign of TomeFormer that incorporates text-conditioned visual feature selection holds the potential to enhance VQA performance.
Ethics Statement
This research aims to enhance both the efficiency and applicability of vision-language generative models via EVLGen. Although our research does not involve human subjects directly, it is important to acknowledge and discuss the broader ethical implications.
Data Bias and Fairness: Our model is trained on publicly available datasets, namely CapFilt, MSCOCO, MSR-VTT, MSVD, and WebVid.
While these datasets are widely used, we acknowledge that we cannot fully ascertain the extent to which they may contain discriminatory, biased, or sensitive material. Given that our model inherits the biases present in these training datasets, there exists the risk of perpetuating or even amplifying existing societal biases. Despite the broad acceptance of these datasets, caution should be exercised.
Real-world Deployment and Responsible Usage: Like all generative models, EVLGen could be misappropriated for creating misleading or harmful content. Thus, it is imperative to implement safety mechanisms to counter such misuse when deploying the model in real-world applications. Special attention should also be paid to ensure that the model does not inadvertently produce outputs that could disclose sensitive or personal information. Finally, while EVLGen is intended as a general-purpose model, its application in contexts that could worsen societal biases or spread misinformation is a pressing concern. Developers and researchers employing EVLGen are advised to be cognizant of these risks and consider incorporating fairness-aware or truth-aware components into their systems.

heading: A Additional Related Works
text: Image-Language Models Vision-language models generally fall into two categories: dual-encoder models and fusion-encoder models. Pioneering works like CLIP (Radford et al., 2021) and
ALIGN (Jia et al., 2021) serve as exemplary dualencoder models, demonstrating exceptional performance in zero-shot classification tasks. These architectures also excel in image-text retrieval, as their features can be pre-computed and stored, allowing for efficient similarity score computation via dot-product operations. Fusion-encoder models (Lu et al., 2019; Tan and Bansal, 2019; Alayrac et al., 2022; Dou et al., 2022b; Li et al., 2022; Dou et al., 2022a; Xu et al., 2023a), such as ALBEF (Li et al., 2021a), mPLUG (Ye et al., 2023), X-VLM (Zeng et al., 2022), and VLMo (Bao et al., 2022), employ cross-attention mechanisms to enable deep interactions between visual and linguistic features. Other designs include concatenating features of each modality before feeding them into a Transformer (Chen et al., 2020; Li et al., 2020; Zhang et al., 2021; Gan et al., 2020; Li et al., 2021b; Cho et al., 2021; Huang et al., 2020, 2021; Shen et al., 2022; Kamath et al., 2021; Yang et al., 2022a; Wang et al., 2022b; Kim et al., 2021; Xue et al., 2021; Wang et al., 2022a, 2023; Xu et al., 2021). These models excel in complex tasks like closedset Visual Question Answering (VQA) and visual entailment.
BLIP-2 (Li et al., 2023a) is proposed to leverage pre-trained frozen ViTs and LLMs to alleviate the computation demands in the full end-to-end training. Under this learning paradigm, Zhang et al. (2023a) introduce visual-prompt transfer learning to mitigate visual-language re-alignment cost when using different LLMs. Jian et al. (2023) propose decoupled language pre-training to alleviate the intensive data requirement in BLIP-2.
More recently, VLM research (Yin et al., 2023; Fu et al., 2023) also explores visual instruction tuning (Xu et al., 2023b; Liu et al., 2023; Ye et al., 2023; Dai et al., 2023a), multi-modal in-context learning (Chen et al., 2022a; He et al., 2023; Shao et al., 2023; Gupta and Kembhavi, 2023; Yang et al., 2022b) and Chain-of-Thought prompting (Zheng et al., 2023; Lu et al., 2022), interleaved image-text generation (Aiello et al., 2024), and hallucination (Li et al., 2023c).
Visual Redundancy The concept of visual redundancy plays a pivotal role in the field of computer vision. It pertains to the phenomenon where semantic information is conveyed through multiple channels, often involving the use of various visual elements like shape and color to represent complex symbols. Recognizing the impact of this
redundancy on deep learning algorithms, there has been a shift towards devising methods to minimize it, thereby enhancing efficiency. For example, IARED2 (Pan et al., 2021) has an interpretable design to dynamically and gracefully remove redundant tokens. In Chen et al. (2022b), the redundancy of ViT is discussed on embedding, attention, and weight levels. Eventful Transformer (Dutson et al., 2023) discusses the temporal redundancy in the temporal dimension.

heading: B BLIP-2 Training Configurations
text: We re-train BLIP-2 from scratch using LAION115M, CCS-14M, MSCOCO from CapFilt dataset (Li et al., 2022). The models are trained on eight A100-80G GPUs. The training configurations are shown in Table 7.

heading: C Technical Details of Token Merging
text: In this section, we briefly summarize the technical designs of Token Merging (ToMe) (Bolya et al., 2023). Token Merging was initially proposed in Bolya et al. (2023) for accelerating ViTs without training. Whereas we re-purpose ToMe to condense the visual features used as language prompts in the LLM. Please refer to Sections 3 of Bolya et al. (2023) for full details.
Strategy. The token merging operations take place in between the attention and MLP blocks of each Transformer layer. ToMe reduces r tokens per layer. And over the L layers of a Transformer, it reduces a total of r × L tokens. In our experiments, we set r = 19 and our TomeFormer has 12 layers.
Token Similarity. The similarities of tokens are defined by the cosine similarity (dot product) of keys of tokens.
Bipartite Matching. The bipartite soft matching algorithm is summarized as follows: • Tokens are randomly partitioned into two sets A
and B. • Each token in set A is linked to the most similar
token in set B.
• Keep links with top r similarities. • Merge tokens with top r links. • Concatenate set A and B back into a single set.

heading: D Details on Our Implementations of BLIP-2 and VideoCoCa
text: Our reported results of our re-trained BLIP-2 are slightly worse than what was reported in Li et al. (2023a). There are mainly three reasons: • We are only able to download 104M image-text
pairs from the original 129M CapFlit dataset. • We intentionally exclude the VG dataset from
our pre-training procedure, as it mainly consists of localized captions. Thus, our re-trained BLIP2 is more challenging when evaluated on GQA, which is built on VG dataset. • The exact dataset weighting is unknown from the LAVIS project, we use a weighting that is based on the size of each pre-training dataset, i.e., CSS14M, LAION115M, MSCOCO. For video captioning in Table 3 and Table 4, because VideoCoCa is not open-sourced, we use a pre-trained model OpenCoCa released by mlfoundations.

heading: E Additional Token Merging Visualization in EVLGen
text: In this section, we provide additional examples of token merging visualization (before and after training) in EVLGen in Figure 5.
The visual features compressed via token merging in the TomeFormer exhibit semantic informativeness even prior to training. This inherent char-
acteristic facilitates EVLGen’s ability to converge quickly in an end-to-end training setup.

heading: F Ablation on Visual Encoders
text: One of the limitations of BLIP-2 is that it requires an extensive stage-1 pre-training for every different vision encoder. This prohibits practitioners from exploring stronger ViTs when they are available. EVLGen offers fast training of models, allowing for exploration of different ViTs as visual encoders.
We conduct an ablation experiment on two ViTs (CLIPL and EVA-ViTG) using 8× RTX-A6000 and the CCS-14M dataset for pre-training. The models are trained for 60,000 steps.
Shown in Table 8, EVLGen is robust to different visual encoders, and the stronger ViT leads to better results. This implies that while EVLGen also requires retraining for different ViTs, but the single-stage training and quick convergence allow it to benefit from a future release of the latest ViTs, given its capability of fast adaptation.

heading: G Ablations on TomeFormer
text: In this section, we provide experimental results in VQAv2, GQA, and OKVQA of EVLGen, by varying hyper-parameter r in TomeFormer. As we can see from Table 9, EVLGen is robust to the choice of r.

heading: Future Work
text: This implies that while EVLGen also requires retraining for different ViTs, but the single-stage training and quick convergence allow it to benefit from a future release of the latest ViTs, given its capability of fast adaptation