SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yiren Jian $^{1}$  Chongyang Gao $^{2}$  Soroush Vosoughi $^{1}$ $^{1}$ Dartmouth College  $^{2}$ Northwestern University
We present a novel methodology aimed at optimizing the application of frozen large language models (LLMs) for resource-intensive vision-language (VL) pre-training. The current paradigm uses visual features as prompts to guide language models, with a focus on determining the most relevant visual features for corresponding text. Our approach diverges by concentrating on the language component, specifically identifying the optimal prompts to align with visual features. We introduce the Prompt-Transformer (P-Former), a model that predicts these ideal prompts, which is trained exclusively on linguistic data, bypassing the need for image-text pairings. This strategy subtly bifurcates the end-to-end VL training process into an additional, separate stage. Our experiments reveal that our framework significantly enhances the performance of a robust image-to-text baseline (BLIP-2), and effectively narrows the performance gap between models trained with either 4M or 129M image-text pairs. Importantly, our framework is modality-agnostic and flexible in terms of architectural design, as validated by its successful application in a video learning task using varied base modules. The code will be made available at https://github.com/yiren-jian/BLIText.
The field of vision-language (VL) learning seeks to create AI systems that mimic human cognition, processing the world through multi-modal inputs. Core research areas in VL include visual-question-answering (VQA), image captioning, image-text retrieval, and visual reasoning. VL learning began with task-specific learning [3, 64] and has since progressed to large-scale image-text pre-training paired with task-specific fine-tuning [50]. Furthermore, contemporary studies have begun exploring the use of off-the-shelf frozen pre-trained large language models (LLMs) in VL models [2, 23, 34, 58], which have delivered impressive results in language generation tasks such as VQA and image captioning.
Present VL models utilizing frozen LLMs are characterized by shared design elements: visual encoders, visual-to-language modules, and frozen LLMs. Except for Flamingo [2], which employs a visual signal at each layer of the frozen LLM via gated cross-attention, the majority of works [6, 34, 41, 46, 58] feed aligned visual features as soft language prompts [29] into the frozen LLMs (see Figure 1 left). The models are then trained end-to-end with an image-conditioned language generation loss using large-scale image-text pairs. This conceptually simple and implementation-wise straightforward design has proven effective. BLIP-2 [34] demonstrates that decoupling the end-to-end training into two stages is crucial for state-of-the-art results. The second stage of training involves standard end-to-end learning, while the first stage of training of BLIP-2 utilizes a learnable module (called Query-Transformer/Q-Former) to selectively choose/query visual features relevant to the corresponding text. This reduces 256 features of an entire image to the 32 most relevant visual features that will be sent into the following parts of the model. Stage 1 of BLIP-2 can be viewed as a refined learnable version of early VL works [3, 38, 71] that use object detectors like Faster-RCNN
Figure 1: left: End-to-end training of X-to-language models (where X can be images, videos, or audio), in which aligned input features are provided as prompts to LLMs. Examples include Frozen [29] and ClipCap [46]. middle: "Forward-decoupled training" as demonstrated in BLIP-2 [34] and X-LLM [6]. For instance, in BLIP-2, the Q-Former is first trained to extract relevant features from the image encoder, and then the selected features are used as prompts for LLM for end-to-end learning. right: We propose "backward-decoupled training", which initially identifies the "reference prompt" for the LLM to generate the target text, followed by mapping input features to the "reference prompt".
[17] to select features from regions of objects (objects in images are likely to be mentioned and thus relevant to the accompanying text). We refer to this strategy as "forward-decoupling" since it uses a heuristic to learn/select which useful features are forward-passed into the subsequent model to mitigate challenges in the end-to-end optimization (shown in Figure 1 middle).
We provide a novel insight to mitigate the challenges in end-to-end optimization by introducing "backward-decoupling" during back-propagation. For a caption  $t$  (e.g., "a cat wearing sunglasses") from VL pre-training dataset  $\mathcal{D}_{\mathrm{VL}}$ , the optimizer first finds the optimal continuous prompt  $p$  for a fixed decoder LLM  $D_{\mathrm{language}}$ :  $p = \operatorname*{argmin}_p \mathcal{L}(D_{\mathrm{language}}(p), t)$ , before further back-propagating into the vision-to-language module (e.g., Q-Former in BLIP-2, or MLP in ClipCap) and the vision encoder (shown in Figure 1 right). We realize that the first stage, optimization of  $p$  given  $D_{\mathrm{language}}$  and  $t$ , is purely linguistic and does not restrict the learning text examples from  $\mathcal{D}_{\mathrm{VL}}$ . Thus, we propose to learn this part independently with the available sentence dataset.
While it's not feasible to learn individual prompts  $p$  for each sentence  $t$  due to the infinite number of possible sentences, we propose to parameterize prompt  $p$  by a Prompt-Transformer (P-Former):  $p = E_{\mathrm{P - Former}}(t)$ . This effectively transforms the learning of  $p$  given  $D_{\mathrm{language}}$  and  $t$  into learning  $E_{\mathrm{P - Former}}$  by  $\operatorname{argmin}_{E_{\mathrm{P - Former}}} \mathcal{L}(D_{\mathrm{language}}(E_{\mathrm{P - Former}}(t)), t)$ . Essentially, this is an autoencoder with the causal LLM  $D_{\mathrm{language}}$  as the decoder. As for P-Former, we use a bidirectional Transformer and the [CLS] representation as the bottleneck. Besides the reconstruction loss, we add a contrastive loss to discriminate each sample. Such a design makes  $E_{\mathrm{P - Former}}$  a semantic sentence embedding model like SimCSE [16] (i.e., semantically similar sentences have similar representations). Once  $E_{\mathrm{P - Former}}$  is learned,  $p = E_{\mathrm{P - Former}}(t)$  will be the "reference prompt" for LLM  $D_{\mathrm{language}}$  to generate  $t$  auto-regressively. The training overview and P-Former details are shown in Figure 2.
Returning to the VL pre-training, we add a complementary loss to minimize the distance between aligned visual features (being used as language prompts) and the "reference prompt" given by P-Former. We expect this to improve the VL pre-training in two ways: (1) We further decouple the VL learning into another stage, as Li et al. [34] suggest that multi-stage training is important to mitigate alignment challenges. (2) A semantically rich space is learned for aligned visual features/prompts by a SimCSE design for our P-Former trained with the unimodal sentence dataset (i.e., semantically similar images are encouraged to align to "reference prompts" with close representations).
Our proposed framework only adds a learning objective on tensors feeding into LLMs as prompts (a.k.a images/multi-modalities as foreign languages [6, 61]). Therefore, our method is agnostic to the input modalities, X encoders, and X-to-language modules (where X can be images, videos, and audio). This could be especially salient for videos, which have much less high-quality paired data [15] compared to image-text pairs. And because P-Former is only trained with the LLM, there is no need to re-train the P-Former for different modalities.
Figure 2: Overview of P-Former. left: The P-Former training resembles an autoencoder, with the bidirectional P-Former as the encoder and a causal LLM (frozen) as the decoder. The objective is to reconstruct input text auto-regressively. The [CLS] representation serves as sentence embeddings, which are projected back to the length of prompts. The contrastive loss at [CLS] mirrors the training of SimCSE [16]. A regularization vocabulary loss is utilized to encourage the prompts to be close to the vocabulary embeddings. right: Overview of bootstrapping VL pre-training with the trained P-Former. The alignment loss introduced by P-Former is agnostic to input modalities, encoders, and X-to-language modules (i.e., modules within the dashed box can be flexible). P-Former is only used during training and not during inference.
In our experiments, we take BLIP-2 as an example and show that our proposed framework improves this latest VL method by great margins in various benchmarks of VQA and image captioning. In Section 4.5, we demonstrate its effectiveness in other modalities (i.e., video) using different vision-to-language modules (i.e., plain Transformer over Q-Former).
We anticipate a growing body of future work within the paradigm of "images/multi-modalities as language prompts with frozen LLMs" due to its simplicity and effectiveness, as demonstrated by BLIP-2. For example, a concurrent work X-LLM [6] extends BLIP-2 from images to videos/speech with more advanced LLMs, augmenting BLIP-2's vision-to-language module Q-Former with Adapters. Because our proposed method is agnostic to input modalities, encoders, and X-to-language modules, it should seamlessly apply to future work within this paradigm of "images/multi-modalities as language prompts with frozen LLMs".
End-to-end vision-language learning Most end-to-end VL pre-training models can be broadly classified into two categories: dual-encoder and fusion-encoder models. Dual-encoder models employ two separate networks for vision and language, with the modality interaction computed via dot-product between visual and linguistic features (e.g., CLIP [50]). Due to the efficient computation of vector dot-product through feature caching, dual-encoder models are effective and highly efficient for image-text retrieval tasks. However, their performance in VQA, captioning, and visual reasoning tasks is limited due to the lack of fine-grained alignment between the two modalities.
Fusion-encoder models, such as ALBEF [32], VLMo [4], and CoCa [69], introduce new fusionTransformer layers to model deep interactions between the two modalities in addition to vision and language encoders. Common designs include concatenating visual and linguistic features before feeding them into a self-attentive Transformer [4, 7, 8, 14, 19, 20, 25, 27, 35, 37, 38, 54, 56, 59, 60, 61, 63, 66, 68, 71] or cross-attending vision and language encoders to compute fused features [2, 11, 12, 30, 32, 33, 40, 43, 44, 57, 65]. The vision encoder can range from simple linear embeddings [27] and ConvNets [19, 20, 25, 54, 60, 63, 68] to Transformers [4, 11, 12, 32, 33, 59, 61, 66], an offline pre-trained object detector like Faster-RCNN [7, 8, 14, 35, 37, 38, 56, 71], or an ensemble of models [42]. The language encoder can be initialized with a BERT-based [26] model or as part of a fusion-Transformer [4, 11, 12, 61, 70]. Most methods utilize three types of losses during pretraining: image-text contrastive (ITC) loss, image-text matching (ITM) loss, and mask language modeling (MLM) loss or language generation (ITG) loss. Fusion-encoder models have shown superior performance in VQA and captioning tasks, though they are less efficient in retrieval tasks. A thorough review of the recent advancements in VL pre-training can be found in Gan et al. [15].
Vision-language learning with frozen language models Large language models, pre-trained on large text corpora, show exceptional performance in language generation tasks. Therefore, incorporating these large frozen language models into VL models can be particularly beneficial for vision-language generation tasks, such as VQA and captioning. Flamingo [2] incorporates visual signals into each layer of a large frozen LLM using cross-attention. In contrast, Frozen [58] finetunes the image encoder to align visual features as soft prompts, which are input into the frozen language model. Recently, BLIP-2 [34] introduced an additional vision-to-language adaptation module Q-former (in conjunction with the frozen ViT [10] and an LLM), proposing a two-stage training process to mitigate the challenges in learning visual-language alignment. The first stage of BLIP-2 training optimizes the Q-former to extract beneficial visual features using ITC, ITM, and ITG losses. In the second stage of BLIP-2 training, all three modules (ViT, Q-former, and LLM) are trained end-to-end with only the parameters in Q-former updated. Despite being trained on 129M image-text pairs and with affordable computational resources, BLIP-2 demonstrates competitive results across multiple benchmarks. Finally, a concurrent work on visual chat-bot X-LLM [6] also adopts a similar architectural design philosophy to BLIP-2. Our proposed framework with P-Former can be applied to models under this paradigm that use soft prompts as the visual-language interface (e.g., Frozen, BLIP-2, X-LLM, etc).
Multi-modal auxiliary data learning Besides using off-the-shelf pre-trained vision encoders (ViT and Faster-RCNN [17, 51]) and language models, it is also interesting to explore how unimodal training can enhance multi-modal models. VLMo [4] demonstrated the benefits of conducting stage-wise pre-training with image-only and text-only data for their proposed model architecture. Li et al. [36] proposed using object tags from detectors as anchor points to bridge unpaired images and text, while Zhou et al. [74] formed pseudo-image-text pairs using an image-text retrieval alignment. Video-language models also leverage image-text pairs by repeating images to create static videos, constructing auxiliary paired datasets for pre-training. Jian et al. [22] showed that contrastive visual learning could also enhance contrastive sentence embeddings, a purely linguistic task. We also show how pure language training can enhance a multi-modal model.
Problem formulation Given an image-text dataset  $\{I, t\} \in \mathcal{D}_{\mathrm{VL}}$  and a unimodal language dataset composed purely of sentences  $\{t\} \in \mathcal{D}_{\mathrm{L}}$ , our objective is to optimize the pre-training of a vision-language (VL) model. This model consists of a pre-trained vision encoder  $E_{\mathrm{vision}}$ , a vision-to-language adaptation module  $\Theta_{\mathrm{V}\to \mathrm{L}}$ , and a frozen pre-trained language decoder  $D_{\mathrm{language}}$ . The goal is to minimize the image-conditioned language generation loss, given that the vision encoder  $E_{\mathrm{vision}}$  is also frozen:
$$
\underset {\Theta_ {\mathrm {V} \rightarrow \mathrm {L}}} {\operatorname {a r g m i n}} \mathcal {L} _ {\text {C r o s s E n t r o p y}} \left(D _ {\text {l a n g u a g e}} \left(\underset {\mathrm {V} \rightarrow \mathrm {L}} {\Theta} \left(E _ {\text {v i s i o n}} (I)\right)\right), t\right) \tag {1}
$$
As Li et al. [34] have noted, end-to-end optimization of Equation 1, visualized in Figure 1 left, can sometimes lead to catastrophic forgetting in LLMs.
3.1 Backward-decoupling and soft prompt pre-training (Training P-Former)
Let's denote the adapted visual features as  $p = \Theta_{\mathrm{V} \to \mathrm{L}}(E_{\mathrm{vision}}(I))$ , which serve as soft prompts for the LLM  $D_{\mathrm{language}}$ . During the optimization, Equation 1 can be decomposed into two parts, visualized in Figure 1 right:
$$
\underset {p} {\operatorname {a r g m i n}} \mathcal {L} _ {\text {C r o s s E n t r o p y}} \left(D _ {\text {l a n g u a g e}} (p), t\right) \tag {2}
$$
$$
\underset {\underset {\mathrm {v} \rightarrow \mathrm {L}} {\Theta}} {\operatorname {a r g m i n}} \mathcal {L} _ {\mathrm {M S E}} \left(\underset {\mathrm {v} \rightarrow \mathrm {L}} {\Theta} \left(E _ {\text {v i s i o n}} (I)\right), p\right) \tag {3}
$$
Equation 2 essentially asks "What is the optimal soft prompt  $p$  that enables the auto-regressive language model  $D_{\text{language}}$  to generate the sentence  $t$ ." Like all gradient-based deep learning models, depending on the training dataset, learning  $p$  given  $\{D_{\text{language}}, t\}$  could lead to different sub-optimal points<sup>1</sup> (a conventional deep learning problem is usually learning  $D_{\text{language}}$  given  $\{p, t\}$ ). End-to-end
<sup>1</sup>It can be easily verified that there exist multiple different soft prompts for an LLM to generate the same text auto-regressively. In an extreme example, a prompt with 32 tokens and a prompt with 16 tokens padded with 16 empty tokens (zeros vectors) can be both optimized for a LLM to generate the same text.
learning of Equation 1 can only use text  $t$  from image-text dataset  $\mathcal{D}_{\mathrm{VL}}$  to update its intermediate variable  $p$ . However, we observe that the learning of Equation 2 involves no image, thus allowing us to leverage abundantly available unimodal sentences in  $\mathcal{D}_{\mathrm{L}}$ .
Learning  $p$  for each  $t$  in  $\mathcal{D}_{\mathrm{L}}$  without constraint is intractable. Thus, we model  $p$  by a bidirectional Transformer  $E_{\mathrm{P-Former}}$  (named Prompt-Former, or P-Former)  $p = E_{\mathrm{P-Former}}(t)$ . Specifically, we use the output [CLS] hidden state of BERT as a compact representation for  $t$  and project it back to the token length of  $p$ . Equation 2 can thus be reformulated as:
$$
\underset {E _ {\mathrm {P} - \text {F o r m e r}}} {\operatorname {a r g m i n}} \mathcal {L} _ {\text {C r o s s E n t r o p y}} \left(D _ {\text {l a n g u a g e}} \left(E _ {\mathrm {P} - \text {F o r m e r}} (t)\right), t\right) \tag {4}
$$
In essence, Equation 4 describes the training of an autoencoder with the bidirectional P-Former  $E_{\mathrm{P - Former}}$  serving as the encoder, and the auto-regressive LLM  $D_{\mathrm{language}}$  as the decoder. To enhance our model, we include an unsupervised contrastive loss  $\mathcal{L}_{\mathrm{contrast}}$ , acting on the [CLS] representations of sentences to differentiate distinct instances. This loss, combined with our P-Former design, emulates the training of SimCSE [16], a semantic sentence embedding model (i.e., for semantically similar image-text pairs, the predicted prompts by P-Former should also be close). Furthermore, we introduce a regularization loss  $\mathcal{L}_{\mathrm{vocab}}$  to minimize the distance between each token in  $p$  and the closest embedding of the LLM's  $(D_{\mathrm{language}})$  vocabularies. The final objective becomes:
$$
\underset {E _ {\mathrm {P} - \text {F o r m e r}}} {\operatorname {a r g m i n}} \left(\mathcal {L} _ {\text {C r o s s E n t r o p y}} \left(D _ {\text {l a n g u a g e}} \left(E _ {\mathrm {P} - \text {F o r m e r}} (t)\right), t\right) + \mathcal {L} _ {\text {c o n t r a s t}} + \mathcal {L} _ {\text {v o c a b}}\right) \tag {5}
$$
A comprehensive view of the P-Former's architecture and learning losses is presented in Figure 2 left. We emphasize that the optimization of Equation 5 and P-Former training rely only on the text. Upon training the P-Former, Equation 3 can be reformulated as:
$$
\underset {\underset {\mathrm {V} \rightarrow \mathrm {L}} {\Theta}} {\operatorname {a r g m i n}} \mathcal {L} _ {\mathrm {M S E}} (\underset {\mathrm {V} \rightarrow \mathrm {L}} {\Theta} (E _ {\text {v i s i o n}} (I)), E _ {\mathrm {P - F o r m e r}} (t)) \equiv \underset {\underset {\mathrm {V} \rightarrow \mathrm {L}} {\Theta}} {\operatorname {a r g m i n}} \mathcal {L} _ {\text {a l i g n m e n t}} \tag {6}
$$
This new form, depicted in Fig 2 right, minimizes the distance between the aligned visual features and the prompts predicted by the trained P-Former, effectively aligning visual-linguistic representations.
3.2 Preliminary: BLIP-2 forward-decoupled training
While our proposed framework is flexible in regards to the specific architecture of  $\Theta_{\mathrm{V}\rightarrow \mathrm{L}}$  or the learning strategy deployed, for illustrative purposes, we employ BLIP-2 as a case study to demonstrate the applicability of our approach with state-of-the-art learning methods, owing to the strong performance and reproducibility of BLIP-2. In the context of BLIP-2,  $E_{\mathrm{vision}}$  is a ViT-g,  $\Theta_{\mathrm{V}\rightarrow \mathrm{L}}$  is referred to as Q-Former, and  $D_{\mathrm{language}}$  is a OPT2.7B. BLIP-2 proposes a two-stage pre-training process, with the initial stage involving the pre-training of  $\Theta_{\mathrm{V}\rightarrow \mathrm{L}}$  by:
$$
\underset {\Theta_ {\mathrm {V} \rightarrow \mathrm {L}}} {\operatorname {a r g m i n}} \operatorname {I T C} \left(\underset {\mathrm {V} \rightarrow \mathrm {L}} {\Theta} \left(E _ {\text {v i s i o n}} (I)\right), \underset {\mathrm {V} \rightarrow \mathrm {L}} {\Theta} (t)\right) + \operatorname {I T M} \left(\underset {\mathrm {V} \rightarrow \mathrm {L}} {\Theta} \left(E _ {\text {v i s i o n}} (I), t\right)\right) + \operatorname {I T G} \left(\underset {\mathrm {V} \rightarrow \mathrm {L}} {\Theta} \left(E _ {\text {v i s i o n}} (I), t\right)\right) \tag {7}
$$
This is followed by a second stage that involves end-to-end training of Equation 1. The terms ITC, ITM, and ITG in Equation 7 are utilized to guide the Q-Former  $\Theta_{\mathrm{V}\rightarrow \mathrm{L}}$  in extracting visually relevant features that correspond to the associated captions. We refer to this two-step process in BLIP-2 – first determining the visual features to extract and then incorporating the selected visual features into an end-to-end learning framework – as “forward-decoupled training.”
3.3 BLIP-2 forward-decoupled training with pre-trained P-Former
We now describe the full training pipeline when integrating our framework with BLIP-2. The first stage of training involves pre-training the Q-Former with Equation 7 ( $\mathcal{L}_{\mathrm{BLIP2-stage1}} \equiv \mathrm{ITC} + \mathrm{ITM} + \mathrm{ITG}$ ), supplemented with the alignment loss introduced by the P-Former, as defined in Equation 6:
$$
\mathcal {L} _ {\text {B L I P 2 - s t a g e 1}} + \omega_ {1} \times \mathcal {L} _ {\text {a l i g n m e n t}} \tag {8}
$$
Subsequently, the second stage of training, in line with our approach, involves BLIP-2's stage 2, which is the end-to-end training of Equation 1:  $\mathcal{L}_{\mathrm{BLIP2 - stage2}}\equiv \mathcal{L}(D_{\mathrm{language}}(\underset {\mathrm{V}\to \mathrm{L}}{\Theta}(E_{\mathrm{vision}}(I))),t))$  again enhanced with the alignment loss imparted by P-Former in Equation 6:
$$
\mathcal {L} _ {\text {B L I P 2 - s t a g e 2}} + \omega_ {2} \times \mathcal {L} _ {\text {a l i g n m e n t}} \tag {9}
$$
Figure 3 provides a schematic representation of the proposed integration of our framework and P-Former with BLIP-2.
Figure 3: An overview of our framework with BLIP-2, which employs a two-stage training process. The green components represent the alignment loss and modules added by us, which do not require gradients. The blue components are part of the original BLIP-2 structure. P-Former is solely utilized during training and is not required during the inference phase. Our proposed framework, with P-Former, can be seamlessly applied to any models that leverage prompts as the interface for multi-modal-language communications.
3.4 Model pre-training
Training dataset We employ a 12M subset of the pseudo-labeled [33] LAION dataset [52], using only the sentences, for pre-training the P-Former. For VL pre-training, we widely adapted academic setting (since academic institutions lack the resources available to industry researchers to use very large datasets) with approximately 4M image-text pairs. This set comprises the MSCOCO-80K [39], VG-100K [28], CC-3M [53], and SBU-1M [47] datasets.