SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pre-training models Our method is universally applicable to any vision-to-text models that utilize prompts as the interface. Owing to its impressive performance and reproducibility, we chose BLIP-2 as the base model for our primary experiments. Thus, for VL pre-training, the image encoder  $E_{\mathrm{vision}}$  is a ViT-g/14 from EVA-CLIP [13], the LLM decoder  $D_{\mathrm{language}}$  is an OPT2.7B [72], and the vision-to-language adaptation module is a Q-Former [34]. The Q-Former is initialized by BERT-base with 32 learnable queries. Our newly proposed P-Former is a base Transformer initialized by BERT-base.
Pre-training details The P-Former is trained on a system with  $3 \times$  RTX-A6000 (48GB) GPUs, using PyTorch [48]. We trained for five epochs with a linear warm-up and cosine scheduling, using a batch size of 384 ( $3 \times 128$ ), and AdamW as the optimizer. The initial learning rate is set to  $1e^{-4}$ , with a minimum learning rate of  $1e^{-5}$ , a warm-up learning rate of  $1e^{-6}$ , and 2000 warm-up steps. The VL pre-training is performed on a server equipped with  $8 \times$  RTX-A6000 (48GB) GPUs, using PyTorch. We developed the code based on the LAVIS project [31]. Predominantly, we employed the default configuration files provided by BLIP-2 of LAVIS. Both the stage 1 and stage 2 training ran for 10 epochs with linear warm-up and cosine scheduling, using a batch size of 1024 ( $8 \times 128$ ), and AdamW as the optimizer. The weight decay is set to 0.05, the initial learning rate is  $1e^{-4}$ , the minimum learning rate is  $1e^{-5}$ , and the warm-up learning rate is  $1e^{-6}$ . The key distinction is that stage 1 and stage 2 incorporate 5000 and 2000 warm-up steps, respectively. We set  $\omega_{1} = 10$  and  $\omega_{2} = 100$  while training BLIP-2 OPT $_{2.7B}$  with our P-Former.
Computational overhead considerations Incorporating  $\mathcal{L}_{\mathrm{alignment}}$  from Equation 8 and 9 introduces only a minimal computational overhead, attributable to an additional forward pass of the P-Former (Transformer-base) at each iteration. To illustrate, in our experimental settings using BLIP-2  $\mathrm{OPT}_{2.7\mathrm{B}}$ , the training time for stage 1 saw a modest increase from 2,669 minutes to 2,743 minutes. Similarly, for stage 2, the training time increased marginally from 1,862 minutes to 1,880 minutes. Thus, our methodology's overall computational burden remains manageable despite its enhancements (the only additional cost is pre-training of the P-Former, which only needs to be done once for an LLM).
Given the impressive performance and accessibility of the BLIP-2 model, coupled with its open-source nature, we primarily employ it as our base model. We aim to demonstrate how our proposed "backward-decoupling" strategy, along with the learned P-Former, can enhance the baselines across various image-to-text generation benchmarks. In Section 4.5, we further extend the applicability of our framework to other modalities, utilizing different base models.
4.1 Zero-shot image-to-text generation
We assess the performance of our pre-trained models on zero-shot VQA, encompassing GQA [21], OKVQA [45], and VQAv2 [18], without any task-specific fine-tuning. As per BLIP-2, we append text prompts to visual prompts prior to their processing by the frozen LLM. Both for the baseline BLIP-2 and our model, the text prompt used is "Question: Short answer": The results, as detailed in Table 1, suggest that our proposed framework significantly enhances the zero-shot VQA performance of BLIP-2 trained with 4M image-text pairs. Remarkably, the gap between the BLIP-2 trained with 4M and 129M image-text pairs is largely bridged by our method.
Models	#Pretrain
Image-Text	Pretrain
Uni-Text	VQAv2	OK-VQA
test	GQA
test-dev
val	test-dev
FewVLM [24]	9.2M	-	47.7	-	16.5	29.3
Frozen [58]	3M	-	29.6	-	5.9	-
VLKD [9]	3M	-	42.6	44.5	13.3	-
Flamingo3B [2]	1.8B	-	-	49.2	41.2	-
OPT2.7B BLIP-2 [34]	4M	-	46.8	45.6	25.9	30.5
OPT2.7B Ours	4M	✓	52.6	52.2	30.0	34.0
OPT2.7B BLIP-2† [34]	129M	-	53.5	52.3	31.7	34.6
4.2 Fine-tuned image captioning
We further fine-tune our pre-trained model for MSCOCO [39] image captioning, employing the text prompt "a photo of". Following BLIP-2, we fine-tune the model for 5 epochs using a batch size of  $1024(8\times 128)$ , AdamW with an initial learning rate of  $1e^{-5}$ , minimum learning rate of 0, warm-up learning rate of  $1e^{-8}$  and 1000 warm-up steps, with linear warm-up and cosine scheduling. We evaluate our fine-tuned model on the Karpathy test split of MSCOCO. Also, zero-shot transfer results on the NoCaps dataset [1] are reported. Shown in Table 2, our framework improves BLIP-2 in all metrics, with greater improvements in CIDEr compared to SPICE.
Table 1: Comparison with different methods on zero-shot VQA  ${}^{ \dagger  }$  : numbers taken from Li et al. [34].
Models	#Pretrain
Image-Text	NoCaps Zero-shot (validation set)	COCO Fine-tuned
Karpathy test
in-domain
C	S	near-domain
C	S	out-domain
C	S	overall
C	S	B@4	C
OSCAR [38]	4M	-	-	-	-	-	-	80.9	11.3	37.4	127.8
VinVL [71]	5.7M	103.1	14.2	96.1	13.8	88.3	12.1	95.5	13.5	38.2	129.3
BLIP [33]	129M	114.9	15.2	112.1	14.9	115.3	14.4	113.2	14.8	40.4	136.7
OFA [60]	20M	-	-	-	-	-	-	-	-	43.9	145.3
Flamingo [2]	1.8B	-	-	-	-	-	-	-	-	-	138.1
SimVLM [63]	1.8B	113.7	-	110.9	-	115.2	-	112.2	-	40.6	143.3
OPT2.7B BLIP-2 [34]	4M	115.3	15.0	111.0	14.6	112.5	14.0	111.9	14.5	41.8	140.4
OPT2.7B Ours	4M	118.3	15.3	114.7	14.9	114.1	14.1	115.1	14.8	42.3	141.8
OPT2.7B BLIP-2† [34]	129M	123.0	15.8	117.8	15.4	123.4	15.1	119.7	15.4	43.7	145.8
Table 2: Comparison with different captioning methods on NoCaps and COCO. All methods optimize the cross-entropy loss during fine-tuning. C: CIDEr, S: SPICE, B: BLEU. †: numbers taken from Li et al. [34].
4.3 Zero-shot image-text retrieval
While our proposed method primarily focuses on refining visual prompts for a frozen LLM to generate corresponding text, it may not prove as beneficial for image-text retrieval tasks (the ITC and ITM losses are principally responsible for these tasks). Nevertheless, we present results on zero-shot
MSCOCO, and zero-shot Flickr30K [49] image-to-text and text-to-image retrievals. We compare two models trained with  $\mathcal{L}_{\mathrm{BLIP2-stage1}}$  (ITC, ITM and ITG) and  $\mathcal{L}_{\mathrm{BLIP2-stage1}} + \mathcal{L}_{\mathrm{alignment}}$ , without any further task-specific fine-tuning. As expected, Table 3 reveals that the newly introduced  $\mathcal{L}_{\mathrm{alignment}}$  offers limited benefits for retrieval tasks. However, it does not negatively impact the performance.
Task	Pre-training objectives	Image → Text	Text → Image
R@1	R@5	R@1	R@5
Flickr30K	LBLIP2-stage1	94.3	99.8	82.9	95.5
LBLIP2-stage1 + Lalignment	93.7	99.7	83.0	95.8
MSCOCO	LBLIP2-stage1	78.4	93.8	60.5	83.0
LBLIP2-stage1 + Lalignment	78.7	94.5	60.4	82.8
4.4 Ablation studies
Impact of alignment loss weights We investigate the influence of  $\omega_{1}$  and  $\omega_{2}$  in Equation 8 and 9.  $\omega_{1} = 0$  and  $\omega_{2} = 0$  refers to BLIP-2, and  $\omega_{1} = 10$  and  $\omega_{2} = 100$  refers to our default configuration of BLIP-2 + P-Former. The alignment loss introduced by the P-Former proves beneficial in both stages of VL pre-training, as shown in Table 4.
Alternate language model In this section, we substitute the decoder-based  $\mathrm{OPT}_{2.7\mathrm{B}}$  model with an encoder-decoder-based FLAN-T5XL as the new LLM. The experiments are conducted with a limited computational budget on  $3\times$  RTX-A6000 and for 5 epochs on both stage 1 and stage 2. The results, displayed in Table 5, verify the effectiveness of our framework with another LLM.
Table 3: Comparison with different image-to-text and text-to-image retrieval methods.
ω1 ω2	VQAv2 val	OK-VQA test	GQA test-dev
0	0	46.8	25.9	30.5
10	0	51.4	29.2	32.8
0	100	50.4	28.7	33.0
10	100	52.6	30.0	34.0
Table 4: Ablations on  ${\omega }_{1}$  and  ${\omega }_{2}$  of Equation 8 and 9 (using OPT2.7B as LLMs).
Models	#Pretrain
Image-Text	VQAv2
val	OK-VQA
test	GQA
test-dev
Flan-T5XL BLIP-2‡	4M	48.3	31.5	36.4
Flan-T5XL ours‡	4M	54.9	35.7	40.3
Flan-T5XL BLIP-2†	129M	62.6	39.4	44.4
Table 5: Experiments using Flan-T5XL as LLM.  $\ddagger$ : using much less GPUs/epochs compared to Sec.4.1.  $\dagger$ : from Li et al. [34].
Effect of P-Former's pre-training sentence datasets In our primary experiments, we utilize a dataset containing 12M sentences for P-Former training. We investigate the impact of the pre-training sentence dataset for P-Former by re-training it with 4M sentences from our VL pre-training datasets. We then train BLIP-2 + P-Former and report zero-shot VQA results in Table 6. This examination underscores that both the implicit decoupling of BLIP-2's two-stage training into a 3-stage training (pre-training of P-Former), and the employment of additional unimodal sentences contribute to the improved outcomes.
P-Former	#Pretrain Sentences	VQAv2 val	OK-VQA test	GQA test-dev
×	-	46.8	25.9	30.5
✓	4M	51.7	28.2	32.3
✓	12M	52.6	30.0	34.0
Table 6: Ablations on sentence datasets used to train P-Former (using OPT2.7B as LLMs). The first row w/o P-Former is baseline BLIP-2.
BLEU-4	CIDEr	ROUGE
NITS-VC [55]	20.0	24.0	42.0
ORG-TRL [73]	32.1	49.7	48.9
LITG	29.3	56.6	48.2
LITG + Alignment	30.9	60.9	49.1
Table 7: VATEX English video captioning. Baseline is a sequential model (I3D  $\rightarrow$  Transformer  $\rightarrow$  OPT2.7B), training end-to-end with ITG.
4.5 Video captioning
Our framework is modality-agnostic with respect to the visual encoder and vision-to-language adaptor, making it applicable to other modalities, such as video. Consequently, we establish a video learning
pipeline, with the vision encoder set as a frozen I3D [5] video encoder, the vision-to-language adaptor as a Transformer-base, and the LLM decoder as the  $\mathrm{OPT}_{2.7\mathrm{B}}$  (also frozen). We then train this model on the VATEX [62] English training set and evaluate it on the validation set. This dataset contains 26K videos for training. The experiments are conducted on an RTX-A6000. Initially, we train the model solely using  $\mathcal{L}_{\mathrm{alignment}}$  for 10 epochs with the P-Former, followed by end-to-end learning with  $\mathcal{L}_{\mathrm{ITG}}$  for an additional 10 epochs.
Our baseline, represented in Table 7, is competitive with two well-established video captioning models: MITS-VC [55] and ORG-TRL [73]. It is noteworthy that the current state-of-the-art on this benchmark, VideoCoCa [67], is trained on 10M videos, in contrast to our model, which is trained on merely 26K videos. Furthermore, the integration of P-Former and  $\mathcal{L}_{\mathrm{alignment}}$  enhances the CIDEr score by 4.3 (from  $56.6 \rightarrow 60.9$ ).
Despite being a smaller-scale experiment without large-scale pre-training, we demonstrate that our learning framework can be generalized to another modality (i.e., video-learning), employing a different vision-language adaptor (i.e., a plain Transformer as opposed to a Q-Former).
Despite the modality-agnostic nature of P-Former and its ability to adapt to various encoders and vision-to-language adaptors, the unimodal language pre-training remains contingent on the choice of the frozen LLM. This necessitates re-training of the P-Former for different language decoders such as  $\mathrm{OPT}_{2.7\mathrm{B}}$  and FLAN-T5 $_{\mathrm{XL}}$ . Moreover, incorporating P-Former primarily enhances image-to-text generation tasks such as VQA and image captioning, while it falls short in improving image-text retrieval tasks. Finally, our methodology primarily assists in bootstrapping prompt-based VL pretraining, i.e., providing aligned visual features as soft prompts to LLMs. Its application to Flamingo remains unclear due to its cross-attention basis and non-open-source status. Nevertheless, given the simplicity of sequential modules of prompt-based models (as demonstrated by recent works such as Frozen, BLIP-2, X-LLM, etc.), we anticipate that our framework will be broadly applicable to most future work in the academic setting.
6 Conclusion and discussion
This paper introduces a novel optimization framework for enhancing vision-language models based on large, frozen LLMs. We observe that the end-to-end image-to-text pre-training can be backwardly decoupled: initially determining the "ideal prompt" that triggers the LLM to generate the target text (which can be trained in an unsupervised fashion), followed by the alignment of visual features to the prompt. To this end, we train a P-Former, which functions similarly to a semantic sentence embedding model, to predict prompts to which visual features should align. Experimental results demonstrate that including alignment loss (via P-Former) in the BLIP-2's framework significantly narrows the performance gap between models trained with 4M and 129M image-text pairs.
The key contributions of this paper are as follows:
- Contrary to most prior studies, which decouple VL pre-training into (1) learning which visual features to forward into language modules and (2) conducting end-to-end learning with the selected visual features (dubbed "forward-decoupling"), we propose an innovative perspective of VL decoupled-training from a backward viewpoint. We bifurcate the training into (1) determining the "ideal prompt" for the LLM to generate the text and (2) aligning visual features to that prompt.
- We introduce the P-Former, designed to predict the "ideal prompt," which is trained using a unimodal sentence dataset. This exhibits a novel application of unimodal training in enhancing multi-modal learning.
- Our proposed training framework substantially enhances a robust and recent baseline (BLIP-2), bridging the gap between models trained with 4M and 129M image-text pairs using accessible hardware ( $8 \times$  RTX-A6000 in less than 4 days). This considerably lowers the entry barriers to VL pre-training research and is expected to attract interest from groups with limited resources.
- The proposed framework generally applies to different modalities (images, videos, audio, etc.), vision encoders, and vision-to-language modules.
Lastly, we address the commonly asked questions by the reviewers in Appendix A, B, C, D, and E.
A Intuition and motivation behind P-Former
In this section, we summarize the intuitive explanation and motivation on why learning an ideal language prompt helps more than using visual ones as in the counterpart models.
- In our experiments with base models like BLIP-2, the architecture consists of three sequential components: (1) ViT, (2) VL-connector, and (3) LLM decoder. Since we use a frozen LLM for generation, optimizing closer to the LLM decoder becomes more pivotal for achieving optimal generation quality.
- The unique design of P-Former mirrors a sentence embedding model. This means the prompts predicted by the P-Former carry rich semantics. Therefore, during evaluations on unfamiliar images, the model boasts an improved generalization capability.
- BLIP2's studies indicate that direct end-to-end optimization of the sequential model can sometimes lead to catastrophic forgetting. Our approach adds an additional layer of complexity by decomposing the 2-stage BLIP2 training into 3 stages, further addressing this optimization challenge.
- For BLIP2, optimization of soft prompt is learned only using text from image-text pair, while our decoupled training allows for leveraging additional unimodal data for optimizing these soft prompts
B Justification for lack of ablation experiments w/ and w/o the P-Former
We purposely omitted experiments with and without the P-Former module (e.g., using a randomly initialized prompt  $p$ ). This omission was driven by the following considerations:
- Random initialization and learning without P-Former: Our initial approach was to directly learn from a randomly initialized prompt  $p$  without incorporating the P-Former. But, upon testing, we identified a significant challenge. For a smaller model variant like opt-2.7b, which possesses a hidden size of 2560, if we employ 32 tokens as soft prompts for an expansive dataset with 4M sentences, the resultant model would have to accommodate an overwhelming 327B parameters. This would have computational implications and potentially overfit, as learning from such a vast parameter space can dilute the essential semantic connections between various sentences.
- P-Former's efficiency in parameterization: The P-Former emerged as a solution to this parameter explosion problem. Instead of requiring a unique prompt for each data point in the dataset, the P-Former parameterizes the soft prompt  $p$  using a semantically-rich Transformer model. This design ensures that the total number of parameters remains fixed at 110M. The major advantage here is scalability. Whether working with a dataset of 4M, 12M, or even larger (e.g., 129M) or LMs with varying decoder sizes, the P-Former guarantees a consistent number of parameters, making the model more computationally efficient and preventing the loss of essential semantic relationships.
In brief, our experimentation strategy was driven by the dual goals of maintaining computational efficiency while preserving rich semantics. The challenges posed by direct learning from a randomly initialized prompt emphasized the need for a more structured approach, leading to the birth of the P-Former concept.
C Qualitative analysis on VQA
In this section, we incorporate qualitative comparisons for the GQA and OKVQA datasets, allowing us to offer more nuanced insights. In Figure C.1, we show several examples comparing our model's response with BLIP-2 and the ground truth (GT). From these examples, it can be observed that there is greater agreement with GT by our model.
It should be noted that the abstract semantic reasoning of our model can sometimes lead to artificially low scores for our model when looking for an exact match. For instance, asking "What occupation might he have?" with a picture of a person driving a forklift generates the answer "forklift operator" by our model, whereas the correct exact answer in the GT is stated as "forklift driver." Though these two answers are semantically identical, they will count as a wrong generation by our model.
D Additional discussion of the results
In this section, we provide more interpretation of the results. For instance, Table 1, in addition to underscoring the potency of our proposed framework in bolstering the zero-shot VQA performance,
Figure C.1: Qualitative analysis on success and failure cases of GQA and OKVQA.
particularly when trained with 4M image-text pairs, shows that our method manages to considerably close the performance gap between the BLIP-2 trained on different scales: 4M and 129M image-text pairs. This suggests that the effectiveness of our model is not solely a function of the amount of training data but rather the methodology itself. In essence, this table illustrates how strategic modifications and improvements can achieve comparable results to models trained on much larger datasets.
Similarly, Table 2 provides insights into our model's adaptability. When we fine-tune our pre-trained model for a specific task like MSCOCO image captioning, the results reflect an overall enhancement over BLIP-2 across all metrics. The pronounced improvement in CIDEr, as opposed to SPICE, indicates that our model is adept at recognizing and generating more relevant and contextually accurate descriptions of images. The additional data on zero-shot transfer to the NoCaps dataset further substantiates the model's capability to generalize and adapt to newer, unseen data.
Finally, while our model's primary design goal is to refine visual prompts for text generation, Table 3 offers a perspective on its performance in the retrieval domain. Even though the model was not specifically optimized for retrieval tasks, it is evident that the introduced modifications do not compromise the retrieval performance, attesting to the model's robustness.
E LLM-dependence of the stage-1 pre-training
It should be noted that our stage-1 pre-training needs to be repeated for each LLM, if  $\omega_{1} \neq 0$ . However, as evidenced in Table 4 ( $\omega_{1} = 0$  and  $\omega_{2} = 100$ ), our approach achieves competitive results even without the alignment loss in stage-1, focusing the alignment solely on stage-2.
F Acknowledgement
The authors would like to thank Mu Li and Yi Zhu from Boson AI for their YouTube and Bilibili videos on paper reading, which greatly inspired this work.