Bootstrapping Vision-Language Learning with Decoupled Language Pre-training
Authors: Yiren Jian
Chongyang Gao
Soroush Vosoughi
Venue: NeurIPS 2023 spotlight (2023)
Keywords: vision-language pretraining
multi-modal learning
uni-modal auxiliary learning

Abstract: We present a novel methodology aimed at optimizing the application of frozen large language models (LLMs) for resource-intensive vision-language (VL) pre-training. The current paradigm uses visual features as prompts to guide language models, with a focus on determining the most relevant visual features for corresponding text. Our approach diverges by concentrating on the language component, specifically identifying the optimal prompts to align with visual features. We introduce the Prompt-Transformer (P-Former), a model that predicts these ideal prompts, which is trained exclusively on linguistic data, bypassing the need for image-text pairings. This strategy subtly bifurcates the end-to-end VL training process into an additional, separate stage. Our experiments reveal that our framework significantly enhances the performance of a robust image-to-text baseline (BLIP-2), and effectively narrows the performance gap between models trained with either 4M or 129M image-text pairs. Importantly, our framework is modality-agnostic and flexible in terms of architectural design, as validated by its successful application in a video learning task using varied base modules. The code will be made available at https://github.com/yiren-jian/BLIText.

Bootstrapping Vision-Language Learning with
Decoupled Language Pre-training
Yiren Jian!
Chongyang Gao?
Soroush Vosoughi'
Dartmouth College
Northwesterrn University
Abstract
We present novel methodology aimed at optimizing the application of frozen large language models (LLMs) for resource-intensive vision-language (VL) pre-training. The current paradigm uses visual features as prompts to guide language models, with a focus on determining the most relevant visual features for corresponding text. Our approach diverges by concentrating on the language component, specifi- cally identifying the optimal prompts to align with visual features. We introduce the Trompt-Transforne which is trained exclusively on linguistic data, bypassing the need for image-text pairings. This strategy subtly bifurcates the end-to-end VI. training process into an additional, separate stage. Our experiments reveal that our framework signifi- cantly enhances the performance of a robust image-to-text baseline (BLIP-2), and effectively narrows the performance gap between models trained with either 4M or 129M image-text pairs. Importantly, our framework modality flexible in terms of architectural design, as validated by its successful application in a video learning task using varied base modules. The code will be made available at htttts:///gtthtt www. commyiren-aee jian/BLineeeeee BLIText
(P-Former), model that predicts these ideal prompts,
and
1 Introduction
The field of vision-language (VL.) learning seeks to create Al systems that mimic human cognition, processing the world through multi-modal inputs. Core research areas in VL include visual-question- answering (VQA), image captioning, image-text retrieval, and visual reasoning. VL learning began with task-specific learning 3 64 and has since progressed large-scale image-text pre-training paired with task- specifie fine-tuning tuning [50]. Furthermore, contemporary studies have begun exploring the use of off- thellf frozen pre- pre-trained large language models (LLMs) VL models [2 23 34 58). which have delivered impressive results in language generation tasks such as VQA and image captioning.
Present VI. models utilizing frozen L.MSs are characterized by shared design elements: visual encoders, -isual-to-language modules, and frozen L.Mss. Except for Flamingo (2, which employs
a visual signal at each layer of the frozen LLM via gated cross-attention. the majority of works
66 34 46 58 feed aligned visual features as soft language prompts [29 into the frozen LLMs (see Figure left). The models are then trained end-to-end with an image-conditioned language generation loss using aarge-scale image-tex text pairs. This conceptually simple and implementation. wise straightforward design has proven effective. BLIP-2 34] demonstrates that decoupling the end-to-end training into two stages is crucial for ofate-of-thh-ar results. The second stage of training involves standard end-to-end learning, while the first stage of training of BLIP-2 utilizes learnable module (called Query-Tran Trwnsformerr
to selectively choose/query visual features relevant to the corresponding text. This reduces 256 features of an entire image to the 32 most relevant visual features that will be sent into the following parts of the model. Stage of BL.P-2 can be viewed as
refined learnable version of early VL works 3 38 71 that use object detectors like FCNN
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

cat sanglanses
Selected Viual Features
"fund prompts"
Prompts
X-L Connector
Q: Forner/Onjent Detector
X-+i Connector
K Connector
sunglorees
wearing sunglosurs
Encoder
SB
Decoder
Decoder
Encoder
Selected Visual Festures
What best prompts"
I
MIN-
Inpue
X
End-to-end
Forward-decouplec Training
Backward-decoupled Training
Figure I: left: End-to-end training of X-to-language models (where X can be images, videos, or audio), in which aligned input features are provided as prompts to LLMs. Examples include Frozen [29] and ClipCap [46] middle: "Forward Prooward-ddeee deccupedddcouppee training" as demonstrated in BLIP-2 [34] and X-I.MM [6]. For instance, BL.I-22, the Q-Former is first trained to extract relevant features from the image encoder, and then the selected features are used as prompts for LLM for end-to-end learning. right: We propose boecwardd-ccouppg
training", which
initially identifies the "reference prompt" for the LLM to generate the target text, followed by mapping input features to the "reference prompt".
1r to select features from regions of objects (objects in images are likely be mentioned and thus relevant to the accompanying text). We refer to this strategy as "forward-decoupling since it uses heuristic learn/select which useful features are forward-passed into the subsequent model to mitigate challenges in the and-to-end optimization (shown in Figure] middle).
We provide novel insight to mitigate the challenges in end-to-end optimization by introducing "backward-decouplingg during pack-propagation. For a caption (e.g.. "a cal wearing sunglasses" ") from VL pre-training dataset DvL the optimizer first finds the optimal continuous prompt for fixed decoder LLM Danguugge p = argmin, L ianguage (p). t). before further back-propagating into the vision-to- saiio-to-languaee language module (c.g.. Q-Former in BLIP-2, MLP in ClipCap) and the vision encoder (shown in Figure right). We realize that the first stage, optimization of p given Danguuge and f, is purely linguistic and does not restrict the learning text examples from DvL. Thus, we propose to learn this part independently with the available sentence dataset.
While 's not feasible to learn individual prompts for each sentence due to the infinite number of possible sentences, we propose to parameterize prompt p by a Drompting-Transfrmmee
(P-Former): P Ep.Fommer()) This effectively transforms the learning of p given Diangugee and / into learning Ep Former by argmin Er.senz L( Daangugge (Ep. Former(()) (1)), 1). Essentially, this is an autoencoder with the causal LLM D Dianuugge as the decoder. As for P- P-Former, the [CLS] representation as the bottleneck. Besides the reconstruction loss, we add a contrastive loss to discriminate each sample. Such a design makes Ep-Fomer semantic sentence embedding model like SimCSE [6] (i.e., semantically similar sentences have similar representations) Once Ep-romer is learned, p Ep-romee (t) will be the "reference prompt" for LLM Dtunguage to generate t auto-regressively. The training overview and P-Former details are shown in Figure
bidirectional Transformer and
Returning to the VL pre-training, we add complementary loss to minimize the distance between aligned visual features (being used as language prompts) and the "reference prompt" given by Former. We expect this to improve the VI. pre-training in two ways: (1) We further decouple the VI. learning into another stage, as Li et al. [34] suggest that multi-stage training is important to mitigate alignment challenges. (2) A semantically rich space is learned for aligned visual (eatures/prompp) by a SimCSE design for our P-Former trained with the unimodal sentence dataset (i.c., semantically similar images are encouraged to align to "reference prompts" with close representations).
Our proposed framework only adds learning objective on tensors feeding into LLMs as prompts as foreign languages [6] 61). Therefore, our method is agnostic to
(a.k.a laarg////mliimmdddlle the input modalities, X encoders, and X-to-language modules (where X can images, videos, and audio). This could be especially salient for videos, which have much less high-quality paired data [15 compared to image-text pairs. And because P-Former only trained with the L.M,, there is no need to re-train the P-Former for different modalities.

---

LIM Lons
cst weaving songlasses
Vocab
LLM Decoder
PC
Allgrment
LLM Decoder
Contrast
X2L Connector
P-former
Transforer- based
Encodder
POOTPd (CLS] cat weaning sunglasses
Apput REXT
cat weoring sunglosses
Impor
Figure 2: Overview of P-Former. left: The P-Former training resembles an autoencoder, with the bidirectional P-Former as the encoder and a causal LLM (frozen) as the decoder. The objective is to reconstruct input text sute-reggessivelly The [CLS] representation serves as sentence embeddings, which are projected back to the length of prompts. The contrastive loss at [CI.S] mirrors the training of SimCSE [16]. A regularization vocabulary loss utilized to encourage the prompls to be close to vocabulary embeddings. right: Overview of bootstrapping VL pre-training with trained P-Former. The alignment loss introduced by P-Former agnostic to input modalities, encoders, and X-to-language modules (i.e., modules within the dashed box be flexible). P-Former only used during training and not during inference.
In our experiments, we take BLIP-2 an example and show that proposed framework improves this latest VL method by great margins in various benchmarks of VQA and image captioning. In Section 4.5 demonstrate its effectiveness in other modalities (i.e., video) using different vision-to-language modules (i.e., plain Transformer over Q-Former).
We anticipate growing body of future work within the paradigm of "images/multtimodaalll as language prompts with frozen LLMs" due to its simplicity and effectiveness, as demonstrated by BLIP- 2. For example, concurrent work X-LLM [6] extends BLIP-2 2 from images videos/speech with more advanced LLMs, augmenting BLIP-2's
module Q-Former with Adapters. Because our proposed method is agnostic to input modalities, encoders, and X-to-language modules, should seamlessly apply to future work within this paradigm of "imaaes/mulltimoddiiiees as language prompts with frozen L.MMs".
2
Related work
End-to-end vision-language learning Most end-to-end VL pre-training models can be broadly classified into two categories: dual-encoder and fusion-encoder models. Dual-encoder models employ two separate networks for vision and language, with the modality interaction computed via dot-product between visual and linguistic features (c.g.. CLIP (50). Due the efficient computation of vector dot-product through feature caching, dual-encoder models are effective and highly efficient for image-text retrieval tasks. However, their performance VQA, captioning, and visual reasoning tasks is limited due to the lack of fine- fine-grained aligmment between the two modalities.
Fusion-encoder models, such ALBEF 32 VLMo and CoCa 699 introduce new fusion- Transformer layers to model deep interactions between the two modalities in addition to vision and language encoders. Common designs include concatenating visual and linguistic features before feeding them into a self-attentive Transformer
20
25
27
35
37
38
54
[6] [59
60
61
63
66
68
71
cross-attending vision and language encoders compute fused features
11 30
32
33
40
43
57 65].
65). The vision encoder can range from simple linear embeddings
27 and ConvNets 19
20
25
54
60
63
68 to Transformers [4] [T 12
32
33
59
61
66) an
or ensemble model as part 61 77]. Most methods utilize three types of losses during pre- training: image-text contrastive (ITC) loss, image-text matching (ITM) loss, and mask language modeling (MLM) loss or language generation (ITG) loss. Fusion-encoder models have shown superior performance VQA and captioning tasks, though they are less efficient in retrieval tasks. thorough review of the recent advancements in VL pre-training can be found in Gan et al. [15].
offline pre-trained object detector like Faster-RCND
7 g 14, 35 37
38
56
of models [42]. The language encoder can be initialized with a BERT BERee-based
of Tiiton-Tranformer 4 I
12
3

---

Vision-language learning with frozen language models Large language models, pre-trained on large text corpora, show exceptional performance in language generation tasks. Therefore. incorporating these large frozen language models into VI. models can be particularly beneficial for vision-language generation tasks, such as VQA and captioning. Flamingo 2 incorporates visual signals into each layer of a large frozen LLM using cross-attention. In contrast, Frozen (58] fine- tunes the image encoder to align visual features as soft prompts, which are input into the frozen language model. Recently, BLIP-2 (34) introduced an additional vision-to-language adaptation module Q-former (in conjunction with the frozen ViT [0] and an LLM), proposing a two-stage training process to mitigate the challenges in learning visual- visual-language alignment. The first stage of BLIP-2 training optimizes the Q-former to extract beneficial visual features using ITC. ITM, and ITG losses. In the second stage of BLIP-2 training, all three modules (ViT, Q-former, and LLM) trained end-to-end with only the parameters in Q-former updated. Despite being trained on 129M image-text pairs and with affordable computational resources, BLIP-- demonstrates competitive results across multiple benchmarks. Finally, a concurrent work on visual chat-bot X-LLMM [6] also adopts similar architectural design philosophy to BLIP-2. Our proposed framework with P- P-Former can be applied to models under this paradigm that use soft prompts as the visual-language interface (e.g,, Frozen, BLIP-2, X-LLM, etc).
Multi-modal auxiliary data learning Besides using off shellf pre-trained vision encoders (ViT and Faster-RCNN 17 (II) and language models, it is also interesting to explore how unimodal training can enhance multi-modal models. VI.Mo 4 demonstrated the benefits of conducting stage-wise pre-training with image-only and text-only data for their proposed model architecture. Li et al. (36) proposed using object tags from detectors as anchor points to bridge unpaired images and text, while Zhou et al. [74] formed ppcudo-image-tex pairs using an image-text retrieval alignment.
Video-language models also leverage image-text pairs by repeating images to create static videos, constructing auxiliary paired datasets for pre-training. Jian et al. [2] showed that contrastive visual learning could also enhance contrastive sentence embeddings, a purely linguistic task. We also show how pure language training can enhance a multi-moda model.
3 Methodology
Problem formulation Given an image-text dataset { E DvL, and a unimodal language dataset composed purely of sentences {t} DL. our objective to optimize the pre-training of vision- language (VL) model. This model consists of pre-trained vision encoder Evision, vision-to-langeen lo-language adaptation module and a frozen pre-trained language decoder Dianguuge- The goal is to minimize V the image-conditioned language generation loss, given that the vision encoder Evision is also frozen:
argmin LcrossEntropy (Druggagge V (Friision (T)),,
As Li et al. 34 have noted, end-to-end optimization of Equation | visualized in Figure Left, can sometimes lead to catastrophic forgetting in LLMs.
3.1
Backward--decuupling and soft prompt pre-training (Training P-Former)
Let's denote the adapted visual features as p p= - (Frvssion (7)), which serve as soft prompts for the v-L.
can be decomposed into two parts, visualized in LLM D Dragguage During the optimization, Equation !
Figure right:
ar argmin Ccrosstinropyy (Drr (Dimguugge (p),t)
(2)
ar argmin LMSF
θ (Evision (T) (I)).p)
(3)
V-vl.
Equation 2 essentially asks "What is the optimal soft prompt p that enables the auto-regressive language model Diangaage to generate the sentence t." Like all gradient-baser deep learning models, depending on the training dataset, learning p given (Duunguage: 1) could lead to different sub-optimal points] (a conventional deep learning problem is usually learning Drangugge given {p. t)). End-to-end
It can be casily verified that there exist multiple different soft prompts for an LLM to generate the same text auto-regressively. In an extreme example, prompt with 32 takens and a prompt with 16 tokens padded with 16 empty tokens (zeros vectors) be both optimized for LLM to generate the same text.

---

learning of Equation 1 can only use text t from image-text dataset DvL to update its intermediate variable p. However, we observe that the learning of Equation 2 iinvolves no image, thus allowing us to leverage abundantly available unimodal sentences in Dr.
Learning/ p for each & in Dr. without constraint is intractable. Thus, we model p by bidirectional Transformer Fo-Fomer (named Prompt-Former, or P-Former) Fpp-orerrt) (t). Specifically, we use the oulput [CLS] hidden state of BERT a compact representation for and project back the token length of p. Equation 2 can thus be reformulated as:
argmin CaussEEnropy
(D language Ep Former (t)). tt)
Ep.iomer
In essence, Equation 4 describes the training of an autoencoder with the bidirectional P-Former Ep-Forer serving as the encoder, and the auto-regressive LLM D Dianuugg the decoder. To enhance our model, we include an unsupervised contrastive loss Lcuntrast acting on the [CLS] representations of sentences to differentiate distinet instances. This loss, combined with our P-Former design, emulates the training of SimCSE [16], semantic sentence embedding model (i.e., for semantically similar image-text pairs, the predicted prompts by P-Former should also be close). Furthermore, we introduce a regularization loss Lvaaab to minimize the distance between each token in p and the closest embedding of the LLM's (Dinngunge) vocabularies. The final objective becomes:
argmin CCrossEnmrppy (Danguage (Ep-Fommer(t)) t) + Ceontss + Lvocab))
(5)
Fip-Fonner
comprehensive view of the P-Former's architecture and learning losses is presented Figure 2 left.
We emphasize that the optimization of Equation and P-Former training rely only on the text. Upon training the P-Former, Equation 3 can be reformulated as:
argmin LMSE
(ovision())) E.p Former (t)) argmin Lalignment
9
This new form, depicted in Fig right, minimizes the distance between the aligned visual features and the prompts predicted by the trained P-Former, effectively aligning lisual-linguistic representations.
3.2 Preliminary: BLIP-2 forward-decoupler training
While our proposed framework flexible in regards to the specific architecture of O the learning strategy deployed, for illustrative purposes, we employ BLIP-2
V-1.
study demonstrate the
applicability of our approach with state-of-the-ar learning methods, owing to the strong performance
and reproducibility of BLIP-2. In the context of BLIP-2, Evision is ViT-g, e is referred to as
V-+
Q-Former, and Diangunge is OPB BLIP-2 proposes two-stage pre-training process, with the initial stage involving the pre-training of θ by:
V-L
argmin ITC( θ V-L
(Frvision (T)) θ (1) ( V-L
ITM( (Frvision (T), + PTG θ (F), (T), (7)
V-L
This is followed by a second stage that involves end-to-end training of Equation] [ The terms ITC, ITM, and ITG in Equation 7 are utilized to guide the Q-Former θ in extracting visually relevant
V-L
features that correspond to the associated captions. We refer to this two-step process in BLIP-2 - first determining the visual features to extract and then incorporating the selected visual features into an end-to-end learning framework as fforward-decouple training."
3.3 BLIP-2 foroardd training with pre-trained P-Former
We now describe the full training pipeline when integrating our framework with BLIP-2. The first
stage of training involves pre-training the Q-Former with Equation 7 (CBlup2-stgee ITC + ITM ITG), supplemented with the alignment loss introduced by the P-Former, as defined in Equation 6
LBLI stagel w1 X Laligmment
(8)
Subsequently, the second stage of training, line with our approach, involves BLIP-2's stage 2, which is the end-to-end training of Equation [ LRI.PP-sttge22 = L C(Diamgunge ( ② (Evision (T))), ()), V-51.
again enhanced with the alignment loss imparted by P-Former in Equation 6
LBLIP2-sage2 W2 X Caligmment
Figure 3 provides a schematic representation of the proposed integration of our framework and P P-Former with BLIP-2.
5

---

Stage 1
Loss
Alignment
Loss
Image Encoder
QtFormer
Former
P-Former (hetnar))
Input Image
DM..0m
KISSA
Learaalle Quiies
wearing
sunglosses
Stage
00
Project
A cat weoring sunglosses
Image Encoder
Q+Former
LLM Decoder
Input Image
Po..o Learnalle Quiies
Alignmant Loss
[CLS] cot wearing sunglasses
P-Former
(honwwa))
Figure 3: overview our framework with BLIP-2, which employs a two-stage training process. The green components represent the alignment loss and modulcs added by us, which do not require gradicnts. The blue components are part of the original BLIP-2 structure. P-Former is solely utilized during training and is not required during the inference phase. Our proposed framework, with P- F-Fomer, be seamlessly applied any models that leverage prompts the interface for cilii-oodllllnnnnnee language communications.
3.4 Model pre-training
Training dataset We employ 12M subset the pseudo-labeled [33] LAION dataset [52], using only the sentences, for pre-training the P-Former. For VL pre-training, we widely adapted academic setting (since academic institutions lack the resources available to industry researchers to use very large datasets) with approximately 4M image-text pairs. This set comprises the MSCOCO-8OK (39]. VG-100K (28] CC-3M [53], and SBU-IM [47] datasets.
Pre-training models Our method is universally applicable to any vision-to-text models that utilize prompts as the interface. Owing its impressive performance and reproducibility. we chose BLIP--2 as the base model for our primary experiments. Thus, for VL pre- traininiing. the image encoder Evision a V from EVA-CLIP 13,, LLM decoder Dhangunge an OPT2.7R [72), and the vision-to- language adaptation module is a Q-Former (34). The Q-Former is initialized by BERT-base with 32 learnable queries. Our newly proposed P-Former a base Transformer initialized by BERT-base.
Pre-training details The P-Former is trained on a system with x RTX-A6000 (48GB) GPUs, using PyTorch [88). We trained for five epochs with a linear warm-up and cosine scheduling, using batch size of 384 (3 128), and AdamW as the optimizer. The initial learning rate is set to le with minimum learning rate of 15-5 warm-up learning rate of 16-6, and 2000 warm-up steps. The VL pre- is performed on server equipped with 8 x RTX-A6000 (48GB) GPUs, using Py PoTrcc We developed the code based on the LAVIS project [I. Predominantly, we employed the default configuration files provided by BLIP-2 of LAVIS. Both the stage and stage 2 training ran for 10 epochs with linear warm-up and cosine scheduling, using batch size of 1024 (8 x 128), and AdamW as the optimizer. The weight decay is set to 0.05, the initial learning rate is 1e-1, the minimum learning rate 1e -5 and the warm-up learning rate is 1e 6. The key distinction that stage 1 and stage 2 incorporate 5000 and 2000 warm-up steps, respectively. We set W1 10 and w2 = 100 while training BLIP-2 OPT2.B with our P-Former.
Computational overhead considerations Incorporating Laligmment from Equation 8 and introduces only a minimal computational overhead, attributable an additional forward pass of the P-Former (Transformer-base at each iteration. To illustrate, in our experimental settings using BLIP-2 OPT278. training time for stage saw modest increase from 2,669 minutes to 2,743 minutes. Similarly, for stage 2, the training time increased marginally from 1 1,862 minutes to 1,880 minutes. Thus, our methodology's 's overall computational burden remains manageable despite its enhancements (the only additional cost is pre-training of the P-Former, which only needs to be done once for LLM).

---

Experiments
Given the impressive performance and accessibility of the BLIP-2 model, coupled with its open- source nature, we primarily employ it as our base model. We aim to demonstrate how our proposed breckward-decopliig
strategy, along with the learned P-Former, can enhance the baselines across various image-to-tedt generation benchmarks. In Section [45] we further extend the applicability of our framework to other modalities, utilizing different base models.
4.1 Zero-shot image-to-text generation
We assess the performance of our pre-trained models on zero-shot VQA, encompassing GQA 21 OK OKQQA 45 and VQAv2 [8], without any task-specific fine-tuning. As per BLIP-2, we append text prompts to visual prompts prior to their processing by the frozen LLM. Both for the baseline BLIP-2 11P-2aam and our model, the text prompt used is "Question: Short answer:". The results, as detailed in T suggest that our proposed framework significantly enhances the zero-shot VQA performance of BLIP-2 trained with 4M image-tett pairs. Remarkably, the gap between the BLIP-2 trained with 4M and 129M image-text pairs is largely bridged by our method.
#Pretrain Image-Text
Pretrain Uni-Text
VQAv2 test-dev
OK-VQA test
GQA test-dev
Models
Few VLM 24
9.2M
47.7
16.5
29.3
Frozen 588
3M
29.6
5,9
VLKD 9
3M
42.6
44.5
13.3
Flamingo3B
1.B
49.2
41.2
OPT BLIP-2
34
4M
46.8
45.6
25.9
30.5
OPT Ours
4M
52.6
52.2
30.0
34.0
2PT BLIP-2
34
129M
53.5
52.3
31.7
34.6
Table
Comparison with different methods on zero-shot VQA
numbers taken from Li al.
4.2
Fine-tuned image captioning
We further fine-tune our pre-trained model for MSCOCO 39 image captioning, employing the text prompt "a photo of Following BLIP-2, we fine-tune the model for epochs using batch size of 1024 (8 x 128), A with an initial learning rate of 1 minimum learning rate of 0, warm-up learning rate of 1e and 1000 warm-up steps, with linear warm-up and cosine scheduling. We evaluate our fine-tuned model on the Karpathy test split of MSCOCO. Also, zero-shot transfer results on the NoCaps dataset []] are reported. Shown in Table 2 our framework improves BLIP-2 in all metries, with greater improvements in CIDEr compared to SPICE.
NoCaps Zero- shot (validation set) near-domain
COCO Fine-tuned Karpathy test
Models
#Pretrain Image-Text
in-domain S
out-domain
overall
S
C
S
C
S
B@4
C
OSCAR 38
4M
80.9
11.3
37.4
127.8
VinVL
5.7M
103.1
14.2
96.1
13.8
88.3
12.1
95.5
13.5
38.2
129.3
BLIP A3
129M
114.9
15.2
112.1
14.9
115.3
14.4
113.2
14.8
40.4
136.7
OFA 60
20M
43.9
145.3
Flamingo
1,8B
138.1
SimVLM
53
1.8B
113.7
110.9
115.2
112.2
40.6
143.3
OPT BLIP-2
L21
4M
115.3
15.0
111.0
14.6
112.5
14.0
111.9
14.5
41.8
140.4
0PT Ours
4M
118.3
15.3
114.7
14.9
114.1
14.1
115.1
14.8
42.3
141.8
27T BLIP-21
129M
123,0
15.8
117.8
15.4
123.4
15.1
119.7
43.7
145.8
Table Comparison with different captioning methods on NoCaps and COCO. All methods optimize cross-entropy loss during fine-tuning. C: CIDEr, S: SPICE, B: BLEU, 1; numbers taken from Li al.
34
4.3 Zero-shot image-text retrieval
While our proposed method primarily focuses on refining visual prompts for a frozen LLM to generate corresponding text, may not prove as beneficial for image-text retrieval tasks (the ITC and ITM losses principally responsible for these tasks). Nevertheless, present results zero-shot
7

---

image-to-text and text-to-ine -inage retrievals. We compare two models trained with LBLJP2-stnpe stngel (ITC, ITM and ITG) and LBIPP + Lligmments without any further task-specific fine-tuning. As expected, Table 3 reveals that the newly introduced Laligmment offers limited benefits for retrieval tasks. However, does not negatively impact the performance.
MSCOCO, and zero-shot Flickr30K 14
Pre-training objectives
Image R@ 1
Text R@S
Text R@
Image R@5
Task
Flickr30K
Suuir--tuell Iariez-ngee + Eiligement
94.3
99.8
82.9
95.5
93.7
99.7
83.0
95.8
Caripz-nnel CuLiv2-sagel Calignment
78.4 78.7
93.8
60.5
83.0
MSCOCO
94.5
60.4
82.8
Table Comparison with different image-to-text lext-to-image retrieval methods.
4.4 Ablation studies
Impact of alignment loss weights We investigate the influence of w1 and we in Equation S and 9 Wi = 0 and W2 0 refers to BLIP-2, and w1 = 10 and W2 100 refers to our default configuration of BLIP-2 + P-Former. The alignment loss introduced by the P-Former proves beneficial in both stages of VL pre-training, shown in Table 4
Alternate language model In this section, we substitute the decoder- based OPT2.78 model with an deleeeddddedde--bbbabbe FLAN-T5xl the new LLM. The experiments are conducted with limited computational budget on RTX-A6000 and for 5 epochs on both stage 1 and stage The results, verify the effectiveness of our framework with another L.LMM
displayed in Table
VQAv2
OK-VQA test
GQA
#Pretrain Image-Tex:
VQAv2 OK- VQA GQA
W1 W2
val
test-dev
Models
val
test
test-dev
46.8
25.9
30.5
Flan-T5x: BLIP-2+ 4M
48.3
31.5
36.4
100
50.4
51.4
28.7
29.2
32.8
33.0
Flan-T5xi ourst 4M
54.9
35.7
40.3
10 100
52.6
30.0
34.0
Flan-T5xx BLIP-- 129M
62.6
39.4
44.4
Table
Ablations on wi wa Equa-
Table 5:
Experiments using Flan- 15xt
LLM. 4; using much
tion and (using OPT2m LLMs).
less GPUs/epochs compared to See 4.1
T: from Li et al. 34]
Effect of P-Former's pre-training sentence datasets In our primary experiments, we utilize dataset containing 12M sentences for P-Former training. We investigate the impact of the pre-training sentence dataset for P-Former by re-training with 4M sentences from our VL pre-training datasets. We then train BLIP-2 P-Former and report zero-shot VQA results in Table 6 This examination underscores that both the implicit decoupling of BLIP-2's two-stage training into 3-stage training (pre-training of P- P-Fomeer), and the employment of additional unimodal sentences contribute to the improved outcomes.
#Pretrain VQAv2 OK-VQA GQA test-dev
|BLEU- CIDEr ROUGE
P-Former Sentences
val
test
NITS-VC 55 ORG-TRL 73
20.0
24.0
42,0
46.8
25.9
30.5
32.1
49.7
48.9
4M
51.7
28.2
32.3
LIG
29.3
56.6
48.2
12M
52.6
30.0
34.0
LIG Latignment
30.9
60.9
49.1
Ablations on sentence datasets used to train P-Former (using OPT27 as LLMs), The first row w/o P-Former baseline BLIP-2.
Table 6:
Table VATEX English video captioning. Baseline sequential model (13D - Transformer - OPT22)).
training end-to-end with ITG.
4.5
Video captioning
Our framework mogality-agnostic with respect to the visual encoder and lisigagto adaptor, making it applicable to other modalities, such as video. Consequently, we establish video learning

---

pipeline, with the vision encoder set as frozen 13D [5] video encoder, the vision-to-languun -o-nague adaptor as Transformer-base,, and the LLM decoder as the 2PT2.7 (also frozen). We then train this model on the VATEX [6] English training set and evaluate it on the validation set. This dataset contains 26K videos for training. The experiments are conducted on an RTX-A6000. Initially, we train the model solely using Aaligmment for 10 epochs with the P-Former, followed by end-to-end learning with CIGG for an additional 10 epochs.
Our baseline, represented in Table 7 is competitive with two well-established video captioning models: MITS-VC [55] and ORG-TRL 773.. It is noteworthy that the current state-of-the-ee on this benchmark, VideoCoCa [7], is trained on 10M videos, in contrast to our model, which is trained on merely 26K videos. Furthermore, the integration of P-Former and Calignment enhances the CIDEr score by 4.3 (from 56.6 + 60.9).
Despite being a smaller-scale experiment without large-scale pre-training, demonstrate that our learning framework can be generalized to another modality (i.e., video-learning), employing a different vision- languae adaptor (i.e., plain Transformer as opposed a Q-Former).
5
Limitations
Despite the modality-agnos nature of P-Former and its ability to adapt various encoders and vision-to-lan vision-to-languae adaptors, the unimodal language pre-training remains contingent on the choice of the frozen LLM. This necessitates re-training of the P-Former for different language decoders such as 277 and FLAN-T5xL- Moreover, incorporating P-Former primarily enhances image-to-text generation tasks such as VQA and image captioning. while falls short in improving image-text retrieval tasks. Finally, training, i.e., providing aligned visual features as soft prompts to LLMs. Its application to Flamingo remains unclear due its cross -ots-tattn basis and non-open-soure source status. Nevertheless, given the simplicity of sequential modules of cooppt-baser models (as demonstrated by recent works such as Frozen, BLIP-2, X-LLM, etc.), we anticipate that our framework will be broadly applicable to most future work in the academic setting.
methodology primarily assists in bootstrapping prompt-based VL pre-
6 Conclusion and discussion
This paper introduces a novel optimization framework for enhancing vision-language models based on large, frozen LLMs. We observe that the end-to-end image-to-text pre-training can be backwardly decoupled: initially determining the "ideal prompt" that triggers the L.MM to generate the target text (which can trained in an unsupervised fashion), followed by the alignment of visual features to the prompt. To this end, we train P-Former, which functions similarly to a semantic sentence embedding model, to predict prompts to which visual features should align. Experimental results demonstrate that including alignment loss (via P- P-Former) in the BLIP-2's framework significantly narrows the performance gap between models trained with 4M and 129M image-text pairs.
The key contributions of this paper are as follows:
Contrary to most prior studies, which decouple VL pre-training into (1) learning which visual features to forward into language modules and (2) conducting end-to-end learning with the selected visual features (dubhed "forwardddecouppling"" we propose innovative perspective of VI. decoupled-trainingg from a backward viewpoint. We bifurcate the training into (1) determining the "ideal prompt" for the LLM to generate the text and (2) aligning visual features to that prompt. We introduce the P-Former, designed to predict the "ideal prompt," which is trained using unimodal sentence dataset. This exhibits novel application of unimodal training in enhancing multi-modal learning.
Our proposed training framework substantially enhances robust and recent baseline (BLIP-2), bridging the gap between models trained with 4M and 129M image-text pairs using accessible hardware (8 RTX-A6000 less than 4 days). This considerably lowers the entry barriers to VL pre-training research and is expected to attract interest from groups with limited resources.
The proposed framework generally applies to different modalities (images, videos, audio, etc.), vision encoders, and sasion-tollaguage modules.
Lastly, we address the commonly asked questions by the reviewers Appendix A
B
and E

---

References
[1] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948-8957, 2019.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in Neural Information Processing Systems, 35: 237766, 2022.
[3] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086, 2018.
[4] Hangbo Bao, Wenhui Wang. Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with toixstturroo-modlltt--yoooo 32897-32912, 2022.
Advances in Neural Information Processing Systems, 35:
[5] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6299-6308, 6308, 2017.
[6] Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-IIm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages, 2023.
[7] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020.
[8] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-languaal tasks via text generation. In International Conference on Machine Learning, pages 1931-1942. PMLR, 2021.
[9] Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation clip via vision-language knowledge distillation. In Findings of the Association for Computational Linguistics: ACI 2022, pages 2383-2395, 2022.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[11] Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, Jianfeng Gao, and Lijuan Wang. Coarse-to-fine vision-language pre-training with fusion in the backbone. In Advances in Neural Information Processing Systems, 2022.
[12] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng. et al. An empirical study of training end-to-end langaaa transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18866-181766 2022.
[13] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636 2022.
[14] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-languae representation learning. Advances in Neural Information Processing Systems, 33:66 10666666600 16-6628, 2020.
10

---

[15] Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision- language pre-training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, 14(3-4); 19:163-3522 2022.
[16] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods Natural Language Processing (EMNLP), 2021.
[17] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[18] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1904-6913, 2017.
[19] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint Siti::20040008444 2020.
[20] Zhicheng Huang. Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12976-12985, 2021.
[21] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[22] Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Non-linguistic supervision for contrastive learning of sentence embeddings. In Advances in Neural Information Processing Systems, 2022.
[23] Yiren Jian, Tingkai Liu, Yunzhe Tao, Soroush Vosoughi, and Hongxia Yang. Simvlg: Simple and efficient pretraining of visual language generative models. arXiv preprint arXiv22310.0391, 2023.
[24] Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision- languae models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2763-2775, 2022.
[25] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulate detection for end-to-end multi-modal understanding, In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[26] Jacob Devlin Ming-Was Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186, 2019.
[27] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference Machine Learning, pages 5583-5594. PMLR, 2021.
[28] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[29] Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.

---

[30] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang. Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, and Luo Si. mPLUG: Effective and efficient vision-language learning by cross-modal skip-connections. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7241-7259 Association for Computational Linguistics, December 2022.
[31] Dongxu Li, Junnan Li, Hung Le, Guangsen Wang. Silvio Savarese, and Steven C. H. Hoi. Lavis: library for language- vision intelligence, 2022.
[32] Junnan Li. Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances neural information processing systems, 34:9694 9705, 2021.
[33] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified dision-langual understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[34] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2:
Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. 2023.
[35] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXXiv/908... 0955503557 2019.
[36] Liunian Harold Li, Haoxuan You, Zhecan Wang. Alireza Zareian, Shih-Fu Chang, and Kai-Wei Chang. Unsupervised vision-ande lainn-a pre-training without parallel images and captions. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5339-5350, 2021.
[37] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and generation via moosl contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2592-2607, 2021.
[38] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for lasion-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
[39] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[40] Haogeng Liu, Qihang Fan, Tingkai Liu, Linjie Yang. Yunzhe Tao, Huaibo Huang, Ran He, and Hongxia Yang. Video-teller: Enhancing cross-modal generation with fusion and decoupling. arXiv preprint 2..Xi:2330000999) 2023.
[41] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 230.::2300004855 2023.
[42] Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Pris- mer: A vision-language model with an ensemble of experts. arXiv preprint arXiv:2303.02500 2023.
[43] Tingkai Liu, Yunzhe Tao, Haogeng Liu, Qihang Fan, Ding Zhou, Huaibo Huang, Ran He, and Hongxia Yang. Video-csr: Complex video digest creation for risual-language models. arXiv preprint arXiv:2310.0506/ 2023.
[44] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for lisoon-and-language tasks. Advances in neural information processing systems, 32, 2019.
12

---

[45] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[46] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09755 2021.
[47] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24, 201 1.
[48] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[49] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana I Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[51] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[52] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[53] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 2556-2565, 2018
[54] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Chang, Zhewei Yao, and Kurt Keutzer. How much can CLIP benefit vision-and language tasks? In International Conference on Learning Representations, 2022.
[55] Alok Singh, Thoudam Doren Singh, and Sivaji Bandyopadhyay. Nits-ve system for vatex video captioning challenge 2020. arXiv preprint arXiv:2006. 04058, 2020.
[56] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. V1-bert: Pre- training of generic iisual-linguise representations. In International Conference on Learning Representations, 2020.
[57] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference Empirical Methods in Natural Lun- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100-5111, 2019.
[58] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212. 2021.
[59] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: generative image-to-text transformer for vision and language. Transactions on Machine Learning Research, 2022. ISSN 2835-8856,
13

---

[60] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai. Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple requence-to-sqquence learning framework. In International Conference on Machine Learning, pages 23340.23340. PMLR, 2022.
[61] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[62] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for aideo-and-languagee research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581-4591, 2019.
[63] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. SimVLM: Simple visual language model pretraining with weak supervision. In International Conference on Learning Representations, 2022.
[64] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[65] Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, and Nan Duan. Bridgetower: Building bridges between encoders in vision-language representation learning. arXiv preprint arXiv:2206.0880 086577 2022.
[66] Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing inter-modality: Visual parsing with self-attention for rision-and-language pre-training. Advances in Neural Information Processing Systems, 34:4514 4528, 2021.
[67]
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video-text modeling with zero-shot transfer from contrastive captioners, 2022.
[68] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 521-539. Springer, 2022.
[69] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
[70] Yan Zeng, Xinsong Zhang, and Hang Li. Multi multi-grained vision language pre-training: Aligning texts with visual concepts. In International Conference on Machine Learning, pages 25994- 26009. PMLR, 2022.
[71] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588, 2021.
[72] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[73] Ziqi Zhang, Yaya Shi, Chunfeng Yuan, Bing Li, Peijin Wang, Weiming Hu, and Zheng-Jun Zha. Object relational graph with teacher-recommendde learning for video captioning. In Proceedings of the IEEÉ/CVF conference on computer vision and pattern recognition, pages 19278-132883 2020.
[74] Mingyang Zhou, Licheng Yu, Amanpreet Singh, Mengjiao Wang, Zhou Yu, and Ning Zhang. Unsupervised vision-and-langen Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16485-16494, 2022.
pre-training via retrieval-based muuli-granula alignment. In
14

---

A Intuition and motivation behind P-Former
In this section, we summarize the intuitive explanation and motivation why learning ideal language prompt helps more than using visual ones as in the counterpart models.
In our experiments with base models like BLIP-2, the architecture consists of three sequential components: (I) VIT, (2) VL-connector, and (3) LLM decoder. Since we use a frozen LLM for generation, optimizing closer to the LLM decoder becomes more pivotal for achieving optimal generation quality.
The unique design of P-Former mirrors sentence embedding model. This means the prompts predicted by the P-Former carry rich semantics. Therefore, during evaluations on unfamiliar images, the model boasts an improved generalization capability.
BLIP2's s studies indicate that direct end-to-end optimization of the sequential model can sometimes lead catastrophic forgetting. Our approach adds additional layer of complexity by decompos- ing the 2-stage BLIP2 training into 3 stages, further addressing this optimization challenge. For BLIP2, optimization of soft prompt is learned only using text from image-text pair, while decoupled training allows for leveraging additional unimodal data for optimizing these soft prompts
B Justification for lack of ablation experiments w/ and w/o the P-Former
We purposely omitted experiments with and without the P-Former module (e.g., using randomly initialized prompt p). This omission was driven by the following considerations:
Random initialization and learning without P-Former: Our initial approach was to directly learn from a randomly initialized prompt without incorporating the P-Former. But, upon testing, we identified significant challenge. For a smaller model variant like opt-2.7b, which possesses a hidden size of 2560, if we employ 32 tokens as soft prompts for an expansive dataset with 4M sentences, the resultant model would have to accommodate an overwhelming 327B parameters.
This would have computational implications and potentially overfit, as learning from such vast parameter space can dilute the essential semantic connections between various sentences.
P-Former's s efficiency in parameterization: The P-Former emerged as a solution to this parameter explosion problem. Instead of requiring a unique prompt for each data point in the dataset, the P- P-Former parameterizes the soft prompt p using a semantically -rich Transformer model. This design ensures that the total number parameters remains fixed at 110M. The major advantage here is scalability. Whether working with dataset of 4M, 12M, or even larger (e.g.. 129M) LMs with varying decoder sizes, the P-Former guarantees a consistent number of parameters, making the model more computationally efficient and preventing the loss of essential semantic relationships.
In brief, our experimentation strategy was driven by the dual goals of maintaining computational efficiency while preserving rich semantics. The challenges posed by direct learning from a randomly initialized prompt emphasized the need for more structured approach, leading to the birth of the P- P-Former concept.
C Qualitative analysis on VQA
In this section, we incorporate qualitative comparisons for the GQA and OKVQA datasets, allowing us offer more nuanced insights. In Figure C.I we show several examples comparing our model's response with BLIP-2 and the ground truth (GT). From these examples, it can be observed that there greater agreement with GT by our model.
should be noted that the abstract semantic reasoning of our model can sometimes lead to artificially low scores for our model when looking for exact match. For instance, asking "What occupation might he have?" with a picture of a person driving a forklift generates the answer "forklift operator" by model, whereas the correct exact answer in the GT is stated "forklift driver." Though these two answers are semantically identical, they will count as a wrong generation by our model.
D Additional discussion of the results
In this section, we provide more interpretation of the results. For instance, Table T. in addition to underscoring the potency of our proposed framework in bolstering the zero-shot VQA performance,
15

---

GQA
OKVQA
What ' croswalk near the lamp made of?
type clothing the men weading??
snowy?
device?
Aughat Concretie GT:
BLIP-
BLPP somy
BLIP-2 black GIT:
Clar
GT:
the picture the
On which
Is there tomato In this photo?
type would this
your
What grade are these lods
house?
BUP- both GT: left
BLIP yeu GTT: no
BUP 2: me GT: plummlrrrrr
SLJP-2 middl school GTT: third
Which reflective?
kind
of
looks oraage old
What type of picture taking?
What de call this unimal?
new?
BLIP-: 2 computer monber
BUP-2: minror walle
BLJP2: cal
Dur
doar
occupation
group of these What?
Dur:me
Our: black
Duc forklit operatir
Our hard
road the same that of the flag?
does happy man hold?
Is being spread?
havling lunch?
BLP-2:
Durino no
controllars game GT: will controller
BUP-2 cheese Our monareta
restaurant GTT: outeide
Which room it?
en top of the computer undemeath
What the purpose those umbeetlas?
What are the round that used as
pina in the photo?
BUI-2 both
BLP-2: computer GT: router
BUP-2 they for Our: for shade bleck
SuP-2 plra crust
OT: office
GT: olives
Figure C.I: Qualitative analysis on success and failure cases of GQA and OKVQA.
particularly when trained with 4M image-text pairs, shows that method manages to considerably close the performance gap between the BLIP-2 trained on different scales: 4M and 129M image-text pairs. This suggests that the effectiveness of model is solely function of the amount of training data but rather the methodology itself. In essence, this table illustrates how strategic modifications and improvements can achieve comparable results models trained much larger datasets.
Similarly, Table 2 provides insights into model's adaptability. When fine-tune pre-trained model for specific task like MSCOCO image captioning, the results reflect an overall enhancement BLIP-2 across all metrics. The pronounced improvement in CIDEr, as opposed SPICE, indicates that our model is adept at recognizing and generating more relevant and contextually accurate descriptions of images. The additional data zero-shot transfer the NoCaps dataset further substantiates the model's capability to generalize and adapt to newer, unseen data.
Finally, while our model's primary design goal is to refine visual prompts for text generation, Table 3 offers a perspective on its performance the retrieval domain. Even though the model was not specifically optimized for retrieval tasks, it is evident that the introduced modifications do not compromise the retrieval performance, attesting to the model's robustness.
E
LLM-dependence of the stage-1 pre-training
should be noted that our stage-1 pre-training needs to be repeated for each LLM, if w1 * 0. However, evidenced in Table 4 (wi and W/ 100), our approach achieves competitive results even without the alignment loss in stage-1, focusing the aligmment solely on stage-2.
Acknowledgement
The authors would like to thank Mu Li and Yi Zhu from Boson AI for their YouTube and Bilibili videos on paper reading, which greatly inspired this work.
16


Reviews:
[{"review_id": "SGuiSUKi1a", "reviewer": "Reviewer_3ni9", "summary": "The paper introduces a novel approach for optimizing the application of large language models in resource-intensive vision-language pre-training. Unlike the traditional approaches of using visual features as prompts to guide language models, the paper focuses on identifying optimal prompts to align with visual features. This is achieved through the introduction of the P-Former, a module trained only on linguistic data, eliminating the need for image-text pairings.", "questions": "- Could the authors provide more details and clearer insights into the P-Former part of the framework and size?\n- Across all experiments, it can be seen that more data helps BLIP-2 to outperform the P-Former. I wonder if the P-Former will improve the best performance of BLIP-2 if the pre-training image-text data is larger (129M)?\n- In Table 3, it can be observed that the performance of the proposed model is lower when it comes to retrieval. Could you interpret these results and elaborate more on this limited performance?\n- What is the reasoning behind the need for a separate pre-training of the P-Former on language data? Why do additional unimodal sentences contribute to the performance in Table 6?\n- Since the framework is presented as modality agnostic, do you think it can handle language-only tasks, such as machine translation? If so, which parts of the framework would need to be adapted to handle this?", "limitations": "The authors have provided a sufficient discussion of the limitations of their work in section 5. Due to the usage of pre-trained backbones, I would also encourage the authors to discuss any potential negative societal impact of their work.  ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality:\n   The paper demonstrates a high level of originality in its approach to vision-language learning. By introducing the P-Former, the paper presents a unique perspective in contrast to the standard approach of using visual features as prompts. Also, the focus on a modality-agnostic framework expands the applicability to various VL tasks, further enhancing the originality of the paper.\n2. Quality:\n   The paper exhibits a good level of quality in various aspects. The methodology is well-formulated, and the introduction of P-Former is well-justified. Also, it provides sufficient details about the experiments and demonstrates the robustness and flexibility of the framework across different VL tasks.\n3. Clarity:\n   The paper is written with clarity and precision. The abstract, introduction, and related work sections provide a clear context and properly position the paper in the literature. The technical concepts are well-explained, including the P-Former and the decoupled language pre-training approach. The use of figures and tables further help in understanding the experiments and results.\n4. Significance:\n   The significance of the paper is in the decoupled language pre-training approach, with the P-Former's ability to predict ideal prompts, addressing resource-intensive challenges in this field. Also, it offers a fresh perspective on how to leverage large LMs in VL learning scenarios. ", "weaknesses": "- The idea of the P-Former is interesting, however, it seems to lack an intuitive explanation and motivation. Why learning an ideal language prompt helps more, compared to using visual ones as in the counterpart models?\n- There seems to be a lack of some ablations, which naturally arise as questions, for example, experimenting with and without the P-Former module, maybe just by using randomly initialized prompt p, using different sizes/types of language backbones, different ways of initializing the prompt p at the beginning, etc.\n- Also, there is a lack of qualitative analysis of the experiments. I would recommend including and analyzing qualitative results in comparison to existing approaches. Presenting visual examples of the model's performance in both successful and failure cases can make the paper stronger.\n- Some experiments lack stronger interpretation. I would also encourage the authors to provide more interpretation of some results (e.g. tables 1, 2, 3), rather than describing the tables, to enable the readers to gain better insights into the behavior of the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Seems to lack an intuition on why learning an ideal language prompt helps?**\n\nRe: \n- Models like BLIP2 consist of three sequential components: (1) ViT, (2) VL-connector, and (3) LLM decoder. Given that we use a LLM for generation, optimizing closer to the LLM (i.e., prompts, comparing to visual features) becomes more pivotal for achieving optimal generation quality.\n- The design of P-Former mirrors a sentence embedding model (lines 158 to 163). This means the prompts predicted by the P-Former carry rich semantics. Therefore, the model boasts an improved generalization capability.\n- BLIP2's studies indicate direct end-to-end optimization can sometimes lead to catastrophic forgetting. Our approach decomposes the 2-stage training into 3 stages, further addressing this optimization challenge.\n- BLIP2's (implicit) refinement for soft prompts is exclusively achieved through the utilization of textual content from image-text pairs. In contrast, our decoupled training approach empowers us to harness supplementary unimodal data, facilitating the enhancement of these soft prompts in a more comprehensive manner.\n\n**W2: Lack some ablations, e.g., w/o the P-Former module, maybe randomly initialized prompt, different backbones, etc.**\n\nRe: That is a great question which highlights critical facets of the experimentation process. \n- *Random Initialization and Learning Without P-Former*: Our initial approach was, as you mentioned, to directly learn from a randomly initialized prompt p without incorporating the P-former. But, upon testing, we identified a significant challenge. For a smaller model variant like opt-2.7b, which possesses a hidden size of 2560, if we employ 32 tokens as soft prompts for an expansive dataset with 4M sentences, the resultant model would have to accommodate an overwhelming 327B parameters. This would not only have computational implications but also learning from such a vast parameter space can dilute the essential semantic connections between various sentences.\n- *P-Former's Efficiency in Parameterization*: The P-Former emerged as a solution to this parameter explosion problem. P-Former parameterizes the soft prompt $p$ using a semantically-rich model. This design ensures that the total number of parameters remains fixed at 110M. The major advantage here is scalability. Whether we're working with a dataset of 4M, 12M, or 129M, or any LM decoder's size, the P-Former guarantees consistent parameters, making the model more computationally efficient and preventing the loss of essential semantic relationships.\n\nFuture iterations of our research will undoubtedly delve deeper into the areas you've highlighted, providing a more comprehensive understanding of the P-Former's capabilities.\n\n**W3: Lack of qualitative analysis**\n\nRe: We value the feedback. We've incorporated qualitative comparisons for datasets such as GQA and OKVQA, allowing us to offer more nuanced insights (in uploaded PDF). We have demonstrated several examples comparing our model's response with BLIP-2 and the ground truth (GT). We will include a section on qualitative analysis in the appendix of our revised paper.\n\n**W4: Provide more interpretation of some results, rather than describing the tables.**\n\nRe: Thank you for the suggestion. We agree that more interpretation of the results will make our paper stronger. We will update our results section to include such a discussion. \n\n**Q1: Provide more details and clearer insights into the P-Former**\n\nRe: This is similar to our response to W1 and W2. Please see our responses above.\n\n**Q2: if the P-Former will improve BLIP-2 in 129M dataset setting?**\n\nRe: Conducting experiments with the 129M dataset presents challenges for us, given that we possess a max of 8 GPUs, while the original results reported by BLIP2 on the 129M dataset utilized 16 GPUs about 9 days. The primary goal of our method is to streamline the training process and make efficient use of the available training data. As such, we anticipate that our approach might show modest gains in a 129M setting, particularly if the model undergoes extensive training. In fact, a key motivation behind P-Former is to reduce the dependence on vast multi-modal datasets and models. This approach not only simplifies the training process but also democratizes participation, ensuring that research in this area isn't solely the domain of entities with access to significant computational resources.\n\n**Q3: Elaborate more on this limited performance in retrieval?**\n\nRe: The retrieval tasks primarily rely on the ViT and Q-former. Since P-former is designed exclusively for language generation by the LLM, it doesn't influence the retrieval processes, especially tasks that rely heavily on the contrastive objective. Our results in Table 3 aim to show that introducing P-Former doesn't negatively affect existing functionalities.\n\n**Q4: reasoning for a separate pre-training of the PFormer on language? Why do additional sentences contribute?**\n\nRe: (1) We drew inspiration from BLIP2’s observations that a 2-stage training is easier to optimize than end-to-end. Our strategy introduces an additional pre-training stage for the P-former, and our results confirm its benefits. (2) Introducing additional unimodal sentences enhances the learning of equation 4. With a fixed language decoder $D$, additional text $t$ inputs facilitate the learning of a better P-former $E$. \n\n**Q5: Language-only tasks, e.g., machine translation?**\n\nRe: Absolutely, our framework can be adapted for language-only tasks, including machine translation. The key adjustments would involve simply substituting the ViT with a language encoder and swapping the VL connector for a language-to-language connector. While a few adjustments are necessary, translations have proven effective when utilizing an encoder coupled with cross-attention, as exemplified in *'Attention is All You Need'*. For tasks focused solely on language, cross-attention might be the more intuitive approach.\n\n"}, {"review_id": "0qZkvRYqr8", "reviewer": "Reviewer_jFy5", "summary": "To improve image-to-text generation, this paper proposes a proxy model P-Former to predict LLM prompts and uses it as an auxiliary loss in BLIP-2 to align selected features with LLM prompts. Results show promising results, especially in 0-shot VQA tasks.\n", "questions": "As mentioned in “Weaknesses”, instead plugging P-Former in BLIP-2 as an auxiliary loss (IMO more like an incremental work), it might be more interesting to verify P-Former’s effect by directly predicting the LLM prompts in a captioning model?\n\nIn line 216, also mention the cost to train P-Former?\n\nIn line 196, elaborate the size of each dataset?\n\nminor comment: in equation 7, the parentheses seem incorrect?\n", "limitations": "No specific concern.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Nice novelty by introducing a proxy model for LLM prompts prediction to enhance image-to-text generation.\n\nThe experiments, ablations and analysis are comprehensive, showing improvements in both VQA (more significant) and captioning (less significant) tasks, and no improvement in image-text retrieval.\n\nWell written and easy to read.\n", "weaknesses": "This paper only shows the effect of the LLM prompt prediction in an incremental way, i.e., as an auxiliary loss in the existing BLIP-2. It would be more interesting if we could show the effectiveness of P-Former in a cleaner (simpler) setup.\n\nminor: The paper title says “Bootstrapping Vision-Language Learning”, which might be generic according to the results of this paper, such as no improvements on image-text retrieval tasks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 and Q1: This paper only shows the effect of the LLM prompt prediction in an incremental way, i.e., as an auxiliary loss in the existing BLIP-2. It would be more interesting if we could show the effectiveness of P-Former in a cleaner (simpler) setup.**\n\nRe: A potential pitfall in bypassing the LM loss entirely is that errors originating at the prompt level might be amplified as they navigate through the LM decoder. Given the end goal is caption/text generation, the LM loss remains a pivotal factor for ensuring the proficiency of such a model. Consequently, it may be challenging to completely forgo the LM loss, especially when considering a language generative model.\n\nFurther emphasizing the versatility of our method, it's adaptable across various modalities, including image, video, and audio. Moreover, it can be integrated into any VL model that employs prompts as interfaces. To illustrate this adaptability, we reference Section 4.5 where we highlighted the efficacy of our approach in video captioning, **leveraging a model distinct from BLIP2**. \n\n**Q2: In line 216, also mention the cost to train P-Former? In line 196, elaborate the size of each dataset? minor comment: in equation 7, the parentheses seem incorrect?**\n \nRe: These are thoughtful suggestions. We will make the necessary updates in our final version. Thank you!\n"}, {"review_id": "dgHddS2wfZ", "reviewer": "Reviewer_8epX", "summary": "This paper introduces a prompt-transformer (P-Former), which is pretrained on text corpus, that can trigger the LLM to generate better text prompts to align with the vision-and-language models with better visual features. Empirical experiments on the top of BLIP-2 shows promising results on several tasks. ", "questions": "1. Where is the gain from? I assume the stage 1 in the Figure 3 and Line 211, is just taken from the pretrained BLIP model, right? If not, do you take pretrained BLIP checkpoint for continuing pretrain 10 epochs? And do you have a comparison for the stage 1 pretraining, compared to the BLIP-2? The main issue here is, to try to make an alignment with BLIP-2 to see where the difference/improvement comes from? stage 1 or stage 2.\n\n2. Is the P-Former is agnostic to the different Vision-encoders? if changing a vision encoder, does the framework still work?", "limitations": "See the questions part.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper introduces an separate stage for P-Former, to trigger LLM to generate text prompts to align with visual features, verifies the ideas on the BLIP-2, and shows promising results on several tasks.", "weaknesses": "1. May need more comprehensive experiments to show where is the gain from. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1 and Q1: Where is the gain from? I assume the stage 1 in the Figure 3 and Line 211, is just taken from the pretrained BLIP model, right?**\n\nRe: Our approach in stage 1 is grounded in equation (8) and comprises dual learning objectives: the first one originates from BLIP2, while the second alignment loss is introduced by our P-Former. We have not adopted pretrained BLIP2 weights; we only employ its objective functions. The only learnable parameters in equation (8) are those in Q-Former, which are randomly initialized.\n\n**If not, do you take pretrained BLIP checkpoint for continuing pretrain 10 epochs? And do you have a comparison for the stage 1 pretraining, compared to the BLIP-2? The main issue here is, to try to make an alignment with BLIP-2 to see where the difference/improvement comes from? stage 1 or stage 2.**\n\nRe: Regarding the checkpoint continuation and comparison with BLIP-2: We initiate our vision-to-language model, ViT-Qformer-OPT, with a pretrained ViT and OPT, whereas the Q-former remains randomly initialized. A preliminary \"stage 0\" serves to train the P-former, which is subsequently frozen. Our subsequent approach can be summarized thusly:\n- Stage 1: We train the ViT (which remains frozen) and the learnable Q-former using equation (8), with the Q-former being initialized randomly.\n- Stage 2: Here, we employ the Q-former from stage 1 and train the ViT-Qfromer-OPT tandem via equation (9). \n\nTo succinctly delineate our method from BLIP2: Our stage 1 is represented by equation (8). Our stage 2 is represented by equation (9). By contrast, BLIP2's stage 1 corresponds to equation (8) but with w1 set to 0, and BLIP2's stage 2 matches equation (9) but with w2 set to 0.\n\n**Where does the difference/improvement come from? stage 1 or stage 2.**\n\nRe: By introducing w1 and w2 in equations (8) and (9) respectively, we've enabled an ablation to answer this question. As depicted in our ablation study (Table 4), we examine cases with w1=0 and w2=0. By setting w1 or w2 to zero, we effectively bypass the influence of the P-former, making the approach analogous to the original BLIP-2 training. The results indicate that the alignment loss introduced by our P-former contributes to both stage 1 and stage 2.\n\n\n**Q2: Is the P-Former agnostic to the different Vision-encoders? if changing a vision encoder, does the framework still work?**\n\nRe: Our framework is intrinsically adaptable to different encoders. This is illustrated in section 4.5 where we transition from the attention-driven ViT to a convolution-centric I3D model for video encoding. Notwithstanding this shift, our results (displayed in Table 7) remain very competitive on video-to-language generation.\n"}, {"review_id": "e1rB80PGUF", "reviewer": "Reviewer_T7A1", "summary": "This paper introduces a vision-language pre-training method with the help of the proposed Prompt-Transformer. As the first step, the P-Former is optimized to learn the \"optimal\" soft prompt that can guide the LLM to generate the target texts. After that, the trained P-Former is frozen and used to train the visual adaptor of VLM.", "questions": "None", "limitations": "The authors discussed three limitations in the paper. I am mainly concerned about the last one that the method cannot handle cross-attention models. With the development of VLM, cross-attention or joint modeling could be a very important branch of methods.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The method is novel in that it utilizes an intermediate P-Former to effectively optimize the VLM.\n2. The method achieves impressive results in terms of the performance of training data efficiency.", "weaknesses": "1. In Table 1, it is not fair to only list the numbers of image-text pairs considering that P-Former also consumes language data for training. Similarly, in the paragraph of Line 216, the overhead of training P-Former should also be taken into consideration.\n2. After reading the paper, it is still a bit unclear why, in essence, P-Former is effective. Does it function as some type of knowledge distillation? Please provide more discussion about this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: In Table 1, it is not fair to only list the numbers of image-text pairs considering that P-Former also consumes language data for training. Similarly, in the paragraph of Line 216, the overhead of training P-Former should also be taken into consideration.**\n\nRe: We agree that it's crucial to present a more comprehensive view of data consumption, factoring in both image-text pairs and the language data leveraged in P-Former training. While our original rationale was centered around the ease of accessibility and abundance of unimodal language data relative to image-text pairs, we understand the necessity of providing a balanced representation. Additionally, it's essential to underscore that the overhead of training the P-Former, although present, is a one-time process for a given LLM. This means it does not necessitate repeated training for different tasks (e.g., image/video/audio or QA/caption) provided the same OPT-2.7b is utilized. In our revised manuscript, we will ensure that these points are clearly articulated, and the computation overhead section is enhanced.\n\n**W2: After reading the paper, it is still a bit unclear why, in essence, P-Former is effective. Does it function as some type of knowledge distillation? Please provide more discussion about this.**\n\nRe:\n- In our experiments with base-models like BLIP-2, the architecture consists of three sequential components: (1) ViT, (2) VL-connector, and (3) LLM decoder. Given that we use a frozen LLM for generation, optimizing closer to the LLM decoder becomes more pivotal for achieving optimal generation quality.\n- The unique design of P-Former mirrors a sentence embedding model, as evidenced in lines 158 to 163. This means the prompts predicted by the P-Former carry rich semantics. Therefore, during evaluations on unfamiliar images, the model boasts an improved generalization capability.\n- BLIP2's studies indicate that direct end-to-end optimization of the sequential model can sometimes lead to catastrophic forgetting. Our approach adds an additional layer of complexity by decomposing the two-stage BLIP2 training into three stages, further addressing this optimization challenge.\n- For BLIP2, optimization of soft prompt is learned only using text from image-text pair, while our decoupled training allows for leveraging additional unimodal data for optimizing these soft prompts\n\n\nWe will include a discussion of this intuition in our revised manuscript.\n\n"}, {"review_id": "N05ranlreH", "reviewer": "Reviewer_FD1R", "summary": "This paper propose to add a new component, a P-Former, to the BLIP-2 vision-language pre-training framework. The P-Former is a sentence encoder that learns to project texts into the input space of a LLM. During vision-language pre-training, an additional alignment loss is applied between BLIP-2 Q-Former's visual feature and the P-Former's text feature, which improves the alignment between the Q-Former and the frozen LLM, leading to improved image-to-text generation performance.", "questions": "- Is the P-Former finetuned during stage-1 and stage-2 pre-training?", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper explores an interesting direction to improve BLIP-2 pre-training with an additional text encoder. The text encoder serves as a surrogate of the LLM to help better aligning the Q-Former with a frozen LLM. This could open up future research opportunities. \n\n- The experiments show some good improvements w.r.t. the original BLIP-2 on image-to-text generation tasks, which verifies the improved alignment with the LLM. It is also good to see improvements with both OPT and FlanT5.", "weaknesses": "I enjoy the core idea of the paper. However, the paper still needs much improvement in multiple aspects as detailed below.\n\n1. The motivation and problem formulation is a slight misrepresentation of the actual method. The actual role of the P-Former is to serve as a surrogate of the LLM to help the Q-Former better align with the frozen LLM in both stage-1 and stage-2 pre-training.\n\n2. The proposed method has a significant conceptual difference from BLIP-2: the stage-1 pre-training is *not* LLM-agnostic. This means that the more expensive stage-1 pre-training needs to be performed for every LLM. The proposed method thus tradeoff the Q-Former's flexibility for better alignment with the LLM. This point needs to be highlighted.\n\n3. The pre-training of the P-Former is rather complex, where the intuition of each training loss is not clear. There needs to be more discussion and ablation study on the multiple losses.\n\n4. The effect of the P-Former alignment may or may not maintain given more pre-training data and longer pre-training epochs. This is hard to verify given limited compute, so I'm not expecting the authors to provide answers.\n\n5. The writing could be improved. Personally, I feel many equations to be unnecessary because they add additional burdens to understand the method. Simple plain text would be good enough to describe the method clearly. The \"forward-decoupled\" and \"backward-decoupled\" concepts are also not intuitive to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: The motivation and problem formulation is a slight misrepresentation of the actual method. The actual role of the P-Former is to serve as a surrogate of the LLM to help the Q-Former better align with the frozen LLM in both stage-1 and stage-2 pre-training.**\n\nRe: The P-Former does indeed function as a surrogate for the LLM, assisting the Q-Former to align optimally with the frozen LLM during both stage-1 and stage-2 pre-training. The driving force behind our methodology, however, was to develop a decoupled training approach. Our primary intention was to create an optimal prompt, learned by the P-Former, to guide the VL-connector (e.g., Q-Former) effectively.\n\n**W2: The proposed method has a significant conceptual difference from BLIP-2: the stage-1 pre-training is not LLM-agnostic. This means that the more expensive stage-1 pre-training needs to be performed for every LLM. The proposed method thus tradeoff the Q-Former's flexibility for better alignment with the LLM. This point needs to be highlighted.**\n\nRe: You've correctly identified a departure in our approach from BLIP-2: our stage-1 pre-training is indeed not LLM-agnostic. This difference will be highlighted  in our revised manuscript. However, as evidenced in the 3rd row of Table 4 (w1=0 and w2=100) from our ablation study, our approach achieves competitive results even without alignment loss in stage-1, focusing the alignment solely on stage-2.\n\n**W3: The pre-training of the P-Former is rather complex, where the intuition of each training loss is not clear. There needs to be more discussion and ablation study on the multiple losses.**\n\nRe: The contrastive loss is employed to enhance the semantic richness of the P-Former’s output, ensuring similar images/sentences yield similar prompts. Furthermore, our findings indicate that the inclusion of vocabulary loss (which can be viewed as a regularization term) slightly improves our results (without vocabulary loss: GQA:33.5, OKVQA: 29.5, VQA: 51.7. With vocabulary loss: GQA: 34.0, OKVQA: 30.0, VQA: 52.6). \n\n**W4: The effect of the P-Former alignment may or may not be maintained given more pre-training data and longer pre-training epochs. This is hard to verify given limited computation, so I'm not expecting the authors to provide answers.**\n\nRe: Conducting experiments with the 129M dataset presents challenges for us, given that we possess a max of 8 GPUs, while the original results reported by BLIP2 on the 129M dataset utilized 16 GPUs about 9 days. The primary goal of our method is to streamline the training process and make efficient use of the available training data. As such, we anticipate that our approach might show modest gains in a 129M setting, particularly if the model undergoes extensive training. In fact, a key motivation behind P-Former is to reduce the dependence on vast multi-modal datasets and models. This approach not only simplifies the training process but also democratizes participation, ensuring that research in this area isn't solely the domain of entities with access to significant computational resources.\n\n**W5: The writing could be improved. Personally, I feel many equations to be unnecessary because they add additional burdens to understanding the method. Simple plain text would be good enough to describe the method clearly. The \"forward-decoupled\" and \"backward-decoupled\" concepts are also not intuitive to understand.**\n\nRe: We are grateful for your feedback. We will revisit our manuscript to ensure greater clarity, potentially simplifying certain sections for enhanced comprehension. Complex concepts like “forward-decoupled” and “backward-decoupled” will be elaborated upon to provide a more intuitive understanding.\n\n**Q1: Is the P-Former fine-tuned during stage-1 and stage-2 pre-training?**\n\nA: The P-Former remains frozen during both stage-1 and stage-2. Learned during stage 0, the P-Former is designed to predict semantically rich soft prompts for individual instances during the subsequent stages. Its fixed state during these stages is pivotal for generating alignment losses, directing the outputs of the VL-connector (e.g., Q-Former in BLIP2).\n"}]

Author rebuttal:
Dear Reviewers:

We sincerely thank you for your thoughtful reviews and constructive feedback on our paper. We are heartened to see that our proposal of adding a new component, a P-Former, to the X-language (X being any modality) pre-training framework resonated well with all of you. Your appreciation of our exploratory direction, the improvements shown in our experiments, and the potential impact on future research is highly encouraging. Your insights have aided in refining our understanding, and we have addressed each of your comments individually.  

In addition to our detailed response, we've incorporated qualitative comparisons of our method and BLIP-2 for specific datasets, such as GQA and OKVQA (see attached pdf). We will include a section on qualitative analysis in the appendix of our revised paper.

Meta-review: This paper proposes a new approach for vision-language learning, in particular visual language models. To this end, the paper introduces a new transformer that generates prompts that optimally align with the visual features. The reviewers highlighted the papers originality, experimental completeness, unique ideas and relevance to the community. In addition, the extensive rebuttal and discussion have further cleared *all* of the remaining questions. Recommending spotlight.
Decision: Accept (spotlight)