Fusion or Defusion? Flexible Vision-and-Language Pre-Training
Year: 2023

Abstract:
Existing approaches in the vision-and-language pre-training (VLP) paradigm mainly deploy either fusion-based encoders or dual-encoders, failing to achieve both effectiveness and efficiency in downstream multimodal tasks. In this paper, we build a flexible VLP model by incorporating cross-modal fusions into a dualencoder architecture, where the introduced fusion modules can be easily decoupled from the dual encoder so as to switch the model to a fusion-free one. To better absorb cross-modal features from the fusion modules, we design a cross-modal knowledge transfer strategy along with other comprehensive pre-training tasks to guide the training process, which can further strengthen both the fusion-based and fusionfree representation learning. Extensive experiments conducted on various downstream visionlanguage tasks show that our proposed model is well-equipped with effectiveness as well as efficiency, demonstrating a superior performance compared with other strong VLP models.

heading: 1 Introduction
text: With the great development of self-supervised pretraining in both the community of natural language processing (Devlin et al., 2019; Raffel et al., 2020) and computer vision (Dosovitskiy et al., 2021; Bao et al., 2022a), recent researches have also witnessed the success of Vision-and-Language Pretraining (VLP). VLP learns generic multimodal representations from large-scale image-text pairs and can be further finetuned on various downstream Vision-Language (VL) tasks, including image-text retrieval (Lin et al., 2014), visual question answering (Goyal et al., 2017), visual reasoning (Suhr et al., 2019) and visual entailment (Xie et al., 2019).
The core of VLP resides in modeling the interaction between image and text representations. Most of the mainstreams first represent the input image
∗Equal contribution. †Corresponding authors.
via pre-trained deep feature extractors, then feed the derived visual features along with the text embeddings into multi-layer Transformers(Vaswani et al., 2017), in which cross-modal attention is used to fuse multimodal representations. Despite demonstrating superior performances on downstream VL tasks, the fusion-based methods need to jointly encode image and text representations, significantly degrading the efficiency in retrieval tasks with massive candidates of image-text pairs.
To make VLP models applicable in real-world scenarios, another line of methods independently encode text and image with dual encoders, shown in Fig. 1(a), in which cross-modal fusion is conducted by lightweight modules such as dot production. Thanks to the dual-encoder architecture, encoded features of image and text can be precomputed offline for inference efficiency. Nevertheless, independent encoding with shallow interaction fails to fully exploit the cross-modal interaction, making the performance far from satisfactory in VL classification tasks that require a strong ability of multimodal reasoning.
There are some recent works that attempt to keep
5105
both effectiveness and efficiency in downstream VL tasks. In particular, Wang et al. (2021b) empower a dual-encoder model by distilling knowledge from a fusion-based model. Although the distilled dualencoder learns useful knowledge from cross-modal fusions while keeping its efficiency, this kind of method needs to pre-train a fusion-based model as a teacher and the performance is severely limited by the ability of the teacher model. VLMo (Bao et al., 2022b) introduces mixture-of-experts to encode various modalities with a modality-agnostic Transformer, which can be used as either a fusion encoder or a dual encoder. However, to fully train such a sparse model with experts towards different modalities, not only the image-text pairs but also massive images and text are required.
In this paper, we propose a unified and flexible VLP model named FOD, which incorporates cross-modal fusions into a dual-encoder architecture for achieving both efficacy and efficiency in multimodal scenarios. Specifically, we adopt a dual architecture with one image encoder and one text encoder, in which cross-modal fusions are placed in the text encoder side. Considering that conventional fusions are based on either concatenation (Kim et al., 2021; Singh et al., 2022) or cascading (Li et al., 2021a; Dou et al., 2022) that can’t be directly decoupled from the boarding encoder, we employ a parallel-style fusion module to model cross-modal interactions, shown in Fig. 1. In this way, FOD can explicitly capture the complex interaction between modalities during training while switching the fusion-based text encoder to a fusionfree one by removing the fusion module.
In order to retain more cross-modal knowledge in FOD when the fusion modules are removed, we
further design a cross-modal knowledge transfer strategy that forces both the unimodal features of image and text to approximate the multimodal representation produced by the fusion-based encoder. Intuitively, since paired image and text describe the same object in different views, we can naturally associate a set of relevant images when given a caption (and vice versa). Thus, if the text feature learns to “associate" its related images and absorbs them to enhance itself, the enhanced text feature can become closer to the relevant image candidates (and also farther to the unrelated ones) in inference. A concrete example illustrating this intuition is shown in Fig. 2.
We evaluate our model on both image-text retrieval tasks and vision-language understanding tasks. Experimental results show that our model outperforms other VLP methods on all downstream VL tasks, and even performs competitively with models that use a larger order of magnitude of data for pre-training. Thanks to the detachable fusion module and the strategy of knowledge transfer, our model can be flexibly switched to a fusion-free pattern to enjoy a much faster inference speed of retrieval while retaining most of the performance.

heading: 2 Related Work
text: Without considering the ways of visual feature extraction, the approaches of vision-language pretraining can be divided into two categories based on the interaction form between image and text. The first category, fusion-based model, explicitly utilizes deep fusion layers with cross-modal attention to model the interaction of images and texts (Tan and Bansal, 2019; Lu et al., 2019; Su et al., 2019; Li et al., 2019; Chen et al., 2020; Li et al.,
2020, 2021b; Gan et al., 2020; Zhang et al., 2021; Huang et al., 2020, 2021; Kim et al., 2021; Li et al., 2021a; Wang et al., 2021c; Li et al., 2022; Zeng et al., 2022; Wang et al., 2022). These models perform well on vision-language understanding tasks due to the ability of capturing deep cross-modal features. However, for vision-language retrieval tasks, the fusion-based methods need to encode all the possible image-text pairs to find the most relevant candidate, resulting in extremely high time cost.
The second category, dual-based model, utilizes a visual encoder and a text encoder to separately encode images and text, while the interaction between images and text is modeled by cosine similarity or linear projection (Radford et al., 2021; Jia et al., 2021; Yao et al., 2021). Although dualbased models are effective for retrieval tasks since features can be pre-computed and cached offline, the shallow interaction is insufficient to tackle the vision-language understanding tasks that require complex VL reasoning. Besides, training a dualbased model often necessitates a large number of image-text pairs (e.g. 300M for Filip (Yao et al., 2021) and 1.8 B for ALIGN (Jia et al., 2021)).
Recently, some researchers have devoted themselves to investigating a unified model that is wellperformed on vision-language understanding tasks while maintaining the efficiency towards retrieval tasks (Wang et al., 2021b; Liu et al., 2021; Wang et al., 2021a; Bao et al., 2022b; Dou et al., 2022). To achieve this, one line of the works leverage knowledge distillation, in which a fusion-encoder
model is pre-trained as a teacher model to guide the training of a dual-encoder model (Wang et al., 2021b), but the performance is inevitably limited by the teacher model. Other efforts attempt to train a modality-agnostic encoder with shared parameters, which can be used as either a fusion encoder or a dual encoder (Wang et al., 2021a; Bao et al., 2022b). Despite the benefits of modeling all the modalities into a single encoder, it is hard to fully train such a huge model and a large number of training samples in different modalities are required. Different from these methods, we incorporate a detachable cross-modal fusion module into a dualencoder architecture, which can easily remove the fusion module in inference and switch to a fusionfree model. More importantly, our model does not rely on teacher models or massive data in other modalities.

heading: 3 Model Architecture
text: As shown in Fig. 3, FOD is in a transformer-based dual-encoder architecture that includes a visual encoder and a text encoder. The text encoder can be flexibly switched between a fusion-based pattern and a fusion-free pattern. For the fusion-based pattern, cross-modal fusions are incorporated into the text encoder to model multimodal interactions. For the fusion-free pattern, the fusion module is decoupled from the text encoder so as to get rid of the cross-modal calculation. During training, both fusion-based and fusion-free patterns are involved in the learning process, while in inference, the text encoder will be switched to one of the two patterns
according to the type of downstream tasks. In the following sections, we introduce the visual encoder and the two patterns of the text encoder, followed by the pre-training strategies.

heading: 3.1 Visual Encoder
text: We utilize Vision Transformer (Dosovitskiy et al., 2021) to build the visual encoder. Given a 2D image I ∈ RC×H×W , we first reshape I into a sequence of 2D image patches V p ∈ RN×(P 2·C), where (H,W ) is the original image resolution, C is the number of channels, (P, P ) is the patch resolution, and N = HW/P 2 is the number of patches.
V p = [vp1 ; · · · ; vpN ]. (1)
Then we flatten the patches and embed them to V e ∈ RN×D with a trainable linear projection ω ∈ R(P 2·C)×D, where D is the hidden size.
V e = [vp1ω; · · · ; vpNω]. (2)
We also prepend a learnable embedding V ecls ∈ RD to the patch embeddings V e. Besides, positional information is also important for path representations. Therefore, the embedded patches V̄ are obtained by summing [V ecls;V
e] and learnable 1D position embeddings Vpos ∈ R(N+1)×D. Finally, we obtained visual features V by encoding V̄ with the visual encoder VE.
V̄ = [V ecls; v e 1; · · · ; veN ] + Vpos, V = VE(V̄ ). (3)

heading: 3.2 Text Encoder
text: As mentioned before, there are two patterns of the text encoder: fusion-free text encoder and fusionbased text encoder. These two patterns are both based on Transformers (Vaswani et al., 2017) and share all the fusion-free parameters except the output linear projection in the last encoding layer.
Given the input text t = {tcls;w1; · · · ;wS}, we first embed t to T 0 ∈ RS×D via a word embedding matrix and a position embedding matrix. Then the text embedding T 0 can be fed into different patterns of the text encoder to produce different output features.

heading: 3.2.1 Fusion-free Text Encoder
text: In this pattern, the text encoder skips the crossmodal fusions and outputs text-only features. The
text encoder is a L-layer Transformer, and the output of the l-th layer T l is computed as follows:
T ls = MSA(T l−1, T l−1, T l−1), T̂ l = LN(T ls + T l−1), T l = LN(MLP(T̂ l) + T̂ l),
T = TL,
(4)
where MSA, LN and MLP are shot for Multi-Head Self-Attention, layer normalization and multi-layer perceptron respectively, T is the final features of the fusion-free text encoder.

heading: 3.2.2 Fusion-based Text Encoder
text: To fully capture vision-and-language interactions, both self-attention and cross-modal attention are considered in the fusion-based encoder. Specifically, in the l-th layer, we separately compute the fusion-free self-attention and the image-fused cross-attention, and then sum them up to produce the multimodal features. The detailed process is shown as follows:
M0 = T 0, M ls = MSA(M l−1,M l−1,M l−1), M lc = MCA(M l−1, V, V ), M̃ l = 1
2 × (M ls +M lc),
M̂ l = LN(M̃ l +M l−1),
M l = LN(MLP(M̂ l) + M̂ l),
M = ML,
(5)
where MCA is Multi-Head Cross Attention, V is the final visual features produced by the visual encoder. The MCA, LN and MLP modules are reused from the fusion-free text encoder. Notably, the cross-modal attention is introduced in a parallel manner, which is parameter-efficient and can be easily decoupled from the encoder.
In addition, the cross-modal fusions can also be placed in the visual side to build a fusion-based visual encoder, or be placed in both sides for deeper interaction. We will discuss this in the experiment section.

heading: 4 Pre-training Strategies
text: FOD is jointly trained with three different strategies, namely fusion-free learning, fusion-based learning and cross-modal knowledge transfer, which are complementary to each other.

heading: 4.1 Fusion-free Learning
text: For this strategy, we utilize image-text contrastive learning to train the dual architecture with the ability of unimodal encoding, which is not only beneficial to other cross-modal learning strategies, but also the basis for applying the model to downstream retrieval tasks.

heading: 4.1.1 Image-Text Contrast
text: We select Vcls and Tcls produced by visual encoder and fusion-free text encoder to compute the loss of contrastive learning. In order to have more negative examples here, we maintain two queues to store the most recent K image and text representations computed by momentum encoders like MoCo (He et al., 2020). For convenience, we denote these representations in queues as V kcls and T k cls, where k ∈ {1, · · · ,K}. For each image representation V jcls and text representation T j cls in the current batch, the image-to-text similarities pi2tj and text-to-image similarities pt2ij are computed by:
si2tj,k = g(fv(V j cls)) ⊤g(ft(T k cls)), st2ij,k = g(ft(T j cls)) ⊤g(fv(V k cls)),
(6)
pi2tj = exp(si2tj,j/σ)∑K k=1 exp(s i2t j,k/σ) , pt2ij = exp(st2ij,j/σ)∑K k=1 exp(s t2i j,k/σ) ,
(7)
where fv and ft are linear projections, g is L2 normalization, and σ is a learnable temperature parameter. Let yi2t and yt2i denote the ground-truth ont-hot similarity, where positive pairs have a probability of 1 and negative pairs have a probability of 0. The image-text contrastive loss Litc is defined as the cross-entropy H between p and y:
Litc = 1 2 ×
[ H(yi2t, pi2t) +H(yt2i, pt2i) ] . (8)

heading: 4.2 Fusion-based learning
text: For this strategy, we apply image-text matching (ITM) and mask language modeling (MLM) to the fusion-based text encoder for learning both coarsegrained and fine-grained cross-modal fusions.

heading: 4.2.1 Image-Text Matching
text: ITM focuses on coarse-grained multimodal learning, which aims to predict whether a pair of image and text is matched or not. Since the imagetext pairs in a batch are all positive, we sample global hard negative image-text pairs from all input batches on all the GPUs based on the similarity scores calculated in Eq. 7. Then we feed the final hidden vector of the fusion-based encoder
Mcls into a binary classifier to predict a two-class probability pitm. Given the ground-truth label yitm ∈ {0, 1}, the image-text matching loss Litm is defined as the cross-entropy H between yitm and pitm:
Litm = H(yitm, pitm). (9)

heading: 4.2.2 Masked Language Modeling
text: MLM predicts masked tokens on the image-fused text features, which serves as the fine-grained crossmodal learning. Formally, we randomly mask 15% of the tokens in the text sequence t with a whole word masking strategy (Cui et al., 2021) and denote the input embedding of the masked text as T̄ 0. Then the model is trained to predict the masked tokens based on the final outputs M̄ by feeding T̄ 0 into the fusion-based encoder. The detailed process is similar to Eq. 5. Let ymask denote the groundtruth label of the masked tokens, and pmask denote the models’ prediction for the masked tokens, then the masked language modeling loss is defined as the cross-entropy H between ymask and pmask:
Lmlm = H(ymask, pmask). (10)

heading: 4.3 Cross-modal Knowledge Transfer
text: In our preliminary experiments, we observe that if the ITM loss is removed from the training process, the performance in retrieval tasks would dramatically degrade. From the perspective of feature distributions, we believe that ITM can better close the spatial distance between the unimodal features of image and text, which encourages us to explicitly utilize ITM to enhance unimodal representations.
To achieve this, we further design the strategy of cross-modal knowledge transfer (CKT). Given an image-text pair, we can first extract its image Vcls, text Tcls and multimodal representations Mcls. Obviously, Mcls is the most comprehensive feature that describes the image-text pair among them, but only Vcls and Tcls are used to compute similarity score in retrieval tasks. In this case, if we enhance the text feature to actively associate its related images by transferring knowledge from Mcls to Tcls, it will be easier to find the relevant image candidates based on the enhanced text feature in inference (and similar for Vcls). Thus, we force both Vcls and Tcls to approximate Mcls via meansquared loss in the last layer, which are calculated as follows:
LI2M = MSE(fv(Vcls), ft(Mcls)), LT2M = MSE(ft(Tcls), ft(Mcls)),
(11)
where fv and ft are the linear projections used in Eq. 6. We do not freeze Mcls in knowledge transfer so that multimodal and unimodal features can be jointly trained.

heading: 5 Experiment

heading: 5.1 Pre-training Settings

heading: 5.1.1 Datasets
text: Following previous works (Chen et al., 2020; Kim et al., 2021), we use four well-known image captioning datasets for pre-training: SBU Captions (Ordonez et al., 2011), Microsoft COCO (Lin et al., 2014), Visual Genome (Krishna et al., 2017) and Google Conceptual Captions (GCC) (Sharma et al., 2018). Since images in GCC and SBU are provided
in url format and some of them are inaccessible, we only collected 3.4M images, which is around 600K less than the original settings. In the experiments, we term the setting of 3.4M images as 3M.

heading: 5.1.2 Implementation Details
text: For model settings, the visual encoder adopts the same architecture as ViT-Base (Dosovitskiy et al., 2021) and we initialize it with pre-trained weights of Beit (Bao et al., 2022b). The text encoder is modified on Bert-Base (Devlin et al., 2019) by adding a multi-head cross attention and we initialize it with pre-trained weights of uncased-bert-base. For hyper-parameter settings during pre-training, the resolution of input images is 256 × 256 and the patch size is 16 × 16. RandAugment (Cubuk et al., 2020) is applied to the input images. We use AdamW optimizer (Loshchilov and Hutter, 2017) with weight decay of 1e-2 and the learning rate is
warmed up to 1e-4 over the first 1k steps. We pretrain for 300K steps on 32 NVIDIA A100 GPUs with a batch size of 2048.

heading: 5.2 Downstream Vision-Language Tasks

heading: 5.2.1 Image-Text Retrieval Tasks
text: The vision-language retrieval tasks include imageto-text retrieval and text-to-image retrieval. We evaluate our model on the Karpathy and Fei-Fei (2015) split of MSCOCO (Lin et al., 2014) and Flickr30K. During fine-tuning, we preserve the loss of image-text contrastive learning, image-text matching and cross-modal knowledge transfer. For a better comparison with various methods, we have two settings in the inference phase, namely “Dual” and “Re-Rank”.
For the “Dual” setting, we use Eq. 6 to precompute images and text representations separately, and compute the similarity scores of all possible image-text pairs by dot production. For the “ReRank" setting, we first utilize the similarity scores derived from Eq. 6 to select the top-k candidates, and then predict the final results by calculating their ITM scores (pitm).

heading: 5.2.2 Visual Question Answering
text: The VQAv2 (Goyal et al., 2017) task requires to predict answers based on the given pair of an image and a question. Following Cho et al. (2021) and Li et al. (2021a), we treat VQA as an answer generation problem. In order to compare fairly with other methods, we restrict the answer generation space
to the same candidate set (Kim et al., 2021; Bao et al., 2022b) during inference.

heading: 5.2.3 Natural Language for Visual Reasoning
text: The NLVR2 (Suhr et al., 2019) task asks the model to predict whether a text correctly describes a pair of images. We follow previous work (Li et al., 2021a; Zeng et al., 2022) to extend the fusion-based encoder to enable reasoning over image pairs and feed the encoded vector of the input pair into a classification layer to predict answer.

heading: 5.3 Main Results

heading: 5.3.1 Image-Text Retrieval Results
text: Table 1 and Table 2 show the results of fine-tuned and zero-shot image-text retrieval on MSCOCO and Flickr30K. For a fair comparison, only basesize models pre-trained on the standard 4M data are selected as the compared models. In this setting, our model achieves state-of-the-art performance on both datasets, and even performs competitively with CLIP, ALIGN and BLIP that are pre-trained on a larger order of magnitude of data. Furthermore, thanks to the designed parallel-style fusions and cross-modal knowledge transfer strategy, more cross-modal knowledge is retrained when the fusion module is decoupled in inference, narrowing the gap between “Dual" and “Re-Rank" settings. Detailed analysis of performances between “Dual" and “Re-Rank" settings are given in Appendix.

heading: 5.3.2 Vision-Language Understanding Results
text: The VQA2 and NLVR2 are categorized as understanding tasks since they both require the ability of VL reasoning, and the results are shown in Table 3. Our model achieves the best performances on both tasks among all the competitors that are also
in base-size and pre-trained with the standard 4M data, and even outperforms models pre-trained on more data like SimVLM and BLIP, demonstrating the effectiveness and efficiency of our model.

heading: 5.4 Ablation Studies

heading: 5.4.1 Different Designs of Fusions
text: We incorporate cross-modal fusions into a dual architecture to better model vision-language interactions. Conventional fusions are based on two kinds of methods, namely concatenation and cascading. (1) Concatenation jointly encodes the concatenation of image and text, which is quadratic in time complexity and twice in memory consumption. (2) Cascading first uses self-attention to encode the text input and then fuses it with image via crossattentions, which has a strong dependency between cross-attention and self-attention. Table 4 reports the ablation results of different fusions, our design that incorporates cross-modal fusions in a parallel manner outperforms other methods on retrieval tasks, showing that parallel-style fusion can switch our model into the “Dual” setting more flexibly.

heading: 5.4.2 Cross-modal Knowledge Transfer
text: We conduct the ablation experiments towards the strategy of cross-modal knowledge transfer, which is shown in Table 5. The objectives of I2M and T2M are defined in Eq. 11. From the results we can observe that: (1) I2M specifically improves the performance on image-text retrieval (TR) while T2M is beneficial for the text-image (IR) side, which are consistent with their intuitions; (2) I2M and T2M are complementary to each other. Adding both I2M and T2M during training can further bring
improvements for retrieval tasks while keeping the performances on VL understanding tasks.

heading: 5.4.3 Fusions on Both Sides
text: Intuitively, in addition to placing cross-modal fusions in the text encoder, we can also add the fusion modules into the visual side in a similar way. In this setting, ITM and downstream classifications are based on the concatenation of the multimodal features produced by both text and visual encoders. Fig. 4 shows the results of placing fusions in different sides, from which we find that when fusions are placed on both sides, the performance unexpectedly drops on all downstream tasks. We analyze that one possible reason comes to the difference between text and vision in self-supervised learning. It is obvious that BERT naturally works better in selfsupervision than ViT, and thus we can utilize the MLM task from BERT to learn fine-grained crossmodal interaction. When it comes to the visual side, self-supervised tasks are much more complex than MLM, inevitably making it more difficult to train such a VLP model.

heading: 6 Qualitative Analysis
text: We further provide a qualitative analysis by using Grad-CAM (Selvaraju et al., 2017) to illustrate the per-word visualizations of the cross-modal attention maps of the fusion-based encoder. As shown in Fig. 5, from the visualizations we observe that when conducting image-text matching tasks, our model can focus on specific regions in an image according to different words in each sentence, including objects, actions, attributes and background. More examples are given in Appendix.

heading: 7 Conclusion
text: In this work, we propose a flexible VLP model that incorporates cross-modal fusions into a dualencoder architecture. The fusion module can be easily decoupled in inference, enabling the model to be switched between fusion-based and a fusionfree patterns according to different scenarios. Extensive experiments conducted on both image-text retrieval and vision-language understanding tasks show that our model is well-equipped with effectiveness and efficiency compared with existing VLP models.
Limitations
The findings of this study have to be seen in light of some limitations. (1) It is non-trivial to extend our model for generation tasks. Since the main focus of this work is to improve both effectiveness and efficiency of the dual-encoders, text-decoder is not considered in model design. In the future, autoregressive mechanisms will be consider to applied in model architecture so that the model can be directly used for generation tasks like image captioning. (2) There may be disadvantages of the model in region-level VL tasks such as Object Detection. The reason is that these tasks require images in high resolution and fine-grained annotations of bounding boxes, which are non-trivial in generic VLP settings. To solve this problem, exploring different levels of granularity between image-text pairs is a promising direction and will be considered as the future work.

heading: Acknowledgements
text: This research is supported by National Natural Science Foundation of China (Grant No.62276154), Research Center for Computer Network (Shenzhen) Ministry of Education, Beijing Academy of Artificial Intelligence (BAAI), the Natural Science Foundation of Guangdong Province (Grant No. 2023A1515012914), Basic Research Fund of Shenzhen City (Grant No.JCYJ20210324120012033 and JSGG20210802154402007), the Major Key Project of PCL for Experiments and Applications (PCL2021A06), and Overseas Cooperation Research Fund of Tsinghua Shenzhen International Graduate School (HW2021008). Jingang Wang is funded by Beijing Nova Program (Grant No.20220484098).

heading: A Appendix
text: A.1 Statistics of Pre-training Datasets
In the expriments, we use four widely-used datasets for pre-training: SBU Captions (Ordonez et al., 2011), Microsoft COCO (Lin et al., 2014), Visual Genome (Krishna et al., 2017) and Google Conceptual Captions (Sharma et al., 2018). Due to the inaccessible problem of url, we can only collected 3.4M images, which is around 600K less than the original settings (Kim et al., 2021; Li et al., 2021a; Bao et al., 2022b). Details are shown in Table 6. Intuitively, if we can access to the full 4M data, our model could perform better.
A.2 Implementation Details
Image-Text Retrieval. Different from pretraining, we set the resolution of images to 384× 384 and use the tasks of ITC, ITM and CKT in image-text retrieval. The batch size is 256 and the initial learning rate is 1e-5. For Flickr30K and MSCOCO, we finetune 10 epochs and 5 epochs respectively. For the “Re-Rank" setting that selects top-k candidates, k is set to 128 for Flickr30K and 256 for MSCOCO following Li et al. (2021a) and Zeng et al. (2022).
Visual Question Answering. For visual question answering, most methods convert VQAv2 to a classification task by preserving the most frequent 3192 answers in datasets. However, this will prevent some data from being used for fine-tuning because their answers are not in the candidate set. Thus, we follow previous work (Cho et al., 2021; Li et al., 2021a; Zeng et al., 2022) and treat VQA as an answer generation problem. More specifically, we predict the probability distribution on the vocabulary of the first token, and select the top-k candidates with the highest probability from the distribution. Finally, we use language-modeling loss to predict the final answer from the top-k candidates. For a fair comparison, we restrict the answer generation space to the same candidate set (Kim et al., 2021; Bao et al., 2022b) during inference.
We finetune our model for 8 epochs with 256 batch size and the learning rate is 2e-5. The resolution of images is set to 576× 576 (Dou et al., 2022) and k is set to 128.
Natural Language for Visual Reasoning. For NLVR2, we follow previous work (Li et al., 2021a; Zeng et al., 2022) and extend the fusion-based encoder to enable reasoning over image pairs, in which an additional pre-training step is applied for training model to reason the relations among text and images. Then, we fine-tune the model for 15 epochs. The batch size is 128, learning rate is 2e5 and the resolution of the input image is set to 384× 384.
A.3 Performance Retaining
For VL retrieval tasks that involve massive candidates of image-text pairs, it is crucial for a VLP model to have the ability of acting as a dual-encoder for efficient inference. Table 7 reports the comparisons between “Re-Rank” and “Dual” settings on retrieval tasks. Our model performs best in terms of performance retraining when switched from “ReRank” to “Dual” setting, showing the effectiveness of the designed parallel-style fusions and crossmodal knowledge transfer strategy.
A.4 Inference Speed
We further evaluate the inference time of our models and other compared methods on MSCOCO dataset. All the models are evaluated on a single A100 GPU. From the results reported in Table 8, we can observe that our model is well-equipped both efficacy and efficiency in retrieval tasks. Notably, when our model is switched to the fusionfree (dual) pattern, it can still achieve a comparable performance compared with other methods while enjoy a much faster inference speed.
A.5 Visualization
We provide more examples of per-word visualizations of our fusion-based encoder finetuned on VL retrieval tasks, as shown in Fig. 6. The visualizations suggest that our model can focus on specific
regions of the image according to different words in text when conducts image-text matching.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
The ’limitation’ section on page 9.
7 A2. Did you discuss any potential risks of your work? This study is a fondamental work based on public datasets, and does not involve privacy, ethics and other dangerous related content.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 5.2/5.2/Appendix A.1
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The datasets used in this paper are widely used in corresponding fields.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 5.2/5.2/Appendix A.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 5.2/5.2/Appendix A.1
C 3 Did you run computational experiments? 5.3/5.4 Appendix A.3/A.4/A.5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 5.1.2 / Appendix A.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5.2 / Appendix A.2
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We report the average results of many experiments
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5.1.2/ 5.2 / Appendix A.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, autoregressive mechanisms will be consider to applied in model architecture so that the model can be directly used for generation tasks like image captioning. (2) There may be disadvantages of the model in region-level VL tasks such as Object Detection. The reason is that these tasks require images in high resolution and fine-grained annotations of bounding boxes, which are non-trivial in generic VLP settings. To solve this problem, exploring different levels of granularity between image-text pairs is a promising direction and will be considered as the future work