Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone 
Authors: Zi-Yi Dou
Aishwarya Kamath
Zhe Gan
Pengchuan Zhang
Jianfeng Wang
Linjie Li
Zicheng Liu
Ce Liu
Yann LeCun
Nanyun Peng
Jianfeng Gao
Lijuan Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: vision-language pre-training
VQA
image captioning
object detection

Abstract: Vision-language (VL) pre-training has recently received considerable attention. However, most existing end-to-end pre-training approaches either only aim to tackle VL tasks such as image-text retrieval, visual question answering (VQA) and image captioning that test high-level understanding of images, or only target region-level understanding for tasks such as phrase grounding and object detection. We present FIBER (Fusion-In-the-Backbone-based transformER), a new VL model architecture that can seamlessly handle both these types of tasks. Instead of having dedicated transformer layers for fusion after the uni-modal backbones, FIBER pushes multimodal fusion deep into the model by inserting cross-attention into the image and text backbones to better capture multimodal interactions. In addition, unlike previous work that is either only pre-trained on image-text data or on fine-grained data with box-level annotations, we present a two-stage pre-training strategy that uses both these kinds of data efficiently: (i) coarse-grained pre-training based on image-text data; followed by (ii) fine-grained pre-training based on image-text-box data. We conduct comprehensive experiments on a wide range of VL tasks, ranging from VQA, image captioning, and retrieval, to phrase grounding, referring expression comprehension, and object detection. Using deep multimodal fusion coupled with the two-stage pre-training, FIBER provides consistent performance improvements over strong baselines across all tasks, often outperforming methods using magnitudes more data. Code is released at https://github.com/microsoft/FIBER.

Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone
Zi-Yi Dou'l, Aishwarya Kamath Zhe Gan Pengchuan Zhang Jianfeng Wang'
Linjie Li", Zicheng Liu', Ce Liu', Yann LeCun', Nanyun Peng", Jianfeng Gao Lijuan Wang!
"Microsoft *University of California, Los Angeles NNw York University
{ædou, viwletpeng//cs.... ucla.edu, edu, yaish, www.uleenuuuueeeee..oe. pengchuanzhang@fb.. com (zhgan,jiante ialanf,,innlll linjliizzint liuce, jfgao, , www.uaaajjmmiiccccc wicroooot... com
Abstract
Vision-language (VL) pre-training has recently received considerable attention. However, most existing end-to-end pre-training approaches either only aim tackle VL tasks such as image-text retrieval, visual question answering (VQA) and image captioning that test high-level understanding of images, only target region-level understanding for tasks such as phrase grounding and object detection. We present FIBER FFusion-In-the -nt-----------oboneee Backboneebasee transformERR) architecture that can seamlessly handle both these types of tasks. Instead of having dedicated transformer layers for fusion after the uni-modal backbones, FIBER pushes multimodal fusion deep into the model by inserting cross-attention into the image and text backbones, bringing gains in terms of memory and performance. In addition, unlike previous work that is either only pre-trained on image-text data on grainrrined data with box-level annotations, we present two-stage pre- training strategy that uses both these kinds of data efficiently:
VL model
(i) coarse- gaasseegrained pre-training based on image-text data; followed by (ii) fine-grained pre-training based image-text-box data. We conduct comprehensive experiments on a wide range of VL tasks, ranging from VQA, image captioning, and retrieval, to phrase grounding, referring expression comprehension, and object detection. Using deep multimodal fusion coupled with the two-stage pre-training, FIBER provides consistent performance improvements over strong baselines across all tasks, often outperforming methods using magnitudes more data. Code available at : thub. cww/micmmmccccceeec.ccc......... croooft/IBo...
1 Introduction
Inspired by the success language model pre-training 51, 42], coupled with the unification of architectures used in the NLP and computer vision communities [12, 4], lision-language pre-training
(VLP) [62, 45, 33, 6] has been receiving an increasing amount of attention. It has been proven that VLP can establish state-of-the-art performance visual question answering [3], visual reasoning [60], image captioning, and image-text retrieval [41]. The pre-training objectives commonly used for these tasks, such image-text matching, image conditioned masked language modeling and image-text constrastive learning, require multimodal understanding at the image level. Typically, this means the pre-training done using images at lower resolution (e.g., 384x384) making possible to scale up training by using large batch sizes.
Recently, it has also been shown that tasks such as image classification and object detection (OD), which have been traditionally viewed as vision-only tasks, can benefit from being cast as VL tasks [50, 25, 34, 26]. Inspired by MDETR [26], GLIP [34] reformulates standard classification- based OD as phrase grounding. This opens up the possibility to leverage VI.P for OD, and vice versa,
"Equal Technical Contribution
Project Lead
Work done while at Microsoft
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Ist-stage pre-training
2nd-stage pre-training
TTC
Alignment
OD HEAD
FIBER
MLM
FIBER
Localization
ITM
by person
youung
flrown
coarse-grained
fine-grained
1
fine-tuning
fine-tuning
VQA
Image Captioning
Image Text Retrieval
Detection
Phrase Grounding
What is the young person dring?
Several blue vents compground
A browa cat that is being brusked
Person Blcycle. Hair dryer. Bot... Shocs, Tort.
young penson swings but ball thrown by older
Figure 1: The proposed coarse-to to- fine pre- pre-training framework for vision- lasson-lanuuage tasks. We first perform soasse-grained pre- pre-training with image-text data VQA, image captioning retrieval tasks, and then perform fine-grained pre pre-training with imaee-text-bor data for phrase grounding and object detection tasks. The same FIBER architecture is used for both stages. OD: object detection. MLM: masked language modeling. ITM: image-text matching. ITC: image-text contrastive loss.
and this unification has led impressive performance on several established OD as well as phrase grounding benchmarks [49]. Since these tasks involve fine-grained image understanding between regions in the image and phrases in the text, and also require prediction of precise bounding boxes at the output, the pre-training typically involves using high resolution input images (e.g. 800x1.3333
Existing multimodal architectures typically do support both kinds of tasks. Specifically, the fully end-to-end VLP models such as ALBEF [32], METER [13], and SimVLM VLM [67] can achieve the state of the (SoTA) on image-level understanding tasks, but non-trivial extend them for region-level VL tasks because predicting bounding boxes is typically hard in end-to-end settings. On the other hand, MDETR [26] and GLIP [34] are designed to predict bounding boxes, but have not been shown to support tasks such image captioning and retrieval. Further, fine- fine-graines pre- training not only requires data with bounding box annotations that cumbersome to acquire, but the requirement of high input image resolution makes pre- traininii very costly, especially when using standard Transformer architectures [63] that have quadratic complexity in the size the image. A natural but challenging question arises: can we have a unified frumework for efficient VL pre-truining that benefits both image-level and region-level VL. lasks (e.g., both VQA and OD)?
We answer this question by proposing two ideas: (i) novel model architecture that can handle various types of tasks and pre-training strate- gies (high and low resolution inputs, image and region level outputs) more efficiently than pre- vious work (see Section 3.1 and 4), and (ti) two-stage pre-training pipeline.
FFN
FFN
M
XM
Cross-Ati
Cress-Att
Self-Aff
Self-Aft
In terms of architecture, we present FIBER, shown in Figure 2, which performs deep mul- timodal fusion in the backbone. Specifically, instead of having a few dedicated transformer layers on top of the image and text encoders for is commonly done in previous work [36, 6, 13, 26, 34]), we propose directly insert cross- crossaatennionn modules into the image and text backbones. Additionally, we support the ability to switch between a dual encoder (for fast image retrieval) and fusion encoder (for VQA and captioning) readily, by switching on off the cross-attention modules. With the same model architecture, by simply adding an object detection
FEN
FFN
NN Mix
K&NG-
Self-Att
Sei-Att
Vision Input
Language Input
fusion (e.g.,
Model architecture for FIBER. Swin former is used the image backbone, simplified here for illustration purposes.
Figure 2:

---

Fgb
FiN
Vaice Input
Language Inpul
Valua Input
Lunguage
Video taput
Languaze
Valua Input
Lunguazz Iaput
(a) VQA/Visual Reasoning
(b) Retrieval
(c) Captioning
(d) Grounding/OD
Figure 3: FIBER be readily adapted to various downstream VL tasks, ranging from VQA, image captioning and retricval, phrase grounding and object detection (OD).
head (e.g., Dynamic Head [9]) on top, FIBER can be readily extended to visual grounding, referring expression comprehension and (open-vocabulary) OD tasks well.
By considering the nature different VI. tasks, FIBER pre-trained with a coarse-to-fine to-fine two-stage pipeline, as detailed in Figure 1. Specifically,
During coarse-graine pre-training. FIBER takes low-resolution x384)384) images as input, and pre-trained with image-text matching, masked language modeling, and image-text contrastive losses, used in previous work [13, 66, 64]. The pre-trained model can then be directly finetuned for VQA and image captioning tasks (Figure 3a and 3c). By switching off the cross-attention modules, FIBER also automatically functions as a dual encoder for fast image-text retrieval (Figure 3b).
the coarse pre-trained model as initialization, in addition to randomly initialized parameters for the OD head. At this stage, the model takes high- resolution x800 1,333) images as input, and pre-trained with bounding box localization loss and word-region alignment loss, as used in GLIP [34]. We use image-text-box data with ground-truth box annotations for pre-training and the model can be directly fine-tuned for grounding and detection tasks (Figure 3d).
During fite-grained pre-training, FIBER
Compared fine-grained pre-training, coarse-grained pre-training is easier scale up, it only requires paired image-text data which can be easily harvested from the web. Crucially, we show that re-using all the parameters from our coarse-grained pre-trained model for fine-graines pre-training alleviates the requirement for large amounts of bex-level annotated data. In our experiments, we show that on fine-grained tasks such as Flickr30k Entities, FIBER using coarse- coarse-grained pre-training achieves gains even over previous SoTA (GLIP [34]) that uses 25 x more box-level annotated images during the ane-grained pre-trate pre-training stage. We also show that our architecture is much more efficient in terms of training time on OD tasks, as compared to GLIP.
FIBER is the first end-to-end VLP model that can support VL tasks encompassing image-level and region-level outputs. We conduct experiments on VQAv2 [3], NLVR² [60]. COCO captioning [41], NoCaps [1], COCO and Flickr30k image-text retrieval [49], as well as on phrase grounding [49], referring expression comprehension [75], COCO and LVIS detection [17], and suite of 13 object detection in the wild datasets [34]. We show that our model can provide consistent performance
improvement over strong basclines (e.g., METER [13] and GLIP [34]) across tasks.
Related Work
VLP for Classical VL Tasks. ViLBERT [45] and LXMERT [62] were the first two methods to introduce using transformers for VLP. Since then, we have witnessed a boom of VLP methods [33, 30, 59, 73, 22, 71, 81, 38, 7, 35J. Early methods mainly focus on the use of pre-trained object detectors to extract image region features offline, such as UNITER [6], OSCAR [36], VILLA [15] and Vi9). More recently, end-to-end VLP methods that use the image directly as input have become popular. In these approaches, convolution networks or vision transformers [12] are used as the image backbone,
with additional transformer layers for modeling multimodal fusion [24, 23, 28, 68, 32, 64]. Prominent examples along this line include ViLT [28], ALBEF [32], Sim VLM [67]. METER [13], X-VLM [77]
and BLIIP [31]. These models have achieved the current SoTA on major VI. benchmarks such as
VQA and image captioning. However, they cannot be directly used for tasks such as object detection.
3

---

Model
VQA
O(n m) Retrieval'
Captioning Grounding
End2End
ViLBERT [45], LXMERT [62], UNITER [6]
OSCAR [36]. VinVL [79]
X
PixeIBERT [24], CLIP-VIL [57], VILT [28]
X
X
X
CLIP [50]" ALIGN [25]
X
V
X
X
VI-T5 [7]
X
X
METER [13]] SiM [67]
X
ALBEF [32], FLAVA [58], VLMo [66]
V
X
X
X
BLIP [31], CoCa [74]] Flamingo [2]
J
X
X
MDETR [26], GLIP [34]
UniTAB [70], X-VLM [77], OFA [65]
X
FIBER
Table 1: Comparison among different VLP models. FIBER is the only VLP model that can support all tasks considered. (f) VQA is used as representative VL classification task. (#) O(1 + m) retrieval denotes model backbones process inputs O(n + m) times given n images and m text sentences during image-text retrieval. (+) Here, we mainly focus on what tasks CLIP can be directly used for.
VLP for Vision Tasks, Recently, it has been shown that image-text data can be used to learn image encoders from scratch [10, 54J. By performing large-scale contrastive pre-training, CLIP [50] and ALIGN [25] display strong zero-shot image classification capabilities. While these models mainly tackle image-level understanding tasks, MDETR [26] extends the end-to-end OD model DETR [4], and uses contrastive learning along with an alignment loss to learn correspondences between image regions and text phrases, opening up the possibility to tackle tasks such phrase grounding and long-tailed OD using VL models. This has inspired many follow-up works to further enhance the pre-training [37, 72, 46, 69], among which GLIP [34] shows that OD also be cast VI. task (i.e., phrase grounding). However, it has not been shown how traditional VL tasks such as VQA, captioning and retrieval can be well supported in GLIP [34] and MDETR [26].
Unified VL Modeling. There have been few recent attempts that try to develop unified VI. models. VL-T5 [7] unifies VI. tasks as text generation; however, pre-trained object detectors are used for image feature extraction, so the model cannot be end-to-end pre-trained. UniT [20] proposes a multimodal multi-task framework with unified transformer; however, it can only support VQA and object detection tasks, but not captioning and grounding. GPV [18] proposes a general-purpose vision system, and FLAVA [58] presents VL system similar METER [13]: however, they did not evaluate on grounding and detection tasks, and their performance on other VL tasks is still far from SoTA. TA [70] and OFA [65] reformulate grounding as sequence generation task, by borrowing ideas from Pix2Seq [5]. However, these approaches have not been demonstrated to work standard OD benchmarks, and also cannot be used dual encoders for fast image retrieval. Our model is the first work that can support not only VQA, image captioning and O(n + m) retrieval, but also visual grounding and object detection, with impressive performance across all tasks. A detailed comparison is provided in Table 1.
3 Method
In this section, first describe the proposed model architecture in Section 3.1. We then illustrate our two-stage pre-training paradigm in Section 3.2, followed by fine-tuning strategies for all the tasks supported by FIBER in Section 3.3.
3.1 Fusion in the Backbone
Feedforward
The architecture of FIBER is shown in Figure 2. Different from models that stack a modality fusion module on top of the vision or language backbones [6, 13], we insert multimodal fusion inside the backbones, and include a gating mechanism for the cross-modal layers (shown in Figure 4). Specifically, at each encoding layer, we have:
Cross-Atiention
Self-Attention
2.K
ã = SELF- SELF-ATT(Z),
Figure 4: Illustration of performing fusion in the backbone. (ar, y) are the (image, text) or (text, image) representations, and is learnable scalar.
+ ã CROSS-ATT(E.
œ x + FFN

---

where is learnable parameter initialized to O. For simplicity, we insert the same number of cross-attention layers into the vision and language backbones.
By inserting cross-attention layers with the gating mechanism, we enable cross-modal interactions without affecting the original computational flow of the backbones the beginning of model training. Also, we can easily switch off the interactions by setting a to 0, and the backbones can be used in the dual-encoder setting. addition, compared to stacking a large number of transformer layers on lop of the backbones, our approach of inserting cross-attention layers is relatively light-weight and thus more memory- -effeerefffiiint.... To illustrate, both GLIP [34] and METER [13] use an additional 10M modality fusion parameters for a base-size model, while FIBER only adds about 26M parameters.
During training, the fusion module of FIBER only consumes half of the FLOPs needed by METER (12.35 vs. 24.04 GFLOPs for one instance). We experimented with two other model variants for fusion in the backbone, the details of which are provided in Appendix.
3.2 Coarse-to-Fine Pre-training
We divide VL tasks into two categories based on whether or not need generate region-level outputs on the image side. While these two kinds of tasks are characteristically different, they both require fusion between the vision and language modalities, and we hypothesize that sharing as many parameters as possible between the model used for these two sets of tasks will be beneficial. Based on this motivation, we propose a two-stage pre-training paradigm, where we first pre-train models with image-level objectives on images at low resolution, and then perform further pre-tran pre-training with region-level objectives where the input images are at a higher resolution. In this way, the coarse- grained supervision from the first stage can provide good initialization for the second stage for all the shared parameters. FIBER with the same architecture (Swin Transformer [43] and RoBERTa [42]) is used as the backbone for both stages of pre-training.
Coarse-grained Pre-training. For tasks like VQA and captioning. it has been demonstrated [32, 13, 66] that masked language modeling (MLM), image-text matching (ITM), and image-text contrastive (ITC) objectives are helpful for ViT-based VLP models. Following previous work, we use all the three objectives during pre- pre-training. Specifically,
For ITC, the inserted cross-attention modules are switched off, so FIBER functions a dual encoder. Given a batch of N image-caption pairs, we first compute their representations with our vision and language encoders independently without modality fusion, and then maximize the similarities between N positive image-text pairs while minimizing the similarities between the rest N N negative pairs, via a contrastive loss.
For MLM and ITM, the inserted cross-attention modules are switched on, FIBER now functions as a fusion encoder. For MLM, we randomly mask 15% of the input tokens and the model is trained reconstruct the original tokens. For image-text matching, the model is given an image-text pair and predicts whether they are matched. Following VLMo [66], we sample global hard negatives based on the similarities computed from the above ITC loss.
Fine-grained Pre-training. Most existing VL architectures [6, 62, 26, 31, 65, 7] use vanilla transformers both for encoding the vision as well as language inputs. However, in contrast to tokens in text, the entities of interest in images do not all occur the same scale. Being able to accurately model the image at different scales is especially important for tasks such as object detection and phrase grounding. To handle this, it is typical in object detection literature use input images at higher resolutions (800x 1333), which becomes problematic when using vanilla transformers that scale quadratically in the length of the input sequence. As mentioned earlier, we use Swin Transformer [43] as our image encoder, which provides hicrarchical representations of the image while having linear complexity in the size of the image. We combine these multi-scale representations using an FPN [39] for object detection training. For -en-igri pre-training, we switch on the cross-attention modules, using FIBER
fusion encoder. This ensures that the image representations that are passed to the FPN are already text-aware, and is a crucial difference compared to GLIP [34], where the image-text fusion takes place in the object detection head. Once the text-aware image features are extracted by the Swin backbone and image-aware text features are extracted using RoBERTa [42], the image features after the FPN are fed to Hend [9] which predicts a set of regions. Just as in [34], we compute the dot product between the image region features RTA and the contextualized
5

---

token representations Tía to compute the grounding score:
TA. A = (I,T, TAA = OD-HEAD((Ta). SCROUNDING = RIA TiA
(2)
where RIA represents regions that are text aware, produced using the OD-Head that takes input IIA. which are image representations that are already text-aware and Tia are the text features that have already attended the image features. The typical object detection model has a classification head that predicts the label of the object, and localization head that predicts the bounding box. We follow GLIP [34] by substituting the classification head with the grounding score SGroUNDING- The localization loss is composed of two parts: centerness loss and GloU loss, which are used to supervise the box prediction. Taken together, FIBER learns the correspondence between regions the image and phrases in the text, making it possible to tackle tasks such as phrase grounding and object detection using the same framework. We use ATSS framework [80] in our paper, but our method can be combined easily with other object detectors such as Faster-RCNN [52] and RetinaNet [40] as well.
3.3 Adaptation to Downstream Tasks
We now describe how we adapt FIBER to different downstream tasks as depicted in Figure 3.
For VI. classification tasks such as VQA, we use FIBER as fusion encoder. Specifically, the top M layers the vision and language backbones interact with each other and produce multimodal representations. The final layer representations of the two modalities are concatenated together to generate the final outputs for tasks such as VQA and visual reasoning.
For retrieval tasks, we switch off the inserted cross-attention modules use FIBER as a dual encoder for fast image-text retrieval.
For captioning, we adapt FIBER by only keeping the image-to-text cross-attentions and using causal masks in the decoding side. The representations of the final image encoding layer fed into the cross-attention modules. In this way, the model is turned into a seq2seq model [61, 8] and performs captioning in an auto-regressive way.
For phrase grounding, object detection and referring expression comprehension, we use FIBER as fusion encoder, and the OD-Head introduced during fine-grained pre-training receives image features that already language aware due the multimodal representations extracted by FIBER. The pre-trained model is directly used without any modifications for these tasks.
Experiments
Pre-training Datasets. Following previous
work [6, 28, 32, 13, 64, 66). we perform coarse- grained pre-training on COCO [41]. Concep- tual Captions [56]. SBU Captions [47]. and Vi- sual Genome [29]. The four datasets consist of about 4M images in total. For fine-grained pre- training, we use two data sources: data curated by MDETR [26] after removing the COCO im- ages, and the Objects365 [55] detection dataset, together consisting of about 0.8M images. We ensure that we exclude any data that exists in the validation or test splits of downstream tasks.
Type of Fusion
COCO
GPU-hours V100 (32GB)
Sec/Iter
Va12017
No Fuse
53.9
511
1.31
GLIP-B [34]
54.6
840
2.14
FIBER-B 54.5
540
1.38
Table 2: Object detection on COCO [41], without vision- language pre-training. We initialize the text encoder and image backbones using a pre-trained RoBERTa and Swin transformer pre-trained on ImageNet222. Our proposed FIBER model achieves the same performance as GLIP [34] while taking much less time to train.
Architecture. We adopt Swin-Base [43] and RoBERTa-Base [42] as our vision and text backbones, which are initialized with weights from uni-modal pre-training. We insert cross-attention blocks into the top 6 blocks of the vision and text encoders. The input resolution is 384 x 384 for coarse- graiined pre-training and 800 x 1, 1.333 for fine-graing -neiner pre- pre-traimine training. Using a hierarchical vision transformer enables us to efficiently tackle these high resolution tasks, which would be expensive in models such as BLIP [31] that rely on the vanilla transformer architecture. In METER [13], which does explore using a Swin transformer as the image encoder, the multi-modal fusion occurs in layers specifically designed to align the modalities, only after the image and text features are extracted from the uni-modal backbones. This in contrast to approach where the hicrarchical image features that are used in the FPN for fine-graine training

---

#Pretrain Images
VQA22
NLVR
Flickr30k
COCO
Model
test-dev test-std dev test-P IR@1 TR@1 @R@1 TR@1
Base-size models pre-trained on COCO. VG. SBU, and CC datasety
UNITER-B B [6]
4M
72.70
72.91
77.18
77.85
72.5
85.9
50.3
64.4
VILLA-B [15]
4M
73.59
73.67
78.39
79.30
74.7
86.6
UNIMO-B [35]
4M
73.79
74.02
ViLT-B [28]
4M
71.26
75.70
76.13
64.4
83.5
42.7
61.5
ALBEF-B [32]
4M
74.54
74.70
80.24
80.50
82.81
94.31
56.8"
73.1
VLMo-B [66]
4M
76.64
76.89
82.77
83.34
79.3
92.3
57.2
74.8
HETER-Swin-le [13]
4M
76.43
76.42
82.23
83.47
79.02
92.4
54.85
72.96
X-VLM [77]
4M
78.22
78.37
84.41
84.76
86.9
97.01
63.4"
81.2
Models pre-trained on more data and/or with larger size
VLMo-L [66]
4M
79.94
79.98
85.64
86.86
84.5
95.3
60.6
78.2
BLIPeageaa [31]
129M
78.25
78.32
82.15
82.24
87.51
97.2
64.1"
81.2
SimVLM-B B [67]
1.8B
77.87
78.14
81.72
81.77
SimVLM-H H [67]
1.8B
80.03
80.34
84.53
85.15
FIBER-B
4M
78.55
78.46
84.59
85.52
81.44
92.90
58.01
75.38
Table 3: Results on VL classfiication and retrieval. We also include models pre-trained on more data and/or with larger size. FIBER and VLMo use dual encoders for retrieval. (f) ALBEF, X- VLM, and BLIP first use its dual encoder to obtain top-k candidates, and then use its fusion encoder to re-rank the candidates. Our retrieval results with re-ranking are provided in Table 4. All the other models use fusion encoders.
are already language aware, due to the multi-modal fusion being in the backbone. This also lets us avoid adding additional "language-aware deep fusion layers" [34] as part of the OD head in GLIP, resulting in 1.5x faster training while maintaining performance as shown in Table 2. While in principle would be possible use the image features extracted by METER's backbone for object detection, it would be necessary as in GLIP to add additional layers to make the visual features "language-awe aware" for good detection performance, especially on datasets with limited training data and with rare and infrequent objects.
Implementation Details. We perform coarse-grained pre-training for 100k steps with 4,096 batch size on 64 A100 GPUs We use Adam [44] with the peak learning rates of le-4 for the backbones and 5e-4 for the cross-modal parameters. We use linear warmup over the first 1k steps and linear decay. For fine-grained pre-training, we train for 800k steps on 64 V100 GPUs, with batch size of 64. We use a learning rate of le-5 for the language backbone, and le-4 for the rest of the model with a weight decay of 0.01. We use linear warmup over the first 2k steps and then a constant learning rate, with two learning rate drops by factor of 10 67% and 89% of the total number of steps.
4.1 Results on Downstream Tasks
Vision-Language Classification. We first experiment on two representative VI. classification tasks, including VQAv2 [3] and NLVR2 [60]. As reported in Table 3, we achieve the best performance compared to other models in the same setting. It is worth noting that FIBER pre-trained with 4M images can achieve better performance than BLIP trained with 129M images and SimVLM trained with 1.8B images. The results indicate that introducing fusion modules into the backbone is an effective alternative to appending them on the top of uni-modal backbones.
Image-Texx Text Retrieval. In Table 3 we report image retrieval performance in the dual encoder setting, achieving competitive performance on both Flickr30k [49] and COCO [41] retrieval tasks. However, previous work has shown that fusion encoders obtain superior performance, albeit at the cost of efficiency as it involves feeding every image-text pair into the model. To illustrate, on the COCO test data, ranking the similarities between 5K images and 25K captions requires the model to process each image-caption pair 75M times, whereas the dual encoder model only needs 30K forward passes. As shown in Table 4, the fusion encoder can indeed surpass the dual encoder on retrieval tasks by large margin. In addition, directly ensembling the two models by summing their similarity scores together for each image-caption pair can bring huge improvements.
7

---

Flickr30k
COCO
Model
IR@1
IR@5
IR
10
TR@1
TR@5
TR
@
IR@5
I 10
TR@1
TR@5
TR@10
FIBER-ITC
$1.44
96.72
98.48
92.90
99.50
99.90
58.01
83.45
90.11
75.38
94.04
97.36
FIBER-ITM
84,10
97.54
98.88
95.10
99.60
99.90
59.03
84,04
91.03
75.14
93.88
97.36
FIBER-ITCHH Ensemble
90.96
98.44
99.14
96.00
99.70
100.00
69.73
90.66
94.59
80.10
95.60
97.98
ALBEF [32]
82.8
96.7
98.4
94.3
99.4
99.8
56.8
81.5
89.2
73.1
91.4
96.0
X-VLM [77]
86.1
97.4
98.7
96.8
99.8
100.0
63.1
85.7
91.6
80.4
95.5
98.2
1eoER-Rerank-10
90.94
98.16
98.48
85.80
99,60
99.90
68.71
87.69
90.09
79.66
95.34
97.36
PIBER-ReranK-20
90.10
98.38
99.14
95.90
99,80
100.00
69.32
89.52
93.33
79.78
95,20
97.66
Rerank-50
91.08
98.50 51)
99.37
96.
99.70
100.00
69.58
90.41
94.35
79.98
95.40
97.76
TIBER-Rerank-100
91.02
98.54
99.34
96.00
99.70
100.00
69.63
90.54
94.47
80.06
95.60
97.96
Table 4: Additional results on image-text retricval, where (i) the fusion encoder is used for retricval, or (ii) the dual encoder is first used to obtain top-k candidates, and then the fusion encoder is used to re-rank the candidates. We also provide full set of results on all evaluation metrics.
#Pretrain Images
COCO
NoCaps Val
NoCaps Test
Model
B@4
M
C
C
C
S
Models trained without CIDEr optimization
UFO-B [64]
4M
36.0
28.9
122.8
22.2
80.7
12.5
78.8
12.5
VITCAP [14]
4M
36.3
29.3
125.2
22.6
METER CLIP-B [13]
4M
38.8
30.0
128.2
23.0
X-VLM [77]
4M
39.8
133.1
VinVL-BB [79]
5.7M
38.2
30.3
129.3
23.6
BLIPC [31]
129M
39.7
133.3
109.6
14.7
LEMON-B [21]
200M
40.3
30.2
133.3
23.3
106.8
14.1
SimVLM-B [67]
1.8B
39.0
32.9
134.8
24.0
94.8
13.1
FIBER-B
4M
39.1
30.4
128.4
23.1
88.6
13.0
86.0
12.9
FIBER GOLD-G
4M
40.3
30.7
133.6
23.6
92.8
13.4
90.6
13.4
Models trained with CIDEr optimization
VITCAP [14]
4M
41.2
30.1
138.1
24.1
89.2
12.7
X-VLM [77]
4M
41.3
140.8
VinVL-B [79]
5.7M
40.9
30.9
140.4
94.3*
13.1"
92.5*
13.1"
LEMON-B [21]
200M
41.6
31.0
142.7
25.1
FIBER-B
4M
42.8
31.0
142.8
24.3
96.7
13.4
94.1
13.4
FIBER-GOLD-B
4M
43.4
31.3
144.4
24.6
99.2
13.7
97.1
13.8
Table 5: Results of base-size models on image captioning. We grey models pre-trained on larger magnitudes of data. Numbers with *** are obtained with constrained beam search during inference and without VLP. The complete results on all metrics are provided in Appendix. B@4; BLEU@ @4, M: METEOR, C: CIDEr, S: SPICE.
Further, we explore combining the strengths of both strategies by performing re-ranking as in [16, 31, 32]. Specifically, we first retrieve the top-k most similar instances using the dual encoder setup, and then add the similarity scores between the given instance and the top-k candidates provided by the fusion encoder the original scores to perform retrieval. From Table 4, we provides a balance between efficiency and performance, and that just re-ranking the top- instances achieve comparable performance with ensembling.
that this strategy
Image Captioning. We also evaluate our models on COCO [41] and NoCaps [1] captioning to test whether FIBER can be adapted to generation tasks. As in Table 5, FIBER can achieve better performance than models trained on the same data with and without CIDEr optimization [53]. We find
that integrating GOLD [48] into FIBER bring significant improvements, outperforming models trained with hundreds of millions of images. Notably, we establish the absolute state-of-the-at CIDEr scores on COCO for base-size models. Considering that FIBER is not pre-trained to perform captioning, the results demonstrate the strong generalization ability of FIBER.
Phrase Grounding, Our fine-grained pre- pre-training stage incorporates Flickr30k entities grounding data, and we achieve 87.4 on the Recall metric on the test set without any subsequent fine-tuning. This not only surpasses the current SoTA [34] using smaller sized model (Swin-B compared to their Swin-L.), but also uses 25x less fine-grained data. Our FIBER model is able to leverage the image-text gaasiegr pre- pre-training stage better, instead of relying expensive lseudo-labelling of large
8

---

Image
#Pretrain Images (fne-grained)
Flickr30k Val
Flickr30k Test
Model
Backbone
R @
R@5
R@ 10
R @@1
R@5
R@ 10
Visual-BERT [33]
ResNet-101
120k
70.4
84.5
86.3
71.3
85.0
86.5
MDETR [26]
EN-B5
200k
83.6
93.4
95.1
84.3
93.9
95.8
GLIP [34]
Swin-B
860k
85.7
95.0
96.2
86.1
95.5
96.4
Models pre-trained on more data and/or with larger size
GLIP [34]
Swin-L
27M
86.7
96.4
97.9
87.1
96.9
98.1
FIBER-B w/o C.G. VLP
Swin-B
860k
87.1
96.1
97.4
87.4
96.4
97.6
Swin-B
860k
86.2
96.0
97.6
86.5
96.4
97.7
Table 6: Phrase grounding performance Flickr30k entities dataset. We reproduce GLIP-Base sized results. and GLrgeLarge sized results taken from [34]. FIBER with Base size outperforms a GLIP-L which trained with 25x more fine- mne-grained data on the R@ I metrie. Further, FIBER without coarse grained VL pretraining outperforms GLIP-B when trained on the same fine-graine data.
Pre-training data
RefCOCO
RefCOCO+
RefCOCOg
Model
Im-Txt Im-Txt-Box
val
testA
testB
val
testA
testB
val
test
MDETR-B [26]
87.51
90.40
82.67
81.13
85.52
72.96
83.35
83.31
UNICORN-B B [70]
88.29
90.42
83.06
80.30
85.05
71.88
83.44
83.93
Models pre-trained on more data and/or with larger size
UNITER-L [6]
81.41
87.04
74.17
75.90
81.45
66.70
74.86
75.77
VILLA-L [15]
82.39
87.48
74.84
76.17
81.54
66.84
76.18
76.71
OFA-L [65]
90.05
92.93
85.26
84.49
90.10
77.77
84.54
85.20
FIBER-B
90.68
92.59
87.26
85.74
90.13
79.38
87.11
87.32
Table 7: Results on referring expression comprehension datasets.
web-scale corpus and subsequent high-resolution training on this generated fine-grained data in [34]. We also compare our approach without using any coarse-grame gaarse-graindd VL training (image encoder initialized to Swin-B weights from IN22K, and text encoder initialized pre-trained RoBERTa), and even in this setting, we are able to outperform a similarly sized GLIP model (GLIP-B), proving that fusion in the backbone is better at capturing fine-grained image-text understanding.
Referring Expression Comprehension (REC). con- trast to many previous works [6, 15, 45] that tackle the REC task by re-ranking object proposals provided by an off-the-shelf detector, the bounding box for the given referring expression. Us- ing proposed two stage pre-training, FIBER achieves better performance than current SoTA [65] that uses a Large sized model. Notably, on RefCOCOg [76], which contains much longer referring expressions than in Ref- COCO/ReCCOCO# [27], we observe more than 2 points boost over OFA-L. On the challenging testB split of both RefCOCO and RefCOCO+, FIBER outperforms current SoTA, OFA-L.
follow [26] to directly predict
20
A3
Figure 5: Few-shot results on the aggregated ODinW datasets.
Object Detection. We report FIBER results two stan-
dard object detection benchmarks, COCO [41] and LVIS [17], in zero-shot transfer' as well as fine-tuned settings in Table The LVIS dataset consists of long-tail of object classes, and is a popular test-bed for evaluating models on their generalization capabilities and robustness class imbalance. On the APr metric, which is the Average Precision on
Following [50, 78], we consider zero-shot transfer to mean that during pre-training we may have seen relevant data but it is not used for training for the task of interest. For instance, our coarse-graines pre-training includes some images from COCO (without any box information), but we do not have any COCO images in our fine- fine-grained training that
train the object detection head.

---

COCO Val 2017
LVIS MiniVal
Model
ODinW
AP
APr
APc
APf
AP
Prro-shot/Fine-uun
Zero-sho//Finetune
Kero-shot/Fine-tum
Mask R-CNN [19]
126.3
-34.00
-33,,9
/33.3
MDETR [26]
/20.9
-124.9
-24.3
124.2
GLIP-T [34]
46. 76.71
17.71
19.5/ -
31.0/-
24.9/ -
44.4/63.9
GLIP-B [34]
48. 1/57.0
17.0/31.3
23.9/48.3
35.9/56.9
29.1/51.0
44.8/65.8
Models pre-trained on more data and/or with larger size
GLIP-L [34]
49.8/60.8
28.2/
34.3/
41.5/-
37.3/-
52.1/68.9
FIBER-B
49.3/58.4
29.5/50.0
32.2/56.9
40.1/58.1
35.8/56.9
47.0/65.9
Table 8:
Zero-shot transfer and fine-tuning results for object detection COCO, LVIS and the average over 13 datasets for object detection in the wild. Detailed scores the 13 datasets presented in the Appendix. FIBER achieves better AP across board compared similarly sized GLIP-B, trained on the same amount of fine-graine grained data. On rare objects in LVIS, FIBER outperforms GLIP-L trained on 25x more fine ine-grained data. Results without coarse-grained pre-training are provided in the Appendix.
rare objects, FIBER outperforms GLIP-L which is bigger model and also trained with 25 more fine-grained data.
We also report zero-shot and fine-tuned results on a suite of 13 ODinW (object detection in the wild) datasets, spanning various domains and show consistent performance improvements over previous SoTA. Additionally, in Figure 5, we report few-shot results aggregated across these 13 datasets and show better data efficiency over GLIP-B trained with the same fine- one-grained data.
Ablation Study. In Appendix ?? and ??, we have provided detailed ablations that guided our architecture design, including ablations on fusion strategies, pre-training objectives, architecture for captioning, and additional results on open-ended VQA, and detailed few-shot ODin results. Due to the space limit, these ablations and additional results are only provided in the Appendix. Some important observations are summarized below. (i) Co-attention works similarly to merged attention for fusion the backbone. (éi) Adding gating parameter in co-attention allows the addition of fusion in more layers, and also gives better performance than merged attention. (fii) Adding co-attention in the last layers provides a balance between performance and efficiency. (iv) MLM, ITM with hard negative mining, and ITC are all important pre-training objectives for training FIBER-style models.
5 Conclusion
We propose (i) FIBER, novel architecture and (ii) a coarse-to-fine pre-training pipeline. We perform extensive experiments and show consistent improvements over strong baselines across diverse set of tasks. The results demonstrate the effectiveness of FIBER coupled with our pre-training strategy, by setting new SoTA scores while at the same time reducing the requirement of expensive box-level annotations. Future directions include scaling our models and extending our framework to other modalities.
The approach introduced in our work can potentially inherit undesirable societal biases that exist in our pre-trainin -reitrain data. Careful debiasing and filtering of data should be undertaken before real-life deployment of our work. Additionally, pre-trate pre-training can induce environmental costs, and minimizing these costs is avenue that we plan to explore further.
Acknowledgement
We would like thank Nguyen Bach, Jiayuan Huang, and Luis Vargas for their support. We also thank Wenhui Wang. Li Dong, Furu Wei, Bin Xiao, and Lu Yuan for their helpful discussions. We also thank Liunian Harold and Te-Lin Wu for their feedback on the manuscript. Aishwarya is supported in part by the National Science Foundation under NSF Award 1922658. Zi-Yi is supported in part by the DARPA Machine Common Sense (MCS) program under Cooperative Agreement N66001-19-2-4033 and NIH RO1HL152270.
10

---

References
[1] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: Novel object captioning scale. In ICCV, 2019. 3,
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yama Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint, 2022. 4
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In /CCV, 2015, 3,7,
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 1,4
[5] Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. In ICL,R, 2022. 4
[6] Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy, Faisal Ahmed. Zhe Gan, Yu Cheng, and Jingjing
Liu. UNITER: Universal image-text representation learning. In ECCV, 2020. 1, 2, 3, 4, 5, 6, 7.9
[7] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and and--anddaaaad lan-uag tasks text generation. In ICML, 2021. 3, 4,
[8] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn decoderr for statistical machine translation. In EMNLP, 2014. 6
Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In CVPR, 2021. 3,5
[10] Karan Desai and Justin Johnson.
VirTex: Learning visual representations from textual annotations. In
CVPR, 2021.
[11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In NAACI, 2019. 1
[12] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2021. 1,3
[13] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang. Lijuan Wang, Chenguang Zhu, Pengehuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. An empirical study of training end-to-end vision-and- ann-uagu transformers. In CVPR, 2022. 2, 3, 4, 5, 7,8
[14] Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang. Yezhou Yang, and Zicheng
Liu. Injecting semantic concepts into end-to-end image captioning. In CVPR, 2022. 8
[15] Zhe Gan, Yen-Chur Chen, Linjie Li. Chen Zhu. Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and- lnd-lane representation learning. In NeurlPS, 2020. 3, 7,9
[16] Gregor Geigle, Jonas Pfeiffer, Nils Reimers, Ivan Vulié, and Iryna Gurevych. Retrieve fast, rerank smart: Cooperative and joint approaches for improved cross-modal retrieval. Transuctions of the Association for Computational Linguistics, 10:503-521, 2022. 8
[17] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In CVPR, 2019. 3,9
[18] Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems. In CVPR, 2022. 4
[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017. 10
[20] Ronghang Hu and Amanpreet Singh. UniT: Multimodal multitask learning with unified transformer. In ICCV, 2021.
[21] Xiaowei Hu, 7he Gan, Jianfeng Wang, Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In CVPR, 2022.
[22] Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang. Lei Zhang, Jianfeng Gao, and Zicheng Liu. VIVO: Surpassing human performance in novel object captioning with visual vocabulary pre-training, In AAAI, 2021. 3
[23] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In CVPR, 2021. 3
[24] Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu, Fixel-BERT Aligning image pixels with text by deep multi-mods transformers. arXiv preprint, 2020. 3,4
11

---

[25] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V Le. Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, 2021. 1.4
[26] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR-modulated detection for end-to-end multi-modal understanding. In ICCV, 2021. 4,
[27] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014.
[28] Wonjae Kim, Bokyung, Son, and Ildoo Kim. ViLT: Vision-and-lanuuee transformer without convolution
or region supervision. In ICML, 2021. 3, 4, 6,7
[29] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. Visual Genome: Connecting language and vision using crowdsoureed dense image annotations. IJCV, 2017. 6
[30] Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-VI. A universal encoder for vision and language by cross-modal pre-training. In AAAI, 2020. 3
[31] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint, 2022. 3, 4, 5, 6, 7,8
[32] Junnan Li, Ramprasaath Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, 2021. 2, 3, 4, 5,5, 6, 7,8
[33] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. VisualBERT: A simple and performant baseline for vision and language. arXiv preprint, 2019. 1,3,9
[34] Liunian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training In CVPR, 2022. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
[35] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning. In
ACL, 2021. 3,7
[36]
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, I Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020. 3,4
[37] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjic Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. ICLR, 2022. 4
[38] Yehao Li, Yingwei Pan, Ting Yao, Jingwen Chen, and Tao Mei. Scheduled sampling in language pretraining with decoupled uncoder-decoder network. In AAAI, 2021. 3
[39] Tsung-Yr Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongic. Feature pyramid networks for object detection. In CVPR, 2017. 5
[40] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. TPAMI, 2020.
[41] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 1, 3, 6, 7, 8,
[42] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized hert pretraining approach. arXiv preprint, 2019. 1, 6,6
[43] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 5,
[44] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018. 7
[45] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViI.ERTT: Pretraining task-agnostic visiolinguistic
representations for iision-and-languree
tasks. In NeurlPS, 2019. 1,3,4,9
[46] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. SLIP: Self-supervision meets language- image pre-training. In CVPR, 2022. 4
[47] Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2Text: Describing images using 1 million captioned photographs. photograass
In NeurlPS, 2011. 6
[48] Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In ICLR, 2021. 8
12

---

[49] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k Entities: Collecting region-to-hraae correspondences for richer image-to-senlence models. In /CCV, 2015. 2,
[50] Alec Radford, Jong Wook Kim, Chris Hallaey, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICMI., 2021. 4,
[51] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 2020.
[52] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. TPAMI, 2016. 6
[53] Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR. 2017. 8
[54] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, 2020.
[55] Shuai Shao, Zeming Li. Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Ohjects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019. 6
[56] Piyush Sharma, Nan Ding, Scbastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
[57] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-language tasks? In ICLR, 2022. 4
[58] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In CVPR. 2022. 4
[59] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre-training of generic visual-linguistic representations. In ICLR, 2019. 3
[60] Alane Suhr, Stephanie Zhou, Ally Zhang, Tris Zhang, Huajun Bai, and Yoav Artzi, A corpus for reasoning about natural language grounded in photographs. In ACI,, 2019. 1,3,7
[61] Ilya Sutskever, Oriol Vinyals, and Quoe Lc. Sequence sequence learning with neural networks. In NeurIPS, 2014.
[62]
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In EMNLP, 2019. 1,
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017. 2
[64] Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, and Lijuan Wang. UFO: A unified transformer for vision- languala representation learning. arXiv preprint, 2021. 3, 6,
[65] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to- sequence learning framework. arXiv preprint, 2022. 4, 5,9
[66] Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. VLMo: Unified vision-language pre-training with mawaere--o-omddlll-e-eeeeeee arXiv preprint, 2021. 3, 4, 6.7
[67] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. VLm Simple visual language model pretraining with weak supervision. In ICLR, 2022. 2, 3, 7.8
[68] Hongwei Xue, Yupan Huang, Bei Liu. Houwen Peng. Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing inter-modality: Visual parsing with self-attention for vision-language pre-training. In NewrlPS, 2021. 3
[69] Jianwei Yang, Chunyuan Li. Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning image-text-label space. CVPR, 2022.
[70] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Crossing the format boundary of text and boxes: Towards unified vision-language modeling. arXiv preprint, 2021. 4, 9
[71] Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. TAP: Text-aware pre-training for text-vqa and text-caption. In CVPR, 2021.
[72] Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: finenerained interactive language annuage-image pre-traing me--rrining. In ICLR, 2022.
13

---

[73] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE- ViL: Knowledge enhanced vision-language representations through scene graph. In AAAI, 2021.
[74] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Scyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. arXiv preprint, 2022.
[75] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L. Berg. MAtNee::
Modular attention network for referring expression comprehension. In CVPR, 2018. 3
[76] Licheng Yu, Patrick Poirson, Shan Yang. Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV, 2016. 9
[77] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In ICML, 2022. 3, 4, 8,8
[78] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and
Lucas Beyer. LiT: Zero-shot transfer with locked-image text tuning. In CVPR, 2022. 9
[79] Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang. Lijuan Wang. Yejin Choi, and Jianfeng Gao. VinVL: Revisiting visual representations in vision- laniua- models. CVPR, 2021. 3, 4.
[80] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. Bridging the gap between anchor- based and anchor-free detection via adaptive training sample selection. In CVPR, 2020, 6
[81] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision- language pre-training for image captioning and vqa. AAAI, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
14

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "e0lQlR-XgN0", "reviewer": "Reviewer_SULZ", "summary": "This paper proposes a new vision-language model that can deal with both  VL tasks and region-level understanding tasks.\nThe proposed model, Fiber inserts cross-attention to the model to learn multimodal fusion.\nThe pretraining leverages two kinds of data: coarse-grained image-text pairs and fine-grained image-text-box data.\n", "questions": "Why do not compare the performance with X-VLM, which is a recent vision-language pre-trained model?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\t1. The model is novel. The proposed fusion in the backbone is a good architecture by inserting cross-attention layers. I like this solution which is more light-weight than previous fusion modules. The authors demonstrate this in terms of parameter size (Fiber adds 26M parameters while METER adds 110M).\n\t2. Performance is good. It outperforms METER by around 2/% on VQA and NLVR. Also, it has a good performance on region-level understanding tasks.\n\nWeaknesses:\n\t1. My biggest concern is a recent work X-VLM [1]. Could the authors distinguish the proposed method and X-VLM model? The motivation to leverage both coarse- and fine-grained features is similar. And the authors did not discuss the differences in the model architecture and performance. Especially, X-VLM is a one-stage model and I feel like the one-stage model is easier to implement and use.\n\n\n[1] X-VLM: Multi-Grained Vision Language Pre-Training (ICML 2022)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XRtCzJDthe5", "reviewer": "Reviewer_Gvbu", "summary": "This paper proposes FIBER, a multimodal transformer architecture with two sets of cross-attentions: image-to-text and text-to-image. The model is pre-trained with two stages. The first stage uses image-text pairs to learn vision-language interaction with some standard pre-training objectives. The second stage uses images with bounding box annotations to learn language-grounded localization. The pre-trained model is evaluated on a diverse set of vision-language and object detection benchmarks.", "questions": "- The coarse-to-fine two-stage pre-training is claimed as a key contribution. However, from the phrase grounding experiments in Table 6, it seems that coarse-grained pre-training does not help much. Does coarse-grained pre-training help the other two localization tasks, i.e., REC and OD?\n\n- When training with object-level bounding box annotations, how does FIBER convert object names into texts? It is an important piece of information that need to be discussed in the paper.\n\n- The paper claims that the image-text fusion mechanism from FIBER is better than that of GLIP. Could the authors provide some evidence to support this claim?\n\n- FIBER can support training and inference with higher-resolution images due to the Swin Transformer. However, this also increases the training and inference time. It would be good if the authors could make a thorough comparison between Swin Transformer and ViT for vision-language tasks. Similarly, it is unclear how much gain does RoBERTa provide over BERT.\n\n", "limitations": "The authors have addressed the potential social impact. It would be also good to see some discussions on the technical limitation of the proposed method.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is well-written and very easy to follow. The paper clearly states its improvement over existing methods: an additional text-to-image cross-attention mechanism and an additional stage of fine-grained pre-training. The paper does a good job to highlight the architectural differences between FIBER and existing models. The evaluation covers a sufficient number of downstream tasks.\n\nWeaknesses: I have two major concerns about the paper as explained below.\n1. The paper puts together multiple techniques to build a model. However, most of the techniques have been proven to work well by existing literature. Therefore, the insight from this paper is not clear to me. The paper does not have enough ablation experiments to analysis what effect each technique brings, and how the proposed architectural design compares with some alternative design choices. To put it short, I find it hard to learn something from this paper that can inspire future research.\n2. The paper needs to be more careful when claiming better performance over existing models. The number of pre-training images is an important factor for pre-training performance, but not the only factor. FIBER uses Swin Transformer and RoBERTa, which are more powerful unimodal backbones than some of the existing ones. From the appendix, it also seems that FIBER uses higher-resolution images when finetuning on the downstream tasks, compared to most existing models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NiFg9_DGDzO", "reviewer": "Reviewer_fMci", "summary": "This paper proposes to fuse visual and textual features with cross attention in the backbones for vision-language pre-training, which makes the model applicable to both high-level vision-language understanding tasks (e.g., VQA, captioning) and region-level image understanding tasks (e.g., object detection, phrase grounding). Moreover, the model is pre-trained in two-stage coarse-to-fine manner (i.e., first high-level VL tasks then region-level VL tasks). Experiments are conducted on a wide range of VL tasks to demonstrate the superiority of the proposed method.", "questions": "Pls check the weakness section.", "limitations": "The pre-trained model in this paper can not be directly transferred to all the considering VL tasks in zero-shot manner.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n(1)\tThe proposed model is the first VLP model that can be applicable to both high-level and region-level downstream VL tasks.\n\n(2)\tA two-stage pre-training strategy is proposed to pre-train the model in coarse-to-fine manner.\n\n(3)\tSuperior performances are achieved compared with SOTA methods on a wide range of VL tasks.\n\nWeaknesses\n\n(1)\tSome related works that jointly support VQA, Retrieval, and captioning are missing:\n[A] Unified vision-language pre-training for image captioning and vqa[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07): 13041-13049.\n[B] Scheduled sampling in vision-language pretraining with decoupled encoder-decoder network[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(10): 8518-8526.\n\n(2)\tAs shown in Table 5, it is better to indicate the type of vision/text backbone of each run in other tables (Table 3, 4, 6, 7), pursuing a fair comparison under same vision/text backbone.\n\n(3)\tThe ablation study about the effect of the number of layers to additionally fuse cross-modal features in the backbones is missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept