Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new vision-language model that can deal with both  VL tasks and region-level understanding tasks.
The proposed model, Fiber inserts cross-attention to the model to learn multimodal fusion.
The pretraining leverages two kinds of data: coarse-grained image-text pairs and fine-grained image-text-box data.

strengths_and_weaknesses: Strengths:
	1. The model is novel. The proposed fusion in the backbone is a good architecture by inserting cross-attention layers. I like this solution which is more light-weight than previous fusion modules. The authors demonstrate this in terms of parameter size (Fiber adds 26M parameters while METER adds 110M).
	2. Performance is good. It outperforms METER by around 2/% on VQA and NLVR. Also, it has a good performance on region-level understanding tasks.

Weaknesses:
	1. My biggest concern is a recent work X-VLM [1]. Could the authors distinguish the proposed method and X-VLM model? The motivation to leverage both coarse- and fine-grained features is similar. And the authors did not discuss the differences in the model architecture and performance. Especially, X-VLM is a one-stage model and I feel like the one-stage model is easier to implement and use.


[1] X-VLM: Multi-Grained Vision Language Pre-Training (ICML 2022)
questions: Why do not compare the performance with X-VLM, which is a recent vision-language pre-trained model?
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to fuse visual and textual features with cross attention in the backbones for vision-language pre-training, which makes the model applicable to both high-level vision-language understanding tasks (e.g., VQA, captioning) and region-level image understanding tasks (e.g., object detection, phrase grounding). Moreover, the model is pre-trained in two-stage coarse-to-fine manner (i.e., first high-level VL tasks then region-level VL tasks). Experiments are conducted on a wide range of VL tasks to demonstrate the superiority of the proposed method.
strengths_and_weaknesses: Strengths

(1)	The proposed model is the first VLP model that can be applicable to both high-level and region-level downstream VL tasks.

(2)	A two-stage pre-training strategy is proposed to pre-train the model in coarse-to-fine manner.

(3)	Superior performances are achieved compared with SOTA methods on a wide range of VL tasks.

Weaknesses

(1)	Some related works that jointly support VQA, Retrieval, and captioning are missing:
[A] Unified vision-language pre-training for image captioning and vqa[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(07): 13041-13049.
[B] Scheduled sampling in vision-language pretraining with decoupled encoder-decoder network[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(10): 8518-8526.

(2)	As shown in Table 5, it is better to indicate the type of vision/text backbone of each run in other tables (Table 3, 4, 6, 7), pursuing a fair comparison under same vision/text backbone.

(3)	The ablation study about the effect of the number of layers to additionally fuse cross-modal features in the backbones is missing.

questions: Pls check the weakness section.
limitations: The pre-trained model in this paper can not be directly transferred to all the considering VL tasks in zero-shot manner.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes FIBER, a multimodal transformer architecture with two sets of cross-attentions: image-to-text and text-to-image. The model is pre-trained with two stages. The first stage uses image-text pairs to learn vision-language interaction with some standard pre-training objectives. The second stage uses images with bounding box annotations to learn language-grounded localization. The pre-trained model is evaluated on a diverse set of vision-language and object detection benchmarks.
strengths_and_weaknesses: Strengths: The paper is well-written and very easy to follow. The paper clearly states its improvement over existing methods: an additional text-to-image cross-attention mechanism and an additional stage of fine-grained pre-training. The paper does a good job to highlight the architectural differences between FIBER and existing models. The evaluation covers a sufficient number of downstream tasks.

Weaknesses: I have two major concerns about the paper as explained below.
1. The paper puts together multiple techniques to build a model. However, most of the techniques have been proven to work well by existing literature. Therefore, the insight from this paper is not clear to me. The paper does not have enough ablation experiments to analysis what effect each technique brings, and how the proposed architectural design compares with some alternative design choices. To put it short, I find it hard to learn something from this paper that can inspire future research.
2. The paper needs to be more careful when claiming better performance over existing models. The number of pre-training images is an important factor for pre-training performance, but not the only factor. FIBER uses Swin Transformer and RoBERTa, which are more powerful unimodal backbones than some of the existing ones. From the appendix, it also seems that FIBER uses higher-resolution images when finetuning on the downstream tasks, compared to most existing models.
questions: - The coarse-to-fine two-stage pre-training is claimed as a key contribution. However, from the phrase grounding experiments in Table 6, it seems that coarse-grained pre-training does not help much. Does coarse-grained pre-training help the other two localization tasks, i.e., REC and OD?

- When training with object-level bounding box annotations, how does FIBER convert object names into texts? It is an important piece of information that need to be discussed in the paper.

- The paper claims that the image-text fusion mechanism from FIBER is better than that of GLIP. Could the authors provide some evidence to support this claim?

- FIBER can support training and inference with higher-resolution images due to the Swin Transformer. However, this also increases the training and inference time. It would be good if the authors could make a thorough comparison between Swin Transformer and ViT for vision-language tasks. Similarly, it is unclear how much gain does RoBERTa provide over BERT.


limitations: The authors have addressed the potential social impact. It would be also good to see some discussions on the technical limitation of the proposed method.
review_title: null

Meta-review: metareview: This paper proposes a two-stage pretrain visual language model which can deal with both high level and region level downstream tasks.  Experiments show significant improvement SotA models.  Main concerns from reviews are some missing references while the author gave detailed comparisons in the responses.  Although reviewer Gvbu's opinion is still somewhat conservative, I think the novelty of the paper is clear and the comparison to the SotA is sufficient.
confidence: Certain
Decision: Accept