Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposed a pre-trainable generic representation for visual-linguistic tasks call VL-BERT. VL-BERT extend BERT by changing the input from subsequent sentence to image regions and modify the caption words has the additional visual feature embedding. The authors pre-train the VL-BERT on the conceptual caption dataset and Wikipedia and book corpus dataset, empirical results show that the VL-BERT achieve the SOTA performance on the VCR, VQA and refer expression tasks. 

As the authors mentioned in Table 5, pre-training the visolinguistic representation for vision and language tasks is very popular recently, and 5~6 similar works have appeared recently. One of the nice features I found on this work is it's joint train with text-only corpus and faster RCNN weight. While ViLBERT designs for easier extendable for other modalities, VLBERT is more focus on the representation learning on the vision and language, since the caption input also combines with the visual feature embedding. 

Overall the paper is well written and performs extensive experiments/ablations. There is some specific point that is not clear to me or needs further clarifications from the authors. 

1: The authors mentioned the improvement over tuning the visual parameters, I wonder what is the details on that? is the region proposal network's weight fixed? if not, how to avoid the shift on the proposal layer? Is the model still has the visual genome target or objective? Which layer is fixed/updated? and what is the optimizer and learning rate scheduler? 

2: I notice there is a change in the textual input which take visual feature embeddings. I wonder what is the performance without these features? What is the visual feature input for textual corpus? 

3: For the Masked RoI classification with Linguistic Clues, what if there are overlapped regions? what if the detection label from faster rcnn is incorrect? will this introduce any noise? 

4: For VCR tasks, it seems the VL-BERT_base w/o pre-training is performed similar compare to the with pre-training (only 0.7% lower on val of Q->A) I wonder what is the reason of this? Is this show the pre-training is not important for the VCR tasks? 

5: The VCR tasks also have the object bounding box correspondence, is VL-BERT take any of this supervision for input? If not, how does the VL-BERT learn the correspondence? 

6: For refer expression tasks, the VL-BERT_base is actually worse than ViLBERT on the detected regions. It's not a fair comparison since other models use bert-base model. 

Overall, I think this paper is well written and has solid experiment results. It will be great if the authors can further clarify the above questions. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: ### Summary:

This paper propose a new model for learning generic feature representations for visual-linguistic tasks by pretraining on large-scale vision and language datasets like Conceptual Captions and language-only datasets like BookCorpus and English Wikipedia. They demonstrate that the pre-training procedure can help improve performance on down-streaming tasks like visual question answering, visual commonsense reasoning. 

Overall I liked the design choices made in the presentation. Although the paper doesn't provide insights around what the representations have learned and how they differ from representations learned / used by existing methods, they have provided substantial evidence to suggest that pre-training helps in a lot of downstream tasks. 

Although it's hard to evaluate the paper without putting it in context with other concurrent works that have come out recently, I tried my best to evaluate the merits of the paper in isolation. 

### Strengths:

- The paper explores an interesting direction of learning generic feature representations for visual-linguistic tasks for down-streaming tasks. Traditionally, people learn feature representations from scratch for each downstream task which might not always be possible if the training data is limited.
- The paper does a decent job mentioning all the concurrent work in the space of learning multi-modal representations that have come out very recently. They distinguish the proposed method from existing work and also compare the performance of the proposed approach with concurrent work on downstream tasks showing performance on-par or better than existing methods.
- I liked some of the design choices made in the paper. (1)  Instead of training a separate  transformer network for each type of input segments (question, ans, caption, image, etc). This makes the model easily extensible to other tasks as long as the correct segment embeddings are used to identify different input sources. (2) They also use a separate embedding for visual features instead of a common embedding  for both language tokens and visual tokens.
- Unlike the pre-training task in concurrent work, the model was pre-trained not just on multi-modal datasets like conceptual captions but also on text-only corpus like BookCorpus and English Wikipedia. The authors claim that this leads them to learn better representations for longer sentences which they found useful for VCR task.

### Weaknesses:

- The authors claim that attention mechanism in cross-modal transformer by concurrent approaches is restrictive but doesn't give substantial evidence that this is true. What are the limitations for cross-modal attention mechanisms compared to a single transformer model as described in this paper.
- On the contrary, by having a cross modal architecture, they can pre-train each modality separately on unaligned data. For instance, the text only transformer can be trained using large text corpora similar to BERT while the image only transformer can be trained on big datasets like OpenImages, ImageNet etc
- While the paper gives a lot of empirical evidence that pre-training helps, it would have been interesting to develop an understanding of what the model is actually learning and how are these representations better than learning representations from scratch for each task. For instance maybe the authors can visualize attention similar to [1].

### Other questions:

- When training on text-only datasets, what is the input on Visual Feature Embedding since there are no associated images. The authors mention that for non-visual elements, the features are extracted on the whole image. It's still unclear what the associated visual features are for text-only datasets.
- One of the pre-training tasks is masked ROI classification but it assigns a hard label to each ROI feature. It might be interesting to instead try learn the entire probability distribution (the output of a pre-trained classifier) by either minimizing the KL-divergence or by using softmax with soft-targets.
- While the model was learnt on text-only data, as mentioned in the above section, will the model help from image-only datasets such as large-scale classification datasets?
- While the models are tested on vision-and-language datasets, will these generic representations also be useful for unimodal tasks?
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: # 1. Summary
The paper introduces a pre-training procedure for visual-linguistic representations. The model is an extension of BERT (with transformer backbone) to deal with visual input. Images are encoded using object detectors which regions are masked at pixel level. Experiments show state-of-the-art results on different downstream tasks.    

Strengths of the paper:
* State-of-the-art results on 3 vision-language tasks
      
The weak reject decision was mainly guided by the following two weaknesses of the paper:
* Clarity of the paper needs to be improved to make the readers understanding the details of the model (see point 2 below)
* Limited novelty: the paper is an extension of BERT to the visual domain (see point 3 below)

      
# 2. Clarity
The paper reads quite well, although some points need to be improved:
* How were words split in sub-words (Sec 3.2)?      
* "For each input element, its embedding feature is the summation of four types of embedding, ...": it is not clear how you sum embeddings. E.g., token embedding has 30k dimensions while image one has 2048 dimensions.
* "It is attached to each of the input elements, which is the output of a fully connected layer taking the concatenation of visual appearance feature and visual geometry embedding as input" -> this is not clear; what output are we talking about? What is the geometry embedding? I suggest to describe the two features first and then say at the end of the paragraph that the representation is the concatenation.
* "For the non-visual elements, the corresponding visual appearance features are of features extracted on the whole input image" -> what is the intuition of having the full image here? Some terms do not need to have an image associated (e.g., verbs or articles). Do you take care somehow of that?
* Once textual embeddings are masked by [MASK], the related visual embedding (whole image) is also masked? To my understanding the answer is no: what's the intuition of this?
* Segment embedding: is this important? This should be easy to show with an experiment in the ablation study of Table 4?
* It seems that there is a semantic asymmetry of input to the loss during training when considering only the text information (bookscorpus) and the image-text information (conceptual captions): how is training coping with this? Doesn't it make more sense to have 2 pre-training phases: first on text information only and then on image-text information?


# 3. Novelty and Motivation
The novelty of the paper is quite limited. It strongly relies on transformer networks and then recent success of BERT in the NLP domain. The proposal is an extension of these two ideas to visual domain.

Moreover, there is a body of concurrent work that is very similar to the proposed idea with slight differences (ViLBERT, VisualBERT, LXBERT, UNITER, B2T2), i.e., using transformers with masking operation on the RoIs. It is not clear what is the intuition related to the differences between the methods, i.e.
* Why one is better than the other; why should someone prefer this pre-training technique wrt others?
* Why a unified network (this work) is preferred wrt a two-stream one (ViLBERT, LXMERT)?
It seems that everything heavily depends on the experiments and empirical results obtained by trying many variants during the prototyping phase. It is missing a bit of understanding and intuition on the reasons why this technique should be used.


# 4. Experimentation
Experiments are the strength of the paper showing state-of-the-art results on 3 vision-language tasks. Some additional analysis is missing:
* If masking is conducted on the raw pixel, this makes training much slower since you need to perform inference many times. What is the impact in terms of accuracy? Did you carried out an experiment showing that it is better to mask raw pixels instead of conv maps?
* How long is the model trained for?
* What is the performance/accuracy on the pre-training tasks?
* How important is the segment embedding?
* Footnote 2 should be in the main text (Sec 4.1). It is too hidden, but very important to let the reader knowing about it.

review_title: Official Blind Review #3

Meta-review: The paper proposed a new pretrained language model which can take visual information into the embeddings. Experiments showed state-of-the-art results on three downstream tasks. The paper is well written and detailed comparisons with related work are given. There are some concerns about the clarity and novelty raised by the reviewers which is answered in details and I think the paper is acceptable.
Decision: Accept (Poster)