Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposed a cross-modal retrieval augmentation for the multi-modal classification task (VQA). The authors first introduce a transformer-based image caption retrieval architecture that achieves decent performance. Then, the authors proposed to use the retrieval model to retrieve relevant visual and textual information as augmentation. The proposed method experiment on 3 existing method (Visual Bert, ViLBERT, and Movie + MCAN) and show good improvements over the baseline model. 

My major concern about this paper is the lack of novelty and experiment comparison. The proposed image caption retrieval architecture is not novel at all. Most existing method (ViLBERT, UNITER, VLBERT etc.) has a similar transformer objective, while the image fine-tuning are from Pixel Bert. In the experiment section (Table 2), the author even didn't compare these methods. 

In terms of the VQA performance, pre-training on the conceptual caption actually hurt the performance of visual Bert and ViLBERT. Could the authors explain why a larger dataset can not help with the model? Is Cross-Modal Retrieval pre-training necessary for VQA?  

In terms of the speed, the deep fusion model for image retrieval is super slow, since the model need to calculate the score for each pair. What is the size of pool when computing the retrieved captions? What is the time complexity? 


review_title: Interesting paper, but the proposed method is not novel enough and problems in experiment comparison. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper explores a new direction, to utilize the searched results (image caption pair) to improve downstream multimodal learning tasks. They first pre-trained a cross-modal model using the contrastive learning on the image caption dataset. Then they use the pre-trained model to search the relevant terms for image or text input, and augment the searched results as the input for the downstream multi-modal tasks.

Concerns:

1. The authors claim that the trained alignment model, DXR, "achieves state-of-the-art image-caption retrieval performance" on COCO and MIRFlickr. However, plenty of cross-modal retrieval methods achieve better performance than DXR. For example, Oscar [a] and Unicoder-VL [b] could achieve significantly better than DXR on both COCO and MIRFlickr. In my mind, this could not be regarded as a novel contribution. Also, how different cross-modal pre-trained models perform as the DXR is also an interesting direction for exploration.

2. The authors claim that XTRA "achieved state-of-the-art performance", which is also not convincing. Oscar [a] and UNITER [c] achieve significantly better than XTRA on COCO VQA val set. Please carefully survey the literature and claim the contribution.

3. The visual-linguistic pre-training methods [a,b,c] also aim to distill the external knowledge from the large-scale image caption dataset to the downstream tasks, but the solution is to provide a pre-trained model as the initialization which is pre-trained on the large scale image-caption dataset. These methods should also be surveyed in the literature, and may be compared in the experiment part.

[a] Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks, ECCV 2020

[b] Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training, AAAI 2020

[c] UNITER: UNiversal Image-TExt Representation Learning, ECCV 2020
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors present a method to use unstructured external knowledge sources to improve visual question answering and image-caption retrieval. The proposed method can achieve somewhat improvement for visual question answering, but drop the performance for image-caption retrieval with a more complex model. Some concerns are as follows:

1. The authors claim that the proposed method achieved state-of-the-art performance on both COCO and Flickr30k  image-caption retrieval. However, their retrieval scores are lower about 10 than the state-of-the-art counterparts, such as TERAN. The statement is not correct.
2. Although the authors stated the proposed method uses raw images as input, the adopted backbones (i.e., image/text encoders) should be frozen to extract the features for the following components in their pipeline, which is similar to the other feature-based methods (e.g., TERAN) that also can be seen as freezing their backbones (e.g., Faster R-CNN) during their training and inference stages. Thus, the inputs between the proposed method and other methods have no essential difference. What is the significance to design such a much more complex model for image-caption retrieval? What are the advantages of the proposed method comparing prior superior methods? I am confused that if it is worthy to adopt such a complex model with worse performance.
3. It is interesting to see that the proposed method could improve the performance of VQA. However, Table 3 does not give us a throughout comparison. There are many results missed in the table, such as different training types for Flickr30K, some results for Movie+MCAN, etc. From the results, we also could draw that the improvement of the proposed method is very limited for a good VQA method, i.e., Movie+MCAN with Vanilla. The experiments could not significantly demonstrate the significance and advantages of the proposed method.
review_title: The advantages and significance of the proposed method are unclear and confusing.

Meta-review: The paper discusses the problem of how to augment cross-modal retrieval for the task of multi-modal classification -- it uses image caption pairs to improve downstream multimodal learning, and shows improvement in the task of visual question answering. However, the paper has the following weaknesses: (a) lack of novelty, (b) lack of thorough empirical evaluation, (c) the complex model did not give significant gains.
Decision: Reject