Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a model for image captioning, utilizing external knowledge with a graph-based model. The full model contains a transformer (a text tokenizer + a ViT based patch tokenizer), and a graph attention network to get the external knowledge from conceptnet 5.0. The paper claims to achieve state-of-the-art performance on smaller dataset, and can generalize to unseen object categories.

strength_and_weaknesses:  - The paper is clear overall.
 - The authors provides good ablation study in table 1 to validate their major claim.
 - It is interesting and novel to me to incorporate external knowledge to image captioning tasks, and achieve competitive results.

clarity,_quality,_novelty_and_reproducibility: I appreciate the ablation studies in table 1, but it is not explained in detail in the experiment sessions. E.g., I am curious to know how sensitive the model is to the quality of the knowledge graph. What about we train a transformer based text model and feed their embeddings to the model? Will it achieve similar results?

summary_of_the_review: I think the paper passed the threshold for acceptance, based on its novelty.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a Vision language model that can use the external knowledge graph ConceptNet. The model consists of a transformer model with a graph-based model to embed graph information. The authors claim the proposed model can outperform state-of-the-art models in image captioning.

strength_and_weaknesses: Strengths:
- The model design is simple and intuitive.
- The proposed method does not require a pretrained object detector.
- The authors provide some ablation studies.

Weaknesses:
- Experiments:
    - No qualitative results.
    - No analysis of how this added knowledge actually increases the "common sense" of the captioning model.
    - Some of the ablation experiments results are only described but not shown in any table. For example, the VinVL + knowledge graph result.
    - The authors only show performance on two tasks, whereas, for such a general vision language model, it is important to see the generalization ability on many different vision language tasks.
    - The result table of image caption retrieval is too simple. Should include more baselines and ablations.
    - The authors say the experiments show that GMML is helpful however, if we look at Table 1, for all methods with GMML, there is no method without GMML to compare to. This seems suspicious to me.
    - The numbers in the tables are weird and the baselines’ scores are far lower than the numbers reported in the original papers. The SOTA performance is lower than what was 5 years ago.
- Paper presentation
    - I am seeing many superscripts in which I am expecting footnotes. For example, for method C, and Oscar in Table 1. However, I don’t see any footnotes. Is that expected?
clarity,_quality,_novelty_and_reproducibility: Novelty: the novelty is limited because most of the components are borrowed from existing papers.
Reproducibility: the authors provide many implementation details; it is likely that it can be reproduced by an expert in the field.
summary_of_the_review: There are many problems in this paper as I described in the weaknesses, thus I suggest reject.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to identify semantic concepts that are not explicitly represented in the given image for caption generation by incorporating knowledge from an external knowledge graph (e.g., ConceptNet). Specifically, a ViT is trained to extract multimodal representations from image-text pairs with multiple pre-training objectives (i.e., Image Text Matching, Word Patch Alignment, Masked Toke Loss and Group Mask Model Learning), and the external information from a pre-trained knowledge graph built on the basis of ConceptNet is injected into the last layer of the ViT through cross attention. Compared with state-of-the-art models of vison-and-language pre-training, the proposed method achieves better captioning performances on MSCOCO.
strength_and_weaknesses: Strengths:

1.	This paper proposes to leverage useful information from an external knowledge graph for identifying underlying semantic concepts not represented in the images for caption generation.

2.	The proposed method has achieved better results in image captioning by training on a relatively small dataset compared with several state-of-the-art models of vision-and-language pre-training.

Weaknesses:

1.	It is unclear that the proposed model should be considered as a captioning model or a vision-and-language pre-training model. If it is a captioning model, it should be compared with state-of-the-art baselines of image captioning. Otherwise, it should be evaluated on more cross-modal understanding tasks like the other pre-training models. However, none of these is included in the paper.

2.	The idea of leveraging external knowledge graph for cross-modal understanding tasks is not new and has been adopted in other works like [R1-2]. The authors should discuss the differences between the proposed method and these works. Moreover, for me, the key idea between KG-BART and the proposed method is almost the same except that KG-BART is for NLP tasks.

[R1] Relational Reasoning using Prior Knowledge for Visual Captioning, arxiv 1906. 01290v1.
[R2] Concept Propagation via Attentional Knowledge Graph Reasoning for Video-Text Retrieval, ACM MM 2022.

3.	For efficiency, it seems unpractical to cross attention over all the nodes in the knowledge graph in the last layer of ViT.

clarity,_quality,_novelty_and_reproducibility: 1.	Clarity: Satisfied. Most of the details about the proposed method are clarified. 

2.	Novelty: Not novel enough. The idea of incorporating external knowledge graph into multimodal understanding tasks is not new. The model architectures and training objectives adopted in the proposed method are not new either.

3.	Reproducibility: Satisfied. Code is not available, but an expert can reproduce the results following the details in the paper.

4.	Quality: Satisfied. The paper is well organized. 

summary_of_the_review: I lean to negative for the reasons as follows: 

1) The idea of leveraging external knowledge graph for multimodal understanding tasks is not new and has been adopted in previous works. 

2) No new architecture designs or training strategies are proposed in the paper. 

3) Though the authors claim that the proposed model is a vision and language model which can learn multimodal image representations from image-text pairs, evaluations on more multimodal understanding tasks are not included in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose to pretrain a vision language transformer with external commonsense knowledge to improve on downstream tasks where external knowledge is relevant. They validate their approach on the downstream task of image captioning and image retrieval and show improvement when the model is trained with commonsense knowledge. 
strength_and_weaknesses: Strengths - 
The motivation and intuition behind incorporating external commonsense knowledge during pretraining is interesting .The authors also use  a detector free approach to image modeling thus training an end to end system. 

Weaknesses -

The role of GMML is not very well experimentally evaluated. What happens if GMML is not used at all? And the only losses used are ITM + MTL + WPA ? Is it not very convincing what advantages it brings when knowledge is incorporated . 

Qualitative visualization of what commonsense knowledge nodes were important for the downstream task ?

The contributions of the paper are not significant and not well illustrated in experimental results.


clarity,_quality,_novelty_and_reproducibility: The paper is well written and motivated. The  experimental results are not clear. Why is experiment E - not E+KG, assuming KG means adding knowledge graph to the pretraining. 
summary_of_the_review: The paper proposes an interesting idea but the contributions are not sufficient and the experimental results do not fully convey the addition of commonsense knowledge and its effetcs. Other downstream tasks such as VCR or VQA might be more reasonable the effect of commonsense knowledge on pretraining.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new Vision language model that combines the strengths of Transformers with graph-based models to convey external common sense knowledge. The motivation and intuition of the paper are interesting. However, the reviewers had many concerns about the contributions, experiments, ablation studies, and analysis of the results. The authors addressed some of them but the reviewers are not fully convinced.
Decision: Reject