Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper extends the Contrastive Language-Image Pre-training (CLIP) model with a knowledge-based pre-training framework. The framework introduces knowledge-based objectives in the pre-training process and utilizes diverse types of knowledge graphs as training data.

Contributions:
- A framework to utilize knowledge graph to vision-language pretraining
- Continued pretraining to save computation resources
- Improved performance upon CLIP on vision-language multi-modal tasks as well as uni-modal tasks.
strengths_and_weaknesses: Strengths:

- The framework is intuitive and clearly presented.
- The improvement of the CLIP model is significant and promising.

Weakness:
- The lack of proper ablation study. For the vision-language or uni-modal tasks in the evaluation, not much reasoning is required to perform the task. So, it is important to answer the question: if the improvement in Knowledge-CLIP comes from the knowledge-based objective or just from more training data. An ablated baseline would be CLIP continued pretrained on the same datasets which Knowledge-CLIP was pretrained.
- Another ablation study should be on the three objectives to measure their contributions to the final result.
- The motivation in figure 1 is not accurate. The example only shows that CLIP fails to capture negation, not all semantics. From the same figure, we can see that CLIP performs well in differentiating cars from planes. Authors need to be more specific on their claims.
questions: - Equation 13 is confusing to me. Does it have a simple but meaningless solution that G^L(t) = Y(-,-,t)?

limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a vision-language pre-training framework that incorporates knowledge information by pre-training on multiple knowledge graph datasets (i.e., VisualSem, Visual Genome, and ConceptNet).
They unify all the possible triples of different modalities (e.g., image-relation-text, image-relation-image, text-relation-text), encode each modality with its corresponding CLIP encoder (i.e., Image encoder, Text encoder), and concatenate all the embeddings with special token <head>.
Then, multi-modal encoder (i.e., 4 layer transformer model) encodes it and uses embedding of <head> token as a representation.
For the training objectives, the paper utilizes three different objectives including entity-entity, entity-relation, and graph-entity loss.
The pre-trained model consistently outperforms baselines including UNITER, OSCAR, CLIP, ALBEF on various multi-modal tasks.
strengths_and_weaknesses: ### Strengths

1. It is interesting that how framework encodes multiple features of different modalities into one representation.
2. The performance shows the effectiveness of the approach.

### Weakness

1. Lack of analysis
    - Want to see the performance with each training objective for understanding the effectiveness of each objective. (e.g., without L_E2E, without L_E2R, without L_G2E)
    - Want to see the impact of L_KD, which is the loss term of KL distance between original CLIP and the proposed model.

2. Lack of intuition for each objective
    - E2E: Want to see the intuition behind exploiting contrastive learning between (head, relation)-masked triple and (tail)-masked triple.
    - G2E: Want to see the intuition behind applying contrastive learning between before and after graph-propagation on (tail).
questions: 1. Figure 2: L_E2L -> L_E2R ?
limitations: I really like the approach but the lack of analysis is the limitation of this paper.

I am willing to increase the score when more analysis about training objectives are provided.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In order to address the semantic alignment issue in multimodal tasks, this paper suggests the Knowledge-CLIP framework. Experiments reveal that the proposed approach is practical.

strengths_and_weaknesses: (a) Model
+ The use of "graphs" for multimodal pre-training is practical. In essence, the model benefits from having strong reasoning skills.
- However, the details of the model are not well presented, including data processing, sample visualization, etc.

(b) Experiments
+ The experiments in the article are adequate and verify that the learning graph can help the model improve multimodal learning.
- Which of these loss functions affects the results the most, and is it graph-based loss? If not, the model appears to be oriented east-west.
- In Figure 1, because the template "not a photo of ｛}" is inherently misleading. In addition, the car below is between yellow and green in color. As a result, this is not a particularly good example to demonstrate what the author is trying to say.
- There is not much evidence to demonstrate that the proposed methods can really help the model for modal alignment. It is difficult to verify this conclusion just by show only the SOTA results (without some newest SOTA baselines ) and lacking visual graphs/tables.
- Except for VQA, there is no obvious advantage of the effect of the model in this paper.

(c) writing
+ The reviewer likes the motivation in the Introduction section.
- There aren't enough ablation experiments to confirm that every design is rational. 
- Many earlier works, such as VLPs in e-commerce, also address the issue of alignment from different techniques.  The "co-occurence" is the key factor that is the same with this article (no matter the word-patch alignment, or using RoI-tags). The paper's references, however, are insufficient.

(d) others
- The SOTA performance was obtained by ignoring a large number of SOTA models. The authors  should give a justifiable explanation.


Update after rebuttal

Thanks for your detailed response. I stick to my score.
questions: Please see and answer "Strengths and Weaknesses" for the issues I mentioned. Additionally, the reviewer suggests the authors take the time to consider why Multimodal Pre-training requires Graph (or knowledge).  The authors claimed that focusing on the graph can help semantic alignment at first, but the second half of the paper, just shows that continue learning with graph will have some improvement to CLIP.

limitations: I recommend the authors give more robust explanations. Besides, this article still has many self-justifying explanations so far, and I hope the author will give more clear and reasonable illustrations. Overall, this article does a lot of experimenting, which is why I gave it a original POSITIVE rating. However, the article's flaws are really obvious, and I hope that the author will thoroughly improve them based on the comments. I will further check the rebuttal.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: - The paper proposes a new Contrastive Language-Image Pre-Training method that incorporates semantic knowledge graphs (knowledge-CLIP). The method builds on the existing CLIP architecture and introduces three KG-aware pretraining objectives on top: Entity-Entity loss, Entity-Relation loss, Graph-Entity loss. The authors experiment with this method using multiple knowledge graphs and show improved performance across multimodal, unimodal, and KG-based downstream tasks.
strengths_and_weaknesses: Strengths 
- The paper is generally well-written and easy to understand. 
- The problem they are solving (incorporating background semantic knowledge into language-image model) and their proposed method (KG-aware model architecture and training objectives) are well motivated. The idea of considering a triplet set that contains different forms, (Img, Rel, Img), (Img, Rel, Text), (Text, Rel, Text), is interesting.
- The experiments are solid (covers three types of KGs; covers both multimodal and unimodal downstream tasks) and shows that the proposed method makes moderate improvements over the baseline.

Weaknesses
- While the authors show improved numbers on benchmark datasets, it would be nice to also show and discuss how the proposed knowledge-CLIP model is *qualitatively* improving over the baseline CLIP. For example, in Intro and Figure 1, the authors motivates this paper by arguing that the baseline CLIP only captures text-image co-occurrence and fails to adjust for negation in text, etc. - is this issue solved in the proposed knowledge-CLIP model? Some existing work that combines text and KG (e.g. https://arxiv.org/abs/2104.06378) has done closely-related analyses such as adding negation or changing entities in text to see if the KG-augmented method can robustly handle them. It would be very interesting if the authors perform such analysis on the proposed knowledge-CLIP model that combines image, text and KGs.
questions: - Please find my main questions and suggestions mentioned in the "Weaknesses" section above. 
- Will the code/data/trained models be released?
- For Figure 1b, what is the expected prediction when changing "yellow" to "green"? I wonder "a photo of green __" may not really make sense for the given image because there is no green object in the image?
- For graph-based loss (Equation 11), do you compute Y(-,-,t) for all the triplets using knowledge-CLIP? Would  that be very expensive? Also, what is the size of the graph considered in the GNN - is it the entire KG or some subgraph?
- A suggestion for the presentation: In Figure 2, maybe make the font consistent across A (left panel) and B (right panel)?
limitations: - The authors addressed the potential negative societal impact (in Appendix)
review_title: null

Meta-review: metareview: Authors have improved their paper in such a way that multiple reviewers raised their original score. With varying degrees of excitement, the reviewers all agree that this paper should be accepted

Strengths:
- Interesting framework for multimodal pretraining integrating language, vision and knowledge graphs
- Strong empirical results on multiple benchmarks
- Ablation study demonstrates the benefit of each proposed component and objective
- Paper is well-written

Weaknesses:
- No consistent ones seem to remain, as the reviewers effectively responded to most of the concerns of reviewers in the rebuttal
confidence: Certain
Decision: Accept