Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a pre-training framework, called VALM, to jointly train on image-text data. The novelty of this work, compared to previous works in similar field, is how the image-text pairs are created. While previous works use pre-curated image-text aligned pairs, this work instead uses images retrieved using text as a query and then jointly fuses them through attention layers. The claim is that this will help the model perform better on tasks requiring visual commonsense reasoning.
strength_and_weaknesses: **Strengths**:

-- The empirical results in the paper are quite strong and surpass pre-trained text models (GPT) as well as vision-language models (VisualBERT) on 4 reasoning task datasets and several language understanding/modeling tasks. While the differences in language-only tasks is small, the gains compared to baselines in visual reasoning tasks is large as claimed.

-- The proposed model (esp visual knowledge fusion component) is novel and elegant and simple, which will serve as motivation for following works.

-- The paper is clearly written and easy to follow.

**Weaknesses**:

-- The main weakness in the method is the use of frozen image retrieval component. The concern is that if this component is not end-to-end trained with the rest of the model, will the model quality be limited by the quality of image retrieval. While the reviewer acknowledges that empirical results show a large gap between CLIP and VALM on all tasks, it is worth wondering if the gains are due to embedding multiple images with text in VALM (compared to single image-text pair used originally in CLIP). An ablation with different values of k (number of images retrieved) will be helpful here.
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear, quality of writing and results is high. The method is novel and authors mention they will be releasing the code on github later.
summary_of_the_review: Overall, quite an interesting and novel idea in the space of vision-language models. The claims are verified by empirical results. VALM outperforming on tasks compared to other vision-language models is a strong result. The weakness related to understanding why this method works better than other vision-language models and whether frozen image retrieval will limit the method's effectiveness is something the reviewer would like to understand and engage with the authors further.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: 
This paper improves a language model's performance on pure language tasks about visual concepts by augmenting its internal representation with dynamically retrieved images.

(motivation)
Global context (external knowledge about entities, relations, etc.) has been incorporated into pre-trained language models (PLMs), but so far it has not been visual. This paper incorporates external visual knowledge into their PLM (called VaLM) to
1) prevent VaLM from hallucinating inconsistent statements and
2) give VaLM visual knowledge that is harder to obtain from text.

(approach)
VaLM is a standarder decoder style PLM, used like any standard PLM and trained on the CC-100 dataset (text only). The difference is that the second to last self attention layer is replaced with a Visual Knowledge Fusion Layer (VKFL). First the VKFL uses CLIP to retrieve K (4 or 8) relevant images from a database of 200M natural images (from LAION) where relevance is according to the text input VaLM has seen so far. Second the VKFL fuses these relevant images with VaLM's hidden state at the previous layer to produce the output hidden state - the tokens are the same, they just have additional visual knowledge.

(experiments)
Results show that
1. When asked to complete text prompts about the typical color, shape, and relative size of objects, VaLM outperforms both PLMs and pretrained vision-language models by a large margin.
2. VaLM outperforms baselines at pysical commensense QA (PIQA).
3. VaLM is at parity with and sometimes better than baselines at traditional language understanding and language modeling tasks.
4. Ablations show disabling image retrieval or retrieving random (not relevant) images hurts performance.
5. Ablations also justify design decisions in the attention mechanism.

VaLM improves language modeling performance by adding global image context internally.
strength_and_weaknesses: 
Strengths
===

Incorporating visual background knowledge in a language only model makes a lot of sense and seems novel. Furthermore, VaLM is a simple and effective realization of the idea in practice.

The writing is clear and straightforward, without frills.



Weaknesses
===

There are couple areas where the model should have been evaluted:

* There is limited analysis of the images retrieved by the retrieval module. Using the figure 1 example, the idea is that it will say the sky is blue because the retrieved images tend to be blue. Are the retrieved images consistent with its answer? That is, given that it says the sky is blue, does it actually retrieve images where the sky is blue? If the retrieved images are manually substituted with alternative images where the sky is green then does VaLM say the sky is green? It would be good to have a systematic evaluation of the retrieved images, but it would also help to simply provide examples.

* A related concern is about the images recalled for each piece of a sentence. As I understand it, the visual knowledge fusion layer recalls a different set of K images for each sentence part (e.g. different images for "The color", "The color of", "The color of sky", etc.). How do the recalled images vary over the course of the sentence? Do they stay the same when non-semantic words like "of" are used? Do they change appropriately when semantic words like "sky" are used?

* There is no evaluation of how this impacts the model's runtime. How long does a forward pass take when generating every word requires a kNN lookup?


There are also some points where the presentation could be clearer:

* The motivation in 3.2 ("object properties rarely appears in text corpora") seems like it should also belong in the introduction. To me this is a key reason to expect this approach to be helpful.

* The text doesn't mention what the "Majority" row in Table 4 means.


Finally, here are some minor suggestions and comments:

* The attention mechanism over K retrieved images is novel, but a fairly straightforward extension of the attention mechanism once one has already decided to retrieve relevant images. The major contribution is in retrieving relevant images.

* Can VaLM be used as a vision-language model? Essentially, what if it was given vision and langauge tasks (e.g., VQA) and then the image retrieval module was replaced with a module that simply returned the image(s) associated with the VL task example? Would it perform well at the VL task? In general, does it treat the retrieved images as global / abstract context or does it also consider them as specific context local to those images?
clarity,_quality,_novelty_and_reproducibility: 
Novelty and Significance: The idea is novel, timely and very relevant to recent progress in transformers. It provides a crisp solution to a clear problem. There are also many potential directions to both build on this work and apply it to improve current state of the art solutions.

Quality: The experiments provide strong support that VaLM works well, though further investigation of how the retrieval module is working would be good to add in this paper.

Clarity: The presentation is very simple and clear.

Reproducibility: The information in the paper is enough to implement the model. It provides hyperparameter details as well as the prompts used for evaluation. A code release is also promised.

summary_of_the_review: 
The novelty, potential significance, quality, clarity, and potential reproducibility of this work are all high, so it is a very strong submission.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To augment language models with relevant visual information, a visually-augmented language model (VALM) is proposed in this paper. The core idea is to retrieve relevant images by CLIP model and then fuse them to the second last Transformer layer of PLM.
strength_and_weaknesses: Strengths:
1. The research problem of how to inject visual information into language modeling for visual-demand tasks is interesting. The method is intuitive and novel to a certain degree.
2. In many datasets, the performance of the proposed model outperforms compared methods/baselines 

Weaknesses:
1. Indexing images, retrieving images and fusing image features to PLM all take extra computation and time as opposed to original PLM, especially retrieving I think. The analysis of additional training and inference time brought by the proposed method should be added.
2. Lack of experimental comparison against recent visual-augmented language model works. For example, Vokenization and iACE.
3. The database image quality matters for the retrieval and downstream task performance. But the ablation/analysis is missing. I mainly have two questions: (1) Does the dataset have to be an image-text dataset, such as ImageNet? Theoretically, it can also be an image-only dataset since CLIP is already trained. (2) Does the size of the database matter? How about we only take 100M, 10M, or even 1M data from Laion?
4. Some unclear illustrations. (1) In Sec2.3, is z_ij the image feature output from the same CLIP image encoder used in retrieval? (2) In the 3rd last sentence of Sec3.4, shouldn't it be "adopting image-specific bias outperforms directly sharing the bias"?
clarity,_quality,_novelty_and_reproducibility: Clarity: Good but some details are missing.
Quality: Good.
Novelty: Good.
Reproducibility: Since it requires a retrieving system on large-scale image datasets, reproducing might takes more effort.
summary_of_the_review: My main concern is about the lack of experimental comparison with previous methods and the efficiency problem. So my initial rating is borderline reject.

---------- After rebuttal --------
My main concerns have been solved and I'd like to raise the score from borderline below to borderline above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a novel architecture to do visual-augmented language modeling. Before each prediction of the next word, this architecture queries the most relevant images w.r.t. the current received part of the sentence using a pretrained CLIP model and a large image database. The authors trained this model on a large text corpus. They then showed that this model outperforms other models on multiple visual-language benchmarks, including Memory Color, Object Shape, Relative Size, and ColorTerms, by noticeable margins. The authors further show some ablation studies that the use of the image-retrieval module is helpful.
strength_and_weaknesses: Strength:

This work proposes a new architecture for augmenting language modeling with auxiliary visual cues. The authors tested multiple evaluation datasets for their proposed and other candidate models. The improvement in these datasets compared to the other models tested in this work is significant. The authors also show some ablation studies that the proposed image retrieval module helps the performance.

Weakness:

The biggest worry I have is about the evaluation in this work, which is critical to be resolved before I can fully back the acceptance of the paper. The pure language model tested in this work is a GPT-2 retrained by the authors on the same text corpus. And the other visual augmented models are all pretrained models from other papers. None of these comparisons can be perfectly fair on the training datasets. However, it seems that the model size is controlled, as this model uses a pretrained CLIP model during its training and is augmented by a large image database. If the training dataset can never be perfectly controlled, why not try models trained on much larger text corpus, such as the pretrained OPT models? Besides this, the numbers on these benchmarks seem to be also lower than the numbers I can find in other papers. For example, in the paper “Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?” (about the Memory Color dataset), Table 4 shows numbers much higher than the numbers reported in this paper. Can the authors explain this difference?

Another minor issue about the evaluation is: what would be the current up-limit on these evaluation datasets from pure-language models? Have people tested the largest pretrained models on these datasets? Are these problems really hard for the models to resolve?

To help correctly evaluate the innovation of this work, can the authors also comment on how different this fusion layer proposed here is from the fusion layer in the Google Flamingo paper?

This is more of a question instead of an issue. It would be great to see how reducing the number of images in the auxiliary image database will influence the performance. I would imagine that having these images during the real-time inference makes the inference very slow, about which I cannot find any time estimation from the paper. So this could be an issue for real-world applications. To be clear, I don’t think this issue needs to be addressed in this work right now, but I want to get a sense of how annoying this is.

Finally, it would be good to know how the scaling will influence the results. Will large models like 350M make the performance better?

clarity,_quality,_novelty_and_reproducibility: The writing is clear. There is some novelty, but more explanations are needed to separate it from other publications. The paper has enough details to reproduce the results, though I think the reproduction would require many computation resources and time. So it would be great to have the pretrained models also released by the authors.
summary_of_the_review: This work proposes a new architecture that achieves better performance than other models on the benchmarks they tested. The current version needs more justification about how real this improvement is and how innovative this architecture is.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a pre-training framework to augment the visual information to the text pretrained models. The method is based on a an image retrieval module given a textual context and a fusion layer to comsume the visual information. The experiments are carefully done and compared with essential baselines. The reviewers are generally happy about the paper. Although reviewer 9KzG did not raise the score in the end, I feel his concerns are addressed in the rebuttal stage. I therefore would recommend acceptance of this paper to ICLR.
justification_for_why_not_higher_score: I think this is a solid paper, but the tasks are mostly vision + language tasks, the method is expensive and relies on a good retriever, making the applications of such model limited.
justification_for_why_not_lower_score: n/a
Decision: Accept: poster