Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a way to train a few-shot image-text neural network in which the text encoder weights are frozen, but used to train the image encoder from scratch. It provides an interesting training framework and evaluates few shot performance on image classification and vqa tasks, also studying several qualities of interesting, e.g. rapid task adaptation, encyclopedic knowledge, and fast concept binding.
main_review: The training paradigm is really cool, and definitely novel. The merit of this paper is strong as having few-shot learners that can adapt to tasks they've never seen before is very desirable. The paper is also generally well written and easy to follow. The primary drawback for the significance of this work is that the task performance is very low, but the authors do acknowledge that beating SOTA is not their intention. Also, the paper properly compares to "blind" baselines to prove that the network actually makes use of visual information, which is a necessary comparison and further verifies the merit of their work. 

Other positives include that comparison to prior work is thorough and there are clear distinctions made between this papers contributions and those of related work.

Otherwise, there are some questions + comments:
- L34: what is meant by "ordered sets"? do you just mean pairs of images and words as input? it's a bit confusing. 
- L73-80: the terminology of prefix vs. prompt vs. infix is not clear. Could you clarify the differences and revise the writing for clarity?
- L122: it seems like the term "n" is used to reference different concepts/quantities throughout the paper, which gets confusing. In L22, is the number of visual prefix tokens = n? this should be explicitly stated. Then, in tables 1,2 n refers to the number of shots. Then, L274 refers to n as the number of images of the referents of words.
- L130-138: is the difference between y and t that y are text tokens in their raw form, and t are their embeddings? from the notation it seems like they're the same thing, especially since x is defined on i_1, i_2
- L30-138: why is there swapping between i_1, i_2 and then generalizing it to i_1, i_2, ...., i_n?
- L136: can you define the relative position encoding? it's not defined/clear to me. Furthermore, did you ever actually check changing the image prefix positions? I would be curious to know how well the relative position encoding works, and how much the input ordering is needed as an inductive bias.
- L154: how do we know that the "different ways of inducing the task to the model" aren't potentially harmful or biased to different groups?
- Figure 4: The purpose of injecting nonsense words isn't explained early enough (it's only ever formally mentioned on L241-244), can you please address that sooner in the text. Also, why isn't there an answer in the upper row for the "Question from ImageNet"?
- Number of shots vs. number of inner shots. Are there two differences here? The first being that number of inner shots is per category, and the second being that the "full" example isn't provided for inner shots? I am trying to connect it to the VQA example in the number of shots definition, in which the answer is provided in addition to the image and question.
- Number of repeats. Repeating samples doesn't feel well motivated, and isn't really discussed in the text. Why is this done? I am not understanding how this helps us "explore how the model integrates visual information"
- L174: is the word "fast" supposed to precede general knowledge? what about this approach makes the access to general knowledge fast? or is the point that we don't have to retrain the language model to be task specific? It just seems a bit redundant with "rapid adaptation" and "fast binding"
- Table 1,2 Frozen_train-blind: why do these results do worse with n=4 shots?
- Table 1,2: clarifying question- are all of these results with having 2 visual prefixes?
- Table 1: How much VQAv2 data is mixed in?
- L189-191. This is an important take-away, but it is not new - see Visual Dialogue without Vision or Dialogue, Massiceti et al. Maybe you already are familiar with related work showing this, but nothing is cited here, so it would be good to mention this confirms results already shown in prior work.
- L207: Did you ever look at increasing the number of shots until full SGD training performance was reached? is it possible?
- Fast binding: could the purpose of fast binding be better motivated? right now the use of nonsense words doesn't really provide intuition of how this is useful. Do you think we could extend fast binding to other languages? That seems like it would be an impactful use case.
- Table 3: It seems like Frozen and Frozen (real name) plateau as both the number of inner shots or repeats increases, or performance even gets worse (repeats = 5, frozen real name). Why is this? This happens both for the 2-way and 5-way experiments.
- L241-242: What does "this allows the system to express its answers with word-like tokens" mean?
- L251-255 states that Frozen improves performance with more examples, which feels like an exaggeration of the results, given that the performance plateaus after 3 which is not addressed in the analysis.
- Fast-VQA and Real-Fast-VQA: Not enough dataset details were provided on this in the main text, additional details on the number of samples etc would be helpful. Also, I'd reconsider using "Fast" in the name, it's a bit confusing. And "Real-fast" sounds like the data is now really fast. Is the word fast being used because the data is evaluated in a few shot setting?

Style suggestions, typos, etc.
- L34: remove "e.g."
- L136: the ellipses here are an awkward way to end a sentence
- L149: people that aren't native English speakers may not know what a "stress test" is. I'd suggest rewording.
- L153: remove "at"
- L215: the e.g. feels awkward here, consider removing
- L258-259, 265: wording a bit weird/hard to follow
- L283: "a model is reminded what the important entities in the question look like" is a bit confusing- is this because the question now has the real category names included? I would reiterate that explicitly.
- L325-326: what does this mean? 
- L327: adding a hyphen after language may help the flow here. 
limitations_and_societal_impact: 1. There isn't analysis of samples that were already correctly classified with the blind models. What did those reveal, did you look at them? This would provide better insight to this issue and also help researchers build upon this work to specifically target these biases.
2. The classification baselines only explore at most a 5-way classification task, and with performance so low, it isn't particularly promising that results would extend well to a more realistic setting.
3. The role of the relative position encoding is not studied, and so it is unclear how necessary the image-text embedding ordering is for the experimental results.
4. The plateau or worsening of performance as the number of shots or repeats increases is not addressed, and is a bit concerning. Why would performance get worse with more examples?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new way to reuse large pre-trained language models and condition their generation on visual input. The authors build on the idea of pretext tuning and make pretext a (CNN) function on an image. By training the CNN for VQA, keeping the language model decoder fixed, the authors bring visual information into the input space of the language model. The authors propose several benchmarks for few-shot classification and captioning and show that their model demonstrates promised capabilities and achieves better than random results.
main_review: While it was standard for the computer vision community to reuse large pre-trained CNNs for image/video or even multi-modal tasks, doing the same with language models was not possible. This paper proposes to reuse the large pre-trained language models in a simple and elegant way. 
The paper is very well written and clearly conveys the main point. I like how the authors demonstrate numerous capabilities that are enabled by such design, in the Experiments. I believe that this work may pave the way for new interesting research in multi-modal learning, especially given that the authors introduce new benchmarks.

The only drawback I see in this paper is the weak results in the introduced benchmarks. While this is not critical at this stage, it would be useful to have a better idea why the results are low, i.e. is it the difficulty of the task or simply not very capable individual components that are comprising the pipeline.
limitations_and_societal_impact: -
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to utilize the zero-shot/few-shot generalization ability of frozen language models in a multi-modal setting, by projecting visual input features into the text input feature space, through moderate pre-training on Conceptual Captions (the LM is frozen during pre-training). 

The paper designs several interesting experiment settings to show the model’s ability of task adaptation, utilizing pre-trained encyclopedia knowledge, and novel concept binding. 

My main concern is that it is hard to judge whether the results in these experiments are significant or somewhat trivial. But given the novelty of the setting and the promise, I would lean towards acceptance but I look forward to the author’s response.
main_review: Pro:

The picture the paper tries to paint is appealing and makes sense. By projecting the visual features into the same features space as the input embedding space of a frozen LM, it is reasonable to expect that the Frozen LM can make use of this visual input information and we could tap into the great zero-shot/few-shot generalization ability of the Frozen LM.

The paper presents three interesting experiment settings that truly explore the potential usage of the model, i.e. to quickly adapt to new tasks and new concepts.



Cons:
1. The paper somehow falls short as it does not deliver the same level of zero-shot ability as the original GPT3 on NLP tasks. But I won’t count this as a reason for rejection, as the authors have used reasonable methods and the unsatisfying results just show that this is an interesting and hard problem.

2. Hard to judge whether the results are meaningful or trivial. The main metric to judge the significance of the results is the difference between Frozen and Frozen_blind. However, as all numbers are so far away from a reasonable baseline model, it is hard to judge the meaningfulness of the difference between model numbers.
For example, on VQA, a blind Frozen (without seeing the image) gets 33.3 while a Frozen gets 38.2. The numbers are so far away from reasonable baselines (for example, on VQA, a language-only model without any visual information can get over 40% acc), it makes it hard to say whether the 5 points improvement is significant or not.

Detailed comments:
3. Line 126-127: it is stated that fine-tuning the LM hurts performance, which is somehow in contrast to what is observed in V&L model pre-training (e.g., ViLBERT, VisualBERT, LXMERT). I wonder what would be the possible reason.

4. Practically, if we just want a model with a good zero/few-shot ability on V&L tasks, we don't really need to fixate on keeping weights fixed during pre-training on multi-modal data? I see that the reason this paper keeps the weights frozen is simply because the performance is worse. I wonder if there are other reasons that we want to keep the weight frozen.

5. Frozen_train-blind: Table 1 says Frozen_train-blind is mentioned in Section 4.1, but I cannot seem to find it mentioned in the main text.

6. What is Frozen VQA? It says “mixing in data from VQA”. Does it mean mixing in VQA during pre-training?

7. Table 1: The Frozen-finetune is a bit confusing, as it could mean fine-tuning LM during Conceptual Caption pre-training or fine-tuning on VQA with few examples. Judging from the context, I assume it is the former but it would be helpful to be clearer on this.

limitations_and_societal_impact: Nothing I could think of.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new GPT-2 style image-language model, Frozen, which is trained in two stages: 
1. common GPT-2 pre-training on huge text-alone corpora
2. training of a Resnet image encoder while freezing the GPT-2 and text embedding weights, on the task of image captioning (i.e. language modeling conditional on two image encoding vectors with the same dimension as token embeddings).

Since GPT-2 exhibits few-shot abilities in language-alone tasks, the idea is Frozen can also benefit from the GPT-2 language pertaining and exhibit *multimodal* few-shot abilities by the minimal design of adding a visual encoder. Note that the freezing does not make training more efficient, since gradients have to pass through all Transformer layers to the visual encoder.

The Frozen model is applied to zero/few-shot learning of two downstream tasks: VQA and few-shot image classification. 
main_review: **Significance**: The work is at the intersection of two important topics: multimodal (vision-language) learning and few-shot learning. Given the surprising zero/few-shot abilities of recent large language models (e.g. GPT-2/3, T5), it is natural to investigate how such abilities can be instantiated in a multimodal setup, which requires the learner not only for rapid task adaption and knowledge transfer, but also fast binding between visual features and language concepts. Such a study should be very impactful and pertaining both to cognitive science and AI, given how humans possess such strong abilities but not machines yet. 

The work also relates to an important debate in NLP: can language models learn meanings from form (text corpora) alone? By leveraging a frozen language model to "heavy-lift" a visual encoder head, it probes to what extent can language learning enable learning and binding to visual concepts.

**Originality**: The work is interesting and original.

**Clarity**: The paper is well written and the main idea is simple and easy to follow. The figures are illustrative, though I was *slightly* confused by the left part of Figure 4 and what "0-repeats, 0-shots, ..., 0-repeats, 2-inner-shots" mean.

**Quality**: It is interesting how few-shot enables some boosts in results compared to zero-shot. However, my main concern is the poor experimental results in general. Please see below. 

limitations_and_societal_impact: Unfortunately, the results seem too low to be claimed surprising. With better experiments I'd definitely raise the score.

- For VQA (Table 1), the gap between Frozen and Frozen (train-blind) seem very small, which suggests maybe the majority of correctly answered questions are thanks to language-alone learning? I'd like to see some non-trivial examples. 

- For few-shot classification (Table 3, 4), the accuracies are merely 50-something and 20-something percents for two and five way classification, while random baseline would achieve 50% and 20%. This makes me hard to believe Frozen achieves "fast concept binding". Besides, I am not sure if the Real-Name result (66%/34%) can prove the model achieves "slow concept binding" either, given the baseline's 84%/62% performance. 

- Given the poor lag behind visual-language baselines, is it possible to make a fairer comparison by controlling the training data to Frozen and baselines? For example, make sure Frozen and Oscar receives the same amount of training data (text corpora for language-alone pretraining, paired image-captions for visual encoder training), and then study the zero/few-shot test case? Showing the baselines performing a lot better with more training data isn't very helpful to determine how Frozen behaves. 

- Conceptually, why would language learning first enable visual learning later? Why not the other direction (visual learning first, language learning later), given that's more like how humans learn? I'm also curious about such a Frozen variant: mix the two training stages (language modeling, language modeling conditional on visual features) together and learn GPT-2 and visual encoder together. If such a variant is worse than Frozen, it better justifies the two-stage approach with freezing. 

- Is the visual encoder pre-trained anyway? It is not clearly stated in the main paper. If not, it might be interesting to evaluate the visual encoder alone, testing how much better at vision tasks it has been than a random encoder, thanks to the feedbacks from the language model. I'm also curious what if a pre-trained ResNet is used (frozen or fine-tuned) what will happen. If so, what type of pre-training is used, based on what data?
review_title: null

Meta-review: This work presents a proof-of-concept for multimodal few-shot training of frozen language models — where the parameters of a pre-trained language model are kept fixed while an image encoder is trained to prompt it to generate text. Despite its simplicity, the approach demonstrates surprising generalization capabilities of the model when presented unseen visual concepts, although as pointed out by the authors themselves, the results are still far from what current models can achieve on these tasks and more work is needed to refine these ideas further. 
Decision: Accept (Poster)