Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper introduces a new pre-trained vision-and-language model (VLM), Flamingo, built upon pre-trained vision and language models. The Flamingo model can take arbitrarily interleaved visual data and text as input and do open-ended text generation. 
The Flamingo model shows promising results on multiple multimodal tasks in few-shot learning and can even outperform some SOTA models on some tasks. 
strengths_and_weaknesses: Strengths: 
-
- This paper introduces a new pre-trained VLM built upon existing powerful pre-trained vision models and language models, and archives promising few-shot learning results on a wide array of vision-and-language tasks. 
- Useful tricks are proposed for bridging pre-trained vision models and language models, including vision feature resampling, inserting cross-modal attention into language models, etc.
- A new dataset is also mentioned in the paper to pre-train the Flamingo model. 
- The paper is well written in general and easy to understand. 

Weaknesses: 
-
- Taking arbitrarily interleaved visual data and text as input and generating text isn't new for VLM models; the CM3 [1] model does it already and even can do image generation in addition to text generation. Since CM3 is released months ago before the NeurIPS submission, the discussion about it in the paper isn't fair ("CM3 follows the paper's similar approach"). The existence of CM3 does not invalidate some of the main contributions of Flamingo, but it has to be fairly discussed, especially considering the similarities those two models have, e.g., the way to interleave and encode visual and text data in a sequence.
- However, lots of implementation and dataset details are missing (especially the dataset), so reproducing the results in the paper by a third party is nearly impossible. 
- No data and model are attached, and no plan is mentioned in the paper. If the authors are not going to release the model and the data, I am not sure what the point is as no one else can reproduce the results (you have to trust the results reported in the paper blindly). Most researchers do not have the resources to re-train the model, and even with sufficient resources, a third party cannot reproduce the results without access to the data used in the paper.
NeurIPS advocates open source research and cares so much for reproducibility, so non-reproducible research is not acceptable. 
In that case, this paper would be mainly a PR and flag-planting paper for a big institute/company, and does minimal good (and arguably hurt) the research community. I think such kinds of research have to convince people first about the validity and reproducibility of their results before being accepted to top conferences.
- Figure 1 is too good to be true. I know those results are most likely to be cherry-picked and supposed to be good. But without a detailed and fair analysis, they could be very misleading. 

[1] CM3: A Causal Masked Multimodal Model of the Internet, Aghajanyan et al., 2022. 
questions: - Do the authors plan to release the model and/or the dataset used to train the model? If so, when?
- Do the authors plan to release an API like what OpenAI does to allow people to play with your model? 
- How can one trust the results in the paper (not by faith) if no one can reproduce the results?
limitations: - Missing details of the dataset introduced in the paper. No plan for the data and model release. 
- Inadequate reproducibility.

See the weaknesses mentioned above for details.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces an interesting way of combining large, pre-trained vision and language models for solving a wide range of multi-modal tasks. The proposed model, Flamingo, is able to ingest a sequence of text tokens interleaved with images and/or videos, and produces text as output. The authors further demonstrate the efficiency of this model on a wide range of visual-language benchmarks such as few-shot image captioning, reasoning and question-answering tasks where it outperforms many SOTA models. 
strengths_and_weaknesses: **Strengths:**
- The architecture for combining pre-trained visual and language models, while preserving their knowledge, is novel and would be beneficial for the community in adapting large pre-trained models effectively. 
- The paper empirically justifies almost all the aspects of design choices via adequate evaluations.
- The proposed method demonstrates impressive performance on few-shot image captioning, reasoning and question-answering tasks.
- The paper is very detailed, well-written and easy to follow..

**Weaknesses:**
I do not have major concerns with the paper but a suggestion to improve the readability of the paper. I have put my questions in the `questions` section below.
- It’s hard to navigate the appendix of the paper. Could you please mention the exact sections of appendix for specific details while referencing it in the main paper? For e.g. see Appendix A.1.3 instead of just Appendix A.
questions: - The few-shot VQA performance of Flamingo is impressive. However, I have a feeling that the model needs to be guided with text prompts to answer certain aspects related to an image. Is it possible for the model to answer a certain question without building the context? For e.g. In Fig.1 re: Flamingo image, what would be model response if you directly ask $\textit{what is the difference between these three images?}$ without the initial leading questions. Similarly, in Fig. 11 re: street sign in NYC, what is the response if you directly ask $\textit{Which park is nearby?}$

- What is the model performance if you don't use pre-trained vision and language models i.e. training them from scratch with Perceiver Resampler etc.? Does the proposed framework not help in extracting useful low-level features from images and text?

- Is it possible to visualise how textual embeddings map (or attend) to visual embeddings? I reckon, given the task (query prompt) this attention will change for the same image.

- How does the pre-trained visual or language model affects (or limits) the performance of Flamingo? You have mentioned this in the limitations section but have you tried ablating those components? or do you have a failure example?

- I am surprised that spatial grid positional encodings did not help while flattening 2D spatial features to a 1D sequence. In general, this should result in the loss of valuable spatial information. Can authors provide some insights on this?

- What's the reason behind choosing NFNet for visual encoder? Did you try transformers (e.g. ViT) as a vision encoder where you can directly utilise tokens as inputs to Perceiver Resampler?

- What does datasets $D_{m}$ refer to in Eq. 2? I thought the model is trained on only M3W dataset, constituting data from multiple documents. After some sifting through Appendix A, I found the relevant details in Sec. A.3.3. I think it would be a good idea to mention it in the Sec. 2.4 of main paper before introducing the equation.

- For training, do you need the data to be arranged in certain task-specific order for it to generalize to the tasks in few-shot manner during evaluation?

- Out of curiosity, what’s the idea behind naming this model Flamingo? 
limitations: Authors have adequately highlighted the limitations and societal impacts of this work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to align a pretrained image encoder and a large pretrained generative text encoder to the same multimodal space, by training an additional perceiver resampler and gated cross-attention layers. The method is simple and effective. Experiments on various vision-language multimodal tasks show that the proposed model called Flamingo outperforms previous methods under the zero-shot and few-shot settings, and also achieves SOTA performances on several tasks under the finetuned setting.

strengths_and_weaknesses: Strengths
1. The paper is quite well written and easy to follow.
2. The proposed method is simple and effective. 
3. The proposed vision-language pretraining dataset with interleaved images and texts is novel, and would be valuable to the community.
4. Extensive experiments on various tasks and model sizes show the efficacy of the proposed model.

Weakness:
1. One of my main concerns is about the novelty. Aligning a pretrained image encoder and a pretrained text encoder to the same multimodal space has already been studied in previous works, though not at a scale as large as Flamingo. In particular, a very similar idea is proposed in [1], which aligns the CLIP image encoder and a pretrained BART model to the same model space via distillation methods with light additional projection heads. Similar to Flamingo, [1] also exhibits remarkable zero-shot performance on image caption and open-end VQA tasks. It would be better if this submission can clarify more on the connection and difference with [1] and other similar works.
2. The good performance of the proposed models depends heavily on the large model size and large pretraining dataset size. This makes the pretraining and inference of these models very expensive.
3. Some details about the datasets and experimental settings on downstream tasks are not very clear. E.g, 
(1) it is not clear what are the sources for collecting the interleaved image-text dataset.
(2) Previous dual-stream vision-language models like CLIP are efficient for large-scale image-text retrieval because features can be precomputed. However, the image and texts are intertwined in the proposed Flamingo, and it is not clear how the proposed model can be used for retrieval tasks.

[1] Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation, ACL Findings 2022.


questions: Will the trained models and pretraining datasets be released?
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work introduced a new vision-language pretraining model, Flamingo, that aims at few-shot generalization on interleaved visual and textual data. The main architecture is based on Perceiver Resampler to connect a frozen language model and a visual encoder. In order to generate text conditioned on visual representations, the authors design a gated cross-attention module that takes in visual outputs within the pre-trained LM. The model is trained via multi-task joint training. A large-scale interleaved text and image dataset, M3W, is collected to enhance the few-shot capabilities. Experiments show that the proposed model outperforms prior state-of-the-art models on 16 tasks and surpasses the fine-tuned version on 6 tasks.
strengths_and_weaknesses: Strengths:
- The demonstrated few-shot capabilities on vision-language tasks are impressive and can refresh current VL performance on various multimodal benchmarks.
- The proposed architecture and gated cross-attention injected language models are technically sound and can be borrowed by future research on vision-language modeling.
- Ablation studies on the proposed architecture demonstrated the effectiveness of each module design, which also provides further guidance on future VL architecture design.
- The interleaved image-text dataset M3W is beneficial to VL community for future research on zero-shot learning.

Weaknesses:
- Similar to GPT-3 and other generative pretraining models, Flamingo requires a huge amount of model parameters (the smallest one is 1.4B and the performance drop is significant). How to distill the model to a smaller size may be worth discussing.
- Most dataset and tasks are perception level understanding, such as VQA. How is the reasoning capability of the model, for example on Visual commonsense Reasoning?
questions: - The model can only be used in uni-modal/multi-modal to text tasks, will and how the model can be generalized to a unified VL architecture, e.g. text to image, image to image?
- Can prompt engineering techniques that are used for GPT-3 (or similar models) be applied to Flamingo?
limitations: The authors provide sufficient analysis of the potential limitations of the proposed method, including massive architecture seeking, and legacies of large language models.
The requirements of huge amounts of TPUv4 hardware make the model hard to be reproduced by acamadic researchers.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The Flamingo model is a general few-shot solution for multimodal tasks. As input, the model can accept a multimodal prompt. The prompt contains a few examples of the desired task, with text, images, or videos as tokens. The model determines the task dynamics from the prompt and infers the desired task output. This model relies on three key ideas: (i) A frozen Pretrained Language Model (PLM) for training extensive knowledge, (ii) Reducing visual input to a fixed number of visual tokens with Perceiver, and (iii) Training on a web-based noisy image-text dataset (ALIGN) and a cleaner dataset (LTIP).
strengths_and_weaknesses: Strengths:
The paper is generally of high quality. Various tasks are studied in depth. Compared to previous few-shot methods, the performance is significantly better. Additionally, the work includes an extensive ablation study of different design choices, including pertaining, training data mixtures, mixing different tasks, and architecture choices. Moreover, the method is efficient and uses frozen PLM and vision encoders. The only trainable parts are intermediate modules (e.g., Perceiver and gated XATTN-Dense attention). 

Notably, I find few-shot supervision, and steering of large-scale PLM to solve different tasks to be human-like and an important research direction. 


Weaknesses:



(W1) Data access is a major concern. As far as I know, the ALIGN dataset is not publicly available, so future research in this area will be limited. Data is a key factor for training, as indicated in the paper. 

(W2) A similar work, "Multimodal Few-Shot Learning with Frozen Language Models," has many related ideas (e.g., frozen PLM, few-shot multimodal prompt solution). In addition, the study introduces a relevant benchmark, Open-Ended miniImageNet 2-Way Tasks.
While the paper does cite this work, the paper didn't elaborate enough on the fundamental differences between the studies. Is it only using new inputs such as videos, as mentioned on L275?

(W3) This is not a weakness, but a matter of taste. I found it a bit exhausting to go back and forth between the main paper and the appendix, especially with the appendix's size.  A concise but formal description of the method in the main paper (e.g., GATED XATTN-DENSE) would be greatly appreciated. 
questions: Given the limited access to large-scale training data (e.g., ALIGN), how do the authors see future research on this topic?  Are the authors able to provide results from public data, such as LAION?

limitations: Yes
review_title: null

Meta-review: metareview: This paper proposed Flamingo, a visual-language pretrained model, which is built based on existing powerful pretrained pure language models and pure image models.  By fixing parameters of the existing langauge model and visual model, the proposed model is further pretrained with additional perceiver and gated cross-attention components, on  a mixture of vision and language datasets.  The model can take as input an sequence of interleaved text and image/vidios, and generate text output.  The model demonstrated its performances on a range of open-ended and close-ended visual-language tasks, in zero-shot or few-shot settings.  Reviewer KruY proposed stronge concerns on the reproducibility of the work because of not releasing the source codes and datasets and the lack of some dataset details.  Nevertheless, other reviewers all agree to accept the paper because of the contribution of the paper to the community while being aware of the reproducibility problem.  I think the paper is good enough and represent a new sota in a range of tasks in this area, and is acceptable.
confidence: Certain
Decision: Accept