Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a method for few-shot learning for video understanding tasks. In a nutshell, it works by first extracting information from a given video using pre-trained image and language models: it uses CLIP to extract "visual tokens" for objects, events and attributes, as well as a captioning model (BLIP) to extract image description of a few frames of the video. These automatically generated annotations are then assembled together to form a textual description representing the video by notably keeping some information about the order of events. Since videos are represented by simple text, it allows to use the few-shot capability from an existing language model (InstructGPT in this case) to do few-shot learning on video understanding tasks: video captioning and video question answering. The authors evaluate this approach on five different benchmarks. The model is also used as a pseudo labeler to improve video-text retrieval system.

strengths_and_weaknesses: **Strengths**

- The paper is clearly written and easy to follow
- Few-shot learning in multimodal setting is a very important endeavor 
- The proposed method is a good example of repurposing a large language model to do few-shot learning for videos in a simple way that does not involve any training with video and language dataset. It provides a very strong baseline for such models.


**Weaknesses**

- A few clarifications are required around the generation of the prompt (see Question section)
- Conceptually, the approach is quite similar to the PICA paper [61]. It would be good to better clarify the difference with that approach? Is it mainly the video setting and the fact that one has to deal with temporal information? If yes, then would it be possible to show what happens if only a single frame from each video is used instead of multiple ones?
- Some ablations are missing (see Question section for specific suggestions)
- Some results seems a little bit inconsistent and raise some questions (see Question section for more details)
- Should Flamingo be also compared against in Table 2 for consistency (since they provide results for YC2 and VATEX as well?)
 - It would be interesting to see how the performance evolve with number of prompts for all benchmarks but mainly 5 shot results are given (see request in the Question section)
- Limitations of the approach: 
  - This approach is brittle as it is bottlenecked by the extraction of visual entities and the captioning model. 
  - Such approach may not scale well to long term video understanding (mentioned in the conclusion) since one would be quickly limited by the size of the prompt of Language Models. Also temporal information is only given with words like "First", "After"... How would that be scale to longer videos? Could the authors comment on that?


**Summary of review**

Overall the paper introduces an interesting method and has good results on video benchmarks. However there are a few important questions that are important to address in the rebuttal to help confirm my current weak accept decision.
questions: **Clarifications**

- L144: "We then select top 5 visual tokens": is it top 5 per frame, or top 5 total? In the Figure 2, it seems there are only 3 for the first frame. Can the authors clarify this? It is also said that 8 frames are used but when I check the Appendix, I can see that there are at most 4 temporal description (First, Then, After that, Finally). Is there some filtering happening on top of the top 5?

- L224: For the BLIP baseline, is there additional training involved or is the proposed technique of "stitching the sampled frames" is a pure inference only trick? This was not clear to me as the term "loss" was mentioned. Can the authors clarify what is meant here?

- Table 2: As far as I know UniVL and BLIP are not "few-shot" methods. How do the authors obtain 10-shot results for these baselines?

- Table 2: How come VidIL can be less good than BLIP (for MSRVTT) knowing that the BLIP captions are used as input in the prompt? Is that because the language model is not able to properly use the information provided by the BLIP model in that case?

- When using in-context selection, are you also ordering the samples in a certain way to account for the recency bias in big language models? (putting the most similar last in the prompt might help)

**Additional ablations**

- How important is the "temporal aware prompt" technique quantitatively? Unless I missed that ablation I could only see the qualitative example of Figure 3. Since this trick seems one of the contribution of the paper, I believe it would be important to properly demonstrate that it brings something.

- How does the performance change with the number of sampled frames?

- How does the performance evolve with the number of shots? What is the maximum number of shots you can fit in the usual prompt of the model?

**Questions about experiments**

- In Table 6, adding event seems to hurt performance (Line 3). Can you provide the numbers for "Frame+Object+Attribute"?
- Table 7: is that possible to put a column "Prompt size" and a column "Number of shots" to better distinguish between these two parameters. I think it will make it clearer. How come in-context selection can actually hurt?

limitations: Some important limitations are not mentioned:

- Performance potentially highly bottlenecked by the image level "text" extraction
- Replacing videos by text in the first place is limited and may not scale well when moving to longer videos (the approach would always be limited by the size of the prompt of the language model). Having Large Language models that could directly attend high level features instead could help alleviate this issue. 

The authors do discuss the societal risk, notably the ones inherited from large language models that might be amplified here by using them on videos.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposed a new model for video-language tasks with few-shot ability. The task is decomposed into hierarchical level, to first translate objects, frame captions and their relations using CLIP model. Then a langauge model is instructed with prompts to compose the video frame. 
strengths_and_weaknesses: The proposed model is not end-to-end. It is based on using CLIP and InstructGPT3. The approach is simply based on extracting candidate objects and captions frame by frame using CLIP encoder, and then creating a prompt to feed to InstructGPT3 using in-context learning to achieve few-shot performance. Therefore, there is minimal novelty in the architecture
questions: 1- Line 28: Only the encoder model with bidirectional attention are limited to understanding tasks, but with causal attention, they are capable of both understanding and generation. In fact, autoregressive models have shown promising few-shot performances, because they are more parameter efficient in compare to encoder-decoder models 

2- Is the few-shot ability arises from using language model at the end, or from decomposing the tasks to two models?

3- Line 116: it is mentioned that the proposed model is not required to be pretrained on any video, compare to Flamingo. However, it is mentioned otherwise in other parts of the paper!

4- is there an ablation on the decomposition of video contents into frame, visual token level, video level on the few-shot performance?

5- Figure 2: in visual token level, each frame includes multiple objects. How the template format organizes objects of one frame in order. how the order of objects is defined for each frame? (refer to “few-shot context” box). Is there an ablation on the ordering of visual tokens?

6- is there an ablation on different task instruction formats? how does wording of the template affects performance?

7- Table 7: increasing shots also increase standard deviation. why model performance variance increase by more training examples?
limitations: The proposed model is not end-to-end. It is based on using CLIP and InstructGPT3. Therefore, there is minimal novelty in the architecture.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a few-shot learning approach for video and language tasks.
The approach relies on pre-trained and frozen models: first, vision models are used to extract for each video frame: (A) a caption and (B) a visual attributes (Objects, Events, Attributes) from a set of pre-defined ones.
A text-only prompt is then constructed showing few-shot examples of how to perform a vision-language task utilising information the generated captions and attributes.
Finally, the prompt is fed to a large language model (InstructGPT) to solve the task defined by the few-shot examples.
The model is evaluated on several video benchmarks such as: video captioning, video question answering and next event prediction.
strengths_and_weaknesses: **Strength**:

- The approach is relatively easy to follow
- One strength of the approach is that it does not rely on video data for training. More precisely, it does not require any training at all (in the sense of gradient descent training) as the approach mostly builds on top of pre-existing pre-trained/foundational models.
- The VLEP (Next event prediction task) results are strong and they provide a new state-of-the-art number on that task.
- The method is more interpretable than standard VLM models.
- The approach is highly modular. It is straightforward to update it when improved captioning models or language models are available.

**Weaknesses**:

- I believe the approach to be highly bottlenecked by the expressiveness of the vision foundational models. As a concrete example, I do not see how this approach can work well on traditional action recognition benchmarks (i.e. Kinetics or Something Something) when the task requires low-level temporal understanding of actions. Also the approach does not work for task where there is no useful signals to leverage from frame captions and visual attributes. For instance such task could be fine-grained spatial visual question answering. MSVD-QA and MSRVTT-QA are not really fine-grained VQA dataset, so there are no questions requiring spatial understanding such as "'What is on the left of the car?' or 'What is on bottom the table?'. Right now I do not see how the provided frame captions and attributes are enough to answer fine-grained question about the visual content.

- Why are the ablation studies only reported on MSVD-QA? The results would have been more convincing with several evaluation datasets.
For example in Table 7, we do not see clear improvement when the number of shots increase. As the ablation study is only reported on one dataset, I'd assume the conclusion were not even better on the other datasets? Also I find that the number of in-context examples VS the true few-shot number in this Table 7 is not clear. It would have been clearer to have two columns, one for both. It also look like the in-context selection is not really beneficial for more than 5 shots.

- Table 3: Clarify the real number of shot used in that table. What is the number of in-context vs the number of shots the retrieval system has access to. Is it 5-shot in the sense of 5 in-context shot?

- Table 6: It looks like frame captions are enough to get most of the useful signal, as it only improves the MSVD-QA top-1 accuracy of 1.2% when adding the different visual attributes. This is disappointing knowing all the emphasise around the need for the visual attributes in the approach section.

- In the Related work section, the authors wrote L 110:
‘We are the first work to leverage prompting a frozen language model for tackling few-shot video-language tasks with a unified textual representation’.  I believe this is an overclaim as [1] have already done something in a similar spirit. It looks to me this paper, mostly extend the work of [1] with non-vqa tasks and video instead of images.

[1] Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., & Wang, L. (2022, June). An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 3, pp. 3081-3089).
questions: 1. About the use of temporal marker: how does it work when you have more frames than temporal markers?
2. I was curious to know the impact of the choosen language models. Why did you choose InstructGPT over GPT-3? Is it because InstructGPT is much more lightweight to run? Or is it because InstructGPT is performing better than GPT-3 at following the few-shot examples?

typo:
L 276 : ‘video-captain’

limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a few-shot Video-language Learner termed VidIL based on image- language modeling. A temporal order aware prompt is also proposed by decomposing videos into a hierarchical structure. Without pre-training on videos, the proposed VidIL is evaluated on a variety of few-shot video-to-text tasks, such as domain-specific captioning, question answering, and future event prediction. 
strengths_and_weaknesses: Strengths:
+ A few-shot Video-language Learner termed VidIL is proposed based on image- language modeling.
+ A temporal order aware prompt is proposed by decomposing videos into a hierarchical structure.
+ A variety of few-shot video-to-text tasks are included for performance evaluation. 

Weaknesses:
- The technical novelty of this paper is very limited. CLIP, BLIP, and InstructGPT are respectively deployed for visual tokenization, frame captioning, and conditional text generation with few-shot prompt. Excluding this, the contributions made by the authors are rather minor. I even have doubt that the performance of VidIL is mainly due to the deployment of CLIP, BLIP, and InstructGPT. 
- The proposed VidIL can still be considered as prompt learning for large-scale cross-modal pre-training models. Therefore, other prompt learning methods should be included in the main experiments. Some examples are:\
[a] Align and Prompt: Video-and-Language Pre-training with Entity Prompts, arXiv:2112.09583, 2021.\
[b] CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models, arXiv:2109.11797, 2021. 
- The method part should be reorganized in a more formal way. Although the source code is given, an outlined algorithm would help the readers easier to capture the proposed method.  

questions: See my questions in weaknesses.
limitations: Yes.
review_title: null

Meta-review: metareview: While the reviewers are divided, I agree with those with accept. The paper introduces an interesting alternative without training on videos and numbers useful for the community. The use of pretrained models in a clever way (without finetuning) is not a weakness but a contribution. A reviewer raised a concern about lacking analysis on how the proposed pipeline contributes to the few/zero-shot capability but it is already widely accepted that large-scale pretrained language models can do well in few/zero-shot settings with proper prompts. Also, the use of CLIP is just a way of extracting textual/categorical representation from the input video using a pretrained network and I believe the authors chose to use CLIP mainly because it is trained on a large-scale dataset with an open vocabulary. I think engineering this component is not the main focus of the paper and the lack of ablations on this should not discount the paper’s novelty. 
confidence: Certain
Decision: Accept