Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed to leverage the power of a frozen bidirectional LM to tackle the zero-shot video question-answering task. Adapters and projection layers are added to the model as the only trainable weights, so it's a very lightweight optimization. Also, the frozen backbones enable us to fine-tune the model without forgetting the knowledge of the pretrained backbones, so it can achieve much better performance compared to fine-tuning the whole model. The proposed model achieves state-of-the-art performance on many zero-shot VideoQA datasets. The experiments in the fully-supervised setting also show that it's possible to apply the same method with much training data and achieve state-of-the-art results.
strengths_and_weaknesses: Strengths:
- The paper is well-written. The flow is easy to follow and the method is clear and simple.
- The authors carefully design the prompting templates for using the frozen model out-of-the-box.
- The ablation study is comprehensive, we can clearly see that a frozen LM and adapters and the speech modality are necessary to to achieve good performance.
- The result shows the effectiveness of this proposed method: it can outperform a fine-tuning baseline or an autoregressive baseline by a large margin.
- The same method can also be applied to the fully-supervised setting and it can achieve state-of-the-art results when compared with other supervised baselines.

Weaknesses:
- The idea of utilizing a frozen LM module is not new (e.g. https://arxiv.org/abs/2106.13884 which is a paper using autoregressive LMs to perform image-text few-shot learning). Although this method mainly focuses on VideoQA and BiLM, its novelty is still limited. However, this paper has a good engineering value, as no one has tried to use a frozen BiLM on VideoQA tasks. This paper is the first to do that can achieve surprisingly good results.
questions: - Is it possible to adapt a pretrained speech module to this framework? As audio is crucial information in videos. Merely adding ASR transcripts to the model can not fully preserve the audio information. If we can incorporate pretrained models like wav2vec, this direction would be much more interesting.
limitations: The authors adequately addressed the limitations and potential negative societal impact in the Conclusion section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper addresses zero-shot video question answering by leveraging pre-trained language models and visual encoders. The proposed model encodes videos with CLIP and projects them into language. The language is encoded with a bidirectional language model (e.g. BERT) and trained with a masked language modeling objective. The paper only re-trains the visual projection and adapter layers on a large video-captions dataset, and it shows that freezing the language model and the visual encoder leads to better results than re-training everything. Extensive experiments are conducted and results are reported on 8 videoQA datasets.
strengths_and_weaknesses: - The paper is well-motivated, well-written, and easy to understand. The ideas proposed in the paper (joining pre-trained models together and use them for zero-shot videoQA without retraining) are simple and elegant. Multiple experimental results and ablations on 8 datasets support the hypothesis of the paper.

- The results of the paper are significant to the community: 
  1. Using frozen language models (with no visual information training at all) can lead to better performance than fine-tuning such models on visual-language datasets probably due to catastrophic forgetting.
  2. Smaller language models like BERT and its variants can outperform larger language models based on GPT.
  3. Models pre-trained for different tasks can be made to work together by only fine-tunning some adapter layers.

questions: - The drop in performance when removing the suffix is surprising. Does something similar happen when removing the prefix ([CLS])?
- How important is the prompt design for the zero-shot performance? Have you tried with different prompt templates?
- Have you tried using the model for image VQA?
limitations: Already discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper mainly concerns the problem of zero-shot video question answering (VideoQA). It proposes FrozenBiLM, which performs video-text(-speech) pre-training over a frozen language model to enable it with visual understanding capability while preserving its language generalization ability. The resulted model could perform zero-shot VideoQA out of the box. The paper also demonstrates the capacity of FrozenBiLM in terms of few-shot/fully-tuned learning. New SotAs are set on a wide range of mainstream VideoQA benchmarks.
strengths_and_weaknesses: Despite that adapting a frozen language model for VL tasks is not new (e.g., [88]), nor is adaptor-based transfer learning for Transformers (e.g., [26] and LoRA [r1]), the paper studies this simple yet important baseline on the challenging problem of VideoQA. The overall idea is convincing and well-executed. The experimental results are solid and adequately support the claims from the paper w.r.t. the importance of freezing the language model and adaptors on cross-domain transfer learning. The paper flows well and is easy to follow.

Note that some relevant concurrent work such as Flamingo [r2] is missing in the related work, considering its relevance and the similar model design. Other related work on zero-shot/few-shot VideoQA include [r3] [r4].

[r1] Hu et al., LoRA: Low-Rank Adaptation of Large Language Models. arXiv 2021.

[r2] Alayrac et al., Flamingo: a visual language model for few-shot learning. arXiv 2022.

[r3] Xu et al., Videoclip: Contrastive pre-training for zero-shot video-text understanding. EMNLP 2021.

[r4] Wang et al, Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners. arXiv 2022.
questions: Please address the following questions:

i) In line 217, it is evident that a simple averaging over multiple tokens does not preserve the semantic structure of the label/phrase (e.g., "man riding horse" vs. "horse riding man"). Hence, the resulted model could suffer in the wild when *rare* events/relationships happen (e.g., "horse riding man"). It would be great if the authors can shed some lights on possible solutions and perhaps other strategies they have attempted in the experiments to alleviate this issue. One of the related works is [r5].

ii) Considering that not all existing methods have speech transcripts as model input and the fact that transcripts are extremely informative esp. on instructional videos, the comparison seems unfair (e.g., in Tab. 6). Please comment on this and also denote methods with speech input in the result tables.

[r5] Salazar et al., Masked Language Model Scoring. ACL 2020.
limitations: Appears to be sufficient.
review_title: null

Meta-review: metareview: The submission introduces a zero-shot VQA model that combines frozen video and bidirectional language models by training additional projection and adaptor layers. The method significantly outperforms related previous work that uses only uni-directional language models. While the approach is somewhat incremental technically, reviewers found the results to be significant and thought that the main claims of the paper are well supported by thorough ablations. There are no significant concerns raised by the reviews, and overall this is solid work, so I recommend acceptance.
confidence: Less certain
Decision: Accept