Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes OmniVL, a vision-language foundation model that supports both image-language and video-language pretraining in a unified framework. The model allows for evaluation of vision-only tasks, multimodal retrieval, and multimodal generation based tasks like captioning and VQA. The model achieves comparable or SOTA performance across many benchmark datasets.
strengths_and_weaknesses: Strengths
- The proposed framework unifies different modality objectives, downstream task formulations, and input data types.
- Given the current move toward "foundation" models, this work seems like an obvious next step toward general multimodal models that are performant in a variety of downstream tasks, and is new to incorporate video data at the same time.
- There are comprehensive experiments and performance is either comparable or SOTA on some downstream evaluation tasks.
- The decoupled pretraining is a sensible idea and has a large impact on performance given the ablation study

Weaknesses
- While the paper states that their unified vision-language contrastive (UniVLC) is a novel contribution, it is a straight forward extension of the existing UniCL loss. 

questions: - Why does OmniVL perform more poorly on the in-domain subset of the NoCaps captioning task?
limitations: I don't think the authors adequately addressed limitations of the approach or negative societal impact. Computational resources were broadly mentioned, but the societal consequences sentences are hand wavy and it gives the impression the authors didn't think critically about this.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a new foundation model for image-language and video-language tasks.
First, it applies a single unified transformer-based visual encoder for both image and video inputs. Second, they propose a decoupled joint pre-training of image-language and video-language to effectively leverage spatial and temporal dimensions.
Third, they propose a unified vision-language contrastive loss to leverage different sources of data.
The authors argue they achieve a good performance on many different types of tasks.

strengths_and_weaknesses: Strengths:
	1. The motivation is good. The authors aim to build a foundation model for image-language and video-language tasks, which is a popular research topic.
	

Weaknesses:
	1. The so-called decoupled paradigm separates the image-language pretraining and video-language joint pretraining. The two-stage pre-training is a bit complicated. End-to-end pre-training is easier to train.
	2. Several important baselines are omitted. For example, FLAVA serves as an important foundation model but there are few comparisons. Why? I found that FLAVA outperforms Omnivl on ood101, DTD, and Flowers. Also, for image captioning tasks, the state-of-the-art is already 140+. Although different model sizes and pre-training data are used, the authors should compare with more recent methods (like SimVLM [1], OFA [2]).
	3. Several important datasets are omitted. For example, Imagenet is an important dataset for linear probing but this paper did not report its performance. 
	4. Because a lot of important baseline models and datasets are missing, the comparison is not comprehensive.

[1] SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
[2] OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
questions: 	1. The performance comparison is confusing. Why do you report the performance of SimVLM in table7 but do not report it in table6? As we know, SimVLM is able to do both VQA and captioning. This is common for other baselines in other tables. 
	2. In Section2, the authors claim FLAVA is concurrent work, which is not true. FLAVA is proposed last year and their results should be included and discussed.


limitations: 1. include more studies and conduct a fair and comprehensive comparison
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a unified transformer based model for Image-Language and Video-Language tasks by performing both image-language and video-language pretraining. The authors propose a decoupled joint pretraining image-language and video-language in order to obtain a boost in performance on multiple downstream tasks. The model is pretrained in a decoupled way, where first it is pretrained on image-language and then on video-language different from prior works. Moreover, the authors propose a new loss function that leverages image-text, video-text, image-label and video-label information. Finally, the authors test their proposed approach on multiple benchmarks.
strengths_and_weaknesses: Strengths
The paper tackles an important problem and proposes an interesting system that achieves good results. The overall paper is fairly well written and the overall message is well understood.

Weaknesses
While the overall idea is interesting and the results look promising, I feel like the ablation part of the paper could be more comprehensive. So, I have several questions related to these ablations (please see below)

Missing citations
* Miech, Antoine, et al. "Thinking fast and slow: Efficient text-to-visual retrieval with transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
* Croitoru, Ioana, et al. "Teachtext: Crossmodal generalized distillation for text-video retrieval." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
* Bogolin, Simion-Vlad, et al. "Cross Modal Retrieval with Querybank Normalisation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
questions: 1. Can you give more details on the data used for the experiments from Table 9? How many samples are there? Do you use the "label" anotation?

2. Does the Table 9 experiments assume a zero-shot setup, or the model is fine-tuned on the downstream task? If it is finetuned, can you also provide numbers without pre-training?

3. I find a bit strange the fact that "Joint" pre-training performs worse than "Image-only" and also that "img2vid" brings an improvement while "joint" doesn't. Do you have any insights about this? Also, I would suggest including some of the "img2vid" results in the main paper.

4. The decoupled learning seems to bring a quite significant gain. I would emphasize this aspect a bit more.

5. Can you give more details about the "*" from the tables. From my understanding 14M* means that the model was pre-trained on 14M text-image pairs as well as on text-video data. Is this correct? If yes, please make it more obvious in the paper. Also, if this is indeed the case, I think it would be fair to include the numbers only with the 14M samples to have a clear comparison on the amount of performance the video data brings.
limitations: The limitations are briefly discussed while the potential societal impact is left for future studies.
review_title: null

Meta-review: metareview: After the authors’ rebuttal and long discussion between reviewers and authors, the paper unanimously receives positive rates thanks to reasonable proposed ideas and thorough experiment evaluation. The camera-ready version may need to be updated to fully reflect reviewers’ comments and authors’ answers to them. 
confidence: Certain
Decision: Accept