Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces a new multi-task benchmark for video and language understanding tasks. The benchmark is organized in a fashion that is reminiscent of the recent GLUE and Super-GLUE datasets in the NLP community, combining 11 video and language datasets that are diverse, challenging, and hard-to-game. By covering a wide range of tasks, the benchmark aims to measure the effectiveness of a universal video and language system beyond just a single task. Finally, the paper is complemented with a series of experimental results using baseline models that showed that there is a significant gap between the baseline models and the human performance, suggesting that there is ample room for improvement that can be obtained using the benchmark as the guide.

strengths: The benchmark is well-motivated and the baseline evaluations are well-designed. I particularly welcome the focus on the multi-modal aspect of the benchmark, combining the NLP and vision. Overall, the paper makes a robust contribution to the community.
weaknesses: .
correctness: The experimental setup for understanding the baseline architecture's performance on the benchmark is conducted in a thorough and reasonable manner, testing whether the individual video or the language component of the dataset is enough to achieve a high enough performance, and investigating how different ways to fuse these multimodal components into an embedding. The experiments also point to the potential of joint learning between different tasks, which is interesting in the context of the recent trend to create a single universal model that can perform robustly across multiple tasks. 
clarity: The paper is well-written and presented.
relation to prior work: The paper adequately cites and identifies a missing link in the current language and video datasets, modeling itself after the recent GLUE and SuperGLUE benchmarks that aim to combine multiple datasets and tasks to evaluate models in a more general setting. 
additional feedback: .
review_title: Review for VALUE benchmark

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes a unified dataset for comprehensive video and language understanding.
Particularly, they unify 11 different existing datasets, targeting three cross-modality tasks, namely video retrieval, video question answering, and video captioning. 

strengths: 1.	This dataset could serve for pre-training large deep models.
2.	This dataset is able to support various datasets, making it possible to test a method from different aspects.
3.	Adequate materials are well provided, including the starting code, the extracted visual features, the original annotations, etc.

weaknesses: 1.	The contribution is limited. It seems this work is a kind of integration of other datasets. The integrating process seems to require moderate and routine efforts. 
2.	The motivation is not strong enough compared with other works aiming at unifying existing datasets. For example, the AMASS dataset unifies different motion capture datasets because those datasets have different settings (e.g. markers set), which hinders the training of deep models. Those authors have to propose creative methods to perform the integration.
3.	Concerns about different granularities of different datasets. To explain a little, the authors unifying a variety of datasets for different tasks, some of them focused on more general situations (those in TV and movies, etc.) and some limited to more restricted scenes (\e.g. YouCook2). It’s suggested that the unifying process as well as the evaluation should be carried based on dataset granularity.

correctness: Some claims need more evidence to support. For example, in Line 232 “QA tasks benefit more from subtitle channel than video.” This is caused by the choice of datasets, meaning that answering video-related questions from these datasets requires little focus on visual content and the model could cheat using language bias. For some datasets where “vision really counts”, e.g. the FineGym dataset, things may be different.
clarity: The structure is clear but lacks necessary figures. Figures are suitable to give a better overview of the dataset and could display the experiment results visually.
relation to prior work: This work is an assemble dataset of some previous datasets. 
I hope to see more clarification about the main difference except this dataset is large.

additional feedback: Please consider the semantic and temporal granularity when targeting at unifying datasets.
For example, scenes in TV shows and movies are more general,
while there are some datasets built for a specific domain, e.g. MPII Cooking, FineGym, Epic Kitchen. Please discuss them in related work.
review_title: A comprehensive dataset  is built through unifying existing datasets but lacks novelty and more detailed processing.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a new benchmark for video-and-language understanding evaluation. It assembles 11 video-and-language datasets over three tasks, text-to-video retrieval, video question answering, and video captioning. The benchmark contains a diverse collection of videos with their subtitles. Simple evaluation metrics are also defined for the three tasks. Various baseline methods are evaluated, including models with and without pretraining, different video input channels, different video representations, and fusion methods. Task transferability evaluation and multi-task learning evaluation are also conducted. This paper is the first work to provide a unified benchmark and evaluation for different video-and-language tasks. It also provides analysis on different baseline methods. I think it will shed light on future research on video-and-language, and will make significant contributions to this field.
strengths: 1) This is the first work to propose a benchmark for evaluating video-and-language models on different tasks. As research on multi-task learning and video-language pretraining is making great progress recently, this new benchmark is of high important to the research field. 
2) The datasets are of high diversity, and current methods can not perform very well on the evaluation metrics, indicating that there is much work to be done and more advanced approaches are necessary.
3) There are many experiments and analyses on components of the baseline models, as well as the transferability and multi-task learning. Those experiments provide inspiring information and will shed light on future research.
4) Unified evaluation metrics, pre-extracted features, benchmark, starter code, and leaderboard are provided, so the benchmark is well ready to be investigated and make an impact to researches in this field.
weaknesses: 1) It would be good to provide some statistics or intuitive explanations on the diversity of those videos. For example, the distribution of the video lengths and keywords.
2) What is the meta-average score? Is it the average of evaluation scores on all tasks and datasets? If so, I think there are some potential drawbacks of this metric. Evaluation scores for different tasks may have different ranges. When averaging the scores of different tasks, the averaged score may be dominant by the one with the largest range. For example, in Table 2 the Video QA accuracy has a small range compared to retreival and captioning scores, so the difference in VQA performance might be neglected in the final meta-average score. Since the scores of different evaluation metrics are not comparable in numbers, I doubt does it make sense to take the average of them?
3) When averaging the scores of different datasets to calculate the meta-average score, is the size of test sets taken into consideration? For example, the YC2C leaderboard test set contains only 1.6K videos, while the TVC leaderboard test set contains 10.8K videos. When calculating the meta-average, are the scores of those datasets simply averaged or weighted averaged?
4) The retrieval performance AveR of How2R dataset is very low. Does it mean that the retrieval task defined on this task is impossible? Are there issues with the task definition on this dataset?
correctness: I have doubt about the meta-average score which takes the average number of different evaluation metrics. It seems not very reasonable to take average of different metrics since numbers of different evaluation metric scores are not comparable. Also, the meta-averaged score may be dominated by tasks whose evaluation scores have large ranges and neglect tasks whose evaluation scores have small ranges. In addition, by taking the average of different datasets, the samples in those datasets with a smaller test set will get higher weight on the meta-average score.
clarity: The paper is well written and explained clearly.
relation to prior work: It is clearly discussed how this work differs from previous contributions.
additional feedback: I am impressed by the new benchmark, and I believe that it will make a high impact on video-and-language research. My major concern is the definition of meta-score seems unreasonable and may lead to undesired behaviors. The meta-score may be dominated by tasks whose evaluation metrics have the highest range. Also, samples from small datasets will get higher weights when calculating the meta-average score. If the authors can address or explain this in the rebuttal, I will be happy to raise the score.

**Post-rebuttal Updates**
The authors addressed my concerns well. I am convinced by the discussions on the meta-score and the proposed average rank score in the rebuttal. So I raised my rating to acceptance.
review_title: Review

Meta-review: The paper provides an extensive benchmark and thorough evaluation for several video and language understanding tasks. The contribution has the potential to accelerate future research.
Decision: Accept