Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper proposes a benchmark on activity recognition dataset to measure robustness against the spatial and temporal corruptions. The paper establishes Mini Kinetics-C and Mini SSV2-C for the evaluation purposes. This is the first paper that evaluates corruption robustness of spatial-temporal video models.
strengths: * This is a good benchmark. The paper performs corruption on two of the very popular action recognition datasets – Kinetics and Something Something V2
* Multiple types of corruptions can be performed at different levels.
* Well written paper with extensive analysis and discussion for temporal and spatial corruptions.
* The contribution seems to be significant in the field of action recognition and video understanding. 

weaknesses: No major weaknesses that are not already accounted for in the paper. In fact, the authors include a refreshingly up-front discussion of their results. Potentially the authors can throw some light on the computation required since action recognition datasets tend to be massive and computationally hungry.
correctness: They seem correct to me.


clarity: The paper is very well written.
relation to prior work: Its relation to prior works is clearly discussed.


additional feedback: It would be interesting to analyze which actions particularly fail on each dataset and like wise for each model.
review_title: Review

final_score: rating: 3: Clear rejection
review_content: summary and contributions: The authors propose a benchmark for video corruption robustness of spatio-temporal models. They implement 12 versions of synthetic corruptions - 6 spatial corruptions and 6 temporal corruptions and subsequently create two corrupted datasets (Mini Kinetics-C and Mini SSV2-C). They find that models with higher clean accuracy also have higher corruption accuracy. They also find training for individual corruptions does not provide robustness to other corruptions.
strengths: The strengths of this paper is that there is a thorough systematic evaluation of many models (8) on many different corruption types (12). The authors also create two new corrupted datasets - Mini Kinetics-C and Mini SSV2-C - which will be useful for the community to benchmark on moving forward.
weaknesses: To me, it is not clear what the contributions of this paper are (apart from the synthetic datasets).
The results from the headline experiments - that models with higher clean accuracy have higher corrupted accuracy, and that training for one corruption does not give robustness to other corruptions - have already been studied in the literature in the image classification setting [12, 40]. Thus, is the takeaway simply that "nothing changes in the spatio-temporal modeling setting"? If so, this should be stated more clearly and the paper should be re-organized accordingly.

What I believe would be a more fruitful direction would be understanding what sorts of distribution shifts are likely to arise in video data? The authors acknowledge that [40] find that robustness to synthetic shifts do not imply robustness to natural distribution shifts (which is ultimately what we care about). The presented counterexample from [17] is entirely unconvincing - it is conceivable that being robust to synthetic blurs also gives some robustness to real-world blurs. In general, however, real-world distribution shifts are not so closely aligned with existing synthetic counterparts. For example, videos can vary greatly in subject/object pose, lighting, composition of the scene, weather elements, etc., among many others. Given the comprehensive study of [40] showing that in general synthetic benchmarks do not track real-world use cases, would it not be better to study these natural distribution shifts directly? Or, the authors can disprove previous work such as [40] wrong by collecting data from real-world natural distribution shifts and showing that the proposed synthetic benchmarks track them very closely. 

In summary, the major flaw of this work is that it lacks motivation for why synthetic shifts should be considered. I believe this can be remedied by collecting a few examples of natural distribution shifts and showing whether synthetic performance does or does not track it. Additionally, the authors should consider what is different/unique about the spatio-temporal modeling setting (as opposed to image classification) and tailor their analyses, so as not to repeat similar work done previously in image classification.
correctness: For the synthetic shifts, why are only half implemented spatially and half implemented temporally? Is there any restriction for coming up with a spatial version of the temporal shifts, and vice versa? If this was possible, one additional interesting analysis would be to see how robust models are to the same shift, but a spatial version vs a temporal version.

Relatedly, it would be interesting to split up figure 2 into spatial corruptions and temporal corruptions and see if the trends line up with each other.
clarity: Paper is clear to read. When reading results, I find the rPC hard to interpret (and mostly unnecessary) - looking at the scatter plots such as Figure 2 is far more useful.
relation to prior work: Already addressed in weaknesses.
additional feedback: N/A
review_title: Not clear what the contributions of the benchmark are

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors propose a benchmark for evaluating the robustness of spatiotemporal deep learning models against real-world spatial and temporal corruptions. Specifically, they have created subsets of the Kinetics and SSV2 datasets (called Mini Kinetics-C and Mini SSV2-C respectively), and released code for comprehensively simulating various artifacts into the video data. They demonstrate that this benchmark can be used to assess robustness to common video artifacts across various models including CNN and Transformers.
strengths: To my knowledge, while many prior works have assessed the robustness of 2D image classification networks with respect to common spatial artifacts, none have explored the effect of artifacts on video classification / action recognition. This benchmark will be the first, which will further understanding about the effect that both spatial and temporal artifacts have on action recognition tasks.
weaknesses: The authors do not provide model definitions, nor the code used to actually evaluate each model on the Mini Kinetics-C and SSV-C datasets. This limits the reproducibility of the benchmark results in the paper, and the leaderboard found in the Github repository.
correctness: The authors correctly state in the title and throughout the text that this is a benchmark (not a dataset). The evaluation methods and experiment design are sound, and make sense for the goal which is to assess the robustness of action recognition models.
clarity: It is correctly stated in the title and throughout the text that the authors are releasing a benchmark, and not a dataset.
relation to prior work: The authors provide a comprehensive list of prior action recognition approaches utilizing both CNNs and Transformers, which are the models used for comparison in their benchmark. Other image-based benchmarks which intend to assess the robustness of image classification models are also discussed (i.e. ImageNet-C).
additional feedback: •	In Section 2.2, I recommend including another work by D Tran, et al. in which video classification networks were further improved by decomposing 3D spatiotemporal convolutions into 2D spatial and 1D temporal convs with an activation between (R(2+1)D networks): D Tran, et al. “A Close Look at Spatiotemporal Convolutions for Action Recognition” DOI:10.1109/CVPR.2018.00675
-	How is the frame rate conversion performed exactly? Is the video data re-interpolated? What interpolation scheme is used? Please clarify this in the text.
-	How did the authors decide on standard deviation of 0.1 and 0.2 for training with Gaussian noise? Can you show an example of clean images, with std=0.1 noise, and with std=0.2 noise in the supplementary information? Please clarify this in the text.
-	The authors state that “Gaussian data augmentation has been widely used to improve the robustness of models in image-related tasks,” yet they only cite one work (ref. 13). Can the authors provide other examples? Otherwise, I suggest revising the wording here.
review_title: The authors propose the first benchmark for assessing the robustness of action recognition models against real-world spatial and temporal corruptions

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a benchmark for testing the robustness of action recognition models in videos against different types of data corruptions which may occur in real life. In addition to corruptions which are spatial in nature and were previously considered for images, the paper introduces several types of temporally-related corruptions which may occur in the process of capturing and transmitting videos. Using the benchmark, the authors conduct a study on the relations between robustness, generalization, and models' type and efficiency. 
strengths: Studying the robustness of models against corruptions that may occur in real life applications is important both from a scientific and a practical point of view. This work does that in the context of video classification and is a natural next step following similar benchmarks and studies in the context of image classification. I think that the benchmark that the authors created could be useful to the computer vision community in the pursuit of developing accurate and robust models. 
weaknesses: I think that the main contribution of this work is the benchmark itself. I would therefore expect some more details such as: how were the corruptions implemented? how did you determine the severity scale? What are the different severity levels for compression?
I couldn't find any of that, not even in the supplementary material, expect for a referral to the code in Github. 

I think that from the results presented in the paper, it is difficult to draw a clear conclusion regarding the effect of model capacity/computation on robustness. The fact that performance on clean data is correlated with mPC and both are correlated with model capacity is not surprising. We know that by and large more capacity leads to better generalization. The more interesting metric to look at, in my opinion, is rPC vs. capacity/computation. With respect to that, Figure 3 does not show a significant correlation. Furthermore, while there is a drop in rPC between 3D ResNet-50 to 3D ResNet-18, it is not significant. I think that more direct comparisons of that sort, between similar models with backbones of different sizes are necessary. Also, it would be interesting to include transformer-based models in such an experiment. Currently only one such model is considered.    
correctness: The claims seem to be correct. I can't say much about the benchmark construction because the authors did not provide sufficient details. 
clarity: The paper is well written. 
relation to prior work: Prior work is well covered. 
additional feedback:
review_title: A useful benchmark to test robustness of video classification models against data corruption

Meta-review: This paper presents a novel benchmark for evaluating robustness against spatio-temporal video corruptions. While the reviewers generally agree that the paper addresses a novel important question, it received split reviews of (3,8,6,7) where the main concern raised by one reviewer is the lack of real corruption data to prove that the proposed synthetic dataset really aligns with the real-world corruption cases.
The reviewers and AC had an intensive and fruitful discussion on this point. Importantly, we all agree that the concern totally makes sense and we all have to be aware of this fundamental issue. At the same time, we also understand that sometimes collecting ideal data is quite difficult and demanding, and we need to find out a reasonable compromise to go forward step by step. From this viewpoint, three reviewers clearly support that the paper opens up a novel research direction and has enough contributions to be published in its current form. The AC considers their conclusion is valid and would like to recommend acceptance.
In addition to that, as a part of NeurIPS, papers in this track are expected to lead this kind of fundamental and scientific discussion in terms of data, so I'd like to encourage authors to consider this issue thoroughly and enhance the discussion properly (in particular, the limitation of the dataset).
Decision: Accept