Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present their work on analyzing various action recognition models. Specifically, they present a toolkit that can be used to evaluate action recognition models, an evaluation of 74 released video action recognition models and design choices that can be used to improve action recognition models.

While the article was interesting, my large concern was in the lack of discussion/summarization of the results which could be used to further research in this area.

strengths: *The presented work is an interesting area and the article is reasonably well written
*The benchmarking results are thorough and supplementary information provides good context that can be used to choose different models
*Important observation that existing models learn based on background
weaknesses: *While the benchmarking is thorough, it uses only Kinetics-400. Some discussion about other datasets such as Moments in Time, VLOG would have been a good addition.
*Not much discussion that could be used to help researchers chart out new problems to solve using their results
*Many of the tools described in this article have already been released and/or published
correctness: I don't have a reason to doubt the correctness.
clarity: Yes 
relation to prior work: Yes
additional feedback: None
review_title: Benchmarking video action recognition models

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, the authors introduced a new toolkit for human action evaluation on an existing dataset. In particular, their tool generates synthetic video frames by using image segmentation models to segment the boundary of human regions and using a video-inpainting method to fill the gap of the pixels in human boundaries. Then they evaluated various existing models to validate how much existing models are influenced by background rather than human-related regions. The authors also claimed that the models trained on longer time sequences are less dependent on the scene but rather focused on the human actions. 
strengths: The manuscript is very easy to read, and this reviewer clearly enjoyed reading it. Also, the findings (e.g. Lines 226-233) are interesting. 

The authors demonstrated the validity of their proposed models on various video model architectures (followed by the MMAction2 [6] implementation).

weaknesses: One of the weaknesses of this paper is that it is unclear what the contributions of this paper are to the datasets or benchmarks. The authors invented the toolkit using the two existing models for image segmentation and video inpainting. However, combining these two approaches is not enough for a solid contribution. Though it is interesting to see the human-centric analysis of existing models using the toolkit, the authors' plan for this benchmark (or datasets) is not specifically mentioned in the manuscript. In Lines 168-169, the authors said the swap pairs would be released. Would the authors also release the toolkit?

Some other limitations are also written in the sections below. 
correctness: In Lines 179-180, the authors said they did not train any models but tested the existing models on the newly generated videos. While the frames shown in Figure 1 look natural, this reviewer is curious whether the original models show low performances on their datasets (Background Only Videos, Human Only Videos, Action Swap Videos) due to some artifacts. Namely, the models haven't been exposed to the synthesized video frames. As an example, HAcc performances are relatively lower than BAcc performances. This reviewer wonders if the percentage of the synthesized pixels also matters to the performance. 
clarity: Figure 1 Right and Section 3.4: Are Random/Close/Far/Same related to the data distribution? If yes, how do the authors make the distinctions between the three categories? For example, in Line 149, the authors defined "Close" when the background is swapped with a video from a class that has a similar background. In line 160, it is more specified with "low L1 distance" but is still vague with "low". 

In Lines 157-164: the authors computed the average prediction probability of each class. Does the class indicate an action category or a scene category? If that refers to an action category, averaging scores per class relies on the assumption that videos in each class will share a similar scene, which could be correct but not always. This reviewer suggests computing the background similarity based on individual scene scores instead of average scene scores per action category.   

In Line 156, why do the authors only keep the videos with all the frames having human masks taking 5-50% of the pixels? That makes the benchmark unnatural and less interesting. What percentage of the videos satisfy the criterion? Did the authors use the subsets for testing the models in the results section?
relation to prior work: The authors showed the related work in section 2 by clarifying the distinction between the proposed work.
additional feedback: In Tables, I wonder if * can be replaced with some other character that might be middle-aligned instead of top-aligned. A better suggestion might be to show the statistics of the different settings by showing std. 

In Figure 3 (left), some dots are not covered with black lines.
review_title: Review of Enabling Detailed Action Recognition Evaluation Through Video Dataset Augmentation

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper:
- proposed a novel method to quantify the influence of background on model predictions
- proposed several metrics i.e. Background-Only Ratio (BOR), Human-Only Ratio (HOR), Swap Human Accuracy (SHAcc), Swap Background Error (SBError) in addition to accuracy
- benchmarked a variety of models (74 trained activity recognition models) on an augmented dataset and provide insights about what makes certain models robust to background
- studied the impact of scene background on action recognition 

strengths: - Novel method to quantify the influence of background using Human-Only Ratio (HOR), Background-Only Ratio (BOR), and SHAcc and SBErr on action swap methods (same, random, close, far)
- Combined image segmentation and video inpainting to generate background-only, human-only and action-swap dataset
- Provided new metrics on top of accuracy which would be useful to quantify the influence of background on model predictions.
- Provided some valuable insights into design choices that can improve or worsen the influence of background on model predictions based on their evaluation using BOR and HOR i.e. activity recognition models lean towards using background to predict action and high accuracy models tend to focus on human better than other models. 

weaknesses: - While one of the main highlights of the paper was that it introduces a human-centric Analysis Toolkit, it is confusing as there is no public link nor code provided in the supplementary materials at this point of review. 
- The paper analyzes 74 trained activity recognition models, but there were only a total of 19 architectures trained on different dataset, backbones, pretraining, resolution, sampling, clip length, stride, number of clips. Only the effect of dataset, sampling, and clip length were discussed. The number of clips, backbone, pretraining and resolution might also play a part in model performance and should be looked into.
- Table 1 and 2 compared models from different backbones i.e. ResNet50, ResNet101, ResNet152, TimeSformer which might not provide a far comparison. Backbone seems to play a part as models with better performance were also trained on larger backbones. 
Table 3 compared the different models but details about their backbone/ pretraining were not provided. 
- I cannot find any explanation of normal-scale dataset and large-scale dataset even though they were analysed in section 4.4 and used in Table 1, Table 2 and Figure 3 in the main paper and Table 3-6 in the supp. materials.
correctness: - I noticed that action swap will lead to unrealistic scenes i.e. floating face/ person in demo videos and I am concerned if that might affect accuracy. 
- I looked through the provided dataset and in some background-only videos, the person is still visible. This might affect the evaluation for BOR. Perhaps the paper could also mention this as a limitation i.e. the robustness of the dataset also depends on the accuracy of the image segmentation and in-painting model. 
- L257-261 “using a larger dataset just to improve accuracy is not an ideal training strategy” seems like a large generalization. What characteristics of the large scale dataset makes it not ideal? I.e. are there too many duplicates of similar backgrounds?
- L260-261  “background fixation of the model became worse when using a large-scale dataset incorrectly” - This seems to be a rather bold statement given (1) there is no explanation of what constitutes a large-scale dataset in the paper (2) there is only a single example given. Perhaps more experiments could be run to support this claim. The authors could also expound on how the large scale dataset is used incorrectly? And conversely, how to use it correctly? 

clarity: - In general, it is a very well written and concise paper
- I think Swap Human Accuracy (SHAcc) and SBErr (SBE) should be added and explained in section 3 under methodology. The first mention in Table 3 was slightly confusing.
- While the paper analyzes 74 trained activity recognition models, it is slightly confusing as only a few architectures were discussed in the main section. Table 3 in the supplementary materials showed that there were only 19 architectures but the model variants were trained on different dataset (normal-scale and large-scale), backbones, resolution, sampling, clip length, stride, number of clips, totaling 74 models. Perhaps some of the details about the models in the supplementary materials could be moved to the main section for clarity. 

relation to prior work: - For sections 4.2-4.4, there are quite a number of insights. More literature could be reviewed to see if other works might agree/ concur with the current observations, or if the observations are new and not discussed previously.
additional feedback: - It is a strong submission and provides a way to quantify the longstanding problem of the influence of background in human-centric problems. 
- Additionally, the paper could provide suggestions on how to combat the problem of background reliance from a data perspective i.e. train on their augmented action-swap dataset or collect better data with more diverse scenes? 
- The paper has some useful and interesting insights. Perhaps they could be summarised to give some recommendations on how to train a more robust activity recognition model i.e. use more frames for training, adopt TSM design that could exploit temporal difference to improve HOR etc 
- The modified Kinetics-400 was provided in the supplementary materials. I think it could serve as a new benchmark dataset to evaluate the different metrics. This can be added into the main paper as a feature.
review_title: Review of "Enabling Detailed Action Recognition Evaluation Through Video Dataset Augmentation" 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper describes a new approach to address the problem of capturing the impact of visual background cues for action recognition.
To this end, the paper relies on recent advances in image/video segmentation and inpainting to segment the human figure in each frame. This allows separating the person that performs the executed action from the background.
The paper proposes to use the two cues, background and the human figure, to test in three variations: Human-only, background-only, and action-swap. 
The evaluation is done extensively for various backbones and shows the impact of background vs. action under various aspects.
strengths: 
The discussion about the role of background vs. real action recognition and which visual cues play which role in action classification has been around for a long time. 
The proposed method and the respective data could help to assess this question for existing and future models in a consistent and reproducible manner. 

With that respect, the proposed dataset could have a significant impact on the deeper analysis of action classification models and architectures, mainly doing for action recognition what ObjectNet did for ImageNet. 

Furthermore, the availability of the used software could allow expanding those tests also to other datasets.   

Note that the assessment of strengths is based on the assumption that the newly generated data and software will be made available to serve as a new benchmark for the community. 
weaknesses: There are some questions open for me that could have a direct impact on the quality and usefulness of the proposed work:

- Do the new derivates (human-only, bg-only, action swap) from the Kinetics400 dataset comprise the same amount of video data? Or are they just subsets? In this case, what's the amount of videos overall and per class of the new distilled datasets?

- For the action swap, how do you make up for camera motion (or align it) in the original video as well as in randomly chosen background videos?

- Why exclude larger views >50% of pixels? Wouldn't that exclude e.g. applying make-up or other actions which are usually recorded in a close-up.

Indicate pertaining:
It is hard to draw any conclusion without knowing which backbones rely on which pertaining. Please add this information to the text and (ideally) tables. 
 
Human Action vs. Background for Human-Object Interaction
Another problem is revealed in Tab 4 which shows that using only the human might make limited sense in case of human object interactions and when it comes to tool usage. Here it would be good to open the discussion if things like screw drivers are part of 'using screwdriver' or not and how to capture that. 
On a side note, I'm also not clear why scuba diving is not recognized. Shouldn't the diving suit give enough information here? 

The relevant person might be to small
As all human figures are segmented, including bystanders that perform different actions, this might be an additional distractor. Would there be a good way of handling that?



Minor:
- Fig. 2 might profit from having the same scale for all four plots for easier comparability.

- It might be good to have some qualitative results in the main paper.
correctness: 
The claims made in the paper look correct and valid. The proposed metrics are sound and make sense for the task at hand.
clarity: The paper should be more explicit about the size of the newly generated data if it deviates from the original Kinetics-400 size. 
relation to prior work: There are several references missing with regards to two aspects: 

1) A similar idea has been tried for synthetic data, e.g. SURREACT: Synthetic Humans for Action Recognition from Unseen Viewpoints, Gül Varol, Ivan Laptev and Cordelia Schmid, Andrew Zisserman, IJCV 2021 and SURREAL, Learning from Synthetic Humans, Varol et al. CVPR 2017. Perhaps there might be more work in the synthetic or AR action area that I might have missed.

2) Discussing the role of background has a long tradition in action recognition actually. It would have been good to cover the discussion a bit, e.g.:
- Human Action Recognition without Human, Yun He et al., ECCVW 2016, https://openreview.net/forum?id=ryeujDzo
- Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition, J. Choi, NeurIPS 2019
- CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning, Rohit Girdhar and Deva Ramanan, ICLR 2020
- and many more ... 

additional feedback: Please see the questions under weakness and clarify how and which data you plan to release. 

Justification of final rating: As there was no response by the authors and there is no clear idea how the data would be made available (which is obviously the main purpose of the track), I lower my score from 7 to 4. 

Justification of final, final rating: After detailed discussion and the release of the toolkit and the respective validation videos, I agree with the other reviewers that this package has value for the community and I'm a happy to raise my score to the initial value.
review_title: Role of background in action recognition

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a new metric for studying the impact of background scene on the performance of action recognition models and provides a toolkit for computing it. The tool uses existing human segmentation and in-painting models to place humans in different background scenes to evaluate the impact of pose vs. background on model performance. The paper then uses this tool to evaluate the background-dependence of 74 existing action recognition models.

strengths: - This paper provides a helpful tool for analyzing a specific, known limitation of action recognition models -- background dependence. It uses this tool to produce a dataset in which humans are separated from background scenes and placed in new background scenes, to analyze the background-dependence of each model. It evaluates 74 different action recognition models on this dataset to understand how and if these models depend on scene background during classification.

- The paper's measurements are thorough, and the evaluation metrics succinctly capture the problem at hand, i.e. measuring how models perform when they see humans only vs. background only.


weaknesses: - Overall comment: While I appreciated the thorough evaluation of the background-scene dependence problem in this paper, the paper could be more thoughtful in its motivation and presentation of results. The tables and figures can be hard to read, and the authors do not explain well why they choose to present certain results but move other results to the Appendix. Finally, the authors provide few insights for addressing the problem highlighted in their work.

Specific comments:
- Results are reported in a confusing manner. For example, the paper claims to have evaluated on 74 different models (c.f. 4.1), but results on far fewer than 74 models are reported in the main paper body, with no explanation of why particular results are reported in main body vs. the supplementary. An example is Table 1, which reports results on 12 models and does not even cover all of the model designs listed in 4.1. Similarly, Figure 2 only presents results on 4 models -- why are these models chosen? This presentation should be better motivated and more systematic.

- Table 1 -- many results are said to be averaged across different settings. What settings? Why are no error bars reported?

- Figure 2: 
1. Color scheme must be more distinct. It is impossible to distinguish between "3" and "4" in the leftmost 3 figures.
How can you be sure that the number of frames used for training is the cause of the higher BOR in the leftmost Fig 2 graph, rather than the model structure itself? An ablation of model structure vs. different frame rates is never presented, so it's impossible to disentangle these two factors.
2. Error bars are not provided. Given the tight range of y-axis points in all four graphs (e.g. range is < 3 percentage points), it seems likely that the results may not be statistically different.

- Figure 3:
1. Bottom right figure: should x-axis be "Swap background err" rather than "Swap background acc"?
2. Color for TSN changes between the left and right figures, which was confusing.

- Table 4:
1. what is the "background category"? Is this defined as the label of the action originally performed in this scene, before the human was removed?
2. How to read this table? Are the elements in each line separate or related somehow (e.g. a belly dancer placed in "exercising arm" setting?)

- The paper analyzes a specific problem (background-dependence in action recognition models), but provides no tangible suggestions for future work beyond a few comments in Section 4. The research community would certainly benefit from hearing more future work suggestions towards improving this problem based on the analysis in this paper.
correctness: The paper's claims are well-motivated and appear to be correct
clarity: - Overall, the paper is well-laid out but could use more narrative text motivating specific actions and evaluations. For example, Section 4 drops the reader right into experimental details and evaluation without first explaining why the evaluation is ordered the way it is or motivating the particular experiments run. 

- The story of the paper is a bit confusing. The intro/abstract focus a lot on the need to reduce the cost of video dataset development, but the rest of the paper then focuses on/evaluates the effect of background scene on action recognition model performance via augmentations of an existing dataset. Based on the intro/abstract, I thought the paper was going to propose ways of creating new (training) datasets from existing video datasets, but instead it focuses on developing a specific validation set. Just a note that I found this hard to follow.

- Some small grammar errors (subject/verb alignment, etc), especially in intro (lines 23-24). "Tabel" --> "Table", line 214
relation to prior work: - Prior work is well-covered.

- Question: why can't you analyze the weights of a model trained on a specific dataset (para 3 of "Human-centric analysis" in Sec 2)? This assertion is made but not backed up. GradCAM (or related techniques) seem like they would still be appropriate for analyzing these models.
additional feedback: - It seems like low-hanging fruit would be to run HAT on an action recognition TRAINING dataset to produce a dataset of human-only videos and observe if background-dependence decreases in this new model. Why did the authors not perform this analysis?

- Why is it important that action recognition models detangle human action from the background scene? It seems like context would be important in distinguishing say, holding a baseball bat vs. holding a rolling pin. It is not clear from the paper why this separation is needed, rather it is just implied that it is important. This should be better motivated in the abstract/introduction.

- Section 3: could use more narrative text up front introducing WHY you create the three types of videos. The reader must intuit the motivation as being because "background-only" and "human-only" enable the creation of "swap".
review_title: Review of "Enabling Detailed Action Recognition Evaluation Through Video Dataset Augmentation"

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Many CV researchers know (or guess?) that the action recognition models tend to see the ``background scene'' of the video, rather than the human action itself.
However, a thorough inspection of the issue is hard to be conducted, because of the lack of a valid dataset.
The authors detour this dataset problem by adopting off-the-shelf inpainting/segmentation models.
Extensive experiments are conducted, and the results are reported thoroughly.

strengths: i) Important and relevant problem:
We all know that there's a big difference between just guessing and actually seeing it with concrete experimental results.
From this perspective, the paper's empirical results clarify the CV researcher's vague guess, which stimulates further research on the problem.
I really appreciate the author's effort in conducting such valuable experiments.

ii) Convincing experimental settings:
I think Only background, Only Human-action, Swap Action setting is reasonable, and the presented empirical results are interesting.

iii) Extensive experiments on different models:
Exhaustive experiments are conducted and the results are fairly reported.

Overall, I think that the experiments are convincing, and have a meaningful impact on the CV community.
weaknesses: i) Straightforward method and expectable empirical results: 
Utilizing off-the-shelf segmentation/inpainting models is quite straightforward, lacking technical novelty.
Besides, the empirical results in the paper double-check the well-known prior, rather than suggesting novel insights.
As the paper is not proposing a new dataset, I think at least some level of technical novelty or some new findings is needed.

ii) Suboptimal behavior of the off-the-shelf models:
Using off-the-shelf models has an inherent limitation due to the suboptimal performance of the adopted off-the-shelf models.
It is explicitly mentioned in the paper as well and can be clearly seen in the videos in the supplementary materials.
Thus, I think that the suboptimal behavior of the off-the-shelf model may affect the experimental results in some sense, but we do not exactly know ``how much'' it affected the results.

iii) camera--ready release claim:
The authors noted that "We plan to offer the official implementation of the toolkit by the release of the camera-ready version of the paper." in their supplementary materials.
As the track is about datasets and benchmarks, I think it should be open at the review period, not at the camera-ready period.
I think it is the most critical issue in this submission.

I really appreciate the authors' effort in conducting such thorough experiments, and think the empirical results are valuable for the CV community. (Much higher than the acceptance threshold) However, unless the toolkit is provided with sufficient documentation, I cannot recommend acceptance of the paper.

correctness: Reasonable experimental settings, but as it uses off-the-shelf ML models to preprocess video, the empirical results here might not exactly match with the results on real videos.
clarity: The paper is well-written.
relation to prior work: Related works are clearly presented.
additional feedback: I'll be happy to raise my rating if the authors provide access/good documentation of the toolkit.
review_title: Review

Meta-review: metareview: The paper provides an evaluation toolkit and leaderboard for comparing current action recognition models on the famous Kinetics-400 dataset for their background bias. This is achieved by using publicly available training weights and carefully curating a validation set to quantitatively assess the impact of such bias.

Before the rebuttal, the reviewers were on the negative side due to writing confusion as well as the fact that the toolkit was not submitted with the first version.

Luckily to the authors, the reviewers were happy to engage actively with the revised version that seems to have resolved their concerns. They all recommend acceptance.

As an AC, I think the authors should have provided their toolkit with the first submission for thorough and detailed assessment, as this is their main contribution. But, in light of all reviewers recommending acceptance, I believe this is sufficient proof of the value to the research community and would thus recommend acceptance of this work as a poster. 

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept