Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper curates existing synthetic datasets to compile a new dataset to benchmark video action recognition.
When using the resulting dataset for pre-training action recognition models, it appears that synthetic pre-training benefits the performance on real datasets differently.
The paper explores the idea of a bias towards objects behind these performance discrepancy.

strengths: (+) The paper proposes a compiled and curated version of multiple synthetic video action recognition datasets, by combining classes, normalizing the frame rate, or balancing the number of samples per class.

(+) The paper provides a valuable list of advantages of synthetic datasets, discussing privacy and ethics issues.
weaknesses: (-) Motivation for not reporting state-of-the-art performance not convincing  
The paper only reports results on a pre-training with a subset of Kinetics. The authors justify this with an emphasis on fair comparisons between synthetic and realistic pre-trained models. Still, it would have been interesting to know what is the empirical upper bound is at the moment and what is needed to bridge this performance gap (e.g., is it more samples? better rendered videos? more diverse videos?)

(-) Unclear goal of the benchmark given the evaluated method
* If the emphasis of the paper is to compare synthetic and realistic pre-trained models, how come there is no domain adaptation methods that were assessed?
* On related topic, it is hard to grasp the main message of the benchmark. What would the authors recommend for future works working on such benchmark? e.g., should the models focus on better leveraging synthetic data, better transferring knowledge, mitigating dataset biases?

(-) Biases metric scores in Table 2 would benefit from a more thorough analysis  
It is a bit hard to understand and interpret the scores. For example, Mini-SSV2 has a representation bias score very close to HMDB51 but is considered as focusing more on temporal movement. Similarly, IkeaFA has a very high performance with ImageNet backbone but is considered as focusing more on temporal movement. What is the explanation for these two datasets contradictory?

(-) Statements can be overclaimed  
It would be preferable to avoid terms like "theorized" as the paper does not propose any theory but rather some hypothesis on why some datasets might benefit more from a synthetic training.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: n.a.
review_title: Benchmark based on compiled datasets shows the utility of synthetic data for pre-training in video action recognition

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: 1. The authors introduce SynAPT, a novel benchmark of synthetic video clips compiled from existing datasets 
2. The authors provide the first comprehensive study of the transferability of action representations based on synthetic videos to diverse downstream task, showing a close relationship between the sim-to-real gap and the object/scene bias
3. The authors address ethical concerns with existing datasets



strengths: 1. SynAPT provides a useful benchmark for the action recognition community, while minimizing issues caused by ethical concerns and object/scene bias of real videos
2. The extensive study of the sim-to-real gap in action recognition addresses important challenges in the research community
3. The experimental design, including choice of evaluation dataset and model architecture, is well thought out and shows an understanding of the field.
weaknesses: 1. For some dataset (especially UAV-human), linear probing accuracy is extremely low for both real and synthetic datasets, and performance is even lower for the "scratch" model. The 1.26 accuracy for R(2+1)D seems suspiciously low, even for a randomly initialized model, on a dataset with 22,476 video clips.
2. All backbones are initialized randomly, without ImageNet (Line 200), without justification. I would like to see at least the "scratch" model with ImageNet initialization.
correctness: 1. Points to an important challenge with synthetic training: "domain adaptation techniques... assume the same label set for both synthetic data and real data" (Line 45). However, I'm not sure this is quite true. For example, there are many recent methods which use unsupervised pre-training on human skeleton data, then finetune the learned features to predict unseen actions (e,g. https://arxiv.org/abs/1911.12409 and https://arxiv.org/abs/2010.05599)
2. The experimental design is reasonable, with the authors using several large and well-known action recognition datasets to evaluate transferability from synthetic data. The authors train several architectures, TSN, I3D, and R(2+1)D, which are a good choice since they are very commonly used in the community.
3. The dataset is constructed soundly by combining existing data to form a class-balanced synthetic dataset combination
clarity: The paper is well-written and well-organized.
relation to prior work: Yes. Unlike previous works, the authors conduct a study of the transferability of learned representations from models trained on fully synthetic videos to downstream tasks. SynAPT is constructed from three previous synthetic datasets (ElderSim, SURREACT, and PHAV). 
additional feedback: The authors should provide the script to generate the dataset. They should also provide evaluation code so that future researchers can fairly compare their models on SynAPT.
review_title: Review for the SynAPT benchmark

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper constructed a benchmark dataset SynAPT by combining the existing synthetic datasets. They pre-trained different CNN-based models with the benchmark dataset to evaluate the transferability of the video representation based on synthetic data. By comparing the results obtained from real datasets like Kinetics, they hypothesize that the performance gap from representation between the real and synthetic datasets is highly related to contextual bias and static objects. They finally conduct an ablation study to show the impact of the hyperparameters in dataset construction.
strengths: This paper explores a very important direction that we train with synthetic video datasets and test with real data. It can alleviate the privacy, ethical and data scarcity issues in ML world, especially for the action recognition task. The transferability of the pretrained models based on synthetic data is an important but under-explored topic for action recognition. The experimental results show that training with synthetic datasets can obtain more general video representations than the Kinetics dataset on several downstream classification tasks. 
weaknesses: Though it is an important direction to explore the transferability of representation based on synthetic data overall, this paper lacks several key elements. There are several things to be clarified and provided:
1. The dataset documentation is missing. There is no detail for dataset construction, availability and maintenance in either Supplementary or web URL link. It makes the benchmark unreproducible.
2. The reason to choose the 150 classes for dataset construction is not clear. Are these classes randomly chosen from three synthetic datasets? or based on a specific standard. Although there is a sensitivity study in Chapter 4.3, the plots in Figure 3 show that the accuracies are not saturated when the classes increase from 30 to 150. 
3. The representation bias of the proposed SynAPT and Kinetics are missing. Table 2 shows the bias of the downstream classification datasets based on ImageNet-pretrained model. However, the bias of training datasets including SynAPT and Kinetics is missing. The relationship between the bias of training data and test data is ignored, which makes the conclusion a bit weak.
4. Why the Kinetics is used as the representative of the 'real dataset'? To my knowledge, Kinetics is composed of public Youtube video dataset, which has similar size as SSV2 dataset. It is not surprising that Kinetics video representation can transfer better to HMDB51 and UCF101, because these 2 small datasets are fully/partially extracted from Youtube as well. 
5. The transformer-based deep learning models are missing. The transformers like Timesformer, Swin, Mvit demonstrate superior performance on action recognition tasks in recent years. 
correctness: There is no documentation of dataset construction and the chosen classes are not well explained. Hence, the construction of the current dataset is problematic. 

Using the Kinetics dataset as the only 'real dataset' to compare with the synthetic dataset can make the hypothesis/conclusion weak. Though it is the most popular dataset in action recognition, we cannot get a general conclusion with such a single dataset.
clarity: There needs improvement in the writing. Table 1 shows the most important results in this paper, but the analysis based on it is little. It loses the connection with the assumption in the last paragraph of Chapter 4.2. 

The motivation for only using Kinetics for comparison and the construction of the dataset needs to be further clarified.
relation to prior work: The discussion on prior work is sufficient.
additional feedback: I would suggest the authors use more datasets for pre-training, to improve the generalizability of the hypothesis. At the same, exploring the relationship of the bias between training and test data.
review_title: An important work to study the transferability of synthetic dataset in action recognition but lacks some key elements

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper uses existing synthetic datasets to pre-train models and then transfer to large scale action recognition datasets with disjoint label sets. The different datasets considered have a range of scene-object bias with the synthetic pre-training outperforming (or performing equally well wrt) the kinetics dataset pre-training on low scene-object bias datasets.
strengths: 1. Interesting analysis on the strength of pre-training using synthetic datasets vs real datasets from the downstream action recognition datasets. Results show models pre-trained on synthetic dataset performing equally well or outperforming the models pre-trained on kinetics when the datasets have low scene-object bias. A variety of downstream tasks are considered.
2. The downstream tasks that follow the pre-training have disjoint label sets. This shows that the SynAPT pre-training could be useful for a wider range of downstream tasks as there is no label constraint.
3. Interesting insights are provided into the effect of representation bias on the transferability and of number of classes on the performance.
weaknesses: 1. For the sake of completeness, it might be interesting to report if similar/different results are observed in the transfer learning for the gap between the synthetic and real dataset pre-training on datasets with the same label set. 
Further, since the label set is different, it might be nice to report a comparison with no pre-training. This would solidify that the pre-training indeed helped improve the results in the first place.

2. Is the choice of the real dataset (kinetics) well balanced with the choice of the combination of synthetic datasets? Are there differences between the two datasets beyond them being real/synthetic that could be biasing results? Would choosing a different "real" dataset affect the result?

correctness: overall correct. some remarks are listed in the weakness section.
clarity: very clear. enjoyed reading the analysis.
relation to prior work: clear.
additional feedback: listed in weakness section.
review_title: Introduces SynAPT, a benchmark for pre-training action recognition models from existing synthetic datasets with extensive analysis

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a systematic study on the pre-training potential of synthetic data on action recognition. A new action recognition benchmark (synthetic data) is proposed to mitigate the privacy, ethics, and copyright issues caused by real videos. Based on the study, several analyses are conducted, which could inspire future studies on synthetic action recognition research.
strengths: 1.	This paper is the first to systematically study the pre-training potential of synthetic data on action recognition.
2.	Extensive experiments have shown that the synthetic dataset can work well on most downstream datasets, compared with Kinetics dataset.
3.	This paper is well written.

weaknesses: 1.	Missing more ablation studies. AFAIK, the most current video action recognition works sample 16/32 frames for each clip, while the authors sample 8 frames in this paper. The authors are suggested to conduct experiments on 16 frames.
2.	Missing the experiments on transformer-based models, AFAIK, most current state-of-the-art models are transformer-based (e.g, TimeSformer, vivit, MViT). So, it is important to conduct experiments on such models.

correctness: Correct
clarity: Yes, this paper is well written.
relation to prior work: Yes, clear relationship to prior work is discussed.
additional feedback: It is better to provide which 150 classes in Kinetics dataset are chosen, this will help the follower who will use this benchmark.
review_title: Interesting study but missing some key evaluations

Meta-review: metareview: This paper had 5 reviews. In response to the reviewers' comments, the authors performed additional experiments. Considering that the dataset consists of synthetic videos, there are no concerns related to ethics, privacy, and copyright. Public documentation for benchmark reproducibility is available on GitHub. Overall, the authors responded satisfactorily to the reviewers' comments, and all reviewers were satisfied with their response. The paper is well written and its relation to prior work are clear. The paper is original and of significance to deep learning and action recognition.  
confidence: 5: The area chair is absolutely certain
Decision: Accept