Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In this paper, authors create a novel, large-scale dataset (named VFP290K) for the detection of fallen persons composed of fallen person images collected in various real-world scenarios. They empirically demonstrate the effectiveness of the features through extensive experiments analyzing the performance shift based on object detection models.
strengths: 1. Presents a relatively diverse and extensive dataset on fallen person detection task. The dataset is publicly available for download and research purposes too. 
2. The visualization and evaluations presented in the paper are very extensive.

weaknesses: 1. One of the limitations of the work has to be a better motivation for the work. I understand that falls can be fatal and some of the numbers showcased in the introduction are well grounded on numbers from WHO and CDC. However, I would like some discussion as to how a fallen person detection task can come handy in avoiding some serious health related consequences in such situations. 

2. While the most of the paper is very well written, I find the Ethics and Social Impact paragraph (in section 3.5) lacking some explanations. Can the authors put some lights on what sort of agreements were agreed upon by the volunteers and how has the anonymization process taken place for people who did not volunteer and were still in the frames (say while a volunteer is on the frame some non-volunteers might well be in frame too). Further clarification is also necessary on the line: ``we discarded the videos, if a person did not wish to be in.''.
correctness: The technical aspects of the work seem correct. 
clarity: The paper is very well written. 
relation to prior work: Yes, the same has been discussed clearly.
additional feedback: Please refer to weakness section.
review_title: VFP290K: A Large-Scale Benchmark Dataset for Vision-based Fallen Person Detection

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Authors release a large-scale dataset for fallen person detection in CCTV cameras. The dataset is curated under various conditions including lighting, camera heights and backgrounds. They perform an analysis on the metadata and feature-level distribution and demonstrate a good variability of the dataset. Experiments using popular object detection models including one-stage, two-stage and transformer-based were conducted to benchmark the performance in general and under different environment conditions. 
strengths: -	The dataset achieves 1st place at the AI Grand Challenge 2020 in South Korea, which is operated by the Government of South Korea.
-	The dataset is the largest up-to-date for fallen person detection, significant larger than the 2nd largest dataset (2x).
-	The dataset considers various conditions (day/night, camera heights, backgrounds) that other datasets lack.

weaknesses: -	Although demonstrate clear rules (Occlusion Rule and Overlap Rule) for quality control for the annotation procedure, there still exists ambiguities:
o	How would annotator categorize people during their falls (not completely on the ground)? There should be a clear definition for this binary classification problem.
o	Authors said the annotations for each frame are cross-check between 5 different students but did not specify the rule for interrater agreement and the criteria to accept an annotated frame.
-	As the dataset is composed of frames from multiple videos, there might be redundancies where consecutive frames are similar if the frames-per-second (fps) of the captured videos is high, which the authors do not specify. Train/test split should also be split video-wise, not frame-wise, which the reviewer found no information.

correctness: The construction of the dataset is reasonable and considerate. The author tries to extend the variety of the dataset.
clarity: The paper is structurally well-written and easy to follow.

relation to prior work: The authors demonstrate a good comparison to other similar dataset, highlighting their strengths against others.
additional feedback: None
review_title: A large dataset with high variety for fallen person detection.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introduced a large-scale dataset for studying the problem of fallen person detection. Unlike existing datasets, the authors collect data in diverse environmental and lighting conditions. They collect their data with a single GoPRO HERO 5 camera and 15 volunteers that perform different fall scenarios. The data was collected in 39 different locations in South Korea, totaling 178 video clips. This way, the authors aim to address the existing limitations of existing methods for falling person detection which lack robustness to environmental and lighting conditions. The dataset contains annotations for both fallen and non-fallen persons.
strengths: - the dataset serves a clear purpose and facilitates study of an important application of object detection systems
- the authors conduct a comprehensive study of baseline object detectors and publicly release their data, the evaluation code and the trained baseline models
- the dataset seems to be substantially more diverse than existing datasets in this area
weaknesses: - the challenges that this dataset addresses are important for the specific application of fallen person detection, however they don't seem to facilitate more fundamental study of the challenges that arise in the underlying algorithms that are used here (namely deep learning based object detectors). The algorithmic challenges that are of interest to the Neurips community (limited generalization due to lack of background / lighting / viewpoint / occlusion diversity in the training data) have been addressed by many large-scale object detection benchmarks in the past.
- While the background and lighting conditions are diverse, the set of persons that perform the fall scenarios seems quite small (15). Here, the algorithms could still be prone to varying appearance across humans (e.g. elderly vs young, different skin color). 
- While i understand that it can be challenging to obtain real falling person scenes, and it is much more efficient to let volunteers perform the fall scenarios, it has the disadvantage that an algorithm performing well on exclusively acted fall scenarios might not generalize to real footage of falling persons. Hence it would be good to rely on real footage at least for the test data.
correctness: Yes, except for the two points:
- only 15 volunteers do not capture the varying appearance of humans
- all fall scenarios are staged, which might look different from real footage of fallen persons
clarity: Yes
relation to prior work: Yes
additional feedback: The dataset might be better suited for an application oriented computer vision audience.
review_title: Great dataset for more robust fallen person detection, but limited broader impact

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a large-scale dataset Vision-based Fallen Person (VFP290K) for the detection of fallen persons, It consists of 
294,714 frames of fallen persons extracted from 178 videos, including scenes in 49 locations. This is 3x more than [35]. Also the dataset does vary in light condition (day vs night), camera height (low vs high)and  background (street vs park vs building). The authors investigate in the experimental section the usefulness of their dataset for the task of binary classification between fallen and not-fallen person in several experimental settings.
strengths: The dataset seems to be the largest and the most varied from the literature. The experimental evaluation shows that on this dataset the problem of detecting fallen persons can be solved with moderate success by state-of-the-art object detectors (Faster-RCNN, RetinaNet, YOLO). The experimental evaluation with this respect seems to be well carried.
weaknesses: -My main concern in building this dataset is that it contains videos "performed" artificially by the volunteers. It is not clear that a model trained on this dataset will generalize well on real cases, not performed. 

-It would have been nice to see some experiments also on the anomalous event detection task.
correctness: Seems to be correct.
clarity: The paper is written mostly clear, there are some sentences that sound a little bit awkward: 
-"Given that many detection systems rely on a comprehensive dataset comprising fallen person images collected under diverse environments and in various situations is crucial."
-"We weigh on the street than a park or building for the background since the CCTV is likely to be installed more on the street environment."
relation to prior work: Seem to be well addressed.
additional feedback: See weaknesses.
review_title: The paper compiles a fallen-person benchmark in various settings

Meta-review: The reviewers acknowledge the novelty of the fallen person detection dataset. Although it has some drawbacks such as limited number of subjects and acted events, this is very difficult to have real data for the problem. Overall, the data and provided analyses show unique aspects and this can be a good resource for the community.
Decision: Accept