SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Benchmarking computer vision models for occlusion-aware video action recognition can lead to significant advancements in various application domains like development of enhanced surveillance systems which take occlusion into account, autonomous driving among others. These enhanced surveillance systems might cause some privacy concerns. For synthetically occluded datasets since, we sample objects randomly from the PASCAL VOC dataset the occluder can often present an unrealistic appearance both since the texture of occluder is significantly different from that of rest
Figure 14: Class wise distribution of samples in UCF-19-Y-OCC dataset.
of the scene but also, the motion of exhibited by the objects is relatively simple as opposed to the complicated motions followed real life occluders.
A.10 Datasheet for Dataset
Motivation The core motivation behind this dataset was to study the effect of natural occlusion on video action recognition models, which could further help in moving the field forward by studying the performance of models systematically and in a real world setting.
Composition Content and Composition The instances in the dataset consists of videos in which actions are completely or partially occluded. The UCF-101-Y-OCC dataset consists of 19 classes, each class consists of 30 clips each lasting 5 second. The UCF-101-O and K-400-O consists of all the videos in test split of UCF and Kinetics dataset occluded synthetically.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? Yes for K-400-O and UCF-101-O, No for UCF-101-Y-OCC.
Are there recommended data splits (e.g., training, development/validation, testing)? No, all the proposed datasets are for evaluation only.
Are there any errors, sources of noise, or redundancies in the dataset? No, The entire UCF-101-Y-OCC dataset was annotated by the authors.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? UCF-101-Y-OCC is composed of YouTube videos. UCF-101-O and K-400-O are self-contained.
Is there a label or target associated with each instance? Yes, a class label representing the action is associated with each dataset.
Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of
Figure 15: Qualitative results for occlusion localization. Each row represents a video followed by localization of occlusion in each of the above frames.
government identification, such as social security numbers; criminal history)? No.
Does the dataset identify any subpopulations (e.g., by age, gender) No.
Figure 16: Qualitative results for occlusion localization of CTx-Net (augmented). Each row represents a video followed by localization of occlusion in each of the above frames.
Figure 17: Examples of videos in which Video MAE is unable to classify correctly whereas MViT and MViTv2 are able to correctly classify.
Figure 18: Visualization of images used for training occluder model
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? Yes, the individual might be identifiable from YouTube.
Collection process Each instance of UCF-101-Y-OCC was extracted from publically available videos on YouTube. The entire video was then viewed to ensure the action occurring is relevant as well as a part of it is occluded. UCF-101-O and K-400-O are composed of test split of UCF-101 and K-400 datasets.
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? YouTube API was the only external source of data used for UCF-101-Y-OCC. Annotations for UCF-101-Y-OCC were performed manually by the authors.
If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? Not applicable since UCF-101-O and K-400-O contain entire test split of UCF-101-O and K-400-O.
Over what timeframe was the data collected? 2 Months for UCF-101-Y-OCC.
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? YouTube was used for UCF-101-Y-OCC. UCF-101 and K-400 were synthetically occluded to obtain UCF-101-O and K-400-O.
Were the individuals in question notified about the data collection? Data was publicly available on YouTube.
Labelling/Preprocessing/cleaning The video collected from YouTube was the segmented temporally into 5 seconds segments which contained occlusion. UCF-101-O and K-400-O have annotations same as UCF-101 and K-400 whereas UCF-101-Y-OCC was annotated manually.
Uses Has the dataset been used for any tasks already? For testing the performance of current models when exposed to occluded actions.
Is there a repository that links to any or all papers or systems that use the dataset? NA
What (other) tasks could the dataset be used for? Action Recognition under occlusion is the only intended task for the proposed dataset.
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses No.
Distribution Will the dataset be distributed to third parties outside the entity (e.g., company, institution, organization) on behalf of which the dataset was created? No
How will the dataset be distributed (e.g., tarball on website, API, GitHub)? Will be available along with the code through download link.
When will the dataset be distributed? During the review process.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? Dataset will be distributed under Creative Commons License.