SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 11: Distractor vs Background-Context Sensitivity: Highest performance drops are observed when background is masked (no context), whereas presence of distractor objects in background has slightly less effect.
Methods	UCF-24	JHMDB-21
Clean	Distractor	No Context	Clean	Distractor	No Context
Abs	δr	Abs	δr	Abs	δr	Abs	δr
MOC[46]	54.4	38.6	0.71	26.6	0.49	77.2	53.7	0.70	49.5	0.64
YOWO[39]	48.8	43.8	0.90	32.2	0.66	85.7	73.5	0.86	66.5	0.78
VCAPS[13]	75.5	67.7	0.90	53.1	0.70	65.7	61.8	0.94	47.5	0.72
Figure 8: Occlusion Robustness across multiple severity levels. The outermost envelope represents most robust model. Radial Axis represents vmAP-0.5 score. For O-UCF, VCAPS is most robust, whereas for O-JHMDB, YOWO is most robust across several categories.
Table 12: O-UCF Benchmark: Illustrates the full benchmark across 9 severity levels of static occlusions and different trajectories in dynamic occlusions.
Methods	Occ As Aug	BG1	BG2	BG3	Circle	Sin	Avg
FG1	FG2	FG3	FG1	FG2	FG3	FG1	FG2	FG3
MOC[46]	×	54.7	37.0	34.7	52.7	35.6	31.4	53.3	33.1	22.5	42.9	39.9	39.8
YOWO[39]	×	38.6	32.6	27.3	39.1	32	27.1	37.9	32.8	26	24.1	25.7	31.2
VCAPS[13]	×	51.4	44.2	37.3	50.7	43.6	37.6	52.4	40.7	35.7	31.8	39.1	42.2
MOC[46]	✓	48.3	43.6	39.6	47.3	43.1	39.1	47.2	44.6	38.5	48.1	45.6	44.1
YOWO[39]	✓	46.5	45.3	43.4	46.1	45.3	43.4	46.5	45.7	44.3	43.8	43.6	44.9
VCAPS[13]	✓	54.7	51.1	48.7	54.8	50.4	47.6	54.2	50.7	47.8	43.4	45.3	49.9
Table 13: O-JHMDB Benchmark: Illustrates the full benchmark across 9 severity levels of static occlusions and different trajectories in dynamic occlusions.
Occ As Aug	BG1	BG2	BG3	Circle	Sin	Avg
FG1	FG2	FG3	FG1	FG2	FG3	FG1	FG2	FG3
MOC[46]	×	48.1	35.1	30.5	46	33	24.6	41.4	33.7	22.6	39.8	33.9	35.3
YOWO[39]	×	52.5	48.6	47.3	50.6	47.3	45.8	48.4	47.2	45.5	45.7	47	47.8
VCAPS[13]	×	50.7	37.1	20.5	47.3	34.5	22.8	49.5	34.7	21.9	38.8	33.8	35.6
MOC[46]	✓	59.6	55.9	53.3	58.9	54.3	52.9	58.5	54.8	52.2	57.2	57.2	55.9
YOWO[39]	✓	71.1	70	69.3	69.2	70.3	66.9	68.4	67	65	65.7	65.1	68
VCAPS[13]	✓	60.8	59	54	61.2	56.9	53.9	61.5	57.8	54.4	55.3	58.2	59.7
A6. Full Benchmark Analysis
In Tables 12, 13, we present the full benchmark of the proposed O-UCF and O-JHMDB datasets across the 9 severity levels of static occlusions and circular/sinusoidal dynamic motions.
The only difference is VIS involves instance-classification which could be solved by just a single-frame object detection. However, action detection requires higher-level action-classification which requires temporal reasoning. The per-frame localization task of both the problems is fundamentally identical.
Figure 9: Background Bias: As the occlusion severity over background is increased, the performance of an ideal detector should not drop (dashed line). Our VCAPS-Mvitv2 shows highest robustness of all studied models. (X axis): Different severity levels of background occlusions. (Y Axis): Relative Robustness  $(\delta_r)$
Table 14: Static vs Motion Trends: 2D backbones work well on Static Occlusions whereas 3D backbones work best under Dynamic Occlusions. S- Static Occluders. D- Dynamic Occluders. For analysing clear effects, results are shown with only one occluder over an actor (no bg occlusion).
UCF-24	JHMDB-21
Methods	Backbone	Type	Clean	S	D	D-S	Clean	S	D	D-S
MOC[46]	DLA34 [79]	2D	54.4	28.7	27.3	-0.4	77.2	34.4	34.0	-1.1
YOWO[39]	YOLO +ResNext	2D+3D	48.8	24.0	25.2	1.2	85.7	54.6	55.2	0.6
VCAPS[13]	i3D [7]	3D	75.5	42.5	46.6	4.1	65.7	23.0	23.9	0.9
Background Bias: For an ideal action-detector, it is expected that occlusions in the background wont impact the localization performance in the actor region. We observe this trend in Fig9, where the performance of an ideal detector would be a black line parallel to x-axis. In the graph, most of the methods suffer negligible drops on increasing the occlusion severity in the background. Note that MOC[46] suffers most drop across different severity levels. Our VCAPS-Mviv2 obtains the highest robustness across all occlusion severity levels for both O-UCF & O-JHMDB datasets.
A7. Motion Analysis
In the main manuscript, we had experimented with realistic motion on the OVIS-dataset. We had also experimented with synthetic motions on O-UCF and O-JHMDB datasets, and show those results in Table 14. We observe that in case of MOC using the 2D DLA34 backbone, the performance is better in case of static occlusions, (i.e. 28.7 vs 27.3 on O-UCF). This shows that 2D backbones work well if occluder is static. On the other hand, YOWO uses a combination of 2D backbone (YOLO) and a 3D backbone (ResNext). This shows a slight positive improvement of  $1.2\%$  in UCF and  $0.6\%$  in JHMDB, thereby indicating that 3D backbone seems to be helping. The best improvement is evident by purely switching to a 3D based backbone as in the case of VCAPS, where gains as much as  $4.1\%$  can be observed. From this, it can be seen that 3D backbones might help networks reason about actor locations even under the challenging scenarios when the occluders might be moving. However, we note that these differences are close within a small margin of error. Also, our experiments with different type of trajectories in Tab1213, i.e. circle and sinusoids show inconsistent-trends thereby indicating that there is no particular motion type to which networks are more sensitive.
A8. Real-UCF dataset
We have curated Real-OUCF dataset for realistic occlusion scenarios.
Video Selection Process: The original videos in UCF-24 dataset were mostly of sporting events, but were of significantly lower resolution, i.e. 240 by 320. Furthermore, they consisted of only a single actor. Now, we have significantly upgraded that test set to reflect multiple actors which mutually occlude each other with a high degree of overlap as high as  $99\%$ . First, we scraped the videos matching keywords like "riding bike" etc. One curious way we were able to get such overlap ratios
was by searching specifically for events like tandem surfing, tandem diving etc. In tandem-events, two actors try to synchronize with each other. A lateral captured viewpoint, which captures multiple actors (with one actor behind another), offers very challenging yet realistic conditions for occlusions. Finally, our curated videos are consisting of professional sporting events like Olympics, as well as casual settings, for eg, people just playing cricket in a park.
Annotation Criteria: In spatio-temporal action detection, there are two questions that the annotations try to answer 1) what is the time interval during which the action occurs 2) what is the spatial location of the actor in each frame of the action-interval. Therefore, we first temporally crop the scraped videos to obtain action start and end times. Finally, for all the frames in between this interval, we spatially localize the actors using the CVAT annotation tool. One subtle thing is that action-detection does not try to label all the actors in the video. For eg, if some people are standing, and some people are doing some useful activity like biking, we are only concerned with the biking. This is in line with the annotation procedure of official UCF24.
A9. Dataset Samples
In this section, we show some qualitative samples from our proposed-datasets and benchmarks. Specifically, Fig1011 shows the realistic occlusion dataset which we have collected for evaluating robustness to real-world occlusions. Next, Fig1415contains the synthetic dataset samples from O-UCF and O-JHMDB consisting of controlled occlusions. Fig1213 contains the exhaustive instance-level annotations of UCF-24 videos we have released to facilitate our proposed benchmark of action-segmentation.
Figure 10: Our Real-OUCF: contains realistic occlusions with instance-level action annotations.
Figure 11: Our Real-OUCF: contains realistic occlusions with instance-level action annotations.
Figure 12: Our instance-level annotations for UCF-24: We propose a new benchmark of instance-level action segmentation and release official annotations.
Figure 13: Our instance-level annotations for UCF-24: We propose a new benchmark of instance-level action segmentation and release official annotations.
Figure 14: Different occluder trajectories: are illustrated for our proposed O-UCF and O-JHMDB datasets.
Figure 15: Nature of Occluders: Occluders in the proposed O-UCF & O-JHMDB datasets belong to either indoor/outdoor samples.
A10. Datasheets For Datasets
O-UCF, O-JHMDB were created to perform systematic benchmark study of occlusions in video action detection. Real-OUCF was created to evaluate sota action-detectors on real-world occlusions.
These were created by a research-group whose identity shall be released after the review process.
Dataset contains videos of several people performing actions.
For each instance, we provide an instance-level segmentation mask along with our datasets.
The idea for a real-world occlusion dataset is that it is easy to train an existing network using synthetic occlusions as a data augmentation. However, there is a need of consistent real-world tet set to benchmark all the future approaches against, and that is the test set which our Real-OUCF provides.
A very detailed care has been taken to make our annotations noise free. In the rare case that a correction is needed, the updated annotations will be posted on our official link (Shared on top of this manuscript).
All the videos of our dataset were obtained after hand picking from YOUTUBE, and then scraping them. Due to offline availability of the dataset, our dataset is now self-contained.
Yes, it is spatio-temporal action detection. It primarily considers human actors.
The classes in our Real-OUCF dataset primarily belong to sports such as basketball. Several videos of the dataset are picked from official sporting events like NBA, Olympics where it might be possible to identify famous athletes by face only. However, all the videos we use are already there in the public domain.
Data was raw-video directly observed with human eyes.
Manual human curation
Graduate students, who were graciously supported by research grants with warm thanks.
Dataset was collected over a period of 6 months from Dec22-May23.
Pre-processing was done by temporally cropping a video into smaller clips which indicate start/end of the action. After auto-labelling spatio-temporal masks via SAM, we manually annotate/refine masks using the CVAT tool.
Link shall be provided soon for the raw data.
Yes, we use normal Segment Anything and Computer Vision Annotation Tool. https://github.com/facebookresearch/segment-anything https://github.com/opencv/cvat
Yes, for our benchmark study on the impact of occlusions in spatio-temporal video action detection.
Amodal mask completion, Occlusion Robustness Testing etc.
Just before the start of the neurips 2023 conference.
No, its free for use by all parties. However, the discretion to update more video samples in future, and refine the existing annotations over time lie with the original dataset authors.
Dataset will be supported by the authors/research group of this paper.
By email-address, the details shall be revealed post-review.
Changes if any, will be posted once a year to our github repo.
Yes, this shall be useful to give reliable comparisons on different dataset versions in existing literature.
We also release a python script which allows to create on-the-fly semi-realistic occlusions consisting of static occlusions at different severity levels and different motions of the occluder. Anyone could use this to generate infinite occluder variations.
For realistic occlusions, once could always collect more real world samples or expand the count of existing classes. Note that the test set we provide is already significantly larger than traditional action-detection datasets. [61, 33]
We are grateful to you for taking the time to read this datasheet and reviewing this manuscript.