On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
Authors: Rajat Modi
Vibhav Vineet
Yogesh S Rawat
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: mortal computation
occlusion
capsules
transformers
action detection
TL;DR: First benchmark study of occlusions in video action-detection, 5 new benchmark datasets and shows that transformers + some capsule layers  are significantly robust to occlusions, islands of agreement can emerge in realistic images.

Abstract: This paper explores the impact of occlusions in video action detection. We facilitate
this study by introducing five new benchmark datasets namely O-UCF and O-
JHMDB consisting of synthetically controlled static/dynamic occlusions, OVIS-
UCF and OVIS-JHMDB consisting of occlusions with realistic motions and Real-
OUCF for occlusions in realistic-world scenarios. We formally confirm an intuitive
expectation: existing models suffer a lot as occlusion severity is increased and
exhibit different behaviours when occluders are static vs when they are moving.
We discover several intriguing phenomenon emerging in neural nets: 1) transformers
can naturally outperform CNN models which might have even used occlusion as a
form of data augmentation during training 2) incorporating symbolic-components
like capsules to such backbones allows them to bind to occluders never even seen
during training and 3) Islands of agreement (similar to the ones hypothesized in
Hinton et Al’s GLOM) can emerge in realistic images/videos without instance-level
supervision, distillation or contrastive-based objectives(eg. video-textual training).
Such emergent properties allow us to derive simple yet effective training recipes
which lead to robust occlusion models inductively satisfying the first two stages of
the binding mechanism (grouping/segregation). Models leveraging these recipes
outperform existing video action-detectors under occlusion by 32.3% on O-UCF,
32.7% on O-JHMDB & 2.6% on Real-OUCF in terms of the vMAP metric. The code for this work has been released at https: //github.com/rajatmodi62/OccludedActionBenchmark.

On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
Rajat Modi'; Vibhav Vineet', Yogesh Singh Rawat CRCV, University of Central Florida and Microsoft Research²
Abstract
This paper explores the impact of occlusions in video action detection. We fa- cilitate this study by introducing five new benchmark datasets namely O-UCF and O-JHMDB consisting of synthetically controlled static/dynamic occlusions, OVIS-UCF and DVIS-JHMDB consisting of occlusions with realistic motions and Real-OUCF for occlusions in cralisticwoorld scenarios. We formally confirm an intuitive expectation: existing models suffer a lot occlusion severity is increased and exhibit different behaviours when occluders are static vs when they are moving. We discover several intriguing phenomenon emerging in neural nets: 1) transform- ers can naturally outperform CNN models which might have even used occlusion as form of data augmentation during training 2) incorporating like capsules to such backbones allows them to bind to occluders never even seen during training and 3) Islands of agreement can emerge in realistic images/videos without instance-level supervision, distillation or contrastive-based objective video-textual training). Such emergent properties allow us derive simple yet effective training recipes which lead to robust occlusion models inductively satis- fying the first two stages of the binding mechanism els leveraging these recipes outperform existing video action-detectors under oc- clusion by 32.3% on O-UCF, 32.7% on O-JHMDB & 2.6% on Real-OUCF terms of the vMAP metric. The code for this work has been released at https: //github. com ra http://aaaaaaaooooooocococcacca.
seuupinn/eegrggttion) Mod-
Introduction
Deep learning(4) [1] has led to significant advances in object detectionn/semmenttiion for both image [16] 19 and video domain (4). Such deep neural networks are turn widely used self-driving cars and safety critical scenarios. A key concern for such applications is whether they are able to perform well when encountering realistic occlusions: e.g. are they able reliably localize pedestrian even when an occluder (say dog) comes in front of him. However, one major limitations being existing dataset test split doesn't 't contain such occlusions. This raises concern, whether these models will be robust to real-world occlusions or not.
One would suspect that the inherent inductive biases of these architectures would be enough to induce natural occlusion robustness. To verify the hypothesis, we run two preliminary setups: (Fig. TT (A) Firstly, superimposing single occluder (eg bus) over the actor, and, (B) Then, we analyze the
performance if the occlusion is shifted to background (i.e. no occlusion over actor at all). We observe
a relative drop of 20-50% across multiple existing ofate-of-the-ant approaches 46 13
This verifies
"Corresponding Author, email: rajatmodi @ ucf.edu *Grouping pixels is a perceptual phenomenon of the visual-cortex 59]
Assigning group to class is group is identified by multiple-names then neural-net can learn this one- many
property of language. mapping via contrastive objectives (75) But grouping should also emerge without language since organisms continue to perceive objects even without cars/tongue.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Effect Severiy
Didrrador vs Context
( Fat 15-0%)
(0) FG2 (3-40%)
(RI PG3 660-009
(n No
MOC
VIAPO
Figure I: A Toy Experiment. (i-iii) superimposing a single occluder (bus) on an actor and varying its size results in drops large 50%. (iv) even simple occluder (cat) in background results in 30% drop. (v) highest drops are observed if background is entirely masked. (vi-ix) Clean refers to all methods evaluated on unoccluded test sets. Best viewed when zoomed in.
our hypothesis that existing approaches are not robust to occlusion. It opens up the area that there is a need study these models behaviour under different types and severity of occlusion.
We conduct first benchmark study of occlusions spatio-temporal video action detection. We investigate following aspects: a) Effect of different severity level of occlusions, b) Robustness of different network backbones, c) Effect of pre-training and trainable parameters. d) Effect of static dynamic occlusions on models. Our study yields the following key insights: 1) Performance decreases when severity increases on actor region, but increasing severity on background does not have much impact 2) Transformers which have never CNN-based models which uses occlusions as a data augmentation. 3) a significant component of transformer's 's robustness (and other architectures as well) comes from pre-training large-scale datasets 4) Existing models perform better on static occlusions than realistic dynamic occlusions.
occlusions during training can outperform
Based on our study, we investigate techniques to make the existing video action detection models robust against occlusion. First, we experiment with augmentations and found them to be effective for all the models in improving their robustness. We further utilize some of these insights and develop transformer-based token masking strategy that provides a robust video action detection model which outperforms all existing methods. We make the following contributions in this work:
We analyze existing approach behaviours occlusions and conduct first benchmark study on occlusions spatio-temporal action detection.
Towards this, we propose five novel datasets, i.e. O-UCF, O-JHMDB for static occlusions, OVIS-UCF, OVIS-JHMDB for realistic occluder motions and Real-OUCF for real world occlusions to benchmark our study.
To improve robustness of existing approaches, we show how occlusion as augmentation improves robustness. Furthermore, we propose token-masking to further improve robustness transformer backbones.
Related Work
Action Detection (STVAD): Involves localizing an actor in each video frame and also predicting action class Existing methods[66 58 [1] adopt tubelet- based approach: action trajectories can thought of as a sequence of actor bounding boxes spread over time. These tubes can be predicted by progressively refining predictions over center-key Kerte--eeffrme444 [7] or directly learning a tubelet as a feature volume [86). However, these methods require additional post-processing by linking these smaller tubelets across time. Other architectures like VideoCapsuleNet [3] frame action-detection as a per-pixel localization task.
Occlusions in Videos: Recently, OVIS dataset 54 was proposed for Video Instance Segmenta-
introduce occlusion robustness by temporally averaging actor-centric embeddings in DETR-like architectures However, instance- recognition during VIS can be performed by object-detection in single frame whereas action- classification in STVAD requires more than frame. Approaches like [4] 70 explore occlusions
tion (VIS) under realistic occlusions. Methods like 76.
32 3

---

Different Occlusion Severity Levels.
Ocoluder Distribution
B01
Indoor
Outdoor
Different Occluder Motion
BG2
B63
F01
FG2
FG3
Circle
Linear
Zoom-In
Realistic Occlusion Samples.
Figure 2: Sample video framee from proposed benchmark datasets. (i) occlusion severity increases across severity levels and both actor/background region. (ii) occluders are sampled from both indoor/outdoor splits. (iii) Our O-UCF and O-JHMDB simulate 6 dynamic occluder motions like circle, linear, zoom-in etc. (iv) Similary, the proposed Real-OUCF is a dataset for realistic scenarios where multiple actors mutually-occlud each other. Best viewed in color.
for action-recoggition. In comparison, we additionally focus on the impact of occlusions on localiza- tion ability of SOTA action-detcctors.
Occlusion in Images: CompositionalNens 40 explore -oouuttens for image classification through EM-based mixture models. Deepioting 85 caters object-detection under occlusions by proposing a voting layer to gather contributions from visible object parts. While such studies are restricted images, we study occlusions under videos.
Regularization Tenchniques: Dropout 62 have been shown to act as an implicit regularizer (69 and reduce network overfitting. Moreover, dropout has been found effective for increasing occlusion robustness [21). We further show that masking input tokens in transformers [1] is also an effective training strategy for this task.
3 The Video Occlusion Datasets
The aim of this study is study occlusions in spatio-temporal video action detection. We present five different benchmark datasets to study this problem. Two of them (O-UCF and O-JHMDB) synthetically generated to systematically study this problem by manipulating single occlusion parameter at a time. Similarly, OVIS-UCE, OVIS-JHMDB consist of occluders exhibiting realistic motions. Furthermore, we also present real-world dataset (Real-OUCF) to validate findings. We use three different parameters to create our synthetic datasets, 1) Indoor/Outdoor Occluders 2) Severity of Occlusion Static/Dynamic motions of occluders.
3.1 Occluder Selection
To keep the generated occlusions realistic, we first crop out several objects from the Pascal- -oCCl images using ground- -oound-truth pixel-level annotations. Out of 2413 cropped instances, we shortlist 900 possible occluders which correspond to commonly-occurien distributed among two separate categories namely indoor objects eg, chair, table) and outdoor objects (eg, aeroplane, ship etc.). Finally, we seftly blend these occluders over the video pixels using RGBD maps of these objects. This blending correctly simulates very real object (occluder) being present over in the frame, without having resort to intensive data collection procedure.
objects. Next, these objects are

---

Table Benchmark dataset statistics
Statistics
O-UCF
OVIS-UCF
O-JHMBB
OVIS-JHMDB
Real-OUCF
Classes
24
24
21
21
24
Severity Levels
9
Occ. Motions
6
6
6
Oce. Type
Synthetic 20306 2284
Semi-Rest 20306 2284
Synthetic 5896 928
Semi-Real 5896
Real 1743
Total Videos
Instances
928
6920
3.2 Severity of Occlusion
In a video, an actor occupies certain portion of the frame as it moves over time. Formally, we define such an actor-region (FG) the tightest bounding box enclosing all the ground truth boxes of the actor's trajectory over time. The remaining region is termed as background(BG),
Occlusion severity is defined as the fraction of the total area which is occupied by an occluder during occlusion. We experiment with three severity levels, namely (0-20%), 2 (20-40%). and 3 (40-60%) respectively. Occlusion levels in the actor region are prefixed by FG, i.e. FG1/2/3, whereas occlusions in the background are prefixed by BG, i.e. BG 1/2/3. A combination of these actor/background severity levels yields 9 levels of severity.
3.3 Types of Occlusion
Static Occlusions: Refer to the occlusions where the occluders occupy fixed position in the frame and don't move over time. Our dataset consists of 9 severity combinations of actor/background regions as explained in the previous section.
Existing occlusion-bases datasets are mostly image-based and are therefore unable to study the effects of the movement of occluders in the video. For such dynamic cases, pick a fixed severity level (i.e. FG2, BG3) and vary the motion of the occluders from a particular starting point. Our test set consists of two types of motions, i.c. circular and sinusoidal. The training set consists of linear, zoom-in, zoom-out, or random motions. Note that the motion in the train/test set is mutually-exclusive, i.e. type of motion in one dataset split is not present in another for fairness. Similarly, an occluder moving in the actor region never mixes over to the background region but instead gets wrapped around over the course of its trajectory.
Dynamic Occlusions:
3.4 Benchmark datasets
The statistics for five proposed benchmark datasets are illustrated in Table O-UCF contains of 24 action classes, along with 20306 testing samples. Similarly, O-JHMDB contains 21 action clases with 928 samples. Both of these datasets consist of static occlusions in 9 levels of severity, controlled occluder trajectories in train set, and 2 trajectories in the test set. Our OVIS-UCF/OVISS JHMDB datasets consist of occluders with realistic motions from OVIS superimposed on top of original UCF/JHMDB datasets. Note that we term this dataset as semi-realistic because although the occluders themselves are synthetically placed on the frames, occluder trajectories are naturally occuring as observed in OVIS dataset. Real-OUCF consists of 1743 fully-realistic occlusion videos which were hand-picked from Youtube for 24 action classes. These videos were then cropped temporally using LossLessCurt 2. to precisely localize start/end time step of each action. Such shorter duration clips then need to spatio-temporally annotated. Therefore, we feed-forward all such clips through auto-label generator of GroundedS appropriately constructed textual query. Since SAM(38] a foundational model, it segments all the persons in an image. However, we are only concerned with the people who are actually performing actions. In order to remove such excessive false positives predicted by GroundedSAM we manually suppress/refine per-frame instance-lever masks using the CVAT Annotation Tool Finally, we end up with 64.1% 1.1%% annotated instances being occluded.
in order to localize "person" class with an
Experiments and Analysis
Studied models: We experiment on three most recent SOTA methods in video action detection. Namely, we pick MOC[46), YOWO[39 and VideoCapsuleNee[[] whose official Github implementa- tions and evaluation protocols have been open-sourced for both UCF and JHMDB datasets. All

---

Table O-UCF Benchmark: Performance across three occlusion severity levels. For particular actor-region occlusion (FG), averaged results across BG1/2/3 levels are reported.
FGI
FG2
FG3
Clean
Oce
6
6,
Occ
6a
6,
Occ
6
Ö,
MOC|46 [46]
54.4
53.6
0.99
0.99
35.2
0.81
0.65
29.5
0.75
0.54
YOWO 39
48.8
38.5
0.90
0.79
32.5
0.84
0.67
26.8
0.78
0.55
VCAPS 13
75.5
51.5
0.76
0.68
42.8
0.67
0.57
36.9
0.61
0.49
Table 3: O-JHMDB Benchmark: Performance across three occlusion severity levels. For a particular actor-region occlusion (FG), averaged results across BG1/2/3 levels are reported.
FGI
FG2
FG3
Clean
Occ
8.
6,
Occ
6a
6,
Occ
6g
8,
MOC
77.2
45.2
0.68
0.59
33.9
0.57
0.44
25.9
0.49
0.34
YOWO 39
85.7
50.5
0.65
0.59
47.7
0.62
0.56
46.2
0.61
0.54
VCAPS 13
65.7
49.2
0.84
0.75
35.4
0.70
0.54
21.7
0.56
0.33
these models are tested on the original clean UCF24 & JHMDB-21 datasets, which do not contain any occlusions. Also, evaluate these models on occ versions, i.e. our O-UCF/O-JHMIB datasets. All our transformer-bant facilitate appropriate convergence. The best performing models for downstream testing selected from a separate hold-out validation set.
models have been trained for 1.5x more epochs than CNN based models to
Metries A well accepted metric for action-detection is v-mAP at 0.5 ioU threshold, it also measures the temporal consistency of the prediction across time. Assume that under occlusions vmAP drops from V to V Therefore, we report absolute robustness 6, = (1 V-V") and relative robustness 5, (1 V=)"). Note that both 0 8ay o, < 1, with greater value denoting more robustness.
4.1 Results and Analysis
We present the result of our benchmark on the baselines and analyse several interesting trends.
Increasing occlusion severity over actor region reduces performance. In Tab we occlude actor region with multiple occluders. It can be clearly observed that as the occlusion severity is increased, the performance of all the action-detectors is reduced. The notable thing is that VCAPS is most robust severe occlusion (i.e. FG3) (in terms of absolute scores) the larger O-UCF dataset (i.e. 36.9%) and YOWO is most robust to occlusions on smaller JHMDB dataset (i.c. 46.2%). The larger performance difference in MOC and YOWO could be attributed to the treatment of action- detection as a box-regression problem in /YOWOO 339 vs dense-semantic mask prediction in the VideoCappuleNee([3] Regressing the 4 coordinates of a box has minimum probability of error as = 0.25, whereas dense-prediction reduces this error to I/n, where n is the no of pixels being predieted and n 4. As long least four pixels are predicted along the boundaries of the box, the method performs relatively wel Also capsules have been shown to be extremely robust to occlusions on Multi MNIST and we believe similar inductive bias extends to videos.
Are backbones with more parameters necessarily more robust? To answer this question, we tried CNN/Transformet--eee
backbones two of our best performing methods, i.c. YOWO & VCAPS. In Table 5 it is evident that Mvitv2-S with 50% less parameters (35M) can outperform other 3D backbones like ResNext using large as 89M parameters (i.e. 67.3% on O-UCF & 86.8% on O-JHMDB). This shows that the internal attention mechanism in transformers doesn' 't necessarily need more parameters improve robustness. Note that VCAPS with Mvitv2-S backbone (67.3, O-UCF) largely outperforms YOWO which also uses Mvitv2-S backbone (31.2, O-UCF). A crucial architectural difference is that the VCAPS-Mvitv2 uses additional layers of capsules, whose qualitative effects shall be revealed in a later section.
'That is why predicting dense mask and then fitting rectangle around it during evaluation is better than directly regressing box. (see supplementary.)
5

---

Table Effect of realistic occluder-motion on OVIS-UCEFOVISSJEMMD datasets. Models in
general perform better on static occlusions than dynamic occlusions.
OVIS-UCK
OVIS-JHMDB
Backbone
Static
Dynamic
Static
Dynamic
MOC 46
[99]
14.8
12.4
26.8
31.9
YOWO 39
ResNext 77]
16.5
10.1
37.6
35.3
VCAPS 13
i3D
32.3
21.7
20.9
17.3
39
Resnet 8 118
17.3
11.8
28.0
24.2
VCAPS
Resnet 8818
33.2
21.9
7.7
7.3
O-UCF
O-JHMDB
70
VCAPS- Mvitv2
6
85
VCAPS- Mvitv2 w
75
65
YOWO-W W
40
MOC-w
VCAPS-W
55
YOWO-w/o
YOWG-W w
VCAPS-W/G
VCAPS-W
20
YOWO-w/o
35
VCAPS-Mvitt2 VCAPS-W/o
MOC-W
VCAPS Mvltv2 w/o
MCC-w/o
MOC-w/o
0
15
Accurady
Accuracy
Figure 3: Effect of pretrained weights: (CAPS-Mvitye outperforms all models both O-UCF and O-JHMDB. (X Axis:) Accuracy without occlusions. (Y Axis): Relative Robustness of Model. TighRight corner of each plot corresponds to most robust model.
Pre-training improves transformer's robustness to occlusions. To investigate why transformers perform so well out of the box, we train them from scratch without pre-trained weights and test them on D-UCF/O-JHMDB datasets. From Fig 3 (i), it be observed that transformers (i.e. VCAPS-Mvitv2) perform poorly without weights compared to other models. However, on training with Kinetics 400 weights, the transformers achieve state-of-the-are results. Therefore transformers are not naturally more robust, but pre-training helps to improves their intrinsic-robustness. This also reveals the glaring dependency of transformers on large amounts of data and compute for pre-training.
Increasing parameter size in same model family improves occlusion robustness By universal- approximation theorem [31], larger neural net can learn better representations. We investigate whether similarly increasing the model parameter size (and hence internat-reprreeenttiinaa capacity) could lead to better occlusion robustness. In Fig 4 right), it can be that changing the backhones from Resnet 18-50-100 101 follows this trend. This shows that increasing parameter size in the same model family generally leads to improved occlusion robustness.
Effeet of occluder motion on backbones. We experiment with realistic occluder motions by super- imposing occluders from OVIS datasets upon UCF/JHMDB videos, resulting in benchmark datasets called OVIS-UCF and OVIS-JHMDB respectively. In Tab14] we observe that in general models are performing better during static occlusions than dynamic occlusions. The differences still hold when the backbones are made consistent [9). This shows that resolving actor location becomes difficult when both occluders and actors move together. Therefore, reasoning about temporal motion properly remains a worthwhile architectural pursuit. We had also experimented with controlled occluder motions like circle, sinusoids. For those results, refer the reader to the supplementary.
From this section, we conclude that using transformers as a backbone and training using pre-trained weights can greatly improve robustness.
5 Improving Robustness under Occlusion
Next, we investigate if we can make these models robust against occlusion. Augmentation is a well studied technique to induce robustness models against distribution shifts. We experimented with synthetic occlusions as augmentations to make the existing models robust. We generate augmented samples by superimposing occluders at random locations, various severity levels and motion. Training existing baselines on these augmented samples allows to improve occlusion robustness (Table

---

O-UCF
VCAPS-R100
O-JHMDB
YOW
6
VCAPS-R50
YOWO-R18
YOWO-R50
VCAPS-R18
51
TOWO-RIO1
35
VCAPS-R50
VCAPS-R101
YOWO-R18
YOWO-R50
25
16
VCAPS-R18
40
45
50
23
60 Accuracy
40
70 Accuracy
Figure 4: Effect of number of model parameters: (i) Increasing parameter size within the same model family yields more robustness. (X Axis:) Accuracy without occlusions. (Y Axis): Relative Robustness of Model. Top-Right corner of each plot corresponds to most robust model.
Table 5: Effect of Backbones: Transformers outperform other backbones in less than half the parameters. Clean evaluation is done on UCF-24 and JHMDB-21 test sets.
O-UCF
Real-OUCF Occ
O-JHMDB
Methods
Backbone
#Params (M)
Clean
Occ
Clean
Occ
Resnet [8]
11.7
44.7
26.9
4.4
66.1
51.8
YOWO 39
YOLO ResNext
89
48.8
31.2
7.6
85.7
63.7
MVITv2-S
35
69.4
48.2
9.1
81.3
68.2
Resnetl 8 [18]
11.7
57.7
36.1
6.4
35.2
18.3
VCAPS
ResNext [74]
89
61.2
34.7
10.8
49.3
27.7
MVITv2-S[45
35
83.1
67.3
13.1
93.1
86.8
Next, make some very surprising observations. In Table6 Swapping the backbone of one of the methods (VCAPS) with a transformer (i.e. Mvitv2) yields the highest performance across both UCF/JHMDB. A non-trivial observation is that transformer- based network trained without augmentation outperforms all the other methods which are trained using occlusion as an augmentation (67.3 vs 49.9 on UCF). Training the transformers further with occlusion- augmentation improves the performance from 67.3% to 81.6%. This is notable because this performance on a occluded test (i.e. 81.6%) comes close how the detector functions in the absence occlusion,i.e, (83.1% on UCF). The remaining gap of 1.5% can be bridged by explicit occlusion modelling. Therefore, transformers trained with occlusions as augmentation strategy can significantly improve robustness.
5.1 Importance of Capsules
Here, explore the importance of capsules towards occlusion modelling. In Tat we remove all the capsule layers from VCAPS-Mvvtv? Note that the architecture then reduces to a standard 3D [00, and the performance drops from 67.3% to 64.1%. Therefore, capsules are crucial component for the performance of VCAPS-Mvitv2.1
Emergent Object/Occluder separation in capsules: Next, we qualitatively explore the behaviour of capsule-based models towards occlusions. We choose our best-performing VCAPS-MvitV model which has never seen occlusions during training and feed-forward an occluded sample during inference in Fig5] The activation maps of some of the capsules the primary layers have been visualized. Each capsule has an activation map of size Hx W, where each value denotes the confidence about the location of a particular entity (object). Applying a threshold of 0.7 results a binary mask. It can be seen that some of the capsules selectively look at the objects (i.e. actors), and other capsules focus on occluders (eg, chair, sofa). Note that the capsules have never seen occluders during training, but are still able to focus them. Therefore, architectural constraints like binding [17] object-specific pixels to particular slots (eg, capsules) (36) allows behaviour like unsupervised object discovery to emerge 44).
'Quantitative improvements are not observed the number of capsule layers are increased further than two. One reason that capsules were originally invented for rigid bodies (23] 30 and iideo-capsules would need to enforce dynamic deformable reference frames. 63 (3 learn canonicl frames in case of multiple
instances. 778
7

---

Table 6: Effect of Augmentation: Transformer based backbones outperform other CNN based models which have even used occlusion as an augmentation. Clean evaluation is done on UCF-24 and JHMDB-21 test sets. w/o: without augmentation, w: with augmentation.
O-UCF
Real-OUCC
O-JHMDB
Methods
Backbone
Type
clean w/o
w/o
clean w/o w
MOC[46
DLA34 79
CNN
54.4 39.8 44.1
3.1
5.8
77.2 35.3 55.9
YOWO 39
YOLO[56]+ + ResNex(
CNN
48.8 31.2
44.9
7.6
9.0
85.7 47.8 68.0
VCAPS
i3D
CNN
75.5
42.2
49.9
9.9
10.8
65.7
35.6 59.7
VCAPS 13
2 45
Transformer 83.1 67.3 81.6
12.1
93.1 86.8 90.5
Object Cansules
Occluder Carsules
pepresenatioonl
Collipo
Figure 5: Emergent object/occluder separation in capsules. (top left) An occluded video is forwadeeddd through VCAPS-Mvitv2 and activations of primary layer capsules are visualized. Capsules can (i) parse an actor into constituent body parts without instance-level supervision [22 28 (ii) segment multiple actors & objects of action (volleyball net). (iii) show evidence of focusing on occluders never seen during training (iv) undergo representational collapse, where a single capsule starts representing multiple objects (wholes) when number of objects in the scene are greater than number of capsules in the network. Best viewed in color.
Representational collapse: Each capsule is meant to represent one entity (object) only. If there are more objects (i.e. occluders) in video than the capsules in a network, single capsule gets forced to encode multiple disjoint objects together. This problem can't 't be solved without increasing the number of capsules further and retraining the network again This similar issue arises in the standard object detection literature where the maximum number of objects that a network can detect is equal to the number of object proposals was originally trained with [5 57
5.2 GLOM: Islands of agreement can emerge from token representations, [26
We qualitatively explore the effectiveness of our VCAPS-Mvity2 model presented in Tables 9b Specifically, we assume that each of the output token representation from Mvitv2 equivalent to a basund activity vector [26]. We perform t-sne reduction (64] of each vector to three dimensions, and linearly project obtained components to the RGB range of [0, 256) in Fig Note that obtained maps are not simple patch-based correlations among ooken-representation//// (52), but rather lower dimensional clustering of vectors themselves plotted as identical colors [8]. Furthermore, our 2CAPS-Mvitv2 has only received actor-level annotations/ action-label supervision and not any form of occluder masks [83] contrastive-
supervision (68] or distilled knowledge [6 during training. The only used prior is Kinetics pretrained weights learnt from action-recognition.
This suggests that the lower layers in our VCAPS-Mvity2 (i.e. transformer layers) perform semantic grouping of pixels into objects 75 (object discovery). Next, higher layers (capsule layers) selectively segregate objects/occluders into instance-specific 5 Thus transformers and capsules complement each other to achieve first two stages of the binding process Finally, our decoder reasons to suppress occluder-specific features and focus/infill on actor-centric regions during localization, i.c. Fig 7 From this section, we conclude that two capsule layers stacked on top of transformer-baa based backbones can help improve cottuusiiooooouunnesss.... The neuro- neuro-symbolic advantage of capsules
'This differs from how the brain iteratively parses variable number of objects in a constant number of neurons, although it may take more time(50) In contrast, parallel models of attention [65] and capsules 60) have assumed singular fixation. This requirement forces object queries to compete through self attention [S] for explaining perceptual parts of input 49. However, competing queries raise memory requirement linearly for each object that needs to [5) Therefore, while neural nets can encode a finite number of learnable neurons, parallel decoding suffers a fundamental memory bottleneck (5) 67 (that can't onn't be solved by retraining at scale). Surprisingly. neural fields aren't it constrained by this limitation because they enforce single world-specing coordinate frame instead of separate speciic canonical frames. (80 78
8

---

Figure 6: Emergent islands of agreement on O-JHMDB dataset 15:: Output token representations of our (CAPS-Mvitv2[]) can be thought of as column-based vectors 26 4 All the vectors belonging to the same object should agree among themselves. Reducing dimensionality of these vectors to three using t-sne shows evidence of such islands of agreement as belonging identically-colored regions. Note that obtained islands show both occluders as well as actors. Our video-based model is only trained with actor-level annotation/action class labels and occluder-specific masks.
Train
Test
Transformer Encoder
Capsules
Decoder
4
Figure 7: Proposed Token Masking (from left to right) An input video is partitioned into tokens and fed to transformer. (ii) During training, random number of tokens blacked out. (iii) During inference, all the tokens are switched on and the actor localized under occlusions.
Table 7: Token Masking: Improves performance of models trained even using occlusion as
augmentation.(D) dropout on intermediate network layers. (T)- proposed token masking
O-UCF
O-JHMDB
Backbone
Mode
clean
aug
aug+mask
clean
aug
aug+mask
466
DLA34 [79]
D
54.4
44.1
47.4
77.2
55.9
61.3
YOWO 39
ResNext
D
48.8
44.7
46.8
85.7
68.0
72.8
[3
i3D 7
D
75.5
49.9
51.5
65.7
59.7
62.5
VCAPS
Mvitv2 45
D
83.1
81.6
81.9
93.1
90.5
91.7
VCAPS
Mvitv2 45
82.2
92.4
combined with innate natural robustness of transformers forms the basis of our best model, i.e. VCAPS-Mvitv2.
5.3
Token masking
Till now, we have seen that architectures possessing a transformer based backbone with a few capsule layers top are highly robust to occlusions. Now, we propose loken-masking to improve their innate robustness. Given a video dimensions T x И W, a transformer creates L spatio-temporal patches each of which is projected to common dimensionality D during input. Our idea is that randomly masking (blacking out) some of the tokens during training introduces additional occlusion robustness in the model. Mathematically, generate iid bernoulli random variables
M₁ Bernoullipp) Vi [1,
where p is the masking probability. Each M; is repeated D times, so that a particular token P fully masked before input. Next, Me is multiplied element wise with the input sequence to give final
masked input I' @ M where I.M are input sequence vector and masking vector of dimensions !.x respectively, and @ is element-wise hadamard operation. During inference, none the tokens masked, and the input occluded video sequence is directly feed-forwarded through the network.
Quantitative Results In Table it can be seen that such token-masking improves even the per- formance of models which have used occlusion data augmentation from 81.9% to 82.2% on

---

Table 8: Comparison with existing methods: Comparison of our method across existing supervised approaches, *; denotes results using CSNI52 backbone.
UCF-24
JHMDB-21
Backbone
f-mAP
V-mAP
f-MAP
V-mAP
Methods
2D
3D
0.5
0.2
0.5
0.5
0.2
0.5
Yang al. [37
75.0
76.6
Li et al. [66
78.0
82.8
53.8
70.8
77.3
70.2
Kopuklu et al. (39
80.4
75.8
48.8
75.7
88.3
85.9
Zhao et al. 86
81.3
85.3
60.2
82.3*
81.8
80.7
Duarte et al. 13
78.6
97.1
80.3
64.6
95.1
Kumar et al.
69.2
95.3
71.9
68.1
96.8
68.4
Tao al. 73
83.7
86.7
Ours
81.2
98.6
83.1
93.0
98.1
92.8
Table Ablations and evaluation of proposed ooken-masking on Realistic Occlusions.
(a) Capsules stacked on transformer backbone improve occlusion robust- ness. Testing is done on O-UCF.
(b) Realistic Occlusions: Models trained with token- masking oulperform other models Real-OUCF. f- Training with occlusion augmentation dropout. f-
Training with occlusion augmentation & token- masking.
w/o caps
w/ caps
VCAPS (i3D)
VCAPS! (Mvitv2)
VCAPS (MvitV2)
MOC!
YOWO!
64.1
67.3
Occ ||
7.2
9.6
11.7
14.3
O-UCF and 91.7% to 92.4% on O-JHMDB. Note that for VCAPS(Mvitv2), masking some of the tokens during training performs better than dropping some of the intermediate network neurons. This observation in line with how learning objectives similar to pixel-level reconstruction in masked image modelling (MIM) 20 yields better sele-suue representations.
Comparison With existing Methods: We compare with existing methods on f-mAP and v-mAP. In Tat8 our method obtains 83.1 1 vmAP on UCF-24 thereby indicating most localization robustness. On JHMDB-21, we obtain 92.8% in terms the absolute v-mAP score, which is significantly better than other existing methods. We acknowledge that TubeR [86] and ST-Mixer 73 are slightly better than our method in terms of f-mAP scores on UCF-24 dataset.
Realistic Occlusions: Furthermore, we evaluate our baseline models and the token- masked models on the Real-OUCF dataset. table we observe that the models trained with occlusion as augmentation perform better than models without augmentation. As evident in Table models trained with our token- masking approach also outperform other models in realistic scenarios.
Conclusion
We have conducted the first-ever benchmark study evaluate the impact of occlusions in spatio- temporal action detection. This study provides several interesting key insights including, i) Models perform better on static occlusions vs dynamic occlusions in realistic scenarios. ii) Transformer- based models possess greater natural robustness compared to other models using occlusion as data augmentation. iii) Pre-training improves robustness of transformers more than CNN models. (iv) Robustness can be improved further by leveraging components like capsules and training jointly with occlusions as data augmentation. (v) Recipes like simply masking some input tokens of transformers during training can introduce additional robustness and obtain ofte--of-theaar performance under realistic occlusions. Our benchmark, datasets and code have been released at this link
Limitations: We have only focused on annotating visible regions of the occluded-objects in line with official COCO protocol 47]. Another direction could also focus on predicting missing regions, i.e. amodal-segee segmentation. Classically, cormantic-segmentaaan only one object [9]. However, this is not true for occlusions. Inductively, this symmetry issue has been resolved in works like Maskformer2[9] where multiple objects per pixel can be predicted by removing the softmax assumption. We note that there still significant gap left bridge existing models for improving robustness to realistic occlusion 9b
has assumed that one pixel can belong to
10

---

7 Acknowledgement
This research is based upon work supported in part by the Office of the Director National Intelligence (Intelligence Advanced Research Projects Activity) via 2022-21102 and in part by University of Central Florida seed funding. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies, either expressed or implied, ODNI, IARPA, or the US Government. The US Government is authorized to reproduce and distribute reprints for govemmennal purposes notwithstanding any copyright annotation therein.
References
[1] Computer vision annotation tool. htttps://ithhb. cot.tth...cooo..ooooooooooo.o. wwwp://ww/wppcccaaaaeeeee 2013.
[2] Losslesscut. GitHub repository, 2013.
[3] Adrià Caelles, Tim Meinhardt, Guillem Brasó, and Laura Leal-Taixé. Devis: Making deformable transformers work for video instance segmentation. arXiv preprint arXiv:2207.11103, 2022.
[4] Yuxuan Cai, Yizhuang Zhou, Qi Han, Jianjian Sun, Xiangwen Kong, Jun Li, and Xiangyu Zhang, Reversible column networks. arXiv preprint drXiv:2212.11690 2022.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 213-229, Springer, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe-vupe vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-96600 2021.
[7] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 6299-6308, 2017.
[8] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. arXiv preprint arXiv:2211.0972 2022.
[9] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1290-1299, 2022.
[10] Özgün Çiçek, Ahmed Abdulkadir, Soeren Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Assiteer-Assisted Intervention-MICCAA 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part 11 19, pages 424 432. Springer, 2016.
[11] Ishan Dave, Zacchaeus Scheffer, Akash Kumar, Sarah Shiraz, Yogesh Singh Rawat, and Mubarak Shah. Gabriellav2: Towards better generalization in surveillance videos for action detection. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 122-132, 2022.
[12] Mingyu Ding, Yikang Shen, Lijie Fan, Zhenfang Chen, Zitian Chen, Ping Luo, Joshua B Tenenbaum, and Chuang Gan. Visual dependency transformers: Dependency tree emerges from reversed attention. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14528-14539, 2023.
[13] Kevin Duarte, Yogesh Rawat, and Mubarak Shah. Videocapsulenet: A simplified network for action detection. Advances in neural information processing systems, 31, 2018.
[14] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88 (2):303-338, 2010.

---

[15] Nicola Garau, Niccoló Bisagno, Zeno Sambugaro, and Nicola Conci. Interpretable part-whole hierarchies and conceptual-sene semantic relationships in neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13689-13699 2022.
-ementu
[16] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[17] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[20] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[21] Mingjie He, Jie Zhang, Shiguang Shan, Xiao Liu, Zhongqin Wu, and Xilin Chen. Locality-aware channel-wise dropout for occluded face recognition. IEEE Transactions on Image Processing, 31:788-798, 2021.
[22] Hinton. Using relaxation to find a puppet. In Proceedings of the 2nd Summer Conference on Artificial Intelligence and Simulation of Behaviour, pages 148-157, 1976.
[23] Geoffrey Hinton. Some demonstrations of the effects of structural descriptions in mental imagery. Cognitive Science, 3(3):231-25D) 1979.
[24] Geoffrey Hinton. How to represent part-whole hierarchies in neural network. arXiv preprint arXiv:2102.12622 12627, 2021.
[25] Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345, 2022.
[26] Geoffrey Hinton. How to represent part-whole hierarchies in neural network. Neural Computation, pages 1-40, 2022.
[27] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 150:::50000250 02031, 2015.
[28] Geoffrey E Hinton. Relaxation and its role in vision. 1977.
[29] Geoffrey E Hinton. Connectionist learning procedures. Machine learning, pages 555-610. Elsevier, 1990.
[30] Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In International conference on learning representations, 2018.
[31] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural networks, 255):359-366, 1989.
[32] De-An Huang, Zhiding Yu, and Anima Anandkumar. Minvis: A minimal video instance segmentation framework without video-based training. arXiv preprint arXiv.2208.02245, 2022.
[33] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black. Towards understanding action recognition. In International Conf. on Computer Vision (ICCV), pages 3192-3199, December 2013.
[34] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. Advances in neural information processing systems, 29, 2016.
12

---

[35] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland. Clemens Meyer, Simon A A Kohl, Andrew J Ballard, Andrew Cowie. Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure predic- doi: 121008//41586-021--0000
tion with AlphaFold. Nature, 596(7873):583-55) 2021.
[36] Daniel Kahneman, Anne Treisman, and Brian Gibbs. The reviewing of object files: Object- specific integration of information. Cognitive psychology, 24(2):175-219 1992.
[37] Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari, and Cordelia Schmid. Action tubelet detector for spatio-temporal action localization. In ICCV, 2017.
[38] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02664, 2023.
[39] Okan Köpüklü, Xiangyu Wei, and Gerhard Rigoll. You only watch once: A unified enn architecture for real-time spatiotemporal action localization. arXiv preprint arXiv:/91 1.06644, 2019.
[40] Adam Kortylewski, Ju He, Qing Liu, and Alan L Yuille. Compositional convolutional neural networks: A deep architecture with innate robustness to partial occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8940-89493 2020.
[41] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 64(6),84-90, 2017.
[42] Akash Kumar and Yogesh Singh Rawat. End-to-end semi-supervised learning for video action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14700-14710, 2022.
[43] Mariana Lenharo. Decades-long long bet on consciousness ends-and and it's philosopher 1, neuroscientist Nature, 2023.
[44] Tianhong Li, Lijie Fan, Mingmin Zhao, Yingcheng Liu, and Dina Katabi. Making the invisible visible: Action recognition through walls and occlusions. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 872-881, 2019.
[45] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4804-4814, 14, 2022.
[46] Yixuan Li, Zixu Wang, Limin Wang, and Gangshan Wu. Actions as moving points. In European Conference on Computer Vision, pages 68 84. Springer, 2020.
[47] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 755-755. Springer, 2014.
[48] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303,, 004999 2023.
[49] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 13:11555511538 2020.
13

---

[50] Tim Meinhardt, Alexander Kirillov, Laura Taixe and Christoph Feichtenhofer. Track- former: Multi-object tracking with transformers. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8844 8854, 2022.
[51] Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz. Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. Advances Neural Information Processing Systems, 14.23296-23308 2021.
[52] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[53] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[54] Jiyang Qi. Yan Gao, Yao Hu, Xinggang Wang, Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: benchmark. International Journal of Computer Vision, 130(8), 2022.
[55] Arthur Reber, William B Miller, and Františck Baluška. Consciousness: unicellular organisms know the secret. Nature, 620(7972)377-37, 2023.
[56] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 779-788, 2016.
[57] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[58] Mamshad Nayeem Rizve, Ugur Demir, Praveen Tirupattur, Aayush Jung Rana, Kevin Duarte, Ishan R Dave, Yogesh Rawat, and Mubarak Shah. Gabriella: An online system for real-time activity detection in untrimmed security videos. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 4237-4244. IEEE, 2021.
[59] Pieter R Roelfsema and Roos Houtkamp. Incremental grouping of image elements in vision. Attention, Perception, & Psychophysics, 73:2542-2572, 2011.
[60] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances neural information processing systems, 30, 2017.
[61] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint 1rXiv:1212.0400. 2012.
[62] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
[63] Weiwei Sun, Andrea Tagliasacchi, Boyang Deng, Sara Sabour, Soroosh Yazdani, Geoffrey Hinton, and Kwang Moo Yi. Canonical capsules: Self-supervised capsules in canonical pose. Advances Neural information processing systems, 11-29933-5500) 2021.
[64] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11)), 2008
[65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[66] Tuan-Hung Vu, Anton Osokin, and Ivan Laptev. Tube-cnn: Modeling temporal evolution of appearance for object detection in video. arXiv preprint arXiv:1812.02619, 2018.
14

---

[67] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 8741-8750, 2021.
[68] Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pages 1 11686-11695 2022.
[69] Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In International conference on machine learning, pages 10181-10192. PMLR, 2020.
[70] Daniel Weinland, Mustafa Özuysal, and Pascal Fua. Making action recognition robust to occlusions and viewpoint changes. In European Conference on Computer Vision, number CONF, 2010.
[71] Stephen Wolfram et al. new kind of science, volume 5. Wolfram media Champaign, IL, 2002.
[72] Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai. In defense of online models for video instance segmentation. In European Conference on Computer Vision, pages 588-605, Springer, 2022.
[73] Tao Wu, Mengqi Cao. Ziteng Gao, Gangshan Wu, and Limin Wang. Stmixer: A one-stage sparse action detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14720-14729, 2023.
[74] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[75] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18134-18 18144, 2022.
[76] Shusheng Yang, Xinggang Wang, Yu Li, Yuxin Fang, Jiemin Fang, Wenyu Liu, Xun Zhao, and Ying Shan. Temporally efficient vision transformer for video instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2885 2895, 2022.
[77] Xitong Yang, Xiaodong Yang, Ming- Yu Liu, Fanyi Xiao, Larry S Davis, and Jan Kautz. Step: Spatio-tempora: progressive learning for video action detection. In IEEE Conference Computer Vision and Pattern Recognition (CVPR),201)
[78] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[79] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2403- 2412, 2018.
[80] Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprint arXiv:2107.07905 2021.
[81] Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chieh Chen. k-means mask transformer. In European Conference on Computer Vision, pages 288 307. Springer, 2022.
[82] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 918-833. Springer, 2014.
[83] Guanqi Zhan, Weidi Xie, and Andrew Zisserman. tri-layer plugin to improve occluded detection. arXiv preprint 22110:2210.10046, 2022.
15

---

[84] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprini arXin:2003.03605, 2022.
[85] Zhishuai Zhang, Cihang Xie, Jianyu Wang, Lingxi Xie, and Alan L Yuille. Deepvoting: A robust and explainable deep network for semantic part detection under partial occlusion. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1372-13800 2018.
[86] Jiaojiao Zhao, Yanyi Zhang. Xinyu Li, Hao Chen, Bing Shuai, Mingze Xu, Chunhui Liu, Kaustav Kundu, Yuanjun Xiong, Davide Modolo, et al. Tuber: Tubelet transformer for video action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13598-13607, 2022.
16

---

8 Appendix
A1. Broader Impact Statement
[2 showed that successive filters of a convnet could act as general forms of edge, shape and texture detectors. In Fig 6 we now illustrate that higher layers of neural net can learn to group pixels into objects at semantic level without any explicit localization supervision/ multi-modal alignment. This shows that we could learn non- pan-aaametriic object-queries the highest levels in the architecture (via umsupervised-clustering into lower decoder- Jay In nature, there can be a large number of objects present in the retinal frame (eg, leaves of a tree). Depending on the granularity of fixation (eg, separation between multiple trees), we might care only about a subset of those classes 47 However, the objects we don't care about still exist, even though an object query might not bind to them. Therefore, the encoding process in a neural net should not prevent separate islands of finer objects (eg, leaves) from getting formed
A decade ago
instead of directly injecting learnable parameters/proposals
The representational collapse issue observed in 5.15.1 presents a memory-scaling bottleneck. One way to get around it is to consider the behaviour of self-replicating cellula--atoomator [7] like GLOM [24] unfolding a singular embedding creates dynamic connectionist hardware on-the-fly [29] This would also allow us spiritually succeed the simply copying a singular embedding lesser number of times on a weaker hardware would allow the student model to exist. A key question that thus remains unanswered for our community how to compress the entire knowledge of neural-net into this singular entity (eg, biological seed) and the mechanism behind its unfolding 335 (which is an inverse of the protein-folding problem).
distillation setup, i.e.
A2. Mortal computation requires self-replicable embeddings
On the other hand if we want to pack similar levels of intelligence in a brain-like interface which consumes less than 25 watts daily, we have to take an alternate intelligence can be encoded in a single cell (embedding) much like how the genetic code of a human gets encoded in the DNA. In nature, multiple copies of a zygote (cell) lead to emergence of animal's internal organs. Similarly, in our computers multiple copies of this single replicable embedding shall lead to networks whose structure gets discovered 'on-the-fly' according to local hardware constraints.
setooically-plassibbe route (
has been well established in distillation literature that [27] a sighhrr-parmmetriized teacher network can teach a lower- lower-parameter student network to obtain similar performance. If all the knowledge of the teacher could be compressed into a singular embedding, then we won't 't need distillation at all. Simply copying the single embedding many times weaker student hardware would be enough to allow the student model to exist. Although the student model would be weak due to lesser copying, the replication step for both student/teacher stems from common learnt singular representation" These poanecctionistnneworrtt
229 which start their existence from a single embedding and only remain in the memory as long as the hardware is powered on (hence the name mortal) would require learning algorithms other than backpropogation: where the parameters of each layer could be updated without knowing the precise feed-forward cathemmticcl-functoonn 25 and allow dynamic growing of the network on a smaller hardware 34]. We remain optimistic for the future that such mortal computation offers to humanity 25
A3.
Additional Supplementary Material
This section discusses the supplementary materials in addition to the main manuseript. The supple- mentary material contains seven sections:
A4. proposes a new task called oetion-segmentationn involving instance- localization of actor in a video and presents a benchmark to streamline further research in the field.
A5._
explores the importance of background context in action detection.
A6. presents the full benchmark and analyzes the background bias property in existing detectors.
"This section is meant to he philosophical and optimistic in nature.
This shall resolve the issue of oversegmentation/ inability to distinguish between phot-s that SAM still faces when given a grid of input point prompts at very- fine granularity. (38)
7 AAnd instead encode the complete phrt- hierarchy of a scene 12
An undeniable fact of nature is that imelligence/consciossees in humans emerges from self-replication of a singular cell (zygote). It still remains to be seen whether singular prokaryotic organisms like amoeba themselves
are conscious [55 43)
17

---

A7. analyzes our results for synthetic occluder motions on O-UCF & O-JHMDB datasets.
A8. discusses more detail about the video-collection and annotation process of our curated RUCF dataset.
A9 presents some qualitative samples from the proposed three Benchmark datasets, along with UCF-101 instance-level annotations. All the datasets, benchmarks and codes for this work will be released for free public usage at https:://// htt.:::ss//nnomoouss... 4open
A1O. provides the NeurIPS recommended datasheet explaining the dataset collection mechanism and other important details.
A4. A New Instance Level Benchmark
Traditionally 46 [6]. spatio-temporal action-detection has relied on predicting bounding boxes across an actor for each frame. A much harder task instead would be a finer-grained localization, i.c. predicting instance-level mask for actor instead of only bounding boxes. Surprisingly, to the best of our knowledge, only one approach i.e. VideoCapsuleNet [3] is able to solve the much harder task of instance-level action segmentation by adapting the network trivially out-of-the-boxs
We argue that research the field of instance-level reciio-seemmntattiee has largely been inhibited due to the lack of proper instance-level actor annotations for standard vettoondetection datasets[61] While the popular JHMDB 33 dataset consists of instance puppet masks, the larger UCF-24 dataset only has bounding box annotations. To rectify this, we release the instance-level annotations for UCF-24 some of whose samples have been illustrated in Fig Our insight that these annotations will now allow the existing research in action-detection and Video Instance Segmentation 772 to progress concurrently due to inherently similar problem formulations 9 Finally, we note that the results of VCAPS on instance-level benchmark in Tat are significantly lower than on the bounding-boo level benchmark in T2 This indicates that instance- localization task is significantly harder task than isolating 'broader level" bounding boxes.
Annotation Procedure for UCF-24:We generate instance-level segmentation masks on UCF101-24 videos leveraging the recent SOTA in Video Instance segmentation (72). To make sure that our instance-level action tubes are temporally coherent, we perform inference over successive chunk sizes of 100 frames, with temporal overlap of t t= 30 frames. Next, we run sliding window for smoothening the predicted tube to remove any stray pixel-level artifacts. Finally, we manually refine the obtained segmentation masks using the CVAT tool.
Table 10: Instance Level Benchmark O-UCF: A benchmark showing VCAPS model trained using our instance-level annotations on UCF-24 datasets and evaluated on clean/occluded test-set.
BG1
BG2
BG3
Occ As Aug Clean FGI
FG2 FG3
FGI FG2 FG3
FGI FG2 FG3 Cirele
Sin Avg
VCAPS I3 VCAPS 13
x
34.7
29.3
21.0
28.3 28.3
13.7
19.1
13.5 29.2 20.0 13.1
13.3
18. 1 19.9
41.1
36.4
29.5
23.5
36.3
28.1
24.4
34.9 28.2 23.8
22.6
25.8 28.5
AS. Preliminary Experiments
In Figl of the main manuscript, we had run a set of preliminary experiments which served as motivation for this benchmark study. One notable result has been illustrated in Table [T] Distractor refers to the setting when there is just one single occluder in the background of video. No Context refers to the setting when all the background pixels of the video have been blackened, thereby making it easier for the network 46 13] to classify the remaining pixels as an actor. It can be clearly observed that the highest drops (i.e. 8, for No Context case) are observed when the background entirely masked. This shows that existing networks are highly biased to the background information while trying to reason about an actor location. We note that this dependency on background counter intuitive to the desirable behaviours of learning ceaet((actorcceentti representations 86).
18

---

Table 11: Distractor Background-Contel Sensitivity: Highest performance drops are observed when background is masked (no context), whereas presence of distractor objects in background has slightly less effect.
UCF-24
JHMDB-21
Distractor
No Context
Distractor
No Context
Methods
Clean
Abs
6,
Abs
6,-
Clean
Abs
8,
Abs
6,
MOC[46]
54,4
38,6
0.71
26.6
0.49
77.2
53.7
0.70
49.5
0.64
YOWO
48.8
43.8
0.90
32.2
0.66
85.7
73.5
0.86
66.5
0.78
VCAPS 13
75.5
67.7
0.90
53.1
0.70
65.7
61.8
0.94
47.5
0.72
Q-UCF
O-JHMD6
PG2//861
FG2/062
NO0 YOHO VDAPS
POC VVDAP
F62/863
FGL/B62
FO2/BR
FO1/
FG2/BCL
FG1/DG1
PC3/OGA
FOL/DG1
Average
Average
FOS/BO3
5in
F03/B05
85n
Circle
Circle
Occlusion Robustness across multiple severity levels. The outermost envelope represents most robust model. Radial Axis represents vmAP-0.5 score. For O-UCF, VCAPS is most robust, whereas for O-JHMDB, YOWO is most robust across several categories.
Figure 8:
Table 12: O-UCF Benchmark:
Illustrates the full benchmark across severity levels of static occlusions and different trajectories in dynamic occlusions.
BGI
BG2
BG3
Methods
Oce As Aug
FG1
FG2
FG3
FG1
FG2
FG3
FGI
FG2
FG3
Circle
Sin
Avg
MOC
X
54.7
37.0
34.7
52.7
35.6
31.4
53.3
33.1
22.5
42.9
39.9
39.8
YOWO 39
X
38.6
32.6
27.3
39.1
32
27.1
37.9
32.8
26
24.1
25.7
31.2
VCAPS 13
51.4
44.2
37.3
50.7
43.6
37.6
52.4
40.7
35.7
31.8
39.1
42.2
MOC 46
48.3
43.6
39.6
47.3
43.1
39.1
47.2
44.6
38.5
48.1
45.6
44.1
YOWO 39
46.5
45.3
43.4
46.1
45.3
43.4
46.5
45.7
44.3
43.8
43.6
44.9
VCAPS 13
54.7
51.1
48.7
54.8
50.4
47.6
54.2
50.7
47.8
43.4
45.3
49.9
Table 13: O-JHMDB Benchmark: Illustrates the full benchmark across 9 severity levels of static occlusions and different trajectories in dynamic occlusions.
BG1
BG2
BG3
Oce As Aug
FG1
FG2
FG3
FG1
FG2
FG3
FGI
FG2
FG3
Circle
Sin
Avg
MOC 46
x 48.1
35.1
30.5
46
33
24.6
41.4
33.7
22.6
39.8
33.9
35.3
YOWO 39
x
52.5
48.6
47.3
50.6
47.3
45.8
48.4
47.2
45.5
45.7
47
47.8
VCAPS 13
X
50.7
37.1
20.5
47.3
34.5
22.8
49.5
34.7
21.9
38.8
33.8
35.6
MOC 46
59.6
55.9
53.3
58.9
54.3
52.9
58.5
54.8
52.2
57.2
57.2
55.9
YOWO 39
71.1
70
69.3
69.2
70.3
66.9
68.4
67
65
65.7
65.1
VCAPS 13
60.8
59
54
61.2
56.9
53.9
61.5
57.8
54.4
55.3
58.2
59.7
A6.
Full Benchmark Analysis
In Tables 12 13 we present the full benchmark of the proposed O-UCF and O-JHMDB datasets across the 9 severity levels of static occlusions and circular/sinusoidal dynamic motions.
Thhe only difference VIS involves instance-Classifiaatie which could be solved by just a single-frame object detection. However, action detection requires higher-level action-classification which requires temporal reasoning. The per- frame localization task of both the pyoblems is fundamentally identical.

---

11
17
O-UCF
0.8
677
0.7
0.8
0.8
8000
BG1
Backgrounc Severity
800
8931 Background Beverity
DG2
VCAPS
VCAPS
Figure Background Bias: As the occlusion severity over background is increased, the performance of an ideal detector should not drop (dashed line). Our VCAPS- MvPS-2 shows highest robustness of all studied models. (X axis): Different severity levels of background occlusions. (Y Axis): Relative Robustness (6,)
Table 14: Static vs Motion Trends: 2D backbones work well on Static Occlusions whereas 3D backbones work best under Dynamic Occlusions. S- Static Occluders. D- Dynamic Occluders. For analysing clear effects, results are shown with only one occluder over an actor (no bg occlusion).
UCF-24
JHMDB-21 21
Methods
Backbone
Type
Clean
S
D
D-S
Clean
S
D
D-S
4OC
DLA34 (79
2D
54.4
28.7
27.3
-0.4
77.2
34.4
34.0
-1.1
YOWO 39
YOLO +ResNext
2D+3D
48.8
24.0
25.2
1.2
85.7
54.6
55.2
0.6
VCAPS
i3D 7
3D
75.5
42.5
46.6
4.1
65.7
23.0
23.9
0.9
Background Bias: For an ideal action-detector, is expected that occlusions in the background wont impact the localization performance in the actor region. We observe this trend in Fig where the performance of an ideal detector would be black line parallel x-axis. the graph, most of the methods suffer negligible drops on increasing the occlusion severity in the background. Note that MOC 46) suffers most drop across different severity levels. Our VCAPS-Mviin obtains the highest robustness across all occlusion severity levels for both O-UCF & O-JHMDB datasets.
A7. Motion Analysis
In the main manuscript, we had experimented with realistic motion the experimented with synthetic motions on O-UCF and O-JHMDB datasets, and show those results in T4] We observe that in case of MOC using the 2D DLA34 backbone, the performance is better in case static occlusions, (i.e. 28.7 vs 27.3 on O-UCF). This shows that 2D backbones work well if occluder static. On the other hand, YOWO a 3D backbone (Res.eex).. This shows a slight positive improvement of 1.2% in UCF and 0.6% in JHMDB, thereby indicating that 3D backbone seems be helping. The best improvement is evident by purely switching to 3D based backbone as in the case of VCAPS, where gains much as 4.1% 1% can be observed. From this, can be seen that 3D backbones might help networks reason about actor locations even under the challenging scenarios when the occluders might be moving. However, we note that these differences are close within a small margin of error. Also, our experiments with different type of trajectories in 12 i.e. circle and sinusoids show inconsistent- trends thereby indicating that there is no particular motion type to which networks are more sensitive.
We had also
a combination of 2D backbone (YOLO) and
A8. Real-OUCF dataset
We have curated Real-OUCF dataset for realistic occlusion scenarios.
Video Selection Process:
The original videos in UCF-24 dataset were mostly of sporting events, but were of significantly lower resolution, i.e. 240 by 320. Furthermore, they consisted of only a single actor. Now, we have significantly upgraded that test set to reflect multiple actors which mutually occlude each other with a high degree of overlap as high as 99%. First, we scraped the videos matching keywords like "riding bike" etc. One curious way were able to get such overlap ratios
20

---

was by searching specifically for events like tandem surfing, tandem diving etc. In mondem-eventte two actors try to synchronize with each other. A lateral captured viewpoint, which captures multiple actors (with one actor behind another), offers very challenging yet realistic conditions for occlusions. Finally, our curated videos are consisting of professional sporting events like Olympics, as well as casual settings, for eg, people just playing cricket in park.
Annotation Criteria: In opatioltemporal action detection, there two questions that the annotations try to answer 1) what is the time interval during which the action occurs 2) what is the spatial location of the actor in each frame of the action -itooniiterval.. Therefore, videos to obtain action start and end times. Finally, for all the frames in between this interval, we spatially localize the actors using the CVAT annotation tool. One subtle thing that oction-detection does not try to label all the actors in the video. For eg, if some people are standing and some people are doing some useful activity like biking, we are only concerned with the biking. This is in line with the annotation procedure of official UCF24.
first temporally crop the scraped
A9. Dataset Samples
In this section, we show some qualitative samples from our proposed-datasets and benchmarks. Specifically, Fig T shows the realistic occlusion dataset which we have collected for evaluating robustness to real-world occlusions. Next, Epontann the synthetic dataset samples from O-UCF and O-JHMDB consisting of controlled occlusions. containss the exhaustive instance- level annotations of UCF-24 videos we have released to facilitate our proposed benchmark of action-segmentation.
21

---

SCRINMAGES
Figure 10: Our Real-OUCF: contains realistic occlusions with instance-level action annotations.
TENNIS NARPHOUUE
T TENNIS WARFH
T TENNIS SARPHOLS
TENNIS SARPHOLUS
Figure 11: Our Real-OUCF: contains realistic occlusions with instance-level action annotations.
22

---

CLNE DOWING
Figure 12: Our instance-level annotations for UCF-24: We propose new benchmark of instance- level action segmentation and release official annotations.
23

---

Figure 13: Our instance-level annotations for UCF-24: We propose new benchmark of instance- level action segmentation and release official annotations.
24

---

Circular
Linear
Zoom-in
Sinusoicil
Figure 14: Different occluder trajectories: are illustrated for our proposed O-UCF and O-JHMDB datasets.
25

---

Indoor Occluders
Outdoor Occluders
Indoor Occluders
Outdoor Occluders
Figure 15: Nature of Occluders: Occluders in the proposed O-UCF & O-JHMDB datasets belong to either indoor/outdoor samples.
26

---

A10. Datasheets For Datasets
Motivation
For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.
O-UCF, O-JHMDB created to perform systematic benchmark study of occlusions in video action detection. Real-OUCF was created to evaluate sota action-detectors on real-world occlusions.
Who created this dataset (e-g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
These were created by research-grous whose identity shall be released after the review process.
Composition
What do the instances that comprise the dataset represent (e-g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
Dataset contains videos of several people performing actions.
Is there label target associated with each instance? so, please provide description.
For each instance, provide an instance-level segmentation mask along with our datasets.
Are there recommended data splits (e.g., training, veveloomenntvalliaatin,, testing)? so, please provide description of these splits, explaining the rationale behind them.
The idea for a real-world occlusion dataset that it is easy to train an existing network using synthetic occlusions as a data augmentation. However, there is need of consistent real-world tet set to benchmark all the future approaches against, and that is the test set which our Real-OUCF provides.
Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide description.
very detailed care has been taken to make our annotations noise free. the rare case that correction is needed, the updated annotations will be posted on our official link (Shared on top of this manuscript).
Is the dataset self-contained, or does it link to otherwise rely external resources (e.g., websites, tweets, other datasets)? it links or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
All the videos of our dataset were obtained after hand picking from YOUTUBE, and then scraping tue to offline availability of the dataset, our dataset is now self-contained.
Does the dataset relate to people? If not, you may skip the remaining questions in this section.
Yes, it is spatio-temporal action detection. It primarily considers human actors.
27

---

Does the dataset identify any subpopulations (e.g., by age, gender)? so, please describe how these subpopulations are identified and provide description of their respective distributions within the dataset.
No.
Is it possible to identify individuals (i.e., one more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
The classes in our Real-OUCF dataset primarily belong to sports such as basketball. Several videos of the dataset are picked from official sporting events like NBA, Olympics where might be possible to identify famous athletes by face only. However, all the videos we use are already there in the public domain.
Collection Process
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data verifie?d so, please describe how.
Data was raw-video directly observed with human eyes.
What mechanisms or procedures were used to collect the data (e.g., hardware appa- ratus sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?
Manual human curation
Who was involved in the data collection process (e-g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
Graduate students, who were graciously supported by research grants with warm thanks.
Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e-g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
Dataset was collected over a period of 6 months from Dec22-May23.
Preprocessingicdannnng//alleege
Was
any preprooessinngicaannngglaaaingn of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? so, please provide a description. not, you may skip the remainder of the questions this section.
Pre-processing was done by temporally cropping video into smaller clips which indicate start/end of the action. After auto-labelling spatio-temporal masks via SAM, manually annotate/refine masks using the CVAT tool.
Was the "raw" data saved in addition to the preproeeseddcceene//bbeeee data (e.g., to support unanticipated future uses)? so, please provide a link or other access point to the "raw" data.
Link shall be provided soon for the raw data.
28

---

Is the software used to preprocessichean/labll the instances available? If so, please provide link other access point.
we use normal http://ghhhcooooooooooootsss//st https///g/gghhhh...o////////////
Yes,
Segment Anything and Computer Vision
Annotation
Tool.
Uses
Has the dataset been used for any tasks already? so, please provide a description.
Yes, for our benchmark study on the impact of occlusions in spatio-temporal video action detection.
What (other) tasks could the dataset be used for?
Amodal mask completion, Occlusion Robustness Testing etc.
Distribution
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? so, please provide description.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub) Does the dataset have digital object identifier (DOI)?
When will the dataset be distributed?
Just before the start of the neurips 2023 conference.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? so, please describe this license and/or ToU, and provide link other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
No, its free for use by all parties. However, the discretion to update more video samples in future, and refine the existing annotations over time lie with the original dataset authors.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.
No. its free for use by all parties. However, the discretion to update more video samples in future, and refine the existing annotations over time lie with the original dataset authors.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.
No. its free for use by all parties. However, the discretion to update more video samples in future, and refine the existing annotations over time lie with the original dataset authors.
Maintenance
29

---

Who will be supporiinghhosiinnmmiinninng the dataset?
Dataset will be supported by the authors/research group of this paper.
How can the owner/curator/mangger of the dataset be contacted (e.g., email ad- dress)?
By email-address, the details shall be revealed post-review.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated users (e.g., mailing list, GitHub)?
Changes if any, will be posted once year to our github repo.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will enforced.
Will older versions of the dataset continue to be supportee/hosted/maiitaaneel If so, please describe how. not, please describe how its obsolescence will be communicated to users.
Yes, this shall useful give reliable comparisons on different dataset versions in existing literature.
If others want to extend/augmmentbbild on/contribute to the dataset, is there mech- anism for them to do so? If so, please provide a description. Will these contributions be validatee/veriffee??? If so, please describe how. If not, why not? Is there process for communicainggddiiiibbtiigg these contributions to other users? so, please provide description.
We also release a python script which allows to create on -the-fly fly semi-realistic occlusions consisting of static occlusions different severity levels and different motions of the occluder. Anyone could use this to generate infinite occluder variations.
For realistic occlusions, once could always collect more real world samples expand the count of existing classes. Note that the test set we provide already significantly larger than traditional action-detection datasets. [61 33
Any other comments?
We are grateful to you for taking the time to read this datasheet and reviewing this manuscript.
30


Reviews:
[{"review_id": "2bTWJMjr3v", "reviewer": "Reviewer_cTJi", "summary": "", "questions": "", "limitations": "The authors have adequately addressed the limitations concerning the dataset in their paper.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper presents the first systematic analysis of the detrimental effects of occlusion and proposes corresponding solutions.\n\n2. The visual analysis of the capsule module is intriguing and provides valuable insights.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "p1lL0f0By8", "reviewer": "Reviewer_NCed", "summary": "", "questions": "", "limitations": "Limitations addressed.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Interesting study about impact of occlusion in spatiotemporal action localization.\n- Useful datasets / benchmarks.\n- Good experimental breadth with several models analyzed.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "LncgsqbPDj", "reviewer": "Reviewer_2htk", "summary": "", "questions": "", "limitations": "The authors are not as forthcoming about context-specific limitations of the work and only pose limitations relevant to the community at large (lines 264-269).", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors claim it is the first video occlusion dataset that focuses on the localization ability of SOTA spatial-temporal action detectors. Interestingly, Mvitv2 is robust to occlusions even when not exposed to them. It is a relevant pursuit considering the issues in self-driving cars.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ypODzosMqz", "reviewer": "Reviewer_yxN6", "summary": "", "questions": "", "limitations": "yes.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Three datasets are proposed for evaluating video action detection methods in the occlusion situations. \n\nThe impact of different aspects are evaluated, including backbone (2D vs 3D, CNN vs transformer), type of occlusion (static vs dynamic), the use of capsule and the use of occlusion as data augmentation. Based on these analyses, an improved training recipe is proposed. \n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: It seems the authors have tried their best to address the issues raised by the reviewers.  Unfortunately, the first reviewer did not give a second round of comments based on the rebuttal.    The three datasets proposed in the occlusion settings could be interesting for researchers who are doing video action detection.   The authors are encouraged to further improve their work based on the reviews.  This decision has been made in agreement with the AC and SAC.
Decision: Accept (Poster)