SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rajat Modi $^{1*}$ , Vibhav Vineet $^{2}$ , Yogesh Singh Rawat $^{1}$ , CRCV, University of Central Florida $^{1}$ , and Microsoft Research $^{2}$
This paper explores the impact of occlusions in video action detection. We facilitate this study by introducing five new benchmark datasets namely O-UCF and O-JHMDB consisting of synthetically controlled static/dynamic occlusions, OVIS-UCF and OVIS-JHMDB consisting of occlusions with realistic motions and Real-OUCF for occlusions in realistic-world scenarios. We formally confirm an intuitive expectation: existing models suffer a lot as occlusion severity is increased and exhibit different behaviours when occluders are static vs when they are moving. We discover several intriguing phenomenon emerging in neural nets: 1) transformers can naturally outperform CNN models which might have even used occlusion as a form of data augmentation during training 2) incorporating symbolic-components like capsules to such backbones allows them to bind to occluders never even seen during training and 3) Islands of agreement can emerge in realistic images/videos without instance-level supervision, distillation or contrastive-based objectives $^{2}$ (eg. video-textual training). Such emergent properties allow us to derive simple yet effective training recipes which lead to robust occlusion models inductively satisfying the first two stages of the binding mechanism (grouping/segregation). Models leveraging these recipes outperform existing video action-detectors under occlusion by  $32.3\%$  on O-UCF,  $32.7\%$  on O-JHMDB &  $2.6\%$  on Real-OUCF in terms of the vMAP metric. The code for this work has been released at https://github.com/rajatmodi62/OccludedActionBenchmark.
Deep learning[41] has led to significant advances in object detection/segmentation for both image[16, 19] and video domain[84]. Such deep neural networks are in turn widely used in self-driving cars and safety critical scenarios. A key concern for such applications is whether they are able to perform well when encountering realistic occlusions: e.g. are they able to reliably localize a pedestrian even when an occluder (say a dog) comes in front of him. However, one major limitations being existing dataset test split doesn't contain such occlusions. This raises a concern, whether these models will be robust to real-world occlusions or not.
One would suspect that the inherent inductive biases of these architectures would be enough to induce natural occlusion robustness. To verify the hypothesis, we run two preliminary setups: (Fig. 1) (A) Firstly, superimposing a single occluder (eg bus) over the actor, and, (B) Then, we analyze the performance if the occlusion is shifted to background (i.e. no occlusion over actor at all). We observe a relative drop of  $20 - 50\%$  across multiple existing state-of-the-art approaches [46, 13]. This verifies
*Corresponding Author, email: rajatmodi@ucf.edu
${}^{2}$  Grouping pixels is a perceptual phenomenon of the visual-cortex[59]. Assigning a group to a class is a property of language. If a group is identified by multiple-names, then a neural-net can learn this one-many mapping via contrastive objectives[75]. But grouping should also emerge without language since organisms can continue to perceive objects even without ears/tongue.
Figure 1: A Toy Experiment. (i-iii) superimposing a single occluder (bus) on an actor and varying its size results in drops as large as  $50\%$ . (iv) even a simple occluder (cat) in background results in  $30\%$  drop. (v) highest drops are observed if background is entirely masked. (vi-ix) Clean refers to all methods evaluated on unoccluded test sets. Best viewed when zoomed in.
our hypothesis that existing approaches are not robust to occlusion. It opens up the area that there is a need to study these models behaviour under different types and severity of occlusion.
We conduct first benchmark study of occlusions in spatio-temporal video action detection. We investigate following aspects: a) Effect of different severity level of occlusions, b) Robustness of different network backbones, c) Effect of pre-training and trainable parameters. d) Effect of static vs dynamic occlusions on models. Our study yields the following key insights: 1) Performance decreases when severity increases on actor region, but increasing severity on background does not have much impact 2) Transformers which have never seen occlusions during training can outperform CNN-based models which uses occlusions as a data augmentation. 3) a significant component of transformer's robustness (and other architectures as well) comes from pre-training on large-scale datasets[7]. 4) Existing models perform better on static occlusions than realistic dynamic occlusions.
Based on our study, we investigate techniques to make the existing video action detection models robust against occlusion. First, we experiment with augmentations and found them to be effective for all the models in improving their robustness. We further utilize some of these insights and develop a transformer-based token masking strategy that provides a robust video action detection model which outperforms all existing methods. We make the following contributions in this work:
- We analyze existing approach behaviours on occlusions and conduct a first benchmark study on occlusions in spatio-temporal action detection.
- Towards this, we propose five novel datasets, i.e. O-UCF, O-JHMDB for static occlusions, OVIS-UCF, OVIS-JHMDB for realistic occluder motions and Real-OUCF for real world occlusions to benchmark our study.
- To improve robustness of existing approaches, we show how occlusion as augmentation improves robustness. Furthermore, we propose token-masking to further improve robustness of transformer backbones.
Spatio-Temporal Action Detection (STVAD): Involves localizing an actor in each video frame and also predicting an action class. Existing methods[66, 58, 11] adopt a tubelet-based approach: action trajectories can be thought of as a sequence of actor bounding boxes spread over time. These tubes can be predicted by progressively refining predictions over a center-keyframe[46, 77] or directly learning a tubelet as a feature volume[86]. However, these methods require additional post-processing by linking these smaller tubelets across time. Other architectures like VideoCapsuleNet [13] frame action-detection as a per-pixel localization task.
Occlusions in Videos: Recently, OVIS dataset[54] was proposed for Video Instance Segmentation (VIS) under realistic occlusions. Methods like [76, 32, 3] introduce occlusion robustness by temporally averaging actor-centric embeddings in DETR-like architectures[5]. However, instance-recognition during VIS can be performed by object-detection in a single frame whereas action-classification in STVAD requires more than one frame. Approaches like [44, 70] explore occlusions
Figure 2: Sample video frames from proposed benchmark datasets. (i) occlusion severity increases across 9 severity levels and both actor/background region. (ii) occluders are sampled from both indoor/outdoor splits. (iii) Our O-UCF and O-JHMDB simulate 6 dynamic occluder motions like circle, linear, zoom-in etc. (iv) Similar, the proposed Real-OCUF is a dataset for realistic scenarios where multiple actors mutually-occlude each other. Best viewed in color.
for action-recognition. In comparison, we additionally focus on the impact of occlusions on localization ability of SOTA action-detectors.
Occlusion in Images: CompositionalNets[40] explore occlusion-robustness for image classification through EM-based mixture models. DeepVoting [85] caters to object-detection under occlusions by proposing a voting layer to gather contributions from visible object parts. While such studies are restricted to images, we study occlusions under videos.
Regularization Ten Techniques: Dropout [62] have been shown to act as an implicit regularizer [69] and reduce network overfitting. Moreover, dropout has been found effective for increasing occlusion robustness [21]. We further show that masking input tokens in transformers [51] is also an effective training strategy for this task.
3 The Video Occlusion Datasets
The aim of this study is to study occlusions in spatio-temporal video action detection. We present five different benchmark datasets to study this problem. Two of them (O-UCF and O-JHMDB) are synthetically generated to systematically study this problem by manipulating a single occlusion parameter at a time. Similarly, OVIS-UCF, OVIS-JHMDB consist of occluders exhibiting realistic motions. Furthermore, we also present a real-world dataset (Real-OUCF) to validate our findings. We use three different parameters to create our synthetic datasets, 1) Indoor/Outdoor Occluders 2) Severity of Occlusion 3) Static/Dynamic motions of occluders.
3.1 Occluder Selection
To keep the generated occlusions realistic, we first crop out several objects from the Pascal-VOC[14] images using ground-truth pixel-level annotations. Out of 2413 cropped instances, we shortlist 900 possible occluders which correspond to commonly-occurring objects. Next, these objects are distributed among two separate categories namely indoor objects (eg, chair, table) and outdoor objects (eg, aeroplane, ship etc.). Finally, we softly blend these occluders over the video pixels using RGBD maps of these objects. This blending correctly simulates a very real object (occluder) being present over in the frame, without having to resort to an intensive data collection procedure.
Table 1: Benchmark dataset statistics
Statistics	O-UCF	OVIS-UCF	O-JHMDB	OVIS-JHMDB	Real-OUCF
Classes	24	24	21	21	24
Severity Levels	9	9	9	9	-
Occ. Motions	6	6	6	6	-
Occ. Type	Synthetic	Semi-Real	Synthetic	Semi-Real	Real
Total Videos	20306	20306	5896	5896	1743
Instances	2284	2284	928	928	6920
3.2 Severity of Occlusion
In a video, an actor occupies certain portion of the frame as it moves over time. Formally, we define such an actor-region (FG) as the tightest bounding box enclosing all the ground truth boxes of the actor's trajectory over time. The remaining region is termed as background(BG).
Occlusion severity is defined as the fraction of the total area which is occupied by an occluder during occlusion. We experiment with three severity levels, namely 1 (0-20%), 2 (20-40%), and 3 (40-60%) respectively. Occlusion levels in the actor region are prefixed by FG, i.e. FG1/2/3, whereas occlusions in the background are prefixed by BG, i.e. BG 1/2/3. A combination of these actor/background severity levels yields 9 levels of severity.
3.3 Types of Occlusion
Static Occlusions: Refer to the occlusions where the occluders occupy a fixed position in the frame and don't move over time. Our dataset consists of 9 severity combinations of actor/background regions as explained in the previous section.
Dynamic Occlusions: Existing occlusion-based datasets are mostly image-based and are therefore unable to study the effects of the movement of occluders in the video. For such dynamic cases, we pick a fixed severity level (i.e. FG2, BG3) and vary the motion of the occluders from a particular starting point. Our test set consists of two types of motions, i.e. circular and sinusoidal. The training set consists of linear, zoom-in, zoom-out, or random motions. Note that the motion in the train/test set is mutually-exclusive, i.e. one type of motion in one dataset split is not present in another for fairness. Similarly, an occluder moving in the actor region never mixes over to the background region but instead gets wrapped around over the course of its trajectory.
3.4 Benchmark datasets
The statistics for five proposed benchmark datasets are illustrated in Table 1. O-UCF contains of 24 action classes, along with 20306 testing samples. Similarly, O-JHMDB contains 21 action classes with 928 samples. Both of these datasets consist of static occlusions in 9 levels of severity, 4 controlled occluder trajectories in train set, and 2 trajectories in the test set. Our OVIS-UCF/OVIS-JHMDB datasets consist of occluders with realistic motions from OVIS superimposed on top of original UCF/JHMDB datasets. Note that we term this dataset as semi-realistic because although the occluders themselves are synthetically placed on the frames, occluder trajectories are naturally occurring as observed in OVIS dataset. Real-OUCF consists of 1743 fully-realistic occlusion videos which were hand-picked from Youtube for 24 action classes. These videos were then cropped temporally using LossLessCut[2], to precisely localize start/end time step of each action. Such shorter duration clips then need to be spatio-temporally annotated. Therefore, we feed-forward all such clips through auto-label generator of GroundedSAM[48] in order to localize "person" class with an appropriately constructed textual query. Since SAM[38] is a foundational model, it segments all the persons in an image. However, we are only concerned with the people who are actually performing actions. In order to remove such excessive false positives predicted by GroundedSAM[48], we manually suppress/refine per-frame instance-level masks using the CVAT Annotation Tool[1]. Finally, we end up with  $64.1\%$  of annotated instances being occluded.
Studied models: We experiment on three most recent SOTA methods in video action detection. Namely, we pick MOC[46], YOWO[39] and VideoCapsuleNet[13] whose official Github implementations and evaluation protocols have been fully open-sourced for both UCF and JHMDB datasets. All
Table 2: O-UCF Benchmark: Performance across three occlusion severity levels. For a particular actor-region occlusion (FG), averaged results across BG1/2/3 levels are reported.
FG1	FG2	FG3
Clean	Occ	δa	δr	Occ	δa	δr	Occ	δa	δr
MOC[46]	54.4	53.6	0.99	0.99	35.2	0.81	0.65	29.5	0.75	0.54
YOWO[39]	48.8	38.5	0.90	0.79	32.5	0.84	0.67	26.8	0.78	0.55
VCAPS[13]	75.5	51.5	0.76	0.68	42.8	0.67	0.57	36.9	0.61	0.49
Table 3: O-JHMDB Benchmark: Performance across three occlusion severity levels. For a particular actor-region occlusion (FG), averaged results across BG1/2/3 levels are reported.
FG1	FG2	FG3
Clean	Occ	δa	δr	Occ	δa	δr	Occ	δa	δr
MOC[46]	77.2	45.2	0.68	0.59	33.9	0.57	0.44	25.9	0.49	0.34
YOWO[39]	85.7	50.5	0.65	0.59	47.7	0.62	0.56	46.2	0.61	0.54
VCAPS[13]	65.7	49.2	0.84	0.75	35.4	0.70	0.54	21.7	0.56	0.33
these models are tested on the original clean UCF24 & JHMDB-21 datasets, which do not contain any occlusions. Also, we evaluate these models on occ versions, i.e. our O-UCF/O-JHMDB datasets. All our transformer-based models have been trained for  $1.5\mathrm{x}$  more epochs than CNN based models to facilitate appropriate convergence. The best performing models for downstream testing are selected from a separate hold-out validation set.
Metrics A well accepted metric for action-detection is v-mAP at 0.5 ioU threshold, as it also measures the temporal consistency of the prediction across time. Assume that under occlusions vmAP drops from  $V$  to  $V'$ . Therefore, we report absolute robustness  $\delta_{a} = (1 - \frac{V - V'}{100})$  and relative robustness  $\delta_{r} = (1 - \frac{V - V'}{V})$ . Note that both  $0 \leq \delta_{a}, \delta_{r} \leq 1$ , with greater value denoting more robustness.
We present the result of our benchmark on the baselines and analyse several interesting trends.
Increasing occlusion severity over actor region reduces performance. In Tabs2,3, we occlude actor region with multiple occluders. It can be clearly observed that as the occlusion severity is increased, the performance of all the action-detectors is reduced. The notable thing is that VCAPS is most robust to severe occlusion (i.e. FG3) (in terms of absolute scores) on the larger O-UCF dataset (i.e.  $36.9\%$ ), and YOWO is most robust to occlusions on smaller JHMDB dataset (i.e.  $46.2\%$ ). The larger performance difference in MOC and YOWO could be attributed to the treatment of action-detection as a box-regression problem in MOC[46]/YOWO[39] vs dense-semiotic mask prediction in the VideoCapsuleNet[13]. Regressing the 4 coordinates of a box has a minimum probability of error as  $\frac{1}{4} = 0.25$ , whereas dense-prediction reduces this error to  $1/n$ , where  $n$  is the no of pixels being predicted and  $n >> 4$ . As long as at least four pixels are predicted along the boundaries of the box, the method performs relatively well<sup>3</sup>. Also capsules have been shown to be extremely robust to occlusions on Multi MNIST[60], and we believe similar inductive bias extends to videos.
Are backbones with more parameters necessarily more robust? To answer this question, we tried CNN/Transformer-based backbones on two of our best performing methods, i.e. YOWO & VCAPS. In Table 5, it is evident that Mvitv2-S with  $50\%$  less parameters (35M) can outperform other 3D backbones like ResNext using large as 89M parameters (i.e.  $67.3\%$  on O-UCF &  $86.8\%$  on O-JHMDB). This shows that the internal attention mechanism in transformers doesn't necessarily need more parameters to improve robustness. Note that VCAPS with Mvitv2-S backbone (67.3, O-UCF) largely outperforms YOWO which also uses Mvitv2-S backbone (31.2, O-UCF). A crucial architectural difference is that the VCAPS-Mvitv2 uses 2 additional layers of capsules, whose qualitative effects shall be revealed in a later section.
3That is why predicting a dense mask and then fitting a rectangle around it during evaluation is better than directly regressing a box. (see supplementary.)
Table 4: Effect of realistic occluder-motion on OVIS-UCF/OVIS-JHMDB datasets. Models in general perform better on static occlusions than dynamic occlusions.
Backbone	OVIS-UCF	OVIS-JHMDB
Static	Dynamic	Static	Dynamic
MOC[46]	DLA34[79]	14.8	12.4	26.8	31.9
YOWO[39]	ResNext[74]	16.5	10.1	37.6	35.3
VCAPS[13]	i3D[7]	32.3	21.7	20.9	17.3
YOWO[39]	Resnet18[18]	17.3	11.8	28.0	24.2
VCAPS[13]	Resnet18[18]	33.2	21.9	7.7	7.3
Figure 3: Effect of pretrained weights: VCAPS-Mvitt2 outperforms all models on both O-UCF and O-JHMDB. (X Axis:) Accuracy without occlusions. (Y Axis): Relative Robustness of Model. Top-Right corner of each plot corresponds to most robust model.
Pre-training improves transformer's robustness to occlusions. To investigate why transformers perform so well out of the box, we train them from scratch without pre-trained weights and test them on O-UCF/O-JHMDB datasets. From Fig 3 (i), it can be observed that transformers (i.e. VCAPS-Mvit2) perform poorly without weights compared to other models. However, on training with Kinetics 400 weights, the transformers achieve state-of-the-art results. Therefore transformers are not naturally more robust, but pre-training helps to improve their intrinsic-robustness. This also reveals the glaring dependency of transformers on large amounts of data and compute for pre-training.
Increasing parameter size in same model family improves occlusion robustness By universal-approximation theorem [31], a larger neural net can learn better representations. We investigate whether similarly increasing the model parameter size (and hence internal-representational capacity) could lead to better occlusion robustness. In Fig 4(right), it can be seen that changing the backbones from Resnet  $18\rightarrow 50\rightarrow 101$  follows this trend. This shows that increasing parameter size in the same model family generally leads to improved occlusion robustness.
Effect of occluder motion on backbones. We experiment with realistic occluder motions by superimposing occladers from OVIS datasets upon UCF/JHMDB videos, resulting in benchmark datasets called OVIS-UCF and OVIS-JHMDB respectively. In Tab14, we observe that in general models are performing better during static occlusions than dynamic occlusions. The differences still hold when the backbones are made consistent[19]. This shows that resolving actor location becomes difficult when both occladers and actors move together. Therefore, reasoning about temporal motion properly remains a worthwhile architectural pursuit. We had also experimented with controlled occluder motions like circle, sinusoids. For those results, we refer the reader to the supplementary.
From this section, we conclude that using transformers as a backbone and training using pre-trained weights can greatly improve robustness.
5 Improving Robustness under Occlusion
Next, we investigate if we can make these models robust against occlusion. Augmentation is a well studied technique to induce robustness in models against distribution shifts. We experimented with synthetic occlusions as augmentations to make the existing models robust. We generate augmented samples by superimposing occluders at random locations, various severity levels and motion. Training existing baselines on these augmented samples allows to improve occlusion robustness (Table 6).
Figure 4: Effect of number of model parameters: (i) Increasing parameter size within the same model family yields more robustness. (X Axis:) Accuracy without occlusions. (Y Axis): Relative Robustness of Model. Top-Right corner of each plot corresponds to most robust model.
Table 5: Effect of Backbones: Transformers outperform other backbones in less than half the parameters. Clean evaluation is done on UCF-24 and JHMDB-21 test sets.
Methods	Backbone	#Params (M)	O-UCF	Real-UCF	O-JHMDB
Clean	Occ	Clean	Occ	Occ
YOWO[39]	Resnet18[18]	11.7	44.7	26.9	4.4	66.1	51.8
YOLO + ResNext[74]	89	48.8	31.2	7.6	85.7	63.7
MVITv2-S[45]	35	69.4	48.2	9.1	81.3	68.2
VCAPS[13]	Resnet18[18]	11.7	57.7	36.1	6.4	35.2	18.3
ResNext[74]	89	61.2	34.7	10.8	49.3	27.7
MVITv2-S[45]	35	83.1	67.3	13.1	93.1	86.8
Next, we make some very surprising observations. In Table6, Swapping the backbone of one of the methods (VCAPS) with a transformer (i.e. Mvitv2) yields the highest performance across both UCF/JHMDB. A non-trivial observation is that a transformer- based network trained without augmentation outperforms all the other methods which are trained using occlusion as an augmentation (67.3 vs 49.9 on UCF). Training the transformers further with occlusion-based augmentation improves the performance from  $67.3\%$  to  $81.6\%$ . This is notable because this performance on a occluded test set (i.e.  $81.6\%$ ) comes close to how the detector functions in the absence of occlusion,i.e.  $(83.1\%)$  on UCF. The remaining gap of  $1.5\%$  can be bridged by explicit occlusion modelling. Therefore, transformers trained with occlusions as a augmentation strategy can significantly improve robustness.
5.1 Importance of Capsules
Here, we explore the importance of capsules towards occlusion modelling. In Tab9a, we remove all the capsule layers from VCAPS-Mvitv2. Note that the architecture then reduces to a standard 3D Unet[10], and the performance drops from  $67.3\%$  to  $64.1\%$ . Therefore, capsules are a crucial component for the performance of VCAPS-Mvitv2.
Emergent Object/Occluder separation in capsules: Next, we qualitatively explore the behaviour of capsule-based models towards occlusions. We choose our best-performing VCAPS-MvitV2 model which has never seen occlusions during training and feed-forward an occluded sample during inference in Fig5. The activation maps of some of the capsules in the primary layers have been visualized. Each capsule has an activation map of size  $\mathrm{H} \times \mathrm{W}$ , where each value denotes the confidence about the location of a particular entity (object). Applying a threshold of 0.7 results in a binary mask. It can be seen that some of the capsules selectively look at the objects (i.e. actors), and other capsules focus on occluders (eg, chair, sofa). Note that the capsules have never seen occluders during training, but are still able to focus on them. Therefore, architectural constraints like binding[17] object-specific pixels to particular slots (eg, capsules)[36] allows behaviour like unsupervised object discovery to emerge[49].
4 Quantitative improvements are not observed if the number of capsule layers are increased further than two. One reason is that capsules were originally invented for rigid bodies [23, 30] and video-capsules would need to enforce dynamic deformable reference frames.[63, 53]/ learn separate-canonical frames in case of multiple instances.[78]