SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shresth Grover
IIT Kanpur shrgo@iitk.ac.in
Microsoft Research vibhav.vineet@microsoft.com
Yogesh S Rawat  
University of Central Florida  
yogesh@crcv.ucf.edu
In this work, we study the effect of occlusion on video action recognition. To facilitate this study, we propose three benchmark datasets and experiment with seven different video action recognition models. These datasets include two synthetic benchmarks, UCF-101-O and K-400-O, which enabled understanding the effects of fundamental properties of occlusion via controlled experiments. We also propose a real-world occlusion dataset, UCF-19-Y-OCC, which helps in further validating the findings of this study. We find several interesting insights such as 1) transformers are more robust than CNN counterparts, 2) pretraining make models robust against occlusions, and 3) augmentation helps, but does not generalize well to real-world occlusions. In addition, we propose a simple transformer based compositional model, termed as CTx-Net, which generalizes well under this distribution shift. We observe that CTx-Net outperforms models which are trained using occlusions as augmentation, performing significantly better under natural occlusions. We believe this benchmark will open up interesting future research in robust video action recognition. Code is publicly available at https://shroglck.github.io/rev_unseen.
Video action recognition is a challenging and important task in computer vision with numerous real-world applications, including security, robotics, sports analysis, and human-computer interaction [42, 39]. Recent years have witnessed significant progress in this field, facilitated by the availability of large-scale datasets [20, 8, 40], enabling the learning of increasingly complex models [1, 7, 11, 13, 44, 30], which have demonstrated human-level performance. However, the robustness of such models for real-world applications, particularly in the presence of occlusion, has not been explored [38, 6]. Given the dynamic nature of the environment, occlusion is an inherent property of real-world videos and needs to be studied.
To study this problem, we propose three benchmark datasets for analyzing the robustness of deep learning models against occlusion, namely UCF-101-O, K-400-O, and UCF-19-Y-OCC. The first two are derived from UCF-101 license CCLA-BY 4.0) [40] and Kinetics-400 (license CCLA-BY 4.0)[20, 8] respectively, and are created by synthetically occluding the actions in the videos using objects from the Pascal VOC dataset [10]. These benchmarks facilitate systematic analysis of occlusion effects by examining occluder properties such as severity and occluder dynamics. Additionally, we introduce a benchmark dataset with natural occlusion, UCF-19-Y-OCC, which comprises real-world videos containing natural occlusion. This dataset, is collected from YouTube, with categories based on a subset of classes from UCF-101, enables validation of observations made in the synthetic setting within a real-world environment.
We investigate the effects of occlusion on seven different video action recognition models, encompassing various aspects such as network size and architecture (transformer vs CNN). Through extensive
Figure 1: Comparison of relative robustness, accuracy, and model parameters for video action recognition models under occlusion. Left: UCF-101-O and right: K-400-O dataset. Size of circles indicate number of parameters in the model.
experimentation, we aim to assess the generalization capabilities of these models when confronted with both synthetic and natural occlusion. Our findings yield several interesting insights. Firstly, we observe that transformer based models exhibit significantly higher robustness to occlusion compared to CNN based models (Figure 1). Additionally, all models demonstrate superior resilience to temporally consistent forms of motion compared to more chaotic motion patterns. Natural occlusion likewise has a more detrimental impact on the robustness of CNN based models than transformer based models. Similarly, synthetic occlusion appears to disproportionately affect the performance of CNN-based models compared to transformer-based models. These experiments collectively illustrate a lack of generalization among current state-of-the-art models when faced with occlusion.
Data augmentation is a widely employed technique to address the issue of distribution shift. We investigate its impact on robustness of models against occlusion. Our experiments reveal a lack of generalization in models trained with data augmentations. Furthermore, we observe that such augmentations are insufficient in achieving generalization to real-world occlusion. We also propose CTx-Net, a compositional model for robust video action recognition under occlusion. Compositionality, helps captures interdependencies among parts, mimicking the robustness of the human visual system against distribution shifts [2, 15, 4]. Compositionality has shown good robustness properties in image-based tasks such as classification and object detection [24, 25, 22, 23, 41, 5], and we extend this to videos with the help of transformers. To validate its effectiveness, we extensively evaluate it on the proposed benchmark datasets, demonstrating its robustness in video action recognition under occlusion across both synthetic and natural scenarios.
We make the following contributions in this work,
- We study the problem of video action recognition under occlusion; this is the first work focusing on this problem to the best of our knowledge.
- We propose three benchmark datasets, K-400-O UCF-101-O, and UCF-19-Y-OCC, to study occlusion in video action recognition. K-400-O and UCF-101-O are created with the help of synthetic occlusions for a systematic study and UCF-19-Y-OCC consists of videos with real-world occlusions.
- We develop a simple transformer-based compositional model CTx-Net which generalizes well under occlusion.
Video action recognition Current state-of-the-art video action recognition can be essentially broken down into two distinct ways. Firstly, CNN based models,[45, 49, 13, 18, 34, 12].Secondly, models that are based on the vision transformer[1, 31, 11, 28]. The majority of the currently available CNN-based methods make use of 3D CNNs. However, 3-D filters typically have many parameters and require the use of large-scale datasets such as Kinetics [20] in order to be trained effectively. As a result of the restricted receptive field of CNN architectures in general, they are unable to accurately model motion, which causes their performance to deteriorate as a direct result of this limitation.
In recent years, transformer architecture [46] has steadily gaining popularity as a direct result of the successes of vision transformers. Since then, there has been a surge in the amount of work done in this field, with models such as MViT, MViTv2, Swin [11, 28][31] and Timesformer[3] being able to give state-of-the-art performance on video-related tasks without significantly increasing the amount of memory overhead. [33] demonstrates the robustness of vision transformers on related tasks in the image domain, which inspired us to use transformer-based models in the video domain. Recently, [44] showed the ability of transformer models to run on relatively fewer data. Recently proposed [61, 17, 56, 57, 60] show that data augmentation is helpful in increasing robustness and data efficiency of video action recognition models.
Occlusion reasoning The existing works in image domain focus on occlusion for classification and image segmentation. In [14], the authors proposed using binary variables to infer visible cells. Hsiao and Herbert[19] modelled occlusion by using 3D relationship of objects with corresponding bounding boxes. Recent works on pixel level occlusion model use probabilistic methods as proposed by George et al. [16]. Another probabilistic approach by Yang et al.[54] introduces occlusion prior modeled by Markov random field in the domain of object tracking. Tighe et al. [43] introduce an inter class occlusion prior to parse scenes and refine pixel level labels. OFNET by Lu et al. [32] considers relevance between occlusion contours and pixel orientation Kortelewski et al. [22, 23] proposed a deep convolution based compositional network for occlusion reasoning at high level features. [58] studies the problem of scene de occlusion by obtaining the ordering graph of objects in the scene. In the context of the video domain, the challenge of occlusion has been investigated within the realms of pose estimation and video instance segmentation. Addressing this challenge, Li et al. [27] delve into the realm of action recognition under occlusion by leveraging radio signals to estimate poses. Meanwhile, Cheng et al. [9] tackle the issue of occlusion-induced ambiguity in 3D pose estimation by employing optical flow to evaluate the status of keypoints. Yang et al. [53] adopt a self-supervised paradigm to enhance pose estimation. Qi et al. [35] introduce a dedicated dataset specifically tailored for occluded instance segmentation. Ke et al.[21] introduce BCNet, which introduces a novel branch for inferring occluder information. In a similar vein, Lazarow et al. [26] introduce OCFusion, introducing an occlusion-aware module to signify occlusion relationships among mask proposals. Zhang et al.[60] propose a self-supervised approach that aims to estimate occlusion by recovering the temporal ordering of objects within videos.
Recently, there have been various efforts to combine compositional models with deep neural networks. Liao et al. [29] proposed to integrate compositionality into DCNNs by regularizing feature representations of DCNN's to cluster during learning. Zhang et al. [59] demonstrated that part detectors emerge in DCNNs by restricting the activations in feature maps to have localized distributions. However, these approaches have not been shown to enhance the robustness of deep models to occlusion. In [24, 22], the authors propose a CNN based compositional model for robustness against occlusion in images.
3 Benchmarking occlusion
In order to assess the robustness of current state-of-the-art models for video action recognition, we introduce three benchmarks. These datasets encompass a range of occlusion scenarios, including both synthetic and natural occlusions. The first two benchmarks, UCF-101-O and K-400-O, are synthetically curated and enable controlled studies of occlusion and serve as extreme cases for evaluating the impact of occlusion on action recognition models. Additionally, we introduce UCF-19-Y-OCC, a dataset comprising real-world videos with natural occlusions. This dataset provides an opportunity to examine the performance of models under more realistic occlusion conditions.
3.1 Design parameters
We study three key properties of occlusion: the type of occluder (including its shape and object class), the severity of occlusion (measuring the extent to which each frame is occluded), and the dynamics of the occluder (capturing its motion characteristics). By analyzing these properties, we aim to gain a comprehensive understanding of the impact of occlusion on video action recognition.
Type of occluders To ensure diversity among the occluders, we select 50 random objects from the PASCAL VOC dataset [10]. The objects are then masked out using segmentation masks, resized to
Figure 2: Samples video frames from proposed occluded benchmark datasets. First row: UCF101-O (left: 'Drumming' and right: 'Jump Rope'), second row: K-400-O (left: 'Air Drumming' and right: 'Arm Wrestling'), and third row UCF-19-Y-OCC (left: 'Mopping' and right: 'Skate Boarding'). Rows: 4,5,6,7,8:- Represent occlusion variations. Row 4: Represents occluder exhibiting linear motion, Row 5: Represents occluder exhibiting circular motion, Row 6: Represents occluder exhibiting random motion, Row 7: Represents static occluder with severity  $20 - 40\%$ , Row 8: Represents static occluder with  $40 - 60\%$  occluder severity
the desired dimensions, and finally pasted onto the frame at specific locations. This process allows us to maintain variability and realism of occlusion in environment.
Severity of occlusions In our study, we consider a range of occlusion severities from  $0 - 60\%$ . The severity of occlusion is quantified by calculating the average number of pixels occupied by the occluder throughout the video. For the proposed benchmark datasets, occlusion severities of  $40 - 60\%$  are employed, while severities of  $0 - 40\%$  are used during the training phase for augmentation experiments. This approach allows us to assess the impact of varying occlusion levels on video action recognition performance.
Dynamics of occluders We examine both static and dynamic occluders. For static occluders, the position of occluder is randomly chosen and remains constant throughout the video. To simulate dynamic motion, we employ four motion types with varying temporal coherence: linear, circular, and random. For linear motion, the occluder starts at a randomly selected coordinate and follows a linear path with a randomly determined slope. In the case of random motion, the occluder's position is randomly selected without considering its previous frame position. For circular motion, the occluder follows a circular path centered around a randomly chosen point. The center point is selected randomly for each video and remains consistent across all frames. In the benchmark datasets, we utilize static, random, and circular motions, while linear motion is used for augmentations. This allows us to assess the impact of different occluder dynamics on action recognition performance.
3.2 Benchmark datasets
We introduce three benchmark datasets to comprehensively investigate the challenge of occlusion in video action recognition. UCF-101-O consists of 3783 videos distributed across 101 action classes.
Figure 3: Variation of performance of action recognition models with changing occlusion severity on the proposed benchmark datasets. Left: UCF-101-O, and right: K-400-O dataset.
This includes all the videos in the test split 1 of UCF-101. The dataset incorporates occlusions that vary in size between  $40 - 60\%$ . The motion of occluders is randomly chosen from three categories: random motion, circular motion, and static. The occluders are selected randomly from a pool of 50 objects sourced from PASCAL VOC [10]. K-400-O consists of 40,000 video clips distributed across 400 action classes. It also includes all the testing videos from Kinetics-400. The selection of occlusion parameters and properties are same as UCF-101-O.
UCF-19-Y-OCC focuses on real-world occlusions and consists of publicly available videos from YouTube that feature natural occlusion. The dataset includes a total of 19 classes selected from the UCF-101 dataset, with each class consisting of 30 video clips of 5 seconds duration. The selection of classes was based on the likelihood of natural occlusion occurrence, while ensuring a diverse representation of actions to avoid bias towards specific classes. We conducted targeted searches on YouTube to curate a collection of videos pertinent to the specified class. From these we kept only those instances which contained occluded action sequences. We then extracted 5-second clips, within this duration, clips were incorporated into our dataset only if they encompassed a minimum of 2 seconds during which the action remained occluded. Figure 2 show sample video frames from the UCF-101-O, K-400-O dataset and UCF-19-Y-OCC.
3.3 Video action recognition models
We study several existing video action recognition methods with a wide range of properties for this benchmark. We use CNN based R50[50], R2P1D[45], I3D[49] and X3D[12] and transformer based MViT[11], MViTv2[28] and VideoMAE[44] as our baseline models. This helps in making a comparison based on transformer and CNN based models. Further usage of lightweight models x3d allows us to compare the effect of model size to robustness to occlusion. Except for VideoMAE, all employed models underwent fully supervised training on the training split of the K-400 dataset. For the evaluation on UCF-101, all models were initialized with weights derived from the preliminary K-400 training phase. Sequentially, they were fine-tuned, on the training subset of the UCF-101 dataset. VideoMAE model underwent self-supervised training on K-400 and subsequently transitioning to fully supervised training on the K-400 training segment. On UCF-101, weights originating from fully supervised training on the K-400 training subset were employed. Augmented models were initialized with pre-trained K-400 weights. Following the introduction of synthetic occlusions to each video, each model was fine-tuned in a fully supervised manner using the UCF-101 training split.
3.4 Evaluation metric
We use accuracy and robustness scores as the metrics for evaluation. The robustness metric is defined in two ways, relative and absolute [38]. The absolute robustness score is given as  $\gamma_p^a = 1 - \frac{A_c - A_p}{100}$  where  $\gamma_p^a$  is the absolute robustness score for severity level p,  $A_{c}$  is the accuracy on the clean dataset, and  $A_{p}$  is the accuracy on the occluded dataset with severity level p. Relative robustness is defined as  $\gamma_p^r = 1 - \frac{A_c - A_p}{A_c}$ , where  $\gamma_c^a$  is the relative robustness score.
Figure 4: T-sne feature analysis under occlusion comparing CNN and transformer architecture. Left two plots: R2P1D features on UCF-101 and UCF-101-O respectively, and right two plots: MViT features on UCF-101 and UCF-101-O respectively.
3.5 Preliminary benchmark analysis
The evaluation of existing action recognition models on the proposed benchmarks is shown in Figure 3. We use clips of  $224 \times 224$  for all datasets with 10 clips uniformly sampled across the temporal dimension, applying center crops to these clips. We observe that all the models show a drop in performance under occlusion, and the drop increases with the increase in severity. Our findings indicate a lack of generalization to occlusion in the existing baseline models. We also observe that transformer based models outperform CNN based models in both accuracy and robustness. This decline in accuracy with occlusion severity mirrors observations in the image domain [33] and video domain [38, 55], but the impact is more pronounced in videos.
We find Transformer-based models (MViT, VideoMAE, and MViTv2) exhibiting higher robustness to occlusion compared to R50, R2P1D, X3D, and I3D, which experience significant accuracy degradation with increasing occlusion severity. Additionally, in Table 1 we show accuracy results on real-world benchmark UCF-19-Y-OCC, revealing the lack of robustness in baseline models when exposed to natural occlusion. We analyze this further with the help of T-SNE plots (see Figure 4) and visualize the loss of discriminative power caused by occlusion in both CNN and transformers based models. These findings emphasize the need for future studies in action recognition to incorporate occlusion as a fundamental consideration.
4 Compositional model for robustness
We propose a simple transformer based compositional model which disentangles the occluder and activity features. Compositional models have been found effective in image domain [47] [22] and we extend this to videos. They act as efficient part detectors, where parts contribute to the overall class score of the region of interest. This enables the network to focus on visible parts and mitigate the impact of occlusion, resulting in robustness to occlusion in the image domain. The compositional net comprises three main components: vmf kernels  $(\mu_1,\mu_2,\dots ,\mu_n)$ , occluder kernels  $(\beta_{1},\beta_{2},\dots ,\beta_{n})$  and class mixture models  $(A_{1},A_{2},\ldots ,A_{n})$ . The primary objective is to obtain a generative model  $p(F|y)$ , where F represents the feature representation for a video and y represents the action label. The vmf kernels aid in identifying spatially relevant regions for the actions, while the class mixture model facilitates action classification by categorizing the features into different activities.
4.1 CTx-Net
The proposed CTx-Net takes a video input  $X \in \mathbb{R}^{T \times H \times W}$ , where  $H$  is the height,  $W$  is the width, and  $T$  is the number of frames and outputs class probabilities. The model aims to disentangle occluded features from other features to mitigate their impact on the final class score. We build upon [22] by i) Employing a transformer backbone for modelling spatio temporal properties inherent to videos. ii) Usage of a collection of spatial and spatio temporal features for modelling occlusion.
Video level representation To effectively model temporal information, we utilize transformer architecture [46] as a backbone. The positional encoding preserves spatial information, while the transformer captures temporal dynamics. The output of the transformer network, denoted as  $F_{tr} \in \mathbb{R}(H' \times W' \times T' + 1) \times C'$ , represents the video's feature representation, where  $H', W'$ , and  $T'$  are the height, width, and temporal dimensions, and  $C'$  is the number of feature dimensions. We combine
the feature embedding with the class embedding to integrate both information. We use MViT as the backbone in our model.
$$
F _ {C} = F _ {t r} + c l ^ {\text {e n c o d i n g}} \tag {1}
$$
where  $F_{C} \in R^{(H'W'T') \times C'}$  represents integrated features. Following this, we aggregate the temporal information by averaging across temporal dimension, thus obtaining a feature representation  $F_{l}$  which contains both temporal and spatial information.
The feature map obtained from the transformer is then used to train the centers of the vmf cluster as described in [22]. The actions that are closest to the center of the cluster correspond to similar rudimentary actions, such as moving hands vertically. The mixture models are also learned in a manner similar to that presented in [22]. The mixture model learns actions movements which vote to compute the total class score for the entire action.
Occluder kernel The occluder kernel is obtained by clustering the vmf likelihoods. We explore occluder kernels for both temporal and spatio-temporal aspects. Temporal kernel: We obtain  $F_{c}$  as described earlier. Subsequently, mean average pooling across the temporal dimension is applied to obtain  $F_{l}$ . The similarity between these features and the learned vmf kernels is computed to obtain  $L_{l}$ . We use features  $l_{p}$  from  $L_{l}$ , where  $F_{l}$  and  $L_{l} \in \mathbb{R}^{(H^{\prime} \times W^{\prime}) \times C^{\prime}}$ , for training the occluder kernels. Spatial kernel: We compute the similarity between features  $F_{c}$  and the vmf kernels, resulting in  $L_{c}$  where  $F_{c}$  and  $L_{c} \in \mathbb{R}^{(H^{\prime} \times W^{\prime} \times T^{\prime}) \times C^{\prime}}$ . From  $L_{c}$ , we sample the likelihood  $l_{c}$ , with dimensions  $R^{C^{\prime}}$ . Spatial and temporal kernels are then combined to create spatio-temporal kernels. To train the occluder kernels, we utilize a set of random images that are unrelated to any specific action.
End to end training To incorporate the compositional model with a transformer, we use it as the classification head by replacing classical fully connected head. The differentiability of the model allows it to be fully trained by backpropagation. The trainable parameters are  $T = \{\omega, \Lambda, A_y\}$ . The loss function is similar to the one used in [22] and is defined as,
$$
\mathcal {L} \left(y, y ^ {\prime}, F, T\right) = \mathcal {L} _ {\text {c l a s s}} \left(y, y ^ {\prime}\right) + \gamma_ {1} \mathcal {L} _ {\text {w e i g h t}} (\omega) + \gamma_ {2} \mathcal {L} _ {\text {v m f}} (F, \Lambda) + \gamma_ {3} \mathcal {L} _ {\text {m i x t}} (F, A _ {y}), \tag {2}
$$
where  $\mathcal{L}_{\text{class}}(y, y')$  is the cross entropy loss between network output and the target output  $\mathcal{L}_{\text{weight}}(\omega)$  is the regularization term for the transformer network and  $\mathcal{L}_{\text{vmf}}, \mathcal{L}_{\text{vmf}}$  regularize the parameters of the compositional head [48], where