SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {v m f} (F, \Lambda) = - \sum_ {p} \max  _ {k} \log \left(p \left(f _ {p} \mid \mu_ {k}\right)\right), \tag {3}
$$
$$
\mathcal {L} _ {m i x} (F, A _ {y}) = - \sum_ {p} \left(1 - z _ {p}\right) \log \left[ \sum_ {k} \alpha_ {p, k, y} ^ {m} p \left(f _ {p} \mid \lambda_ {k}\right) \right]. \tag {4}
$$
Here  $y$  is the predicted class score by the model  $y^\prime$  is the class label  $\omega$  are the network weights.  $\Lambda = \{\lambda_k = \{\sigma_k,\mu_k\} |k = 1\dots K\}$  .  $F$  is the feature map obtained from the transformer network, which is pooled across the temporal dimension.  $\Lambda$  and  $A_{y}$  are the parameters for the mixture model.
5 Experiments and analysis
Training Setup The parameters for compositional net have been trained in the same way as in [25]. We learn the parameters of  $n = 5$  occluder models  $\beta_{1}, \ldots, \beta_{n}$  in an unsupervised manner. We set the number of mixtures as 2. The mixing weights of the loss are determined empirically and are set to be  $\gamma 1 = 0.1, \gamma 2 = 5, \gamma 3 = 1$ . We train for 15 epochs using stochastic gradient descent [37] with momentum  $r = 0.9$  and a learning rate of  $l_r = 1e^{-4}$ .
Augmentation for robustness Data augmentation provides a useful technique to solve distribution shift in the dataset and has been studied under various contexts [36]. We analyze its effectiveness for robustness against occlusion. We experimented with two different models, R2P1D and MViT, and use data augmentation by synthetically occluding video with random objects distinct from the objects used for benchmarking. We use linear motion with size varying between  $20 - 40\%$ .
5.1 Discussion and analysis
In Tables 1 and 2 we can observe that all the models suffer a performance drop under occlusion, with transformer based models performing better than CNN based models. Next, we analyze the effect of
Table 1: Comparison of performance on UCF101, UCF101-O and UCF-101Y-OCC datasets.
Dataset	UCF-101	UCF-101-O	UCF-19-Y-OCC
Models	Aug	Top-1-Acc	Top-1-Acc	γa	γr	Top-1-Acc	γa	γr
R50	×	89.4	39.6	0.50	0.44	50.9	0.61	0.57
R2P1D	×	88.3	48.9	0.61	0.55	49.3	0.61	0.55
X3D	×	91.2	62.4	0.71	0.68	56.3	0.65	0.62
I3D	×	89.1	59.4	0.70	0.67	52.9	0.64	0.59
MViT	×	93.5	81.9	0.89	0.88	64.3	0.71	0.69
VideoMAE	×	96.0	79.2	0.83	0.82	65.4	0.69	0.68
MViTv2	×	96.5	83.5	0.87	0.86	66.3	0.70	0.69
R2P1D	✓	83.0	71.4	0.89	0.86	37.5	0.55	0.45
MViT	✓	92.7	84.3	0.92	0.91	59.7	0.67	0.64
MViTv2	✓	95.7	88.3	0.92	0.92	65.3	0.69	0.68
VideoMAE	✓	95.8	87.1	0.91	0.91	64.2	0.68	0.67
CTx-Net	×	93.4	86.4	0.93	0.92	67.4	0.74	0.72
Table 2: Comparison of accuracy and robustness score of studied models on the K-400-O dataset. CTx-Net2 uses MviTv2 backbone and CTx-Net-mae uses Video MAE as pretrained backbone.
Dataset	K-400	K-400-O
Model	Top-1-Acc	Top-1-Acc	γa	γr
R2P1D	70.8	24.4	0.57	0.34
R50	70.6	31.2	0.61	0.44
I3D	74.6	33.6	0.59	0.45
X3D	73.1	34.3	0.61	0.45
MViT	76.4	54.4	0.78	0.71
MViTv2	77.6	56.7	0.79	0.73
VideoMAE	78.4	58.1	0.79	0.74
CTx-Net	75.9	58.4	0.79	0.77
CTx-Net2	77.4	59.7	0.82	0.77
CTx-Net-mae	76.8	58.2	0.81	0.76
Table 3: Performance comparison showing accuracy with different severity levels and occluder motion on UCF-101-O dataset. Here L0 is  $0\%$ , L1 is  $(0 - 20)\%$ , L2 is  $20 - 40)\%$ , and L3 is  $(40 - 60)\%$  occlusion. S and D denote Static and Dynamic occluders respectively.
Models	L0	L1 -S	L1-D	L2-S	L2-D	L3-S	L3-D	Avg Acc
R50	89.4	84.7	52.9	71.3	34.3	46.7	16.5	56.5
R2P1D	88.3	79.3	45.3	55.3	27.2	33.3	19.3	49.7
I3D	89.1	85.2	56.7	71.7	40.2	46.8	23.7	59.1
X3D	90.3	85.3	46.6	73.5	34.4	47.0	21.8	56.9
MViT	93.5	91.8	86.6	87.2	80.3	80.2	70.1	84.2
MViTv2	96.5	94.5	87.2	89.5	82.3	81.3	72.4	86.2
VideoMAE	96.0	94.7	87.8	91.2	80.1	81.5	66.7	85.4
CTx-Net	93.4	92.4	89.7	87.6	82.4	82.2	78.9	86.7
occlusion properties on the performance and also conduct controlled experiments in which we fix occlusion properties to study their effects.
Effect of occluder dynamics In Table 3 and 4 we show the impact of motion of occluders on models performance. We can observe most of the models excel with static occluders as compared with dynamic occluders. Table 4 demonstrates that both CNN-based and transformer-based models experience performance decline with increasing occluder motion complexity, particularly with random motion. This highlights the greater sensitivity of models to temporally incoherent motion compared to temporally coherent motion.
Table 4: Accuracy of models with occluders exhibiting different motions as well as occluders belonging to different classes; M1: straight line, M2: Random motion, M3: Circular Motion, M4: Static Motion, S1: Desktop, S2: Motorcycle, S3: Human, and S4: Cat. AA-S: average accuracy for shapes, and AA-M: average accuracy for motion variations.
Models	S1	S2	S3	S4	AA-S	M1	M2	M3	M4	AA-M
R50	47.2	36.9	40.3	40.8	41.1	57.1	6.4	33.7	67.4	41.3
R2P1D	31.2	33.6	40.8	29.3	33.7	42.8	14.6	32.5	59.3	37.3
I3D	41.3	49.1	46.2	44.8	45.3	58.6	16.2	44.6	67.5	46.6
X3D	42.3	41.6	37.4	43.4	41.2	50.9	16.7	34.7	69.3	42.3
MViT	83.8	81.1	72.2	80.4	79.3	80.1	79.3	78.3	84.3	80.5
MViTv2	85.8	83.7	75.2	82.1	82.6	83.9	77.3	81.3	88.3	82.7
VideoMAE	79.8	85.2	73.8	78.2	79.2	84.1	67.3	78.1	89.1	79.6
CTx-Net	87.7	85.2	74.3	83.2	82.6	85.2	81.2	84.3	86.3	84.2
Figure 5: Class-wise performance analysis. Confusion matrix for (i) MViT, (ii) MViT+data augmentation, and (iii) CTx-Net on UCF-19-Y-OCC dataset.
Effect of occlusion severity To investigate the impact of occlusion severity, we vary the severity of the occluder from  $0\%$  to  $60\%$  (Figure 3 and Table 3). Notably, transformer-based models demonstrate significantly higher robustness to occlusion severity compared to CNN-based models.
Effect of type of occluder We vary the occluder's shape from spatially primitive (desktop) to spatially complex (human). We observe that baseline models perform poorly when the occluder is in the form of a human (Table 4). This pattern is also observed in CTx-Net, which consistently outperforms CNN-based and transformer based baselines across all occluders. Interestingly, X3D and R50 exhibit worse performance when the occluder is in the form of a motorcycle (less complex but more area).
Effect of synthetic and natural occlusions Comparing the occluded datasets UCF-101-O, K-400-O, and UCF-19-Y-OCC in Tables 1 2 and Figure 5, we observe that CNN-based models have low relative robustness scores for synthetic occlusions. However, for natural occlusions that appear intermittently in the video, CNN-based models exhibit higher relative robustness scores than in the case of synthetic occlusion. This suggests that CNN models are less resilient to spatially constant occluders than to occlusions that dynamically enter and exit the frames. On the other hand, transformer based models demonstrate a larger decrease in relative robustness scores for natural occlusions that sporadically appear throughout the video, as opposed to synthetic occlusions that persist throughout. This indicates that transformer-based models are more susceptible to dynamic natural occlusions than to spatially constant occlusions. More specifically in UCF-101-O, a dataset with synthetic augmentation, synthetic occluders cause a distribution shift that is constant across training and test sets which results in increased robustness for data augmented methods. However, this is not the case for UCF-19-Y-OCC where augmented models are less robust, indicating a difference in distribution shift caused by natural and synthetic occlusions.
Effect of pretraining Next, we study the effect of pretraining on robustness. We experiment with both CNN and transformer based models. To test this we train models from scratch, on UCF-101 dataset and compare their performance with models finetuned on UCF-101 dataset, these models weights were initialized with pretrained weights on K-400 dataset. Figure 6 shows that pretraining clearly helps in increasing the performance of models and making it more robust to occlusion. Furthermore, pretraining increases relative robustness of transformer based models more than CNN based models.
Figure 6: Effect of pretrained weights on models performance for UCF-101-O dataset. Left: accuracy, middle: relative robustness  $\gamma^r$ , and right: absolute robustness  $\gamma^a$ .
Table 5: Ablations to study the effect of different components of CTx-Net on UCF101-O dataset.
Dataset	UCF-101	UCF-101-O
Models	Top-1-Acc	Top-1-Acc	γa	γr
CTx-Net (CNN)	24.2	10.3	0.86	0.43
CTx-Net (spatial occluder kernels)	90.3	84.2	0.94	0.93
CTx-Net (w/o class encoding)	91.2	83.3	0.92	0.91
CTx-Net (pretrained with occluded data)	85.2	75.4	0.89	0.88
CTx-Net	93.4	86.4	0.93	0.92
5.2 Improving robustness against occlusion
We observe that augmentation does help with robustness against occlusion in synthetic benchmarks for both transformer and CNN based models but fails to generalize to natural occlusion (Table 1). The proposed CTx-Net, a compositional model, improves upon the performance on synthetic occlusion as well outperforms all the models on natural occlusion.
Ablations on CTx-Net We perform ablations of the proposed CTx-Net on UCF-101-O dataset to demonstrate the effects of the chosen components on its robustness to occlusion and report its results in Table 5. CNN vs transformer backbone: First, we compare the effect of CNN vs transformer backbones on the network. We observe that CNN based network is not able to model long dependencies, which are essential. Spatial vs spatio-temporal kernels: We perform experiments with spatial occluder kernel as opposed to the spatio-temporal occluder kernel. We observe a drop in performance, demonstrating the importance of spatio-temporal nature of occlusion. Impact of class token: We also demonstrate the importance of using class encoding during classification using CTx-Net. As we can see from Table 5 the class encoding captures information from all patches to determine the class as in [51], [52]. Training on occluded data: We also analyze the performance of CTx-Net when trained using occluded data. We can clearly see that there is a significant drop in performance as compared to vanilla CTx-Net. This can be attributed to the presence of occladers causing the class mixture model to treat occluder as an essential property for action detected.
6 Conclusion and findings
In this work, we focus on video action recognition under occlusion. To the best of our knowledge, this is the first study focusing on this problem. To study this problem, we propose two synthetic benchmarks datasets based on UCF-101 and Kinetics-400 datasets and also a dataset containing naturally occluded videos. We observe several interesting findings such as 1) transformers more robust than CNNs, 2) pretraining helps all models, and 3) augmentation helps, but does not generalize to real-world occlusions. We also propose a simple transformers based compositional model which outperforms existing methods and even augmentation based models on synthetic benchmarks as well as on the proposed naturally occluded benchmark. All three benchmark datasets and code is publicly available at this [link].
[1] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6836-6846, 2021.
[2] Fred Attneave. Some informational aspects of visual perception. Psychological review, 61(3):183, 1954.
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021.
[4] Irving Biederman. Recognition-by-components: a theory of human image understanding. Psychological review, 94(2):115, 1987.
[5] Elie Bienenstock, Stuart Geman, and Daniel Potter. Compositionality, mdl priors, and object recognition. Advances in neural information processing systems, 9, 1996.
[6] Sofia Broome, Ernest Pokropek, Boyu Li, and Hedvig Kjellström. Recur, attend or convolve? on whether temporal modeling matters for cross-domain robustness in action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4199-4209, January 2023.
[7] Jiezhang Cao, Yawei Li, Kai Zhang, and Luc Van Gool. Video super-resolution transformer. arXiv preprint arXiv:2106.06847, 2021.
[8] Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset, 2019.
[9] Yu Cheng, Bo Yang, Bo Wang, Wending Yan, and Robby T Tan. Occlusion-aware networks for 3d human pose estimation in video. In Proceedings of the IEEE/CVF international conference on computer vision, pages 723-732, 2019.
[10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[11] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6824-6835, 2021.
[12] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203-213, 2020.
[13] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211, 2019.
[14] Tianshi Gao, Benjamin Packer, and Daphne Koller. A segmentation-aware object detection model with occlusion handling. In CVPR 2011, pages 1361-1368. IEEE Computer Society, 2011.
[48] Jianyu Wang, Zhishuai Zhang, Cihang Xie, Yuyin Zhou, Vittal Premachandran, Jun Zhu, Lingxi Xie, and Alan Yuille. Visual concepts and compositional voting. arXiv preprint arXiv:1711.04451, 2017.
[49] Xianyuan Wang, Zhenjiang Miao, Ruyi Zhang, and Shanshan Hao. I3d-lstm: A new model for human action recognition. In IOP Conference Series: Materials Science and Engineering, volume 569, page 032035. IOP Publishing, 2019.
[50] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[51] Jiangtao Xie, Ruiren Zeng, Qilong Wang, Ziqi Zhou, and Peihua Li. So-vit: Mind visual tokens for vision transformer. arXiv preprint arXiv:2104.10935, 2021.
[52] Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu. Multiclass token transformer for weakly supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4310-4319, 2022.
[53] Di Yang, Yaohui Wang, Antitza Dantcheva, Lorenzo Garattoni, Gianpiero Francesca, and François Brémond. Self-supervised video pose representation learning for occlusion-robust action recognition. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 1–5. IEEE, 2021.
[54] Menglong Yang, Yiguang Liu, Longyin Wen, Zhisheng You, and Stan Z Li. A probabilistic framework for multitarget tracking with mutual occlusions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1298-1305, 2014.
[55] Chenyu Yi, Siyuan Yang, Haoliang Li, Yap-peng Tan, and Alex Kot. Benchmarking the robustness of spatial-temporal models against corruptions. arXiv preprint arXiv:2110.06513, 2021.
[56] Sangwoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[57] Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, and Jinhyung Kim. Videomix: Rethinking data augmentation for video classification. arXiv preprint arXiv:2012.03457, 2020.
[58] Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, and Chen Change Loy. Self-supervised scene de-occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[59] Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8827-8836, 2018.
[60] Yumeng Zhang, Gaoguo Jia, Li Chen, Mingrui Zhang, and Junhai Yong. Self-paced video data augmentation by generative adversarial networks with insufficient samples. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1652-1660, 2020.
[61] Yuliang Zou, Jinwoo Choi, Qitong Wang, and Jia-Bin Huang. Learning representational invariances for data-efficient action recognition. Computer Vision and Image Understanding, 227:103597, 2023.
All three benchmark datasets and code will be made publicly available at this [link]. We include the following results and details in this supplementary,
1. We provide classwise performance of CTx-Net on UCF-101 and UCF-101-O dataset.
2. We provide additional results for data augmented transformer.
3. We provide a detailed network architecture of the proposed CTx-Net.
4. Qualitative Results for CTx-Net.
5. We provide tsne visualizations for effect of occlusion.
6. We further provide the confusion matrix on UCF-101-Y-OCC dataset for other models as well.
7. Limitation and Social Impact.
8. Datasheet for the proposed datasets.
A.2 Class wise Performance
In figure 7 we provide the classwise performance for the proposed CTx-Net on UCF-101 and UCF-101-O dataset. The result shows the relative robustness score for each class. As expected for most of the classes, the relative robustness score is less than 1 which shows the negative effect of occlusion on the task of action recognition. Further, a few outlying classes have robustness score more than 1 which shows that the model performs better on occluded frame as compared to clean frames. We can also notice from the plots that for most of the classes the score is close to 1 which shows the robustness of the proposed CTx-Net irrespective of classes.
In Figure 13 performance of MVit+data augmentation on out of distribution occluders is shown. It is performed by including a specific number of occluders during test time which were present during training as well. From the Figure we can clearly see that the proposed CTx-Net has minimal effects on performance while varying the distribution of occluders whereas the performance of data augmented model falls as the number of out of distribution occluders increase. This shows data augmentations inability to generalize well.
A.4 UCF-19-Y-OCC dataset
The UCF-19-Y-OCC is composed of 19 classes. The classes included are - Band Marching, BenchPress, Biking, Playing Cello, Baby Crawling, Walking with dog, Drumming, Playing Flute, Hand stand Pushups, Kayaking, Mopping Floor, Nunchucks, Pizza Tossing, Pushups, Skateboarding, Skiing, Soccer Juggling, Soccer Penalty, Surfing. All of these actions are also present in UCF-101 dataset. Figure 14 shows the distribution of number of clips per class in the dataset. Figure 13 shows the confidence matrix plot for I3D and X3D on UCF-19-Y-OCC dataset. It can be clearly seen that these methods do not have enough discriminative properties in case of natural occlusion.
A.5 Effects of Occlusion on feature representation
From Figure 12 we can see that the feature representations learned by X3D and I3D are quite discriminative in case of clean frames, as they are able to provide distinct cluster in the tsne plot, whereas for UCF-101-O we can see that the feature representation does not have enough discriminative power given the lack of distinct clusters.
A.6 Qualitative Results
Figure 15 shows qualitatively the localization of occladers in a video. Each row represents frames in a video, followed by localization of occladers performed by CTx-Net. We observe that CTx-Net
Figure 7: Class wise performance of CTx-Net on UCF-101 dataset using relative robustness as the metric.
is able to localize occluders exhibiting a variety of shapes and motions effectively. Additionally, we present qualitative findings pertaining to the CTx-Net model trained on data augmented with synthetic occlusions, as shown in Figure 16. Notably, we observe an interesting phenomenon: the model tends to identify certain attributes as occlusions even within video segments that remain unoccluded throughout. This behavior is rooted in the utilization of augmented data to train a class mixture model within the CTx-Net architecture. Consequently, this affects the model's ability to generalize effectively to non-occluded scenarios. Our study also extends to the comparative analysis detailed in Table 3 and Table 4. Particularly, we analyze instances where VideoMAE's performance is lower in contrast to MViTv2 and MViT. Visualizations of videos that exhibit correct classification by MViT and MViTv2, yet are misclassified by VideoMAE, are shown in Figure 17. These observations contribute to a comprehensive understanding of the strengths and limitations exhibited by the models under study.
Figure 8: Overview of the CTx-Net architecture. A transformer network used to extract features. Class token is then aggregated with feature tokens, followed by feature pooling using average pooling, followed by this a convolution with vMF kernels  $\mu_{n}$  followed by non-linear vMF activation  $\mathcal{N}(.)$ . The resulting vMF likelihood L is used to compute the occlusion likelihood O using the occluder kernels  $\beta$ . Furthermore, L is used to compute the mixture likelihoods  $E_{y}^{m}$  using mixture models  $A_{y}^{m}$ . O and  $E_{y}^{m}$  compete in explaining L the orange box and are combined to obtain the final class score.
Figure 9: Visualization for the patches in a video activated by three different vmf kernel. Left column: represents actions which comprises hand movements closer to upper body, right column: represents action which comprises hand movements closer to lower body
A.7 Model architecture
Figure 8 shows the model architecture of the proposed CTx-Net which uses a transformer backbone. To calculate the class score, first the feature and class tokens are obtained for the given video. Following this, the feature token and class tokens are aggregated. Then vmf likelihood of the obtained features is then calculated. Class models are used to obtain class likelihood for each part detected. Similarly the occluder model is then used to obtain likelihood of which fea-tures are occluded.Both
Figure 10: Visualization for the patches in a video activated by different components of mixture model for two different classes.
Figure 11: Samples from occluded benchmark datasets. First row: UCF101-O, second row: K-400-O, and third row UCF-19-Y-OCC.
Figure 12: T-sne analysis of features under occlusion comparing I3D and X3D. Starting from left, (i) I3D features on UCF-101, (ii) I3D features on UCF-101-O, (iii), X3D features on UCF-101 and (iv) X3D features on UCF-101-O.
Figure 13: Class-wise performance analysis. Confusion matrix for (a) I3d, and (b) X3d. (c) Performance of the proposed CTx-Net and MViT+data aug on out of distribution occluders.
these scores are then combined to obtain the final class score. We also visualize the patches which activate the vmf and class mixture models the most. In Figures 9 and 10 we can see that the vmf kernels learned corresponds to more fundamental movements like moving the hand up, whereas class mixture model capture same actions being performed from differing points of views.
A.8 Occluders
From Figure 18 we can see some of the images that were used for training the occluder kernel. These are a randomly selected, out of distribution images in which no action seems to take place. Hence, this helps in separating out the random occlusion that occur in the video. Figure 11 also shows the occluded samples from the proposed datasets. UCF-101-O and K-400-O showing different severity of occlusions used
A.9 Limitations and Societal Impact