SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Effect of Augmentation: Transformer based backbones outperform other CNN based models which have even used occlusion as an augmentation. Clean evaluation is done on UCF-24 and JHMDB-21 test sets. w/o: without augmentation, w: with augmentation.
Methods	Backbone	Type	O-UCF clean w/o w	Real-OUCF w/o w	O-JHMDB clean w/o w
MOC[46]	DLA34[79]	CNN	54.4	39.8	44.1	3.1	5.8	77.2
YOWO[39]	YOLO[56]+ ResNext[74]	CNN	48.8	31.2	44.9	7.6	9.0	85.7
VCAPS[13]	i3D[7]	CNN	75.5	42.2	49.9	9.9	10.8	65.7
VCAPS[13]	Mvitv2[45]	Transformer	83.1	67.3	81.6	10.5	12.1	93.1
Figure 5: Emergent object/occluder separation in capsules. (top left) An occluded video is feed-forwarded through VCAPS-Mviv2 and activations of primary layer capsules are visualized. Capsules can (i) parse an actor into constituent body parts without instance-level supervision[22, 28] (ii) segment multiple actors & objects of action (volleyball net). (iii) show evidence of focusing on occluders never seen during training (iv) undergo representational collapse, where a single capsule starts representing multiple objects (wholes) when number of objects in the scene are greater than number of capsules in the network. Best viewed in color.
Representational collapse: Each capsule is meant to represent one entity (object) only. If there are more objects (i.e. occluders) in a video than the capsules in a network, a single capsule gets forced to encode multiple disjoint objects together. This problem can't be solved without increasing the number of capsules further and retraining the network again<sup>5</sup> This similar issue arises in the standard object detection literature where the maximum number of objects that a network can detect is equal to the number of object proposals it was originally trained with [5, 9, 57].
5.2 GLOM: Islands of agreement can emerge from token representations.[26]
We qualitatively explore the effectiveness of our VCAPS-Mvittv2 model presented in Tables7,9b. Specifically, we assume that each of the output token representation from Mvittv2 is equivalent to a column-based activity vector [26]. We perform t-sne reduction [64] of each vector to three dimensions, and linearly project obtained components to the RGB range of [0, 256) in Fig6. Note that obtained maps are not simple patch-based correlations among token-representations[52], but rather lower dimensional clustering of vectors themselves plotted as identical colors[81]. Furthermore, our VCAPS-Mvittv2 has only received actor-level annotations/ action-label supervision and not any form of occluder masks [83], contrastive-textual supervision[68] or distilled knowledge[6] during training. The only used prior is Kinetics pretrained weights learnt from action-recognition.
This suggests that the lower layers in our VCAPS-Mviv2 (i.e. transformer layers) perform semantic grouping of pixels into objects[75] (object discovery). Next, higher layers (capsule layers) selectively segregate objects/occluders into instance-specific slots 5. Thus transformers and capsules complement each other to achieve first two stages of the binding process[17]. Finally, our decoder reasons to suppress occluder-specific features and focus/infill on actor-centric regions during localization, i.e. Fig 7. From this section, we conclude that two capsule layers stacked on top of transformer-based backbones can help improve occlusion-robustness. The neuro-symbolic advantage of capsules
<sup>5</sup>This differs from how the brain iteratively parses variable number of objects in a constant number of neurons, although it may take more time[50]. In contrast, parallel models of attention[65] and capsules[60] have assumed singular fixation. This requirement forces object queries to compete through self attention[5] for explaining perceptual parts of an input[49]. However, competing queries raise memory requirement linearly for each object that needs to be decoded[5]. Therefore, while neural nets can encode in a finite number of learnable neurons, parallel decoding suffers a fundamental memory bottleneck[5, 8, 67] (that can't be solved by retraining at scale). Surprisingly, neural fields aren't constrained by this limitation because they enforce a single world-specific coordinate frame instead of separate object-specific canonical frames.[80, 78]
Figure 6: Emergent islands of agreement on O-JHMDB dataset[15]: Output token representations of our VCAPS-Mviv27 can be thought of as column-based vectors[26, 4]. All the vectors belonging to the same object should agree among themselves. Reducing dimensionality of these vectors to three using t-sne shows evidence of such islands of agreement as belonging to identically-colored regions. Note that obtained islands show both occluders as well as actors. Our video-based model is only trained with actor-level annotation/action class labels and no occluder-specific masks.
Figure 7: Proposed Token Masking (from left to right) An input video is partitioned into tokens and fed to a transformer. (ii) During training, a random number of tokens are blacked out. (iii) During inference, all the tokens are switched on and the actor is localized under occlusions.
Table 7: Token Masking: Improves performance of models trained even using occlusion as augmentation.(D)- dropout on intermediate network layers. (T)- proposed token masking
Backbone	Mode	clean	O-UCF	clean	O-JHMDB
aug	aug+mask	aug	aug+mask
MOC[46]	DLA34 [79]	D	54.4	44.1	47.4	77.2	55.9	61.3
YOWO [39]	ResNext	D	48.8	44.7	46.8	85.7	68.0	72.8
VCAPS[13]	i3D [7]	D	75.5	49.9	51.5	65.7	59.7	62.5
VCAPS	Mvitv2 [45]	D	83.1	81.6	81.9	93.1	90.5	91.7
VCAPS	Mvitv2 [45]	T	-	-	82.2	-	-	92.4
combined with innate natural robustness of transformers forms the basis of our best model, i.e. VCAPS-Mvitv2.
5.3 Token masking
Till now, we have seen that architectures possessing a transformer based backbone with a few capsule layers on top are highly robust to occlusions. Now, we propose token-masking to improve their innate robustness. Given a video of dimensions  $T \times H \times W$ , a transformer creates  $L$  spatio-temporal patches each of which is projected to a common dimensionality  $D$  during input. Our idea is that randomly masking (blacking out) some of the tokens during training introduces additional occlusion robustness in the model. Mathematically, we generate  $L$  iid bernoulli random variables
$$
M _ {i} \sim \operatorname {B e r n o u l l i} (p) \forall i \in [ 1, L ] \tag {1}
$$
where  $p$  is the masking probability. Each  $M_{i}$  is repeated  $D$  times, so that a particular token  $P$  is fully masked before input. Next,  $M_{L}^{D}$  is multiplied element wise with the input sequence to give final masked input  $I' = I \odot M$  where  $I, M$  are input sequence vector and masking vector of dimensions  $\mathbb{R}^{L \times D}$  respectively, and  $\odot$  is element-wise hadamard operation. During inference, none of the tokens are masked, and the input occluded video sequence is directly feed-forwarded through the network.
Quantitative Results In Table 7 it can be seen that such token-masking improves even the performance of models which have used occlusion as a data augmentation from  $81.9\%$  to  $82.2\%$  on
Table 8: Comparison with existing methods: Comparison of our method across existing supervised approaches,*: denotes results using a CSN152 backbone.
UCF-24	JHMDB-21
Methods	Backbone	f-mAP	v-mAP	f-mAP	v-mAP
2D 3D	0.5	0.2	0.5	0.5	0.2	0.5
Yang et al.[37]	✓	75.0	76.6	-	-	-	-
Li et al. [46]	✓	78.0	82.8	53.8	70.8	77.3	70.2
Kopuklu et al.[39]	✓	80.4	75.8	48.8	75.7	88.3	85.9
Zhao et al.*[86]	✓	81.3	85.3	60.2	82.3*	81.8	80.7
Duarte et al.[13]	✓	78.6	97.1	80.3	64.6	95.1	-
Kumar et al.[42]	✓	69.2	95.3	71.9	68.1	96.8	68.4
Tao et al.[73]	✓	83.7	-	-	86.7	-	-
Ours	✓	81.2	98.6	83.1	93.0	98.1	92.8
(a) Capsules stacked on transformer backbone improve occlusion robustness. Testing is done on O-UCF.
Table 9: Ablations and evaluation of proposed token-masking on Realistic Occlusions.
| w/o caps	w/ caps
VCAPS (MvitV2)	64.1	67.3
(b) Realistic Occlusions: Models trained with tokenmasking outperform other models on Real-OUCF.  $\dagger-$  Training with occlusion augmentation & dropout.  $\ddagger-$  Training with occlusion augmentation & token-masking.
MOC†	YOWO†	VCAPS† (i3D)	VCAPS‡ (Mvitv2)
Occ	7.2	9.6	11.7	14.3
O-UCF and  $91.7\%$  to  $92.4\%$  on O-JHMDB. Note that for VCAPS(Mvitv2), masking some of the tokens during training performs better than dropping some of the intermediate network neurons. This observation is in line with how learning objectives similar to pixel-level reconstruction in masked image modelling (MIM)[20] yields better self-supervised representations.
Comparison With existing Methods: We compare with existing methods on f-mAP and v-mAP. In Tab8, our method obtains 83.1 vmAP on UCF-24 thereby indicating most localization robustness. On JHMDB-21, we obtain  $92.8\%$  in terms of the absolute v-mAP score, which is significantly better than other existing methods. We acknowledge that TubeR[86] and ST-Mixer [73] are slightly better than our method in terms of f-mAP scores on UCF-24 dataset.
Realistic Occlusions: Furthermore, we evaluate our baseline models and the token-masked models on the Real-OUCF dataset. In table 6, we observe that the models trained with occlusion as augmentation perform better than models without augmentation. As evident in Table 9b, models trained with our token-masking approach also outperform other models in realistic scenarios.
We have conducted the first-ever benchmark study to evaluate the impact of occlusions in spatiotemporal action detection. This study provides several interesting key insights including, i) Models perform better on static occlusions vs dynamic occlusions in realistic scenarios. ii) Transformer-based models possess greater natural robustness compared to other models using occlusion as data augmentation. iii) Pre-training improves robustness of transformers more than CNN models. (iv) Robustness can be improved further by leveraging components like capsules and training jointly with occlusions as data augmentation. (v) Recipes like simply masking some input tokens of transformers during training can introduce additional robustness and obtain state-of-the-art performance under realistic occlusions. Our benchmark, datasets and code have been released at this link.
Limitations: We have only focused on annotating visible regions of the occluded-objects in line with official COCO protocol[47]. Another direction could also focus on predicting missing regions, i.e. amodal-segmentation. Classically, semantic-segmentation has assumed that one pixel can belong to only one object[19]. However, this is not true for occlusions. Inductively, this symmetry issue has been resolved in works like Maskformer2[9] where multiple objects per pixel can be predicted by removing the softmax assumption. We note that there is still a significant gap left to bridge in existing models for improving robustness to realistic occlusions9b.
This research is based upon work supported in part by the Office of the Director of National Intelligence (Intelligence Advanced Research Projects Activity) via 2022-21102100001 and in part by University of Central Florida seed funding. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the US Government. The US Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
[1] Computer vision annotation tool. https://github.com/opencv/cvat, 2013.
[2] Losslesscut. GitHub repository, 2013.
[3] Adrià Caelles, Tim Meinhardt, Guillem Brasó, and Laura Leal-Taixe. Devis: Making deformable transformers work for video instance segmentation. arXiv preprint arXiv:2207.11103, 2022.
[4] Yuxuan Cai, Yizhuang Zhou, Qi Han, Jianjian Sun, Xiangwen Kong, Jun Li, and Xiangyu Zhang. Reversible column networks. arXiv preprint arXiv:2212.11696, 2022.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages 213-229. Springer, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[7] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[8] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. arXiv preprint arXiv:2211.09788, 2022.
[9] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022.
[10] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, pages 424-432. Springer, 2016.
[11] Ishan Dave, Zacchaeus Scheffer, Akash Kumar, Sarah Shiraz, Yogesh Singh Rawat, and Mubarak Shah. Gabriellav2: Towards better generalization in surveillance videos for action detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 122–132, 2022.
[12] Mingyu Ding, Yikang Shen, Lijie Fan, Zhenfang Chen, Zitian Chen, Ping Luo, Joshua B Tenenbaum, and Chuang Gan. Visual dependency transformers: Dependency tree emerges from reversed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14528-14539, 2023.
[13] Kevin Duarte, Yogesh Rawat, and Mubarak Shah. Videocapsulenet: A simplified network for action detection. Advances in neural information processing systems, 31, 2018.
[14] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes (voc) challenge. International journal of computer vision, 88 (2):303-338, 2010.
[84] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605, 2022.
[85] Zhishuai Zhang, Cihang Xie, Jianyu Wang, Lingxi Xie, and Alan L Yuille. Deepvoting: A robust and explainable deep network for semantic part detection under partial occlusion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1372-1380, 2018.
[86] Jiaojiao Zhao, Yanyi Zhang, Xinyu Li, Hao Chen, Bing Shuai, Mingze Xu, Chunhui Liu, Kaustav Kundu, Yuanjun Xiong, Davide Modolo, et al. Tuber: Tubelet transformer for video action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13598-13607, 2022.
A1. Broader Impact Statement
A decade ago $^{6}$ , [82] showed that successive filters of a convnet could act as general forms of edge, shape and texture detectors. In Fig 6, we now illustrate that higher layers of a neural net can learn to group pixels into objects at a semantic level without any explicit localization supervision/ multi-modal alignment. This shows that we could learn non-parametric object-queries at the highest levels in the architecture (via unsupervised-clustering), instead of directly injecting learnable parameters/proposals into lower decoder-layers[5]. In nature, there can be a large number of objects present in the retinal frame (eg, leaves of a tree). Depending on the granularity of fixation (eg, separation between multiple trees), we might care only about a subset of those classes[47]. However, the objects we don't care about still exist, even though an object query might not bind to them. Therefore, the encoding process in a neural net should not prevent separate islands of finer objects (eg, leaves) from getting formed $^{7}$ .
The representational collapse issue observed in Sec5.1 presents a memory-scaling bottleneck. One way to get around it is to consider the behaviour of a self-replicating cellular-automaton [71] like GLOM[24]: unfolding a singular embedding creates dynamic connectionist hardware on-the-fly[29]. This would also allow us to spiritually succeed the computationally-expensive distillation setup, i.e. simply copying a singular embedding lesser number of times on a weaker hardware would allow the student model to exist. A key question that thus remains unanswered for our community is how to compress the entire knowledge of a neural-net into this singular entity (eg, biological seed) and the mechanism behind its unfolding[35](which is an inverse of the protein-folding problem).
A2. Mortal computation requires self-replicable embeddings
On the other hand if we want to pack similar levels of intelligence in a brain-like interface which consumes less than 25 watts daily, we have to take an alternate biologically-plausible route[26]: intelligence can be encoded in a single cell (embedding) much like how the genetic code of a human gets encoded in the DNA. In nature, multiple copies of a zygote (cell) lead to emergence of an animal's internal organs. Similarly, in our computers multiple copies of this single replicable embedding shall lead to networks whose structure gets discovered 'on-the-fly' according to local hardware constraints.
It has been well established in distillation literature that [27] a higher-parameterized teacher network can teach a lower-parameter student network to obtain similar performance. If all the knowledge of the teacher could be compressed into a singular embedding, then we won't need distillation at all. Simply copying the single embedding many times on a weaker student hardware would be enough to allow the student model to exist. Although the student model would be weak due to lesser copying, the replication step for both student/teacher stems from a common learnt singular representation<sup>8</sup>. These 'connectionist-networks' [29] which start their existence from a single embedding and only remain in the memory as long as the hardware is powered on (hence the name mortal) would require learning algorithms other than backpropagation: where the parameters of each layer could be updated without knowing the precise feed-forward mathematical-functions[25] and allow dynamic growing of the network on a smaller hardware[34]. We remain optimistic for the future that such mortal computation offers to humanity[25].
A3. Additional Supplementary Material
This section discusses the supplementary materials in addition to the main manuscript. The supplementary material contains seven sections:
- §A4. proposes a new task called action-segmentation involving instance-level localization of actor in a video and presents a benchmark to streamline further research in the field.
- §A5. explores the importance of background context in action detection.
- §A6. presents the full benchmark and analyzes the background bias property in existing detectors.
<sup>6</sup>This section is meant to be philosophical and optimistic in nature.
<sup>7</sup>And instead encode the complete part-whole hierarchy of a scene[12]. This shall resolve the issue of oversegmentation/ inability to distinguish between part-wholes that SAM[38] still faces when given a grid of input point prompts at a very-fine granularity.[38]
An undeniable fact of nature is that intelligence/consciousness in humans emerges from self-replication of a singular cell (zygote). It still remains to be seen whether singular prokaryotic organisms like amoeba themselves are conscious [55, 43]
- §A7. analyzes our results for synthetic occluder motions on O-UCF & O-JHMDB datasets.
- §A8. discusses more detail about the video-collection and annotation process of our curated Real-OUCF dataset.
- §A9. presents some qualitative samples from the proposed three Benchmark datasets, along with UCF-101 instance-level annotations. All the datasets, benchmarks and codes for this work will be released for free public usage at https://anonymous.4open.science/r/OccludedActionBenchmark-B9E2.
- §A10. provides the NeurIPS recommended datasheet explaining the dataset collection mechanism and other important details.
A4. A New Instance Level Benchmark
Traditionally[46, 86], spatio-temporal action-detection has relied on predicting bounding boxes across an actor for each frame. A much harder task instead would be a finer-grained localization, i.e. predicting instance-level mask for an actor instead of only bounding boxes. Surprisingly, to the best of our knowledge, only one approach i.e. VideoCapsuleNet[13] is able to solve the much harder task of instance-level action segmentation by adapting the network trivially out-of-the-box.
We argue that research in the field of instance-level action-segmentation has largely been inhibited due to the lack of proper instance-level actor annotations for standard action-detection datasets[61]. While the popular JHMDB[33] dataset consists of instance puppet masks, the larger UCF-24 dataset only has bounding box annotations. To rectify this, we release the instance-level annotations for UCF-24 some of whose samples have been illustrated in Fig1213. Our insight is that these annotations will now allow the existing research in action-detection and Video Instance Segmentation [72] to progress concurrently due to inherently similar problem formulations<sup>9</sup>. Finally, we note that the results of VCAPS on instance-level benchmark in Tab10 are significantly lower than on the bounding-box level benchmark in Tab12. This indicates that instance-level localization task is significantly harder task than isolating 'broader level' bounding boxes.
Annotation Procedure for UCF-24: We generate instance-level segmentation masks on UCF101-24 videos leveraging the recent SOTA in Video Instance segmentation [72]. To make sure that our instance-level action tubes are temporally coherent, we perform inference over successive chunk sizes of 100 frames, with a temporal overlap of  $t = 30$  frames. Next, we run a sliding window for smoothening the predicted tube to remove any stray pixel-level artifacts. Finally, we manually refine the obtained segmentation masks using the CVAT tool.[1]
Table 10: Instance Level Benchmark on O-UCF: A benchmark showing a VCAPS model trained using our instance-level annotations on UCF-24 datasets and evaluated on clean/occluded test-set.
Occ As Aug	Clean	BG1 FG1 FG2 FG3	BG2 FG1 FG2 FG3	BG3 FG1 FG2 FG3	Circle	Sin	Avg
VCAPS[13]	×	34.7	29.3	21.0	13.7	28.3	19.1	13.5
VCAPS[13]	✓	41.1	36.4	29.5	23.5	36.3	28.1	24.4
A5. Preliminary Experiments
In Fig 1 of the main manuscript, we had run a set of preliminary experiments which served as a motivation for this benchmark study. One notable result has been illustrated in Table 11. Distractor refers to the setting when there is just one single occluder in the background of a video. No Context refers to the setting when all the background pixels of the video have been blackened, thereby making it easier for the network[46, 13] to classify the remaining pixels as an actor. It can be clearly observed that the highest drops (i.e.  $\delta_r$  for No Context case) are observed when the background is entirely masked. This shows that existing networks are highly biased to the background information while trying to reason about an actor location. We note that this dependency on background is counter-intuitive to the desirable behaviours of learning object actor-centric representations[5, 86].