SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Point tracks: Although object tracks based on bounding boxes allow probing some physical properties of objects, such as object permanence, solidity, and coarse motion, they do not fully describe articulated or non-rigid objects, thin objects that are not axis-aligned, or out-of-plane rotation. A better understanding of physical interactions arises if we can track how object surfaces move and deform over time. To this end, we annotate point tracks on object surfaces following the protocol of TAP-Vid [19]. Annotators were instructed to select points spanning all the different parts of the objects labelled in the object tracking task. Points that are occluded are simply marked as occluded and not tracked. For translucent objects (e.g. glass cups), we only consider points to be 'visible' if they belong to the surface closest to the camera. The annotated points are dense in time (30fps). Table A2 in the appendix gives the distribution of points that are moving or static, as well as those on videos with moving cameras.
Action segments with action-relevant objects: To capture temporal understanding and enable grounding over time, we annotate the videos with temporal segments belonging to a fixed set of templated labels, e.g. putting something into something, similar to [26]. These are associated with action-relevant object tracks, i.e. objects involved in the action. The action boundaries are defined based on contact with action-relevant objects. For instance, when a person puts sugar in a tea, the putting something into something action starts when the person picks up the spoon and ends when the person puts down the spoon. If, after putting the sugar, the person starts stirring with the same spoon,
Task	Output	Metric	Baseline	Score
Object tracking	box track	Avg. IoU	SiamFC [8]	0.67
Point tracking	point track	Avg. Jaccard	TAP-Net [19]	0.40
Temporal action localisation	list of action segments	mAP	ActionFormer [57]	0.16
Temporal sound localisation	list of sound segments	mAP	ActionFormer [57]	0.15
multiple-choice videoQA	answer (1 out of 3)	top-1 accuracy	SeViLA [55]	0.46
grounded videoQA	list of box tracks	HOTA [40]	MDETR [34] + Stark [52]	0.10
Table 4: Computational tasks and top-performing baselines in the Perception Test: the model receives a video with audio, plus a task-specific input (e.g. the coordinates of a bounding box for the object tracking task), and produces a task-specific prediction, evaluated using dedicated metrics.
this defines a new segment as the type of action changed. The frequency of actions across the entire dataset is included in the appendix, Fig. A1(right).
Sound segments with sound-relevant objects: Similarly to the action segment annotations but applied to the audio modality, we collect sound segment annotations grounded in object tracks. By watching the video and listening to the audio, the annotators define temporal sound segments and label them from a list of 16 audio segment labels. For each sound, the annotators also identify the object (or objects) involved in making the sound, or specify that these are out of the camera's field of view. For example, if an object is placed on the table making an audible sound, then both the object track and the table track are associated with the sound segment. The frequency of sounds across the entire dataset is included in the appendix, Fig. A2
Question-answers for video-level reasoning: Different from the existing VQA datasets, which rely on crowd-sourced questions and answers, our team designed the questions per script to cover different types of reasoning [54]: descriptive, explanatory, predictive, counterfactual, and to cover aspects that are important for operating in the real world, e.g. understanding task completion, detecting changes, and so on. The answers for all the questions per video were provided by crowd-sourced participants. As we are interested in non-ambiguous evaluation, we favour the multiple-choice setup over the open-language answer setup. To define challenging negative options, we partly relied on human annotators, partly sampling from the correct answers of other videos in the same type of script. Table 3 bottom and Figure A3 in the appendix show the distribution of question-video pairs into perception skills, skill areas, and type of reasoning.
Question-answers with answer-relevant objects: As another way to connect high-level and low-level scene understanding capabilities, we define questions or tasks in language form, with answers given as object tracks. Similar to the multiple-choice question-answers above, our team defined the questions, and human raters selected the answers from the existing object tracks. The grounded questions are associated with skill areas and types of reasoning.
5 Computational tasks and baseline results in the Perception Test
Computational tasks: We defined six computational tasks based on the annotations available in the Perception Test. We summarise in Table4 the task definitions (outputs, metrics) and the performance of top-performing baselines. It can be observed that the Perception Test combines lower-level dense prediction tasks like object and point tracking, whose outputs are box and point trajectories, with higher-level tasks like video question answering. For all the tasks, the video and audio are available as inputs, together with a task specification where applicable, e.g. the coordinates of a box to track for object tracking, or a language question and options for multiple-choice videoQA. More details about the task definitions are included in the appendix. Note that many other computational tasks can be defined based on the available annotations, e.g. grounded temporal action/sound localisation.
Baselines: Ideally, a single model should be able to perform all the tasks in the Perception Test. Since such a model is not available in the literature, we include results obtained with per-task baselines on the validation split for all the six tasks in the Perception Test; see Table 4 for a summary of top-performing baselines and their average performance, and the appendix for more details. When selecting these baselines, we favoured strong-performing models that can be evaluated in a zero-shot or few-shot setting, as our focus is on generalisation. However, for action and sound localisation, such models do not exist in the literature, so fine-tuning on our set of classes was necessary. For the mc-vQA task, we also provide a human baseline and fine-tuned evaluation to further assess the difficulty of the dataset for humans and for SOTA video-language models, respectively.
Object tracking: The overall performance of SiamFC [8] (UniTrack [50] implementation) on our benchmark confirms the findings from [21] that simple Siamese trackers are better when probed zero-shot than more complex recent trackers, e.g. Stark tracker [52] obtains 0.56 mean IoU on the Perception Test vs 0.67 for SiamFC. Even for SiamFC, the tracking performance drops when the camera and/or the objects are moving. The results for the different categories of objects (involved in actions or in sounds, etc.) are included in Table A3, aggregated based on camera motion.
Point tracking: The performance of our baseline TAP-Net [19] is a bit lower on the Perception Test compared to the performance reported by the authors on the Kinetics dataset [36] (0.466 vs 0.401); see detailed results in Table A4. We attribute this drop in performance mainly to the increased video length in our benchmark (23s in Perception Test compared to 10s in Kinetics).
Action localisation: The confusion matrix for our fine-tuned baseline ActionFormer [57] (Fig. A4) shows that the model struggles mostly with rare action classes that are confused with more frequent ones, and it also confuses pretend actions with their non-pretend versions, e.g. ironing something vs pretending to iron something. Using multimodal inputs does not increase the performance significantly, as summarised in Table A5 top. Overall, ActionFormer's performance on our benchmark is lower compared to other benchmarks (15.56 mAP on Perception Test vs 22.7 mAP on EPIC-Kitchens [14]), most likely due to the presence of adversarial actions and our limited training set. We hope to see in the near future models that can handle open-vocabulary action classes (similarly to open-vocabulary object detection [35]), so that fine-tuning is no longer necessary.
Sound localisation: We adapted the same ActionFormer model [57] to perform the localisation task in the audio modality. The best performance is obtained when features from both video and audio modalities are used as input; see Table A5, bottom.
Multiple-choice vQA: We report results for two strong recent video language models: Flamingo [3] in zero-shot and few-shot setups, and SeViLA [55] in zero-shot and fine-tuned regimes. We also include a dummy frequency-based baseline and a human baseline. For the frequency baseline, given that each question-options pair is defined over multiple videos, we keep as answer the option that is most frequently the correct answer in the training set. One can also compute this baseline on a random subset of training examples for each question, see Table A6 to obtain a fairer dummy baseline for models using few-shot evaluation.
Human baseline. We ran a small study for the mc-vQA task with human participants. We used 126 questions from the dataset, with one video per question selected at random. We recruited 30 crowd-sourced participants (half male, half female, with advanced English skills), different from the raters annotating the videos. Each participant answered a subset of 42 questions, resulting in 10 answers per question. The performance per area and type of reasoning is detailed in Figure 2. The overall average accuracy was  $91.4\%$ . The mistakes occurred in situations difficult to judge from the given viewpoint, e.g. if a configuration of objects would be stable (without seeing the end of the video), or in edge cases where humans overlooked details happening very early on in the video. It is worth noting that the participants did not require any training, which is similar to a zero-shot setup. The median time spent to answer 42 questions was 30 minutes.
It can be observed that both Flamingo and SeViLA are far from human performance when evaluated 0-shot or few-shot and cannot outperform the 8-shot dummy frequency baseline; see Figures [2], [3], [A5] and Table [A6]. On many skills in the Memory, Physics, and Abstraction areas, their performance is below the 8-shot frequency dummy baseline, and in a few cases, e.g. (Piaget) conservation task, collision, or counterfactual reasoning, they are even below the pure random baseline. For counterfactuals, our qualitative investigation shows that Flamingo tends to latch on the visible elements in the video, failing to imagine the alternate reality that counterfactual questions require; e.g. for videos where a person writes some letters on the paper, the (counterfactual) question posed is: What would be the order of the written letters if the person had written them in reverse order?. Flamingo often selects the written order as correct. Interestingly, the larger versions of the model (due to larger language branches) seem to fare worse overall, which might point to overfitting issues. However, we leave an in-depth analysis for future work. Fine-tuning SeViLA leads to better results compared to all-shot frequency baseline, but mainly in the Semantics area (Fig. [3]). SeViLA's 0-shot/ fine-tuned scores on Perception Test (Fig. [3]) are significantly lower than on NExTQA benchmark [5]: 46.2 vs 63.6 for zero-shot, and 62.0 vs 73.8 on fine-tuned. We attribute this to the diversity of skills and the hard negative options included in the Perception Test.
Grounded-vQA: As no existing model can perform this task, we created a baseline by running MDETR [34] on the middle frame and then tracking the predictions using the Stark [52] object tracker. The performance is poor; see Table A7 and Figure A6. The failures are caused mainly by poor detection results – since the tasks are temporal in nature, extracting seed boxes from the middle frame is not sufficient to solve the tasks, calling for models capable of dealing with both spatial and temporal dimensions.
Figure 2: 0-shot human baseline compared to 8-shot Flamingo-3B, 0-shot and fine-tuned SeViLA, and random baseline on the validation set. In 0-shot and 8-shot regimes, both Flamingo and SeViLA are far from the 0-shot human baseline. SeViLA fine-tuning improves results to some extent, but the gap to human performance is still substantial.
Figure 3: Performance on the validation set across skills for the 0-shot and fine-tuned SeViLA compared to frequency dummy baseline. The black dashed line indicates the random baseline.
We propose a diagnostic benchmark for multimodal models, to probe for memory, abstraction, physics, and semantic capabilities, across visual, audio, and text modalities, using real-world videos purposefully designed and filmed to show interesting perceptual situations. Solving the tasks requires different types of reasoning: descriptive, explanatory, predictive, and counterfactual. The videos are densely labeled with six types of annotations (objects and point tracks, action and sound segments, multiple-choice and grounded video question-answers). We are open-sourcing the videos and the annotations in the train and validation splits, together with per-task baseline results and evaluation code. A challenge server is available to evaluate models on the held-out test split. In principle, any model can be evaluated on our benchmark, either in a zero/few-shot setting or by fine-tuning on our limited train split. An ideal perception model would be able to perform all the tasks in our benchmark. Our results suggest that state-of-the-art zero-shot or few-shot video-language models are not able to outperform a dummy frequency-based baseline, whereas humans in the same setting are nearly perfect. We discuss limitations, and ethical and societal aspects in the appendix. We hope that our work will contribute to understanding models' limitations (through direct evaluation and interpretability analysis supported by the different types of annotations) and narrowing down areas of improvement to guide research towards general perception models.
We are grateful to Luis Piloto, Kenneth Marino, Luyu Wang, Felix Hill, Martin Chadwick, Lucy Campbell-Gillingham, Boxi Wu, Drew Jaegle, Pauline Luc, Marianne Monteiro, Anna Bulanova, Radu Isac, Muqthar Mohammad, Vijay Vibha Tumala, Mahesh Maddinala, Yiwen Luo, Alina Kuznetsova, Aida Nematzadeh, Lisa Anne Hendricks, Aishwarya Agrawal, Nando de Freitas, Matt Botvinick, Shane Legg, and Relja Arandjelovic for providing insightful input on this project.
[45] Madeline Chantry Schiappa, Shruti Vyas, Hamid Palangi, Yogesh S Rawat, and Vibhav Vineet. Robustness analysis of video-language models against visual and language perturbations. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?id=A79jAS4MeW9
[46] Gunnar A. Sigurdsson, Gül Varol, X. Wang, Ali Farhadi, Ivan Laptev, and Abhinav Kumar Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. ArXiv, abs/1604.01753, 2016.
[47] Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple-object tracking with transformer. arXiv preprint arXiv: 2012.15460, 2020.
[48] Andrea Tacchetti, Leyla Isik, and Tomaso Poggio. Invariant Action Recognition Dataset, 2019. URL https://doi.org/10.7910/DVN/DMTOPG.
[49] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks, 2022. URL https://arxiv.org/abs/2208.10442
[50] Zhongdao Wang, Hengshuang Zhao, Ya-Li Li, Shengjin Wang, Philip Torr, and Luca Bertinetto. Do different tracking tasks require different appearance models? Advances in Neural Information Processing Systems, 34:726-738, 2021.
[51] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777–9786, June 2021.
[52] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10428-10437, 2021.
[53] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview Transformers for Video Recognition. arXiv e-prints, art. arXiv:2201.04288, January 2022.
[54] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=HkxYzANYDB
[55] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. arXiv preprint arXiv:2305.06988, 2023.
[56] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, pages 9127–9134, 2019.
[57] Chenlin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. In European Conference on Computer Vision, 2022.