Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Authors: Viorica Patraucean
Lucas Smaira
Ankush Gupta
Adria Recasens Continente
Larisa Markeeva
Dylan Sunil Banarse
Skanda Koppula
Joseph Heyward
Mateusz Malinowski
Yi Yang
Carl Doersch
Tatiana Matejovicova
Yury Sulsky
Antoine Miech
Alexandre Fréchette
Hanna Klimczak
Raphael Koster
Junlin Zhang
Stephanie Winkler
Yusuf Aytar
Simon Osindero
Dima Damen
Andrew Zisserman
Joao Carreira
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: evaluation
multimodal video models
perception
reasoning
diagnostic
memory
abstraction
physics
semantics
TL;DR: Comprehensive evaluation benchmark for multimodal video models focused on skills (memory, abstraction, physics, semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual)

Abstract: We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, BEiT-3, or GPT-4). Compared to existing benchmarks that focus on computational tasks (e.g. classification, detection or tracking), the Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities, to provide a comprehensive and efficient evaluation tool. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime. For these purposes, the Perception Test introduces 11.6k real-world videos, 23s average length, designed to show perceptually interesting situations, filmed by around 100 participants worldwide. The videos are densely annotated with six types of labels (multiple-choice and grounded video question-answers, object and point tracks, temporal action and sound segments), enabling both language and non-language evaluations. The fine-tuning and validation splits of the benchmark are publicly available (CC-BY license), in addition to a challenge server with a held-out test split. Human baseline results compared to state-of-the-art video QA models show a substantial gap in performance (91.4% vs 46.2%), suggesting that there is significant room for improvement in multimodal video understanding.
Dataset, baselines code, and challenge server are available at https://github.com/deepmind/perception_test

Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Viorica Päträucean'' DeepMind
Lucas Smaira DeepMind
Ankush Gupta DeepMind
Adrià Recasens Continente DeepMind
Larisa Markeeva Deep
Dylan Banarse
Skanda Koppula
Joseph Heyward
DeepMind
DeepMind
DeepMind
Mateusz Malinowski
Yi Yang DeepMind
Carl Doersch
Tatiana Matejovicova DeepMind
Yury Sulsky DeepMind
DeepMind
DeepMind
Antoine Miech DeepMind
Alex Frechette
Hanna Klimezak
Raphael Koster DeepMind
Junlin Zhang DeepMind
DeepMind
DeepMind
Stephanie Winkler
Yusuf Aytar DeepMind
Simon Osindero DeepMind
Dima Damen University of Bristol
DeepMind
Andrew Zisserman University of Oxford, DeepMind
João Carreira'
DeepMind
Abstract
We propose novel multimodal video benchmark the Perception Test to eval- uate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, GPT-4). Compared to existing benchmarks that focus computational tasks (e.g. classification, detection or tracking), the Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities, provide comprehensive and efficient evaluation tool. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot few-shot or limited finetuning regime. For these purposes, the Perception Test intro- duces 1 1.6k real-world videos, 23s average length, designed to show perceptually interesting situations, filmed by around 100 participants worldwide. The videos are densely annotated with six types of labels (multiplecchoice and grounded video question-anse anniiess,, object and point tracks, temporal action and sound segments), enabling both language and non-language evaluations. The fine-tuning and valida- tion splits of the benchmark are publicly available (CC-BY license), in addition to challenge server with held-out test split. Human baseline results compared to state-of-the-art video QA models show a substantial gap in performance (91.4% vs 46.2%), suggesting that there is significant room for improvement in multimodal video understanding. Dataset, baselines code, and challenge server are available at https://gittult com hoom///deppiind/perrgtoooooeoee
"Corresponding author: viorica@google.com com, sshared senior contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Introduction
Significant progress in multimodal models has been made recently due to large-scale training on multimodal data. Models like Flamingo 41. SeViLA [55], BEiT-3 49]. GPT-4 [ show remarkable versatility, dealing with diverse data sources and tackling new tasks by observing only handful of examples. This is major departure from specialised models that are typical in computer vision, e-g- image or action classifiers
SX 20) object detectors [T3,, or object trackers
opening up the path
towards general perception and reasoning models.
Benchmarking these models a robust and efficient way is key to expanding their capabilities, by allowing researchers to rank model design and training choices, and identify areas for improvement. Many perception-relatee
benchmarks exist, for example Imagenet for image classification [[7], Kinetics for video action recognition 36) Audioset for audio event classification (24). TAO for object tracking [6]. or VQA for image question-answering answering 28), to name only a few. While these benchmarks have led to amazing progress, they all target restricted aspects of perception, focusing on specific computational tasks: e.g. image benchmarks discard the temporal dimension, visual question- answering tends to focus only high-level semantic scene understanding, and object tracking focuses on lower-level, texture-based cues. Gluing several datasets together 39 45 to benchmark more general models (as is done in Flamingo, SeViLA, BEiT-3, or GPT-4) improves coverage, but results in an expensive evaluation procedure that still misses important general perception abilities, e.g. physics understanding or memory. Few existing benchmarks even define tasks over both audio and visual modalities [29]. much less more complex combinations of modalities and tasks. Furthermore, most prior work provides large training sets and thus benchmark models for in-dataset capabilities.
In this work, we propose the Perception Test - a benchmark formed of purposefully designed, filmed, and annotated real-world videos that aims comprehensively assess the capabilities of multimodal perception models across different skill areas (Memory. Abstraction, Physics, Semantics), types of reasoning [54] (descriptive, explanatory, predictive, and counterfactual) and modalities (video, audio, text). Our benchmark draws inspiration from diagnostic synthetic datasets like CATER (25) or CLEVRER (54), behavioral tests like the Visual Turing Test 41 23
experiments in developmental 31, and motor-free perception screening tests used for children or adults 42
psychology
1 6
To avoid benchmark overfitting, propose to benchmark any representation or model, pre-trained with any external dataset or task, of any scale available. The Perception Test itself contains small training set that can optionally be used for fine-tuning task decoders or prompting the model, and the rest is used for evaluation (public validation and held out test sets). In this regime, can more robustly assess the transfer abilities of these models, such that improvement on the benchmark can more reliably predict generalisation to real-world operation.
pnnerralisatinnfoccsse evaluation regime. We aim
The dataset contains 11.6K real-world videos, densely annotated with 190K object and 8.6K point tracks, 73.5K temporal action segments, 137K temporal sound segments, 38K multiple-choice video question-ansee answerr (mc-vQA) pairs and 6K grounded video guestion-answer (g-vQA) pairs, enabling both language and non-language evaluations, to ensure thorough assessment; see Figure I and Table 3 Having multiple types of annotations per video is useful also for analysis and explainability purposes, the correlations between successes and failures across tasks may uncover model biases.
We open-source the videos and annotations in the training and validation splits. An evaluation server is made available together with the videos from the held-out test split. Since currently there no model that can tackle all the evaluation tasks in our benchmark, we provide baseline results for per-task models: object tracking, point tracking, temporal action localisation, temporal sound localisation, multiple-choice video question-answerinn -nswering, and grounded video guestion-answerte For the mc-vQA task, the performance is mapped across skill areas (memory, abstraction, physics, semantics), and types of reasoning (descriptive, explanatory, predictive, counterfactual) to obtain a comprehensive diagnostics report.
In the next section (section [], we discuss related work in more detail, highlighting what sets the Perception Test apart in the rich landscape of multimodal benchmarks. In sections 3 and 4 we describe the videos and annotations participants involved in filming the videos. In section 5 we introduce the computational tasks enabled
the Perception Test, with details about the diversity of
by these annotations, together with evaluation metries and baselines, including a human baseline. We conclude with summary and directions of future work in section X

---

EXAMPLE1
Sounds Hiting Celons vabiil
Dropping
Moving sommothing ing around tylosa)
Puling something on top aomothing (mar book!
Actions
Purting sometting someiinng
O
Mulliple-cheice video QA Physies, Predictive
contguvattnn Vtery arbbie
The confgwaaton viely usstzble.
Question: confgurccion objects ofter placing the objess?
c) One connee juigo the stoMyy of cho contguration
EXAMPLE 2
Input
Ares: Memory, finssoring Explanetery
Question: What changed on the toole whie the carnero was looking eway?
Options:
a) The mobile and clip suopped positios.
b) The bettle and warch were removed and and mobble mere odkeec
c) The mobile in arised
wns remouved
Aneue Memery. Resscriing:: Descriptive
Question: trook the objects that were added to the table whil comero was looking away.
Grounded video QA annotations
EXAMPLE
Input
Multiple-choice Counterfactuel person objects backpock obyect objects leptop
EXAMPLE 4
EXAMPLE 5
choice
ple-choice
Reasoning: Explanatory
Abstroction Reasoning: Descriptive onaa persen puts on the table have the some colow?
Question:
Whot octions porson complete why?
Queation
Options:
a) The persor noobog next the massad traide the cup
Options:
n) the person redd to actter laseet becouue they didiet we conteine wnough pour
al b) SE c)
contalncr
Figure 1: The Perception Test contains types of annotations (ohject point tracks, action & sound segments, multiple-choice videoQA and grounded videoQA) and tasks spanning skill areas (Memory, Ashtraction, Physies, Semantics, and types of reasoning (Descriptive, Explanatory, Predictive, Counterfactual). See the presentation video htttps://githut: cactuuu.coom/// wwt.cooo///eeeeeeeeeeeee wtweeeeiinddpeeeepptteee
for examples.
Related work
large number of perception-related benchmarks exist the literature, covering various compu- tational tasks or modalities. We focus the discussion here video benchmarks and highlight the differences between the Perception Test and prior work, in terms data collection process, covered modalities, and available annotations and tasks.

---

Dataset
Source
Skills
videos
Dens
L(s)
Charades
C.R
10,000
14
30
SSv2
C.R
AS
108,499
4
Ego4D-v2
MS
205,534
9"
4921
CLEVRER"
C,Y
P
60,000
N/A
5
Perception Test
C.R MAPS
11,620
761
23
Table 1: Characteristics of different datasets compared to the Perception Test. Dataset sources:
Scripted (C), Real (R) and Synthetic (Y). Skill areas: Memory (M), Abstraction (A), Physics (P),
Semantics (S). Dens: Average number of annotations per video. L: Average video length in seconds. 'number of annotated clips, 'reported for hand-objects subset with the highest density of annotations, 'reported for ELM NLQ subset with highest average clip length. "; Annotations are extracted directly from the simulator.
Existing real-world benchmarks rely on one of the following data sources: (i) Videos collected from the web repositories like Youtube, e.g. Kinetics (301, ActivityNet 19, VGGSound [T]. HVU [
ActivityNet-QU [56]. tGIFQA [3]; (ii) Videos collected on demand, filmed by volunteers doing
arbitrary activities in indoor or outdoor scenes, e.g. SPIC-KITCHENS [1.. Ego4D
(2 (iii) Videos
collected on demand, filmed by crowd-sourced participants doing actions described in pre-defined
scripts, mostly in indoor scenes, e.g. Charades [i)
v2 (SSv2) 26 Invariably,
all real-world benchmarks use crowd-sourced annotations to enable various computational tasks like action classification, object detection, or video captioning, name only few.
Annotating publicly available videos is useful for training. However, using this approach for general perception evaluation has multiple drawbacks. Large quantities of data would need to be amassed and carefully filtered and annotated to accumulate (statistically) sufficiently diverse samples showing perceptually interesting situations that require skills like memory, abstraction, physics, and semantics understanding. In addition, some types of data are simply not available, e.g. situations showing incorrect execution of simple tasks like tying shoe laces. As we aim assess more diverse skills, chose to design video scripts that show perceptually interesting and diverse situations and film these with crowd-sourced participants from different places in the world ensure diversity of video content and appearance. Different from Charades where the scripts were designed by crowd-sourced workers, our scripts are designed by our research team, similar did not aim to obtain an exhaustive coverage of simple actions like in SSv2. Instead, we designed more complex scripts to probe for more advanced reasoning skills beyond action classification.
Gomething-something (v2). However,
few research works have highlighted the need for robust diagnostics benchmarks, e.g. CATER (55) CLEVRER (54 IntPhys 44 Physion 7. Their authors developed synthetic datasets to evaluate in a more systematic way, across different levels of difficulty, models' abilities to reason about intuitive physics (object collisions, motion, object permanence). We share the same motivation of creating diagnostic test, and we aim to cover aspects related to memory, abstraction, intuitive physics, and semantics, using real-world videos. To achieve this, in addition designing the video scripts, our team also designed the questions for each script type for the high-level tasks (mc-vQA and g-vQA); the answers per video were provided by crowd-sourced annotators. Given that our videos filmed in real world scenes using common household items, the distributions of objects, actions, and sounds in our benchmark have significant overlap with standard computer vision datasets (e.g. 99.01% % of the words in our benchmark also appear in VQA22 (27), hence the domain gap between the Perception Test and existing large-scale training datasets should be minimal.
Table I summarises the characteristics of the Perception Test compared to previous efforts. It can be observed that the Perception Test has better coverage of skill areas and higher density of annotations Size-wise, it is comparable to Charades, but smaller than Ego4D or S$v2. We emphasise that the Perception Test is not designed to be large-scale training dataset. Instead, it is an evaluation benchmark, with limited fine-tuning or prompting data, meant to assess the transfer capabilities of models.
We count every labeled box, point, temporal segment, or question separate annotation

---

3 Videos in the Perception Test
Inspired by how human perception screening tests are carefully designed by experts in developmental psychology or medicine (e.g. [2] we designed video scripts and tasks to diagnose the perception skills of our models.
Our goal was not to obtain an exhaustive coverage of activities or types of scenes. Instead, we selected four areas Memory, Abstraction, Physics, Semantics - within which several skills should be tested (see Table first column) through tasks that require different types of reasoning: descriptive, explanatory. predictive, or counterfactual [54 The skills selection took into account blind spots of existing benchmarks, weaknesses of current models, and aspects that are important for real-world scene understanding.
Scripts design:
We then created scripts describing simple situations games that can be easily performed by any one person non-professional actor) using the items available in a regular household, or items that can be easily crafted if not available (e.g. letters or geometric shapes crafted from paper or cardboard). Each script consists of a brief description of the scene, followed by description of the actions to be performed, together with specification of the camera placement (static camera one viewpoint; static camera 2 viewpoints; static camera and moving camera). To enhance content diversity, each script had considerable room for variability in the number of objects be included in the scene or types of actions to be performed, or order of actions.
We prioritised situations where we can test high-level concepts like memory through low-level tasks like object tracking and the other way around: low-level physics understanding probed through high-level tasks like prestion-answering In addition, we included in each script elements that could make the situations more interesting and challenging. For example, in cooking scripts (e.g. making tea, making salad), we added distractor actions 466. i.e. actions not relevant for making tea and that have no impact on the outcome of the making tea sequence, like clapping hands, or hitting a kettle with a spoon; this allows probing for understanding of causal relations between actions. We also included distractor objects in the scene description, i.c. objects that are not relevant for the current script, but which are relevant for other scripts, like tomatoes present on the table during the make tea activity 48.. For all the scripts, we also asked participants to include in the scene some adversarial configurations of objects e.g. shoe on the table. This allows us to probe models for understanding of spatial relations of objects when the language biases are not valid. Finally, some of the script variations include adversarial actions (26), i.e. incorrectly executed actions. For example, when making the tea, all the steps are done normally, but one is incorrectly executed, like pouring water from an empty kettle. this way, we can probe for understanding of task completion, in a more complex setup than the adversarial action classification used in SSv2 dataset [26)
Table and Figure I show examples of situations included in the scripts to probe for different skills in the different areas and types of reasoning. Note that the vidcos associated with a script allow defining tasks and questions across multiple skill areas. All-in-all, we designed 37 scripts, each with 2-5 variations, to obtain a diverse dataset. Having multiple variations per script allows us to ask the exact same question with the same set of options, and the correct answer depends on the specific script variation in this way, we can avoid language biases in questions that give away the answer [77] Examples of videos included in the dataset can be found in the presentation video at https: gi thub. com wtt./oooo/eeeeeeeeeeo dttp:/ideeeeddeee////ee/ httpm//ddapppiiddeeeeeppppiiooii testt
Video filming: Ensuring diversity of participants and scenes depicted in the videos consideration when developing the benchmark. Using a crowdsourcing pool, we selected around 100 participants from differrnt countries of different ethnicity and gender and aimed to have diverse representation within each video script. We include in the appendix details about the self-reported demographics of participants. Each script variation was filmed by at least dozen of different participants, using most often a mobile-phone camera, resulting in high-resolution audio-viste assets. For scripts to be filmed from two different viewpoints, the recording was most often done sequentially by repeating the script; a few participants recorded simultaneously using two filming devices. About 15% of the videos were filmed with a moving camera. Most of the videos were filmed indoors in the living room or kitchen, with small number being filmed in the bathroom or outdoors (about 1%). Most of the activities are performed on a tabletop, but some are also performed on the floor or chair. To avoid privacy concerns, we instructed the participants to not record their faces or
critical
5

---

(Skill Area Skill
Example of situations and questions or tasks
(M) Visual discrimination
Ohjects are shown in front the camera, with some shown more than once. Task:
Detect which objects were shown multiple times.
(M) Change detection
The camera is filming a table, then looks away for a few seconds, then looks back at the table. Some changes may have occurred. Task: Explain what changed.
(M) Sequencing
Objects are put in a backpack. Task: List their order.
(M) Event recall
A person indicates region on the table with the hand, then puts objects inside and outside the region. Task: List the objects put inside the region.
(A) Object, action
&
A person turns lamp on and off. Task: Count the number of times the illumination
event counting
changed in the scene.
(A) Feature matching
A person puts wooden letters on the table. Task: Which letters have the same colour?
(A) Pattern discovery
Geometric shapes are shown in pattern. Task: What shape will be shown next? person puts multiple cups all facing upwards and one facing downwards. Task:
(A) Pattern breaking
Indicate the object that breaks the pattern.
(P) Object permanence
A person plays a cups-game with 3-4 cups by hiding a small object under one of the
cups, then shuffles the cups. Task: Predict where is the hidden object after shuffling.
(P) Spatial relations &
A person puts bookmark in a book, then puts the
or another book in a
containment
backpack. Task: Where is the bookmark at the end?
(P) Object attributes
A person writes on a piece of paper. Task: Is the paper lined or plain?
(P) Motion & occluded in-
A person moves an occluder object in front of small object, sometimes moving also the small (oceluded) object. Task: Was the small object moved?
teractions
(P) Solidity & collisions
A person launches objects against blocker object, sometimes removing the blocker. Task: Does the object fall off the table?
(P) Conservation
person pours an equal amount of water in 2 identical glasses, then pours all part
of the water from one glass a taller or wider glass. Task: How much water is the last glass?
(P) Stability
A person puts objects on top of each other in a stable or unstable configuration. Task: Predict if the configuration will he stable after placing the last object.
(S) Distractor actions &
A person makes tea, and does also some distractor actions unrelated making tea, e.g. rotating a knife. Task: Identify the distractor action(s).
objects
(S) Task complction & ad-
A person ties shoe laces, but sometimes pretends to tie, or ties the lace of one shoe to the lace of the other shoe, Task: Detect if the action is done correctly.
versarial actions
(S) Object & part recogni-
person conceals small object one of their hands, then shufiles the hands. Task: Identify in which hand is the object held.
tion
(S) Action & sound recog-
All scripts. Task: Detect the actions and sounds in the video from pre-defined list.
nition
(S) Place recognition
All scripts. Task:
Detect where is the action taking place.
(S) State recognition
A person uses an electric device. Task: Indicate if the device is on.
(S) General knowledge &
Some objects are shown to the camera, some multiple times. Task: Given list of arbitrary statements or word puzzles, some requiring general knowledge to solve, select the statement that contains a reference to the second distinct object shown.
Language
Table 2: Examples of scripts probing for different skills in the four areas in the Perception Test: (M): Abstttatac,,,,, (P):Physics, (S):Semantics.
voices. This is not a limitation of the dataset since the focus in our scripts is on object interactions.
The participants gave their consent for the data to be used, published, and stored for perpetuity.
Splits: The Perception Test contains 1 11609 videos (with audio), 23s average length. is divided into a small training split (2184 videos, 20% of the data) that can be used for fine-tuning or prompting, a validation split (5900 videos, ~ 50% of the data), and a held-out test split (3525 videos, ~ 30% of the data) available through the evaluation server. We optimised to obtain good balance across all annotation types and camera motions across the splits; see section 5 in the appendix.
4 Annotations in the Perception Test
We annotate these videos with six types of annotations to cover low-level and high-level aspects, both spatial and temporal, and enable language and non-language evaluations: object and point tracks, temporal action and sound segments, multiple-choice and grounded video question-answers, We include summary of the number of annotations in Table and visualisations in Figure 0

---

Annotation type
classes
# annot
# videos
Rate (fps)
Objects tracks
5101
189940
11609
Point tracks
NA
8647
145
30
Action segments
63
73503
11353
30
Sound segments
16
137128
11433
30
mc-vQA
132
38060
10361
NA
g-VQA
34
6086
3063
Area
# vidcoQA
Reasoning
# Vi
Memory
7256 (36)
Descriptive
31536
(106)
Abstraction
12737 (58)
Explanatory
4513
(14)
Physics
23741 (80)
Predictive
1278
(7)
Semantics
24965 (82)
Counterfactual
733
(5)
Table 3: Top: Annotations in the Perception Test. Each object or point track contains frame-level annotations certain frame rute, e-g. each point is annotated on every frame, at 30 fps. Action and sound segments are annotated at the original video frame rate. # classes refers to the number of unique object names for object tracks and the number of unique questions for multiple-choie videoQA (mc-vQA) and grounded videoQA (g-vQA). Bottom: Number of videoQA pairs and (unique questions) per area and type of reasoning. Note that one question may be counted in multiple areas if it tests more than one skill. Each question is assigned a unique type of reasoning.
Object tracks: Object tracks represent the root annotation of our benchmark. All the other annota- tions, except for multiple-choice vQA, are linked or grounded into object tracks. In the annotation process, we instructed annotators to focus on the objects that the person interacts with and the objects that are in the immediate vicinity of the area where the person is performing actions, which act as distractor objects. We annotated boxes at 1fps throughout the video, which gives a good trade-off between density of annotations and annotation cost. When the objects are occluded, the annotators marked an approximate position of the boxes. Some ambiguous classes still remain, like liquids being poured or objects being torn. The object names were defined from open vocabulary. The annotators typically included object attributes as well (colour, material), resulting in a large number of unique names. A list of the most frequent words (object or attributes) is included in the appendix, Fig. A (left), together with the distribution of object tracks into various categories, e.g. objects involved in actions or sounds correlated with camera motion (Table
Cups-game subset: We isolate the videos corresponding to the cups-game scripts, as they can be an interesting subset for probing object trackers' abilities to reason about motion, object permanence, or occluded interactions when different factors may influence the difficulty of the task, e.g. identical vs non-identical objects used in the game, transparent vs non- transsaapp objects, or number of objects used. This subset contains 598 videos, with 483 videos where the cups are identical, and 113 videos where the cups are transparent. Most of the videos have 3 cups (451 videos), 132 videos have 2 cups, and 34 videos have 4 cups. We also provide a visibility mask for each video showing when the hidden object is occluded.
Point tracks: Although object tracks based on bounding boxes allow probing some physical proper- ties of objects, such as object permanence, solidity, and coarse motion, they do not fully describe articulated or non-rigid objects, thin objects that are not axis-aligned, or out-of-plane rotation. A better understanding of physical interactions arises if we can track how object surfaces move and deform over time. To this end, we annotate point tracks on object surfaces following the protocol of Tid-Vid [19. Annotators were instructed to select points spanning all the different parts of the objects labelled in the object tracking task. Points that are occluded are simply marked as occluded and not tracked. For translucent objects (e.g. glass cups), we only consider points to be "visible" if they belong to the surface closest to the camera. The annotated points are dense in time (30fps). Table A2 in the appendix gives the distribution of points that are moving static, as well as those on videos with moving cameras.
Action segments with action-relevant objects: To capture temporal understanding and enable grounding over time, we annotate the videos with temporal segments belonging to a fixed set of templated labels, e.g. putting something into something, similar to 26). These are associated with action-relevan object tracks, i.e. objects involved in the action. The action boundaries are defined based on contact with action-relevant objects. For instance, when a person puts sugar in tea, the putting something into something action starts when the person picks up the spoon and ends when the person puts down the spoon. If, after putting the sugar, the person starts stirring with the same spoon,
7

---

Task
Output
Metric
Baseline
Score
Object tracking
box track point track
Avg. IoU
SiamFC
0.67
Point tracking
Avg. Jaccard
TAP-Net 19
0.40
Temporal action localisation
list of action segments list of sound segments
mAP
ActionFormer
[57
0.16
Temporal sound localisation
mAP
ActionFormer
[57
0.15
multiple-choice videoQA
answer (1 out of 3) list of box tracks
top-I accuracy
SeViLAA
0.46
grounded videoQA
HOTA 440 MDETR [34] +Stark
52
0.10
Table 4: Computational tasks and lop-performing baselines in the Perception Test: the model receives a video with audio, plus a task-specific input (e.g. the coordinates of a bounding box for the object tracking task), and produces task-specific prediction, evaluated using dedicated metrics.
this defines new segment as the type of action changed. The frequency of actions across the entire dataset is included in the appendix, Fig. (right).
Sound segments with sound-relevant objects: Similarly to the action segment annotations but applied to the audio modality, we collect sound segment annotations grounded in object tracks. By watching the video and listening to the audio, the annotators define temporal sound segments and label them from a list of 16 audio segment labels. For each sound, the annotators also identify the object (or objects) involved in making the sound, or specify that these are out of the camera's 's field of view. For example, if an object is placed on the table making an audible sound, then both the object track and the table track are associated with the sound segment. The frequency of sounds across the entire dataset included in the appendix, Fig. A2
Question-answers for video-level reasoning: Different from the existing VQA datasets, which rely on crowd-sourced questions and answers, our team designed the questions per script to cover different types of reasoning [54]: descriptive, explanatory, predictive, counterfactual, and to cover aspects that are important for operating in the real world, e.g. understanding task completion, detecting changes, and so on. The answers for all the questions per video were provided by crowd-sourced participants. As we are interested in non-ambiguous evaluation, we favour the multiple-choice setup over the apen-language answer setup. To define challenging negative options, we partly relied on human annotators, partly sampling from the correct answers of other videos in the same type of script. Table B bottom and Figure A3 in the appendix show the distribution of question-vide -vidoo pairs into perception skills, skill areas, and type of reasoning.
question-answers with answer-relevant objects: As another way to connect high-level and low- level scene understanding capabilities, we define questions tasks in language form, with answers given as object tracks. Similar to the multiple-choiee question-answer above, our team defined the questions, and human raters selected the answers from the existing object tracks. The grounded questions are associated with skill areas and types of reasoning.
5 Computational tasks and baseline results in the Perception Test
Computational tasks: We defined six computational tasks based on the annotations available in the Perception Test. We summarise in Table 3 the task definitions (outputs, metrics) and the performance of top-performing baselines. It can be observed that the Perception Test combines lower-level dense prediction tasks like object and point tracking, whose outputs are box and point trajectories, with higher-level tasks like video question answering. For all the tasks, the video and audio are available as inputs, together with task specification where applicable, e.g. the coordinates of box to track for object tracking, or language question and options for multiple-choice videoQA. More details about the task definitions are included in the appendix. Note that many other computational tasks can be defined based on the available annotations, e.g. grounded temporal action/sound localisation.
Ideally, a single model should be able to perform all the tasks in the Perception Test. Since such a model is not available in the literature, we include results obtained with per-task baselines on the validation split for all the six tasks in the Perception Test; see Table g for summary of top perporrffrmng baselines and their average performance, and the appendix for more details. When selecting these baselines, we favoured strong- pernorrirfoominggggo models that can be evaluated in a zero-shot or few-shot setting, as our focus is on generalisation. However, for action and sound localisation, such models do not exist in the literature, so fine-tuning on our set of classes was necessary. For the mc-vQA task, we also provide human baseline and fine-tuned evaluation to further assess the difficulty of the dataset for humans and for SOTA video- languae models, respectively.
Baselines:
8

---

The overall performance of SiamFC 8 (UniTrack (50 implementation) on our benchmark confirms the findings from |2I that simple Siamese trackers are better when probed zero-shot than more complex recent trackers, e.g. Stark tracker 52 obtains 0.56 mean IoU on the Perception Test vs 0.67 for SiamFC. Even for SiamFC, the tracking performance drops when the camera and/or the objects are moving. The results for the different categories of objects (involved in actions or in sounds, etc.) are included in Table A3 aggregated based on camera motion.
Object tracking:
The performance of our baseline TAP-Net compared to the performance reported by the authors on the Kinetics dataset [366] (0.466 vs 0.401); see detailed results in Table A4 We attribute this drop in performance mainly to the increased video length our benchmark (23s Perception Test compared to 10s in Kinetics).
Point tracking:
19 is bit lower on the Perception Test
Action localisation: The confusion matrix for our fine-tuned baseline ActionFormer shows that the model struggles mostly with rare action classes that are confused with more frequent ones, and it also confuses pretend actions with their non-pretend versions, e.g. ironing something vs pretending to iron something. Using multimodal inputs does not increase the performance signifi- cantly, as summarised in Table A3 top. Overall, ActionFormer's ' performance on our benchmark is lower compared other benchmarks (15.56 mAP on Perception Test vs 22.7 mAP on EPIC- Kitchens [4),, most likely due to the presence of adversarial actions and our limited training set. We hope to see in the near future models that can handle open-vocabulary action classes (similarly to open- vocabulcay object detection (35 D. so that fine-tuning is no longer necessary.
(Fig. A4
Sound localisation: We adapted the same ActionFormer model [57 to perform the localisation task in the audio modality. The best performance is obtained when features from both video and audio bottom.
modalities are used as input; see Table A5
We report results for two strong recent video language models: Flamingo [ in zero-shot and few-shot setups, and SeViLA [55 in zero-shot and fine-tuned regimes. We also include a dummy rrequency-based baseline and a human baseline. For the frequency baseline, given that each question-options pair is defined over multiple videos, we keep as answer the option that is most frequently the correct answer in the training set. One can also compute this baseline on random subset of training examples for each question, see Table A6 to obtain fairer dummy baseline for models using few-shot evaluation.
Multiple-choice vQA:
Human baseline. We ran small study for the mc-vQA task with human participants. We used 126 questions from the dataset, with one video per question selected at random. We recruited 30 crowd-sourced participants (half male, half female, with advanced English skills), different from the raters annotating the videos. Each participant answered a subset of 42 questions, resulting in 10 answers per question. The performance per area and type of reasoning is detailed in Figure The overall average accuracy was 91.4%. The mistakes occurred in situations difficult to judge from the given viewpoint, e.g. if a configuration of objects would be stable (without seeing the end of the video), or in edge cases where humans overlooked details happening very early on in the video. is worth noting that the participants did not require any training. which is similar to a zero-shot setup. The median time spent to answer 42 questions was 30 minutes.
It can be observed that both Flamingo and ViLA are far from human performance when evaluated O-shot or few-shot and cannot outperform the 8-shot dummy frequency baseline; see Figures 2 3 AS and Table A6 On many skills in the Memory, Physics, and Abstraction areas, their performance is below the 8-shot frequency dummy baseline, and in a few cases, e.g. (Piaget) conservation task, collision, counterfactuals, our qualitative investigation shows that Flamingo tends to latch on the visible elements in the video, failing to imagine the alternate reality that counterfactual questions require; e.g. for videos where a person writes some letters on the paper, the (counterfactual) question posed is: What would be the order of the written letters the person had written them in reverse order?. Flamingo often selects the written order as correct. Interestingly, the larger versions of the model (due to larger language branches) seem to fare worse overall, which might point to overfitting issues. However, we leave an in-depth analysis for future work. Fine-tuning SeViLA leads to better results compared to all-shot frequency baseline, but mainly in the Semantics area (Fig. 3 SeViLA's O-shot/fine-tunel scores on Perception Test (Fig. 3 are significantly lower than on NEx QA benchmark [5T]: 46.2 vs 63.6 for zero-shot, and 62.0 vs 73.8 on fine-tuned. We attribute this to the diversity of skills and the hard negative options included in the Perception Test.
counterfactual reasoning, they are even below the pure random baseline. For

---

Qro As no existing model can perform this task, we created baseline by running MDETR [4 on the middle frame and then tracking the predictions using the Stark [52] object tracker. The performance is poor; see Table A7 and Figure A6 The failures are caused mainly by poor detection results since the tasks are temporal in nature, extracting seed boxes from the middle frame is not sufficient to solve the tasks, calling for models capable of dealing with both spatial and temporal dimensions.
Areas
Memory 10
Reasoning
Descriptive
Counterfactual
Abstraction
D-shot human baseline
B-shot Flamingo
D-shot SeViLA
Fine- tuned SeVILA
Physics
Random
Predictive
Figure 2: O-shot human baseline compared to 8-shot Flamingo-33 O-shot and fine-tuned ViL, and random baseline on the validation set. In O-shot and 8-shot regimes, both Flamingo and SeViLA are far from the O-shot human baseline. SeViLA fine-tuning improves results to some extent, but the gap to human performance is still substantial
Morooy Asstrcciee. Plysis Sorerixs
Al-chol funetura Seuta
change detection
event recall
sequencing
visua Ulscrimnaaiin
action counting
event counting
feature matching
object counting
pattern breaking
pattern eiscovery
collis on
conservation
containment
material
motion
occlusion
object attriibutes
object object pormmnneen
quantity
coldity
pattie relations
sction recogning adversaria ial action colour aecoggaann distractor action
disur sclor objest
general knowledge
language
object recognition
part recognition on
place recognition
recognition
task completion
Figure 3: Performance on the validation set across skills for the O-shot and fine-tuned SeVA
compared to frequency dummy baseline. The black dashed line indicates the random baseline.
6 Conclusion
We propose diagnostic benchmark for multimodal models, probe for memory, abstraction, physics, and semantic capabilities, across visual, audio, and text modalities, using real-world videos purposefully designed and filmed show interesting perceptual situations. Solving the tasks requires different types of reasoning: descriptive, explanatory, predictive, and counterfactual. The videos are densely labeled with six types of annotations (objects and point tracks, action and sound segments, multiple-choice and grounded video question-answers). We are open-sourcing the videos and the annotations in the train and validation splits, together with per-task baseline results and evaluation code. A challenge server is available to evaluate models on the held-out test split. In principle, any model can be evaluated on our benchmark, either in a zero/few-shot setting or by fine-tuning on our limited train split. An ideal perception model would be able to perform all the tasks in our benchmark. Our results suggest that state-of ta-te-of-the-an zero-shot or few-shot video-language models are not able to outperform a dummy frequency-base baseline, whereas humans in the same setting are nearly perfect. We discuss limitations, and ethical and societal aspects in the appendix. We hope that our work will contribute to understanding models' limitations (through direct evaluation and interpretability analysis supported by the different types of annotations) and narrowing down areas of improvement to guide research towards general perception models.
10

---

Acknowledgments
We are grateful to Luis Piloto, Kenneth Marino, Luyu Wang. Felix Hill, Martin Chadwick, Lucy Campbell-Gillingham,, Boxi Wu, Drew Jaegle, Pauline Luc, Marianne Monteiro, Anna Bulanova, Radu Isac, Mugthar Mohammad, Vijay Vibha Tumala, Mahesh Maddinala, Yiwen Luo, Alina Kuznetsova, Aida Nematzadeh, Lisa Anne Hendricks, Aishwarya Agrawal, Nando de Freitas, Matt Botvinick, Shane Legg, and Relja Arandjelovic for providing insightful input on this project.
References
[1] A. Aguiar and R. Baillargeon. Developments in young infants' reasoning about occluded objects, Cognitive Psychology, 45:267-336, 2002.
[2] Lean-Baptise Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25-37, 2020.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brack, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neurul Information Processing Systems. 2022. URL httpps://opeureviee
[4] lean-Baptiste Alayrac. Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Montciro, Jacoh Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot leamnngg 2022. arrg//44.2222....../ 14198
URL (RLLhhtts://axxxt http://aaxvvvv/.. org/ abs 2204.2024422
[5] Humam Alwassel, Silvio Giancola, and Bernard Ghanem. TSP: Temporally-sensitive pretraining of video encoders for localization tasks. In Proceedings of the IEEE/CVF International Conference Computer
Vision Workshops, pages 3173-3183, 2021.
[6] Renée Baillargeon. Physical reasoning in young infants: Seeking explanations for impossible events. British Journal of Development Psychology. 12:9-33, 1994.
[7] Daniel Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiao-Ya Yu Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Fan- Yun Sun, Fei-Fei Li. Nancy Kanwisher, Josh Tenenbaum, Dan Yamins, and Judith E. Fan. Physion: Evaluating physical prediction from vision in humans and machines. Joaquin Vanschoren and Sai- Sai-Kit Yeung. editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
proceedings. neurips.cc/ httt:p::/ddaaaeets-baaee wwwbbbaratt-rounddee .ttl
paper hoper/202111 2021 wwtp////.00444343444404070000--0 www.://eeeeeec-aatatttteeaaaaeee
[8] Luca Bertinetto, Jack Valmadre, João Henriques, Andrea Vedaldi, and Philip HS Torr. Fully-convolutional siamese networks for object tracking. arXiv preprint 1606v:1606.090
2016.
[9] Fabian Caba Heilbron, Victor Escorcia, Bemard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[10] Luka Cehovin, Aleš Leonardis, and Matej Kristan. Visual object tracking performance measures revisited. IEEE Transactions on Image Processing, 25(3): 1261-1244 2016.
[11] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020.
[12] Deirdre M Cooke, Kryss McKenna, Jennifer Fleming, and Ross Darnell. The reliability of the occupational therapy adult perceptual screening test (ot-apst). British Journal of Occupational Therapy, 2005.
517,
[13] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengehen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7373-7383 June 2021.
11

---

[14] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari. Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. Eurupean Conference on Computer Vision (ECCV), 2018.
[15] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro. Toby Perrett Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for BHICCKITCHENS-III 100. International Journal of Computer Vision (IJC)), 130:33-55, 2022.
[16] Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: A large-scale benchmark for tracking any object. Lecture Notes in Computer Science, pages 436 454, 2020. ISSN 1611-3349
doi: 10.1007/978- 3-788-8-300----- 3-008-855-0----8--77777888888 7_26. httpt/t://ax..oo www.://o...........o00/......0 1007/
078-3-030-1 -78-3-030-58558-7- _26
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lcce, 2009.
[18] Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jürgen Gall, Rainer Stiefelhagen, and Lue Van Gool. Large scale holistic video understanding. In European Conference on Computer Vision, pages 593-610. Springer, 2020.
[19] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens Continente, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-vid: A henchmark for tracking any point in video. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL URI. htttp:://openrevieee http://pwwerrieeeeeeeeerrrrraaaa
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[21] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Mingzhen Huang, Juchuan Liu, Yong Xu, et al. Lasot: A high-quality large-scale single object tracking benehmark. International Journal of Computer Vision, 129(2):439-46 2021.
[22] Marianne Frostig and David Horne The Frostig program for the development of visual perception: Teacher's guide. Follett Publishing Company in collaboration with Curriculum Materials... ..., 1965.
[23] Donald Geman, Stuart Geman, Neil Hallonquist, and Laurent Younes. Visual Turing test for computer vision systems. Proceedings of the National Academy of Science, 112(12)331183322 March 2015. doi: 10. 101107/pas.142225550 1422933112.
[24] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeles dataset for audio events. In 2017 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 776-780, 2017. doi: 10.1 10.1000CAAPPP2217795222...
[25] Rohit Girdhar and Deva Ramanan. CATER: diagnostic dataset for Compositional Actions and TEmporal Reasoning. In ICLR, 2020.
[26] Raghav Goyal, Farzaneh Mahdisoltani, Guillaume Berger, Waseem Gharbieh, Ingo Bax, and Roland Memisevic, Evaluating visual "common sense" using fine-grained classification and captioning tasks, 2018. URL.httpa httctpp://openrevveee www.nneen//onneeeeee wttfforrmmmdd-xx9... kwf
[27] Y. Goyal, T. Khot, D. Summers- Stay, D. Batra, and D. Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6325-6334, Los Alamitos, CA. USA, jul 2017. IEBE Computer Society. doi: 11110//VVV..20...6777 URL https: //doi. ieseccomputersoccietty... wttp:/.......................... 1109/ CVPR. 2017.670
[28] Yash Goyal. Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding Visual Question Answering. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[29] Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Q. Chavis, Autonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh K. Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Z. Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do. Morrie Doulaty, Akshay
12

---

Erapalli, Christoph Feichtenhofer, Adriano Fragomeni. Qichen Fu, Christian Fuegen, Abrham Gebreselasie. Cristina González. James M. Hillis, Xuhua Huang, Yifei Huang. Wenqi Jia, Weslie Yu Heng Khoo, Jáchym Kolár, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullic Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran K. Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang. Xindi Wu, Takuma Yagi. Yunyi Zhu, Pablo Arbelácz, David J. Crandall, Dima Damen, Giovanmi Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard A. Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianho Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3, 000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2022.
[30] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3761, 2022.
[31] Eileen Mavis Hetherington, Ross D. Parke, and Virginia Otis Locke. Child psychology: A contemporary viewpoint, 5th ed. McGraw-Hill, 1999.
[32] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Puttern Analysis and Machine Intelligence, 43(5): 1562-1577, 2019.
[33] Yunseok Jang, Yale Song, Chris Dongjoo Kim, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Vidco Question Answering with Spatio-Tenne Temporal Reasoning. I/CV, 2019.
[34] Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. Moetr-modulated detection for end-to-end multi-modal understanding. arXiv preprint arXiv:2104.1276 2021.
[35] Prannay Kaul, Weidi Xie, and Andrew Zisserman. Multi-modal classifiers for open-vocabulary object detection. In JCML, 2023.
[36] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vjjaannrasimmhn, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017.
[37] Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Roses are red, violets are bluc... but should vqa expect them to? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2776-2785, June 2021.
[38] Zhenyang Li, Ran Tao, Efstratios Gavves, Cees G. M. Snock, and Amold W. M. Smeulders. Tracking by natural language specification. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7350-7358, 2017. doi: 10.1 119/CVPR.2011..77
[39] Paul Pu Liang. Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng. Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee. Yuke Zhu et al. Multibench: Multiscale beuchmarks for multimodal representation learning. Thirty-ffthh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[40] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal- Taixé, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision, pages 1-31, 2020.
[41] Mateusz Malinowski and Mario Fritz. Towards visual turing challenge. arXiv preprint 1rX.v:1410.8027, 2014.
[42] Nancy A Martin and Morrison F Gardner. Test of visual perceptual skills. Academic Therapy Publications Novato, CA, 2006.
[43] OpenAI. Gpt-4 technical report, 2023.
[44] Ronan Riochet, Mario Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP, 03 2018. doi: 10.11099TAAM1222233338333
13

---

[45] Madeline Chantry Schiappa. Shruti Vyas. Hamid Palangi, Yogesh S Rawat, and Vibhav Vineet, Robustness analysis of video-language models against visual and language perturbations. In Thirty-sixth Conference on Neural Information Processing Syxtems Datasets and Benchmarks Track, 2022. URL https://
ovenreviev. net/eerummiddd A
[46] Gunnar Sigurdsson. Gül Varol, Wang, Ali Farhadi, Ivan Laptev, and Abhinav Kumar Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. ArXiv, abs/ (bb//1604.01533, 2016.
[47] Peize Sun, Jinkun Cao, Yi Jiang. Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple- tracking with transformer. arXiv preprint arXiv: 2012.15460, 2020.
[48] Andrea Tacchetti, Leyla Isik, and Tomaso Poggio. Invariant Action Recognition Dataset, 2019. URL //dps: org 10. /910/DVN/DMC
[49] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subbojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks, 2022. URI htttps://arxii org/ abs////////2 2208/ 10442
[50] Zhongdao Wang, Hengshuang Zhao, Ya-Li Li, Shengjin Wang, Philip Torr, and Luca Bertinetto. Do different tracking tasks require different appearance models? Advances in Neural Information Processing Systems, 34:726-78 738, 2021.
[51] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question antwerrnn explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777-977779 June 2021.
[52] Bin Yan, Houwen Peng. Jianlong Fu, Dong Wang, and Huchuan Learning spatio-temporal transformer visual tracking. 202/ IEFF/CVF International Conference on Computer Vision (ICCV), pages 10428- 10437, 2021.
[53] Shen Yan, Xuchan Xiong. Anurag Arna,, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview Transformers for Video Recognition. arXiv e-prints, art. Stt::2201.042880 Jamuary 2022.
[54] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenen- baum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2020. /ttp: http:////..ooaaooeeeeee////////a
[55] Shoubin Yu, Jaemin Cho, Prateck Yadav, and Mohit Bansal, Self-chained image-language model for video localization and question answering. arXiv preprint arXih:22305006698 2023.
[56] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynct-qa: A dataset for understanding complex web videos via question answering. In AAAI, pages 9127 9134, 2019.
[57] Chenlin Zhang, Jianxin Wa, and Yin Li. Actionformer: Localizing moments of actions with transformers.
In European Conference Computer Vision, 2022.
14


Reviews:
[{"review_id": "97diZQXi9N", "reviewer": "Reviewer_7rcX", "summary": "", "questions": "", "limitations": "Limitations are discussed. The authors made it very clear that the goal of the introduced dataset is not to be used as training data, but rather as a benchmark to evaluate complex tasks.\n", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The proposed benchmark sets a new ambitious goal for video understanding which goes beyond what existing video benchmarks can evaluate. Indeed, most benchmarks such as Kinetics, and even some considered as difficult for video understanding such as SSv2, can in reality be solved by using visual cues that are unrelated to real temporal understanding. By designing a benchmark with these problems in mind, and introducing *adversarial* elements such as distractor actions, the authors propose a new challenge for video understanding.\n\n- Interesting video understanding benchmarks for physics and memory existed before, for example CATER and CLEVRER, but are synthetic and mostly consist in 3d shapes moving around. The perception Test consists in natural videos but still evaluate the capabilities of a model to understand the basic principal of physics and perform memorisation tasks.\n\n- For each task, a baseline is proposed. In particular the Flamingo model is evaluated on does not perform much better than random compared to an humain eval baseline, which indicates that there is a lot of progress to do in the future on the path to real video understanding.\n\n- A challenge is available along with a leaderboard.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "PAZl7Ycmv9", "reviewer": "Reviewer_zp8s", "summary": "", "questions": "", "limitations": "Limitations and potential societal impact are discussed in supplemental.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The benchmark considers multiple visual perceptual skills (broadly on Memory, Abstraction, Physics and Semantic understanding; further detailed in the paper) which are not all together present/considered in existing benchmarks. Further the data sources are real world videos and usage of 37 scripts (with 2-5 variations) specifically designed by the authors. While the number of videos is much lesser than existing datasets such as Something-something V2 or Ego4D-V2, the videos have higher 'density' of annotations (in terms of object tracks, point tracks, question-answers, etc). This makes it a more comprehensive benchmark than existing works (although as stated in paper, the proposed benchmark mainly serve as a test for 'transfer' or zero/few-shot capabilities of pretrained models and not as a base training source itself).\n\n2. The data collection and annotation is well described and goes through multiple rounds of annotation in addition to diversity in participants to ensure data quality and diversity. The addition of distractor actions, distractor objects, adversarial configurations of objects and actions also reduces potential language or visual biases and increases complexity of tasks in the dataset.\n\n3. The tasks and associated expected outputs, metrics and considered baselines are well described for evaluation and reproducibility.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "1uLWoWgEUW", "reviewer": "Reviewer_nYjf", "summary": "", "questions": "", "limitations": "The authors have discussed the limitations and potential negative societal impact of their work in the Appendix, section Limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.The proposal of a comprehensive benchmark that assesses the capabilities of multimodal perception models across different skill areas, types of reasoning, and modalities. This contributes significantly to the research community by providing a holistic evaluation tool.\\\n2.The design of the benchmark to avoid overfitting and to assess the transfer abilities of models pre-trained with any external dataset or task, which enhances the relevance of the benchmark to real-world applications.\\\n3.The provision of baseline results for various per-task models, spanning object tracking, point tracking, temporal action localization, temporal sound localization, multiple-choice video question-answering, and grounded video question-answering. These baseline results serve as reference points for future research and establish a benchmark for performance comparisons.\\\n4.The open-sourcing of the videos and annotations in the train and validation splits, as well as the evaluation code, which promotes reproducibility and transparency in research.\\\n5.The availability of a challenge server to evaluate models on the held-out test split, which encourages researchers to compare their models with state-of-the-art methods and to improve their models' performance.\\\n\\\nIn summary, the strengths of the submission include its comprehensive evaluation approach, focus on avoiding overfitting, provision of baseline results, open-sourcing of resources, availability of a challenge server. These strengths add to the significance and quality of the submission.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "u5a6hHMErd", "reviewer": "Reviewer_t6Pk", "summary": "", "questions": "", "limitations": "The authors explicitly mention that this work is an evaluation benchmark and not a dataset suitable for training. Therefore, there are fewer videos in this work compared to the majority of the related work they compare against.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The area of multimodal learning is growing fast, and a benchmark for testing the reasoning abilities of multimodal models in difficult situations can benefit the community.\n2. The authors demonstrate that state-of-the-art methods are still lagging behind humans and even another simple baseline when it comes to video question answering. Therefore, this benchmark can provide a good opportunity for researchers to develop more capable multimodal models.\n3. The paper is very well-written and easy to follow.\n4. This work fills the gap in the literature by providing a benchmark of more difficult scenarios in a multimodal setup.\n5. The authors provide a live evaluation server for the benchmark.\n6. This benchmark includes six different types of annotations which makes it possible to define more tasks (in addition to the tasks that the authors described)  to evaluate multimodal models", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "wEYssAD9ln", "reviewer": "Reviewer_mVd6", "summary": "", "questions": "", "limitations": "* 1 model per task seems small to make any kind of conclusions from the benchmark, limiting the insights for future work. I know the limitation was mentioned in L248, but I would assume there are more action recognition, VQA models, etc that could be used.\n* Some models needed fine-tuning or training from scratch to work on the benchmark.\n* Because there are few insights provided or discussion on what this means for current approaches, the utility of the dataset is not motivated in its entirety. I think there is great potential though, so this is an area of improvement. \n", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "**Summary**\nI think the dataset is clear in its motivation and collection process. I also think the different aspects of the dataset are very useful to better understand where models are strong and weak and what areas of improvements can be made. \n\n**Details**\n* When filming videos, they ensured ethnic variety to prevent undesirable biases. \n\n* The details on the demographics (location, ethnicity, camera, etc) were provided. \n\n* Table 2 provides good clarity on the scripts discussed.\n\n* The supplementary provides details on the models that were evaluated on the benchmark. \n\n* On some tasks they provided human results, giving us another baseline to compare to that is useful for future work. They also provide a frequency-based baseline, allowing for further comparisons.\n\n* In supp Section 5, there is a detailed explanation of how the datasets were split to ensure less bias in the distribution of variations for video. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: __Summary of paper and reviews.__
This work introduces a dataset of scripted real world videos collected to test 4 different skills and 4 types of reasoning over 3 modalities. Instances are densely annotated to simultaneously support 6 different tasks. A strong baseline from the literature is evaluated for each task and a leaderboard will support future evaluations.

Reviews were generally positive, agreeing that the dataset poses a difficult new task with limited weaknesses.

__Strengths.__
* The benchmark provides a vision (unified perception) that will drive progress because no current model even runs on all of its tasks, yet each task individually is difficult but in reach from the state of the art.
* The presentation is clear and the dataset released with a public leaderboard and test set.
* Multiple rounds of annotation ensure the quality of the diverse and dense annotations while careful script design aims to prevent dataset biases.
* One task (grounded videoQA) is new.

__Weaknesses.__
* Few models (usually 1) are evaluated for each of the 6 tasks, so the paper provides limited insight about how different approaches compare.
* Though collected in the real world (not simulated), the videos still lack diversity, so they do not test generalization to open ended settings.

__Decision justification.__
This paper provides a high quality dataset that presents multiple new challenges, so it is a clear accept.
Decision: Accept (Poster)