Slot-VLM: Object-Event Slots for Video-Language Modeling
Authors: Jiaqi Xu
Cuiling Lan
Wenxuan Xie
Xuejin Chen
Yan Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Language Modeling
Object-Centric Representation
Event-Centric Representation
Video Understanding
Primary area: machine_vision
TL;DR: In this work, we propose a framework that advocates the construction of semantically decomposed video tokens, which encourages object-wise and event-wise token representations, for effective video-language modeling.

Abstract: Video-Language Models (VLMs), powered by the advancements in Large Language Models (LLMs), are charting new frontiers in video understanding. A pivotal challenge is the development of an effective method to encapsulate video content into a set of representative tokens to align with LLMs. In this work, we introduce Slot-VLM, a new framework designed to generate semantically decomposed video tokens, in terms of object-wise and event-wise visual representations, to facilitate LLM inference. Particularly, we design an Object-Event Slots module, i.e., OE-Slots, that adaptively aggregates the dense video tokens from the vision encoder to a set of representative slots. In order to take into account both the spatial object details and the varied temporal dynamics, we build OE-Slots with two branches: the Object-Slots branch and the Event-Slots branch. The Object-Slots branch focuses on extracting object-centric slots from features of high spatial resolution but low frame sample rate, emphasizing detailed object information. The Event-Slots branch is engineered to learn event-centric slots from high temporal sample rate but low spatial resolution features. These complementary slots are combined to form the vision context, serving as the input to the LLM for effective video reasoning. Our experimental results demonstrate the effectiveness of our Slot-VLM, which achieves the state-of-the-art performance on video question-answering.

Slot-VLM: Object-Event Slots for Video-Language Modeling
Jiaqi Xu Cuiling Lan", Wenxuan Xie', Xuejin Chen', Yan Lu² University of Science and Technology of China,
Microsoft Research Asia
xuj jujiaqiimat edu.cn, {culan, yanluieeimaaummmmmcmmmmmm
x xjchen99Rustc.
Abstract
Video-Language Models (VLMs), powered by the advancements in Large Lan- guage Models (LLMs), are charting new frontiers in video understanding. pivotal challenge is the development of an effective method to encapsulate video content into set of representative tokens to align with LLMs. In this work, we introduce SLo,-VLM, new framework designed to generate semantically decomposed video tokens, in terms of object-wise and event-wise visual representations, facilitate LLM inference. Particularly, we design Object-Event Slots module, i.e., OE- Slots, that adaptively aggregates the dense video tokens from the vision encoder to set of representative slots. In order to take into account both the spatial object details and the varied temporal dynamics, we build OE-Slots with two branches: the Object-Slots branch and the Event-Slots branch. The Object-Slots branch fo- cuses on extracting object-centric slots from features high spatial resolution but low frame sample rate, emphasizing detailed object information. The Event-Slots branch engineered to learn event-centric slots from high temporal sample rate but low spatial resolution features. These complementary slots are combined form the vision context, serving as the input to the LLM for effective video reasoning. Our experimental results demonstrate the effectiveness of our Slot-VLM, which achieves the state-of-the-an performance on video
Introduction
Recently, Large Language Models (LLMs) have gained significant progress exceptional ability to comprehend, reason with, and generate human language text. Such amazing capabilities have stimulated the wave of research on extending the models Vision Language Models, enabling the vision reasoning ability.
39
31
They present
For image understanding, MiniGPT-4 [5T] leverages a Q-Former and projector to align frozen vision encoder with a frozen advanced LLM, where Q-Former converts the visual input into a fixed- length learned visual query tokens (32 tokens). LLaVA 25 and MiniGPT-v2 I0 directly use the gridded visual tokens (after projection) as the LLM input. Image-text pairs are leveraged to align the visual model and the LLM in training. For handling videos, one straightforward way is to stack the tokens from sampled frames and feed them into the LLM [19 12 50 This is challenging when densely sample frames (e.g., with the purpose of preserving more information) or when we sample abundant frames for long videos. For example, for video of 10 minutes, when we sample at frame per second and each frame uses 32 (or 256), we will have 19,200 (or 153,600) tokens total. On the one hand, the large number of tokens increases both the memory and computational requirement. On the other hand, there is apatio-tempon redundancy and the features are not representative.
* "This work done when Jiaqi intern at Microsoft Rescarch Asia. 2This paper is the result of an open source research project starting from October, 2023.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Visual features
Pooling
(a)
Q-Former
[c]
LLM
Object-centric
(d)
LLM
Event-1
Event-2
Event-3
Event-centric
tokens
Figure 1: Illustration of methods for aligning visual features with LLM. Previous methods (a) and (b) leverage pooling or Q-Former to aggregate visual tokens, where each generated token contains coupled semantics. In contrast, we propose to generate semantically decoupled object-centric tokens illustrated in (c), and event-centric tokens illustrated in (d), align with the LLM.
Video-ChatGPT (28) reduces the number of video tokens to 356 by separately performing spatial pooling and temporal pooling, which suffers from the loss of visual details. Video-LLaMA [48] and VideoChat [20] compress the video tokens using Q-Former, where set of learnable queries aggregate the information from video tokens through cross attsettttnntoon and self-attention. These strategies generate reduced tokens. However, each token contains semantically mixed/entangles information (see the visualization analysis Q-Former in Appendix G.I) and this may bring inefficiency to video-language modeling. As we know, the text words as the input LLMs have semantics and disentangled. Intuitively, to better align the video tokens with the language imput of L.Mss for building VI.Ms, the generated video tokens that could act similarly words desired.
In this work, as illustrated in Figure [ we aim to generate semantic-centric tokens from video features to comply with LLMs for effective video-language modeling. By leveraging slot attention 27 35), which converts an image into object-centric representations, i.e., slots, we design an Object- Event Slots module (i.e., OE-Slots module), that generates set of semantically decomposed video tokens (a.k.a. slots) from video features, and takes such slots as input to LLMs. This is somewhat similar to human visual reasoning, where the brain converts the visual perception to high-level object representations to facilitate further reasoning. We dub the scheme powered by OF-Slots module as Slot-VLMM. Figure 2 shows the flowchart of Slot-VIs VL.. Particularly, OE-Slots module consists of two branches: Object-Slots and Event-Slots for decomposed spatial and temporal modeling, focusing spatial objects and temporal varied dynamics (event), respectively. The Object-Slots branch extracts object-centric slots from high spatial resolution features but sampled at low frame rate. The Event-Slots branch extracts event-centric slots from high temporal resolution but low spatial resolution features. The two sets of slots are combined together as the input to the LLM for video reasoning. In instruction-tuning we fine-tune OE-Slots module and the projection layer to align the visual features with LLM.
In summary, have three main contributions:
A semantic-centric token refers to token that represents semantically meaningful entity, such as an object, event (from temporally consistent segment).

---

High spatial & lose frama rete.
Object-Slots
SoA Alientinn
Outout
00
D B
xx N₁
#1
High frame rate & low sparial ree.
Input
W
Features
Po..
Text
20th X
Event-Stots
Imput
Object-Evant Stots (OE Stots)
L-MM-+ff Output
Figure 2: Flowchart of our proposed Slot-VLM for video understanding. Slot- VLM consists of a frozen image encoder, learnable Object-Event Event Slots module (i.e., OE-Slots module), projection layer, and frozen LLM. The image encoder encodes the input video of frames into a sequence of image features, resulting in extensive (H x W T') video tokens. In order to obtain semantically decoupled and compact (reduced) video tokens as the vision context for aligning with LLM, our OE-Slots module learns aggregate those tokens to object-centric tokens and event-centric tokens through the Object-Slott branch and the Event-Slots branch, respectively. The Object-Slots branch operates low frame rate << T') but high spatial resolution order capture spatial objects through slot attention on each sampled frame. The Event-Slot branch operates at high frame rate but low spatial resolution (m
w, where h < H, w < W) in order to capture temporal dynamics through slot attention over each spatial position. The learned slots (tokens) from two branches are projected and inputted to LLM for video reasoning, together with the text.
framework, Slot-VLM, for effective video-language modeling, where generate semantically decomposed tokens to comply with LLMs. To the best of our knowledge, this work is the first to explore the use of learnable semantically decoupled visual tokens to align with LLMs. This work provides new insight that 'disentangled" video tokens are beneficial for video- modeling.
We propose
We design an OF-Slots module that encourages object-centric and event-centric visual represena- tions for modeling spatial objects and temporal dynamics. This moves step towards the compact and semantically meaningful token representations in VLMs.
Experimental results demonstrate the effectiveness of our OE-Slots designs, achieving state-of-the- art performance on video question answering tasks.
We hope that this work will inspire more designs towards generating compact and semantically interpretable visual tokens for effective video-language modeling.
2 Related Work
Visual Language Models In considering the strong comprehension, reasoning, and generation ability of the LLMs, there is surge of investigations on exploiting LLMs and vision encoders to build
visual-language models, enabling the vision reasoning ability 19 51
25 20
49
48
37
BLIP-2 19 leverages frozen pre-trained CLIP image encoders 332 and frozen LLMs to bootstrap
language-ing image pre-training, with the modality gap bridged by Querying Transformer (Q-Former) and a projector. Q-Former extracts a fixed number (e.g., 32) of output features (tokens) from the image feature, acting as information bottleneck to facilitate the feeding of most useful information to LLMs. Other works like MiniGPT-4 [51) and Video-LLaMA [48] similarly leverage Q-Former to encode visual tokens, followed by image-tex or video-text pair instruction tuning.
Compared with image reasoning, is more challenging for video-language modeling, where the excessive number of visual tokens from video would enlarge the computation and memory cost to the I.MM inference and thus constrains the application VLMMs. Some works sparsely sample frames and concatenate the tokens from these frames as the input to LLMs [19 12 50). However, this brings

---

difficulty for understanding videos with high temporal dynamics or long videos, with high risk of losing valuable temporal information.
Miteo-ChatGPT 28 reduces the number of visual tokens by performing spatial and temporal pooling separately. LLaMA-VID [2) represents each frame with two distinct tokens: context token and content token, where context token is obtained by aggregating the visual features based on the text query while the content token is obtained by pooling the features. However, being task-agnostic, pooling would result in loss of helpful information. Moreover, the semantic-agnostic pooling inevitably results in the mixing of features of different semanties, where such coupled representations may bring difficulty for video reasoning. Q-Former provides an elegant way for generating reduced number of tokens. However, we found the learned tokens are still coupled. with abundant overlapped information among them. When aligning with the 'disentangled' word text tokens, such entanglement may result in poor compatibility with L.LMs. Chat-UniVi [17] merges the dense visual tokens by clustering and uses the averaged token features in a cluster as the cluster representation. With clustering being parameter-free and task-agnostic, this is sub-optimal. It is still an open question on how to generate video tokens for constructing effective VLMs.
In this work, explore the use of semantically decoupled tokens by learning spatial object-centric slots and temporal event-centric slots as video representation. We reveal that using semantically decoupled tokens as the imput to LLMs are effective for video-language modeling.
Object-Centriy Learning Humans naturally decompose the observed environment into entities at the appropriate level of abstraction to act in the world (35]. To obtain semantically decoupled
representations, object-centric representation learning has garnered significant interest 6 27
36 35
Slot Attention 27 is a representative work. A slot attention module is designed to interface with features to produces a small set of abstract representations, named as slots, where each slot represents an object or a scene. It demonstrates encouraging object decomposition ability in simulated data but fails on real-world images (27). To successfully learn object-centric slots in real-world images, DINOSAUR (35 learns the slots from pre-trained features by reconstructing the features. Such pre-trained features can be obtained from self- self-supervised learning techniques like DINO [8) MAE [15] or vision- languuge pre-tr teettraiiing like CLIP [32]. These features usually have a high level of homogeneity within objects and thus facilitate the forming of object-centric representations.
In this work, leverage slot attention to learn semantically decoupled tokens as the input LLMs and investigate the effectiveness of aligning these 'concept tokens' with LLM input. Note that, interestingly, words in human language
abstractions of concepts, with different words tending to be disentangled. We intend to bridge the connection between vision and language by representing video with 'concepts' in terms of object-centric slots and event-centric slots.
3 Proposed Slot-VLM
As illustrated in Figure 2 our Slot-VLM consists of a frozen image encoder, a trainable Object-Event Slots (OE-Slots) module, projection layer, and a frozen LLM. Given video sequence, we extract frames at a speed of frame per second (FPS) and obtain a video V of T frames. The video V together with the text question (user instruction) is input to Slot- LLM, and Slot-VLM outputs the generated text answer.
In order to build an effective VLM for video understanding, we propose to encapsulate the dense video tokens into a small set of semantically decoupled tokens to align with the LLM. Our OE- Slots module converts the dense video tokens from the vision encoder into set of object-centric slots (tokens) through the Object-Slots branch, and a set event-centric slots (tokens) through the Event-Slots branch, which are then projected the input to LLM for video reasoning.
Intriguingly, our exploitation of object-centric visual representations enhance large language model (LLM) reasoning mirrors certain aspects of the cognitive processes humans use for visual reasoning. Human visual reasoning initially through the primary visual cortex (V1) and subsequently integrated in higher-level visual processing areas, resulting in the formation of complex visual object representations. Such high-level object representations together with brain-stored knowledge are then used for logical reasoning and inference interpret observations R4 3 Similarly, Slot VLM generates object-centric and event-centric token representations to provide the vision source for the LLM reasoning, where the
complex cognitive process. Visual signals are processed

---

LLM stores rich knowledge and has strong reasoning ability, and interpretable-ten text generation ability. Such exploration makes us move small step towards brain-like visual reasoning.
3.1 Visual Feature Extraction
To extract visual features of a video, we simply use an image-based model to get per-frame feature. Following (28), we utilize vision-language pre-trained CLIP (ViT-L/14) vision model [32] as our vision encoder. For each frame, the vision encoder outputs H x W visual tokens, with the dimension of a token denoted by D. where H W 16 for the CLIP vision encoder. For sampled video of T frames, we have H x W x T visual tokens. Taking a video of 3 minutes as an example, the total number of video tokens 16 x 16 x 180 = 46, 080. The large number of tokens would incur significant computational and memory costs during L.MM inference. Considering the redundancy from the dense tokens, we aim semantically exploit the the spatial structure and and temporal dynamics to reduce the number of tokens for efficient LLM inference.
3.2 Object-Event Slots (OE-Slots) Module
actually produce tokens wherein semantics remain entangled within each token, rather than targeting at generating semantics decoupled representations. In contrast, the word tokens, which are the input to LLMs, are more semantically decoupled. In this work, we explore the using of semantics-decoupled video representations for effective iido-language modeling.
Previous methods, which aggregate the video tokens by pooling (28) Q-Former [50] 40
We design a OE-Slots module that encapsulates the dense video tokens from the vision encoder into a small set of semantic-centric tokens to facilitate the LLM inference. It is challenging to learn semantics-centric representations from the dense video features. DINOSAUR (35] uses slot attention on the pre-trained image feature learn object-centric representations without supervision. It is challenging to directly extend the slot learning to video features, given the substantial number of video tokens (e.g., 46,080 for a 3-minute video), which increases memory and computation requirements, and the optimization difficulty.
In our work, effectively learn semantics-centric representations from abundant video tokens, design an Object-Evenn Slots module comprising two branches. The an Object-Slots branch operates at low frame rate to capture spatial object-wise slots for each sampled frame, while the Event- Slots branch operates at high frame rate but low spatial resolution to capture temporal event-wise slots. This dual-branch design is inspired by the SlowFast network [13, employing a Slow pathway for low frame rate spatial feature learning and a Fast pathway for high frame rate motion capture using a lightweight subnetwork. Unlike SlowFast which focuses on refining grid-wise features, our Object-Event Event Slots module aims to generate a small set of semantically meaningful object-centric slots and event-centric slots, serving as semantically meaningful input into LLM.
Object-Slots Branch As shown in Figure of the Object-Slots branch, for a video of H x W x T dense video tokens, sample the features at low frame rate but high (original) spatial resolution to obtain H x W x t video tokens. denotes the number of uniformly sampled frames, which we set to 8 by default.
For the i-th frame, we have a set S; of H x W tokens. S, is taken as the input to slot attention module 27 35 to generate N, object-centric slots O = (or,1,.... oi,N.) ,, (we also refer to them as spatial slots). Please see Appendix for more detailed formulation. Particularly, slot attention uses an iterative mechanism to map from the input tokens to the slots. At each iteration, slots attention uses cross attention with attention coefficients that are normalized over the slots (where slots are the queries) to aggregate token information. The normalization over the slot introduces competition between the slots for promoting the forming of decoupled representations.
To distinguish different frames, we add learnable temporal position embedding p: to each slot of the i-th frame and obtain the updated object slots as O = (o..1.... or,N.,. where o == 0.j + p;. We concatenate the updated slots of all the frames and obtain / x N, slots O (O,,..., O,} (0),1; .1, O1,N. .............................. O, linear projection layer (S-Proj.) transforms each token to facilitate the alignment with slots from the Event-Slots branch and the alignment with the LLM.
5

---

Note that we perform spatial object slot learning on the high spatial resolution but low frame rate features in order capture the spatial objects and reduce the number of slots, which proportional to the number of frames.
Event-Slots Branch As shown in Figure 2 of the Event-Slots branch, for video of H W x T dense video tokens, we sample the features at high frame rate but low spatial resolution to obtain T video tokens. We obtain the spatial down-sampled tokens by averaging pooling with a stride of 4, therefore h = H/4 = 4 and w W/4- = 4.
We perform temporal slot learning along temporal axis. To be aware of different frames, add learnable temporal position embedding to each token. For the k-th spatial position, where k ===1.... .,..., and m h w, we have set Fx of T tokens from the T frames. Fx is taken as the input to slot attention module to generate N₂ event-centric slots & = [e ex,N. (we also refer them as temporal slots). Note that each spatial position from h positions corresponds to a large local patch in the pixel space, 56x56 patch for a video of 224 224 spatial resolution. This allows us to observe the temporal dynamics of large local region x56 pixels), which helps infer the evolution of part of an event within that region. Temporal slots aggregate the temporal tokens for explaining parts of the input, similar to identifying events. Thus, we name the learned slots as event-centric slots. We leverage temporal slots to aggregate semantically consistent temporal segments, akin how spatial slots aggregate spatially consistent regions. While these temporal segments within a slot might not form a complete event, they are likely part of the same event.
We concatenate the slots from each of the m spatial positions and obtain {E....,, Em) = (e1,1, e1,2,,,,,....... eI.NN...... N. J- A linear projection layer (F-Proj.) transforms each token to facilitate the alignment with slots from the Object-Slots branch and the alignment with the LLM.
N₂ slots &
Note that we perform temporal event slot learning the high frame rate but low spatial resolution features in order to capture the temporal dynamics, and reduce the number of slots, which is proportional to the spatial feature resolution.
3.3 Connection to LLM
We concatenate the object-centric slots from the Object-Slots branch and the event-centric slots from the Event-Slots branch and obtain N = t x N, + m x Ne slots. A linear projection layer (Proj.) transforms each token align with the LLM. The projected tokens and the text instruction are taken as the input to LLM for video reasoning.
3.4 Training Strategy
We divide the training procedure into three stages: slot attention pre-training, single branch instruction tuning, and two branch joint instruction tuning. We use the three stage training to better optimize the model. Stage 1 aims to pre-train the slot attention modules (with the objective of reconstructing the imput features) to facilitate the learning of object-centric and event-centric slot representations, i... the forming of semantically decomposed tokens/slots. Stage 2 separately trains the Object- Slots branch and the Event-Slots branch to facilitate the system focusing on the optimization each branch separately, which eases the optimization. Stage 3 jointly optimizes the two branches and the projection layer. Please see Appendix B for more details.
Experiments
4.1 Implementation Details
Following Video-CharGPT (2 we employ the pre-trained CLIP vision model of ViT-L/14 32
as the vision encoder. Note that the model size of CLIP ViT-L/14 (which we use) is 303M, which much smaller than CLIP ViT-G/14 (1012M) as used by MovieChat [37] LLaMA-VID 22), and Video-LLaMa (23). Specifically, we extract features from the penultimate layer, yielding an array of H W visual tokens for each video frame. We use the Vicuna (7B) from the LLaVA model 25 to serve as our LLM. We sample at fps from a video to obtain T frames, and we resize each frame to 224 222 resolution. In our experiments, we set N. and N. to by default unless otherwise specified.
The OE-Slots module generates N t No m N 8 x 8 + 16 x 8 192 slots (tokens)

---

Table 1: Comparison with the state-of-thee art methods for video QA. All these models use Vicuna- 7B as the LLM. Different methods may use different datasets for pre-training. Moreover, for the instruction tuning, different methods adopt different instruction data as illustrated in the second column. For example, (EV)45.IMM)) denotes the instruction data comprises about 1 11,000 pairs of video instructions pairs and 5.1 million pairs of image instructions. Connector denotes the method for connecting the vision features and the LLM. See Table for the number of video tokens.
Instruction Data
MSVD QA Score
MSRVTT QA Score
ActivityNet QA Score
Average
Model
Connector
(# of Pairs)
Acc.
Acc.
Acc.
Acc.
Score
Video a. 48
11K(V)5.IM(I
Q-Former
51.6
2.5
29.6
1.8
12.1
1.1
31.1
1.8
Video Chat 20
11K(V)+7<()
Q-Former
56.3
2.8
45
2.5
26.5
2.2
42.6
2.5
Video- Chaa-Ch
28
100K(V)
Pooling
64.9
3.3
49.3
2.8
35.2
2,7
49.8
2,9
Char-UniVi
2M(V)=333(()
Clustering
65
54,6
3.1
45.8
3.2
55.1
3.3
Video-LLaVA
23
100K(V)
64.8
58.3
40.7
54.6
Video-LLaVA
100K(V)+66K())
70.7
59.2
3.5
45.3
3,3
58.4
BT-Adapter
26
100K(V)
Temporal Adaptor
67
3.6
51.2
2.9
46.1
3.2
54.8
3.2
LLaMA-VID 22
(065())))
Q Former&
69.7
3.7
57.7
3.2
47.4
3,3
58.3
3,4
VideoChar2
21
0.SM()+LLM())
Q Former
70
3.9
54.1
3.3
49.1
3,3
57.7
3.5
MovieChat
S7
11K (V)+5.1M
Borge+Q-Formeer
75.2
3.8
52.7
2.6
45.7
3.1
57.9
3.2
Slot-VLM (Ours)
100K(V)
Object-Event Slots
74.9
3.8
69.6
3.4
48.3
3.4
64.3
3.5
Slot-VLMM (Ours)
(VK(())))
Dhject-Fyent Slots
75.9
3.8
69.6
3.5
49.4
3.4
65.0
3.6
from 16 16 x T dense video tokens. To reduce training cost, both the image encoder and LLM frozen in our training.
All models are trained using a single NVIDIA A100 80GB GPU. The linear projection layer S- S-Proj.. F-Proj. and Proj. consists of 1024, 1024, and 4096 neurons, respectively. We adopt a network structure that is similar to the slot attention from [6 to build the slot attention modules, with learnable slot initialization. More details please refer to Appendix
4.2 Data Details and Evaluation Metrics
We use the Video Instruction Data, collected by (28). for video instruction tuning. This comprehensive dataset comprises approximately 100K video text gquestion-and-answe from the ActivityNet dataset with average video length of 180 seconds. The dataset is characterized by diverse array of question types. Alternatively, similar to Video-LLaVA (23), we could also incorporate 665K image-text instruction dataset from LLaVA v1.5 (24 for enhancing the instruction
pairs, which are generated
tuning of the Object-Slots branch and mark the model as Slot-VLM' We report both the results of Slot-VLM and Slot- VLM* in Table Note that all our other results including ablation studies and visualization are obtained without using the 665K pairs.
We evaluate the performance on three open-ended video question-answering (QA) benchmarks like MSVD-QA(9). MSRVTT-QA 44 and NetivityNet-QAA 77) We evaluate models using accuracy (%)
and average score metrics by (28], employing ChatGPT judge prediction correctness. ChatGPT reviews each QA pair and issues a binary correctness verdict and a similarity score from 0 (least) to 5 (most) (See Appendix C for more details). We also evaluate our models on three multi-choice QA
benchmarks, including Egoschema| 29] NEXT-QA 43) and STAR 42
(See Appendix IT.
4.3 Comparison with the State-of-the-Arts
We evaluate the performance of our scheme against the state-of-the-art methods on three zero-shot open-ended video QA benchmarks. Table [ the results. All these models use Vicuna-7B (11 as the LLM.
For video-language modeling, there is no converged standard on the training datasets, including the data for pre-training and that for instruction tuning. Generally, it not very fair for comparisons. For example, Video-LLaMA [48] uses Webvid-2M short video dataset and CC595k image caption datasets for pre- tre-training to enable video features to contain as much visual knowledge as possible; VideoChat2 (21) performs per-training of two stages for vision- language alignment (15M image captions from CC3M and CC12M and 10M video captions form WebVid- 10M) and vision-language connection (adding 2M image captions, and 10M video captions from Intern Vid 40),, respectively. Similar to Video-ChatGPPT 28), we do not perform such visual-text alignment pre-training. For instruction tuning, different methods use different data (see the second column in Table [
7

---

We have the following observatiioss/concluiio 1) Our Slot- V consistently achieves the best aceuracy on all the three benchmarks, outperforming all other methods, even though the volume of vision-text pairs used for our instruction tuning (0.7M pairs) is much less than many models such as VideoChat2 [2T] (1.9M pairs), Univ-UniVi 17 (2.4M pairs), and MovieChat [37] (5.1M pairs). The average performance of our Slot-VL.M" outperforms Video-L.LaVA by 6.6% in accuracy, while achieving the similar scores. 2) In comparison to the methods highlighted in gray that utilize the same 100k video-text pairs for instruction tuning, our VLM consistently outperforms its competitors, surpassing and 13.1% % on ActivityNet-QU. Furthermore, it exceeds Video-LLaVA 23 by 10.1%. %, 11.3%, and 7.6% %, and B BT-Apteter [26] by 7.9%. 18.4%. and 2.2% across the same benchmarks, respectively, demonstrating the high efficacy of our framework. 3) Compared with UniVi [77] that leverages clustering to aggregate/compres tokens, our Slot-VIM VI. outperforms it significantly by 10.9% on MSVD-QA, 15.0% % on MSRVTT-QA, and 3.6 % ActivityNet-QA in accuracy, respectively. Note that Chat-UniVe uses 2 million video-text pairs and 433K image-text for instruction tuning while we only 100K video-text pairs and 665K image-text pairs. 4) Comparing VLMT with Slot- VLM, we can see that the incorporation of 665K image-text pairs brings an average of 0.7% gain in accuracy.
28) by 10% on MSVD-QA, 20.3% % on MSRVTT-QA,
4.4 Ablation Studies
Table Ablation studies on the effectiveness of our Slot- We compare our schemes powered by slot attention with the schemes powered by Q-Former under our framework. The FLOPs and number of parameters for the reduced token generation module are presented.
In-domain
MSVD-QA
Model
#FLOPs (G)
#Param. (M)
Acc.
Score
Acc.
Score
Spatial- Qeoomororrrraeee VLM Object-Slot- -LLM
40.1
2.48
70,9
3.57
24.9
50.8
Spatial branch
46.5
2.69
73.1
3.71
20.8
13.6
Temporal-Qfore Qformrr VLM
46.0
2.64
72.6
3.62
26.2
50.8
Temporal branch
Event- Slot-VI.M
47.1
2.67
73.1
3.67
20.7
13.6
ST-Qformer-V VLM
29.7
2.07
66.9
3.40
51.1
101.6
Two branches
Vlot-VLM
48.8
2.75
74.9
3.76
41.6
27.3
We study the effectiveness of our semantics-centric designs, two branch design, and the hyper- parameter choices, on the test set of the Video Instruction Data from (28). which we refer to as In-domain, and on MSVD-QA [9]. We name our scheme with only the Object-Slots branch as Object-Slot-VLN. the scheme with only the Event-Slots branch as Event-Slot- VLM, and our final scheme as Slot- VLM
Effectiveness of using Semantics-centric Tokens Under our framework, we replace our slot attention modules for generating reduced tokens by Q-Former [1,, using similar training strategies and computational cost (FLOPs). Similarly, we name the schemes with the spatial branch, the temporal branch, and two branches as Foralll-oooor--VVee
Cemporal-QFormeer VLM, and ST-QFormer- shows the results, the number of FLOPs, and the number of parameters
VLM, respectively. for the reduced token generation modules, indicating the fairness for comparisons. Note we compute the FLOPs by taking a video of 100 frames. The number of parameters for Object/Event Slots module is much smaller than that of the Spatial/Tempores Temporal-Qformer module because the Slot Attention uses shared parameters for its L 3 iterations. We can that the schemes using slot attention outperform those using Q-Former in both the single branch setlings and the two branch setting, Object-Slot-VL., outperforms Peatial-FForme--VI.. by 6.4% on the indomain test data (In-domain) and 2.2% on MSVD-QA, respectively. Event-Slot- VL.M outperforms -emporal--Foormrr- VL by 1.1% % and 0.5%. We visualized the spatial attention maps of learned Q-Former and found that Q-Former is not capable of decoupling visual tokens to semantically meaningful instances (Please see Appendix G.2).
In addition, for the two-branch scheme ST-QFormer-VILL VLM, we found that tuning the Q-Formers and projection layers even with the single-brance trained parameters as initialization cannot lead to satisfactory results, which are poorer than the single-branch schemes. That may be caused by the difficulty in aligning the output tokens of the two branches.
Effectiveness of our Two-branch Design As shown Table our VLM with two branches outperforms our angle-branch schemes Object-Slot-Ve VLM by 2.3% on dn-domain/MVVD-QA
8

---

Table Ablation study on the effectiveness of joint spatial-temporal slots learning vs. two branch design. The FLOPs and number of parameters for the reduced token generation module are presented.
In-domain
MSVD-QA
Model
#FLOPs (G)
#Param. (M)
Acc.
Score
Acc.
Score
Slot-Joint-V VL.M
46.7
2.66
72.8
3.68
1304.8
13.7
Slot-VLM
48.8
2.75
74.9
3.76
41.6
27.3
i
0,2
0
0,5
0
0,7
0
01,1
0
0,3
0
01,5
01,6
01,7
OL,
1
3
5
7
Figure Visualization of spatial attention masks from the Object-Slots branch for video examples, respectively. We have = 8 frames as shown in 8 rows, indexed by i, where = 1, 1,...,4 t. The first column shows the original frame. The second to the ninth columns show the cross attention mask (from slot attention) for the N. = 8 object-centric slots O; {o..1..... OI.N. J. We can see that even though not perfectly segmented. some meaningful slots have been formed. For example, the slots marked by red, purple, green, and blue in the first video (left) correspond to 'background; "human body", 'head', and 'barbell'. Note that the slots in a frame is unordered and exchangeable.
and Event-Slot-VLM by 1.7% in accuracy, respectively. Our semantics-centrie tokens from the two branches are complementary.
An alternative design is to learn the same number of slots (i.e., 192) from the dense H x W x T tokens instead of from the two branched tokens. We dub the scheme as Moi-Jonnt-VLL, As discussed in Section 3.2 directly extending the slot learning to dense video features raises increases memory and computation requirements, and the optimization difficulty. Table B shows the comparison between Slot-Joint- VLM and our two-branch design BLM. We can see that the computational complexity of the reduced token generation module of Slot-VLM is much lower than that of Slot-Joint-VLA while MLM achieves superior performance.
Influence of
Appendix shows the influence of
4.5 Visualization
Object- Cirect-Centrictee Slots We visualize the spatial cross-attention masks from our Slot-VLM for indicating the forming of object-centric slots for each frame, with two video samples shown in Figure We can see that some meaningful slots have been formed. For example, in the first video, the slots marked by red, purple, green, and blue in the first video (left) correspond to 'background", 'human body', "head", and 'barbell'. For comparison, the scheme Peatial-QFormedd VLM in Appendix G.2 We observe that our object-centric slots from Slot-VLA are better semantically decoupled.
also visualize the cross-attention masks of Q-Former from
Moreover, we found the instruction tuning can further promote the decoupling of slots (see Ap- pendix G.3 where the well decoupled representations may better align with L.M..
Event-Centric Slots Figure 4 a) visualizes the temporal cross-attention masks from our Event- Slots branch for indicating the forming of event-centric slots for each spatial position (in total

---

E4
86
By
Eg
E10
E11
E12
E13
E14
915
616
8/
8/
84
6%
866
E
Eg
By
610
E11
612
P13
$14
E1s
E16
(8)
(b)
Figure 4: Visualization of temporal attention mask for = h 16 spatial positions from (a) Event-Slott Slots branch and (b) Temporal-QFormer-
Qoormer- VLM, respectively. For simplicity, we also refer to slot as query here. For the k-th spatial position, we denote the of learned temporal queries by & Take the 13-th spatial position of the query set E13 as an example (as marked by red box in (a) and blue box in (b)). For this spatial position, the models generate N₂ 8 slots/queries by aggregating the temporal visual tokens. The attention masks for &13 are denoted by a map of rows and N columns, with the visibility indicating which queries this temporal position belongs The higher the visibility, the greater the affinity between this temporal position and the query. We can see that in our Slot-VLM,, similar contents tend to be allocated the same slot, i.e., different slots capture different contents (events) and present decoupled semantics. In contrast, in Temporal -orralrrrrrrrrrrrrr VLM, different contents are usually assigned to the same query or are uniformly assigned to different queries. Note that for Temporal- -oorrerr-rrrrrr---r VLM, we only show the mask of one head to save space, where similar observations can be found from other heads. glimpse of the original video can be found Figure See Figure T0 for the enlarged visualization of E13-
Figure 5: A glimpse of the original video used in Figure 4 For visualization purpose, we only show the frames down-sampled at factor of 8, which is 1/8fps.
x w = 16 positions). Similarly, Figure 4bb visualizes the temporal cross-attention masks from Temporal- QFormer-VLMM for indicating the forming of query tokens for each spatial position. We Temmporl--QForme--VLee
can see that in our Slot- VLM, along the temporal axis, similar contents tend to be allocated to the same slot. In other words, different slots capture different contents (events) and present decoupled semantics. In contrast, in the Temporal-QFormer -iFormer-VLMM different contents are usually assigned to the same query or are uniformly assigned to different queries. The queries learned from Q-Former do present decoupled semantics. Figure 5 shows a glimpse of the original video used in Figure
5 Conclusion
In this work, we introduce new framework, Slot-VLM, that aims to generate a small set of semanti- cally decoupled video tokens comply with LLM for effective video reasoning. Particularly, we design a branch Object-Evee Event Slots module to learn object-centric slots and event-centric slots to jointly capture the spatial object details and temporal dynamics. These semantic-cenin slots are taken as the input to LLM for effective video reasoning. Experimental results demonstrate the superiority of our framework and show the effectiveness of using semantics-decoupled representations for aligning with LLM. However, our current representations are still not perfect where object instances and events are not ideally segmented. We anticipate this work will inspire more investigations towards semantic-centrie visual representations for video-language modeling.
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-2377 227366, 2022.
[2] Ziyi Bai, Ruiping Wang, and Xilin Chen. Glance and focus: Memory prompting for multi-event video question answering. Advances in Neural Information Processing Systems, 36, 2024.
[3] Moshe Bar. Visual objects context. Nature Reviews Neuroscience, 5(8):617-629, 2004.
[4] Apratim Bhattacharyya, Sunny Panchal, Reza Pourreza, Mingu Lee, Pulkit Madan, and Roland Memisevic. Look, rememher and reason: Grounded reasoning in videos with language models. In The Twelfth International Conference on Learning Representations.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
[6] Christopher Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint 1901v.1901.113900 113990 2019.
[7] Fabian Caba Heilbron. Victor Escorcia. Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970, 2015.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe-sup vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660), 2021.
[9] David Chen and William B Dolan. Collecting highly paraliel data for paraphrase evaluation. In Proceed- ings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 190-200, 2011.
[10] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09477 2023.
[11] Wei-L in Chiang, Zhuohan Li, Zi I. Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* ChatGPT quality, March 2023.
[12] W Dai, Li, D Li, AMH Tiong. Zhao, W Wang, B Li, P Fung. and S Hoi. Instructblip: Towards general- purpose Asion-languuae models with instruction tuning. Advances in Neural Information Processing Systems, 2023.
[13] Christoph Foichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.. Slowfast networks for vidco recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6202 6211, 2019.
[14] Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou. Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14773-14783, 2023.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[16] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06833 2023.
[17] Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08044 2023.
[18] Kumara Kahatapitiya. Kanchana Ranasinghe, Jongwoo Park. and Michael S Ryoo. Language repository for long video understanding. arXiv preprint ArXiv:2403.146)) 14622, 2024.
11

---

[19] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-ingn pre-training with frozen image encoders and large language models. arXiv preprini arXiv:2301. 125977 2023.
[20] KunChang Li. Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang. Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06335. 2023.
[21] Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video understanding benchmark. 2023.
arXiv preprint arXiv:2311.1700
[22] Yanwei Li, Chengyao Wang. and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, arXiv preprint 1xXiv2331..7044 2023.
[23] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video- Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023.
[24] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 2023.
[25] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 2023.
[26] Ruyang Liu,, Chen Li, Yixiao Ge, Ying Shan, Thomas H Li, and Ge Li. One for all: Video conversation is feasible without video instruction tuning. arXiv preprint arXiv:2300. 15785, 2023.
[27] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskly, and Thomas Kipf. Object-centric learning with slot attention. Advances Neural Information Processing Systems, 33:1 13:11525-1151 1538,, 2020.
[28] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.00422 2023.
[29] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36, 2024.
[30] Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid. Verbs in action: Improving verb understanding in video-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15579-15599 2023.
[31] OpenAL. Chatgpt: Large language model for human style conversation. hntpps:/ccaat oww.ai............... com 2023.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[33] Kanchana Ranasinghe, Xiang Li, Kumara Kahatapitiya, and Michael S Ryoo. Understanding long videos in one multimodal language model pass, arXiv preprint arXiv:2403. 1699.8 2024.
[34] Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature nearoscience. 2(1 101199111- 1999.
[35] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf. Thomas Brox, et al. Bridging the gap to real-world object-centric learning. arXiv preprint MiXX::2209118664 2022.
[36] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E learns to compose. International Conference on Learning Representations, 2022.
[37] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE Conference Computer Vision and Pattern Recugnition, 2024.
[38] Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888 11898, 2023.
12

---

[39] Hugo Touvron. Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[40] Yi Wang. Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Intervvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942 2023.
[41] Wang, Kunchang Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.031191, 2022.
[42] Bo Wu. Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world vidcos. arXiv preprint 2xXiv:205.09711 2024.
[43] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question question-answeriinn answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9777-9786, 2021.
[44] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition, pages 5288 5296, 2016.
[45] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Yc, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304. 14178, 2023.
[46] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained amage-language model video localization and question answering. Advances in Neural Information Processing Systems, 36, 2024.
[47] Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang. Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple Ilm framework for long-range vidco question-amse answering. arXiv preprint arXiv:2312.17235,, 2023.
[48] Hang Zhang, Li, Lidong Bing. Video-LLaMA: An instruction-tume -uuned audio-visual language model for vidco understanding. arXiv preprint arXiv:2336.028855 2023.
[49] Hang Zhang, Li, Lidong Bing. Video-LLaMA: An instruction tuned audio-visie visual language model for video understanding. arXiv preprint arXiv:2306.0288 02858, 2023.
[50] Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen. Shuai Shao, Xinxin Zhu, Zehuan Yuan, Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint arXiv:2305. 16103, 2023.
[51] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv: 10592, 2023.
13

---

A More Details about Object-Slots Modeling
As shown in Figure of the Object-Slots branch, for video of H W x ' dense video tokens, we sample the features at low frame rate but high (original) spatial resolution to obtain H x x video tokens. denotes the number of uniformly sampled frames, which we set to 8 by default. In our framework, H = W 16. For the i-th frame, we have a set S, of M = Н x W tokens. S; is taken as the input to slot attention module 27 35 to generate No object-centric slots O {o.,1, oi,N. (we also refer to them spatial slots).
Slot attention uses iterative mechanism map from the input tokens to the slots. At each iteration, slots attention uses cross attention with attention coefficients Am,a that are first normalized over the slots
Manr Eim Mmm.
Am.
M = D MS.) . (Q.)" E R MXN.
(1)
Such normalization over the slots introduces competition among the slots for promoting the forming of decoupled representations. To aggregate the input tokens their assigned slots, weighted mean is used as follows for updating slots as:
c = .((.) NNXD, Wm.n
Amn ΣEi
k()). (()), and v(-) are Icarnallc linear transformations. Gated Recurrent Unit (GRU) and MLP parameterized by hig is used to further updated the slots Q, := h Q., 2.). I. such iterations are performed to obtain the final N, slots for this frame. Note that we initialize the slots from learnable Gaussian distributions. L such iterations are performed to obtain the final N,, slots for this frame. Note that we initialize the slots from learnable Gaussian distributions.
To distinguish different frames, we add learnable temporal position embedding pa to each slot of the i-th frame and obtain the updated object slots as O fou........ ON,N, J. where oL, == o. +p. We concatenate the slots of all the frames and obtain x No slots O {O1. ..... O.} {01.1, 01.2 O1,No Or.N, }. A linear projection layer (S- Proj.) transforms each token to facilitate the alignment with slots from the Event-Slots branch and the alignment with the LLM.
Note that the slot attention mechanism for learning event-centric slots is similar to that for learning object-centric slots and we will no longer elaborate on that.
B More Implementation Details about Training
We use three stage training to better optimize the model. Stage aims to pre-train the slot attention modules (with the objective of reconstructing the input features) to facilitate the learning of object-centric and event-centric slot representations, i.e., the forming of semantically decomposed tokens/slots. Stage 2 separately trains the Object-Slots branch and the Event-Slott branch to facilitate the system focusing the optimization of each branch separately, which eases the optimization. Stage jointly optimizes the two branches and the projection layer. All models are trained using a single NVIDIA A 110 80GB GPU. Adam used as the optimizer.
Stage We first pre-train the two separately. This promotes the formation of object-centric and event-centric slot representations. Similar to DINOSAUR [35]. we use transformer decoder to reconstruct features (rather than images) obtained by CLIP ViT-L/14 by taking the outputted slots (ahead the projection) from Slot-Attention input.
modules of the Object-Slots branch and the Event-Slots branch
We train the models for 60 epochs with learning rate le-4. We use the Slot Attention module in SLATE The dimension of each slot set to 1024. Following (36), the number of iterations is set 3. We use the vidco frames from the 100K Vidco Instruction Data (28 for training. Position embeddings in the Event-Slots branch are discarded during the pre- pre-traning to case the training.
Stage 2 We perform instruction tuning on the single branch schemes, separately, For example, we fine-tune the Object-Slots branch and the projection layers using the instruction pairs from 28). by loading the parameters from the first stage. We use the 100K Video Instruction Data from 28 to train our single branch schemes, respectively. We set the number of epochs to 3.
The Object-Slots 2e-5. We adopt the cosine annealing learning rate. We set the batch size to 40 and train on a single A100 GPU.
branch and the projection layer Proj. are tuned. We set the learning rate to
Stage 3 We fine-tune the BloStots module and the projection layer using the instruction pairs, by loading the parameters from OE-Slots module from the second stage. By default, we use the Video Instruction Data for instruction wningg We set the number of epochs to 3.
14

---

Table 4: Comparison with the state-of-the-arr methods for video QA in terms of the used number of video tokens. 'Varied length' denotes the number varied with the video length. '-' denotes we did not find information from their papers. denotes the likely number based the paper's description
Model
# of vidco tokens
Video LLaMA
Video Chat 20
Video-ChatGPT
28
356 Varied length
Chat-UniVi 17
Video-LLaVA 23
Video-LLaVA 23
BT-Adapier 26
256 Varied length 1536*
LLaMA-VID 22
VideoChat2 21
MovicChat 37
Slot-VLM (Ours)
192
Slot-VLMM (Ours)
192
When incorporate 665K image-text pairs for training "LMM,, use 665K image-text pairs and 100K video-text pairs jointly to train the framework. When the input is an image-text pair, the Event-Slots branch dropped in training. With the increased training data, we set the number of epochs to
The Object-Slots branch/Event-Sue branch and the projection layer Proj. are tuned. The training hyper- parameters used in this stage are the same as those in the second stage.
Note that our three-stage training is easy to implement. Previous methods usually also use multiple stage training. For example, VideoChat2 (2I] adopted a progressive three stage training using much more data than ours. LLaMA VID 2
Mideo-LLaVA 23 Chat UniVi 1 all adopted two stage training.
The numbers trainable parameters in the three stages are 27M, 37M, and 33M, respectively. The three stages training our Moo model takes about 15, 12, and hours respectively. The three stages training of our SLot-VLM' model takes about 15, 12, and 46 hours respectively.
C More Details about Evaluation Metrics
In our study the open-ended video question answering, adopt the evaluation metrics of accuracy and average score as established by (28). assess accuracy our model's predictions, we utilize ChaiGPT (chatgp(35-turbo) as an evaluator. ChatGPT processes each question alongside the corresponding ground truth and the model's predicted answer. It then provides binary "yes" or "no" judgment on the correctness of the prediction for accuracy assessment. Additionally, ChatGPT assigns integer score ranging from 0 to to quantify closeness the predicted answer to ground truth, with indicating no similarity and 5 denoting a close match.
D Number of Video Tokens Used by Different Models
Table shows the performance gain is not due to the final number of video tokens input the LLMs, where Slot-VLM uses only 192 tokens that is much smaller than some other models.
E
More Ablation Studies
Influence of High Spatial Resolution for the Object-Slots Branch and High Frame Rate for the Event- Slots Branch We study the influence spatial resolution the visual features in the Object-Slots branch and the influence frame rate the visual features the Event-Slots branch, respectively. As shown Table 5 when we reduce the spatial resolution from 16 x 16 to 4 x 4 for the Object-Slots branch, the performance drops by 4.6% 33%% on In-domain/MSVDA. VDD-A/AMVVDDAVA in accuracy. When we reduce the frame rate by factor of 8 from ffs 1/8fps for the Event-Slots branch, the performance drops by 4.3% on In-domain/MSVD-O.. in accuracy.
Influence of Hyperparameter N. We study the influence the number object-centric slots N. of frame for Object-SUl Slot- VLM and show the results in Table We can see that as the increase of the number of slots from 4 to 16 for a frame, the performance increases and saturates when N, = 8 in terms of score in In do-momiin. Therefore, by default, we set N. = 8.
15

---

Table 5: Ablation study on the influence of high spatial resolution for the Object-Slots branch and high frame rate for the AieentSSloss branch. Object-Slor Srooct-Sloo--oee VLM (4 x 4) denotes the spatial resolution is reduced from 16 x 16 to x Vvent-Slot-VILM (7/8) denotes the frame rate is reduced by a factor of 8.
In-domain
MSVD-QA
Model
Acc.
Score
Acc.
Score
Object-Slot- Slotet--MM (4x4)
41.9
2.53
69.8
3.55
Object-Slots branch
Object-Slot- VLM
46.5
2.69
73.1
3.71
Event-Slot- VLM (178)
42.9
2.59
71.6
3.63
Event-Slots branch
Fvent-Slot-VIl
47.1
2.67
73.1
3.67
Table 6: Ablation study the influence of the number of object-centric slots No and the influence of the number of event-centric slots Ne, respectively.
In-domain
MSVD-QA
Model
Acc.
Score
Acc.
Score
Object-Slot-VInn (4slots)
42.6
2.54
72.7
3.67
Object-Slots brunch
Object-Slot-VIl VLM (8slots)
46.5
2.69
73.1
3.71
Object-Slot-VLi VLM (16slots)
47.7
2.69
72.9
3,70
Event-Slot- VL.M (4slots)
46.4
2.66
74.2
3.69
Event-Slots branch
Event-Slot- VLMM (8slots)
47.1
2.67
73.1
3.67
Event-Slot- VL.M (16slots)
46.3
2.68
74.4
3.69
Influence of Hyperparamettrr N₂ We study the influence the number event-centric slots N. of spatial position for Srent-Slot-VLLN and show the results in Table We can see that as the increase of the number of slots from 4 to 16 along the temporal direction, the performance increases in terms of score in In- In-domain and achieves good performance when N. 8. For simplicity, we set N. = by default.
Influence of Frame Sampling Rate in the Object-Slots Branch and Stride for Pooling in the Event-Slots Branch We conducted ablation studies using single branch settings, For the Object- Slots branch, we tested three frame sampling rates: 4 frames (Ohject-Slot- Mijeect--lotVVLMMM VL.M (4 frames)), frames, and frames per video. For the Event-Slots branch, we tested four pooling strides to achieve spatial resolutions of 2x2 (Event-SSot-Vil VL. (2x2)), 4x4, and 8x8.
Table 7 shows the results. Increasing the sampled frames or spatial resolutions improves performance but increases the number of video tokens and the computational cost. By default, we use sample frames and 4x4 spatial resolution to balance complexity (192 tokens total) and performance.
Table 7: Ablation studies on sampling different number of temporal frames for the Object-Slots branch, and on using different spatial resolutions for the Event- branch
In-domain
MSVD-QA
Model
# video tokens
Acc.
Score
Acc.
Score
Object-Slot Sooet-SlotVLLM (4 frames)
42.1
2.55
72.4
3.65
Object-Slots branch
Object-Slot- VLM Slo--VLMMM (8 frames)
64
46.5
2.69
73.1
3.71
Object-Slot-Ve M (16 frames)
128
46.8
2.7
74.3
3.72
Event-Slot-V LLM (2x2)
32
39.3
2.44
70.9
3.55
Event-Slots branch
Event-Slot- VLM (4x4)
128
47.1
2.67
73.1
3.67
Event-Slot-N -LM (8x8)
512
50.4
2.79
76.6
3.77
Influence of Three Stages Training Stage 1 aims pre-train the slot attention modules (with the objective of reconstructing the input features) to facilitate the learning of object-centric and event-centric slot representations, i.c., the forming of semantically decomposed tokens/slots. Stage separately trains the Object-Slott branch and the Event-Slots branch to facilitate the system focusing on the optimization of each branch separately, which eases the optimization. Stage 3 jointly optimizes the two branches and the projection layer. Table the ablation study the influence of each training stage. We have the following observations/conclusioi only one joint training stage as Model-1, the performance drops by 1.8% and 2.4% in accuracy In-domain and MSVD-QA QA,, respectively. When compared with other methods as shown in Table L on MSVD-QQ QA. our Model-1 with one joint training stage achieves 72.5% in accuracy, ranks the second best among all other methods. It is still effective over many other methods but is less effective than our final scheme with three stages training. 2) Skipping either Stage I or Stage leads to performance drop.
1) If there
16

---

Table Ablation study on the three-stage training on In-domain and MSVD-QA. "No" denotes that this stage training is not used while "Yes" denotes that this stage training is used. We report the accuracy (%) and score.
In-domain
MSVD-QA
Models
Stage
Stage 2
Stage 3
Acc.
Score
Acc.
Score
Model-1
No
Yes
46.9
2.69
72.5
3.67
Model-2
No
Yes
Yes
74.6
3.76
Model-3
Yes
No
Yes
44.2
2.64
73.5
3.69
Model-4 (final)
Yes
Yes
Yes
48.8
2.75
74.9
3.76
Table 9: Performance (accuracy) comparrisn on the subset of Egoschema. ViperGPT[38]
Method
Sevila
Video-LLaVA22 mPLUG-OW145)
LLoVi
Accuracy
15.8
25.7
36.8
33.8
50.8
55.8
F
Comparison with the State-of-the-Arts on Multi-choice datasets
We also evaluate our models on multi-choice QA benchmarks, including Egoschemal 29 NEXT-QA 43 and STAR[42)
Egoschema. For EgoSchema, have tested the performance the 500-question subset and Table shows the results. Mlot-VLM significantly outperforms Sevila by 30% and outperforms the comparable model Vidco-LLaVA by 19% accuracy.
NEXT-QA. For NEXT-QA, Table shhows
shows our Slot-VLM VLM competitive to VideoChat2, We believe using more training videos like VidcoChat2 (1.1 million) would further enhance performance.
STAR, For STAR, Table 11 shows the comparisons. Different from 14 2 46 4. our model is tested in zero shot manner, without accessing STAR during training. Our Slot-V VLM achieves competitive performance with generalizable models of Flamingo-9B and Intern Video. Note that Intern Video uses much larger number of videos (12 million) than ours, whereas our model uses only 100K videos for training.
G More Visualization
G.1 Visualization of Q-Former Attention Maps from BLIP2
BLIP2 19 is representative visual language pre- me-trained model that aggregates image features (from CLIP encoder) by Q-Former into 32 learnable query tokens. Q-Former employs a 12-head tross-attention mechanism aggregate visual features into set of queries, which subsequently processes queries by self-attention module and a feed-forward layer. We analyze whether such well-trained Q-Former has learned decoupled semantics for the 32 query tokens. Figure visualizes the learned cross-attention masks (laid out on the original image) corresponding to the 32 queries (32 columns, q qh.2, 32.32) from 12 heads (shown in 12 rows, h 1, 12) for images (a) and (b) respectively. We for different heads. For head, usually information was allocated into only a queries. However, there no obvious evidence that different queries have learned decoupled semantics. In contrast, as shown in Figure our learned spatial slots have more remarkable decoupled semanties.
that query focuses different regions
G.2 Visualization of Q-Former Spatial Attention Maps from Spatial- VLM
VLM. Q-Former uses cross-attention of 12 heads to aggregate the visual features. In Figure [ we show the masks for 12 heads separately as shown in the x 1 grids separated by green lines. For each head, we show all the 8 frames in different rows; the first column shows the original frame; the second the ninth columns show the cross attention mask (from Q-Former) with each one being a mixture of spatial features. A feature in a spatial position is usually allocated to a couple of queries instead of one. Compared with the spatial attention masks from our Slot- VLM as shown in Figure it is less semantically decoupled for the learned queries.
We visualize the spatial attention maps the learned queries of Q-Former from our
queries. We can see that the queries are not clearly decoupled,
17

---

Table 10: Performance (accuracy) compari-
Table 11: Performance (aceuracy) compari-
son on NEXT-QA.
son on STAR.
Dataset specitic trained (In domain)
Accuracy
Dataset specific trained (In domain)
Accuracy
14
57.2
14
51.1
GF(uns) 2
58.8
GF(uns) 2
53,9
Sevila 46
73.8
Sevila 46
64,9
Zero-shot (Gieccralization)
Accuracy
LRR
70.5
Vidco|1
49.1
Zero shot (Generalization)
Accuracy
Mistral(7B)
51.1
Flamingn-9BI
41.8
VFC 30
51.5
Intern 41
41.6
LLoVi
54.3
Slot-VLM
42.7
MVU(13B)
55.2
ViperGPT
60.0
LangRepo(12B)) 12B)
60.9
VideoChatt[[1
61.7
Slot- VI.M
62.0
G.3 Visualization of Spatial Attention Maps for Spatial Slots from Stage 1 and Stage 2
In the first stage pre- pre-training, slot attention is learned by reconstructing the features. In the second stage. LLM is incorporated for video instruction tuning, Here, take spatial slots examples compare the slot attention masks from stage and stage 2. Figure @ and Figure show the visualization for two examples, respectively, Interestingly, we observe that after the instruction tuning, the learned slots are better decoupled, where spatial position usually contributes to multiple slots in stage but only contributes to a very few slots in stage 2. Similar phenomenons are obtained for temporal slots.
G.4 Visualization of Learned Event-Centric Slots
Figure 10 shows the enlarged visualization of attention masks for the 13-th spatial position that is previously shown Figure
H Discussion on the Application on Long Video Understanding
For everal-minute-long videos, Slot-VI MLM can still capture important objects and events information for video understanding. Experiments on the Aetivity dataset, where the average length of video clips is 111.6 seconds, and the maximum length is 285 seconds, show the superiority of our Slot-VLM over many other methods (see Table note that the amount of instruction tuning data much smaller than that in VideoChat2). scenes with more objects, some similar objects would be gathered into the same slot. Even not being perfect, our semantics decoupled representations presents its superiority to pooling- based and Q-former based strategics where semantics are mixed in the generated tokens for vidco understanding.
For much longer vidcos (c.g., hour-long vidcos), it would be difficult to represent the dynamic scenes with only a small set of event slots. We could extend our framework to well serve long video understanding by partitioning the long-video chunks of shorter duration, where the object-centric and event-centric slots generated for each chunk and these slots from chunks are concatenated sequentially as the input LLM. We leave this as future work.
I Limitations
In this work, we introduce a new framework, SLot-VLM, that aims to generate small set of event-centric video tokens to comply with LLM for effective video reasoning. Currently, leverage slot attention leam object-centric slots and event-centric slots. However, the learned slot representations still not perfect where object instances are still not perfectly segmented and events are not perfectly partitioned. With such imperfect slot representations, our framework achieves the state-of-the the-thh-rt performance, demonstrating strong potential. We believe the advancement of unsupervised object-centric representation learning in future could further enhance the performance of our framework.
For much longer videos (e.g., hour-long videos), would be difficult represent the dynamic only a small set of event slots. Our framework extensible support those very long videos but we have not implemented and experimented on such cases. For example, for a three-hour-loo hour-long video, we could partition the video into chunks with each chunk being seconds and summarize dense video features into sparse N, object-centric and Ny event-centric tokens for each chunk and concatenate the tokens of chunks sequentially
with
18

---

9h12 0h,20aa
0b:32
3
12
9h,32
1
2
6
11
(b)
Figure 6: Visualization of spatial attention masks from the Q-Former BLIP2 for two images in (a) and (b) respectively, We show the learned query masks for the 12 heads in 12 rows, respectively. In each row, we show the masks for the 32 queries. Note that the first column show the original image repeated by 12 times. There is no obvious evidence that different queries have learned decoupled semantics.
the input LLM for inference. This is feasible in practice by borrowing experience from LLaMA-VID [2]] which designed for long-form video understanding. LLaMA-VID encodes frame into two tokens at a sample rate of frame per second to support three-hour-long videos. In our framework, when we have N. + N, 192 tokens for each chunk, by setting the duration of chunk have tokens per frame average, facilitating the handling of nour-long-vides as LLaMA-VID does.
96 seconds, we equivalently
J Impact Statements
This paper endeavors to make notable advancements in video-language modeling, field integral to enhancing interaction with and comprehension of video content. We aspire drive positive innovation, yet remain aware that technological progress may have unforeseen side effects. We upon users use Al responsibly. When we release code and models, we would set a safeguard by requiring that users adhere to usage guidelines.
19

---

Figure 7: Visualization of spatial attention masks from our baseline scheme for 8 frames of video. Since there are 12 heads in Q-Former, we plot 12 x 4) sets of attention masks separated by green lines. For each head, we have / = 8 frames as shown in each row; the first column shows the original frame; the second to the ninth columns show the cross attention mask (from Q-Former) for the 8 queries. We can that the queries are not clearly decoupled, with each one being mixture of spatial features. A feature in spatial position is usually allocated to couple of queries. Compared with the spatial attention masks from our LLM as shown in Figure 3 is less semantically decoupled for the learned queries.
-aern-FFooeerrrreeee VLM
O1 0,2
OL
01,4
0
0,,6
0,,7
OLB
0,2
0LA
01.5
0
0,,
3
4
6
(a)
(b)
Figure 8: Visualization of spatial attention masks from (a) the stage I pre-training, and (b) the stage after instruction tuning. We have t frames as shown 8 rows, indexed by i, where = 1..... .t. respectively. The first column shows the original frame. The second to the ninth columns show the cross attention mask (from slot attention) for the N. = object-centric slots O, = {o..1..... ON,N. Interestingly,
that after the instruction tuning, the learned slots much more decoupled, where a spatial position usually contributes to multiple slots in stage but only contributes to a very few slots in stage 2.
20

---

O1,1
0
LL
0
015
016
0L,7
OLS
01,1
0
01,3
01.5
01,6
0
0,,8
2
4
(a)
(b)
Figure 9: Visualization of spatial attention masks from (a) the stage I pre-training, and (b) the stage after instruction tuiigg We have respectively. The first column shows the original frame. The second to the ninth columns show the cross attention mask (from slot attention) for the N, object-centric slots O = (o..1..... O..N. Interestingly, we can see that after the instruction tuning, the learned slots are much more decoupled, where spatial position usually contributes to multiple slots in stage but only contributes to very few slots in stage 2.
frames as shown rows, indexed by i, where 1,....4. I,
21

---

E E3
E3
Time
Figure 10: Enlarged visualization of temporal attention mask for the 13-th spatial position from (a) our Event-Slots branch and (b) Temporal-QFoommrr- VLM,, respectively. For simplicity, we also refer to slot query here. For the 13-th spatial position, denote the set of learned temporal queries by E13. For this spatial position, the models generate Ne = 8 queries by aggregating the temporal visual tokens. The attention masks for E13 are denoted by map of I rows and N. 8 columns, with the visibility indicating which queries this temporal position belongs to. The higher the visibility, the greater the affinity between this temporal position and the query. We similar contents tend to be allocated to the same slot, i.e., different slots capture different contents (events) and present decoupled semanties. In contrast, in Cimporal-QFormer-- are usually assigned to the same query or are uniformly assigned to different queries.
that in our Slot-V LLM,
different contents
22

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We aim to generate semantic-cenee centric tokens from video features to better comply with LLMs for effective video- video-language modeling, providing new insight that 'disentangled" tokens are beneficial for ladeu-lng modeling.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Appendix !
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (c-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to- peech-to-texx text system might not used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
applicabll, the authors should discuss possible limitations their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: There is theory in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and wros--referenced.
23

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 4 and Appendix [B have presented necessary information to reproduce the main experimental results of the paper. The used datasets are all public datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments. No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing model checkpoint, or other means that are appropriate to the research performed.
example, contribution
While NeurIPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (c.g.. with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes].
Justification:
We will release the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines btttpe://nips.cc/ppil cc/public/
Galdes//CoeeSubni htteee//oooeuuuaissiioooooo.....
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : //nnps//......///// cc/public/ out.dess/CooeeSttmieesssooooooo for more details.
24

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters. how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We have described the necessary information in Section Appendix and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Following other papers in this field, we did not report emror bar to save the computational cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please see Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
25

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies
Answer:
[Yes]
Justification: have read the NeurlPS Code of Ethics and confirm that we follow that.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e-g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a dircct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feodback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please see Section | When release code and models, would set safeguard by requiring that users adhere to usage guidelines.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use usel-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
26

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly eredited and are the license and terms of usc explicitly mentioned and properly respected?
Answer: [Yes]
Justification: have cited original papers of the datasets. We have checked licenses of used datasets and followed them.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperewi thcode. cwm/ca////aa//oc daaa/dattstt has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country of data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
27

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
28


Reviews:
[{"review_id": "Jx8Tnmv9Am", "reviewer": "Reviewer_jNDE", "summary": "This paper proposed Solt-VLM, which aims to generate a small set of semantically decoupled video tokens to comply with LLM for effective video reasoning. It design a dual-branch Object-Event Slots module to learn object-centric slots and event-centric slots to jointly capture the spatial object details and temporal dynamics. Experimental results demonstrate the superiority of our framework and show the effectiveness of using semantics-decoupled representations for aligning with LLM.", "questions": "please see weakness", "limitations": "authors provide limitation discussion in the paper", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the main contribution of this paper is introducing slot attention as the multimodal connector between LLM and visual encoders. Even though it is more like a solid technical implementation based on previous works, I think it is a good try on the video-language domain and provides more insights into spatial-temporal modeling in MLLM. Also, the paper writing is good and the main motivation makes sense, the results show both efficiency and effectiveness.", "weaknesses": "- The proposed Slot-VLM is tested on open-ended MSRVTT-QA, MSVD, and ActivityNetQA with automatic ChatGPT-based metrics. Although the metrics is applied by some previous works, I don’t think this is a stable, reliable, and explainable enough evaluation which may vary a lot across GPT versions (check conclusion in [1]) and may generate different responses even for the same input. \n- Also MSRVTT-QA/MSVD/ActivityNetQA is not designed from an object interaction perspective, which may not be effective enough for validating the object-centric design. \n- Based on the previous two points, I would suggest testing on recent multi-choice QA benchmarks, including STAR [2] / NExT-QA [3] / Egoschema [4], which are annotated with clear answer choices, and designed from a human-object interaction view. And compared with recent model works like Sevila [5],  MIST [6], LRR [7], GF [8], which are also based on query-former, or iteratively attention on regions in the video.\n\n[1] FreeVA: Offline MLLM as Training-Free Video Assistant, Arxiv24.  \n[2] Star: A benchmark for situated reasoning in real-world videos, NeurIPS 2021.  \n[3] Next-qa: Next phase of question-answering to explaining temporal actions, CVPR 2021.  \n[4] Egoschema: A diagnostic benchmark for very long-form video language understanding,  NeurIPS 2023.  \n[5] Self-chained image-language model for video localization and question answering, NeurIPS 2023.  \n[6] Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering, CVPR 2023.  \n[7] Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024.  \n[8] Glance and focus: Memory prompting for multi-event video question answering, NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback on our insights into spatial-temporal modeling in MLLM, the main motivation, and the effectiveness of method, as well as the quality of our writing. We have carefully considered your valuable comments and suggestions and are committed to incorporating them into our revised manuscript. Please find our detailed responses below.\n\n**Q1**: The proposed Slot-VLM is tested on open-ended MSRVTT-QA, MSVD, and ActivityNetQA with automatic ChatGPT-based metrics. Although the metric is applied by some previous works, I don’t think this is a stable, reliable, and explainable enough evaluation which may vary a lot across GPT versions (check conclusion in [1]) and may generate different responses even for the same input.\n\n**A1**: We acknowledge your comment and will include the comparisons on multi-choice QA benchmarks in our manuscript (see Response A3 below).\n\n\n**Q2**: Also MSRVTT-QA/MSVD/ActivityNetQA is not designed from an object interaction perspective, which may not be effective enough for validating the object-centric design. \n\n**A2**: Thank you for your comment and suggestion. We explored additional datasets for validation as discussed in Response A3 below. \n\nNote that our framework is expected to generally work well, since in general each image in a video is a capture of the world consisting of objects and backgrounds, where object-centric representations can be considered as the basic units. This can also be intuitively explained by human visual reasoning where object representation acts as a basic and fundamental representation. For humans, visual signals are processed initially through the primary visual cortex (V1) and subsequently integrated into higher-level visual processing areas, resulting in the formulation of complex visual object representations. Such high-level object representations together with brain-stored knowledge are then used for logical reasoning in brain. Similarly, our Slot-VLM generates object-centric and event-centric representations to provide the high-level vision source for effective LLM reasoning, where object-centric design would be generally helpful. The strong performance of our Slot-VLM (Table 1) also demonstrated the effectiveness of our design.\n\nIn addition, for the three datasets, we found that there are plenty of video-question pairs that are related to querying the states/actions of objects and relations. For example, what is a dog doing? What’s the shape of the table? What are the animals that appear in the video? What is behind the person sitting in the video? Our object-centric design provides an effective bridge for visual and language modeling.\n\n\n**Q3**:  Based on the previous two points, I would suggest testing on recent multi-choice QA benchmarks, including STAR [2] / NExT-QA [3] / Egoschema [4], which are annotated with clear answer choices, and designed from a human-object interaction view. And compared with recent model works like Sevila [5], MIST [6], LRR [7], GF [8], which are also based on query-former, or iteratively attention on regions in the video.\n\n**A3**: Thank you for the helpful suggestions and we will add the comparisons in our revision. \n\nFor EgoSchema, we have tested the performance on the 500-question subset and Table A-1 shows the results. Slot-VLM significantly outperforms Sevila by 30% and outperforms the comparable 7B model Video-LLaVA by 19% in accuracy.   \n\nTable A-1 Performance (accuracy) comparison on the subset of Egoschema. \n| Method   | ViperGPT[9] | Sevila | Video-LLaVA | mPLUG-Owl | LLoVi [10] | Slot-VLM |\n|:--------:|:------:|:------:|:-----------:|:---------:|:--------:|:--------:|\n| Accuracy | 15.8   | 25.7   | 36.8        | 33.8      | 50.8 | **55.8**     |\n \n[9] ViperGPT: Visual Inference via Python Execution for Reasoning\n \n[10] A Simple LLM Framework for Long-Range Video Question-Answering\n\n\nFor STAR, Table A-2 shows the comparisons. The referred four works [5-8] report the in-domain accuracy where training is performed on STAR. In contrast, without accessing STAR during training, our model is tested in zero shot manner. Our Slot-VLM achieves competitive performance with generalizable models of Flamingo-9B and InternVideo. Note that InternVideo uses a much larger number of videos (12 million) than ours, whereas our model uses only 100K videos for training.   \n\nTable A-2 Performance (accuracy) comparison on STAR. \n|                            |      |\n|:--------------------------:|:----:|\n| **Dataset specific trained (In domain)**   |      |\n| MIST [6]                       | 51.1 |\n| GF(uns) [8]                   | 53.9 |\n| Sevila [5]                    | 64.9 |\n| LRR [7]                       | 70.5 |\n| **Zero-shot (Generalization)** |      |\n| Flamingo-9B                | 41.8 |\n| InternVideo                | 41.6 |\n| Slot-VLM                   | **42.7** |\n\nFor NExT-QA, Table A-3 shows our Slot-VLM is competitive to VideoChat2. We believe using more training videos like VideoChat2 (1.1 million) would further enhance performance. \n\nTable A-3 Performance (accuracy) comparison on NExT-QA. \n|                            |      |\n|:--------------------------:|:----:|\n| **Dataset specific trained (In domain)**   |      |\n| MIST [6]| 57.2 |\n| GF(uns) [8]| 58.8 |\n| Sevila [5]| 73.8 |\n| **Zero-shot (Generalization)** |      |\n| InternVideo [11]| 49.1 |\n| Mistral(7B) [12]| 51.1 |\n| VFC [13]| 51.5 |\n| LLoVi [10]| 54.3 |\n| MVU(13B) [14]| 55.2 |\n| ViperGPT(GPT-3.5) [9]| 60.0 |\n| LangRepo(12B) [15]| 60.9 |\n| VideoChat2 [16]| 61.7 |\n| Slot-VLM| **62.0** |\n\n[11] InternVideo: General Video Foundation Models via Generative and Discriminative Learning\n\n[12] Mistral 7B\n\n[13] Verbs in Action: Improving verb understanding in video-language models\n\n[14] Understanding Long Videos in One Multimodal Language Model Pass\n\n[15] Language Repository for Long Video Understanding\n\n[16] MVBench: A Comprehensive Multi-modal Video Understanding Benchmark", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LxtnbLqzEg", "reviewer": "Reviewer_gzPw", "summary": "This paper introduces a new framework called Slot-VLM that aims to aggregate spatial and temporal relationships between frames for more effective video understanding with Large Language Models (LLM). The Slot-VLM approach aims to generate video tokens that are semantically disentangled for better alignment with the frozen LLM. Specifically, the paper proposes the Object-Slots module to extract object-centric slots from high spatial resolution features that are sampled with a low frame rate. Additionally, it also introduces the Event-Slots module, which helps to aggregate the temporal dynamics between frames by learning event-centric slots from features with low spatial resolution but sampled at a high frame rate. The authors demonstrate the benefits of their proposedSlot-VLM approach by evaluating across three open-ended video question-answering datasets including MSVD-QA and ActivityNet-QA, where it outperforms existing video-language models by a significant margin.", "questions": "Please see above-mentioned limitations.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The model figures are informative and especially helpful in helping the reader to understand the different stages of the learning algorithm as well as the intuition behind each stage. The slot attention approach, from the aggregation of video tokens to the combination of object and events-centric slots as well as their input to the LLM, is described well.\n\n2) While the introduced Slot-VLM bears strong similarities to existing work on hierarchical understanding of videos such as the SlowFast approach, the concept of generating semantically-disentangled video tokens to align with frozen LLMs is relatively interesting and novel. In contrast to existing video-language models which often rely on pooling or compressing information into learnable queries, the ability to generate interpretable attention maps is particularly helpful to visualize what the model is focusing on.\n\n3) In this paper, the authors conduct comprehensive comparisons with state-of-the-art video-language models. These evaluations highlight the limitations of existing work under relatively fair settings of using the same visual and language models as well as similar training data. The results also further emphasize the improvements brought by Slot-VLM.", "weaknesses": "1) The proposed Slot-VLM approach appears to be much more effective at aggregating disentangled spatial and temporal relationships between frames, as evidenced by the performance gap between its achieved results and those of Video-ChatGPT and Video-LLaVA in Table 1. However, it is unclear how much of the performance gain is also due to the final number of video tokens that are passed into the underlying LLM for reasoning and generation. It would be helpful to include a comparison of the different number of video tokens used among the different video-language models as well as the number of trainable parameters during each of the training stages.\n\n2) It is also unclear how scalable such an approach will be in handling much longer videos. Currently, it appears that 8 frames are used for the object-slots module. Along with using 8 event-centric slots in total across shorter videos, this may limit the performance of the proposed Slot-VLM approach on much longer videos, such as those used in the EgoSchema evaluation benchmark. Similarly, is there also a more efficient way to aggregate spatiotemporal relationships over more frames beyond just increasing the number of object and event-centric slots since this will increase the computational demand in the LLM?\n\n3) It may be beneficial to the reader to include additional ablation experiments and analysis over the low frame sampling rate used in the Object-Slots Branch as well as the stride used for pooling across the spatial and temporal axes in the Event-Slots Branch. Currently, these hyperparameters appear to be selected based on final downstream performance but there is hardly any discussion on them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive suggestions, and positive feedback on the novelty and interestingness of our proposed concept, the ability to generate interpretable attention maps, the comprehensive comparisons, and the clarity of presentation. The detailed responses are provided below.\n\n**Q1**: Include a comparison of the different number of video tokens used among the different video-language models as well as the number of trainable parameters during each of the training stages.\n\n**A1**: We will include these in our revision.  Table B below shows the performance gain is not due to the final number of video tokens, where our Slot-VLM uses only 192 tokens that is much smaller than some other models.\n\nTable B: The number of video tokens used. ‘Varied length’ denotes the number varied with the video length. ‘-’ denotes we did not find information from their papers and will investigate their code in future to fill.  * denotes the likely number based on the paper’s description.\n\n| Model                    | Number of video tokens |\n|:------------------------:|:-----------------:|\n| Video LLaMA              | 256               |\n| Video Chat               | -                 |\n| Video-ChatGPT            | 356               |\n| Chat-UniVi               | Varied length     |\n| Video-LLaVA              | -                 |\n| Video-LLaVA$^†$  | -                 |\n| BT-Adapter               | 256               |\n| LLaMA-VID                | Varied length     |\n| VideoChat2               | 1536*         |\n| MovieChat                | -                 |\n| Slot-VLM (Ours)          | 192               |\n| Slot-VLM$^†$ (Ours) | 192           |\n\n\nDifferent models have different numbers of training stages and trainable parameters per stage. Most papers lack detailed information. For simplicity, we present the trainable parameters for each stage of Slot-VLM and the likely numbers for VideoChat2 based on the paper's description in Table C. Our Slot-VLM uses far less trainable parameters than VideoChat2 and requires less computing resource in training. \n\nTable C: The numbers trainable parameters (M: million; B: billion) in each stage for VideoChat2 and our Slot-VLM. \n\n|    Stage    | Stage 1 | Stage 2 | Stage 3 |\n|:-----------:|:-------:|:-------:|:-------:|\n| VideoChat2  |   101M  |   492M  |    7B   |\n| Slot-VLM    |   27M   |   37M   |   33M   |\n\n**Q2**: It is also unclear how scalable such an approach will be in handling much longer videos. \n\n**A2**: **Our approach is directly applicable to videos of a few minutes**. We have validated our framework on ActivityNet-QA (see Table 1), where the average length of video clips is 111.6 seconds, and the maximum length is 285 seconds, which is comparable to the reviewer referred EgoSchema (180 seconds). \n\nWe also tested a 500-question subset of EgoSchema. Table D shows that our Slot-VLM achieves superior performance, which outperforms Video-LLaVA by 7% in accuracy. \n\nTable D: Performance (accuracy) comparison on the subset of Egoschema. \n| Method | ViperGPT | Sevila | Video-LLaVA | mPLUG-Owl | LLoVi | Slot-VLM |\n|:------:|:------:|:------:|:------:|:-----:|:------:|:------:|\n| Accuracy | 15.8   | 25.7   | 36.8     | 33.8  | 50.8 | **55.8**     |\n\n**For handing even longer videos, our framework can be extended.**  For example, for a three-hour video, we could partition the video into chunks, each of $\\tau$ seconds, summarizing dense video features into $N_s$ object-centric and $N_f$ event-centric tokens for **each** chunk. These tokens can then be concatenated sequentially as input to LLM for inference.  At the setting of $N_s+N_f=192$ tokens per chunk and $\\tau=96$ seconds (1fps) duration per chunk, we equivalently have 2 tokens per frame on average, facilitating hour-long video handling. Note that LLaMA-VID (Li et al., 2023b) also encodes each frame into two tokens but overlooks the exploration of temporal correlation (where ours outperforms LLaMA-VID in three benchmarks in Table 1). This strategy ensures that the total number of video tokens is proportional to the video length in a manageable and tolerable way. More efficient strategies for aggregating spatiotemporal relationships over longer videos will require further research and future efforts. \n\n\n**Q3**: Additional ablation over the low frame sampling rate used in the Object-Slots Branch as well as the stride used for pooling in the Event-Slots Branch. \n\n **A3**: We conducted ablation studies using single branch settings. For the Object-Slots branch, we tested three frame sampling rates: 4 frames (Object-Slot-VLM (4 frames)), 8 frames, and 16 frames per video. For the Event-Slots branch, we tested four pooling strides to achieve spatial resolutions of 2x2 (Event-Slot-VLM (2x2)), 4x4, and 8x8.\n\nTable E shows the results. Increasing the sampled frames or spatial resolutions improves performance but increases the number of video tokens. By default, we use 8 sample frames and a 4x4 spatial resolution to balance complexity (192 tokens in total) and performance.\n\nTable E: Ablation studies on sampling different number of temporal frames for the Object-Slots branch, and on using different spatial resolutions for the Event-Slots branch.\n\n| Model                       | # Video tokens | In-domain (Acc./Score) | MSVD-QA (Acc./Score) |\n|:---------------------------:|:--------------:|:----------------------:|:--------------------:|\n| Object-Slot-VLM (4 frames)  | 32             |     42.13/2.55            |   72.36/3.65      |\n| Object-Slot-VLM (8 frames)  | 64             | 46.5/2.69              | 73.1/3.71            |\n| Object-Slot-VLM (16 frames) | 128            |  46.82/2.70          |      74.29/3.72     |\n| Event-Slot-VLM (2x2)        | 32             |       39.28/2.44          |    70.86/3.55       |\n| Event-Slot-VLM (4x4)        | 128            | 47.1/2.67              | 73.1/3.67            |\n| Event-Slot-VLM (8x8)        | 512            |   50.35/2.79          |     76.56/3.77     |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oylg00NUEl", "reviewer": "Reviewer_wQQx", "summary": "This paper aims to learn a higher level of abstract representation as the input tokens to LLM. The paper proposes a dual-branch (object- and event-centric) to extract the concepts and uses the three-stage training paradigm. The proposed Slot-VLM is evaluated on three VQA datasets and has shown state-of-the-art performance.", "questions": "See weakness. The claim of event-centric slot is my main concern.", "limitations": "Yes, the authors raise the limitations of this paper in two directions: imperfect concept discovery and extension for hours-long videos. \n\nHowever, it would enhance the paper much more if the authors could provide more insights into what other semantic concepts the proposed model lacks, which would complement the existing object- and event-centric representations (Although I am not convinced the model actually captures the “event” semantic)", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The idea of decomposing the semantics of input tokens to LLM is interesting and crucial for human-like reasoning.\n- The evaluation results show strong performance on three datasets (Table 1) and better efficiency (4.4), which validates that such structured representations can improve the performance of VQA.\n- The extensive ablation study (Section D) is meaningful and comprehensive enough.\n- The details of reproducing the method are comprehensive.", "weaknesses": "My main concern is that the authors claim the temporal branch of Slot-VLM as the event branch. The slot attention focuses on local regions temporally, which makes it only capture low-level local motion instead of actual events with high-level semantics. Moreover, the visualization of temporal attention masks (Figures 8 and 10) can barely provide cues that the model can learn events-centric representation (L53, L55, L65). Thus, I believe the event-centric slot is overclaimed in the entire article. I hope the authors can clarify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback and helpful suggestions. We greatly appreciate your recognition of the interestingness and cruciality of the idea, strong performance, meaningful and comprehensive ablation study, and comprehensive reproducibility details. We have carefully considered your valuable suggestions and comments and will incorporate them into our revised manuscript. Please find our detailed responses below.\n\n**Q1**: My main concern is that the authors claim the temporal branch of Slot-VLM as the event branch. The slot attention focuses on local regions temporally, which makes it only capture low-level local motion instead of actual events with high-level semantics. Moreover, the visualization of temporal attention masks (Figures 8 and 10) can barely provide cues that the model can learn events-centric representation (L53, L55, L65). Thus, I believe the event-centric slot is overclaimed in the entire article. I hope the authors can clarify this.\n\n**A1**: Thank you for your constructive comments. We will revise the descriptions for clarity. Our intention is to conceptually align event slots with events at a high level, as defined in lines 203-204 of the manuscript: \"**Temporal slots aggregate the temporal tokens for explaining parts of the input, similar to identifying events. Thus, we name the learned slots as event-centric slots.**\" More precisely, we leverage temporal slots to aggregate semantically consistent temporal segments, akin to how spatial slots aggregate spatially consistent regions, termed object-centric slots. While these temporal segments within a slot might not form a complete event, they are likely part of the same event.\n\n**About local region**: We understand your concern regarding the focus on local regions. This was a careful consideration in our design process. To alleviate this, instead of learning event slots for each spatial position along the temporal dimension, we divide the feature map into 4x4=16 non-overlapped spatial regions (downsampling), where each region corresponds to a large local region of 56x56 pixels when the frame resolution is 224x224. Observing the temporal dynamics of such a large local region (56x56 pixels) allows for partial inference of event evolution within the region, although it may not be perfect. Note that the temporal aggregation through slots is performed on CLIP features, which possess high-level semantics, thus enhancing the slots’ ability to capture high-level semantics. Moreover, by utilizing the learned slots for all 16 large local regions, the LLM can achieve a global perception of temporal dynamics.\n\n**About visualization**: We recognize that the current technique using slot attention for forming objects (spatial) and action/event (temporal) segmentation are not yet satisfactory. In Figure 10 (a), the good news is that similar contents tend to be allocated to the same slot and different slots capture different contents. The patterns from slots (Figure 10 (a)) are much better than those obtained from Q-Former (see Figure 10 (b)). We acknowledge that our work is still far from forming idea events, but we have taken a small step towards that goal. The superior performance demonstrates the strong potential of our proposed idea of decomposing the semantics of video tokens to align with LLM. More joint efforts from the community are still needed to push this field forward.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BxJxswX6w2", "reviewer": "Reviewer_KJVB", "summary": "This paper proposes to use object-centric slot representations as visual representations in vision language models. Specifically, it adopts the popular video LLM paradigm and concatenates a pre-trained visual encoder with a pre-trained LLM. It uses a slot attention method to group visual features into object slots and event slots before feeding them into the LLM. These slot representations compress visual features into a small set of tokens and stand out in terms of efficiency. The final model, Slot-VLM, performs well in video question answering tasks and has the potential to handle long videos.", "questions": "1. A related work, ChatUniV in CVPR 2024, uses clustering algorithm to merge visual tokens, thereby reduce computational cost. Slot attention can be regarded as an upgraded (learnable) clustering algorithm. Apart from this, what is the key difference between Slot-VLM and ChatUniV?\n2. SlowFast network is initially proposed to address the video redundancy problem. While in Slot-VLM, the slot-attention seems to also reduce redundancy by aggregating the visual tokens into compact slot representations. In that case, is it still necessary to use SlowFast?", "limitations": "Please refer to weaknesses and questions above.\nIn summary, the object-centric inductive bias used in this work is interesting. It also shows potential on handling long-videos given the token compression strategy and the impressive performance on standard benchmarks. However, the claim of Object-Event Slots in the title is questionable. 1) The object-slots seem to not effectively group objects according to visualization. 2) Event-slots. See weakness point 1.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It's an intuitive and good idea to apply object-centric inductive bias to vision-language models.\n2. The experiments in the main paper and appendix are sufficient, assessing various aspects of the method.\n3. The performance is surprisingly good.", "weaknesses": "1. Are the temporal slots really event slots? Since there are m spatial positions after downsampling, what if an object moves around from one spatial patch to another? Then the event cannot be effectively captured by the slots, since the temporal slot attention happens in the fixed spatial position along the temporal dimension.\n2. The visualization seems poor. It was expected that slot attention, as a learnable method, should performs better than naive clustering algorithms, as used in ChatUniV. However, the results shown in Figure 3 do not really group objects as expected. It still looks similar to the visualization of ChatUniV.\n3. Regarding object-centric/slot-attention in videos, there are some related works that are not discussed. For example, <Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities, NeurlPS 2023>, <Unsupervised Open-Vocabulary Object Localization in Videos, ICCV 2023>.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback regarding the idea, sufficiency of experiments, and the surprisingly good performance. We greatly appreciate your insights and suggestions and will thoroughly incorporate them into our revised manuscript. Below, we provide detailed responses.\n\n**Q1**: Are the temporal slots really event slots? Since there are m spatial positions after downsampling, what if an object moves around from one spatial patch to another? Then the event cannot be effectively captured by the slots, since the temporal slot attention happens in the fixed spatial position along the temporal dimension.\n\n**A1**: Thank you for the insightful question. We understand the concern. Our intention with the term “event slots” is to conceptually align these slots with events at a high-level, as mentioned in lines 203-204 of the manuscript: “Temporal slots aggregate the temporal tokens for explaining parts of the input, similar to identifying events. Thus, we name the learned slots as event-centric slots.”\n\nWe have considered this question during our design. To alleviate it, instead of learning event slots for each spatial position, we divide the feature map into 4x4=16 non-overlapped large spatial regions (downsampling), with each region corresponding to a large local region of 56x56 pixels when the frame resolution is 224x224. **This allows us to observe the temporal dynamics of a large local region (56x56 pixels), which helps infer the evolution of part of an event within that region.** \n\nWe acknowledge that this method might not perfectly capture the entire object dynamic, but it allows for a partial understanding of object evolution within a large local region. Through leveraging the learned slots from all 16 large local regions, the LLM can achieve a more comprehensive perception of global temporal dynamics.\n\nThe temporal slots aggregate semantically consistent temporal segments, akin to how spatial slots aggregate spatially consistent regions (object-centric slots). While the segments within a slot might not encompass a complete event, they are likely part of the same event. We will revise the manuscript to clarify this design consideration and improve the explanation of how event slots work in our model.\n\n\n**Q2**: The visualization seems poor. It still looks similar to the visualization of ChatUniV. What is the key difference between Slot-VLM and ChatUniV?\n\n**A2**: Thank you for your insightful question. Yes, the visualization of slot attention is still not very satisfactory. Actually, the unsupervised object-centric representation learning remains a challenging task in the field. \n\nOur Slot-VLM significantly outperforms ChatUniV (see Table 1) due to two main reasons. \n\n1) **Slots can preserve more original information than clustering centers.** Slot attention is trained to reconstruct the original features from slots. In contrast, ChatUniV uses the average token within each cluster to represent the corresponding cluster, where the averaging can result in a loss of fine-grained details, whereas the detailed visual information is crucial for accurate video understanding and reasoning. \n\n2) **Slots are learnable and can be jointly optimized within the entire framework, whereas clustering (being not learnable) in ChatUniV is task-agnostic.** Our ablation study below shows that freezing the slot attention subnetworks at Stage 3 in our framework leads to a noticeable drop in performance when compared with our final scheme with slot attention jointly tuned (see Table F).  Note that we will add the results that freezing the slot attention subnetworks at Stage 2 and Stage 3 once we obtain, which we guess they are lower than that freezing slot attention subnetworks only at Stage 3.\n\nTable F: Ablation study on the influence of end-to-end optimization of the slot attention modules. Slot-VLM (slot frozen) denotes the slot attention modules are frozen in Stage 3.\n\n| Models                 | In-domain (Acc./Score) | MSVD-QA (Acc./Score) |\n|:----------------------:|:----------------------:|:--------------------:|\n| Slot-VLM (slot frozen) |     46.2/2.68                   |   74.0/3.74                   |\n| Slot-VLM               | 48.8/2.75              | 74.9/3.76            | \n\n\n**Q3**: Add related works regarding object-centric/slot-attention in videos.\n\n**A3**: Thank you for the helpful suggestion and we will add the following discussion.\n\nSome works have also explored object-centric learning in videos. VideoSAUR [A] extends the method of DINOSOUR [27] to video-based object-centric learning by incorporating temporal feature similarity loss, which introduces a motion bias for each frame. Fan et al. [B] propose an unsupervised method for open-vocabulary object localization in videos, where they run only once slot-attention on the entire spatial-temporal features to obtain video slots for all frames. \n\nIn contrast, we leverage slot attention to learn semantically decoupled tokens as the input to LLMs and investigate the effectiveness of aligning these ‘concept tokens’ with LLM input. \n\n(Note [A] and [B] denote the referred two papers, respectively.)\n\n\n**Q4**: Is it still necessary to use SlowFast?\n\n**A4**: Yes, we had demonstrated the necessity of our two-branch design in line 309-320 of our manuscript. Table G shows that directly extending slot learning to spatial and temporal features increases memory and computation requirements, and the optimization difficulty. \nTable G: Ablation study on the effectiveness of joint spatial-temporal slots learning vs. our two-branch design. The FLOPs and number of parameters for the reduced token generation module are presented.\n\n| Model          | In-domain (Acc.) | MSVD-QA (Acc.) | #FLOPs (G) | #Param. (M) |\n|:--------------:|:---------:|:-------:|:----------:|:-----------:|\n| Slot-Joint-VLM | 46.7      | 72.8    | 1304.8     | 13.7        |\n| Slot-VLM      | 48.8      | 74.9    | 41.6       | 27.3        |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Area Chair,

Thank you very much for your great efforts and insightful feedback on our paper. **We are grateful for your positive feedback on our motivation/insights (Reviewer jNDE, KJVB), idea novelty and interestingness (Reviewer gzPw, wQQx, KJVB), comprehensive experiments and method effectiveness (Reviewer jNDE, gzPw, wQQx, KJVB), and paper writing and comprehensive details (Reviewer jNDE, gzPw, wQQx).**

We have carefully considered each of your comments and suggestions. We provide detailed responses to address the concerns. Particularly, we summarize our responses regarding each reviewer’s main concerns/comments below:

1)	Based on **Reviewer jNDE**’s concern on ChatGPT-based metrics and suggestion on testing on recent multi-choice QA benchmarks, we have conducted additional zero-shot testing on recent multi-choice QA benchmarks and provide results (see A3). These results demonstrate the robustness and generalizability of our approach.

2)	Based on **Reviewer gzPw**’s suggestions and comments, we have added the ablation study on the influence of frame sampling rate and strides, respectively (see A3). We also show that our method is scalable in handling longer videos, e.g., EgoSchema (see A2). This scalability ensures the applicability of our method to diverse video datasets.

3)	Regarding the questions of **Reviewer wQQx and KJVB** about the event slots, we will clarified the definition and add more explanation in the revision (see A1 to Reviewer wQQx and A1 to Reviewer KJVB). Temporal slots aggregate the temporal tokens for explaining parts of the input, similar to identifying events. Thus, we name the learned slots as event-centric slots. More precisely, we leverage temporal slots to aggregate semantically consistent temporal segments, akin to how spatial slots aggregate spatially consistent regions, termed object-centric slots. While these temporal segments within a slot might not form a complete event, they are likely part of the same event. 

4)	Regarding the questions of **Reviewer KJVB**, we have analyzed the reason why Slot-VLM significantly outperforms ChatUniV (see A2) and demonstrated the necessity of the two-branch design (see A4). The analysis underscores the advantages of our architectural choices in enhancing video-language modeling.

We believe that incorporating your valuable insights will significantly enhance the quality and clarity of our paper. We hope our responses adequately address your concerns and look forward to any further feedback you may have. Please feel free to share any additional questions or concerns!


Many thanks,

All authors

Meta-review: The paper received split ratings, 3 weak accepts and 1 borderline reject.

The rebuttal successfully addressed most of the concerns from the 3 reviewers, including additional benchmarks. The borderline reject reviewer's remaining concern is due to the misleading claim of the paper, which the authors promised to fix. The AC does not see a major reason to turn down the majority of reviewers' opinions.

As mentioned during the reviewer discussions, the authors are advised to include the new experimental results in the final version of the paper. Also revise the claim of the event slot in the paper, to more accurately reflect its capabilities and limitations.
Decision: Accept (poster)