Slot-VLM: Object-Event Slots for Video-Language Modeling

Authors: jiaqi  xu
cuiling  lan
wenxuan  xie
xuejin  chen
yan  lu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-03 11:12:40

Abstract: Video-Language Models (VLMs), powered by the advancements in Large Language Models (LLMs), are charting new frontiers in video understanding. A pivotal challenge is the development of an effective method to encapsulate video content into a set of representative tokens to align with LLMs. In this work, we introduce Slot-VLM, a new framework designed to generate semantically decomposed video tokens, in terms of object-wise and event-wise visual representations, to facilitate LLM inference. Particularly, we design an Object-Event Slots module, i.e., OE-Slots, that adaptively aggregates the dense video tokens from the vision encoder to a set of representative slots. In order to take into account both the spatial object details and the varied temporal dynamics, we build OE-Slots with two branches: the Object-Slots branch and the Event-Slots branch. The Object-Slots branch focuses on extracting object-centric slots from features of high spatial resolution but low frame sample rate, emphasizing detailed object information. The Event-Slots branch is engineered to learn event-centric slots from high temporal sample rate but low spatial resolution features. These complementary slots are combined to form the vision context, serving as the input to the LLM for effective video reasoning. Our experimental results demonstrate the effectiveness of our Slot-VLM, which achieves the state-of-the-art performance on video question-answering.

Main paper: 
 introduction: 
 Recently, Large Language Models (LLMs) have gained significant progress [5, 39, 31] . They present exceptional ability to comprehend, reason with, and generate human language text. Such amazing capabilities have stimulated the wave of research on extending the models to Vision Language Models, enabling the vision reasoning ability.For image understanding, MiniGPT-4 [51] leverages a Q-Former and a projector to align a frozen vision encoder with a frozen advanced LLM, where Q-Former converts the visual input into a fixedlength learned visual query tokens (32 tokens) . LLaVA [25] and MiniGPT-v2 [10] directly use the gridded visual tokens (after projection) as the LLM input. Image-text pairs are leveraged to align the visual model and the LLM in training. For handling videos, one straightforward way is to stack the tokens from sampled frames and feed them into the LLM [19, 12, 50] . This is challenging when we densely sample frames (e.g., with the purpose of preserving more information) or when we sample abundant frames for long videos. For example, for a video of 10 minutes, when we sample at 1 frame per second and each frame uses 32 (or 256), we will have 19,200 (or 153,600) tokens in total. On the one hand, the large number of tokens increases both the memory and computational requirement. On the other hand, there is spatio-temporal redundancy and the features are not representative. Video-ChatGPT [28] reduces the number of video tokens to 356 by separately performing spatial pooling and temporal pooling, which suffers from the loss of visual details. Video-LLaMA [48] and VideoChat [20] compress the video tokens using Q-Former, where a set of learnable queries aggregate the information from video tokens through cross-attention and self-attention. These strategies generate reduced tokens. However, each token contains semantically mixed/entangled information (see the visualization analysis of Q-Former in Appendix G.1) and this may bring inefficiency to video-language modeling. As we know, the text words as the input to LLMs have semantics and are disentangled. Intuitively, to better align the video tokens with the language input of LLMs for building VLMs, the generated video tokens that could act similarly to words are desired.In this work, as illustrated in Figure 1 , we aim to generate semantic-centric tokens 2 from video features to comply with LLMs for effective video-language modeling. By leveraging slot attention [27, 35] , which converts an image into object-centric representations, i.e., slots, we design an Object-Event Slots module (i.e., OE-Slots module), that generates a set of semantically decomposed video tokens (a.k.a. slots) from video features, and takes such slots as input to LLMs. This is somewhat similar to human visual reasoning, where the brain converts the visual perception to high-level object representations to facilitate further reasoning. We dub the scheme powered by OE-Slots module as Slot-VLM. Figure 2 shows the flowchart of Slot-VLM. Particularly, OE-Slots module consists of two branches: Object-Slots and Event-Slots for decomposed spatial and temporal modeling, focusing on spatial objects and temporal varied dynamics (event), respectively. The Object-Slots branch extracts object-centric slots from high spatial resolution features but sampled at low frame rate. The Event-Slots branch extracts event-centric slots from high temporal resolution but low spatial resolution features. The two sets of slots are combined together as the input to the LLM for video reasoning. In instruction-tuning, we fine-tune OE-Slots module and the projection layer to align the visual features with LLM.In summary, we have three main contributions: In order to obtain semantically decoupled and compact (reduced) video tokens as the vision context for aligning with LLM, our OE-Slots module learns to aggregate those tokens to object-centric tokens and event-centric tokens through the Object-Slots branch and the Event-Slots branch, respectively. The Object-Slots branch operates at low frame rate (t ≪ T ) but high spatial resolution in order to capture spatial objects through slot attention on each sampled frame. The Event-Slots branch operates at high frame rate but low spatial resolution (m = h × w, where h < H, w < W ) in order to capture temporal dynamics through slot attention over each spatial position. The learned slots (tokens) from two branches are projected and inputted to LLM for video reasoning, together with the text.• We propose a new framework, Slot-VLM, for effective video-language modeling, where we generate semantically decomposed tokens to comply with LLMs. To the best of our knowledge, this work is the first to explore the use of learnable semantically decoupled visual tokens to align with LLMs. This work provides a new insight that 'disentangled' video tokens are beneficial for video-language modeling. • We design an OE-Slots module that encourages object-centric and event-centric visual represenations for modeling spatial objects and temporal dynamics. This moves a step towards the compact and semantically meaningful token representations in VLMs. • Experimental results demonstrate the effectiveness of our OE-Slots designs, achieving state-of-theart performance on video question answering tasks.We hope that this work will inspire more designs towards generating compact and semantically interpretable visual tokens for effective video-language modeling.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We aim to generate semantic-centric tokens from video features to better comply with LLMs for effective video-language modeling, providing a new insight that 'disentangled' tokens are beneficial for video-language modeling.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Visual Language Models In considering the strong comprehension, reasoning, and generation ability of the LLMs, there is a surge of investigations on exploiting LLMs and vision encoders to build visual-language models, enabling the vision reasoning ability [1, 19, 51, 25, 20, 28, 49, 48, 37] .BLIP-2 [19] leverages frozen pre-trained CLIP image encoders [32] and frozen LLMs to bootstrap language-image pre-training, with the modality gap bridged by a Querying Transformer (Q-Former) and a projector. Q-Former extracts a fixed number (e.g., 32) of output features (tokens) from the image feature, acting as an information bottleneck to facilitate the feeding of most useful information to LLMs. Other works like MiniGPT-4 [51] and Video-LLaMA [48] similarly leverage Q-Former to encode visual tokens, followed by image-text or video-text pair instruction tuning.Compared with image reasoning, it is more challenging for video-language modeling, where the excessive number of visual tokens from video would enlarge the computation and memory cost to the LLM inference and thus constrains the application of VLMs. Some works sparsely sample frames and concatenate the tokens from these frames as the input to LLMs [19, 12, 50] . However, this brings difficulty for understanding videos with high temporal dynamics or long videos, with high risk of losing valuable temporal information.Video-ChatGPT [28] reduces the number of visual tokens by performing spatial and temporal pooling separately. LLaMA-VID [22] represents each frame with two distinct tokens: context token and content token, where context token is obtained by aggregating the visual features based on the text query while the content token is obtained by pooling the features. However, being task-agnostic, pooling would result in loss of helpful information. Moreover, the semantic-agnostic pooling inevitably results in the mixing of features of different semantics, where such coupled representations may bring difficulty for video reasoning. Q-Former provides an elegant way for generating reduced number of tokens. However, we found the learned tokens are still coupled, with abundant overlapped information among them. When aligning with the 'disentangled' word text tokens, such entanglement may result in poor compatibility with LLMs. Chat-UniVi [17] merges the dense visual tokens by clustering and uses the averaged token features in a cluster as the cluster representation. With clustering being parameter-free and task-agnostic, this is sub-optimal. It is still an open question on how to generate video tokens for constructing effective VLMs.In this work, we explore the use of semantically decoupled tokens by learning spatial object-centric slots and temporal event-centric slots as video representation. We reveal that using semantically decoupled tokens as the input to LLMs are effective for video-language modeling.Object-Centric Learning Humans naturally decompose the observed environment into entities at the appropriate level of abstraction to act in the world [35] . To obtain semantically decoupled representations, object-centric representation learning has garnered significant interest [6, 27, 36, 35] . Slot Attention [27] is a representative work. A slot attention module is designed to interface with features to produces a small set of abstract representations, named as slots, where each slot represents an object or a scene. It demonstrates encouraging object decomposition ability in simulated data but fails on real-world images [27] . To successfully learn object-centric slots in real-world images, DINOSAUR [35] learns the slots from pre-trained features by reconstructing the features. Such pre-trained features can be obtained from self-supervised learning techniques like DINO [8] , MAE [15] , or vision-language pre-training like CLIP [32] . These features usually have a high level of homogeneity within objects and thus facilitate the forming of object-centric representations.In this work, we leverage slot attention to learn semantically decoupled tokens as the input to LLMs and investigate the effectiveness of aligning these 'concept tokens' with LLM input. Note that, interestingly, words in human language are abstractions of concepts, with different words tending to be disentangled. We intend to bridge the connection between vision and language by representing video with 'concepts' in terms of object-centric slots and event-centric slots.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please see Appendix I.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed slot-vlm: 
 As illustrated in Figure 2 , our Slot-VLM consists of a frozen image encoder, a trainable Object-Event Slots (OE-Slots) module, a projection layer, and a frozen LLM. Given a video sequence, we extract frames at a speed of 1 frame per second (FPS) and obtain a video V of T frames. The video V together with the text question (user instruction) is input to Slot-VLM, and Slot-VLM outputs the generated text answer.In order to build an effective VLM for video understanding, we propose to encapsulate the dense video tokens into a small set of semantically decoupled tokens to align with the LLM. Our OE-Slots module converts the dense video tokens from the vision encoder into a set of object-centric slots (tokens) through the Object-Slots branch, and a set of event-centric slots (tokens) through the Event-Slots branch, which are then projected as the input to LLM for video reasoning.Intriguingly, our exploitation of object-centric visual representations to enhance large language model (LLM) reasoning mirrors certain aspects of the cognitive processes humans use for visual reasoning. Human visual reasoning is a complex cognitive process. Visual signals are processed initially through the primary visual cortex (V1) and subsequently integrated in higher-level visual processing areas, resulting in the formation of complex visual object representations. Such high-level object representations together with brain-stored knowledge are then used for logical reasoning and inference to interpret observations [34, 3] . Similarly, Slot-VLM generates object-centric and event-centric token representations to provide the vision source for the LLM reasoning, where the LLM stores rich knowledge and has strong reasoning ability, and interpretable-text generation ability. Such exploration makes us move a small step towards brain-like visual reasoning.To extract visual features of a video, we simply use an image-based model to get per-frame feature. Following [28] , we utilize vision-language pre-trained CLIP (ViT-L/14) vision model [32] as our vision encoder. For each frame, the vision encoder outputs H × W visual tokens, with the dimension of a token denoted by D, where H = W = 16 for the CLIP vision encoder. For a sampled video of T frames, we have H × W × T visual tokens. Taking a video of 3 minutes as an example, the total number of video tokens is 16 × 16 × 180 = 46, 080. The large number of tokens would incur significant computational and memory costs during LLM inference. Considering the redundancy from the dense tokens, we aim to semantically exploit the the spatial structure and and temporal dynamics to reduce the number of tokens for efficient LLM inference.Previous methods, which aggregate the video tokens by pooling [28] or Q-Former [50, 49] , actually produce tokens wherein semantics remain entangled within each token, rather than targeting at generating semantics decoupled representations. In contrast, the word tokens, which are the input to LLMs, are more semantically decoupled. In this work, we explore the using of semantics-decoupled video representations for effective video-language modeling.We design a OE-Slots module that encapsulates the dense video tokens from the vision encoder into a small set of semantic-centric tokens to facilitate the LLM inference. It is challenging to learn semantics-centric representations from the dense video features. DINOSAUR [35] uses slot attention on the pre-trained image feature to learn object-centric representations without supervision. It is challenging to directly extend the slot learning to video features, given the substantial number of video tokens (e.g., 46 ,080 for a 3-minute video), which increases memory and computation requirements, and the optimization difficulty.In our work, to effectively learn semantics-centric representations from abundant video tokens, we design an Object-Event Slots module comprising two branches. The an Object-Slots branch operates at low frame rate to capture spatial object-wise slots for each sampled frame, while the Event-Slots branch operates at high frame rate but low spatial resolution to capture temporal event-wise slots. This dual-branch design is inspired by the SlowFast network [13] , employing a Slow pathway for low frame rate spatial feature learning and a Fast pathway for high frame rate motion capture using a lightweight subnetwork. Unlike SlowFast which focuses on refining grid-wise features, our Object-Event Slots module aims to generate a small set of semantically meaningful object-centric slots and event-centric slots, serving as semantically meaningful input into LLM.Object-Slots Branch As shown in Figure 2 of the Object-Slots branch, for a video of H × W × T dense video tokens, we sample the features at low frame rate but high (original) spatial resolution to obtain H × W × t video tokens. t denotes the number of uniformly sampled frames, which we set to 8 by default.For the i-th frame, we have a set S i of H × W tokens. S i is taken as the input to slot attention module [27, 35] to generate N o object-centric slots O i = {o i,1 , . . . , o i,No } (we also refer to them as spatial slots). Please see Appendix A for more detailed formulation. Particularly, slot attention uses an iterative mechanism to map from the input tokens to the slots. At each iteration, slots attention uses cross attention with attention coefficients that are normalized over the slots (where slots are the queries) to aggregate token information. The normalization over the slot introduces competition between the slots for promoting the forming of decoupled representations.To distinguish different frames, we add learnable temporal position embedding p i to each slot of the i-th frame and obtain the updated object slots as Note that we perform spatial object slot learning on the high spatial resolution but low frame rate features in order to capture the spatial objects and reduce the number of slots, which is proportional to the number of frames.O i = {o i,Event-Slots Branch As shown in Figure 2 of the Event-Slots branch, for a video of H × W × T dense video tokens, we sample the features at high frame rate but low spatial resolution to obtain h × w × T video tokens. We obtain the spatial down-sampled tokens by averaging pooling with a stride of 4, therefore h = H/4 = 4 and w = W/4 = 4.We perform temporal slot learning along temporal axis. To be aware of different frames, we add learnable temporal position embedding to each token. For the k-th spatial position, where k = 1, . . . , m and m = h × w, we have a set F k of T tokens from the T frames. F k is taken as the input to slot attention module to generate N e event-centric slots E k = {e k,1 , . . . , e k,Ne } (we also refer to them as temporal slots). Note that each spatial position from h × w positions corresponds to a large local patch in the pixel space, i.e., 56×56 patch for a video of 224×224 spatial resolution. This allows us to observe the temporal dynamics of a large local region (56×56 pixels), which helps infer the evolution of part of an event within that region. Temporal slots aggregate the temporal tokens for explaining parts of the input, similar to identifying events. Thus, we name the learned slots as event-centric slots. We leverage temporal slots to aggregate semantically consistent temporal segments, akin to how spatial slots aggregate spatially consistent regions. While these temporal segments within a slot might not form a complete event, they are likely part of the same event.We concatenate the slots from each of the m spatial positions and obtain m × N e slots E = {E 1 , . . . , E m } = {e 1,1 , e 1,2 , . . . , e 1,Ne , . . . , e m,Ne }. A linear projection layer (F-Proj.) transforms each token to facilitate the alignment with slots from the Object-Slots branch and the alignment with the LLM.Note that we perform temporal event slot learning on the high frame rate but low spatial resolution features in order to capture the temporal dynamics, and reduce the number of slots, which is proportional to the spatial feature resolution.We concatenate the object-centric slots from the Object-Slots branch and the event-centric slots from the Event-Slots branch and obtain N = t × N o + m × N e slots. A linear projection layer (Proj.) transforms each token to align with the LLM. The projected tokens and the text instruction are taken as the input to LLM for video reasoning.We divide the training procedure into three stages: slot attention pre-training, single branch instruction tuning, and two branch joint instruction tuning. We use the three stage training to better optimize the model. Stage 1 aims to pre-train the slot attention modules (with the objective of reconstructing the input features) to facilitate the learning of object-centric and event-centric slot representations, i.e., the forming of semantically decomposed tokens/slots. Stage 2 separately trains the Object-Slots branch and the Event-Slots branch to facilitate the system focusing on the optimization of each branch separately, which eases the optimization. Stage 3 jointly optimizes the two branches and the projection layer. Please see Appendix B for more details.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
implementation details: 
 Following Video-ChatGPT [28] , we employ the pre-trained CLIP vision model of ViT-L/14 [32] as the vision encoder. Note that the model size of CLIP ViT-L/14 (which we use) is 303M, which is much smaller than CLIP ViT-G/14 (1012M) as used by MovieChat [37] , LLaMA-VID [22] , and Video-LLaMA [23] . Specifically, we extract features from the penultimate layer, yielding an array of H × W visual tokens for each video frame. We use the Vicuna (7B) from the LLaVA model [25] , to serve as our LLM. We sample at 1 fps from a video to obtain T frames, and we resize each frame to 224×224 resolution. In our experiments, we set N o and N e to 8 by default unless otherwise specified.The OE-Slots module generates All models are trained using a single NVIDIA A100 80GB GPU. The linear projection layer S-Proj., F-Proj. and Proj. consists of 1024, 1024, and 4096 neurons, respectively. We adopt a network structure that is similar to the slot attention from [36] to build the slot attention modules, with learnable slot initialization. More details please refer to Appendix B.N = t × N o + m × N e = 8 × 8 + 16 × 8 = 192 slots (tokens)We use the Video Instruction Data, collected by [28] , for video instruction tuning. This comprehensive dataset comprises approximately 100K video text (question-and-answer) pairs, which are generated from the ActivityNet dataset with an average video length of 180 seconds. The dataset is characterized by a diverse array of question types. Alternatively, similar to Video-LLaVA [23] , we could also incorporate a 665K image-text instruction dataset from LLaVA v1.5 [24] for enhancing the instruction tuning of the Object-Slots branch and mark the model as Slot-VLM † . We report both the results of Slot-VLM and Slot-VLM † in Table 1 . Note that all our other results including ablation studies and visualization are obtained without using the 665K pairs.We evaluate the performance on three open-ended video question-answering (QA) benchmarks like MSVD-QA [9] , MSRVTT-QA [44] , and ActivityNet-QA [7] . We evaluate models using accuracy (%) and average score metrics by [28] , employing ChatGPT to judge prediction correctness. ChatGPT reviews each QA pair and issues a binary correctness verdict and a similarity score from 0 (least) to 5 (most) (See Appendix C for more details). We also evaluate our models on three multi-choice QA benchmarks, including Egoschema [29] , NExT-QA [43] and STAR [42] (See Appendix F).We evaluate the performance of our scheme against the state-of-the-art methods on three zero-shot open-ended video QA benchmarks. Table 1 shows the results. All these models use Vicuna-7B [11] as the LLM.For video-language modeling, there is no converged standard on the training datasets, including the data for pre-training and that for instruction tuning. Generally, it is not very fair for comparisons. For example, Video-LLaMA [48] uses Webvid-2M short video dataset and CC595k image caption datasets for pre-training to enable video features to contain as much visual knowledge as possible; VideoChat2 [21] performs per-training of two stages for vision-language alignment (15M image captions from CC3M and CC12M and 10M video captions form WebVid-10M) and vision-language connection (adding 2M image captions, and 10M video captions from InternVid [40] ), respectively. Similar to Video-ChatGPT [28] , we do not perform such visual-text alignment pre-training. For instruction tuning, different methods use different data (see the second column in Table 1 ).We have the following observations/conclusions. 1) Our Slot-VLM † consistently achieves the best accuracy on all the three benchmarks, outperforming all other methods, even though the volume of vision-text pairs used for our instruction tuning (0.7M pairs) is much less than many models such as VideoChat2 [21] (1.9M pairs), Chat-UniVi [17] (2.4M pairs), and MovieChat [37] (5.1M pairs). The average performance of our Slot-VLM † outperforms Video-LLaVA † by 6.6% in accuracy, while achieving the similar scores. 2) In comparison to the methods highlighted in gray that utilize the same 100k video-text pairs for instruction tuning, our Slot-VLM consistently outperforms its competitors, surpassing Video-ChatGPT [28] We study the effectiveness of our semantics-centric designs, two branch design, and the hyperparameter choices, on the test set of the Video Instruction Data from [28] , which we refer to as In-domain, and on MSVD-QA [9] . We name our scheme with only the Object-Slots branch as Object-Slot-VLM, the scheme with only the Event-Slots branch as Event-Slot-VLM, and our final scheme as Slot-VLM.Effectiveness of using Semantics-centric Tokens Under our framework, we replace our slot attention modules for generating reduced tokens by Q-Former [19] , using similar training strategies and computational cost (FLOPs). Similarly, we name the schemes with the spatial branch, the temporal branch, and two branches as Spatial-QFormer-VLM, Temporal-QFormer-VLM, and ST-QFormer-VLM, respectively. Table 2 shows the results, the number of FLOPs, and the number of parameters for the reduced token generation modules, indicating the fairness for comparisons. Note we compute the FLOPs by taking a video of 100 frames. The number of parameters for our Object/Event-Slots module is much smaller than that of the Spatial/Temporal-Qformer module because the Slot Attention uses shared parameters for its L = 3 iterations. We can see that the schemes using slot attention outperform those using Q-Former in both the single branch settings and the two branch setting.Object-Slot-VLM outperforms Spatial-QFormer-VLM by 6.4% on the indomain test data (In-domain) and 2.2% on MSVD-QA, respectively. Event-Slot-VLM outperforms Temporal-QFormer-VLM by 1.1% and 0.5%. We visualized the spatial attention maps of learned Q-Former and found that Q-Former is not capable of decoupling visual tokens to semantically meaningful instances (Please see Appendix G.2).In addition, for the two-branch scheme ST-QFormer-VLM, we found that tuning the Q-Formers and projection layers even with the single-branch trained parameters as initialization cannot lead to satisfactory results, which are poorer than the single-branch schemes. That may be caused by the difficulty in aligning the output tokens of the two branches. We can see that even though not perfectly segmented, some meaningful slots have been formed. For example, the slots marked by red, purple, green, and blue in the first video (left) correspond to 'background', 'human body', 'head', and 'barbell'. Note that the slots in a frame is unordered and exchangeable.and Event-Slot-VLM by 1.7%/1.8% in accuracy, respectively. Our semantics-centric tokens from the two branches are complementary.An alternative design is to learn the same number of slots (i.e., 192) from the dense H ×W ×T tokens instead of from the two branched tokens. We dub the scheme as Slot-Joint-VLM. As discussed in Section 3.2, directly extending the slot learning to dense video features raises increases memory and computation requirements, and the optimization difficulty. Table 3 shows the comparison between Slot-Joint-VLM and our two-branch design Slot-VLM. We can see that the computational complexity of the reduced token generation module of Slot-VLM is much lower than that of Slot-Joint-VLM while Slot-VLM achieves superior performance.Influence of Hyper-parameters Appendix E shows the influence of hyper-parameters.Object-Centric Slots We visualize the spatial cross-attention masks from our Slot-VLM for indicating the forming of object-centric slots for each frame, with two video samples shown in Figure 3 . We can see that some meaningful slots have been formed. For example, in the first video, the slots marked by red, purple, green, and blue in the first video (left) correspond to 'background', 'human body', 'head', and 'barbell'. For comparison, we also visualize the cross-attention masks of Q-Former from the scheme Spatial-QFormer-VLM in Appendix G.2. We observe that our object-centric slots from Slot-VLM are better semantically decoupled. Moreover, we found the instruction tuning can further promote the decoupling of slots (see Appendix G.3, where the well decoupled representations may better align with LLM. . For this spatial position, the models generate N e = 8 slots/queries by aggregating the temporal visual tokens. The attention masks for E 13 are denoted by a map of T rows and N e columns, with the visibility indicating which queries this temporal position belongs to. The higher the visibility, the greater the affinity between this temporal position and the query. We can see that in our Slot-VLM, similar contents tend to be allocated to the same slot, i.e., different slots capture different contents (events) and present decoupled semantics. In contrast, in Temporal-QFormer-VLM, different contents are usually assigned to the same query or are uniformly assigned to different queries. Note that for Temporal-QFormer-VLM, we only show the mask of one head to save space, where similar observations can be found from other heads. A glimpse of the original video can be found in Figure 5 . See Figure 10 for the enlarged visualization of E 13 .Figure 5 : A glimpse of the original video used in Figure 4 . For visualization purpose, we only show the frames down-sampled at a factor of 8, which is 1/8fps.h × w = 16 positions). Similarly, Figure 4 (b) visualizes the temporal cross-attention masks from Temporal-QFormer-VLM for indicating the forming of query tokens for each spatial position. We can see that in our Slot-VLM, along the temporal axis, similar contents tend to be allocated to the same slot. In other words, different slots capture different contents (events) and present decoupled semantics. In contrast, in the Temporal-QFormer-VLM, different contents are usually assigned to the same query or are uniformly assigned to different queries. The queries learned from Q-Former do not present decoupled semantics. Figure 5 shows a glimpse of the original video used in Figure 4 .Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 3, 4, and Appendix B have presented the necessary information to reproduce the main experimental results of the paper. The used datasets are all public datasets. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 In this work, we introduce a new framework, Slot-VLM, that aims to generate a small set of semantically decoupled video tokens to comply with LLM for effective video reasoning. Particularly, we design a dual-branch Object-Event Slots module to learn object-centric slots and event-centric slots to jointly capture the spatial object details and temporal dynamics. These semantic-centric slots are taken as the input to LLM for effective video reasoning. Experimental results demonstrate the superiority of our framework and show the effectiveness of using semantics-decoupled representations for aligning with LLM. However, our current representations are still not perfect where object instances and events are not ideally segmented. We anticipate this work will inspire more investigations towards semantic-centric visual representations for video-language modeling.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] . Justification: We will release the code. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We have described the necessary information in Section 3, 4, Appendix B and E.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Following other papers in this field, we did not report error bar to save the computational cost.Guidelines:• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Please see Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As shown in Figure 2 of the Object-Slots branch, for a video of H × W × T dense video tokens, we sample the features at low frame rate but high (original) spatial resolution to obtain H × W × t video tokens. t denotes the number of uniformly sampled frames, which we set to 8 by default. In our framework, H = W = 16. For the i-th frame, we have a set Si of M = H × W tokens. Si is taken as the input to slot attention module [27, 35] to generate No object-centric slots Oi = {oi,1, . . . , oi,N o } (we also refer to them as spatial slots).Slot attention uses an iterative mechanism to map from the input tokens to the slots. At each iteration, slots attention uses cross attention with attention coefficients Am,n that are first normalized over the slotsAm,n := e Mm,nNo l=1 e M m,lEQUATIONSuch normalization over the slots introduces competition among the slots for promoting the forming of decoupled representations. To aggregate the input tokens to their assigned slots, weighted mean is used as follows for updating slots as:EQUATIONk(•), q(•), and v(•) are learnable linear transformations. Gated Recurrent Unit (GRU) and MLP parameterized by h ϕ is used to further updated the slots as Qi := h ϕ (Qi, Qi). L such iterations are performed to obtain the final No slots for this frame. Note that we initialize the slots from learnable Gaussian distributions. L such iterations are performed to obtain the final No slots for this frame. Note that we initialize the slots from learnable Gaussian distributions.To distinguish different frames, we add learnable temporal position embedding pi to each slot of the i-th frame and obtain the updated object slots as Oi = {oi,1, . . . , oi,N o }, where oi,j := oi,j +pi. We concatenate the slots of all the t frames and obtain t × No slots O = {O1, . . . , Ot} = {o1,1, o1,2, . . . , o1,N o , . . . , ot,N o }. A linear projection layer (S-Proj.) transforms each token to facilitate the alignment with slots from the Event-Slots branch and the alignment with the LLM.Note that the slot attention mechanism for learning event-centric slots is similar to that for learning object-centric slots and we will no longer elaborate on that.We use three stage training to better optimize the model. Stage 1 aims to pre-train the slot attention modules (with the objective of reconstructing the input features) to facilitate the learning of object-centric and event-centric slot representations, i.e., the forming of semantically decomposed tokens/slots. Stage 2 separately trains the Object-Slots branch and the Event-Slots branch to facilitate the system focusing on the optimization of each branch separately, which eases the optimization. Stage 3 jointly optimizes the two branches and the projection layer. All models are trained using a single NVIDIA A100 80GB GPU. AdamW is used as the optimizer.Stage 1 We first pre-train the two Slot-Attention modules of the Object-Slots branch and the Event-Slots branch separately. This promotes the formation of object-centric and event-centric slot representations. Similar to DINOSAUR [35] , we use transformer decoder to reconstruct features (rather than images) obtained by CLIP ViT-L/14 by taking the outputted slots (ahead the projection) from Slot-Attention as input.We train the models for 60 epochs with a learning rate 1e-4. We use the Slot Attention module in SLATE [36] . The dimension of each slot is set to 1024. Following [36] , the number of iterations L is set to 3. We use the video frames from the 100K Video Instruction Data [28] for training. Position embeddings in the Event-Slots branch are discarded during the pre-training to ease the training.Stage 2 We perform instruction tuning on the single branch schemes, separately. For example, we fine-tune the Object-Slots branch and the projection layers using the instruction pairs from [28] , by loading the parameters from the first stage. We use the 100K Video Instruction Data from [28] to train our single branch schemes, respectively. We set the number of epochs to 3.The Object-Slots branch/Event-Slots branch and the projection layer Proj. are tuned. We set the learning rate to 2e-5. We adopt the cosine annealing learning rate. We set the batch size to 40 and train on a single A100 GPU.We fine-tune the OE-Slots module and the projection layer using the instruction pairs, by loading the parameters from OE-Slots module from the second stage. By default, we use the Video Instruction Data for instruction tuning. We set the number of epochs to 3. Video LLaMA [48] 256 Video Chat [20] -Video-ChatGPT [28] 356 Chat-UniVi [17] Varied length Video-LLaVA [23] -Video-LLaVA † [23] -BT-Adapter [26] 256 LLaMA-VID [22] Varied length VideoChat2 [21] 1536* MovieChat [37] - SlotIn our study on the open-ended video question answering, we adopt the evaluation metrics of accuracy and average score as established by [28] . To assess the accuracy of our model's predictions, we utilize ChatGPT (chatgpt35-turbo) as an evaluator. ChatGPT processes each question alongside the corresponding ground truth and the model's predicted answer. It then provides a binary "yes" or "no" judgment on the correctness of the prediction for accuracy assessment. Additionally, ChatGPT assigns an integer score ranging from 0 to 5 to quantify the closeness of the predicted answer to the ground truth, with 0 indicating no similarity and 5 denoting a close match.Table 4 shows the performance gain is not due to the final number of video tokens input to the LLMs, where our Slot-VLM uses only 192 tokens that is much smaller than some other models.Influence of High Spatial Resolution for the Object-Slots Branch and High Frame Rate for the Event-Slots Branch We study the influence of spatial resolution of the visual features in the Object-Slots branch and the influence of frame rate of the visual features in the Event-Slots branch, respectively. As shown in Table 5 , when we reduce the spatial resolution from 16 × 16 to 4 × 4 for the Object-Slots branch, the performance drops by 4.6%/3.3% on In-domain/MSVD-QA in accuracy. When we reduce the frame rate by a factor of 8 from 1fps to 1/8fps for the Event-Slots branch, the performance drops by 4.3%/1.4% on In-domain/MSVD-QA in accuracy.We study the influence of the number of object-centric slots No of a frame for Object-Slot-VLM and show the results in Table 6 . We can see that as the increase of the number of slots from 4 to 16 for a frame, the performance increases and saturates when No = 8 in terms of score in In-domain. Therefore, by default, we set No = 8. Influence of Hyperparameter Ne We study the influence of the number of event-centric slots Ne of a spatial position for Event-Slot-VLM and show the results in Table 6 . We can see that as the increase of the number of slots from 4 to 16 along the temporal direction, the performance increases in terms of score in In-domain and achieves good performance when Ne = 8. For simplicity, we set Ne = 8 by default.Influence of Frame Sampling Rate in the Object-Slots Branch and Stride for Pooling in the Event-Slots Branch We conducted ablation studies using single branch settings. For the Object-Slots branch, we tested three frame sampling rates: 4 frames (Object-Slot-VLM (4 frames)), 8 frames, and 16 frames per video. For the Event-Slots branch, we tested four pooling strides to achieve spatial resolutions of 2x2 (Event-Slot-VLM (2x2)), 4x4, and 8x8.Table 7 shows the results. Increasing the sampled frames or spatial resolutions improves performance but increases the number of video tokens and the computational cost. By default, we use 8 sample frames and a 4x4 spatial resolution to balance complexity (192 tokens in total) and performance. Influence of Three Stages Training Stage 1 aims to pre-train the slot attention modules (with the objective of reconstructing the input features) to facilitate the learning of object-centric and event-centric slot representations, i.e., the forming of semantically decomposed tokens/slots. Stage 2 separately trains the Object-Slots branch and the Event-Slots branch to facilitate the system focusing on the optimization of each branch separately, which eases the optimization. Stage 3 jointly optimizes the two branches and the projection layer. Table 8 shows the ablation study on the influence of each training stage. We have the following observations/conclusions. Dataset specific trained (In domain) Accuracy MIST [14] 57.2 GF(uns) [2] 58.8 Sevila [46] 73.8 Zero-shot (Generalization) Accuracy InternVideo [41] 49.1 Mistral(7B) [16] 51.1 VFC [30] 51.5 LLoVi [47] 54.3 MVU(13B) [33] 55.2 ViperGPT(GPT-3.5) [38] 60.0 LangRepo(12B) [18] 60.9 VideoChat2 [21] 61.7 Slot-VLM 62.0 Dataset specific trained (In domain) Accuracy MIST [14] 51.1 GF(uns) [2] 53.9 Sevila [46] 64.9 LRR [4] 70.5 Zero-shot (Generalization) Accuracy Flamingo-9B [1] 41.8 InternVideo [41] 41.6 Slot-VLM 42.7In the first stage pre-training, slot attention is learned by reconstructing the features. In the second stage, LLM is incorporated for video instruction tuning. Here, we take spatial slots as examples to compare the slot attention masks from stage 1 and stage 2. Figure 8 and Figure 9 show the visualization for two examples, respectively. Interestingly, we observe that after the instruction tuning, the learned slots are better decoupled, where a spatial position usually contributes to multiple slots in stage 1 but only contributes to a very few slots in stage 2. Similar phenomenons are obtained for temporal slots.Figure 10 shows the enlarged visualization of attention masks for the 13-th spatial position that is previously shown in Figure 4 .For several-minute-long videos, Slot-VLM can still capture important objects and events information for video understanding. Experiments on the ActivityNet-QA dataset, where the average length of video clips is 111.6 seconds, and the maximum length is 285 seconds, show the superiority of our Slot-VLM over many other methods (see Table 1 , note that the amount of instruction tuning data is much smaller than that in VideoChat2).For scenes with more objects, some similar objects would be gathered into the same slot. Even if not being perfect, our semantics decoupled representations presents its superiority to pooling-based and Q-former based strategies where semantics are mixed in the generated tokens for video understanding.For much longer videos (e.g., hour-long videos), it would be difficult to represent the dynamic scenes with only a small set of event slots. We could extend our framework to well serve long video understanding by partitioning the long-video to chunks of shorter duration, where the object-centric and event-centric slots are generated for each chunk and these slots from chunks are concatenated sequentially as the input to LLM. We leave this as future work.In this work, we introduce a new framework, Slot-VLM, that aims to generate a small set of event-centric video tokens to comply with LLM for effective video reasoning. Currently, we leverage slot attention to learn object-centric slots and event-centric slots. However, the learned slot representations are still not perfect where object instances are still not perfectly segmented and events are not perfectly partitioned. With such imperfect slot representations, our framework achieves the state-of-the-art performance, demonstrating strong potential. We believe the advancement of unsupervised object-centric representation learning in future could further enhance the performance of our framework.For much longer videos (e.g., hour-long videos), it would be difficult to represent the dynamic scenes with only a small set of event slots. Our framework is extensible to support those very long videos but we have not implemented and experimented on such cases. For example, for a three-hour-long video, we could partition the video into chunks with each chunk being τ seconds and summarize dense video features into sparse Ns object-centric and N f event-centric tokens for each chunk and concatenate the tokens of chunks sequentially as the input to LLM for inference. This is feasible in practice by borrowing experience from LLaMA-VID [22] , which is designed for long-form video understanding. LLaMA-VID encodes each frame into two tokens at a sample rate of 1 frame per second to support three-hour-long videos. In our framework, when we have Ns + N f = 192 tokens for each chunk, by setting the duration of a chunk to τ = 96 seconds, we equivalently have 2 tokens per frame on average, facilitating the handling of hour-long-video as LLaMA-VID does.This paper endeavors to make notable advancements in video-language modeling, a field integral to enhancing our interaction with and comprehension of video content. We aspire to drive positive innovation, yet we remain aware that technological progress may have unforeseen side effects. We call upon the users to use AI responsibly. When we release code and models, we would set a safeguard by requiring that users adhere to usage guidelines.Figure 7 : Visualization of spatial attention masks from our baseline scheme Spatial-QFormer-VLM for 8 frames of a video. Since there are 12 heads in Q-Former, we plot 12 (3 × 4) sets of attention masks separated by green lines. For each head, we have t = 8 frames as shown in each row; the first column shows the original frame; the second to the ninth columns show the cross attention mask (from Q-Former) for the 8 queries. We can see that the queries are not clearly decoupled, with each one being a mixture of spatial features. A feature in a spatial position is usually allocated to a couple of queries. Compared with the spatial attention masks from our Slot-VLM as shown in Figure 3 , it is less semantically decoupled for the learned queries. For simplicity, we also refer to slot as query here. For the 13-th spatial position, we denote the set of learned temporal queries by E 13 . For this spatial position, the models generate N e = 8 queries by aggregating the temporal visual tokens. The attention masks for E 13 are denoted by a map of T rows and N e = 8 columns, with the visibility indicating which queries this temporal position belongs to. The higher the visibility, the greater the affinity between this temporal position and the query. We can see that in our Slot-VLM, similar contents tend to be allocated to the same slot, i.e., different slots capture different contents (events) and present decoupled semantics. In contrast, in Temporal-QFormer-VLM, different contents are usually assigned to the same query or are uniformly assigned to different queries.NeurIPS Paper ChecklistJustification: There is no theory in this paper.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.A semantic-centric token refers to a token that represents a semantically meaningful entity, such as an object, or an event (from a temporally consistent segment). 


Decision: The paper received split ratings, 3 weak accepts and 1 borderline reject.

The rebuttal successfully addressed most of the concerns from the 3 reviewers, including additional benchmarks. The borderline reject reviewer's remaining concern is due to the misleading claim of the paper, which the authors promised to fix. The AC does not see a major reason to turn down the majority of reviewers' opinions.

As mentioned during the reviewer discussions, the authors are advised to include the new experimental results in the final version of the paper. Also revise the claim of the event slot in the paper, to more accurately reflect its capabilities and limitations.

Reviews: review: paper_summary: paper_summary: This paper proposed Solt-VLM, which aims to generate a small set of semantically decoupled video tokens to comply with LLM for effective video reasoning. It design a dual-branch Object-Event Slots module to learn object-centric slots and event-centric slots to jointly capture the spatial object details and temporal dynamics. Experimental results demonstrate the superiority of our framework and show the effectiveness of using semantics-decoupled representations for aligning with LLM.
strength_weakness: strength_weakness: - The proposed Slot-VLM is tested on open-ended MSRVTT-QA, MSVD, and ActivityNetQA with automatic ChatGPT-based metrics. Although the metrics is applied by some previous works, I don’t think this is a stable, reliable, and explainable enough evaluation which may vary a lot across GPT versions (check conclusion in [1]) and may generate different responses even for the same input. 
- Also MSRVTT-QA/MSVD/ActivityNetQA is not designed from an object interaction perspective, which may not be effective enough for validating the object-centric design. 
- Based on the previous two points, I would suggest testing on recent multi-choice QA benchmarks, including STAR [2] / NExT-QA [3] / Egoschema [4], which are annotated with clear answer choices, and designed from a human-object interaction view. And compared with recent model works like Sevila [5],  MIST [6], LRR [7], GF [8], which are also based on query-former, or iteratively attention on regions in the video.

[1] FreeVA: Offline MLLM as Training-Free Video Assistant, Arxiv24.  
[2] Star: A benchmark for situated reasoning in real-world videos, NeurIPS 2021.  
[3] Next-qa: Next phase of question-answering to explaining temporal actions, CVPR 2021.  
[4] Egoschema: A diagnostic benchmark for very long-form video language understanding,  NeurIPS 2023.  
[5] Self-chained image-language model for video localization and question answering, NeurIPS 2023.  
[6] Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering, CVPR 2023.  
[7] Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024.  
[8] Glance and focus: Memory prompting for multi-event video question answering, NeurIPS 2023.
questions: questions: please see weakness
limitations: limitations: authors provide limitation discussion in the paper
paper_summary: paper_summary: This paper introduces a new framework called Slot-VLM that aims to aggregate spatial and temporal relationships between frames for more effective video understanding with Large Language Models (LLM). The Slot-VLM approach aims to generate video tokens that are semantically disentangled for better alignment with the frozen LLM. Specifically, the paper proposes the Object-Slots module to extract object-centric slots from high spatial resolution features that are sampled with a low frame rate. Additionally, it also introduces the Event-Slots module, which helps to aggregate the temporal dynamics between frames by learning event-centric slots from features with low spatial resolution but sampled at a high frame rate. The authors demonstrate the benefits of their proposedSlot-VLM approach by evaluating across three open-ended video question-answering datasets including MSVD-QA and ActivityNet-QA, where it outperforms existing video-language models by a significant margin.
strength_weakness: strength_weakness: 1) The proposed Slot-VLM approach appears to be much more effective at aggregating disentangled spatial and temporal relationships between frames, as evidenced by the performance gap between its achieved results and those of Video-ChatGPT and Video-LLaVA in Table 1. However, it is unclear how much of the performance gain is also due to the final number of video tokens that are passed into the underlying LLM for reasoning and generation. It would be helpful to include a comparison of the different number of video tokens used among the different video-language models as well as the number of trainable parameters during each of the training stages.

2) It is also unclear how scalable such an approach will be in handling much longer videos. Currently, it appears that 8 frames are used for the object-slots module. Along with using 8 event-centric slots in total across shorter videos, this may limit the performance of the proposed Slot-VLM approach on much longer videos, such as those used in the EgoSchema evaluation benchmark. Similarly, is there also a more efficient way to aggregate spatiotemporal relationships over more frames beyond just increasing the number of object and event-centric slots since this will increase the computational demand in the LLM?

3) It may be beneficial to the reader to include additional ablation experiments and analysis over the low frame sampling rate used in the Object-Slots Branch as well as the stride used for pooling across the spatial and temporal axes in the Event-Slots Branch. Currently, these hyperparameters appear to be selected based on final downstream performance but there is hardly any discussion on them.
questions: questions: Please see above-mentioned limitations.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper aims to learn a higher level of abstract representation as the input tokens to LLM. The paper proposes a dual-branch (object- and event-centric) to extract the concepts and uses the three-stage training paradigm. The proposed Slot-VLM is evaluated on three VQA datasets and has shown state-of-the-art performance.
strength_weakness: strength_weakness: My main concern is that the authors claim the temporal branch of Slot-VLM as the event branch. The slot attention focuses on local regions temporally, which makes it only capture low-level local motion instead of actual events with high-level semantics. Moreover, the visualization of temporal attention masks (Figures 8 and 10) can barely provide cues that the model can learn events-centric representation (L53, L55, L65). Thus, I believe the event-centric slot is overclaimed in the entire article. I hope the authors can clarify this.
questions: questions: See weakness. The claim of event-centric slot is my main concern.
limitations: limitations: Yes, the authors raise the limitations of this paper in two directions: imperfect concept discovery and extension for hours-long videos. 

However, it would enhance the paper much more if the authors could provide more insights into what other semantic concepts the proposed model lacks, which would complement the existing object- and event-centric representations (Although I am not convinced the model actually captures the “event” semantic)
paper_summary: paper_summary: This paper proposes to use object-centric slot representations as visual representations in vision language models. Specifically, it adopts the popular video LLM paradigm and concatenates a pre-trained visual encoder with a pre-trained LLM. It uses a slot attention method to group visual features into object slots and event slots before feeding them into the LLM. These slot representations compress visual features into a small set of tokens and stand out in terms of efficiency. The final model, Slot-VLM, performs well in video question answering tasks and has the potential to handle long videos.
strength_weakness: strength_weakness: 1. Are the temporal slots really event slots? Since there are m spatial positions after downsampling, what if an object moves around from one spatial patch to another? Then the event cannot be effectively captured by the slots, since the temporal slot attention happens in the fixed spatial position along the temporal dimension.
2. The visualization seems poor. It was expected that slot attention, as a learnable method, should performs better than naive clustering algorithms, as used in ChatUniV. However, the results shown in Figure 3 do not really group objects as expected. It still looks similar to the visualization of ChatUniV.
3. Regarding object-centric/slot-attention in videos, there are some related works that are not discussed. For example, <Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities, NeurlPS 2023>, <Unsupervised Open-Vocabulary Object Localization in Videos, ICCV 2023>.
questions: questions: 1. A related work, ChatUniV in CVPR 2024, uses clustering algorithm to merge visual tokens, thereby reduce computational cost. Slot attention can be regarded as an upgraded (learnable) clustering algorithm. Apart from this, what is the key difference between Slot-VLM and ChatUniV?
2. SlowFast network is initially proposed to address the video redundancy problem. While in Slot-VLM, the slot-attention seems to also reduce redundancy by aggregating the visual tokens into compact slot representations. In that case, is it still necessary to use SlowFast?
limitations: limitations: Please refer to weaknesses and questions above.
In summary, the object-centric inductive bias used in this work is interesting. It also shows potential on handling long-videos given the token compression strategy and the impressive performance on standard benchmarks. However, the claim of Object-Event Slots in the title is questionable. 1) The object-slots seem to not effectively group objects according to visualization. 2) Event-slots. See weakness point 1.

Comments: title: Thank you
Response to the questions
Response of rebuttal and further questions
Thanks for your detailed response and experiments
Citations in response A2
comment: Thank you very much for your great efforts and valuable feedback. We will follow your advice to revise the claim of the event slot to more accurately reflect the capabilities and limitations. Particularly, we will reflect its functioning on "large local regions (e.g., patches of 56x56 pixels)" and capturing of portions of events. 

We appreciate your constructive feedback, which contributes a lot to the improvement of our work!

Thank you!
We are very grateful for your valuable suggestions and feedback! Incorporating these results and clarifications are very helpful to make this paper more solid and comprehensive.  Thank you very much for your great efforts and time!
I thank the authors for their response. 
My main concern about the clarity of "event slots" has not been addressed well, which is the major claim of the paper. As mentioned by reviewer KJVB, the paper only considers "events" that represent the temporal change within a patch. This would limit the proposed method to a simple domain of activities. For example, such a definition of event slots would not be able to handle a scenario where a subject manipulates objects with hands (e.g., Assembly-101 and Ego4D) or kitchen scenarios (e.g., Epic-kitchen) where the subjects and their hands move consistently. I would suggest that the authors revise the claim of the event slot as the term can be misleading.
Moreover, from the rebuttal by the authors "slots is performed on CLIP features, which possess high-level semantics, thus enhancing the slots’ ability to capture high-level semantics" is not very convincing. The statement should be substantiated with some qualitative results that can demonstrate the event slots actually capture high-level semantics, as the current ones do not really show meaningful semantics.
Thank you very much for your comprehensive efforts in addressing my concerns. In particular, I find your response on extending the proposed approach to much longer videos insightful. I also appreciate your efforts on the additional ablation experiments, given the time and computational constraints on your end, as well as providing more evaluation results on additional QA benchmarks. Thus, I will retain my original rating.
Thank you very much for your insightful questions and constructive suggestions, which are very helpful in improving our work.  We will add the clarification related to event slots into the revised manuscript. 

**About the visualization**: As far as we know, DINOSAUR [27] as you referred, which learn the object-centric representation on pre-trained feature space (e.g., DINO, CLIP features) by reconstructing the features, was the state-of-the-art solution on real world data to obtain object grouping results. As the visualization shown in [27], the performance also depends on the complexity of the image contents. It can work well on simple datasets such as MOVi-C, MOVi-E (see Fig. 2 in [27]) but is still not perfect on more complicated images (e.g., COCO). We use the same technique of DINOSAUR to learn slots by reconstructing pre-trained CLIP features. Even though the visualization is still not good, we can observe the learned slots tend to group the features of similar semantics. Our Slot-VLM achieves superior performance, demonstrating strong potential. We believe that more advanced object-centric learning method would further enhance the performance of our framework, and we are eager to have a try if we can use a better object-centric learning method. 

[27] Bridging the Gap to Real-World Object-Centric Learning, ICML'23.

**About preservation of information**: Thank you for the good question. The dimension of a slot is 1024. Slot-attention module generates compact slot representations, where the number of output slots is much smaller than the number of input tokens. Actually, the embedding dimension is **not** low.  Take the Object-Slots branch as an example. For a video frame with feature of 16x16x1024 (height x width x dimension), i.e., 16x16=256 tokens, slot attention taken the 256 tokens as input to generate 8 slots with each of 1024 dimensions. After going through the projection layers, the slots for t (t=8) frames are sequentially concatenated to have 64 slots, as the input to LLM, with each slot as a token. 

Slot attention module is optimized to enable slots to reconstruct the input as better as possible, even though the bottleneck/compression would lead to loss of some information. Each slot is a weighted summarization of the input. In contrast, the forming of clustering center (using pooling/averaging operation) is not optimizable to preserve information. Therefore, at the same number of output tokens, slots are able to preserve more information.   

**About SlowFast and Slot-Joint-VLM**: Yes, the total number of slots are the same. For the cross-attention operation in slot attention module, we know the complexity is proportional to the number of Key/Value (i.e., number of input tokens). Our decomposition of the spatial-temporal tokens to spatial and temporal enables the slot-attention to operation along each dimension (spatial or temporal), which greatly reduced the number of Key/Value (input tokens) to infer slots. In addition, we have conducted temporal down-sampling in the spatial branch (Object-Slots branch) while preserving the spatial resolution. Similarly, we have conducted spatial down-sampling in the temporal branch (Event-Slots branch) while preserving the temporal resolution. This further reduces the complexity. We will add more analysis to our revision.

Thank you again for your great efforts to make our paper clear and solid. Any questions/comments are warmly welcomed!
I appreciate the detailed rebuttal from the author.

Regarding the definition of the event slots, please ensure to add the clarification into the paper, as this definition is somehow misleading.

As for the poor visualization result, I understand that object-centric learning methods struggle to handle real-world scene. But some related works, e.g., <Bridging the Gap to Real-World Object-Centric Learning> already shows decent object grouping results on real-world data. Therefore, I'm still a bit confused about the visualization. If the visualization is not good, then the object-slot and event-slot definition is a bit more questionable.

For the conceptual comparison with Chat-UniV, the authors mentioned that "Slots can preserve more original information than clustering centers." and "Chat-UniV ... averaging can result in a loss of fine-grained details". As far as I know, slot-attention module usually uses a very compact slot representation with low embedding dimension to form a information bottleneck. I'm curious about the implementation of the slot attention in the paper. What is the dimension? How do you use the slots to input to the LLM? Why does it preserve more abundant information than average pooling visual tokens?

And the last point about SlowFast and Slot-Joint-VLM, since there would be a hyper-parameter to define the number of slots in your model, I wonder if you define the total number of slots as the same in the two settings in Table G, what brings the significant difference in the FLOPs?

Thanks.
Thank you very much for your constructive suggestions. We will include these results in our revised manuscript. 

Moreover, we will follow your insightful suggestions to present more comprehensive comparisons, including comprehensive results and model/training information. 

Thank you again for your great efforts to help us improve!
Thanks authors for their every effort to provide more results on multi-choice QA benchmarks/datasets.

Happy to see the proposed Slot-VLM achieves comparable performance with limited resources.
Overall, I would highly suggest including that multi-choice exp as a main result in the final version, considering it is a more stable/interpretable evaluation and can provide more grounded hints for further work. 

On the other hand, I would also suggest including a more inclusive comparison of those multi-choice QA benchmarks, for example,
(1) SeViLA achieves higher performance on zero-shot nextqa/star with flant5-3B
(2) Videochat2 achieves higher zero-shot performance on star with the similar LLM.

Beating absolute sota models generally is not the main research focus/interest from my view, but the effectiveness of the proposed slot design (which is already included in paper ablation studies), however, I believe a comprehensive table (e.g. including connector/LLM types/extra pre-training) provides more interest/solid insights for future module designs. 

Thanks for the effort again, and I am willing to increase my score based on these rebuttal results.
For the schemes in our response A2, the paper for ViperGPT is "ViperGPT: Visual Inference via Python Execution for Reasoning".
 
The paper for LLoVi is "A Simple LLM Framework for Long-Range Video Question-Answering".
 
The paper for mPLUG-Owl is "mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality".