Slot State Space Models
Authors: Jindong Jiang
Fei Deng
Gautam Singh
Minseung Lee
Sungjin Ahn
Venue: NeurIPS 2024 poster (2024)
Keywords: State-Space Models
Object-Centric Learning
Video Understanding Models
Spatial-Temporal Reasoning
Primary area: deep_learning_architectures
TL;DR: We propose SlotSSM which incorporates independent mechanisms into State Space Models to preserve or encourage separation of information int object-centric learning and visual reasoning.

Abstract: Recent State Space Models (SSMs) such as S4, S5, and Mamba have shown remarkable computational benefits in long-range temporal dependency modeling. However, in many sequence modeling problems, the underlying process is inherently modular and it is of interest to have inductive biases that mimic this modular structure. In this paper, we introduce SlotSSMs, a novel framework for incorporating independent mechanisms into SSMs to preserve or encourage separation of information. Unlike conventional SSMs that maintain a monolithic state vector, SlotSSMs maintains the state as a collection of multiple vectors called slots. Crucially, the state transitions are performed independently per slot with sparse interactions across slots implemented via the bottleneck of self-attention. In experiments, we evaluate our model in object-centric learning, 3D visual reasoning, and long-context video understanding tasks, which involve modeling multiple objects and their long-range temporal dependencies. We find that our proposed design offers substantial performance gains over existing sequence modeling methods. Project page is available at \url{https://slotssms.github.io/}

Slot State Space Models
Jindong Jiang" Rutgers University
Fei Deng Rutgers University
Gautam Singh Rutgers University
Minseung Lee KAIST
Sungjin Ahn"
KAIST
Abstract
Recent State Space Models (SSMs) such as S4, S5, and Mamba have shown markable computational benefits in long-range temporal dependency modeling. However, in many sequence modeling problems, the underlying process is inher- ently modular and it is of interest to have inductive biases that mimic this modular structure. In this paper, we introduce SlotSSMs, novel framework for incorpo- rating independent mechanisms into SSMs to preserve or encourage separation of information. Unlike conventional SSMs that maintain a monolithic state vec- tor, SlotSSMs maintains the state as a collection of multiple vectors called slots. Crucially, the state transitions are performed independently per slot with sparse interactions across slots implemented via the bottleneck of self- atfentttenion. In experi- ments, we evaluate our model in object-centric learning, 3D visual reasoning, and long-context video understanding tasks, which involve modeling multiple objects and their long-range temporal dependencies. We find that our proposed design offers substantial performance gains over existing sequence modeling methods. Project page is available at htttts://s github.ie io/
Introduction
State space models (SSMs) have recently emerged promising class of sequence models, achieving remarkable success in language modeling [27 58 24 50 9 due to their long-term memory capability whose attention mechanisms also
and computational efficiency. Compared Transformers facilitate capturing long-range dependencies, SSMs are more efficient during both training and inference. Notably, SSMs offer parallel training with sub-quadratic complexity, and recurrent generation with constant cost per time step. These benefits have motivated the application of SSMs to sequences of other modalities such as audio [9 and video 10
Typically, SSMs use a monolithic state vector to summarize all past information. This design can struggle model sequences with modular underlying structures, which common in physical processes and real-world dynamics. For example, physical objects largely follow independent dynamics based on their own properties, with strong interactions happening only sparsely (e.g., when objects come in close contact). A monolithic state vector would excessively entangle the dynamics of different entities, thereby hurting generalization. could beneficial to incorporate inductive biases for independent mechanisms [20] into the sequence modeling architecture.
Recent progress in object-centric learning 46 54 36 has led several methods for discovering
modular object-centric structures and modeling their dynamics from videos with no or only weak
supervision (39 [3 57). Similar to RIMs (20). they build modularity into the RNN architecture
to separately keep track of the dynamics of each object. However, RNNs are prone to vanishing
"Correspondence to jindong. jiangerutgers edu and sungj in ahn@kaist. ac. kr.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1:K
1:K
1:K
1:K
Self Altention
Feedlfowwrd
Self Attention
A
Self Attention
SSM
RNN
SSM
ANN
SSM
Input
Input
Ingut
Inpul
Inpui
Input Reedout
22--1
SlotSSMs (Ours)
(b) Single State SSMs
(c) Transformer-based Models
(d) RNN-based Models
Figure 1: SlotSSMs existing models. (a) SlotSSMs incorporate modularity through independent state transitions and sparse interactions via self-attention. (b) Traditional SSMs utilize monolithic state vector for all past information. (c) Multi-slot Transformer-
models offer modularity but with high computational complexity. (d) Multi-slot RNN-based models have modular states but can't parallelize training (red lock). SlotSSMs combine parallelizable training, memory efficiency, and modularity for efficient temporal modeling.
gradients [51] and are not amenable to parallel training, making it hard to scale these methods up to modeling long-range effects that span hundreds of time steps.
In this paper, we propose Slot State Space Models (SlotSSMs), novel and general SSM framework that have built-in inductive biases for discovering and maintaining independent mechanisms. Instead of using monolithic state vectors, SlotSSMs maintain a set of modular slot states whose transition dynamics designed be largely independent, with only sparse interaction across slots introduced through the bottleneck of self-attention. The number of slots can be flexible across the layers of SlorSSMs, allowing slots to have different level of abstraction at each layer. Furthermore, SlotSSMs inherit the strengths of SSMs, namely parallelizable training, memory efficiency, and long-range reasoning capabilities, giving it an advantage over methods based on RNNs and Transformers.
Our contributions are summarized as follows. First, we propose SlotSSMs, a novel and general architecture that incorporates independent mechanisms into SSMs for modeling inherently modular physical processes. Second, we show that SlotSSMs can be specialized to solve object-centric learning tasks. It achieves comparable better performance than existing RNN- methods and the Transformer baseline that we develop, while being more computationallyy efficient. Third, we further investigate the abilities of SlotSSMs general sequence modeling framework, demonstrating its advantages in video understanding and prediction, long-range reasoning, and 3D visual reasoning.
Preliminaries
A state space model (SSM) defines a mapping between input sequence ei.r E RxDD and an output sequence fur E T'DD X via the recurrence [28] 58,
h; A,h,-1+ B,e. y: C,h..
(I)
Here, T' is the sequence length; e,, M: E ER are input and output vectors at time and h, R the hidden state summarizing the history est. The matrices A, # B, HxD and designed with learnable parameters in specific ways that encourage modeling long-range range dependencies while maintaining computational efficiency. For example, A, commonly takes a diagonal or dlock-diggon form, with its (complex) cigenvalues distributed close to the unit
C, PxH
circle initialization [25] 27 29
26
58 50]
When A, B,, C, are time-invariant (constant over t), the computation of f1.r can be parallelized, enabling efficient training. Recent works [24 9 further show that conditioning these matrices on the input e, does not hinder training efficiency. They employ learnable functions A: RRD R H
H X B: RDD
C: RD + DxH to generate input-dependent matrices:
A, = A(e))
T B(e), C, = C(e))
(2)
This allows the model to selectively emphasize or ignore information based on the input, leading to more flexible sequence modeling.

---

SSM
SlotSSM
h
A(e)
h +
B(e)
Alr
Cirl
C(e)
Figure 2: SSM vs SlotSSM. SlotSSM encourages modularity by maintaining set of separate slot state representations, each updated independently using separate transition matrices and input matrices, allowing for more efficient scalable modeling of complex sequences with inherent modular structures.
Due to the Dolock-)diagone structure A, limiting cross-dimensional information flow, SSMs are typically interleaved with mixing layers (e.g.. linear projections MLPs) to mix information across dimensions. Alternatively, using dense B, and C₁ matrices can also enhance mixing.
3 Slot State Space Models (SlotSSMs)
Standard SSMs use monolithic vectors for inputs, outputs, and hidden states, and mix information across all dimensions. This lack of modularity could cause difficulties in modeling real-world dynamics such as object interactions, where underlying process consists of multiple entities and inherently modular [20]. In this section, we present slot state space models (SlotSSMs), new class of SSMs with built-in inductive biases for encouraging and preserving modularity.
Our key idea is to maintain set of separate slot state representations (called slots in short), and process the slots independently and symmetrically. To do this, format the input vector e, R as
a concatenation of K slot representations { RR D. 1P1A KK
k== where D = D/K. The output ye E R
and hidden state h2 E R are formatted similarly:
e, = concat [8) [8.....,8
K y: = concat [y!.....y:" ]]
h; = concat [h!....,,
h
(3)
where y E DR and h RRH, are the output and the hidden state corresponding to slot sf with H. H/K. In this section, we focus preserving modularity when the input already complies with the slot format. When coupled with a slot encoder, the SlotSSM can help encourage the emergence of modularity from unstructured inputs such video frames,
will discuss in Section
To preserve modularity, we make sure that SlotSSM do not mix information across different slots. More precisely, the hidden state h and output S only integrate information from the history of the corresponding input slot a As illustrated in Figure (Right), this can be achieved by making A,, B,, C, block-diagonal, where the k-thh block is only conditioned the k-th slot:
A, diag (()))) B, = diag (B(s)))
C, diag ((C())))
(4)
Implementation details. The SlotSSM formulation in Equation is general and accommodate various choices of the A, B, C' functions. In our implementation, we adopt those from Mamba [24]. Specifically, A(st), ((s)), C(sf) are themselves block-diagonal matrices with D. blocks, one for
each slot dimension. The i-th blocks A (s?) E R N and B' (s/) E Nx are obtained by
discretizing their continuous-tiin time counterparts A (i) and B(i ((s(stt) using the time step A0) and the zero-order hold (ZOH) rule:
A (1) (t)), B ( ZOH (s)) A B ((( i= L.
(5)
Here, N = H,/ D. is the hidden state size per slot dimension, ( E NNN is an input-inde learnalle model parameter, and A RD - R. B R N are learnable functions imple- mented as neural networks. Similarly, the i-th block ((
(sf)) is computed by the learnable function

---

Slot Mixer
MLP
MLP
Maj
Adee
Self Attention
SlatSSM
Aijc
Arjs
Bii
Mij
Linear
CLSi
CLS
Slot Encoder
Transformer
CT.S
C
Figure 3: Sequence modeling with SlotSSM. Each layer includes Slot Encoder, SlotSSM, and Slot Mixer. The Slot Encoder uses Transformer to extract slots from inputs. The SlotSSM independently updates the slots via separate transitions. The Slot Mixer introduces inter-slot interactions through self-attention.
C RD +RINN For simplicity and efficiency, (*)) and C() are shared across all S iis D.. and ( is parameterized as diagonal matrix.
Modular Sequence Modeling with SlotSSM
The SlotSSM proposed in Section 3 are designed to preserve modularity when the input is already separated into slots. In this section, we complement SlotSSM with a slot encoder that extracts slot representations from unstructured inputs (Section J.J.. and a slot mixer that introduces sparse interactions across slots (Section 4.2). We then present sequence modeling architecture (Section 4.3) that encourages discovery of underlying modular processes by stacking these components.
4.1
Slot Encoder
We assume the unstructured input æ; at each time step is represented as a sequence of M tokens:
2t (x)
M.M
xt m
RDD=
For example, image inputs can be CNN feature maps (M is the number of cells in the feature map), embeddings of non-overlapping image patches (M is the number of patches), as proposed in To extract K slot representations from æt,
VIT [2]
K learnable CLS 3 tokens (CLS,
R D2 1= as queries and perform cross-attention with the input tokens through a Transformer [63
{CLS/)K + Transformer (q {CLS kv (x
[x J==1)
The Transformer also includes self-attention within the CLS tokens, allowing them to communicate with each other and capture information from different parts of the input, thereby facilitating the emergence of modularity. The slot representations are then obtained by applying a linear projection the corresponding output embeddings of the CL.S tokens:
Linear(CLss)) A == 1, 1,..,,KK K.
(8)
4.2
Slot Mixer
The slot encoder obtains slot decomposition purely based single time steps, which can be subop- timal. addition, the SlotSSM processes each slot fully independently, making hard to correct mistakenly decomposed slots model interarctions across slots. To resolve both issues, interleave SlotSSM with slot mixers.
Following the tradition of VIT, use "CLS" distingish learnable tokens from observation tokens.

---

The slot mixer consists of two residual blocks, and is applied to the outputs {y/ fk=1 of the SlotSSM. The first block introduces interaction across slots through self-attention [63], whereas the second block uses MLP to further process the gathered information within each slot:
.... ( Vi -- (y) yt + SelfAuun(LNNy)) ..., LN(D()) MLP(LN(y.
(9)
(v)
yi ) -- (y) y + (MIP(INN()))
(10)
Here, LN(-) denotes layer normalization [2]. Because y carries information from the entire history of each slot, provides the opportunity to refine the slot representations based on temporal dynamics.
4.3 Sequence Modeling Architecture
We now present generic architecture for modeling sequences with modular underlying processes. Given sequence of unstructured inputs Fı.r, our goal is to obtain a set of K₁ modular representations at each time step and each layer I that summarizes all underlying processes up to time t.
In general, the number of slots K2 at each layer can be different, potentially allowing fewer but more abstract slots at higher layers. To accommodate this, we insert slot encoder wherever the number of slots changes, and repurpose it to extract a different number of slots from existing slot representations. This is achieved by treating the slots output from the previous layer as keys and values in Equation When the number slots does not change, we can simply copy the slots from the previous layer.
As shown in Figure 3 our proposed architecture stacks the (optional) slot encoder, SlotSSM, and slot mixer together at each layer. Variables layer denoted with the subscript 'IK". The slot mixer's output from layer / - 1, 1 Ki_1 serves as input to layer I. The initial input is { where Ko == M. The computations at each layer = 1,... L are:
K, { / k=1 = ElotEnr (fzf/-1 1k=e)
(11)
& Ki W
(h ( k=1 SloisSM ((/))) (he (11-11)
Kg
(12)
{ k SlotMixer ((2())
(13)
The final output 1 ) can be used for various tasks, such as predicting the next observation and the properties of underlying processes (e.g., position, velocity).
Object-Centric Learning with SlotSSM
In this section, we present concrete example of adapting the generic sequence modeling architecture proposed in Section 4 to solve specific task. We consider the task of object-centric representation learning from unannotated videos of interacting objects, a typical example of sequences with modular underlying structures. The goal is to obtain representation for each individual object that captures relevant attributes such as object position, size, shape, color, etc. without any object-level annotation.
5.1
Object-Centric SlotSSMs (OC-SlotSSMs)
we make slight modifications to our sequence modeling archi- tecture facilitate the discovery of modular structures. We call the resulting model OC-SlotSSMs. First, we use the same number slots across all layers. is thus unnecessary to have slot encoder per layer. However, we find it helpful to still have it, but in another form that encourages iterative refinement of the slots. Specifically, we use the slots output from the previous layer (0/ K /=
Inspired by previous works 46 67
as queries, and provide the input tokens {a m=1 keys and values. Second, we introduce competition among slots the attention layers of the slot encoder. We achieve this by using inverted attention 61 67 which is essentially cross attention with the Softmax operation performed over the queries instead of the keys. This has the effect of softly assigning each input token to a slot, thereby promoting modularity. The computation at each layer 1,..., .,, L can be summarized as follows:
5

---

( ) InvAttn q - 1 kv = {arfjo the 3M
(14)
(u ( {arj )
SlotSSM (0() (11-111)
(15)
SlotMixer
(v. (=)))
(16)
We note that the queries in the first inverted attention layer are the learnable CLS tokens {CLS) )
k=1-
5.2
Training Pipeline
Following previous works in object-centric learning |4 pipeline. Given a sequence video frames (o, E sequence modeling architecture by applying CNN encoder each frame Og and adding positional embedding for each feature map cell. The output slots {æt 1K are each decoded into an object
adopt auto-encoding training x3 1T we obtain the input 2 to our <R
39
57],
image of TR. xx33 and an alpha mask af E RR#xxx1 by spatial broadcast decoder [66]. The final reconstruction 6, E R H x3 is given by the alpha-composition of the object images:
K
exp(a²)
o o = (a)p)er(aaip.. 6,
of
(17)
Σj= exp(a!)
The training objective to minimize the reconstruction error L = +i 160 - -0:
Related Work
State Space Models (SSMs). Popularized by S4 (27.. SSMs have attracted growing interest in language modeling and as a sequence modeling framework in general. The original S4 follows the HiPPO theory 22 to parameterize and initialize the state transition matrices, which is quite mathematically involved. Most recent works have proposed simplified versions that use diagonal tran- 26 58 and pure RNN formulation (i.e., without reliance ODE discretization) 30 (50 9.. Several works have proposed hybrid architectures of SSMs and Transformers to incorpo-
sition matrices 29
rate their complementary strengths 775 48
17
32 24 In addition to language modeling. SSMs have
been applied to various domains, including time-series generation 73). audio generation (19] visual
classificiation and generation [49 40 32 65
7117 and reinforcement learning 47 10 52). Our
study introduces the first SSM with inductive biases for modeling inherently modular processes.
Object-Centric Learning. Object-centric learning seeks to discover modular structures and indepen- dent mechanisms (00 such as objects and their relations from multi-object images and videos with
weak or supervision B 22 38
23 15
14 [6 (] 35 37
6
[4] [1] 64 68
53
Recent and compet-
34]
works are predominantly based on the Slot Attention
46 model, which uses a GRU 5
itive attention mechanisms to iteratively refine slot representations [54 57
39
13
69
55
36
70).
SI and the train- ing must be done in sequential way. These weaknesses render them incapable of scaling up to long-range videos. Additionally, hardware parallelization for video object-centric learning has been explored in (56), however, it incurs quadratic cost unlike ours. Our SlotSSMs framework can be specialized to address object-centric learning tasks effectively. By integrating SSMs at its core, SlotSSMs benefit from parallelizable training and possess remarkable long-term memory capabilities. Moreover, versatile framework, SlotSSMs are well-suited suited to tackle other tasks such as long-range range visual reasoning.
However, GRUs and RNNs in general are prone to vanishing gradient issues
7 Experiments
We present an extensive evaluation of our models across variety of tasks. Section 7.1 need for modular latent states through a multi-object video prediction task. Section the advantages of SlotSSMs over Transformers and RNNs using newly proposed long-context rea-
illustrates the 7.2 demonstrates
soning benchmark. Section 7.3 investigates the object-centric learning capabilities of OC-SlotSSMs.
Finally, Section 7.4 showcases the 3D visual reasoning capabilities using the CATER benchmark [18).

---

9M680 crovee
Toyn
Bouncing Ball Video Prediction
3.0
SlotSSM
Mean Squared Error
Single State SSM
Saee SBM:
2.5
SingleSSM (Split)
2.0
Transformer
Transforrr
1.5
RIM
RIM Sartrers Trre
1.0
0.5
0.0
5
10
15
20
25
30
Time Step (Every 2nd Step Shown)
Time step
Figure 4: Multi-Object Video Prediction Task. Left: Generated video frames at every second step, showing 10 of the 20 total frames generated. Green color indicates ground-truth and red color indicates predictions. Right: MSE over 20-frame autoregressive rollout, given 10 context frames. SlotSSM demonstrates its efficiency in modeling multi-object dynamics.
7.1 Multi-Object Video Prediction
We begin with a multi-object video modeling task to demonstrate the benefit of incorporating modularity into state space.
Dataset and Task. We utilize the bouncing balls video dataset introduced by [62), which consists of videos of white balls bouncing off each other in an empty window. Each ball has random initial positions, velocities, and masses, governing their interactions. The task is conditional video generation, specifically p(x++1:7 I:T+ x1:r). This task is inherently modular as it requires models to remember each object's s attributes and interaction rules.
Experimental Setup. We train models 20-frame sequences using seacher-foreing and binary cross-entropy loss. At test time, given T = 10 context frames, the model autoregressively predicts W = 20 future frames using its outputs. Performance is evaluated using Mean Squared Error (MSE) between predicted and ground-truth images.
Models. We employ the SlotSSM architecture described in Section 4.3 We use the same number of slots across layers and apply the Slot Encoder only at the first layer. We compare our model against several baselines: Single State SSM, which shares the same architecture but uses monolithic state; Single State SSM (Split), which uses a Single State SSM with multi-slot encoder and decoder-ske slots concatenated in SSM, then split into multiple slots for the decoder; RIM[20], slot-based RNN model with separate RNN weights per slot that introduces sparse slot updates and interactions based on input attention values; Transformer, vanilla Transformer model with single input embedding per time step; and Slansraasfrm,,r, Transformer model with multiple input slots at each time step.
All models share the same encoder and decoder architectures. The encoder is Transformer described in Section 4.1] using a single CLS token for single- state models. The decoder consists of three Transformer layers with self-attention for image patches and cross-attention to query the slots. We use six slots for all slot-based models. For RIM, we set k = 4 for top-k active modules as in the original paper. We carefully match hyperparameters across baselines to ensure comparable model sizes, except for RIM, which inherently requires a larger model due to separate RNN weights per slot. Additional implementation details are in Appendix C
Results. Figure 4 compares model performances, showing that SlotSSM outperforms all baselines. including slight improvement over SlorTransformer, The significant gap between SlotSSM and Single State SSM underscores the importance of modular slot states for effective multi-object dynamics learning, as also evidenced by the comparison between Transformer and SrnnTransformer SlotSSM also significantly outperforms Single State SSM (Split), which uses the same modular encoder and decoder, highlighting that modularity in temporal modeling tte core contribution of SlotSSM critical for improved performance. While the RIM model performs better than other single-state basclines, it still lags behind SlotSSM and SloTTransformer.
7

---

(a) Long- CongtCt Construction
(b) Model Efficiency
2 3
TT-
T
Inference Latency (
Seconds (log scale)
2,000
0.400
0.100
0.020
Patchifier
0.005
80
160
Model
Sequence Length (log scale)
SlotSSM
SlotTransformer
SlotRNN
RIM
Input Sequence
Figure 5: Long-Context Construction and Model Efficiency in the Blinking Color Balls Benchmark. Left: We construct cong-sequence inputs by patchifying the context images. Righr: Comparison of model inference
latency with batch size 6. SlotSSM demonstrates computational efficiency for long oong-eqquence processing tasks.
7.2 Long-Context Reasoning
We now evaluate the long-context reasoning capabilities of SlotSSM. To enable rigorous assessment in multi-object setting, we propose the novel Blinking Color Balls Benchmark.
Blinking Color Balls Benehmark. This benchmark has two variants Earliest Color and Most Frequent Color-each consisting of image sequences with context images X1:7-1 and a target image X7. In each context image, one ball is randomly selected and assigned non-white color from five options while others remain white. The coloring of balls in the target image Xy depends on the variant: in Earliest Color, each ball's color is its earliest assigned non-white color in the context (remaining white if none); in Most Frequent Color, each ball's color is the non-white color assigned most frequently during the context (ties broken by earliest assignment; remaining white if none).
To create a long-range reasoning task, we further patchify each context image into non-overlapping P x P patches and flatten them into sequence of length p2, shown in Figure(S(a) With context length T 1, the total input sequence length is L = (T - 1) x p2, Models must identify and track objects from partial patch views while remembering and counting color assignments, making the task highly challenging. The final task is to predict the target image given this long sequential input.
Experimental Setup. We evaluate models on Earliest Color with T = 6 and Most Frequent Color with T E {6. 11], using patch sizes E {4, 8, 16). This yields input sequence lengths L E f80, 160, 320, 640, 1280, 2560). The Most Frequent Color with T = 11 requires stronger memorization and reasoning capabilities due to longer context and more color assignments,
Models. We employ the same encoder, decoder, and the SlotSSM architectures as in Section 7.1 For slot encoding, each image patch is treated as an image and processed by the slot encoder. The slots from the last time step are provided to the decoder to predict the full target image. We compare our SlotSSM against several baselines: Single State SSM, SlorTransformer. and RIM. Additionally, we introduce novel slot-based design called SlotRNNs, which shares RNN weights across slots and uses self- attentttnn layers between time steps as the slot mixer. SlotRNNs can viewed as a special case of RIMs with shared weights and dense state updates. Empirically, SlotRNNs exhibit more stable training and improved performance compared RIMs. For fair comparison, all slot-based models use six slots, and we carefully match model sizes as in Section 7.1
Results. Figure shows that SlotSSM outperforms Single State SSM, SlotRNN, and RIM across all sequence lengths. For shorter sequences (80 and 160), Single State SSM and SlotR have relatively low eιror rates but degrade significantly beyond 320 frames. Surprisingly, RIM fails to generalize at any sequence length, likely due to optimization issues from separate weights per slot; our SlotRNN ad- dresses this by sharing weights across slots while maintaining modularity. SlotTransformer performs competitively up to 640 frames. However, SlotSSM demonstrates superior long-range reasoning, especially at 1280 and 2560 frames. where other models cannot run due to memory or optimization constraints. Figure ()(b) highlights SlotSSM's 's computational efficiency. SlorTransformer's inference latency increases rapidly with sequence length due to quadratic complexity, SlotSSM maintains stable and efficient inference across all lengths. Due to Tronsfrrorrrr''ss high memory usage, we used a batch size of 6 for latency evaluation. Qualitative comparisons in Appendix B.3 provide further insights into the models' strengths and weaknesses.

---

Earllest Color Frames
Frequent Color-
Most Frequent
0.06
Mean Squared Error
Mean Squared Errer
0.04
0.04
0.03
0.03
0.00
0.02
Mean
0.02
0.02
0.01
0.01
0.01
80
320
1260
80
320
1260
160
640
2560
Sequence Length !log
Sequence Length (log scale)
Length (log
SlotSSM
SlotTransformer
Single State SSM
SlotRNN
RIM
Figure 6: Long-Context Reasoning in Blinking Balls Benchmark. SlotSSM maintains consistent performance across sequence lengths from 80 to 2560, whereas baseline models show degraded performance or fail to complete training due to high memory and computational requirements.
(TV)A(T)
SAM
00-3383M
0.44
0.2
FG-ARI
Position
Color
FG-ARI
mloU
Posttion
Time Step
Figure 7: Object-Centric Learning Results. Left: Qualitative comparison of segmentation masks on MOVi-A. OC-SlotSSM demonstrate less object spliting and better boundary adherence. Right: Quantitative evaluation unsupervised object segmentation altribute prediction. OC-SlotSSM outperforms SAVi on most metrics.
7.3 Unsupervised Object-Centris Learning
In this section, we evaluate the performance of the Object-Centric SlotSSMs (OC-SlotSSM) variant in unsupervised object-centric representation learning.
Datasets. We evaluate OC-SlotSSM on the MOVi-A and MOVi-B subsets the MOVi video dataset 21. which contain videos of up to 10 objects moving in 3D environment. MOVi-B adds complexity over MOVi-A by including a wider variety object types and multi-colored backgrounds.
Tasks. Following prior object-centric learning works 46 36), we evaluate models two downstream tasks: unsupervised object segmentation and attribute prediction. For segmentation, we report FG- ARI and mloU metrics. For attribute prediction, we measure the quality of representations by inferring object properties: we report prediction accuracy for discrete attributes (e.g., object shape) and R2 for continuous attributes (c.g., object position).
Models. We compare OC-SlotSSM to SAVi [39] an RNN-based object-centric learning approach. Both models use CNN encoder to extract image features as input tokens æt 0 m = MM which are processed by their respective attention mechanisms innranttms-iieertedd inverted attention OC-SlotSSM and slot attention in SAVi-r to produce slots. These slots are then used to reconstruct the image and generate per-object segmentation masks via spatial broadcast decoder, with reconstruction as the training objective. For unsupervised object segmentation, we directly use the object masks obtained during training. For attribute prediction, we match slots to object IDs using Hungarian matching based on segmentation masks, then use linear heads and 2-layer MLPs to predict discrete and continuous attributes, respectively, keeping the slots frozen.
Results. Results in Figure 7 demonstrate that OC-SlotSSM consistently outperforms SAVi in unsupervised object segmentation both MOVi-A and MOVi-B. The qualitative comparison (Figure 7 left) shows that OC-SlotSSM generates masks with tighter object boundaries and fewer object splitting, which also leads to improved attribute prediction accuracy (Figure right). Furthermore, we empirically found that OC-SlotSSM exhibits superior stability during training compared to SAVi, which tends collapse into a single slot representing the entire scene when trained long enough. This collapse is not reflected in the validation loss, so we apply early stopping based on manual inspection. In contrast, OC-SlotSSM does not suffer from this instability, demonstrating robustness in learning object-centric representations.

---

Table 1: Performance CATER Snitch Localization Task.
No Pre-train
Pre-train
Model
Top-1 Ace (%)
Top-S Ace (%)
Top-1 Acc (%)
Top-5 Acc (%)
Single State SSM
10.27
27.21
41.15
65.70
SlanTrassformer
41.09
62.24
49.21
70.24
SlotSSM
25.64
45.03
54.73
74.42
OC-SlotSSM
61.58
84,00
69.27
90.48
7.4 3D Visual Reasoning
Finally, explore the application of SlotSSM and OC-SlotSSM to 3D visual reasoning tasks using the CATER benchmark [1].
CATER Benchmark. CATER consists of 300-frame video episodes of objects moving in 3D environment. The movement can lead to partial occlusions and even complete coverage of smaller objects by larger ones. The primary task is snitch in the final frame. The snitch is always present but may be occluded. Models must reason about its location based on the last visible position and other objects' movements. Success in this task demonstrates models' capacity for complex visual reasoning in dynamic 3D environments.
cnttiiaation-predicttin the golden snitch's 's location
Experimental Setup. We consider two experiment settings: direct training and pre-training + fine- tuning. direct training, models are trained end mod-to-end on the snitch localization task. pre-training + fine-tuning, models are first pre-trained on video inputs using reconstruction objective, then fine-tuned on the task-specific signal. During pre- pre-training, we randomly sample frames from the 300-frame videos. For direct training and fine-tuning, we split the sequence into 50 non-overlapping segments of frames each, randomly selecting one frame from each to create 50-frame sequence spanning the entire video. At test time, we evenly sample 50 frames by skipping every 6 frames. The snitch's final location is quantized into 6x6 grid, framing the problem as a classification task.
Models. We evaluate the performance of SlotSSM, OC-SloISSM, Single State SSM, and Slot- Transformer. We exclude RNN-based baselines, our preliminary experiments reveal that they are unstable when handling long video inputs and prone to collapse to constant output. For the visual pre-training setting, we employ spatial broadcast decoder to reconstruct the input images. During downstream training/fine-tunin
we feed the slots from the final step to transformer predictor with single CLS token, followed by a linear layer on the output CLS token to predict the snitch's position.
Results. Table T presents the Top-1 and Top-5 accuracy on the CATER Snitch Localization task. Consistent with our previous findings, SlotSSM outperforms Single State SSM, highlighting the importance of modular latent structures. Comparing SlotSSM with SlorTransformer, we see notable differences between direct training and pre-training settings: in direct training, SlofTransformer surpasses SlotSSM, possibly due to optimization advantages from direct access to all previous states; however, SlotSSM benefits more from pre-training likely due to the explicit memory capacity of SSM states, consequently, pre-trained SlotSSMs outperforming their SlorTransformer counterparts.
Remarkably, OC-SlotSSM achieves the highest accuracy, outperforming all baselines by large margin in both direct training and pre-training settings. This performance gain may be attributed to the explicit decomposition into object-centric representations, which facilitates reasoning about object properties, relationships, and interactions.
8 Conclusion
In this work, we presented SlotSSMs a novel approach to incorporating modular structure and inductive biases into State Space Models for improved sequence modeling. By maintaining collection of independent slot vectors and performing state transitions independently per slot with sparse interactions via self-attention, SlotSSMs effectively captures the inherent modularity present in many real-world processes. The experimental results in object-centric video understanding and video prediction tasks demonstrate the substantial performance gains offered by SlotSSMs over existing sequence modeling methods.
10

---

Acknowledgements
This research was supported by GRDC (Global Research Development Center) Cooperative Hub Program (RS-2024-00436166) and Brain Pool Plus Program (No. 202/H1D3/2203110 103645) through the National Research Foundation Korea (NRF) funded by the Ministry of Science and ICT.
References
[1] Titas Anciukevicius, Christoph Lampert, and Paul Henderson. Object-centric image gen- eration with factored depths, locations, and appearances. arXiv preprint arXiv:2004.00644 2020.
[2] Jimmy Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06453 66450000.065,,. 2016.
[3] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint 19X1v:1901.11300 2019.
[4] Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. TransDreamer: Reinforcement learning with Transformer world models. Deep RL Workshop NeurlPS 2021, 2021.
[5] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: IXX.33542..355, 2014.
[6] Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. arXiv preprint arXhv:1911.09033 2019.
[7] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convo- lutional neural networks. In Proceedings of AAAI, 2019.
[8] Shmuel Bar David, Itamar Zimerman, Eliya Nachmani, and Lior Wolf. Decision S4: Effi- cient sequence-based RL via state spaces layers. In International Conference on Learning Representations, 2023.
[9] Soham De, Samuel Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402. 19427, 2024.
[10] Fei Deng, Junyeong Park, and Sungjin Ahn. Facing off world model backhones: RNNs, Transformers, and S4. Advances in Neural Information Processing Systems, 36, 2024.
[11] Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative ICLR 2021: The Ninth International Conference on Learning Representations, 2021.
graph networks. In
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[13] Gamaleldin F. Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Cur- tis Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real- world videos. ArXiv, abs/2206.07764, 2022.
[14] Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-VE V2: Inferring unordered object representations without iterative refinement. arXiv preprint arXiv:2 00400999099. 2021.
[15] Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative scene inference and sampling with object-centric latent representations. In ICLR 2020 Eighth International Conference on Learning Representations, 2020.
11

---

[16] SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, and Ge- offrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pages 3225-3233, 2016.
[17] Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry Hippos: Towards language modeling with state space models, In International Conference on Learning Representations, 2023.
[18] Rohit Girdhar and Deva Ramanan. CATER: and T'mpprrl Reasoning. International Conference on Learning Representations, 2020.
A diagnostic dataset for Compositional Actions
[19] Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. It's raw! Audio generation with state-space models. In International Conference on Machine Learning, 2022.
[20] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. ArXiv, abs/ htos/999..1093. 2021.
[21] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam H. Laradji, Hsuch-Ti Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. arXiv preprint SAKKv:2203.035777 2022.
[22] Klaus Greff, Raphaël Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. arXiv preprint N0Kiv:/903,0040 2019.
[23] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pages 6691-6701, 2017.
[24] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprini urXiv:23/2.00733 2023.
[25] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. HiPPO: Recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems, 2020.
[26] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 2022.
[27] Albert Gu Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representattinss 2022.
[28] Albert Gu, Isys Johnson, Karan Goel, Khaled Kamal Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous time models with linear state space layers. Advances in Neural Information Processing Systems, 2021.
[29] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. In Advances in Neural Information Processing Systems, 2022.
[30] Ankit Gupta, Harsh Mehta, and Jonathan Berant. Simplifying and understanding state space models with diagonal linear RNNs. arXiv preprint arXiv:2212.00768, 2022.
[31] Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 116-8163,, 2024.
[32] Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. In ECCV, 2022.
12

---

[33] Yasamin Jafarian and Hyun Soo Park. Learning high fidelity depths of dressed humans by watching social media dance videos. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 12753-12762, 2021.
[34] Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving Object-centric Learning with Query Optimization. In International Conference on Learning Representations (ICLR), 2023.
[35] Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. In Advances in Neural Information Processing Systems, 2020.
[36] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. Advances in Neural Information Processing Systems, 36, 2024.
[37] Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In International Conference on Learning Representations, 2019.
[38] Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P. Burgess. Simone: View-invariant, temporally-absttrcced object representations via unsupervised video decomposition. arXiv preprint arXiv:2106. 033444 2021.
[39] Thomas Kipf. Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. arXiv preprint arXiv:2111.12544 2021.
[40] David M Knigge, David W Romero, Albert Gu, Efstratios Gavves, Erik J Bekkers, Jakub Mikolaj Tomezak, Mark Hoogendoom,, and Jan-jakob Sonke. Modelling long range dependencies in ND: From task-specific to general purpose CNN. In International Conference on Learning Representations, 2023.
[41] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pages 8606-8616, 2018.
[42] Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman. Discovering important people and objects for egocentric video summarization. In 2012 IEEE conference on computer vision and pattern recognition, pages 1346 1353. IEEE, 2012.
[43] Zhiqi Li. Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's s-cce-vieww representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pages 1-18. Springer, 2022.
[44] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Jindong Jiang, and Sungjin Ahn. Improving generative imagination in object-centric world models. In International Conference on Machine Learning, pages 41 4124, 2020.
[45] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference Learming Representations, 2020.
[46] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
[47] Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning. arXiv preprint arXiv:2303.03982. 2023.
[48] Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via gated state spaces. In International Conference on Learning Representations, 2023.
13

---

[49] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah. Tri Dao, Stephen Baccus, and Christopher Ré. S4ND: Modeling images and videos as multidimensional signals with state spaces. In Advances in Neural Information Processing Systems, 2022.
[50] Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, 2023.
[51] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, 2013.
[52] Mohammad Reza Samsami, Artem Zholus, Janarthanan Rajendran, and Sarath Chandar. Mas- tering memory tasks with world models. In The Twelfth International Conference on Learning Representations, 2024.
[53] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl- Johann Gabriel Tong He, Zheng Zhang, Bernhard Scholkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning, arXiv preprint DxXiv:220911866 2022.
[54] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-e learns to compose. In International Conference on Learning Representations, 2022.
[55] Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural Systematic Binder. In International Conference on Leurning Representations, 2023.
[56] Gautam Singh, Yue Wang. Jiawei Yang. Boris Ivanovic, Sungjin Ahn, Marco Pavone, and Tong
Che. Parallelized spatiotemporal binding, 2024.
[57] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprint arXiv:2205. /4065, 2022.
[58] Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In International Conference Learning Representations, 2023.
[59] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaík, Paul Tsui, James Guo. Yin Zhou, Yuming Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[60] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long Range Arena: A benchmark for efficient Transformers. In International Conference on Learning Representations, 2021.
[61] Yao-Hung Hubert Tsai, Nitish Srivastava, Hanlin Goh, and Ruslan Salakhutdinov. Capsules with inverted dot-product attention routing. In International Conference on Learning Representations, 2020.
[62] Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv: 1802. 1005, 2018.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[64] Julius von Kügelgen, Ivan Ustyuzhaninov, Peter V. Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. arXiv preprint arXiv:2004. /2906, 2020.
[65] Jue Wang. Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Selective structured state-spaces for long-form video understanding. In CVPR, 2023.
14

---

[66] Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerehner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprini urXiv:/901.000
2019.
[67] Yi-Fu Wu, Klaus Greff, Gamaleldin Fathy Elsayed, Michael Curtis Mozer, Thomas Kipf, and Sjoerd van Steenkiste, inverted-attention transformers can learn object representations: Insights from slot attention. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023.
[68] Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Generative video transformer: Can objects be the words? In International Conference on Machine Learning, pages 11307-11318. PMLR, 2021.
[69] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsuper- vised visual dynamics simulation with object-centric models. arXiv preprint arXiv:221000586,, 2022.
[70] Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object- centric generative modeling with diffusion models. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[71] Jing Nathan Yan, Jiatao Gu, and Alexander M. Rush. Diffusion models without attention, 2023.
[72] Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, al. Mm-ego: Towards building egocentric multimodal Ilms. arXiv preprint 24 2024.
[73] Linqi Zhou, Michael Poli, Winnie Xu, Stefano Massaroli, and Stefano Ermon. Deep latent state space models for time-series generation. In International Conference on Machine Learning. 2023.
[74] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint 2xXhv:2401.09417 2024.
[75] Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. Efficient long sequence modeling via state space augmented Transformer. arXiv preprint arXiv:2212.08136, 2022.
15

---

A Limitations & Broader Impact
Limitations SlotSSMs' success illustrates the importance of designing architectures that align with the problem domain's underlying modular structure. It also paves the way for future research in modular and object-centric sequence modeling. However, has some limitations that future studies could address. First, compared to Transformer architectures, we find that SlotSSMs could benefit more from a pre- pre-training phase in visual reasoning tasks. For example, in the 3D visual reasoning task, SlotSSMs underperform Transformer models when trained without pretraining. However, when combined with task- -ree pretraining, SlotSSMs demonstrate significant improvement, enabling them outperform Transformer models. We note that this effect of task-free pre-training is more prominent in SlotSSMs than in Transformer baselines. This suggests that for tasks with sparse training signals, the sequential nature of SlotSSM performs better with a pre-training phase learn effectively utilize information from all time steps. We believe this phenomenon is worth further investigation in future research. Second, while the proposed architecture is applicable beyond video modeling-for example, to other modalities such as text and audio-this study has not explored these possibilities. remains matter for future work. Thrid, due our academic research lab's s computing resource constraints, we were unable to significantly scale up the proposed model to industry-scale in terms of model size and data size. Scaling up SlotSSMs could uncover additional properties or limitations that are not evident at the current scale of experimentation. Lastly, future studies should investigate the effect of increased visual complexity in videos. As an initial step, we present preliminary exploration in Appendix [D.] where SlotSSMs are applied to natural video scenes. These experiments illustrate how modularity can emerge through the independent mechanisms of SlotSSMs in real- world scenarios. We hope these findings will inspire future research on the industry-scale adoption of SlotSSMs.
Impact Statement The introduction of SlotSSMs, novel framework that incorporates independent mechanisms into State Space Models (SSMs), has the potential to significantly impact the field of sequence modeling. By leveraging the modular structure inherent in many real-world processes, SlotSSMs offers a more intuitive and effective approach to modeling long-range temporal depen- dencies in object-centric video understanding and prediction tasks. The substantial performance gains demonstrated by SlotSSMs over existing sequence modeling methods highlight the importance of designing architectures that align with the underlying structure of the problem domain. This breakthrough could lead to the development of more efficient and accurate models for wide range of applications, such robotics, autonomous vehicles, and video surveillance systems. Moreover, the success of SlotSSMs in capturing the modular nature of real-world processes could inspire further research into modular and object-centric sequence modeling. This could result in the development of even more advanced architectures that can better handle the complexity and diversity of real-world data. Because this is general backbone architecture for sequence modeling, it doesn't 't raise direct ethical concerns. However, its ethical implications depend on the way downstream application developers use the model.
B
Blinking Color Balls Benchmark
B.1
Motivation
Real-world videos are often inherently modular, involving multiple dynamic entities and their interac- tions across time. However, existing long-range reasoning tasks, such as those in the Range Arena Benchmark [60], are typically designed to focus on single-object settings and recognizing a single dynamic pattern in the observations. To bridge this gap and facilitate more comprehensive evaluation, we propose the Blinking Color Balls Benchmark, a long-range visual reason benchmark desgined in multi-object setting.
B.2 Dataset Design
We provide an illustrative example of the dataset design in Figure 8 Each episode of the dataset contains a context-target pair XT). At each timestep in X all bouncing balls are first colored white, and then one ball is randomly picked and colored with one of 5 non-white colors. This process is repeated for all context frames, and it is represented in the rows in Figure top). Note that the object picking and coloring are performed independently for each timestep, thus one ball could
16

---

Farliest Colar Rules
Most Frequent Color Rules
Color Assignment
22
33
24
25
I6
Context Sequence
Target Images
Figure 8: Blinking Color Balls Benchmark Overview. Left: Context frames with independent random ball picking and color assignments for each frame. Top figures indicate the sequential color assignment. Right: Target image for the Earliest Color and Most Frequent Color variants. Top figures indicate the color assignment rules.
be selected none or multiple times and colorized with the same or different colors across different timesteps.
The target images are then constructed with two rules: Earliest Color and Most Frequent Color. The Earliest Color rule picks the earliest non-white color assigned to the ball as the final color, while the Most Frequent Color rule counts the assignment of each non-white color and picks the color with the highest count (if there are ties, the earlier color among the highest is chosen). In Figure g we differentiate the two datasets using the same context sequence, which will result in different target images based on the rule. Note that regardless of the color assignment, the objects are moving and follow the physical bouncing rules throughout the full sequence. More image samples can be found in Figure
Finally, as illustrated in Figure 5(a), we transform the conditional image generation task into long-range reasoning task by using patchified context images as input. Instead of providing the T 1 context images directly the model, we flatten non-overlapping patches of the original images to create a long input sequence. Given P x P patches per image, the context length becomes L LT - 1) P Note that patchification is used intentionally construct long sequences for the benchmark; SlotSSMs in general do not inherently require patchified inputs and instead use a Slot Encoder to extract slots as input each time step.
B.3 Challenges and Qualitative Comparison
The Blinking Color Balls tasks pose significant challenges for the models, as they required to reason about the object movement and color assignment rules from partial views of objects in temporally distant image patches. We can define two levels of challenges: (1) identifying the objects from image patches and predicting their future positions based on their dynamics, and (2) determining the final color assignment of each object based the given rules. The first challenge is relatively simple, as it primarily involves learning the dynamics of objects from the past two frames prior to the target time step. However, the second challenge is particularly difficult, it requires the model
Context Sequence
Target
Context Sequence
Target
(a) Earliest Color Variant
(b) Most Frequent Color Variant
Figure 9; Blinking Color Balls Samples.
17

---

Context Frames
GT
SlotSSM
SlotTrans
SlotRNN
SingleSSM
RIM
Time Step
Figure 10: Blinking Color Balls Qualitative Comparison. Results shown for the Most Frequent Color variant with a sequence length of 80 frames.
to reason over the entire input sequence, necessitating the identification of an object's history from partially observed patches in a long-range context.
Figure 10 presents qualitative comparison of the models' performance the task. The results reveal a clear categorization of the models based on their capability to address the two levels of challenges. The baseline RIM model successfully predicts the object positions in the target image but struggles with learning color assignment rules. Consequently, it predicts the color white that generally have the highest appearance probability for all objects. Note that the rendered images based on the argmax of the logits over the color categories. Models such as SlotRNN and Single State SSM demonstrate the ability to learn color assignments, but they make mistakes in some cases. In contrast, SlotSSM and SloTTransformer successfully achieve both accurate position prediction and color assignment.
C
Additional Implementation Details
C.1 SlotSSMs and OC-SlotSSMs
Slot Encoder. The main difference between the SlotSSMs and OC-SlotSSMs variants the design of the Slot Encoders as illustrated in Figure 11 The Slot Encoder in SlotSSMs is implemented as a multi-layer transformer with self-attention and cross-attention modules. Given the input tokens M X = {w]] 1m=1. the structure of each layer in the Slot Encoder can be delineated into three modules:
C. SellAttn(C))
(18)
C, CrossAttn (q C,, kv X1).
(19)
C. = MLP((C.)
(20)
We use 3 layers in all our experiments. Note that we also apply skip connections and layer normaliza- tion in the input for all three modules, but have omitted them in the equations cross-attention used here employs softmax normalization over the attention weights applied the input tokens:
for brevity. The regular
Q = WQ(C.).
K W=Wx(?),
== WV((),
(21)
QKT
C softmax
axis= v.
(22)
VD
In the OC-SlotSSMs layers, the Slot Encoder is implemented as single inverted attention layer. This layer differs from the regular cross attention by the way attention weights are normalized:
18

---

Slot Mixer
MLP
MLP
%/2
Mij
Self Altention
SlotSSM
Sly
Mip
Linear
CLS
CLSSg
Slot Encoder (Optional for layer /
Transformer
Droas Atteninn
CLS
CLSS
(a) SlotSSMs
Slot Mixer
MLP
MLP
Mi
Self Attention
SlotSSM
A
A
Vei
Slot Encoder (Inverted Attention)
(b) OC-SlotSSMs
Figure SlotSSMs vs OC-SlotSSMs.
Wo(C,), K WX),,,,
V Wv((k),
(23)
QKT
tmax
VD quiriequeries'
(24)
Ar,
Aij
(25)
Σ
NK
Aij
C
AV
(26)
The inverted attention layer applics softmax normalization over the queries, introducing a competition among the query tokens the attention the input tokens and thereby promoting disentanglement for the input tokens.
SSM Blocks. For the implementation of the SSM models, we leverage recent advances in linear state space models and design our SSM block in SlotSSM based on the Mamba architecture (24). The block-diagonal transition of slots is implemented as parallel runs of SSM blocks that share the same model weights.
k= (h (hii1 Ki PB ke
= SlotSSM ((
(ha Ki
(27)
vij ha MambaBlock 1k hi-1)). Vk E (1...,,K)
(28)
19

---

Dataset & Models
Module
Hyperparameter
Blinking Color Balls (SlotSSMs)
MOVi-A (OC-SlotSSMs)
General
Batch Size
128 300K
24
Training Steps
500K
Sequence Length
{80, 160, 320, 640, 1024, 2048] AdamW 0.1
Optimizer
AdamW
Weight Decay
0.1
Learning Rate
8e-4
3e-4
Slot Encoder
Input Tokenizer
MLP (Patcchiyyy
Flatten( CCNN (iiguu))
Encoder Type
Self- Self-Cross Attention
Inverted Attention
Applied Layers
First Layer
All Layers
Hidden Size
64
192
Dropout
0
Heads
4
SlotSSM
Hidden Size
64
192
# Slots
6
11
SSM Model
Mamba Block
Mamba Block
State Size
16
16
State Expand
1.25
1.25
Slot Mixer
Dropout
0
Heads
4
Table 2: Hyperparamettess of our model used in experiments.
We include pseudo-code the Mamba block implementation in Algorithm For a more detailed description of the Mamba architecture and its underlying principles, refer the readers to the original paper [24].
C.2 Baseline Models
We use the official implementation of RIM from GitHub as well as the SAVi implementation from STEVE
[ We describe the implementation the proposed baselines SlotRNN and Slo(Transformer in the following.
SlotRNN. SlotRNN adopts similar design to SlotSSM, but replaces the SSMs with GRUs (5].
In this architecture, the slots are processed in parallel across different slots at each time step and sequentially across time steps. The implementation of each layer is summarized follows.
(= Slot Encoder (=f(--1)==1) Kg
(29)
h
GRU shps h n1-11),
Vk :L....;K..
(30)
(hip) K;
T== SelfAttention (hij) K; Kr
(31)
{ (2
(32)
SlotTransformer. SlofTransformer uses the same SlotEncoder as SlotSSM obtain slot represen- tations. At each time step, the slots from the current step are concatenated with the slots from all previous time steps. This combined sequence then processed using a Transformer with causal mask in time dimension which ensures that each slot can only obtain information from prior or current time steps. The implementation of each layer is summarized as follows:
"happ://githuu.ccmm/// http://iiwh...cmm//////mnnnn11// 19/RM/
bbunss//gthhooocommiiigggaaaaaan
20

---

Table 3: The CNN encoder architecture used for object-centric learning.
Layer
Kernel Size
Stride
Padding
Channels
Activation
Conv
5 x 5
2
2
192
ReLU
Conv
192
ReLU
Conv
5 x 5
1
2
192
ReLU
Conv
5 x
192
None
Table 4: Spatial broadcast decoder architecture for image reconstruction object-centric learning, it outputs RGB and alpha-mixing logits.
Layer
Kernel Size
Stride
Padding
Channels
Activation
Slot Normalization
Positional Embedding
Conv TrnnTsosposee2d
5 x 5
2
2 (Output Padding: 1)
64
ReLU
Conv Transpooopo
x
2 (Output Padding: 1)
64
ReLU
Conv TonnTraasposedd
5
2 (Output Padding: 1)
64
ReLU
Conv SonnTrasspoe2d
5
2 (Output Padding: 1)
3
None
{
SlotEnc
(a 11 )
(33)
Transformer
(s (
ki u 1 (
(34)
C.3
Blinking Color Balls Experiemtns
show the hyperparameters used in the experiments in Table 2
Input Tokenizer. Each patch in the input sequence is treated as an image and further split into non-overlapping patches of size 4. Each patch then augmented with spatial and temporal positional embeddings, followed by an MLP layer to compute the final tokens for the Slot Encoder.
self-cross attention layer with positional embeddings as input and slots as context. Given the positional embeddings P: = {p HW Jmm and slots from SlotSSM S. = {sf K each layer of the transformer decoder can described as follows:
Decoder. During image decoding,
P. SelfAttn(Pi)..
(35)
P. CrossAttn (q= Pr, kv = S.)
(36)
P. = MLP(P,)
(37)
We use total of 3 layers, and the final pixel logits are computed using a linear head.
Training Objective. During training, transform the image prediction problem into pixel-wise classification task. Specifically, for target image E IR xxW x3 we compute quantization by categorizing each pixel into one of discrete color categories:
x (i, = (ixx(i))) ViE (1,2, H) (1,2.... W)
(38)
where Q 3 Cs the quantization function that maps 3-dimensional color vector one of the color categories in the set C = discrete color category. This a lossless quantization process since the raw images are generated with the same set of discrete colors. The final training objective is the cross-entropy loss between the model output ēN and the target x N:
c.). Each I represents color vector corresponding
?
21

---

H
W
L =
2 N
log.
(39)
C.4 Unsupervised Object-Centrie Learning Experiments
The hyperparameters used in the experiments are presented in Table 2 Table details the structure of the spatial broadcast decoder described in Section 5.2
To compute the input tokens, the input images are first processed by CNN network to generate a 2D feature map. The architecture of the CNN network is described in Table We use a downsampling factor of 2, resulting in an output 2D feature map of size 64 for an input image size of 128 128. The 2D feature map is then flattened into sequence of length 4096 and provided to the inverted attention mechanism.
D Additional Results
D.1 Emerging Modularity in SlotSSMs
To gain further insights into the learned representations of the slot-based models, we investigate how the slots are utilized the image generation process. This can be done by visualizing the attention mechanisms in the decoders.
Figure 12 presents the results of this analysis. For the transformer decoders used in the video prediction and blinking color balls tasks, we compute the argmax over the slots in the cross-attention map (Eq. 36). which represents the attention of the positional tokens the slots employed to obtain information for reconstruction at each position. In the case of the spatial broadcast decoder, we take the argmax over the alpha-mixing logits a; (Eq. [7]. The visualizations reveal that each slot tends to specialize in representing a specific object or a coherent part of the scene. This emerged object-centric representation allows the model efficiently capture the dynamics and interactions of the objects, leading to improved performance in tasks such as video prediction and reasoning in the blinking color balls benchmark.
Interestingly, even though the slot encoder used the video prediction and blinking color balls benchmarks does not explicitly enforce spatial disentanglement constraints like the inverted attention mechanism OC-SlotSSMs does, the models still learn to represent the sequences an object- centric manner. This emergent modularity suggests that the SlotSSM design naturally encourages the model to discover and exploit the underlying structure of the data which is crucial capability for modeling complex visual inputs such real-world videos.
D.2 Real- World Videos Depth Estimation
In this additional experiment, we conduct preliminary evaluation of SlotSSMs to assess its perfor- mance in real-world videos with increased visual complexity. Our aim is to observe how SlotSSMs utilize the modular representations to interpret and process real-world video data. Following previous works in object-centric learning [13], we evaluate this through a depth estimation task.
Datasets and Tasks. We select three datasets that represent distinct real-world application scenarios to observe the behavior of SlotSSMs across diverse contexts:
1. TikToo Dancing Dataset comprises dynamic videos of individuals dancing with variety of movements and poses.
33
Commonly used for content creation tasks [31, this dataset
2. UT Egocentric Video Dataset 42
Often utilized for egocentric action recognition [72),
this dataset consists of first-person view videos that involve interactions with objects in the environment.
3. Waymo Open Dataset 59
Primarily used for autonomous driving applications 43 this
dataset includes videos captured from autonomous vehicles navigating traffic scenarios with diverse environmental conditions.
22

---

The primary task across these datasets is to estimate the depth of each pixel in the video frames. However, it is important emphasize that our objective is not to develop depth estimation models that compete with existing specialized approaches. Instead, our main focus is to use this task, manageable with our lab resources, to showcase the emerging modularity in SlotSSMs for real-world video inputs.
Models. We compare OC-SIotSSM, which uses inverted attention in the Slot Encoder, against SAVi++ [3], an RNN-based object-centric learning method. Similar to the setting in Section 5 both models use a CNN encoder to extract input tokens, which are processed using inverted attention for OC-SlotSSM and slot attention for SAVi++ produce slot representations. These slots then used to reconstruct the image using spatial broadcast decoder, with MSE loss as the training objective.
The quantitative results in Table show that OC- consistently outperforms the SAVi++ baseline across all datasets, demonstrating its superior video modeling capabilities. More importantly, illustrated by the attention patterns in Figure 13 unsupervised scene decomposition emerges during training. This demonstrates that SlotSSM is able to utilize the modular representations discover and exploit the latent structure of the input complete the task, while the SAVi++ baseline does not demonstrate the same level of emergent modularity.
Results.
Table 5: Depth Estimation MSE (1) on Real-World Datasets.
Model
UT Egocentric
Waymo
TikTok
SAVi+
0.589
0.804
1.412
OC-SIoISSM (Ours)
0.464
0.653
1.180
Algorithm Mamba Block. The algorithm receives T'length sequence of the same slot across time $1:7 E T The algorithm outputs the updated slots S1:7- Note that the model imposes the diagonal structure on the matrix.
1: Input: 8 E R.TxD
2: Block params: SSM linear Sg. Sc, Sa: Transition matrix A E DxN, LayerNorm LN; Linear Linear1, Linearg; 1D Conv Conv1D
3: for = ... T in parallel
4: 8; rCs; = Linear (s,)
5:
= SiLU (SoVVDDo
6:
SSM block:
7:
B R + Ss(s)
8:
Ce TXN t Sc(s)
9:
A R TXD t SoftPlus(Paranee + (s)ss)
10:
A. B E RXDNN discretizoA,,AABB
ho 0xN
for
1... parallel (scan)
# GPU hardware accelerated y < SSM(A, (s).
13:
h; = A, o h Bys;
Hadamard product for diagonal A.
14:
yt = C.h,
15:
for
= 1... T in parallel
16:
y: y, SiiUU(res)
y: Linearg(y:)
return y
23

---

Multi-Object Video Prediction
Blinking Color Balls Benchmark
Object-Centric Learning
eite
Ele
lle
B
Figure 12: Emerging Modularity in SlotSSMs. Object-centric representations naturally emerged accommodate the underlying structure of the data.
24

---

Input Video
OC olossssssMe
(Ours)
SAV++
TiKTok Dataset
Input Vidéo
OC Slo-s5s
(Ours)
SAV
Waymo Dataset
Input Video
OC-9lo1SSMs
(Ours)
SAV
UT Egocentric Dataset
Figure 13: Emergent Scene Decomposition from Depth Estimation Tasks. Colors represent the ID of slots used for predicting each position. SlotSSM capable of exploiting the inherent modular structure of real-world videos efficient inference, without explicit segmentation supervision. For more examples please visit our project website.
25

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatiin right after your answer (even for NA),
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main claims made in the paper, accurately reflecting its contributions and scope. The claims are supported by experimental results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed in the section Conclusion and Limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
26

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: In this paper, we focus on a thorogh empirical study of the proposed architec- ture and do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
All the architectural details are explained in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
27

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will make our work open-source upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
mublic/guidess httpg/gie//ooeeuuiiiiieeeseeee
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. www.c/pwwwp.pccucclocooooo wwtp:///giiiisgeee http//oeeeuaiiisiiooooooooooeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are included in the Appendix and our open source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
28

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not include error bars in our reported results due to the prohibitively high computational cost of running multiple experimental trials. However, we have provided detailed experimental setups and parameters the appendix. Additionally, we will open source our code to facilitate reproducibility and allow others to verify our findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are included in the Appendix and our open source code.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////aurrrss.....oooo//tttt
Answer: [Yes]
Justification:
We conducted in the paper conforms to the NeurIPS Code of Ethics in every
respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
29

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed this in the Impact Statement section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our work is not about image generators or pretrained language models and uses synthetic and non-risky datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
30

---

Answer: [Yes]
Justification: We used existing dataset in our work and the original papers that produced the datasets. are cited in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppsppppeaaaaeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
31

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
32


Reviews:
[{"review_id": "a4tt7ZrJT4", "reviewer": "Reviewer_ERbD", "summary": "This paper presents Slot State Space Models (SlotSSMs), a novel framework that integrates modular structures and inductive biases into State Space Models (SSMs) to improve sequence modeling. SlotSSMs maintain a collection of independent slot vectors and perform state transitions independently per slot, with sparse interactions managed through self-attention. This approach effectively captures the inherent modularity present in many real-world processes. Authors demonstrate substantial performance gains in object-centric video understanding and video prediction tasks, highlighting the importance of modularity in sequence modeling. Additionally, the authors introduce Object-Centric SlotSSMs (OC-SlotSSMs), which leverage inverted attention to further enhance the discovery of modular structures. Extensive experiments across multiple tasks, including multi-object video prediction, long-context reasoning, unsupervised object-centric learning, and 3D visual reasoning, validate the effectiveness and versatility of the proposed models. The paper also includes a detailed analysis of the emerging modularity in SlotSSMs, showcasing their ability to naturally discover and exploit the underlying structure of the data.", "questions": "1. Given the current computational constraints mentioned in the paper, what are the potential strategies for scaling SlotSSMs to handle larger datasets and model sizes? Are there specific optimizations or approximations that you are considering to improve scalability?\n2. Have you conducted any preliminary experiments or theoretical analyses on applying SlotSSMs to other data modalities, such as text or audio? What challenges do you anticipate in these domains, and what benefits might SlotSSMs bring?\n3. How do you anticipate SlotSSMs would perform on datasets with higher visual complexity? Have you identified any specific challenges or limitations in such scenarios, and what strategies might you employ to address them?", "limitations": "Already discussed by the authors.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This approach is original in its design, utilizing independent slot vectors with sparse interactions managed through self-attention.\n- A thorough evaluation of the proposed models across multiple challenging tasks, including multi-object video prediction, long-context reasoning, unsupervised object-centric learning, and 3D visual reasoning. The experiments are well-designed and provide robust evidence of the models' effectiveness.\n- The authors provide a detailed analysis of the emerging modularity in SlotSSMs, offering valuable insights into how the models learn to capture the underlying structure of the data, with visual aids.\n- The success of SlotSSMs in capturing modular structures suggests promising directions for further research in modular and object-centric sequence modeling. This could lead to the development of even more advanced architectures capable of handling complex, real-world data.", "weaknesses": "- While the potential for applying SlotSSMs to other modalities (e.g., text, audio) is mentioned, the paper does not provide experiments or theoretical analyses in these areas. Including preliminary results or theoretical discussions on applying SlotSSMs to different modalities would strengthen the paper and demonstrate broader applicability.\n- In the related work section, mention also another interesting work utilizing SSMs for Neuromorphic Cameras: \"State Space Models for Event Cameras\". Nikola Zubić, Mathias Gehrig, Davide Scaramuzza.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We sincerely thank you for your positive recommendation and thoughtful comments!\n\n> … the paper does not provide experiments or theoretical analyses in other modalities (e.g., text, audio). …  What challenges do you anticipate in these domains, and what benefits might SlotSSMs bring?\n> \n\nThank you for the insightful suggestions. We agree that additional experiments on other modality would make the paper stronger. We’re planning to explore this aspect in our next project. We will include additional discussion in the following about modality-agnostic generality of SlotSSM in our revision:\n\n- **Audio modality.** A key feature of SlotSSM is its ability to leverage the slot representations to exploit the modular structure of input data for downstream tasks. This is especially relevant for videos, which are inherently modular, yet require the model to uncover this structure. We anticipate this principle extending to audio inputs, where multiple acoustic sources may exist, and the model must inherently decompose these sources to understand the context. In such scenarios, slots can represent the decomposed acoustic sources.\n- **Text modality.** Another important aspect of SlotSSM is the introduction of additional memory units, the slots, at each time step. Unlike conventional SSM-based language models that use a single state for each input token, slots can store additional pertinent information extracted from past data. Therefore, applying SlotSSM to process language tokens could potentially enhance the capabilities of existing language models, improving performance on tasks such as long-context reasoning.\n\n> In the related work section, mention also another interesting work Neuromorphic Cameras: \"State Space Models for Event Cameras\". Nikola Zubić, Mathias Gehrig, Davide Scaramuzza.\n> \n\nThank you for suggesting this. We will mention it in the related work.\n\n> … what are the potential strategies for scaling SlotSSMs to handle larger datasets and model sizes? … How do you anticipate SlotSSMs would perform on datasets with higher visual complexity?\n> \n\nWe appreciate this important question. We first want to highlight that our experiments have already demonstrated the significantly superior efficiency of the SlotSSM design compared to both transformer and RNN baselines. To further improve the efficiency in large scale datasets and model size, we can consider adopting the following strategies: \n\n1. Utilizing pre-trained visual encoders for visual encoding and apply SlotSSM on top. This improves both representation quality and computation cost.\n2. For object-centric learning, utilize a stronger image decoder, such as autoregressive transformer decoders or diffusion decoders. This improves reconstruction capability to handle visually more complex data.\n3. Reducing the number of tokens for deeper layers to capture increasingly abstract information. This reduces the memory consumption for larger model size.\n\nMoreover, recent methodologies like SAVi++ [1] suggest leveraging cross-modality signals to facilitate object-centric learning on real-world videos without extensive scaling of training. Inspired by this, we conducted an additional experiment where the model received RGB video inputs and predicted the depth of each frame. We provide qualitative results in rebuttal PDF Fig 1.\n\nWe compared OC-SlotSSM with SAVi++ on three real-world datasets: UT Egocentric Videos, Waymo Autonomous Driving Videos, and TikTok Dancing Videos. Qualitative visualizations on the TikTok Dataset are provided in the rebuttal PDF Fig 1. Our OC-SlotSSM model demonstrated its ability to exploit the inherent modular structure of the videos to accomplish the task, with unsupervised scene decomposition emerging during training. Below, we provide a quantitative comparison.\n\n|      Prediction MSE ( $\\downarrow$ ) ( $\\times 10^{-3}$ )          | UT Egocentric       | Waymo Autonomous Driving     | TikTok\n|:----:|:-----:|:----:|:----:|\n| SAVi $++$          | 0.5885 | 0.804 | 1.412 |\n| OC-SlotSSM (Ours) | **0.4640** | **0.653** | **1.180** |\n\n\nWe see that our OC-SlotSSM consistently outperformed the SAVi++ baseline across datasets, showcasing its superior video modeling capabilities for this task. During the experiment, we also noted that OC-SlotSSM maintained better training stability, particularly with longer sequences. In our revised manuscript, we will include more qualitative results on the emergent scene decomposition and detailed discussions comparing the performance and scalability of OC-SlotSSM and SAVi++.\n\n[1] SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos, https://arxiv.org/abs/2206.07764", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kxsHEMC38H", "reviewer": "Reviewer_Ae5c", "summary": "This paper presents SlotSSMs, an extension of SSMs such that their states encourage information separation. This is in contrast to conventional SSMs whose state is monolithic. The authors evaluate slotSSMs in object-centric video understanding and video prediction tasks involving multiple objects and long-range temporal dependencies.", "questions": "Aside from the previous weaknesses, there are some other aspects I would like to mention / clarify:\n\n- In Eq. 8, are these Linear layers different per slot or are they the same everywhere? \n- As far as I understand, in Eq.7 the “CLS tokens” are basically learnable embeddings. Is there a reason to call these CLS tokens instead? This might be confusing, as this implies that the input is always the CLS, which I think is not the case here.\n- The authors go about how their method can use a different number of slots at each layer, but then go and use the same number of them at all layers. It would perhaps be better to introduce the slots refinement module as it is. In the current setting, when reading this it feels as if having a different number of slots will be required, which can be counterproductive for the impact of the paper (readers can think that the method is too complicated to use in practice). \n- There are several typos / misspellings in the paper.\n- As far as I know, the A matrix of Mamba is not input dependent. Could you confirm if you made it input dependent in Slot SSMs? Also, if you are using only Mamba, why not call the paper Slot Mamba?\n- In many aspects, your proposed method is similar to the idea of using heads but for SSMs. Could you please comment on this? Also, I am aware that multiple works have introduced heads to SSM / long-conv models as well, e.g., in multi-headed Hyena [1]. \n\n[1] https://arxiv.org/abs/2310.18780", "limitations": "The authors clearly state a limitations section for the method. However, the limitations stated there obey more to a future work section and does not really discuss the limitations of the existing method. I would encourage the authors to revise this section and state the limitations of the submitted work.\n\n### Conclusion\n\nWhilst I acknowledge the novelty of this paper, I as of now have many concerns that I believe should be addressed before the paper is ready for publication. I therefore am unable to support acceptance. With that being said, I am happy to increase my score should these concerns be addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- As far as I know, this is the first work to propose the use of slots and object-centric modeling in SSMs.\n- The paper is well structured and easy to read. The exhibition and definition of the different SlotSSM components is very clear (up to some small observations). \n- The paper contains multiple experiments that illustrate the power of their proposed method. It also incorporates multiple ablation studies that illustrate the contribution of each of the proposed modules.", "weaknesses": "- My biggest concern lies in the evaluation of the method. For example, in Fig 4, the loss of SlotSSMs is relatively low and comparable to that of SlotTransformers. However, if we look at the predictions, these do not look good for any of the methods. I wonder if these methods and dataset are the right place on which SlotSSMs should be evaluated. Note that this is also the case in Fig 7, where all of the predictions are also quite bad. I would encourage the authors to think about settings on which SlotSSMs could really solve an open issue as this would increase the impact of the paper.\n- Next, it seems that the authors evaluate other features than what is often evaluated in the literature. For example, in the MOVi paper, the authors evaluate object segmentation, for which SAVi gets 82% accuracy. I do not see this back in the paper, unless this is the first column of Fig 7 right, in which case the matrics are very low in comparison to the original paper. This, in combination with the reconstructions shown in the paper, I doubt that the Slot SSMs would be any better than SAVi. Also, the authors should consider including more recent methods in the comparison as well.\n- In addition, it would be nice to see –as is often the case in slot papers– to check what the slots are actually learning. If this is not as easy for SSMs as for Transformers, then this should be stated as well in the limitations.\n- I also have concerns regarding the reproducibility of the method. There are several experimental details missing which, in its current form, I would argue would make this paper irreproducible. Given that the metrics in this paper are also lower than what is shown in other papers, I consider this to be of vital importance. I would encourage the authors to add these details in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We sincerely appreciate your insightful feedback!\n> Fig 4 predictions do not look good\n\nWe would like to rectify an error in Fi.g 4. We mistakenly used Single State SSM (Split)'s figures for \"Ours,\", and therefore the images does not represent our model's prediction accuracy. We have included an revised figure in rebuttal PDF Fig 2 left with enhanced visualization. The updated figures show more clearly that our model outperforms the baselines.\n> Fig 7 SAVi segmentation metrics are very low comparing to the original paper.\n\nOur experiment uses a different setting. The SAVi paper uses two additional and privileged training signals: (1) label conditioning, where the segmentation of each object in the first frame is given as input, and (2) an additional modality, where optical flow, instead of RGB reconstruction, is the prediction target, making the model less sensitive to high-frequency details in the RGB domain.\n\nIn contrast, our Fig 7 (also see rebuttal PDF Fig 2 right) uses a fully unsupervised setting where learning is through video reconstruction. The original SAVi paper also provide such setting. A direct comparison can be made by comparing the **MOVi-A FG-ARI in our Fig 7** with the **MOVi FG-ARI of the SAVi (uncond. w/o flow) in Fig 3(a) of SAVi paper [1]**. The SAVi paper achieved ~62%, while we reproduce 63.98%. Thus, we are confidence that our implementation aligns with the original results. Also, we would like to emphasize that object-centric learning is just one possible application of the SlotSSM unlike SAVi. The proposed SlotSSM is a more general sequence modeling architecture.\n> Fig 7 Qualitative prediction looks bad.\n\nFig 7 shows results for MOVi-B, a challenging dataset for all models due to its greater visual complexity compared to MOVi-A. We include MOVi-A results in rebuttal PDF Fig 2 right for comparison to more clearly show that our model outperforms SAVi.\n\nTo improve robustness in visually complex datasets, recent approaches, such as SAVi++ [2], suggest using cross-modality signal in training. We have included new experiments in this setting, please refer to rebuttal PDF Fig 1 and our response to reviewer ERbD for more details.\n> Authors: Additional comments on evaluation metrics.\n\nWe hope our above responses adequately addressed the concerns on evaluation metrics. Here, we clarify the motivations behind our evaluation. Our evaluation aims to highlight three key benefits of SlotSSMs:\n\n1. Improved visual reasoning ability by using modular representations.\n2. Improved long-context reasoning ability by employing SSM.\n3. The emergent of object-centric representations from video reconstruction (OC-SlotSSMs).\n\nOur Multi-Object Video Prediction task verifies #1, Long-Range Reasoning task proposes a novel benchmark to highlights both #1 and #2, Object-Centric Learning task verifies #3, and 3D Visual Reasoning task showcases both #1 and #3.\n> It would be nice to check what the slots are actually learning.\n\nIn SlotSSM, we can inspect the attention pattern in the decoder to see what each slot is learning. Results are shown in the rebuttal PDF Fig 3. Our findings suggest that SlotSSMs captures semantically meaningful components in different slots. Thank you for the suggestion. We will add this result in the revision.\n> There are several experimental details missing.\n\nWe fully agree and will include detailed description of our design and experiments. Additionally, we are committed to releasing our code upon acceptance.\n> Linear layers per slot or the same? … are “CLS tokens” learnable tokens and why the naming?\n\nThe linear layers are shared for all slots. The CLS tokens are learnable embeddings. Following the tradition of ViT, we use “CLS” to indicate that they are not observation tokens. We will clarify these in the revision.\n> slots number for different layers can be different in description but use same in practice … can be counterproductive for the impact\n\nThank you for the thoughtful suggestions. The current description unifies the formulation of the first layer and the following layers. However, we agree that it could be complicated and will clarify this part in the revision.\n> the A matrix of Mamba is not input dependent?\n\nThe A matrix is not input dependent. However, the discretized $\\bar{A}$ is input dependent via the input-dependent $\\Delta$, i.e., $\\bar{A}=(I-\\frac{\\Delta}{2} \\cdot A)^{-1}(I-\\frac{\\Delta}{2} \\cdot A)$. We will clarify this in the revision. \n> why not call the paper Slot Mamba?\n\nAlthough we use Mamba as the base backbone, the proposed key contribution is not limited to Mamba's specific architecture. Instead, it is generally applicable to the parallel-trainable SSM architecture class. We use \"SSM\" to refer to the general SSM selection.\n> proposed method is similar to the idea of using heads but for SSMs.\n\nIt might be seen that way. However, they are actually quite different. In multi-head methods, heads are continuously mixed through projection and MLP layers, making it difficult to learn independent mechanisms. Conversely, SlotSSM encourages independent learning by explicitly splitting representations into slots that interact independently with input features and across time, communicating only through self-attention layers after temporal propagation. \n> the limitations obey more to a future work, does not really discuss the limitations\n\nWe appreciate the suggestions and will revise the limitations section. For example, we will include our finding that in the 3D visual reasoning task, SlotSSM achieves suboptimal performance without task-free pre-training. This suggests that for tasks with sparse training signals, the sequential nature of SlotSSM performs better with a pre-training phase to learn to effectively utilize information from all time steps.\n> There are several typos / misspellings in the paper.\n\nWe will carefully review the manuscript and fix the grammatical issues.\n\n[1] https://arxiv.org/abs/2111.12594\n\n[2] https://arxiv.org/abs/2206.07764", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O63x2IJvC6", "reviewer": "Reviewer_YJW6", "summary": "This paper introduces Slot State Space Models where instead of having a single monolithic state, the SSM state is divided into K different - each ideally representing a separate object in the scene. Each SSM state is evolved independently while interacting with each other through self attention. The complete architecture consists of Slot Encoder - to encoder images into slots, SlotSSM - to evolve the state of each SSM state, and Slot Mixer - to capture interactions between slots. Through experimentation they show the proposed architecture is useful for various long range reasoning and object-centric tasks.", "questions": "- One of the issues with applying object-centric models to videos is the problem of temporal consistency - a given object may be represented by different slots into neighbouring timesteps. Did the authors face this issue in their video experiments and is there a way to address this in SSM based models which consider parallel training across timesteps? In RNN based method, this is addressed by initializing the slots of the future timestep by slots from the previous timestep. \n- Would it be possible to apply top-k attention like used in RIMs in the proposed architecture? Do the authors see any benefit to doing that in this architecture.", "limitations": "The authors have discussed limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of incorporating modularity in State Space Models is interesting and combines the strenghts of SSMs (fast parallel training, constant memory inference) with that of modular architectures (factorization, compositionality, ood generalization). \n\nThe paper is well written and experimental results demonstrate the efficacy of the approach.", "weaknesses": "The paper has introduced a multi-layer architecture where the number of slots can vary per layer but this architecture has not been used in any of the experiments. Also if this architecture requires the user to specify the number of slots separately for each user, its utility seems limiting since it is hard to estimate this beforehand and would require abundant hyperparameter tuning. \n\nThe authors have stated that the interaction between slots is sparse. If the interaction is done using QKV self-attention, then each slot can interact with every other slot and hence the interaction should be dense. I don't understand how such an interaction can be sparse. \n\nIt would be nice to see an ablation which investigates the effect of number of slots. It seems that most experiments use 6 slots. Specifically, it would be interesting to see how the performance scales with increasing number of slots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We sincerely appreciate for your valuable insights!\n> The paper has introduced a multi-layer architecture where the number of slots can vary per layer but this architecture has not been used in any of the experiments.\n\nIn our current design, only our first layer changes the number of slots from # of input tokens to # of slots. We considered this model description to be more general since it can unify the formulation of the first layer and following layers. However, we agree that this could bring confusion to the readers, and we will revise this part in our revision.\n> requirement for selecting number of slots limits the model since it is hard to estimate beforehand and would require abundant hyperparameter tuning … It would be nice to see an ablation of number of slots.\n\nIn general, using more slots leads to better performance in reasoning tasks. The number of slots presents a trade-off between performance and efficiency. In practice, we recommend starting with a redundant number of slots and gradually reducing it until performance drops. To investigate the effect of the number of slots, we conducted an ablation study using 1, 2, 4, 6, and 8 slots in the Long-Context Reasoning task with sequence lengths of 320 and 640. The results are provided in the following table:\n\n|Prediction MSE ( $\\downarrow$ ) ( $\\times 10^{-2}$ )|1 slot|2 slots|4 slots|6 slots|8 slots|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|Count Color: Length 320|4.379|1.170|1.009|0.965|**0.879**|\n|Count Color: Length 640|5.021|2.089|1.036|1.039|**0.857**|\n\nWe see that the results do present an improving performance with increasing number of slots. We will include the ablation study in our revision.\n> If the interaction is done using QKV self-attention, then each slot can interact with every other slot and hence the interaction should be dense. I don't understand how such an interaction can be sparse.\n\nThe term 'sparse' is used to contrast with the dense interaction in single state models, where all dimensions are continuously mixed across time and space through MLP or attention layers, making it difficult to isolate independent mechanisms into separate units. With SlotSSM, these representations are divided into independent slots that interact solely through self-attention layers after temporal updates, encouraging the learning of independent mechanisms. The term 'sparse' refers to the isolation of the dedicated interaction module and the independence in all other modules. We will clarify this part in the revised version.\n> Did the authors face temporal consistency issue in their video experiments and is there a way to address this in SSM based models which consider parallel training across timesteps? In RNN based method, this is addressed by initializing the slots of the future timestep by slots from the previous timestep.\n> \n\nWe appreciate the reviewer's question. Firstly, please note that segmentation metrics like FG-ARI and mIoU in our object-centric learning experiment are evaluated temporally, and thus explicitly measure slot temporal consistency. The results demonstrate that our model achieves better consistency than the baseline in these tasks. We will explain how SlotSSM address the temporal consistency in the following:\n\nTo allow for parallel training and fast inference, SlotSSM does not reuse previous slots for future initialization. Instead, temporal consistency is progressively promoted through deeper layers. In SlotSSM, the first slot encoder layer obtains frame-wise bottom-up information, enabling only partial temporal consistency from shared slot initializations across time. These slots are then refined by the SSM and Mixer layers, where the SSM injects temporal information and the mixer layer provides global awareness. Importantly, the mixer layer can route information between slots according to the updated temporal information to enable consistent capturing of moving objects/parts. These processes are repeated layer by layer to promote temporal consistency progressively. Optionally, we can also apply the slot encoder at deeper layers to further refine the input attention, as done in our OC-SlotSSM variant.\n\nIn our rebuttal PDF, we include the Fig 3 `Emerging Modularity in SlotSSMs` to qualitatively show that this temporal consistency is achieved by both SlotSSM and OC-SlotSSM models across tasks. Additionally, the new real-world video results in the PDF demonstrate that temporal consistency is achieved on real-world videos as well. We will include a more detailed discussion in our revision.\n> Would it be possible to apply top-k attention like used in RIMs in the proposed architecture? Do the authors see any benefit to doing that in this architecture.\n\nWe thank the reviewer for providing the insightful comment. We believe applying the top-k selection in SlotSSM would be possible and it would be an interesting topic. Potentially, it can improve the generalization and efficiency when number of slots is large. However, we note that directly applying RIM’s top-k selection might be suboptimal as it could pose optimization challenges. Our experience with SlotRNN vs RIM in Long-Context Reasoning tasks showed RIM to be significantly harder to optimize. We suspect the use of top-k selection, which restricts learning of the full set of slots at each iteration, may impact training stability. Therefore, we will emphasize the need to investigate approaches to stabilize training for top-k SlotSSM. Some of the possible approaches will be the following:\n\n- Initially training with full slots, followed by test-time top-k selection. The test-time top-k selection can be done by selecting the top-k slots with largest output norms as usually done in MoE literature [1].\n- Training with full slots initially, then fine-tuning the model on top-k selections post-training, essentially introducing an alignment stage for adapting to top-k selection.\n\n[1] A Closer Look into Mixture-of-Experts in Large Language Models, https://arxiv.org/pdf/2406.18219", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their insightful and positive feedback! We are encouraged that they find our work **novel** (ERbD, Ae5c), **interesting** (YJW6), and **offers valuable insights** (ERbD). They also highlighted its **potential to facilitate future research** and **lead to more advanced architectures** (ERbD). We are pleased that they recognized our empirical evaluation as
**thoroughly conducted** (ERbD), **demonstrating models' effectiveness** (ERbD, YJW6), and our paper **well-written** and **easy to follow** (Ae5c, YJW6).

We provide additional experiment results in our uploaded pdf. These experiments are done to respond to reviewer **ERbD’s question about model performance on visual complex dataset**, **Ae5c’s question about prediction quality and suggestion to check what the slots are actually learning**, and **YJW6’s question about slots’ temporal consistency**. A detailed overview of the PDF is the following:

1. **In Figure 1**, we conduct additional experiments for depth estimation on three real-world video datasets to showcase SlotSSM’s ability to handle visually complex real-world videos. In the PDF, we use TikTok dataset as an example to show the emergent unsupervised scene decompositions and depth prediction results, demonstrating that SlotSSM is able to utilize the modular representations to discover and exploit the latent structure of the input to complete the task. Quantitative results is shown in the rebuttal.
2. **In Figure 2**, we provide revised figures for video prediction and object-centric learning experiments to better showcase SlotSSM’s superior performance.
3. **In Figure 3**, we visualize the attention pattern in the decoder to investigate what each slot is learning and how they contribute to the final prediction. Results demonstrate the emergent modularity in slot representations as well as their temporal consistency.

We will respond to each reviewer’s concerns and questions separately below.

Meta-review: Overall, after reading the reviews and the rebuttal, I think the paper is in great shape and should be accepted. Echoing what the reviewers said, I think the method is sound, and the paper does a great job at ablating and understanding how the proposed slotSSM structure works, showing good results. 
Please consider all the feedback received, and incorporate in the camera ready all the changes/additional results mentioned in the rebuttal.
Decision: Accept (poster)