ConvT3: Structured State Kernels for Convolutional State Space Models
Authors: Jaeyoung Hong
Yun Young Choi
Joohwan Ko
Minseon Gwak
Venue: ICLR 2026 Poster (2026)
Keywords: Spatiotemporal modeling
Video modeling
Physical system modeling
Tridiagonal Toeplitz tensor
Long-range sequence modeling
Primary area: learning on time series and dynamical systems
TL;DR: We propose ConvT3, a ConvSSM with extended state kernels structured by tridiagonal Toeplitz tensors, enabling efficient linear-time training, stable parameterization, and state-of-the-art performance in video and PDE modeling.

Abstract: Modeling long spatiotemporal sequences requires capturing both complex spatial correlations and temporal dependencies.
Convolutional State Space Models (ConvSSMs) have been proposed to incorporate spatial modeling in State Space Models (SSMs) using the convolution of tensor-valued states and kernels. 
Yet, existing implementations remain limited to $1\times 1$ state kernels for computational feasibility, which limits the modeling capacity of ConvSSMs.
We introduce a novel spatiotemporal model, ConvT3 (ConvSSM using Tridiagonal Toeplitz Tensors), designed to equivalently realize ConvSSMs with extended $3\times 3$ state kernels.
ConvT3 structures a state kernel for its corresponding tensor to be composed as a structured SSM matrix on hidden state dimensions and a constrained tridiagonal Toeplitz tensor on spatial dimensions.
We show that the structured tensor can be diagonalized, which enables efficient parallel training while leveraging $3\times 3$ state convolutions.
We demonstrate that ConvT3 effectively embeds rich spatial and temporal information into the dynamics of tensor-valued states, achieving state-of-the-art performance on most metrics in long-range video generation and physical system modeling.

Published as a conference paper at ICLR 2026
CONVT3: STRUCTURED STATE KERNELS FOR CONVO- LUTIONAL STATE SPACE MODELS
Jaeyoung Hong*.l.2, Yun Young " Ko², Minseon Gwak
1 Solverx, Seoul National University, Uniivrsitty of Massachusetts Amherst, 4 Pohang University of Science and Technology
ABSTRACT
Modeling long spatiotemporal sequences requires capturing both complex spa- tial correlations and temporal dependencies. Convolutional State Space Models (ConvSSMs) have been proposed to incorporate spatial modeling in State Space Models (SSMs) using the convolution of tensor-valued states and kernels. Yet, existing implementations remain limited to x 1 state kernels for computational feasibility, which limits the modeling capacity of ConvSSMs. We introduce novel spatiotemporal model, Conv T3 (ConvSSM using Tridiagonal Toeplitz Tensors), designed to equivalently realize Conv with extended 3 x 3 state kernels. C structures state kernel for its corresponding tensor to be composed structured SSM matrix on hidden state dimensions and a constrained tridiagonal Toeplitz tensor spatial dimensions. We show that the structured tensor be diagonalized, which enables efficient parallel training while leveraging 3 x 3 state convolutions. We demonstrate that ConvT3 effectively embeds rich spatial and temporal information into the dynamics of tensor tensor-valued states, achieving state-of-the- th-tth-artttttto performance on most metrics in long-range video generation and physical system modeling.
INTRODUCTION
Modeling spatiotemporal dynamics is foundational for numerous domains, including videos (Finn
al. 2016 Ho et al. 2022, Wasim et al. 2024), physical systems (Nguyen et al. 2023 Li et al.
Choi al.) (200), and weather forecasting (Lam et al.) 2023; Bodnar et al. [024]] Pathak et al. 2022). In video generation, maintaining long-term memory and consistency is crucial for ensuring realistic outputs. Similarly, modeling physical systems such fluid flow thermodynamic processes requires fine-grained spatial reasoning within temporal dynamics, as spatial interactions evolve in such systems. Consequently, spatiotemporal modeling has drawn intense interest as a key challenge for domains that require capturing intricate dependencies across space and time.
2024
Across visual and physical domains, sequence models offer compelling paradigm for spatiotemporal modeling, capturing the underlying dynamics that link input sequences to output sequences. The primary objective of spatiotemporal sequence modeling is integrate spatial feature extraction with temporal representation. Early approaches like Convolutional Recurrent Neural Networks (ConvRNNs), exemplified by ConvLSTM (Shi al. 2015), introduced valued hidden states, updated with convolutions, to capture spatial patterns. However, ConvRNNs inherit limitations of RNNs, including serial training and difficulty in modeling long-range dependencies. Transformer- based models (Yan et al. 2023 Park al., 2023) Patraucean et al. 2024) have also been adapted to video domains, employing factorized attention or patched-bart base processing to handle spatiotemporal inputs. Although powerful, the quadratic-tine computational cost of attention restricts the feasible spatiotemporal context, especially as the data dimensionality and spatial/temporal resolution increase.
Convolutional State Space Models (SonySSMs) (Smith et al. 2023) combine tensor-valued states from ConvRNN with State Space Models (SSMs), providing expressive modeling capability with linear- time efficiency. Although ConvSSMs conceptually allow arbitrary kernel sizes for the state,
Equal contribution
Corresponding author; mi 25eon250postech

---

Published as a conference paper at ICLR 2026
input, output, and feedthrough convolutions, ConvS5 (Smith al. 2023), the practical implementation of ConySSM, restricts the state kemnels to pointwise x convolutions. This constraint is necessary to avoid exploding computation in parallel scans with larger kernels, but it fundamentally limits the learned state dynamics from effectively capturing spatiotemporal context. Such limitations motivate alternative designs that can leverage extended state kernels while maintaining efficient training.
In this paper, we propose a ConvSSM using Tridiagonal Toeplitz Tensors (ConvT3), which equiva- lently implements ConySSM with 3 x 3 state convolution. We first reformulate the convolution operations in a ConvSSM as tensor contractions, where state, input, output, and feedthrough tensors correspond to their respective kernels. The key idea of ConvT3 is to structure the state tensor using two components: a diagonalizable SSM matrix and constrained tridiagonal Toeplitz tensor. The proposed structuring rule ensures ConvT3's diagonalizability and correspondence to the extended 3 x 3 kernels. Finally, we develop an efficient training algorithm using linear-complexity parallel seans with stable parameterization, which naturally generalizes to higher-dimensional data.
We empirically validate the effectiveness of ConvT3 modeling videos and physical dynamics. On the long-range Moving-MNIST benchmark (Srivastava et al. (0155). ConvT3 consistently outperforms existing sequence models and achieves state-of-the-art results in video generation. On PDEBench datasets (Takamoto et al., 2022). ConvT3 attains the best accuracy in physical system modeling, while also exhibiting superior training stability compared to ConvS5, Ablation studies further confirm that the modeling performance of ConvT3 arises from its structured 3 x 3 state kernels rather than parameter growth. Together, these results demonstrate that Conv'T3 provides a scalable, stable, and effective framework for modeling complex spatiotemporal dynamics.
2
PRELIMINARIES
Before introducing the equations, we summarize in Table symbols used for values and operations, with notations distinguished according to the type of value.
2.1
CONVOLUTIONAL. STATE SPACE MODELS WITH POINTWISE STATE KERNELS
ConvSSM A ConvSSM using general-size convolution kernels was conceptually suggested by Smith ctt al. (2023), where SSM combines convolutional operations for spatial feature extraction. Let (((t) E I "x"' be a U-channel wwo-dimensional data at time t, where H and W denote spatial height and width. A continuous-time ConySSM is formulated with state tensor (t)) xxx where denotes the hidden state dimension, and an output tensor (t) E Mx XXU
as
x²(t) = A * (t) + B u({), (t() C (t) D + (t)),
(1)
(2)
where * denotes zero-padded convolution that preserves the tensor shapes, and X'(1) A 4 (1).
The convolution kernels A E CPxPxkxXX, xkx, B E CPxUxka C e CuxPxke x D xUxUxkp x&D are referred to as the state, input, output, and feedthrough kernels, respectively. The
onput-output tensor sequence can remain real-valued despite operating in the complex domain by parameterizing the kernels with conjugate pairs Gu et al. (2022).
ConvS5 A ConvS5 (Smith al. 2023) implements ConvSSM with pointwise state kernel, i.e.,
kA = 1, to apply parallel scans to the discretized ConVSSM
X++1 = A Xx B x Uk.
34 C X + D * Uk,
where Ae CPxxxxx and B E CPxUxkgxXs represent the state and input kernels discretized for a timescale parameter A E Rº' by discretization methods, such as zero-order hold. The computation of the state tensor sequence X1. from the input tensor sequence U1 enabled by parallel scans for an element 9% (gk,ay bs.b) = (A. B Ux) and binary associative operator defined by
q: 43 (4j,a qi,a Qj.a q8,b + 93,6),
(5)
where denotes the convolution of kernels and + is clementwise addition. Since the operation produces growing kernels across scans, ConvS5 parameterizes A Cxxxx x1 to maintain training feasibility on long sequences by preventing kernel growth during parallel scans.
2

---

Published as a conference paper at ICLR 2026
E CNNNNgN N2
D
Convolution
Tensor Contraction
IC * V
IC V
Tridiagonal Toeplitz Tensor
Input Tensor
3 Kerne
K. v
D
NyNa
D2
W CMMXNaX0,
Dr
D
Ni Ke (N,
IC CDoxDixáx
Ln | Fotten
K KOx..N.
Output Tensor
Figure 1: Equivalence between 3 convolution and its tensor contraction formulation using a TT tensor. For interpretability, the TT tensor is depicted with block TT matrices in the flattened form. The convolution operator K. can be expressed as structured tensor contraction K.V.
However, this design restricts the expressivity of the state dynamics by reducing A to a pointwise state kernel, leaving the spatial modeling to 13, C, kernels and deeper stacking of layers.
2.2 EIGENDECOMPOSITION OF TRIDIAGONAL TOEPLITZ MATRICES
As preliminary, we recall the eigendecomposition of tridiagonal Toeplitz (TT) matrices. Define tridiag (Ir, dr, ur) E CNNN, where Ir, dr, ur E are the lower, diagonal, and upper entries:
dT
WT
l
dp
uT
T= tridiag(11, dr, ur)
(6)
lT
dT
WT
lT
dr
It is well known that TT matrix admits a closed-form eigendecomppsition, with the ith eigenvalue A and its corresponding eigenvector zς are given by
A; = dr + 2 2 br cos
ir
N+1):
Tij = (IT/uT)" 9/2 sin
Vjz
(8)
N+T
where zij denotes the jth entry of the ith eigenvector, for i, (1,...,N).
Thus, one can obtain the eigenvalues and eigenvectors of 7 using its entries Ir, dr, ur without further computation. In particular, TT matrices with the same off-diagonal ratio share common eigenbasis.
2.3 CONVOLUTION AND TRIDIAGONAL. TOEPLITZ TENSORS
Convolutions are linear and shift-invariant, so they can be rewritten as matrix/tensor operations with structured matrices/tensors. In 1D, they correspond to multiplication with Toeplitz matrices; in higher dimensions, to tensor contraction with Toeplitz tensors. The Toeplitz structure arises from the shift-invariance, while the convolution kernel size determines the number of nonzero off-diagonals.
For a 3 x kernel K E CD. xx3x3 (with D, output and D; input channels), 2D convolutions can be written the tensor contractions with TT tensors. Specifically, we call T CD.xD, xN, X.N xNxxx. a TT tensor if each slice T Ta.ri E N2xN22 and Ty.nonda.J2 E x is TT matrix for (I....,N1) 12, E [2,...,N2} such that j -i-iil 1, - < 1. Values outside the tridiagonal patterns are zero.
all q .., (.), r E {I...., D.}. i1.91
Then, for input tensor V CNNx,, the convolution with K. can be equivalently written
1. KV CNXXxxDD.

---

Published as a conference paper at ICLR 2026
Qu Qu
Parallel scan
Discretize
Arriz
Vis
Nonlinear
Qu Qw
B
U4
C
Training parameter
Fixed variable
Operation
Instantiated variable
Layer I/O
Algorithmic flow of parallel computation in a Conv T3 layer for length-L input and output tensor sequences UI.t. Ui I xU The proportionality constraints in ConvT3 induce form that allows parallel scans to be applied.
Figure 2:
X where K. E CC.xDD, XN1 x N2 is the associated TT tensor whose entries are induced by K. The operation between the two tensors is matrix- purriixmultipliaatinn- like contraction along each D,, Ni. N2 dimension. This correspondence between convolution and tensor contraction is illustrated in Figure
METHOD
In this section, three types of tensor operations are employed: (1) Elementwise multiplication along all dimensions, denoted by @: (2) Tensor product resulting in dimension extension, denoted by @; and (3) Tensor contraction over certain dimensions, where the operator symbol omitted. The contraction dimensions are specified using Einstein notation in Appendix A.1
3.1 STRUCTURED STATE KERNELS OF CONVT3
Our goal is to construct a ConySSM state kernel A of size larger than 1 x 1, while having a diagonalizable form, thus enabling linear complexity parallel scan. Also, wish employ the structure of a well-known state matrix like S5 along the state dimension to guarantee performance. We operate with diagonalizable state matrix and structured TT tensor achieve such goals. We define the state tensor A corresponding to the state kernel A by the following method:
A (R,
(10)
where R E CxP is diagonalizable matrix, S CPxPXH*HXWXX is proportionality- constrained TT (PTT) tensor, and f denotes the composition rule to construct a state tensor. Specifically, the PTT tensor S satisfies two proportionality conditions with some nonzero ratios aH, aw C:
Is OHH USggrnnndedddd
(ii) Lspnrd aw U.S US (11)
for all g E Niw Jul < 1.
PJ, insin E (l...,, HJ. iws In (I,..., W] such that lin jnl
Moreover, for efffccent computation in (14.. we impose that the PTT tensor is diagonal along the hidden P x P dimension, i.c., each slice S.,.ka do dw is diagonal.
Importantly, can decompose H and S using their properties. First, since Rt is diagonalizable, there exists an invertible matrix QP E Cx such that
R= = QPA@F.
(12)
where A E PxP is diagonal.
Next, since X and III the spatial slices share common eigenbases QH E CHxH and Qw CW xW for the height and width dimensions, where the eigenbases are uniquely determined by ah and ow. Thus, S can decomposed as
(Tp Qu Qw) E (Ir s Qu @ Qw) &PPP
(13)
4

---

Published as a conference paper at ICLR 2026
where Ip PxP denotes the identity matrix, & CPXPXHXXXWXX such that each slice Sn kusw E CPP, E Eq.reidmeJh E CH and Eq...in.fiin E CWW are diagonal for 9, E {I...., P], In {I..... H), bh E (l....,W') (Refer to Appendix A.2 for derivation).
Based on the decomposed factors, the composition rule f is defined as follows:
f (R(QP.A) S(Qu.Qw.))) (Qp @ QH Qw) @ In Iw) ( ( Q# @Qw (14)
We formally define the proposed model, ConvT3, as Definition Definition 1. A ConvT3 is defined as
(t) = A (() Bu) )(1) - c Du(I).
(15)
where A = Q(AA @ Ia @ ( ( E CР*PXи*XXXXX constructed by the composition and DD E xUXUxx*x>* rule f in 14), 1. @PXUXHAHHWXH WWW E (UXPXH*BXWXX
We next show that the ConvT3 defined above equivalent to ConvSSM with a 3 x 3 state kernel.
Theorem ConvT3 ConySSM with a 3 state kernel A E CP
The key idea behind the proof (Appendix A.3) is that the state tensor of ConvT3, defined as
A = (Qp & QH @ Qw ((A @ IH @ Iw) @ (Q @ Qu @ Qw)
(16)
retains a PTT structure. The middle factor (A @ TH @ Iw) O E, contracted with QH and Qw. is itself PTT tensor, since A is block-diagonal and & preserves the Proportional Toeplitz pattern along the spatial dimensions H and W. The contraction with Qp does not break this structure: Qp also only acts on channel dimensions. As result, the full tensor A a PTT tensor. By the equivalence between 2D convolution operations and tensor contractions with TT tensors (9) this establishes that ConvT3 is indeed a ConvSSM with 3 x state kernel.
3.2 PARALLEL TRAINING OF CONVT3 IN LINEAR TIME
We show that ConvT3 can be diagonalized,, enabling the of linear-time parallel scan in sequence length (Proof is provided in Appendix A.4)
Theorem 2. A ConvT3 can be diagonalized
Br (t) V(t) CT) +DU(t), (DUU))
(17)
where
= 3 @ By B
C
(18)
under the change of state (1( e-xx((), with = Qp 8 Qu @ Qw The contraction dimensions of tensor contraction are stated by Einstein notation in Appendix A.I
Since the transformed state tensor Ar (18) is diagonal, parallel scans using the operator in 5 be applied to the transformed ConvT3 (17). enabling linear-time complexity sequence length. See
Appendix B for full complexity analysis.
For equivalent state dynamics, transformation on B and C using Q are performed before and after the scan. In practice, the transformation Qp for the hidden dimension omitted, since the state along the P dimension can be assumed to be trained in a diagonalized form, analogous to diagonal SSMs. This implementation detail is essential for reducing computational complexity, as changing the kernel operation of B, C to tensor products would be inefficient. Thus, the effective transformation reduces to the spatial transformation QH и @ Qw, which is sufficient for preserving the desired equivalence. The overall algorithmic flow for parallel training of ConvT3 is shown in Figure
3.3 PARAMETERIZATION OF CONVT3 FOR TRAINING STABILITY
Overview Learnable Parameters and Stable Reparameterization. In continuous SSMs, stability follows from the Hurwitz condition applied to the diagonalized state matrix, since negative real parts
5

---

Published as a conference paper at ICLR 2026
Training Parameter CPP
Hurwitz Condition
N' CPxP, Re(A) <
Negative Roal Part
(Stable Eigenvalues of R)
A f(R,S)
Diagonalize
Training Parameter
Positivity Condition
E 22 XX
A₁ f(N,
Softmax & Linear Comb.
(Positive Eigenvalues of S)
Stable State Tensor
f1. Fa. f4 E IRP
Figure Reparameterizztioon of ConvT3 state kernel for stability. The A (eigenvalues of R) is conditioned to have negative real parts, while the (eigenvalues of S) is conditioned to be positive. This ensures the stability of the resulting state tensor constructed from their combinations. Here, 6 denotes the vector whose components are the e from all effective channels, and indicates that the value is reparameterized.
guarantee contractive temporal dynamies. For ConvT3, the same principle applies, i.c., the stability of Ar is ensured when the real part of its diagonal values remains strictly negative.
As shown in Figure 3 the diagonalized state tensor Ar can attain stable dynamics by enforcing two conditions: (1) A E C PxP has negative real parts, and (2) & are strictly positive. These conditions ensure the negativity of Re(Ar). ), and thus guarantee the stability of ConvT3. We now describe the reparameterization methods used to ensure that these conditions are satisfied.
Hurwitz Condition. the learnable A as
To guarantee the negativity of real part of A during training, we reparameterize
A' Re(A'] soopplusRe((AA)
which automatically keeps R(A') < 0 at all times.
Positivity Condition. The other condition for stability is strict positivity of >> 0, where & the eigenvalue of the PTT tensor S X
We construct & by extending the Toeplitz eigenvalue formula to the PTT tensor by
E En Eg.r (0,H, O.W.) din Siwnter
(19)
where
Eg.r (O O.) agrr byr cos 0H,, + Cg,r cos OW dg cos 0H, cos OW
(20)
for 8.
H+1 and BW Tiw. with the channel indices q.r E (1...., P), the spatial indices W+1
ins {I,..., H} and iw- E {I,.... W). and the Kronecker delta 5..- The scalar coefficients agrr: byrr. Cg.r, dgrr E C along with the off-diagonas proportions induces the 3 3 state kernel, where ag.r corresponds to the center value and byr Cq.r, dy,, with the side values of the kernel.
For simplicity, we drop the indices (q, r) with the understanding that the following derivations apply all rq. Thus, (20) is rewritten
c@0H BW + cos 8 + coos A.W. + d cos Hfd cos A.W.
(21)
To guarantee positivity all discrete (0 OW), it suffices to impose positivity over the continuous
domain (0, T) x (0, r). Since is bilinear in cos 0 cos OW E (-1,1) 1), positivity is enforced by checking the four extreme points:
E ++bc++dxx
62 =a d 0,
b+ d 0. - c 0. Here, can fix the center coefficient
1, reducing redundant degrees of freedom, which implies
6 62 =

---

Published as a conference paper at ICLR 2026
To satisfy all positivity constraints automatically, reparameterize the four extreme values via softmax:
4 4 softmax(e), 62, E3, 6a)i:
which ensures + - and Σ: c 4. The coefficients (b, c, d) are then recovered using the inverse linear transformation:
b b= (c + - 4)/4, c= - -
d d= 44/4.
Stable State Tensor Ar. Finally, combining the Hurwitz-stable A' and the positive eigenvalues E yields diagonalized state operator that satisfies (e(Ar) elementwise. Thus, the ConvT3 state tensor is guaranteed to remain stable throughout training while retaining fully learnable temporal and spatial dynamics.
3.4 GENERALIZATION TO N-DIMENSIONAL CONVT3
The correspondence between convolution and Toeplitz tensors can extend ConvT3 to higher di- mensions. While 2D convolutions are represented by TT tensors for two spatial dimensions, N- dimensional convolutions naturally induce TT tensors for N spatial dimensions. In higher-order TT tensor, the tridiagonal pattern also arises along each of the N spatial axes for an N-dimensional kernel with kernel size of 3. Thus, for input tensor in N N-dimensional space, convolution can be represented as a tensor contraction with the -ddimensional TT tensor followed by the contraction over the channel and spatial indices.
This implies that the Conv T3 formulation, defined for 2D inputs and PTT tensors, admits a natural theoretical extension to arbitrary spatial dimensions. The peportiinnaliyycossrrrinnn structure and parallel scan mechanism remain applicable, as they depend only on the PTT property along each axis rather than the dimensionality itself.
4 RELATED WORKS
ConvRNN-basse Models ConvRNNs capture local spatiotemporal correlations and are widely used in video prediction and physical simulations (Ballas et al. 2015); early variants such as ConvLSTM (Shi et al. 2013) and PredRNN (Wang et al. 2017) improved prediction by integrating sertial-temprall cues and introducing spatiotemporal memory. Building on these, SwinL.TM (Tang
al. 2023) embeds Transformer blocks within LSTM cells to model local and global dependencies, and later PredRNN (Wang al. 2022) improves long-term stability via memory decoupling.
Transformer-based Models Transformers, with self-attention, excel at global dependency modeling and are widely used in spatiotemporal tasks (Lee et al., 2024 Vaswani et al. 2017). Recent variants include TECO, which preserves long-tern temporal consistency via VQ-based non-autorrgressvve framework (Yan et af. (223);; PredFormer, a purely Transformer model with competitive results across benchmarks (Tang al.] 2024b); and ViT, hybrid that pairs lightweight temporal recurrence with spatial Transformer blocks to balance efficiency and accuracy on long sequences (Päträucean al. 2024).
SSM-hased Models SSMs provide linear-time sequence modeling with strong memory reten- tion (Zhang al., 2024 Huang et al. 2025 Liu et al. 2024). Early S4 (Gu al. 2021) and S5 (Smith et al. 2022) paved the way: ConvSSM (Smith et al. 2023) added spatial modeling via convolutions; S4ND (N (guyynn al. (022) extended SSMs multidmensional signals for vision; Selective SSM (Wang et al. 2023) prunes information-spee -pprrse sparse tokens; and VMRNN (Tang et al. 2024a) leverages Mamba modules (Gu Dao 2023) to capture short- and long-term dynamies.
EXPERIMENTS
In Section 5.1 evaluate the spatiotemporal modeling capability of ConvT3 long- range video generation task. Section [.22, we evaluate ConvT3 on complex physical system modeling. In Section present ablation studies architecture and hyperparameters.
For all tasks, we used off-diagonal proportions a = aw = -1, meaning the state kernel is skew- symmetric across spatial dimensions. We initialized by.r, Cg.r; dg.r zero to ensure equivalence with
7

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation the Moving- -NNITT dataset (Srivastava et al. 100 frames, and then show results after generating 800 and 1200 frames. Bold: best performance. Underline: second-best performance. Trained on 300 frames
2015). We condition
100
- 800
100 + 1200
Method
FVD 1
PSNR
SSIM
LPIPS
FVD
PSNR
SSIM
LPIPS
Transformer (Vaswani al. 2017
159
12.6
0.609
0.287
265
12.4
0.591
0.321
Performer Choromanski
2021
234
13.4
0.652
0.379
275
13.2
0.592
0.393
CW-VAE Saxena el al.
104
12.4
0.592
0.277
117
12.3
0.585
0,286
ConvLSTM
2015
128
15.0
0.737
0.169
14.1
0.706
0.203
ConvS5 (milh el al. 2023
72
0.761
0.156
14.5
0.678
0,230
ConvT3
79
16.1
0.776
0.146
118
15.2
0.746
0.179
Trained on 600 frames
Transformer
13.7
0.672
0.207
91
13.1
0.631
0.252
Performer
93
12.4
0.616
0.274
243
12.2
0.608
0.312
CW-VAL
94
12.5
0.598
0.269
107
12.3
0.590
0.280
ConvL LTM
91
15.5
0.757
0.149
137
14.6
U.727
0.180
ConvS5
47
16.4
0.788
0.134
71
15.6
0.763
0.162
ConvT3
36
17.7
0.823
0.104
56
16.7
0.795
0.131
ConvS5 at initialization. The other experiment setups for each task are provided in Appendix Implementation is available at wtltttps://gtthubb htt.:ub....coo/ottt....... www.wttwwindeveeeee dewe-cnvT3/
5.1 LONG-RANGE VIDEO MODELING AND GENERATION
Video modeling requires capturing both spatial structures within each frame and temporal dynamics across long sequences. To evaluate such capability, consider the Moving-MNNI MNIST dataset (Srivas tava et al. 2013), a standard benchmark where digits move within 2D frame.
Following the task setup in ConvS5 (Smith al. 2023 we train models to reconstruct 300 600 input frames, and evaluate them at test time by 400, 800, and 1200 future frames from 100 observed frames. Baselines include Transformer (Vaswani al. 2017). Performer (Choromanski et al.) 2021
ConvLSTM Shi et al. 2015), CW-VAE (Saxena et al. 2021 and ConvS5.
ConvT3 consistently outperforms existing baselines, showing significant improvements across nearly all metrics. When trained on 300 frames, ConvT3 achieves the best PSNR, SSIM, and LPIPS scores and second-bes FVD scores for both 800- and 1200-frame generation. With longer training sequences of 600 frames, ConvT3 further amplifies its advantage, achieving the best scores for all metrics and prediction lengths. Overall, ConvT3 demonstrates state-of-the-ar performance on the Moving -NNIST benchmark, validating the effectiveness of structured 3 state kernels for long-range spatiotemporal modeling.
5.2 PHYSICAL SYSTEM MODELING
5.2. PARTIAL DIFFERNNIIAL EQUATION MODELING
Physical system modeling requires capturing the underlying spatiotemporal dynamics that govern complex phenomena, often expressed as PDEs. To assess ConvT3 on accurate prediction of physical dynamics, we use the PDEBench dataset (Takamoto al. 2022). following the prediction task setup in MPP (McCabe et al. 2024). Among the 2D datasets in PDEBench, we exclude the computationally heavy compressible Havier-Stokes case and focus on the Shallow-Water and Diffusion-Reaaae Reaiiionn datasets, which require accurate modeling of nonlinear PDE dynamics.
For training, models are provided with the first 16 time steps grid trajectories and optimized to predict the next single step. For ConvS5 and ConvT3 in this task, replaced the attention layers in AVIT by ConvS5 and ConvT3 layers, allowing direct comparison of spatiotemporal modeling ability within the same backbone. Performance is measured in terms of normalized root square error (NRMSE), along with inference time. Baselines include ENO (Li et al. 2020), UNet (Ronneberger al. 2015), and AVIT (McCabe al. 2024). and ConvS5.
8

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation Shallow- Water and Diffusion-Reactinn datasets (Takamoto al. 2022) Complex parameters were counted as two reals. Inference time was measured on A100 GPU. Bold: best performance. Underline: second-best performance. Dash: Not provided by source.
NRMSE
Time (s)
Model
# Params
Shallow-Ward
SeffusionnRRection
Train Step
Evaluation Step
AVIT-B FNO-B
116M
0.00047
0.0110
115M
0.00246
0.0599
UNet
7M
0.083
0.84
FNO
927K
0.0044
0.12
AVIT-Ti
7M
0.00053
0.0090
303 (2.31 x)
2.74 (2.06x)
ConvS5
6M
0.00035
0.0106
131 (1.00x)
1.33 (1.00x)
ConvT3
6M
0.00033
0.0087
151 (1.15x)
1.51 (1.14x)
As shown in Table [ ConvT3 achieved the best accuracy on both Shallow-Water and Diffusion- Reaction datasets while using significantly fewer parameters compared to large baselines. On Shallow- Water, ConvT3 performs comparably to ConvS5, while on Diffusion-Reacteo ConvT3 achieved a substantial performance gain over ConvS5. Moreover, ConvT3 maintains efficiency close to ConvS5, demonstrating both effectiveness and scalability in modeling complex physical dynamics.
5.2.2 TRAINING STABILITY
ConvS5 often exhibited training instability, whereas ConvT3 remained stable under the same ex- perimental and model configurations. As representative instance shown in Figure + ConvT3 maintains a smooth loss curve, while ConvS5 suddenly spikes. This behavior persisted across multiple random seeds, suggesting it is not from initialization effects.
5.3 MODEL ABLATIONS
We conduct ablation study the standard Moving-MNIST task (Srivastava al. 2015). predicting 10 frames from 10 observed frames.
In Table 3 evaluate variant of ConvT3, named MiniT3, which shares the same kernel slice across P x P hidden dimensions. This variant introduces structural alternation from ConvS5 to ConvT3, while minimizing the additional learnable parameter. Specifically, ConvT3 increases 3P parameters, but MiniT3 only increases 3 parameters per layer. Notably. MiniT3 significantly outperformed ConvS5 with only 24 additional parameters, demonstrating that the superior performance of ConvT3 arises from its structural improvements rather than parameter increase.
In Table we check the influence of the off-diagonal proportions a and aw with eemmeyc and symmetric state kernels. The results showed nearly identical performance, implying that the proportions can be set to fixed, arbitrary values.
In Table and Table we compare ConvS5 and ConvT3 on various numbers of ConvSSM blocks and B, C kernel sizes. Conv'Ts outperformed ConvS5 in most cases, especially when either or C kernels were x 1 sized, and thus the spatial modeling effect was minimal.
6 DISCUSSION
This work proposed ConvT3, which extends the state kernels of ConySSM from pointwise to 3 x 3. This extension enriched the spatial representation within the state dynamics and enabled the model to capture more complex interactions. As a result, ConvT3 achieved significant performance gains in downstream tasks. Another strength of ConvT3 lies in its efficient training algorithm. The algorithm scales linearly with sequence length, which makes ConvT3 feasible for long-range modeling. Experimental evaluations confirmed the inference speed, highlighting that the proposed method maintains efficiency without compromising modeling power. Furthermore, the parameters carefully designed with stability in mind. By constraining the parameter space, ConyT3 avoided

---

Published as a conference paper at ICLR 2026
Table 5: (Ablations: Number of blocks.)
Table (Ablations: Minimal parameterization.)
MiniT3 outperformed ConvS5 despite the minimal increase in parameter numbers.
Conv'T3 generally achieved better performance than ConvS5 under the same number of blocks.
Model
#Blocks
MSE +
MAE
Model
#Params
MSE 4
MAE
2
13.15
27.08
ConvS5
24M 24M (+24)
11.57
23.25
4
11.92
24.64
MiniT3
10.87
21.64
ConvS5
6
11.13
22.76
8
11.57
23.25
Table 4: (Ablations: Off-diagonal proportion.)
12
10.68
21.80
Different proportions gave similar results.
2
12.91
25.75
ag, ow
MSE
MAE |
4
11.82
25.31
ConvT3
6
11.11
23.38
(Symmetric)
10.97
22.12
8
10.99
22.15
(Skew-symmetric)
10.99
22.15
12
10.57
21.77
Table 6: (Ablations: Kernel size.) The introduction of a 3 x 3 state kernel A, in contrast to relying solely on same-sized B and C kernels, resulted in notable performance gains. Bold: better performance under the same B and C settings. (More cases are provided in Table
Model
A kernel
B kernel
C. kernel
MSE
MAE
1 1x1
1 x
14.98
30.60
1 1x
3 x
12.74
24.07
ConvS5
1
3
1
12.13
24.35
1 x 1
3 x
3
11.57
23.25
3
1x1
1x1
14.68
29.65
3 x 3
1 1x
3x3
11.17
23.04
ConvT3
3
3 3
1x1
11.69
23.46
3 x 3
3 3
3x3
10.99
22.15
unstable dynamics that can hinder learning. This stability-oriented design was reflected in the experiments, where the model consistently exhibited smooth convergence and across datasets.
In theory, ConvT3 can be generalized to N-dimensional data by extending the PTT construction as described in Section 3.4 However, our empirical validation was conducted on 2D datasets, which the most prevalent and practically important modalities. The absence of experiments on higher- dimensional data remains a limitation of this study. Furthermore, we adopted fixed proportionality conditions for the parameterization of PTT tensors. While this choice ensured tractability, it may limit flexibility. A natural extension would be to allow these proportionalities to be learnable parameters, potentially improving adaptability while retaining structural interpretability.
ETHICS STATEMENT
This research was conducted in compliance with ethical guidelines. At the current stage of the study, no foresccable ethical risks or socictal harms have been identified.
REPRODUCIBILITY STATEMENT
We provide link access the source code. For theoretical results, all assumptions are clearly stated, and complete proofs are included in the appendix.
REFERENCES
Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations. arXiv preprint 151i 1ate://51..4444. 06643,, 2015.
Cristian Bodnar, Wessel P Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, et al. Aurora: foundation model of the atmosphere. arXiv preprint arXiv:2405.
13063, 20t.:22355..33 2024.
10

---

Published as a conference paper at ICLR 2026
Yun Young Choi, Seongyoon Kim, Soowhan Kim, and Jung-Il Choi. Multiple parameter identification using genetic algorithm in vanadium redox flow batteries. Journal of Power Sources, 450:227684, 2020.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with Performers. In International Conference on Learning Representations, 2021.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, 29, 2016.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.0033: 2021.
Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Advances in Neural Information Processing Systems, volume 35, pp. 35971 1-3993,,, 2022.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 1188633-8666,, 2022.
Yizhou Huang, Yihua Cheng, and Kezhi Wang. Trajectory mamba: Efficient attention-mamne forecasting model based selective ssm. arXiv preprint arXiv:2503. 2025.
Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsherger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and Peter Battaglia. Graphcast: Learning skillful medium-range global weather forecasting, 2023. URL
abs/ /elo//221... 2212.127999
Minho Lee, Yun Young Choi, Sun Woo Park, Seunghwan Lee, Joohwan Ko, and Jacyoung Hong. Enhancing topological dependencies in spatio-temporal graphs with cycle message passing blocks. In The Third Learning on Graphs Conference, 2024.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametrie partial differential equations. arXiv preprint ar 20t::000000.9 2020.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics Physics-informen informed neural operator for learning partial differential equations. ACM/IMS Journal of Data Science, 1(3):1-27, 2024.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang. Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. Advances in neural information processing systems, 37: 37:103031-103066 2024.
Michael McCabe, Bruno Régaldo-Saie Blancard, Liam Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, et al. Multiple physics pretraining for spatiotemporal surrogate models. Advances in Neural Information Processing Systems, 37:1 19301-119335. 2024.
Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah. Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensionall signals with state spaces. Advances in neural information processing systems, 35:2846-288 2861, 2022.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint -xX1:23011.1343.. 2023.
11

---

Published as a conference paper at ICLR 2026
Jungin Park, Jiyoung Lee, and Kwanghoon Sohn. Dual-path adaptation from image video transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2203-2213, 2023.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcast- net: global data-driven righ-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. 1214 2022.
Viorica Päträucean, Xu Owen He, Joseph Heyward, Chuhan Zhang, Mehdi SM Sajjadi, George- Cristian Muraru, Artem Zholus, Mahdi Karami, Ross Goroshin, Yutian Chen, et al. Trecvit: A recurrent video transformer. arXiv preprint arXiv:24122 14294, 2024.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 241. Springer, 2015.
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autoencoders. Advances in Neural Information Processing Systems, 34:29246-29257, 2021.
Xingjian Shi, Zhourong Chen, Hao Wang. Dit- Yan Yeung. Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in nearal information processing systems, 28, 2015.
Jimmy Smith, Shalini De Mello, Jan Kautz, Scott Linderman, and Wonmin Byeon. Convolutional state space models for long-range spatiotemporal modeling, Advances in Neural Information Processing Systems, 36:80690-80729, 2023.
Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208.04933,, 2022.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video 2015. doi: 10.48550/arXiv.5500
representations using Istms. arXiv preprint 04681.
Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. PDEBench: An Extensive Benchmark for Scientific Machine Learning. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks, 2022.
wttp:///wwwwww////.../o.ooo 07182
Song Tang, Chuang Li, Pu Zhang, and RongNian Tang. Swinlstm: Improving spatiotemporal prediction accuracy using swin transformer and Istm. In Proceedings the IEEE/CVF international conference on computer vision, pp. 13470-13479, 2023.
Yujin Tang, Peijie Dong. Zhenheng Tang, Xiaowen Chu, and Junwei Liang. Vmrnn: Integrating
vision mamba and Istm for efficient and accurate spatiotemporal forecasting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5663-5673, 2024a.
Yujin Tang, Lu Qi. Fei Xie, Xiangtai Li, Chao Ma, and Ming-Hsuar Yang. Predformer: Transformers are effective spatial-temporal predictive learners. arXiv preprint 10X1v:2410.04733, 2024b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. Se- lective structured state-spaces for long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6387-6397, 2023.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predmnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2026
Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip S Yu, and Ming- sheng Long. Predmn: recurrent neural network for spatiotemporal predictive learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(2):2208-22 2225, 2022.
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang. and Fahad Shahbaz Khan. Videogrounding-ding Towards open-vocabulary spatio-temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18909-18918, 2024.
Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel. Temporally consistent transformers video generation. International Conference on Machine Learning, pp. 39062-39098. PMLR, 2023.
Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation. In European Conference on Computer Vision, pp. 265-282. Springer, 2024.
13

---

Published as a conference paper at ICLR 2026
Supplementary Material
CONTENTS
Introduction
Preliminaries
2.1
Convolutional State Space Models with Pointwise State Kernels
2.2
Eigendecomposition of Tridiagonal Toeplitz Matrices
2.3
Convolution and Tridiagonal Toeplitz Tensors
3
Method
Structured State Kernels of Cony'33
3.2
Parallel Training of ConvT3 in Linear Time
3.3
Parameterization of Conv for Training Stability
Generalization to V-Dimensiona Conv
Related works
5
Experiments
5.1
Long-Range Video Modeling and Generation
5.2
Physical System Modeling
8
5.2.1
Partial Differential Equation Modeling
5,2.2
Training Stability
5.
Model Ablations
6
Discussion
Proofs
15
A.I Explicit Form of Tensor Contractions in Section
15
A.2 Proof of (13)
15
A.3 Proof of Theorem
17
Proof of Theorem
17
Computational Complexity of ConvT3
18
Supplementary Results
19
Experiment Setup
14

---

Published as a conference paper at ICLR 2026
A
PROOFS
Notation Definition
Scalar
A
Matrix
Block matrix
Tensor
A
Convolution kernel
Convolution (eesso--Tensor)
Convolution -eemel-Kernel
Elementwise multiplication
Tensor product
Direct sum Kronecker delta
Table 7: Major notations
A.1 EXPLICIT FORM OF TENSOR CONTRACTIONS IN SECTION
The explicit tensor contractions are described by Einstein notation follows:
Equation
12
Rq. = -g
Equation
13
Sg Qi Q. oh Egrrtidii (Q (Q") "iudo
Equation
I6
da
Q. Ci.kk Q. 11 Minh Vunia
Equation
18). Diagonalized state tensor
AT dichhi A.r St
Equation
Transformation tensor
Qu Qu QW Dua
A.2 PROOF OF 13
We derive 13 using the following theorems for general TT and PTT tensors. In the proofs of the theorems, we use block matrices, i.e., flattened forms of tensors, for clarity of exposition.
We first introduce the definition of a commutation matrix, which will be used in Theorem to show that a block TT matrix admits a closed-form eigendecomposition when its outer blocks share common eigenvector.
Definition 2. The (m, commutation matrix ((m,n) {0, 1) maxmn is the unique permutation matrix satisfying K vec(A) vec(A?) for all A Cm
Theorem 3. Let T E Comxam be block TT matrix with blocks L,, D. U E Çm sharing a common eigenbasis. Then T admits closed-form eigendeccomposition of the form
T Q
(22)
where Q and A are explicitly constructed the proof.
Proof of Theorem Suppose that the blocks admit eigendecompositions of the form
L ==QA-QT D= QA U= =QQAQ-11
15

---

Published as a conference paper at ICLR 2026
with common eigenvector matrix Q and diagonal matrices Ag diag(AL.1," AL,m), Ap diag(Ap.1. ... AD.m), and Au = diag(Au,1: ; ;Umm). Then, T can be written with a block tridiagonal matrix A as
T T= ==QQQ-1
(23)
where = diag(Q......................... ......... Q) and A tridiag AD,
Now, define a permuted matrix A' by
: K Ak((mm)
(24)
where K ((m,n) {0, 1 maxxin is the (m, n commutaaion matrix. The permuted matrix A' decom- poses as the direct sum of tridiagonal Toeplitz matrices T3 for = 1,
A' = Ti @ ... + Tm
where @ denotes the direct sum and each component T3 E Cuxn is given by T. tridiag(A (AL, AD. AU,)).
Since T; is TT matrix, it has closed-form eigendecomposition,
T, = Q,AQ,
where A = .. Msn)) is a diagonal eigenvalue matrix with Pa,k Ap.
h+I)) and Q; = [v] [r],-.. U is the matrix of the corresponding eigenvectors with (1) (AL.s/ ) 1/2 sin jkn n+I
eVAL,Aus cos( ka
Since the permuted matrix A' is the direct sum of T, for 1, m, A' has closed-form eigendecomposition of the form
A' = QAQQQ
(25)
where Q =1 E ........... 4 Q.. and A = A₁ @ ① Am-
Combining
the block TT matrix T has closed-form eigendecomposition since K(m,)) K (n.m) Inn-
T Q Q-1
(26)
where Q = = Q''' = ((am) (m(m) and A' ((.m)) (K
Now, we obtain Theorem when constraining the proportionality of the block entries.
Theorem 4. Let T E Camxnm be block TT matrix with blocks L, D,U E Cm sharing com- mon eigenbasis Q. If = aU for arbitrary a E C, then admits a closed- forsed-form eigendecomposition of the form
Qno
Q
Tno
inn
T=
A
(27)
Q
Qnn
Q
Qm
Oan Q
where Q depends only on the matrix size and a.
Proof. Using the notation of Theorem when AD tridiag(AL,, i, AU.s)
aU, each TT matrix Tj satisfies AL.1 = aAU. By (8), this implies that all Tj share the same Qi..,
eigenbasis Q, i.e., Q = Q ... = Qm E Cmxm Therefore, Q = diag ( ...,. Q) E C is uniquely determined by a. Since Q = (Q.g(QQ..... Q) Comxnm substituting into the decomposi-
tion 26 yields 27 proving the claim.
Since the PTT tensor in 13 has proportionality conditions for both spatial dimensions, applying Theorem too to S yields Corollary 4.1 which represents the explicit form of tensor contractions in (13
Corollary 4.1. A PTT tensor CPxP*HxH*W>W can be decomposed as
Q &f Sy (Q") (QW)-1 12.j2
where QH E HxH QW CWW and E XXXP
16

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF THEOREM 1
Proof. We first show that the state tensor A defined in ① is a TT tensor.
Agriinsiniio = Qg iin Qiy (Asg & hh (Z Miiji )
(28)
-g (As, 3 Sh (Q
For all in.5n in E {l..... HI such that li - -jal 1, all slices Ja are zero, since S has zero value, which gives the tridiagonality condition for the H dimension. Likewise, the state tensor A is tridiagonal along the W dimension.
For all in, 1/2 E {l...., H 11,,,
1}, the following equation holds due to the Toeplitz condition for S.
Ag = Qg (As @ Saddin ((Q")) )er
= (A 2 Sanyy Lo Ho (Q
(29)
= Qpp
Likewise, the condition is satisfied for the lower diagonal and the dimension, proving the Toeplitz condition for A.
Since 9 and A TT tensor, it is equivalent to a x P x kernel operation.
A.4 PROOF OF THEOREM 2
Proof. By definition I ConvT3 is given by
x²(1) 2((A In @ Iw) o (())))))()(()(u)))
(30)
('t) C ((t) D
Suppose a state transformation (t) - x-x((tt. Substituting X((t) (tr(t) yields
Qx (t) (((A @ IH @ ( (t) + BU((),
(31)
)'()) Qxr(() DU()).
By performing tensor contraction with Q- we obtain
(t) ((A IH Iw) ())
(32)
Therefore, the transformed system is
(Ar,Br,Ce Br, Cr, D) = ((A In @ @ Q CQ. D),
which completes the proof.
17

---

Published as a conference paper at ICLR 2026
B COMPUTATIONAL COMPLEXITY OF CONVT3
The computational complexity of parallel scan ConvT3, given input Br U1. E CLXHxWxP OLLPHW), identical to the ConvSSM model with pointwise state kernel. This is because the diagonalized form of the state tensor A allows identical parallel scan operations for the two cases.
The total computational complexity for single pointwise state kernel ConvSSM layer O(LPHW TB+Tc), with TB the cost for imput kernel convolution, and Te the cost for ouiput kernel convolution. Since TB O HW) and Te O(LPUK)HW) the total cost will be O(LPHWU
k()) (Smith et al. 2023).
For ConvT3, we have additional matrix multiplication operations with QH and Qw (Check Figure for details), so the total cost is O O(LPHWU(k,, + However, matrix multiplication with the Q matrix the same as discrete sine transform (DST) type-I, and thus the matrix multiplication can be implemented via fast Fourier transform (FFT)-based routines. Instead of ((H2) complexity due to dense matrix multiplication, the DST type-I has complexity of O( log H). Thus. with FFT-based routines, the total complexity of ConvT3 is reduced to OLLPHWU(K + k2) + LPHWHWWWW (log + log W)).
if naively implemented.
Notice two main things: First, the additional computational cost of ConvT33 does not scale with the input dimension U, which is normally the same scale as the state dimension P. meaning Cony becomes more efficient as the hidden dimension of the model grows. Second, although the cost of ConvT3 scales more according to spatial size compared to ConvS5, the cost ratio grows logarithmi- cally, thus there does not exist point where ConvT3 becomes unfeasible.
Table Table g and Table T0 provide inference-time and memory comparisons between single ConvS5 5 and ConvT3 layers across different state sizes, hidden dimensions, sequence lengths, and image resolutions, respectively.
Table 8: Inference time and memory usage under different state sizes.
Time [ms]
Memory [GB]
State size P, Hidden dim U
ConvS5
ConvT3
ConvS5
ConvT3
512
52.7 (1.00x)
65.8 (1.25x)
4.26 (1.00x)
5.22 (1.23x)
256
20.1 (1.00x)
28.0 (1.39x)
2.13 (1.00x)
2.62 (1,23x)
128
10.5 (1.00x)
13.8 (1.31x)
1.09 (1.00x)
1.34 (1.23x)
64
7.2 (1.00x)
9.3 (1.29x)
0.58 (1.00x)
0.70 (1.21x)
Table 9: Inference time and memory usage under different sequence lengths.
Time [ms]
Memory [GB]
Sequence length L.
ConvS5
ConvT3
ConvS5
ConvT3
100
100.2 (1.00x)
138.6 (1.38x)
10.46 (1.00x)
12.39 (1.18x)
50
50.8 (1.00x)
67.3 (1.32x)
5.25 (1.00x)
6.28 (1.19x)
20
20.9 (1.00x)
27.9 (1.33x)
2.13 (1.00x)
2.62 (1.22x)
10
11.9 (1.00x)
15.4 (1.29x)
1.09 (1.00x)
1.40 (1.28x)
Table 10: Inference time and memory usage under different image resolutions.
Time [ms]
Memory [GB]
Image resolution H W
ConvS5
ConvT3
ConvS5
ConvT3
32x32
77.2 (1.00x)
160.6 (2.08x)
8.25 (1.00x)
11.75 (1.42x)
16x
20.9 (1.00x)
27.8 (1.33x)
2.13 (1.00x)
2.62 (1.23x)
8x8
7.7 (1.00x)
9.7 (1.26x)
0.60 (1,00x)
0.70 (1.16x)
4x4
7.0 (1.00x)
8.0 (1.14x)
0.22 (1.00x)
0.24 (1.08x)
18

---

Published as a conference paper at ICLR 2026
C SUPPLEMENTARY RESULTS
Table 11: Full results on the Moving- Moving-MNIST dataset (Srivastava al. 2015). The number of training frames, 300 or 600, is specified in the table header. The evaluation task is to condition on 100 frames, and then generate forward 400, 800, and 1200 frames.
Trained un 300 frames
100
400
Parums
PSNR
SSIM
Trapsfomerr
13.5
Porfomerr
164M
111 +
13.4
0.1
0.633
0.002
0.288
0.001
CW-YAF
20M
78 +
12.7
0.1
0.611
0.002
0.264 = 0.001
Con-LSTM
20M
57
1
3
16.9
0.2
0.796
0.004
0.113
0.000.000
Com-SM
init)
ConuS5
20M
26
18.1
0.830 2 0.003
0.094
0.002
ComvS5 (seproduced)
21M
26
17.9
0.003
0.097
0.001
Com/T3
21M
33
=
18.0 1
0.1
0.828
+ 0.003
0.096 0.002
Trained on 600 framess
100
400
Method
Purams
PVD
P
LPPS
Transfommer
164M
1
15.0 1
0.1
0.741
0.002
0.138
0.001
Perfomeer
CW-VAE
20M
+
2
12.9
0.1
0.621
0.004
0.242
0.001
Com-LTMM
20M
39
1:
17.3
0.2
0.812
0.005
0.100 2: 0.003
ComSSM (mandom init)
20M
81
15.5
0.1
0.743
1:43::00
0.002
0.163
0.003
Com
20M
0.532
0.003
Com-S5 (seproduced)
19.5
Con-T3
21M
16
19.8 1 0.1
0.871 0.002
0.066
0.000
Trained 300 frames
100 -
Method
Parums
PVD 4
PSNR 1
SSIM 1
LPIPS
Transurmer
164M
159
1:
12.6
0.1
0.809
0.002
0.287
0.001
Performer
164M
234
+
1
13.4
0.1
0.652
+ 0.006
0.379 # 0.002
CW-VAR VAE
20M
104
12.4
0.592
0.002
0.277
0.002
CTM
128
ComSSSM trandom init)
20M
287
+
13.6
0.1
0.577
0.001
0.293 0.001
ComS5
20M
72
1
16.0
0.1
0.761
1: 0.005
0.156
0.003
Com-5 (roproduued))
21M
=4
16.0
0.1
0.767
1: 0.004
0.152 = 0.001
Con-T3)
21M
79
16.1
2
0.1
0.776
0.004
0.146
0.002
Trained on 608 frames
100
Mathed
Pamms
FVD
PSNR
SSIM 1
1.PIPS
Traisformer
164M
2
13.7
0:207
0.003
Performer
164M
93
2:
12.4
0.1
0.616
0.002
0:274
0.001
VAE
20M
94
+
12.5
0.9
0.598
4 0.004
0.269
0.001
STM
20M
0.000
ComoSSM (random init)
20M
145
0.696
0.002
0:218
Comu55
20M
23
18.1
0.1
0.832
2 0.003
0.092
0.003
Com-55 (reproduced)
21M
35
3
17.6
0.2
0.819
0.005
0.109
0.000000 0.005
Conv13
21M
30
17.7
# 0.1 0.823
0.008
0,104
0.001
100 - 1200
Method
Parans
FVD
PSNR
SSIM
I.PIPS
Transfommer
164M
265 +
8
12.4 + 0.1 0.591
+ 0.002
0.321
+ 0.002
Performer
275
13.2
0.382
0.001
0.393
CW-VAR
117
12.3
0.585
0.002
0.286
0.001
ConSL.SM
20M
187 + 6
14.1
0.1
0.706
4 0.008
0.203
+ 0.001
CSM (candom init)
20M
511 8
13.3
0.515
4 0.000
0.348 : 0.001
187
14.5
0.678
0.003
0.230
ComyS5 (eproduced)
130
5
14.9
0.1
0.721
4 0.004
0.198
118
15.2
0.748
0.004
0.179
0.002
Trained frumes
100
1200
Method
Params
FVD
PSNR
SSIM
LPPPS 4
Transfommr
164M
91 +
13.1
13.1:00 D.1
0.631
+ 0.004
0.252 + 0.002
Performer
164M
243 7:
12.2
0.1
0.608
2 0.001
0.312 : 0.002
CW-VAR
0..3
0.580
Com-LSTM
20M
137 9
14.6
0.11
0.727
0.004
0.180 0.003
Can-SM (random ini)
20M
215
13.4
0.1
0.600
0.287 2 0.001
ComS55
20M
71 9
15.6 0.1
0.763
0.002
0.002
0.163 0.003
ComSS5 (repoduced))
2IM
06
0.1
0.791
+ 0.000
0.136 + 0.005
56
16.7
0.795
0.008
19

---

Published as a conference paper at ICLR 2026
ConvS5
ConvT3
0.5
0.4
Volid NRMSE
NRMSE
0.3
0.3
0.2
0.C
200
000
0.0
100
Epech
Epoch
Figure 4: Training loss curves of ConvT3 and ConvS5 in representative run on the Diffusion Reaction dataset, illustrating their training stability.
Table 12: (Ablations: Kernel size.) The introduction of a 3 3 state kernel A. in contrast to relying solely on same-sized B and kernels, resulted in notable performance gains. Bold: better performance under the same B and C settings.
Model
A kernel
B kernel
C. kernel
MSE 4
MAE J
1x1
1 1x1
1 1x1
14.98
30.60
X 1
1 1x
3 x
12.74
24.07
ConvS5
1x1
5
10.56
21.03
X 1
3x3
1x1
12.13
24.35
1
3x3
3x3
11.57
23.25
1 1XL 1
3x3 3
5X5
11.49
22.83
5x5
1x1
11.82
23.52
1
5
3x3
12.48
24.43
1
5
5
13.56
24.73
3
1 1x
1x1
14.68
29.65
3 3
1x1
3x3
11.17
23.04
Conv T3
3 3 3
1x1
5X5
9.91
20.38
3 3
3x3
1x1
11.69
23.46
3 3
3x3
3x3
10.99
22.15
3 3
3x3
5X5
11.34
22.43
3 3
5x
1x1
11.48
23.09
x 3
5x5
3x3
12.76
24.47
3 x
5
5x5
13.20
24.57
20

---

Published as a conference paper at ICLR 2026
D EXPERIMENT SETUP
Table 13:
Experiment configuration for ConvT3 and ConvS5 on long-range Moving-MNIST expeni ments. Overall model structure is ResNet.
Hyperparameters
Moving-MNIST-3300
Moving-MNNIS
Params
21M
21M
Input Resolution
64 x 64
64 x 64
Latent Resolution
16 16
16 x 16 8
Batch Size
8
Sequence Length
300
600
LR
1 10
1 10-3
LR Schedule
cosine 5k 300K
cosine 5k 300K
Warmup Steps
Max Training Steps
Weight Decay
1 x 10-5
10-5
Encoder
Depths
64, 128, 256
64, 128, 256 1
Blocks
Decoder
Depths
64, 128, 256
64, 128, 256
Blocks
Hidden Dim (U)
256
256
State Size (P)
256
256
ConvSSM
B Kernel Size
3 x 3
3 x 3
Kernel Size
x 3
x 3
Layers
8
Table 14: Experiment configuration for ConvT3 and ConvS5 PDEBench experiments. Learning rate for ConvS5 on Diffusion-Reaction was reduced due to training instability.
Hyperparameters
Shallow- -Wter
Diffusion-Reaction
Params
6M
6M
Input Resolution
128 128
128 128
Latent Resolution
8 x 8
8 x 8 8
Batch Size
Sequence Length
16 + 1
16 - 1
LR
5 x 10-4
x 10-1 / x 10
LR Schedule
cosine
cosine
Warmup Steps
10k
10k
Max Training Steps
IM
IM
Weight Decay
1 x 10-3
x 10-5
Hidden Dim (U)
384
384
State Size (P)
384
384
ConvSSM
Kernel Size
1
1
C Kernel Size
X 1
x 1
Layers
6
21

---

Published as a conference paper at ICLR 2026
Table 15: Experiment configuration for ablation studies. Overall model structure is ResNet.
Hyperparameters
ConvS5
ConvT3
Input Resolution
64 x 64
64 64
Latent Resolution
16 x 16
16 x 16
Batch Size
16
16
Sequence Length
10 + 10
10 + 10
LR
1 10-3
10-3
LR Schedule
cosine
cosine
Warmup Epochs
10
10
Max Training Epochs
200
200
Weight Decay
x 10-5
1 x 10
Encoder
Depths
64, 128, 256
64, 128, 256
Blocks
Decoder
Depths
64, 128, 256
64, 128, 256
Blocks
Hidden Dim (U)
256
256
State Size (P)
256
256
B Kernel Size
x 3 1
x
ConvSSM
C Kernel Size
3 x 3 3
3 3
Layers Off- Oif-Diagonal Proportion
2/4/6/8/ 12
2/4/6/8 12
N/A
-1,-1/1,1
Mini (Kernel Sharing)
N/A
Yes / No
22


Reviews:
[{"review_id": "YuQTALqSK5", "reviewer": "Reviewer_pYFy", "summary": "The paper proposes ConvT3, a convolutional state-space model (ConvSSM) that realizes a 3×3 state kernel while keeping linear-time parallel scans for long sequences. The key idea is to structure the state kernel as (i) a diagonalizable SSM matrix over hidden channels and (ii) a tridiagonal Toeplitz (TT) tensor over spatial axes. This structure enables efficient training while capturing richer spatial dynamics than prior 1×1 state kernels (e.g., ConvS5). On Moving-MNIST, ConvT3 achieves state-of-the-art results across most metrics; on PDEBench, it attains the best accuracy with efficiency close to ConvS5.", "questions": "See the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The paper shows how to move from 1×1 to 3×3 state kernels in ConvSSMs without losing linear-time scans, via a structured (diagonalizable) state tensor and tridiagonal Toeplitz (TT) formulation. \n\n+ Training stability: The paper uses a Hurwitz condition-based parameterization (constraining eigenvalues to have negative real parts) and a positive eigen-tensor construction to keep dynamics stable during training. This is validated by the training loss curve (Figure 3).\n\n+ Ablations on kernel size and minimal parameterization: MiniT3 outperformed ConvS5 despite the minimal increase in parameters, and the kernel-size ablations show the 3×3 state kernel A is the main driver of gains, not B/C alone.", "weaknesses": "- Limited benchmarks: Moving-MNIST and PDEBench tasks show the benefits, but they are relatively synthetic. Adding natural video datasets or harder physics targets (e.g., Navier–Stokes) would strengthen the paper. \n\n- Compute and throughput details: The paper states linear-time scans and that efficiency is close to ConvS5, but the actual comparisons are limited. Per-component speed and throughput are missing. Also, efficiency measurement across sequence lengths and resolutions would make the efficiency claims stronger.\n\n- Ablation: The kernel-size studies of A, B, and C are helpful, but additional ablations such as 3×3 vs 5×5 state kernels would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "moH2dmxqIh", "reviewer": "Reviewer_a6wG", "summary": "This paper proposes a ConvSSM method using Tridiagonal Toeplitz Tensors, which equivalently implements a ConvSSM with 3 $\\times$ 3 state convolution. Compared with traditional ConvSSM, this method give a constrained tridiagonal Toeplitz tensor. This method can avoid exploding computation in parallel scans with larger kernel and learn state dynamics from effectively capturing spatiotemporal context.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a new method which are used in statistical theory and leverage this method into state space model.\n2. This paper gives the proofs in detail and gives some theorems and definitions.\n3. The experiments prove that this method is good enough compared traditional methods on video generation tasks and complex physical system modeling.", "weaknesses": "1. This paper introduces an excessive number of notations, with some lacking sufficient detail—for instance, the formula in line 196 is not adequately elaborated.\n2. The paper fails to theoretically validate the effectiveness of ConvT3 in comparison to ConvSSM. It is suggested that the authors supplement a dedicated theoretical analysis section, rather than relying solely on experimental results.\n3. Regarding the experimental validation, only two experiments are presented. However, classification and detection tasks are standard benchmarks in computer vision. Additionally, traditional SSM-based methods (e.g., Vmamba) have been applied to CV tasks. It is recommended to include more comparative experiments covering these typical tasks.\n4. Certain theoretical details require clarification. Specifically, the calculation process of Equation 9 and its underlying rationale are not sufficiently explained.\n5. It would be valuable to provide a comparison of time complexity/cost between the proposed method and transformer-based approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9mMLVlF1aV", "reviewer": "Reviewer_V85B", "summary": "The paper proposes an extension of the ConvSSM by enabling a larger state-space kernel size, which increases from the original value of $1$ to $3$. The model is parameterized via a tridiagonal Toeplitz tensor and asymptotic stability is guaranteed. The model is empirically validated on long-range video generation (Moving-MNIST) and physical system modeling (PDEBench) tasks.", "questions": "1. Is $P = 3$ a special number? That said, since the theory is heavily grounded in the tridiaognal Toeplitz matrix theory, I imagine that it could be hard to generalize this framework to a larger $P$ per Galois theory; is this true? How does $P = 2$ compare to $3$?\n2. How does the memory usage of ConvT3 compare to that of ConvS5?\n3. In Table 1, you seem to show that ConvT3 works much better than ConvS5 in the long-horizon regime (see 800 frames versus 1200 frames). Do you have a good intuition in that?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The derivation of the ConvT3 block is rigorous and well-grounded in tridiagonal Toeplitz matrix theory.\n* The empirical results look promising, and the model seems to gain performance at a small cost of computing.\n* The ablation is thorough and insightful. It demonstrates the importance of four design choices separately.", "weaknesses": "* It is not thoroughly analyzed in the manuscript why increasing the kernel size from $1$ to $3$ in the latent space improves the model. While a complete theory may not be very accessible without an extensive amount of work, the paper would benefit from a dedicated discussion (theoretical or empirical, but on a more insightful toy example where further analysis can be done) that illustrates the benefits of a larger convolution kernel.\n* The mathematical derivation in section 3.3 is dry and purely technical. It is hard to tell what the final parameterization method is with a first glance. I would recommend that the author(s) start with a clear and accessible method and then use a theorem to show its stability property. (The proof can be dumped into an appendix.)\n* While ConvT3 is as efficient as ConvS5 in the $\\mathcal{O}$-notation, the constant inside is also important. This governs how the ConvT3 model is less computationally efficient in practice. It would be good to show some analysis of this kind or show some scaling plots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)