TRecViT: A Recurrent Video Transformer
Authors: Viorica Patraucean
Xu Owen He
Joseph Heyward
Chuhan Zhang
Mehdi S. M. Sajjadi
George-Cristian Muraru
Artem Zholus
Mahdi Karami
Ross Goroshin
Yutian Chen
Simon Osindero
Joao Carreira
Razvan Pascanu
Venue: Accepted by TMLR ()

Abstract: We propose a novel block for causal video modelling. It relies on a time - space - channel factorisation with dedicated blocks for each dimension: gated linear recurrent units (LRUs) perform information mixing over time, self-attention layers perform mixing over space, and MLPs over channels. The resulting architecture TRecViT is causal and shows strong performance on sparse and dense tasks, trained in supervised or self-supervised regimes, being the first causal video model in the state-space models family. Notably, our model outperforms or is on par with the popular (non-causal) ViViT-L model on large scale video datasets (SSv2, Kinetics400), while having 3x less parameters, 12x smaller memory footprint, and 5x lower FLOPs count than the full self-attention ViViT, with an inference throughput of about 300 frames per second, running comfortably in real-time. When compared with causal transformer-based models (TSM, RViT) and other recurrent models like LSTM, TRecViT obtains state-of-the-art results on the challenging SSv2 dataset. 
Code and checkpoints are available online \url{https://github.com/google-deepmind/trecvit}.

Published in Transactions on Machine Learning Research (12/2025)
TRecViT: A Recurrent Video Transformer
Viorica Pätrăuccan"
viorica @googgle com hezu hezu@google. com heywardj hegwardj@googgee com chubonz chuhonz@ggoogl com msaijodi @google. com graurar u @googgle com zholus @google. com mahdika &gooogle com goroshin @google, com yutianc @google. osinueero @google. com jouoluis @gougle. ruzp @google. com
Xu Owen He*
Joseph Heyward"
Chuhan Zhang
Mehdi S. M. Sajjadi
George-Cristian Muraru
Artem Zholus
Mahdi Karami
Ross Goroshin
Yutian Chen
Simon Osindero
João Carreira
Razvan Pascanu
Google DeepMind
Reviewed on OpenReview: https: // openreview. net/ forum? id= Md=Mmm,6Ytba b1H
Abstract
We propose novel block for cousal video modelling. Tt relies time spae-pppac-ehanee factorisation with dedicated blocks for each dimension: gated linear recurrent units (LRUs) perform information mixing over time, self-attention layers perform mixing over space, and MLPs over channels. The resulting architecture TiTeViT is causal and shows strong perfor- mance on sparse and dense tasks, trained in supervisod or self-supervised regimes, being the first causal video model in the state-space models family. Notably, our model outperforms or is par with the popular (non-causal) ViViT-L model large scale video datasets (SSv2, Kinetics400), while having 3x less parameters, 12x smaller memory footprint, and 5x lower FLOPs count than the full self-attention ViViT, with an inference throughput of about 300 frames per second, running comfortably in real-time. When compared with causal transformer- based models (TSM, RViT) and other recurrent models like LSTM, TRecVil obtains stato-of-tho-ars results on the challenging SSv2 datasct. Code and chockpoints are available online https: bttpp://github. httpttthu...mmmooooo.oett ctm//poggee-eeeepnnnn/teeeeeeei
1 Introduction
Vidco understanding requires low-level scene understanding (c.g. how objects move) and high-level reasoning (e.g. causal relations between events) over a signal that is high-dimensional, can be noisy, and contains high correlations and redundancies both spatial and temporal dimensions. Efficient video modelling needs high- capacity models that can represent the sheer diversity and richness of real-world vidcos, while operating causal manner with reasonable compute and memory footprint both training and during inference time these efficiency requirements are critical for Robotics augmented reality applications. Convolutional neural networks (Carreira & Zisserman 2017b Feichtenhoter et al. 2019 Lin et al. 2019 Kwon et al, 2020) have been a successful family of causal models for video, but their scaling capabilities (in both data and parameters) are limited due to their inductive biases (locality, invariance). Recurrent neural networks, e.g. (Srivastava et al.] 2015 Patraucean et al., 2016) have some desirable properties for video modelling (constant inference cost per timestep independent the length the video, causality for unidirectional
*Core contributor. 'Correspodding author.

---

Published in Transactions on Machine Learning Research (12/2025)
models), but they are slow to train due to their sequential nature and have difficulties in learning over long complex sequences. Transformers (Vaswani et al. 2017) have emerged as very powerful family of models for all modalities, with impressive scaling capabilities. However, they have significant memory footprint and latency due to the quadratic complexity of the self-attention operation and their performance degrades when using causal self-attention masks. Recently, a new family of linear recurrent networks (Gu et al. 2020; Gu & Dao, 2023 Orvieto et al. 2023b; Beck ct al.] 2024h, referred to as State Space Models (SSMs), has emerged as an answer to the quadratic complexity of sell-attention and the slow training of RNNs, with promising results for vision and language
(De et al.) 2024 Li et al., 2024 However, none of the existing video SSM architectures can run in causal manner, their performance strongly depends on bidirectional
operation.
In this paper, we propose a hybrid architecture that combines the best of all worlds. It alternates gated linear recurrent units (LRUs) (De et al. 2024) applied over time, with self-attention blocks over space, and MLP over feature channels. As opposed to space and channels, time has natural order ( ('arrow-of-time)) that LRUs can implicitly and efficiently model in causal manner with ((N) complexity in the number of input frames at training time and (1) complexity at inference time, making it possible to process in real-time videos that extend even indefinitely. Space, on the other hand, has a fixed limited dimension, for which the quadratic cost of self-attention is more accessible. From a practical perspective, using self-attention over space allows us to naturally process in parallel all the pixels of a given frame, without having to commit to a particular scanning order (Li et al. 2024 making better use of hardware when parallel resources are available. Importantly, by restricting the LRUs to temporal-only recurrence, this factorisation reduces the sequence length by about two orders of magnitude compared to models that apply recurrence across both 2024 Such models typically require bidircctional scanning to allain strong performance, preventing them from operaling in causal manner.
space and time (Zhu et al. 2024 Li ct al.
To further limit the self-attention cost, we use spatial patches as introduced in the successful VIT (Dosovisskiy ct al. 2 model. But, compared to cxisting video transformer models, c.g. ViVIT (Arnab et al. 2021), the patches do not have a fixed temporal extent. Instead, the embeddings of the spatial palches are integrated continuously into the hidden state of the gated LRUs, providing persistent memory of the entire temporal sequence up to the current frame, leading to a causal architecture. Furthermore, similar to convolutional networks, the parameters of the LRUs are shared over space, preventing the number of parameters from exploding as the resolution of the video increases.
We refer to the resulting model as Temporal Recurrent Video Transformer (TRecViT). TRecViT is highly flexible and can address various video understanding tasks, both sparse (e.g. video classification) and dense (e.g- point tracking), trained in a supervised or self-supervieed manner, e.g. using masked auto-encoding. In all our experiments, we use a causal setup that respects the arrow of time, so the model is suitable for any downstream applications, from c.g. video classification where we have offiine access to the vidcos, to e.g. Roboties, where online processing is required. Overall, model is significantly more efficient in both memory footprint and FLOPs compared to vanilla transformers, and obtains stato-of-the-art performance when compared to causal models, while comfortably running in real time (e.g. throughput of about 300 frames per second for point tracking task).
We propose a causal video architecture that uses a novel factorisation interleaving LRUs and VIT blocks. We run ablations for the building blocks to find optimal hyperparameters and present extensive experiments on multiple tasks and training regimes, showing the versatility and strong performance of our model.
Contributions:
2 Related work
Transformers for Vidco. Proposed initially as language models, transformers (aaswani et al. 2017 have quickly become the dominant architecture across multiple modalities (images, audio, video). Transformer blocks alternate between a spatial mixing block represented by self-attention and a (feature) channel mixing block, typically represented by gated MLP. Given that the self-attention layer treats the input tokens as a set, positional encodings must be used in order to specify the location of each token. This also implics that no parsing order is needed, unlike the case with RNNs. Vision transformers (VIT) (Dosovitskiy al. 2021
Liu
2

---

Published in Transactions on Machine Learning Research (12/2025)
et al.
split images into fixed number of patches that are projected into an embedding space to obtain lokens and these are then processed by regular transformer. Several works extended ViT to video (Arnab
2021
Bertasius et al., 2021: Li et al. 2021 Fan et al. 2021 Patrick et al. 2021 e.g. by replacing the regular image patches with spatio-temporal oncs. The main challenge with transformers, particularly for video, is the quadratic complexity in the number of input. tokens. Multiple approaches have been proposed to address this: e.g. factorisations of the self-attention operation (Arnab et al. 2021, Bertasius et al. 2021
et al. 2021
iterative attention (Jaegle al. 20221, sparse sampling of the input frames Piergiovanni al. 2023 and
distributed self-attentionn operations across different devices (Liu et al,, 2024 Our proposed model uses novel space-time factorisation, where the temporal dimension is handled by LRUs and the spatial dimension by self-attention, resulting in a causal efficient video architecture.
As these models scale successfully to large number of parameters, their data needs are efficiently met by using self-supervised pre-training like masked autoencoding (MAE) (Tong et al., 2022 or contrastive learn- ing (Zhao et al. 2024). Due to the factorisation used in our architecture, using such pre-training strategies is straightforward and we include successful experiments with MAE pre-training in Section 5
SSM, type of Linear Recurrent Model.
While transformers (Vaswani et al. 2017 can be efficiently
parallelised during training, at inference they need to pay quadratic cost in the sequence length. On
the other hand, recurrent networks (Elman 1990 Hochreiter & Schmidhuber 1997a; Mikolov et al.] 2010
Bahdanau et al. 2014 Sutakever et al. 2014) are compact and efficient at inference but slow at training.
State Space Models (SSMs) (Gu et al. 2020 2021 Orvieto et al. 2023b), a particular type of linear recurrent
networks, have recently been proposed as an answer to the scalability problem of RNNs, and have shown strong performance in language and other long-range dependencies tasks (De et al. 2024
Gu & Dao 2023
SSMs, like S4 (Gu et al.
2021 S4D (Guu et al.] 2022), or Mamba (Gu & Dao 2023 have been introduced
as particular discretizations of continuous time linear system. On the other hand, the linear recurrent unit (LRU) (Oricto et al. 2023b) was designed by identifying the minimal set of changes to a vanilla RNN Elman 1990) that allows to obtain the same key properties as the S4D architecture (Gu et al. 2021 Specifically, the nonlinear recurrence typical of a recurrent, model was removed to improve the scalability and controllability of the system, as the linearity allows the recurrent matrix to be diagonalised through cigenvalue decomposition and absorbing the (dense) cigenvectors matrix into the neighbouring layers. This gives direct access to the eigenvalues of the Jacobian of the transfer function characterising the system. By parametrising the diagonal weight matrix containing the eigenvalues such that all entries are constrained to be below one, the system is guaranteed to be stable, bypassing exploding gradients issues. However, using only linear recurrence can greatly limit the expressivity of the layer. In (Orvieto et al., 2023a), the authors show that by using these layers within a typical transformer structure that alternates linear recurrences with point-wise nonlinearities (e.g. the MLP block), the overall architecture recovers expressivity through depth and can be shown to be universal approximator of finite sequence-to-sqqecee maps.
Improving on the LRU, the gated LRU De et al. 2024 introduces gating mechanisms similar to LSTM or GRU architecturos, to filter the input sequence, or, for the recurrent gate, to control the rate of the information decay. Importantly, different from LSTM or GRU, these gates do not depend on the previous state, which would prevent parallelisation at training time. A series of recent works rely on formulating the lincar recurrence as a form of lincar attention, such as Mamba 2 (Dao & Gu, 2024], Gated Delta Networks (Yanget et al., 2025), Networks fvon Oswald et al., 2025), , akin previous observation, providing more direct connection to attention. These works exploit this connection to show that the recurrence can be seen as update rule of a local objective, similar to interpretability works on in-contezt learning. In this line of work, it is also worth noting xLSTMs (Beck et al. 2024). which connects SSMs more explicitly to traditional non-linear models such as LSTMs (Hochreiter & Schnidhuber 1997b). In our work, we use gated LRUs, but we expect similar results when using other gated SSM blocks like Mamba or xLSTM within our factorisation.
SSMs for Video. While SSMs have mostly been explored in language, several architectures like S4 and Mamba have also been adapted to image and vidco understanding ZZang et al.) 2024 and generation Gao
et al. 2024h. ViS4mer IIslam & Bertasius] 2022] uses ViT image encoder to process videos frame by frame,
and integrates their representations over time using S4 blocks at the top. TranS4mer (Islam et al. 2023
3

---

Published in Transactions on Machine Learning Research (12/2025)
Book
VIT block
Maingg
LRU
BDO
LRUP
TR.D
Self Alt-Atttiinn Eppatian Mixing!
ConulD
Tokens
Reshepe (BxT.
Linear projaction of image patches
Getect (Tempora Mikinn)
(Bok T.D)
RMShoom
Geted LRU
t+1
T. DD
Figure 1: Left: TiTcVIT archilecture. Each video frame is divided into non-overlapping patches that are linearly projected into token embedding space. We then add a learnt spatial positional encoding, The tokens are passod through gated lincar recurrent units (LRUs) that share paramcters across space. The outputs of the recurrent blocks are then processed by a ViT block. The recurrent operation followed by VIT is repeated N times. Right: TiecViT block. The input is a batch of videos, each frame with N tokens. We apply recurrent units over temporal tubes integrate information over time, and self-attention and MLP across tokens within each frame. Note that the recurrent units share parameters, but the information is not mixed across temporal tubes. Similarly, the ViT blocks share parameters, but the information is not mixed across frames. There are skip connections around each block, not included here to reduce clutter.
uses self-attention over short clips and integrates these with gated S4 blocks. More recently, the Mamba architecture was extended to images and videos by having it process a flattened ID sequence of image or video patches. This requires defining a processing order for the patches, and different orders have been proposed, e.g. bidirectional and following a column or row order (Zhu et al. 2024 Li et al.) 2024 Shi all factorisation naturally uses
2025 Lu et al. 2.25). As opposed to these Mamba-based architectures, the arrow-ol-time to decide the scanning order, resulting in causal model. Another important benefit of our hybrid architecture is that we can initialise the VIT blocks with strong existing pre-trained weights. This leads to strong performance even at larger scale, as opposed to VideoMamba (Li et al. 2024 where the authors report severe overfitting issues, requiring distillation from smaller models when training in a supervised fashion or distillation from CLIP features (Radford et al. 2021 for self-supervised training.
Causal video models. All the models mentioned above process the videos in an offline manner, i.e. require access to all frames at the same time, making their application in streaming setups (c.g. Robotics, virtual reality) sub-optimal. A few works have proposed adaptations of convnets and Transformers to causal operation (Lin et al. 2019 Zhao et al. 2023 Yang et al.) 2022). In (Zhao et al. 2023), the authors store the keys and values from previous frames to perform cross attention with queries from the current frame, but they still use a non-causal temporal transformer for action recognition tasks. The closest to our work is RVIT (Yang et al. 2022), which proposes a linear attention-basel gating for processing videos recurrent manner. In the experiments section, we show that our recurrence based on LRUs leads to theea- results in causal operation settings and competitive performance with non-causal models.
3 TRecViT Architecture
Let X E [o, 1|T>XXx x3 be an RGB video with T frames and H x W pixels. The video frames are split into N non-overlapping patches o of size n x n x 3, with E {1,T) and & {1,N}. J. Let a be the tokens obtained after the linear projection of the patches and the addition of the spatial positional encoding, with token size x d, where d is the token feature dimension. A temporal tube is a sequence {a/[tt I,T) containing the tokens from the same spatial location across frames.
Notations:
4

---

Published in Transactions on Machine Learning Research (12/2025)
The proposed architecture, TiT,VIT, is composed of repeated identical blocks, each performing a sequence of ion mixing steps across the different. dimensions of the video signal: time, space, and channels; see Figure ! The mixing over the time dimension is handled by gated linear recurrent units (LRUs), similar to the one introduced in (De et al, 2024) for language. Each spatial token associated with an LRC that processes the lokens within the same temporal tube over time, without mixing the information across temporal tubes, so each LRU has its own state we use LRU1, LRU2,.. in Figure [ to highlight this. The LRUs share parameters over space, similar to a convolutional network. When applying this temporal mixing operation, the space dimension is transposed into the batch dimension.
The mixing over spatial and channel dimensions is handled by standard ViT block, which first performs the spatial mixing through a self-attention operation, then the channel mixing by using an MLP. When performing the spatial and channel mixing, the time dimension is transposed into the batch dimension. We first perform temporal mixing, followed by spatial and channel mixing. We found this time-space order produce better results compared to space-time order. We hypothesise that, by applying LRUs first, this allows them to focus on more local, easier to model, information at the first layer, instead of operating directly on features that mix information across the entire frame.
Empirically, we show that this time-space-channel factorization and choice of building blocks is more efficient for understanding temporal dynamics compared to video transformer approaches (e.g. ViViT (Arnab et al. 202Ip) or pure SSM models. By applying self-attention over the spatial dimensions, we allow all tokens to attend to all the other tokens parallel, without having to commit to a particular order (unlike in VideoMamba). We employ strong transformer blocks from ViTs for this operation, including their Imagenet pre-trained weights. The recurrence of the temporal processing enables efficient frame-by-frame inference over long videos, with constant memory footprint and causal operation.
3.1 Gated LRUs for Video
We adopt the gated variant of the LRU (De et al. 2024 to design our proposed block for video modelling. Although LRUs have been proposed for language, we hypothesisc that they can effectively model vidco signals, given the continuous nature of the video and LRUs' inspiration from time continuous systems. We run extensive analysis and ablations for the different components and hyperparameters of the gated LRC block to find a configuration that works well for video; see analysis below and ablations in section 5.1
The gated LRU operation described by the below equations:
is oW + ba)
input gate recurrence gate
(1)
T1 = (WAxe + bx)
(2)
A = CA)
(3)
h = 10h--1 + V1
X2
+ (1e
① x.)
(4)
where he E RRd the state of the LRU, d E Rdd is a vector containing the eigenvalues of the (diagonal) recurrence matrixD E Rd is the input gate controlling whether x, Rd is integrated within the state hy of the LRU or not, and r. Rd is the recurrence gate. The weights and biases of the LRU (We Raxd WA E Rdddd by E IRd, by E Rd) are initialized using LeCun init [LeCun et al. 2012)
The (learnable) recurrence weights are passed through a sigmoid function to ensure they are between 0 and 1, and are initialised such that aA) is sampled uniformly in [Aminn Amax]. These recurrent weights are raised to the power C. Tt, which effectively acts as a gate controlled by re (equation equation 2 rt. is defined as a lincar projection, with parameters W,, and bx, followed by a sigmoid function to ensure again the range [0, 1]. By raising element-wise aA)) to rt, the effective recurrence weight at some position j can change between the j-th entry of a (A)) when the corresponding gate entry is 1 and 1 when the gate entry is 0.
Similar De al. 2024), we implement the recurrence weights A exp(-C -sp(-C-softplnea))) softplus(A) rz), which is mathematically equivalent but numerical ly more stable.
5

---

Published in Transactions on Machine Learning Research (12/2025)
The additional constant cocfficient C R, typically set to 8 as in (De et al, 2024 increases the range to be between of (A)C to 1, providing additional flexibility. E.g. if (A) is 0.9 and we set C == 8, we extend the range from [0.9, 1J to [0.43, 1]. More importantly, we change the learning dynamies (e.g. gradient norms) and resolution we have over the range during learning. Specifically, for Iy in some fixed interval and similar magnitude W,, as it. is the case at initialisation, a higher value of implies ) will concentrate more towards the edges of the range. Note also that this is the dynamic range which the recurrent weights can vary during inference as function of the input tokens.
In (De et al. 2024 the authors found that setting Amin 0.9 and Amax 0.999 leads to the best results. An cigenvalue of 0.9 implics that it will take at least 10 time steps for the information to decay to roughly 35% of its magnitude, while for an eigenvalue of 0.999 it will take 1000 time steps to decay by the same amount. When using the same range for video modelling, we observed that the eigenvalues are pushed significantly towards Amin during training, with a small number cigenvalues becoming smaller than m see Figure
We experimented with extending the range and obtained better results with Amin = 0.6; see Table This leads faster decay of information initially and might reflect the importance for videos of having enough recurrent units focused short term information, in order to disentangle fast changing dynamics from slow ones.
Start of training
End training
Start of training
End of training
25
30
15
10
15
10-
0.7 0.6
00
Griffin range [0.9 0.9991, final PSNR 30.9
Extended range [0.6 0.999). final PSNR 31,2
Figure 2: Distribution of the eigenvalues of the recurrent matrix at the beginning and end of training on long video memorisation task (see subsection 5.4 for different initialisation ranges.
Finally, note that when diagonalising the recurrence matrix, the cigenvalues À could, in theory, have complex values. We conducted experiments using complex eigenvalues, but we did not see improvements compared
to using only real eigenvalues. The same observation was made in De et al. 2024 Gu & Dao 2023 as well.
3.2 Video block based on gated LRU
We use the gated LRU in a similar block structure as the one employed in DDe et al. 2024h, see Figure D Given 1D input (temporal tubc), the block first applies normalisation layer, then the signal is routed two different paths. On the first one, it gets linearly projected to same dimensionality d and then the GeLU activation is applied. On the other path, the signal is also linearly projected to the same dimensionality d, then we apply a 1D convolution followed by the gated LRU described in equation ! The output of the LRU and the GeLU branch are element-wise multiplied and then linearly projected to the same dimension d. Note that, line with (De al. 2024), we use a separable convolution, which allows mixing information only over time, not over channels. We sweep the width of the convolutional kernel and find that window of gives the best results, similar to (De al., 2024). Different from (De et al., 2024), we do not use an MLP block after the LRU for feature mixing. We apply the MLP after the self-attention block, as done in VIT.
Given the diagonal form of the recurrence, on device, the gated LRU computations are memory-bound, i.e. the data transfer takes longer than the actual computations done on that data. Similar to (De et al.) 2024h we use a specialisod Pallas (Brddbury et al. 2018) kernel that minimizes the number of bytes that need be moved between HBM and VMEM (the Vector Processing Unit's cache). The parameters added by the linear projections within the block, well as the parameters of the convolutto and the LRU, are learned.

---

Published in Transactions on Machine Learning Research (12/2025)
500
VIVT
VIVIT L
VIVT
VIVIT
VIVI (factorised SA]
VIVIT L (factorised SA)
400
VIVI (factorised encoder!
VIVT (factorised encoder)
Posk Memory (GB)
TRacVIT tours)
50
TRecVIT tours)
300
TFLOPS
40
200
30-
20-
100
10-
10
Number Frames
Number Frames
Memory comparison; Right: FLOPs comparison. Our model demonstrates increasingly greater memory and compute savings compared to ViVIT full self-attention as the number of frames in- creases. The elficiency of ViVIT factorised self-atiention is very similar to TRecViT's, but its performance is significantly lower. The factorised encoder variant of ViVIT-L is very efficient, but it performs well mainly on sparse tasks.
Figure 3: Left:
4 Training TRecViT
The proposed architecture can be trained in a supervised or self-supervised regime. Given a tokenised video input, the output of ViT will have the same dimension and shape as the input, meaning that we can easily recover the spatio-temporal structure of the input video, which can be useful for dense tasks like pixel reconstruction, depth estimation, or point tracking. At inference time, the architecture can be applied over all the video frames at once, or frame-by-frame by carrying over the state of the LRUs. Depending on the task, one can choose to keep all the outputs from all time-steps to make a prediction (similar to ViViT), or just the outputs from the last, step, given that the LRU integrates the previous history in its state. In our experiments, we use mainly the former for fairer comparison with ViVIT, but we also experiment with the latter to analysc LRU's capability of remembering over very long context; see subsection
4.1 Self-supervised pre-training
Given the factorised nature of the proposed architecture and the redundancy present in the video signal, it comes natural to apply masked auto-encoding to enable self-supervised pre-training from scratch on large- scale unlabelled datasets.
We follow the same recipe as in the original VideoMAE paper (Tong et al. 2022 tube masking where a 2D random mask is generated and repeated for all the frames in the video. For our architecture, this is equivalent to dropping temporal LRU's The training objective is simply L2 reconstruction error of the entire frames. We sweep the value of the masking ratio and we find that 0.90 leads to best performance on downstreeam tasks. When using the pre-trained representations for downstream tasks, we keep all the tokens of the video and we add decoder or readout head that is fine-tuned for the respective tasks.
Specifically, use
4.2 Computational complexity and efficiency analysis
We discuss here the computational complexity of the proposed architecture and its efficiency compared to different variants of ViVIT. Our model's parameter count (111M) falls between that of ViViT-B (90M) and ViVIT-L (310M). We focus on ViVIT-L as the main point of comparison in our efficiency and SOTA analysis (Section 5.2 as this is the strongest representative of the ViVIT family reported in the original VIVIT paper. Furthermore, since our architecture is suitable for both sparse and dense video tasks, we compare the efficiency benefits mainly against the more general full self-attention ViViT-L variant, which utilised by influential follow-up works (c.g., VideoMAE Tong et al. (2022)) over the specialised Factorisod Encoder (FE) version, which models temporally only one token per frame, being suitable for classification tasks, but not for dense tasks.

---

Published in Transactions on Machine Learning Research (12/2025)
We consider video with T frames, each frame with N spatial tokens of dimension D, and a model with L layers. The full self-attention ViVIT flattens the video sequence and computes attention scores between all T. N pairs of tokens followed by a token-wise MLP block, resulting in a complexity DD-INN with the first term associated to self-attention dominating the overall computation. For our hybrid architecture, we consider the complexity of the ViT blocks and the gated LRU blocks separately. The L ViT blocks perform spatial attention over the N tokens within each frame, resulting OLLT(N3 + ND))) complexity. The L gated LRU blocks perform temporal processing across the T frames for all N tokens in parallel, contributing O(LTND²) complexity, Hence the total complexity for the proposed model is O(LTN D + LTND²). The main difference is that ViViT full self-attention scales quadratically with the mumber of frames (T²) and the number of spatial tokens (N²),, whereas our model scales linearly with the number of frames (T") and quadratically with the number of spatial tokens (N²).
This difference is reflected in the memory footprint and FLOOPs count of the two types of models; see Figure B The profiling results are obtained by cost and memory analysis of lowered Jax HLO on CPU backend to be aligned with the theoretical numbers (JaxAuthors] 2018). We consider as inpul. video of size 224 x 224 and we vary the length of the video to analyse the savings provided by our architecture as the length of the video increases. The peak memory and number of flops for TiecViT are significantly lower as the number of frames increases, e.g. at 32 frames (the number of frames typically used in video classification experiments). TieeViT's peak memory is 12x smaller than that of ViViT-I full self-attention and the FLOOPs count is 5x lower. When going to 64 frames, the peak memory is ~44 smaller and FLOPs count is 8x lower. This is due to the fact that ViT maintains its history in a fixed-sized compressed state, whereas full self-attention Transformer-basee models store in cache the keys and values for the full history. Similarly, the FLOPs count grows exponentially for VIVIT due to its quadratic complexity, where each input token atlends to every other loken in the video. The factorised self-atlention and factorised encoder variants are significantly more efficient than the full self-attention and comparable to ViT. However, the performance of the factorised self-attention version is significantly worse (see ablations in Table 2 whereas the factorised encoder is tailored specifically for sparse tasks, modelling only one class token per frame.
5 Experiments
We present results for supervised vidco classification and self-supprvised masked auto-cncoding with frozen representations evaluated on two downsttremm lasks: video classification and point tracking. To analyse the memory capabilities of our model, we also include a reconstruction task of frames seen in the distant past. Using the same task, we study the generalisation capabilities to longer sequences than seen during training. We follow the VIT scaling configurations and, unless otherwise stated, we use the Base version for our model for all our experiments. We specify the number of parameters for all models considered in our experiments, and we include in the supplementary material all the model configurations, training hyperparameters, and data augmentations used in all experiments.
Datasets: We use mainly two large-scale real-world datasets for ablations and for SOTA comparison on the supervised video classification task. Kinetics400 Carreira & Zisserrann 2017a contains 241,512 videos F across train, validation, and test splits, 10s-long (25fps), spanning 400 classes. This dataset is known to require modelling appearance for successful action recognition. To challenge our model's capability of understanding motion, we also use SSv2 datasct (Goyal et al.) 2017p, which contains 220,847 shorter vidcos (2-6s long), sampled at 12fps, representing 174 classes. This dataset includes actions that differ in finer notion-related details, requiring a deeper temporal understanding, e.g. pouring something into something vs pretending to pour something into something.
5.1 Ablations
Gated LRU block: We ran multiple ablations for the components of the gated LRU block used in TRecViT using SSv2 supervised classification as task, see Table [
We study the impact of the different components and
2Kinetics a dynamic dataset (videos may be removed from Tube). Our current version has 241,512 videos, compared to 267, ,00 videas reported (Arnah et. all 2021). so a decrease of almost 10%, noticeable in the final performance.
8

---

Published in Transactions on Machine Learning Research (12/2025)
of the initialisation range for the cigenvalues of the recurrent matrix, mentioned in section 3.1 We report top-1 classification accuracy, i.e. the percentage of videos for which the model's 's highest-confidenne prediction matches the ground truth label. The results show that the skip connection is essential to successful training; without it, we need to lower the learning rate for the training to take off at all, and the final performance is still very poor. Removing the input and recurrent gales leads to slightly reduced performance, in line with the observations on SSM in language. The input gate adds selectivity on the input, while the recurrent gate modulates how far in the past the model looks. Removing the 1D convolutional layer leads to bigger decrease, which again is in line with the original Griffin paper that mentioned the important role of the ConvID layer in extracting local temporal features.
Ablation
Top-1(%)
Gated LRU block [0.6, 0.999) (ours)
66.7
Griffin range 0.9, 0.999]
66.2
No skip connection (lower Ir)
15.3
No input gate, recurrent gate
66.0
No convolutional layer
64.9
Table 1: Ablation on Gated LRU components and eigenvalue initialisation. Task: SSv2 classification. Metrie: top-1 accuracy (higher is better).
Training from scratch: We run experiment compare the proposed factorisation against simpler adaptations of LRU to video, similar to the ones used in VIVIT and VideoMamba, i.e. patchify and flatten the entire vidco, add spatio-temporal positional embeddings, and apply 1D self-attention or SSM blocks as done in language lasks. Figure g compares the proposed TiecViT with these two baselines, with all models being trained from scratch on supervised classification on SSv2. We consider the Small version for all models as the larger Base version shows stability issues when trained from scratch, as reported other works as well Li et al.) 2024 Arnab al.) 2021). As expected, the performance on this challenging dataset when training from scratch is far from SOTA, but it clearly shows that the proposed factorisation has superior video modelling capabilities compared to baselines, ViVIT-S with full self-attention being the closest competitor. PureLRU's performance is very poor, which is in line with the findings from VidcoMamba that bidirectional (non-causal) processing of the input is needed for good performance.
Same factorisation, different temporal blocks: We compare our proposed hybrid architecture against other factorised variations that have similar efficiency; see Figure $ the appendix for memory footprint and FLOPs comparison. The results are included in Table 2 and clearly show that the proposed hybrid architecture is superior in performance and in training throughput compared to architectures using LSTM or factorised self-attention to integrate temporal information. The model using ConvlD 1D as temporal module is faster to train, but its accuracy is far from competitive.
We include in the appendix ablations by running supervised classification on SSv2 using different hyperparameter values: temporal dimension of video patches, minimal radius for eigenvalue initialisation, window size for the 1D convolution in LRU. We also include an experiment considering multiple seeds and report mean and variance.
Additional ablations:
Temporal module (w skip)
Top-1 (%)
Steps per sec (training)
VIVIT factorised self-attention
62.4
8.3
VIT-LSTM
63.7
6.0
ViT-ConvID
40.3
10.5
TRecViT
66.7
8.9
Table 2: Comparison of different temporal modules used in hybrid setups similar to our proposed TiecVIT. Task: SSv2 classification. Metric: Top-1 accuracy (higher is better).
9

---

Published in Transactions on Machine Learning Research (12/2025)
SSV2 from
TRecVIT-S WVTT-5
WMTS FE PureLRU-S
Top-1 Accuracy
0.25
0.15
0.05
0.00
0
10000
20000
50000
40000
Training steps
Figure 4: TiTeViT compared to baselines supervised video classification on SSv2 dataset, trained from scratch. The plot shows the evolution of the evaluation accuracy training progresses (higher is better).
5.2 SOTA comparison on supervised video classification
We compare the performance of TiecViT against top models in the literature, considering both causal and non-causal architectures; see Table g for SSv2 results and Table ! for Kinetics400 results. On SSv2, our model achieves stato-of-tho-art performance compared to causal baselines (TSM, RViT, causal ViViT) and outperforms or compelitive with all non-causal baselines. Notably, outperforms the popular ViViT-L by 2.3% despite having about 3x less parameters.
On Kinctics400, TReeVIT outperforms convolutional architectures (I3D) and some Transformer-basete ar- chitectures (TimeSlormer, ViVIT-B, causal ViVIT-L). Compared to RVIT and ViViT-I, the performance is competitive, but slightly lower. We attribute this in part to the reduced number of videos in the dataset as mentioned above when training both TRecViT and ViVIT-L on the same number of videos, the perfor- mance gap is significantly reduced. In addition, this result could also reflect the difference between the two datasets mentioned above (appearance vs motion) and highlights that TiecViT is superior at modelling mo- tion compared baselines, but on Kinetics400 where the appearance is enough for successful classification, the performance is par.
5.3 Self-supervised masked autoencoding
We use Kinetics400 for self-supervised pre-training from scratch and we report results on multiple downstream datasots and tasks by fine-tuning attention readout heads on top of frozen representations. We choose this setup, as opposed to fine-tuning end-lo-end, as the performance this case more clearly reflects the quality of the pre-trained representations. As mentioned in the previous section, we use large masking ratio (0.90), which makes pre-training very efficient. We report the number of parameters for every model considered. Note that the number of parameters for TiecVIT is different from the one reported in the previous section due to the addition of the readout heads.
Video classification: We report video classification accuracy as downstream task using attention readout heads on SSv2 and Kinetics400. We compare the performance against VideoMAE-L (Tong et al., 2022) in Table P Our model obtains slightly better performance on both datasets compared to this strong baseline, despite having almost 3x less parameters.
To demonstrate that model can handle dense(r) tasks well, evaluate the same frozen MAE representations for the point tracking task. We use the recurrent architecture in MooG (van Stecnkiste et al. 2024) as readout head due to its simplicity. MooG uses light cross-attention layers to process the embeddings of each frame in order, and the readout state is carried over time. We finetune the readout head using MOVi-E dataset (Greff et al. 2022) as done popular point tracking works (Doersch ct al. 2023). We evaluate these finc-tuned representations on two datasets: Perception Test (Pätráuccan et al. 2023) and DAVIS dataset (Pont- [Pont-Tuset Tuset al. 2017) with point tracks extracted (Doersch et al. 2022). We report average Jaccard metric, i.e. the percentage of predicted points that fall within certain threshold
Point tracking:
10

---

Published in Transactions on Machine Learning Research (12/2025)
Model
Pre- Train
Top-1 (%)
Param [M)
FLOPs (T)
Mem
(G)
non-causal
I3D
K400
51.3
25.0
N/A
N/A
SlowPast R50
K400
61.9
34.1
0.19
3.35
SlowFast R101
K400
63.1
53.3
0.32
4.20
MSNet
IN-21K
64.7
54.6
0.07
6.54
TimeSformer-I
IN-21K
62.4
121.4
5.1
>2
VidTr-L
63.0
N/A
10.5
N/A
ViViT-Lau
63.9
310.8
7.75
6.11
Mformer-B
IN-21K
66.5
114.0
1.10
7.3
MVIT-Ba2
K400
67.1
36.6
0.51
10.7
MVIT-Be4
K400
67.7
36.6
1.36
24
MViT-B-24gz
K600
68.7
36.6
1.36
24
VideoMamba
IN-21K
68.4
74.0
0.33
N/A
VideoMambaPro
IN-21K
69.4
72.0
2.2
N/A
causal
TSM
K400
63.3
42.9
0.19
5.98
eViViT-La2
TN-21K
64.4
310.8
N/A
N/A
RViT-La2
K400
66.1
72.0
1.34
2.12
RVTT-XLes
K400
67.9
107.7
3,99
2.33
TRec V/T
IN-21K
66.8
111.3
1.44
1.79
TRex V47
IN-2IK
68.2
111.3
2.89
36
Table 3: Performance comparison on SSv2 dataset, considering causal and non-causal basclines. cViViT-L is causal VIVIT model that trained using causal attention masking for the self-attention block. We do not include FLOPs and Mem for this model they will heavily depend the implementation of the causal masking. Where present, the subscript indicates the number of frames used at inference time. Metrie: Top-1 accuracy (higher is better). *Result reported in Yang et al. 2022). **For TRecViT, the memory at inference independent of the number of frames, here we run causally on the entire sequence in parallel, hence the memory increase.
Model
Pre-Train
Top-1 (%)
Param (M)
non-cousal
13T)
IN-1K
72.1
25.0
TimeSformer
IN-21K
78.0
121.4
Mforner-B
IN-21K
79.7
114.0
VIVIT-L
IN-21K
80.3
320.0
VIVIT-B
IN-21K
78.1
91.2
VIVIT-I.
IN-21K
78.7
310.9
causal
RVYT-XL
IN-21K
81.5
107.7
cViViT-L
IN-21K
76.3
310.9
TRecVaT
IN-21K
76.5
111.2
Table Performance of TRecVIT compared to convolutional and transformer- based baselines on Kinetics400 dataset. For fair comparison, we trained ViViT-B and ViViT-I. models on the current Kinetics400 dataset version; footnote 2 The numbers in gray correspond to results obtained the original larger Kinetics400 dataset, reported by their authors. Metric: Top-1 accuracy (higher is better).
from the ground truth points (Dorssh et al. 2022 for TiecViT compared with MooG and VideoMAE; see Table TiT obtains better performance on both datasets compared to basclines, which reinforces the observation that our proposed model has strong motion modelling capabilities. We include qualitative results for this task in the appendix.
5.4 Long video memorisation task
Transformer models for language are known be excellent at retrieving information from context, as they cache the keys and values for the entire history. On the other hand, LRUs SSMs and RNNs in general
11

---

Published in Transactions on Machine Learning Research (12/2025)
Model
Datasct
Top-1 (%)
Param (M)
VideoMAR
Kinetirs400
45.8
3:00
TRec VIT
Kinetics400
46.0
128
VideoMAE
SSv2 SSv2
53.7
330
TRec VET
53.9
128
Table 5: Performance of TRecVIT compared to VideoMAE on video classification using frozen MAE repre- scntations, pre-trained on Kinctics400. Metric: Top-1 accuracy (higher is better).
Model
Num frames train
Num frames
PSNR
ViVIT-L
96
96
32.2
VTT
96
96
29.1
VIVIT-L
64
96
15.1
TRec
64
96
26.4
Table 6: Comparison between ViT and ViVIT-L on the long video memorisation task, when the models are evaluated on the same number of fraies seen in training (top) on longer sequences (bottom). Metric: Peak Sognal-To-Noise Ratio (PSNR); higher is better.
struggle with such meede-in-the-haee compressed history kept in their recurrent state JJeassi et al., 2024 De et al. 2024
style tasks as they need to perform the retrieval based on the
We are interested studying this aspect in the video domain as well. We set up a simple reconstruction task where the model has to remember the frame seen a given time-step the past. For example, we train the models with sequences of length T = 96 frames to reconstruct the 16th frame. Using the same task, also analyse the generalisation capabilities to sequences longer than those seen during training, iee. the models are trained with sequences of length T 64 frames to reconstruct the 16th frame, and then are evaluated on video sequences with T = 96 frames to reconstruct the same 16th frame.
We employ Walking Tours dataset (Venkataramanan et al. 2024), which contains hour-long vidcos, and the scenery changes constantly, hence we are guaranteed that the video frames seen most recently will be very different compared to the frames seen earlier on. We scale the videos to 224 224 pixels. We adopt ViVIT-L as baseline, and train both models using Imagenet pretrained weights. For ViViT-L, we keep all the outputs from all T lime sleps and apply temporal pooling and a 1 x convolution to get the expected shape for the reconstructed frame. For TRecViT, we simply keep the output of the last layer at time step T and reshape it to the expected shape. When evaluating on longer sequences, the ViVIT model needs to adapt the positional encoding - we use interpolation to nearest neighbour to obtain the desired length; cubic interpolation led to worse results. TRecViT can run on any sequence length without modification.
Table B includes quantitative results measured in terms of Peak Signal-to-Noise Ratio (PSNR), which is defined based on the mean squared error (MSE) between the pixel values of the original video and the re- constructed video; more quantitative and qualitative results are included in the Appendix. When evaluated on the same sequence length, ViViT-L is better than Tic as it manages to better preserve the higher
Model
Dataset
AJJ
Param (M)
MooG
DAVIS
0.687
35
VideoMAE
DAVIS
0.703
330
TRec VYT
DAVIS
0.706
128
MooG
Perception Test
0.760
46.5M
VideoMAE
Perception Test
0.761
330M
TRec
Perception Test 0.783
128M
Table 7: Performance of TRecViT compared to baselines on point tracking task on DAVIS and Perception Test datasets. All models use frozen representations evaluated using the readout head from MooG. In line with MooG, use 8 frames for the DAVIS dataset and 16 frames for the Perception Test. Metric: average Jaccard (AJ); higher is better.
12

---

Published in Transactions on Machine Learning Research (12/2025)
frequencies in the visual signal. However, when evaluated on longer sequences, ViVIT's PSNR drops signifi- cantly, showing strong artefacts, whilst TRecViT's output quality remains quite satislactory (see qualitative results in the appendix, Figures g and g We consider this to be very promising result that warrants
further invostigation into the memorisation capabilities of our model we leave this as future work.
6 Conclusion
We propose a novel causal video architecture TRecViT that alternates gated linear recurrent units (LRUs) modelling the temporal dynamics in the video with ViT blocks modelling the spatial and channel dimensions. Notably, this is the first causal architecture in the SSM video models family. The proposed model outper- forms or obtains compelitive performance compared to strong causal and non-causal baselines on supervised and self-supervised tasks, while having much smaller number of parameters and significantly reduced mem- ory footprint and FLOPs count compared to vanilla Transformer video models. Our study indicates that temporal linear recurrence combined with spatial self-attention is a natural and effective parameterisation for video modelling, given the sequential nature of videos as defined by the arrow-of-time.
Limitations and Future Work: While this study presents the first exploration into leveraging LRUs for the video domain, some key aspects necessitate further investigation. First, we plan to conduct a deeper investigation into the memorisation limitations of our architecture, particularly in needle-in-haystane prob- lems. Second, we aim to understand our model's capabilities in multi-step generative context, specifically by integrating it within vidco diffusion models. Finally, given LRUs' inspiration from continuous time sys- tems, we plan to explore its capabilities of modelling variable frame rate videos across various complex applications, including video-language tasks and Robotics.
Acknowledgments
We would like to thank Çağlar Gülgehre, Jörg Bornschein, Zhitao Gong, Daniel Zoran, and Andrew Zisserman for providing insightful feedback on this work.
References
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In 2021 IEEE/CVE /CV International Conference on Computer Vision (ICCV), pp. 6816-6826, 2021. doi: 10.1109/IICI CBL///CCCCV22222222 202..0266222200666666
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXix:1409.04 047.0473, 2014.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, arXiv preprint arXii:2405.0455 045177 2024.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video under- standing? In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Muchine Learning Research, pp. 813 824. PMLR, 18 24 Jul 2021. URL https::/ mlr, www.ppeesssvveeee https////1///brrssuslaa.a. html
James Bradbury, Roy Frostig, Peter Hawkins, Malthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang. JAX: compos- able transformations of Python+ programs, 2018. URL http://github. ctm/ coo//gooleejaa
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017a.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinctics dataset. In 2017 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4724 44333, 2017b. doi: 10.1109/CV 2000/VV..220...00
13

---

Published in Transactions on Machine Learning Research (12/2025)
Tri Dao and Albert Gu. Transformers are ssms: generalized models and efficient algorithms through struc- tured state space duality. In Proceedings of the 4 1stt International Conference on Machine Learning, 1CML'24. JMLR.org, 2024.
Soham De, Samuel L.. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Ar- naud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient language models, 2024. URL
ltops://arxiv.. orgo ha////b///2/ 2402. 19427
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens Continente, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-vid: A benchmark for tracking any point in a video. In Thirty-sizth Conference on Newral Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openrevieer htt.://eee...eeeoommmmoooooaaaaa
Carl Doersch, Yi Yang, Mel Vecerík, Dilara Gokay, Ankush Gupta, Yusuf Aytar, João Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In ICCV, pp. 10027-10088, 2023. URL
http://doooorrgg////
2023, .09233
Alexcy Dosovitskiy, Lueas Beyer, Alexander Kolcsnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openrevier. http:///www.a.ttoooooooooooooooa
Jeffrey L Elman. Finding structure in time. Cognátive Science, (4(2):179-211, 1990.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recog- nition. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6201-6210, 2019. doi: 10. N..1099ICCVV.200 2019.00630)
Yu Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jie, Yujie Zhong, and Lin Ma. Matten: Video genera- tion with mambe-attention. ArXiv, abs/ 00s224055,3025, 2024. URL https://api semanticscholar.co org/
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, He- una Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Muelter-Freitag, et al. The' something something' video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Docrsch, Yihun Du, Daniel Duckworth, David Flect, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In CVPR, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXio preprint arXiv:2312. 2.00552 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems, volume 33, pp. 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00390 2021.
Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. Xix preprint 2rXiv:2206.111 11893, 2022.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 988):1735-1780. 1997a.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 988):1335-1700. November 1997b. ISSN 0899-7667. doi: 10.1162/neco. (LL16//nnco..197..88..11 URL ftttp://doi..og/10..1111 neco
1997 1997.1197.90 9.8. 1735
Md. Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. European Conference on Computer Vision, 2022. URL https://api www.eeeannnccccooooooooooooooeee
247940203
Md Mohaiminul Islam, Mahmudul Hasan, Kishan Shamsundar Athrey, Tony Braskich, and Gedas Berta- sius. Efficient movie scene detection using state-space transformers. Tn 2023 IEFE/CVP Conference doi: 10.1109/CVPR52222.
Computer Vision and Pattern Recognition (CVPR), pp. 18749-18758, 2023.
2023.01798.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Tonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carrcira. Perceiver IO: A general architecture for struc- tured inputs & outputs. In International Conference on Learning Representations, 2022. URL https:
The JaxAuthors. Jax documentation, 2018. URL htt.t//s//aa...aaaaaaoo..///////
htmlHmodule-jax stages
Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach. Repeat after me: Transformers better than state space models at copying. arXiv preprint 002::2402200003, 2024.
Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. Motionsquecze: Neural motion feature learning for video understanding. In European Conference on Computer Vision, 2020. URL https:
//api. Slpiiiemmattcsccolla. org/CorpusID 220647295
Yann A. LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient BackProp, pp. 9 48. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642-35289-8 doi: 10.1007/ 978-3-642- __3. 3. URL https:://oo: owg/oo.......... 1007/978-71111 0422252299-888
Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding, 2024. URL
06977
Xinyu Li, Yanyi Zhang, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convolutions. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13557-13567, 2021. htttss//:://app www.sseeeaanncllseeeeeeeeeeeeeee wttp://acccccaaaooooooooorooolaa
233387838
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding, In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ringattention with blockwise transformers for near-infinite context. In The Twelfth International Conference on Learning Reeprssenaatinss, 2024. URL https:// openreview net/
nee/ferunnune forumm WSRHHlH44
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFE/CVP International Conference on Computer Vision (ICCV), pp. 10012-10022, October 2021.
Hui Lu, Albert A. Salah, and Ronald Poppe. Snakes and ladders: Two steps up for videomamba. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), 2025.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Tomás Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH 11th Annual Conference of the International Speech Communication Association, pp. 1045 1048, 2010.
Antonio Orvicto, Soham De, Caglar Gulcchre, Razvan Pascamu, and Samnel L. Smith. On the universality of linear recurrences followed by noulinear projections. arXiv preprial ar arXiv:2307.1188 118888 2023a.
Antonio Orvieto, Samuel L Smith, Albert Gu, Amushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. arXiv preprint arXiv:2303.063/
2023b.
Viorica Patraucean, Ankur Handa, and Roberto Cipolla. Spatio-temporal video autoencoder with differen- tiable memory. In 2016 International Conference on Learning Representations (ICLR) Workshop track, 2016.
Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F. Henriques. Keeping your eye on the ball: trajectory attention in video transformers. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393.
A. J. Piergiovanni, Weicheng Kuo, and Anelia Angelova. Rethinking video vits: Sparse video tubes for joint image and video learning. In IEEE/CV Conference on Compuler Vision and Pallern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 2214 2224. IEEE, 2023. doi: 10.1109/CVPR52229, 2023.00220. URL
10leg///..... 10. 11199//VPP55229..... 2023. 00220
Jordi Pont- Just,, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.0000 2017.
Viorica Paträucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Mate- jovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimezak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. ion test: diagnostic benchmark for multimodal video models. In Advances in Neural Information Processing Systems, 2023. URL httl//ttt///peereeee// /ttp:///ppee/ee/e////mmee wwwparrrieeeeeeeeouuurtteeteeeee www./eeennneeeeeeee http//ooruuii---HEEXEPPPPP
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. Tn Marina Meila. and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. $748-8763. PMLR, 2021. URL http::/pproceedigg/t
mlr,
html
Shijun Shi, Jing Xu, Lijing Lu, Zhihang Li, and Kai Hu. Self-supervised controlnet with spatio-temporal mamba for real-world video super-resolution. 2025 IEEE/CVE Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 7385-7395 2025. URL https://api. http://app...mmaatcccccccoo..ale wwwg/oorrooooee Orgg/orpusID:
279075115
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video represen- tations using Istms. In Proceedings of the 32nd International Conference International Conference Machine Learning Volume 37, ICML'15 15, pp. 843-852. JMLR.org, 2013.
Tlya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. Tn Advances in Newral Information Processing Systems, pp. 3104-3112, 2014.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VidcoMAE: Masked autocncoders are data-cfficient learners for self-supervised video pre-training. Advances in Neural Information Processing Systems, 2022.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Sjoerd van Steenkiste, Daniel Zoran, Yi Yang, Yulia Rubanova, Rishabh Kabra, Carl Doersch, Dilara Gokay, Joseph Heyward, Etienne Pot, Klaus Greff, Drew A. Hudson, Thomas Albert Keck, Joao Carreira, Alexey Dosovitskiy, Mehdi S. M. Sajjadi, and Thomas Kipf. Moving off- off-thh-grid: Scene-grounded video represen- tations. In The Tiittt- Annual Conference on Neural Information Processing Systems, 2024. URL http:://pperreviet aepeaureieee.net/fouuniiiiieee mett/forumiidrrjSPDDaaat
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki M Asano, and Yannis Avrithis. Is imagenet worth 1 video? learning strong image encoders from 1 long unlabelled video. In International Conference on Learning Representations, 2024.
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Agüera Arcas, and João Sacramento. Mesanet: Sequence modeling by locally optimal test-time training, 2025. URL
www.ii.org//bb.. wwwgg/bb//5066.. 05233
Jiewen Yang, Xingbo Dong, Liujun Liu, Chao Zhang, Jiajun Shen, and Dahai Yu. Recurring the transformer for video action recognition. In 2022 IEEE/CVS Conference on Computer Vision and Pottern Recognition doi: 0ttp://CCVPR2222222222..1.222
(CVPR), pp. 14043 14053, 2022.
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025. URL https://
openreview
Hanwei Zhang, Ying Zhu, Dan Wang, Lijun Zhang, Tianxiang Chen, Ziyang Wang, and Zi Ye. A survey on visual mamba. Applied Sciences, 14(13), 2024. ISSN 2076-3417. doi: 10.3390//pp141133633 URL
https://wwz http:::///wwmm.p......./ www./o/mmm/./ /4//227777001171///////6663
Long Zhao, Nitesh Bharadwaj Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hormung, Florian Schroff, Ming-Hsuan Yang, David A Ross, Huisheng Wang, Hartwig Adam, Mikhail Sirotenko, Ting Liu, and Boqing Gong, Video- Prism: foundational visual encoder for video understanding. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceed- ings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 60785 60811. PMLR, 21 27 Jul 2024. URL httts: mlt.sresss///pp press/ v235/ zhao24f html
mlr
Yucheng Zhao, Chong Luo, Chuanxin Tang, Dongdong Chen, Noel Codella, and Zheng-Jun Zha. Streaming video model. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 14602 14612, June 2023.
Lianghui Zhu, Beneheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bonaaectional stale space model. In Porty-firsl International Conference Machine Learning, 2024. URL
17


Reviews:
[{"review_id": "mw05xBdNHJ", "reviewer": "Reviewer_xPAN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes TRecViT, a causal video architecture for video understanding. Its core contribution is a new temporal module that alternates gated LRUs along the temporal dimension, yielding a causal architecture that can operate in streaming or robotics oriented settings. The authors present TRecViT as the first LRU based video model and provide detailed analyses showing large efficiency gains relative to ViViT. They further demonstrate versatility across tasks and training regimes, including supervised classification on SSv2 and K400, MAE style self-supervised pretraining, and point tracking. \n\nOn the positive side, the architecture is causal and efficient, achieving strong throughput while delivering strong performance with relatively few parameters compared to transformer baselines, and its ability to support video classification, video prediction, and dense point tracking indicates that it can serve as a general purpose video backbone. \n\nOn the negative side, the novelty may be perceived as largely an architectural recombination of existing components, namely ViT style spatial transformers and gated LRUs, rather than a new modeling principle. The results on Kinetics 400 are mostly competitive rather than clearly superior to strong transformer baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s main claims that the proposed time–space factorization with LRUs yields a causal video backbone, and that this backbone achieves state-of-the-art or competitive performance against both causal and non-causal baselines under comparable settings, are supported by extensive quantitative experiments on SSv2 and K400.", "audience_yn": "Yes", "audience_explanation": "Video understanding is an important and active research area, and the design of video architectures is definitely an important topic within it. This paper proposes a new architecture and validates it with extensive experiments, so its findings are very likely to be of interest to the TMLR audience.", "requested_changes": "1. Although the temporal shift operation in TSM supports an online setting, it can essentially be viewed as a channel-wise 1D convolution with fixed kernel weights, so I am not fully convinced that it should be regarded as causal model. It would be helpful if the authors could clarify this point. \n\n2. I also find the comparison between TRecViT and ViViT in terms of FLOPs and parameter counts somewhat overclaimed or potentially misleading. TRecViT appears to be designed on top of a ViViT-B, or more precisely ViT-B backbone, so it should primarily be compared against ViViT-B rather than ViViT-L. Moreover, since TRecViT introduces additional modules, it naturally ends up with more parameters than ViViT-B, which should be acknowledged. Finally, TRecViT is consistently compared against the full self-attention variant of ViViT. For fair comparison, it would be more appropriate to evaluate it against the factorized variant of ViViT (ViViT-FE Base).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SQgLTbdfPi", "reviewer": "Reviewer_KCcQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method that combines gated-LRU units for temporal modeling with ViT blocks for spatial modeling, resulting in an effective video representation learner. It also seems to be the first causal video model within the state-space–modeling framework. The extensive set of experiments demonstrates that the proposed approach is not only computationally efficient, but also outperforms several memory-intensive baselines such as ViViT. This highlights both the effectiveness and practicality of the method for large-scale video understanding.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1) The claims made in the paper are supported by clear, convincing, and comprehensive empirical evidence. The authors propose a new way of modeling videos through the combination of gated-LRU temporal modules and ViT-based spatial modeling. This design is both effective and computationally efficient, and the experiments consistently validate these claims across multiple benchmarks.\n\n2) The paper is also clearly written and well organized, making the technical contributions accessible.", "audience_yn": "Yes", "audience_explanation": "1) The findings would be of interest to the TMLR audience. The work introduces an alternative to self-attention–based video models, offering significantly reduced computational and memory costs while still achieving competitive or superior performance. This opens new research directions in video modeling, especially in the context of state-space models and causal temporal architectures.\n\n2) The method shows strong empirical performance in the self-supervised setting and even surpasses several supervised baselines on video classification tasks. This makes the proposed approach further usable or extendable for different video tasks.", "requested_changes": "1) The paper mentions that the LRUs are \"space-shared\". Does this imply that LRU1 through LRU4 are actually the same module with shared parameters? If so, it is unclear why they are annotated differently in Figure 1. Clarifying this point in the paper would be very helpful.\n\n2) The proposed architecture applies **temporal modeling first** using LRUs, followed by \"spatial modeling\" with ViT blocks. Is there a specific motivation for choosing this order? In principle, one could also apply spatial modeling first and then temporal modeling on top of the ViT-encoded representations. It would be useful for the paper to discuss why the chosen ordering is preferable.\n\n3) The introduction suggests that this is the first work to perform \"causal temporal modeling\" within the state-space–modeling framework. However, the paper would benefit from a clearer explanation of how the gated-LRU implicitly enforces causal video modeling, and how this mechanism differs from prior approaches.\n\n4) Would it be accurate to view this method as an extension of Griffin to the video domain? Although the underlying LRU building block follows the formulation proposed by De et al., the way temporal and spatial information are combined in this work goes beyond a straightforward adaptation. De et al. focus on language modeling with a Transformer-like architecture built from gated-LRUs, while this paper applies LRUs to sequences of visual tokens. It is not entirely clear whether parameters such as \"C\" behave identically when the input consists of visual rather than textual embeddings. Additional explanation, or even a small experiment, would help clarify whether the same parameterization generalizes appropriately across modalities.\n\n5) Since LRUs model stable temporal dynamics, and the current self-supervised objective focuses on reconstructing the _current_ frame, it raises the question of what modifications would be required to enable \"future-frame extrapolation\". Understanding how well the LRU dynamics extend to multi-step prediction would be highly interesting.\n\n6) Finally, it would be useful for the paper to discuss any \"limitations\" of the proposed approach. Highlighting scenarios where the method may underperform, or tasks for which LRUs may not be ideal, would help readers better understand when and where the model is most appropriate.\n\nMinor:\n1)  The text inside Figure 3 is quite small. Increasing the font size slightly would make it much easier to read.\n2)  The paper does not clearly explain which evaluation metrics are used for each task. Adding a short section summarizing the metrics would help readers understand the comparisons.\n3)  Section 5.4 refers to Figure 8, but this figure does not appear in the main paper. It seems to have been moved to the supplementary material. It would be better to include it in the main paper for clarity.", "broader_impact_concerns": "As such, the paper itself does not pose any direct technological concerns. However, video modeling techniques in general could potentially be misused, for example, for surveillance without consent. That said, in this particular case, a broader impact statement may not be strictly necessary in my opinion", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x42AxStjHZ", "reviewer": "Reviewer_GSRR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces TRecViT, a novel recurrent Transformer model for video modeling that works in both supervised or self-supervised settings. It leverages gated Linear Recurrent Units (LRUs) for temporal mixing, while employing ViT's self-attention mechanisms and MLPs for spatial and channel mixing, respectively. This TRecViT model is the only causal model within the state-space family within the video domain.  As a result, it achieves strong performance while using significantly less memory, fewer FLOPs, and reaching real-time inference speeds (~300 FPS). Overall, the model shows strong promise for efficient and scalable video applications.\n\nStrengths:\n\n1. TRecViT is a casual video model that combines SSM's recurrence (gated LRUs) with Transformer (ViT). Each framework is modeled by ViTs, while the recurrent state efficiently models temporal information. This hybrid design unifies SSM recurrence with Transformer structure, enabling to model temporal, spatial, and channel information. \n\n2. The gated LRU provides data-dependent information flow with respect to recurrent and forget gating for updating hidden state. The constant C expands the effective forget range, allowing to flexibly model short-term and long-term temporal dependencies.\n\n3. TTRecViT supports both supervised learning for label prediction and self-supervised learning via VideoMAE task for representation learning.\n\n4. The memory footprint and the number of FLOPs of TRecViT against multiple baselines.\n\n5. The paper includes extensive ablation studies that evaluate various component of the proposed method, clearly demonstrating the advantages of the architecture.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, most claims are supported by strong experiments, including baseline comparisons, ablations, and multi-task evaluations. \n\nHowever, the efficiency claims are not fully demonstrated. This paper only includes the memory and runtime analysis based on its temporal block (gated LRU) rather than the overall model, which includes the memory and time consuming ViT components.", "audience_yn": "Yes", "audience_explanation": "Yes. The paper addresses an important topic in video modeling by combining SSM's recurrence with ViT. It provides a novel way to model both temporal, spatial, channel information. The effective efficient architectures are highly relevant to TMLR’s audience.", "requested_changes": "1. Move LRU Background Out of the Method Section.  Section 3.1 discusses the The background on LRUs, which is not the contribution of this paper and not part of the proposed method. I recommend moving the LRU background to the Related Work section, so that Section 3 focuses exclusively on the proposed architecture.\n\n2. Add Clear Notation Before Referring to Figure 1. In page 3, Section 3 describes the model pipeline and refers to Figure 1, but the necessary notation (e.g., video data, temporal tube, spatial tokens) is only introduced much later in Section 3.2. Note that Figure 1 has variable notations, making readers must jump to Section 3.2 to find their meanings. Therefore, the notations are better introduced at the beginning of Section 3, maybe having a formal problem definition with variable notations. \n\n3. Figure 3 only evaluates the gated LRU module, whereas we are interested i the overall efficiency of the entire architecture. Moreover, the ViT blocks contribute substantial cost. Please provide full-model memory and FLOPs comparisons, not just temporal module  gated LRU.\n\n4. Lack of theoretical complexity analysis: The paper does not provide a theoretical analysis of the model’s computational complexity, making it difficult to clearly understand how the proposed architecture scales compared to RNNs or Transformers (e.g., ViViT).\n\n5. Missing discussion of inherited limitations. Since the model combines recurrent units with Transformer components, it should have both advantages and disadvantages. As discussed in introduction, RNNs are slow to train due to their sequential processing, while Transformers suffer from quadratic complexity in self-attention.These limitations should be explicitly discussed for TRecViT.\n\n6. The paper does not report essential architectural hyperparameters (e.g., embedding dimension, number of heads, number of layers, patch size for embedding). It only provides the model parameters.", "broader_impact_concerns": "This research focuses on developing efficient video modeling architectures. It does not directly involve sensitive data, human subjects, or high-risk deployment scenarios. It does not have broader impact concerns.", "additional_comments": "1. In introduction, it states \"However, none of the existing proposed architectures can run in a causal manner, their performance strongly depends on bidirectional operation\". Many established sequence models are designed for causal modeling. The manuscript should revise this statement to avoid over-claiming or more accurately describe the objects \"the existing proposed architectures\".\n\n2. Fix the typo in page 6 \"then we apply a 1D convolution followed by the gated LRU described in equation equation 4\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)