LinMU: Multimodal Understanding Made Linear
Authors: Hongjie Wang
Niraj Jha
Venue: Accepted by TMLR ()

Abstract: Modern Vision-Language Models (VLMs) achieve impressive performance but are limited by the quadratic complexity of self-attention, which prevents their deployment on edge devices and makes their understanding of high-resolution images and long-context videos prohibitively expensive. To address this challenge, we introduce LinMU (Linear-complexity Multimodal Understanding), a VLM design that achieves linear complexity for the language model decoder without using any quadratic-complexity modules while maintaining the performance of global-attention-based VLMs. LinMU replaces every self-attention layer in the language model decoder with an M-MATE block: a dual-branch module that combines a bidirectional state-space model for global context (Flex-MA branch) with localized Swin-style window attention (Local-Swin branch) for adjacent correlations. To transform a pre-trained VLM into the LinMU architecture, we propose a three-stage distillation framework that (i) initializes both branches with self-attention weights and trains the Flex-MA branch alone, (ii) unfreezes the Local-Swin branch and fine-tunes it jointly with the Flex-MA branch, and (iii) unfreezes the remaining blocks and fine-tunes them using LoRA adapters, while regressing on hidden states and token-level logits of the frozen VLM teacher. On MMMU, TextVQA, LongVideoBench, Video-MME, and other benchmarks, LinMU matches the performance of teacher models, yet reduces Time-To-First-Token (TTFT) by up to 2.7$\times$ and improves token throughput by up to 9.0$\times$ on minute-length videos. Ablations confirm the importance of each distillation stage and the necessity of the two branches of the M-MATE block. We also conduct distillation on various VLM backbones to validate the universality of LinMU. The proposed framework demonstrates that state-of-the-art multimodal reasoning can be achieved without quadratic attention, thus opening up avenues for long-context VLMs that can deal with high-resolution images and long videos.

Published in Transactions on Machine Learning Research (05/2026)
LinMU: Multimodal Understanding Made Linear
Hongjie Wang
henggiccoang@prrinctoo...ct
Department of Electrical and Computer Engineering Princeton University
Niraj K. Jha
jha@princeton.cor edu
Department Electrical and Computer Engineering Princeton University
Reviewed on OpenReview: https: // openreview. net/ met/ forum? id- 6BYATSNrab
Abstract
Modern Vision-Language Models (VLMs) achieve impressive performance but are limited by the quadratic complexity of self-attention, which prevents their deployment on edge devices and makes their understanding of high-resolution images and long-context videos prohibitively expensive. To address this challenge, we introduce LinMU (Linear-complextty Multimodal Understanding), a VLM design that achieves linear complexity for the language model decoder without using any quadratic-complexity modules while maintaining the per- formance of caaball-ateentoon-aaeeee VLMs. LinMU replaces every self-attention layer in the language model decoder with an M-MATE block: a dual-branch module that combines bidirectional state-space model for global context (Flex-MA MA branch) with localized Swin- style window attention (Local-Swin branch) for adjacent correlations. To transform a pre- trained VLM into the LinMU architecture, we propose a three-stage distillation framework that (i) initializes both branches with self-attention weights and trains the Flex-MA branch alone, (ii) unfreezes the Local-Swin branch and fine-tunes it jointly with the Flex-MA branch, and (ifi) unfreezes the remaining blocks and fine-tunes them using LoRA adapters, while regressing on hidden states and token-level logits of the frozen VLM teacher. On MMMU, TextVQA, LongVideoBench, Video-MME, and other benchnarks, LinMU matches the per- formance of teacher models, yet reduces Time- Co-FFist-Tooen (TTFT) by to 2.7x and improves token throughput by up to 9.0x on minute-length videos. Ablations confirm the importance of each distillation stage and the necessity of the two branches of the M-MATE block. We also conduct distillation on various VLM backbones to validate the universalityy of LinMU. The proposed framework demonstrates that state-of-the-arl muodll reason- ing can be achieved without quadratic attention, thus opening up avenues for long-context VLMs that can deal with high-rosolution images and long videos.
1
Introduction
Recent advances Linion-Language Models (VLMs) Liu et al., 2023 Li et al. 2024 Chen et al.] 2024d:
Wang et al.] 2024b Liu al. 2025) have delivered outstanding multimodal understanding performance across wide range of image- and video-centric benchmarks, often matching surpassing sully-super systems through only few-shot prompting, VLMs have been widely deployed in various domains, including but not
limited to medicine [Tanno et al. 2025 remote sensing (Kuckreja et al. 2024). robotics (Kim et al. 2024).
aulonomous driving (Jiang el al. 2025 agriculture (Arshad et al. 2025 and science education (Lu el. al.
2022). However, these achievements are accompanied by substantial computational overhead. Since global self-attention scales quadratically with the number of tokens, processing minute-length, high-resolution vidcos requires aggressive frame sampling or clusters of high-end GPUs, thus limiting the practical deployment of current stale-of-the-at models and motivating research into linear-complexity alternatives.

---

Published in Transactions on Machine Learning Research (05/2026)
Traditional VLM with Attention
LinMU with M-MATE
Quadratic Complexity O
Linear Complexity (
M-MATE replaces Attention
HIGH COST
LOW COST
Attention
M-MATE
Long Video
Long Video
"This image/video
"This image/video describes...
describes...
High-Res Image
High-Res Image
Figure 1: By replacing quadratic-complexity attention layers with our proposed M-MATE layers, LinMU
achieves linear complexity without sacrificing model performance.
In this article, we seek break the quadratic barrier for multimodal understanding. We propose LinMU (Cinear-complxxity Multimodal Understanding). framework that replaces every self-attention layer in VLM with linear-complexity module while preserving model performance, spired by the recently introduced MATE block in LinGen (Wang et al.) 2025b a), we propose the M-MATE (Multimodal-MATE) block, a lincar-cost replacement for self-attention, as the building block a multimodal Transformer. The M-MATE block consists of two complementary branches: (i) a Flex-MA brauch based on the masked bidirectional Mamba2 model (Dao Gu (024) for capturing long-range dependencies efficiently, and (ii) Local-Swin branch using 3D Swin attention (Liu al.) 2022) with fixed small window size focus short-range spatiotemporal correlations. Each M-MATE block runs in O time with respect to token count, inheriting linear scalability from the state-space model (SSM) and fixed-size window operations. By replacing self-attention with M-MATE throughout the network, LinMU can handle significantly longer contexts than traditional Transformers for a given compute budget.
shown in Fig. [ In-
Adopting fully linear architecture risks degrading accuracy. Hence, we employ a careful knowledge distilla- tion strategy transfer the capabilitics of a pretrained attention-basee teacher model LinMU. Specifically, distill from a strong VLM leacher to a LinMU student by reusing the teacher's attention weights to ini- tialize our M-MATE branches and then tne-tuning the student model in stages. We design three-stage distillation pipeline: (1) train the Flex-MA branch alone to mimic the teacher's global attention behavior, (2) train the Local-Swin branch together with the Flex-MA branch to recover local correlation modeling, and (3) jointly fine-tune the entire M-MATTE-augmented model (with other backbone layers tuned via low-rank adapters) to close any remaining performance gap. During distillation, we optimize combination of token- level and sequence-level losses, aligning the student's predictions with the teacher's outputs (in the spirit of cross-entropy with teacher soft targets and using teacher-generated sequences pseudo-labels) (Wang) et al.) 2024a). We also include hidden feature alignment loss in the distillation objective to enable faster convergence. This staged distillation framework ensures smooth transition from quadratic-complexity at- tention to linear-come complexxyy M-MATE blocks without sacrificing understanding capability. We summarize our contributions as follows:
We present LinMU, the first multimodal (vision-language) model that achieves linear complexity in imput length by replacing self-attention with the M-MATE block without sacrificing model perfor- mance. LinMU enables long-context video and image understanding (potentially minutes of vidco) on modest hardware, providing much better scalability than the standard VLMs.
We develop a multi-stage distillation framework transfer knowledge from standard Transformer- based VLM into linear architecture. By reusing the teacher's projection weights to initialize
2

---

Published in Transactions on Machine Learning Research (05/2026)
masked bidirectional Mamba2 model (Flex-MA branch) and 3D Swin-attention (Local-Swin branch), and by employing carefully designed distillation losses, we preserve the accuracy of the original teacher model to great extent.
We demonstrate, using challenging image and video understanding benchmarks, that LinMU achieves performance on par with the teacher models (i.e., "VILA-&B-Viddr and Qwen2.5- 1ren2.5--VL7BB. "L--B-Instrucd while significantly improving efficiency. In our experiments, LinMU maintains competitive perfor- mance on tasks like MMMU, LongVideo
and Video-M MME, and enables markedly faster in- ference terms of TTFT and token throughput, thus validating linear scalability. We also present ablation studies that quantify the impact of each distillation stage and loss term, and confirm the importance of M-MATE components (Flex-MA and Local-Swin branches) in maintaining teacher performance.
Overall, LinMU illustrates that can make multimodal Transformers linear without compromising their remarkable understanding capabilities. This opens the door to scaling up context lengths (e.g., using more video frames or higheroresolution features) and deploying VLMs in real-time or resource-consrrained settings, which was previously infeasible due to quadratic costs.
We organize the remainder of this article as follows. Section 2 surveys recent progress on VLMs and prior efforts to make them more efficient. Section 8 introduces the LinMU framework, detailing the M-MATE block and three-stage distillation pipeline. Section presents comprehensive experiments to evaluate the performance and efficiency of LinMU, along with ablation studies that target the distillation stages, loss terms, and block components. Finally, Section concludes the article and outlines directions for future work.
2 Related Work
In this section, we situate our contribution within three complementary lines of research. First, we review the evolution of modern VLMs, with emphasis on the higl-capacity Transformer-based architectures (Sec- tion 2.1 Second, we discuss methods that improve the efficiency of VLMs primarily through token pruning and structural compression while keeping the underlying attention backbone unchanged (Section 2.2 Third, survey linear-complexity alternatives to attention, including state-space and recurrent architectures and their deployment on VLMs, which are most closely related to our proposed framework (Section 2.3
2.1 Large LisionnLanguage Models
Recent years have seen rapid progress in large VLMs, which couple a high-capacity language backbono with a visual encoder through a multimodal projector and stacks of Transformer atlention layers. Early models, such as Flamingo 4Alayrac et al., 2022). BLIP-2 [Li et al. 2023a), MiniGPT-4 (Zhu et al. 2023) and LLaVA (Liu et al. 2023), established the now-standard recipe of using a frozen or lightly-tuned vision encoder to produce visual tokens that are injected into frozen or instruction-tunn large language model (LLM).
Contemporary open-source models push this paradigm to higher capacity and broader modalities. Qwen2-VL (Wang et al. 2024b) and the more recent Qwen2.5-VL (Bai et al.) 2023) series extend the Qwen language backbone with a streamlined encoder in the Visual Transformer (ViT) style and improved visual projector, supporting images, documents, and vidcos at native resolution and long contexts. Qwen2.5-VI. achieves competitive performance on diverse benchmarks, such as MMMU (Yue et al.) (022)) and Math- Vista (Lu et al. 2023). while remaining fully Transformer-basee with quadratic attention in the LLM decoder.
The LLa VA family follows a similar design but is trained almost entirely from open data. LLaVA-NeXT Liu 2024b) improves reasoning, optical character recognition (OCR), and world knowledge over LLaVA-1. (Liu et al. (024a) and extends to vidco understanding. L.LVA-OOeVisiion (Li et al., 2024) further pushes performance by training on native-resolution images and providing a complete open training pipeline.
et al.
3

---

Published in Transactions on Machine Learning Research (05/2026)
Parallel to these, NVILA (Lu et al, 2025 introduces "scalo-then-compress paradigm designed to opti- mize the efficieney-accuracy frontier. Unlike traditional models that trade off resolution for speed, NVTLA first scales up spatial and temporal inputs to capture fine-grained details and then compresses these visual tokens to maintain high throughput during inference. Meta's Llama-3.22 3.2- Vision models (Dubey et al.) 2024) are optimized for high-accuracy image reasoning and document understanding with long context windows. InternVT-2.55 (Chen et al. 2024c) optimizes the visual encoder (InternViT) specifically for semantic align- ment, which can reduce the number of visual tokens needed to convey the same amount of information compared to vanilla CLIP encoders.
As discussed above, recent VLMs tend to explore unified architectures that handle images, documents, and videos through scaling of parameter-count and context length. However, across almost all of these models, the multimodal backbone still built from standard self- and cross-attention Transformers. Hence, both computation and memory scale quadratically with the total number of tokens, which becomes a growing bottleneck as models move to high-resolution, multi-image, and vidco inputs.
2.2 Efficient Vision-Language Models with Attention
To mitigate the cost. of quadratic attention without changing the core architecture, a large body of work focuses on improving the efficiency of VLMs through token reduction, structural pruning, and other model-compression techniques. These methods can be categorized into strategies that operate on the token sequence (e.g., pruning and compression)) network structure (e.g., layer pruning, distillation), and at system level (e.g,, quantization, KV-cache optimizations).
Visual token pruning and compression. A dominant line of work observes that visual tokens are heavily redundant relative to text tokens and seeks to reduce the number of vision tokens that reach the LLM. Fast (Chen et al.] 2024b) proposes a plug-and-play inference procedure that discards roughly half of the image tokens after the second layer, yielding substantial speedups while incurring minimal accuracy loss. Eerruction-Guider Visual Token Pruning (Huang et al. 2024) introduccs a group-wise token pruming module based on allention rollout, using the instruction to guide which vision tokens are pruned both within the vision encoder and the LLM. LLaVA-PruMergee (Shang et al. 2025 proposes an adaptive reduction of visual tokens using the sparsity of class-token attention. VLatt (Chen et al. 2024a) introduces a Visual Context Compressor and a new training scheme, demonstrating that up to 70% of visual tokens can be removed during testing while only incurring minor degradation. SparseVT. (Zhang et al.) 2024a) proposes text-guided, training-free free pruning mechanism that ranks visual tokens using self-attention with selected text tokens and adaptively determines sparsífication ratios per layer. Related works, such as FEATHER (Endo lt al.] (2025) and CoViPAL (Tang et al.) 2025), refine these ideas by considering layer-wise token importance and contextual dependencics. PruncVid (Huang et al. 2025) and LA-SiSScrso (Sun et al. 2025) further deploy the token pruning idea to Video-centric VLMs.
Architectural compression. Beyond token-level sparsification, several works compress VLMs the architecture level. Short-LVLM (Ma et al. 2025) proposes a new pruning framework that identifics re- dundant layers while compensating for cross-modal feature changes, providing training-fres compression that is model-agnostic and compatible with token-level methods. E.CoFT.P (Sung et al., 2023) proposes a coarse-to-fine layer-wise pruning scheme for large VLMs that first computes a global importance score via zeroth-order gradient approximations to assign adaptive sparsity ratios across layers, and then performs local unstructured pruning within each layer. Complementary to pruning, various works distill the visual reprosentation space of large VLM teachers (o.g. CLIP ViT-L/14) into lightweight students using small- or mid-scale datasets, with a particular focus on preserving relative geometry in the joint vision language embedding space to maintain open- vocabulary out-of distribbttiiioon generalization (Li et al. 2023b
Quantization and key-value (KV) cache optimization. Another line of efficiency work targets numer- ical precision and cache storage rather than architecture. NVILA employs quantization in both the vision encoder and the LLM backbone to reduce TTFT and improve token throughput. MBQ
-allance Quantization) (Li et al.) 20255) observes that vision and language tokens in large VLMs have very differ- ent sensitivity to quantization noise and thus uses gradient- sensitivity indicators for each modality. ReKV (Di et al. 2025) proposes training-free approach that enables efficient streaming video question-
4

---

Published in Transactions on Machine Learning Research (05/2026)
answering via in-context video window KV-cache retrieval. However, such pure window-based perception
pattern for inputs incurs performance degradation.
In summary, most of the current efficiency efforts for VLMs retain the Transformer attention backbone. While they dramatically reduce constant factors, the underlying complexity with respect to the remaining tokens remains quadratic, and the attention mechanism itself is unchanged. Our work complementary: Instead of compressing the token sequence or pruning layers on top of attention, we investigate a linear-complexity module that can directly replace attention layers in existing VLM backbones.
2.3 Limear-Complexity Alternatives to Attention in VLMs
To address the quadratic complexity of attention, one line of work explores sparse or factorized attention to handle longer inputs. In vision, local window attention and factorized space-time attention have been used in video Transformers, such as Vision Longformer (Zhang et al., 2021) and Video Swin Transformer {Liu et al. 2022). ; to reduce the cost of global attention. These approaches limit each token's receptive field (e.g., attending only within a local patch or frame window), which yields linear or near-linear complexity per layer but often requires adding some global layers not hurt performance. For example, Video Swin Transformer achieves linear scaling by restricting attention non-overlapping windows, then shifting windows between layers to propagate information. Such designs be very effective, but purely local attention cannot cap- ture long-range dependencies unless some global mechanism is present. For natural language processing, BigBird (Zahoer et al., 2020) and Longformer (Beltagy et al. (020)) similarly use sparse attention patterns (sliding windows, random or global tokens) to reach approximate linear complexity, with trade-ofls in com- pleteness of context modeling. In contrast, LinMU's M-MATE block provides a global sequence mixing through the MA-Flex branch and local neighbor focus through the Local-Swin branch, together achieving full coverage of dependencies with linear cost.
Another line of work explores different computational patterns to achieve linear complexity, including linear attention (Katharopoulos et al.) 2020). SSMs (Gu et al. 2021 Gu & Dao 2024), and recurrent architec- tures (Pngg et al., [023] Sun et al. 2023). They have been explored for language modeling and other modalities, but are hardly competitive with Transformers at scale without. involving quadratic-complexity layers.
Specifically, Mamba (Gu & Dao 2024 introduces a selective SSM architecture that conditions the SSM parameters on the input and implements hardware-aware recurrent scan, achieving linear complexity in soquence length and significantly higher throughput than Transformers for long sequences. Its successor Mamba2 (Dao & Gu, 2024) has the attention format and is compatible with the efficient attention ker- nels, such as FlashAttention (Dao et al.) 2022) and XFormers (Lefaudeux et al. 2022). Building on this, VMamba (Liu ct al.] 2024c) adapts Mamba to visual data by introducing Visual State-Space blocks and a 2D Selective Scan module. VMamba traverses images along multiple scan directions to capture global context. with linear complexity. Subsequent works, such as Multi-Scale VMamba (Shi et al., 2024) and Spatial-Mamba (Xiao al.) 2025), refine scanning strategies and multi-scale context aggregation.
Several recent works apply linear-time architectures directly to multimodal learning and VLMs. VT-Mamba (Qaao et al. 2024) replaces the Transformer LLM with a pretrained Mamba. language model and introduces a MudtiModal Connector with a Vision Selective Scan module, demonstrating competitive performance with Transformer-basee multimodal LLMs while enjoying linear scaling in sequence length and faster inference. VisualRWKV (Hou et al. 2025b) extends the RWKV linear recurrent neural network to multimodal learning, proposing data-dependent recurrence, a "sandwich" visual prompting scheme, and a 2D image scanning mechanism to handle visual tokens.
To obtain better performance, hybrid architectures that interleave quadratic- attention and linear- complexity SSM layers are beginning to appear in both text-only and multimodal settings. Nemotron-Nano-2 models (Basant el al. 2025) replace some of the self-attention layers of a Transformer with Mamba-2 lay- ers, achieving higher throughput on long-context reasoning while maintaining Transformer-leeel accuracy. Nemotron-Nano-2 VL (Deshmukh et al. 2025) extends this hybrid Transformer-Mambey design to vision- language tasks, such as video understanding and document intelligence, demonstrating that linear-time modules can be integrated into competitive multimodal systems.
sctting
5

---

Published in Transactions on Machine Learning Research (05/2026)
As discussed above, most existing tomear-comllxxtty VLMs typically take one of two approaches: (i) replacing attention layers of the entire backbone with linear-complexity alternative, e-g., Mamba or RWKV, which often lags behind Transformer-basee based VLMs on complex vision understanding tasks due to the adjacency preservation issuc (see Sec. 3.10; ; or (ii) hybrid architectures that interleave attention layers with linear- complexity layers, which still sullers from overall quadratic complexity. By contrast, LinMU replaces all attention layers in Transformer-basse
VLMs without compromising performance, especially on complex vision tasks. By building on the M-MATE block design and a tailored distillation schedule, we show that such replacement is indeed possible for pre-trained VLMs complex vision-language understanding tasks.
3 Methodology
In this section, we first describe our proposed architecture, LinMU, and then illustrate the distillation framework to transform a global-attention- basod pro-trained VLM into LinMU.
3.1 The LinMU Architecture
Despite the efficiency advantages of linear sequence layers, as in Mamba and RWKV, known challenge is applying them to large-scale vision tasks. Given the relatively better hardware efficiency of Mamba models among the lincar sequence layers (Dao & Gu 2024), we focus on Mamba in the rest of this article. When Mamba is led image or video tokens (lattened from 2D/3D), it struggles with the adjacency preservation issue: Spatially and temporally adjacent tokens become non- noi-beigh in the flattened sequence, causing the model's performance to degrade due to the inherent correlation precision decay over long distances (Wang 2025b). Prior works attempt to alleviate this problem through better token ordering (Hu et al.) 2024 et al. He et al. 2024) or by mixing some regular attention layers with SSM layers (Basant. et al., 2025). However, these solutions either do not fully fix the adjacency preservation issue at large scale or reintroduce quadratic components.
Inspired by the MATE block in LinGen (Wang et al. in VLMs with an M-MATE block. As shown in Fig. 2 LinMU is an auto-regressive VLM composed of two parts: Vision Encoder and Token Processor (i.c., LM Decoder). Because the implementation of the Vision Encoder could be quite different across variant VLM backbones, to enable generalization
2025b., we propose replacing the self-attention block
leave the Vision Encoder unchanged and replace attention layers in the Token Processor with our proposed M-MATE blocks.
The proposed M-MATE block builds on two branches: (I) The Flex-MA branch that uses bidirectional Mamba2 for its linear complexity and throughput benefits. By applying a mask in the reverse direction, it flexibly achicves causal correlations and global correlations for text tokens and vision tokens, respectively. We mitigate adjacency preservation via Rolary Major-Scan (RMS) proposed in LinGen. (II) The Local- Swin branch that uses 3D local Swin attention to directly tackle the adjacent correlations immediate neighbor patches. The windows are fixed at small size because it only needs to model adjacent correlations, which keeps the complexity of this branch linear and makes it very efficient. In this way, LinMU inherits the best of both worlds: global linear mixing and local precise attention.
modeling
3.1.1 FA Branch
This branch is built on Mamba2 (Dao & Gu 2024), an SSM designed as a linear-complextty alternative to attention. We choose Mamba2 for its linear-time complexity and its Transformer-fomnat parameterization. The latter makes it compatible with modern efficient attention kernels, such as Fltstttnontionn and xFormers. In addition, this Transformer format makes it easier to reuse altention weights in pre-trained models for initialization.
In this branch, the input token sequence (c.g., sequence of visual and textual embeddings) is first rearranged by an RMS operation (Wang el al. 2025b). RMS permutes the lokens in a way that enables spatially or temporally neighboring vision tokens to remain closer in the sequence (mitigating the adjacency preservation issue of vanilla Mamba). The main difference between RMS and other existing special scan methods is that it achieves this with only tensor reshaping operations and much more friendly to hardware, thus is much more efficient and incurs much less extra latency WWag et al. 2025b). RMS only changes the order of vision
6

---

Published in Transactions on Machine Learning Research (05/2026)
FFN
X N
Image/ Videos
Reshape
Rev-RMS
Visian Encoder
Rev
is the in this video doing?
Mamba2
Mamba2 Rev
vision tokens
text tokens
Swin
Rev
LM Decoder
Vision Mask
output tokens
Rotary Major Scan (RMS)
Reshapo
The woman in this video is drinking coffee...
Flex-MA
Local-Swin
M-MATE block
Figure 2: The overview of LinMU, in which the Vision Encoder is untouched but all the attention layers in the Language Model (LM) Decoder replaced with proposed M-MATE blocks. Residual/gate connections are not shown in this figure for brevity. The M-MATE block contains two parallel branches: (1) an Flex-MA branch built around bidirectional Mamba2 augmented with reverse direction token mask and RMS loken rearrangement, and (2) Local-Swin branch that implements a local 3D Swin Attention with a fixed window size. The Flex-MA branch provides efficient long-range sequence mixing with linear complexity, while the Local-Swin branch focuses on adjacent spatial ttemppal correlations preserve local consistency. Both branch outputs summed produce the M-MATE block output that replaces the original self-attention output. Other parts of the LMM Decoder (e-g., multi-layer perceptron, layer norms, etc.) remain unchanged.
tokens. Assume the vision token tensor shape in the latent space T H x W (where T 1 if it represents image), the index of token T(t [v] [xr] is always for image) in the rearranged 1D sequence in the I-th layer is given by
t.(H +)+ W +x,
if
mod
6. 6.( W) x H +
if mod
y. (T- W)+ +.
if mod
H) H)
if mod 4 4=
Four different scanning patterns alternate in consecutive layers. When dealing with images, it reduces to two different patterns from four. The permuled sequence is then fed to a bidirectional Mamba2 layer, which computes an output sequence of the same length. Neither like the original (causal) Mamba that was designed for unidirectional language modeling, nor the pure bidircctional version that produces full "attention-like" correlation map for vision tasks, we employ a vision-token mask My in the reverse direction, flexibly producing causal maps for text tokens and full maps for vision tokens, shown in Fig. 3

---

Published in Transactions on Machine Learning Research (05/2026)
Receptive Field ("Attention Map")
Reverse Direction
Vision
Vision
Text
Normal Direction
Vision
Native Mamba2
Text
Vision
Bidirectional Mamba2
Masked Flexible Bidirectional Mamba2
Figure 3: The receptive field comparison between native Mamba2 (suitable for text-only tasks), bidircctional Mamba2 (suitable for vision-only lasks), and our proposed masked flexible bidirectional Mamba2 (suitable for multimodal tasks).
The masked bidirectional Mamba2 performs global, linar-time sequence mixing via recurrent state updates run forward and backward. It does not compute explicit token-to-token similarity matrices but attains a full-context receptive field suitable for multimodal understanding tasks. Importantly, the computation cost of Mamba2 scales as O(Nd)) (where N the number of tokens and is the dimension of embeddings); which is linear in N (Dao & Gu 2024). The outputs of Mamba2 are finally inverse- permuted back (to undo RMS) and passed on. Overall, the Flex-MA branch is responsible for capturing global dependencies (even across hundreds of thousands of tokens) al low cost, albeit in a somewhat blurred fashion due to the limited precision of SSMs for very long ranges.
Let u, Rdd be the token with index the sequence after RMS permutation. A selective SSM layer defines input-conditioned parameters (A,, ; B,, C₁) and updates as follows:
h = A,h + B.u, yr = C,
(1)
h A + B. @ uu), yf C.h,
(2)
Note that My vision token mask that retains only vision tokens. We fuse the directions with a learned gate:
y, Flex W. W.grr
gr 2 h + g: 10h h
3.1.2 Local-Swin Branch
In parallel with the Flex-MA branch, each M-MATE block includes a local 3D swin attention module, as shown in Fig. 4 It focuses on vision tokens and masks out text tokens. This is form of multi-head attention restricted to local neighborhoods spatially and temporally, inspired by Swin Transformer's shifted window approach (Liu et al. 2022). In practice, given the tokens arranged in 3D index (frame height x width for video, or 1 x height x width for images), the Local-Swin branch divides them into small windows (e.g., 16 x 4 x for long videos) and computes standard aetetion only among tokens within each window. Because window size is fixed and very small (we use 16 x 4 x 4 volumes, for example), this attention is lincar in N (cach token attends to constant number of neighbors). The windows are shifted in consccutive layers (similar Swin) allow cross-window connections over multiple M-MATE blocks. The Local-Swin branch thus excels at modeling short-range and medium-range correlations: Tt ensures that immediately

---

Published in Transactions on Machine Learning Research (05/2026)
Division
Combination
Normal Windows from Spatial View (No Need for te-Combination)
Normal Windows from Temporal View (No Need for Re-Combination)
Normal Attention Windows
A
C
C
Shifted and Re-Combined Windows from Spatial View
Shifted and Re- Ce-Comi Windows from Temporal View
Shifted Attention Windows
Window division, shifting, and re-combination in the local 3D Swin attention. After window shifting, tokens around the image/video edge are combined to form new attention windows, ensuring the total number windows remains constant.
Figure 4:
adjacent pixels/patches patches and nearby frames are consistently processed, something the Flex-MA branch may not guarantee due to token rearrangement to ID) sequences and long-range decay.
In the Local-Swin branch, vision tokens are reshaped Fxxxxxxd For window size (T, s, and stride equal to window (non-overlap), we compute multi-head attention locally inside each (r 8) block follows:
QK Van
Allnwin (Q. K, V) softmax
V.
In alternating layers, we use a shift of (lT/2) /2]. [s/2]. [s/2]) enable cross- cross-windes window links. The per-layer cost is O(N -2s ie., linear in N for fixed local window size.
The output of this branch is fused with the Flex-MA branch output with learned weight to produce the final M-MATE block output, shown in Alg. D By design, the M-MATE output has the same shape a standard attention output. Hence, it can be fed into the subsequent feed-forward network (FFN) and residual connections exactly as the original Transformer layer.
Crucially, both branches of the M-MATE block operate parallel, and their combined cost still linear. The cost of the masked bidirectional Mamba2 is O(N) in sequence length, and the cost of the local 3D Swin attention is also O(N) (with the fixed window size). By replacing all L self-attention layers in Transformer with the proposed M-MATE block, the overall complexity w.r.t. token number becomes O N), instead of O(L N2), enabling dramatic speedups for long sequences.
3.2 Distilling Pre-trained VLMs to LinMU
Although our proposed architect ure ure achieves linear complexity, directly training a VLM with the new archi- tecture from scratch would be very costly. We, therefore, propose a progressive distillation framework that converts a pretrained Transformerbbasod VLM (teacher) into LinMU (student) by replacing each quadratic self-attention layer with an M-MATE block of identical iaput/outpig size, as described in Alg. Fig. illustrates the overall replacement and the three-stage distillation schedule.

---

Published in Transactions on Machine Learning Research (05/2026)
Algorithm M-MATE block forward
Require: tokens (zri==1 vision token mask My
1: Flex-MA branch
{ur} + (M))((z))
[vi <- (
[yblex] <- (ev-RMM(((v))
Local-Swin branch
6: (vli)] +- My O (z:})
b alternately shifted fixed-size windows D padding from vision tokens to the full sequence
7: [y] + Pad((v(=))
8: return y, = (A Flox + )
> A is learnable
Teacher Model
Student Stage
Student Stage 2
Student Stage 3
Leoks segi L sup
XN
XN
x N
x N
FFN
FFN
FFN
FFN
Lhid
Flex-MA branch
Local-Swin branch %
Flex-MA branch
Local-Swin branch
Flex-MA branch
Local-Swin branch
Self-Attention
Figure 5: Our proposed three-stage distillation pipeline to replace attention layers in pre-trained VLMs with linear-compleiity M-MATE blocks.
3.2.1 Weight Reuse Initialization
Simply swapping attention for randomly initialized M-MATE blocks degrades accuracy. Following the weight- reuse spirit in MambaInLlama (Wang et al. 2024a) and LinGen-Uni (Wang et al. 2025a). initialize student M-MATE blocks from the teacher attention weights wherever possible.
K: W e Rdxd and output projection W6 Raxd For the Flex-MA (masked bidirectional Mamba2) branch,
Let the teacher attention layer & have (fatteend) projection matrices Q W
We W WB < W W W/.
W W
(4)
where (W&, WT, Wx) the dominant lincar projections in Mamba2 (accounting for over 90% parameters). while the state-transition parameters (e.g., A) use standard initialization and gales use identity initialization.
We initialize the Local-Swin branch as the teacher attention restricted to fixed local windows over vision tokens as follows:
(W2 Swwin w W& K.Swin V.Swia
we
+ (W&. W Wt, W(),
(5)
while enforcing the Swin window mask. Intuitively, some the teacher's attention heads are likely specialized local structure (as commonly observed in carly Transformer layers); hence, the Local-Swin branch can inherit them. This weight reuse gives the student a head start, often starting in regime not too far from the teacher's function.
10

---

Published in Transactions on Machine Learning Research (05/2026)
Algorithm 2 Progressive Distillation from Transformer VLM to LinMU
Require: Frozen teacher fr (self-attention); student fs (M-MATE); dataset D; temperature т; steps (S1, S2, Sy) for each stage
Require: Loss weights Ahid, Atole Asoq: Asup
Ensure: Distilled LinMU fs
1: Weight reuse init: for cach replaced layer L, apply Eq. parameters in the standard way.
4
and Eq. 5
initialize remaining Mamba2
2: Freeze all teacher params; freeze student backbone and vision
Stage 1 (Flex-MA only): unfreeze Oplexi disable/freeze Local-Swin.
for == to S1 do
5: Sample minibatch (x, ~ D
6: ) + [T(x)
D. no grad
7: (h (zs,)) + fs(x)
Compute Liid by Eq. @ and Look by Eq.
( AndaCaid Crok
D Eq. 10
10: Update Oriex with (L
11: end for
12: Stage (Flex-MA + Local-Swin): unfreeze Oswini keep backbone frozen.
13: for s
S2 do Sample minibatch yst) D
14:
15:
(hT), (zr, + fr(x)
no grad
16:
(be), (zs.e) + fs(x)
17:
Compute Luid (Eq. 6), Liok (Eq. 7
18: ( + Apidhhid + MokkStol
D Eq.
19: Update (OFlex, Oswin) with (2
20: end for
21: Stage 3 (LoRA fine-tune): insert LoRA in backbone; unfreeze OLORA and all M-MATE params.
22: for s= to S3 do
23:
Sample minibatch ytt D
24:
{2T,a} + fr(x) and ý f- Decode(fr. (fr,
> no grad
25:
(zs,e} t fs(x)
26:
Compute Lrok (Eq. 7) Lung (Eq. 80, and Loup (if y available)
27:
((3) + AtokCook Awaaa fren + Aoup Csup
Eq
13
28:
Update (OM-MATE, (LORA) with (3)
29:
end for
30:
return fs
3.2.2
Three-stage progressive distillation
We keep the teacher fT frozen and train the student fs via three stages that progressively increase trainable capacity. Let A denote the set of replaced layers.
Common loss terms. We use (i) layer-wise main path matching between the output of the M-MATE block and the replaced attention layer:
Chid
h 1T.1
(6)
A
where (bg and (b are the output of the student M-MATE block and the leacher attention module at the E-th layer, respectively. (ii) token-level knowledge distillation (KD) with temperature T:
72
T KL 11 pT.)) pf. = softmax(2rs/7), p5, softmax(zs./T),
Look
Σ
(7)
T
11

---

Published in Transactions on Machine Learning Research (05/2026)
where z,, are vocabulary logits at generation step t and T is the output length. We use this loss to keep the student aligned end-task predictions with the teacher model. (iii) sequence-level KD using a teacher- decoded pseudo target y = Decode(fr,x):
T Σ
Leeq
log ps (Dt
x. Fc))
(8)
T
where ie the output of the student model. This two-level KD (Ltok and Leeq, which are often denoted word-level and sequence-level loss, respectively) has been shown to be effective for distilling LLMs, as it balances imitating the probability distribution with matching the single best sequence. (iv) ground-truth task loss when the ground truth y is available:
Suu = Lrask (0,ak(0,, yot
(9)
Stage 1: Flex-MA-only distillation. ters Oftex and freeze the Local-Swin branch. The objective emphasizes module-level imitation while retaining end-lask aligument:
We freeze the entire student except the Flex-MA branch parame-
min ( Apid Lind + Atok Look
(10)
Spiex
Matching (b \N forces the Flex-MA branch to approximate the teacher attention effect each replaced layer, providing strong and stable signal even when only small subset of parameters is trainable.
Stage 2: Joint distillation of Flex-MA and Local-Swin. We unfreeze Local-Swin parameters Oswin and train both branches jointly while koeping the rest of the backbone frozen. We initialize the Local-Swin branch as aforementioned and allow it to learn to model the finer local details that the Flex-MA branch might have missed due to the adjacency preservation issue. The Stage- objective is:
min 0Flcx GSwin
(2) = Auid Lhid + Aenk Lrtk
(11)
Because the Local-Swin branch starts from the teacher's weights and the Flex-MA branch has been trained in Stage 1, the joint optinization in Stage 2 converges quickly. We find that if we atlempt to train both branches from the start, the Flex-MA branch is less stable and harder to converge; by staging it, each branch Iearns its rolc: Flex-MA first learns global context, then Local-Swin fills in local consistency. By the end of Stage 2, the student's M-MATE blocks can produce outputs very close to the teacher's 's attention outputs for each layer.
Stage LoRA fine-tuning of the backbone. the final slage, we unfreeze the rest the student model (the vision encoder and the corresponding projector are still fixed). However, instead of full fine- tuning (which could be too costly and risk distorting the pretrained weights), we apply low-rank adaptation (LoRA) (Hu et al. 2022) to these layers. For frozen weight. W |ddou Xdin LoRA parameterizes
W W + AW,
AW BA, A B R.dout
(12)
and only (A, B) are trainable. We train all M-MATE parameters plus LoRA parameters & of FFNs the backbone with an objective that shifts emphasis output-level alignment and (optionally) ground-truth supervision:
min SXLMATE.PLORL
(3) Atok Liok + >ssg Laog + Asuu Souu
(13)
Note that at this stage, we incorporate Leeq and Loup for better output distribution aligument but exclude Lid because other blocks in the student backbone are also boing trainod. This stage yields the final LinMU model. Notably, because LoRA updates are lightweight, the original knowlodge in the backbone is largely preserved.
12

---

Published in Transactions on Machine Learning Research (05/2026)
We set Ahid Aeok Aseg Asup = 0.5 in all the three stages mentioned above. In summary, weight reuse provides a strong functional prior, while the three-stage schedule stabilizes optimization: Flex-MA first learns to reproduce long-range/ /oonggg//ottttuall effects, Local-Swin then specializes to local visual structure, and LoRA fine-tuning adapts the remaining backbone to the new lincar-complextty blocks with minimal parameter updates. Through this distillation pipeline, LinMU learns to laithfully emulate the original model's attention-driven reasoning, but using only linear-complexity blocks. The outcome a model that, during inference, requires no quadratic attention computation and is, therefore, much more efficient on long sequences (e.g., dealing with high-resolution images or long videos).
4 Experiments
In this section, we first introduce the general experimental setup in Sec. 4.1 including backbones, datasets, and benchmarks that we use to deploy our framework, perform distillation, and evaluate performance. Then in Sec. 4.2 we compare the performance and efficiency of our proposed framework with baseline models, demonstrating that LinMU achieves a better balance between accuracy and efficiency. Finally, we provide ablation experimental results to validate the effectiveness of the techniques proposed and design choices made in our framework in Sec 4.3
4.1 Experimental Setup
Backbones. For most experiments, we use Video-SB as the teacher model to deploy our proposed distillation framework and achieve linear complexity. It is an 8B-parameter VLM that exhibits competitive performance on many image and video understanding benchmarks. For image benchmarks, we use NVILA- 8B the teacher model instead; it is checkpoint before the video-instruction tuning stage and focuses more image tasks. To demonstrate the generalization ability of LinMU, we further perform distillation on Qwen2.5- 1-55-L--BB-Instrree (Bai et al.] 2025), which has strong performance across various perception tasks, including but not limited to long document understanding, video grounding, and OCR.
Datasets. Referring to the image and video instruction-tuniin datasets of NVILA, we involve (1) the single-image and multi-image subsets of OneVision-1.6M in LLaVA-OneVisionn (Li al. 2024). which in- clude around 1M samples; (2) ELaVa-Videe 178k (Zhang et al., 2024b), which provides around 178K videos; (3) the training set of ActivityNet-QA (excluding the test sct), which includes around 60K question-answer (QA) pairs on 6K videos as our distillation dataset. ActivityNet-QA is only involved in Stage 3. Note that Qwen2.5-Ve VL-7B-L---sttt standing. Complete distillation of it requires building a comprehensive dataset including all its capabilities, while our datasets here focus more on the domain that we target (i.c., multimodal undorstanding). Distilling 2wen2.5--V--BB-Insse
performs well across many downstream tasks, including long document under-
a proof of concept. that our proposed framework can be deployed on various VLM backbones with a relatively standard model architecture design.
suite of image and video understanding tasks to verify that LinMU achieves comparable performance to the teacher model while gaining efficiency. (1) MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning): A multimodal reason- ing benchmark (Yue et al.) 2024) spanning many academic subjects (e.g., science, math, engineering), where models answer questions that require understanding images like diagrams, charts, tables, and fig- ures alongside text. (2) VextVQA: A visual QA benchmark (Singh et al. 2019) focused on reading and reasoning about text in images (scene text), not just recognizing objects. (3) ActivityNet-QAA: A video QA benchmark (Yu et al.) 2019) built Activity videos, testing whether model can an- swer natural-language questions about actions/events in videos, often requiring temporal understanding. (4) LongVidooBench: A benchmark (Wu et al. 2024) designed for long-form vidco understanding, em- phasizing long-context temporal reasoning (Iracking events across minutes-long videos rather than short clips). (5) MLVU (Multi-task Long Video Understanding: A multi-task long-video understanding bench- mark (Zhou et al., 2025) that evaluates modcl's ability to comprehend lengthy vidcos across tasks like QA and other video- language understanding settings, stressing temporal reasoning over extended context. (6) Video-MME. (Video Multi-Modal Evaluation): A comprehensive evaluation suite FFu et al. 2025 for video-
Benchmarks. We evaluate LinMU and basclines
13

---

Published in Transactions on Machine Learning Research (05/2026)
Table 1: Performance comparison between the student LinMU model, the teacher NVILA model, and other baseline modcls. Note that GPT-4o is strong closed-source bascline for reference.
MMMU
TextVQA
ActivityNet-QQA
LongVideoBench val
MLVU m-aV
Video-MME
Models
Size
pro
acc.
score
test
w/o sub.
w/sub.
Transformer- VLMs
CPT-40
64.7
51.9
77.4
61.9
66.7
66.7
64.6
71.9
77.2
LLaVA-OV
7B
42.8
24.1
78.3
56.6
56.5
64.7
58.2
61.5
Qwen2-VL
8B
46.6
30.5
84.3
55.6
56.8
65.5
63.3
69.0
InternVL2
8B
42.6
29.0
77.4
54.6
64.0
56.3
59.3
Linear- Complezity Quodratic Lineor Hybrid VLMs
VL-Mamba
2.8B
48.9
ViRWKV-HM
8.7B
56.7
LoVgLLaVA
9B
34.4
51.9
53.3
51.6
Vo
53.5
65.4
VAMBA
55.9
65.9
Teacher and Student Models
NVILA
8B
44.4
27.8
80.1
60.9
3.7
57.7
58.7
70.1
64.2
70.0
LinMU-NV
8B
44.6
27.3
79.3
60.1
3.6
57.4
58.8
69.4
64.5
70.1
language models that uses standardized tests (often multiplc-choice) to mcasure general video understanding across diverse calegories and video lengths.
Training recipe. The distillation training is done on 8xA100 GPUs, using the AdamW optimizer. In Stages and 2, we use a Learning Rate (LR) of 5c-4 for M-MATE parameters (teacher backbone frozen). In Stage 3, we apply LoRA (rank-8) the backbone and use a smaller LR (2e-5). We stop Stage after 10K steps (when the Flex-MA branch's hidden alignment loss plateaus), Stage after 15K more steps, and Stage after 25K steps. The total wall time for distilling NVILA-Video-SB is about 50 hours. We use the same training recipe for Swee2.5-VVL-75 Instrucct...
4.2 Better Balance between Accuracy and Efficiency
Performance compared to teacher: Table ! summarizes the performance of the student LinMU model against the tcacher NVTLA model image and vidco benchmarks. We also include GPT-4o HHust et al. 2024) 88 strong closed-source baseline, well as other typical open-source baselines, including LLaVA OneVision [i et al. 2024), Qwen2-VL (Wang et al. 2024b), and InternVL2 (Chen et al., 2024d), and limerr-omplexity or guadratic-linear lenetti--iiner-yyybrre bybri baselines, such as VL-Mamba (Qiao et al. 2024), VisualRWKV. HM Hou ct al. 2025a). LongLLaVA VA (Wang et al. 2024ch, LongVU (Shen et al. 2024 and VAMBA (Ren et al. 2025), for reference. LinMU achieves performance that is very close the teacher across the board, even outperforming the teacher model on a few benchmarks.
For example, ActivityNet-QAA LinMU reaches 60.1% accuracy, essentially matching NVILA's 60.9%. On TextVQA, LinMU scores 79.3%, only slightly below the teacher's 80.1%. On the test sets of MMMU (58.8% vs. 58.7%), and across both settings of Video-MME (64.5%
(44.6% vs. 44.4%) and LongVideoBent vs. 64.2%; 70.1% vs. 70.0%), LinMU achieves slightly better performance than NVILA. Notably, LinMU outperforms many other models of similar scale on these benchmarks, benefiting from the strong performance of the teacher model. For instance, outperforms the prior 7B model, I.L.VA-OnVission and the 8B model, InternVL2. It is also on par with the recent Qwen2-VI. LL model.
MMMU and Video-MME tests complex multi-step reasoning; MLVU and LongVideoBentie specifically target long video understanding, where the advantage of linear complexity is expected to be most useful. After distillation, LinMU remains competitive on these benchmarks, while achieving linear complexity without quadratic attention. This indicates that our distillation successfully transfers the needed capability from the teacher to the linear student.
Inference efficiency: We measure the efficiency of VLMs in terms of TTFT and token throughput. TTFT reflects the time required to generate the first output token after the VLM receives the input, and token
14

---

Published in Transactions on Machine Learning Research (05/2026)
(a)
TTFT vs Input Length (4 better)
(b)
Token Throughput vs Input Length (f better)
NVILA LinMU
100
Token throughput (tokens/s)
80
2.7x
9.0x
40
20
NVILA
LinMU
210
2
2
233
2
225
2
210
2
212
2
2
2
2
Input length (tokens)
Input length (tokens)
Figure Efficiency comparison between the quadratic-complextty teacher model NVILA and the linear- complexity student model LinMU: how (a) TTFT (lower is better) and (b) token throughput (higher is better) scales the imput length increases. 2 input tokens correspond to 85-second 24fps video 224x224 resolution with 14x 14 spatial compression rale and 8x temporal compression rate.
throughput describes the rate which output tokens are generated. Note that standard attention with modern efficient kernels, such FlashAttention, already has linear memory footprint with respect to sequence length. Thus, replacing these atlention layers with M-MATE blocks does not yield significant memory savings. Thus, focus the latency reduction and throughput improvements they provide.
The scaling curves of TTFT and token throughput with respect to the input sequence length are shown in Fig. 0 Results are measured on NVIDIA H100 GPUs. Note that we exclude the vision encoding time here to highlight the LM Decoder cost that we are most interested in. Longer input sequences correspond to higher-resolution images or longer vidcos. 216 input tokens correspond to an 85-sccond 24fps video (2048 frames) 224x 224 resolution with 14x14 spatial compression rate and 8x temporal compression rate. Fig. 6 validates the complexity gain of LinMU over the standard attention-based VLM architecture. The TTFT of LinMU scales much more slowly than that of NVILA the input length increases, achieving 2.7x speed-up when dealing with minute-length videos, even at low resolution. The speed-up of LinMU in terms of token throughput is more significant: 9x higher than that of NVILA when dealing with a mimute-length video at 224x224 resolution. LinMU maintains almost the same token throughput. as the input sequence length increases, whereas NVILA's token throughput drops significantly. Benefiting from the complltyyyllxxity of LinMU, the longer the input sequence is, the more speed-up LinMU can achieve, The efficiency gains imply LinMU can enable longer-context reasoning (e.g., analyze entire movie continuously) and can be deployed in scenarios where NVILA would be too slow (e.g., real-time robot perception with limited compute). Note that LinMU achieves these gains while maintaining competitive performance relative to NVILA across various tasks.
hour-long surveillance video
4.3 Ablation Studies
We conduct ablations to support our design choices in the distillation process, justify the M-MATE block compositions, and validate the universality of our proposed LinMU framework.
Effect of distillation stages. We record performance improvement across the three distillation stages to validate their effectivencss in Table 2 Even with only initialization, LinMU can finish some simple understanding lasks. As distillation progresses, its performance across various tasks gradually improves. In particular, Stage 1 forces the masked bidirectional Mamba2 to approximate global attention patterns, while Stage trains the model to fit local patterns using the Local-Swin branch. Stage 3 allows the backbone adapt and calibrate the interactions between M-MATE blocks and the rest of the model. If we train Lin in single stage (i.e., unfreeze both branches and the backbone from scratch, and train the student
15

---

Published in Transactions on Machine Learning Research (05/2026)
Table 2: Performance comparison between models different training status. The performance of LinMU gradually improves as the distillation procceds. Finishing distillation in single stage leads to lower performance under the same compute budget.
Training Status
Activity Ance.
LongVideo VidggeeeeBBBcc...
MLVU
Video-MME: /
Initialization
39.3
29.4
48.7
43.2
Stage 1
48.7
38.5
58.3
52.9
Stage 1+2
56.7
52.7
65.2
63.4
Stage 1+2+3
60.1
58.8
69.4
70.1
All in one stage
57.3
54.2
66.8
67.0
Table 3: Performance comparison between different distillation loss settings. Reducing the corresponding factor to zero means eliminating that loss. The default setting (in the first row) achieves the best balanced performance across various tasks.
Ahid
Asup
ActivityNet Q.acc. Long VideoBenheen
MLVU
Vidco-MME w/uub.
0.5
0.5
0.5
60.1
58.8
69.4
70.1
0.5
0.5
0
58.7
58.6
69.1
69.8
0.5
0
0.5
59.2
57.1
68.5
68.2
0.5
0.5
58.1
56.6
67.9
68.6
0.5
0.5
60.8
56.9
68.2
69.3
0.5
1
0.5
59.4
58.2
68.8
69.5
0.5
0.5
59.9
58.9
69.1
69.4
model end-to-end), the student converges much more slowly and achieves lower final performance. This indicates that the stage-wist wise distillation is important.
Loss function choices. We conduct distillation experiments with different loss function settings to investi- gate the effectiveness of each loss term and compare their performance in Table 3 Note that the token-level KD loss Look is the standard distillation loss; hence, it is always retained with Aook = 0.5. Results presented in Table 3 indicaae that eliminating any of the hidden feature loss Laid, sequence-level loss Lueq: and ground- truth task loss Lsup incurs performance degradation. Without the hidden feature loss Lhid, the student model converges more slowly and reaches a lower final performance under the same compute budget. Removing the sequence-level loss Leeq makes it harder for the student model to mimic the teacher model's behavior. The ground-truth task loss Lsup is espocially helpful for improving the student performance on well-defined tasks with ground-truth labels, such as ActivityNet-QA. Our combined loss gives the best results, echoing findings from prior distillation research. We further explore increasing the factors these loss terms from 0.5 Although increasing Aoup improves the student performance on ActivityNet-QAA. the performance other tasks suffers. Overall, our default setting (i.c., the first row in Table 3) achieves the best balance among various tasks.
Window size of the Local-Swin branch. We conduct a distillation ablation on the Local-Swin branch with different spatial and temporal window sizes, and report the results in Table 4 As shown, decreasing the window size along either dimension yields only marginal speedups, since the Local-Swin branch already highly efficient. In contrast, enlarging the window spatially or temporally substantially incrcases latency while providing only limited accuracy gains. This is expected: the Local-Swin branch is primarily responsible for capturing short-range and localized correlations, whereas medium-range dependencies are effectively modeled by the Flex-MA branch.
M-MATE block components. We validate the effectiveness and cost of the Flex-MA branch and the Local-Swin branch each M-MATE block in Table 5 If we drop the Local-Swin branch entirely (making LinMU a pure Mamba2-based based model), performance dogrades: 51.3% on LongVidooBench (vs. 58.8% for full LinMU) and 62.7% on Video-MME (vs. 70.1% for full LinMU). This is expected, as a pure SSM struggles with fine local details in vision due to the adjacency preservation issue. The Local-Swin branch addresses this issue
16

---

Published in Transactions on Machine Learning Research (05/2026)
Table 4: Performance and efficiency comparison of different window sizes of the Local-Swin branch. We only compare their performance after Stage 2 to save computational resourcos. TTFT is mcasured at 32K input sequence length.
Window Size ActivityNet-OO QAace.
Long Benchh
MLVU
Video-MME /sub.
TTFT
32x4x4
56.6
52.9
65.2
63.4
2.043
16x8x8
56.8
52.6
65.3
63.2
2.421
16x4x4
56.7
52.7
65.2
63.4
1.723
16x2x2
56.5
52.8
64.8
63.3
1.689
8x4x1
56.7
52.2
64.6
62.9
1.692
Table 5: Performance and efficiency comparison between different M-MATE branch settings. TTFT (s) and token throughput (tokens/s) are measured at 32K input sequence length.
FA Local-Swin
MLVU
Video-MMEw /subb.
TTFT
token throughput
Yes
Yes
58.8
69.4
70.1
1.723
93.46
Yes No
62.9
62.7
1.549
101.48
No Yes
45.4
36.1
0.987
133.43
and improves the model performance on dealing with complex vision lasks (e.g., long video understanding) noticeably. Note that the Local-Swin branch maintains the linear complexity with respect to the number of tokens and only introduces tiny extra latency (e-g., 10% TTFT increment). On the contrary, removing the Flex-MA branch reduces TTFT and boosts token throughput significantly. However, this makes LinMU pure
model with fixed small window size. As shown in Table 5 in this case, the model performance degrades significantly on all benchmarks. This indicates that it is infeasible to build a powerful VLM solely using window attention with small window sizes. These ablations confirm that the dual- branch design of M-MATE critical for success: The Local-Swin branch addresses adjacent correlations, and the Flex-MA branch addresses medium-range and long-range correlations efficiently.
VLM backbone besides NVILA. To validate the generalization ability of our proposed framework, we further perform distillation Qwen2.5- VLL-B--Instrr Instruct and transform it into the LinMU architecture to achieve linear complexity. Performance comparison between the teacher and student models is shown in Table LinMU-Qwen still maintains the performance of the teacher model across various benchmarks. We further benchmark the efficiency gain brought about by LinMU in Fig. 7 Tt illustrates that the TTFT and token throughput of LinMU-Qwen scale much more slowly than the teacher model as the number of video frames (i.c., the input sequence length) increases. LinMU achieves 2.3x (4.9x) speed-up in terms of TTFT (token throughput) compared to the teacher model for 512-frame 256px videos. As the input video becomes longer or higher-resolution, LinMU delivers larger speedups.
4.4 Training Cost Analysis
Our total distillation and fine-tuning cost about 400 A100 hours, which approximately equals 5.6 H100 GPU days. This is significantly lower than the cost of training VLM from scratch. Specifically, NVILA initializes its vision encoder from SigCLIP and its LM decoder from Qwen2, but still uses 208 H100 GPU days to complete training. Qwen2.5-VL-7B initializes its LM decoder with Qwen2.5 LLMs and still uses about 4. tokens to train its LM decoder, which requires at least thousands of H100 GPU days. The proposed distillation pipeline not only saves the training cost by 100-1000x but also outperforms the teacher model on some benchmarks, such as Video-MME, as shown in Table and Table f
5 Limitation
In this article, we focus on replacing the self-attention layers of the LM decoder with our proposed M-MATE blocks, while leaving the vision encoder unchanged. The vision encoder also possibly involves global self- attention layers, which have quadratic complexity with respect to the number of patches tokens. Based
17

---

Published in Transactions on Machine Learning Research (05/2026)
Table 6: Performance comparison between the teacher Qwen2.5-VL S---------------E and the student LinMU model. After distillation, LinMU maintains the good performance of Qwen most tasks.
MMMU
VQx
Long BongVidcBeench
MLVU
Hidco-MME
Models
Size
pro
val
val
Lest
m-avg
w/o sub. w/sub.
Qwen2.5-VL.
7B
38.3
84.9
56.0
57.3
70.2
65.1
71.6
LinMU-Qwer
7B
37.3
83.2
55.4
56.5
69.6
65.2
71.8
(a)
TTFT vs # Video Frames (4 better)
(b)
Token Throughput vs Video Frames (f better)
18
16-
Qwen LinMU
Token throughput [tokens/s)
50
14
40
12
17
2.3x
TTFT
4.9x
8-
6-
20
Qwen
LinMU
10
256 Number of video frames
128
Number of video frames
Figure 7: Efficiency comparison between the quadratic-complexity teacher model Qven2.5-VL--7--Inste
and the linear-complexity student model LinMU: how (a) TTFT (lower is better) and (b) loken throughput (higher is better) scales the number of input video frames. The metrics are measured on the captioning task for 224x224 videos. The corresponding input sequence length for 128, 256, and 512 video frames is 16K, 32K, and 64K.
the current results the LM decoder, we speculate that we also replace the global self-attention layers in the vision encoder with proposed M-MATE blocks and distillation pipeline, and we leave it as part of future work. However, it is also worth noting that: (1) The complexity of the vision encoder only affects TTFT. The token throughput, which is usually more important for rapid interactions, is solely determined by the complexity of the LM decoder.
(2) The latency of the vision encoder in many modern VLMs is already almost linear with respect the number of patches, benefiting from efficient attention kernels like FlashAttention (Dao et al. 2022). Specifically, the vision encoder of Qwen2.5-VL has 31 layers with window attention and only one layer with full attention. We also measured the latency of the vision encoder in NVILA under different numbers of imput video frames, shown in Table 7 The latency is almost linear with respect to the number of frames (i.e., the number of patches).
6 Conclusion
We introduced LinMU, a multimodal understanding model that achieves linear computational complexity sequence length by distilling Transformer self-attention layers into M-MATE blocks. LinMU demonstrates that we maintain high accuracy challenging image-language and video-language tasks while elimi- nating the quadratic overhead of attention. Through carefully crafted distillation process, which includes reusing pretrained weights, progressively training branches, and aligning the student with the teacher's is be- havior at multiple levels, LinMU effectively learns to replace atlention with a combination of an efficient SSM and localized attention. Our experiments demonstrate that LinMU matches the performance state-of- the-art teacher VLMs (including NVILA-8B-VV Vidoo and Breen..-VL--BB-Insteee
on multiple benchmarks, and substantially improves inference efficiency and scalability. These results are an encouraging sign that Transformer-qualiy, multimodal reasoning can be achieved with computationally scalable architectures.
18

---

Published in Transactions on Machine Learning Research (05/2026)
Table 7: TIFT of NVILA contributed by the vision encoder under different numbers of video frames. The rosolution is 224,224, and latencies measured single H100 GPU.
Number Video Frames
32
64
128
256
Input Length (tokens)
1K
2K
4K
8K
Encoding Latency (s)
0.2694
0.5372
1.068
2.137
Moving forward, this work opens up several exciting new directions. An immediate next slep is to explore combining LinMU with token compression or adaptive imput selection to further improve efficiency, Another direction is to extend lincar multimodal modeling to native generative tasks, such as auto-regressive imago generation driven by a VLM. In addition, while we distilled from 7-8B teacher models, this framework could also be applied larger teachers (e.g., 33B) to train an efficient yet powerful student, or even specialist models in medical or robotics domains where long video analysis is needed. Finally, as linear architectures mature, important goal is to develop training paradigms that optimize linear models from scratch while maintaining competitive accuracy. LinMU's distillation results indicate that much of the sell-attention representational capacity can be transferred to linear form; a longer-term objective is to design dedicated training schemes (rather than directly inheriting Transformer recipes) that achieve this without relying on a Transformer teacher. We hope our work motivates broader exploration of attention-fres or attention-lite multimodal models, making tigh-performing VLMSs more accessible, faster, and better suited to ever-growing context lengths in real-world applications.
Acknowledgments
This work was supported in part by Qualcomm and in part by NSF under Grant No. CCF-2203399. We also appreciate the assistance provided by Hongxu Yin from NVIDIA Research.
19

---

Published in Transactions on Machine Learning Research (05/2026)
References
Jean-Baptiste Alayrac, Jeff Donahuc, Panline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 2--2316--3736, 2022.
Muhammad Arbab Arshad, Talukder Zaki Jubery, Tirtho Roy, Rim Nassiri, Asheesh K Singh, Arti Singh, Chinmay Hegde, Baskar Gauagathyssuramaninn, Aditya Balu, Adarsh Krishnamurthy, et al. Leveraging vision language models for specialized agricultural tasks. In IEEE/CVE Winler Conference on Applications of Computer Vision, pp. 1820-63323 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijic Wang, Jun Tang, et al. Qwen2.5-VL technical report. arviv preprinl ar 2rv:222002. 13923, 2025.
Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, Aleksander Ficek, al. NVIDIA Nemotron Nano 2: An accurate and efficient hybrid Hamba-Transforee reasoning model. ar preprint
arXiv:2508.144 14444, 2025.
Ix Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 2xir:2004.051500 2020.
Jieneng Chen, Luoxin Ye, Ju Hc, Zhao- Yang Wang, Daniel Khashabi, and Alan Yuille. Efficient large multi- modal models via visual context compression. Advances in Neural Information Processing Systems, 37: 73986-74007, 2024a.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. European Conference on Computer Vision, pp. 19-35 Springer, 2024b.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries open-source multimodal models with model, data, and test-lime scaling. Xix preprint arXiv:24 12.05271, 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. InternVI: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In IEEE/CVF Conference on Compuler Vision and Pallern Recognilion, pp. 24185 24198, 2024d.
Tri Dao and Albert Gu. Transformers SSMs: Generalized models and efficient algorithms through structured state space duality. arXiv preprint Xxx::24055.21060 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with TO-awareness. Advances in Nearal Information Processing Systems, 35: 16344 16359, 2022.
Amala Sanjay Deshmukh, Kateryna Chumachenko, Thomas Rintamaki, Matthieu Le, Tyler Poon, Da- nial Mohseni Taheri, Ila Karmanov, Guilin Liu, Jarno Seppanen, Guo Chen, et al. NVIDIA Nemotron Nano V2 VL. arXiv preprint Stt::211100922 2025.
Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoynan Li, Tao Zhong, Hao Chong, Bolin Li, Wanggui Hc, Fangxun Shu, and Hao Jiang. Streaming video trieval. Xin preprint ar orXiv:2503.00540 2025.
with in-context video KV-cache re-
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandoy, Abhishok Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Llama 3 herd of models, arXio e-prints, pp. Xiv- 2024.
20

---

Published in Transactions on Machine Learning Research (05/2026)
Mark Endo, Xiaohan Wang, and Serena Yeung-Levy. Feather the throttle: Revisiting visual token pruning for vision-language model acceleration. In IEEE/CVF International Conference on Compuler Vision, pp. 22826 22835, 2025.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. In IEEE/CVF Compuler Vision and Pallern Recognition Conference,
pp. 24108 24118, 2025.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In Pirst Conference on Language Modeling, 2024.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. Xix preprint orXi:2211100030 2021.
Haoyang He, Yuhu Bai, Jiangning Zhang, Qingdong He, Hongxu Chen, Zhenye Gan, Chengjie Wang, Xiang- tai Li, Guanzhong Tian, and Lei Xie. MambaAD: Exploring state space models for multi-class unsupervised anomaly detection. Advances in Neural Information Processing Systems, 17:71162-7111 2024.
Haowen Hou, Fei Ma, Zihang Li, and Fei Richard Yu. VisualRWKV-HI -H: Enhancing linear visual-language models via hybrid mixing, Information Pusion, 124:103336, 2025a.
Haowen Hou, Peigen Zeng, Fei Ma, and Fei Richard Yu. VisualRWKV: Exploring recurrent neural networks for visual language models. In 31st International Conference on Compulational Linguisties, pp. 10423 10434, 2025b.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. ToRA: Low-rank adaptation of large language models. In International Conference Learning Representations, volume 1, pp. 3, 2022.
Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Björn Ommer. ZigMa: A DiT-style zigzag Mamba diffusion model. In European Conference Computer Vision, pp. 148-166. Springer, 2024.
Kai Huang, Hao Zou, Ye Xi, BoChen Wang, Zhen Xie, and Liang Yu. IVTP: Instruction-guided visual token pruning for large vision-language models. In European Conference on Computer Vision, pp. 214 230. Springer, 2024.
Xiaohu Huang, Hao Zhou, and Kai Han. PruneVid: Visual token pruning for efficient video large language models. In Findings of the Association for Computational Länguisties: ACL 2025, pp. 19959-19973, 2025.
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. ar preprint arXiv:2410.2199 21276 2024.
Sicong Jiang, Zilin Huang, Kangan Qian, Ziang Luo, Tianze Zhu, Yang Zhong, Yihong Tang, Menglin Kong, Yunlong Wang, Siwen Jiao, et al. A survey on vision- saiio-languugc-action models for autonomous driving. arXiv preprinl 2tf::2506.24444. 2025.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Mochine Learning, pp. 5156-5165. PMLR, 2020.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. OpenVLA: An open-source vision-language- action model. arXiv preprint arXiv:2406.0920 2024.
Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Nascer, Abhijit Das, Salman Khan, and Fa- had Shahbaz Khan. GeoChat: Grounded large lanionnl model for remote sensing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17831-27844 2024.
21

---

Published in Transactions on Machine Learning Research (05/2026)
Benjamin Lefandeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xFormers: A modular and hackable transformer modelling library. https:
2022.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Lla VAAOneVision: Easy visual task transfer. orXiv preprint (xXiv:240800332) 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Internalional Conference Machine Learning. pp. 19730-199 19742. PMLR, 2023a.
Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, et al. MBQ; Modality-balanced quantization for large ussionllnguagg models. In IEEE/CV Computer Vision and Pattern Recognition Conference, pp. 4167-4177, 2025.
Xuanlin Li, Yunhao Fang, Minghua Liu, Zhan Ling, Zhuowen Tu, and Hao Su. Distilling large dision-language model with outtof-distribution genoralizability. In IEEE/CVF International Conference on Computer Vision, pp.
2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892-34911 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved basclines with visual instruction tuning, In IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 26296 26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Lee LLaVA-NeXT. Improved reasoning, OCR, and world knowledge, Jamuary 2024b. URL https///lav--22.gthhub 1o/
Yue Liu, Yunjie Tian, Yuzhong Zhao, Honglian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. VMamba: Visual state space model. Advances in Neural Information Processing Systems, 37:103031 103063, 2024c.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2202-3211. 2022.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dachong Li, et al. NVILA: Efficient frontier visual language models. In Computer Vision and Pattern Recognition Conference, pp. 4122-14880 4134, 2025.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, In The 36th Conference Newral Information Processing Systems, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arxiv:2310, 02255, 2023.
Ji Ma, Wei Suo, Peng Wang, and Yanning Zhang. Short-LVLM: Compressing and accelerating large vision- language models by pruning redundant layers. In 33rd ACM International Conference on Multimedia, pp. 2025.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, al. RWKV: Reinventing rnss for the transformer era. arXiv preprinl Xiv:2330 13048, 2023.
22

---

Published in Transactions on Machine Learning Research (05/2026)
Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. VI.-Mamba: Exploring state space models for multimodal learning. Xiv preprint. or arXiu:2403. 13600, 2024.
Weiming Ren, Wentao Ma, Huan Yang, Cong Wei, Ge Zhang, and Wenhu Chen. VAMBA: hour-long videos with hybrid caamba-raansfrmert In IEEE/CVF International Conference on Computer Vision, pp. 21197-212211 21208, 2025.
Understanding
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. TlaVA-PruMerge: Adaptive token reduction for efficient large multimodal models. In IEEE/CVF International Conference on Computer Vision, pp. 18877228677, 2025.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, el. al. LongVU: Spatiotemporal adaptive compression for long video-langnage understanding. arXiv preprint arX
2024.
Yuheng Shi, Minjing Dong, and Chang Xu. Multi-scale Vmamba: Hierarchy in hierarchy visual state space model. Advances in Neural Information Processing Systems, 37:25687 25708, 2024.
Amanproct Singh, Vivek Natarjan, Moet Shah, Yu Jiang. Xinlci Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8317-8326, 2019.
Boyuan Sun, Jiaxing Zhao, Xihan Wei, and Qibin Hou. LLAVA-Scissor: Token compression with semantic connected components for video LLMs. arXiv preprint arXiv:2506.21862 2025.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive Network: A successor to transformer for large language models. arXiv preprint arXin:337.08621, 2023.
Yi-Lin Sung, Jachong Yoon, and Mohit Bansal. ECoFLaP: Efficient coarsc-to-fine layer-wise pruning for vision-language models. arXiv preprint ar arxi::2310.022 029998, 2023.
Zicong Tang, Ziyang Ma, Suqing Wang, Zuchao Li, Lefei Zhang, Hai Zhao, Yun Li, and Qianren Wang. CoViPAL: Layer-wise contextualized visual token pruning for large vision-language models. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 20701-20714, 2025.
Ryutaro Tanno, David G.T. Barrett, Andrew Sellergren, Sumedh Chaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Charles Lau, Tao Tu, Shekoofeh Azizi, et al. Collaboration between clinicians and vision- language models in radiology report generation. Nature Medicine, (2):599 608, 2025.
Hongjie Wang, Chih- Yao Ma, Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, Peter Vajda, Xiaoliang Dai, and Niraj K. Jha. LinGen-Uni: A universal lincar-complexity framework for high-resolution minuto-length text-to-video generation. Research Square Preprinl, 2025a.
Hongjie Wang, Chih- Yao Ma, Yen-Cheng Liu, Ji Hou, Tao Xu, Jialiang Wang, Felix Juefei-Xu, Yaqiao Luo, Peizhao Zhang, Tingbo Hou, Peter Vajda, Niraj K. Jha, and Xiaoliang Dai. LinGen: Towards high- resolution minute-length text-to-vidco generation with lincar computational complexity. In Computer Vision and Pultern Recognition Conference, pp. 2578 2588, 2025b.
Junxiong Wang, Daniele Paliotta, Avner May, Alexander Rush, and Tri Dao. The Mamba in the Llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems, 37:62432- 62457, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing lasion-language model's perception of the world any resolution. Xiv preprint orXiv:2409. 12191, 2024b.
23

---

Published in Transactions on Machine Learning Research (05/2026)
Xidong Wang, Dingjic Song, Shunian Chen, Chen Zhang, and Benyou Wang. LongLLaVA: Scaling multi- modal LLMs to 1000 images efficiently via hybrid architecture. orXiv preprinl or arxin:2409.0020 028809 2(5):6, 2024c.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. LongVideoBench: A benchmark for long-context interleaved vidco-language understanding. Advances in Neural Information Processing Systems, 37:28828- 28857, 2024.
Chaodong Xiao, Minghan Li, Zhengqiang ZHANG, Deyu Meng, and Lei Zhang. Spatial- Effective visual state space models via structure-aware state fusion. In The Thirleenth International Conference Learning Representations, 2025.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activity QA: A datasct for understanding complex web videos via question answcring. In AAAI Conference on Artificial Intelligence, volume 33, pp. 9127 9134, 2019.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI, In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 9556 9567, 2024.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big Bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33:17283- 17297, 2020.
Pengehuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi- scale vision Longformer: A new vision transformer for high-resolution image encoding. In IEEE/CVF International Conference on Computer Vision, pp. 2998-3008, 2021.
Yuan Zhang, Chun-Kai Fan, Jumpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohci Nakata, Kurt Keutzer, et al. SparseVLM: Visual token sparsification for efficient vision-language model inference. arXiv preprint arXiv:2410.04411, 2024a.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, arXiv preprint arXiv:2410.. 02713, 2024b.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong, Bo Zhang, et al. MIVU: Benchmarking multi-task long video understanding. In IEEE/CVP Computer Vision and Pattern Recognition Conference, pp. 13691 13701, 2025.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhosciny. MiniGPT-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXio:2304.1052 2023.
24


Reviews:
[{"review_id": "RfhZHx0C88", "reviewer": "Reviewer_azyg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to transform a pretrained multimodal model (image-text) into one with linear complexity with respect to input length. The approach replaces self-attention layers in the language model backbone with modules composed of two parallel branches: a bidirectional Mamba2 block (Flex-MA) and a fixed-window-size Swin attention (Local-Swin). To maintain performance, the authors employ a three-stage progressive distillation process from the original \"teacher\" model to the modified \"student\" model: first finetuning the Flex-MA branch, then both branches, and finally the whole model via LoRA. The method is evaluated on both image and video understanding benchmarks.\n\n**Strengths:**\n\n- The method significantly improves efficiency and token throughput for large VLMs (8B parameters), scaling better with input length than standard transformer architectures.\n- It effectively preserves downstream performance compared to the teacher model used.\n\n**Weaknesses:**\n\n- There are significant concerns regarding the accuracy of the \"linear complexity\" claims, particularly regarding the vision encoder and existing literature.\n- The evaluation lacks comparisons to other linear-complexity baselines, relying solely on comparing to the teacher model.", "claims_and_evidence_yn": "No", "claims_explanation": "While the empirical results regarding performance preservation are convincing, the claims regarding **linear complexity** and **novelty** are not fully supported by the evidence provided:\n\n1. **Complexity Claims:** The authors claim to achieve \"pure linear complexity in input length.\" However, the method appears to leave the vision encoder untouched. Vision encoders also typically rely on attention blocks that scale quadratically with input length. Therefore, the _entire_ model effectively remains quadratic, even if the language model part is linear. Figure 6 supports this observation: while efficiency improves over the teacher, the correlation with input length does not appear strictly linear.\n2. **Novelty Claims:** The paper claims to be \"the first multimodal model that achieves pure linear complexity.\" This contradicts the paper's own related work section, which cites other VLMs that replace attention layers, like VL-Mamba (Qiao et al., 2024).\n3. **Comparative Evidence:** The distilled models are compared exclusively to the original teacher model. To support claims of efficacy, the method should also be compared against other approaches that replace attention with linear-complexity operations.", "audience_yn": "Yes", "audience_explanation": "Efficient processing for Large Vision-Language Models (VLMs) is a highly relevant topic for the TMLR audience. The ability to improve token throughput and scale to longer inputs is critical for practical deployment. If the claims are clarified, the distillation strategy and architecture modifications (Mamba2 + Swin) would be of significant interest to researchers working on efficient architectures and to practitioners.", "requested_changes": "**Major Changes:**\n\n1. **Rectify Complexity Claims:** The authors must accurately qualify the \"linear complexity\" claims. If the vision encoder retains quadratic attention, the text must reflect that only the language backbone is linear, or the \"pure linear\" claim must be removed. \n2. **Correct Novelty Statement:** Remove or modify the claim of being \"the first\" to achieve this complexity, specifically acknowledging the prior work cited in the related work section.\n3. **Expand Baselines:** Include comparisons (performance and efficiency) against other methods that replace attention with linear-complexity operations, rather than comparing solely against the teacher model. This will help put in perspective the effectiveness of the proposed architecture.\n\n**Minor Changes:** \n\n4. **Unify Presentation:** The evaluations for the Qwen teacher setup and NVILA setup appear to use identical benchmarks (comparing Table 1 and Table 5). I recommend merging these or clarifying the distinction to improve readability. In Figure 7, please change the x-axis to represent the \"number of tokens\" directly rather than the \"number of video frames,\" to allow for easier interpretation of efficiency scaling. Similarly, figure 7 could also be merged with figure 6.", "broader_impact_concerns": "No specific ethical concerns were identified in the draft. The work focuses on architectural efficiency for existing open models.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qX1OULA1iZ", "reviewer": "Reviewer_7owK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes LinMU, an efficient Vision-Language Model that converts existing quadratic-complexity self-attention layers into linear-complexity modules. The core design of LinMU lies in the M-MATE block, a drop-in replacement for attention that uses a dual-branch architecture: the Flex-MA branch captures global context efficiently with Mamba2, while the Local-Swin branch handles precise local correlations. On top of that, the paper introduces a three-stage knowledge distillation pipeline that effectively transfers knowledge from a pre-trained teacher VLM to the new LinMU student architecture.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper's central claims are generally supported by clear and convincing evidence.\n-  The main contribution of the paper \"LinMU achieves linear complexity while maintaining the performance of the teacher model\" is supported by Table 1&5 (for comparison with teacher models and other VLMs)  and Figure 6&7 (for improved efficiency)\n\n- The effectiveness of the proposed dual-branch M-MATE is supported by results in Table 4.\n\n- The effectiveness of the proposed three-stage distillation is supported by results in Table 2.", "audience_yn": "Yes", "audience_explanation": "- The paper proposes an effective and practical solution to enhance the efficiency of VLMs.", "requested_changes": "-  The paper reports a total distillation time of ~50 hours on 8 A100 GPUs. While the distillation stage already involves a large amount training data, it would be better to compare the current setting with training LinMU with such data without teacher model. Also, it would be worthwhile to compare the training costs of distillation and standard training recipe of VLM.", "broader_impact_concerns": "n/a", "additional_comments": "- While the entire LinMU framework is effective, it is parasitic; it heavily relies on a powerful, pre-trained, quadratic-attention teacher model, which may limit its generalization.", "extra_scores": {}, "extra_text": {}}, {"review_id": "767yMmNMV0", "reviewer": "Reviewer_ciny", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Core Contributions:\n1. Proposes LinMU, a VLM with pure linear complexity by replacing self-attention with the dual-branch M-MATE block (Flex-MA for global dependencies, Local-Swin for local correlations).\n2. Develops a three-stage distillation framework (initialize with teacher weights, train Flex-MA, joint train both branches, LoRA fine-tune) to preserve teacher performance.\n3. Matches state-of-the-art VLM performance on multiple benchmark, while boosting TTFT by 2.7× and token throughput by 9.0× for long videos.\n\nStrengths:\n1. Linear Complexity with Competitive Performance: The primary strength is achieving true linear complexity with respect to sequence length without the significant performance drop typically associated with non-attention architectures. It matches the accuracy of strong teacher models like NVILA and Qwen2.5-VL.\n2. Effective Hybrid Design (M-MATE): The dual-branch design combines the strengths of SSMs (global mixing, linear cost) and window attention (local precision), mitigating the adjacency preservation issue that plagues flattened 1D sequence models in vision tasks.\n3. Robust Distillation Strategy: The proposed three-stage distillation pipeline, combined with weight reuse from the teacher, ensures stable convergence and effective knowledge transfer, avoiding the high cost of pre-training a linear VLM from scratch.\n4. Significant Efficiency Gains: The empirical results demonstrate substantial improvements in inference speed and throughput, particularly for long-context scenarios, making the model suitable for edge deployment or resource-constrained settings.\n\nWeakness:\n1. Reliance on Teacher Models: The framework is fundamentally a distillation method. It requires a high-quality, pre-trained quadratic VLM teacher to function. It does not explore training a linear VLM from scratch, meaning its performance is likely capped by the teacher's capabilities.\n2. Complexity of Implementation: Compared to a simple drop-in replacement, the three-stage training pipeline with specific freezing schedules and loss combinations is relatively complex to implement and tune.\n3. Limited to Decoder Modification: The method currently focuses on replacing attention in the Language Model decoder. The Vision Encoder remains unchanged, which might still be a bottleneck in fully end-to-end processing for extremely high-resolution inputs.\n4. Fixed Window Limitations: While the Local-Swin branch is efficient, using a fixed small window size might still limit the modeling of medium-range dependencies that fall outside the window but are too subtle for the global Mamba branch to capture perfectly.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The evidence is accurate and clear to support the claims.\n\nThe claims of Linear Complexity is supported by theoretical grounding on section 3.1 and empirical scaling.\nThe claims of Performance Parity is supported by multi-benchmark validation.\nThe claims of Architectural Necessity is supported by clear ablation study.", "audience_yn": "Yes", "audience_explanation": "I believe the findings would be of significant interest to TMLR's audience.\n\n1. This paper addresses fundamental scalability bottleneckss: the quadratic complexity of self-attention. As multimodal models increasingly handle high-resolution images and long-form video, computational scaling becomes a primary research concern.\n2. This paper provide a novel integration of SSMs and Attention: the M-MATE block combines SSMs (Mamba2) with localized 3D Swin attention in a way that eliminates quadratic operations entirely.", "requested_changes": "1. Comparison with Existing Linear/Efficient VLMs\nTable 1 compares LinMU primarily against standard, quadratic Transformer-based VLMs (NVILA, Qwen2-VL, LLaVA). While it is impressive that LinMU matches their performance, the paper lacks a direct performance comparison with other existing linear or efficient architectures mentioned in the Related Work, such as VL-Mamba, Cobra, or VisualRWKV.\n\n2. Extend Ablation on Window Size in Local-Swin Branch\nThe paper uses a fixed 16×4×4 window size but does not justify its choice. Add ablation studies testing different window sizes (e.g., 8×4×4, 32×4×4) to show how local window scale impacts performance-efficiency tradeoffs. This is critical to validating the Local-Swin branch’s design rationale.\n\n3. Complexity Analysis of the Vision Encoder\nThe paper’s title and abstract claim \"Multimodal Understanding Made Linear.\" However, Section 3.1 explicitly states, \"we leave the Vision Encoder unchanged.\" Many state-of-the-art VLMs utilize ViTs that possess quadratic complexity with respect to the number of patches per image/frame, or the number of frames if joint space-time attention is used.  If the Vision Encoder becomes the bottleneck at very long context lengths due to quadratic scaling, this limitation must be acknowledged.", "broader_impact_concerns": "This article does not include a section on Broader Impact, and I do not believe there is any need to discuss any Broader Impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)