Efficient-SAM2: Accelerating SAM2 with Object-Aware Visual Encoding and Memory Retrieval
Authors: Jing Zhang
Zhikai Li
Xuewen Liu
Qingyi Gu
Venue: ICLR 2026 Poster (2026)
Keywords: Segmemt Anything Model
Efficient Deep Learning
Model Acceleration
Primary area: infrastructure, software libraries, hardware, systems, etc.

Abstract: Segment Anything Model 2 (SAM2) shows excellent performance in video object segmentation tasks; however, the heavy computational burden hinders its application in real-time video processing.
Although there have been efforts to improve the efficiency of SAM2, most of them focus on retraining a lightweight backbone, with little exploration into post-training acceleration.
In this paper, we observe that SAM2 exhibits sparse perception pattern as biological vision, which provides opportunities for eliminating redundant computation and acceleration:
i) In mask decoder, the attention primarily focuses on the foreground objects, whereas the image encoder in the earlier stage exhibits a broad attention span, which results in unnecessary computation to background regions.
ii) In memory bank, only a small subset of tokens in each frame contribute significantly to memory attention, and the salient regions exhibit temporal consistency, making full-token computation redundant.
With these insights, we propose Efficient-SAM2, which promotes SAM2 to adaptively focus on object regions while eliminating task-irrelevant computations, thereby significantly improving inference efficiency.
Specifically, for image encoder, we propose object-aware Sparse Window Routing (SWR), a window-level computation allocation mechanism that leverages the consistency and saliency cues from the previous-frame decoder to route background regions into a lightweight shortcut branch.
Moreover, for memory attention, we propose object-aware Sparse Memory Retrieval (SMR), which allows only the salient memory tokens in each frame to participate in computation, 
with the saliency pattern reused from their first recollection.
With negligible additional parameters and minimal training overhead, Efficient-SAM2 delivers 1.68$\times$ speedup on SAM2.1-L model with only 1.0\% accuracy drop on SA-V test set, where SWR and SMR provide 1.83$\times$ and 1.78$\times$ speedups, respectively.

Published as a conference paper at ICLR 2026
EFRICIENI-SAM2: ACCELERATING SAM2 WITH OBJECT-AWARE VISUAL ENCODING AND MEMORY RETRIEVAL
Jing Zhang' Zhikai Li Xuewen Liu Qingyi Gu
'Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences (zhangjing2024, zhikai.li. 1iuxuewen2023, qingyi gel.igge@eaiaea... cn
ABSTRACT
Segment Anything Model 2 (SAM2) shows excellent performance in video object segmentation tasks; however, the heavy computational burden hinders its applica- tion in real-time video processing. Although there have been efforts to improve the efficiency of SAM2, most of them focus on retraining lightweight backbone, with little exploration into post-training acceleration. In this paper, we observe that SAM2 exhibits sparse perception pattern as biological vision, which provides opportunities for eliminating redundant computation and acceleration: i) In mask decoder, the attention primarily focuses on the foreground objects, whereas the image encoder in the earlier stage exhibits broad attention span, which results in unnccessary computation to background regions. ii) In memory bank, only a small subset of tokens in each frame contribute significantly memory attention, and the salient regions exhibit temporal consistency, making full-token computa- tion redundant. With these insights, propose EfficientSAAM2, which promotes SAM2 to adaptively focus on object regions while eliminating task-irrelevant computations, thereby significantly improving inference efficiency. Specifically, for image encoder, we propose object-award Sparse Window Routing (SWR). a window-level computation allocation mechanism that leverages the consistency and saliency cues from the previous-frame decoder to route background regions into a lightweight shortcut branch. Moreover, for memory attention, we pro- pose object-aware Sparse Memory Retrieval (SMR), which allows only the salient memory tokens in each frame to participate in computation, with the saliency pattern reused from their first recollection. With negligible additional parame- ters and minimal training overhead, Efficient-SAM2 delivers 1.68x speedup on SAM2.1-L model with only 1.0% accuracy drop SA-V test set, where SWR and SMR provide 1.83x and 1.78 speedups, respectively. Code is available at: bttps://gitee com wiw.iggnnnn019//iiiiio
INTRODUCTION
Segment Anything Model (SAM) (Kirillov et al., 2023) was introduced a vision foundation model, enabling promptable segmentation following user's instruction (Li et al., 2024b; Chen et al., 2024), and its successor, SAM2 (Ravi et al., 2024; Jiaxing & Hao, 2025), extends this framework with video understanding capabilities. Leveraging its memory mechanism, SAM2 can reference long-term historical information to process streaming video sequences, achieving remarkable performance in video object segmentation (VOS) (Ding et al., 2024) and video object tracking tasks (Cuttano et al., large-scale visual backbone and frame-level
2025; Yang al., 2024). However, the reliance memory interaction incurs prohibitive computational costs and latency (Zhou et al., 2025; Xiong al., 2024a; Sun et al., 2025), which limits its applicability in real-time video processing.
The primary latency bottlenecks of SAM2 lie in its image encoder and memory attention block. To address this, Ed (Zhou al., 2025) distills lightweight model architecture and introduces
"Corresponding authors: {zhikai.li, qingyi.gu Betoviigl@@aaaaa..oo @ia.ac.cn.

---

Published as a conference paper at ICLR 2026
spatial perceiver for memory compression, achieving edge-level efficiency. However, its high training costs and degraded performance hinder practical deployment, A second line of research accelerates vision Transformer (ViT) backbones by dynamically merging similar tokens (Bolya & Hoffman, 2023; Norouzi et al., 2024); yet these approaches not only fail to adapt to SAM2's hierar- architecture, but also struggle with segmentation tasks (Kienzle et al., 2024), resulting in severe performance degradation and undesirable practical acceleration.
chical,
In this work, we observe that SAM2 exhibits a sparse perception pattern as biological vision; how- ever, its default processing fails to capitalize on this sparsity, leading to redundant computation.
Object-focused attention in mask decoder vs. Broad attention span in image encoder. During mask decoding, the prompt-to-image attention ex- hibits object preference towards prompt's interests. As shown in Figure 1, attention is focused on the foreground objects and potential distractions, while the background features are significantly suppressed. In contrast, the image encoder in the earlier stage is unaware of prompt's 's interest and exhibits board at- tention span, thus producing useless computations.
Prompts
Image Encoder
Decoder
Object A
Object B
Figure 1: attention coverage, but the mask decoder fo- cuses narrowly prompt-relevant objects.
The image encoder exhibits broad
Video Stream
Slegerale aedd
Cross-Attention
Memory Queue
t-19
E-20
1-25
t-24
Recollaction
CSa0 968
CS=t 999
SS=0.998
CS:0 967
CSa0 897
Temporally Consistent Memory Focus
Figure 2: In memory attention, the memory frame exhibits concentrated attention distribution, sug- gesting redundancy in the memory bank, and its saliency pattern remains temporally consistent, as evidenced by high cosine similarity (CS) to its first recollection.
Temporally-consssseen saliency in memory bank vs. Repeated full-token recollections. In memory attention, entire memory frames are repeatedly recalled modulate image features. How- ever, we observe that the useful information is sparsely distributed, with only small subset of tokens actively providing cues. As shown in Figure 2, in the default mode, a memory frame is recomputed for every incoming frame before it exits the memory queue; yet the attention pattern shows strong temporal consistency, frequently focusing on a fixed set informative tokens, which provides opportunity prune unnecessary computation.
With the above insights, we propose Efficient-SAM2. an object-aware computation scheduling and optimization framework accelerate SAM2, with a particular focus on improving the efficiency of both image encoder and memory attention. Specifically, for image encoder, we introduce object- aware Sparse Window Routing (SWR), which performs computation allocation at the window level and integrates naturally with SAM2's window attention. It assigns background windows to a lightweight shortcut branch to reduce redundant computation, where the router leverages the con- sistency and saliency cues provided by previous frame's mask decoder. Here, the shortcut branch consists of two linear layers with negligible parameters, which can trained efficiently. addi- tion, for memory attention, we propose object-aware Sparse Memory Retrieval (SMR), which selects only the salient tokens in the memory bank to participate in attention computation, effectively re- ducing the overall token cost. Notably, leveraging temporal consistency, the saliency pattern of memory frame cached during its first participation in memory attention and can be reused across subsequent frames without recomputation. Our contributions summarized follows:
We propose Efficient- a post-training sparse acceleration scheme for SAM2. ad- dresses the contradiction between SAM2's sparse perception pattern and its dense compu- tation, highlighting redundant operations in image encoder and memory attention, and thus offering new insights for efficient acceleration.
2

---

Published as a conference paper at ICLR 2026
With the above insights, we we propose dedicated acceleration schemes for both image encoder and memory attention. For image encoder, we allocate computation at the window level lightweight shortcut branch. For memory attention, we restrict computation to only the salient memory tokens in each frame, with saliency determined by the memory attention map from their initial participation.
by routing background windows
Sfficient-SAM2 shows superior performance-speen trade-off across multiple VOS bench- marks, while adding negligible additional parameters and incurring only minimal train- ing cost. For instance, Efficient-SAMM2 delivers 1.68x end-to-end speedup on SAM2. 1-L model with only 1% accuracy drop on SA-V test set; individually, SWR and SMR provid- ing 1.83x and 1.78 x speedups, respectively.
RELATED WORKS
Segment Anything Model. SAMs (Kirillov et al., 2023; Carion al., 2025), owing their robust segmentation and tracking capabilities, have emerged as versatile foundational models in computer vision. In particular, SAM2 (Ravi al., 2024) introduces a streaming memory mechanism that infuses long-range range historical context into the current frame, further extending its capabilities from image to video. Recently, SAM2 has been widely applied to semi-supervised video object segmen- tation (Ding et al., 2024), video object tracking (Yang et al., 2024; Videnovic et al., 2025), and refer- ring video object segmentation (Cuttano et al., 2025), consistently outperforming prior task-specific models. However, the large image encoder and computationally intensive memory mechanism in SAM2 impose substantial computational burdens and inference latency (Li Gu, 2023), severely limiting its widespread deployment in real-world applications (Liu et al., 2024a; Li et al., 2024a).
Lightweight SAM2. To reduce the computational cost of SAM2's image encoder and memory mechanism, existing methods have conducted valuable explorations (Lv et al., 2024; Zhang et al., 2025). EfficientTAM (Xiong et al., 2024h) designs a lightweight image encoder and leverages pool- ing operations within its memory mechanism reduce computational overhead. EdgeTTAM (Zhou et al., 2025) introduces a spatial perceiver module to compress memory and retrains the whole net- work end-to-end to accommodate this module. Although these methods have achieved significant success, they require costly retraining on large-scale datasets. In contrast, post- post-training accelera- tion techniques. which avoid end-to-end retraining, are more flexible and generalizable for Efficient SAM2. Unfortunately, post-training acceleration tailored for SAM2 remains largely unexplored.
Post-training Acceleration. Post-training acceleration (Li et al., 2023a; Liu et al., 2025), valued for its low implementation overhead, has been extensively studied across diverse model architec- tures (Liu al., 2026; Li et al., 2023b; Liu et al., 2024b). For Vision Transformer (ViT) architec- tures, ToMe (Bolya et al., 2022) dynamically merges similar tokens using a general matching algo- rithm, reducing computational redundancy without altering the model structure. ALGM (Norouzi et al., 2024) further introduces two-stage local-to-globe merging strategy for segmentation mod- els, significantly enhancing pixel-level task performance. For Diffusion Transformer (DiT) archi- tectures, ToMe4DM (Bolya & Hoffman, 2023) leverages the temporal characteristics of diffusion, introducing randomness within local windows during token matching to accelerate diffusion models without any retraining. While these methods accelerate VIT and DiT models successfully (Li al., 2023c; 2022c), they fail on SAM2. The windowed attention in image encoder renders existing token merging and matching strategies incompatible, resulting in suboptimal accuracy and speedup.
METHOD
3.1 PRELIMINARIES
The Pipeline of SAM2. SAM2's image encoder Eing adopts Hiera (Ryali et al., 2023; Li et al., 2022a) as the backbone to hierarchically encode multiscale image features, with stages denoted as {So. S2), the image feature F, is obtained as follows:
(1) where I, is the current frame input. Then F serves as image embedding F,, which interact with the memory bank M, in the memory attention module A obtaining memory-conditioned image
F = (F F F = Eing (I:),
3

---

Published as a conference paper at ICLR 2026
feature FM as follows:
FM = M.).
(2)
Next, the mask decoder performs cross- coss-attentioon between prompt embedding P. and FM, with Fr and F help to provide high-resolution details, deriving the output O, follows:
O2 = M F. F Pt).
Finally, O2 and F are fused and encoded as memory the memory encoder Emem as follows:
M₁ Fmeem F Ot).
Image Encoder with Window Attention. SAM2 predominantly adopts window attention (Li et al., 2022b; Bolya et al., 2024) in its encoder to capture fine-grained local structure. For a window attention module in layer 7, denote the inputs as X E R RRIHXXM xw xc where (H, W) is the feature height and width, and channel dimension, the process as follows:
x Windowwartition (X'-1; w) E NR xhxwxC
(5)
W
X W MHSA(X IR Nw xxxwxxxxxx
(6)
X WindowUnnarritionn(XX
w) E RIXHXWXC
where WindowPartitionn)) pads and partitions image features into non-overlapping windows while WindowUnpartitionn perform reverse operation. (h, denotes the window size, Nw represents the number of windows. MHSA(-) denotes Multi-Heaa Self-Attention. Thus, attention is computed locally within the windows, leaving windows mutually independent.
Mask Decoder. The mask decoder predicts three candidate masks along with their confidence scores the highest score. Additionally, it returns an occlusion score Subj. where Subj 0 indicates object absence, Subj 0 indicates presence, and the absolute value |saij depicts the tracking confidence.
8iou E [0, 1] and outputs the mask with
Memory Bank. At moment M, consists of prompted frames MZ and First-In-First-On (FIFO) queue M2 of previous frames (Zhou et al., 2024), represented as:
Me = MP e M MM + [Me-m, Memmm Mc-1),
(8)
where denotes queue length, and @ denotes concatenation between sets. In semi-supervised VOS task, the length of M keeps 1. Equation 8 is the default setting in which memory sampling interval At 1. For more general formulation, the queue consists every At frame plus the last frame, i.c., My = {M (( m- A), M(n-n- (1((nmm-1)amm m-1))tt- MnAt]. Here n = [(t 2)/ For simplicity, we present subsequent formulations based on At 1, while they generalize other cases.
3.2 OBJECT- SPARSE WINDOW ROUTING
The disparity between the image encoder's broad attention span and the mask decoder's object- focused attention reveals significant encoding redundancy. To address this inefficiency, we propose object-aware Sparse Window Routing, window-level computation scheduling scheme that routes windows through separate pathways based on their relevance to objects. This divide-and-conquer handling of windows aligns naturally with window attention's locality, thereby effectively reducing redundancy and preserving model performance. Specifically, our method consists of two compo- nents: an object-aware router for predicting object-relevant windows, and lightweight shortcut branch for processing background windows efficiently.
Object-Aware Router. Assuming that the video stream is continuous both spatially and tempo- rally, the router accurately predicts object-relevant windows Way following the Spatial-Temporal Consistency of Object and Perceptual Saliency of Object, formulated as:
Waj Wpred Weatient-
W Wpeed are windows that cover the previous frame's prediction masks, thereby ensuring spatial- temporal continuity of the object, formulated as:
Ppedd (Wr 3(x, y) E W., 1,,1( =1), O 01-10-
(10)
4

---

Published as a conference paper at ICLR 2026
Image Encoder
Memory Attention
Window
Router
Video Stream
Lightweight Shortout Branch
Memory Retriever
Window Ameriion Chobne-Amantinn Scaling
Memory
Bank
Object-Aware Sparse Window Routing (SWR)
Objoct-Aware Sparse Memory Retrieval (SMR)
0 SpatioTTmmporal Consistency of Object
Sparse Mask Queue (FIFO)
Cache
From t-1
P-5
Ressed From Ther Fisl Recillccion
Decader
Retrieval
O1-1j
O
prod
Key
Sallency
Plorcephual Sallency Object
W
Value
Pattern
From t-1
Memory Attention
Top, (A).))
Mask Decoder
Qutry
saliest
Racdlaction
Figure 3: Overview of Efficient-SAM2. For image encoder, we introduce object-aware Sparse Win- dow Routing (SWR), which assigns object-irreleant background windows to a lightweight shortcut branch based on spatial-temporal consistency and perceptual saliency of the object, thus reducing encoding redundancy. For memory attention, we propose object-aware Sparse Memory Retrieval (SMR), which builds a FIFO mask queue to retrieval most salient memory tokens, in which the saliency patterns reused from their first recollection, thereby reducing the computational cost.
Here, O.-1,; E {0, 13 #x .j = 0, 1,2 are mask predictions. Inspired by SAM2. 1++ (Videnovic et al., 2025), which reveals that the mask decoder could capture distractors in predictions with lower Siou. all three alternative masks are utilized to enhance distractor awareness. Morcover, to address motion-induced boundary escape, dilation is applied to O,-1 for robust window estimation.
When tracking confidence low or no mask is predicted, Waalint broadens perception view by adding windows with high artes-attention saliency from previous frame's 's mask decoder as follows:
(W&
Σ S Th. if Sabj Oobj. 0:20a
Waatien
(11)
0,
otherwise,
where Ouy is threshold for prediction confidence, determines the cumulative attention threshold. For each window, its saliency score a, E Aw,1-1 is computed as:
Aw = {as aa = Σ A-112z,0 E i N.
(12)
(K.))EW:
where A,-1 " is the average cross-attention weight of image tokens in the mask decoder. This saliency-guides selection protects both re-tracking and anti-interference capabilities, effec- tively handling challenges including object occlusion, reappearance, and dynamic backgrounds.
The shortcut branch is designed for efficient background feature processing and alignment. Specifically, it consists of only two linear layers, which is formulated
Lightweight Shorteut Branch.
- X + W
(13)
where Wdown E Rd and Wap E dxd, denote the froneprooection and up- up-projection matrices, respectively, with d, = d. The parameter count of the branch module is merely de + 2d, which is substantially smaller than the full transformer block (12d? + 13d).
The shorteut module is trained through low-cost reconstruction with small number of training samples. The reconstruction loss function is defined as:
= P3 F 1
(14)
where Fir is the feature from the original teacher model, and F is the feature from the student model with the shortcut branch. We adopt memory- conditioned features as the reconstruction tar- get for two key considerations. Directly reconstructing image embeddings would deviate from the
5

---

Published as a conference paper at ICLR 2026
shorteut branch's purpose, which is to provide efficient path for background feature alignment. Meanwhile, features from the mask decoder are ineffective due to its strong suppression effect on background information. Therefore, choose memory- condittond feature for its moderate back- ground information, enabling effective branch learning and robust feature alignment.
3.3 OBJECT-AWARY SPARSE MEMORY RETRIEVAL
We empirically observe two properties of memory attention: (i) the sparsity in memory bank: only small subset of memory tokens receives dominant attention; and (ii) the temporal consistency of saliency pattern: attention distributions over the same memory frame remain highly similar across consecutive video frames queries (FM,t, MM,++1........... FM....mmm.....
Let A E R denote the cross- -oos-nttenttion matrix at layer between image embedding FM and memory feature Mr where is the number of image tokens and is the number of memory tokens. The temporal consistency property is characterized by:
sim (A, 1, Vj (1,2, m,,,
(15)
where sim(:;) measures the cosine similarity.
Saliency Pattern Recognition in First Recollection. Leveraging these properties, we propose a layer-wise sparse memory retrieval scheme. Upon memory frame's first recollection in layer i.e., the cross-attention with M at moment I, we compute and cache its sparse pattern once, and subsequently reuse it later time steps avoid redundant computation. Specifically, given memory frame M with K tokens, its saliency pattern S/_1 is identified as follows:
s TopK (A/-1) L(1 - s) KJ),
(16)
where TopK(u, k) returns the indices of the top-k largest elements of vector v, A/_ E R repre- sents the averaged attention weights of M,-1 in layer l of the memory attention module, and is the predefined sparsity ratio. Then S added to the cached retrieval queue for memory retrieval in subsequent (m 1) time steps.
Saliency Pattern Reuse for Memory Retrieval Following a FIFO saliency pattern caching scheme, in current frame processing, the retrieval pattern queue S is formulated as follows:
= ( St S
(17)
which is used for sparse memory retrieval and updated at every time steps. Thereby, the sparse memory bank M is obtained as follows:
M {M M -i-m+1--.... M, (M-1),
(18)
M MM- Ist 2
(19)
where -sy_, E 1) is the sparse mask constructed from S and @ denotes clement-wise multiplication. Consequently, except for the important prompt frame M" kept unchanged, only the latest memory frame M,-1 maintains its complete form for sparsity pattern computation, while all other frames are distilled through layer-adaptive sparse retrieval. Compared to the dense memory bank Mr, the sparse representation M My ① M significantly reduces the computational complexity from O((m + Kdd to O(2NKd + (m 1)Nkd) in each memory attention layer, where :: = Ks < K, and d the feature dimension. Consequently. SMR reduce the memory redundancy well preserve important latest memory to ensure performance.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Models and Datasets. Our experiments incorporate SAM2.1-BB 1- and SAM2.1-L model with mem- ory intervals of and 5, denoted SAM2.1-B+ar. SAM2.1-B+aN SAM2.1-LAcst and 1AM2.1-LArse We follow the standard semi-supervised VOS protocol, where the ground-truth masks on the first frame are available and serve prompts. For evaluation, well-established

---

Published as a conference paper at ICLR 2026
Table Performance and speedup comparison of different acceleration methods the SAM2.1-B4 model with memory interval At=1 and At=5. The evaluation metric is J&.F. In 'IE' and 'MA' columns, 'v' indicates efficiency optimization for Image Encoder (IE) and Memory Attention (MA) components, while 'x' indicates original setting. Speedup Ratio is reported component level. 'x' denotes the distillation method. Our proposed method achieve significant speedup while maintaining competitive performance with the original SAM2 models.
SAM2, I-B+ Atm1
SAM2. I-B+ Aemb
Method
IF
MA
Speeduu
SA.V
SA.V
DAVIS
MOSE val
SA-V
SA-V
DAVIS
MOSR
test
2017 val
test
2017 val
val
Original
77.7
77.2
89.7
73.6
79.6
79.8
88.6
73.5
TTMe
/
x
1.36 x
55.3
55.0
74.5
53.5
57.5
57.2
74.3
54.0
1.05
67.0
ToMe4DM
/
x
1.00
62.7
63.8
80.9
59.5
63.
65.0
80.2
58.3
SWR(ours)
1.69
75.0
74.3
71.2
76.9
76.5
88.6
70.5
McmPool
x
2.14x
72.3
72.8
86.6
71.0
74.6
74.2
85.6
69.2
SMR-random
1.73x
72.2
77.4
73.1
SMR unilorm
X
1.78x
76.8
76.6
89.6
72.3
78.6
77.4
88.5
73.1
SMR(ours)
1.82x
77.8
77.4
89.7
73.2
79.4
78.4
88.6
73.5
EdgeTAM*
/
1.63x
72.1
71.8
86.4
69.9
73.0
73.4
86.2
69.5
Effcient-SAMZ(ous))
1.74x
75.5
70.9
76.6
88.5
70.1
Table 2: Results of different acceleration methods on the SAM2.1-L model with memory interval AL and Al Our methods demonstrates effective performance-speed trade-offs.
SAM2.1-L At=1
SAM2. I-L A125
Mothod
MA
Speedup
SA-V
DAVIS
MOSE
SA-V test
SA-V val
DAVIS
MOSE
test
val
2017 val
val
2017 val
val
Original
79.8
78.3
89.9
74.5
80.9
79.7
89.9
74.7
ToMe
1.43x
47,8
72.4
67.0
44.7
48.1
49.2
67.0
42.2
ALGM
/
x
1.76x
54.9
$3.5
88.4
67.0
56.5
55.7
87.6
66.1
ToMe4DM
x
1.14x
71.6
65.2
SWR(ours)
x
1883
79.0
76.4
89.9
73.4
79.8
77.7
90.7
73.4
MemPool
/
2.04x
72.6
73.2
87.8
71.3
74.4
74.9
87.2
69.8
SMR-random
X
1.75x
78.9
76.6
89.8
73.8
79.7
78.2
89.9
74.4
SMR-uuiform
73.9
74.5
SMR(ours)
1.78x
79.6
77.7
89.9
74.2
80.2
78.8
89.9
74.6
Emiciet-SAMZ(our
1.80%
78.8
75.5
89.7
72.6
79.0
77.7
90.7
73.1
VOS benchmarks: SA-V (Ravi al., 2024) test and validation sets, DAVIS 2017 (Pont-Tuset al., 2018) validation set, and MOSE (Ding et al., 2023) validation set. See Appendix A for details.
Baselines. We build three aspects of baselines for comprehensive comparison. First, for visual back- bone acceleration, we include token merge methods ToMe (Bolya al., 2022), ALGM (Norouzi et al., 2024), and ToMe4DM (Bolya & Hoffman, 2023). Second, for efficient memory attention, we implement the memory pooling (Xiong et al., 2024b) method (MemPool), and SMR variants with random and uniform masks (SMR-random and SMR-uninn uniforrf..).. Third, we include a distillation- based method Edge which compresses both image encoder and memory attention.
Implementation Details We adopt the default setting of original SAM2 with SAM2.1 checkpoints and follow standard inference pipeline of VOS. We perform inference GPU. The details of speed benchmark are provided in Appendix B. For SWR, we set prediction confidence Oorj 5, and the window saliency threshold T 0.7. For shortcut branch learning, we implement simple reconstruction pipeline based on the inference process to achieve adaptation and alignment with the backbone. This approach requires only 30 unlabeled samples from SA-V train dataset, and completes in approximately 1 hour on an RTX A6000 GPU. For memory retrieval, we apply sparsity ratio & = 0.95 each memory frame across memory attention layers. With prompt frame and latest frame maintained dense, the overall sparsity ratio achieves Se a 0.68. We provide details baseline reproduction Appendix
single NVIDIA A6000
4.2 MAIN RESULTS
Results of SAM2.1-B+ Models. Table presents performance and speedup comparisons of vari- acceleration methods on the SAM2.1-B+ model with memory intervals Al and Al 5.
7

---

Published as a conference paper at ICLR 2026
Our methods demonstrate excellent balance between performance and efficiency. SWR, which focuses on Image Encoder (IE) acceleration, achieves a 1.65x speedup ratio at At while main- taining strong performance. For example, in SA-V test the performance achieves 75.0, with only 2.7 drop from the original model. In DAVIS, it reaches 89.4, with only 0.3 accuracy drop. SMR, targeting Memory Attention (MA) optimization, delivers 1.82 speedup while preserving perfor- mance comparable to the original model, achieving 77.8 on SA-V test (merely 0.1 drop) and no accuracy drop on DAVIS. Collaboratively, our Efficient-SAM2 provides a 1.71x speedup at At with minimal performance degradation. In contrast, other methods such as ToMe offer acceleration but suffer significant performance drops, while MemPool achieves higher speedup x2) but with considerable performance loss.
Results SAM2.1-L Models. Table 2 illustrates the performance of acceleration methods on larger S-M..1-L model. On this larger model, our methods demonstrate even more significant advantages. SWR achieves 1.80x speedup A/=1, while maintaining performance nearly identical to the original model, with SA-V test score of 79.0 (only 0.8 lower than the original 79.8) and DAVIS score of 89.8 (just 0.1 below the original 89.9). SMR also perform excellently, reaching a 1.78 x speedup with performance metries close original model. Our Efficient- method provides a 1.75x x inference speedup while remaining competitive on key metrics. Notably, at At=5, our SWR method even surpasses than original model's s performance on DAVIS datasets (90.7 vs. 89.9), suggesting the potential of suppressing noises.
Across all models and datasets, experimental results with SMR and its variants (SMR-random and SMR-uniform) provide evidence for token-level sparsity hypothesis in SAM2's memory bank. Ablation studies with various sparsity ratios demonstrate the sparse memory configurations can maintain or even exceed the performance of the original dense memory.
4.3 ABLATION STUDY
Speedup Analysis. In Figure 4(a), our method accelerates both B+ and L variants without de- grading accuracy. For the Large model, we achieve module-wise speedups of up to 1.80 and 1.78x, with an end-to-end speedup of 1.68x. For the Base+ model, the two major modules reach 1.86x and 1.65x, and the over- all pipeline gains 1.59x, which shows balanced latency reduction. In end-to-end comparisons (Figure 4(b)), Efficient-S AifeiientSSAM2..11 SA....1 and -L shift the Pareto frontier, delivering markedly lower in- ference time at comparable or slightly better per- formance, while alternatives (ToMe, MemPool, EdgeTAM) achieve smaller speedups or incur larger accuracy drops. Thus, our SMR/SWR variants provide a superior speed-performance trade-off, achieving up 1.8x acceleration without sacrificing segmentation quality.
SAM2-L
Efficient-
SAM2-L
1.83x
1.78x
1,68x
SAM2-B+
IE
Efficient- SAM2-B+
ME+MD
1.67x
1.86x
1.59x
Others
50
100
150
20o
250
Inference Time (ma)
(a) Speedup Analysis of Modules
80
Ours-L
Ours-B+
SAM2-L
SAM2-B+
JBF on SA-V test
EdgeTAM
SAM2-B-
SAM2-L
ToMo-Bt
ALGM-L
SWR-BB+
SWR-L
MemPool-S+
MemPoolLL
SMR-B-
SMR-L
Ours-Br
Curs-L
Ablation for SWR. We compare the SWR shortcut variants, as shown in Table 3, which includes straight bypass without additional pro- cessing (Identity), attention-basee network (At- tention). standard Feedforward Network (FFN), and FFN with bottleneck setting (Bottleneck wins well- accuracy-speed trade-off. FFN). The three learned shorteut variants yield comparable improvement over Identity at low train- ing cost, underscoring the effectiveness of the proposed training pipeline. Among Bottleneck-FFF retains nearly all achievable accuracy while incurring substantially lower computational and pa- rameter overhead. Moreover, as shown in Table 4, we examine the effects of two enhancement components in Wpeed selection, ie.. including all predictions (All Pred.). and performing mask di- lation (Dilation). Including all predictions yields modest but consistent gains compared with using
100
150
200
250
End-to-End Inference Time (ms)
(b) Speed- PreeedPerformance Comparison
Figure 4: Detailed speedup analysis. Our method
8

---

Published as a conference paper at ICLR 2026
J&F Sparsity
SAM2 (Baseline)
Efficient-SAM (Ours)
Spensity
0.86
0.8
78-
Window Sparsilly
78.
Memor Sparsity
Object A
J&F on SA-V
0.50
0.8
75-
0.75
0,2
74
0.70
JBF
Object B
0.08
73
0.0
0.1
0.5
0.7
0.5 0.7
0.95
(a) Grid Search for r
(b) Grid Search for
Figure 5: Sparsity analysis of SWR and SMR. As grows, the window sparsity decreases but stays over 0.6, while the accuracy steadily increases. As the memory sparsity in- creases, the performance surprisingly surpasses the baseline and then declines,witt s=0.95 offers a satisfied trade-off.
Figure 6: SWR preserve attention focus on object-relevant regions in the Image Encoder, while blur- ring attention to unimportant back- ground regions.
only prediction with maximum Siou (e.g., +0.5 on SA-V test, +0.7 on MOSE), suggesting improved distractor awareness. Dilation produces larger increments (e.g.. +1.4 on SA-V test, +1.6 on MOSE).
Table Comparison of shortcut variants in SWR. All trainable branches outperform the Identity, with Bottleneck-FFN offers parameter-effiieett yet effective option.
Table Ablation study of SWR components. Including all predictions (All Pred.) outper- forms single prediction; Performing dilation exhibits stable improvement.
SWR
Param. (M)
Training Hours
SWR
SAM2. I-B+ (At 1)
Model
J&F
Shortcut
SA-V
SA-V
DAVIS
MOSE
Identity
73.0
All Pred.
Dilation
2017 val
val
Attention
0.35
1.05
75.2
SAM2.1-B4
FFN
0.81
1.10
75.0
73.6
73.7
69.5
BoRteNeck-PFN
0.20
0.97
75.0
74.1
73.8
89.2
70.2
x
75.0
73.8
89.5
71.1
Identity
0
75.7
75.0
74.3
89.4
71.2
Attention
0.58
1.15
78.5
SAM2.1-L
Original (Bascline)
1.33
79.2
77.7
77.2
89.7
73.6
0.33
1.10
79.0
Sparsity Analysis. For SWR, window spar- sity is defined as the ratio of the number of Way to total windows (25 for Base+ and 16 for Large), which depends on adaptive object oc- cupancy and the tunable saliency threshold т. We reports the average window sparsity and the corresponding module speedup across datasets (Table 5), showing the cross-dataset robustness of the module speedup ratio. Furthermore, A grid search for (Figure 5(a)) indicates that r=0.7 provides a balanced trade-off between accuracy and sparsity. For SMR, the memory spar- sity is defined as 7. A grid search for (Figure 5(b)) shows the accuracy remains above the dense baseline until s zs 0.95, confirming memory redundancy and validating the effectiveness of SWR.
Table 5: Cross-dataset analysis of SWR module. with content-adaptive window sparsity, SWR tains consistent speedups across datasets.
SAM2.1-R+
SAM2 I-I.
Dataset
# Videos
Sparsity
Specdup Sparsity
Spoodup
SA-V test
150
0.80
3.12x
0.75
3.24x
SA-V
155
0.77
2.96x
0.74
3.14x
DAVIS 2017 val
30
0.61
2.17x
0.50
2.03x
MOSE val
2.48x
5 CONCLUSION
In this paper, we present Efficient-SAMM2. a post-training acceleration framework for SAM2 that addresses the mismatch between SAM2's inherently sparse perception patterns and its dense pro- cessing pipeline. By examining attention concentration in mask decoder and temporal consistency of salient memory tokens, we identify two key inefficiencies and introduce two lightweight, object- aware components to resolve them. SWR reallocates computation at the window level by routing background windows, identified through previous-frame saliency and decoder-derived consistency signals, into lightweight shortcut branch, while preserving computation for object-relevant regions. SMR confines memory attention to salient tokens only, caching each memory frame's 's saliency pat- tern upon first use and reapplying it across subsequent frames, thereby avoiding redundant recom- putation. Experiments across multiple VOS benchmarks demonstrate that Afficient-SAMe achieves superior accuracy-effciency ofade-off without expensive retraining or architectural redesign.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported in part by the Strategic Priority Research Program of Chinese Academy of Sciences under Grant Number XDB 100000; in part by the National Natural Science Foundation of China under Grant Number 62276255; in part by the Postdoctoral Fellowship Program of CPSF under Grant Number G2C20251175.
REFERENCES
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the /EEF/CVF conference on computer vision and pattern recognition, pp. 4599-4603, 2023.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
Daniel Bolya, Chaitanya Ryali, Judy Hoffman, and Christoph Feichtenhofer. Window attention is bugged: How not to interpolate position embeddings. In ICLR, 2024.
Nicolas Carion, Laura Gustafson, Tunn-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:25 1...111117199 2025.
Yi-Chia Chen, Wei-Hua Li, Cheng Sun, Yu-Chiang Frank Wang, and Chu-Song Chen. Sam4mllm: Enhance multi-modal large language model for referring expression segmentation. In European Conference Computer Vision, pp. 323 233-300. Springer, 2024.
Claudia Cuttano, Gabriele Trivigno, Gabriele Rosi, Carlo Masone, and Giuseppe Averta. Samwise: Infusing wisdom in sam2 for text-driven video segmentation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. -995--305, 2025.
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip H. S. Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. 2023 JEEE/CVF Interna- lional Conference on Computer Vision (ICCV), pp. 20167-20177, 2023. URL httpp:://aa semant cscholasc aaroolrroooo org/ Cttp//orpussseee TD: 256598368.
Shuangrui Ding, Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Yuwei Guo, Dahua Lin, and Jiaqi Wang. Sam2long: Enhancing sam 2 for long video segmentation with training- 2024.
free memory tree. arXiv preprint arXiv:2410.
Zhang Jiaxing and Tang Hao. Sam2 for image and video segmentation: A comprehensive survey. arXiv preprint arXiv:2503.1 12.5033..771/ 12781, 2025.
Daniel Kienzle, Marco Kantonis, Robin Schön, and Rainer Lienhart. Segforme+++: Efficient token- merging strategies for high-resolution semantic segmentation. In 2024 IEEE 7th International Conference Multimedia Information Processing and Retrieval (MIPR), pp. 75-81. IEEE, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 4015-4026, 2023.
Yanghao Li, Chao- Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022a.
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recogni- tion, pp. 4804-4814, 2022b.
Zhikai Li and Qingyi Gu. I-vit: Integer-only quantization for efficient vision transformer infer- ence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17065- 17075, 2023.
10

---

Published as a conference paper at ICLR 2026
Zhikai Li, Liping Ma, Mengjuan Chen, Junrui Xiao, and Qingyi Gu. Patch similarity aware data-free quantization for vision transformers. In European conference on computer vision, pp. 154-170. Springer, 2022c.
Zhikai Li, Mengjuan Chen, Junrui Xiao, and Qingyi Gu. Psaq-vit v2: Toward accurate and gen- eral data-free quantization for vision transformers. IEEE Transactions on Neural Networks and Learning Systems, 2023a.
Zhikai Li, Xiaoxuan Liu, Banghua Zhu, Zhen Dong, Qingyi Gu, and Kurt Keutzer. Qft: Quantized full-parameter tuning of Ilms with affordable resources, arXiv preprint arXiv:2310.07147, 2023b.
Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17227-17236, 2023c.
Zhikai Li, Xuewen Liu, Jing Zhang, and Qingyi Gu. Repquant: Towards accurate post-
training quantization of large transformer models via scale reparameterization. arXiv preprint arXiv:2402.05668, 2024a.
Zhikai Li, Jing Zhang, and Qingyi Gu. Privacy-preserving sam quantization for efficient edge intel- ligence in healthcare. arXiv preprint arXiv:2410.01813. 2024b.
Haofeng Liu, Erli Zhang, Junde Wu, Mingxuan Hong, and Yueming Jin. Surgical sam Real-time
segment anything in surgical video by efficient frame pruning. arXiv preprint arXiv:2408.079. 2024a.
Xuewen Liu, Zhikai Li, and Qingyi Gu. Dilatequant: Accurate and efficient diffusion quantization via weight dilation. arXiv preprint arXiv:2409. /4307, 2024b.
Xuewen Liu, Zhikai Li, and Qingyi Gu. Cachequant: Comprchensively accelerated diffusion mod- els. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 23269- 23280, 2025.
Xuewen Liu, Zhikai Li, Jing Zhang, Mengjuan Chen, and Qingyi Gu. Ptq4arvg: Post-training quantization for autoregressive visual generation models. arXiv preprint arXhv:2601.21238 2026.
Chengtao Lv, Hong Chen, Jinyang Guo, Yifu Ding, and Xianglong Liu. Ptq4sam: Post-training quantization for segment anything. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 15941-159511 2024.
Narges Norouzi, Svetlana Orlova, Daan De Geus, and Gijs Dubbelman. Algm: Adaptive local- then-global token merging for efficient semantic segmentation with plain vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15773-15782, 2024.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Dorkine-Hornung and Lue Van Gool. The 2017 davis challenge on video object segmentation, 2018. URL https: //arxiv.org/aaaeeee.eeee 1170417400000 00675.
Nikhila Ravi, Valentin Gabeur, Yuan-Tine Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 007 2024.
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Ag- garwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al. Hiera: A hierarchical vision transformer without the bells-and-whistle In International conference machine leam- ing, pp. 29441-222222 29444. PMLR, 2023.
Xiaorui Sun, Jun Liu, Hengtao Shen, Xiaofeng Zhu, and Ping Hu. On efficient variants of segment anything model: A survey: X. sun et al. International Journal of Computer Vision, 133(10): 7406-7436, 2025.
11

---

Published as a conference paper at ICLR 2026
Jovana Videnovic, Alan Lukezic, and Matej Kristan. A distractor-awart memory for visual object tracking with sam2. In Proceedings of the Computer Vision and Patter Recognition Conference, pp. 14255-24264, 2025.
Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiao- liang Dai, Dilin Wang, Fei Sun, Forrest landola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16111-16121, 2024a.
Yunyang Xiong, Chong Zhou, Xiaoyu Xiang, Lemeng Wu, Chenchen Zhu, Zechun Liu, Saksham Suri, Balakrishnan Varadarajan, Ramya Akula, Forrest landola, et al. Efficient track anything. arXiv preprint arXiv:2411.18933, 2024b.
Cheng-Yes Yen Yang, Hsiang-Wae Wei Huang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Samu- rai: Adapting segment anything model for zero-shot visual tracking with motion-aware memory. arXiv preprini arXi::2411.1192. 2024.
Jing Zhang, Zhikai Li, Chengzhi Hu, Xuewen Liu, and Qingyi Gu. Saq-sam: Semantically-aligne quantization for segment anything model. arXiv preprint arXiv:2503.065/5, 2025.
Chong Zhou, Chenchen Zhu, Yunyang Xiong, Saksham Suri, Fanyi Xiao, Lemeng Wu, Raghuraman Krishnamoorthi, Bo Dai, Chen Change Loy, Vikas Chandra, et al. Edgetam: On-device track anything model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 13842, 2025.
Junbao Zhou, Ziqi Pang, and Yu-Xiong Wang. Rmem: Restricted memory banks improve video ob- ject segmentation. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16602-1861, 2024.
12

---

Published as a conference paper at ICLR 2026
A EVALUATION BENCHMARKS
SA-V (Ravi al., 2024) The SA-V dataset consists of 50.9K video clips with 642.6K masklets for video segmentation tasks. Its validation set contains 293 masklets across 155 videos, while the testing set comprises 278 masklets from 150 videos. Both evaluation sets feature challenging scenarios including small objects, occlusions, and reappearing targets, providing a robust benchmark for assessing video segmentation models.
DAVIS (Pont-Tuset et al., 2018) is a widely-used benchmark for video object segmentation, consist- ing of 50 high-quality video sequences for training and 30 sequences for validation. Each sequence is annotated with pixel-accurate ground truth masks, featuring diverse object categories and chal- lenging scenarios such as occlusions and appearance changes. The dataset has evolved through multiple versions, with DAVIS 2017 extending the benchmark to multi-object scenarios.
MOSE (Ding al., 2023) dataset contains 2,149 video clips spanning 36 object categories, with total of 431,725 high-quality segmentation masks. Specifically designed for complex motion sce- narios, it includes 5,200 annotated objects. The dataset has demonstrated its value through extensive evaluation of state-of-the methods.
B SPEEDUP BENCHMARK
The speedup evaluation are conducted RTX A6000 GPU Float32 precision. For fair com- parison, we use 20 videos randomly sampled from SA-V test set, including single object and multi- object cases. We choose float32 as the default precision for speed evaluation rather than bfloat 16 (BF16) because, on our tested hardware and software stack, the actual acceleration of the BF16 path falls far short of theoretical expectations, yielding little to no end-to-end gains even under aggressive sparsification. For example, even after dropping about 80% of windows and 68% of memory tokens, the end-to-end inference time remains essentially unchanged. We attribute this to the pipeline be- ing dominated by GPU-side irregular memory access and control-flow overheads, rather than dense operators that can be efficiently accelerated by tensor cores. To ensure fair comparison and to highlight the genuine end-to-end benefits of algorithmic sparsification, therefore report speed with float32 as the baseline.
We provide module-level speedup for all baselines in Table I and Table 2, where the speedup are computed based on afficiency-optimizee module. For example, the reported IE speedup is the Image encoder inference time relative original model, including the additional computational cost intro- duced by methods. For some methods, e.g., ALGM, ToMe4DM, the speedup results fall far below theoretical expectations. We hypothesize that this situation arises because the additional computa- tion introduced increases the CPU scheduling overhead, resulting in insignificant accelcration for GPU computation bottlenecks. Specifically, we observed that EdgeTAM'ss VOS inference pipeline differs from the previous testing methodology. To ensure fair comparison, we separately evalu- ated the acceleration ratio relative to the original model BFloat precision. The Table reports module-level acceleration ratios, specifically the inference time ratios of the image encoder, memory attention, and the introduced memory perceiver compared to the original SAM2-Bases model.
We also shows end-to-end speed analysis in Figure 4, where the number of EdgeTTM is converted based on the speedup ratio to align with the numerical ranges of other methods. We attribute this gap not to insufficient GPU compute, but to pipeline overheads introduced by routing/merging and sparsification: the workload shifts from compute-boune dense kernels to memory- and control- bound execution with irregular indexing and seatter scatter-gather.
Table 5 shows fine- -raeeggr sub-module level speedup across different dataset. As shown in Figure 3, our SWR accelerate only deeper final stage layers of image encoder, which account for 60% of total time overhead in the image encoder. This metric provides the most direct reflection of how sparsity affects the acceleration ratio.
13

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS OF BASELINES
In our experiments, we implemented and evaluated various acceleration methods on SAM2.1 models with different configurations. For all experiments, we maintained consistent settings across different memory intervals (At=1 and At=5) to ensure fair comparisons.
ToMe: We implemented ToMe with different pruning ratios, 0.2 for SAM2.1-B+ and 0.1 for SAM2.I-L for comparable speedup ratio. To adapt to segmentation task, we prunes a fixed proportion of tokens at each layer and recover spatial structure through unmerge operations in the final layer.
ToMe4DM:
We set the pruning ratio to 0.5 for this method. However, its acceleration effect was limited because ToMe4DMM requires matching, merge, and unmerge operations each layer, introducing significant computational overhead. This explains why it only achieved a 1.00x speedup on SAM2. I-++, effectively providing no acceleration.
ALGM: We implemented ALGM following its original two-stage merging approach for segmentation models, which includes local conditional pooling and global merging with adaptive pruning rates, followed by unmerging restore spatial structure. We set the sim- ilarity threshold for local merging to 0.7. Unlike the original method, we performed two rounds of global merging to increase the pruning rate. Despite these optimizations, ALGM showed limited acceleration in our experiments (only 1.05x on SAM2.1-B+).
MemPool: For the memory pooling method, we configured the pooling kernel size as 2x2. While this method achieved the highest speedup ratios (2.14x on SAM2.I-B4 and 2.04 on SAM2. I-L), it also showed notable performance degradation, particularly on DAVIS and SA-V test sets.
D QUALITATIVE COMPARISON AND ANALYSIS
We show tracking instance with different prompt in Figure Efficcient-AAM2 show comparable performance with original SAM2 model. Figure 7(c) shows the contribution of dilation operation, where motion induced boundary escape leads to excessively focused attention on ambiguous area, degrading attention to other important area.
EfficlentSAAE
SAM2 (Basaline]
Efficient-
SAM2
EnficientSAM22 (Ouns
Efficient-
Object B Object A
Mask Prediction
Image Encoder
Attention Mask Decoder
Figure 7: Qualitative comparison and analysis. (a) shows Efficient-SAM22 comparable segmenta- tion quality with original SAM2 model. (b) and (c) illustrate the object-focusing effect of proposed SWR. while SWR blurs background regions, it preserves object-relevant responses in the image encoder, and maintains consistent attention focus in the mask decoder.
E FURTHER DISCUSSION OF EFFICIENT-SAM2
In our SWR, window routing is used to accelerate the deeper backbone (i.e., stage 2), achieving an excellent accuracy accuracy-efffciency balance. We also attempted to extend the SWR scheme to accelerate the shallow stages inference (which account for about 40% the backbone's 's compute latency) and found that SWR can also adapt to the window structure in shallow layers. As shown in Table 6, accelerating more layers yields greater efficiency gains; however, due to the lack of initial aggrega- tion for background windows, this variant incurs adopt it. Nevertheless, we believe that with an improved design of the bypass network, this variant has the potential further recover accuracy, which we leave for future work.
pronounced accuracy drop.
did not
14

---

Published as a conference paper at ICLR 2026
Table 6: Results of Efficient-SAMM variants. denotes extend SWR to all encoder stages, which provides more efficiency gain but compromises performance.
SAM2.1- B+A
SAM2.1-Lasa.
Method
MA
SA.V
Speedup
Speedup
lest
val
2017 val
val
lest
2017 val
Original
77.7
77.2
89.7
73.6
79.8
89.9
SWR SWRt
74.3
71.2
79.0
74.0
72.0
88.3
69.3
2.62
77.4
75.0
89.7
73.2
3,17
EfficiontSAMM. SAM2
75.5
74.5
89.3
70.9
1.74
78.8
75.5
89.7
72.6
1,80
74.2
72.8
88.0
69.5
2.39
76.5
74.9
89.7
72.8
2.67
SHORTCUT TRAINING DETAILS.
We present detailed training settings of shorteut branch in Table
Table 7: Training details of shortcut branch.
config
value
Data
Selected from SA-V train
# Videos
30
Video Length
<300
Optimizer
Adam W
Sampling Stride
3 for base+, 4 for Large
Learning Rate
0.0001
Total Steps
495
Epochs
3
Update stride
32
Memory stride
{1,3,5) for epoch 0,1,2
15


Reviews:
[{"review_id": "m9cEBVF85c", "reviewer": "Reviewer_DdRT", "summary": "The paper aims to reduce the high inference cost of SAM2 in video segmentation. Specifically, they propose two modules: Object-aware Sparse Window Routing (SWR), which skips background windows in the image encoder based on object masks and saliency, and Object-aware Sparse Memory Retrieval (SMR), which selects only salient memory tokens and reuses their mask across frames. Together, these modules accelerate SAM2 by up to 1.75× with minimal accuracy loss on benchmarks such as SA-V, DAVIS, and MOSE.", "questions": "1. What is the specific layer index after which the router divides foreground and background tokens? How many subsequent modules benefit from reduced computation? It would be helpful to include an ablation varying the routing layer index to assess its impact.\n\n2. How is speedup defined in this paper? It would be clearer to disclose it in two aspects—FLOPs reduction and throughput improvement—to give a more complete view of efficiency.\n\n3. What is the intuition behind using two different temporal intervals (∆t =1,5) in the experiments? It seems the performance difference might largely stem from the frame rate (FPS) of the original benchmark. For high-FPS datasets like SA-V, increasing the interval could naturally yield greater gains since the memory bank contains more diverse frames.\n\n4. For SMR, have you evaluated a variant that selects memory frames only when object presence is confident (akin to SAM2Long’s strategy)? It would be informative to report the gain from this filtering.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper makes a solid technical contribution by streamlining the SAM2 model. The object-aware pruning of the image encoder and the introduction of a background shortcut for non-foreground patches are both clever ideas that substantially reduce computation.\n\n2. The ablation study is comprehensive. It not only analyzes the proposed components in isolation but also integrates other efficient methods (e.g., ToME) into their framework for comparison, which provides valuable insights.", "weaknesses": "1. The proposed routing mechanism heavily relies on the assumption of temporal consistency in video streams, meaning that no significant camera shaking or viewpoint shift occurs. This limits the method’s applicability in real-world scenarios with dynamic motion. It would be interesting to see comparisons with SAM2 on datasets such as MOSEv2[1] and SeCVOS[2], which feature frequent viewpoint transitions.\n\n\n\n\n2. Performing grid search on only one benchmark is not sufficient to demonstrate robustness. It would strengthen the paper to include additional grid search curves across multiple benchmarks (in Figure 5).\n\n\n\n\n\n\n\n[1] MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenes\n[2] SeC: Advancing Complex Video Object Segmentation via Progressive Concept Construction", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3gBwG9PwUq", "reviewer": "Reviewer_gSJw", "summary": "This paper proposes Efficient-SAM2, a post-training acceleration framework to address the significant computational bottleneck of SAM2 in real-time video object segmentation. The authors identify a mismatch between SAM2's dense computation and its inherently sparse perception, highlighting redundancy in the image encoder's background processing and in full-token memory retrieval. To exploit this, the framework introduces two key components: object-aware Sparse Window Routing (SWR) and object-aware Sparse Memory Retrieval (SMR). SWR dynamically routes irrelevant background windows in the encoder to a lightweight shortcut branch, guided by saliency and consistency cues from the previous frame's decoder. SMR leverages temporal consistency by identifying a sparse set of salient memory tokens during their first recollection and reusing this pattern for subsequent frames, drastically reducing memory attention computations.", "questions": "While the paper presents a valuable approach to making SAM2 more efficient, the design of the SWR and SMR modules appears highly heuristic and is tightly coupled to assumptions about temporal continuity. This raises a significant question: Is SAM2's strong, general-purpose segmentation performance fully preserved? The evaluation is currently confined to standard VOS benchmarks. To truly validate that these heuristics do not compromise the model's robustness, I would ask the authors to provide evaluations on more diverse datasets, particularly on challenging \"in-the-wild\" videos, which would better test the limits of these heuristic assumptions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation for reducing SAM2's computational overhead is well-grounded and intuitive for video object segmentation\n2. The post-training approach is practical, enabling efficient adaptation by leveraging the generalized parameters of the pre-trained SAM2.\n3. The method achieves a good speed-performance trade-off, delivering a speedup of nearly 2x while incurring only a minimal and acceptable performance degradation of approximately 1%.", "weaknesses": "1. The SWR component is heavily dependent on the previous frame's prediction and salient mask. I think this may cause challenges in some cases, such as rapid motion, abrupt scene cuts, or severe occlusions, where this temporal assumption would be violated.\n2. For a video domain paper, the qualitative results with static images are insufficient. Supplemental videos would be significantly stronger to properly demonstrate temporal consistency, failure modes (especially in scenarios mentioned in point 1), and the practical impact of the optimizations.\n3. I think an evaluation is needed to determine if the SMR module, which uses a cached saliency pattern, maintains its effectiveness in long video scenarios where significant appearance and context drift are likely.\n4. The paper lacks a dedicated discussion of its limitations. This omission leaves the impression that the proposed efficiencies might be confined to easy (or trained) scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsIhiUrQni", "reviewer": "Reviewer_FsUz", "summary": "The paper addresses the key motivation of resolving SAM2’s high computational burden that limits its real-time application in video object segmentation (VOS). The  proposed Efficient-SAM2, a post-training acceleration framework, has two core components: 1) Object-Aware Sparse Window Routing (SWR) for the image encoder, which leverages spatial-temporal consistency and perceptual saliency from the previous frame’s mask decoder to route background windows to a lightweight shortcut branch  and preserve full computation for object-relevant windows,2) Object-Aware Sparse Memory Retrieval (SMR) for memory attention, which caches each memory frame’s saliency pattern during its first recollection and reuses it in subsequent frames to only involve salient tokens in computation. Experimentally, Efficient-SAM2 achieves a 1.68× end-to-end speedup on the SAM2.1-L model with only a 1.0% accuracy drop on the SA-V test set.", "questions": "1.What's the actual theory that the equation (11) can reflect the saliency of a window? It should be explained.\n   \n 2.How can the object-aware router adapt to the fast moving object since all the information of prediction and saliency are from the preceding frame?\n   \n 3.Mask decoder module seems to disappear from the figure 3. How is the segmentation mask produced?\n   \n 4.What is the alignment of  background feature processed by lightweight shortcut branch?\n   \n 5.How would Efficient-SAM2 adapt to dynamic prompts (e.g., user-added points in middle frames) that change the focus of attention?\n   \n 6.Can you provide statistical data (e.g., average CS across all frames in SA-V/DAVIS) to quantify how often saliency patterns remain consistent? What is the impact of inconsistent patterns on SMR’s accuracy?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.  Efficient-SAM2 avoids expensive full-model retraining. It adds negligible parameters  and low training overhead, making it flexible for low computational deployment.\n2.  By aligning with SAM2’s natural sparse perception, it eliminates redundancy without compromising core functionality—unlike generic token-merging methods (e.g., ToMe) that cause severe accuracy drops.\n3.  SWR (image encoder) and SMR (memory attention) are independent modules, allowing separate optimization or integration with other SAM2 variants. Their design (window-level routing, cached saliency patterns) is intuitive and supported by qualitative analysis (e.g., Figure 6 shows SWR preserves object attention).\n4.  It maintains strong accuracy across diverse VOS datasets (SA-V, DAVIS 2017, MOSE) and scales to larger models (SAM2.1-L).", "weaknesses": "1. The claimed contribution 1 should be merge with contribution 2 as whole one.\n\n\n2. SWR relies on hyperparameters like the prediction confidence threshold (θₒᵦⱼ=0.5) and saliency threshold (τ=0.7), while SMR depends on the sparsity ratio (s=0.95). The paper does not explore how these parameters generalize to edge cases (e.g., highly cluttered scenes, fast-moving objects) or different datasets.\n\n\n3. The variable symbols cause confusions, especially for different A.\n\n\n4. The codes released as supplementary material fail to run according to SAM-2 environment. The clarity of the code is also limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)