Polyline Path Masked Attention for Vision Transformer
Authors: Zhongchen Zhao
Chaodong Xiao
Hui LIN
Qi Xie
Lei Zhang
Deyu Meng
Venue: NeurIPS 2025 spotlight ()
Keywords: Mamba2
Vision Transformer
Polyline Path Masked Attention
Primary area: deep_learning
TL;DR: We adaptes Mamba2's structured mask to 2D scaning and integrates it into the self-attention mechanism of ViTs as an explicit positional encoding.

Abstract: Global dependency modeling and spatial position modeling are two core issues of the foundational architecture design in current deep learning frameworks. Recently, Vision Transformers (ViTs) have achieved remarkable success in computer vision, leveraging the powerful global dependency modeling capability of the self-attention mechanism. Furthermore, Mamba2 has demonstrated its significant potential in natural language processing tasks by explicitly modeling the spatial adjacency prior through the structured mask. In this paper, we propose Polyline Path Masked Attention (PPMA) that integrates the self-attention mechanism of ViTs with an enhanced structured mask of Mamba2, harnessing the complementary strengths of both architectures. Specifically, we first ameliorate the traditional structured mask of Mamba2 by introducing a 2D polyline path scanning strategy and derive its corresponding structured mask, polyline path mask, which better preserves the adjacency relationships among image tokens. Notably, we conduct a thorough theoretical analysis on the structural characteristics of the proposed polyline path mask and design an efficient algorithm for the computation of the polyline path mask. Next, we embed the polyline path mask into the self-attention mechanism of ViTs, enabling explicit modeling of spatial adjacency prior. Extensive experiments on standard benchmarks, including image classification, object detection, and segmentation, demonstrate that our model outperforms previous state-of-the-art approaches based on both state-space models and Transformers. For example, our proposed PPMA-T/S/B models achieve 48.7%/51.1%/52.3% mIoU on the ADE20K semantic segmentation task, surpassing RMT-T/S/B by 0.7%/1.3%/0.3%, respectively. Code is available at https://github.com/zhongchenzhao/PPMA.

Polyline Path Masked Attention for Vision Transformer
Zhongchen Zhao', Chaodong Xiao?.3, Hui Lin', Xie Lei Zhang 2.3, Deyu Meng" 1.1
Xi'an Jiaotong University The Hong Kong Polytechnic University OPPO Research "aazhou Laboratory (Huangpu) Institute zhongchenzhao@ettu. xj tu.euu.. en, xie. qi @maee. edu. cn
Abstract
Global dependency modeling and spatial position modeling are core issues of the foundational architecture design in current deep learning frameworks. Recently, Vision Transformers (ViTs) have achieved remarkable success in computer vision, leveraging the powerful global dependency modeling capability of the self-attention mechanism. Furthermore, Mamba2 has demonstrated its significant potential in natural language processing tasks by explicitly modeling the spatial adjacency prior through the structured mask. In this paper, we propose Polyline Path Masked Attention (PPMA) that integrates the self-attention mechanism of ViTs with an enhanced structured mask of Mamba2, harnessing the complementary strengths of both architectures. Specifically, we first ameliorate the traditional structured mask of Mamba2 by introducing a 2D polyline path scanning strategy and derive its corresponding structured mask, polyline path mask, which better preserves the adjacency relationships among image tokens. Notably, we conduct a thorough theo- retical analysis the structural characteristics of the proposed polyline path mask and design an efficient algorithm for the computation of the polyline path mask. Next, embed the polyline path mask into the self-attention mechanism of ViTs, enabling explicit modeling of spatial adjacency prior. Extensive experiments on standard benchmarks, including image classification, object detection, and segmen- tation, demonstrate that our model outperforms previous state-of-the-an approaches based on both state-space models and Transformers. For example, our proposed PPMA-T/SIB models achieve 48.7 %/11. 1 % /52.3% % mloU on the ADE20K seman- tic segmentation task, surpassing RMT-T/S/B by 0.7% respectively. Code is available at wtlhtps::/// htt.://ww..ooooooooooo/e////a//a
1 Introduction
The research of foundational models has long been cornerstone of deep learning. In computer vision, Convolutional Neural Networks (CNNs) 20 19 and Vision Transformers (ViTs) 46 31 22
currently represent the dominant architectures. Notably, ViT's have become the most mainstream architecture in large models through the powerful self-attention mechanism, which can capture the non-local self-similarity within global receptive fields. However, the quadratic complexity of the Transformer, when implementing self-attention, severely limits its application in large image processing models. Moreover, as shown in Fig. (b), classic positional encoding methods [46] 31 38]
ViTs lack the explicit modeling capability of spatial distance between image tokens, and largely ignore the important spatial adjacency priors in texture, shape, semantics, and so on. This increases the learning pressure and limits its capability for fine- tineggrained image feature extraction.
Compared to CNNs and Transformers, the recently proposed Mamba 11 achieves linear complexity while maintaining global receptive fields, demonstrating strong potential as the next-generation archi- tecture. Specifically, Mamba follows the State Space Models (SSMs) paradigm and employs the se- lective scan mechanism with the state transition matrix to recursively propagate dependencies among
"oorresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tokens in a sequence. Building on
Cirmre Staminn per
Playline Scamning poneeorennn
this foundation, Mamba2 [6] fur- ther refines the state transition ma- trix into a lightweight structured mask and introduces unified theo- retical framework, Structured State Space Duality (SSD), to bridge SSMs and attention variants. Un- der SSD, the core selective scan mechanism of Mamba2 can be re- formulated as form of structured masked attention, iee, a Linear Attention [33 element-wise multi- plied by the structured mask, as il- lustrated in Fig. (a). Notably, this structured mask explicitly encodes the sequence adjacency of tokens, enabling Mamba2 to match or surpass Transformers across various natural language processing (NLP) tasks. Following its success in NLP, Mamba [ has been rapidly adapted to various visual
Structured Mask
ID Spotial
Adjocecc Modeling
Mamba2
Linear Attention
Scanning Strategy
Weak Globel Dependency Modeling Weak Non-l incar Modoling
Theory
Improvement
Positional Encoding
Playline Path Mask
Weak Sputial Adjacency Madeling
2D Spaital
Adjucency Modcling
VIT
Self-Attention
Strong Glcbal Deperdenzy Modeling
Strong Noo-Lincse Modeling
PPMA
(C)
Figure 1: (a)-(b) Illustration of the modules in Mamba2 and ViT. (c) Our method adapts the structured mask of Mamba2 to 2D scanning and integrates it with ViT's self-attention.
domains, including: high-level tasks (classification, object detection, segmentation [58] 30 49 47)
)
low-level tasks (super-resolution, denoising, deraining [ 12 59 image generation
video
analysis 26), point cloud analysis [27 51, and remote sensing images
(54]
Although Mamba [11 has demonstrated impressive results on certain vision tasks, empirical results on
high-level vision tasks demonstrate that even state-of-the-art (SOTA) Mamba-based backbones (30
49
47 15] still underperform SOTA Transformer-based backbones (57 10] with substantial performance gap. As shown in Fig. (aa.
(a). this gap mainly stems from two issues: (I) 1D Scanning Issue. Mamba's ID scanning strategy arranges the tokens of 2D image into a ID sequence, which inevitably disrupts the inherent spatial adjacency within 2D images and limits the effectiveness of its recursive selective scanning mechanism. (II) Weak Global Dependency Modeling Issue. The linear attention in Mamba2 omnts the non- non-linear softmax layer, leading to a decrease in the precision and stability of global dependency modeling of images.
In this paper, we present Polyline Path Masked Attention (PPMA), a brand-new method that effectively combines the advantages of ViTs and Mamba2. Specifically, illustrated in Fig. (c), to address the ID Scamning Issue when applying current Mamba to 2D images, we propose a novel 2D polyline path scanning strategy and derive an efficient calculation method for its corresponding structured mask, the polyline path mask. Then, we embed the polyline path mask as an explicit positional encoding into the ViT framework. This not only avoids the Weak Global Dependency Modeling Issue of Mamba2, but also alleviates the positional encoding issue of ViTs. As a result, our method fully leverages the powerful global context modeling capability of the self-attention mechanism in ViTs together with the explicit spatial adjacency modeling capability of the polyline path mask inspired by Mamba2, achieving SOTA performance on mainstream high-level vision tasks.
To the best of our knowledge, this is the first work to integrate Mamba2's 's structured mask mechanism into ViTs. The main contributions of this study are summarized follows:
We propose 2D polyline path scanning strategy for visual Mamba, which better preserves the inherent 2D spatial structure of images compared to existing scanning strategies. Building on this, we further derive a novel structured mask, termed polyline path mask, which is more suitable for 2D images than the traditional structured mask used in Mamba2.
We conduct a comprehensive theoretical analysis for the proposed 2D polyline path mask. Specifically, we theoretically prove that it can be decomposed into two 1D structured masks with clear physical meanings (i.e., horizontal and vertical scanning masks). More importantly, by leveraging this decomposability, we derive an efficient a algorithm to reduce its computational complexity from ( in the naive calculation to (N
The polyline path mask can be seamlessly integrated into various attention variants in a plug- and-play manner without introducing a substantial increase in computational complexity. In this paper, we incorporate into the vanilla self-attention and criss-cross attention, deriving the Polyline Path Masked Attention (PPMA).

---

Leveraging PPMA, we construct a hybrid Tranbf--rrasorrmee model. Experimental results demonstrate that our model achieves SOTA performance on standard benehmarks for image classification, object detection, and segmentation.
2 Related Work
Vision Transformers. ViTs have become foundational in large-scale vision models such as SAM 24] and Sora I1. primarily due to their self-attention mechanism that effectively captures the long-range dependency. Moreover, the spatial structural information provided by positional encodings (c.g., APE 46) RPE [31], and RoPE (38)) is also crucial to VITs. However, traditional positional encodings fail to explicitly encode spatial adjacency. Recent works, such as RMT [0] and VVT [39], porpose to incorporate RetNet's input-independee
temporal decay mask 440 into VITs for more explicit spatial modeling based on the Manhattan distance. comparison, Mamba2's ' input-dependent selective structured mask not only explicitly encodes the relative positional information in the spatial space but also captures the semantic continuity in the feature space.
Mamba. As state space model, Mamba introduces an input-dependent selection mechanism into the state transition matrix A, achieving Transformer-level performance with linear complexity on NLP tasks. Building on this foundation, Mamba2 [6] further simplifies the matrix A to scalar enabling more hardware-efficient parallelizable training without sacrificing performance. Moreover, Mamba2 [6] demonstrates that its formulation is mathematically equivalent I-semiseparable structured masked attention, and develops the State Space Duality (SSD) framework to connect structured SSMs and attention variants. Furthermore, Mamba2 points out that other potential structured masked attentions can also be integrated into the SSD framework.
In this paper, we introduce novel structured masked attention, termed polyline path masked attention, tailored for vision tasks. Different from the previous selective SSM framework (53) based on Mamba [1]] our Mamba2-based polyline path mask is more lightweight and can be plugged seamlessly into various attention variants. Morcover, compared to Mamba Vision [17] which naively concatenates Mamba's is blocks and ViT self-attention layers, our method more effectively harnesses complementary strengths of both architectures.
Preliminaries
Mamba2's Recurrent Form. Mamba2 66 initially adopts a recurrent form with linear complexity for sequence modeling. Specifically, Mamba2 employs the selective state space models to map the input sequence E R RxC to the output sequence y R NxC i.e., for i=1:N,
h = a + B,T x,,
y: C,h,,
(I)
where I,, yi E IICC h2 DxC denotes the hidden state, ag E E and B, C, E IxD are input- dependent parameters learned by multilayer perceptron (MLP) layers, the scalar o; serves decay factor bounded in [0, 1]. N, C' and denote the sequence length, channel number, and hidden state dimension, respectively.
Mamba2's Attention Form. Leveraging the SSD framework in Mamba2 6), the recurrent form of Mamba2 in Eq. (I) can be reformulated as its equivalent dual form, i.e., structured masked attention, by eliminating the hidden state h; via substitution:
a; X x xa++1
i>
y = (CB O 1))
L = azj =
1
i= i i<j
(2)
0
where @ denotes the Hadamard (clement-wise) product, B, C' E RNDD, and the ID structured mask
LID E N is a l-semiseparable matrix which can be efficiently calculated with a complexity of
( by the chunkwise algorithm [6]. Mamba2's 's attention form (Eq. (2) enables more efficient
parallelizable training than its recurrent form (Eq. (T). Notably, parameters C' and B in Eq. (2) are learned analogously to the query Q and key
ViTs, respectively. Thus, Eq. 2 reveals that the selective state transition function in Mamba2 is equivalent to the Hadamard product of linear attention map CB and 1D structured mask L 1D Here, the structured mask can be interpreted as form of relative positional encoding (6).
3

---

J
H2V
V2H
(a) Cross Scamnnn
Hilbert Cuve Scanning
(c) Ployline Path Scaming
Figure 2: Compared to existing scanning strategies (a) and (b), which flatten 2D tokens into a ID sequence, our polyline path scanning (c) better preserves the adjacency of 2D tokens.
In this work, we extend the structured masked attention in Mamba2 from ID sequences 2D images. Specifically, we extend the ID structured mask L to the 2D polyline path mask L by introducing novel 2D scanning strategy, and propose an efficient algorithm for computing and applying this polyline path mask _ The proposed - can be substituted into Eq. (2) for replacing L or adopted ViTs as the explicit positional encoding.
Method
In this section, introduce the idea of adapting the structured mask of Mamba2 to 2D scanning and integrating it into the self-attention mechanism of ViTs, achieving an explicit positional encoding. Specifically, 1) introduce the definition of 2D polyline path mask in Sec. 4.1, 2) analyze the theoretical properties the proposed polyline path mask and introduce an efficient algorithm for the proposed polyline path mask in Sec. 4.2 3) apply the polyline path mask standard self-attention and criss-cross attention of ViTs in Sec.
4.1 Definition Polyline Path Mask
As sequence autoregressive framework, visual Mamba starts from employing scanning strategy to flatten a 2D image into a ID sequence image tokens. This scanning strategy plays an important role in Mamba's s performance, since the order of tokens is determined by it. As illustrated in Fig. 2 previous works [58 30 [27] have proposed various scanning strategies for visual Mamba. However, these strategies fail to fully preserve the inherent spatial adjacency of 2D tokens. For example, as shown in Fig. 2 (a) and (b), for two tokens B and C' which are close in an image, previous scanning
strategies 30 27 may cause them to be significantly farther apart in the ID scanning path.
Polyline Path Scanning. To address this limitation, we design a 2D polyline path scanning strategy. Specifically, for each token pair (ærj; xx in the 2D grid, we define their scanning path as the L- shaped polyline connecting them, as shown in Fig. 2 (c). To ensure symmetry in mutual distances, we set two bidirectional polyline paths: vertical-then www-tthhehhhhnnttttt path (V2H solid lines in Fig. (c)) and horizontal wttizoottalttee--eriieeee then-vertical path (H2V dotted lines in Fig. (c)), and their combination the final scanning path. In this way, the adjacency relationship of 2D tokens can be strictly maintained under the Manhattan distance?) Intuitively speaking, tokens close (or far) to each other will be in close (or far) distance on the scanning path, and vice versa. As the example shown in Fig. 2 polyline scanning strategy better preserves the distance between token B and C' compared to the other strategies. An more intuitive example is shown in Fig.
622
633
U3A4
442
Figure 3: An intuitive example illustrating the polyline path mask on 4 grid.
Definition of Polyline Path Mask. Based the proposed polyline path scanning strategy, introduce the polyline path mask. As an example shown in Fig. 3 we define the horizontal and vertical decay factors of each input token T. 0g.j and Br.j. respectively. In this paper, employ two MLP layers learn 04 and Bi.j. respectively" Then, the decay weight of V2H polyline path
2The Manhattan distance between two points Tı, and æ..1 a 2D plane - A| + +li jj 2]. We apply the ReLU and exponential operator after the MLP layers ensure ai.j 8.,5 E 1].

---

from kk to Zi.j is defined as L.j.k. which is the product of all decay factors along that path, i.e.,
Il= din
j <
m= 1
Bad
< k
Cisad = angaBaki, where dijj
j j= 1 BEs,
i = h.
(3)
as.n
j j>b
[I Pn. i k
For example, as illustrated in Fig. 3 the V2H polyline path's decay weight from token a4,4 to #1,1 is L1,1,4,4 11114,,:1,,0000 2012200,301,000000 P Similarly, the decay weight along the H2V polyline path is
defined as k. = k Due the spatial symmetry, evident that Lij.&. C By combining the V2H and H2V polyline paths, the final decay weight is
2D CLCC.
(4)
Note that L. C and 2 are all 4D tensors of size R RHxWXMWW where H and W are the height and width of the feature map, respectively. The polyline path mask, a 2D matrix 2 ER "WWHW be obtained by unfolding the decay weight tensor, ie.. for all i. k. and I,
can
2
(S)
For simplicity, we denote the above tensor-to to-ttrix unfolding operation as = unfold 2))) and its inverse operation as 2 = fold(L2)) in the following sections. More details can be found in Appendix A.2
4.2 Efficient Computation Theory of Polyline Path Mask
According to the definition @ the direct approach to compute the polyline path mask L L22D is to calculate each element individually. However, the large size of the mask and numerous multiplications for each element lead to high computational cost both calculating and applying I,"D, To address this issue, we present a decomposition theorem for matrices structured as L Based on this, we further design an efficient algorithm for performing multiplication on _ For simplicity, we focus our theoretical study on L. which is similar the case of 2 Complete proofs of the theorems are provided in Appendix A.3 and A.5
Theorem 1 (Matrix Decomposition). For any matrix M E ER HWWHW and M = fold (M), if for Vi,j, I, AĄ's E wxW and B' E H.HHH s.t., Misikl = [A x [B',, i,ke then M can be decomposed as:
M M X M = =MA GAON BMB
(6)
where MA, M MA MB ERHW which satisfy
A k=i
(B' j=l
unfold((MM ===unfold[MM' s.L., MA Mi kii
MB
(7)
(0
j+l
M = un==oooo(MM B), s.l., M A', M B
B!
(8)
As defined in Eq. (3), the polyline path mask L satisfies the conditions in Theorem with [A
arjja and [B') 3. Thus, based on Theorem 1 the polyline path mask L can be decomposed as L= L= x - @ LV. Moreover, for the complexity of computing L, we have:
Corollary (Mask Complexity). The complexity of directly computing polyline path mask L with (3) and (5) is (NV)) which can be reduced to (N2)) by applying Theorem 1 where N HxW.
For matrices in the form of Eq. (7), when performing multiplication operations, we have: Theorem 2 (Efficient Matrix Multiplication). For matrices A M defined in Eq. YDER"W the following equation holds:
y == M x
M
X
Z,=B''X, d. Y. A'xZ Zi
where y "W X unvec( E R
unvoc()))R"AA HxW, Y unuvccly) E HxW Z &R and the operator vec(-) vectorizes matrix by slucking its columns and unvec(-) is its inverse operator.
5

---

2D Ployline Path Mask
Alga
1D
Alga Step 2833
1D
Algn Step 483
Step
Vertical Mask
Horizonaal Mask
1
1
y
1 y = 2D x
unvec
B
A
YOC
Z
Figure 4: Illustration of the efficient algorithm for utilizing the proposed polyline path mask. Left: Naive computation of matrix multiplication. Right: An intuitive illustration of Algorithm
Based on Theorem 2 we can design AI- gorithm computing the matrix mul- tiplication between polyline path mask L and the vector æ. Note that the in- volved matrices A' and B' are sym- metric matrices with lower triangular parts being s-semiseparable, defined in Mamba2 [6]. This will lead to a substantial reduction in complexity stated in the following corollary.
Algorithm Efficient Masked Attention Computation.
Input: decay factors a, 8 of L, vector æ RER#W
1: Compute X = unvec((a)) ER HxW
Compute B' E RxH where for 1 =1:W, J=A=BBAR
[
3: Compute Z E FxxW where Z, -B' XX..;;
Compute A' E WxW where for i= [A 1.1
5: Compute Y IR HxW where Y..: A' Zu.:;
Output: y= ve)
Corollary 2 (Masked Attention Complexity). The computational complexity of the matrix multiplica- tion between polyline path mask and vector i.e., y= Lax can be reduced from ((N2) (NN by Algorithm]] 7 and further reduced to O(N) by applying the chunkwise algorithm of Mamba2 [ steps 3 and 5 in Algorithm
Remarks. Intuitively, illustrated in Fig. Algorithm shows that the 2D polyline path scanning on 2D tokens (i.e., Lœ) can be decomposed as the ID vertical scanning along each column of X (i.c., Z, B' x.. followed by the 1D horizontal scanning along each row of Z (i.e., Y., A' Z..)) This equivalence offers an intuitive understanding of the physical meaning of the decomposed polyline path mask L = LV and enables its natural extension to 3D higher-dimensional tokens, as detailed in Appendix C.2
4.3 Polyline Path Masked Attention
The proposed polyline path mask can be seamlessly integrated into various attention variants in plug-and-play manner. In this section, we integrate it into two softmax- self-attention layers: vanilla attention [9] and criss-cross attention (22. Notably, theorems and algorithm given in Sec. 4.2 guarantee that integration of polyline path mask does not substantially increase the computational complexity of the original attention mechanism. More applications, such as the polyline path masked linear attention with a complexity of O(N), are provided in Appendix A.7
Polyline Path Masked Vanilla Attention. The polyline path mask L is integrated into vanilla attention via a Hadamard product with the attention map, i.e., for query Q, key K, and value V:
PPMVA (x) = (softmax QKT) ) V,
(10)
where Q, K, V VERHWXC Based on Corollary Eq. 10 maintains the complexity of ((2)).
Polyline Path Masked Criss-Cross Attention. The original criss-cross attention
22 employs the
sparse attention over tokens located in the same row or column, achieving a complexity of ( )). In this work, we follow RMT [10] to decompose criss-cross attention into the vertical attention over each column followed by the horizontal attention over each row. The polyline path mask 2D is applied to the decomposed criss-cross attention through the Hadamard product, that is:
PPMCCA (ær) ((SH SV ② L2D V ((SH x SV) VOOLVV+ ((S**S x XX
oivv, (11)
where horizontal and vertical attention maps S SV ER satisfy the form in Eq. with
A softmaxí O...K K. .....) and B' softmax e.. K.I.). and Q K. E HxWxC are tensor forms

---

Down Sampling
Stem
PPMCCA Block
PPMCCA Block N2
Dona Toluuro
PPMCCA Block x N3
PPMVA Block x N
x N
X
Masked Attention Map
softmax(QK*)
2
Polyline Path Masked Vanilla Attention Figure 5: Overall architecture of the Polyline Path Masked Attention based Vision Transformer.
Polyline Path Masked Criss-Cross
of Q, K, respectively
22 Based on Theorem
we can reformulate the left part Eq. T1)
as:
(S X LVV ((S x
SV
2 L
-($ Sv L o M
(12)
= =(S 1
S
o ( v= OL x (SV LLV
Note that matrices SH i H and ŠV vi also satisfy the form in Eq. 77). Thus, the computational
complexity of Eq. 12 can be reduced to ((NE) by Algorithm Similar conclusions can also be
derived for the right part of Eq. (TI) Thus, the complexity of Eq. (II maintains (
4.4 Overall Architecture
Based the proposed Polyline Path Masked Attention, construct hybrid -aanbaa2-rrasrrreer backbone for vision tasks. As illustrated in Fig. [5] our backbone adopts the four-stage hierarchical architecture. Following RMT [10), employ Polyline Path Masked Criss- Attention in the first three stages, and Polyline Path Masked Vanilla Attention in the final stage. Moreover, we develop model in three scales: tiny (PPMA-T), small (PPMA-S), and base (PPMA-B).
5 Experiments
To validate the effectiveness of our method, we conduct series of experiments on mainstream benchmarks for image classification (Sec. (Sec. object detection and instance segmentation (Sec. 5.2
and semantic segmentation (Sec. 5.3). Comparison methods include advanced CNN-based [34 3
42 SSM-based 30
53
49
47 and Transformer-based backbones [31 8 17
16
57 10). For fair
comparison, we reproduce the experimental results of RMT [10] with the same experimental settings
ours. We also perform comprehensive ablation studies the structured mask design in Sec. 5.4
More detailed experimental settings and results can be found in Appendix B
5.1 Image Classification on ImageNet-1K
Settings. We evaluate the classification performance of our method on ImageNet- K 77] Following the same training strategy as [10 44). we train our models from scratch for 300 epochs with the input size of 224x224, We use the adaptive AdamW optimizer with cosine decay learning rate scheduler (batch s02c=1024, initial learning ratc=0.001, weight decay=0.05).
Results. The comparison results presented in Table I show that our method achieves state-of-the-an (SOTA) performance compared to other advanced models based on various architectures across tiny, small, and base scales. Specifically, PPMA-S achieves 84.2% % top-1 accuracy, surpassing 2DMamba-
(53) by 1.4%. MLLA-T (15) by 0.7% Mamba Vision-T2 17 by 1.5%, and RMT-S 10 by 0.2% % with similar FLOOPs PPMA-T achieves 82.6% top-1 accuracy, outperforming the most competitive RMT-T 10 by 0.2% without extra training tricks. Moreover, our РPMA-B also surpasses other SOTA based SSM- and Transformer based backbones.
7

---

Table Image classification performance on the ImageNet- IK validation set.
#Param.
FLOPs
Top-1
#Param.
FLOPs
Top-I
Model
Arch.
Model
Arch,
(M)
(G)
(%)
(M)
(G)
(%)
RogNetY-1.66
34
CSM
78.0
NAT-T
28
BBfNet-B3
81.6
BiFormer-S 57
Trans.
26
4.5
Vim-T 58
76.1
RMT-S
27
4.5
84.0
MSVMambu-M
12
79.8
PPMA-S
27
4.9
84.2
BiFormen-T 57
NAT-M [6
Trans.
20
13
2.7
2.2
81.4
81.8
RegNetY-8G 34
ConvNeXt-S
32
CNN
50
39
8.0
8.7
81.7
83.1
SMT-T 29
12
2.4
82.2
EftNet-B5
30
9.9
RMT-T 10
14
2.5
82.4
VMamba-S
8.7
PPMA-T
14
2.7
82.6
2DMamba-S
SSM
50
8.8
RegNetY-4G
34
CNN
80.0
GrootVL-S
51
8.5
84.2
ConvNeXi-T
32
82.1
MILAA- [5
43
7.3
LffNet-B4
42
82.9
Spatial-Mamba-S
17
43
7.1
84.6
VMamba-T
30
30
4.9
82.6
Swin-S 3T
50
8.7
83.0
2DMamba-T
53
SSM
31
4.9
82.8
NAT-S 116
51
7.8
83.7
GrootVL-T
49
30
4.8
83.4
CSWin-B
Transs.
78
15.0
84.2
Spatial-Mamhar
27
4.5
83.5
MambaVision-BB 11
98
15.0
84.2
MLLA-T 15
25
4.2
83.5
BiFormer- [57
57
9.8
84.3
Swin-T 31
Trans.
29
4.5
82.1
iFormer-B [37
48
9.4
84.6
CSWin-T
23
4.3
82.7
RMT-B [T]
54
9.7
84.9
MambaVision-
5.1
82.7
РPMА-B
54
10.6
85.0
Table 2: Object detection and instance segmentation performance with Mask R-CNN on COCO val2017. FLOPs are calculated with input resolution of 1280 800.
18 detector
Mask R- C-CNN 1x schedule
Backbone
#Param. (M)
FLOPs (G)
AP*
AP
AP
APM
APm
50
AP
Vim-T T [58]
45.7
63.9
49.6
39.2
60.9
41.7
MSVMamha-N
36
32
201
43.8
65.8
47.7
39.9
62.9
42.9
MPVIT-XS 25
30
231
44.2
66.7
48,4
40.4
63.4
43.4
RMT-T [1]
33
218
46.7
68.6
51.6
42.1
65.3
45.2
РPMA-T
218
68.7
51.7
65.9
45.7
ResNet-50 [19
44
260
38.2
58.8
41.4
34.7
55.7
37.2
ConvNeXt-
48
262
44.2
66.6
48.3
40.1
63.3
42.8
MLLA-T
255
46.8
69.5
51.5
42.1
66.4
45.0
GrootVL-T 49
49
265
47.0
69.4
51.5
42.7
66.4
46.0
VMamba-T [0
50
271
47.3
69.3
52.0
42.7
66.4
45,9
Spatial-Mambet
216
47.6
69.6
52.3
42.9
66.5
46.2
Swin-T [3]
48
267
43.7
66.6
47.7
63.3
42.7
CSWin-T X
42
279
46.7
68.6
51.3
42.2
65.6
45.4
BiFormer-S (57
47.8
69.8
52.3
43.2
66.8
46.5
RMT-S [1]
262
48.8
70.8
53.4
43.6
67.4
47.3
PPMA-S
46
263
49.2
70.7
54.0
43.8
67.4
47.1
ResNet-101 19
63
336
40.4
61.1
44.2
36.4
57.7
38.8
ConvNeXt-S
348
45.4
67.9
50.0
41.8
65.2
GrootVL-S 49
341
48.6
70.3
33.3
43.6
67.5
47.1
VMamba-S
80
70
349
48.7
70.0
53.4
43.7
67.3
47.0
Spatial-Mamba-S
63
315
49.2
70.8
54.2
44.0
67.9
47.5
MLLA-S
319
49.2
71.5
53.9
68.5
47.2
Swin-S 31
69
359
45.7
67.9
50.4
41.1
64.9
44.2
CSWin 8
54
342
47.9
70.1
52.6
43.2
67.1
46.2
BiFormer-B
48.6
70.5
53.8
43.7
67.6
47.1
RMT-B [0]
73
373
72.0
55.7
69.2
49,0
PPMA-B
73
374
51.1
72.5
55.9
45.5
69.7
49.1
Mask R-CNN schedule
ConvNeXI-S
82
70
348
47.9
70.0
52.7
42.9
66.9
46.2
GrootVI-S
49
70
341
50.1
71.2
54.9
44.6
68.7
47.8
VMamba-S
30
349
49.9
70.9
54.7
68.2
47.7
MLLA-S [5]
63
319
50.5
71.8
55.2
44.9
69.1
48.2
Spatial-Mambar
47
63
315
50.6
71.5
55,4
44.7
68.6
48.2
NAT-S [16
70
330
48.4
69.8
53.2
43.2
66.9
46.4
Swin-S S
359
48.5
70.2
53.5
67.3
46.6
CSWin S
54
342
50.0
71.3
54.7
44.5
68.4
47.7
RMT-B [10
73
373
52.2
72.9
57.0
46.1
70.4
49,9
PPMA-B
73
374
52.6
73.3
57.5
46.3
70.3
50.2

---

Table Semantic segmentation performance with UPerNet 48 and 'MS' represent sangle-scale and multi-scale testing, respectively.
segmentor on ADE20K val set. 'SS'
#Param.
FLOPs
ml)U({))
#Param. (M)
FLOPs
(l)
Backbone
Backbone
(M)
(G)
SS
MS
(G)
SS MS
LocalVim-T [2
181
43.4
44.4
BiFormer-S 157
49.8 50.8
MSVMamba-M
36
875
45.1
45.4
RMT-S [T0
56
49.8 49.7
NAT-M [16
900
45.1
46.4
PPMA-S
56
984
51.1 52.0
RMT-T [10
43
977
48.0
48.8
ResNet-10
101
19
85
1030
42.9 44.0
PPMA-T
43
983
48.7
49.1
ConvNeXt-S
32
82
1027
48.7 49.6
ResNet-50 19
67
953
42.1
42.8
VMamba-S 30
82
1028
50.6 51.2
Con-NeXt-T
32
939
46.0
46.7
Spatial-Mamha-S [47
73
992
50.6 51.4
VMamba-T T 30
949
48.0
48.8
GrootVL-S 49
82
1019
50.7 51.7
2DMamba-T
[33
950
48.6
Swin-S
81
47.6 49.5
GrootVL-T
49
60
941
48.5
49.4
NAT-S [16
82
1010
48.0 49.5
Spatial-Mambar
47
57
936
48.6 49.4
Mamba MambaVision 122
84
1135
48.2
Swin-TT 31
60
945
44.4 45.8
CSWin-S 8
65
1027
50.4 51.5
MambaVision-1
17
55
945
46.6
BiFormer-B 57
51.0 51.7
NAT-T [66
58
934
47.1
48.4
RMT-B [TO]
83
1051
52.0 52.1
CSWin-S 8
60
959
49.3
50.7
PPMA-B
83
1137
52.3 53.0
5.2 Object Detection and Instance Segmentation on COCO
Settings. We evaluate our method for object detection and instance segmentation tasks on MSCOC02017 (28 using the MMDetection library (2). Following previous work (35), we ini- tialize the backbone with ImageNet- IK pretrained weights and adopt Mask R-CNN [8] as the basic framework. The models are trained for 12 epochs (1x schedule) and 36 epochs with multi-scale imputs (3x schedule) using AdamW optimizer (batch size=16, learning ratc=0.0001, weight decay=0.05).
Results. The results presented in Table show that our model outperforms existing methods on most evaluation metrics. Under the same experimental settings, PPMA-T achieves box mAP of 47.1 % and mask mAP of 42.4%, surpassing the SOTA Transformer-basee backbone RMT-T [TO by 0.4 % and 0.3 % in the x schedule, respectively. Moreover, РРМА-B - achieves box mAP of 51.1 % and mask mAP of 45.5%, surpassing the SOTA SSM-based backbone MLLA-S [SS] by 1.9% and 1.3 % the x schedule, respectively. Furthermore, PPMA-B maintains its superior performance under the 3x multi-scale training schedule.
5.3 Semantic Segmentation ADE20K
Settings. We evaluate the semantic segmentation performance of our method on ADE20K [56] using the MMSSgmentatiin library 4 Following the settings in previous works (55, we initialize the backbone with ImageNet- K pretrained weights and adopt UPerNet [48] as the basic framework. The input size of images is set 512 512 and all models trained for 160K iterations with Adam optimizer (batch size=16, learning rate=6 x 10-5, weight decay=0.05),
Results. The semantic segmentation results are summarized in Table 3 Our method consistently outperforms previous methods under all settings. Compared to SOTA Transformer-based counterparts, PPMA-T/S/B surpass RMT-T/S/B by 0.7 0.7% % mloU in the Single-Scale (SS) setting and 0/.%//..%//99/ mloU in the Multi-Scale (MS) setting. Compared to SOTA SSM-based methods, PPMA-T/S/B surpass them by least 3.6 2...%...5//..... //.66 % in SS mloU, respectively.
5.4
Ablation Study
Polyline Path Mask Design. To verify the effectiveness of the proposed polyline path mask, we conduct an ablation study on ImageNet-IK and ADE20K using PPMA-T as the backbone. Under the same experimental settings, compare various structured masks embedded into the softmax-base self-attention layers by Hadamard product, including: no mask (baseline). RMT decay mask [IO], cross scan mask [30], Hilbert sean mask (27), V2H polyline path mask, and our final 2D polyline path mask. As shown in Fig. 6 our polyline path mask L compared to the RMT decay mask, can selectively capture the semantic continuity in the image. Compared to the cross scan mask and Hilbert sean mask, the polyline path mask better preserves the spatial relationships between 2D tokens, alleviating the rangerange forgetting issue. Experimental results in Table 4 show that the 2D

---

Table 4: Ablation study of structured mask designs in PPMA-T on ImageNet- IK and ADE20K.
Structured Mask
Paraam. (M) FLOPs (G) Throughput (imgs/s) Top-1 (%) mloU SS (%)
Baseline (w/o mask)
14.33
2.65
1779
82.28
47.78
+ RMT Decay Mask
14.33
2.65
1650
82.35
48.01
+ Cross Scan Mask
14.34
2.71
1100
82.44
48. 48.14
+ Hilbert Scan Mask
14.34
1091
82.44
V2H Polyline Path Mask
14.34
2.71
1203
82.44
48.57
+ 2D Polyline Path Mask
14.34
2.71
1034
82.60
48.73
Shared Decay factors =
14.33
1124
82.37
48.27
Different Decay factors (0i_. + Pi.s)
14.34
2.71
1034
82.60
48.73
Input Image
(b) W/o Mask
(c) RMT Decay Mask
Cross Scan Mask (e) Hilbert Scan Mask
Polyline Path Mask
Figure 6: Illustration of various structured masks.
(b) Horizontal Decay Factor a Figure 7: Visualizations of the decay factors and the polyline path masked attention maps of the well-trained PPMA model. In each input image, the query token is marked by a red box.
(c) Vertical Decay Factor B
(d) Original Attention Map
(e) Polyline Path Mask 20
(f) Polyline Path Masked Attention Map
(a) Input Image
polyline path mask 2 boosts the baseline by 0.32 % top-1 accuracy on ImageNet- IK and 0.95% % SS mloU on ADE20K, respectively. Visualization results in Fig. further demonstrate that our polyline path mask 2D effectively suppresses the falsely highlighted areas in the original attention maps. More visualizations and detailed discussions are provided Fig. 12 and Sec. C.I
Horizontal and Vertical Decay Factors. In our model, we employ different decay factors (a..) + Bi,g) capture semantic similarity between adjacent tokens along horizontal and vertical directions, respectively. As illustrated in Fig. (b) and (c), the learned decay factors a and 33 effectively capture semantic continuity in horizontal and vertical directions, respectively. Table Sablee|shov shows that replacing different decay factors with a shared decay factor (as,, = Bt.,) results in significant performance drop, highlighting the importance of modeling horizontal and vertical decay factors separately.
Conclusion
In this paper, we argue that the key component of Mamba2 model is its structured mask, which explicitly encodes the spatial distance information through the recursive propagation mechanism and captures the semantic continuity in sequences through the selective mechanism. Building on this insight, we propose to extend the structured mask from ID text sequences to 2D images. To this end, propose novel 2D polyline path scanning strategy with its corresponding structured mask tailed for images. To achieve SOTA performance on high-level vision tasks, we integrate the polyline path mask into the powerful self-attention mechanism of VITs.
Limitations. Although the proposed efficient algorithm optimizes the integration complexity, inevitably incurs additional GPU memory occupation and lower throughput, as shown in Table We plan to alleviate this limitation through further engineering optimizations, such as CUDA-based or Triton-base implementations, in the future work.
10

---

Acknowledgments
We would like to thank all anonymous reviewers for their constructive suggestions for improving this paper. This work was supported in part by the National Key R&D Program of China under Grant part by the Major Key Project of PCL under Grant PCL2024A06; in part by Tianyuan Fund for Mathematics of the National Natural Science Foundation of China (Grant No. 12426105); in part by the China NSFC projects under contract 62476214.
2024YFA1012000;
References
[1] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, al. Video generation models as world simulators. OpenAl Blog, 1:8, 2024.
[2] Kai Chen, Jiaqi Wang, Jiangmiao Pang, cl. MMDetection: Open mmlab detection toolbox and benchmark. 2019.
arXiv preprint /rXi6
[3] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. Conditional positional encodings for vision transformers. The Twelfth International Conference on Learning Representations, 2023.
[4] MMSegmentation Contributors. Mmsegmentatiin, an open source semantic segmentation toolbox, 2020.
[5] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, et al. Randaugment: Practical automated data augmentation with a reduced search space. In CVPRW, 2020.
[6] Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In Proceedings of the IEFE international conference on computer vision, 2024.
[7] Jia Deng, Wei Dong, Richard Socher, al. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2009.
[8] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, et Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedingy of the IEEE/CVE conference on computer vision and puttern recugnition, 2022.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An image is worth 16x16 words: Trans- formers for image recognition scale. In The Tenth International Conference Learning Representations, 2021.
[10] Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, and Ran He. Rmt: Retentive networks meet vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5641-5651, 2024.
[11] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00752: 2023.
[12] Hang Guo, Yong Guo, Yaohua Zha, Yulun Zhang, Wenbo Li, Tao Dai, Shu-Tao Xia, and Yawci Li. Mambairv2: Attentive state space restoration. arXiv preprint arXiv:2411. 152669 2024.
[13] Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: simple baseline for image restoration with state-space model. In Proceedings of the European conference on computer vision, pages 222-241. Springer, 2024.
[14] Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cnt: Convolu- tional neural networks meet vision transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022.
[15] Dongchen Han, Ziyi Wang, Zhuofan Xia. Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song, Zheng, and Gao Huang. Demystify mamba in vision: A linear attention perspective. arXiv preprint arXiv:2405.16605, 2024.
[16] Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.
[17] Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba-transforma transformer vision backbone. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2025.
11

---

[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, 2017.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Deep residual learning for image recognition.
In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2016.
[20] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4700-4700 47000 2017.
[21] Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan, arXiv preprint XXiv:2403.09338, 2024.
[22] Zilong Huang. Xinggang Wang. Yunchao Wei, Lichao Huang, Humphrey Shi, Wenyu Liu, and Thomas S. Huang. Conet: Criss-cross attention for semantic segmentation. IEEE transactions on pattern analysis and machine intelligence, pages 1-1, 2020.
[23] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mnss: Fast autoregressive transformers with linear attention. Proceedings of the IEEE international conference on computer vision, pages 5156-5165, 2020.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEEE international conference computer vision, pages 4015-4026, 2023.
[25] Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. In Proceedings of the IEEE/CVF conference on computer vision and paitern recognition, 2022.
[26] Kunchang Li, Xinhao Li, Yi Wang. Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In Proceedings of the European conference on computer vision, pages 237 275. Springer. 2024.
[27] Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: simple state space model for point cloud analysis. In Advances Neural Information Processing Systems, 2024.
[28] Tsung-Yi Yi Lin, Michael Maire, Serge Belongic, al. Microsoft coco: Common objects in context. In Proceedings of the European conference on computer vision, 2014.
[29] Weifeng Lin, Ziheng Wu, Jiayu Chen, Jun Huang, and lianwen Jin. Scale-aware modulation meet transformer. In Proceedings of the IEEE international conference on computer vision, 2023.
[30] Yue Liu, Yunjic Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. In Advances in Neural Information Processing Systems, pages 103031-103063, 2024.
[31] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF international conference computer vision, 2021.
[32] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, et al. A convnet for the 2020s. In Proceedings of the IEEEJCVF conference computer vision and pattern recognition, 2022.
[33] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging, arXiv preprint
2019.
[34] Ilija Radosavovic, Raj Prateek Kosaraju. Ross Girshick. Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. 2020.
[35] Dai Shi. Transnext: Robust foveal visual perception for vision transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 17773-17785 2024.
[36] Yuheng Shi. Minjing Dong, and Chang Xu. Multi-scale vmamba: Hierarchy in hierarchy visual state space model. arXiv preprint ar arxiv:2405. /4174, 2024.
[37] Chenyang Si, Weihao Yu, Pan Zhou, Yiehen Zhou, Xinchao Wang, and Shuicheng YAN. Inception transformer. In Advances in Neural Information Processing Systems, 2022.
12

---

[38] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:12706; 2024.
[39] Weixuan Sun, Zhen Qin, Hui Deng, Jianyuan Wang, Yi Zhang, Kaihao Zhang, Nick Barnes, Stan Birchfield, Lingpeng Kong, and Yiran Zhong. Vicinity vision transformer. IEFE Transactions on Pattern Analysis and Machine Intelligence, (110))12633-12649, 2023.
[40] Yutao Sun, Li Dong. Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to Transformer for large language models. ArXiv, abs/2307.08622 2023.
[41] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[42] Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the IEEE international conference computer vision, 2019.
[43] Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai, Yu Wang. Zhenguo Li, and Xihui Liu, Dim: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint arXiv:2405. 14224, 2024.
[44] Hugo Touvron, Matthieu Cord, Matthijs Douze, et al. Training data-efficient image transformers distillation through attention. In Proceedings of the IEFE international conference on computer vision, 2021.
[45] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Proceedings of the European conference on computer vision, 2022.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[47] Chaodong Xiao, Minghan Li, Zhengqiang Zhang, Deyu Meng, and Lei Zhang. Spatial- mamba: Effective visual state space models via structure-aware state fusion. In The Fourteenth International Conference on Learning Representations, 2025.
[48] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference computer vision, 2018.
[49] Yicheng Xiao, Lin Song, Shaoli Huang, Jiangshan Wang. Siyu Song, Yixiao Ge, Xiu Li, and Ying Shan. Grootvl: Tree topology is all you need in state space model. arXiv preprint arXiv:2406.02395, 2024.
[50] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, et al. Cutmix: Regularization strategy train strong classifiers with localizable features. Proceedings of the IEEE international conference on computer vision, 2019.
[51] Guowen Zhang, Lue Fan, Chenhang He, Zhen Lei, ZHAO-XIANG ZHANG, and Lei Zhang. Voxel mamba: Group-free state space models for point cloud based 3d object detection. Advances in Neural Information Processing Systems, 37:8 1889-9918,,,9 2024.
[52] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, et al. mixup: Beyond empirical risk minimization. In The Seventh International Conference on Learning Representations, 2018.
[53] Jingwei Zhang, Anh Tien Nguyen, Xi Han, Vincent Quoc- Huy Trinh, Hong Qin, Dimitris Samaras, and Mahdi S Hosseini. 2dmamba: Efficient state space model for image representation with applications on giga-pixel whole slide image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2025.
[54] Sijie Zhao, Hao Chen, Xueliang Zhang, Pengfeng Xiao, Lei Bai, and Wanli Ouyang. Rs-mamba for large remote sensing image dense prediction. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[55] Zhun Zhong, Liang Zheng, Guoliang Kang, al. Random erasing data augmentation. In AAAI, 2020.
[56] Bolei Zhou, Hang Zhao, Xavier Puig, et al. Scene parsing through adc dataset. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2017.
[57] Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson Lau. Biformer: Vision transformer with li-le routing attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.
13

---

[58] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.0%4 241477 2024.
[59] Zhen Zou, Hu Yu, Jie Huang, and Feng Zhao. Freqmamba: Viewing mamba from a frequency perspective image deraining. Proceedings of the 32nd ACM International Conference on Multimedia, pages 1905-1914 2024.
14

---

Supplementary Material
Appendix
Efficient Computation Theory for Polyline Path Mask Applications;
Appendix A.
Notations;
Appendix A.
Definition of Polyline Path Mask;
Appendix A.3
Theorems and Proofs;
Appendix
4
Complexity Analysis of Mamba2 Attention Form;
Appendix
A.5
Complexity Analysis of Polyline Path Mask;
Appendix
A.6 6
Complexity Analysis of Polyline Path Mask Multiplication;
Appendix
A.
Applications of Polyline Path Masked Attention;
Appendix
Experimental Details;
Appendix B.1
Implementation Details;
Appendix B.2
Training Settings for ImageNet-IK;
Appendix
B.3
Training Settings for Downstream Tasks;
Appendix
B
Throughput Comparison;
Appendix
8.5
Visualization;
Appendix
Discussion;
Appendix
Selectivity Polyline Path Mask;
Appendix
3D Extension of Polyline Path Mask;
Appendix
Limitations;
A Efficient Computation Theory for Polyline Path Mask Applications
A.1 Notations
we employ large number of notations both for clarity and central tool in stating and proving our theorems, including:
Following Mamba2 [6].
Dimensions. We generally use N. H, W, C, D as the superscript letters of IR to denote the sequence length, the height of the feature map. the width of the feature map, channel number, and hidden state dimension, respectively. The sequence length of the 2D feature map (i.e., the number of tokens) is N H W.
Matrices and Tensors. Following convention, we use non-bolded lowercase letters, bolded lowercase letters, bolded uppercase letters, and bolded calligraphy letters to denote scalars, vectors, matrices, and 3D or higher dimensional tensors, respectively.
Indexing. We use indexing i to refer to range 1, + 2,..., when i < j and i 1.....; + when j. For example, for any scalar a, we let duz for i < j denote the sequence (ai+1, 0442, ... a.). For shorthand, let a for i denote the product 0 X 01+2 X
We let " d::j for >
Tensor Unfolding. We use the operator vec(-) to vectorize a matrix by stacking its columns and the operator unvec(-) its inverse operation. We use the operator unfold(.) to unfolds 4D tensor L H>WWilWW to a 2D matrix E IR #WxHW where [[]ss-) -xa =
C. and the operator fold(-) as inverse operation.
Tensor Multiplication. For 2D matrices,
the symbol to denote the matrix multiplica- tion and the symbol @ to denote the Hadamard (element-wise) multiplication. For multiplication operations involving 3D higher dimensional tensors, einsum(-) to denote the tensor multiplication on the given dimension, which is commonly used in modern tensor libraries such as PoTorch. For example, einsum( monsummnn,,,. nm' K) denotes the matrix multiplication Q x K
the Einstein summation notation
15

---

61.2
P
622
83
5,9 02223322333
Figure 8: An illustration of the V2H polyline path scanning on a x grid (with a total of 9 tokens). There 81 scanning paths. Each scanning path (red polyline) corresponds decay weight in the polyline path mask L.
A.2 Definition of Polyline Path Mask
For each token pair (wsi.j ax.)) in the 2D grid, the decay weight of the vertical-then-horizonie (V2H) polyline path from æi, æk is defined C.J.K.l. which is the product of all decay factors along that path, i.c.,
Tl Oi,
jel
i= k. I mm
Budt
i<
LigAh = aagaBBRR. where anja
j ==1 1, Bak,
(13)
1Pm Os,,
j j>{
Bn.
i>
where aija and Baku are horizontal and vertical decay factors bounded in the range [0, 1]. For ER HaWstho into 2D matrix the polyline path mask
convenience, we unfold the 4D tensor L e BR iie.
L L=unfold(C).
(14)
auaaaa 21 31. 012131 01228112 012.311.
a1 0112 an.iaB.. w.w
012131i1 a1222812 01.w31.
a.waiBaa wa www.w
an an w.w.B
aи.ι amaaBiaa anwnddaa aH.128H12 aнaalнas aнwaadнa
amaa aR. an. B. aнaлdни1 ан2аннн2 aH2. Bs.ww
LoR, a..a www.w
aHwaIKH, aww
3m..ww.
(15)
An intuitive example illustrating the polyline path scanning on 3x3 grid is presented in Fig. For the 9 tokens in the 2D grid, there are 81 V2H scanning paths connecting them. The V2H scanning path between each token pair marked by the red polyline, which corresponds decay weight in
In some contexts, is always clear that the notation as:j means au and the superseript omitted.
16

---

Lemma
Conclusion
The complexity x can be reduced to ((N) by chunkwise Algorithm
The complexity Mamba2 Oth) Mambs2(0) (ca' o gooox
Preliminary
Conclusion complexity Polyline
Definition
oge"
Property
Application [Complexit)
Theorem 1
Conclusion 3.1
Matrix Decomposition M A
The complesity of Folylina Path Mascad Standard self- Attention is d(V") PPRVVA(0) (sottman(("") ) (34" 1dy
Conclusion 3.2 Attention isCUN) PPMLACO (ox* @ 22n)y
complexity Polyline
Theorem Efficient Multiplication -=xxx e Zaz d' Y, A'E,
spedial al case
Conclusion 3.3
The complesity of Polyline Path Masced Criss Cross Attention o(15) PPMCEA(K) = (15* 5 ""yy
Conclusion
Theorem 3
The complexity Polyline Masked Decomposable Attention
((QK)) VL D (L)KvV)
quadratic PANDAC)) - ( 1
Figure 9: An overall illustration the efficient computation theory and corresponding applications.
the polyline path mask L. The V2H polyline path mask L e 900 paths in Fig. g is defined as:
constructed based on the scanning
01.1228-122
01.:1822
011a0a2
01..0
01.1
a1 01.1:29::5.2
01.2:311.11
01.2:8:112
01223311
01211811
0122
01,2381223
01.2:111.11
01.2
01223
01..11
01329112
01331
41
01,38
01,31
01
02.1:1221.1
42.12/2122
02.18f213
02.11/22.1
02.12222.22
02.13/22
0q.1
02.12
L=
0222132111
00.208212
012a331.
0221822.
02.23322
02233822
02.22333.3
02.31321.11
02.39/122
00,3992133
0231
09339
02333223
09.3n
02,3
03,11581.1
03.1206..2
03,13551.3
0311882.1
03.12322.2
03.135a2.
01.11
03,1223...2
151.1
03,290112
03,23551,3
0322182211
0323053.2
03238823
03,21
03,22333.2
03228833
03,31 151,1
43,3291122
03,339513
0533192211
03,32
05333/92,
03,31955,1
03,3:253,2
05.39953.33
(16)
A.3 Theorems and Proofs
In this section, we present three theorems and their proofs, which will be used to optimize the computational complexity in the following applications A.. Specifically, we present decomposition theorem for matrices structured as the polyline path mask L. Based on Theorem [ we present an efficient matrix multiplication theorem 2 for performing multiplication Then, we present an equivalent computation theorem 3 for the masked linear attention. An overview illustration is provided in Fig. 9 which summarizes the theorems and their corresponding applications in polyline path masked attention.
the polyline path mask L.
Note that the polyline path mask defined in Eq. 15 is a matrix with special structures. Here, we present decomposition theorem for matrices structured as L.
Theorem 1 (Matrix Decomposition). For any matrix M E RR HWxHW and M = fold (M), if
for Vi. I, WĄ' E WxW and B' E HxH s.t., Mijj.k. = [A]] x [B'],s i,k then M can be decomposed as:
M - M X M B = MA o M
(17)
where M M MA M HWW which satisfy
( A' k=i M
BB'
j=l
M un=unfold M =n=unfold(MM s.t,, M....
0 k-i' Jiid
(0
j
(18)
unfolddd(MMi M unf= s.t,, M A', M.
M
B' (19)
17

---

Proof. Let us first prove M M = M in Eq.
For clarity, let i |u/W] + 1. mod W. For 1: HW
mod W. k [ + I mod W, m w
and e 1 HW, we have HW
Σ
M M B
Σ
M
M B J.m
m.nkk
m=m
H
w
M J.mmm m.lkk
+
Σ
MA
by Mm M
m.nkk
m=1
n=Innf
H
M.jmm MijmMMmi.. m.lkkd
(20)
Σ M Mij.M.Maa.
b.kk
m-m limii
M M I.L.I
A Misal
M
According to Eq. 19
we have
M. M
M
iA -Miiaa Mu
(21)
Thus, Theorem is proven.
For matrices of the form given in Eq. 18). when performing multiplication operations, we have:
Theorem 2 (Efficient Matrix Multiplication. For malrices MA. MM defined in Eg. (88) Va E R HW the following equation holds:
= M A
x M
x
Z B X. Y,
A' Zi
(22)
where y EER HW X (
Y=unvvel y HxW
HKW
Proof. The left part of Eq. (22 can be calculated by M B x a and
M x z. For clarity,
let i Lu/ j. mod W. Lu/W + 1. mod W. For HW, we have
HW
H
W
M B
x
ΣΣ
M
B
X
D=
A= t=1
H
H
M k.jk
Xk. +
Σ
MB
M.j.K.aa
(23)
k= 1=111111
M Kijk
X
Σ B
X
1=1
Zej.
The final results in Eq.
23 is equivalent to Z., = Bd x X.
Then, for a = 1: HW, we have
W
yn
M
2, Σ Σ
M
M.j.K.Za. Zu.
M j.j.i.l
Zi,1+
Σ
M
M.j.k.I Zk.1
(24)
w
w
M.j.i
Zt,1 =
A Zi,1
1=1
Yyj-
The final results in Eq. 244 is equivalent to Y., A' x Zi... Thus, Theorem is proven.
18

---

Theorem 3. For any matrices Q, K ER
LER NxV
VER NxC
tte following equation holds:
Y ==((K)) LV
+ Ky einsum ('nd,nc + ndc', K, V)
C KV einsum
('nn, - mdc' L,K' ('nd, - mc', Q, LVV)
(25)
Y einsum
Proof. 1) the left part of the Eq.
25 can be rewritten as:
D
s QK where Sm,
Σ
Qm Kn,ds
(26)
N
Y m.c
Σ
mn Sm, Vn,
Σ
Lmn Σ d=l
Qmakknd Vn.
==1
2) the right part of the Eq. can be rewritten as:
250
nnd
KV
Kn,VVn.
N
N
CKV
Σ
L L
JCV
Lmn Kn,d Vn,es
m,d.c
nddie
n=1
1=
N
Ym,e
Σ
Q m CKV
Qm,d A=1
Σ
Lmm Knd KndVno Vn,e
midic
(27)
d=1
D
N
ΣΣ
a Knd Vn,e
d=1n=1
N
= Σ
Lm, EQm Vn,e-
Thus, Theorem 3 is proven. Here, the computational complexity of the left part of Eq. (25 is O(N3). The computational complexity of the first and third lines in right part of Eq. (25 is O(N). And the computational complexity of the second line in right part of Eq. (25) is (N Thus, if we can reduce the complexity of computing KV from O(N2) to O(N), then the complexity of computing Y = ((QKT) O L) V can be reduced to O(N).
A.4 Preliminaries: Complexity Analysis of Mamba2 Attention Form
In this section, present the efficient algorithm proposed in Mamba2 (6] for its attention form, achieving computational complexity of O(N).
Mamba2's Attention Form. Mamba2's attention form (i.c., structured masked attention) given by the SSD framework is formulated as:
a
aj+1
i>j
Y = (CB o 1 x, LY anj
i= i <<j
(28)
0
where X, Y YYERN are the input and output sequences, respectively, B, C E RND are input- dependent parameters learned by multilayer perceptron (MLP) layers. The 1D structured mask L E NXV is a 1-semiseparable matrix (6], and the scalar a; serves decay factor bounded in the range [0, 1]. In Mamba2, parameters C' and B in Eq. (28) correspond to the query Q and key K in ViTs, respectively. Therefore, Eq. 28 reveals that the selective state transition function in
Mamba2 is equivalent to the Hadamard product of a linear attention map CB and ID structured
mask L LDD
Naive Computation. As defined Eq. (28 the straightforward computation structured masked attention has a complexity of O(N)). In contrast, the complexity of linear attention Y (C = C X) can be reduced from O(N2) to O(N) by the associative prop- erty matrix multiplication. However, this approach not directly applicable to Eq. (28) because of the introduction of the Hadamard product.
19

---

Lemma 1. Ler L NxN be a I-semiseparable matrix and X E NxC be a matrix, the complexity of computing Y LX can be reduced from ( to ((N) by using the chunkwise algorithm in Mamba2 (o
Efficient Computation. Based on Theorem Eq. 28 can be computed follows:
B einsum 'ndd nc ndc', B,
C nX e einsum ('nn,nde ndc + mdc', L L Bx*)
(29)
Y einsum 'ndd, mdc - mc', , )x
Note that L is I-semiseparable matrix, and the second line of Eq. 29 can be reformulated
as matrix multiplication. Therefore, by applying Lemma [ the complexity of computing C can be reduced from O(N2) to O(N). Moreover, the complexity of computing L is O(N).
Consequently, the overall computational complexity of Eq. (28) is ((N.
A.5 Complexity Analysis of Polyline Path Mask
In this section, we analyze the complexity of computing the polyline path mask L, stated in Corollary [ with detailed explanation.
Corollary (Mask Complexity). The complexity of directly computing polyline path mask L via Eq. (66) and (14) is O(N which can be reduced to O(N2) by applying Theorem 1 where
N = H X.W.
Naive Computation. According to the definition in Eq. (5) the polyline path mask E HWXxH is large in size, and each element requires numerous multiplications, resulting in high computational cost. The most straightforward way to compute L is to calculate each element L,, individually. Hence, the total complexity of computing matrix is N² times the complexity of computing each element L. As defined in Eq. (16) and Eq.
i [u/W] + 1, j u mod W and Lo/W] + 1, (14) Lu, Li..k. = a. where mod W. There are || jl and |k
multiplication operations in Oiigh and Bea,,, respectively. Here, i, k range from 1 to H, and j, range from W. Therefore, the complexity of La, O(N)). Consequently, the overall complex of directly computing L is (..
Efficient Computation. [= and decomposed as:
The polyline path mask L satisfies the conditions in Theorem with Thus, based on Theorem [ the polyline path mask can be
L L L 1 LV.
(30)
For example, as illustrated in Fig. 10 [0(a), (a), the polyline path mask Eq. (16) can be decomposed as:
01.28 0
01.10 00a
muaaaa.
01.2281122
0.258118
0121
01.229222
4.23912
aiassusa
01,328u1.2
0133/113
61.2/
0sa38
01
01,38
09.12
05
osnalbss
L-
02.22821.2
02250215
02.2992.2
agasdpas
02.0/ba.22
0231/553
0s.22la1.3
08.133a,
assalasa
05.20/m1.2
0s
auaadsa,
assalusa
Josanfann
0539/An1
00.a asaaaaaa
0sa
osaafasa
toraa onna maa
0
0
Puga 0
0
Pus. 0
mis
0
Buza
0
Bus
mass
Raa
6312
6113
Pega
0322
0223
Bana
09.3
09.3
Bana
03,13
05,13
05.1:3
Baga
0
Pas
03.22
05,22/
055.2:5
8
0
Paga
0
P
05,822
0
0
Pan.
0
0.a.
0
01.1
01.15
01,25
01221
01.12
01222
01
01,23
01.13
01,21
01,12
01,22
0112:3
Pan,
0
8
1.2
0
8.3
0.23
113.1
0.0
01.35
812.1
8u22
02.15
02,1
0211
Paa.3
Paa.1
825.
8a3.1
823
02.32
Paa.
8225
Pas.1
0131
01.32
61.33
921a
521.2
8233
Pasa
8233
01,11
03,12
05,13
161,1
0.
0.0.3
01.11
03.92
00.95
03.50
00.52
09.03
09.90
03,32
03,233
3:1.1
8.12
852.2
8525
Pu5.
055.
00.31
03,22
00,33
03,31
00,52
03,33
00.31
03,32
03,3.3]
P1:11
Pan.3
Pu82
3.33
Auaa
(31)
20

---

A
A-
L
Polyline Path Scanning
(a) Polyline Path Mask Decomposition
X
X
1
X
X
B
X
4
X
B²
X
A
L
1
(b) Polyline Path Mask Multiplication
Figure 10: (a) Illustration of the decomposition of the polyline path mask (b) Illustration of the
multiplication between the polyline path mask L and vector æ. (Algorithm
Nalve Computation
Efficient Computation (Time)
120
Effic lfficient Computation (Memoryy
100%
100%
100%
10.0% 100%
100%
100%
consuming
Rabotrve memory (90)
70.25
60
60
40
5.9%%
55.3%
40
64
96X96
128x128
160X160
192x392
224w224
Figure 11: The comparison of the relative time consuming and memory usage between the naive computation and efficient computation (Algorithm 2 of Læ.
Note that there are HxW² non-zero elements in H and each non-zere zon-zero element Ossj::l requires (N+) multiplication operations. Thus, the complexity of computing L and L is O(N2). Similarly, the complexity of computing f and LV also O(N2). Thus, the complexity of computing L can be reduced to ((N2) by Eq. (30).
A.6 Complexity Analysis of Polyline Path Mask Multiplication
this section, we analyze the complexity computing the matrix multiplication between the polyline path mask L and the vector Fig. [I presents the comparison of speed and memory usage between the naive computation and efficient computation of Læ. Compared to the naive computation approach, Algorithm 2 achieves substantial speed-up and significantly reduced GPU memory consumption, especially when the shape of L and is large.
stated in Corollary 2 and Algorithm 2 with detailed explanation.
21

---

Corollary 2 (Masked Attention Complexity). The computational complexity of the matrix multiplica- tion between polyline path mask and vector æ, i.e., y= La, can be reduced from ( to ( by Algorithm? 2 and further reduced to ((N) by applying the chunkwise algorithm of Mamba2 16 to steps 3 and 5 in Algorithm 2
Naive Computation. Typically, the polyline path mask L E RR is rank-N matrix. Thus, the most direct approach to compute La requires a computational complexity of O(N3).
Efficient Computation. As mentioned above, the polyline path mask L can be decomposed as
L XL', where " and LV satisfy the defimition in Eq. (18) with [A and [B'),
Thus, based on Theorem 2] we can design Algorithm for computing the matrix multiplication between polyline path mask L and the vector æ. As shown in Algorithm 2 computing B x X has a complexity of O H Thus, the complexity of computing Z (i.e., step 3 in Algorithm 2 is O(HW). Similarly, the complexity of computing Y (i.e., step 5 in Algorithm
2 is O(HW2) Thus,
the computational complexity of Algorithm 2 is ( ), where N H x W.
As illustrated in Fig. I0 (b), the matrices A and B are symmetric matrices, and their lower triangular parts are both
Algorithm Efficient Masked Attention Computation.
Input: decay factors 3 of the polyline path mask L. vector E R NWW; Compute X unvec(()) ER #Rw
1 Isseisseparable matrices as de-
2: Compute B' T HxH where for 1= 1: :: [B] 11 11.==Baa.1
fined in Mamba2 [6]. Therefore,
Compute Z E R HXW where Z., B¹ X
by applying Lemma] T the com-
4: Compute A' E RxW where for i = 1: H, [A 1s.
plexity of computing B' x X and can be reduced from O(11²)
5: Compute Y E R HxW where Y.. = A' Ze..
Output: yevec
to O(H), and the complexity of computing A' Zi. can be reduced from O(W3) OWW.. Consequently, the overall complexity of computing Læ is O(N).
A.7 Applications of Polyline Path Masked Attention
The proposed polyline path mask can be seamlessly integrated into various attention variants in plug-and-play manner. As illustrated in Fig. 9 theorems and algorithm given in Sec. Sec.[A.5)an and See. [A6 ensure that integrating the polyline path mask does not substantially increase the computational complexity of the original attention mechanism. In this section, we introduce several Polyline Path Masked Attention (PPMA), including Polyline Path Masked Vanilla Attention (PPMVA), Polyline Path Masked Linear Attention (PPMLA), Polyline Path Masked Criss-Cross Attention (PPMCCA), and Polyline Path Masked Decomposed Attention (PPMDA).
Basic Paradigm. The basic Polyline Path Masked Attention (PPMA) is implemented by performing a Hadamard multiplication with the attention map. Specifically, given query Q. key K, and value V e IR NWCC. PPMA is formulated as:
PPMA (x) = (Attn (1 K) O L2D) V
(32)
= (Attn( Q, K) O L) V +
()
1) Polyline Path Masked Vanilla Attention, According to Eq. 32), the polyline path masked vanilla attention is formulated as:
PPMVA (X)= (softmax QK 122) V,
(33)
Based on Corollary 1 the computation of 2 has a complexity of O(22). Thus, Eq. 33 maintains the complexity of (N2)).
2) Polyline Path Masked Linear Attention. Similar to Mamba2's attention form (Eq. 28 the
polyline path masked linear attention is formulated as:
PPMLA (x)= = ((QK) o 12n)
((QKT)
(34)
22

---

Based on Theorem we can compute ((QKT) @ L) V as follows:
K einsum ('md, mc -> mdc', K, V)
C kV einsum ('nm, mdc mdc', L, K.V
(35)
Y einsum (md,, mdc + mc', Q, Q,
Eq. 35 shows that the computational complexity of Eq. 34 depends computing C KV
Based
on Corollary 2 and Algorithm 2 the computational complexity of the second line in Eq. 335 can be reduced from (NN to ((N), Thus, the computational complexity of Eq. maintains (NN).
3) Polyline Path Masked Criss-Cross Attention. The original criss-cross attention 22 employs sparse attention over tokens located in the same row or column, achieving computational complexity of (NN)). In this work, following RMT [0), we decompose criss-cross attention into vertical attention over each column followed by horizontal attention over each row. The polyline path masked criss-cross attention is formulated
PPMCCA (x)= ((SH SV O - X LLL ((SH (S
(36)
where horizontal and vertical attention maps SH SV E satisfy the form in Eq. (88) with K. I,,, of Q, K. respectively 12 Based on Theorem we can reformulate the left part of Eq. (36) as:
A' softmax Q.... K. and B' softmax Q
and Q, K. HxWXC are tensor forms
((S x SV) @ L) V = ((S x sV)
o [L x x)))
((S (S o SV
о L @ 17) V
(S" iH
SV
(37)
(S LH) (SV
GL XV
(s" L") ((SV . xV).
Note that matrices S Q and SV @ ΦLV also satisfy the form (i.e. A M in Eq. 18)
Thus, the computational complexity of Eq. (37) can be reduced to ( by Algorithm 2 Similar conclusions can also be derived for the right part of Eq. (36). Thus, the overall computational complexity of Eq.336 maintains 3)
4) Polyline Path Masked Decomposed Attention, For general decomposable attention which can be decomposed as S Si x 52, where S E Te. NxD and S2 E DDN, the polyline path masked decomposed attention is formulated as:
PPDA (X)= = ((S) x S2) c (D) ((S) x S2) O L)V + x i
(38)
According to Theorem 3 we can compute ((S) S2) O L) V as follows:
S einsum (mnd mc - mdc', S.T V)
C sv einsum (nn, mdc' L, 5%
(39)
Y einsum (nd, mdc
Based on Corollary 2 and Algorithm 2 the computational complexity of Eq. (39 can be reduced from (N2)) to O(ND). Thus, the computational complexity of Eq. is ((N)).
B Experimental Details
B.1 Architecture Details
As illustrated in Fig. 5 our backbone adopts the same four-stage hierarchical architecture as RMT 10. where the first three stages employ Polyline Path Masked Criss-Cross Attention and the final stage
23

---

employs the Polyline Path Masked Vanilla Attention. Moreover, we develop our model in three scales: tiny (PPMA-T), small (PPMA-S), and base (PPMA-B).
The detailed configurations of PPMA variants are provided in Tab. 5 Following RMT 10 the stem layer consists of five 3 x 3 convolution layers followed by GELU and batch normalization to embed the imput image into 56 x 56 tokens. The downsampling layer consists of 3 x 3 convolution layers with stride reduce the feature map's resolution. Moreover, we follow RMT [TO] and incorporate
RoPE 38
CPE 3.. and LCE (57] into the PPMA blocks. All other configurations also follow
RMT
10
Code is available htttt:://gpttmmiee www.etttu...ommmiieie
com hctp://ooggcheaazza/////M
B.2
Training Settings for ImageNet-IK
To ensure reproducibility and consistency with prior work, we follow the training strategy of RMT T0] and DeiT 44 Specifically, we employ various data augmentation techniques, including RandAug- ment [5) Mixup [52] (prob=0.8), CutMix (50) (prob=1.0). Random Erasing (55) (prob=0.25). For model optimization, we use the AdamW optimizer with cosine decay learning rate scheduler and train our model 300 epochs from scratch. The initial learning rate, weight decay, and batch size are set 0.001, 0.05, and 1024, respectively. The drop path rates for PPMA-T, PPMA-S, and PPMA-B are set to 0.1, 0.15, and 0.4, respectively. We also adopt training techniques from RMT I0 including Label Smoothing (0.1) 41 and Exponential Moving Average (EMA) 33
B.3 Training Settings for Downstream Tasks
For experiments on the ADE20K (56) and MSCOC02017 28 datasets, we follow the training
settings of TransNeXT (35), and utilize the MMDetection 2 and MMSegmentation
4 libraries
for training. Specifically, in the MMDetection (2) library, we adopt Mask R-CNN [18] as the basic framework and use the AdamW optimizer with an initial learning rate of 0.0001 and a weight decay of 0.01. The model is trained for 12 epochs with batch size of 16 using the standard 1 x schedule. In the MMSegmentation
library, we adopt UPerNet [48] as the basic framework and use the Adam optimizer with the initial learning rate of 6 x 10 and the weight decay of 0.01. All models are trained for 160K iterations with batch size of 16 on the ADE20K dataset. The input size of images is set to 512 x 512.
B.4 Throughput Comparison
To evaluate the inference speed of our model, we measure the throughput of SPMA-T/SIE on an A800 GPU with batch size of 64 and the image resolution of 224 x 224. As shown in Table the inference throughput PPMA-T/S/B decrease by 37%/30% compared to RMT-T/S/B, respectively. This is mainly caused by the additional GPU kernel launches and memory transactions required to compute the polyline path mask. As shown in Table 6 the CUDA implementation of TransNe achieves significant speedup over the PyTorch implementation. In our implementation, the polyline path mask is currently computed using PyTorch. Similar to TransNeXt, our implementation can also be optimized through engineering efforts, such using CUDA Triton-based implementations, to accelerate inference speed.
B.5 Visualization
The visualizations of the polyline path masked attention map are shown in Fig. 12 Input images are taken from the ImageNet-1K validation set, and the query token is marked by red box on each input
#Param. (M)
FL.OPs
Model
Blocks
Channels
Heads
Ratios
(G)
PPMA-T PPMA-S PPMA-B
[2, 2, 8, 2]
[64, 128, 256, 512)
[4, 4, 8, 16)
[3. 3, 3, 3]
14
2.7
33 4, 18, 4]
[64, 128. 256, 512)
[4, 4, 8, 16
[4, 4, 3, 3.
27
4.9
[4, 8, 25,
[80, 160, 320, 512]
[5, 5, 10, 16]
[4, 4, 3. 54
10.6
Table 5: Detailed Architectures of the Polyline Path Masked Attention based Vision Transformer.
24

---

Table 6: Comparison of inference speed across different models on ImageNet- IK. Throughput is
measured on an A800 GPU with a batch size of 64.
Model
#Param. (M)
FLOPs (G)
Throughput (imgs/s)
Top- (%)
BiFormer- T 57
13
2.2
1602
81.4
SMT-T 29
12
2.4
636
82.2
RMT-T 10
14
2.5
1650
82.4
TransNeXt-M (PyTorch)
35
13
2.7
742
82.5
TransNeXt-M (CUDA)
35
13
2.7
1299
82.5
PPMA-T
14
2.7
1034
82.6
CMT-S [4
25
4.0
848
83.5
Max ViT-T
45
31
5,6
826
83.6
SMT-S 29
20
4.8
356
83.7
BiFormer-5
57
26
4.5
766
83.8
RMT-S [00
27
4.5
876
84.0
TransNeXt-T (PyTorch) 35
28
5.7
508
84.0
TransNeXt-T (CUDA) 35
28
5.7
947
84.0
PPMА-S
27
4.9
612
84.2
SMT-B 29
32
7.7
237
84.3
BiFormer-B 57
57
9.8
498
84.3
CMT-B [4]
46
9.3
447
84.5
TransNeXt-1 X (PyTorch) 35
50
10.3
266
84.7
TransNeXt-T (CUDA) 35
50
10.3
436
84.7
RMT-B [10
54
9.7
457
84.9
PPMA-B
54
10.6
362
85.0
image. The decay factors and attention maps are generated by the second block of the first stage in the PPMA-T model trained on the ImageNet-1K K training set.
Input-dependent Decay Factor. As shown in Fig. 12 (b) and (c). the decay factors a and /3 learned by the network can roughly capture the edge information of objects in the feature map: decay factors edges tend be smaller (approaching zero), whereas those homogeneous regions tend be larger (approaching one). Moreover, the supervised training encourages the decay factors a and / to focus on horizontal and vertical edge information, respectively.
Polyline Path Mask. As shown in Fig. 12 (e), the polyline path mask, generated by the cumulative multiplication of decay factors, effectively captures the semantic continuity in the feature space. It maintains continuity in homogeneous regions sharing the same semantics and shows discontinuity at the edges between regions of different semantics.
Polyline Path Masked Attention Map. Fig. 12 (d) shows that attention maps from shallow layers in typical VIT models often struggle focus tokens relevant the query token. In contrast, Fig. 12 (f) demonstrates that integrating the polyline path mask 2 successfully suppresses interference from distant and irrelevant tokens, resulting in more semantically accurate masked attention maps.
C Discussion
C.1 Selectivity of Polyline Path Mask
Compared to previous state-space models (SSMs) such as RetNet [40), the primary contribution of Mamba [II] and Mamba2 [6] the introduction of a selective mechanism into the structured mask. which leads to significant performance improvements. However, current studies still lack deep understanding of this crucial selectivity mechanism.
In this work, we argue that the selective mechanism in Mamba explicitly models the semantic continuity in sequences, which corresponds to the local smoothness prior in images. Building this insight, we adopt the selective structured mask of Mamba2 and naturally generalize it into a 2D polyline path mask for Vision Transformers (ViTs).
Semantic Continuity in Sequence. Similar to self-attention maps in ViTs, the structured mask L NxN also be viewed weighting matrix that maps input tokens X e B NxC to output
25

---

POST
(a) Input Image
(b) Horizontal Dacay Factor a
(c) Vertical Decay Factor f
(d) Original ttention Mag softmax((K")
(e) Polyline Path Mask 1
(f) Polyline Path Masked A Aleetion Map
Figure 12: Visualizations of the decay factors and the polyline path masked attention maps of the well-trained PPMA model. each input image, the query token is marked by a red box.
26

---

tokens Y W along the sequence length dimension. In this weighting matrix L, a larger decay weight Li indicates a greater influence the imput token X; on the output token y, and vice versa. In Mamba2, Li,j computed as the cumulative multiplication of decay factors at to achieve linear complexity. As result, if any factor is close to zero, Li approaches zero; conversely, Lisi approaches one only when all decay factors are close to one.
For most semantic-related tasks, an ideal structured mask should model semantic continuity in the sequence: it should maintain continuous between connected tokens with the same semantic, while breaking between tokens with different semantics. This enables the aggregation and separation of tokens according to their semantics. Accordingly, decay factors should ideally be larger in homogeneous regions and smaller at heterogeneous regions. As illustrated in Fig. T2 (b) and (c), the decay factors learned through supervised training align well with this assumption.
Local Smoothness Prior in Images. In natural images, spatially adjacent patches are more likely to belong to the same object and share similar semantics. This local smoothness prior plays a crucial role in natural image processing tasks, especially those requiring fine- mnn-ggaiied feature extraction. The selectivity of polyline path mask aligns naturally with this prior by modeling semantic continuity within homogeneous regions and allowing discontinuities at object edges. Experimental results also show that integrating the polyline path mask yields significant performance improvements on the ADE20K semantic segmentation task.
C.2 3D Extension of Polyline Path Mask
Based on the decomposability, we naturally extend the 2D poly- line path mask to 3D applications. As illustrated in Fig. [3] the 3D polyline path mask L [30 can be decomposed as the multipli- cation of three ID structured masks, L XL x L., representing the horizontal, vertical, and depth scanning masks, respectively. Specifically, for each token pair (æ.,,,k, mt,m.) in the 3D grid, the 3D polyline path mask is defined as:
Y255 Depth
Vertical
(40)
where L the tensor form of matrix L3D, a, 3, and y are the decay factors along the horizontal, vertical, and depth axes, respectively. Compared to the cross-scanning strategy [30), the 3D polyline path scanning strategy better preserves the adjacency relationships of 3D tokens.
05,55
Horizontal
Figure 13: Illustration of the 3D extension of polyline path mask.
C.3 Limitations
In this work, we introduce learnable, input-dependent polyline path mask the explicit positional encoding for ViTs, replacing the fixed decay mask in RMT [TO.. Experiments on high-level tasks demonstrate the superiority of our method, particularly in fine-grained segmentation benchmarks, where PPMA-T/S/B outperform RMT-T/S/B by 0.7%//..% 13%//.%%%% SS mloU on ADE20K, respectively.
Notably, our carefully designed polyline path mask 2 is decomposable described in Eq. (30). enabling efficient computation via algorithms 2 to optimize the computational complexity. However, despite these optimizations, the large size of the mask L still inevitably incurs extra GPU memory occupation and slower inference speed compared to RMT [1O] As shown in Table the inference throughput of PPMA-T/S/D decrease by 37%/30% % in comparison with RMT-T/S/B. respectively. This limitation can be mitigated through engineering optimizations, such as CUDA or Triton-based implementations, which we plan to investigate in the future work.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately describe the proposed method (Poly- line Path Masked Attention), theoretical contributions (Efficient Computation Theory), and experimental results (experiments on image classification, object detection and segmentation tasks), which align with the content presented in the paper (Sections and 5).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in the Appendix, where the authors acknow ledge constraints such the proposed model has lower throughput than some existing models with similar FLOPs.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
28

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results in the paper are accompanied by a complete set of clearly stated assumptions and formal proofs. While the full detailed proofs are presented in the Appendix for readability, the main paper includes simple version aid understanding.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes. The paper provides sufficient details to ensure the reproducibility of its main experimental results. The overall architecture clearly described in Section Furthermore, Section 5 outlines the experimental settings in detail. These descriptions are sufficient for the reproduction to verify the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
29

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We provide a GitHub repository in the abstract in the anonymised version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public/guidess Code/udmieSeeeeee httpuuimmissssoooooooo httt:///nn...e/ooeeeeebbaaiissao
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https;
7 //nips. wttpccccpppccccccccc
httpubiiccguidsss../ guides/ Cooeeubmisseeee siioiciicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are presented in Section 5.1, 5.2 and 5.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The experiment for this task is time-consuming. Referring to previous work, there is no such experimental data.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational cost have been presented in the Appendix for readability
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully reviewed the NeurlPS Code of Ethics and confirm that our research fully adheres its principles.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
31

---

This paper presents work whose goal is to advance the field of Deep Learn- ing and Computer Vision. None the potential societal consequences we feel must be specifically highlighted here.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve pretrained models, generative tools, or scraped datasets that carry a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly credited all existing assets used in our work, including publicly available datasets and code repositories.
Guidelines:
The answer NA means that the paper does not existing assets.
32

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaeed, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The primary new asset is the source code for the proposed methods, which is open source. Documentation is assumed be provided alongside the code its repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
33

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non non-standary components.
Please refer to our LLM for what should or should not be described.
34


Reviews:
[{"review_id": "mYVx0ras1g", "reviewer": "Reviewer_8Ffr", "summary": "The authors propose a novel method called ​​Polyline Path Masked Attention (PPMA)​​, which effectively integrates the structured masking concept from Mamba2 with the self-attention mechanism in Vision Transformers (ViT). Specifically, the paper first introduces an innovative ​​2D Polyline Path Scanning​​ strategy that better preserves the 2D adjacency relationships among image tokens. Building upon this, the authors derive a corresponding ​​polyline path mask​​ formulation. The experimental results demonstrate that the proposed PPMA model achieves state-of-the-art (SOTA) performance across various scales.", "questions": "See my  comments on the weakness.", "limitations": "While the paper presents a well-motivated fusion of Mamba and ViT with thorough theoretical analysis and extensive experiments, several key questions remain unaddressed. The network architecture design (Figure 5) lacks justification for its hybrid use of PPMCCA and PPMVA Blocks, with no ablation study to validate this choice. The learned decay factors (α_ij and β_ij) in the polyline path mask are not analyzed for interpretability (e.g., correlations with image semantics or edges), missing an opportunity to strengthen the method’s transparency. Presentation issues arise, such as relegating Figure 5 to the appendix and an unclear description of attention stages in Section 4.4. Discrepancies in RMT benchmark results versus the original paper raise reproducibility concerns, while the absence of a larger PPMA-L model leaves open whether scaling could close the performance gap with RMT in tasks like COCO detection/segmentation. Addressing these would solidify the work’s rigor and practical insights.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. The research motivation is well-justified, as combining the strengths of Mamba and ViT represents a promising direction. \n2. The paper provides theoretical analysis of the proposed polyline path masking and derives its computationally efficient algorithm, demonstrating solid foundational work.\n3. Extensive experiments have been conducted across multiple vision tasks with comprehensive comparisons against various baseline methods.\n\n\nWeaknesses:\n1. In Figure 5, why is the network structure designed this way? Why not use only PPMCCA Blocks or only PPMVA Blocks? If this network architecture is optimally designed, could the paper provide corresponding analysis?\n2. The decay factors (α_ij and β_ij) in the polyline path mask are learned through MLP. The paper could briefly discuss whether these learned factors exhibit any interpretable patterns, such as potential correlations with the semantic content or edge structures of the image.\n3. It is recommended to relocate Figure 5 from the appendix to the main text to provide readers with more intuitive visual understanding.\n4. Revise “where the first three stages employ Polyline Path Masked Attention” in Section 4.4 to “where the first three stages employ Polyline Path Masked Criss-Cross Attention”\n5. Why do the RMT results in some tables differ from those reported in the original paper, while other models remain consistent with the results in the RMT paper?\n6. I'm curious whether there is a PPMA-L model, because as the model size increases, the performance gap between PPMA performance and RMT is narrowing, such as the effect of object detection and instance segmentation in COCO val2017.", "was_revised": true, "final_justification": "The authors have addressed all my questions.", "rebuttal": "We sincerely thank the reviewer for acknowledging our **research motivation, theoretical analysis with efficient algorithm design, and comprehensive experiments**. Please see below for our point-by-point responses to your comments.\n\n- **Question 1 (Weaknesses 1)**: In Figure 5, why is the network structure designed this way? \n\n  > **Answer**: **The utilized network structure in Figure 5 strictly follows the structure design of previous SOTA work in the same category, RMT ([8] in the manuscript),  for a fair comparison.**\n  >\n  > This hybrid design balances computational efficiency with performance. In the early stages of the network, the token sequence length is large. Employing criss-cross attention (PPMCCA) in these stages significantly reduces computational complexity, since it is a sparse attention variant. In the final stage, the token sequence is much shorter, so using full vanilla self-attention (PPMVA) introduces only marginal computational overhead while enhancing the model’s ability to capture global dependencies. Such a design should be superior to using only PPMCCA or PPMVA, but whether it is the optimal design still requires further verification. \n  >\n  > In this paper, **the overall network architecture is not the focus of our research**. Therefore, to facilitate fair comparisons with previous methods, we chose this design and leave the exploration of better overall structures for future work.\n  >\n  > Following the reviewer’s suggestion, we will appropriately incorporate the aforementioned network structure analysis into the final version of the paper if it is accepted.\n\n- **Question 2 (Weaknesses 2)**: The paper could briefly discuss whether these learned factors exhibit any interpretable patterns, such as potential correlations with the semantic content or edge structures of the image.\n\n  > **Answer**: We appreciate the reviewer's insightful comment. Actually, we have already provided a discussion on our insight about decay factors in **Appendix B.6 and C.1**, through detailed experimental observations and theoretical analysis. Our conclusions are fully consistent with the reviewer's speculation: **these factors do exhibit correlations with the semantic content or edge structures of the image.**\n  >\n  > **1) Interpretable Visualization Results.** The supervised learning process encourages the horizontal decay factors ($α_{ij}$) and vertical decay factors ($β_{ij}$) to focus on horizontal and vertical edge information, respectively. As shown in Figure 7 of Appendix B.6 (particularly the cup example in the 8th row), we observe that the learned decay factors exhibit interpretable patterns that correspond well to image structures such as edges and semantic boundaries.\n  >\n  > **2) Theoretical Insight: Connection to the Local Smoothness Prior in images.**  As further analyzed in Appendix C.1, an ideal polyline path mask should align well with the *local smoothness prior* commonly assumed in natural images. This implies that:\n  >\n  > - Within homogeneous regions that share the same semantics, the decay factors ($α_{ij}$ and $β_{ij}$) should be large (closer to 1), maintaining continuity in the polyline paths.\n  > - At semantic boundaries or edge structures, these decay factors should be small (closer to 0), allowing the mask to appropriately capture discontinuities.\n  >\n  > As suggested by the reviewers, we will include a brief discussion on the interpretability of decay factors in the final version of the paper, with appropriate references to the relevant experimental results and analyses in the Appendix.\n\n- **Question 3 (Weaknesses 3)**: It is recommended to relocate Figure 5 from the appendix to the main text.\n\n  > **Answer**: We will relocate Figure 5 (or a simplified version) to the final version if it is accepted.\n\n- **Question 4 (Weaknesses 4)**: Revise “where the first three stages employ Polyline Path Masked Attention” in Section 4.4 to “where the first three stages employ Polyline Path Masked Criss-Cross Attention”.\n\n  > **Answer**: Thank you for pointing out the issue.  We have revised it as suggested.\n\n- **Question 5 (Weaknesses 5)**: **Why do the RMT results in some tables differ from those reported in the original paper?**\n\n  > **Answer**: As the most important competing counterpart of our method, **we reproduced RMT results under the same experimental settings as our method** for a fair comparison.  Specifically, we trained both our models (PPMA) and the RMT baselines using a batch size of **1024**, due to hardware constraints—we only have access to a single 8-GPU (RTX 4090 with 24GB memory) machine, which cannot accommodate larger batch sizes because of limited GPU memory.  \n  >\n  > **The reason for the discrepancies between our reported RMT results and those in the original paper lies in the batch size used during training**. The model weights provided in their GitHub repository were trained with a batch size of **2048** using **16 A100 GPUs (80GB memory)**. It is worth noting that, possibly due to an oversight by the RMT authors, the RMT paper states that a batch size of 1024 was used on ImageNet-1K, which differs from the information disclosed in the GitHub repository.  In contrast, we have rigorously verified through multiple repeated experiments that the results we provide truly reflect the expected performance of RMT under this configuration (batch size = 1024). Batch size can affect  classification results by **0.1% to 0.2%**, which also affects downstream COCO detection results since they rely on ImageNet-1K pretrained weights.\n  >\n  > In conclusion, we are confident that **all experiments in our paper—including RMT baselines—were conducted under fair and reproducible settings**. To ensure full transparency, we released the complete training code, conda environments, training logs, and model checkpoints for both **PPMA and reproduced RMT baselines** in the anonymous codes (hosted on Google Drive) a month ago.\n  >\n  > We will explicitly clarify the experimental settings in the final version to prevent any potential confusion. Please see the response to the first question of reviewer gCJq for more detailed explanation.\n \n- **Question 6 (Weaknesses 6)**: **I'm curious whether there is a PPMA-L model, because as the model size increases, the performance gap between PPMA performance and RMT is narrowing, such as the effect of object detection and instance segmentation in COCO val2017.**\n\n  > **Answer**: We  fully acknowledge the reviewer's valuable point that evaluating PPMA-L is indeed crucial for validating our method's performance on large-scale models. However, due to **limited computational resources** in our lab (equipped with only 8×RTX 4090 GPUs), we were unable to conduct PPMA-L experiments under standard benchmark configurations. Therefore, to maintain scientific rigor, we opted not to include PPMA-L results in the current manuscript.\n  >\n  > That being said, **we have in fact conducted a series of PPMA-L experiments on smaller-scale datasets.** Specifically, we have conducted preliminary comparisons between **PPMA-Large (PPMA-L)** and **RMT-Large (RMT-L)** (previous SOTA method) on the **ImageNet100** dataset (using a smaller batch size of 512). The results (summarized in the table below) show that **PPMA-L outperforms RMT-L by 0.3%** top-1 accuracy, suggesting that the performance gap does not vanish at larger scales and that PPMA’s effectiveness generalizes across model sizes. \n  >\n  > \\begin{array}{lccc}\\hline \\text{Model}&\\text{Param. (M)}&\\text{FLOPs (G)}&\\text{Top-1 Acc.}(\\\\%) \\newline\\hline \\text{RMT-L}  & 94.5 & 18.2 & 88.9 \\newline \\text{PPMA-L} & 94.6 & 18.6 & \\mathbf{89.2} \\newline \\hline\\end{array}\n  >\n  > In the final version, we will incorporate these additional experimental results into the appendix."}, {"review_id": "QE56MSq3pv", "reviewer": "Reviewer_xW4s", "summary": "This submission presents a method that builds upon Mamba2 for vision. It defines polyline \"scanning paths\" based on the L-shaped manhattan path between pairs of image/spatial locations. In the \"dual\" attention formulation this is an attention mask based on the product of decay factors along the polyline. There is an efficient way to compute this mask based on decomposing it into 1D vertical and horizontal scanning. The forward pass is computed using the dual form as structured mask attention, as part of an overall backbone architecture with 4 stages with two attention different variants using this mask. It's tested and compared to other architectures on ImageNet-1K, COCO, and ADE20K.", "questions": "- **Q1**: Is [36] intended to be included as one of the SSM-based comparison methods in the citation around line 261 in Section 5?\n- **Q2**: Around line 215 in Section 4.2 it is stated that the lower triangular parts of $A^k$ and $B^j$ are 1-semiseparable. Is the argument (however brief) as to *why* matrices constructed this way are 1-semiseparable stated explicitly in the paper? I did not see it.\n- **Q3**: Footnote $^2$ on page 4 states \"We apply the ReLU and exponential operator after the MLP layers to ensure $\\alpha_{i,j}, \\beta_{i,j} \\in [0, 1]$. What's the formula that applies these operators to ensure this? Do these values correspond to the `dt_alpha` and `dt_beta` on line 170 of https://anonymous.4open.science/r/PPMA-3948/classification/models/PPMA.py?\n- **Q4**: Which result does https://anonymous.4open.science/r/PPMA-3948/training_logs/classification/ppma_small_log.txt correspond to? Is the Top-1 test accuracy for this run reported as 84.1% or 84.2%?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "There's a number of works that consider masks or other means to impose the spatial prior on vision transformers and state space models. This submission presents experiments on a good selection of comparison methods that also include spatial relationships, such as [6, 40, 44]. (Plus \"Vicinity Vision Transformer\" by Sun et. al. T-PAMI 2023 also considers Manhattan distance, on the attention/transformer side.)\n\n### Strengths\n\n1. Experiments consider a variety of image tasks.\n\nNotably, this includes both image-level labels and dense prediction (instance segmentation with COCO and semantic segmentation with ADE20K). We might expect that the role of local vs global information might be different between these (\"MambaOut: Do We Really Need Mamba for Vision?\" by Yu and Wang, CVPR 2025), and both are empirically tested.\n\nComparisons include backbone size and computation cost to make it clear what confounding tradeoffs might be present in those metrics between different architectures.\n\n2. Gives a computation complexity analysis of both computing the mask and the overall attention.\n\n3. Structure in PPMA intuitively gives higher weight based on locality.\n\n4. Very well-organized code, including training logs\n\n### Weaknesses\n\n5. Presents only a structured attention form of the method, not the state space model form. (Unless I have missed this.) The motivation of the paper emphasizes how this method is constructed to build on Mamba2.\n\n6. Overall accuracy differences on a lot of the comparisons are quite small. (0.1-0.2% in Table 1) While tending to also require a bit more computation (FLOPs) than the base RMT architectures.\n\nGenerally, this seems like a notably solid and well-executed paper with middle-range potential impact and originality.", "was_revised": true, "final_justification": "The answers to Q1-Q3 can be addressed by changes to the camera-ready that the authors describe. Though the statement connected it to Mamba2's mathematical results might benefit from a more mathematically formal description than \"we adopt the same 1D scanning strategy as in Mamba2\" to allow checking this result more carefully.\n\nFor Q4: The authors are correct that this is a standard technique in reporting eval results in the literature now. I have seen it in other contexts too. I do not like that this is the standard, but it is, so it makes sense not to count it against my score, to be consistent with the precedent from esteemed researchers & senior reviewers like our ACs & PCs.\n\nSo I raise to \"5: Accept\"", "rebuttal": "We are honored to receive the reviewer's recognition of our work as **“a notably solid and well-executed paper”**, and the acknowledgment of its **\"potential impact and originality\"**. Our work is inspired by Mamba2 and introduces a new structured masked attention mechanism that enhances the state space duality (SSD) framework in Mamba2. Please see below for our point-by-point responses to your comments.\n\n- **Reference Suggestion:** Plus \"**Vicinity Vision Transformer**\" by Sun et al. T-PAMI 2023 also considers Manhattan distance, on the attention/transformer side.\n\n  > **Response**: We will include a discussion of this work in the related work section.\n\n- **Weaknesses 1**: Presents only a structured attention form of the method, not the state space model form. (Unless I have missed this.) The motivation of the paper emphasizes how this method is constructed to build on Mamba2.\n\n  > **Response**: We acknowledge that our method, PPMA, is only presented in a structured masked attention mechanism, without any traditional state space model form. However, we want to emphasize that the close relationship between state space models and structured masked attention has been explicitly discussed in Mamba2 [4]. As detailed in Section 5.3 and Figure 4 of Mamba2 [4], **\"state space models with scalar-identity structure on the $A$ matrices, and structured masked attention with 1-semiseparable structure on its $L$ mask – are duals of each other with the exact same linear and quadratic forms.\"**  This duality enables more flexible methodological design, and the proposed PPMA is built upon this foundation.\n  >\n  > **Notably, the proposed method is not directly constructed by human intuition alone, but is naturally derived through refinements within the Mamba2 framework**. The proposed polyline path mask $L^{2D}$ is a 2D adaptation of Mamba2’s $L^{1D}$, maintaining the same approach for generating decay factors (an MLP followed by ReLU and an exponential operator). The primary difference is that we replace the original 1D scanning strategy in Mamba2 with our 2D polyline scanning strategy.  Moreover, in terms of the overall architecture, the proposed method remains consistent with Mamba2:\n  > $$\n  > Mamba2: {\\rm{SMA}}\\left( x \\right)  = \\left( {Q} {K^{\\top}} \\odot {L}^{1D} \\right) {x} ,\n  > $$\n  >\n  > $$\n  > Ours: {\\rm{PPMVA}}\\left( {{x}} \\right) \n  > \t\t\t= \\left( {{\\rm softmax}(Q{K^ \\top }) \\odot {L^{2D}}} \\right)V.\n  > $$\n  >\n  > Therefore, we claim that our method is based on Mamba2. We will clarify this more clearly in the revised manuscript.\n  >\n  > ------\n  >\n  > [4] Dao T, Gu A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality[J]. arXiv preprint arXiv:2405.21060, 2024.\n\n\n\n- **Weaknesses 2**: Overall accuracy differences on a lot of the comparisons are quite small. (0.1-0.2% in Table 1) While tending to also require a bit more computation (FLOPs) than the base RMT architectures.\n\n  > **Response**: Thank you for your feedback. We would like to make the following clarifications:\n  >\n  > **1) SOTA baseline makes further improvements quite challenging.** We selected the state-of-the-art RMT model as our baseline without incorporating any incremental tricks.  This ensures the fair comparison with the RMT method on the fundamental modules. However, RMT already achieves very high performance on ImageNet1k, making further improvements quite challenging.\n  >\n  > **2) The proposed PPMA achieves more significant improvements on fine-grained tasks.** As shown in Appendix Figure 7, the polyline path mask effectively suppresses interference from distant and irrelevant tokens, leading to more semantically accurate masked attention maps. This makes PPMA especially effective for dense prediction tasks such as semantic segmentation. For example, PPMA-S models outperform RMT-S by **1.3% SS mIoU** and **2.3% MS mIoU** on the ADE20K dataset, respectively. To further verify this, we additionally conducted experiments on the **Cityscapes** semantic segmentation task (results shown in the table below). Results show that PPMA-T outperforms RMT-T significantly by **0.8% SS mIoU** and **0.6% MS mIoU**, respectively.\n  >\n  > \\begin{array}{lccc}\\hline \\text{Backbone}&\\text{Input Size}&\\text{SS mIoU} (\\\\%) &\\text{MS mIoU} (\\\\%) \\newline\\hline \\text{RMT-T}  & 1024\\times1024 & 81.9 & 82.9 \\newline \\text{PPMA-T} & 1024\\times1024 & \\bf{82.7} & \\bf{83.5} \\newline \\text{RMT-S}  & 1024\\times1024 & 83.3 & 83.9 \\newline \\text{PPMA-S} & 1024\\times1024 & \\bf{83.7} & \\bf{84.0} \\newline \\text{RMT-B}  & 1024\\times1024 & 83.6 & 84.1 \\newline \\text{PPMA-B} & 1024\\times1024 & \\bf{83.9} & \\bf{84.3} \\newline \\hline\\end{array}\n  >\n  > **3) PPMA can degenerate into RMT.** Importantly, we would like to emphasize that **RMT can be viewed as a special case of PPMA**, where all learnable decay factors are fixed constants. Therefore, **our PPMA model is theoretically at least as expressive as RMT**. The learnable decay factors, while offering greater performance potential, inevitably lead to an increase in FLOPs.\n\n  \n\n- **Question 1:** Is [36] intended to be included as one of the SSM-based comparison methods in the citation around line 261 in Section 5?\n\n  > **Answer**: **EffNet-B3 [36] is a CNN-based model**, not an SSM-based method. We included it for comparison following VMamba.\n  >\n  > ------\n  >\n  > [36] Tan M, Le Q. Efficientnet: Rethinking model scaling for convolutional neural networks[C]//International conference on machine learning. PMLR, 2019: 6105-6114.\n\n\n\n- **Question 2**: Why the lower triangular parts of $A_k$ and $B_j$ are 1-semiseparable?\n\n  > **Answer**: As stated in line 255 in Section 4.2, **matrices $A_k$ and $B_j$ correspond to 1D horizontal scanning along each row and 1D vertical scanning along each column, respectively.** Particularly, we adopt **the same 1D scanning strategy as in Mamba2** to construct matrices $A_k$ and $B_j$. This implies that the lower triangular parts of $A_k$ and $B_j$ are 1-semiseparable, since Mamba2 [4] has shown that a matrix constructed in this way is 1-semiseparable.\n  >\n  > To avoid potential confusion, we will revise the text in Section 4.2 to provide a clearer explanation.\n\n\n\n- **Question 3**: Footnote 2 on page 4 states \"We apply the ReLU and exponential operator after the MLP layers to ensure αi,j,βi,j∈[0,1]. What's the formula that applies these operators to ensure this? Do these values correspond to the `dt_alpha` and `dt_beta` on line 170 of PPMA.py?\n\n  > **Answer**: Yes, thank you for carefully reviewing our code. These values indeed correspond to the `dt_alpha` and `dt_beta` on line 170 of the code. The precise formula is:\n  > $$\n  > \\alpha_{i,j}= {\\rm exp}(-A \\times {\\rm SoftPlus}({\\rm MLP}(x_{i,j}))),\n  > $$\n  > where scalar $\\alpha_{i,j} \\in [0,1]$, $x_{i,j} \\in \\mathbb{R}^{1 \\times C}$ denotes the input token, $\\rm SoftPlus(\\cdot)$ is a smooth alternative to $\\rm ReLU(\\cdot)$  to ensure ${\\rm SoftPlus}({\\rm MLP}(x_{i,j}))>0$, and $A$ is a positive parameter. Decay factor $\\beta _{i,j} \\in [0,1]$ is computed using the same formula. Here, this formulation follows the decay factor computation approach in Mamba2 [4] and corresponds exactly to the code.\n  >\n  > ------\n  >\n  > [4] Dao T, Gu A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality[J]. arXiv preprint arXiv:2405.21060, 2024.\n\n- **Question 4**: Which result does `ppma_small_log.txt` correspond to? Is the Top-1 test accuracy for this run reported as 84.1% or 84.2%?\n\n  > **Answer**: It is **84.2%**, achieved at **epoch 276** (`\"test_ema_acc1\": 84.186`).  Specifically, we follow the commonly adopted evaluation protocol on ImageNet-1K, consistent with methods such as RMT, Spatial-Mamba, and VMamba, by reporting the **best validation epoch** (rather than the final epoch) as the final result. Additionally, we adopt the same EMA (Exponential Moving Average) strategy as used in RMT to mitigate overfitting (please refer to Appendix B.2, Line 234)."}, {"review_id": "uXWuo2JVCQ", "reviewer": "Reviewer_gCJq", "summary": "The authors propose a new way of casting attention by incorporating a new structures mask inspired by the Mamba2 architectures into classical self-attention – and demonstrate that this explicit modelling of the spatial adjacency priors present in images can improve results across various vision tasks.", "questions": "**[Q1]:**  RMT-results repeatedly stated lower than in original RMT paper: The results presented in this work for RMT on various tasks is consistently lower than what is stated in the original RMT paper (both arxiv and CVPR version); I’d like the authors to comment on this, as the original RMT results are mostly on par and sometimes better than the results of the method proposed here!   \nExamples:   \n$\\textrightarrow$  Detection: RMT-T stated as 46.7 instead of 47.1 (APb); 68.6 vs 68.8 (APb50); 51.6 vs 51.7 (APb75) | 42.1 vs 42.6 (APm), 65.3 vs 65.8 (APm50), 45.2 vs 45.9 (APm75).  Similar for other model sizes.  \n$\\textrightarrow$  Classification: RMT-B stated as 84.9 instead of 85.0 (orig.), RMT-S as 84.0 instead of 84.1 (orig.)\n\n*Note*: I don’t expect a new method to outperform all others, but this needs clarification!\n\n**[Q2]:**  Given the claim of ‘plug-and-play’ integration of the mask, I would be quite interested how the method performs on a plain ‘monolithic’ ViT (i.e. no down-sampling, etc);  \n$\\textrightarrow$  Can the masking approach be simply integrated, and if yes, how does it perform?  (I think this could provide quite distinct and ‘clean’ insights, given that ViT is a very simply yet powerful architecture)   \n$\\textrightarrow$  Given that it is inspired by Mamba2, and other Vision-Mamba works exist – have the authors ablated how simply switching out their mask would perform in such architectures? \n\n**[Q3]:**  FLOP counts only provide a partial impression of the actual efficiency of a model; While I can see that some empirical throughput is provided in the appendix (Table 3), it would be interesting to add this to the step-by-step ablation in Table 4 of the main paper to have a side-by-side comparison of benefit vs. throughput decrease!\n\n**[Q4]:** What exactly is the difference of your scanning approach to 2DMamba? I see that some comparison is provided in the appendix within the novelty statement, but this seems to relate mainly to the fact that the entire 2DMamba approach is built on Mamba(1), not the scanning strategy itself. Some clarification/contrasting would be valuable here.  \n\n*Minor*:   \nTable 4 and Figure 6 have a slight mismatch that could be improved: Images show the Hilbert map while the Table does not (and instead V2H); Replacing the ‘w/o Mask’ image through the V2H one and adding the Hilbert-Mask results to the Table would improve consistency. \n\n---\n**TL;DR**: I like the paper, and am happy to reconsider my rating once the questions have been addressed -- but especially the first one is an issue that needs clarification!\n\n---\n---\n## Post-Rebuttal Update:   \nAs detailed below, my questions and concerns have been sufficiently addressed, and I have raised my score accordingly.", "limitations": "Limitations sufficiently discussed: The authors briefly discuss some limitations in the conclusion, and add another more detailed discussion section in the appendix – which could be moved to the main paper if space permits in the final version, given that it provides an **honest discussion of actual drawbacks, which I very much appreciate!**", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths  \n**Originality & Significance:**  \n- Clear motivation of the method based on the identified strengths and weaknesses in Mamba and ViT-like architectures – providing the basis for the approach proposed here\n- The authors demonstrate the applicability via improvements across three different vision tasks and for three model sizes\n\n**Quality:**  \n- Contributions are generally well-placed within the wider area of research\n- Quite detailed and rigorous introduction of the polyline path mask computation (including underlying theory how to make it efficient)\n- Appropriate number of experiments across tasks to show versatility of method\n\n**Clarity:**  \n- Contributions and underlying motivation clearly outlined in the introduction\n- Explanations well-supported through a good mix of figures, algorithm and formulas\n- The paper is well written and easy to read and follow; Further details found in the appendix, but the paper provides a set of great visualizations paired with the main formulas to easily and thoroughly explain the main ideas\n\n\n## Weaknesses\n- Inconsistency between the results presented in the RMT paper and the ones stated here for RMT – stated repeatedly lower (see questions)  \n- Paper could benefit from integrating the mask in other architectures – given the claim that it works in a ‘plug-and-play’ manner; see questions  \n- Some smaller inconsistencies in comprehensiveness of results, see questions.", "was_revised": true, "final_justification": "The authors have provided **thoughtful responses to all my questions, including new insights and additional results where required**.  \n\nI have also read through all the other reviewers' questions, and I did not find any additional ones that would raise concerns.  \nThe authors also provide many valuable insights in the appendix, and recommendations to include a few into the main paper were made by me and other reviewers.  \n\nI'd like to also highlight the authors' **proper and honest discussion** of potential limitations of their method, which I value quite highly (as pointed out in my review already).  \n\n**$\\rightarrow$ I do not see any prohibitive concerns, and recommend the paper for acceptance.**", "rebuttal": "We sincerely thank the reviewer for the positive evaluation of our work. We truly appreciate your recognition of the **\"clear motivation\"**, **\"quite detailed and rigorous introduction of the polyline path mask computation\"**, and **\"well-written\" paper paired with \"great visualizations\"**. We feel very fortunate to receive such a high-quality review. Please see below for our point-by-point responses to your comments.\n\n- **Question 1**: RMT-results repeatedly stated lower than those in original paper.\n  > **Answer**: Thank you for the constructive comment. We can clarify this issue from the following three aspects:\n  >\n  > **1) Discrepancy Caused by Batch Size Differences.**  As the most important competing counterpart, **we reproduced RMT results under the same experimental settings as ours**.  Specifically, both RMT and our PPMA are trained with a batch size of **1024**, due to hardware constraints—we only have access to a single 8-GPU (RTX 4090 with 24 GB memory) machine, which couldn't support a larger batch size.  \n  >\n  > **The discrepancies between our reproduced RMT results and those in the original paper stem from batch size differences during training**.  The released RMT weights provided in GitHub repository were trained with a batch size of **2048** on **16 A100 GPUs** (80GB memory). It is worth noting that, possibly due to an oversight, the RMT paper states that a batch size of 1024 was used on ImageNet-1K, which differs from the information disclosed in the GitHub repository. This can be confirmed by the code snippet below:\n  >\n  > ```\n  > checkpoints = vars(torch.load(r\"download/RMT-B.pth\")['args'])\n  > print(f\"gpu_num: {checkpoints['world_size']}, batch_size_per_gpu: {checkpoints['batch_size']}\")\n  > print(f\"batch size: {checkpoints['world_size']*checkpoints['batch_size']}\")\n  > \"\"\"\n  > >>> gpu_num: 16, batch_size_per_gpu: 128\n  > >>> batch size: 2048\n  > \"\"\"\n  > # world_size = the actual number of GPUs used\n  > ```\n  >\n  > Batch size can affect classification results on ImageNet-1K by **0.1% to 0.2%**, which also affects downstream COCO detection results since they rely on ImageNet-1K pretrained weights.\n  >\n  > **2) We made sure to reproduce RMT fairly and without bias**.  In fact, we have rigorously verified RMT results via multiple repeated experiments.  For example, RMT-S was trained on ImageNet-1K three times using the official code, achieving Top-1 accuracy of 83.86%, 83.92%, and 83.95%. We reported the best (84.0%) in Table 1. In addition, we have included the comparison results on **MSCOCO 2017** using Mask R-CNN with the **3× MS schedule** (36 epochs with multi-scale inputs), where PPMA-B outperforms RMT-B.\n  > \\begin{array}{lcccccc}\\hline \\text{Backbone}&AP^b &AP^b_{50} &AP^b_{75} &AP^m &AP^m_{50} &AP^m_{75}\\newline\\hline \\text{RMT-B (original paper)}&52.2&72.9&57.0&46.1&\\bf{70.4}&49.9\\newline \\text{RMT-B (reproduction)}& 52.1&73.1&57.2&46.2&70.3&50.0\\newline \\text{PPMA-B (Ours)}&\\bf{52.6}&\\bf{73.3}&\\bf{57.5}&\\bf{46.3}&70.3&\\bf{50.2} \\newline \\hline\\end{array}\n  >\n  > **3) Theoretical relationship to RMT.** Importantly, we would like to emphasize that **RMT can be seen as a special case of our method**, where all learnable decay factors are fixed constants. This relationship is analogous to that between Mamba2 and RetNet. Therefore, **our PPMA model is theoretically at least as expressive as RMT**, and potentially more powerful in practice.\n  >\n  > ------\n  >\n  > In conclusion, we are confident that **all experiments in our paper—including RMT baselines—were conducted under fair and reproducible settings**. To ensure transparency, we released the complete training code, conda environments, training logs, and checkpoints for both **PPMA and reproduced RMT** in the anonymous code (hosted on Google Drive) a month ago.\n  >\n  > As suggested, we will explicitly clarify the aforementioned experimental settings in the final version.\n\n- **Question 2.1**: About ‘plug-and-play’ claim: how the method performs on a plain ‘monolithic’ ViT (i.e., no down-sampling)? Can the mask be simply integrated, and if yes, how does it perform? \n  > **Answer**: Yes. The integration process can be easily understood by comparing the formulations of self-attention and PPMA:\n  > $$\n  > {\\rm Attention}(x)={\\rm softmax}(QK^\\top)V.\n  > $$\n  > \n  > $$\n  > {\\rm PPMVA}(x)=({\\rm softmax}(QK^\\top)\\odot L^{2D})V.\n  > $$\n  >\n  > Simply performing element-wise multiplication between the attention map and our mask enables seamless integration. Thus, we consider the proposed mask to be 'plug-and-play'.\n  >\n  > **Example on plain ‘monolithic’ ViT:** We use the **DeiT** model (which does not involve down-sampling) as an example. Only a few lines of DeiT code need to be modified:\n  >\n  > ```\n  > mask = self.generate_polyline_path_mask(x) \n  > attn = (q @ k.transpose(-2, -1)) * self.scale\n  > \n  > # attn = attn.softmax(dim=-1) # PPMA just replaces this line with the next line\n  > attn = (0.5*torch.softmax(attn + mask, -1) + 0.5*torch.softmax(attn + mask.permute(0, 1, 3, 2), -1))\n  > y = attn @ v\n  > ```\n  >\n  > Experimental results with and without the polyline path mask on ImageNet-1K are shown in the table below: \n  >\n  > \\begin{array}{lccc}\\hline \\text{Model}&\\text{Param. (M)}&\\text{FLOPs (G)}&\\text{Top-1 Acc.}(\\\\%) \\newline\\hline \\text{DeiT-tiny}&5&1.3&72.2\\newline \\text{DeiT-tiny+PPMA}&5&1.3&\\bf{72.7} \\newline \\hline\\end{array}\n  >\n  > The results show that incorporating the mask can lead to a performance improvement. In the final version, we will include these experimental results into the appendix.\n\n- **Question 2.2**: Have the authors ablated how simply switching out their mask would perform in other Vision-Mamba architectures?\n\n  > **Answer**: We would like to clarify that the proposed **plug-and-play mask is designed for attention variants**, such as vanilla self-attention, linear attention, and sparse attention—not for all architectures.\n  >\n  > Most existing Vision-Mamba works (e.g., VMamba) are based on Mamba1, which cannot be reformulated into an attention variant like Mamba2. Thus, our mask cannot be simply integrated into these architectures.\n  >\n  > As stated in Section 3, Mamba2 can be reformulated as the **structured masked linear attention**:\n  > $$\n  > Mamba2:{\\rm SMA}(x)=(QK^\\top \\odot L^{1D})x,\n  > $$\n  >\n  > Thus, our polyline path mask $L^{2D}$ can be directly used to replace $L^{1D}$, while maintaining a complexity of $\\mathcal{O}(N)$:\n  > $$\n  > {\\rm PPMLA}(x)=((QK^\\top)\\odot L^{2D})V.\n  > $$\n  > However, Mamba2-based vision models are still relatively rare. Therefore, we have not evaluated our mask on them. We consider this a valuable direction for future work as the Mamba2 ecosystem matures.\n\n- **Question 3**: It would be interesting to add throughput to Table 4 to have a comparison of benefit vs. throughput decrease!\n  > **Answer**: Thanks for the suggestion. We have added **throughput** in Table 4. The results show that all models using learnable masks tend to have relatively low throughput.\n  >\n  > \\begin{array}{lccc}\\hline \\text{Model}&\\text{Throughput (imgs/s)}&\\text{Top-1 Acc.} (\\\\%) &\\text{SS mIoU} (\\\\%) \\newline\\hline \\text{Baseline (w/o mask)}&1779&82.28&47.78\\newline \\text{+ RMT Decay Mask}&1650&82.35&48.01\\newline \\text{+ Cross Scan Mask}&1100& 82.44&48.14\\newline \\text{+ Hilbert Scan Mask}&1091&82.41&48.19\\newline \\text{+ V2H Polyline Path Mask}&1203&82.44 &48.57\\newline \\text{+ 2D Polyline Path Mask}&1034&\\bf{82.60}&\\bf{48.73} \\newline \\hline\\end{array}\n\n- **Question 4**: What is the difference between your scanning approach to 2DMamba? \n  > **Answer**: While our scanning strategy is similar to 2DMamba, our core contribution lies in **integrating Mamba2’s structured mask mechanism into ViTs**. The detailed distinctions include:\n  >\n  > **1) Plug-and-play Flexibility.** We would like to reiterate that 2DMamba lacks the plug-and-play flexibility as ours. Specifically, Mamba1-based 2DMamba uses a diagonal matrix $A$ as the decay parameter, whereas our method follows Mamba2 and adopts a scalar $a$ instead. **This distinction is crucial**, as only a scalar decay factor allows for the derivation of the polyline path mask and the efficient algorithm, which in turn supports plug-and-play usage.\n  >\n  > **2) Independent Horizontal and Vertical Decay Factors.** Our method uses **separate** horizontal decay factor $\\alpha_{i,j}$ and vertical decay factor $\\beta_{i,j}$—to capture directional information separately. In contrast, 2DMamba uses **a shared decay parameter $A$**. Ablation study in Appendix B.2, Table 2 (Line 6 vs. Line 7) shows that a shared decay factor leads to a **0.23%** drop on ImageNet-1K.\n  >\n  > **3) V2H+H2V Scanning.** Unlike 2DMamba’s one-way horizontal-to-vertical (H2V) scanning, our method combines **V2H and H2V scanning** to ensure **symmetric decay** between tokens, i.e., the decay weight from token $x_{i,j}$ to $x_{k,l}$ equals its reverse.  Table 4 (Lines 4 vs. Line 5) shows that this improves accuracy by **0.16%** on ImageNet-1K. \n  >\n  > We will revise the appendix to clarify these distinctions. Lastly, since 2DMamba was released very recently and our scanning strategy was developed independently, we consider this a **concurrent work**. \n\n- **Question 5 (*Minor*)**: Table 4 and Figure 6 have a slight mismatch that could be improved.\n  > **Answer**: **Hilbert-Mask results** have been added to Table 4 (please see our response to Question 3). We will revise Table 4 and Figure 6 in the final version as suggested.\n\n- **Question 6**: More detailed limitations in the appendix could be moved to the main paper if space permits in the final version.\n  > **Answer**: Thank you for the suggestion. We will move detailed limitations to the main paper as suggested. \n\nWe hope our responses have addressed your questions satisfactorily. The revision, incorporating your suggestions, has indeed made the paper more comprehensive. We’d be grateful for a higher rating if you feel it’s warranted."}, {"review_id": "Vi1RctROvT", "reviewer": "Reviewer_SWqC", "summary": "This paper proposes a new relative positional encoding, called polyline path scanning/mask, for ViT, where the adjacency of visual tokens in 2d images can be maintained under the Manhattan distance.", "questions": "The paper starts with a discussion on ViT and Mamba (RNN) for CV tasks, and how ViT and Mamba2 can be more effectively combined. However, it turns out later that this is NOT the focus of the research at all. \n\nThe proposed relative position encoding method is orthogonal to Mamba, and can be used in any models that require to encoding position info of visual tokens. The detailed discussion of pros and cons of Mamba2 and Mamba-ViT hybrid models seem unnecessary. Since Mamba-ViT hybrid models have been comprehensively explored in language models, such a discussion seems dilute the novelty of the paper. The paper should have focused on positional encoding for visual tokens, using Mamba2 / VIT as justification and experimentation.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The new positional encoding method, as detailed in Sec. 4, is well-motivated, simple to implemented, and mathematically sound. The effectiveness of well-justified in the empirical study.\n\nWeaknesses: the presentation can be improved. The paper starts with a discussion on ViT and Mamba (RNN) for CV tasks, and how ViT and Mamba2 can be more effectively combined. However, it turns out later that this is NOT the focus of the research at all. The proposed relative position encoding method is orthogonal to Mamba, and can be used in any models that require to encoding position info of visual tokens. The detailed discussion of pros and cons of Mamba2 and Mamba-ViT hybrid models seem unnecessary. Since Mamba-ViT hybrid models have been comprehensively explored in language models, such a discussion seems dilute the novelty of the paper. The paper should have focused on positional encoding for visual tokens, using Mamba2 / VIT as justification and experimentation.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback. We are delighted that you consider our method to be “**well-motivated, simple to implement, and mathematically sound**”. Please see below for our point-by-point responses to your comments.\n\n\n\n- **Questions 1**: The presentation can be improved. The paper starts with a discussion on ViT and Mamba (RNN) for CV tasks, and how ViT and Mamba2 can be more effectively combined. However, it turns out later that this is NOT the focus of the research at all.\n\n  > **Answer**: Thank you for your valuable feedback. We acknowledge that the initial presentation may have caused some confusion, and we will **revise and reorganize the introduction and motivation** in the final version to more clearly convey the core focus of our work.\n  >\n  > We would also like to clarify that, while our core contribution lies in the proposed polyline path mask $L^{2D}$, the motivation of our study is indeed to **leverage the complementary strengths of Mamba2 and ViTs**. Furthermore, the proposed method is not directly constructed by human intuition alone, but is naturally derived through refinements within the Mamba2 framework. Frankly speaking, the polyline path mask can be seen as **a carefully adapted and extended 2D variant** of the structured mask $L^{1D}$ in Mamba2, tailored for image data.\n\n\n\n- **Questions 2**: The proposed relative position encoding method is orthogonal to Mamba, and can be used in any models that require encoding position info of visual tokens. The detailed discussion of pros and cons of Mamba2 and Mamba-ViT hybrid models seems unnecessary. Since Mamba-ViT hybrid models have been comprehensively explored in language models, such a discussion seems to dilute the novelty of the paper. The paper should have focused on positional encoding for visual tokens, using Mamba2 / ViT as justification and experimentation.\n\n  > **Answer**: Thank you very much for your valuable comments. We appreciate your suggestion and will revise the manuscript to clarify the novelty and focus of our work. We agree that the proposed mask, as a form of relative positional encoding, can be applied to various attention mechanisms beyond Mamba2.\n  >\n  > We also agree that **the core novelty of our work lies in the design of the 2D polyline path mask**. In Section 4, we provide a systematic exposition of this design, including its **definition**, **properties** (e.g., decomposability), **theoretical guarantees** (efficient algorithm), and **practical applications** (e.g., PPMVA and PPMCCA).\n  >\n  > In the final version, we will further refine the structure of the *Introduction* and *Methodology* sections to place stronger emphasis on the proposed mask as a general-purpose positional encoding method for visual tokens."}]

Author rebuttal:

Meta-review: All the reviewers unanimously vote for acceptance. After checking the review, the rebuttal, the manuscript, and the discussion, the AC agrees with this assessment.
Decision: Accept (spotlight)