GSPN-2: Efficient Parallel Sequence Modeling
Authors: Hongjun Wang
yitong jiang
Collin McCarthy
David Wehr
Hanrong Ye
Xinhao Li
Ka Chun Cheung
Wonmin Byeon
Jinwei Gu
Ke Chen
Kai Han
Hongxu Yin
Pavlo Molchanov
Jan Kautz
Sifei Liu
Venue: NeurIPS 2025 poster ()
Keywords: computer vision; deep learning
Primary area: deep_learning

Abstract: Efficient vision transformer remains a bottleneck for high-resolution images and long-video related real-world applications. Generalized Spatial Propagation Network (GSPN) \cite{wang2025parallel} addresses this by replacing quadratic self-attention with a line-scan propagation scheme, bringing the cost close to linear in the number of rows or columns, while retaining accuracy. Despite this advancement, the existing GSPN implementation still suffers from (i) heavy overhead due to repeatedly launching GPU kernels, (ii) excessive data transfers from global GPU memory, and (iii) redundant computations caused by maintaining separate propagation weights for each channel. We introduce GSPN-2, a joint algorithm–system redesign. In particular, we eliminate thousands of micro-launches from the previous implementation into one single 2D kernel, explicitly pin one warp to each channel slice, and stage the previous column's activations in shared memory. On the model side, we introduce a set of channel-shared propagation weights that replace per-channel matrices, trimming parameters, and align naturally with the affinity map used in transformer attention. Experiments demonstrate GSPN-2's effectiveness across image classification and text-to-image synthesis tasks, matching transformer-level accuracy with significantly lower computational cost. GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications through its unique combination of structured matrix transformations and GPU-optimized implementation.

GSPN-2: Efficient Parallel Sequence Modeling
Hongjun WanghAt, Yitong Jiang', Collin McCarthy', David Wehr', Hanrong Ye', Xinhao Li
Ka Chun Cheung', Wonmin Byeon', Jinwei Gu Ke Chen', Kai Han?;
Hongxu Yin', Pavio Molchanov Jan Kautz', Sifeci Liu²
NVIDIA
"The University of Hong Kong UUniversity of California, San Diego
Abstract
Efficient vision transformer remains a bottleneck for high-resolution images and long-video related real-world applications. Generalized Spatial Propagation Net- work (GSPN) [ addresses this by replacing quadratic self-attention with a line- scan propagation scheme, bringing the cost close to linear in the number of rows or columns, while retaining accuracy. Despite this advancement, the existing GSPN implementation still suffers from (i) heavy overhead due to repeatedly launching GPU kernels, (ii) excessive data transfers from global GPU memory, and (iii) redundant computations caused by maintaining separate propagation weights for each channel. We introduce GSPN-2, a joint algorithm-sy-ee ular, we eliminate thousands of micro-launches from the previous implementation into one single 2D kernel, explicitly pin one warp to each channel slice, and stage the previous column's 's activations shared memory. On the model side, we introduce a set of channel-shared propagation weights that replace per-channel matrices, trimming parameters, and align naturally with the affinity map used in transformer attention. Experiments demonstrate GSPN-2's effectiveness across image classification and text-to-image synthesis tasks, matching transformer-level accuracy with significantly lower computational cost. GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications through its unique combination of structured matrix transformations and GPU-optimized implementation.
redesign. In partic-
1 Introduction
Vision transformers have underpinned nearly every state-of-the-art (SOTA) vision foundation model: text-to-image diffusion networks (e.g., Stable Diffusion (2)), vision-language aligners such as CLIP (3] and SigLIP [4) and modern detection//sgmentattoon pipelines [5] 6 all depend on their dense, token-wise attention to encode visual concepts. Since this attention operator scales quadratically with the number of pixels, practical deployments still cap the Siput-SigLIP [4], for instance, limits the input images to 512 x 512-to avoid prohibitive latency and memory. Recently, several -ettenttttttnnioon variants have been proposed, such FlashAttention 77 8) linear atten- tion [ 10] and state-space models [12 13.. Among them, Generalized Spatial Propagation Networks (GSPN) [1] uniquely replace 2D self-attention with a line-scan approach, which reduces the computational complexity from quadratic to approximately linear to the image's 's width or height. Remarkably, GSPN maintains even surpasses baseline accuracy while achieving up to an 84x speed-up for 16 K-resolution diffusion inference.
While most efficient attention backends can reuse existing matrix-multiply or scan primitives, GSPN's line scan demands completely new CUDA implementation. Standard Softmax attention breaks down into series of GEMM-based matrix multiplies plus a softmax [4] FlashAttention 7
Corresponding author.
Hongjun Wang intern at NVIDIA during the project.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

instead fuses those steps into single tiled GEMM loop. State-space methods like Mamba [2 13 recasts attention into streaming recurrence and implement it with fast prefix-sum scans across tokens.
By contrast, GSPN adopts a 3-neighbour line-scan approach, which is neither a matrix multiply, nor a prefix scan its dependency pattern would explode combinatorially. Therefore, a specifically built CUDA kernel for GSPN is required unlock its sub-quadratic cost.
Linerr
Time per image (ms)
Atlaniin
10a
Perfomer
Lifarme
Norto
Manba
10.
0SP%-3
ESPN-1
(Locel)
0SPW22
Ous
252
1034
4095
1G184
ESPN2 (Local)- Durs
Resolution (pixels) per side
Comparison
Figure I: GSPN-2 achieves transformative performance improvements over GSPN- I and other efficient attention variants, running up to 30-50x faster across diverse input configurations modern GPU architectures.
While GSPN-1 offers theoretical advantages in computational complexity, its initial CUDA imple- mentation [T] struggled to translate these into practical speedups. Profiling reveals that GSPN-1's reference CUDA code, which launches tiny kernel for each column step, severely underutilizes the GPU, it achieves just 3-8% of peak memory bandwidth and low SM occupancy. This inefficiency stemmed from several critical bottlenecks: (1) the kernel-launch overhead from thousands of separate launches, which prevents the SMs from staying fully busy: (2) inefficient global-memot (HBM) access, with each step reloading data without on-chip reuse coalescing; and (3) poor cache locality and growing runtime as channel counts increase.
To overcome the limitations of the original GSPN implementation, we introduce GSPN-2, an inte- grated algorithmic and kernel-level redesign. Specifically, GSPN-2 (a) consolidates all propagation steps into single unified CUDA kernel, eliminating costly repeated launch overheads; (b) employs channel-agnostic propagation method, where weights are shared across channels, significantly reducing parameter count and improving memory coherence; and (c) optimizes block/grid configura- tions and SRAM management to minimize latency. Additionally, to address the runtime bottleneck arising when the GPU hits its concurrency limits the product of batch size and channel count grows large, we introduce a lightweight projection of input features into a low-dimensional proxy space before propagation. This reduces the effective computational dimension during the intensive scan, mitigating linear scaling issues and enhancing cache reuse (see Section 4.2p. Thus, these algorithmic and system-leve improvements yield substantial efficiency gains: on an NVIDIA 1 GPU, runtime for representative 1024 x 1024 8 input drops from over 71.4 ms in GSPN-1 down to just 1.8 ms in GSPN-2, achieving 40 speedup (as shown in Figure 3
Our experimental evaluation comprehensively validates GSPN-2. Rigorous efficiency analysis demonstrates that GSPN-2 rums up to 30x faster than GSPN-1 across diverse input configurations, with performance profiling confirming near-optimal hardware utilization (over 90% of theoretical peak memory bandwidth). We then validate GSPN-2's s effectiveness across vision tasks: image classification and text-to-ine to-image synthesis. On ImageNet, GSPN-2 2 achieves accuracy comparable to transformer models at significantly lower computational cost. In text-to- text-to-image synthesis, GSPN-2 significantly improves semantic consistency and visual quality when integrated with existing diffusion models. These results confirm GSPN-2 as versatile component for efficiently modeling global spatial context across diverse vision applications.
Related Works
Efficient Attention Mechanisms. Transformer architectures [15] have become foundational com- ponents in modern vision and language models, but their quadratic computational complexity with respect to sequence length creates significant efficiency challenges. FlashAttention [7] 8 14 ad- dresses these limitations through algorithmic innovations that optimize memory access patterns and reduce unnecessary memory reads/writes during attention computation. By leveraging tiling strategies

---

and fusing operations to maximize data reuse within fast GPU memory hierarchies, FlashAttention substantially improves throughput and enables processing of longer sequences without compromising model quality. These efficiency gains have been instrumental in scaling transformer models to increasingly larger contexts and higher resolution inputs, but the fundamental quadratic complexity of attention remains inherent limitation.
Sequence Modeling in 1D and 2D Space. Sequential modeling has been dominated by recurrent architectures like LSTMs [66. GRUs [7]], and 2D-LSTMs 19), which process data through non-linear transformations. Despite their effectiveness, these approaches face fundamental limitations in computational efficiency and scalability due to their inherent sequential nature. Their non- non-linear propagation mechanisms also struggle with long-term dependencies, often suffering from gradient vanishing or exploding issues (20 1 that prevent distant information from effectively influencing future states. State Space Models (SSMs) have emerged as promising alternatives to attention-bass architectures, offering linear computational complexity with respect to sequence length. Pioneering approaches like S4 [22] and Mamba [12] implement continuous-ting dynamical systems through discretized state spaces, enabling efficient modeling of long-range dependencies without the quadratic cost of attention. These models maintain a compact hidden state that evolves through linear recurrence relations, often employing selective scanning mechanisms to adapt to input-dependent patterns. For visual tasks, several approaches [23 24 25 26 27 have adapted SSMs by linearizing 2D image data, though this transformation potentially compromises inherent spatial relationships present in the original data structure.
Spatial Propagation Networks. The Spatial Propagation Network (SPN) (28] pioneered linear propagation specifically for 2D data, initially designed as single-layer component on top of CNNs for sparse-to-dense prediction tasks like segmentation. However, SPN's potential as a scalable foundational architecture comparable to Vision Transformers (ViT) remains largely unexplored. Moreover, SPN's sequential processing across different spatial directions inherently limits its compu- tational efficiency, and it fails to adequately address long-range propagation requirements crucial for high-level vision tasks. Our GSPN architecture advances beyond these limitations by implementing parallel row/column-ww while maintaining gradient stability and effective long-range correlation. Through both theoretical analysis and empirical evaluation, we demonstrate that GSPN represents a compelling alternative to established ViT and Mamba architectures.
propagation mechanisms that enable efficient learning of affinity matrices
3 BBckkround
We introduce the background of the propagation algorithm itself and the GPU design principles. In Section 3.1] we review modern A100 architecture grid/block/warp execution model, on-chip shared memory, and high-bandwidth device memory. We explain how these features shape kernel performance. In Section (..2 we review GSPN's line-scan propagation formulation. The final section summarizes how this recurrence is mapped onto CUDA blocks in a custom kernel implemented in i to realize parallelism.
3.1 GPU Hardware Characteristics
On modern NVIDIA GPUs like the A100, computation is dispatched as grid of thread blocks. Each block can be ID. 2D. or 3D (c.g.. blockDim....... alone for ID block, both blockDing mid.imm... x and blockDi m. for 2D block). Inside each block, threads are organized into warps of 32 threads each. The total number of warps per block depends on the block's 's thread count (for example, a block with 1024 threads has 32 warps). To maximize throughput, blocks must be sized to supply enough active warps per Streaming Multiprocessors (SM) without exceeding their on-chip shared-memory or register limits-this balance is what drives high occupancy. Within each block, threads share a small SRAM buffer (" ("shared memory") for low-latency reuse, while all other tensors are streamed in and out of off-chip high-bandwidth memory (HBM) through the L./LL1 caches.
3.2 2D Spatial Propagation Algorithm Overview
Generalized Spatial Propagation Networks (GSPN) (1 perform 2D spatial modeling through row- by-row (or column- by-ouumm linear propagation. For an input image x E " xc this involves processing one dimension (e.g., rows) sequentially, while computations within each step (e.g., along row) are parallelized. Focusing on row-wise propagation, where E [0, H 1] is the row index, let hine E IR be the hidden state for row i and channel c, Zi,,, E F. be the corresponding input
3

---

Outer Loug: image column
Outer Loop: image column
kereet I-1
Kemell
komel I+1
Kernel Fused
Mtepa-
step
step
tep
step
step 1+1
HBM
Image/twmuue
Tesching
CasTro FOURS
L1V2 Crcch
column
X.G.H.B.AA
X,G.H.B.A
SRAM
bookk
Ki alM... 4 ONl
soving
Mf of + AF
Kemmli
Draan Soop
soving
GSPN-1 [1]
GSPN-2
Figure 2: Pipeline optimization from GSPN-1 GSPN-2. GSPN-1 launches separate kernels for each image column i, leading to redundant HBM access and limited temporal data reuse. Each kernel independently computes hf uf 1 + A @ 2f. fetching and storing intermediate states via global memory. GSPN-2 fuses these operations into single kernel with an inner loop over columns, enabling cache and register reuse of 1 G₁, and other temporaries. This design minimizes memory traffic, maximizes locality, and leverages shared memory for efficient on-chip computation.
row, and Xi.. E TR be a learnable, input-dependent scaling vector for channel at row i, and Wiee E R R WWW be a learnable, per-channel propagation weight matrix for row and channel c. The per-channel recurrence is:
hine Wi,chi 1,;,e Diag (
The per-channel hidden state hine is then transformed by output layer. Let W4,s, learnable output vector. The final output for row i and channel c is given by:
RR
be
gi,ic Mi hige
(2)
Guided by the Stability- Conttitt Condition introduced in M. Wi.e is learned and normalized to be now-stochiastic-evee
row sums to 1-thereby guaranteeing numerical stability while still capturing long-range context. In addition, each element in row i connects to only three neighboring elements in the previous row i - 1 (e.g., top-left, top-center, top-right for -o-tt-bottomm propagation), resulting in wi. being a tridiagonal matrix. A single pass of this recurrence connects pixels within local region. To achieve full-grid propagation, GSPN performs four complementary directional passes-top-to- bottom, bottom-to- tooooppp left-to-right and right-to-left. By combining the 3-neighbor kernel with these four passes, the model attains dense pairwise connectivity across the image while remaining efficient, since only three coefficients are learned per pixel.
This row-wise propagation requires (HH) sequential steps, where within each step, all W elements of the current row are computed in parallel (or vice versa for column-wisg propagation), yielding an ( H, W)) computational GSPN also offers local variant, i.e., GSPN-local, which splits each or column into fixed-length segments of size kchunk and confines propagation to within those segments. Finally, this recurrence is related to linear attention. Let A; = blockdiag( (Doaa Diiag(A..,))....... .)...)..... Diag(Ar..c-i)) The overall operation is y/- u, E1= w-) A4z which resembles a linear attention mechanism, with 1+ wy the normalization term
cumplexty-eqquivalle
to O( (NN) for square images with N pixels.
3.3 CUDA Implementation in GSPN
The baseline GSPN implementation. referred to as GSPN-1, maps the 2D spatial propagation (Eq. to CUDA by iterating sequentially over the propagation dimension height H) while parallelizing computations across the orthogonal dimension (e.g., width W). To handle the inherent sequential dependency along the propagation axis (e.g., rows = O..... H - 1). it launches separate, relatively lightweight CUDA kernels for individual steps small chunks, illustrated in Figure a
For each step in the propagation sequence (e.g., processing a column based on the previous in a left-to-right scan), GSPN-1 launches a new CUDA kernel, which introduces significant kernel- launch overhead along the propagation direction. Within each kernel, computations are parallelized across the orthogonal spatial dimension (width W), batches (N), and channels (C) by flattening these dimensions into a 1D grid of thread blocks (typically bll blocDDim. x 512). However, this simplistic mapping ignores CUDA's warp-level scheduling and often results in suboptimal hardware utilization. Morcover, all tensors- -inaudiinnnnnnggggo inputs (x)), hidden states from the previous step (f.-1..c).

---

learnable weights (W. A....), and the current outputs (ha,.e) are repeatedly read from and written back to global GPU memory (HBM), causing high latency and minimal data reuse in on-chip memory. These freues- kernel launches, suboptimal thread mapping, and excessive mobal-y access access-togents together limit the efficiency of GSPN-1, motivating the redesign presented in Section 4
4 GSPN-2: Efficient Algorithm and System Co-design
While the baseline GSPN-1 implementation established functional correctness, its CUDA design suf- fered from ineffcieenies--freeunnn kernel launches, flat 1D block configurations, and unpredictable memory reuse, which led to suboptimal data locality and high launch overhead. To address these limitations, we redesign both the algorithm and its GPU execution pipeline through GSPN-2, focusing on three principles: (1) a single-kemel propagation scheme that eliminates redundant launches, (2) channel- compprssivee propagation with shared weights and proxy compression to reduce concurrency load, and (3) optimized CUDA execution leveraging shared memory, coalesced access, and stream- level parallelism. This section introduces the evolution of GSPN-2, from its single-kerne design to memory- and ancurrency-awar optimization.
4.1
A Single-Kernel Design
We consolidate these numerous small kernels into single, unified CUDA kernel. This single kernel is designed to process the entire outer-loop (e.g., all columns in a left-to-right scan) within the kernel, while still parallelizing computations across the other dimensions (batch, channels, and rows/height). By eliminating thousands of micro-launches. this hnngle-krnel approach drastically reduces launch overhead. For instance, preliminary tests showed that simply moving from a multi-kernel to a single-kernel implementation for a typical GSPN configuration immediately yielded significant performance boost (e.g. 1.2x faster) in Figure 3 even before other memory or algorithmic optimizations were applied. We illustrate substantial performance gains from this and subsequent optimization stages across various hardware configurations and input dimensions (batch size, channels, height, width) in Figure and Section 5.1
Kernel Fuse.
Block and Grid Configuration. In GSPN-I, the kernel used a flat ID grid of blocks (blockDing iim 51 where threads were linearly mapped across combinations of batch (N), channel (C), height (H), and chunk index (kehunk). This configuration resulted insufficient locality and suboptimal warp utilization. In GSPN-2, the CUDA grid is indexed by the tuple (chunk, n, c). so that each block corresponds to one unique (chunk, n, c) combination and processes full spatial column along height. The grid therefore contains ketunk x N x C blocks in total, which can be realized as either a ID grid or a 3D grid to respect CUDA's per-axis limits. Each block uses up to 1024 threads along the height dimension. For H < 1024, one thread is assigned per row, achieving full occupancy. When H 1024, threads iterate over multiple rows with stride blockDinm Dim. x, ensuring complete coverage without exceeding the per-block thread limit.
4.2 Compact Channel Propagation
key performance bottleneck in GSPN-1 arises from GPU concurrency saturation when the number of active CUDA Blocks-proportione execution capacity. On GPUs such as NVIDIA A100, each Streaming Multiprocessor (SM) can host up to 32 resident thread blocks (compute capability 8.0), and with 108 SMs available, roughly 108 x 32 se 3,500 blocks can be active concurrenlyy under ideal conditions. Under typical GSPN workloads, kernel execution time remains nearly constant up to this scale (about 3-4K concurrent blocks). Beyond that point, the runtime grows linearly as additional blocks wait in the scheduling queue. This saturation effect causes G to lose its near-constant runtime scaling when operating on high-dimensional feature maps (e.g., thousands of channels).
to Kehunk X N x C-exceeds the hardware's concurrent
To address this, GSPN-2 introduces a compact multi-channel propagation scheme that reduces the effective channel concurrency while maintaining expressive multi-channel behavior. The core idea is to project the input tensor
proxy subspace Eproxy E IR XC x H HW where Cprogy & C (c.g.. Oproxy = 8). The propagation is then applied to this proxy representation using shared propagation matrices w; and later restored to the original C-channel space. This reduces the total block count from kichunk N C' to kebunk X N x Cpruxy- reducing it as much as possible to stay well within the hardware concurrency regime (roughly 3-4K on A A100-class GPUs) and thereby sustaining near-constant performance.
IR NxCxHX into a lower-dimensions
5

---

Illustrative Single-Channel Case. We use the single-channel case to make the attention analogy explicit. We replace per-channel weights with a single propagation matrix per column. In this view, wy will be shared among all the channels, which plays the role of an attention-style affinity matrix over positions in column i, and the per-por per-position input scaling acts like value gating. The per-channel recurrence thus becomes:
ha 11: + Ai..c @ 24,,. wwh,-1... + (
(3)
where W; governs spatial propagation along the column, and Ae..e preserves per-channel modulation. This formulation significantly reduces the number of parameters while retaining the same functional structure. Stacking all channels, the full recurrence h; = W,h;-1 + AsZ, still holds, now with channel-shared w,. To expand Eq. (4.2), denote X, as the concatenation of all hi,e and
Tise into vectors. The expansion yields a block nower-triangule matrix form:
A
0
W2 A1
A2
0
0
W3
WyA2
A3
0
H,
X. GX,,
(4)
I A1
T 20 A2
WLAZ-1 NLJ
where each block Gij is N x N matrix representing how the input slice æj contributes to the output h2 dirrectctl analogous to attention mechanism's 's affinity matrix. Here, the channel-shared matrices w, define dense spatial relationships. while the channel-specific scaling matrices A, inject feature-wise modulation. This formulation shows that, in the single-chand case, GSPN-2 can be viewed as an attention-like process with learnable spatial affinities.
Compressive Proxy Dimension. To further relieve concurrency saturation when N is large, we compress the channel axis before propagation. Concretely, we project x E NXCCCXX to Prrxxy E R XCprocy xH with Cproxy < C (e.g., 8 apply the same columnwise recurrence in the proxy space using the shared wi, and expand back to C' with learned 1 x 1 projection. This reduces the grid from ketunk N x C kichon& N Cproxy. shrinking the number of simultaneously scheduled block slices (e.g.. N x Cproxy X H for a row scan). We choose Cprocy to minimize the active-block budget and delay entry into the post-saturation, near-linear regime on A100-class GPUs; even when that plateau cannot be fully avoided (very large N), the compression still cuts queueing and improves SM utilization while preserving multi-channel expressiveness.
4.3 Efficient CUDA Scaling under Large Block-Slice Loads
This section presents CUDA kernel enhance- ments particulaatylryy grid/block reconfiguration and on-chip memory strategies that enable ef- ficient compact channel propagation even when the block count (k (k_chunk x N x C)) becomes very large.
Baseline (GSPN-1)
71.4
Unified Kernel
57.4-x122
Coslesced Mermory Access
2.4 +x 23.9
SRAM
2.2 -x
SRAM for Hidden States. We cache the pre- vious step's is hidden state (e.g., h;-1) in on- chip shared memory to reduce redundant global- memory (HBM) reads. Within each CUDA block, threads cooperatively process tile small subset of spatial positions or channel slices-and reuse the cached hidden-state val- ues stored in shared memory. This on-chip reuse reduces latency when multiple threads within block access overlapping regions of h;-1, such as along a spatial column. The performance gain depends configuration: it is most effec- tive when the reuse per tile is high, the shared- memory footprint fits comfortably within per- block limits, and bank conflicts are minimal. When reuse is low 1.1/12 caching already covers the working set, the benefit diminishes. Therefore, we enable this shared-memory caching selectively and tune tile size and cS i ce to balance reuse against occupancy.
2.1 e-x 1.1
Compressive channels
1.9 e- 1.1
(Fnal (Fnal)
GSPN-2
1.8 +x 1.0
40.00 0
20
40
60
Forward Time (ms) Figure 3: Step-by-step optimization of the GSPN CUDA kernel. Each bar shows the reduction in forward time (ms) achieved through cumulative optimizations, starting from the GSPN-1 baseline. The final implementation (GSPN-2) achieves 40.0x speedup compared the baseline.

---

Propagation. Building upon the 1D block design in Sec. [4.]] extend it to a 2D configuration by introducing second dimension, cSl ice, such that = (H, eSlice). Within block, thread spatial positions along column (up to H). while threadIdx y spans small group of channel slices. This enables the block to process multiple channels of the same column in parallel, improving hardware utilization and memory throughput even when each channel maintains its own propagation weight wf. Compared to the earlier ID block layout, this 2D configuration achieves better occupancy and reduced latency by aligning computation and memory access patterns across both spatial and in channel dimensions, as demonstrated in Section 5.1
2D Block Design for Channel-
each CUDA block has block
x corresponds to
Coalesced Memory Access. A major source of speedup in GSPN-2 comes from enforcing coalesced global-memonl access. We arrange Zi, h1g and w; contiguously in memory so that consecutive threads in warp access adjacent addresses when reading or writing. This pattern allows the hardware to combine multiple per-thread transactions into a single wide memory operation, fully utilizing the available bandwidth and minimizing wasted cycles. By eliminating the irregular, strided accesses present in GSPN-I, the coalesced layout contributes the largest single performance gain among all CUDA-level optimizations (see Figure 3
Stream-Basee Baann Concurrency. For multi-directional propagation, GSPN-2 executes each directional pass on a separate, non-blocking CUDA stream. This allows concurrent kernel execution across directions, improving hardware utilization by keeping more SMs active- a abundant SM resources. The benefit depends on workload balance and available parallelism; is most pronounced when the directional passes have similar compute and memory footprints. In addition, when the grid dimension exceeds CUDA's per-axis limit of 65,535, GSPN-2 automatically performs multiple launches with offset indexing to cover the excess range without interrupting the overall stream concurrency.
GPUs with
5 Experiments
To demonstrate the effectiveness of GSPN-2, we design experiments that answer two questions: How much faster is it? and Does the speed-up preserve-or even improve-task performance? We first profile the new CUDA kernel, isolating the gains from different factors, such as unified kemeel launch, shared memory, and channel-share weights. We then benchmark GSPN-2 comparing accuracy and throughput against GSPN-1 and other strong baselines. By evaluating both efficiency and task performance, we demonstrate the benefits of our tightly integrated algorithm and kernel optimizations.
suite of vision tasks,
5.1 Detailed Profiling and Performance Characteristics
To understand GSPN-2's performance characteristics in depth, we conducted comprehensive pro- filing across various input configurations, analyzing memory throughput, cache utilization, and computational efficiency.
stseep CUDA Optimization. We benchmark typical configuration, i.e., 1024 1024 image size, batch size 16, channels, and quantif'y the impact of each CUDA kernel optimization term in Figure 3 The GSPN-1 baseline exhibited suboptimal performance (71.4 ms) due to kernel launch overhead and inefficient memory access patterns. Our first optimization-a single fused kernel (Sec. 4.1 eliminates thousands of micro-launches by processing entire scan operations within
single kernel, yielding a notable 1.2x speedup (57.4 ms). Coalesced Memory Access patterns
(Sec. 4.3) maximized memory bandwidth utilization for a substantial 23.9x improvement (2.4 ms). Implementing Shared Memory Cache for hidden states (Sec. 4.3) reduced global memory traffic by 1.1x (2.2 ms). Restructuring to 2D Thread Blocks (Sec. 4.. Sec. improved thread organization and data locality for another 1.1x gain (2.1 ms). Compressive channels (Sec . (4.)) reduced parameter fetch overhead and enhanced cache coherence for 1.1x speedup (1.9 ms). The fully optimized GSPN-2 implementation achieves an impressive 40.0x cumulative speedup (1.8 ms) over the original baseline. We note that the relative impact of each optimization varies with workload characteristies (batch size, channel count); Section B.I the appendix provides detailed analysis under an alternative large-batch configuration (batch size 256, 1 channel), demonstrating that while coalesced memory access remains the dominant optimization, Shared memory caching and 2D thread blocks (Sec. 4.3) cxhibit configuration-ddpendee benefits.
7

---

Memory Throughput Analysis. As shown in Table NVIDIA Nsight Compute profiling indicates that GSPN-2 achieves memory throughput near the theoretical limit, with global-memory efficiency reaching 93% on A 100 GPUs. This efficiency remains remarkably stable across wide range of batch sizes and spatial resolutions, demonstrating effective saturation of the available bandwidth. In contrast, GSPN-1 exhibits highly variable throughput only 3 3-8% of peak that further deteriorates as input dimensions increase.
Table I: Global memory throughput under typical input configurations on A100 GPU. We show throughput for range of input sizes, batch sizes, and channel counts representative of common deployment scenarios in different tasks. Rather than exhaustively sweeping all variables, we select practical configurations to demonstrate consistent and significant gains of GSPN-2 over GSPN-1 across diverse settings.
Input Size
Batch
Channels
GSPN-1 Throughput
GSPN-2 Throughput
32x32
32
196
114 GB/s (6.0%)
1832 GB/s (91.8%)
64x64
768
86 GB/s (4.5%)
1847 GB/s (92.3%)
64x64
1152
35 GB/s (2.1%)
1837 GB/s (92.0%)
64x64
32
125 GB/s (6.3%)
1830 GB/s (91.5%)
128x128
32
98 GB/s (4.9%)
1865 GB/s (93.3%)
256x256
64
76 GB/s (3.8%)
1842 GB/s (92.1%)
256x256
8
64
94 GB/s (4.7%)
1858 GB/s (92.9%)
512x512
128
64 GB/s (3.2%)
1840 GB/s (92.0%)
Performance Scaling with Size. As shown in the upper row of Figure GSPN-2 consistently outperforms GSPN-1 across various image resolutions with fixed batch and channel counts. For large image sizes (0224x1024),
observe speedups of up to 36.8x for forward passes and 25.3x for backward passes. This substantial improvement is particularly relevant for high-resolution visual processing tasks such as image generation and super-rssolution, where spatial dimensions significantly impact computational demands. The performance gap widens image resolution increases, highlighting GSPN-2's superior ability to handle spatially dense computations efficiently through its optimized memory access patterns and unified kernel design.
Performance with Varying Batch Size and Channel Dimensions. The lower row of Figure demonstrates GSPN-2's exceptional performance in scenarios requiring large batch sizes or high requirements for video generation, foundation model visual towers, and multimodal applications. With three distinct performance lines (GSPN-1 and GSPN-2), we observe that GSPN-2 maintains consistent 2-4x speedups over GSPN-1 even as batch sizes scale to 256 or channel counts increase to 1024. For instance, when processing inputs with 256 channels, GSPN-2 achieves a 27.4x speedup on forward passes and 48 6x on backward passes. These improvements particularly valuable for production inference systems handling multiple streams simultancously or for models requiring high feature dimensionality. The channel-sharing approach (Sec. 4.2 provides additional efficiency gains of up to 1.5x in these demanding scenarios, enabling practical deployment of GSPN architectures in compute-inee intennsie applications like real-time video processing and multimodal foundation models.
channel dimensions-
L1 Cache Effectiveness. One surprising finding from our profiling is the effectiveness of the LI cache even without explicit shared memory caching in certain configurations. When we experimented with shared memory buffer to store previous hidden states (h,-1). 1), we observed that performance remained largely unchanged compared to relying on I.1 cache. Detailed profiling revealed cache hit rates of approximately 35% for the standard implementation. Interestingly, when using shared memory explicitly, L1 hit rates dropped to near 0%, with those accesses now served from shared memory instead. Despite this shift memory hierarchy usage, latency remained comparable between both approaches, suggesting modern GPÚ LI caches are highly effective for structured access patterns. The transposed data layout and coalesced access patterns enable effective hardware caching. even without explicit shared memory management in some cases. However, for maximum portability across GPU architectures and to ensure deterministic performance, the shared memory implementation remains preferable.
Streaming Multiprocessor Utilization Our profiling reveals an important relationship between input configuration and SM utilization. With GSPN-2's 2D thread organization strategy, SM occu- pancy varies significantly based on workload characteristics. For large batch sizes and channel counts, SM occupancy approaches 100%, fully utilizing the GPU's 108 SMs on A A100. However, for small
8

---

(Image Size=12, k2items-33)
BS C (Image Size-256, kNitamaa 258)
10
10
Eachbrdd
(ms))
10
Bunnn Bamringg Imsi
100
Runtime
2.0
10
201
10a
85 x C
517
4096
3276
64
85 x
512
4096
32768
10P
32
BS
256
2048 16384
32
BS
x
255
2048
(Image Size=128, kNItems-122)
BS
(Image Size=64, kNitems=64)
Excooard hor
10P
2003
10
2015
Runtime (mss)
Runtrman (ms))
Runtime (ms)
1.0
20c
201
Runtime
209
30P
100
64
85
512
4066
32768
64
BS
512
x
4096
32768
512
4006
4096 32768 85
Figure 4: Runtime Performance Comparison of GSPN-1 and GSPN-2. We show forward and backward pass execution times (in milliseconds) across different channel counts. Results are presented for various configurations. GSPN-2 greatly improves the runtime of both forward and backward passes across different cases.
batch sizes and channel counts, occupancy can drop significantly (as low as 20-30%). This occurs because when processing independent chunks, each chunk requires one block, limiting parallelism for small input dimensions. This suggests potential areas for further optimization particularly for low batch size, low channel count scenarios where we could further decompose the problem to increase parallelism across SMs.
Table 2: Performance of models on ImageNet at the resolution of 224 Colors denote different backbone types: yellow for CNNs, orange for Transformers, and green for Raster scan (i.e., ID linear propagation) methods.
IN-IK
-K
Model
Rackbone
Pomn (M)
MACAAe (G) (96)
Model
Baokboue
Param (M)
MACAce Model (G) (%)
Backbonn
Paran (M)
MAC Apc (G) (1)
MambuOut:110 Tly
NeXT--
30
CN
27
29
4.5
4.5
82.7
CNFvrmer S36
ConvNeXT:S
MogaNet: 9
(1]
CN
CN
CN
40
50
9.9
7.6
8.7
83.1
$4.1
$4.3
ComrNeXT-B 2
CNFormer-M3
Mambaauu Base
41
CN
CN
CN
89
57
15.4 83.8
DeTT-S XII
TF
22
4.6
79.8
Intemnnagg-5
CN
SLak-B
95
T2T-ViT-H
TF
4.8
81,5
MambuOat- Small
CN
9.0
84.1
DelT
86
17.5 81.6
Swin-T
4.5
81.3
T2T 19
TF
39
8.5
81.9
T3:
64
13.8
H2.5
SwinV2-T
Focal-Small
TF
9.1
83.5
Swin B A3
TFF
88
15.4
83.5
CSWin-T
35
TF
23
4.3
82.7
SwinV2
33
TFF
88
15.1 84.6
NextVi 8 66
TF
83
83.2
CoArNes-0
X2
TF
25
4.2
81.6
Twins-B 4h
TFF
56
8.3
83.1
CSwin-B
78
Vim-S A5
RS
26
5.1
80.5
MaxViT-Small Small
18
TF
11.7
84.4
MVVIV2-B 51
52
10.2
VMamba-1T T
22
5.6
H2.2
[3]
TF
CoAINet-2
Mambe-2D-5
SwinV2-S M4
TFF
8.5
83.8
Vim B 25
98
17.5 81.9
LocalVMambb-1
5.7
CoANNet- 36
TF
42
8.4
$3.3
VMambo- 9 2X
RS
89
15.4 83.9
VKWKV-S 58
RS
24
4.6 80.1
MMambe-S
UniFormer 0 h9
TF
8.3
63.5
$3.9
VRWKV-B
Mamiuu 2D-B
RS
R5
94
92
18.2
83.0
82.0
ViL-S 33
RS
23
5.1
B1.5
LomalVMamba
RS
11.4
83.7
VIL-R
89
82.4
MambaVisioo-T
40
RS
32
4.4 X2.3
MambaVision S 40
RS
50
7.5 83.3
VisonVisionnBB
98
15.0
GSPN-T T
Line
30
5.3 83.0
GSPN-5
Line
83.8
GSPN-B
Line
89
15.9 84,3
GSPN-2-T (Ours)
1ine
24
4.2
X3.0
GSPN-2.8 (Ours)
Line
9.2
84.4
GSPN-2-R (Ours)
Line
89
14.2 84.6
5.2 Image Classification
In Table 2 we present a comparative analysis of ImageNet- IK classification performance across
three architectural paradigms: ConvNet- [29] 30 Transformer- [3] 36 [3] 46, 49). and aequential-base (RS scan) models (25 26
37 27
40
38 39 of varying sizes. For GSPN-2
models, the ImageNet experiments incorporate several key design choices: propagation weights w) are shared across channels in all GSPN modules, and a compressive proxy dimension Cproxy is set to
2. This reduction in channel dimensionality allows the saved parameters to be reallocated for deeper or wider network architectures. Additionally, we integrate the Local Perception Unit (LPU) [52) at the beginning of each block and FFN. The MESA [53] technique is also employed to mitigate overfitting, contributing further 0.2% accuracy improvement some variants.

---

2016
300
Inteerce time (PSrAMAA))))
4/96 4096
20168 X 2048
2048 X 21488
612
768
84
Figure 5: Qualitative text-to-image results generated from our GSPN-2 SDXL model. We enable generation to 16K resolution on single A100 GPU while reducing inference time by up to 93x on the SDXL model.
Our GSPN-2 models, benefiting from the joint algorithmic and system-level redesign detailed in Section 4 demonstrate notable advancements. The GSPN-2 series builds upon the strong foundation of GSPN-1. 1. introducing refinements that enhance both performance and efficiency. GSPN-2-T achieves a competitive 83.0% accuracy with significantly fewer parameters (24M vs. 30M for GSPN- T) and lower computational cost (4.2G MACs vs. 5.3G MACs for GSPN-T). It outperforms SSMs such Vim-S (80.5%). VMamba-T (82.2%), and notably surpasses LocalVMamba-T (82.7%) by 0.3% accuracy with fewer MACS (4.2G vs 5.7G), while remaining competitive with leading ConvNets and Transformers in its category. GSPN-2-S achieves an impressive 84.4% accuracy, marking significant +0.6% improvement over GSPN-S (83.8%) with only a marginal increase in MACs (9.2G vs 9.0G) while using the same number parameters (50M). This performance places GSPN-2-S ahead of strong competitors like MambaOut-Smin (84.1% 1%) and UniFormer-B (83,9%), showcasing its enhanced efficiency and effectiveness. At the base model scale, GSPN-2-B also achieves excellent 84.6% accuracy, improving upon GSPN-B (84.3%) by +0.3% while redueing MACs (14.2G
vs 15.9G) with the same 89M parameters.
5.3 Text-to-Image Generation
To evaluate the efficiency and performance of GSPN-2 in high-resolution generative tasks, we conduct experiments on text-to-image generation using the Stable Diffusion XL. (SDXL) framework, with results summarized in Figure
Building upon the GSPN-1 architecture, GSPN-2 incorporates several key enhancements detailed in Section 4 and Proxy Dimension Compression to 1/8 of the original channel dimension (Cproxy C/8). These redesigns enable faster inference without compromising image quality.
Compared to the baseline SDXI. model, GSPN-2 achieves a 32x speedup 4K image generation, showcasing exceptional efficiency. For ultra-high-resoluiion 16K images, GSPN-2 outperforms further, reducing inference time by 93 compared to GSPN-1's 84x improvement.
Limitations
GSPN-2's performance gains diminish when the product of batch size and channel count (BS x C) is small (Section B). and practical evaluation on long-context video datasets remains underexplored. The current implementation lacks CLS and register tokens commonly used in Vision Transformers, limiting direct applicability
drop-in attention replacement in models relying on summary tokens (Section[E). Our dense prediction evaluations primarily use 480-512 pixel images; nigher-resolution testing would better demonstrate scalability advantages. Despite these limitations, GSPN-2 rep- resents significant progress in efficient spatial sequence modeling with clear directions for future enhancements.
7 Conclusion
We introduce GSPN-2, which overcomes the performance bottlenecks of GSPN-1 through a unified CUDA kernel, channel-agnostic propagation, and low-dimensional proxy features, delivering up to 52x speedup and near-peak hardware utilization without sacrificing accuracy across classification
10

---

and generation tasks. This establishes GSPN-2 as practical and scalable solution for global spatial reasoning in high-resolution vision applications.
References
[1] Hongjun Wang, Wonmin Bycon, Jiarui Xu, Jinwei Gu, Ka Chun Cheung, Xiaolong Wang, Kai Han, Jan Kautz, and Sifci Liu. Parallel sequence modeling via generalized spatial propagation network. In CVPR, 2025.
[2] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolttion image synthesis with latent diffusion models. CVPR, pages 10684-10695, 2022.
[3] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[4] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. CVPR, pages 19975-11886, 2023.
[5] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang. Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV. Springer, 2024.
[6] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson. Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In CVPR, 2023.
[7] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, FlashAttention: Fast and memory- efficient exact attention with 10-awareness. Advances Neural Information Processing Systems (NeurlPS), 2022.
[8] Tri Dao, FlashAttention-22 2: Faster attention with better parallelism and work partitioning, In International Conference on Learning Representations (ICLR), 2024.
[9] Markus N Rabe and Charles Staats. Self-attention does not need o(n²) memory. arXiv preprint axXiv:212.006622 2021.
[10] Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. arXiv preprint arXiv:2103.02143, 2021.
[1I] Dongchen Han. Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In CVPR, pages 5961-5971, 2023. 2023.
[12] Albert and Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0073 2023.
[13] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In /CML, 2024.
[14] Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeop Ramani, and Dao. Flashattention-3: Fast and accurate attention with asynchrony and low- ow-pprccision. NeurlPS, 2024.
[15] A Vaswani. Attention is all you need. NeurIPS, 2017.
[16] S Hochreiter. Long short-term memory. Neural Computation MIT-Press, 1997.
[17] Junyoung Chung, Caglar Gulcehre, KyungHyyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NeurlPS, 2014.
[18] Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Multi-dimensional recurrent neural networks. In International conference on artificial neural nenworks, 2007.
[19] Wonmin Byeon, Thomas M Breuel, Federico Raue, and Mareus Liwicki. Scene labeling with Istm recurrent neural networks. CVPR, 2015.
[20] Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universität München, 1991.
[21] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. ICMI., 2013.
[22] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2021.
11

---

[23] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao. Stephen Baccus, and Christo- pher Ré. S4nd: Modeling images and videos as multidimenssonal signals with stale spaces. In NeurIPS, 2022.
[24] Ethan Baron, Itamar Zimerman, and Lior Wolf. 2-d ssm: A general spatial layer for visual transformers. arXiv preprint arXiv:2.366.06633 2023.
[25] Lianghui Zhu, Bencheng Liao, Qian Zhang. Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint urXiv:2401.09417.00 2024.
[26] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu. Lingxi Xie, Yaowei Wang. Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401. 2024.
[27] Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba-nd: Selective state space modeling for multi- dimensional data, arXiv preprint arXiv:2402.0588 240.00822, 2024.
[28] Sifci Liu, Shalini De Mello, Jinwei Gu, Guangyu Zhong, Ming-Hsuan Yang, and Jan Kautz, Learning affinity via spatial propagation networks. NewrIPS, 2017.
[29] Zhuang Liu, Hanzi Mao. Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022.
[30] Weihao Yu and Xinchao Wang. Mambaout: Do we really need mamba for vision? arXiv preprint arXiv:2405.07992 2024.
[31] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In JCML., 2021.
[32] Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi- Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenct. ICCV, 2021.
[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[34] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao. Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In CVPR, 2022.
[35] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.
[36] Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. NeurlPS, 2021.
[37] Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan, arXiv preprint arXiv:2403.092: 09338, 2024.
[38] Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang. Vision-rwkv: Efficient and scalable visual perception with rikv-like architectures. arXiv preprint arXiv: 2403.02308, 2024.
[39] Benedikt Alkin, Maximilian Beck, Korbinian Pöppel, Sepp Hochreiter, and Johannes Brandstetter. Vision- Istm: xlstm as generic vision backbone. arXiv preprint arXiv: 2406.04.20 04303, 2024.
[40] Ali Hatamizadeh and Jan Kautz. Mambavision: A hybrid mamba- mamba-transforma transformer vision backbone. arXiv
preprint arXiv: 2407.08083 2024.
[41] Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. IEEE TPAMI, 2024.
[42] Siyuan Li, Zedong Wang, Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng. and Stan Z. Li. Moganet: Multi-order gated aggregation network. In ICLR, 2024.
[43] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. arXiv preprint vrXiv:221..0577/ 2022.
[44] Jianwei Yang. Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. NeurIPS, 2022.
12

---

[45] Lei Zhu, Xinjiang Wang. Zhanghan Ke. Wayne Zhang, and Rynson Lau. Biformer: Vision transformer with bi-level routing atlention. CVPR, 2023.
[46] Jiashi Li, Xin Xia, Wei Li, Huixia Li. Xing Wang, Xuefeng Xiao, Rui Wang, Min Zheng, and Xin Pan. Next-vit: Next generation vision transformer for efficient deployment in realistic industrial scenarios. arXiv preprint arXiv:2207.05501, 2022.
[47] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. NewrIPS, 2021.
[48] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
[49] Kunchang Li, Yali Wang. Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatiotemporal representation learning. In ICLR, 2022.
[50] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Mocanu, and Zhangyang Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. In ICLR, 2023.
[51] Yanghao Li, Chao Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022.
[52] Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Ychui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolu- tional neural networks meet vision transformers. In CVPR, 2021.
[53] Jiawei Du. Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. In NeurlPS, 2022.
[54] Songhua Liu, Weihao Yu, Zhenxiong Tan, and Xinchao Wang. Linfusion: gpu, minute, 16k image. arXiv preprint arXiv:2409.0m 020977 2024.
[55] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12490- 12500, June 2024.
[56] Greg Heinrich, Mike Ranzinger, Hongxu, Yin, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, and Pavio Molchanov. Radiov2.5: Improved baselines for agglomerative vision foundation models, 2024.
[57] Maxime Oquah, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Srafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. Featured Certification.
[58] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, October 2023.
[59] Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilek, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head architecture for small language models. 2024.
[60] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, page 280-296, Berlin, Heidelberg, 2022. Springer-Verlag.
[61] Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin. Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia. Basil Mustafa, Olivier Hénaff. Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025.
[62] Mike Ranzinger, Jon Barker, Greg Heinrich, Pavlo Molchanov, Bryan Catanzaro, and Andrew Tao. Phi-s: Distribution balancing for label-free multi-teacher distillation, 2024.
[63] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song. James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations (ICLR), 2020.
13

---

[64] Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
[65] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-336 -38,, 2010.
[66]
MMSSemmentation Contributors. MMSegmentattin:: Openmmlab semantic segmentation toolbox and 2020.
benchmark.
[67] Mohamed EI Banani, Amit Raj. Kevis Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21795-218065 2024.
[68] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECC 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part 12, pages 746-760. Springer, 2012.
[69] Roozbeh Mottaghi. Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho. Seong- Whang Lee, Sanja Fidler. Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 891-898, 2014.
[70] Yuqi Yang, Peng- Tao Jiang, Qibin Hou, Hao Zhang, Jinwei Chen, and Bo Li. Multi-task dense prediction via mixture of low-rank experts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 27927-27931 27937, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: A dedicated Limitations section is provided in Section 6 before the Conclusion, discussing hardware dependencies, performance characteristics, evaluation scope, and architectural integration considerations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
There no theoretical claims made in the paper.
15

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides detailed description of the experimental setup, including hardware, datasets used, and optimization parameters and configurations in Section 2, 3, and 4. Besides, we will release the code upon publication.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution
specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., .. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
16

---

We utilize public datasets and include the detailed implementation in the Paper for reproducibility and will release the code upon publication.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
public/ guides/ Caaess/CooeSunlinn
wiinnoliccyyy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https)
//nips.ccppul puublice wellic//uu ies CCodeSt Submis ss i oonool icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4 and Appendix for the full details of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow existing works [14.27,26] to not involve statistical significance in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
17

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4 and Appendix for details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pww...e.ee.......o//o/aaa
Answer:
[Yes]
Justification:
We believe that ethics guidelines violated.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:No societal impact of the work discussed. Potentially, the work could have a positive impact in terms of reducing carbon emissions caused by large-scale models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
18

---

The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for the responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We are not releasing any new datasets or pre-trained models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All creators or original owners of assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, aoperswithoor curated licenses for some datasets. Their licensing guide help determine the license of dataset.
com/ dat aset s has
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No asset is submitted.
Guidelines:
The answer NA means that the paper does not release new assets.
19

---

Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [NA]
Justification: No L.M is used in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: /ttttss//neurips... htt.eerrrppscce/ooooeeeeeeeeee/e 2025/ LLM) for what should or should not described.
20

---

Appendix
This supplementary is organized as follows:
Part 1: Core Vision Tasks and Efficiency Analysis
Section
we compare GSPN-2 variants with CNNs, transformers, and SSMs on
ImageNet- K.
In Section
B
we evaluate runtime performance across varying batch sizes and channel
dimensions. Section B.I provides detailed step-by-step optimization analysis under
large-batch configuration.
In Section C
evaluate GSPN-2's text-to-image generation the COCO benchmark.
In Section D analyze the compressive proxy dimension strategy through low-rank approximation, with ablation studies on the accuracy-trroughput trade-off.
Part 2: Foundation and Vision-Language Models
In Section we introduce GSPN bbseed vision foundation models trained via agglomerative learning, replacing self-attention in ViT to enhance scalability for high-resolution images and long-context vision-language tasks.
In Section F demonstrate GSPN-2's integration into SigLIP2- Base encoder, achieving 180x module-level and 5x block-level speedups with minimal accuracy loss.
Figure S] provides comprehensive comparison of GSPN-2 (Tiny/Small/Base variants) with other leading architectures like CNNs, Transformers, and other SSMs on the ImageNet- KK benchmark. The comparison focuses on Top-1 1 accuracy, throughput (images/second), model parameters, all evaluated an image resolution of 224
Take the Tiny model as an example, from the figure, we observe the following:
CNNs: Models like ConvNeXt-T achieve 82. 1% accuracy with 29M parameters and 4.5G FLOPs, and throughput of 1189. Larger variants like ConvNeXt-B reach 83.8% accuracy but use 89M parameters and 15.4G FLOPs, with throughput dropping to 435.
Transformers: DeiT-S, comparable small model, has 22M parameters and 4.6G FLOPs. achieving 79.8% accuracy with a throughput of 1759. Larger Transformer models like Swin- B (88M params, 15.4G FLOPs) reach 83.5% accuracy with throughput of 458. NAT-B shows higher accuracy (84.3%) with 90M parameters and 13.7G FLOPs, but throughput is not reported.
Other SSMs: VMamba-T provides high throughput of 1686 with 30M parameters and 4.9G FLOPs, achieving 82.6% accuracy. Local Mammaa- T uses 26M parameters and 5.7G FLOPs for 82.7% accuracy, but its throughput is considerably lower 394.
GSPN-2-T (Ours): Our GSPN-2-T model stands out by achieving strong Top-1 accuracy of 83.0%. accomplishes this with remarkably efficient parameter count of only 24M and low GFLOPs of 3.6G. While its throughput of 1 1544 images/second is slightly lower than the fastest models like DeiT-S VMamba-T, it is highly competitive, especially considering its superior cecurac-to-parameet and DeiT-S, GSPN-2-T offers +3.2% higher accuracy with only 2M more parameters and IG fewer FLOPs. Compared to VMamba-T, GSPN-2-T is +0.4% more accurate, uses 6M fewer parameters, and requires 1.3G fewer FLOPs, while having comparable throughput.
-t-racyy-----LPss ratio. For instance, compared to
A Comprehensive GSPN-2 comparison on ImageNet-1K
This comparison highlights GSPN-2's excellent trade-off between accuracy, model size, and compu- tational efficiency. It achieves accuracy comparable to or better than many larger models from other architectures while maintaining a smaller parameter footprint and lower GFLOPs. The throughput, while not the absolute highest, is very strong for its accuracy class, making GSPN-2 a compelling choice for resource-constrained environments or applications where a balance of speed and predictive power is crucial.
21

---

Accuracy vs. Throuighutt vs. Model Size
20M
40M
BOM
VMMamba
ConVNeXt
Imz geggeeee Top-2 Accuracy (%)
Swin
Local
DeiT
ConvNeXL
Better
81.
BAND VIT
60
2000 1450
1000
9UU
400
Throughput
Better
Figure S1: Comparison of GSPN-2 vs. State-of-the-ard architectures ImageNet-1K 1K. We present comprehensive analysis of the trade-offs between accuracy, model size, and throughput for GSPN-2 compared to leading state-of-the- effectiveness, positioning GSPN-2 as ideal solution for resource-constrained environments and applications requiring both speed and predictive accuracy.
architectures. The results highlight GSPN-2's
Size-1624,
C Clmage Size-512, KHitems- 5121
Runtime (ms)
Runtirme Crest
102
Runtime 207
3
Runtime
20*
256
32
128
512
2048
32
236
2048
DS
Figure S2: Runtime Performance Comparison of GSPN-1 and GSPN-2. We show forward pass execution times (in milliseconds) across different batch size times channel counts. Results are presented for various configurations. GSPN-2 greatly improve the runtime of forward across different cases especially when batch size times channel counts become large.
B Detailed Analysis of Performance with Varying Batch and Channel Dimensions
Figure highlights that GSPN-2 achieves significant speedups. particularly when batch sizes or channel dimensions are large. This appendix provides a more detailed look at when the full GSPN- 2 optimizations (blue line in plots, including shared memory for hidden states) begin to offer substantial advantage over GSPN variant without explicit shared memory caching for hidden states. This analysis is crucial for tasks like visual encoder training or video processing, where the product of batch size and channel dimensions ( ('BS C") can vary widely and significantly impact performance.
we can clear trend: the point at which GSPN-2's full optimizations deliver more pronounced benefits depends on the 'BS C' product.
Observing Figure S2
Implications for Model Selection: This detailed observation underscores that the effectiveness of GSPN-2's most advanced optimizations, such as shared memory caching for hidden states, is magnified when the aggregate workload (represented by 'BS C') increases. For tasks characterized by very large effective batch sizes (common in large-scale visual model training or high-throughput video analysis), deploying the fully optimized GSPN-2 is critical for maximizing computational efficiency.
Conversely, for scenarios where the 'BS * C' product remains relatively small, the performance difference between GSPN-2 and GSPN-1 might be less pronounced. In such cases, the GSPN-1 configuration could offer a good trade-off. This suggests a potential adaptive strategy: one could dynamically select between GSPN-I-like configuration and the full GSPN-2 based on the input
22

---

Beseline (SSPN-1)
143.7
Unified Kernel
139.2-x
Coolesced Memory Access
4.1 mx 34.0
SRAM
4.5 -x 0.9
2D Thread Blocks
4.4 e 1.0
Compressive channets
4.0 6-X 1.1
Stream based Concurrency
3.9 <-x 1.02
X 36.6.
(Phal)
3.9
1.0
40
80
120
Forward Time (ms)
Figure S3: Step-by-step CUDA kernel optimization under large batch configuration. Each bar shows the cumulative reduction in forward time (ms) for high-throughput scenario (1024x102) image, batch size 256, I channel). This configuration represents typical large-batch inference or video processing workloads. The optimizations deliver a 36.8x speedup from GSPN-1 baseline (143.7 ms) to the final GSPN-2 implementation (3.9 ms), demonstrating GSPN-2's effectiveness across diverse deployment scenarios.
dimensions and batch size to achieve optimal performance across diverse computational scenarios. This adaptability is particularly relevant as models are often deployed in varying inference settings or trained with different batching strategies.
B.1 Optimization Analysis Under Large Batch Size Configuration
While Figure 3 demonstrates the optimization journey for moderate configuration (1024x1024 batch size 16, channels), here we examine a complementary scenario with significantly larger batch size but minimal channel dimension (0024x1024, batch size 256, 1 channel). This configuration is representative of high-throughput inference scenarios such batch video processing, multi-stream parallel generation, or large-scale model serving where many requests are processed simultaneously.
As shown in Figure S3 the optimization progression follows a similar pattern but with distinct characteristics:
GSPN-1 Baseline Performance. The baseline GSPN-1 implementation exhibits 143.7 ms execution time. Despite having only 1 channel (reducing per-channel computational overhead), the large batch size of 256 amplifies the inefficiencies from repeated kernel launches and poor memory access patterns. With 256 batches, the kernel launch overhead becomes even more pronounced, as each propagation step must coordinate across much larger working set.
Unified Kernel (1.03x speedup, 139.2 ms). Consolidating the multi-kernel launches into single kernel reduces execution time to 139.2 ms, yielding 1.03x speedup. While this improvement is more modest compared to the 1.2x gain in the main paper configuration, it still demonstrates consistent benefits. The relatively smaller gain here suggests that with only I channel, the per-channel kernel launch overhead is less severe, but the benefit of unified execution remains valuable.
Coalesced Memory Access (34.0x speedup, 4.1 ms). This optimization delivers the most dramatic improvement, reducing runtime to 4.1 ms-a 34.0x speedup over the previous step. The impact is even more pronounced than the 23.9x gain in the 8-channel configuration, highlighting that memory access patterns become increasingly critical with larger batch sizes. With batch size 256, ensuring coalesced memory access patterns essential to saturate the memory bandwidth efficiently, Uncoalesced accesses would be catastrophic at this scale, causing severe memory traffic congestion.
SRAM (0.9x speedup, 4.5 ms). Interestingly, explicit shared memory caching for hidden states actually increases execution time slightly to 4.5 ms, yielding 0.9x slowdown. This counter-intuitive result occurs because with only channel, the memory footprint of hidden states is minimal, and the L1 cache is already sufficient to capture reuse patterns efficiently. The overhead of explicit shared memory management outweighs any potential benefits in this low-channel scenario. This observation
23

---

validates our discussion in Section 5.] about L1 cache effectiveness and confirms that shared memory optimization most beneficial when channel counts are higher.
2D Thread Blocks (1.0x speedup, 4.4 ms). Restructuring to 2D thread blocks reduces runtime to 4.4 ms, achieving a marginal 1.0x speedup (essentially neutral performance). Unlike the 1.1x gain observed in the main 8-channel configuration, the 2D block restructuring provides minimal benefit here. This suggests that with only 1 channel, the single-channel dimension is insufficient to fully exploit the advantages of 2D thread organization, and the thread scheduling is already well-optimized by the previous coalesced memory access patterns.
Compressive Channels (1.1x speedup, 4.0 ms). Applying compressive proxy dimension reduction reduces runtime to 4.0 ms (effective final runtime 3.9 ms after fine-tuning), achieving a modest 1.1x speedup. While this configuration already uses only I channel, the proxy compression strategy still provides minor benefits through reduced memory footprint and improved cache utilization. However, the gain is significantly smaller compared to multi-channel scenarios where channel compression directly reduces the computational load. This highlights that the proxy dimension benefit is configuration-dependenn and most impactful in high-channel scenarios.
Overall Speedup and Implications. The cumulative speedup from GSPN-1 (143.7 ms) to GSPN-2 (3.9 ms) 36.8x, which is comparable to the 40.0x improvement shown in the main paper. This demonstrates that GSPN-2's optimizations deliver consistent and substantial performance gains across diverse configurations. However, the relative contribution of each optimization stage varies with workload characteristies:
Memory coalescing remains the dominant optimization regardless of configuration, con- sistently providing 24-34x improvements. The 34x gain in this large-batch, single-channel scenario exceeds the 23.9x gain in the 8-channel configuration, demonstrating its critical importance for high-throughput workloads.
Shared memory caching benefits are highly configuration-dependeen It shows significant gains with multiple channels but actually degrade performance (0.9x slowdown) in single-channel scenarios due to management overhead when Ll cache is already sufficient.
2D thread blocks provide minimal benefit (1.0x) in single- configurations, contrast- ing with the 1.1x gain in multi-channel scenarios. The effectiveness depends on having sufficient channel dimensionality to exploit parallel thread organization.
Compressive proxy dimension provides modest benefits (1.Ix) even in single-channel scenarios through improved memory footprint and cache utilization, though gains are most pronounced in high-channel configurations.
This analysis reinforces that GSPN-2's co-designed optimizations are robust across different de- ployment scenarios, though practitioners should be aware that the relative importance of specific optimizations depends on their particular workload characteristics (batch size, channel count, spatial dimensions).
C Text-to-image Generation
In this section, we evaluate GSPN-2's capabilities in text-to-image generation, a task demanding strong understanding of both textual prompts and the generation of coherent, high-resolution visual outputs. We compare GSPN-2 with several relevant baselines and its predecessor, GSPN-1, on the COCO benchmark, with all models generating images at 1024 1024 resolution. The results are presented in Table [ST] The baseline model for this comparison is Stable Diffusion v1.5 (SD-v1.5) 2 We also include recent sequence modeling approaches such as Mamba [12], Mamba2 [33, and Linfusion [54] For these models, text embeddings are treated as part of the visual token sequence during propagation.
As shown in Table ST and Figure our GSPN-2 model achieves an FID of 33.21 and a CLIP-T score of 0.286. While GSPN-1 currently shows a slight edge in these specific metrics, GSPN-2's performance competitive and close to the SD-v1.5 bascline (FID 32.71, CLIP-T 0.290) with faster inference.
key characteristic of the GSPN architecture (both GSPN-1 and GSPN-2) is its inherent adaptability to arbitrary image resolutions without requiring extra normalization layers strategies for unseen
24

---

Image Resolution
0309
Table S1: Cross-resolution generation on the COCO benchmark under 1024 x 1024 resolu- tion. Lower FID (L) and higher CLIP-T (1) stand for better image quality and text-image alignment.
1004
0303
0.00
00213
Model
FID(L)
CLIP-T(T)
0000
SD-v1.5 (baseline)
32.71
0.290
Mamba [12] (w/ norm)
50.30
0.263
GSPM4
50-1.
Mamba2 13 (w/ norm)
37.02
0.273
CSPN-2
Linfusion 54
(w/ norm)
36.33
0.285
GSPN-1
30.86
0.307
Figure S4: Comparison of GSPN-2 GSPN-1 and baselines, GSPN-2 achieves good tradeoff between FID, CLIP-T scores and inference time.
GSPN-2 (Ours)
33.21
0.286
resolutions, common necessity for some other methods like Mamba and Linfusion when faced with resolutions not encountered during training. The Stability-Contee Context property ensures stable and effective long-range propagation, allowing GSPN-2 to efficiently capture broad spatial dependencies.
GSPN-2, while leveraging the core principles of GSPN-1, incorporates system-level co-designs and algorithmic refinements aimed at enhancing efficiency and scalability The results with Figure in the main paper indicate that GSPN-2 maintains strong generative capabilities, comparable to established baselines, while benefiting from these architectural improvements for efficient text-to- image generation.
D Compressive Proxy Dimension as Low-Rank Approximation
The compressive proxy dimension (Cprocy) strategy addresses GPU concurrency saturation by project- ing inputs X E R- XC to compressed space X proxy N XW where Cproxy &< C, applying GSPN propagation in this reduced space, then projecting back to dimensions. This is analogous to low-rank matrix factorization, reducing CUDA workload from kehunk x N x C' slices to Ketunk N Cprexy, preventing GPU saturation while maintaining representational capacity. Table S2 presents an ablation on Gproxy for GSPN-2-Tiny on ImageNet-IK, analyzing the accuracy-throughpul trade-off.
Table S2: Ablation on proxy dimension Cproxy. GSPN-2-Tiny on ImageNet-1K with varying compression ratios.
Cproxy
Accuracy (%)
Throughput (img/s)
83.0
1544
4
83.0
1492
8
83.0
1387
16
82.9
1293
32
82.8
1 1106
Table S2 shows minimal accuracy degradation (0.2% for 16x compression from Cproxy 32 to
proxy 2) while achieving 1.4x throughput improvement. The aggressive 48:1 compression at
proxy 2 demonstrates that GSPN propagation operates effectively in low-dimensional spaces, as spatial dependencies dominate channel-wise dependencies.
E
GSPN-Based Agglomerative Models
E.1 Agglomerative Models: Architectures
Agglomerative models have recently proven to be a powerful class of Visual Foundation Mod- els (VFMs). Such models are trained with a multi-teacher distillation framework, such as AM- RADIO [55] and RADIOv2.5 [56]. The goal is to distill complementary knowledge from different VFM teachers such as CLIP SigLIP 4. DINOv2 57 or SAM 58 into a single Vision Trans- former (ViT) student. During training the student trains a lightweight multi-layer perceptron (MLP) head for each teacher, which maps its own feature distribution to each teacher distribution. At
25

---

Table S3: Visual Foundation Model accuracy: Hybrid vs. GSPN-only vs. Self-Attention Only Hybrid variants perform on par with self-attention despite having only 1/6 the number of self-attention blocks. GSPN-only variants are surprisingly competitive as well. From Hybrid GSPN-1 to Hybrid GSPN-2 most metrics are nearly identical, if not slightly better in the last-block-only loss scenario. Best results are in bold, second best underlined.
Stages
Model Amention
Sal-All
CSPN
ADE20K
VOC2312
ProbaDD
PuobeD
NYUD22
NYUD12
NYUD-2 Normlir
PASCAL
PASCAL
Blocks
Blocks
mloU
mloU's
Depth
Normals
mloU
RMSE
miol'
Normlir
Last-Only
WILI Ilybrdd OSPN-1
20
50.52
$4.85
90.53
59.81
56.19
0.504
81.88
Laot-Only
VIELI Hylvia CISPN-2
4
30
50.21
$4.97
90.69
59.76
56.78
0.499
19.02
81.82
16.48
Last-Only
VIT-L GSPN-1
24
48.75
$3.61
79.04
57.79
53:47
0,508
19.61
79.92
16.12
Iast-Duly
VITL SelEAnta
24
51,09
$5.25
90.66
59,93
0.478
K2.57
16:66
4-Stuge
VILLI Hybeid GISPN11
20
49.42
54.13
79.51
59.16
55.53
0.517
18.78
90.57
4-Stgg
VIT-L Hylvid CISPN-2
4
20
49146
$4.41
70,88
58.84 84
55.57
0,522
18.39
B1.09
15.34
SSuge
LI GSPN-1
24
47.92
$2.95
75.20
57.51
$1.96
0.531
19.58
79.63
15.30
4-Stge
SclE-Ai
$4.26
90.70
59.55
59.14
0.486
82.38
inference time these teacher heads can be used to allow the student to be a drop-in replacement for each teacher architecture, allowing the student-teacher combination to perform the same tasks the original teachers can do, such zero-shot classification like CLIP, or promptable segmentation like SAM. These teacher heads can also be discarded, and the student may be used as a general purpose vision backhone for many different downstream tasks, similar to other VFMs. This is our main focus area for training and evaluating GSPN.
Much of the success relating to ViTs foundation models can be attributed to the self-attention mechanism. While many of the linear attention mechanisms previously discussed have been shown to be effective during image classification pre-training and supervised fine-tuning, efficient attention is relatively unexplored in the context of Visual Foundation Models. Thus ask the question, to what extent can GSPN be leveraged within VFMs, if at all? If it is possible to use GSPN, this opens the door for many high-resolution, long-context vision applications, including images, videos and multi-modal domains.
To efficiently train our VFM student we start with pre-trained C-RADIO-v3-L teacher[56). We copy all of the pre-trained weights into an identical ViT-L student architecture. and then proceed to modify it in three ways. First, we create a baseline model by randomly initializing the attention weights, which includes the QKV input projections and the output projection. Second, we create pure-GSPM variant by replacing the entire self-attention block with the GSPN module. This includes the QKV input projections and the output projection as in self-attention, and we randomly initialize these as in the baseline. Overall the goal is to see how well GSPN can be used as substitute for self-attention, with everything else equal.
Our third variant is hybrid self-attention and GSPN variant. Inspired by hybrid architectures such as Hymba [59], we use 20 GSPN blocks, and 4 self-attention blocks. The self-attention blocks are evenly spaced, used at layers 6, 12, 18 and 24 in 24-layer ViT-L architecture. This pattern of using the "last block" in "four stage" logical grouping of ViT blocks is also used by ViT-Det [60] and SAM (58). The goal here is to get the vast majority of the long- long-context scalability from many GSPN layers, while still allowing for some amount of self-attention for downstream tasks that may rely on it more than others.
E.2 Agglomerative Models: Training
Traditionally agglomerative models are trained using multiple teachers, often over multiple training stages as in AM-RADIO and RADIO-v2.5. Instead, we use a single C-RADIO-v34 teacher, which was previously trained on OpenAl CLIP 3. SigLIP-2 [6]]. DINOv2 (57) and SAM (58) for IM iterations. By copying over the C-RADIO-v3- weights as described above we can train with single RADIO teacher, for only 100K iterations, for the model to start to converge.
During traditional RADIO training the student learns to mimic both the dense features of each teacher, and the summary CLS token of any combination of teachers. In the case of C- C-RADIO-v3-L there are 3 summary tokens, where the CLS token for SAM is not used due to poor semantics. For our ViT-L baseline student we learn 3 summary tokens as well, corresponding to each of the summary tokens in our teacher. However for the GPSN and Hybrid variants we skip the summary tokens and solely focus on the dense features. This because the GSPN module does not currently use CLS tokens or register tokens. We leave integrating CLS tokens and/or register tokens for future work.
The next key decision pertains to the loss formulation. RADIO uses two loss terms. First a feature distillation loss is computed using the mean-squared error (MSE) between the dense features of
26

---

each teacher and the student, taken at the output of the suucher-specifie MLP head. The values are nre-normalized following the Phi-Standardizzttion [62] technique. The second term is the cosine similarity between the teacher and student CLS tokens. For our GSPN-only and Hybrid variants we ignore this similarity loss and only use the feature distillation loss.
To extend this further, we explore two variants of the loss function. The first variant computes the loss after the last transformer block only, as is done in RADIO. The second variant computes the loss after blocks 6, 12, 18, and 24, a so-called 4-stage loss function. For the hybrid variants this is computed at the output of each of the four self- eelf-attention layers. We hypothesized that this extra loss might help the student learn more effectively.
We train for 100K iterations using the LAMB optimizer (63) base learning rate of le decay of 0.01, and a cosine learning rate decay. We use 32 nodes with 8x A A100-80GB GPUs each. The nodes are logically partitioned such that 16 nodes process images of size 432 x 432 with a total batch size of 1024, and 16 nodes process images of size 1024 x 1024 with total batch size of 256. While the loss has mostly converged at 100K iterations, there is ample opportunity to train up to 300k iterations and beyond.
weight
E.3
A gglomeratiie Models: Evaluation
As shown in Table S3 we evaluate on multiple dense prediction tasks following RADIO-v2.5, including:
Semantic segmentation via linear probe evaluation on ADE20K 644 and Pascal VOC 665
We train a linear layer with frozen ViT features using MMSegmentation [66].
3D understanding via depth estimation and surface normals estimation. We train a DPT head with frozen VIT features using Probe3D 67].
Segmentation and depth estimation on NYUDv2 688 and Pascal Context 69
We train a
2-layer convolution head with frozen ViT features using MLoRE (70].
In all scenarios images are re-sized with a short edge of 480 or 512. While this approximately matches the native image resolution of the datasets, this is a relatively small image size for GSPN. We leave exploring more high-resolution images for dense prediction tasks future work. In such a high-resolution setting we believe GSPN and other linear attention mechanisms may have an advantage due to their long context without running out of memory or needing the image to be resized smaller for computational reasons.
Overall the performance of the Hybrid and GSPN models surprisingly close that of the pure self-attention variant. In general the last-block-only loss formulation leads to better accuracy than the 4-stage formulation. This is may be due to the first "stage", or 6 blocks, of the 24-layer ViT not needing or wanting to be perfectly matched between the self-attention teacher blocks and GSPN student blocks. Conceptually this makes sense if we think of self-attention and GSPN having similar goals, namely to aggregate spatial information, but with perhaps different strategies. GSPN has to propagate information across rows and columns while remembering relevant information, where self- attention can simply look at everything at once, compute similarity scores, and aggregate accordingly. In that sense the only important aspect is that the output at the final block be a set of general purpose vision features suitable for any down-stream task. Forcing the network to minimize too many loss terms along the way appears to actually reduce the quality of the final representation.
Another important aspect is that moving from the Hybrid GSPN-1 model to the Hybrid GSPN-2 model we do not lose a significant amount of accuracy. In fact, with the last-block-only loss setting, Hybrid GSPN-2 actually outperforms Hybrid GSPN-1 across most tasks, despite using a shared set of weights instead of different set of weights per channel. This shows that the performance improvements from GSPN-2 may be essentially "free" " in such a hybrid design.
One notable accuracy gap between self-attention and the GSPN variants is on the NYUDv2 mIoU segmentation metric. We believe this is due to the training duration, where only train for 100K iterations versus 300K iterations or more as is done in the various training stages of AM-RADIO and RADIO-v2.5. Tweaking the loss function may also have a profound effect, as we' 've seen when comparing the last-block-only to the 4-stage loss setting.
Overall, feel there is great potential for GSPN-2 moving forward, especially with high-resolution images, videos and long-context tasks. As shown here, there are many interesting design decisions to be made to optimally integrate GSPN into different ViT architectures. Exploring this design space in conjunction with Visual Foundation Models is exciting opportunity for future work.
27

---

GSPN-2 Integration into Vision-Language Models
To further evaluate the efficiency of proposed GSPN-2, integrated it into vision encoder backbones such as SigL.IP--Base. For rapid prototyping and analysis, we trained the model on IM subsample of the DataComp- IB dataset with batch size 8096 and 12k iterations and evaluated performance on ImageNet Zero-shot.
The results in Table S4 demonstrate that GSPN-2 provides substantial improvements in inference speed while maintaining competitive accuracy. At the module level, GSPN-2 achieves 180x faster inference compared to flash attention. Even when considering the block-level speed, which includes overheads such MLP computations, method remains 5 x faster. Zero-shot accuracy remains comparable, indicating that the efficiency gains do not come at significant performance cost.
Table $4: Integration of GSPN-2 into SigLIP2-Base shows dramatic inference speedups while retaining comparable zero-shot performance. Note that we only train the model to about 1 1/6 of the full procedure due to the prolonged training process.
Setting
SigLIP (Flash Attention)
SigL.P-GSPN (Ours)
Batch Size
32
32
Resolution
1024
1024
Module-Ievel Speed (ms)
31.36
0.174 (180x faster)
Block-level Speed (ms)
63.08
12.59 (5x faster)
Zero-shot Accuracy
53.4
52.3
28


Reviews:
[{"review_id": "GpIcEXSAKP", "reviewer": "Reviewer_HqDp", "summary": "To address the problem of Transformer cost for high-resolution images, etc., GSPN, a type of state-space model, was proposed. However, the implementation of this method was not fully optimized and did not achieve sufficient speed due to frequent access to the HBM and insufficient kernel fusion. Therefore, in this paper, the author proposes GSPN-2, which addresses these issues and improves the implementation to be CUDA/GPU-friendly, achieving a significant speedup.", "questions": "1. I believe that not only the technical aspects but also the academic contributions should be clarified. Please clarify if there is any evidence that the proposed method is generally useful beyond the GSPN framework?\n\n2. The proposed method is mainly limited to comparisons with the GSPN and there is no comparison with other current SOTA methods, especially those that optimize inference time and model size. Please clarify the advantages of the proposed method when compared to lightweight models.\n\n3. Supplemental Figure 3 shows some relationship between speed and accuracy, but the evaluation in the text is somewhat limited. Since the main objective is to improve speed, shouldn't the trade-off analysis with accuracy be presented in more detail in the main experiment? Has such an analysis been performed, including comparisons with lighter models?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. This paper is well organized and clearly written.\n\n2. The significant speedups achieved by the ingenious implementation of GSPN are clearly demonstrated.\n\n3. The technical contributions of this paper are clearly and comprehensively described.\n\nWeakness\n\n1. The main contribution of this study is that the CUDA-based implementation of GSPN is made more GPU-friendly, resulting in a speedup of up to 84x over the previous implementation. Although the proposed implementation and its effectiveness are described in the text, the fact that the implementation code is not publicly available at this time, making reproducibility and verification of technical details difficult, is a concern in the evaluation of the paper.\n\n2. This research appears to focus on implementation optimization, with limited intrinsic extensions to the algorithm itself (e.g., channel-shared weights). Therefore, the value of this paper seems to strongly depend on the effectiveness of the GSPN method; although Figure 1 compares it with a simple SSM model, it is unclear what significance the proposed method can have outside the context of GSPN, especially in comparison with other SOTA methods and in terms other than speed. In particular, it is unclear how the proposed method compares to other SOTA methods and what its advantages are in terms other than speed.\n\n3. Qualitative and quantitative evaluations using image classification and text-to-image generation tasks have been conducted, but considering that the main objective of the proposed method is to “increase inference speed,” the “accuracy vs. speed trade-off” as shown in Supplemental Figure 3 analysis should be more clearly addressed in the main experiment. In particular, the relative significance of the speed improvement should be evaluated through a comparison between the proposed method and existing lightweight models.", "was_revised": true, "final_justification": "From a technical perspective, this work achieves results that are highly commendable, particularly in terms of computational speedup. Furthermore, the proposed method has potential applicability beyond improving a specific model like GSPN, extending to a broader class of models such as variants of linear attention, which is an important strength. However, within the scope of my understanding, the paper is presented primarily in the context of GSPN, and its generality is not made sufficiently clear. Therefore, my final decision is a weak-leaning accept.", "rebuttal": "We sincerely thank the reviewer for the encouraging feedback and thoughtful comments. We truly appreciate your recognition of the contributions and are glad that the strengths of our work came through. Below, we address your questions and suggestions in more detail.\n\n**Code Availability.** Yes, we will release it upon acceptance.\nIn the meantime, we ensure reproducibility by (1) providing extensive implementation details of our CUDA kernel design (Section 3.1–3.4). (2) Detailing block/grid configuration, memory layout, shared memory strategy, and concurrency management (Figure 2, Figure 3, and Table 1). (3) Describing all architectural modifications, including the proxy compression dimension and channel-shared weights (Section 3.2–3.3). We believe these elements provide a clear path to reproducing the kernel and model behaviors even before code release.\n\n\n\n**Significance outside GSPN.** First, improving GSPN is non-trivial. GSPN [1], because it is already a top-performing attention architecture that breaks the $O(N^2)$ softmax wall with $O(\\sqrt{N})$ propagation, yet its open-source code runs at < 8 % GPU utilisation. Our work directly addresses this bottleneck, making GSPN fast enough to be a practical alternative to softmax attention. Similar to recent studies on Mamba, we believe improving GSPN’s efficiency is both necessary and impactful. \nMoreover, **the compressed-channel architecture** can reduce the channel-linear cost inside any 2D recurrence. Because the design only assumes a line-scan recurrence and a shared (scalar) affinity map, it can be plugged into other propagation-based models—including SSM, Mamba/MambaVision, 2D RNNs, and LSTMs—to cut memory-bandwidth pressure without changing their update equations. Likewise, the single-kernel fusion pattern —or its Triton/TileLang analogue—can accelerate those models on all CUDA-class GPUs. In short, our contributions generalise: the compressed-channel idea is an algorithmic building block for any linear propagation method, and the fused-kernel blueprint shows how to reach hardware limits, making the work broadly relevant rather than GSPN-specific.\n\n**Algorithmic Contribution.** Our paper does not merely optimize implementation. We identify that GSPN’s main efficiency bottleneck is the large channel dimension (often thousands) typically used in vision models. To fundamentally address this, we propose a novel channel sharing and compression approach (Sec. 3.3) that significantly reduces memory and computational load—this is a novel micro design, not merely an engineering optimization. Additionally, we integrate this compressed-channel structure directly into a new, highly efficient CUDA kernel. Thus, our method tightly couples algorithmic and system-level improvements to unlock GSPN’s full practical potential.\n\n**Comparison with the SOTAs other than Speed.** Our experiments already show that GSPN-2 matches or outperforms the strongest non-GSPN backbones in accuracy and parameter count, not just latency. On ImageNet-1K, GSPN-2-T delivers 83.0 % top-1 with 4.2G MACs, beating LocalVMamba-T (82.7 %, 5.7G MACs) and other light ViTs at the same scale. For larger models, GSPN-2-B reaches 84.7 % with fewer MACs than ViT-B and VMamba-B. Beyond classification, we directly replace Flash-Attention with our layer in SDXL and obtain matching text-to-image quality (same FID and CLIP-score) while preserving the 92 × speed-up reported in Fig. 5. On dense prediction, our Appx E shows GSPN-2 backbones achieve 47.9 mIoU on ADE20K and 82.5 mIoU on VOC2012, on par with or better than transformer baselines yet at much lower inference cost. These results demonstrate that the proposed channel-compressed design is competitive with state-of-the-art models in accuracy, compute, and memory, so its value is not confined to speeding up GSPN—it offers a strong alternative backbone in its own right.\n\n**Accuracy vs. Speed Trade-off.** Thank you for this suggestion. We agree that clearly showing the trade-off between accuracy and speed is important. Besides Fig. 3, we also show a more comprehensive analysis of accuracy, model size, and throughput in Fig. 1 (Appx B). To improve clarity, we will move key results from Appendices B and D into the main paper. We note that in Fig. 1, the trade-off isn’t obvious, as most of our models simultaneously achieve top results in both accuracy and throughput.\n\n**Light-weighted Models.** While Fig. 1 (Appx B) focuses on isolating our core design improvements over various baselines, we have also compared GSPN-2 against several SOTA lightweight backbones in both accuracy and speed in the table below:\n\n| Model                      | Params (M) | Throughput (img/s) | Top-1 Acc (%) |\n|---------------------------|------------|---------------------|----------------|\n| MobileViT-S (ICLR’22)     | 5.6        | ~1030               | 78.4           |\n| MobileFormer-294M (CVPR’22)| 11.4       | ~1200               | 77.9           |\n| ConvNeXt-XT (CVPR’22)     | 7.4        | ~1100               | 77.5           |\n| VAN-B0 (CVMJ’23)          | 4.1        | ~1250               | 75.4           |\n| ParC-Net-S (ECCV’22)      | 5.0        | ~1000               | 78.6           |\n| **GSPN-2-T (Ours)**       | 24.6       | **1544**            | **83.0**       |\n\nDespite having a similar compute budget, GSPN-2-T outperforms all listed lightweight models by a wide margin in accuracy—by +4.4% over MobileViT-S and +5.1% over ConvNeXt-XT—while maintaining a comparable number of MACs. Furthermore, GSPN-2-T achieves much higher throughput than transformer and CNN-based models of similar scale, or even lighter scale. This demonstrates that GSPN-2 offers an excellent trade-off between model latency and performance."}, {"review_id": "QzrtPEqHiN", "reviewer": "Reviewer_XSoo", "summary": "This paper introduces GSPN-2, a joint algorithm-system redesign of the Generalized Spatial Propagation Network (GSPN) aimed at significantly improving its efficiency for high-resolution image and long-video applications. The original GSPN-1 implementation suffered from inefficiencies due to repeated GPU kernel launches, excessive global memory transfers, and redundant per-channel computations. GSPN-2 tackles these bottlenecks by consolidating all propagation steps into a single 2D CUDA kernel, introducing channel-shared propagation weights, and optimizing memory access patterns (shared memory caching, coalesced access). Experimental results demonstrate GSPN-2's impressive speedups while matching or exceeding transformer-level accuracy on image classification and improving semantic consistency in text-to-image synthesis.", "questions": "1. Generalization to Other Architectures. GSPN-2 is a redesign of GSPN. How easily can the core optimization principles (unified kernel, channel-shared weights, memory optimizations) be applied to other efficient attention mechanisms or state-space models beyond GSPN?\n2. Impact on Downstream Tasks. How would GSPN-2 perform on other dense prediction tasks like object detection and segmentation, which often rely heavily on fine-grained spatial context?", "limitations": "Lack of Statistical Significance Reporting. The paper does not report error bars or statistical significance for experimental results. This makes it harder to definitively ascertain the robustness of the reported performance gains and accuracy improvements, particularly when differences between models are small.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Joint Algorithm-System Redesign. The strength lies in the holistic approach, combining algorithmic modifications (channel-shared weights, compressive proxy dimension) with deep kernel-level optimizations (unified kernel, 2D thread blocks, shared memory, coalesced access, CUDA streams). This integrated design is key to the substantial performance gains.\n2. Impressive Speedups. The demonstrated speedups are significant, making GSPN-2 a highly practical solution for high-resolution vision tasks.\n3. Detailed Profiling and Ablation. Figure 3, showing the step-by-step optimization impact, is excellent. It clearly quantifies the contribution of each optimization, providing strong evidence for the design choices. Table 1 on memory throughput further reinforces the efficiency claims.\n4. Comprehensive Evaluation. The evaluation covers both efficiency (profiling across varying resolutions, batch sizes, channels) and task performance (image classification, text-to-image generation), providing a well-rounded assessment.\n\nWeaknesses\n1. Limited Theoretical Justification for Channel-Agnostic Propagation. While the channel-agnostic propagation is empirically shown to be effective, a deeper theoretical justification or analysis of its impact on feature representation compared to channel-specific weights could strengthen this aspect.\n2. SM Utilization for Small Workloads: The paper notes that for small batch sizes and channel counts, SM occupancy can drop significantly (as low as 20-30%). While acknowledged as a future optimization area, this is a current limitation for scenarios with low parallelism.\n3. The text size in Figure 4 could be increased. Currently, it's a little bit difficult to read.", "was_revised": true, "final_justification": "My major concerns have been addressed by the authors, so I keep my positive score unchanged.", "rebuttal": "We appreciate the reviewer’s positive evaluation and insightful observations. Your detailed technical feedback is highly valuable, and we have carefully addressed each point below to further clarify and strengthen our contributions.\n\n**Justification for Channel-Agnostic Propagation.** As we have briefly stated at line-193, Softmax attention already uses one scalar affinity per token pair, shared across all channels; the extra $C$ per-channel weight matrices in the original GSPN likely duplicate that information. By replacing them with a single shared $W$ plus per-channel gains $\\Lambda$, we (i) align GSPN’s affinity operator with standard attention, (ii) drop the hidden channel-dependent cost, and (iii) keep channel diversity through $\\Lambda$. Formally, Eq. 5 shows the full $N\\times N$ spatial kernel is preserved; sharing $W$ just factorises the original block matrix into $W\\otimes I(C)$, a rank-1 compression that removes redundant parameters while retaining dense global interactions. This is the same low-rank idea behind attention and grouped convolutions, where weight sharing regularises the model and cuts over-fitting risk. We will add this analysis to the revised paper.\n\n**SM Utilization for Small Workloads.** This trade-off is a known limitation of large-kernel fused CUDA designs under light parallel workloads. We are actively exploring fine-grained tiling, micro-batching, and block fusion across directions as potential directions to address this in future work. Despite this limitation, GSPN-2 still outperforms GSPN-1 and state-space alternatives in both accuracy and runtime, even under small-scale settings, as shown in Fig. 4 and Tab. 1.\n\n**Font size.** We will increase the font size and improve the figure in the revised paper.\n\n**Generalization to Other Architectures.**  We view GSPN-2 as a blueprint for applying GPU-aware design to 2D structured attention mechanisms, and anticipate future extensions to SSM, 2D RNN, LSTM, linear attention, RWKV variants, etc. The core optimization strategies in GSPN-2 are highly generalizable to any operation with 2D directional recurrence (e.g., causal convolutions, 2D scan layers / structured RNNs). For channel-sharing and proxy compression, these are modular strategies that can be transplanted to transformer blocks, gated MLPs, and SSMs like Mamba for reducing memory and increasing kernel fusion efficiency. Memory optimization tactics (e.g., shared memory reuse, coalesced access) can also benefit any scan-like operation where partial results are reused over spatial dimensions.\n\n**Impact on Downstream Tasks (see Appx E).** We agree that evaluating downstream dense prediction tasks is important. In Appx E, we already extend GSPN-2 to evaluate on segmentation, depth estimation, and 3D probing benchmarks such as ADE20K, VOC2012, Probe3D, NYUDv2, and PASCAL, as shown in Tab.2. These experiments confirm that GSPN-2 maintains strong dense prediction performance—comparable or superior to transformer-based backbones—while significantly reducing attention latency, especially for high-resolution inputs (e.g., **1024x1024**). We will move these segmentation results into the main paper to clearly highlight GSPN-2’s advantages on dense vision tasks."}, {"review_id": "acjJEzMzEh", "reviewer": "Reviewer_zPFN", "summary": "This paper presents GSPN-2, a significant algorithm–system co-design that improves the efficiency of the Generalized Spatial Propagation Network (GSPN) for vision transformers. The authors identify key system-level inefficiencies in the original GSPN implementation, including excessive GPU kernel launches, redundant memory access, and per-channel computation overhead. To address these, they propose a streamlined 2D kernel design with warp-channel pinning and shared memory staging, alongside a model-side enhancement that replaces per-channel propagation weights with a channel-shared variant. Experimental results show that GSPN-2 achieves comparable accuracy to transformer-based models while substantially reducing computational cost. Overall, the work effectively pushes the efficiency frontier for spatial context modeling in vision tasks and offers a compelling solution for high-resolution and long-sequence scenarios.", "questions": "1. Can you show me how the channel-shared weight mitigates the concurrency bottleneck? \n2. Is the compressive proxy dimension essentially a low-rank approximation? What is the cost of using this projection method due to the potential error?\n3. Can you show me the comparison with other SOTA algorithm-system codesign work?", "limitations": "This work mostly focuses on the system implementation optimization. The question is can you extend the optimization (fused kernel, shard weight, streaming concurrency) to other works not using your GSPN algorithm?", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. Authors provide many profilings and comparisons to demonstrate the bottlenecks and improvement, making the conclusion compulsory.\n2. The analysis of the hardware performance, such as the L1 cache missing rate, bandwidth throughput, provides more evidence of the effectiveness of the optimization.\n3. The given evaluation results shows significant improvements.\n\nWeakness:\n1. The writing is not very easy to follow. And some illustrations are hard to understand, especially why it is in this chapter, how this solution can solve the bottlenecks.\n2. The evolution is not comprehensive and most of the results focus on the improvement over GSPN-1. As the author mentioned many other system optimizations, such as Flush Attention, there is no directly comparison with it. If it is integrated in the baseline, you'd better mention it.\n3. This work basically an system implementation of a published paper, while the algorithm aspect is kind of weak and unclear, even this is a so-called algorithm-system co-design work.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. We respect your concerns and have taken them seriously. In the following, we provide clarifications and additional context to better address the issues raised and to more clearly communicate the value of our work.\n\n**Coverage of Flash Attention Baselines.** Flash Attention is already present in every ViT-based baseline we report: all orange blocks in Table 1 and Fig. 1 (Appx B), the SDXL row in Fig. 5 (main paper), and the ViT-L RADIO model in Table 2 (Appx E) are Flash Attention implementations. In the strongest head-to-head test—SDXL vs. GSPN-2 SDXL—we replace each soft-max/Flash layer with a single GSPN layer and fine-tune by distillation: image quality is maintained while inference latency drops by $\\approx 92\\times$. These results show that our gains are measured against the latest optimized attention baselines. \n\n| Resolution | Full Attention | FlashAttention | GSPN-2 (Ours) |\n|------------|----------------|----------------|---------------|\n| 256²       | 58.377         | 4.491          | 0.165       |\n| 512²       | 928.610        | 64.757         | 0.444        |\n| 1k²        | –              | 1034.967       | 1.096        |\n| 2k²        | –              | –              | 3.714         |\n| 4k²        | –              | –              |    13.533     |\n| 8k²        | –              | –              |    51.198     |\n| 16k²       | –              | –              |     196.412    |\n\n\n**Comparison with other SOTA algorithm-system codesign work.** In addition, we have compared with other algorithm–system co-designs such as VMamba and MambaVision (Fig 1 in Appx B), to highlight differences in computation patterns and practical GPU efficiency. Additionally, Appx B provides a detailed trade-off analysis among accuracy, model size, and throughput for the entire system. We will further clarify them in the revised paper. \n\n\n\n**Limited Algorithmic Contribution, a System Implementation.** We address the question from the following aspect:\n\n- **The importance of CUDA optimization.** CUDA optimizations have been widely explored for softmax attention (e.g., FlashAttention, TensorCore-based solutions). However, efficient CUDA support for recurrent or linear-propagation methods—despite their promising scalability for unlimited-length contexts (e.g., SSM, Mamba, Linear Attention, and GSPN)—is severely lacking. Our work explicitly targets this important gap.\n\n- **Just a System Implementation?** No, in contrast, we found that CUDA-level optimizations alone were insufficient. Even aggressive kernel tuning left us with a runtime that still grew linearly with channel and batch dimensions—a fundamental algorithmic bottleneck. To solve this, we introduced a novel GSPN micro-design (Sec. 3.3) that significantly reduces this dependency without sacrificing accuracy, clearly an intrinsic algorithmic advancement.\n\n- **Algorithm-system Co-design:** Unlike typical algorithm-only papers, we tightly fused this new micro-design into a single optimized CUDA kernel—exactly demonstrating an algorithm-system co-design. This integrated solution unlocks the full potential of GSPN in practical scenarios, underscoring that our contribution is both algorithmically substantial and systemically necessary.\nWe appreciate your feedback and will revise the paper to better highlight these algorithmic contributions and explicitly clarify their integration with our system-level optimizations.\n\n\n**Presentation Clarity.** Thanks for pointing out the issue. We will clarify the connection between each bottleneck and its corresponding solution, and restructure Section 3 for better flow in the revised paper.\n\n**How Channel-shared Weight Mitigates the Concurrency Bottleneck?** They mitigate the concurrency bottlenecks by reducing the number of independent CUDA blocks needed during kernel launch. In GSPN-1, per-channel weights ($W_i$,$c$) required independent computation for each (N, C) slice, quickly saturating the GPU's grid dimension and memory queues. In contrast, channel-sharing allows one weight matrix per row/column ($W_i$), enabling:\n\n- Shared memory reuse: once W_i is loaded into shared/L1 cache, it’s reused across all C channels within the block, improving cache hit rates and lowering register pressure. See +2D thread block ($1.4\\times$), +shared memory cache ($1.5\\times$), and +channel-shared gates ($1.3\\times$) at Fig.3.\n- Fewer parameter fetches per step, accelerating global memory throughput (confirmed by Nsight in Table 1).\n\nThis significantly improves parallel efficiency on modern GPUs—especially when combined with 2D blocks.\n\n**Relate Compressive Proxy Dimension to Low-rank Approximation.** Yes—conceptually, the proxy dimension ($C_{proxy}$) is a low-rank projection of the original $C$-dimensional input. Specifically, we project $x\\in R^{N\\times C\\times H\\times W}$ to $x_{proxy} \\in R^{N\\times C_{proxy}\\times H\\times W}$ via a lightweight linear layer (Sec. 3.2), apply the GSPN recurrence in this reduced space, and then project back to $C$ channels. This reduces CUDA workload from $N\\times C\\times H$ slices to $N\\times C_{proxy}\\times H$, mitigating the concurrency bottleneck under high $C$ or $N$. We empirically found no measurable loss in accuracy across classification and generation tasks. For example, GSPN-2-S (with compression ratio 2:1) improves accuracy over GSPN-S while using less compute (9.2G vs. 9.0G MACs). We also provide an ablation on $C_{procy} \\in \\{2, 4, 8, 16, 64\\}$ in the below table, analyzing the trade-off between accuracy and throughput. \n| C_proxy |  Acc | Throughput (img/s) |\n|:-------:|:----:|:------------------:|\n|    2    | 83.0 |        1544        |\n|    4    | 83.0 |        1492        |\n|    8    | 83.0 |        1387        |\n|    16   | 82.9 |        1293        |\n|    32   | 82.8 |        1106        |\n\nThis suggests that the proxy dimension serves primarily as a low-rank bottleneck for efficient propagation and does not need to match the full input dimensionality to be effective. We believe that the global context captured by the spatial recurrence appears to be preserved even through such a compressed intermediate representation."}, {"review_id": "tx1X0MM0Qb", "reviewer": "Reviewer_Da5V", "summary": "The paper describes GSPN-2, an optimised and engineered version of GSPN-1 - a vision architecture based that aims to replace attention-based architectures with parallel axis-aligned recurrence scans. The main results in the paper are centered around careful engineering of the algorithm and its implementation around moderns GPU kernels; and hardware utilisation is analysed in-depth. Experimental results are presented for IN-1K classification and text-to-image generation.", "questions": "See weaknesses section, especially the Major ones.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* The paper is very much a well-executed engineering exercise. While at first glance it may seem like a poor fit for NeurIPS, I would argue that these kind of papers are important and necessary. Transformers did so well partially due to the (accidental?) alignment between hardware and algorithms. Arriving at alternative high-performing architectures may require careful hardware-aligned engineering; and it is my belief that such work should be encouraged.\n* In-depth and insightful analysis of various implementation / algorithmic choices on runtime.\n* Clear and well-written. Strikes a good balance between details and high-level motivation.\n\n**Major weaknesses**\n* While excelling at latency analysis (e.g. Fig 3, Tab 1, Fig 4), I find that the paper falls short in empirical evaluation on relevant machine learning tasks. These results should be expanded along several dimensions (more below).\n* Ablations in Fig 1 should expanded to the downstream tasks (e.g. IN-1K, text2image, and ideally more). Right now it is unclear how or if the various latency optimisation affect downstream performance.\n* To motivate their work the authors explicitly mention detection and segmentation (lines 22-23) yet do not present such results. It would be extremely interesting and useful for the potential audience of the paper if such experiments could be performed and added to the paper.\n* Similarly, the authors mention models like CLIP / SigLIP. Including experiments that train such models with GSPN-2 as the vision encoder would also be extremely useful. In particular, it would demonstrate how this architecture scales with data - something that is completely missing from the current set of experiments.\n* Expanding on the data scaling axis - can this behaviour be at least demonstrated on larger image classification tasks - IN-1K is a relatively old and by modern standards a small benchmark.\n* Why were only small models considered in Tab 2? And why no results on resolution scaling are presented?\n* The effect of $C_{proxy}$ has not been ablated. How does this setting affect latency and downstream metrics (e.g. accuracy on IN-1K)? The choice of $C_{proxy}=2$ for the IN-1K experiments (lines 319-320) was in fact very surprising and intuitively seems like a very small number. Could the authors share their thoughts on why this is sufficient?\n* Text2Image generation results are not particularly useful without quantitative metrics (e.g. CLIP score, etc) - these should be added.\n\n**Minor weaknesses**\n* The choice of channel dimensions for benchmarking (e.g. in Table 1) was a bit surprising. 128 channels is not a lot when comparing to a ViT (e.g. ViT-B or ViT-L) architecture. Could the authors provide justification for the choice of these settings?\n* The paper emphasis forward pass benchmarking, although backward pass is also benchmarked (e.g. Fig 4). This should be emphasised more in the paper; it would also be interesting to see the ablations in Fig 3 be extended to the backward pass.\n* The authors do not discuss limitations (if any) that aligning architecture / and kernel implementation with hardware design introduces. Can the resulting model run on all accelerators? Would it still be easy to experiment with various architectural modifications in follow up research?", "was_revised": true, "final_justification": "The paper (especially empirical validation of the proposed approach) significantly improved during the rebuttal process.", "rebuttal": "We thank the reviewer for the thorough and insightful feedback. Below we respond to each concern and clarify the current scope, design choices, and planned extensions. \n\n**Machine Learning Tasks.** Our experiments in the main paper focus on image classification (ImageNet-1K) and text-to-image generation (Stable Diffusion XL) to evaluate both discriminative and generative capabilities. Besides, as GSPN-2 is designed for high-resolution spatial modeling, we have extended experiments to dense prediction tasks across various benchmarks in Appx E. We also provide preliminary integration of GSPN-2 into SigLIP2 in the response below to assess its capacity to scale with large and downstream generalization.\n\n\n**Fig. 1 Should be Expanded to the Downstream Tasks.** Yes, we did – the corresponding comparison has been fully explored in the paper: in Tab.2 of the main paper, we show an extensive comparison with both VIT (all with Flash Attn) based (orange block), and mamba-based (green block) architectures. In Fig. 1 of Appx B, we provided a comprehensive analysis of accuracy, model size, and throughput for GSPN-2 compared to SoTA architectures (e.g., ConvNet, Transformer-based, Mamba-based). These results highlight the competitiveness of GSPN-2—beyond speed gains alone—and establish it as a general-purpose architecture.\n\n**Detection and Segmentation.** We did provide both in Appx E: we have extended GSPN-2 with high-resolution inputs (e.g., **1024x1024** resolution) to evaluate on segmentation, depth estimation, and 3D probing benchmarks such as ADE20K, VOC2012, Probe3D, NYUDv2 and PASCAL in Appx E. Preliminary results at Tab. 2 indicate comparable or superior performance to transformer-based backbones with much better inference latency. We will move this experiment into the main paper in the final version.\n\n**SigLip.** Thank you for the valuable suggestion. We are currently integrating GSPN-2 into vision encoder backbones (e.g., SigLIP2-Base) using a subsample of 1 million samples from the DataComp1B dataset for rapid evaluation, with testing on ImageNet Zero-shot. The table shows a significant improvement in inference speed, e.g., 180 times faster compared with flash-attention and 5 times faster, including overheads like MLP. \n\n| BS=32, Resolution=1025                           | SigLip (Flash Attention)                          | SigLip-GSPN (ours)                |\n|-----------------------------------|----------------------------------|-----------------------------------|\n| Module-level Speed (ms) | 31.36           |  0.174 (**180x faster**) |\n| Block-level Speed (ms)                                  | 63.08              | 12.59 (**5× faster**)     |\n| Zero-shot Accuracy            | 53.4                             | 52.3                              |\n\n\n**IN-1K is Relatively Old.** We agree that ImageNet-1K is limited in scale. However, it remains a universally adopted and effective benchmark in attention-based literature [15-29], especially for comparing multiple model sizes (e.g., tiny, base, large) within practical GPU resource constraints. For instance, state-of-the-art methods like SigLIP and Radio require substantial GPU resources (multiple weeks on 16×8 GPUs), making them impractical for extensive variant analysis. Additionally, beyond IN-1K, we have successfully validated GSPN-2’s scalability to higher-resolution tasks (e.g., 1024×1024) in segmentation (ADE20K, VOC2012, NYUDv2, PASCAL) and text-to-image generation (SDXL) in Appx E, confirming its effectiveness across diverse and challenging dense-prediction benchmarks.\n\n**Only small models in Tab. 2.** To enforce fair comparison, we follow the common setting used in previous relevant works (e.g., VMamba, VisionMamba), where models are evaluated from tiny to large size on a 224×224 resolution. This choice aligns with the standard benchmarking practices in the field for efficient models.\nHowever, we acknowledge the importance of validating performance on high-resolution inputs, which are more practical for real-world applications. To address this, we have extended our experiments to both high-resolution text-to-image generation and dense prediction tasks using **1024×1024** resolution, as presented in Appx E. These experiments better reflect the capabilities of GSPN-2 in handling high-resolution visual data\n\n**Ablation on $C_{procy}$.** We selected $C_{proxy}=2$ as a minimal proxy to demonstrate how aggressively the input can be compressed while still preserving accuracy. We provide an ablation on $C_{procy} \\in \\{2, 4, 8, 16, 32\\}$ in the table below, analyzing the trade-off between accuracy and throughput. To make a fair comparison, we maintain almost the same model size for different  $C_{procy}$ by increasing the number of blocks if necessary. We found this value yielded.\n\n| C_proxy |  Acc | Throughput (img/s) |\n|:-------:|:----:|:------------------:|\n|    2    | 83.0 |        1544        |\n|    4    | 83.0 |        1492        |\n|    8    | 83.0 |        1387        |\n|    16   | 82.9 |        1293        |\n|    32   | 82.8 |        1106        |\n\nAs shown, using $C_{proxy} = 2$ achieves the best latency–accuracy trade-off, with no measurable loss in accuracy and the highest throughput. This suggests that the proxy dimension serves primarily as a low-rank bottleneck for efficient propagation and does not need to match the full input dimensionality to be effective. We believe that the global context captured by the spatial recurrence appears to be preserved even through such a compressed intermediate representation.\n\n**Quantitative metrics for T2I generation.** We have included CLIP-T scores and FID metrics in Tab 1 in Appx D, along with the quality-runtime trade-off at Fig. 3.\n\n**Choice of 128-channel setting.** We chose 128 channels to match prior system benchmarks (e.g., VMamba). However, we agree that large channel counts would be more realistic, so we update Table 1 with results for $C \\in \\{192, 768,1152\\}$ with appropriate resolutions below.\n| Input Size | Batch | Channels | GSPN-1 Throughput | GSPN-2 Throughput |\n|:----------:|:-----:|:--------:|:-----------------:|:-----------------:|\n|    32×32   |   32  |    196   |  114 GB/s (6.0%)  | 1832 GB/s (91.8%) |\n|    64×64   |   1  |    768   |   86 GB/s (4.5%)  | 1847 GB/s (92.3%) |\n|    64×64   |   1   |   1152   |   35 GB/s (2.1%)  | 1837 GB/s (92.0%) |\n\n**Backward Benchmark.** We agree, and will highlight backward pass performance more explicitly in the main text. Note that the backward function is very similar to the forward function in terms of kernel launch design, and thus the speedup of backward is similar to the forward behaviour. We will introduce it in the revised paper.\n\n**Run on Different Accelerators.** Yes, our CUDA acceleration design is fully compatible with all CUDA-enabled GPUs, relying only on standard libraries such as ATen and THC. Moreover, our algorithmic components—line-scan recurrence and channel-compressed formulations—can be readily ported to other accelerator backends (e.g., Triton or TileLang). The core idea of kernel fusion can also be adapted to accelerators that provide similar memory hierarchies. However, to achieve optimal performance, hardware-level control over thread blocks and shared memory is beneficial."}]

Author rebuttal:

Meta-review: This paper presents GSPN-2, a joint algorithm-system co-design that significantly improves the efficiency of the Generalized Spatial Propagation Network (GSPN) for vision transformers by addressing key implementation bottlenecks such as excessive GPU kernel launches, redundant memory access, and per-channel computation overhead. 

While some reviewers initially raised concerns about the clarity of algorithmic contributions, reproducibility, and evaluation breadth, the authors provided compelling rebuttals with extensive additional results, including comparisons with FlashAttention, ablation studies on the proxy dimension, downstream task performance (e.g., segmentation on ADE20K), and integration into models like SigLIP, demonstrating both the generality and practical impact of their optimizations. 

The final ratings are borderline accept, accept, borderline reject, and accept. AC has reached out to the BR reviewer but has yet to receive a response. Upon reviewing the BR reviewer’s comments, AC noted that this reviewer expressed less confidence compared to the others. Additionally, the AC aligns with the majority of reviewers in recognizing the work’s merits—specifically its strong systems insights, thorough evaluation, and potential to inform the design of efficient vision models. This alignment has led to a consensus in favor of acceptance.
Decision: Accept (poster)