From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics
Authors: Qinshuo Liu
Weiqin Zhao
Wei Huang
Yanwen Fang
Lequan Yu
Guodong Li
Venue: ICLR 2025 Poster (2025)
Keywords: deep neural network
sequential model
state space model
statistical model
Primary area: learning on time series and dynamical systems

Abstract: The depth of neural networks is a critical factor for their capability, with deeper models often demonstrating superior performance. Motivated by this, significant efforts have been made to enhance layer aggregation - reusing information from previous layers to better extract features at the current layer, to improve the representational power of deep neural networks. However, previous works have primarily addressed this problem from a discrete-state perspective which is not suitable as the number of network layers grows. This paper novelly treats the outputs from layers as states of a continuous process and considers leveraging the state space model (SSM) to design the aggregation of layers in very deep neural networks. Moreover, inspired by its advancements in modeling long sequences,  the Selective State Space Models (S6) is employed to design a new module called Selective State Space Model Layer Aggregation (S6LA). This module aims to combine traditional CNN or transformer architectures within a sequential framework, enhancing the representational capabilities of state-of-the-art vision networks. Extensive experiments show that S6LA delivers substantial improvements in both image classification and detection tasks, highlighting the potential of integrating SSMs with contemporary deep learning techniques.

Published as a conference paper at ICLR 2025
FROM LAYERS TO STATES: A STATE SPACE MODEL PERSPECTIVE TO DEEP NEURAL NETWORK LAYER DYNAMICS
Qinshuo Liu"!, Weiqin Zhao"! Wei Huang", Yanwen Fang', Lequan Yu Guodong Lit
'School of Computing and Data Science, The University of Hong Kong Department Electrical and Electronic Engineering, The University of Hong Kong {u3008680, wqzhao98, 98, u3545683) 8ao356633.ccnnct....mm hku. hk aaron. reihuang@ggmail com {lqyu, gelii hk
ABSTRACT
The depth of neural networks is critical factor for their capability, with deeper models often demonstrating superior performance. Motivated by this, significant efforts have been made enhance layer aggregation reusing information from previous layers to better extract features at the current layer, to improve the repre- sentational power of deep neural networks. However, previous works have primar- ily addressed this problem from a discrete-state perspective which is not suitable the number of network layers grows. This paper novelly treats the outputs from layers as states of a continuous process and considers leveraging the state space model (SSM) to design the aggregation of layers in very deep neural networks. Moreover, inspired by its advancements in modeling long sequences, the Selec- tive State Space Models (S6) is employed to design new module called Selec- tive State Space Model Layer Aggregation (S61A).. This module aims to combine traditional CNN or transformer architectures within a sequential framework, en- hancing the representational capabilities of state-of-the-art vision networks. Ex- tensive experiments show that S6LA delivers substantial improvements in both image classification and detection tasks, highlighting the potential of integrating SSMs with contemporary deep learning techniques.
INTRODUCTION
In recent years, the depth of neural network architectures has emerged as crucial factor influencing performance across various domains, including computer vision, natural language processing, and speech recognition. The network models are capable of capturing increasingly complex features and representations from data as they become deeper, and various methods have emerged to utilize larger numbers of layers to improve performance. For instance, the VGG network (Simonyan & Zisser- man, 2015) achieves higher classification accuracy by increasing the number of layers, although its foundation primarily relies on empirical results rather than systematical analysis. Other significant advancements, such as those demonstrated by CNNs (He et al., 201 6a; Ren et al., 2016; Tan & Le, 2020) and Transformers (Brown, 2020; Dosovitskiy, 2020; Touvron et al., 2021; Liu et al., 2021; Wang et al., 2022), showcase how deeper architectures can enhance accuracy and generalization.
Growing evidence indicates that strengthening layer interactions can encourage the information flow of a deep neural network. For CNN-based networks, ResNet (He et al., 2016a) employed skip connections, allowing gradients to flow more easily by connecting non-adjacent layers. DenseNet (Huang et al., 2018) extended this concept further by enabling each layer to access all preceding lay- ers within a stage, fostering rich exchange of information. Later, GLOM (Hinton, 2023) proposed an intensely interactive architecture that incorporates bottom-up, top-down, and same-level connec- tions to effectively represent part-whole hierarchies. Recently, some studies have begun to frame
Authors contributed equally. Corresponding author.

---

Published as a conference paper at ICLR 2025
structured state mabrices
intent and ipptt
trainable paramster
Linear frojectionfamn
Irllence froes a
8
8
D 4
Linear
Linear
Linear
Linear
OUTPUT
Figure 1: Schematic diagram of a Network with Selective State Space Model Layer Aggregation.
layer interactions with recurrent models and attention mechanisms, such as RLA (Zhao et al., 2021) and MRLA (Fang et al., 2023). All of the above models have been shown by empirical evidence to outperform those without interdependence across layers, and their achievements are obtained by treating the outputs of network layers discrete states.
However, the perspective of discrete treatment may not be suitable when a neural network very deep; say, for example, ResNet-152 has 152 layers. Sander et al. (2022) proposed to treat the ResNet as a discretized version of neural ordinary differential equations, i.e. the whole ResNet is considered
continuous process with the outputs from layers being the corresponding states; see also Liu et al. (2020). Moreover, Queiruga et al. (2020) argued that deep neural network models can learn to represent continuous dynamical systems by embedding them into continuous perspective. This motivates us to conduct layer aggregation among numerous layers of a neural network by alternatively assuming a continuous process to the outputs of layers.
Meanwhile, the State Space Model (SSM), a mathematical framework for continuously updating physical systems, enabled the modeling of dynamic processes and temporal dependencies in deep learning (Gu & Dao, 2023; Liu et al., 2024). Then, Mamba, a selective state space model (Gu & Dao, 2023), proposed selective mechanism and hardware- aware algorithm. which was particularly adept at addressing long sequence modeling challenges. The selection mechanism allows the model to filter out irrelevant information and remember relevant information infinitely.
The significance of layer aggregation in deep models and the popularity of SSMs lead us to propose a novel perspective: layer dynamics in very deep networks can be viewed as a continuous process with long sequence modeling task solvable by selective state space model (S6). By leveraging in- teractions between layers, outputs from different layers can be treated as sequential data input for an S6, allowing the model to encapsulate richer representation of the information derived from the original data. By conceptualizing neural networks as state space models, we introduce a novel structure that integrates traditional models into sequential architectures. This approach opens new research avenues that connect traditional statistical methods with contemporary deep learning tech- niques. Our proposed Selective State Space Model Layer Aggregation (S6LA) effectively harnesses the benefits of layer interactions while incorporating statistical modeling into vision tasks, such as those performed by CNNs and Vision Transformers (ViTs). A schematic of our model is illustrated in Figure 1, with parameters (A, A, B) indicating the influence of X on the implicit latent state h. Here - represents the output the 1)-th layer, which can be either a hidden layer in a deep CNN an attention layer in transformer model.
The main contributions of our work are given below: (1) For a deep neural network, we treat the outputs from layers states of a continuous process and attempt to leverage the SSM to design the aggregation of layers. To our best knowledge, this is the first time such a perspective has been presented. (2) This leads to proposed lightweight module, the Selective State Space Model Layer Aggregation (S6LA) module, and it conceptualizes a neural network as a selective state space model (S6), hence solving the layer interactions by the long sequence modelling selective mechanism. (3) Compared with other SOTA convolutional and transformer based layer aggregation models, S6LA demonstrates superior performance in classification, detection, and instance segmentation tasks.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
State Space Models. In the realm of state space models, considerable efforts have been directed toward developing statistical theories. These models are characterized by equations that map 1-dimensional input signal ztt) to an N N-dimensional latent state h(t), with the details provided in Equation 1. Inspired by continuous state space models in control systems and combined with HiPPO initialization (Gu et al., 2020), LSSL (Gu et al., 2021b) showcased the potential of state space models in addressing long-range dependency problems. However, due to limitations in memory and computation, their adoption was not widespread until the introduction of structured state space models (S4) (Gu et al., 2021a), which proposed normalizing parameters into a diagonal structure. S4 represents class of recent sequence models in deep learning, broadly related to RNNs, CNNs and classical state space models. Subsequently, Gu & Dao (2023) introduced the selective structured state space model (S6), which builds upon S4 and demonstrates superior performance compared to transformer backbones in various deep learning tasks, including natural language processing and time series forecasting. More recently, VMMmba (Liu et al., 2024) was developed, leveraging the S6 model to replace the transformer mechanism and employing scanning approach to convert images into patch sequences. Additionally, Graph-Mamba (Wang et al., 2024) represented a pioneering effort to enhance long-range context modeling in graph networks by integrating Mamba block with an input-dependent node selection mechanism. These advancements indicate that state space models have also been successfully applied to complex tasks across various domains.
Layer Interaction. The depth of neural network architectures has emerged as a crucial factor in- fluencing performance. And Figure of Appendix illustrates the enhanced performance of deeper neural networks. To effectively address the challenges posed by deeper models, increasing efforts have been directed toward improving layer interactions both CNN and transformer-based archi- tectures. Some studies (Hu et al., 2018; Woo et al., 2018; Dosovitskiy, 2020) lay much emphasis on amplifying interactions within a layer. DIANet (Huang et al., 2018) employed parameter-shanine LSTM throughout the network's 's depth to capture cross-channel re relationships by utilizing informa- tion from preceding layers. In RLANet (Zhao et al., 2021), recurrent neural network module was used to iteratively aggregate information from different layers. For attention mechanism, Fang et al. (2023) proposed to strengthen cross-layer interactions by retrieving query-related information from previous layers. Additionally, RealFormer (He et al., 2020) and TA-Transformer (Wang et al., 2021) both incorporated attention scores from previous layers into the current layer, establishing connec- tions through residual attention. However, these methods face significant memory challenges due to the need to retain features from all encoders, especially when dealing with high-dimensional data and they may lack robust theoretical supports.
PRELIMINARY AND PROBLEM FORMULATION
3.1 REVISITING STATE SPACE MODELS
The state space model is defined below, and it maps 1-dimensional input signal z(t), continuous process, to an --dimensional latent state (t),, another continuous process:
N (1) = Ah(t) + Bx((),
where NxN and B E Nx1 are the structured state matrix and weight of influence from input to latent state, respectively. We then can obtain the discretization solution of the above equation:
[" (l
ht - -
Together with the zero-order hold (ZOH) condition (Karafyllis & Krstic, 2011), i.c. x(T) is a con- stant intervals [t 1, for all integers f, we have
h² - + (AA)) - T) AB
(3)
As result, the continuous process Equation can be replaced by the following discrete sequence:
h = - + Br¹ with
Ā = AA)(AA) and B AA T) AB. (4)
3

---

Published as a conference paper at ICLR 2025
Following Gu & Dao (2023), refine the approximation of B using the first-order Taylor series:
B = (AA) (exp( AA) 1) AB ~ (AA) AB AB.
(5)
The formulas Ā (A A) and B = [E(A, A, B) are called the discretization rule, where B Lineary (x) is a linear projection of input into --dimension vector, and A = Linearr (z); see Gu al. (2021a); Gu & Dao (2023) for details.
3.2 CNN LAYER AGGREGATION
Consider a neural network, and let X be the output from its tth layer. We then can mathemati- cally formulate the layer aggregation at the /th layer below,
(6) where g6 is used to summarize the first layers, A is the aggregated information, and S produces the new layer output from the last hidden layer and the given aggregation which contains the previous information. The Hierarchical Layer Aggregation proposed (Yu et al., 2018) can be shown to have such similar mechanism which satisfies Equation 6.
A' g (x x
x ,x""),, X f'=
This formulation could be generalized to the special case of CNNs. The traditional CNNs do not contain layer aggregation since the layer output only depends on the last layer output, which over- looks the connection between the several previous layers' influence. DenseNet (Huang et al., 2018) perhaps is the first one for the layer aggregation, and its output at /th layer can be formulated into
X = CCnvv3' [Conv] ( (Concat X X x'-,,,,,
Let A' = Conv1 C X x rewritten into our framework at Equation 6. RLA (Zhao et al., 2021) considers a more convenient additive form for the layer aggregation, and it has the form of A = E/D (x ), where the kernel weights of Conv1 1 form partition of the weights in Conv1'. As a result, a lightweight aggregation can be formed:
))) and X Conv3'(A") and then DenseNet can be
X Conv3'(A 1 (At=3"(A-1+++ Conv1 (onv1i_((**
Without loss of generality, ResNets (He al., 2016a;b) can also be treated layer aggregation. Specifically, we can treat the update of X = X + f -x with applying the update
recursively as A ( + x and X A + +
3.3 ATTENTION LAYERS AGGREGATION
In this section, we show how to generalize the layer aggregation within transformer. Consider simple attention layer with general input X RLxD and output O E RR Its query Q, key K and value V are given by linear projections W, E DD W E DxD and W, D i.c. Q" X. KT W,X and VT W.X. As result, the output O has the following mathematical formulation:
QKT
Self-Attention/ X)= softmax(
V VD
Let X E RLLD with 1 < t < T be the output from tth layer, where L is the number of tokens, D represents the channel of each token, and T the number of attention layers. A vanilla transformer can then be formulated into:
A x + Self-Attention (x* x x'=
= AA= MLP(Norm(A'))
(10)
Note that these simple self-attention layers can only capture the connection between the current layer output and the last output; they are supposed perform better if the information from previous layers can be considered. To this end, we may leverage the idea given by CNN aggregation to concatenate the previous outputs. Specifically, the vanilla transformer at Equation 10 has the form of:
(11) where g is the attention layer, and ft is the Add & Norm layer for the t-th layer. Following Zhao et al. (2021) at Equation 8, we may then use the recurrent mechanism to combine all the outputs given by attention layers, i.e. replacing A = g²(X)) .... X ) by A' A + (
X == (g'g(X))
4

---

Published as a conference paper at ICLR 2025
h
0f
W
Residual
Convid
Avgpool
W
Training
Parameter
Projection
X
X
Figure 2: Detailed operations in S6LA module with Convolutional Neural Network. The green arrow shows the hidden state connection, while the grey arrow indicates layers communications.
3.4 THE FORMULA OF S6LA
Denote a sequence X = (x ,X"), where X is the output from tth layer, say Convolutional layers/blocks Attention layers, of deep neural network, and T is the number of layers. In financial statistics, the price of an asset can be treated as a process with discrete time when it is sampled in low frequency, say weekly data, while it will be treated time the sampling frequency is high, say one-minute data; see Yuan et al. (2023). Accordingly, we may treat X sequence with discrete time as the number of layers T is small even moderate, and all existing methods for layer aggregation in the literature follow this way. However, for a very deep neural network, it is more like the scenario of high-frequency data, and hence continuous process is more suitable for the sequence X (Sander et al., 2022; Queiruga et al., 2020). This section further conducts layer aggregation by considering state space models in Section 3.1; see Figure for the illustration.
process with continuous
Specifically, we utilize the Mamba model (Gu & Dao, 2023) due its effectiveness processing long sequences. This model is based on S6 models and can provide better interpretation how to leverage the previous information and then how to store based on its importance. Moreover, has been demonstrated have better performance than traditional transformers and RNNs. Following its principle, we propose our selective state space model layer aggregation below:
h (
X f=--(1--1"" X
(12)
where ht is a hidden state similar A in Equation 6, and represents the recurrently aggregated information up to (t 1)th layer. We may consider an additive form, as in Equation 8, for h!. Moreover, g" is the relation function between the current SSM hidden layer state and previous hidden layer state with input. As result, similar to Equation 4, the update of h can be formulated as:
N¹ A BX. X x'=
(13)
The choice function ff different for CNNs and Transformer based models, and they are detailed in the following two subsections; see Figures 6 and 3 for their illustrations, respectively.
3.5 APPLICATION DEEP CNNS
For CNNs backbones, we adopt ResNet (He et al., 2016a) as the baseline architecture. We propose concatenate the imput each layer, say X "xW' where and W represent the height and width, and D indicates the embedding dimension. For each CNN layer, the input each block ResNet- comprisng a combination of 1D and convolutional operations-is formed by con- catenating X with the state h E #XXWX from the previous layer, where N is the dimension of latent states. This integration effectively incorporates SSM information into the convolutional layers, enhancing the network's capacity to learn complex representations.
For the specific implementation of S6LA in CNNs, we initialize the SSM state 1t' using Kaiming normal initialization method (He et al., 2015). This initialization technique is crucial for ensuring effective gradient flow throughout the network, and we will further clarify this point ablation studies. Next, employ selective mechanism to derive components, the coefficient B for

---

Published as a conference paper at ICLR 2025
Wa
A
x
WB
X
Training
X
Class token
Concat
Patching
Sput
ipput
X
X
Linear Projection
Figure 3: Diagram of the S6LA architecture with Transformer. The green shows the hidden state connection, while the grey arrow indicates communication between layers. The input consists of image patch tokens (Xg) and class token (Xz)), processed through positional embedding and attention. The class token is cloned as X and parameters WA and Wn update the hidden state (h*). Updated patch tokens are combined with the class token to form the next input (X++1).
input and the interval A as specified in Equation 4. For transition matrix A, the initial setting is same as in Mamba models (Gu Dao, 2023). Then with Equation 4, we can get the next hidden layer he based on the last h f-1 and X for each layer in CNNs.
Utilizing Equation 4, we compute the subsequent latent state h based on the previous state h and the input X for each layer within the CNN architecture. This methodological framework facilitates improved information flow and retention across layers, thereby enhancing the model's performance. Therefore, the specifics of leveraging our S6LA method with CNNs backbones can be outlined as follows:
Input Treatment:
We begin by merging the input X and the hidden state ht-1 through simple concatenation along the feature dimension. This concatenated representation allows us to generate the output O' with the CNNs backbone (such as ResNet).
For the input state update, we define X the sum of X and O For the hidden state, ht derived as a function of these two components, following the formulation provided in Equation 4. The equations are follows with two trainable parameters Wa and Wp (for the layer):
Latent State Update:
h = ( + ABO',
(14)
where A = WW (a(Coov(O')), = W WB(Conv(O"))
Output Computation: The output component Of from the input treatment step, contributes to the next input x and the computation is: X O X!
3.6 APPLICATION TO DEEP VITS
In our exploration of S6LA implementation in deep ViT backbones, we draw parallels between the integration of the state space model (SSM) state and the mechanisms used in convolutional neural networks (CNNs). However, the methods of combining inputs within transformer blocks differ significantly from those in CNNs. Like the treatment of attention mechanism, multiplication combination instead of simply concatenating to deal with the input X' and h- in transformer- baadddd models. This approach enhances the interaction between input features and SSM, enabling richer feature representation. Then the next paragraghs gives the specifics of leveraging S6LA method with ViT' backbones follows:
utilize
Input Treatment:
We begin by combining the class token and input, alongside the application of positional embeddings. Then following the attention mechanism, X Empu E (L++1))x appeared where L is the number of patches and D is the embedding dimension, and it is split into two com- ponents next: image patch tokens X E IR L.D and a class token X E IRI x
x Sigurr Ndd&:Norm (M. AAddd Attn (x*))))))
X X! Split( (ipp))
(15)

---

Published as a conference paper at ICLR 2025
Table I: Comparisons of the Top-1 and Top-5 accuracy the ImageNet- K validation with CNNs. All the results are reproduced by us with 4 GeForce RTX 3090 GPUs with the same param- eters. The bold fonts denote the best performance.
Model
Method
Params
FLOPs
Top-1 Acc.
Top-5 Acc.
Vanilla (He 2016a)
25.6 M
4.1
76.1
92.9
SE (Hu al., 2018)
28.1 M
4.1 B
76.7
93.4
CBAM (Woo al,, 2018)
28.1 M
4.1
77.3
93.7
A (Chen et al., 2018)
34.6 M
7.0B
77.0
93.5
AA (Bello al., 2019)
27.1 M
4.5
77.4
93.6
ResNet-50
NI. (Wang et al., 2018)
29.0 M
4.4
77.2
93.5
GC (Cao al., 2019)
26.9 M
4.1
77.3
93.5
all GC (Cao et al., 2019)
29.4 M
4.2
77.7
93.7
+ ECA (Wang et al., 2020b)
25.6 M
4.1
77.4
93.6
+ RLA (Zhao et al., 2021)
25.9 M
4.3
77.2
93.4
+ MRLA (Fang et al,, 2023)
25.7 M
4.6
77.5
93.7
S6LA (Ours)
25.8 M
4.4
78.0
94.2
Vanilla (He et al., 201
44.5
7.8
77.4
93.5
+ SE (Hu ct al., 2018)
49.3 M
7.8
77.6
93.9
+ CBAM (Woo al., 2018)
49.3 M
7.9
78.5
94.3
AA (Bello al., 2019)
47.6 M
8.6
78.7
94.4
ResNet-101
ECA (Wang et al., 2020b)
44.5
7.8
78.7
94.3
RLA (Zhao al., 2021)
45.0
MRLA (Fang al., 2023)
44.9
8.5
78.7
S6LA (Ours)
45.0 M
8.3
79.1
94.8
Vanilla (He al., 2016a)
60.2
11.6
78.3
94.0
SE et al., 2018)
66.8
11.6
CBAM (Woo
2018)
66.8
11.6
78.8
ResNet- 152
AA (Bello al., 2019)
66.6
11.9
79.0
94.6
ECA (Wang al., 2020b)
60.2 M
11.6
78.9
94.5
RLA (Zhao al., 2021)
60.8 M
12.1 B
78.8
94.4
MRLA (Fang al., 2023)
60.7 M
12.4 B
79.1
94.6
S6LA (Ours)
60.8 M
12.2 B
79.4
94.9
The class token plays a crucial role in assessing the correlation between X and ht-1, Our model setting can effectively bridge the features extracted from the patches with the SSM state by facili- tating a better integration into the hidden state since it could be considered as a summary feature of the image in sequential layers.
Latent State Update: Given the split class token in last step, the hidden state is updated similar to application in CNNs:
ht = (( + ABX
(16)
where and are calculated from class token with selective mechanism:
A = W B (X))
(17)
Output Computation: At the same time, the new patch tokens X are computed as the sum of the previous patch tokens and the product of the previous patch tokens with ht:
X X + WX
(18)
Then the next input, x is derived from the concatenation of the updated patch and class tokens:
X Concat X x
(19)
EXPERIMENT
This section evaluates our S6LA model in image classification, object detection, and instance seg- mentation tasks, and provides ablation study. All models implemented by the PyTorch toolkit on GeForce RTX 3090 GPUs. More implementation details, and comparisons are provided in Appendix D.

---

Published as a conference paper at ICLR 2025
Comparisons of the Top-1 and Top-5 accuracy the ImageNet-1h kK validation set with vision transformer-basee models. All the results are reproduced by us with 4 GeForce RTX 3090 GPUs with the same parameters. The bold fonts denote the best performance.
Table 2:
Backbone
Method
Params
FLOPs
Top-1
Top-5
DeïT-Ti (Touvron et al., 2021)
5.7 M
1.2 B
72.6
91.1
+ MRLA (Fang et al., 2023)
5.7
1.4
73.0
+ S6LA (Ours)
6.1 M
B.5
73.3
92.0
DeîT-S (Touvron al., 2021)
22.1
4.5
79.9
95.0
DeiT
+ MRLA (Fang et al., 2023)
22.1
4.6
80.7
95.3
+ S6LA (Ours)
23.3 M
4.8 B
81.3
96.0
DeiT-B (Touvron al., 2021)
86.4
81.8
95.6
+ MRLA (Fang et al., 2023)
86.5 M
16.9 B
82.9
96.3
+ S6LAA (Ours)
86.9 M
17.1 B
83.3
96.5
Swin-T (Liu ct al., 2021)
28.3 M
4.5 B
81.0
95.4
+ MRLA (Fang al., 2023)
28.9 M
4.5
80.9
95.2
+ S6LA (Ours)
30.5 M
4.5 B
81.5
95.6
Swin-S (Liu al., 2021)
49.6 M
8.7 B
82.8
96.1
Swin
MRLA (Fang al., 2023)
50.9 M
8.7 B
82.5
96.0
+ S6LA (Ours)
52.5
8.7
83.3
Swin-B (Liu et al., 2021)
87.8 M
15.4 B
83.2
96.4
+ MRLA (Fang al., 2023)
89.8
15.5
82.9
+ S6LA (Ours)
91.3 M
15.5
83.5
96.6
PVTv2-BO (Wang al., 2022)
3.4 M
0.6 B
70.0
89.7
+ MRLA (Fang et al., 2023)
3.4
0.9 B
70.6
90.0
+ S6LA (Ours)
3.8 M
0.6 B
70.8
90.2
PVTv2-B1 (Wang al., 2022)
13.1
2.3
78.3
PVTv2
+ MRLA (Fang et al., 2023)
13.2 M
2.4
78.9
94.9
+ S6LA (Ours)
14.5 M
2.2
78.8
94.6
PVTv2-B2 (Wang et al., 2022)
25.4 M
4.0
81.4
95.5
+ MRLA (Fang al., 2023)
25.5 M
4.2 B
81.6
95.2
S6LA (Ours)
26.1 M
4.1 B
82.3
95.9
4.1 EXPERIMENTS ON IMAGE CLASSIFICATION
Backbone. For the dataset, we use Imagenet- IK dataset (Deng al., 2009) directly. For the CNN backbone, we choose different layers of ResNet (He et al., 201 6a). For transformer-based model, DeiT (Touvron al., 2021), Swin Transformer (Liu et al., 2021) and PVTv2 (Wang al., 2022) are considered. We compare our S6LA with baselines and other layer aggregation SOTA methods with different backbones alone as the baseline models.
Experimental settings. The hyperparameter of state space model channel N shown in Section 3.5 introduced control the dimension of feature of h in per S6LA hidden layer module. After comparison of different N = 16, 32, 64 for ResNet, we choose 32 as our baseline feature channel, for others we talk about Section 4.3. order to compare enhanced by S6LA fairly, we use the same data augmentation and training strategies as in their original papers (Zhao et al., 2021; Fang et al., 2023) all our experiments.
baseline models and the models
Main results. The performance of our main results, along with comparisons to other methods, ensure fair comparison, all results for the models listed in
presented Tables and these tables were reproduced using the same training setup our workstation. Notably, our model outperforms nearly baseline models. We specifically compare our S6LA model with other layer interaction methods using ResNets as baselines. The results in Table demonstrate that our S6LA surpasses several layer aggregation methods on CNN backbones, including SENet (Hu et al., 2018), CBAM (Woo et al.. 2018), A Net (Chen al., 2018), NL (Wang et al., 2018), ECA-Net (Wang et al., 2020b), RLA-Net (Zhao et al., 2021) and MRLA (Fang et al., 2023). Additionally, we find that model consistently outperforms them, achieving nearly 2% improvement in Top-1 accuracy
8

---

Published as a conference paper at ICLR 2025
Table 3: Object detection results of different methods MS COC02017. The bold fonts denote the best performance.
Method
Detector
APh
Al
AP
APS
AP
APT,
ResNet-S0 (Hc et al., 2016a)
36.4
58.2
39.2
21.8
40.0
46.2
+
SE (Ilu al., 2018)
37.7
59.1
40.9
22.9
41.9
48.2
+
ECA (Wang al., 2020b)
38.0
40.9
23.4
42.1
RLA (Zhao al., 2021)
38.8
59.6
42.0
22.5
42.9
49.5
MRLA (Fang et al., 2023)
40.1
61.3
43,8
24.0
43.9
52.2
S6LA (Ours)
Faster R-CNN
40.3
61.7
43.8
24.2
44.0
52.5
ResNet--10 (He et al., 2016
38.7
60.6
41.9
22.7
43.2
50.4
+ SE (Hu ct al., 2018)
39.6
62.0
43.1
23.7
44.0
51.4
+ ECA (Wang et al., 2020b)
40.3
62.9
44.0
24.5
44.7
51.3
RLA (Zhan et al., 2021)
41.2
MRLA (Fang el al., 2023)
41.3
62.9
45.0
24.7
45.5
53.8
S6LA (Ours)
41.7
63.0
45.2
24.6
45.6
53,9
ResNet-50 (Ile al., 2016a)
35.6
55.5
38.2
20.0
39.6
46.8
+ SE (Ilu et al., 2018)
37.1
57.2
39,9
21.2
40.7
49.3
+ ECA (Wang ct al., 2020b)
37.3
57.7
39.6
21.9
41.3
48.9
+ RLA (Zhau et al., 2021)
37.9
$7.0
40.8
22.0
41.7
49.2
+ MRLA (Fang el al., 2023)
39.1
58.6
42.0
23.6
S6LA (Ours)
39.3
59.0
41.9
23.7
42.9
51.0
RetinaNet
ResNet-101 (He
2016a)
37.7
57.5
40.4
21.1
42.2
49.5
SE (Hu al., 2018)
38.7
59.1
41.6
22.1
43.1
50,9
ECA (Wang al., 2020b)
39.1
59.9
41.8
22.8
43.4
50.6
+
RLA (Zhao et al., 2021)
40.3
59.8
43.5
24.2
43.8
52.7
+ MRLA (Fang et al., 2023)
41.0
60.0
43.5
24.3
44.1
52.8
S61.A (Ours)
41.2
60.4
43.8
53.0
with only 0.3 FLOPs compared to vanilla ResNet models. Moreover, in comparison with the latest state-of-the-arr method MRLA (Fang et al., 2023), our approach demonstrates fewer FLOPs and higher accuracy. As indicated in Table 2, our S6L.A achieves nearly a 1.5% improvement in Top-1 accuracy vanilla vision transformer-baser backbones such as DeiT (Touvron et al., 2021), Swin Transformer (Liu et al., 2021), and PVTv2 (Wang et al., 2022), with only a slight increase in parameters (+0.2 M) and FLOPs (+0.3 B), all within acceptable limits for hardware. Again, when compared to the latest SOTA method MRLA (Fang et al., 2023), our model shows fewer FLOPs and better performance.
4.2 EXPERIMENTS ON OBJECT DETECTION AND INSTANCE SEGMENTATION
This subsection validates the transferability and the generalization ability of our model on object detection and segmentation tasks using the three typical detection frameworks: Faster R-CNN (Ren et al., 2016), RetinaNet (Lin et al., 2018) and Mask R-CNN (He et al., 2018).
Experimental settings. For the dataset, we choose MS COCO 2017 (Lin et al., 2014) for exper- iments. All the codes are based on the toolkits of MMDetection (Chen al., 2019). The hyperpa- rameter state space model channel N introduced to control the dimension of feature of in per S6LA hidden layer module same to the settings in classification tasks.
Results of object detection and instance segmentation. For the results of object detection task, Table illustrates the details about AP of bounding box with the notation A It is apparent that the improvements on all metrics are significant. Also compared with the other stronger back- bones and detectors, our method outperforms in this task while we only add little parameters and FLOPs which can be overlooked by the servers. Meanwhile, Table 4 illustrates our S6LA method's improvements about AP of bounding box and mask on all the metrics with Mask R-CNN as the framework. Also similar to the advantages in object detection task, our method balance the param- eters and FLOPPs with traditional backbones. From the tables' results, proved that our S6LA model is feasible.
4.3 ABLATION STUDY
Different variants of S6LA. Due to resource limitations, we only experiment with the ResNet- 50 and DeiT models the ImageNet dataset. Our investigation considers several factors: (a) the

---

Published as a conference paper at ICLR 2025
Table 4: Object detection and instance segmentation results of different methods MS COC02017
with Mask R-CNN as a framework. The bold fonts denote the best performance.
Method
Params
Ap
AP
AP
APm
AP
AP
ResNet-50 (He et al., 2016a)
44.2 M
37.2
58.9
40.3
34.1
55.5
36.2
+ SE (Hu et al., 2018)
46.7 M
38.7
60.9
42.1
35,4
57.4
37.8
+ ECA (Wang ct al., 2020b)
44.2 M
39.0
61.3
42.1
35.6
58.1
37.7
+ NL (Wang et al., 2018)
46.5 M
38.0
59.8
41.0
34.7
56.7
36.6
GC (Cao et al., 2019)
46.9 M
39.4
61.6
42.4
35.7
58.4
37.6
RLA (Zhao al., 2021)
44.4 M
39.5
60.1
43.4
35.6
56.9
38.0
MRLA (Fang al., 2023)
44.4
44.0
36.9
57.8
S6LA (Ours)
44.9 M
40.6
61.5
44.2
36.7
58.3
38.3
ResNet-101 (He al., 2016a)
63.2 M
39.4
60.9
43.3
35.9
57.7
38.4
SE et al., 2018)
67.9
40.7
44.3
36.8
59.3
39.2
ECA (Wang al., 2020h)
63.2 M
41.3
44.8
37.4
59.9
NL (Wang al., 2018)
65.5 M
40.8
63.1
44.5
37.1
59.9
39.2
GC (Cao al., 2019)
68.1 M
41.1
63.6
45.0
37.4
60.1
39.6
RLA (Zhao al., 2021)
63.6 M
41.8
62.3
46.2
37.3
59.2
40.1
MRLA (Fang al., 2023)
63.6 M
42.5
63.3
46.1
38.1
60.3
40.6
S6LA (Ours)
64.0 M
42.7
63.3
46.2
38.3
60.5
41.0
influence of X on h (where the opposite is h randomized for each iteration); (b) the hidden state channels set to 16, 32, and 64; (c) the selective mechanism involving the interval A and coefficient method, using simple concatenation instead of multiplication.
B; (d) for the Transformer-basee
From our analysis of the results presented in Tables 5 and 6, several key findings emerge. Firstly, models incorporating our S6LA framework demonstrate superior performance compared those without it. Notably, using a trainable parameter l (h influenced by X) yields better performance. Secondly, regardless of whether we use ResNet DeiT, we find that channel dimension of N 32 yields the best results. Finally, the selective mechanism is crucial for our model; specifically, for the vision Transformer method (in this case, DeiT), the multiplication of X and outperforms simple concatenation used in CNN backbones.
Table 5: The influence of trainable h and se- lective mechanism of A and B.
Table 6: The influence of latent dimension N and the treatment of DeiT-Ti.
Model
Purams
Top-1
Model
Params
Top- 1
S6LA
25.8
78.0
N 16
25.8
77.9
ResNet
w/o trainable h
25.8 M
77.4
ResNet
N 32
25.8
78.0
S6LA
6.1 M
73.3
N 64
25.9 M
77.7
DefT-Ti
w/o trainable h
6.1 M
72.5
N 16
5.9 M
72.7
S6LA
25.8 M
78.0
DeiT-Ti
N 32
6.1 M
73.3
ResNet
w/o selective
25.8
77.3
N = 64
6,3 M
72.9
S6LA w/o selective
73.3
DeiT: (S6LA)
6,1 M
73.3
DeTT-Ti
6.1 M
72.7
DeiT-Ti (Concatenation)
6.1 M
72.6
5 CONCLUSION
In conclusion, we have demonstrated enhanced representation of information derived from the original data by treating outputs from various layers as sequential data inputs to a state space model (SSM). The proposed Selective State Space Layer Aggregation (S6LA) module uniquely combines layer outputs with a continuous perspective, allowing for a more profound understanding of deep models while employing selective mechanism. Empirical results indicate that the S6LA module significantly benefits classification and detection tasks, showcasing the utility of statistical theory in addressing long sequence modeling challenges. Looking ahead, we aim optimize approach by reducing parameters and FLOPs while enhancing accuracy. Additionally, we see potential for integrating further statistical models into computer science applications. suggesting promising convergence in these fields.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1286-3295, 2019.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet squeeze-excitatiee networks and beyond. In Proceedings of the IEEE/CVF international confer- ence on computer vision workshops, pp. 0-0, 2019.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Mmdetection: Open mmlab detection toolbox and benchmark, 2019. URL https: / arxiv. www.wwww.//a....oooo.oo.oooo
org/ abs /90..99....000 07155.
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A 2-nets: Double attention networks. Advances in neural information processing systems, 31, 2018.
Cheng Chi, Fangyun Wei, and Han Hu. Relationnet++:: Bridging visual representations for ob- ject detection via transformer decoder. Advances in Neural Information Processing Systems, 33: 13564-13574, 2020.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 JEEE Conference Computer Vision and Pattern Recognition, pp. 245, 2009. doi: 10. 109/CVPR. 5506844.
Alexey Dosovitskiy. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.
2020.
Yanwen Fang, Yuxi Cai, Jintai Chen, Jingyu Zhao, Guangjian Tian, and Guodong Li. Cross-layer retrospective retrieving via layer attention. arXiv preprini arXiv:2302.03985 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint ar xrXiv:23/2.00752, 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, arXiv preprint wrXiv:2111.00396 2021a.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré, Com- bining recurrent, convolutional, and continuous-time models with linear state-space layers, 2021b. URL https:tt//// / /awpiv org/ abs / 13985.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 201
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual net- works. In Computer Eision-ECCV 2016: 14th European Conference, Amsterdam, The Nether- lands, October 1l 11-14, 2016, Proceedings, Part IV 14, pp. 630-644 645. Springer, 2016b.
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn, 2018. URL https:
//arxiv. org/ aba/ /1t.//103..0 703.0687000000 06870.
Ruining He, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. Realformer: Transformer likes
residual attention. arXiv preprint arXiv:20/2.1174/) 2020.
Geoffrey Hinton. How to represent part-whole hierarchies in a neural network. Neural Computation, 35(3):413-455 2023.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry, Aug-
ment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. $129-8138, 2020.
Jie Hu, Shen, and Gang Sun. Squeeze-and-excitatio networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2018. URL petttp::/aarxta org/ abs/ 06993.
Zhongzhan Huang, Senwei Liang, Mingfu Liang, and Haizhao Yang. Dianet: Dense-and-implicit attention network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 4206-4214, 2020.
lasson Karafyllis and Miroslav Krstic, Nonlinear stabilization under sampled and delayed measure- ments, and with inputs subject to delay and zero-order hold. IEEE Transactions Automatic Control, (7(5):1711:)) 141-1154, 2011.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, 2018. URL https:tpp///cccc arrivv org/ abs/ 100.. 02002.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsich. How does noise help robustness? explanation and exploration under the neural sde framework. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 282-290, 2020.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401. 10100,,0666......000 2024.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Alejandro F Queiruga, N Benjamin Erichson, Dane Taylor, and Michael W Mahoney. Continuous- in-depth neural networks. arXiv preprint 0rXiv:2008.02389, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions pattern analysis and machine intelligence, 39(6): 137-1149, 2016.
Michael E. Sander, Pierre Ablin, and Gabriel Peyré. Do residual neural networks discretize neural ordinary differential equations?, 2022. URL https: /ttt:///xxxii arxiv org/ abs /tdbb//25... 2205. 14612.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015. URL httts:t//axxxii
hat///b////ee 1556.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020. URL httpt:ttt/////w..t.taa /bsrg////////// 905.
1905 11 1.66.
12

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pp. 10347-10357. PMLR, 2021.
Chloe Wang, Oleksii Tsepa, Jun Ma, and Bo Wang. Graph-mamba: Towards long-range graph sequence modeling with selective state spaces. arXiv preprini arXiv:2402.00785 2024.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 24-25, 2020a.
Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu, Eca-net: Ef- ficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1134-11554, 2020b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415-43) 424, 2022.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Yujing Wang, Yaming Yang, Jiangang Bai, Mingliang Zhang, Jing Bai, Jing Yu, Ce Zhang, Gao Huang, and Yunhai Tong. Evolving attention with residual convolutions. In International Con- ference on Machine Learning, pp. 10971-10980. PMLR, 2021.
Sanghyun Woo, Jongehan Park, Joon- YounYoung Lee, and So Kweon. Cham: Convolutional block attention module. In Proceedings of the European conference computer vision (ECCV), pp- 3-19, 2018
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 2402-2412, 2018.
Huiling Yuan, Kexin Lu, Yifeng Guo, and Guodong Li. Har-ito models and high-dimensional har modeling for high-frequency data, 2023. URL https: : / arxiv. org/ /aabs/23333 02896.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 6023-6032, 2019.
Hongyi Zhang. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
Jingyu Zhao, Yanwen Fang, and Guodong Li. Recurrence along depth: Deep convolutional neural networks with recurrent layer aggregation. Advances in Neural Information Processing Systems, 34:10677-10640, 2021.
Yue Zhao, Junzhou Chen, Zirui Zhang, and Ronghui Zhang. Ba-net: Bridge attention for deep con- volutional neural networks. In European Conference on Computer Vision, pp. 297-312. Springer, 2022.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001 13008, 2020.
13

---

Published as a conference paper at ICLR 2025
A
THE CORRELATION BETWEEN ACCURACY AND LAYER SIZE
Classification Accuracy with Different Layer Sizes
84
PVTv2
83
Accurary (%)
81
80
79
Layer Size
Figure 4: The correlation between accuracy and layer size with model PVT-v2.
Figure 4 illustrates that deeper neural networks can get better performance.
B FULL CAPTION OF FIGURES
structured matrices
and
traiaable paramster
Linear fromm
Inflence Bi
pi-1
g
Linear
Linear
Linear
Linear
C-11
OUTPUT
Figure 5: Schematic Diagram of Network with Selective State Space Model Layer Aggre- gation. The green arrow represents hidden state connection, while the grey arrow indicates communication between layers. The updated latent layer is derived from the previous latent layer h'-l and the last input layer X The output of the t-th layer is generated from the input X ( and the latent layer h'.
C PSEUDO CODE
In this section will propose our pseudo of method for CNNs and transformers.
14

---

Published as a conference paper at ICLR 2025
h
0f
W
Residual
Convid
Avgpool
W
Training
Parameter
Projection
X
X
Figure 6: Detailed Operations in the S6LA Module with Convolutional Neural Network. The green arrow represents the hidden state connection, while the grey arrow indicates communication between layers. The temporary output Of is the concatenation of the latent layer ht-1 and the input layer X After passing through the convolutional layers, the updates for the latent layer and the input layer X derived from the last layers.
Algorithm S6LA used in CNNs
Input: Output of the first CNN block X E RIxHxWXD latent state dimension N.
Output: Output of the T-th CNN block X E RIxHxw>D
1: Initial the latent state no, the trainable structured state matrices A;
2: for each t E [1 do
3: h influence X
4: Concatenate the - 1-th output X and the / 1-th latent state h
5: Get the information O concluding the parts with CNNs and residuals;
6: Treat with Conv ld and average pooling to the next step;
7: I/ selective mechanism
8: Through linear projection to get the selective parameters A and B from the last step;
9: // X influence h
10: Calculate the next output X and the update latent state h with X -0-- + x
11: end for
12: return Outputs X
Algorithm S6LA used Vision Transformers
Input: Output of the first CNN block X e IxxLx latent state dimension N. Output: Output of the 7'th CNN block x e glxLkD
Initial the latent state ft,, the trainable structured state matrices A;
2: for each t E [1, T' do
3: Combine the class token with the 1-th output X and add positional embeddings;
4: Attention mechanism with X X Add&:Norm( (Attn(AAtn(()) (ttn(())=))) and get X E IR (l++1))xD, inpur
5: Split X and get the two components X -
6: // h influence X, multiplication not simple concatenation
7: The patch tokens: X X W
8:
// selective mechanism
9:
Through linear projection get the selective parameters A and B from the class token: ( // X influence h
(X!-1)));
11: Calculate the next output XT and the update latent state hT: h² C(AA))-1 ABX:-
and X = Concat( X X!-1).
12: end for
13: return Outputs X
15

---

Published as a conference paper at ICLR 2025
768 768
30
GFLORs induced by S6LA
20
15
522'512
10
3841364
112
768
Input Resolution
Figure 7:
The GFLOPs induced by our method with respect to input resolution with backbone PVT-
v2-b1.
D EXPERIMENTS
D.1 IMAGENET CLASSIFICATION
D.1.1 IMPLEMENTATION DETAILS
ResNet For training ResNets with our method, we follow exactly the same data augmentation and hyper-paramett settings in original ResNet (He et al., 2016a). Specifically, the input images are randomly cropped 224 x 224 with random horizontal flipping. The networks are trained from scratch using SGD with momentum of 0.9, weight decay of le-4, and mini-batch size of 256. The models are trained within 100 epochs by setting the initial learning rate to 0.1, which is decreased by factor of 10 per 30 epochs. Since the data augmentation and training settings used in ResNet are outdated, which are not as powerful as those used by other networks, strengthening layer interactions leads to overfitting ResNet. Pretraining larger dataset and using extra training settings can be an option.
DeiT, Swin Transformer, PVTv2 in DeiT. All models trained for 300 epochs using the AdamW optimizer with weight decay of 0.05. We use the cosine learning rate schedule and set the initial learning rate as 0.001 with batch size of 1024. Five epochs used gradually training. We apply RandAugment (Cubuk et al., 2020), repeated augmentation (Hoffer et al., 2020), label smoothing (Szegedy et al., 2016) with 0.1, Mixup (Zhang, 2017) with 0.8 probability, Cutmix (Yun et al., 2019) with 1.0 probability and random erasing (Zhong et al., 2020) with 0.25 probability. Similarly, our model shares the same probability of the stochastic depth with the MHSA and FFN layers of DeiT/CeiT/PVTVV.
adopt the same training and augmentation strategy as that
up the learning rate at the beginning of the
D.1.2 MODEL COMPLEXITY WITH RESPECT TO INPUT RESOLUTION
Figure illustrates the FLOPs induced by model S6LA with respect to input resolution. We use the model PVTv2-bl as the backbone and then derive the differences under various settings of input resolution. From this, it is apparent that complexity of our method is linear input resolution.
16

---

Published as a conference paper at ICLR 2025
D.1.3 COMPARISONS WITH RELEVANT NETWORKS
Layer-interaction networks For the comparison of layer-inleraction-aass models using CNNs, we first compare our method S6LA with Densenet (Huang et al., 2018), DIANet (Huang et al., 2020), RLA (Zhao et al., 2021) and MRLA (Fang al., 2023). The comparison on the ImageNet- IK validation set are given in Table 1. From the table, it is obvious that our method outperforms in the classification task and also compared with the similar model size of DenseNet, our model performs better.
Other relevant networks The methods in the last part, they all use the same implemental settings from ours in the training of Imagenet. However for some other models, such as BA-Net (Zhao et al., 2022), adopted other settings. It applied cosine learning schedule and label smoothing in their training process. However, the different settings of our method and their method will give the unfair comparison. Therefore, we adopt the results given in MRLA (Fang et al., 2023) since our method settings are same to this paper. The results are given Table
Table 7: Performances of layer-interaction- based models on ImageNet- 1K validation dataset. The bold one denotes the best performance. (The performances of DIANet and DenseNet from the two papers)
Model
Params
FLOPs
Top- Acc.
Top-5 Acc.
ResNet-50 (He et al., 2016a)
25.6 M
4.1
76.1
92.9
+ DIA (Huang et al., 2020)
28.4 M
77.2
RLA (Zhao al., 2021)
25.9 M
4.5
77.2
93.4
MRLA (Fang et al., 2023)
25.7 M
4.6 B
77.5
93.7
+ +66AA (Ours)
25.8 M
4.4 B
78.0
94.2
ResNet-101 (He al., 2016a)
44.5 M
7.8
77.4
93.5
DIA (Huang et al., 2020)
47.6 M
78.0
+ RLA (Zhao al., 2021)
45.0 M
8
78.5
94.2
MRLA (Fang et al., 2023)
44.9 M
8.5
78.7
94.4
+ S6LA (Ours)
45.0 M
8.3 B
79.1
94.8
DenseNet-161 (k=48) (Huang et al., 2018)
27.4 M
7.9
77.7
93.8
DenseNet-264 (k=32) (Huang et al., 2018)
31.8 M
5.9
77.9
93.8
Table Performances of BA-Net model ImageNet-IK validation dataset under our settings. The bold one denotes the best performance.
Model
Params
FLOPs
Top-1 Aec.
Top-5 Acc.
BA-Net-50 (Zhao al., 2022)
28.7 M
4.2
77.8
93.7 94.2
ResNet-50 + S6LA (Ours)
25.8 M
4.4 B
78.0
D.2 OBJECT DETECTION AND INSTANCE SEGMENTATION ON COCO DATASET
Implementation details We adopt the commonly used settings by Hu et al. (2018); Wang et al. (2020b); Cao et al. (2019); Wang et al. (2021): Zhao et al. (2021); Fang al. (2023), which same to the default settings in MMDetection toolkit (Chen et al., 2019). For the optimizer, we use SGD with weight decay of le-4, momentum of 0.9 and batchsize of 16 for all experiments. The learning rate is 0.02 and is decreased by a factor of 10 after 8 and 11 epochs for within the total 12 epochs. For RetinaNet. we modify the initial learning rate 0.01 avoid training problems. For the pretrained model, we use the model trained in ImageNet tasks.
Results For the experiments of detection tasks in the similar settings models, it is illustrated in Table 4 and 9. For almost backbone models, our model performs better and it proves that model
17

---

Published as a conference paper at ICLR 2025
is also useful in detection tasks. For some other implemental settings such ResNeXT (Xie al., 2017) RelationNet++ (Chi et al., 2020), our method also outperforms object detection results.
Table Object detection results of different methods on MS COC02017. The bold one denotes the best performance.
Backbone Model
Aph
pbb AP
AP 75
AP
APh
APh
30
ResNet-100 (He et al., 2016a) ResNeXT-100132x44 (Xie al., 2017)
37.7
57.5
40.4
21.2
42.2
49.5
39.9
59.6
42.7
22.3
44.2
52.5
RelationNet++ (Chi et al., 2020)
39.4
58.2
42.5
S6LA (Ours)
40.3
61.7
43.8
24.2
44.0
52.5
D.3 VISUALIZATIONS
To investigate how S6LA contributes to representation learning in convolutional neural networks (CNNs), we utilize the ResNet-50 model (He al., 2016a) our backbone. In this study, visualize the feature maps using score- weighee visual explanations generated by ScoreCAM (Wang al., 2020a), as illustrated in Figure and
We specifically focus on the final convolutional layer of the ResNet-50 model and our S6LA- enhanced model. This choice is grounded our observation that feature maps from the ini- tial three layers of both models exhibit remarkable similarity. In the visualization, the first column presents the original images, the second column displays the ScoreCAM images, and the third col- umn showcases the combination of the original images and their corresponding ScoreCAM. Both two images our analysis are randomly selected from the ImageNet validation set, ensuring a di- verse representation of the data. According the definition of the CAM method, areas highlighted in warmer colors indicate stronger contributions to the classification decision.
From visualizations, it is evident that models enhanced with S6LA exhibit larger regions, which align more closely with the classification labels. In contrast, the vanilla ResNet-50 model struggles identify all relevant object compared our method. This disparity suggests that our approach not only improves the localization of important features but also enhances the model's overall classification performance.
The findings presented in the figure provide direct evidence of the efficacy of our method in the classification task. By leveraging S6LA, we can significantly improve the interpretability CNNs, allowing for better insights into how these models make decisions based on the features they learn. summary, our results highlight the advantages incorporating S6LA into standard architectures like 5esNet-50, ultimately leading to more robust and accurate classification outcomes.
18

---

Published as a conference paper at ICLR 2025
Res-55
mage
Ddr
Image + 2om
Res RLA
ResNet-50 + MRLA
Image
ResNet-50 + SGLA
Figure 8: The visualizations of the feature maps extracted from the end convolutional layer of ResNet, RLA, MRLA and our S6LA. The left ones are original images, the middle colomn is the CAM and the right ones are the combinations of left and middle. Compared with others, the red areas of our method are concentrated in the more critical regions of the object (fish) of classification task.
19

---

Published as a conference paper at ICLR 2025
maage
Frace com
ResNet-50
Irage
ResNet-50 + RLA
magge
Irage cam
ResNet-50 + MALA
Frece
ResNet-50 S6LA
Figure 9; The visualizations of the feature maps extracted from the end convolutional layer of ResNet, RLA, MRLA and our S6L.A. The left ones are original images, the middle colomn the CAM and the right ones are the combinations of left and middle. Compared with others, the red areas of our method are concentrated in the more critical regions of the object of classification task.
20


Reviews:
[{"review_id": "8ouraQSNkS", "reviewer": "Reviewer_nBY3", "summary": "The paper presents a new module named Selective State Space Model Layer Aggregation (S6LA), designed to integrate traditional CNN or transformer architectures into a sequential framework for improving the representational power of advanced vision networks. The paper shows results on image classification and detection and segmentation benchmarks.", "questions": "What kind of results does the table 4 contains? Does it use Mask RCNN 1x schedule, 3x schedule (from implementation details it seems like it is 1x schedule but then why is 3x schedule not included)?\nIn fig. 6 and fig 7, Can the author(s) please comment or write in the caption how the qualitative results for the proposed technique are better than the competition?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Overall the paper seems to be structured.", "weaknesses": "The main issue of the paper is that it is not written with enough clarity and often sidetracks from the main points making it hard to understand the main point. Unfortunately, the lack of clarity in the paper makes it hard to judge the work itself. The idea pitched forward in the paper might be interesting but in its current form, it's remotely comprehensible. I would suggest that the author(s) go over the paper and rewrite it in as simple and clear as possible. Also, please go over the mathematical notation and polish it more. Moreover, please pay attention to the citations to back up the claims. For example, lines 230-234, does not contain any citation or mathematical grounds for the claims made there. Similarly, line 95-96 is cryptic. Moreover, the captions of the figures should also indicate what is happening in the figure (this goes for all the figures included in the paper). I believe that once the paper writing is majorly polished and refined, it may have a higher chance of getting through.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fEkuEtstlL", "reviewer": "Reviewer_1DRX", "summary": "The paper proposes a novel layer aggregation method S6LA that treats the outputs from layers as states of a continuous process and utilizes selective state space models architecture for layer aggregation. The paper reformulates two different layer aggregation processes with the proposed S6LA for ConvNet and Transformer-based backbones. The experiments are conducted on two tasks image classification\nand detection on ImageNet, MS COCO 2017 datasets.", "questions": "I have a question about transformer-based S6LA integration.\n\nIn the paper, equations 15, 18, 19, and Figure 3, $X^{t−1}_{input}$ go through the attention block, add & norm, and split into two parts: the cls embedding goes through the S6LA, image embeddings add with the product of itself with hidden state, and the final output is the concatenated cls embedding and image embedding.\n\n1. Can you provide a clarification that with the S6LA integration, are the MLP blocks excluded from the original Transformer block?\n\n2. If yes, can you clarify the reason for excluding?\n\n3. If no, how it fits into the pipeline detailed in equations 15, 18, 19, and Figure 3? \n\n4. (I am assuming Linear Projection in Figure 3 means MLP block in the Not Exlcuded case) Since in the traditional transformer-based methods,  after attention and add & norm, cls embedding is also fed into the MLP layer along with image embeddings. From Figure 3, the proposed method seems to exclude cls embedding from MLP layers. Can you clarify the reason and potential impact on layer aggregation if this is the case?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and well-organized.\n2. The idea of utilizing a selective state space-based model for seamless layer aggregation with continuous layer output is novel and interesting. The different integration formulation using the proposed S6LA for different backbones(ConvNet and Tranformer) is reasonable.\n3. The quantitative results in the experiment sections show the proposed method's effectiveness over baseline methods.", "weaknesses": "1. The pipeline figure (Figure 3) for the Transformer-based approach can be improved so it will be easier for the reader to follow. For example, the naming labels in Figure 3 should be more aligned with what's in the paper along with some pipeline walkthrough captions.\n2. Selective State Space models are known for slow training due to their nature of backpropagation through time (even with parallel scans). Will this affect the transformer and ConvNet backbone's parallelization training compared to the baseline methods that are Transformer or ConvNet-based?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sC7lGK8N1W", "reviewer": "Reviewer_BEKP", "summary": "The central idea is to treat various layer outputs as continuous states within a state space model (SSM) and introduce the Selective State Space Model Layer Aggregation (S6LA). The outputs from different layers are fed to the state space model as sequential inputs. This approach aims to enhance the representational power of neural networks by aggregating information across layers in a continuous manner, motivated by advancements in SSM for sequence modeling. Experimental results show that S6LA performs well in comparison to standard convolutional and transformer-based layer aggregation techniques in image classification and object detection tasks.", "questions": "* How does the computational cost of S6LA compare to simpler aggregation techniques, and are the improvements in accuracy justified by this cost?\n* The concatenation vs. multiplication of X and h variables seem to have a lot of impact on final performance. Can the authors motivate why different fusion mechanisms are needed for different architecture families i.e., CNN vs ViTs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The approach is novel, application of state space models to layer aggregation in deep neural networks is an interesting concept. \n* S6LA demonstrates notable improvements in classification and detection tasks, especially when compared to existing layer aggregation methods.\n* The S6LA module is compatible with both CNN and transformer architectures.\n* The paper includes extensive experiments on benchmark datasets, showing consistent gains in accuracy.\n* The ablation experiments are detailed and well positioned to study the effect different design elements in the approach.", "weaknesses": "* Introducing state space models to layer aggregation may increase computational complexity, can the authors add run time comparisons of their and competing methods?\n* Authors criticize the RealFormer and EA-transformer approaches for high memory consumption. What is the memory consumption of proposed architecture? It will be good to clarify and compare the memory consumption.\n* SSM models can be unstable in training, do authors observe any unstable training behaviour with the proposed SSM based block?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p849EI8vIE", "reviewer": "Reviewer_kdSR", "summary": "This paper proposes a novel layer aggregation strategy based on the recent S6 block. Two different implementations are conducted for CNN and Transformer architectures. Detailed experiments are delivered on both classification and object detection tasks. The proposed strategy achieves almost the best improvement among current layer aggregation strategies.", "questions": "Could the authors also report the performance improvement on the ADE20K dataset?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed layer aggregation based on the S6 block is delicate and achieves significant improvement on various tasks and model architectures. This paper is easy to understand, and the proposed method is easy to follow.", "weaknesses": "1. The main problem for me is the different designs between the transformer and CNN. There is a lack of valid explanation here as to why the transformer needs a multiplication strategy while CNN requires an additive one. The only reason given in Lines 301-302 from the input perspective is not convincing enough. Additionally, the performance of CNN with a multiplication format of S6LA (used in the transformer) is missing here.\n\n2. The authors claim that Kaiming normal initialization is crucial for the initial hidden state and will verify this point in the ablation study. However, no relevant experiments are found in the main paper.\n\n3. The impact of S6LA on model efficiency is needed, including detailed inference and training FPS and memory occupation.\n\n4. Some minor errors. The authors may consider replacing the \"th\" with a subscript, such as the one in Line 164. Besides, from 0 to t-1, there are a total of t layers instead of t+1, as stated in Line 167.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)