TF-MAS: Training-free Mamba2 Architecture Search
Authors: Yi Fan
Yu-Bin Yang
Venue: NeurIPS 2025 poster ()
Keywords: neural architecture search
training-free
Mamba2
mapping matrix
hyperparameter ranges
Primary area: deep_learning
TL;DR: We design a training-free neural architecture search method for Mamba2.

Abstract: The Mamba-type neural networks have gained significant popularity recently. To effectively and efficiently establish model architectures of Mamba, it is natural to introduce Neural Architecture Search (NAS) methods into Mamba. However, existing NAS methods tailored for Mamba are training-based, leading to substantial time and computational resource expenditure. To address this issue, and considering that Mamba2 is an improved version of the original Mamba, we propose a training-free NAS method specifically designed for Mamba2. Based on rank collapse in stacked State Space Duality (SSD) blocks, we design a proxy that only requires the computation of the transformation matrix and its gradient between two tensors within the network. Additionally, we develop a corresponding search space and introduce a novel approach for determining adjustable hyperparameter ranges. Experimental results show that our method outperforms all existing training-free NAS approaches in terms of both ranking correlation and the performance of search results for Mamba2 architecture. To the best of our knowledge, this is the first training-free NAS method designed for Mamba-type architectures. Our codes are available at https://github.com/fanyi-plus/tf-nas.

TF-MAS: Training-free Mamba2 Architecture Search
Yi Fan
State Key Laboratory for Novel Software Technology
Nanjing University
Nanjing 210023, China
fanyiplus@smail nju. edu.
Yu-Bin Yang'
State Key Laboratory for Novel Software Technology
Nanjing University
Nanjing 210023, China
yangyubindnju. wewpiinnnnnuuuu.....mmm edu.
Abstract
The Mamba-type neural networks have gained significant popularity recently. To effectively and efficiently establish model architectures Mamba, natural introduce Neural Architecture Search (NAS) methods into Mamba. However, existing NAS methods tailored for Mamba are training- based, leading to substantial time and computational resource expenditure. To address this issue, and considering that Mamba2 an improved version of the original Mamba, we propose a training- free NAS method specifically designed for Mamba2. Based rank collapse stacked State Space Duality (SSD) blocks, we design a proxy that only requires the computation of the transformation matrix and its gradient between two tensors within the network. Additionally, we develop corresponding search space and introduce novel approach for determining adjustable hyperparameter ranges. Experimental results show that our method outperforms all existing training- free NAS approaches in terms of both ranking correlation and the performance of search results for Mamba2 architecture. To the best of our knowledge, this is the first training-free NAS method designed for Mamba-type architectures. Our codes are available at httph::t/// //gi thub. com/fanyi plus/ plus/tf-nas
Introduction
Recently, the introduction of Mamba has provided new impetus for research in State Space Models (SSM) [24], and Mamba-type models possess potential to partially replace Transformer [6] 64 29]. Subsequently, Mamba2 [13], developed network performance. However, similar to other types of networks, the practical use Mamba- type networks still necessitates the manual specification of hyperparameters to determine the actual model architecture [14]. This process often involves trial-and-error, which can be time-consuming and labor-intensive. Therefore, it is logical apply Neural Architecture Search (NAS) techniques [52] to the design of Mamba-type architectures. Currently, among the limited work, (50) explores hyperparameters such as network depth and width, while 27 [9 search for scanning orders of image patches for vision-oriented Mamba.
enhancement of the original Mamba. further improves
However, these methods rely heavily extensive training of the networks during the search process. In fact. recent research in the NAS is increasingly leaning towards training- frre NAS (35), also known
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

as zero-shot NAS. These approaches avoid the need to train candidate networks and instead compute a proxy relying only on information from the parameters, gradients, latent variables, etc.. of the network in its initial state. The proxy is correlated with the actual performance of the network. The earliest training-free NAS methods target convolutional networks [31 49 28 followed by methods designed for Transformers [68] 54 20]. Among these, DSS [68] is classic method that originates from a finding suggesting that when stacking increasing attention layers in a model, the rank of the output trends towards [88. Based on this, DSS proposes indirect measurement of the degree to which the rank approaches as a representation of candidate network performance. Later adaptations of the method [69] introduce layer-wise proxies. Unfortunately, no training- training-free NAS methods for Mamba-type tymee networks have yet emerged.
To fill this gap in training-free NAS for Mamba-type networks, we propose TF-MAS. Given that the overall performance of typical Mamba-type model, Mamba2 [33 exceeds that of raw Mamba, our search space is designed based on Mamba2. By theoretically demonstrating that when the core structure, State Space Duality (SSD), in Mamba2 is stacked continuously, its output also trends towards 1. Inspired by DSS, we design training-free NAS method for Mamba2. In this method, we need to compute the transformation matrix between the input tensor of each Mamba2 block and the input tensor of the corresponding SSD. When there are different relationships between the sequence length and feature dimensions of the input tensor of the Mamba2 block, we devise distinct methods to derive the transformation matrices. Once the transformation matrices are obtained, the values of the proxies can be computed. Additionally, we design a search space that includes 4 Adjustable Hyperparameters (AHs). Unlike existing methods, the ranges of these AHs are not artificially set but are determined through precise calculations based on the expected cost for specific tasks. This increases the likelihood of discovering candidate networks with superior performance within the search space.
We construct several NASBenches based on existing pre-trained Mamba2 models, with each dataset containing 500 network architectures sampled from the specific search space along with their corre- sponding performance metrics on different datasets. We compute the ranking correlation between our proxies and actual performance on these NASBenches. The results indicate that our method outperforms all other training-free -ree NAS approaches. Subsequently, under the overhead constraint, establish the search space, conduct searches, and train the search results from scratch. The search results exceed the performance of the original Mamba2 while maintaining computational costs not greater than those of the original networks. These results demonstrate the unique effectiveness of our method in searching for Mamba2 architectures. Furthermore, we conduct additional experiments to investigate rank convergence, transformation matrices, and search spaces to validate the rationality and robustness of our method.
We give an brief introduction about the related work and a discussion about societal impacts in Section and B of the technical appendix, respectively. The contributions of this paper follows:
We introduce, for the first time, a training-free NAS method for the Mamba-type networks.
We design a search space where the range of AHs is dictated by computational costs.
We validate the effectiveness of our method through ranking correlation and the performance of search results.
2 Premilinary
simplified Mamba2 model is illustrated in Figure (exclludn the red areas). consists of several Mamba2 cells arranged in series, with each containing a Mamba2 block. Within the Mamba2 block, the input U with dimension T x W is replicated into three copies. which are processed through fully connected layers, convolutions, and the SiL.U activation function, resulting in three variables: X, B. and C, with dimensions T x P. T x N, and T x N, respectively. These three matrices are then fed into the SSD module, whose computation is defined as follows:
Y (L (CB)) X,
where L is computed from a parameter of the network, and Y is the output. Conceptually, the meaning of W. T, P, and N are the width of network (the dimension of each token), sequence length

---

Mumba2 cell
input embedding
Mamba2 block
Mamba2 cell
Mamba2 cell
output
X
scaling
Mamba2
FC
convld
SiLU
block
B
input
PC
convld
norm
output
PC
convid
SiLUU
Wx
We
proxy calculation
90
8C 3W n
Wallow villox
BC
Welao
8W
TF-MAS layer
TF-MAS in one layer
TF-MAS in layer
TF-MAS
Structure of Mamba2 and the corresponding TF-MAS method. To emphasize the key points, we focus primarily on the components related to the computation of the proxy in the Mamba2 structure, while omitting variables such as Z and dt.
Figure 1:
(token number), feature dimension in SSD, and state space size, respectively. Similar transformer architectures, the SSD module in Mamba2 consists of multiple heads.
B indicates that the SSD module shares the same computational pattern linear attention, where the inputs X, B, and C correspond to the value, key, and query in linear attention, respectively.
Method
3.1
Proxy design
has been shown that if multiple attention modules are concatenated without the inclusion of regularization layers, the rank of the output during forward propagation will converge to 1 as the number of attention modules approaches infinity, a phenomenon known as rank collapse. Furthermore, points out that the forward process of the attention module essentially consistent with that of the SSD module, differing only in the representation of each variable. Therefore, we can reason that stacking multiple SSD modules will also result in rank collapse. A theoretical proof provided in Subsection C.I of the technical appendix, and we also verify this by experiments in Subsection 4.4 This phenomenon will affect the expressiveness of the network. This can be regarded as reason why the Mamba2 network necessitates the use of regularization layers (see Figure IT. However, as described in [88, regularization layers can only partially mitigate rank collapse and cannot entirely eliminate it. Therefore, inspired by the evaluation method for Multi-head Self-Attention (MSA) in ZeroLM 169, we consider designing new proxy evaluate the performance of different Mamba2 models by measuring the rank collapse magnitude.
18
[3]
ZeroLM states that rank collapse magnitude of outputs can be measured by calculating the rank of the weights after network initialization. To reduce the computational overhead of the proxy, further theoretical analysis has transformed this into computing the product of the nuclear norm of the weights and the nuclear norm of the gradients of the weights. In attention module, the weights are typically the transformation matrices from input to query/key/value via fully connected layer, while in Mamba2 block, the operations from U to X, B. and C' are not merely fully connected layer; they also involve convolution and SiLU. Here, we represent the transformation matrices from to X. B, and C' in Mamba2 block W Wn n, and Wc, thus the sizes are W P, W N, and W N, respectively. Due to the existence of convolution and SiLU, W x. W B, and W c not equal to the weight of the fully connected layer. Consequently, to compute the score of the network.
3

---

7B. JW U ac 8c U ac and ac U OC Besides, there are also weights, W in aw. we first need to determine Wx W B, and We. Below, we illustrate the calculation of W as an example. We have
UW = X.
The solution of Equation (2) depends on the shape of U, i.e, the relationship between T and W, and for different cases, we will adopt different measures to determine W x-
Case W. In the absence of prior knowledge about the input data, we can assume that the elements in U are independent and follow normal distribution; the reasonableness of this assumption is elaborated in Subsection C.2 of the technical appendix. It can be proven that, this case, the probability of the matrix being full rank is (the proof is provided in Subsection C.3 of the technical uppendix). Therefore. we can directly compute W = -U X.
Case 2: T < W. can be shown that the probability of each column of X lying within the column space of U is (the proof is provided in Subsection Cf the technical appendix), leading to the conclusion that Equation (2) has infinitely many solutions. To minimize the structural risk of the model, we seek to find W that minimizes the 2-norm, i.e., solve
argmin W
[Wxll s.t. UWx=Xx
(3)
We employ the least squares method for this. First, we perform Singular Value Decomposition (SVD) on U, obtaining U = VIoV2 2. where V, and V2 are orthogonal matrices, and or is diagonal matrix. Based on this, the Moore-Penroed pseudoinverse of U can be computed as U = V20 V where + is the pseudoinverse o. We ultimately obtain
argmin Wx
W x = X.
Case 3: T > W. In contrast to Case 2, the probability of each column of X lying within the column space of is 0 (the proof is provided in Subsection C.3 of the technical appendix), leading to the conclusion that Equation 2 has solutions. Thus, it impossible to find W x that accurately substitutes for the mapping from U to X. Therefore, we seek to find an approximate W x such that the result of the linear mapping as close as possible X. That is, we solve
argmin x
UWX - X
(5)
We again utilize the least squares method, yielding the calculation process consistent with Case 2. Considering that W x is obtained via the pseudoinverse approximation, we quantify how this error propagates and affects proxy reliability in Subsection of the technical appendix.
Now we finish the calculation of W x. The calculations for W B and We follow a similar process, except that all instances of P in the matrix sizes replaced with N. It can be observed that the solutions for Wx. W B. and We essentially resolve a set of matrix equations of the form concatenate all equations where A and B have the
AW B. To speed up the computation, same dimensions along the batch size dimension before computation, making full use of the parallel acceleration advantages offered by devices.
After obtaining x. WB, B. and We c for each Mamba2 block, we can compute the network score using the method described in [69]. However, unlike the approach in [69], these weights are not parameters of the network, and thus their gradients cannot be directly obtained through backprop- agation. Nevertheless, via chain rule and matrix calculus, we can easily calculate their values by DC
0x JW output layer of Mamba2 blocks, which functions similarly to the subsequent linear layer in attention modules; thus, participates in the score calculation as well. Specifically, the calculation formula for our proxy, TF-MAS, is
JC.
AL aW, muc
TE-MS Σ
I Illuuc
should be pointed out that although multiplying parameter-related terms and gradient-related terms has already been applied in existing methods [6)38]] none of these methods have provided

---

theoretical proof. We believe that multiplication can amplify or suppress certain characteristics based on the magnitude of parameters and gradients. When both parameters and gradients are large, their product significantly increases, thereby highlighting this important information in the evaluation metric. This enables architecture search to prioritize network architectures with larger parameters and gradients, as these architectures may have stronger learning ability and adaptability during training.
We presents the pseudocode of our proxy computation process in Section D of the technical appendix.
3.2
Search space
The design of the search space generally follows the design principles outlined in 10] Specifically, have set the following 4 AHs:
Depth (D): the number of Mamba2 blocks.
Width (W): the dimension of each token.
State dimension (N): the dimensionality of the state space.
Number of heads (H): the number of heads in each Mamba2 block.
All 4 AHs are positive integers. We set the dimension of each head to 64, which is consistent with the original Mamba2. This indicates that the number of columns in X is 64H. We denote this search space as SSMamba2 (Search Space of Mamba2). Moreover, to enhance the flexibility of the network architecture, and inspired by [10], extend SSMamba2 to form VWSSMamba2 (Variable Width Search Space of Mamba2). The candidate networks within this search space differ from existing Mamba2; thus, refer to them as VWMamba2 (Variable Width of Mamha2). This design permits the values of N and H to differ across layers, meaning that for specific network, N and H are no longer solely positive integers but rather D-dimensional vectors, where the i-th element represents the values of N and H in the i-th layer. It should be noted that our work follows the same protocol as 18 and uniformly employs skip connections in all candidate networks.
The range of values for each AH is determined by the expected computational cost. Suppose the expected computational overhead is provided by upper limit of parameter number, denoted by en- When constructing SSMamba2, we will reference an existing Mamba2 model, referred to as the reference mode 2 construct a virtual benchmark model. This benchmark model must satisfy two conditions: (a) According to the scaling rule, we want this benchmark model to be proportional scaling of the reference model. (b) The capacity of this benchmark model must equal co.
Let the AHs of the reference model be Dr, W,.. Np. and Hp, and the AHs of the benchmark model be Do. Ws, N₂, and Hp. According to condition (a), the equations D, i:D,,, W, kW, N₂ = AN,, and H, = kH, (where & is a positive constant) should be satisfied. Besides, the parameter number of an existing Mamba2 model can be computed as 2VW + DW + 10DN 387DH+ 2DWN + 193DWH (calculation details can be found in Section[EJof the technical appendix), where V is the dictionary length, a predetermined value. According condition (b), we have We + 2VW, + D,W,, + 10D,N, + 387D, H,, + W,,NNNN, + 193D, W,H,, cp- By substituting D kD, W.- AW,, N₂- AN,, and H₂ kH, into the equation and rearranging, we obtain cubic equation in terms of k,
(2D,W,N, + 19,D,W,H,) 1 + (2VW, + D,W, + 38,D, + 10D,N,) W, = cp- (7)
This equation can be solved using the quadratic formula (it is clear that the equation has a unique solution). Once is obtained, the AHs of the benchmark model can be determined. It is important to note that the AHs determined using this method likely not integers; thus, we refer to this benchmark model as virtual. The benchmark model does not need to be physically constructed; it merely serves as reference for determining the range of AHs. Ultimately, we will take |0.6 x 7: 1 and [1.6 x + J. T as the minimum and maximum values for the AHs, respectively, where the symbol r can represent D. W, N, or H, and Ir is a positive integer. In colloquial terms, this means that each AH can take on all values between 0.6 times and 1.6 times the corresponding benchmark model AH. The choice of the limits are based on previous design experiences of search spaces [10 51]
RResearchers or engineers proposing a new Mamba2- like model architecture typically define a specific model structure for training and testing, which can serve a reference model. Thus, we believe suitable reference model is usually available.
5

---

Table 1: Ranking correlation on VMSSMamba2Bench 2.7B. The names of the datasets LD, HS, OBQA, WG are abbreviations for LAMBADA, HellaSwag, OpenbookQA, and WinoGrande, re- spectively. The names of the proxies #Param, AC, HI, HC are abbreviations for Parameter Number, Attention Confidence, Head Importance, Head Confidence, respectively. For target, G, C, T, M means general, CNN, Transformer, Mamba, respectively. Best results in bold.
LD
LD
HS
PIQA
Arc-E Are-C (ACC) (ACC) (ACC)
WG
OBQA (ACC)
Time (s)
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
#Param
G
-0,458
0.437
0.214
0.310
0.246
0.323
0.378
0.325
0.07
Gradnorm
C
-0.091
0.017
0.043
0.036
0.080
0.027
0.075
0.094
0.96
Synflow [58
C
-0.040
0.033
-0.018
0.066
0.061
0.112
0.013
0.029
1.47
GraSP 60
C
-0.064
-0,004
0.044
0.030
0.024
0.042
0.013
0.030
1.54
Fisher 40
C
-0,081
0.004
0.056
0.052
0.035
0.056
0.079
0.021
1.77
Snip
34
0.024
0.061
-0.007
0.119
0.009
0.029
-0.013
0.035
0.89
Zen-NAS
15
C
-0.038
0.046
0.056
0.098
0.013
0.017
-0.006
-0.013
0.75
ZiCo 36
C
-0.093
0.066
0.061
0.020
0.022
0.052
0.055
0.070
0.71
MeCo
31
C
-0.011
0.057
-0.028
0.038
0.061
0.047
0.061
0.061
0.86
Auto-Prox
62
C
-0,034
0.047
0.016
0.042
0.050
0.069
0.029
0.064
1.09
AC [54
T
-0.130
0.112
0.110
0.114
0.081
0.047
0.049
0.108
0.91
HI 54
T
-0.087
0.082
0.069
0.089
0.045
0.113
0.050
0.092
0.97
HC 54
T -0.117
0.146
0.102
0.111
0.034
0.078
0.095
0.123
0.90
DSS++
69
T -0.118
0.070
0.108
0.087
0.043
0.085
0.091
0.089
2.18
AttnNAS
20
T -0.246
0.267
0.159
0.183
0.122
0.144
0.232
0.228
1.63
TF-MAS (ours)
-0.685
0.661
0.381
0.468
0.339
0.452
0.483
0.519
Additionally, within the search space, we must ensure that the values are multiples of 4. Considering the scales of each AH, we ID 1, !w 32, IN and /H 1.
the expected computational overhead is provided in terms of FLOPs or inference time, becomes challenging to accurately determine the benchmark model using the aforementioned method. To estimate the rough range of AHs, we upproximately assume that both FLOPs and inference time are proportional the number of parameters.
Upon completing the construction of SSMamba2, we maintain the limits for each AH while extending the values of N and H from integers to D-dimensional vectors, resulting in VWSSMamba2.
Finally, question worth investigating how the time complexity of the proxy varies with different AHs. Encouragingly, we derive that when any of W, N. H, P approaches infinity, the time complexity grows linearly. A detailed analysis is provided in Subsection C.7 of the technical appendix. This demonstrates the practicality of our method.
4
Experiments
4.1
Ranking correlation
The effectiveness of a proxy is most directly measured by ranking correlation. This involves sampling several network architectures from the search space, training them, testing their actual performance, and calculating their proxy values. The ranking correlation is just the correlation between the proxy values and the performance of the sampled architectures. A higher ranking correlation indicates that the proxy can more accurately estimate the performance of the network.
To save computation time, some researchers have stored sampled architectures and their corresponding performances to create datasets known as NASBench [3]. To our knowledge, current NASBenches encompass only CNN (65 57 [6] and Transformer (2). with no one specific Mamba. Therefore, we will sample architectures from both SSMamba2 and VWSSMamba2 to construct NASBenches pertaining Mamba. However, training numerous Mamba2 networks comparable to the model sizes reported in [13] from scratch incurs significant time cost. To mitigate this, we employ weight entanglement proposed in [10]. After sampling the architectures, we directly activate the corresponding part of parameters from Mamba2 pretrained models as the parameters of the sampled architectures. Nevertheless, we cannot guarantee that weight entanglement will function effectively under Mamba2. Therefore, for each sampled network, we perform 10 epoches of

---

Table 2: Test performance of search results on SSMamba2Bench 130M and VMSS- Mamba2Bench, 130M. The abbreviations in datasets are consistent with Table [
Best results
bold.
Model Size FLOPs LD
LD
IS
PIQA Arc-E Arc-C WG OBQA
Proxy
(M)
(G)
Mamba-2-1300 2-100
128.932
8.249
0.160
16.79
45.9
31.0
65.0
47.4
20.9
52,6
18.0
opt Mamba2
127.130
8.133
0.171
15.31
49.5
35.8
66,4
51.7
25.1
53.9
19.1
opt VWMamba2
119.104
7.620
0.151
15.20
50.7
36.9
66.4
51.8
25.2
54.7
19.4
opt VWMamba2 w/ bwe
122.572
7.842
0.125
15.08
51.5
40.5
66.9
52.6
25.2
55.2
19.5
fine-tuning using the Pile dataset. Importantly, weight entanglement primarily injects noise into the observed performance values, and additive noise tends to reduce the Kendall's Tau (as formally proved in Subsection C.8 of the technical appendix). Consequently, the empirical correlations reported here should be regarded conservative lower bounds; the actual ranking agreement in noise-free setting expected to be even higher, indicating that the use of weight entanglement is indeed acceptable.
Currently, there are five publicly available pretrained Mamba2 models with parameter number of 130M, 370M, 780M, 1.3B, and 2.7B. We generate a NASBench for each model on SSMamba2 and VWSSMamba2 with 500 architectures, resulting in total of 10 NASBenches, which we denote as SSMamba2Bench X and VMSSMamba2Bench. X (where "X" represents the parameter number). When constructing NASBench, corresponding AHs in the pre-trained model. We evaluate accuracy (where higher is better) on datasets LAMBADA [46), HellaSwag [66]. PIQA 771. Arc-E E 8 Arc-C [ WinoGrande (53), and
set the range of the AHs to be between 0.5 to times the
OpenbookQA 45), while for LAMBADA, we also assess perplexity (where lower is better).
[ the ranking correlation between various existing proxies and performance metrics on VMSSMamba2Benchh.. 2.7B. Kendall's Tau is used as correlation in this paper. Section F of the technical appendix includes results for other NASBenches. We compare our method with other existing proxies, including the parameter number, which often treated as a baseline [35]. Given that method is the first known triiiing-free NAS targeting the Mamba-type models, all other methods are oriented towards CNN or Transformer. is worth noting that for attention map-based proxies such as Attention Confidence [54] and AtnNAS 20), we consider CB² as the attention map, motivated by the duality between attention and SSD. Additionally, some classical proxies require specific operations or structures within the network (c.g., the CNN-oriented NASWOT [44] requires ReLU), which are absent in Mamba2; therefore, we do not include comparisons with these proxies.
apparent that the ranking correlation of existing methods is generally lower than the baseline, with only our method surpassing the baseline. This indicates that while existing methods have played a significant role in predicting the performance of CNNs and Transformers, they struggle to adapt to Mamba2, showcasing the value of our method. Furthermore, where Transformer-orienter methods (albeit not as effective as our proposed method) perform slightly better than CNN-oriented methods. We speculate that this may be due to certain structural similarities between Transformers and Mamba2, which provides inspiration for future improvements in proxies targeting Mamba2, suggesting exploration of insights from existing Transformer-bast based proxies.
observe interesting phenomenon
Moreover, recent studies have highlighted that ranking correlation evaluated only on the top-k candidates can be particularly informative in certain scenarios 67 15). We therefore computed the corresponding metrics and obtained results consistent with those in Table (see Section of the technical appendix), demonstrating the robustness of our proxy.
4.2 Performance of search results
While our method demonstrates better performance than other baselines in the previous subsection, the rank correlation coefficient is not sufficiently high in some datasets for the proxy to considered a fully reliable metric. So validating the feasibility in practical search scenarios is important. In this subsection, we utilize the proposed proxy to conduct evolutionary searches on SSMamba2 and VWSSMamba2. After obtaining the search results, we perform training and testing from scratch. Due to limited computational resources, we use the Mamba2 model with 130M parameters as the reference model, setting the expected computational overhead to "parameter number not exceeding
7

---

Table 3: Ranking correlation on SSMamba2Bench 130M and VMSSMamba2Bench 130M when us- ing and not using the weights of the fully connected layer as W W B. and W The abbreviations in datasets are consistent with Table and "wfc" means weights of the fully connected layer.
LAMBADA
LAMBADA
HS
PIQA
Are-E
Are-C
WG
OBQA (ACC)
Method
(PPL))
(ACC)
(ACC)
(ACC)
(ACC)
(ACC) (ACC)
SSMamba2 w/o wfc
0.677
0.648
0.406
0.451
0.364
0.435
0.517
0,508
SSVWMamba2 w/o
0.675
0.667
0.357
0,450
0.329
0.467
0,485
0,556
SSMamba2 w/ wfe
0.164
0,186
0.121
0.146
0.131
0.147
0.132
0.156
SSVWMamba2 w/ wfc
0.192
0.181
0.132
0.141
0.111
0.162
0.134
0.171
130M," which aligns with the existing pretrained models for comparison purposes. Clearly, under these conditions, the solution to Equation (7) is k = 1, and the benchmark model is just the reference model. The set of values for each AH and the details about the evolutionary method are demonstrated in Section Gof of the technical appendix.
Our search is conducted on NVIDIA Tesla V100 GPUs, with search times on SSMamba2 and VWSSMamba2 being 0.7 day and 0.6 day, respectively. 2 showcases the testing results. It can be observed that our search results, referred to as opt Mamba2 and opt VWMamba2 (where "opt" stands for optimized), demonstrate higher performance compared to the existing Mamba2 model while maintaining slightly lower parameter number, FLOPs, and inference latencies than the 130M model. This suggests that our method is capable of effectively optimizing the model architecture. Notably, the search results for VWSSMamba2 show a slight performance improvement over those for SSMamba2, indicating that making the widths of the layers variable increases the flexibility of the architecture, which is beneficial for constructing better models.
Additionally, inspired by the block-wise evolution described in (69] we repeat the experiments for VWSSMamba2 using an alternative evolutionary method. In this method, instead of calculating the proxy for the entire network at once, we compute the proxy layer by layer (see Section G of the technical appendix for more details). The performance of the VWMamba2 model obtained using this evolutionary method, referred to as opt VWMamba2 w/ bwe (where "w/ bwe" stands for "with evolution") "), is also listed in Table Compared to opt VWMamba2, opt VWMamba2 w/ bwe exhibits a further slight increase in performance. This not only further validates the effectiveness of block-wise evolution as proposed in [69], but also demonstrates that our method continues to be effective at the finer granularity of blocks. In Section of the technical appendix, we provide the architectures of the search results.
block-wis. evolutiono
4.3 Ablation study
Effects of transformation matrices. In Subsection 3.1 we indicated that in the calculation of the proxy, Wx. x. W B. W c. should be regarded as the transformation matrix from U to X, B, and C, including convolution and SiLU, rather than the weights of the fully connected layer with U as input. To validate this statement, we replace W x. W, B. W c in the proxy with the weights of the fully connected layer taking U as input and recalculate the correlation on SSMamba2Bench 130M and VMSSMamba2Bench 1 330M. The results are presented in Table It is evident that this substitution results in a significant drop in correlation for both NASBenches, confirming the appropriateness of our computation method for Wx. x. Wnn B, We.
Effects of each AH. In Subsection 3.2 we introduce 4 AHs in the search space. One question arises: can our method detect the impact of each AH's variation on network performance? In other words, when utilizing the proxy, are all 4 introduced AHs necessary? To address this, we calculate the correlation while varying only one AH at time. Specifically, we sample candidate networks using the 130M pretrained model in the search spaces of both SSMamba2 and VWSSMamba2. When investigating the necessity of a specific AH, all sampled candidate networks have only that AH differing, while the values of the other AHs remain consistent with those of the pretrained model. We then compute the ranking correlation for these candidate networks, with results shown in Table 4 Notably, in both search spaces, the ranking correlations for all four AHs are not near zero, indicating that the answer to this question is affirmative.
8

---

Table 4: Ranking correlation on SSMamba2 and VWSSMamba2 with only one AH varying. The
abbreviations in datasets are consistent with Table
LAMBADA
LAMBADA
HS
PIQA (ACC)
Arc-E (ACC)
Are-C (ACC)
WG (ACC)
OBQA
Search space
AH
(PPL.))
(ACC)
(ACC)
(ACC)
0,340
0.314
0.191
0.206
0.219
0.226
0.257
0.257
W
0,410
0,399
0.276
0.324
0.190
0,256
0.365
0.377
SSMamba2
N
0.128
0.128
0.048
0.069
0.076
0.041
0.092
-0.002
H
0.301
0.217
0.105
0.166
0.156
0.169
0.224
0.193
D
0.302
0.304
0.170
0.241
0.161
0.214
0.281
0.259
W
0.425
0.427
0.266
0.321
0.201
0.339
0.304
0.331
VWSSMamba2
0.092
0.065
0.016
0.102
0.093
0.046
0.083
0.100
H
0.227
0.231
0.158
0.267
0.141
0.206
0.214
0.253
Table 5: Ranking correlation on SSMamba2 and VWSSMamba2 when confronting case and 3 in The abbreviations in datasets are consistent with Table [T]
Subsection 3.1
LAMBADA
LAMBADA
HS
PIQA
Are-E
Are-C (ACC)
WG (ACC)
OBQA
Search space
Case
(PPL))
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
0.566
0.571
0.321
0.379
0.291
0.404
0.405
0.477
SSMamba2
3
0.476
0.485
0.291
0.323
0.302
0.321
0.394
0.437
0.551
0.568
0.354
0.398
0.337
0.430
0.460
0.479
VWSSMamba2
0.478
0.458
0.266
0.356
0.308
0.331
0.346
0.386
4.4 Discussion
Rank collapse stacked SSD blocks. valid the rank collapse in stacked SSD model, construct a stacked SSD blocks with 12 layers. After initial- ization, examin the rank collapse metrics (see Section [ of the technical appendix for the details about the metric) of the input, along with the output of each layer. Additionally, we include stacked atten- tion block, Transformer, and Mamba2 of the same depth for comparison. The results are illustrated in
1.0
SAN
Trasffrmer
0.8
Bank Rark collape metric
SSD
0.4
Figure 2 We observe that the depth of the net- work increases, this distance consistently decreases, though at slower rate stacked SSD. This indicates that similar to stacked attention blocks, there is a rank collapse phenomenon occurring stacked SSD blocks, albeit less pronounced than in stacked atten- tion blocks. This validates the theoretical foundation of the proposed proxy.
0.0
Layer index
Figure 2: Rank collapse metric in different layers of various architectures.
Validation across different cases. In Subsection
3.1 we discuss three cases for deducing W. x. WB and We based on the relationship between sizes and W. In fact, in all the experiments mentioned above, T is fixed at 64, while the minimum value of W is 384. This indicates that all proxies computed so far utilize case (T < W). To validate the effectiveness of case 1, we set T T W 64, and then perform sampling and calculate ranking correlation similar to that in Subsection 4.1 (though in this instance, W fixed). To verify the validity of case 3, we set T = 128 and W = 64, subsequently performing the same operations in case 1. The results are shown in Table where we can see that both cases, the ranking correlation values still exceed those of the baseline. This indicates that method is effective across all three cases.
Validation of AH ranges. Based on existing researches NAS, when capping computational budgets with an overhead limit, the best models often have costs slightly below the upper limit 39 33 Therefore, within given search space, if the overhead of the sampled networks are more

---

2.0
2.0
19
19
13
18
1A
1.5
0.00
1.5
1.
LA
1.3
0.02
13
12
0.01
0.01
1.0
0.00
1.0
0.00
0,575
0,625
Mininum
Minimum
(a)
(b)
Figure 3: The metrics corresponding to different AH ranges in SSMamba2(a) and SSVWMamba2 (b). The values on the axes are arranged this order to ensure that the upper right corner corresponds to larger search ranges. The red boxes indicate the range we use.
Table 6: Test performance of three independent training-testinn runs on opt VWMamba2 w/ bwe. The abbreviations in datasets are consistent with Table while Run ID & 3 are models retrained and tested.
[ Run ID is the model reported in Table 2
Run ID LAMBADA LAMBADA HS
PIQA
Arc-E
Are-C (ACC)
WG (ACC)
OBQA (ACC)
(PPL.)
(ACC)
(ACC)
(ACC)
(ACC)
15.08
51.5
40.5
66.9
52.6
25.2
55.2
19.5
2
15.05
51.2
40.5
66.8
52.6
25.2
55.3
19.4
3
15.10
51.4
40.7
67.0
52.9
25.3
55.5
19.5
likely fall slightly below the upper limit, the chance of discovering superior architectures is greater. The range of the AHs in the search space largely determines this probability. In Subsection 3.2 to establish the ranges of each AH, we introduce a benchmark model and specify that the values each AH should fall between 0.6 to 1.6 times that of the benchmark model. To validate the reasonableness of these factors, employ different factors lower and upper limits form varying search spaces. For each search space, we sample 10 000 network architectures and calculate a metric related to the architecture number parameter number falling within the interval [0.9cg, co], where co represents the expected upper limit of parameter number (see Section the technical appendix for the metric details). The results are presented in Figure 3 evident that when the lower and upper limits for the AHs are set to around 0.6 times and 1.6 times the corresponding values in the benchmark model, the metric tends to be the highest. This indicates that our choice of factors justified.
Test- ffst--erformance stability. It is well known that the test performance of the search results are subject to randomness, which may bias the evaluation. To quantify this variance, we conduct two additional training-testing runs for the reported V VM-MMMaa w/ BWE; the results are given in T Although minor fluctuations exist, opt- Mamba-2-1 2-100M, Opt-Mamba2 and the baseline opt- Pt-VWWT VW.V Mamba2 listed in Table Owing to time and compute constraints, we do not repeat the same experiments for every model; nevertheless, because the type of all candidates are the same, we expect them to exhibit similar level of test-variance.
consistently retains clear margin over
5 Conclusion
We propose the first training-free NAS method designed for the Mamba-type architectures. Based the rank collapse in stacked SSD layers, we develop proxy that requires computation of the transformation matrix between the input of the Mamba2 block and the input of the SSD block. Additionally, design search space where the ranges of the AHs flexibly vary according the expected computational overhead. We validate the effectiveness of our method from two perspectives, ranking correlation and the performance of search results. However, have only applied our method on Mamba2 so far. In the future, as many new Mamba-type models are proposed, we will validate and optimize our method on broader range of models.
10

---

Acknowledgments and Disclosure of Funding
This work is funded by the National Natural Science Foundation of China (Grant No. 62176119), and Jiangsu Graduate Research Innovation Program (Grant No. KYCX24_0262).
References
[1] Mohamed S Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Lane. Zero-cost proxies for lightweight nas. arXiv preprint arXi::2101.08133 2021.
[2] Sarwat Ali and M Arif Wani. Gradient-based neural architecture scarch: A comprehensive cvaluation. Machine Learning and Knowledge Extraction, 5(3):1 ((3):1176-111) 176-1194, 2023.
[3] Malyaban Bal and Abhronil Sengupta. Rethinking spiking neural networks state space models. arXiv e-prints, pages arXiv 2406, 2024.
[4] Ali Behrouz and Farnoosh Hashemi. Graph mamba: Towards learning on graphs with state space models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 19-130. 2024.
[5] Ali Behrouz, Michele Santacatterina, and Ramin Zabih. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint arXiv:2403.1988. 2024.
[6] Kai Biegun, Rares Dolga, Jake Cunningham, and David Barber. Rotrmn Modelling long sequences with rotations, arXiv preprint arXiv:2407.072333 2024.
[7] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432 7439, 2020.
[8] Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCal- lum, Maria Chang, Achille Fokoue-Nkoutche Pavan Kapanipathi, Nicholas Mattei, A systematic classification of knowledge, reasoning, and context within the are dataset. ACL 2018, page 60, 2018.
[9] Guo Chen, Yifci Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang. Kunchang Li, Tong Lu, and Limin Wang. Video mamba suite: State space model as versatile alternative for video understanding. arXiv preprint arXiv:2403. 240.00966.. 2024.
[10] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEF/VVF international conference on computer vision, pages 12270-12280, 2021.
[11] Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: thcoretically inspired perspective. In International Conference Learning Representations (ICLR), 2021.
[12] Xiangxiang Chu, Shun Lu, Xudong Li, and Bo Zhang. Mixpath: A unified approach for one-shot neural architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2023.
[13] Tri Dao and Albert Gu. Transformers are ssms: generalized models and efficient algorithms through structured state space duality. In Proceedings the 41st International Conference Machine Learning. pages 10041-10071, 2024.
[14] Deng, Yuanli Zhang, and Jingrui Zhang- The influence mamha model hyperparameters on training time and accuracy. In 2024 IEEF International Conference on Smart Internet of Things (SmartloT), pages 293-300. IEFF, 2024.
[15] Peijic Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Zimian Wei, Qiang Wang, and Xiaowen Chu. Parze: Parametric zero-cost proxies for efficient nas. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 16327-16335, 2025.
[16] Xuanyi Dong. Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench Benchmarking nas algorithms for architecture topology and size. IEEE transactions on pattern analysis and machine intelligence. 44(7):3634-366 2021.

---

[17] Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference Learning Representations.
[18] Yihe Dong, Jean- -ann-Batistee Cordonnier, and Andreas Loukas, Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference machine learning, pages 2793-2803. PMLR, 2021.
[19] Chao Fan, Hongyuan Yu, Yan Huang. Liang Wang, Zhenghan Yang. and Xibin Jia. Slicemamba with neural architecture search for medical image segmentation. IEEE Journal of Biomedical and Health Informatics. 2025.
[20] Yi Fan and Yu-Bin Yang. Training-free neural architectural scarch on transformer via evaluating expressivity and trainability, In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2024.
[21] Maryam Fazel. Matrix rank minimization with applications. PhD thesis, PhD thesis, Stanford University, 2002.
[22] Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jie, Yujie Zhong, and Lin Ma. Matten: Video generation with mamba-attention. arXiv preprint arXiv:2405.0302. 2024.
[23] Haifan Gong, Luoyao Kang, Yitao Wang, Yihan Wang. Xiang Wan, Xusheng Wu, and Haofeng Li. unmamba: 3d biomedical image segmentation, classification and landmark detection with state space model. In 2025 IEEE 22nd International Symposium on Biomedical Imaging (ISBI), pages 1-5. IEEE, 2025.
[24] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0077 0002222 2023.
[25] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
[26] Zhenfeng He, Yao Shu, Zhongxiang Dai, and Bryan Kian Hsiang Low. Robustifying and boosting training-free neural architecture search. arXiv preprint arXiv:2403.0759) 2024.
[27] Tao Huang, Xiaohuan Pei. Shan You, Fei Wang. Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. CoRR, 2024.
[28] Yi-Cheng Huang, Wui-Hua Li, Chih-Han Tsou, Jun-Cheng Chen, and Chu-Song Chen, Up-nas: Unified proxy for neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (675-1684 2024.
[29] Fady Ibrahim, Guangjun Liu, and Guanghui Wang. A survey mamba architecture for vision applications. arXiv preprint arXiv:2502.077 07 161, 2025.
[30] Yesmina Jaafra, Jean Lue Laurent, Aline Deruyver, and Mohamed Saber Naceur. Reinforcement learning for neural architecture search: review. Image and Vision Computing, 89:57-66, 2019.
[31] Tangyu Jiang, Haodi Wang, and Rongfang Bie. Meeo: zero-shot nas with one data and single forward pass via minimum cigenvalue of correlation. Advances in Neural Information Processing Systems, 36:61020- 61047, 2023.
[32] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. International conference on machine learning, pages 5156-5165. PMLR, 2020.
[33] Junghyup Lee and Bumsub Ham. Az-nas: Assembling zero-cost proxies for network architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5893-5903, 2024.
[34] Namhoon Lee, Thalaiyasingam Ajamthan, and Philip HS Torr Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810. 02340, 2018
[35] Guihong Li. Duc Hoang, Kartikeya Bhardwaj. Ming Lin, Zhangyang Wang. and Radu Marculescu. Zero- shot neural architecture search: Challenges, solutions. and opportunities. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[36] Guihong Li, Yuedong Yang, Kartikeya Bhardwaj, and Radu Marculescu. Zico: Zero-shot nas via inverse coefficient of variation gradients. arXiv preprint arXiv:2301.11300 2023.
12

---

[37] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos. Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Sawarrz, al. Jamba: A hybrid transformer-mambuu language model. arXiv preprint urXiv:2403.. 19887, 2024.
[38] Ming Lin, Pichao Wang. Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for perfforrromeee image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 347-356, 2021.
[39] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 1806009055 2018.
[40] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference Machine Learning, pages 7021-7032. PMLR, 2021.
[41] Xiao Liu, Chenxu Zhang, and Lei Zhang. Vision mamba: A comprchensive survey and taxonomy, arXiv preprint arXiv:2405.044000 2024.
[42] Yuqiao Liu, Yanan Sun. Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search. IEEE transactions on neural networks and learning systems. 34(2):500-570, 2021.
[43] Yash Mehta, Colin White, Arher Zela, Arjun Krishnakumar, Guri Zahergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, and Frank Hutter. Mas-bench-suite: Nas evaluation is (now) surprisingly easy. CoRR, 2022.
[44] Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International conference on machine learning, pages 7588-7598. PMLR, 2021.
[45] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391 2018.
[46] Denis Papemo, Germán Kruszewski, Angeliki Lazaridou, Ngoc-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Bolcda, and Raquel Fernández. The lambada datasct: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1525-1534, 2016.
[47] Daniel S Park. Jaehoon Lee, Daiyi Peng, Yuan Cao, and Jascha Sohl- bih--Dicsttein Towards ingp-guided neural architecture scarch. arXiv preprint arXiv:2011 200:2001.0006,, 2020.
[48] Badri N Patro and Vijay Srinivas Agneeswaran. Simba: Simplified mamba-base architecture for vision and multivariate time series. CoRR, 2024.
[49] Yameng Peng. Andy Song, Haytham M Fayek, Ciesielski, and Xiaojun Chang. Swap-nas: Sample-wise activation patterns for ultra-fast nas. In ICLR, 2024.
[50] Huafeng Qin, Yuming Fu, Jing Chen, Mounim El- Yacoubi, Xinbo Gao, and Feng Xi. Neural architecture search based global- vision mamba for palm-vein recognition. arXiv preprint MxXiv:2408.05743 2024.
[51] Huan Ren, Jiangtao Guo, Shuli Cheng, and Yongming Li. Pooling-based visual transformer with low complexity attention hashing for image retrieval. Expert Systems with Applications, 241:122745,, 2024.
[52] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po- Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 5444411334 2021.
[53] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99 106, 2021.
[54] Aaron Serianni and Jugal Kalita. Training-free neural architecture search for mns and transformers. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
[55] Yao Shu, Shaofeng Cai, Zhongxiang Dai, Beng Chin Oni, and Bryan Kian Hsiang Low. Nasi: Label-and data-agnostic neural architecture search at initialization. arXiv preprint 2rXiv:2109.008 2021.
[56] Yao Shu, Zhongxiang Dai, Zhaoxuan Wu, and Bryan Kian Hsiang Low. Unifying and boosting gradient- based training- free neural architecture search. Advances in neural information processing systems. 15:33001-33011 2022.
13

---

[57] Julien Siems, Lucas Zimmer. Arber Zela, Jovita Lukasik, Margret Keuper. and Frank Hutter. Nas- 301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.09771
4:14, 2020.
[58] Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems. 33:6377-6389, 2020.
[59] Yujin Tang, Peijie Dong, Zhenheng Tang, Xiaowen Chu, and Junwei Liang. Vmrmn: Integrating vision mamba and Istm for efficient and accurate spatiotemporal forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[60] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations.
[61] Junxiong Wang. Daniele Paliotta, Avner May. Alexander Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems, 37:62432- 62457, 2024.
[62] Zimian Wei, Peijie Dong, Zheng Hui, Anggeng Li, Lujun Li, Menglong Lu, Hengyue Pan, and Dongsheng Li. Auto-prox Training- free vision transformer architecture search via automatic proxy discovery. In Proceedings of AAAI Conference on Artificial Intelligence, volume 38, pages 15814 15822, 2024.
[63] Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural archi- tectures for neural architecture search. In Proceedings of the AAAl conference on artificial intelligence, volume 35, pages 10293-10301 10301, 2021.
[64] Qianxiong Xu, Xuanyi Liu, Lanyun Zhu, Guosheng Lin, Cheng Long, Ziyue Li, and Rui Zhao. Hybrid mamba for few-shot segmentation. Advances in Neural Information Processing Systems, 37:73858-73883, 2024.
[65] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench- 101: Towards reproducible neural architecture search. In International conference on machine learning, pages 7105-71 14, PMLR, 2019.
[66] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2019.
[67] Yuming Zhang, Jun Hsieh, Xin Li, Ming-Ching Chang, Chun-Chich Lee, and Kuo-Chin Fan. Mote- nas: Multi-ohjective training-based estimate for efficient neural architecture search. Advances Neural Information Processing Systems. 17:100845-100880 2024.
[68] Qinqin Zhou, Kekai Sheng, Xiawu Zheng. Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-free transformer architecture search. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10894-10903, 2022.
[69] Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Yonghong Tian, Chen, and Rongrong Ji. Training- free transformer architecture search with zero-cost proxy guided evolution. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
[70] Barret Zoph and Quoe V L... Neural architecture search with reinforcement learning. arXiv preprint 1.1i::6611.01700 2016.
[71] Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon mamba: The first competitive attention-free 7b language model. arXiv preprint arXiv:2400.0355,, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
We clearly state in the abstract and introduction that for the first time, we design a training-free neural architecture search method specifically for Mamba-type archi- tectures, and we have conducted validation on Mamba2. Our experiments demonstrate the effectiveness of our method.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section we state that we have only applied our method on Mamba2 so far. And in the future, as many new Mamba-type models are proposed, will validate and optimize our method on broader range of models.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We point out that the rank collapse phenomenon occurs in stacked SSD models in Subsection [3.]] and theoretical proof is provided in Subsection [. of the technical appendix. Besides, for some conclusions used in the proxy design, we also provide proofs in Section[Coo the technical appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: When constructing NASBench, provide the publicly available pre-trained models we use in Subsection 4.1 When conducting architecture search, we detail the evolutionary process and the architectures of the search results in Section Section[]andd Hf of the technical appendix, and the language datasets used are also publicly available. For the two metries used in Subsection 44.4] we provide detailed calculation process in Section and of the technical appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
16

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We consider releasing the NASBenches and code after the paper is published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
Sublic/guides CoodeSubmisssoooooiicc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https
cttppscccppullicgguu guwdes/ http:///odeeaaaassiooooooooo111.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the details of the training and test settings in Subsection 4. and 4.2 of the main body and of the technical appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
17

---

Justification: The Mamba2 model, especially the larger models with more parameters, incurs excessive training costs, making difficult repeat the search and training process multiple times. Additionally, when constructing NASBench, while fine-tuning each model is not significantly costly, completing the fine-tuning for all 500 models still poses a substantial expense. Therefore, it is challenging to conduct repeated experiments for both ranking correlation and architecture search. As an alternative, we construct 10 NASBenches on models of various scales and test the ranking correlation separately, yielding very favorable results. This can be considered an indirect validation of the stability of the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Subsection 4.2, we provide the computational resources and time costs for the search. For ranking correlation, we present the computational time for each proxy in Table For the search results, we list the model size, FLOPs, and inference latency Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp::/neurips. wttplppps..ppppppcppeccpep httppuuiiiietttttcccii.eeeeeeeed
Answer: [Yes]
Justification: We conduct in the paper conform with the NeurIPS Code of Ethics.
18

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We give brief discussion of the potential positive and negative societal impacts of our work in Section B of the technical appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release data models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
19

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The pretrained models and datasets we use are all publicly available, and we cite them properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssii thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We currently have not released any new assets. NASBench will be released as a new asset after the paper is published.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Related work
Significant progress has been made in the area of Mamba-related research recently. The Mamba model [24], novel and efficient sequence modeling framework based on SSM [25], has garnered widespread attention. Later, the introduction of Mamba2 [33 makes the performance improve by 2 to 8 times through enhancements in selective SSM. Mamba2 has demonstrated competitive performance in language modeling against Transformers and has optimized system performance by introducing tensor parallelism. Subsequently, Mamba-type model makes greater progress in vision and multimodal tasks, with excellent models such as Vision Mamba [ЛЛ, SiMBA [48). VMRNN [59] , and MambaMixer [5) Additionally, foundation models like Falcon Mamba 7B 711, .Jabaa [37, VideoMamba (9] have been proposed as attempts to replace Transformer models across various tasks. However, some studies indicate that there is not replacement relationship between Mamba and existing architectures; combining Mamba with current models like Transformer (22), convolutional network 23 graph network (4) recurrent network [6], spiking network (3] may lead to better results. These contributions have not only deepened theoretical understanding of Mamba but also propelled its development through system optimization and application expansion, revealing immense potential across multiple fields. However, many of these models involve numerous hyperparameters that dictate the actual architecture of the models and often require manual tuning.
NAS is initially introduced by [70 and has since led to various optimization strategy- based methods 3C 2 42
63 12] Considering that these methods still require extensive computational resources during the search process, some researchers propose training-free frree NAS, also known as zero-shot NAS, inspired by parameter prunning [34|588 Subsequently, grounded in neural network learning theory, multiple training-free NAS methods for convolution networks have been proposed by analyzing activation pattern 44 49 analyzing gradients (56). utilizing neural tangent kernel [55] [T. viewing
the performance prediction as a Gaussian process [47]. or employ alternative heuristic methods (31].
To achieve better performance, some studies [26 28 33 combine multiple proxies and employ
optimization strategies to switch among them dynamically. Unfortunately, the design of proxies in these methods necessitates consideration of the characteristics of network structures and training principles, making them unsuitable for the prevalent Transformer architecture. To address this, DSS 68. based on the theoretical insights from [8), pioneers NAS method for Transformers that measures the synaptic diversity in MSA modules and synaptic significance in Multi-Layer Perceptron (MLP) modules. Building on this, DSS++ [69 proposes a layer-wise proxy approach. Other researchers 55 20 have evaluated network performance using attention map features. In summary, current training-free NAS designs primarily focus on convolutional networks and Transformers, leaving other architectures, including Mamba, without suitable methods. From another perspective, the existing methods for optimizing Mamba-type models [50 27 [9] all utilize traditional NAS approaches, which involve significant computational overhead. Therefore, research on training- free NAS for Mamba-type models is highly urgent.
B Societal impacts
In the context of the rapid development of artificial intelligence and deep learning technologies, meth- ods for the automatic optimization of Mamba architecture have multidimensional social implications.
From a positive perspective, these methods are expected to significantly enhance the performance and efficiency of Mamba models, propelling deep learning applications in various fields such as natural language processing and computer vision to new heights. For example, in the field of medical image analysis, optimized Mamba architectures can more accurately and swiftly identify lesions, assisting doctors in diagnosis. This improvement can increase the accuracy and efficiency of discase diagnosis, benefiting larger number of patients. In autonomous driving scenarios, optimized architectures can more sensitively perceive the surrounding environment and make more precise decisions, thereby accelcrating the practical application of autonomous driving technologies. This advancement is of great significance for improving traffic safety and alleviating congestion, enhancing people's Sepees travel experiences, and promoting the development of intelligent transportation systems.
Moreover, owning to avoiding training, the method can significantly save computational resources and time costs. Training large deep learning models typically requires substantial computing power and time, while this approach cireumvents the iterative training process. This is particularly advanta- geous in scenarios involving large-scale hyperparameter searches and model architecture selection,
22

---

effectively enhancing resource utilization efficiency. Consequently, researchers and enterprises can allocate more resourees the practical application and further optimization of models, facilitating the rapid implementation of technology.
However, this method also has potential negative implications. Although it does require training, the process of selecting network architectures may still involve data processing and analysis. If data management is inadequately handled, there exists the risk of data privacy breaches, particularly when dealing with datasets that contain personal sensitive information. Such breaches could lead to the exposure of personal information, raising societal concerns regarding data security and privacy protection.
C Theoretical analysis and proofs
C.1 Rank collapse in stacked SSD layers
In this subsection, we proof the rank collapse in stacked SSD layers. Intuitively, based on the rank collapse of stacked attention layers [88] along with the duality of attention and SSD [I3], one could directly draw conclusions. However, it is important emphasize that the attention modules aligned with the forward process the SSD are not the original attention modules, but rather modifications made to them involving two key improvements: (a) masking operations on the attention map; and (b)
the use of linear attention operations [32]. This indicates that the forward computational process of the SSD does not entirely align with the attention modules for the analysis of rank collapse in [18). Consequently, to confirm that stacking SSD modules will indeed lead to rank collapse, we will use mathematical induction prove that after k operations, the distance between the matrix X5 and the rank-1 matrix Y satisfies
Xx Y Baa
(8)
When = 0, assume that the initial matrix Xo has a distance of 3 from Y", that is,
1 Y S 3,
(9)
and suppose that after k operations, the distance satisfies
| Y Ba
(10)
Considering the (A + 1)-th operation
X = (LO X W,
(11)
we analyze [K411 Y+|
I k+1 Y
[ xx) W - YY
L Xx) W - ab"
(L Xx) W- L (ab w + LO (ab'wt) ab" Tb
(12)
< L LO
X&W ab w+) + LO ab w ab
<a X ab" T
+7
Here, and 7 constants related matrices L and W, with 0 < < 1 and y being small constant.
By the inductive hypothesis and the above inequality,
conclude that
X Y a X Y"1 + agaa + B+h y
(13)
As k becomes large, a 1++1 approaches zero, the distance || Xx+1 The proof is now complete.
Y also approaches zero.
This conclusion is derived under the scenario of single head; however, based on the principles of path decomposition outlined in [8]. it is evident that our conclusion also holds true in the case of multiple heads.
23

---

C.2 Reasonableness of the Gaussianity and independence assumption for U
Our assumption of a normal distribution for input elements is based on the absence of prior knowledge about the inputs. This approach is also adopted by some training-free NAS methods for CNNs, such as [38], which generate normally distributed tensors to simulate unknown inputs. To validate this assumption, conducted experiments using network with width W 96 and depth D 3. selecting 1,000 samples and truncating them to meet the condition of token number T 96. We extract the inputs U of each Mamba2 block and use the Halmogorov--mminne test to confirm that these samples approximately follow a normal distribution. Additionally, we directly inspect the ranks of all 3,000 U matrices (1,000 samples with three Mamba2 block inputs each) and find them all to be full-rank matrices. These results demonstrate that our assumption does not lead to errors in practice.
C.3 Probability of full-rank matrix for Gaussian random matrix
In this subsection, we will prove that, for a matrix sampled from an independent standard normal distribution, the probability of it being full rank is 1.
First, consider the case where the matrix is square. When the matrix an x n square matrix, its determinant is continuous and non-constant function that maps the elements of the matrix to real numbers. The set of singular matrices (those with a determinant equal to zero) forms low-dimensional algebraic variety. According to the properties of absolutely continuous probability measures, this set has Lehesgue measure zero. Therefore, the probability that matrix with indepen- dently and identically distributed standard normal elements lies in this zero-measure set is zero. Thus, the probability that the matrix is of full rank is 1.
Next, we consider the case of non-square matrices. For non-square matrix A (assumed to be of size m n), if m nI, the rank of the matrix is most n. Conversely, if < n. the rank is at most m. However, for matrices whose elements are drawn from independent standard normal distributions, the row (or column) vectors are linearly independent with probability 1. When m = n, the probability that the matrix is of full rank is I; when m < n. the probability that the matrix has full row rank is also I; similarly, when m > n, the probability that the matrix has full column rank is 1.
Additionally, we can use mathematical induction for proof. Assume the probability that the first k row vectors are linearly independent is 1. When add the (I 1)-th row, the probability that this row falls within the k-dimensional subspace spanned by the previous & rows is zero. Given the independence of the standard normal distribution, the probability that the newly added row vector is linearly dependent on the original set of row vectors is zero. Therefore, by mathematical induction, we conclude that the probability of all row vectors being linearly independent is I, which implies that the probability of the matrix being full rank is also 1.
C.4 Probability of inclusion in the column space of row-full rank matrices
Let A be an m X n matrix satisfying m < n and having full row rank (i.e., its rank is m). In this section, will verify that the probability of random vector R", sampled from independent standard normal distribution, lying within the column space of A is 1.
Since A has full row rank (rank-m), its row space has dimension m, and consequently, its column space also has dimension m. We denote the column space of A as Col (A). Given that A is an m x n matrix, the column space Col (A) is subspace of gT Moreover, full row rank matrix will have a column space dimension of m. which matches the dimension IR"", Therefore, we have Col (A) RT The random vector is independently sampled from standard normal distribution, which means its distribution covers the entire RTM space, except for a set of measure zero. In RTT. the measure of Lingle-poin sets (such as the origin) or dower-dimensional subspaces is zero. Since Col (A) = R"" the probability that x lies in Col (A) is therefore 1. Thus, we have proven this conclusion.
C.5 Probability of inclusion in the column space of column-full rank matrices
Let A be an m X matrix satisfying m > and having full column rank (rank-n). In this subsection, we will verify that the probability of random vector 2: R sampled from an independent standard normal distribution, lying within the column space of A is O.
24

---

The proof follows a similar structure to that of the previous subsection. Since the matrix A is full rank m X matrix with m > n, it has a rank of Consequently, the column space of A is an n-dimensional subspace. The vector is random vector that is independently sampled from a standard normal distribution, which means its distribution spans the entirety of TR. In TR' any subspace of dimension less than m (such as the column space of A, which has dimension < m) has a Lebesgue measure of zero. Since the distribution of æ is absolutely continuous with respect to Lebesgue measure, the probability that lies in a set of measure zero is also zero. Thus, we have proven the conclusion.
C.6 Error bounds for pseudoinverse approximations
For Case 3, we analyze the upper bound of |UW x X|I and derived the conclusion |UU x X|| < | (the proof is provided below). This implies that as long the Frobe- nius norm of X is not excessively large, the resulting error is small. Similar conclusions apply to B and C. When calculating the proxy, we can preprocess the input data (e.g., multiplying by small constant) to keep X, B, and C' at small values, thereby controlling the error introduced by this approximation.
The remainder of this subsection is devoted to proving the conclusion stated in the preceding paragraph. The least squares solution W x corresponds to the orthogonal projection of X onto the column space of U. The residual matrix R = UWx x - X lies in the orthogonal complement of U. Since orthogonal projection does not increase the norm, the projection of X onto the orthogonal complement (i.e., the Frobenius norm of the residual matrix R) must not exceed the Frobenius norm of the original matrix X:
R F (I- (I-vVT)
xx| S ||
(14)
where I is the identity matrix, and V is the matrix of left singular vectors of U.
If X is entirely within the orthogonal complement of U, then UWx = and |UWx X|| S | || p. reaching the upper bound. If X is entirely within the column space of U, the residual is zero, and the upper bound holds.
In summary, the original incquality is established, and the proof is complete.
C.7 Time-complexity derivation for our proxy
In this subsection, we begin with deriving the time complexity of the Mamba2 proxy for given W, N, H, and P.
Firstly, the complexity of SVD is O (TW min (T, W)), and that of computing the pseudo-inverse is O (min (T, W)). Calculating U has a time complexity of O (WT2). while computing W x. WB, B. We and their gradients has complexities of O (WTP), (WTN), and (WTN) respec- tively. The nuclear norm computation for these matrices has complexities of O (WP min (WP)), O OWN min (WN)), and O (WN min (WN)). Additionally, computing the nuclear norm of W out and its gradient is O (W P min (WP)). Here, T is input-dependent and can be treated as constant. Finally, considering multiple heads, the time complexity for one Mamba2 block is O (WHP min (WP) + WHN min (WN)). So is evident that when any of W, N. H, P approaches infinity, the time complexity grows linearly.
It should be noted that VWSSMamba2 merely expands the search space of SSMamba2 without increasing the scale of candidate networks. The scale of candidate networks determined by the expected computational overhead. Thus, the time overhead does not increase significantly.
C.8 Proof that additive noise tends to decreases Kendall's Tau
In this subsection, we prove that adding noise to performance data tends to decrease the Kendall's Tau between proxy and performance.
Assume performance data is (I1 "2, kn) and proxy data is (V1, ".. ...). The Kendall's Tau coefficient is defined as
C D
(15)
n (n 1)
25

---

where C' is the number of concordant pairs and D is the number of discordant pairs.
When noise . (0, o²) is added each I;, forming oंद = xi + 6j, the comparison between any two samples and j changes. The difference G - follows / (0, 20 ) For a pair (i.j) where 3; > zj. the probability that x > x'y is (=d) V20
where = I; Tjj. For a pair where 24 < Zjj
the probability that 2/ < o is +
where = 2j Iz--
Noise can turn concordant pairs into discordant pairs and vice versa. The expected number of concordant pairs decreases, while the expected number of discordant pairs increases. Thus, the expected new Kendall's Tau r' after adding noise is:
E C'I D
E
<T.
(16)
ア (m 1)
This shows that adding normal noise reduces the expected Kendall's Tau value. The proof is complete.
D Pseudocode of TF-MAS
This section provides the pseudocode of proxy computation process in TF-MAS, listed in Algorithm
Algorithm Proxy of TF-MAS
Input: Network architecture A, input D
Output: Proxy value TF-MAS
W ( ((D Initializa and get W out.
2: U(1), (22) ( (1) x2)
(2) (D)
B
B 2
DD) C(L) C(22 (
Forward propagation and get X, B.C.
@B 3: ax ax ... ax
oc ac
ac ac ac
dc
ac
ac a0
ac
ac
ac
0 ag0)
ac
aw on
aw (2)
dc
+ Back propagation and get gradients of X. B, C. W out
ow.l"
4: (class), class2, -.. class +- Classify by column number X (1) X ..., ..) c(.) (()) C(D))
(D) B()) (())
5: for class in (class class2, ... class do
6: index list Get indexcclass,,,)
7: U list + Get elements(index list).
8: UB. XBCB < Concatenate(U list), Concatenate(Class,,)
/concatenate form the batch dimension. dim (U B), dim (U ()2)2. dim (UB), are the batch size, row number, and column number of U p. respectively.
9:
if dim (U B)2 dim (U B)3 then
10:
W Bm U XBC
11:
else
12:
VI 1. V2 SVD(U 8)-
13:
a < Pseudoinverse(f))
14:
W Bm + V20 V XBCB-
15:
end if
16: end for
17:
(!l)
x W ( (2)
....
W(D) W
(D)
W W
W
(D)
w
(I)
W
C
W(D)
Rearrange by
(class), class OC
class (W W
B2....
BM).
18:
Oc
Oc
ac aw
8c
8C
ac
ow
(.
Bw
aw
OW
UUST ac
U2) oc
D ac
U
Oc
U [ ac
U DJT
Oc 8B
ux
8X
2B ...
(()T De
U2)T dc .
DDT Oc
a
19: TF-MAS + Σ EzE(X.B.Cou))
W luce
JW.
20: return TF-MAS
26

---

Mamba2 coll
mppt
embedding
Mumba2
Mamba2
Mamba22
Im head
output
Mamba2 block
scaling
X
PC
convld
SiLUI
inpot
PC
convid
SILU
SSD
morm (10)
PC
output
(11)
PC
(5) dt
bins
Figure 4: Structure of Mamba2 and the positions of parameters. The parameters are indicated by red numbers enclosed in parentheses. Compared to Figure I this figure provides additional details to illustrate all parameters within the model.
E
Calculation of parameter number in Mamba2
In Subsection hyperparameters V, D, W, N, and H. Here, we present the derivation process for this expression.
provide the expression for the parameter number of Mamba2 given the
Figure 2 illustrates all the parameters contained within the Mamba2 model. The corresponding sizes are as follows:
Embedding: This component includes weight with size of V W, resulting in parameter number of VW.
(2) Normalization after all Mamba2 cells: This component includes a weight with size (and thus a parameter number) of W.
(3) Language model head: This component includes a weight with a size x W, resulting in parameter number of VW.
Normalization in each Mamba2 cell: This component includes a weight with a size (and thus a parameter number) of W.
(5) Fully connected layer in each Mamba2 block: This component includes weight with- out bias. The output from this component serves as the data source for X, B, C, dt, and Z, which have feature dimensions of 64H, N, N, H, and 64H, respectively, while input dimension is W. Therefore, the parameter number for this weight (64H + N + N + H + 64H) W = 2WN + 129WH.
(6) Convolution layer in each Mamba2 block: This component includes one-dimensional
convolution with a kernel size of 4, which includes a weight and bias. As a precursor for generating X, B, and C, the corresponding feature dimension is 64H + N N 2N 64H. Consequently, the parameter number for the weight is (2N + 64H) x 4 = 8N + 256H, and for the bias, it is 2N 64H.
(7) dt bias in each Mamba2 block: This is an independent parameter with a size (and thus parameter number) of H.
(8) A in each Mamba2 block: This is independent parameter with size (and thus parameter number) of H.
(9) Scaling factor in each Mamba2 block: This is an independent parameter with a size (and thus a parameter number) of H.
(10) Normalization in each Mamba2 block: This component includes a weight with size (and thus a parameter number) of 64H.
27

---

Table 7: Ranking correlation on SSMamba2Bench, 130M. The abbreviations are consistent with Best results in bold.
Table
LAMBADA LAMBADA
PIQA
Arc-E
Are-C
WG
OBQA
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
#Param
general
-0.411
0.424
0.248
0.279
0.233
0.281
0.330
0,381
Gradnorm
-0,083
0.041
0.075
0.027
0.005
-0.023
0,040
0.007
Synflow
-0.029
0.042
0.074
0.068
0.001
0.080
-0.016
0.092
GraSP
-0.020
0.051
-0.001
-0.017
-0.004
0.019
0.043
0.059
Fisher
-0.074
0.079
0.038
0.031
0.006
0.044
0.061
0.026
Snip
-0.077
0.045
0.009
0.043
0.011
0.037
0.044
-0.037
Zen-NAS
C
-0.113
0,056
0.069
0.054
0.053
0.035
0.042
0.009
ZiCo
C
-0.065
0.044
0.052
0.096
0.089
0.003
0.047
0.071
McCo
-0.065
0.052
0.007
0.031
0.061
0.088
0.038
0.050
Auto-Prox
-0.013
0.069
0.034
-0.024
-0.013
0.007
0.108
0.034
AC
T
-0,086
0.132
-0,001
0.096
0.035
0.120
0.149
0.081
HI
T
-0.107
0.134
0.063
0.102
0.140
0.070
0.096
0.120
HC
-0.068
0.089
0.106
0.089
0.052
0.124
0.087
0.120
DSS
-0.143
0.099
0.074
0.150
0.109
0.094
0.106
0.157
AttaNAS
-0.230
0.294
0.121
0.168
0.167
0.181
0.232
0.193
TF-MAS (ours) M
-0.677
0.648
0.406
0.451
0.364
0.435
0.517
0.508
(II) Fully connected layer in each Mamba2 block: This component includes a weight without bias. The output dimension the tensor changes from 64H to W; therefore, the parameter number for the weights is 64WH.
The total number of parameters for Mamba2 is the sum of the parameter number from the above components. Notably, components (4)-(11) appear once in each layer, necessitating multiplication by the depth when calculating them. Ultimately, we derive the total parameter number as VW
W + VW+ (W 2WN + 129WH + 8N + 256H + 2N 64H H + H + H 64H 64WH) D W + 2VW + DW + 10DN + 387DH + 2DWN + 193DWH.
For the VWManba2 model, the AHs N and H are longer consistent across layers. Therefore, denote them as N; and H, for the i-th layer, leading to the total parameter number = WW+W+VW-
Σ (W + 2WN, + 129WH, + 8N, - 256H; + 2N, + 64H, H, + H₃ + H₃ + 64H; + 64WH,) = W 2VW + DW Σi (1ON; 387H; 2WN, 193WH;).
F More results about ranking correlation
In this section, we present the ranking correlations for all NASBenches datasets except for VMSSMamba2Bench p.7renchh_.BB.... 2.7B. Tables to correspond the NASBenches related to Mamba2 (i.e., SSMMabbaBennc 130M, SSMamba2Bench__9 330M,, SSMamba2Bench_ _700, and SS- Mamba2Bench 1.3B), while Tables 12 correspond the NASBenches related VWMamba2 (i.e.. VWSSMamba2Bench 1a0-aeeeench__MMMM__2211 V WWSSMamba2Bench 370M, VWSSMamba2Bench _70M, VWSSMamba2Bench _.33B and WWSSMamba2Benchh. 2.7B). can be observed that, regardless of the NASBench, the conclusions drawn are consistent with those presented in Table Therefore, our method achieves strong performance across all NASBench datasets.
Besides, in Subsection[4.1 we point out that ranking correlation evaluated only on the top-k candidates sometimes is necessary. Therefore, also conduct experiments on VMSSSaama2Bennhh 2.7B to evaluate the ranking correlation for the top 20%, top 10%, and top 5% of candidates. The results are presented in Table [6] These results indicate that the ranking correlation for the top-k candidates is comparable to that of the entire candidate set.
G Details of the search process
In Subsection we set the expected computational overhead to "the number of parameters not exceeding 130M". Under this condition, the feasible values for each AH are follows:
28

---

Table 8: Ranking correlation on SSMamba2Bench, 370M. The abbreviations are consistent with
Table I
Best results in bold.
LAMBADA LAMBADA
PIQA
Arc-E
Are-C
WG
OBQA
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
#Param
general
-0.414
0.441
0.233
0.289
0.235
0.320
0.363
0,363
Gradnorm
0,003
0.071
0.058
0,057
0.009
0.051
0.054
0.098
Synflow
-0.035
0.064
0.066
0.044
0.047
0.027
0.074
0.055
GraSP
-0.021
+0.007
0.028
0.047
0.023
0.052
-0.000
0.049
Fisher
-0.065
0.089
0.048
0.094
0.084
-0.030
0.063
0.044
Snip
-0.084
0.028
-0.014
0.033
0.051
0.025
0.028
-0.003
Zen-NAS
C
-0.130
0.084
0.003
0.035
-0.007
-0.014
0.023
0.035
ZiCo
C
-0.055
0.084
0.014
0.019
0.045
0.076
0.059
-0.003
McCo
-0.058
0.058
0.035
0.029
0.052
-0.004
0.052
0.044
Auto-Prox
-0.067
0.024
0.057
0.006
-0,002
0.015
0.041
0.077
AC
T
-0.090
0,124
0.118
0.102
0.068
0.050
0.082
0.044
HI
T
-0.091
0.132
0.064
0.088
0.078
0.054
0.123
0.115
HC
-0.149
0.107
0.038
0.061
0.053
0.058
0.084
0.088
DSS
-0.150
0.146
0.084
0.063
0.069
0.056
0.112
0.171
AttaNAS
-0.248
0.289
0.142
0.214
0.129
0.177
0.233
0.216
TF-MAS (ours) M
-0.666
0.685
0.377
0.419
0.388
0.467
0,472
0.560
Table 9; Ranking correlation on S5Mamba2Bench__750 The abbreviations are consistent with
Table I Best results in bold.
LAMBADA
LAMBADA
HS
PIQA (ACC)
Are-E
Are-C WG
OBQA
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
#Param
general
-0.429
0.439
0.228
0.328
0.260
0.311
0.368
0.359
Gradnorm
-0.016
0.060
0.055
0.036
0.009
0.080
0.077
0.043
Synflow
-0.089
-0.022
-0.030
0.020
0.052
0.037
0.052
0.050
GraSP
-0.067
0.040
0.034
0.057
0.023
0.045
0.039
0.045
Fisher
-0.043
0.069
0.037
0.050
0.038
0.045
0.024
0.128
Snip
C
-0.015
0,052
0.045
-0.010
0.012
0.027
-0.031
0,085
Zen-NAS
C
-0.052
0.125
0.054
0.003
0.070
0.050
-0.001
0.047
ZiCo
-0.032
0.015
0.029
0.109
0.000
0.008
0.032
0.013
McCo
-0.047
0.081
0.048
0.049
0.049
0.040
0.072
0.089
Auto-Prox
C
-0.050
0.023
0.041
0.013
0.050
-0.018
-0.001
0.006
AC
T
-0.112
0,031
0.081
0.077
0.054
0.097
0.103
0.126
HI
-0.093
0.078
0.087
0.061
0.093
0.126
0.106
0.096
-0.106
0.067
0.078
0.111
0.041
0.024
0.080
0.091
DSS
-0.105
0.117
0.104
0.098
0.109
0.095
0.118
0.093
AttaNAS
T
-0.298
0.240
0.181
0.131
0.108
0.222
0.193
0.208
TF-MAS (ours)
M
-0.655
0.684
0.325
0.442
0.396
0.424
0.501
0.520
D: 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24 (13 values)
W: 384, 416, 448, 480, 512, 544, 576, 608, 640, 672, 704, 736, 768 (13 values)
N: 64, 72, 80, 88, 96, 104, 112, 120, 128 (9 values)
H: 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24 (13 values)
For SSMamba2, the size of the search space is given by 13 13 x 13 19, 773. For VWSS-
24
Mamba2, the size of the search space is approximately
Σ 13 x 9
13 5.68 1050
1=12
During the search for opt Mamba2 and opt VWMamba2, our search method employs an evolutionary approach with a population size of 50, iterating for 300 generations. In each iteration, there are three methods for generating new individuals: (1) randomly inheriting from an original individual; (2) randomly selecting original individual for mutation. During mutation, one AH is randomly chosen; the AH an integer, is changed neighboring selectable value, while if the AH is list (only exists in the case where the search space is SSVWMamba2), 1/5 of its elements randomly selected

---

Table 10: Ranking correlation on SSMamba2Bench, 1.3B. The abbreviations are consistent with Table T
Best results in bold.
LAMBADA
LAMBADA
PIQA
Arc-E
Are-C
OBQA (ACC)
WG
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
#Param
general
-0.426
0,414
0.238
0.315
0.215
0.323
0.331
0.404
Gradnorm
-0,082
0.102
0.046
0,028
0.048
0.052
0.043
0.073
Synflow
-0.006
0.035
0.037
0.022
0.032
0.019
0.026
0.051
GraSP
-0.024
0.071
0.007
0.049
0.021
0.071
0.075
0.007
Fisher
-0.085
0.008
0.035
0.017
0.074
0.027
0.041
0.019
Snip
-0.023
0.047
0.079
0.053
0.017
0.036
0.034
0.083
Zen-NAS
C
-0.112
0.003
-0.017
0.073
0.015
0.026
0.058
-0.007
ZiCo
C
-0.042
0.048
0.026
-0.037
0.080
0.024
0.034
0.051
McCo
-0.023
0.083
0.042
0.052
0.016
-0,003
0.011
-0.005
Auto-Prox
-0.067
0.059
0.007
0.028
0.010
0.037
0.054
0.050
AC
T
-0.099
0.162
0.089
0.087
0.103
0.099
0.083
0.135
HI
T
-0.076
0.133
0.010
0.093
0.033
0.049
0.116
0.086
HC
-0.108
0,079
0.040
0.061
0.008
0.036
0.076
0.068
DSS
-0.152
0.140
0.094
0.082
0.119
0.111
0.102
0.111
AttaNAS
-0.202
0.273
0.130
0.163
0.166
0.197
0.249
0.228
TF-MAS (ours) M
-0.667
0.654
0.316
0.467
0.379
0.440
0.513
0.544
Table 11: Ranking correlation on SSMamba2Bench_ The abbreviations are consistent with
Table T Best results in bold.
LAMBADA
LAMBADA
HS
PIQA
Arc-E
Are-C
OBQA (ACC)
WG
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
#Param
general
-0.449
0.456
0.326
0.329
0.232
0.308
0.285
0.356
Gradnorm
-0.035
0.063
0.070
0.032
0.069
0.074
0.034
0.031
Synflow
-0.032
0.076
0.048
0.005
-0.002
0.054
0.036
0.074
GraSP
-0.043
0.047
0.013
0.023
-0.011
0.041
0.088
0.013
Fisher
-0.046
0.082
0.047
0.057
0.034
0.033
0.031
0.092
Snip
C
-0.037
0.023
0.066
0.031
-0.005
0.059
0.027
0.105
Zen-NAS
C
-0.067
0.080
0.091
0.059
0.064
0.038
0.054
0.003
ZiCo
-0.071
0.001
0.058
0.037
0.059
0.008
0.017
0.018
McCo
-0.087
0.051
0.010
0.039
0.100
0.091
0.025
0.039
Auto-Prox
C
-0.061
0.041
0.010
0.058
0.047
0.049
0.007
0.005
AC
T
-0.075
0.153
0.070
0.053
0.078
0.094
0.083
0.102
HI
-0.071
0.088
0.100
0.120
0.041
0.102
0.071
0.076
-0.124
0.087
0.101
0.030
0.110
0.051
0.095
0.113
DSS
-0.073
0.154
0.086
0.109
0.073
0.122
0.088
0.111
AttaNAS
-0.237
0.257
0.140
0.168
0.124
0.179
0.245
0.215
TF-MAS (ours) M
-0.656
0.663
0.408
0.446
0.385
0.429
0.521
0.510
and changed to neighboring selectable values; (3) randomly selecting two original individuals for crossover, where two AHs of the new individual are randomly taken from one original individual. and the remaining two AHs are taken from the other original individual. If the parameter number of new individual exceeds 130M, new individual generated until the parameter number is below 130M.
During the search for opt VWMamba2 w/ bwe using block-wise evolution, we calculate the proxy values for each layer separately, meaning that the outermost summation operation is not performed when applying Equation (6). Once we obtain these proxies, during the mutation process, if list-type AH is selected for mutation, the elements that need to be changed are no longer chosen randomly but are designated as the elements corresponding to the top layers with the minimum proxy values, where a is the number of elements to be changed. In the crossover process, the values of and are no longer wholly inherited from one original individual. For each element value of the new individual, which original individual it inherits from depends which original individual has greater proxy value in the corresponding layer. Thus, the values and H in the new individual are hybrids from the two original individuals. To balance the contributions of both original individuals.
30

---

Table 12: Ranking correlation on VWSSMamba2Bench 130M. The abbreviations are consistent with Table T
Best results in bold.
LAMBADA
LAMBADA
PIQA
Arc-E (ACC)
Are-C
WG
OBQA
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
#Param
general
-0.455
0.437
0.215
0.308
0.242
0.334
0.317
0.336
Gradnorm
-0.021
0.037
0.034
0.073
-0.005
0.027
0.012
0.022
Synflow
-0.019
0.048
-0.008
0.038
0.059
0.016
0.034
0.051
GraSP
-0.004
0.003
0.050
0.075
0.075
0.066
0.049
0.064
Fisher
-0.050
0.031
0.033
0.067
0.043
0.022
0.054
0.096
Snip
-0.069
0.060
0.062
0.014
0.060
0.066
0.060
0.011
Zen-NAS
C
-0,061
0.027
0.044
0.034
0.052
0.069
0.040
0.093
ZiCo
C
-0.100
0.080
0.032
0.001
0.038
0.049
0.043
0.056
McCo
-0.058
0.042
0.011
0.115
0.068
0.009
0.045
0.075
Auto-Prox
-0.027
0.061
0.019
-0.036
0.008
0.011
0.041
0.059
AC
T
-0.060
0.153
0.111
0.095
0.094
0.078
0.096
0.067
HI
T
-0.059
0.099
0.047
0.109
0.039
0.067
0.051
0.109
HC
-0.022
0.088
0.073
0.044
0.061
0.041
0.055
0.078
DSS
-0.166
0.125
0.139
0.114
0.112
0.128
0.104
0.159
AttaNAS
-0.253
0.225
0.097
0.156
0.151
0,204
0.191
0.249
TF-MAS (ours) M
-0.675
0.667
0.357
0.450
0.329
0.467
0,485
0.556
Table 13:
Ranking correlation VWSSMamba2BBnchh_55 The abbreviations are consistent with
Table 1
Best results in bold.
LAMBADA LAMBADA
HS
PIQA (ACC)
Are-E
Are-C (ACC)
WG
OBQA
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
#Param
general
-0.384
0,429
0.209
0.314
0.277
0.308
0.316
0.372
Gradnorm
-0.044
-0.008
0.008
0.066
0.039
0.040
0.037
0.092
Synflow
C
0.005
0.017
0.072
0.005
0.073
0.090
0.051
0.050
GraSP
C
-0.055
0.059
0.087
0.036
0.007
0.028
0.022
0.037
Fisher
-0.028
0.069
0.029
0.032
-0.014
0.083
0.062
0.023
Snip
-0.032
0.006
0.078
0.077
-0.008
0.075
0.014
0.044
Zen-NAS
C
-0.114
0.035
0.087
0.054
0.062
0.050
0.029
0.106
ZiCo
C
-0.073
0.041
0.064
0.006
0.023
0.075
0.012
0.013
MeCo
-0.074
0.076
0.045
-0.011
0.069
0.007
0.006
0.043
Auto-Prox
-0.040
0.027
-0.013
0.030
-0.011
0.014
0.036
0.036
AC
-0.081
0.080
0.082
0.100
0.092
0.100
0.094
0.152
HI
T
-0.076
0.038
0.026
0.061
0.089
0.076
0.046
0.123
HC
T
-0.112
0.157
0.040
0.035
0.063
0.036
0.107
0.081
DSS
-0.127
0.114
0.067
0.088
0,072
0.106
0.152
0.094
AttaNAS
-0.238
0.259
0.097
0.169
0.143
0,236
0.185
0.270
TF-MAS (ours) M
-0.660
0.672
0.378
0.444
0.359
0.486
0.495
0.547
for the remaining two integer-valued AHs, D and W, one original individual is randomly selected to inherit the value of D, while the other inherits the value of W.
H
Search results
In this section, we present the architectures of the search results from Subsection 4.2 Optimized Mamba2:
D: 43
W: 608
N: 104
H: 18
Optimized VWMamba2:
31

---

Table 14: Ranking correlation on VWSSMamba2Bench 280M. The abbreviations are consistent with
Table
Best results in bold.
LAMBADA
LAMBADA
PIQA
Arc-E
Are-C
WG
OBQA
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
#Param
general
-0.435
0.433
0.316
0.260
0.258
0.298
0.292
0.351
Gradnorm
-0,084
0.106
-0.040
0.026
0.019
0.053
0.055
0.080
Synflow
-0.037
0.006
-0.041
0.056
0.101
0.056
0.021
0.109
GraSP
-0.109
0.079
0.032
0.029
0.077
-0.013
0.041
0.097
Fisher
-0.053
0.024
-0.000
0.063
0.056
0.089
0.058
-0.015
Snip
-0.069
-0,010
0.022
0.065
0.043
0.037
0.040
0.097
Zen-NAS
C
-0.018
0.047
0.043
-0.005
0.070
0.046
-0.017
0.060
ZiCo
C
0.009
4.025
-0.013
0.036
0.023
0.072
0.039
0.081
McCo
-0.040
0.077
0.050
0.053
0.060
0.036
-0.025
0.061
Auto-Prox
-0.038
0.021
-0.011
0.048
0.031
-0.034
0.010
0.042
AC
T
-0.138
0.136
0.050
0.128
0.085
0.091
0.096
0.126
HI
T
-0.109
0.086
0.063
0.136
0.050
0.072
0.050
0.064
HC
-0.065
0,059
0.071
0.081
0.041
0.077
0.130
0.099
DSS
-0.100
0.111
0.143
0.081
0.094
0.104
0.075
0.108
AttaÑAS
-0.283
0.249
0.161
0.228
0.141
0.198
0.253
0.204
TF-MAS (ours) M
-0.649
0.698
0.365
0.430
0.364
0,451
0,470
0.557
Table 15:
Ranking correlation on VWSSMamba2Bench 1.3B. The abbreviations are consistent with
Table T
Best results bold.
LAMBADA
LAMBADA
HS
PIQA (ACC)
Arc-E
Arc-C
WG
OBQA
Proxy
Target
(PPL)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
(ACC)
#Param
general
-0.390
0.443
0.309
0.331
0.243
0.331
0.344
0.350
Gradnorm
-0.051
0.063
0.019
0.067
0,036
0.042
0.048
0.067
Synflow
-0.002
0.028
0.030
0.080
0.041
0.030
-0.012
-0.026
GraSP
C
-0.062
0.002
0.018
0.023
-0.007
0.021
0.022
-0.002
Fisher
C
-0.059
0.087
-0.007
0.054
-0.020
0.040
0.038
-0.024
Snip
C
-0.061
+0.012
0.019
-0.018
0.063
0.088
0.002
0.091
Zen-NAS
-0.122
0.030
-0.011
0.010
0.048
0.010
0.037
ZiCo
-0.035
0.052
-0.024
0.042
0.036
0.055
0.042
0.038
McCo
C
-0.037
0.047
0.019
0.050
-0.010
0.019
0.033
0.047
Auto-Prox
C
-0.106
0.017
0.050
-0.001
0.062
-0.012
0.062
0.032
-0.100
0.122
0.099
0.071
0.127
0.114
0.054
0.125
HI
-0.100
0,046
0.090
0.064
0.034
0.081
0.076
0.108
HC
-0.097
0.133
0.071
0.073
0.029
0.045
0.045
0.082
DSS
T
-0.148
0.162
0.078
0.083
0.079
0.098
0.110
0.139
AttaNAS
-0.249
0.247
0.122
0.160
0.177
0.140
0.261
0.206
MAS (ours)
M
-0.681
0.697
0.324
0.464
0.377
0.480
0.489
0.527
Table 16: Ranking correlation on VMSSMamba2Bench 2.7B using varying numbers of candidates.
The abbreviations in datasets are consistent with Table [
LAMBADA LAMBADA
PIQA
Are-E (ACC)
Are-C (ACC)
WG (ACC)
OBQA (ACC)
Candidates
(PPL.)
(ACC)
(ACC)
(ACC)
all
-0.685
0.661
0.381
0.468
0,339
0.452
0.483
0.519
top 20%
-0.693
0.683
0.390
0.483
0.347
0.470
0.486
0.527
top 10%
-0.675
0.653
0.374
0.453
0.319
0.455
0.468
0.500
top 5%
-0.662
0.692
0.366
0.421
0.332
0.439
0.489
0.522
32

---

D: 35
W: 544
N: [104, 96, 112. 72, 72, 64, 80, 64, 96, 80, 80, 120, 88, 80, 96, 64, 104, 72, 104, 64, 80, 88,
104, 104, 80, 112, 64, 112, 72, 120, 104, 120, 112, 112, 88]
H: [22, 21, 22, 28, 22, 21, 31, 19, 29, 20, 26, 21, 24, 23, 28, 16, 25, 18, 30, 22, 31, 21, 20, 27, 25, 24, 26, 23, 21, 18, 21, 25, 29]
Optimized VWMamba2 with block wise evolution:
D: 27
W: 672
. N: [88, 104, 72. 72, 72, 72, 96, 88, 104, 104, 112, 88, 88, 120, 72, 120, 72, 80, 96, 72, 104, 64, 120, 112, 80, 88, 120]
H: [31, 25, 31, 29, 18, 18, 20, 25, 30, 24, 17, 17, 29, 25, 17, 19, 25, 30, 25, 19, 30, 29,
19, 25, 21, 28]
Details of rank collapse metric
we introduce metric to evaluate the extent of rank collapse. This subsection will detail the specifies of this metric. For matrix under evaluation, the motivation of the metric is to measure its distance to the nearest rank-1 I matrix, defined as argmin || 1x I1,,. However, this
In Subsection 4.
æ metric may influenced by the size of and the scale of its elements. address this issue, rank collapse metric is the normalized distance:
argmin X
1x 11,00
metric argmin X
(17)
1,00
J Details of AH ranges evaluation metric
In Subsection we introduce metric to evaluate the range of AHs within the search space. This section will elaborate on the details of this metric. Intuitively, an ideal AH range needs to ensure that as many architectures as possible with their parameter numbers fall within the interval (0.9cp,co) while also maintaining diversity among these architectures by maximizing their pairwise distances. Therefore, our metric first selects architectures whose parameter numbers lie within the interval [0.9co, co] and then considers the AHs as features. We begin by identifying the center of these architectures, followed by calculating the sum distances from each architecture to the center. The magnitude of this sum reflects the rationality of the AH range. To ensure that the metric is not influenced by the sampling count, we normalize this value by the number of samples, yielding the final metric value.
Specifically, for SSMamba2, let m represent the number of architectures whose parameter numbers fall within the interval [0.9cp, co). Let the AHs for the i-th architecture be denoted as D(f), W N(), and H() To ensure consistency in feature scaling, we first normalize these parameters follows:
D
W(( NO
N( No
H(
D = Do
W
A = Ho
(18)
Wo
where D61, WO), NO), and H0 are the normalized features, and Do. Wo. No, and Ho are the AHs of the benchmark model. We then calculate the center values for each feature:
m
m
D D= M (
W
WO
Ñ
NO
H
Σ
AO.
(19)
Based on this, we can compute the metric as follows:
m
(bw (010-D) + (wro) - w)² + (NIO N)² + (A(O) A)
metric
Σ
(20)
m 1=1
33

---

For the SSVWMamba2 search space, the overall calculation process is similar to that of SSMamba2; however, there a critical difference: the values of and h are lists rather than integers. Consequently, the terms (( N) and (A()) - -H) in Equation (20 require adjustment. Considering that the values of N and in SSVWMamba2 are lists, we denote the normalized representations of the i-th architecture as Ñ and A (f) (bold letters). Notably, the dimensions of N (6) and H (5) may not be may not hold. (8) and H to unify their lengths to
the same across architectures, i.e., dim N (5) # dim and dim H of dim H
(i)
Therefore, we consider performing linear interpolation on
(t) D Let the interpolated values be N and H we then replace the terms (ÑO) -N) N) and (A() - H)
in Equation (20) with the average squared distances of their components:
n(
n(
)7 (D
metric (pt) DD) + (Wo: - w)2
2+ +
N int..
N3
A Jui.j
H,
DG3
D
iut.J denote the j-th components of and H H while N, and H, represent the j-th components of N and H, respectively.
Here, N and A (()) iut,
(9)
Intuitively, as more architecture samples fall within the interval [0.9cp, Co.. the summation terms in Equations (20) and (21) will increase, which favors larger computed result. Similarly, if the distances between architectures within this interval are greater, their distances to the center will also increase, further contributing to larger result. Thus, our metric is considered reasonable.
34


Reviews:
[{"review_id": "OXq326tupW", "reviewer": "Reviewer_ykDK", "summary": "This work introduces the first training-free NAS method tailored for Mamba2, an improved version of the Mamba neural architecture. By leveraging rank collapse in SSD blocks, the method uses a lightweight proxy based on transformation matrix gradients. A custom search space and hyperparameter strategy are also proposed. Experiments show superior performance over existing training-free NAS methods in both ranking accuracy and final architecture quality.", "questions": "as my concerns", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1. This paper has strong novelty and limited generality.\n2. The paper proposes the first training-free NAS method for Mamba.\n3. The search space design is interesting but quite complex, which may hinder adoption.\n\nMinor Concerns:\n1. Can you clarify how the method could be adapted when no suitable reference model exists for calculating the search space range?\n2. Given the complexity of your pipeline, do you plan to release any tools or code to help practitioners adopt your method more easily?\n\nComment:\nGiven these strengths and limitations, I lean toward a borderline accept.", "was_revised": true, "final_justification": "Although I am a researcher in NAS, my understanding of Mamba is limited, so I find it difficult to confidently assess whether the issues raised and the proposed solutions in the paper are truly solid and novel. My evaluation was based on the reasoning and explanations presented in the manuscript, which I found to be reasonable. Therefore, I have decided to maintain my original score. However, I kindly suggest that the AC take into account the perspectives of other reviewers who may have more domain-specific expertise.", "rebuttal": "Thank you for your thoughtful questions regarding our manuscript. Here are our responses:\n\n1. Adapting the method without a suitable reference model\n\nThis is a practical question. Researchers or engineers proposing a new Mamba2-like model architecture typically define a specific model structure for training and testing, which can serve as a reference model. Thus, we believe a suitable reference model is usually available. In the rare cases where no reference model exists, researchers can assign values to each Adjustable Hyperparameter (AH) based on experience to form a reference model and determine the benchmark model and AH ranges. Notably, this process does not require training the reference model. If uncertain about AH values, researchers can use the search results as the reference model for a second search. Since the search involves no training, the time cost remains acceptable. We believe such a second search will rarely be necessary.\n\nIn fact, in traditional Neural Architecture Search (NAS) methods, specifying AH ranges is a key part of designing search space, with ranges typically based on the experience of designing existing models. Our method standardizes and refines this process to reduce human-induced biases, with the reference model serving a role analogous to \"the experience of designing existing models\" in traditional NAS. In the rare cases where no reference model is available, researchers can substitute it with their personal experience. The aforementioned second search in such cases is akin to obtaining experience directly from the search result.\n\n2. Plan for releasing tools or code\n\nYes. Considering the protection of intellectual property, we will release the code after our manuscript is accepted. Our code can automatically determine the search space and perform searches given a reference model (which does not need to be pre-trained and can be provided solely in the form of model implementation code and AH values), and it features comprehensive documentation and strong extensibility.\n\nWe appreciate your positive and professional questions, which have given us the opportunity to further clarify the details of our work. We are confident that our responses will address your concerns and enhance your understanding of our manuscript. Looking forward to your further reply."}, {"review_id": "6yBelB7yJn", "reviewer": "Reviewer_kmdV", "summary": "This paper proposes a training-free metric that ranks Mamba-like architectures that vary in terms of depth, width, state dimension and number of heads. The authors firstly identify the issue of rank collapse of the Mamba2 architecture, and based on this, propose a proxy metric that evaluates the degree of the collapse (hence trainability of the model). Afterwards, the authors evaluate the proposed proxy by measuring the rank correlation between the proxy scores and the accuracy of the sampled models on various reasoning datasets. To search for the best architecture in the design space, an evolutionary algorithm, that utilizes the proxy as fitness value, is used. The final benchmark results are promising and the authors conduct analysis and ablation studies on their algorithmic design choices as well.", "questions": "- In case 1 of Section 3.1, the authors assume that the input elements are normally distributed. How relevant is this assumption and is it violated in practice?\n- Is the TF-MAS proxy specifically designed for Mamba2 architectures or is it possible to use that on linear attention Transformer search spaces as well? \n- Why did the authors pick the Mamba2 architecture when recent work on subquadratic models [4, 5] has shown better performance compared to Mamba2?\n\n\n-- References --\n\n[1] Dong et al. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In ICML 2021\n\n[2] Zhou et al. Training-free Transformer Architecture Search. In CVPR 2022\n\n[3] Poli et al. Mechanistic Design and Scaling of Hybrid Architectures. In ICML 2024\n\n[4] Yang et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. In NeurIPS 2024\n\n[5] Yang et al. Gated Delta Networks: Improving Mamba2 with Delta Rule. In ICLR 2025", "limitations": "The authors do not discuss the limitations of their method in detail. Is the limitation of the method being applied only on Mamba2 architectures (mentioned in the Conclusion section) the only one?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The proposed approach proposes a new zero-cost proxy in order to automatically evaluate Mamba2 architectures with varying depth, width, state dimension and number of heads. The idea is novel and an interesting research direction.\n- The problem tackled by the paper is significant and more research towards automating such design choices in subquadratic models is in general necessary in my opinion.\n- The writing is clear and the paper is well-structured, including background and related work, theoretical justifications of their design choices and experiments including standard benchmark results and ablations.\n- Empirical results on commonsense reasoning datasets (Table 2) show promising results.\n\n**Weaknesses:**\n\n- The proposed solution to mitigating the problem of rank collapse and the architectural search space are simple and not novel per se (refer to [1] and [2] as rightfully referenced by the authors). Moreover, is there any reason why skip connections are not included in the architectural design space? In [2], it was shown that skip connections play a crucial role in mitigating rank collapse in Transformers.\n- Table 1 shows the rank correlation between the proxy values and the true performance of the sampled architecture. Despite the results being better than the other baselines, it still seems that the rank correlation coefficient is not high enough in some tasks in order to for the proxy to be a reliable metric.\n- The experimental results do not include multiple runs. As far as I know, the Mamba training can be unstable sometimes and therefore I see the necessity of reporting multiple runs in the experiments. Before evaluating the proxy on expensive benchmarks, I would recommend the authors to evaluate on smaller synthetic ones, such as the MAD benchmark [3]\n- The authors have not released the code, however they mention in the Checklist that they will do so upon acceptance. I think this is a very important part since the paper is also proposing new benchmarks in addition to the methodology.", "was_revised": true, "final_justification": "Most of my concerns were somehow addressed. Some of those not unfortunately, such as the evaluation on the MAD benchmark, which should be relatively fast (also since Mamba blocks are already in the repository code). I have increased my score to encourage the authors that the idea proposed in the paper is interesting, however, I am not entirely content with the execution.", "rebuttal": "Thank you for your detailed and constructive feedback on our manuscript. We have carefully considered your questions and suggestions, and we provide our responses below.\n\n1. Regarding the novelty of the proposed solution to mitigating rank collapse and the architectural search space:\n\nWe acknowledge that the proposed solutions to mitigate rank collapse and the architectural search space may not be entirely novel in themselves. However, the key innovation of our work lies in pioneering the first training-free NAS method tailored for Mamba-type networks. This is a novel and significant step in the field, moving from \"nothing to something\" in the context of Mamba-type networks. Existing training-free NAS methods have not focused on innovating the search space design; instead, they prioritize the reasonableness of the search space. So we also design a search space that is both reasonable and effective for Mamba-type networks. This is necessitated by the fact that existing NAS methods for Mamba-type networks [1, 2] have only explored the search for visual task-specific scan modes and have not ventured into the network architecture search. Our search space design draws inspiration from established works like Autoformer [3] and PiT [4], adapting their principles to the unique characteristics of Mamba networks. The resulting search space is sufficiently large to allow for meaningful exploration, with sizes of 19,773 and 5.68e50 for the two search spaces used in our experiments.\n\n2. Regarding the exclusion of skip connections in the architectural design space:\n\nIn [5], it's indicated that skip connections are important. Yet, even with them, rank collapse can only be mitigated, not fully avoided. Thus, [5] doesn't treat skip connections as an Adjustable Hyperparameter (AH). Instead, all candidate networks in [5] uniformly incorporate skip connections, using the degree of rank collapse to predict network performance. Similarly, in our work, all candidate networks also uniformly use skip connections.\n\n3. Regarding the rank correlation coefficient in Table 1:\n\nWhile our method demonstrates better performance than other baselines, we acknowledge that the rank correlation coefficient is not sufficiently high in some tasks for the proxy to be considered a fully reliable metric. It is important to note that our work represents the first training-free NAS method for Mamba-type networks. We have shown the feasibility of our method in practical search scenarios in Subsection 4.2, which validates its application value. Moving forward, we are committed to refining our method to achieve higher rank correlation coefficients and further enhance search outcomes.\n\n4. Regarding the experimental results and multiple runs:\n\nWe appreciate your suggestion to evaluate our method on smaller synthetic benchmarks like MAD [6]. Although we have not yet conducted experiments on smaller synthetic benchmarks due to time constraints, we have performed additional training and testing runs for opt VWMamba2 w/ bwe. The results are (exp1 is the model reported in the manuscript, while exp2 and exp3 are models retrained and tested.):\n\n| Proxy                     | LD (PPL) | LD (ACC) | HS (ACC) | PIQA (ACC) | Arc-E (ACC) | Arc-C (ACC) | WG (ACC) | OBQA (ACC) |\n| ------------------------- | -------- | -------- | -------- | ---------- | ----------- | ----------- | -------- | ---------- |\n| Mamba-2-130M              | 16.79    | 45.9     | 31.0     | 65.0       | 47.4        | 20.9        | 52.6     | 18.0       |\n| opt Mamba2                | 15.31    | 49.5     | 35.8     | 66.4       | 51.7        | 25.1        | 53.9     | 19.1       |\n| opt VWMamba2              | 15.20    | 50.7     | 36.9     | 66.4       | 51.8        | 25.2        | 54.7     | 19.4       |\n| opt VWMamba2 w/ bwe(exp1) | 15.08    | 51.5     | 40.5     | 66.9       | 52.6        | 25.2        | 55.2     | 19.5       |\n| opt VWMamba2 w/ bwe(exp2) | 15.05    | 51.2     | 40.5     | 66.8       | 52.6        | 25.2        | 55.3     | 19.4       |\n| opt VWMamba2 w/ bwe(exp3) | 15.10    | 51.4     | 40.7     | 67.0       | 52.9        | 25.3        | 55.5     | 19.5       |\n\nThe results from these runs show some fluctuations but generally maintain a performance advantage over Mamba-2-130M, opt Mamba2, and opt VWMamba2. In the future, we plan to conduct more extensive experiments, including evaluations on smaller synthetic benchmarks and multiple training and testing runs for more models.\n\n5. Regarding the release of code:\n\nWe understand the importance of code release for the reproducibility and adoption of our method, especially given the new benchmarks proposed in our manuscript. We have made the difficult decision to release the code upon acceptance of the manuscript, primarily to protect our intellectual property. We hope you understand our position.\n\n6. Regarding the assumption of normal distribution in Section 3.1, Case 1:\n\nOur assumption of a normal distribution for input elements is based on the absence of prior knowledge about the inputs. This approach is also adopted by some training-free NAS methods for CNNs, such as [7], which generate normally distributed tensors to simulate unknown inputs. To validate this assumption, we conducted experiments using a network with width W=96 and depth D=3, selecting 1,000 samples and truncating them to meet the condition of token number T=96. We extract the inputs U of each Mamba2 block and use the Kolmogorov-Smirnov test to confirm that these samples approximately follow a normal distribution. Additionally, we directly inspect the ranks of all 3,000 U matrices (1,000 samples with three Mamba2 block inputs each) and find them all to be full-rank matrices. These results demonstrate that our assumption does not lead to errors in practice.\n\n7. Regarding the applicability of the TF-MAS proxy (only for Mamba2) and limitations of the method:\n\nThe method is designed specifically for Mamba2, and all experiments are conducted on Mamba2. This is the first work to explore training-free NAS for Mamba-type networks. Without prior experience, directly creating a training-free NAS method for various Mamba networks is challenging. So designing such methods for more types of Mamba networks will be future work. Fortunately, the consistent computational patterns between linear attention Transformers and Mamba2, as illustrated in [8], offer a promising way to transfer proxy calculation method from Mamba2 to linear attention Transformers. Besides this, we believe no other obvious limitations in our manuscript.\n\n8. Regarding the selection of the Mamba2 architecture over recent subquadratic models:\n\nWe choose Mamba2 due to its status as a well-established and classic architecture within the Mamba family. Both the original Mamba and Mamba2 have been extensively studied and rigorously validated in numerous follow-up works [9, 10], providing a solid foundation for our research. Mamba2 has also demonstrated superior performance compared to the original Mamba. While recent subquadratic models show promise, they are relatively new and may require more time and extensive validation to achieve the same level of credibility as Mamba2. We are open to exploring these newer models in our future research.\n\nThank you again for your valuable feedback. We believe that your questions have allowed us to better clarify and strengthen our work. Looking forward to your further reply.\n\n-- References --\n\n[1] Huang T, Pei X, You S, et al. Localmamba: Visual state space model with windowed selective scan[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 12-22.\n\n[2] Fan C, Yu H, Huang Y, et al. Slicemamba with neural architecture search for medical image segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2025.\n\n[3] Chen M, Peng H, Fu J, et al. Autoformer: Searching transformers for visual recognition[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 12270-12280.\n\n[4] Heo B, Yun S, Han D, et al. Rethinking spatial dimensions of vision transformers[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 11936-11945.\n\n[5] Dong Y, Cordonnier J B, Loukas A. Attention is not all you need: Pure attention loses rank doubly exponentially with depth[C]//International conference on machine learning. PMLR, 2021: 2793-2803.\n\n[6] Poli M, Thomas A W, Nguyen E, et al. Mechanistic design and scaling of hybrid architectures[J]. arXiv preprint arXiv:2403.17844, 2024.\n\n[7] Lin M, Wang P, Sun Z, et al. Zen-nas: A zero-shot nas for high-performance image recognition[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 347-356.\n\n[8] Dao T, Gu A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality[J]. arXiv preprint arXiv:2405.21060, 2024.\n\n[9] Wang W, Dusparic I, Shi Y, et al. Drama: Mamba-enabled model-based reinforcement learning is sample and parameter efficient[J]. arXiv preprint arXiv:2410.08893, 2024.\n\n[10] Huang W, Pan J, Tang J, et al. Ml-mamba: Efficient multi-modal large language model utilizing mamba-2[J]. arXiv preprint arXiv:2407.19832, 2024."}, {"review_id": "cG9wD63Dq1", "reviewer": "Reviewer_yy6s", "summary": "This paper introduces TF-MAS, the first training-free Neural Architecture Search (NAS) method specifically designed for Mamba2 architectures. The core contribution is a proxy metric based on rank collapse theory in stacked State Space Duality (SSD) blocks, which mirrors the rank collapse phenomenon observed in stacked attention layers. The authors develop transformation matrices W_X, W_B, and W_C that map from the input tensor U to the SSD inputs X, B, and C, handling three different cases based on the relationship between sequence length T and feature dimension W. The proxy combines nuclear norms of these transformation matrices with their gradients to estimate network performance without training.\n\nThe paper also presents a principled search space design with four adjustable hyperparameters (depth D, width W, state dimension N, and number of heads H), where ranges are determined by computational cost constraints rather than arbitrary choices. The authors construct NASBenches for Mamba2 models using weight entanglement and validate their approach through ranking correlation experiments and actual search results, demonstrating superior performance compared to existing training-free NAS methods adapted from CNN and Transformer domains.", "questions": "***Mathematical rigor and correctness***: The statement in Equation (11) of ||U||_{1,\\infty} <= min_{x',y'} ||U-x'y'^T||_{1,\\infty} does not have a proper justification and is very likely wrong (a simple counter example: U=[[2,0],[0,1]], ||U||_{1,\\infty}=2, x'=(1,0)^T, y'=(2,0)^T, so ||U-x'y'^T||_{1,\\infty}=||[[0,0],[0,1]]||_{1,\\infty}=1). Can you provide a rigorous proof of this step or reformulate the approach? Additionally, your gradient calculations assume exact solutions (UW_X = X) but apply to approximate pseudoinverse cases in scenarios 2 and 3.\n\n***Proxy design justification***: The proxy formula in Equation (6) combines nuclear norms of transformation matrices with their gradients, but lacks theoretical justification for why this specific combination should predict network performance. Why is the nuclear norm product the right choice rather than other matrix norms or combinations? How sensitive is the proxy to the approximation errors introduced in Cases 2 and 3 where exact solutions don't exist? Can you provide error bounds showing how pseudoinverse approximations affect proxy reliability?\n\n\n***Weight entanglement validation***: The paper relies heavily on weight entanglement for NASBench construction but provides limited validation of its effectiveness for Mamba2. Can you provide more rigorous analysis comparing architectures trained from scratch versus those using weight entanglement? What is the correlation between their final performance, and how might this bias affect your conclusions?\n\n\n***Computational overhead breakdown***: You mention 1.31 seconds for proxy computation, but how does this scale with model size and search space complexity? Can you provide a detailed breakdown of computational costs for transformation matrix computation, especially for the VWSSMamba2 search space with its exponentially larger size?", "limitations": "No - While the authors discuss some practical limitations and societal impacts, they miss a critical limitation that significantly affects the reliability of their work. The authors acknowledge experimental scope (language modeling only), reliance on weight entanglement, testing on smaller models, and provide thoughtful discussion of societal impacts including both benefits (medical diagnosis, autonomous driving) and risks (data privacy).\n\nHowever, the authors fail to acknowledge that their core theoretical justification contains fundamental mathematical errors. The rank collapse proof in Equation (11) has critical flaws (type mismatches, incorrect inequality directions, logical gaps) that undermine the theoretical foundation of their approach. This represents a significant limitation in the validity and reliability of their method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n\n***Quality***: The theoretical foundation is solid, building on established rank collapse theory from attention mechanisms and properly extending it to SSD blocks. The mathematical treatment of the three cases (T=W, T<W, T>W) for computing transformation matrices is rigorous, with appropriate use of pseudoinverse for under-determined systems. The experimental validation spans multiple model sizes (130M to 2.7B parameters) and datasets, showing consistent improvements.\n\n***Clarity***: The paper is generally well-written with clear motivation. The connection between attention rank collapse and SSD rank collapse is well-established, and the proxy computation process is explained systematically. Figure 1 effectively illustrates the method integration into Mamba2 architecture.\n\n***Significance***: This addresses a genuine gap in the NAS literature - no previous training-free methods existed for Mamba architectures. The computational savings are substantial compared to training-based approaches, and the performance improvements over baseline Mamba2 models demonstrate practical value. The cost-driven search space design is a notable methodological contribution.\n\n***Originality***: The work is highly original as the first training-free NAS for Mamba. The extension of rank collapse theory to SSD blocks is novel, and the systematic handling of different tensor dimension relationships shows theoretical depth.\n\n**Weaknesses**:\n\n***Quality concerns***: The paper contains critical mathematical errors that completely invalidate its theoretical foundation. The rank collapse proof in Equation (11) contains one typo (argmin should be min). More importantly, it contains a severe flaw: the best rank-1 approximation error can usually achieve a smaller composite norm than the original norm. This fundamental mathematical error undermines the core justification for why the proxy should work. Additionally, the reliance on weight entanglement for NASBench construction introduces potential bias, and ranking correlations are moderate (0.3-0.7), suggesting room for improvement.\n\n***Clarity issues***: Some notation becomes heavy, particularly in the mathematical derivations. The relationship between the proxy computation and actual rank collapse could be made more intuitive. The search space construction process, while principled, involves several hyperparameters (0.6x and 1.6x scaling factors) that seem somewhat arbitrary.", "was_revised": true, "final_justification": "The authors have addressed my major concern regarding the mathematical soundness.", "rebuttal": "Thank you for your detailed and constructive feedback on our manuscript. We have carefully considered your questions and suggestions, and we provide our responses below.\n\n1. Mathematical rigor and correctness\n\n(1) About Equation (11)\n\nWe apologize for the error in our previous proof and now present a revised demonstration of the following fact: Repeatedly performing the operation of \"first taking the Hadamard product of the current matrix with a 1-SS matrix and then applying a linear transformation\" will cause the matrix to gradually approach a rank-1 matrix, resulting in rank collapse. It should be noted that in the first paragraph of Subsection 4.4, we have experimentally verified this conclusion, so our method is well-founded.\n\nWe will use mathematical induction to prove that after $k$ operations, the distance between the matrix $\\boldsymbol{X}_k$ and the rank-1 matrix $\\boldsymbol{Y}'$ satisfies $\\Vert\\boldsymbol{X}_k-\\boldsymbol{Y}'\\Vert\\leq\\beta\\alpha^k$.\n\nWhen $k=0$, assume that the initial matrix $\\boldsymbol{X}_0$ has a distance of $\\beta$ from $\\boldsymbol{Y}'$, that is,\n$\\Vert\\boldsymbol{X}_0-\\boldsymbol{Y}'\\Vert\\leq\\beta$, and suppose that after $k$ operations, the distance satisfies $\\Vert\\boldsymbol{X}_k-\\boldsymbol{Y}'\\Vert\\leq\\beta\\alpha^k$.\n\nConsidering the $(k + 1)$-th operation\n$\\boldsymbol{X}_{k+1}=(\\boldsymbol{L}\\odot\\boldsymbol{X}_k)\\boldsymbol{W}$, \n\nwe analyze $\\Vert\\boldsymbol{X}_{k+1}-\\boldsymbol{Y}'\\Vert$:\n\n$\n\\Vert\\boldsymbol{X}_{k+1}-\\boldsymbol{Y}'\\Vert=\\Vert(\\boldsymbol{L}\\odot\\boldsymbol{X}_k)\\boldsymbol{W}-\\boldsymbol{Y}'\\Vert=\\Vert(\\boldsymbol{L}\\odot\\boldsymbol{X}_k)\\boldsymbol{W}\\boldsymbol{a}\\boldsymbol{b}^\\top\\Vert\\leq \\Vert(\\boldsymbol{L}\\odot\\boldsymbol{X}_k)\\boldsymbol{W}-\\boldsymbol{L}\\odot(\\boldsymbol{a}\\boldsymbol{b}^\\top\\boldsymbol{W}^+)\\Vert+\\Vert\\boldsymbol{L}\\odot(\\boldsymbol{a}\\boldsymbol{b}^\\top\\boldsymbol{W}^+)-\\boldsymbol{a}\\boldsymbol{b}^\\top\\Vert \n$\n\n$\\leq\\Vert\\boldsymbol{L}\\odot(\\boldsymbol{X}_k\\boldsymbol{W}-\\boldsymbol{a}\\boldsymbol{b}^\\top\\boldsymbol{W}^+)\\Vert+\\Vert\\boldsymbol{L}\\odot\\boldsymbol{a}\\boldsymbol{b}^\\top\\boldsymbol{W}^+ - \\boldsymbol{a}\\boldsymbol{b}^\\top\\Vert\\leq\\alpha\\Vert\\boldsymbol{X}_k-\\boldsymbol{a}\\boldsymbol{b}^\\top\\Vert+\\gamma\n$\n\nHere, $\\alpha$ and $\\gamma$ are constants related to matrices $\\boldsymbol{L}$ and $\\boldsymbol{W}$, with $0<\\alpha<1$ and $\\gamma$ being a small constant.\n\nBy the inductive hypothesis and the above inequality, we can conclude that $\\Vert\\boldsymbol{X}_{k+1}-\\boldsymbol{Y}'\\Vert\\leq\\alpha\\Vert\\boldsymbol{X}_k-\\boldsymbol{Y}'\\Vert+\\gamma\\leq\\alpha\\beta\\alpha^k+\\gamma=\\beta\\alpha^{k+1}+\\gamma$.\n\nAs $k$ becomes large, $\\alpha^{k+1}$ approaches zero, so the distance $\\left\\Vert\\boldsymbol{X}_{k+1}-\\boldsymbol{Y}'\\right\\Vert$ also approaches zero. The proof is now complete.\n\n(2) About gradient calculations\n\nThe bias of values and its gradients about $\\boldsymbol{W}_X$ only occurs in Case 3. In Case 2, the obtained $\\boldsymbol{W}_X$ strictly satisfies $\\boldsymbol{U}\\boldsymbol{W}_X = \\boldsymbol{X}$. In the second paragraph of Subsection 4.4, we have pointed out that for the vast majority of candidates, only Case 2 is encountered, while Cases 1 and 3 are used in rare situations to enhance the robustness of the method. Additionally, in Item 2(2) below, we have explained that the deviation of $\\boldsymbol{W}_X$ has limited impact on the proxy.\n\n2. Proxy design justification\n\n(1) About Equation (6)\n\nIn fact, multiplying parameter-related terms and gradient-related terms has already been applied in existing methods [1]. Some other methods [2] add the logarithms of the terms, which is essentially a multiplication operation. None of these methods have provided theoretical proof. We believe that multiplication can amplify or suppress certain characteristics based on the magnitude of parameters and gradients. When both parameters and gradients are large, their product significantly increases, thereby highlighting this important information in the evaluation metric. This enables architecture search to prioritize network architectures with larger parameters and gradients, as these architectures may have stronger learning ability and adaptability during training.\n\n(2) About the approximation errors\n\nIn fact, only Case 3 employs approximation. For Case 3, we analyze the upper bound of $\\left\\Vert \\boldsymbol{U}\\boldsymbol{W}_X - \\boldsymbol{X} \\right\\Vert$ and derive the conclusion $\\left\\Vert \\boldsymbol{U}\\boldsymbol{W}_X - \\boldsymbol{X} \\right\\Vert \\leq \\left\\Vert\\boldsymbol{X}\\right\\Vert_F$ (the proof is provided below). This implies that as long as the Frobenius norm of $\\boldsymbol{X}$ is not excessively large, the resulting error is small. Similar conclusions apply to $\\boldsymbol{B}$ and $\\boldsymbol{C}$. When calculating the proxy, we can preprocess the input data (e.g., multiplying by a small constant) to keep $\\boldsymbol{X}$, $\\boldsymbol{B}$, and $\\boldsymbol{C}$ at small values, thereby controlling the error introduced by this approximation.\n\n-- Proof --\n\nThe least squares solution $\\boldsymbol{W}_X$ corresponds to the orthogonal projection of $\\boldsymbol{X}$ onto the column space of $\\boldsymbol{U}$. The residual matrix $\\boldsymbol{R}=\\boldsymbol{U}\\boldsymbol{W}_X-\\boldsymbol{X}$ lies in the orthogonal complement of $\\boldsymbol{U}$. Since orthogonal projection does not increase the norm, the projection of $\\boldsymbol{X}$ onto the orthogonal complement (i.e., the Frobenius norm of the residual matrix $\\boldsymbol{R}$) must not exceed the Frobenius norm of the original matrix $\\boldsymbol{X}$: $\\Vert\\boldsymbol{R}\\Vert_F=\\Vert(\\boldsymbol{I}-\\boldsymbol{V}\\boldsymbol{V}^\\top)\\boldsymbol{X}\\Vert_F\\leq\\Vert\\boldsymbol{X}\\Vert_F$, where $\\boldsymbol{I}$ is the identity matrix, and $\\boldsymbol{V}$ is the matrix of left singular vectors of $\\boldsymbol{U}$.\n\nIf $\\boldsymbol{X}$ is entirely within the orthogonal complement of $\\boldsymbol{U}$, then $\\boldsymbol{U}\\boldsymbol{W}_X=\\boldsymbol{0}$, and $\\Vert\\boldsymbol{U}\\boldsymbol{W}_X-\\boldsymbol{X}\\Vert\\leq\\Vert\\boldsymbol{X}\\Vert_F$, reaching the upper bound. If $\\boldsymbol{X}$ is entirely within the column space of $\\boldsymbol{U}$, the residual is zero, and the upper bound holds.\n\nIn summary, the original inequality is established, and the proof is complete.\n\n3. Weight entanglement validation:\n\nRegarding the impact of weight entanglement on network performance, most existing work has been experimentally verified rather than theoretically proven due to the complexity of neural network models. Fortunately, [3] has been experimentally proven that weight entanglement in Transformers introduces minimal bias. Additionally, [4] has been shown that Mamba2 and Transformers share the same computational pattern in their core modules (attention and SSM). This indirectly indicates that weight entanglement in Mamba2 also does not introduce significant bias.\n\nFurther more, adding noise to performance data tends to decrease the Kendall’s Tau between proxy and performance (we provide proof below). So the actual ranking correlation would be higher than the results published in Subsection 4.1.\n\n-- Proof --\n\nAssume performance data is $(x_1,x_2,\\cdots,x_n)$ and proxy data is $(y_1,y_2,\\cdots,y_n)$. The Kendall's Tau coefficient is defined as $\\tau=\\frac{C-D}{\\frac{1}{2}n(n-1)}$, where $C$ is the number of concordant pairs and $D$ is the number of discordant pairs.\n\nWhen noise $\\epsilon_i\\sim\\mathcal{N}\\left(0,\\sigma^2\\right)$ is added to each $x_i$, forming $x_i'=x_i+\\epsilon_i$, the comparison between any two samples $i$ and $j$ changes. The difference $\\epsilon_i-\\epsilon_j$ follows $\\mathcal{N}(0,2\\sigma^2)$. For a pair $(i,j)$ where $x_i>x_j$, the probability that $x_i'>x_j'$ is $1-\\Phi\\left(\\frac{-d}{\\sqrt{2}\\sigma}\\right)$, where $d=x_i-x_j$. For a pair where $x_i<x_j$, the probability that $x_i'<x_j'$ is $\\Phi\\left(\\frac{d}{\\sqrt{2}\\sigma}\\right)$, where $d=x_j-x_i$.\n\nNoise can turn concordant pairs into discordant pairs and vice versa. The expected number of concordant pairs decreases, while the expected number of discordant pairs increases. Thus, the expected new Kendall’s Tau $\\tau'$ after adding noise is $\\mathbb{E}[\\tau']=\\frac{\\mathbb{E}[C']-\\mathbb{E}[D']}{\\frac{1}{2}n(n-1)}<\\tau$. This shows that adding normal noise reduces the expected Kendall’s Tau value. The proof is complete.\n\n4. Computational overhead breakdown:\n\nWe analyze the time complexity here. The complexity of SVD is O(TW min(T, W)), and that of computing the pseudo-inverse is O(min(T, W)). Calculating $U^+$ has a time complexity of O(WT^2), while computing $W_X$, $W_B$, $W_C$ and their gradients has complexities of O(WTP), O(WTN), and O(WTN) respectively. The nuclear norm computation for these matrices has complexities of O(WP min(WP)), O(WN min(WN)), and O(WN min(WN)). Additionally, computing the nuclear norm of $W_{\\text{out}}$ and its gradient is O(WP min(WP)). Here, T is input-dependent and can be treated as a constant. Considering multiple heads, the time complexity for one Mamba2 block is O(WHP min(WP) + WHN min(WN)). It is evident that when any of W, H, P, N approaches infinity, the time complexity grows linearly.\n\nIt should be noted that VWSSMamba2 merely expands the search space of SSMamba2 without increasing the scale of candidate networks. The scale of candidate networks is determined by the expected computational overhead. Thus, the time overhead does not increase significantly.\n\nThank you again for your valuable feedback. Looking forward to your further reply.\n\n-- References --\n\n[1] Training-free transformer architecture search\n\n[2] Zen-nas: A zero-shot nas for high-performance image recognition\n\n[3] Transformers are ssms: Generalized models and efficient algorithms through structured state space duality\n\n[4] Autoformer: Searching transformers for visual recognition"}, {"review_id": "mFthx278HI", "reviewer": "Reviewer_vzNA", "summary": "This paper, for the first time, propose a Zero-cost NAS for Mamba2 models. Using the rank-collapse property, the propose method desinged a new indicator which shows high rank correlation with the actual validation performance.", "questions": "Please refer to the weaknesses section and provide responses to each point.", "limitations": "The scope and practical applicability of the proposed method may be somewhat limited.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The paper is the first to propose a zero-cost NAS method specifically tailored for Mamba2 models.\n2. The proposed indicator is thoughtfully designed, leveraging the unique structural properties of Mamba2 blocks.\n3. New datasets are introduced to facilitate zero-cost NAS evaluation for Mamba2, which adds practical value.\n\nWeaknesses\n1. The reported rank correlation of the proposed method is approximately 0.68, which may be insufficient for reliable application in practice.\n2. A correlation analysis focused on the top-K candidates would be more appropriate, as final model selection typically depends only on this subset.\n3. In Eq. (6), the indicator aggregates values across blocks without normalization. A more general formulation would consider averaging to ensure comparability across different architectures.\n4. Since Mamba2 is a variant of Mamba, the overall scope and practical applicability of the work may be somewhat constrained.", "was_revised": true, "final_justification": "Although the paper would benefit from additional polishing and broader experimental validation, it represents a promising step toward training-free NAS for emerging long-sequence architectures. I maintain my current score.", "rebuttal": "Thank you for your thorough evaluation of our manuscript. We address your points as follows:\n\n1. Regarding the rank correlation of approximately 0.68:\n\nWe acknowledge that the reported rank correlation of our proposed method is around 0.68, which might be insufficient for reliable practical application. However, it is important to note that our work introduces the first training-free NAS method for Mamba-type networks. Despite the ranking correlation not being sufficiently high, our method provides a feasible approach for training-free NAS in the context of Mamba-type networks. Furthermore, we have demonstrated in Subsection 4.2 that our method yields significant improvements in practical search scenarios, which underscores its applicability. We are committed to enhancing our method in the future to achieve higher ranking correlation coefficients and thereby improve search outcomes.\n\n2. Regarding the suggestion for a correlation analysis focused on the top-K candidates:\n\nWe appreciate your suggestion to focus on the top-K candidates for correlation analysis, as the selection of the final model typically depends on this subset. In response, we have conducted experiments across all search spaces to evaluate the ranking correlation for the top 20%, top 10%, and top 5% of candidates. The results obtained on the VMSSMamba2Bench_2.7B are presented in the table below:\n\n| candidates | LD (PPL) | LD (ACC) | HS (ACC) | PIQA (ACC) | Arc-E (ACC) | Arc-C (ACC) | WG (ACC) | OBQA (ACC) |\n| ---------- | -------- | -------- | -------- | ---------- | ----------- | ----------- | -------- | ---------- |\n| all        | -0.685   | 0.661    | 0.381    | 0.468      | 0.339       | 0.452       | 0.483    | 0.519      |\n| top 20%    | -0.693   | 0.683    | 0.390    | 0.483      | 0.347       | 0.470       | 0.486    | 0.527      |\n| top 10%    | -0.675   | 0.653    | 0.374    | 0.453      | 0.319       | 0.455       | 0.468    | 0.500      |\n| top 5%     | -0.662   | 0.692    | 0.366    | 0.421      | 0.332       | 0.439       | 0.489    | 0.522      |\n\nThese results indicate that the ranking correlation for the top-K candidates is comparable to that of the entire candidate set. Similar trends have been observed in other search spaces.\n\n3. Regarding the aggregation of values across blocks without normalization in Eq. (6):\n\nWe thank you for your suggestion regarding the aggregation of values across blocks in Eq. (6). We opt to directly sum the results from each layer, following the approach adopted by many existing training-free NAS methods, such as those mentioned in [1, 2]. However, we acknowledge that normalizing the values across layers and using an average instead of a sum is a more reasonable approach. We will conduct additional experiments in the future to further refine our method.\n\n4. Regarding the scope and practical applicability of the work:\n\nWe agree that since Mamba2 is a variant of Mamba, the overall scope and practical applicability of our work might be somewhat limited. As the first work to explore training-free NAS for Mamba-type networks, we believe that developing a method applicable to multiple types of Mamba without prior experience is highly challenging. Therefore, extending our approach to a broader range of Mamba networks will be a focus of our future research.\n\nThank you again for your valuable feedback. We are confident that your questions have allowed us to better clarify and strengthen our work. Looking forward to your further reply.\n\n-- References --\n\n[1] Lin M, Wang P, Sun Z, et al. Zen-nas: A zero-shot nas for high-performance image recognition[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 347-356.\n\n[2] Zhou Q, Sheng K, Zheng X, et al. Training-free transformer architecture search[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 10894-10903."}]

Author rebuttal:

Meta-review: A summary of the strengths and weaknesses based on the reviews and rebuttal (including the follow-up discussion among the reviewers and authors) is provided below.


**STRENGTHS**

(1) This paper proposes the first training-free metric for NAS in Mamba2 architectures (Reviewer kmdV,yy6s,vzNA), potentially inspiring future research along this line of work.

(2) The proposed problem is significant (Reviewers kmdV,yy6s).

(3) The paper is written well (Reviewers kmdV,yy6s).

(4) Empirical results are promising with new datasets introduced (Reviewers kmdV,vzNA).


**WEAKNESSES**

(1) The proposed solutions to mitigate the issue of rank collapse and the architectural search space are not new (Reviewer kmdV).

(2) The rank correlation coefficient is not sufficiently high in some tasks for the proposed proxy to be considered a fully reliable metric (Reviewers kmdV,vzNA).

(3) The proposed method has limited applicability as it can only be applied to Mamba2 architectures (Reviewers kmdV,vzNA).

(4) Initially, Reviewer yy6s has pointed out an error about equation (11), which has since been corrected by the authors in their rebuttal.


Weaknesses 1, 2, and 3 are acknowledged by the authors in their rebuttal to Reviewer kmdV.

From the above, the pros (especially strength 1) outweigh the cons.

The authors are encouraged to revise their paper based on the above feedback and that of the reviewers.

Minor note:
The authors can consider providing some more references to the conventional training-free NAS papers, such as the ones below (and see the references therein), and describe how the proposed approach fundamentally differs from such works (besides the difference in models):

Zero-cost proxies for lightweight NAS. ICLR 2021.

Robustifying and Boosting Training-Free Neural Architecture Search. ICLR 2024.

Neural architecture search on ImageNet in four GPU hours: A theoretically inspired perspective. ICLR 2021.

Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search. 2022

Towards NNGP-guided Neural Architecture Search. arXiv 2020.

NASI: Label- and Data-agnostic Neural Architecture Search at Initialization. ICLR 2022.
Decision: Accept (poster)