Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
Authors: Tianyi Chen
Pengxiao Lin
Zhiwei Wang
Zhi-Qin John Xu
Venue: NeurIPS 2025 spotlight ()
Keywords: Mamba
Transformer
Deep learning
Primary area: deep_learning

Abstract: State Space Models (SSMs) have emerged as promising alternatives to attention mechanisms, with the Mamba architecture demonstrating impressive performance and linear complexity for processing long sequences. However, the fundamental differences between Mamba and Transformer architectures remain incompletely understood. In this work, we use carefully designed synthetic tasks to reveal Mamba's inherent limitations. Through experiments, we identify that Mamba's nonlinear convolution introduces an asymmetry bias that significantly impairs its ability to recognize symmetrical patterns and relationships. Using composite function and inverse sequence matching tasks, we demonstrate that Mamba strongly favors compositional solutions over symmetrical ones and struggles with tasks requiring the matching of reversed sequences. We show these limitations stem not from the SSM module itself but from the nonlinear convolution preceding it, which fuses token information asymmetrically. These insights provide a new understanding of Mamba's constraints and suggest concrete architectural improvements for future sequence models.

Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
Tianyi Chen 12 Pengxiao Lin 12; Zhiwei Wang Zhi-Qin John Xu 12.3t
Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University 2 School of Mathematical Sciences, Shanghai Jiao Tong University Shanghai Seres Information Technology Co., Ltd, Shanghai 200040, China
Abstract
State Space Models (SSMs) have emerged as promising alternatives attention mechanisms, with the Mamba architecture demonstrating impressive performance and linear complexity for processing long sequences. However, the fundamental differences between Mamba and Transformer architectures remain incompletely understood. In this work, use carefully designed synthetic tasks reveal Mamba's ' inherent limitations. Through experiments, we identify that Mamba's s nonlinear convolution introduces an asymmetry bias that significantly impairs its ability to recognize symmetrical patterns and relationships. Using composite func- tion and inverse sequence matching tasks, we demonstrate that Mamba strongly favors compositional solutions over symmetrical ones and struggles with tasks requiring the matching of reversed sequences. We show these limitations stem not from the SSM module itself but from the nonlinear convolution preceding it, which fuses token information asymmetrically. These insights provide a new understanding of Mamba's 's constraints and suggest concrete architectural improve- ments for future sequence models.
Introduction
Large Language Models (LLMs) have achieved remarkable progress and are now widely applied
across a broad range of fields (Vaswan all, 2017: Liu all 2018 Devlin all 2018; Radford
at 2019; Touvron et all 2123: OpenAl 2023; Brown et all 2020, Dong et al,, 2122: Garg et all 2022 Davies all 20221).. The performance and inductive biases of such models are largely determined by their underlying architectures. Among these, the Transformer (Vaswani et all 2017)) has become dominant backbone in L.Mss. Its attention mechanism is central to its strong performance (Brown et all 2020; Olsson et al, 2022; (Wan et all 2024)). However, this mechanism also represents one of its major limitations: the computational complexity of attention scales quadratically with sequence length, making Transformers inefficient for long-sequence tasks.
2022: Trinh ef all
To address this issue, various Transformer variants have been proposed reduce the computational cost of attention, including sparse attention (Chrld et all 2019) and linear attention mechanisms
(Katharopoulos all 2021). Mamba (Gu and Dac, 2024 Dao GIT, 2024), which incorporates State Space Model (SSM), has recently garnered significant attention due to its linear complexity with respect sequence length and its superior performance on long-sequence problems. As result,
SSM offering a natural, computation-efficien alternative similar to linear attention have become focal point research.
"Equal contribution.
Corresponding author: xuzhiqin wuw...cuussu....nn.......
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To better understand the behavior of large-scale models and to guide meaningful architectural im- provements, it crucial to investigate the underlying causes of the differences between Mamba and Transformer. Given the inherent complexity of natural language tasks, this study leverages simple but meticulously crafted synthetic data.
When examining architectural details more closely, is the difference between Mamba and Trans- former limited merely to the replacement of attention with State Space Models (SSMs)? In fact, beyond the use of SSMs, Mamba exhibits several fundamental differences from the Transformer architecture. One of the most critical distinctions lies in Mamba's use of nonlinear convolution (O'shea and Nash, 2015).. On one hand, the nonlinear convolution enables Mamba to propagate information within a sequence without relying solely on the SSM unlike Transformer, where Trans- former depends entirely on attention for intra-sequence communication. On the other hand, this convolution fuses token-level information within the sequence, and the SSM operates on this fused representation to perform matching and extraction.
Notably, the nonlinear convolution in Mamba introduces an intrinsic asymmetry due to the asym- metric structure of convolution kernels. This asymmetry is transferred to the fused token represen- tations and consequently affects how information extracted. To better understand this limitation, we devised a composite function task (Zhang et al, 2024716 2025) aimed at evaluating how Mamba handles compositional structures.
The composite function task admits two possible solutions: a composite solution and a symmetric solution. We observe that Mamba exhibits a strong bias toward the composite solution while strug- gling to learn the symmetric one. We find that Mamba struggles to match sequences under order changes-for for example, "1234" vs. "4321". To test this limitation, matching task, where the model must match a sequence with its reversed counterpart. Experimental results confirm that Mamba has difficulty completing this task, whereas Transformer handles it with ease. We further introduced a residual connection, complemented by positional encoding, allowing the SSM to directly handle data without relying on convolution. This modification led to a signifi- cant improvement in Mambas performance on the inverse sequence matching task, confirming that the core issue lies not within the SSM itself, but rather the nonlinear convolution outside the SSM. This work therefore provides a new angle of symmetry to understand fundamental mechanisms of Mamba structure.
designed inverse sequence
The main contributions of this work are as follows:
1. We conduct in-depth analysis of how Mamba solves the composite function task, reveal- ing a fundamental difference in information acquisition between Mamba and Transformer. We show that Mamba tends rely on convolution to retrieve relevant information.
2. Through systematic experiments and observations, we identify Mamba's 's bias toward asym- metric solutions in the composite function task. We demonstrate that this behavior distin- guishes Mamba from Transformer and trace the root cause to the asymmetry introduced by convolution.
3. To further examine this asymmetry bias, we design an inverse sequence matching task, in which Mamba exhibits clear difficulties. We show that these difficulties can be effectively addressed through targeted modifications inspired by earlier findings, leading to sub- stantial performance improvements.
4. By highlighting how nonlinear convolution- based fusion induces an inherent asymmetry in Mamba, we provide insightful guidance for future model improvements and the design of new architectures.
2 Related work
State Space Models (SSM), Mamba, and Their Shortcomings. State space models (SSM) orig- inate from neuromorphic spiking models (Noelken, 2014; Voelker et all 2019) and have gained prominence through several key developments, such as S4 (Gu et all 2022). S5 (Smith et all 2023). the RWKV series (Peng et al,, 2023, 2024), RetNet etal 2023), and GLA (Yang et all 2022)) Among these, Mamba2 (Gu and Dad 2022) stands out by delivering performance competitive with Transformers while requiring significantly lower computational resources. However, numerous stud- ies reveal that Mamba has notable limitations. Research including (Ben-Kish et all 2025; ct all

---

20REX Mnan et all (025)) indicates that Mamba generally underperforms compared to Transform-
ers in tasks involving contecontext understanding, prompting the development of alternative models like DeciMamba, LongMamba, and ReMamba. Similarly, (Park et all, 2022; Waleffe et all 2024) explore Mambas in-context learning abilities and conclude that they fall short of Transformer ca- pabilities. Additionally, (Arora el all 20255 Telassi et all 2022) demonstrate that Mamba struggles with retrieval tasks, such as copying information from the input context. Some studies, such as (Xu cLal 2025), also examine the practical efficiency of Mamba compared to Transformers, noting that despite its theoretical advantages, real-world performance can be lacking. Furthermore, (Ren et all 2022)) introduces COPY task, which exposes a performance bottleneck in Mamba. Our study takes a structural view of Mamba, employing carefully designed synthetic data to explore its behavior from the perspective of symmetry, offering deeper insights into these shortcomings of Mamba and proposing a solution to address them.
Understanding the Mechanism of Neural Network Models. Our work conducts an in-depth inves- tigation into the internal mechanisms of Mamba. In identifying the functional roles of key modules in Mamba presented in this paper, we adopt the commonly used techniques of perturbation and causal intervention similar to research on interpretability in large language models (Vig ct all 2021
Jeoung and Diesner, 2022 Wang all 2022: Conmy all 2023: Merullo et all, 2023; Guo et all
2023% Wang and Weman, 2022; Amsel et all 2022; Li el all 2024 Wang et all 2024). The construc- tion of synthetic datasets in our study is primarily inspired by the works of Poli et all (MI2E) and
Zhang et all (20222). The design of our composite function tasks is adopted from the approach used
in Zhang ct all (2(24))) Additionally, several insightful theoretical studies feedforward neural networks have also informed our theoretical analysis. For instance, various works explore the pref- erences and generalization capabilities of neural networks from perspectives such as regularization, frequency, and dynamics (Xu et all, 20255. R, 2019; Wang et all 2024; [acoo et all 2018, 2020 Arora
2015, 2018; Wi and Sni, 20233 Wang Wu, 2023; Arora all 2022:
LL al 202T WT
201:
Zhu el
201; Arora et all 2012 Ren et all 20224).
3
Preliminary
3.1
Introduction of Mamba
The overall structure of Mamba (Gu and Dad, 2024: Dao and Gm. 2024) is shown in Fig. ML The Mamba block can be divided into three parts: the widely known SSM component, the pre-SSM part, and the post-SSM part. Omitting trivial dimension transformations and setting the batch size to omit the batch dimension, for given input U the block, the internal computation process to obtain the output O can be described as follows:
Pre-SSM
(Ù, 7, dt) Linear(U), UE Rlsdd) R(5.2442h) 7 E R(x:2d)) dt R(s.Na),
(1)
(B, X) = (Convid(()), BE R C e R(aM) , E Hla2d)) X = X o X 2NN.))
(2)
SSM
R,,,, Mask = F(Ao,d() Ao E RM,,, Mask E ((NNas))))
(4)
I Repeat(CB" ,N)),, TE R!Na....)
(5)
S = Mask I,, S E (Hisss)) Y = S YE R(MN))
(6)
Post-SSM
YNorM- RMS(Y (o)))) Ynorm E R(x22d)
(8)
= Linear Linear(YNorna) (Ynorm), Oe R(s.2d)
(9)
where 8 is the sequence length, d is the model's hiddenstate dimension, the SSM hidden dimen- sion, S is the SSM matrix, N₂, is the number of SSM heads, Linear denotes linear transformation.
3

---

Convid denotes a one one-dimensional convolution, denotes nonlinear activation function, F de- notes the function that generates the M aakk, Repeat denotes dimension replication operation, denotes pointwise multiplication, and RMS denotes RMS normalization.
We define the composition of and Convld Id nonlinear convolution and attention score from token j to token i is given by the (i.j) entry of the SSM matrix throughout the paper. For more comprehensive computational details, please refer to the appendix.
3.2 Difference between Mamba and Transformer
Transformer block xL
Q!!l
OIG
FNN
Input
Output
Mamba block X
SSM
delS
Maak/!!
CH X
BCX
X
RMS
Emb
Uy d 3o
Proj
Input
Output
Figure 1: Overview of Mamba and Transformer Blocks. The green trapezoids represent linear mappings. "smax" denotes the softmax function, "FNN" stands for feed-forward neural network, and "LN" represents layer normalization. The meanings of variables specific to the Mamba block explained in the main text.
By incorporating the SSM module, Mamba circumvents the quadratic complexity of attention in Transformers. While SSM significantly improves computational efficiency, the key difference from attention appears limited to the softmax function and the learnable mask. However, notable distinc- tions also exist beyond the SSM itself.
Prior to the attention- like SSM, Mamba applies a nonlinear convolution that fuses information, limit- ing the SSM to operate already mixed representations. Mamba contains z-gate following SSM, Network (FFN), and its preceding nonlinear layer lacks hidden layers.
lacks
As we will demonstrate, nonlinear convolution is a defining feature of Mamba that underlies its fundamental divergence from the Transformer.
3.3 Composite function task
detailed illustration of the composite function task (Zhang et all 20243LF 2025) is shown in Fig. The core idea of the composite function task can be understood with simple real world analogy. Imagine a row of people indexed by numbers. Let's define two functions: n(x):: Find the person n positions to the right of person and m(x): Find the person positions to the left of person æ. The composite function (x()))) means "If you start with person x, go people to their right, and then people their left, who do you end up with?"
In the composite function task, each sequence contains two anchors and one key, the remaining elements in the sequence are randomly sampled from the same range the key. Each anchor is

---

pairs of composite function
sequence example
label
seen
11
96
75 34 46 92
51
32 95
39
2
80 47 49
45
+5
53 75
66
4
54 92 31 24
50
non non-compostion function
3
44 20 22
43
8
symmetric solution
unseen
Which type of solution does Mamba tend to learn?
composite solution
8 10
Figure 2: Illustration of the Composite Function Task. Anchors 2, 3, and (depicted orange) each represent distinct functions. Among the 16 anchor pairs formed, 14 correspond to composite functions derived directly from the sequential application of the individual anchor funetions. The pair "34", highlighted in red, is defined as different function rather than direct composition. The pair "43" is intentionally excluded from the training set. The input to each anchor pair function is referred to as the "key" (indicated in green). Label indicates the output of an anchor pair applied to a key.
assigned a unique function, and a pair of anchors defines composite function, with the key (token before anchor pairs) serving the input. We the token through anchors, yielding a total of 16 possible anchor pairs. Among these, 14 composite functions are defined by directly composing the functions associated with the individual anchors. For the anchor pair "34", however, assign a function that deviates from the standard composition of its components.
The objective is to examine how Mamba handles the unseen anchor pair "43". Two plausible solu- tions exist: one is the symmetric solution, which infers the result of "43" by symmetry from the result of "34"; the other is the composite solution, which computes the result by directly composing the functions associated with "4" and "3".
In all experiments, the loss is calculated exclusively based on the final token of the sequence and its corresponding label. For detailed data and training settings, please refer to the appendix.
3.4 Inverse sequence matching task
To investigate whether Mamba can match token information fused by the nonlinear convolution, we design inverse sequence matching task. The detailed structure of the task is illustrated in Fig. The inverse sequence matching task corresponds to real world situations where people search for the symmetrical counterpart of object text. It is similar asking children to find the symmetrical counterpart of a toy and fit them together. This task can test model's ability to perform symmetry matching. Although it is simple and fundamental, it reflects core capability the model.
Each sample is built from a generating set of three distinct numbers (c.g., {28, 92, 37}). From its six possible permutations, five are randomly chosen to form key sequences, which are concatenated and separated by a token not in the generating set.
Next, one of the five key sequences randomly selected, reversed, and appended to the end as query sequence, separated by non-generating-set tokens so that the query and key sequences would not fuse by convolution. The number of tokens is determined by Mamba's pure convolutional receptive field. For example, in the illustration, six tokens correspond the pure convolutional receptive field (refers exclusively to how many tokens can be accessed through convolution alone,
5

---

generating set
possible key sequence
92
Can Mamba find it?
choosen key sequence 28
82
37
02
28
37
28
37
92
37
28
37
28
92
28
92
37
sequence index
3
6 Inverse
sequance
query sequence
28 92 37 52 92
28
37
46
28
37
92
82
37
92
28
52 37
28
92
46 77 46 52 82 77 32 28 92
37
4
Block the convolution
Figure 3: Illustration of the Inverse Sequence Matching Task. The orange elements denote the generation set, which consists of three distinct numbers randomly selected from the interval [20, 100], as well as all possible permutations thereof. Blue and green indicate selected key sequences from the permutation space, separated by random numbers that do not belong to the generation set. One green key sequence is chosen as the answer sequence. The query sequence (shown in red) is obtained by reversing the answer sequence. The corresponding label identifies the position of the answer sequence. To prevent Mamba from leveraging its nonlinear convolution mechanism to infer answers, we prepend the query sequence with random numbers (outside the generation set) matching the length of Mamba's 's pure convolutional receptive field.
without invoking the SSM) of two-layer Mamba (2 6). Finally, the label is the position index of the answer sequence, i.e., the unreversed version of the query sequence.
Mamba biases composite solution
In this section, will empirically show that Mamba is struggling with the symmetric property in composition but biases composite solutions, and analyze the role of nonlinear convolution.
Initialization scale is shown be critical for Transformer to learn the composite symmetric solution (Zhang ct al, 2024R, 2025).. Therefore, we scan different initialization scales as follows. A parameter W E 2 is initialized Gaussian distribution (0, (1/d))²), where is called initialization rate (Luo all 2021).
We scan Mamba with different layers and with different y. As shown in Fig. HL where the accuracy computed by regarding the label as the composite solution in Fig. Ba the symmetric solution in Fig. db, for small (large initialization). the network fails to capture either composite symmetric solution. The value at each position in the figure computed as the average of three independent random runs. This is consistent with previous study in Transformer (Zhang all, 20246, 2025). For large (small initialization), the network biases the composite function across almost all cases.
4.1 SSM does not function in composite tasks
It requires information from anchors and key to finish a composition task. Mamba has two potential options: utilizing the convolution or the SSM module. We found that in standard Mamba (Dao and Gu. (024)). convolution plays critical role while SSM module does not function.
Information blocking. The information flow analysis is a useful tool to visualize the information exchange on token level (Wang et all, 2024).. Since the SSM module has high similarity with the attention, we treat every element in SSM matrix as the "attention score" information flow in Fig. The result suggests that, within the SSM, tokens at later positions have little attention to the key and anchor information. In the case of the Transformer, failure to retrieve the required information via attention renders it incapable of solving the composite function task. To further verify that Mamba does not utilize the SSM for information propagation, we applied a causal intervention approach (Feng and Steinhardi, 2023 Meng et all, 2022; Vig et all 202C Wang et all, 2024), manually block- ing all information flow from the key and both anchors to the downstream tokens. The (i, element of S represents how much token attends to token If this value is set to 0, it implies that token

---

1.0
5:08
0.09
0.8
0.2
8.17
2.80
0.00
0.26
6.24
0.23
0.41
0.55
6.90
2477
0.
0.05
0.06
0.25
P:4
0.27
0.00
0.00
5.10
6.33
0.07
0.04
D.04
layer num
8.1
0.89
0.06
0.6
8:8
5.03
2.30
0.21
0.24
0.07
0.00
0.00
BCT
5.8
2,00
0.09
0.57
6:26
0.17
0.23
0.55
6.44
241
0.64
0.05
0.67
0.25
946
0
0.07
0.7
8:25
6.30
0.01
0.04
0.7
8C6
6:95
0.5
13 0.08
0.38
0.0
0.1 0.2 0.3 0.4
0.5 0.6
0.7
0.8 0.9 1.0 1.5 2.0
0.1
0.2 0.3 0.4 0.5 0.6 0.7
0.8 0.9 1.0
1.5
2.0
Y
V
aa) Mamba, comp.
(b) Mamba, sym.
Figure 4: Phase diagram of Mamba the composite function task. Accuracy (color) for composite function task under different initialization rates (abscissa) and depths (ordinate). The groundtruth for (a) is composite solution and for (b) is symmetric solution. Detailed model configurations and training settings are provided in the appendix.
layer
81
29
SR
AB
state substitution
(enchor-only
Comeldl
substitution
layer
convolation process
blacking
layer
34
SSM Input
41
key
anchor
anchor
Figure 5: SSM information flow in Mamba for the composite function task. Left: SSM informa- tion flow; green crosses indicate pruned connections. Right: (i) SSM input computation: (ii) state replacement after convolution. Flow is computed from S Mask C (TB B, with line thickness indicating flow magnitude. Attention score from token j to token is given by the (i,j) entry of S The numbers are the outputs of each layer through the models final linear layer and then take the argmax of the resulting logits to obtain the corresponding digit.
i cannot receive information from token j through the SSM. Our blocking mechanism imple- mented by zeroing out the specific entries in S corresponding to the connections we wish to block. As shown in the Fig. R. for various anchor pairs, the output after cutting these connections remains nearly identical to the original output, indicating that the SSM plays little role in transmitting this information.
Information substitution. further verify that Mamba relies on convolution for information transmission, we conducted an information substitution experiment: if Mamba encodes all necessary information through convolution, then transferring the resulting state to another sequence should enable it to produce the same output the original. For all sequences with anchor pairs other

---

than '43', as illustrated in Fig. E. we replaced the post-convolution hidden states of the downstream tokens with those from "43" sequence, where all other elements are identical except for the anchor pair. As shown in Fig. B. we found that the outputs of nearly all anchor pairs collapse to the output corresponding to the "43" anchor pair.
Taken together, these two experiments clearly demonstrate that Mamba solves the composite func- tion task primarily by leveraging convolution extract the necessary information. This lays the groundwork for Mamba's 's inability to reach symmetric solution.
1.00
1.00
Accuracy
0:48
0.40
0:36
0.96
(43) (12) (22) 1231 (44) (23) (22) (23) 241 (31) (3)) (33) 139) (411 (42) (94)
1311 (2)) 1231 1341 (23) 122) 1231 G4) 1331 1221 (33) (34) (62) (42) 1491
Anchor pair
Anchor pair
(a) Blocking
(b) Substitution
Figure 6: Accuracy of each anchor pair under blocking and substitution experiments compared to the original output. Figures (a) and (b) correspond to the blocking and substitution experiments, respectively. The x-axis denotes the anchor pairs, and the y-axis represents accuracy. Each anchor pair is evaluated using 480 randomly generated sequences.
4.2 Nonlinear convolution introduces asymmetry
In this section will examine that the asymmetry weight in convolution introduces asymmetry between symmetric anchor pairs.
We design a task with fully symmetric setting as follows. The symmetric anchor pairs have the same function but they not composite by elementary functions; only function "43" is unseen during the training. This leaves the symmetric solution as the only possible one. As shown in the Fig. Zi, we found that standard Mamba can achieve 100% the test data of "34" function and fails to learn "43" via symmetric property. Additional experimental results provided in the appendix.
To better isolate the effect of convolution's asymmetry, we remove this asymmetry by setting all convolutional kernel weights to 1. As shown in the Fig. Zb, once the asymmetry of convolution is eliminated, Mamba biases learning the symmetric solution to fit the composite task.
This indicates that Mamba's 's preference for asymmetry stems from the inherent asymmetry of its nonlinear convolution. In Mamba, sequence information fused through nonlinear convolution operation, which serves as the input to the SSM module. Consider two sequences, (v), V3, u4) and its reversed counterpart. For the convolution outputs of the above sequences, we define the final token of the result as follows:
original sequence: f =
+ 3
symmetric sequence: g C1 o Ua
+ + 0 + C4 Cuu1 +
the convolution parameters G; are not identical(3 the bias), then f and will generally differ. This means that even for token sequences that symmetric in content, their representations after convolution in Mamba can be significantly different. Such a discrepancy illustrates Mambas inherent asymmetry, fails to preserve the equivalence of symmetric inputs.
The root cause of this asymmetry lies in the non-uniformity of the convolution weights. Since the convolution parameters in Mamba are initialized randomly and trained independently, the ca values typically remain distinct throughout training. As a result, the convolution operation induces persistent asymmetry, where different token orders lead to different outputs. We examined the cosine similarity between the individual parameters of the convolution kernel at both the beginning and end of training, and found that they were largely orthogonal one another, indicating strong and persistent inconsistency throughout the training process. The detailed results can be found in the appendix.
For Mamba network with initialization rate 7 = 0.5, cannot learn composite function task by either composite function or symmetric function. We found that if positional encoding is explicitly

---

included, as shown in the Fig. Ze such Mamba network learns composite task by symmetric function. Therefore, positional encoding is also critical for learning symmetric solution.
34-test
43-test
IConvld.comp.
Standard.comp.
Pos..comp. Pos..ymm.
Standard,comp.
1Convid.sym.
Standard, sym.
Standard,sym.
1.00
1.00
0.8
Accuracy
0.75
Accuracy
0.50
0.50
0.25
(a)
0.25
(b)
0.2
(c)
0.00
100
200
100
150
Epoch
Epoch
Epoch
Figure 7: Test Accuracy across. (a): Fully symmetric setting, i.c. target functions are symmetric but do not composite by elementary functions, standard Mamba with y 0.5. (b): Accuracy computed based on groundtruth of composite solution (comp.) and symmetric solution (sym.) for Mamba with all-one convolution (IConvld) and standard Mamba (Standard structure) with initialization rate y = 1. (c): Similar legend as (b) for Mamba with positional encoding (Pos.) and the standard but with initialization 7 0.5. Details of the data and training setup be found in the appendix.
4.3 Transformer with convolution biases asymmetric solution
The empirical analysis in Mamba reveals that the convolution structure is key component to intro- duce asymmetry. Imagine convolution introduced into the Transformer, does it increase anchor asymmetry and push the model toward learning asymmetric composite solution? We insert lution after the input to the Transformer and before the attention module (applying to Q, K, and how Mamba applies nonlinear convolution before the SSM module. For
V). This is analogous detailed configurations, please refer to the appendix. Previous study(Zhang et all, ROPE(T) has shown that with different 7. Transformer can learn symmetric solution or composite solution in different regimes. As shown in Fig. E, with convolution component, Transformer either can not generalize for small y or fit data by symmetric solution for relative large yy this case, the Transformer exhibits a similar preference for composite function solutions as Mamba and struggles to learn symmetric solutions.
1.0
5.04
2.03
2.62
0:35
5.02
0.63 5:03
0.01
2.03 0.07
0.04 5.02
0.8
0.66
0.4
0.13
0.7
0.69
0.30
3.00
03
0.7
0.58
0.89
8.92
063
0:03
0.01
0.06
0.66
0.77
5.05
0.66
0.14
0.06
8.0
8.05
layer num
0.6
+
0.3
6.22
2344
0.5
8.38
5.00
0:63
6:02
0.11
5.02
5.05
8.02
0.4
8:3
8.14
041
0.03
0.11
0.77 2,62
073
672
0.63
0:57
0:37
043
8:05
033
0.09
0.67
0.04
5.01
0.07
0.66
0.07
685
8:07
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
1.0 1.5
2.0
0.1
0.2
0.3
0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.5
2.0
0.0
Y
Y
(a) Trans.+ conv, comp.
(b) Trans. conv, sym.
Figure 8: Phase Diagram of the Composite Function Task under Different Settings. Accuracy (color) for composite function task under different initialization rates (abscissa) and depths (ordinate). (a) and (b) show the composite and symmetric solution accuracy of the Transformer after adding non- linear convolution.

---

Mamba's challenges in the inverse sequence matching task
This raises critical question: Does the change in token order between sequences like 1234 and 4321 make it inherently difficult for Mamba to attend across reversed patterns and discover the correct reverse sequence?
As shown in Fig. La, for the standard Mamba network, the training accuracy can easily achieve 100%, however, the test accuracy for the case with tokens seen during training or the case with tokens unseen during the training ("OOD"), the network predicts the outcome with a random-guu gugsss level. Additional experimental results can be found in the appendix. For Transformer, or a Mamba network that adds a residual connection from the input of convolution to SSM and the position embedding, as shown in Fig. Eb and Bk, the network can accurately predicts test cases. In addition, for the "OOD" case, the accuracies of such two cases are also significantly larger than random guess. We also conducted additional architectural experiments aimed at mitigating Mamba's 's asymmetry bias. For details, please refer the appendix.
Train
Test
00D-Test
1.00
1.00
Accuracy
0.75
Accuracy
Accuracy
0.75
0.50
0.50
0.50
0.25
0.25
0.25
(a) Mamba
(b) Transformer
(c) Mamba modified
0.00
0.00
100
50
100
200
100
150
200
Epoch
Epoch
Epoch
Figure 9: Accuracy on the inverse sequence matching task across different model architectures. (a), (b), and (c) show the accuracy of Mamba, Transformer, and the modified Mamba with residual connection, respectively, on the inverse sequence matching task under the same settings. "OOD" refers set drawn from a distribution outside the training and standard test ranges.
Conclusion
In this work, we leveraged synthetic data to experimentally and systematically analyze the intrinsic properties the Mamba architecture. Our findings reveal fundamental differences between Mamba and Transformer models, particularly in handling symmetrical patterns and relationships. We identi- fied that Mamba's s inherent asymmetry bias stems from its nonlinear convolution mechanism, which fuses token information asymmetrically before passing it to the SSM module. This architectural con- straint limits Mamba's ability match reversed sequences. These insights provide valuable guidance for designing future sequence models that combine the computational efficiency of SSMs with flexible pattern recognition capabilities of Transformers.
to recognize symmetrical solutions in composite function tasks and to
Limitations
To enable precise investigation and clear illustration of Mamba's 's internal mechanisms and induc- tive biases, this work employs synthetic data rather than real-world datasets. While the synthetic tasks are designed to capture key characteristics of the Mamba architecture, they may not generalize the full diversity of real-world data. Moreover, due the use of synthetic data, the Mamba mod- els used in our experiments are relatively small. Whether the observed biases persist in larger-scale Mamba architectures remains an open question requiring further investigation. Additionally, many existing large-scale models incorporating Mamba do so in conjunction with Transformer compo- nents. In such hybrid architectures, it remains be studied whether Mamba's inductive bias still dominates or is diminished.
10

---

Acknowledgements
This work supported by the National Key R&D Program of China Grant No. 2022YFA 1008200. the National Natural Science Foundation of China Grant No. 92270001, 12371511, 12422119, 2025 Key Technology R&D Program "New Generation Information Technology" Project of Shanghai Municipal Science and Technology Commission, and the HPC of School of Mathematical Sciences and the Student Innovation Center, and the Siyuan-l cluster supported by the Center for High Per- formance Computing at Shanghai Jiao Tong University. and Key Laboratory of Marine Intelligent Equipment and System (Ministry of Education, P.R. China), and SJTU Kunpeng & Ascend Center of Excellence.
References
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention all you need, Advances in neural information processing systems 30 (2017).
P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, I.. Kaiser, N. Shazeer, Generating wikipedia by summarizing long sequences, arXiv preprint arXiv:1801. (2018).
J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.0480 (2018).
A. Radford, Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners (2019).
H. Touvron, Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302. (2023).
OpenAI, Gpt-4 technical report, 2023. ATX1V:2303 08774
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural informa- tion processing systems 33 (2020) 1877-1901.
Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, Z. Sui, A survey on in-context learning, arXiv preprint #xXiv23011.00234 (2022).
Garg, D. Tsipras, P. S. Liang, Valiant, What can transformers learn in-context? case study of simple function classes, Advances in Neural Information Processing Systems 35 (2022) 30583- 30598.
T. H. Trinh, Y. WU, Q. Le, H. He, T. Luong, Solving olympiad geometry without human demon- strations, Nature 625 (2024) 476-482.
A. Davies, Vclikovi, L. Buesing, S. Blackwell, D. Zheng, N. Tomaev, R. Tanburn, P. Battaglia, C. Blundell, A. Juhász, et al., Advancing mathematics by guiding human intuition with ai, Nature 600 (2021) 70-74.
C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al., In-context learning and induction heads, arXiv preprint arXiv:2209.1 11895 (2022).
Z. Wang, Y. Wang, 7. Zhang, 7. Zhou, H. Jin, T. Hu, J. Sun, 7. Li, Y. Zhang, Z.-Q. J. Xu, The buffer mechanism for multi-step information reasoning in language models, arXiv preprint arXiv:2405.15302 (2024).
R. Child, S. Gray, A. Radford, 1. Sutskever, Generating long sequences with sparse transformers, arXiv preprint arXiv: 1904. 10509 (2019).
A. Katharopoulos, A. Vyas, N. Pappas, F. Fleuret, Transformers are rans: Fast autoregressive transformers with linear attention, in: International conference on machine learning, PMIR, 2020, pp. 5156-5165.
11

---

A. Gu, T. Dao, Mamba: Linear-time sequence modeling with selective state spaces, The Conference on Language Modeling (2024).
T. Dao, A. Gu, Transformers ssms: Generalized models and efficient algorithms through struc- tured state space duality, arXiv preprint arXiv:2405.21060 (2024).
K. O'shea, R. Nash, An introduction to convolutional neural networks, arXiv preprint 1rXiv:1511.088
(2015).
Z. Zhang, Z. Wang, J. Yao, Z. Zhou, X. Li, Z. Xu, et al., Anchor function: a type of benchmark functions for studying language models, arXiv preprint arXiv:2401.08300 (2024a).
7. Zhang, P. Lin, 7. Wang, Y. Zhang, Z.-Q. J. Xu, Initialization is critical to whether transformers fit composite functions by inference or memorizing, arXiv preprint arXiv:2405 4,,00000999 (2024b).
Z. Zhang, P. Lin, Z. Wang, Y. Zhang, Z.-Q. J. Xu, Complexity control facilitates reasoning- compositional generalization in transformers, arXiv preprint Stii:::551.0083333 (2025).
A. R. Voelker, Dynamical systems in spiking neuromorphic hardware (2019).
A. Voelker, I. Kaji, C. Eliasmith, Legendre memory units: Continuous-time representation in recur- rent neural networks, Advances in neural information processing systems 32 (2019).
A. Gu, K. Goel, C. Re, Efficiently modeling long sequences with structured state spaces, in: In- ternational Conference on Learning Representations, 2022. URL: https:://ppenrevien net/ Yorun//ddeWWLFoz1vvuu
T. Smith, A. Warrington, S. Linderman, Simplified state space layers for sequence modeling. in: The Eleventh International Conference on Learning Representations, 2023. URL: https:
B. Peng, E. Alcaide, Q. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. Chung, L. Derczynski, X. Du, M. Grella, K. Gv, X. He, H. Hou, P. Kazienko, J. Kocon, Kong, B. Koptyra, H. Lau, J. Lin, K. S. I. Mantri, F. Mom, A. Saito, G. Song, X. Tang, J. Wind, S. Wožniak, Z. Zhang, Q. Zhou, J. Zhu, R.-J. Zhu RWKV: Reinventing RNNs for the transformer era, in: H. Bouamor, J. Pino, K. Bali (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics. Singapore, Findingg-nennnn 9367|
2023, pp. 11048-14077, URL: htttps://accaeeee httpps////allaachooooooooo////// doi:10 1188666/1//22.51111
em.
B. Peng, D. Goldstein, Q. G. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, T. Ferdinan, K. K. GV, H. Hou, S. Krishna, R. M. Jr., N. Muennighoff, F. Obeid, A. Saito, G. Song. H. Tu, R. Zhang, B. Zhao, Q. Zhao, Zhu, R.-J. Zhu, Eagle and finch: RWKV with matrix-valued states and dynamic recurrence, in: First Conference Language Modeling, 2024. URL: https: //openreview www.:eeeooormmmdiooolssit...
Y. Sun, L. Dong, S. Huang, S. Ma, Y. Xia, Xue, Wang. Wei, Retentive network: A successor to transformer for large language models, 2023. URL: attps://aaxiiis orwoooggggboo. aba/ 2307 08621.
arXiv:2307 08621
Yang, B. Wang, Y. Shen, R. Panda, Y. Kim, Gated linear attention transformers with hardware- efficient training, in: R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, F. Berkenkamp (Eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, PMLR, 2024, pp. 56501-56523, prwss/v235/yang222222 /
URL: https::/pproceedinggg mmr.
A. Ben- Kish I. Zimerman, S. Abu-Hussein N. Cohen, A. Globerson, L. Wolf, R. Giryes, Deci- mamba: Exploring the length extrapolation potential of mamba, in: The Thirteenth International Conference on Learning Representations, 2025. URL: https://openreview.nco wttp://wwwww..eeetoor........ IWS15Zy1jæ.
Z. Ye, K. Xia, Fu, X. Dong, J. Hong. X. Yuan, S. Diao, J. Kautz, Molchanov, Y. C. Lin, Longmamba: Enhancing mamba's long-context capabilities via training-free receptive field en- largement, in: The Thirteenth International Conference on Learning Representations, 2025. URL:
12

---

D. Yuan, J. Liu, B. Li, H. Zhang, J. Wang, X. Cai. D. Zhao, Remamba: Equip mamba with effective long-sequence modeling, 2025. URL: arxiv:2408 15496.
wwwp://www.rrg/b////0///0 15496,
J. Park, J. Park, Z. Xiong, N. Lee, J. Cho, S. Oymak, K. Lee, D. Papailiopoulos, Can mamba learn how learn? comparative study on in-context learning tasks, in: ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024. URL: https:/7 openreview wwwrrrevewwneee/ffoooo nwt/forumm arrumiid=vrOOHctaae
R. Waleffe, W. Bycon, D. Riach, B. Norick, V. Korthikanti, T. Dao, A. Gu, Hatamizadeh, Singh, D. Narayanan, G. Kulshreshtha, V. Singh, J. Casper, J. Kautz, M. Shoeybi, B. Catanzaro, An em- pirical study of mamba based language models, 2024, URL: hctps://arxiv www.iv.org/ab//24400
07887 ATX1V:2406 07887
S. Arora, S. Eyuboglu, A. Timalsina, I. Johnson, M. Poli. J. Zou, A. Rudra, C. Ré, Zoology: Measur- ing and improving recall in efficient language models, 2023. URL:
/bss/
2312.04927. arX1v:2312 244V22312.049777
S. Jelassi, D. Brandfonbrener, S. M. Kakade, E. Malach, Repeat after me: Transformers are bet- than state space models at copying, 2024. URL: https://arxiv. http:/ab//402...1111
2rk1
Z. Xu, J. Yan, A. Gupta, V. Srikumar, State space models are strong text rerankers, 2025. URL: http///xwwv.orr//////22222 14354, arXiv:2412. 14354
R. Ren, Z. Li, Y. Liu, Can mamba always enjoy the "free lunch"?, 2024. URL: https://arxiv. 2xg/abs/2410 03810, arkiv:2410.038110
J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, S. Shieber, Investigating gender bias in language models using causal mediation analysis, Advances in neural information processing systems 33 (2020) 12388-12401.
Jeoung, Diesner, What changed? investigating debiasing methods using causal mediation analysis, arXiv preprint arXiv:2206.0070) (2022).
K. Wang. A. Variengien, A. Conmy, B. Shlegeris, J. Steinhardt, Interpretability in the wild: a circuit for indirect object identification gpt-2 small, arXiv preprint (2ti:::2211000999) (2022).
A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, A. Earriga-Alonso Towards automated circuit discovery for mechanistic interpretability, Advances in Neural Information Processing Systems 36 (2023) 16318-16352.
Merullo, C. Eickhoff, E. Pavlick, Circuit component reuse across tasks in transformer language models, arXiv preprint arXiv:2310.08744 (2023).
T. Guo, W. Hu, S. Mei, H. Wang, C. Xiong, S. Savarese, Y. Bai, How do transformers learn in- context beyond simple functions? case study on learning with representations, arXiv preprint arXiv:2310. 10616 (2023).
M. Wang, E. Weinan, Understanding the expressive power and mechanisms of transformer for sequence modeling, arXiv preprint arXiv:2402.00522 (2024).
N. Amsel, G. Yehudai, J. Bruna, On the benefits of rank in attention layers, arXiv preprint 1XXiv2447.166133 (2024).
Z. Li, H. Liu, D. Zhou, T. Ma, Chain of thought empowers transformers to solve inherently serial problems, arXiv preprint a2Xiv:2402.12885 (2024).
X. Wang, A. Amayuelas, K. Zhang, L. Pan, W. Chen, W. Y. Wang, Understanding the reasoning ability of language models from the perspective of reasoning paths aggregation, arXiv preprint arXiv:2402.0322 03268 (2024).
M. Poli, A. W. Thomas, E. Nguyen, P. Ponnusamy, B. Deiseroth, K. Kersting, T. Suzuki, B. Hie, S. Ermon, C. Ré, et al., Mechanistic design and scaling of hybrid architectures, arXiv preprint arXiv:2403. 17844 (2024).
13

---

Z.-Q. J. Xu, Y. Zhang, T. Luo, Overview frequency principle/spectral bias in deep learning, Com- munications on Applied Mathematics and Computation (2025a) 827-864.
Z.-Q.. J. Xu, Y. Zhang, Z. Zhou, An overview of condensation phenomenon deep learning, arXiv preprint arXiv:2504.09484 (2025b).
Z. J. Xu, Y. Zhang, T. Luo, Y. Xiao, Z. Ma, Frequency principle: Fourier analysis sheds light deep neural networks, arXiv preprint arXiv:1901.0633 (2019).
M. Wang, H. He, J. Wang, Z. Wang, G. Huang. F. Xiong, Z. Li, L. Wu, et al., Improving general- ization and convergence by enhancing implicit regularization, arXiv preprint arXiv:2405.20766 (2024).
A. Jacot, F. Gabriel, C. Hongler, Neural tangent kemel: Convergence and generalization in neural networks, Advances in neural information processing systems 31 (2018).
A. Jacot, B. Simsek, F. Spadaro, C. Hongler, F. Gabriel, Implicit regularization random feature models, in: International Conference on Machine Learning, PMLR, 2020, pp. 4631-4640.
S. Arora, S. Du, W. Hu, Z. Li, R. Wang, Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks, in: International Conference on Machine Learning, PMLR, 2019, pp. 322-332.
Arora, N. Cohen, N. Golowich, W. Hu, convergence analysis of gradient descent for deep linear neural networks, arXiv preprint arXiv: 18 10.02281 (2018).
L. Wu, J. Su, The implicit regularization of dynamical stability in stochastic gradient descent, in: International Conference on Machine Learning, PMLR, 2023, pp. 37656-37684.
M. Wang, L. Wu, A theoretical analysis of noise geometry in stochastic gradient descent, arXiv preprint 10.0v:2310000622 (2023).
S. Arora, Z. Li, A. Panigrahi, Understanding gradient descent the edge of stability in deep learning, in: International Conference on Machine Learning, PMLR, 2022, pp. 948-1024.
Z. Li, S. Malladi, S. Arora, On the validity of modeling sgd with stochastic differential equations (sdes), Advances in Neural Information Processing Systems 34 (2021) 127 12712-12725.
Wu, Ma, et al., How sgd selects the global minima in over-parameterivee learning: A dynami- cal stability perspective, Advances in Neural Information Processing Systems 31 (2018).
Z. Zhu, J. Wu, B. Yu, L. Wu, J. Ma, The anisotropic noise in stochastic gradient descent: Its behav- ior escaping from sharp minima and regularization effects, arXiv preprint arXiv:1803.00195 (2018).
S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, R. Wang, On exact computation with an infinitely wide neural net, Advances in neural information processing systems 32 (2019).
Y. Ren, C. Ma, L. Ying, Understanding the generalization benefits of late learning rate decay, in: International Conference on Artificial Intelligence and Statistics, PMLR, 2024, pp. 4465 4473.
Luo, Q.-Q. Xu, Ma, Y. Zhang, Phase diagram for two-layer relu neural networks infinite- width limit, The Journal of Machine Learning Research 22 (2021) 3327-3373.
J. Feng, J. Steinhardt. How do language models bind entities in context?, arXiv preprint arXiv:2310.17191 (2023).
K. Meng, D. Bau, A. Andonian, Y. Belinkov, Locating and editing factual associations in gpt, Advances in Neural Information Processing Systems 35 (2022) 17359-17372.
B. Wang, X. Yue, Su, H. Sun, Grokked transformers implicit reasoners: A mechanistic to the edge of generalization, arXiv preprint arXiv:2405.1507 (2024).
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, our claims clearly reflect the contributions of this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We summarize the limitations of this work at the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This work does not present any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The dataset design, model architecture, and training configurations are de- scribed in detail, ensuring that the results can be reliably reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code necessary to reproduce the experiments in the supple- mentary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://miis.ce/
mub1:c//uiless/odeeebliissss oolio for more details.
While we encourage the release of code and data, understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.cc/pin Sttiii//guidee//oodeeeee cattssummmssiooooo...oo asionPolicyl for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide complete description of the training and testing details in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Although eтror bars are not shown, we conducted multiple runs and report averaged results; see Figures and 8.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The specific computational resources used in our experiments are described in detail in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification: This work meets the requirements of NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is an experimental study based on observed phenomena and does not have significant societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e-g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work is an experimental study based on observed phenomena, and there- fore does not require any safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All referenced work has been clearly and appropriately cited throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the pack- age should be provided. For popular datasets, ssperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: This work is experimental and maalyiis-basee study, and therefore does not produce any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: This work is experimental study based on observed phenomena, and there- fore does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work is an experimental study based on observed phenomena, and there- fore does not involve crowdsourcing or human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
This work does not involve anything beyond what stated in the declaration.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy (fte::////earrrrsccc/c//ooreeees for what should or should not be described.
21

---

A Experiments compute resources
All experiments were conducted on server running Ubuntu 22.04.4 LTS. The system equipped with an Intel Xeon Gold 6133 processor featuring 80 logical cores at 3.0GHz, and 352 GB of RAM. The machine is configured with 8 NVIDIA GeForce RTX 4080 GPUs with 16GB of video memory each.
B Detail of Mamba
The following provides a detailed explanation of the internal computations within the Mamba mod- ule, beginning with the specification of the corresponding dimensional representations.
B == Batch size,
S = Sequence length,
Dm = dimension of model,
Din = dimension of inner model
expand x Dm,
N == dimension of hidden state,
Num, = number of heads,
Ha = dimension of heads,
The computations within the Mamba module can be divided into three components: pre-SSM, SSM, and PSSt-SSM. The following presents detailed derivation for each submodule.
Input: E RBB S. Dm).
Pre-SSM
zxBCdl proje E (B, 2Dn 2NN
(2,xBC, dt) = BxBCdt E (BB (B, S,( Din DNN Nums))
B.BaC. = Convid(:BC) ER BB S, D2,)
Tef BefBesCe Cef SiLU(=BC) RB Dia+
(Eef, Bes, Ces) = Kef Bof E (B. Dim, N. N))
Amnit ~ Uniformm (Amin Amux)
E Numa
Alog log(Ainit) E RNuus
A exp (Alog) &RNuu.
dthmat = exp (rand(Nump)) (log(dt max)m_maax - log(dt_mmni) + log(dt_min)) dtbios = dtbinit + log (- exp (-dtbinit + +) E Numn
e Num.
fb softplus(ds (dt dtvias) (B. 8, Numa).
Ä = o dt (B, S, Num,).
0 = Tef reshapel B, S, Numa, Ha) di dess..eshapeeB. Numa,
(B, S, Numg. Ha)
B reshapo B, N) E (B, 8, 1,
= Cef res.apeeB..BSS... B,
L.
Here, Linear proj denotes linear transformation. Convld refers to one-dimensional convo- lution, and SiLU represents the SiLU activation function. Uniform indicates sampling from uniform distribution, while softplus denotes the Softplus activation function. rand refers to draw- ing a specified number of random values. The symbol denotes element- wise multiplication, and reshape indicates dimensional transformation operation.
22

---

SSM
A Maski repeat(A)
RB, Num,,
A Aummuum cumsum A)
E RR, Numa. S, 8)
cxp(Maskg Acumsum)
E ER Numa S.
P = einsum (" BSHH,, BSHN - BHSS" Ö, B)
E (R. 1, 1.
M einsum (" BHSS BHSS BHSS" P)
ERIB. Numa. 8.8)
= einsum (rauuummmHBBB BHSS,, BSHP - BSHP" M, C S. Numa, Ha)
The Mask, operation is used to zero out the diagonal and upper-triangular elements of a matrix, retaining only the elements below the diagonal. The Mask, sets the elements above the diagonal to negative infinity. The cumsum operation performs cumulative sum along the first of the last two dimensions (i.e., across rows) in matrix of shape (S,S), summing from top bottom. The einsum operation denotes the Einstein summation convention, used for concise and flexible tensor contractions.
Post-SSM
D 1m
E R Din,
ÿ = + of reshape(B,,). S, Num,, Ha) x D Dreshape(D.m:1) ÿ. reshappe(B, S, Di) E ER S, Dma),
E RBB S, Num,, Ha).
y= ğ S2 (.. s. Din).
E R S, Din)
Unorm Us /eaa ( axis = 1) +e R Dm).
W
yout
Linear proj (Umorm)
Here, 1 denotes a vector in which all elements are equal to I.
Output: yout E RR (B, 8, Dm).
C Data setup
C.1 Composite function task
Standard The total dataset comprises 300,000 samples, and each sequence has a fixed length of 8. Each anchor pair in the training set accounts for 5.6% of the total data, while each anchor pair in the test set constitutes 0.6%. The mapping between anchors and their associated functions is as follows: anchor 1 corresponds
shift of +55, anchor 2 +1, anchor 3 -2, and anchor 4 to -8. During training, all 15 possible anchor pairs are included except for pair 43. Among these, all pairs except 34 are derived from the composition of their corresponding single-anchor functions. Notably, the function for pair 34 is manually set to -6, deviating from the correct compositional result of -10. The test set contains both symmetric and compositional instances of pair 43. The distinction between training and test data is governed by the position of the key token. For sequences of length 8, each position associated with a congruence class modulo 8. In the training set, the key token is prohibited from appearing in the position whose modulo-8 class matches its own. For example, key 33 cannot appear in the first position, as it belongs to the congruence class mod which is aligned with the first index. Conversely, in the test set, each key token is required to occupy the position that corresponds exactly to its modulo-8 congruence class. This design ensures that the model has access to the full semantic range of tokens during training while preventing it from relying solely on positional memorization to generalize to the test set.
Full symmetry The total number of datasets is 300,000, and each sequence has a fixed length of 8. Each anchor pair the training set accounts for 4.5% of the total, while each anchor in the test set accounts for 0.5% of the total. To ensure that the only possible solutions symmetric,
23

---

we simultaneously utilize two sets of correspondences between anchors and functions. This setup prevents the model from simply solving for individual anchor functions and instead forces it derive symmetric solutions by understanding the symmetry of anchor pairs. There are a total of five anchors: 0, 1, 3, and 4. The 0 anchor is added to balance the data volume between the two function sets, thus avoiding model bias caused by disparities in data quantity. The first set of correspondences between anchors and functions is as follows: 0 corresponds to +2, corresponds to +5, 2 corresponds to + +1, 3 corresponds to 22, and 4 corresponds to -. The second set of correspondences is: 0 corresponds to 10- I corresponds to +6, 2 corresponds to -7, and 3 corresponds to +3. All anchor pair functions composed of either the first or the second set of correspondences. To facilitate the observation of symmetric solutions, anchor pairs 00, 11, 22, 33, and 44 are excluded from this task. Thus, there are total of 20 anchor pairs in this task. The anchor pairs following the first set of correspondences are 01, 02, 10, 20, 14, 41, 23, 32, 34, 43, a total of 10 pairs. Among them, there are zeros, ones, twos, 4 threes and 4 fours. This design ensures that each anchor has the same amount of data under the two corresponding methods, and the two corresponding methods also have the same amount of data. The anchor pairs following the second set of correspondences are 03, 30, 04, 40, 12, 21, 13, 31, 24, 42. Only 43 is not included in the training set. The purpose of this task is to observe whether the model can discover the symmetry among all anchor pairs and thus output symmetric solutions. The distinction between the training set and the test set in the dataset is based on the position of the key. For sequences of length 8, each position corresponds the training set, for each key, the key does not fall within the congruence class corresponding to its position. For example, the key 33 cannot appear in the first position of the sequence because it would then be in the congruence class modulo 8 of 1, which corresponds to that position. In contrast, in the test set, each key must exactly fall within the congruence class corresponding to its position. This setup allows the model to learn the meaning of all tokens while preventing it from relying solely on memorization to generalize to the test set.
congruence class modulo 8. In
C.2 Inverse sequence matching task
The total dataset consists of 100,000 samples. The sequence length varies with the number of layers in the Mamba model. Each sequence is generated from a generation set consisting of three distinct elements. It contains five different permutations of the elements in the generation set, where each permutation is followed by random token that does not belong to the generation set. Each such permutation is referred to as a key sequence.
One of these five sequences is randomly selected and reversed to form the query sequence, which is appended to the end of the original sequence. Between the key sequences and the query sequence, number of randomly generated tokensequal to the pure convolutional receptive field of Mamba are inserted to prevent the model from retrieving information through convolution. Therefore, the total sequence length can be formally expressed as follows:
S == 5 x (3 + 1) + x Num + 3.
Here, Numuaee denotes the number of layers in the Mamba model. For example, in the case of a two-layer Mamba model, the sequence length is 29. To ensure a fair comparison, Mamba and Transformer models with the same number of layers are trained on identical sequence configurations.
The training set constitutes 80% of the total dataset, while the test set and the out-of-distribution (OOD) set each account for 10%. In the training set, the three elements in the generating set do not all belong the same congruence class modulo 3. In contrast, in the test set, all three elements in the generating set belong to the same congruence class modulo 3. For both the training and test sets, all numerical values in the sequences are drawn from the range 20 100. In the OOD set, however, all sequence elements are drawn from the range 101 to 200, and the generation set is not subject to any congruence constraints.
D Training setup
Unless otherwise specified, all tasks this work adopt the following training parameter settings. The learning rate is initially set to le-5 at the start of training, warmed up to 25 times its initial value within 10 epochs, and then decreases to le-5 via cosine decay at 200 epochs. The training optimizer is AdamW with parameters set as 31 0.9, B2 0.999, eps 1e 8, and weight 1eca
24

---

Meanwhile, gradient clipping is applied with maximum norm of 1. For composite function tasks, the batch size is set 2048, while for inverse sequence matching tasks, the batch size is 1024. The loss function used for training is the cross entropy loss function, which is calculated only for the last token of the model output sequence.
E
Experiment detail and result
E.1
Composite function task
E.1. Phase diagram of Mamba the composite function task
Mamba configurations The dimension of model is set to 32, the dimension of hidden state is set to the default value of 128, and the expansion factor is set to the default value of 2, and the activation function employed throughout the model is the Sigmoid Linear Unit (SiLUU.. To facilitate clear observation, all experiments are conducted using a single head. 'he convolution kernel length its default value of 4. The values reported at different positions represent the mean results obtained using three fixed random seeds. Across these positions, the only variations are in the model's depth initialization.
E.1.2 Experiments with all-one convolution
Mamba configurations To investigate whether the asymmetry of convolution is responsible for Mamba's difficulty in learning the symmetric solution, we select a model configuration under which Mamba successfully learns the compositional solution but fails to learn the symmetric one. Specifi- cally, we use five-layer Mamba model with small initialization(y = 1), while all other settings are consistent with those used in the phase diagram experiments.
E.1.3 Experiments with positional encoding
Mamba configurations To examine whether Mamba's bias for asymmetry in the composite function task arises from its lack of explicit positional encoding, thus encouraging reliance on convolution. We select configuration under which Mamba struggles to learn both the posite and symmetric solutions. Specifically, we use a two-layer Mamba model with standard initialization(y 0.5). All other settings are kept consistent with those used in the phase diagram experiments.
E.1.4 Experiment under full symmetry
Mamba configurations To validate Mamba's difficulty solving the composite function task under fully symmetric settings, set dimension of model to 128, while keeping all other settings consistent with those used in the phase diagram experiments. The number of layers varied across 3, 4, 5, and 6, and standard initialization(? 0.5) is applied.
Extended results For each configuration, experiments are conducted using three fixed random seeds. The results are shown in the Fig. II and Fig. II As can be observed, under all configurations, Mamba consistently struggles to solve the composite function task in the fully symmetric setting.
E.1.5
Cosine similarity among convolution weights
Mamba's asymmetry bias originates from its nonlinear convolution, specifically from the asymme- try of its convolutional kernel parameters. To investigate this asymmetry, we examined the cosine similarity between the convolutional kernels at the beginning and the end of training, as shown the Fig. IZ and found that they are nearly orthogonal. This indicates that the convolutional parameters exhibit strong asymmetry throughout the training process.
E.2 Inverse sequence matching task
Mamba configurations The dimension of model is set to 128, the dimension of hidden state is set to the default value of 128, and the expansion factor is set to the default value of 2. The activation
25

---

2.0
0.4
Layer num
106
1.5
1.55
4
2.4
0.0
2.4
3
0.2
26
200
25
100
200
106
120
203
3.5
1.5
2
1
2
3
Random seed
Figure 10: Accuracy of Mamba under fully symmetric setting. The horizontal axis represents the random seed, while the vertical axis corresponds to the number of layers in the Mamba model.
function used SiLU. clearly isolate the structural differences between Transformer and Mamba and ensure fair comparison, the Transformer model also adopts SiLU as its activation function. The convolution kernel length is set its default value of 4, and the number of heads is fixed at
Transformer configurations The dimension of model is set to 128, consistent with the Mamba configuration. To ensure a fair comparison, the Transformer also adopts a 2x dimensional expansion when generating value vectors, mirroring Mambas setup. Specifically, the dimensions of the query and key vectors are set to 128, while the value vectors have dimension of 256. Additionally, to ensure a fair comparison and maintain a comparable number of parameters between the two models, the Transformers feedforward network (FNN) uses hidden dimension of 128, and, like Mamba, it configured with only a single attention head throughout.
Extended results All experiments are conducted using the same set of random seeds ensure fairness in comparison.
26

---

23"
10"
10
59
50
144
121
5
511
39"
107
Layer num
4
80
g1c
810"
30
16"
30'
22
101
147
3
g'0
10'
12-
437
101
300
350
10
121
39
2
g00
50
38
15"
187
230
150
230
184
11c
30c
380
190
250
2
3
Random seed
Figure 11: Loss of Mamba under fully symmetric setting. The horizontal axis represents the random seed, while the vertical axis corresponds the number of layers in the Mamba model.
To evaluate Mambas difficulty in solving the inverse sequence matching task, we vary the model depth across 3, 4, and 5 layers, and consider different initialization strategies ranging from stan- dard to small initialization(? = 0.5, 0.6, 0.7, 0.8, 0.9, 1.0). The detailed results are shown in the Fig. T and Fig. [e] can be observed that Mamba fails to solve the inverse sequence matching task under nearly all configurations.
For the two-layer Transformer, results under different initialization schemes(7> 0.5, 0.6, 0.8, 0.9, 1.0) are shown in the Fig. DS and Fig. Ia It can be seen that even with small number of layers, the Transformer outperforms Mamba.
The results of the modified two-layer Mamba under different initialization schemes(> 0.5, 0.6. 0.7, 0.8, 0.9. 1.0) are shown in the Fig. IS and Fig. IIA. It can be observed that the mod- ified Mamba not only significantly outperforms standard Mamba, but also substantially surpasses the performance of the Transformer.
27

---

1.00
1.00
0.75
0.75
0
0.50
0.25
0.25
1
1
0.00
0.00
2
0.11
-0.02
1.00
0.09
2
0.01
0.18
1.00
0.10
-0.25
-0.25
-0.50
-0.50
3
-0.11
0.05
0.09
1.00
3
0.02
0.01
0.10
1.00
-0.75
-0.75
2
3
0
-1.00
-1.00
(a) Layer 0 Epoch 0
(b) Layer 0 Epoch 209
1.00
1.00
0.75
0.75
0
-0.10
0.09
0.12
0
0.50
1
-0.10
1.00
0.03
0.03
0.25
1
0.25
0.00
0.00
2
0.09
0.03
1.00
-0.01
2
0.13
0.35
1.00
0.27
-0.25
-0.55
-.55
-0.50
3
0.12
0.03
-0.01
1.00
3
0.21
0.18
0.27
1.00
-0.5
-0.75
0
1
2
3
0
2
11.00
1.00
(c) Layer 1 Epoch 0
(d) Layer 1 Epoch 209
Figure 12: Cosine similarity convolutional kernel parameters between epoch epoch 209 for the two-layer Mamba model. The model configuration is consistent with that used in the phase diagram experiments. this figure, Mamba configured with two layers and initialized with
Experiments with alternative architectural modifications From experiments, observe that Mamba's ' bias toward asymmetry originates from the inherent asymmetry introduced by its nonlinear convolution. However, by introducing residual connection that bypasses this nonlinear convolution, the impact of such asymmetry can be effectively mitigated. This allows Mamba to leverage the SSM module for information extraction and even achieve performance comparable or surpassing that of Transformers on the inverse sequence matching task.
It is important to emphasize that, in order for the SSM to extract information in manner similar to Attention, positional awareness of tokens essential. Therefore, positional embedding must be incorporated. In fact, the addition of positional embedding plays significant role in the inverse sequence matching task as well.
In addition to introducing the residual connection that bypasses the nonlinear convolution, we also conducted another set of architectural experiments. Inspired by the original Mamba architecture, another possible way to inject the raw token information is through "gating mechanism". For example, one might use pointwise product between the original token and the fused token (after nonlinear convolution) as the imput to the SSM. We additionally conducted multiple experiments
28

---

0.5
0.6
1308
0.7
0.8
15c
0.9
1.0
2
3
5
Layer num
Figure 13: Accuracy Mamba under different configurations on the inverse sequence matching task. The horizontal axis represents the number of layers in the Mamba model, while the vertical axis corresponds to the initialization scheme.
this approach, and the results are shown in Fig. MА The following presents the computation process of the gating mechanism. Given the input U to the Mamba block, we have:
(Ù, (ú,2,d)) dt) Linear (U), UE R(add) - E R(e.20+2)) 7 R(s.2d) di E R((.Na),
(B, (, o(Convld(U)), B Blsh) C ((ss)) X R(a2d)
(10)
(B, (B.C.X)oū BEROM p R(aM) X E R(s22).
Finally, B, Ò, and X serve the input to the SSM.
29

---

0.5
0.6
0.7
0.8
0.9
1.0
2
3
4
5
Layer num
Figure 14: Loss of Mamba under different configurations on the inverse sequence matching task. The horizontal axis represents the number of layers in the Mamba model, while the vertical axis corresponds to the intialiiztiin scheme.
30

---

0.5
04
0.6
04
Tacch
Exchk
0.7
04
12
Tacch
0.8
23.2
22
159
Park
0.0
0.9
153
252
Paak
1.0
Tach
Mamba
Transformer
Mamba-modified
Figure 15: Accuracy of Mamba, Transformer, and modified Mamba on the inverse sequence match- ing task. Left: Mamba (2-layer), Center: Transformer (2-layer), Right: Modified Mamba (2-layer); all under varying initialization schemes.
31

---

30"
0.5
8:0-
166
150
50
150
3.00
137
337
30"
0.6
30
0.7
30"
Dech
Cmch
0.8
100
140
0.9
307
1.0
12och
Cocce
Mamba
Transformer
Mamba-modified
Figure 16: Loss of Mamba, Transformer, and modified Mamba on the inverse sequence matching task. Left: Mamba (2-layer), Center: Transformer (2-layer), Right: Modified Mamba (2-layer); under varying initialization schemes.
32

---

10
10
10
06
0.4
10
10"1
50
200
Esoch
101
10
10
10"
5 10
0.
10-
0.2
10-4
1074
35
200
100
150
Esoch
1.0
101
10
10-4
0
0.2
130
200
160
150
200
Epuch
1.0
10
10"
0.8
10"1
5 10
200200000000
15
10
0.0
50
150
130
200
100
150
Esoch
Fooch
2.0
301
10
0.0
0.0
10a
10-
3..36....000
10-4
0.0
50
130
130
200
100
Esoch
101
109
1063110630000 cOn
0.0
10a
10-
15"
0.0
50
100
130
200
Epoch
Esoch
Figure 17: Accuracy (left) and loss (right) curves of Mamba with gate residual connections the inverse sequence matching task. The architecture of gete-ues Mamba is consistent with that of residual Mamba. From top to bottom, the initialization is set 7 0.5, 7 0.6, 7 0.7, 7 0.8. = 0.9, and = 1.0.
33


Reviews:
[{"review_id": "aGHVHulZOI", "reviewer": "Reviewer_UvAr", "summary": "This paper investigates the fundamental limitations of Mamba, a State Space Model (SSM) proposed as an alternate to transformers with linear complexity for long sequences. Through a series of synthetic tasks, the authors reveal that Mamba's convolution module introduces asymmetry bias. The bias impairs the recognition of symmetrical patterns, causing it to struggle with tasks requiring reversed sequence matching, and to favor compositional solutions. This work demonstrates that this limitation may stem specifically from the nonlinear convolution preceding the SSM module, rather than the SSM itself.", "questions": "- How would the inherent asymmetry affect language modeling tasks? What could it mean in the context of language modeling?\n- What other modifications in the architecture could potentially address the asymmetry beyond residual connections?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The limitation of Mamba is clearly demonstrated.\n- The synthetic tasks are meticulously designed to demonstrate the limitations and inherent biases of Mamba.\n- The root cause of the limitation is identified.\n- Suggests a simple architectural change that leads to significant performance improvements.\n\nWeaknesses:\n- Lacks explanation on why Conv1d followed by a non-linearity inherently introduces the asymmetry, especially in Mamba architecture.", "was_revised": true, "final_justification": "I believe this work provides a simple yet effective insight that could lead to a better understanding of the Mamba architecture. While the setup in the paper could be limited, the experiments in the rebuttal verify that this analysis could potentially be extended to language modeling tasks, adding to its value.", "rebuttal": "We are deeply grateful for your recognition of our work. Your appreciation has been a great source of encouragement for us to continue pursuing this line of research, and your insightful suggestions have been invaluable in helping us improve and refine our work. We are more than willing to refine it in accordance with your evey suggestion.\n\n**W1:**\n\nWe sincerely appreciate your valuable suggestions. In accordance with your advice, we will add the corresponding explanations to Section4.2 and include additional experimental results and clarifications in the appendix to elucidate why the nonlinear convolution introduces asymmetry. The following is our explanation, based on which the main text will be revised accordingly.\n\nIn Mamba, sequence information is fused through a nonlinear convolution operation, which serves as the input to the SSM module. Consider two sequences, $(v_1, v_2, v_3, v_4)$ and its reversed counterpart $(v_4, v_3, v_2, v_1)$. For the convolution outputs of the above sequences, we define the final token of the result as follows:\n\n> **original sequence:** $f = c_1\\circ v_1 + c_2\\circ v_2 + c_3\\circ v_3 + c_4\\circ v_4 + \\beta$\n\n> **symmetric sequence:** $g = c_1\\circ v_4 + c_2\\circ v_3 + c_3\\circ v_2 + c_4\\circ v_1 + \\beta$\n\nIf the convolution parameters $c_i$ are not identical, then $f$ and $g$ will generally differ. This means that even for token sequences that are symmetric in content, their representations after convolution in Mamba can be significantly different. Such a discrepancy illustrates Mamba’s inherent asymmetry, as it fails to preserve the equivalence of symmetric inputs.\n\nThe root cause of this asymmetry lies in the non-uniformity of the convolution weights. Since the convolution parameters in Mamba are initialized randomly and trained independently, the $c_i$ values typically remain distinct throughout training. As a result, the convolution operation induces a persistent asymmetry, where different token orders lead to different outputs. We examined the cosine similarity between the individual parameters of the convolution kernel at both the beginning and end of training, and found that they were largely orthogonal to one another, indicating a strong and persistent inconsistency throughout the training process.\n\nMoreover, as shown in the main text, when we manually set every  $c_i$ to an all‑ones vector, the differences vanish; the convolution becomes insensitive to token order, and $f$ equals $g$. Once the convolution ceases to introduce asymmetry, as shown in the experiment, Mamba—under the same experimental settings—no longer favors the composite solution but instead learns the symmetric solution.\n\nThe detailed convolution computation process in Mamba, for an input $V$, to obtain the convolutional output $O$ can be expressed as follows:\n\n$V = (v_1, v_2, ..., v_n),~~~ V\\in R^{(n,m)}, v_i\\in R^m,$\n\n$C = (c_1, c_2, c_3, c_4),~~~ C\\in R^{(4,m)}, c_i\\in R^m,$\n\n$p_i = c_1\\circ v_{i-3} + c_2\\circ v_{i-2} + c_3\\circ v_{i-1} + c_4\\circ v_{i},~~~ p_i\\in R^m,$\n\n$o_i = p_i + \\beta,~~~ o_i\\in R^m, \\beta\\in R^m$\n\n$v_j = \\vec{0}\\in R^{m},~~~ j \\leq 0,$\n\n$O = \\textbf{Conv1d}(V) = (o_1, o_2, ..., o_n),~~~ O\\in R^{(n,m)}$\n\nWhere $C$ denotes the convolution kernel parameters; $\\beta$ is the convolution bias. The four vectors of $C$ are multiplied point‑wise with the corresponding $v_i$ at each position. When the index of $v_i$ is smaller than 1, $v_i$ is set to 0 as padding. \n\n**Q1:**\n\nWe are truly grateful for your question, which has provided valuable guidance for improving our work. \n\nSymmetric relationships are common in real-world tasks—for example, the rotational invariance implied by the word and in natural language. To investigate how Mamba’s inherent asymmetry affects language tasks, we concentrate on tasks that fundamentally rely on symmetric relational structures.\n\nIn order to respond to your inquiry, we converted the inverse sequence matching task into a natural‑language format and carried out additional experiments, thereby providing further evidence for the conclusions in our work. Note that, this task is very similar to a widely used task \"Passkey\" (arxiv: 2305.16300). The experimental results reveal the impact of Mamba’s inherent asymmetry on its performance in language modeling tasks. We first present the main results and findings; a complete task example and a detailed explanation follow below.\n\nTo observe the effect of Mamba’s inherent asymmetry in language modeling with truly practical setting, we conducted experiments on two models: Hunyuan‑TurboS (arxiv: 2505.15431), a well-known large language model with 57 Mamba layers, and DeepSeek-V3, a Transformer-based model with comparable parameter scale and performance. The details and results of the experiment are as follows:\n- **3** rounds of testing, each comprising **100** randomly generated questions.\n- **Average accuracy:** Hunyuan TurboS = **17.6\\%** (19\\%, 17\\%, 17\\%), DeepSeek-V3 = **52.3\\%** (47\\%, 56\\%, 54\\%).\n\nThese results, obtained in a real-world application scenario with large-scale models and natural language inputs, are consistent with the findings from our paper based on small models and simplified data, thus further validating our conclusions. This demonstrates that the asymmetry bias inherent to Mamba, as highlighted in our work, not only significantly impacts small models but remains evident in large language models as well. It suggests that this asymmetry makes Mamba less suitable for natural language tasks that rely heavily on symmetric relations.\n\nExample of the Passkey task used in our test:\n\n> **{%), )%{, %{), %){, ){%**,f...(100tokens)...Each symbol combination above is separated by commas. Identify which combination has a reverse that equals **%{)** and output that combination along with the order in which it appears. The symbol combinations are listed before the block of letters above. Perform only one match and output the result directly—do not think. You are not allowed to think step by step, just output the result directly.\n\n**Remarks**:\n- We replaced the numeric sequences in the original task with combinations of symbols to avoid the influence of tokenization effects.\n- The task input consists of five uniquely ordered symbol combinations followed by 100 randomly generated letters, which serve to prevent Mamba from accessing information directly via convolution. \n- The final prompt is designed to clearly convey the task objective to the model while discouraging it from relying on chain-of-thought reasoning, in order to more closely replicate the conditions under which the small-scale models were tested in our paper.\n\n**Q2**\n\nWe sincerely appreciate your question, which has led us to a deeper investigation and motivated us to conduct further experiments. The inherent asymmetry of Mamba arises from the asymmetry of its convolution operation. To mitigate this issue, one can introduce additional channels for information flow, thus avoiding the original design of Mamba that fuses all information solely through nonlinear convolution.\n\nResidual connections seem to be the most straightforward solution to introduce additional channels. Such connections allow the SSM in Mamba to access both the asymmetrically fused token representations and the original, unfused token information, enabling the model to adaptively choose the appropriate source based on the task. This could help mitigate the limitations caused by the inherent asymmetry, while still preserving the unique characteristics of Mamba.\n\nIn addition, inspired by the original Mamba architecture, another possible way to inject the raw token information is through a \"gating mechanism\". For example, one might use a pointwise product between the original token and the fused token (after nonlinear convolution) as the input to the SSM. We additionally conducted multiple experiments on this approach, and found that it also enables Mamba to achieve 100\\% accuracy on the test set of the reverse sequence matching task. However, its out-of-distribution (OOD) accuracy is slightly lower than that achieved with residual connections. Under standard initialization, the results are as follows:\n\n| Model          | Train Acc. | Test Acc. | **OOD Acc.** |\n| - | - | -| - |\n| Mamba          | 100.0%     | 20.7%     |**19.6%**      |\n| Residual-Mamba | 100.0%     | 100.0%     | **63.6\\%**     |\n| Gating-Mamba | 100.0%     | 100.0%    | **42.9\\%**     |\n\nand the setup is identical to that described in Figure 9 of the original paper.\n\nThis may be due to the fact that, compared to residual connections, this method leads to a deeper entanglement between the original token information and the fused representation produced by the nonlinear convolution. In contrast, residual connections simply add the original and fused token representations. If these two types of information reside in different subspaces, the additive structure still allows for the preservation and independent extraction of each type of information.\n\nThe following presents the computation process of the gating mechanism. Given the input $U$ to the Mamba block, we have:\n\n$( \\tilde{U}, Z, dt) = \\textbf{Linear}(U),~~~ U\\in R^{(s,d)}, \\tilde{U}\\in R^{(s,2d+2h)},Z\\in R^{(s,2d)}, dt\\in R^{(s,N_h)},$\n\n$(B, C, X) = \\sigma(\\textbf{Conv1d}(\\tilde{U})),~~~ B\\in R^{(s,h)}, C\\in R^{(s,h)}, X\\in R^{(s,2d)},$\n\n$(\\tilde{B}, \\tilde{C}, \\tilde{X}) = (B, C, X)\\circ \\tilde{U},~~~ \\tilde{B}\\in R^{(s,h)}, \\tilde{C}\\in R^{(s,h)}, \\tilde{X}\\in R^{(s,2d)},$\n\nFinally, $\\tilde{B}$, $\\tilde{C}$, and $\\tilde{X}$ serve as the input to the SSM."}, {"review_id": "HDuUosxaMx", "reviewer": "Reviewer_mJ4r", "summary": "This paper studies limitations of the Mamba architecture on synthetic tasks, identifying that Mamba struggles to learn symmetrical patterns in sequential data. Through two synthetic tasks, a composite function task and an inverse sequence matching task, the authors demonstrate that Mamba exhibits a strong bias toward compositional solutions while failing to recognize symmetric patterns and relationships. The authors localize the problem to the nonlinear convolution in Mamba layers, which fuses token information in an asymmetric manner before passing it to the State Space Model (SSM) module. Through systematic analysis, they show that the limitation stems not from the SSM itself but from how token information is processed beforehand. The authors demonstrate that adding a residual connection that bypasses the convolution and writes directly to the SSM module can significantly improve performance on symmetry-requiring tasks, providing both mechanistic insights into Mamba's architectural biases and a concrete solution for improving its capabilities.", "questions": "- Where is the convolution operation added in the Transformer in section 4.3?\n\n- Please move figure 5 below figure 4.\n\nTypos:\n\nl157: missing a space between after \"Mamba\"\n\nl164: \"is initializeda Gaussian\" -> \"is initialized as a Gaussian\"\n\nl221: \"There\" -> \"This\"\n\nl226: \"predict\" -> \"predicts.\"\n\nl229: \"network\" -> \"the network\"", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper provides excellent visual aids, particularly the architectural diagrams and task illustrations, which effectively communicate complex concepts.\n\n- The composite function task and inverse sequence matching task are cleverly constructed to isolate and expose specific architectural biases, providing controlled environments that clearly reveal Mamba's symmetry-related limitations.\n\n- The authors employ causal interventions including zero ablation (information blocking) and activation patching (information substitution) to localize the source of Mamba's limitations, providing strong causal evidence for their claims.\n\n- The experiment modifying the convolutional weights effectively removes the asymmetric bias and demonstrates a clear causal relationship between convolution operation and the observed behavioral patterns.\n\nWeaknesses\n\n- The paper lacks some details necessary to understand and replicate the analysis. For instance, it is unclear how the \"attention score\" inside the SSM is computed for the information flow analysis, and the authors do not specify where exactly the convolution operation is added in the Transformer architecture in section 4.3.\n\n- The study relies exclusively on synthetic tasks with relatively small models, making it unclear whether these asymmetry biases persist in large-scale pre-trained Mamba models.\n\n- While the synthetic tasks effectively demonstrate limitations in Mamba architecture, it is unclear whether these limitations meaningfully impact Mamba's performance on language modeling or downstream tasks.", "was_revised": true, "final_justification": "My original concerns on its applicability to large scale pre-trained models have been addressed in the rebuttal. I believe the paper makes a meaningful contribution, and my rating reflects my support for its publication.", "rebuttal": "Your feedback has been both illuminating and inspiring, and we sincerely hope to continue learning through such meaningful exchanges.\n\n**W1:**\n\nWe sincerely thank you for pointing out the shortcomings in our writing. Following your valuable suggestion, we will make the following revisions:\n\n**(1)** In **Section 3.1** and the **caption of Figure 5**, we will add a clear explanation of the computation in the SSM and the definition of attention scores according to the explanation below;\n\n**(2)** In **Section 4.3** and the **Appendix**, we will include details on how to incorporate nonlinear convolution into a Transformer architecture according to the explanation below.\n\n**Attention score:** The attention score from token $j$ to token $i$ is given by the $(i, j)$ entry of the SSM matrix $S$.  due to the structural similarity between Mamba’s SSM module and the attention mechanism. Specifically, $S$ comprises both a learnable mask ($Mask$) and the term $CB^{\\top}$, which is analogous to the $QK^{\\top}$ component in attention. \n\nThe detailed computation has been provided in the Appendix. For your convenience, we also present the full computation process of Mamba involving the SSM matrix below. Omitting trivial dimension transformations and setting the batch size to 1 to omit the batch dimension, for a given input \n$U$ to the block, the internal computation process to obtain the output \n$O$ can be described as follows:\n\nPre-SSM\n\n$( \\tilde{U}, Z, dt) = \\textbf{Linear}(U),~~~ U\\in R^{(s,d)}, \\tilde{U}\\in R^{(s,2d+2h)},Z\\in R^{(s,2d)}, dt\\in R^{(s,N_h)},$\n\n$(B, C, X) = \\sigma(\\textbf{Conv1d}(\\tilde{U})),~~~ B\\in R^{(s,h)}, C\\in R^{(s,h)}, X\\in R^{(s,2d)},$\n\nSSM\n\n$Mask = \\textbf{F}(A_0, dt),~~~ A_0\\in R^{N_h}, Mask\\in R^{(N_h, s, s)},$\n\n$I = \\textbf{Repeat}(C B^{\\top},N_h)),~~~  I\\in R^{(N_h, s, s)},$\n\n$S = Mask\\circ I,~~~  S\\in R^{(N_h, s, s)},$\n\n$Y = SX + X,~~~ Y\\in R^{(N_h, s, 2d/N_h)},$\n\nPost-SSM\n\n$Y_{Norm} = \\textbf{RMS}(Y\\circ\\sigma(Z)),~~~ Y_{Norm}\\in R^{(s, 2d)},$\n\n$O = \\textbf{Linear}(Y_{Norm}),~~~ O \\in R^{(s, 2d)},$\n\nwhere \n> - $S$ : SSM matrix,\n> - $s$ : sequence length,\n> - $d$ : hidden‑state dimension,\n> - $h$ : SSM hidden dimension,\n> - $N_h$ : number of SSM heads,\n> - $\\textbf{Linear}$ : linear transformation,\n> - $\\textbf{Conv1d}$ : one‑dimensional convolution,\n> - $\\sigma$ : nonlinear activation function,\n> - $\\textbf{F}$ : function generates the $Mask$,\n> - $\\textbf{Repeat}$ : dimension‑replication operation,\n> - $\\circ$ : pointwise multiplication, \n> - $\\textbf{RMS}$ : RMS normalization.\n\n**Transformer with convolution (which also mentioned in Question 1):** We insert a convolution after the input to the Transformer and before the attention module (applying it to $Q$, $K$, and $V$). This is analogous to how Mamba applies nonlinear convolution before the SSM module. Ignoring the batch size and number of heads for clarity, the process of computing the attention output $O$ from input $U$ can be described as follows:\n\n\n$[Q; K; V] = \\textbf{Linear}(U),~~~ U\\in R^{(s, d)}, Q\\in R^{(s, d_q)}, K\\in R^{(s,d_k)}, V\\in R^{(s,d_v)}$\n\n$[Q; K; V] \\leftarrow \\textbf{Conv1d}[Q; K; V].$\n\n**W2 & W3:**\n\nWe sincerely thank you for your valuable suggestion, which has helped us further improve our work. Following your suggestions, we conducted experiments in a practical setting using commercially deployed large language models. The results strongly support the conclusions presented in our paper, indicating that the asymmetry bias observed in Mamba persists even in large-scale models.\n\nWe reformulated the inverse sequence matching task into a natural language setting,  which bears resemblance to the well-known \"Passkey\" (arxiv: 2305.16300), highlighting its practical significance in real-world language understanding scenarios. We performed 3 rounds of experiments on each of two models: **Mamba based** Hunyuan‑TurboS (arxiv: 2505.15431) and **Transformer based** DeepSeek-V3, with each round consisting of 100 randomly generated questions. These two models have comparable parameter scales and overall performance. The results of the experiment are as follows:\n\n| Model          | 1st. round Acc. | 2nd. round Acc. | 3rd. round Acc. | Avg. Acc. |\n| - | - | -| - | - |\n| Hunyuan‑TurboS          | 19\\%     | 17\\%     |17\\%      |17.6\\%      |\n| DeepSeek-V3 | **47\\%**     | **56\\%**     | **54\\%**     | **52.3\\%**     |\n\nThe results show a clear performance gap on the inverse sequence matching task and are consistent with our earlier findings on small-scale models. This highlights that the inverse sequence matching task remains a challenge for Mamba-based models and clearly reflects the persistence of Mamba's asymmetry bias in large language models. \n\n **Example of the task:**\n\n> **{%), )%{, %{), %){, ){%**,f...(100tokens)...Each symbol combination above is separated by commas. Identify which combination has a reverse that equals **%{)** and output that combination along with the order in which it appears. The symbol combinations are listed before the block of letters above. Perform only one match and output the result directly—do not think. You are not allowed to think step by step, just output the result directly.\n\n**Q2:**\n\nWe greatly appreciate your suggestion regarding the layout, and we will revise the formatting accordingly by positioning Figure 5 below Figure 4.\n\n**C1:**\n\n**line 157:** For example, in the illustration, six tokens correspond to the pure convolutional receptive field of a two-layer Mamba($2\\times3=6$). \n\n**C2:**\n\n**line 164:** A parameter $W\\in R^{d_1\\times d_2}$ is initialized as a Gaussian distribution $N(0,(1/d_{1}^\\gamma)^2)$, where $\\gamma$ is called initialization rate.\n\n**C3:**\n\n**line 221:** This raises a critical question: ...\n\n**C4:**\n\n**line 226:** ..., the network predicts the outcome with a random-guess level.\n\n**C5:**\n\n**line 229:** ..., the network can accurately predict test cases."}, {"review_id": "z9Fig16XVM", "reviewer": "Reviewer_2GPi", "summary": "This paper studies the Mamba architecture by testing it on a designed synthetic tasks. \nThe authors show that Mamba generalizes to learn composite and not symmetric solutions in that task. They identify the source of this generalization as the nonlinear convolution layer. The paper compares Mamba to Transformers and suggests a simple change (adding a residual connection) that helps Mamba learn the symmetric solution", "questions": "Could you clarify the blocking mechanism used in your experiments? The description in the paper is too brief to understand how information was actually blocked. There has been related work (such as Endy et al., \"Mamba Knockout for Unraveling Factual Information Flow\") that explores more targeted and interpretable blocking methods in Mamba-based models. It would be helpful if you could explain whether your method aligns with or differs from such approaches.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 1, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* The paper focuses on a specific synthetic problem and carefully investigates how Mamba performs on it.\n\n**Weaknesses**\n* I was not convinced that the proposed synthetic tasks are meaningful or impactful for the wider research community. The paper does not make a strong case for why these tasks matter, either in terms of practical relevance or future research directions, moreover they don't explain why learning a composite solution is bad. If this is based on earlier work, the authors should clearly restate why these tasks are important.\n* Some parts of the text are unclear. For example, Section 3.1 (Introduction of Mamba) and Section 3.3 are hard to understand on their own. The figures are missing captions, which makes them harder to interpret. Specifically, the framing of the composite function task in Section 3.3 is confusing, and the cited prior work addresses slightly different problems.\n* The blocking mechanism is not well explained. It would help if the authors provided more detail on how it was implemented and why it works.", "was_revised": true, "final_justification": "Repeating here what I wrote to the reviewers:\n\nRegarding W1, I am still not entirely satisfied with the authors' response. While I now better understand the problem and acknowledge that the authors demonstrate differences in performance in real-world \"large\" models, I still struggle to see the relevance of the synthetic task (as presented in the rebuttal to Reviewer UvAr, Section Q1) to real-world language problems. However, I admit that I am not an expert on such synthetic tasks, so I might be setting the bar too high. Given my lingering doubts about the importance of this synthetic task, it is challenging for me to give a high recommendation for acceptance. *Nonetheless, I note here explicitly that since I am not an expert on synthetic tasks, if the Area Chair believes the task is indeed important, they may disregard this particular point.*\n\nRegarding W2, the paper's lack of clarity remains a significant issue. However, I acknowledge and appreciate that the authors have committed to substantially improving clarity in the manuscript.\n\nConsidering the anticipated improvements in clarity and recognizing that another reviewer sees the synthetic task as important, I am raising my score to a 3.", "rebuttal": "We sincerely appreciate your suggestions. Guided by your comments, we have meticulously revised the manuscript, performed additional experiments, and addressed all of your comments in detail. \n\n**W1:**\n\nWe thank the you for the insightful feedback. They raised a crucial point about the need to clearly establish the significance of our synthetic tasks and their practical relevance. To directly address this, we have conducted new experiments on large-scale models which demonstrate that the architectural biases revealed by our synthetic tasks have tangible, real-world consequences.\n\n- **Bridging Synthetic Tasks to Real-World Impact:** The primary motivation for our synthetic task was to isolate and understand a fundamental architectural property—symmetry—in a controlled environment. We hypothesized that Mamba's inherent asymmetry would lead to challenges in tasks requiring symmetrical or inverse reasoning.\n\n  To validate this, we tested our hypothesis on two prominent, large-scale language models: Tencent’s Hunyuan-TurboS (a hybrid model with 57 Mamba layers) and DeepSeek-V3 (a Transformer-based model of comparable scale). We designed a challenging inverse sequence matching task that requires the model to find a key and return its corresponding value in reverse order. The results are striking:\n\n| Model          | 1st. round Acc. | 2nd. round Acc. | 3rd. round Acc. | Avg. Acc. | Avg. SER|\n| - | - | -| - | - | - |\n| Hunyuan‑TurboS          | 19\\%     | 17\\%     |17\\%      |17.6\\%      |19.2\\%|\n| DeepSeek-V3 | **47\\%**     | **56\\%**     | **54\\%**     | **52.3\\%**     |**3.8\\%**|\nAvg. SER is the symbol error rate on correctly indexed retrievals.\n\n  The significant performance gap between the Mamba-containing model and the Transformer model provides strong evidence that the architectural asymmetry we identified is not a mere theoretical curiosity. It has a clear, negative impact on performance in a practical, retrieval-style task that mimics challenges found in complex reasoning. A real testing example can be found in our response to Reviewer  UvAr **Q1**.\n\n- **The Rationale for Our Synthetic Tasks and Their Implications**\nHaving established the practical impact, we wish to clarify the rationale for our methodology:\n  - **Why use synthetic composite functions:** These functions are fundamental constructs in mathematics and computer science. A model's ability to learn them, especially with regard to properties like symmetry, serves as a powerful diagnostic for its underlying reasoning capabilities. As the reviewer suggested, this approach is grounded in prior work (e.g., Zhang et al. [1], Hang et al. [2]), which has successfully used such tasks to probe the emergent abilities of LLMs.\n\n  - **Architectural bias is critical:** The reviewer asked why failing at this is \"bad.\" It's not about a universal \"good\" or \"bad\" but about fitness for certain tasks, as the no free lunch theorem demonstrates. Our findings show that Mamba's architectural bias results in a specific, measurable deficit in tasks requiring symmetrical information mapping (e.g., A -> B and B -> A). The poor performance on the inverse sequence matching task is the direct consequence. This limitation is critical for applications that rely on bidirectional associations or logical inversions.\n\n**W2:**\n\nWe thank the reviewer for their detailed feedback on the clarity of our manuscript. We acknowledge that Sections 3.1 and 3.3, along with the figures, need improvement. As suggested, we will thoroughly revise these sections and add comprehensive captions to all figures in the updated manuscript to make our methodology much easier to follow. To address the reviewer's specific concern about the composite function task, we offer the following clarification on its framing and purpose.\n\n**composite function task:** Anchors 1, 2, 3, and 4 (depicted in orange) each represent distinct functions. Among the 16 anchor pairs formed, 14 correspond to composite functions derived directly from the sequential application of the individual anchor functions. The pair \"34\", highlighted in red, is defined as a different function rather than a direct composition. The pair \"43\" is intentionally excluded from the training set. The input to each anchor pair function is referred to as the \"key\" (indicated in green). \n\n**A Simple Analogy:**\n\nThe core idea of our task can be understood with a simple real-world analogy. Imagine a row of people indexed by numbers. Let's define two functions: $n(x)$:  Find the person n positions to the right of person $x$ and $m(x)$: Find the person m positions to the left of person $x$. The composite function $m(n(x))$ means \"start at person x, move n steps right, then move m steps left.\" A model can solve this in two ways:\n- Compositional Reasoning: Execute the steps sequentially.\n- Symmetry-based Shortcut: Recognize that $m(n(x))$ is equivalent to moving $n−m$ steps right and that its symmetric counterpart, $n(m(x))$, yields the same result.\nOur task is designed to discover which strategy a model prefers. Moreover, its fundamental goal—evaluating compositional generalization—is deeply connected to established benchmarks like SCAN and COGS [10, 11]. These benchmarks also test a model's ability to combine known components into novel structures. \n\n**W3 & Q1:**\n\nWe appreciate the reference to [5]. This is an excellent paper that provides a more targeted, surgical method for analyzing information flow specifically within Mamba's architecture.\n\nHowever, as our manuscript was submitted on May 15, 2025, this work, which was posted on arXiv on May 30, 2025, was published concurrently and was thus not available to us at the time of writing.\n\nFurthermore, our goal required a simple, model-agnostic method to validate long-range reasoning in both the Mamba-based model and the Transformer [12]. The common technique of inserting random tokens was the most direct and appropriate choice for this purpose.\n\n**Our Blocking Mechanism: A Common and Effective Technique**:\n\nThe SSM mechanism can be expressed as\n\n$Y=SX+X$, with $S=Mask\\circ(CB^{\\top})$,\n\nwhere the matrix $S$ has a shape of $(s,s)$, assuming the head dimension is omitted and $s$ is the sequence length. The $(i,j)$ element of $S$ represents how much token $i$ attends to token $j$. If this value is set to $0$, it implies that token $i$ cannot receive information from token $j$ through the SSM. Our blocking mechanism is implemented by zeroing out the specific entries in $S$ corresponding to the connections we wish to block.\n\nWe will update the manuscript to include this detailed explanation and properly situate our method within the context of established practices.\n\n**References**\n\n[1]Zhang et al. Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing. NeurIPS 2024.\n\n[2]Hang et al. Scalable Complexity Control Facilitates Reasoning Ability of LLMs. arXiv: 2505.23013 (2025).\n\n[3]Xu et al. Overview frequency principle/spectral bias in deep learning. Commun. Appl. Math. Comput. 7 (3): 827–864 (2025).\n\n[4]Dao et al. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. arXiv:2405.21060 (2024).\n\n[5]Endy et al. Mamba Knockout for Unraveling Factual Information Flow. arXiv:2505.24244 (2025).\n\n[6]Wolpert et al. No free lunch theorems for optimization. IEEE Trans. Evol. Comput., 1 (1), 67–82. (1997). \n\n[7]Mohtashami et al. Landmark Attention: Random-Access Infinite Context Length for Transformers. Adv. Neural Inf. Process. Syst. 36 (NeurIPS 2023).\n\n[8]Tworkowski et al. Focused Transformer: Contrastive Training for Context Scaling. Adv. Neural Inf. Process. Syst. 36 (NeurIPS 2023).\n\n[9]Hu et al. Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling. Proc. Int. Conf. Mach. Learn. 42 (ICML 2025).\n\n[10]Lake et al. Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks. Proc. 35th Int. Conf. Mach. Learn. 80, 2879 – 2888 (ICML 2018).\n\n[11]Kim et al. COGS: A Compositional Generalization Challenge Based on Semantic Interpretation. Proc. 2020 Conf. Empir. Methods Nat. Lang. Process. (EMNLP 2020), 9087 – 9105.\n\n[12]Meng et al. Locating and Editing Factual Associations in GPT. Adv. Neural Inf. Process. Syst. 35 (NeurIPS 2022)."}, {"review_id": "3pQVZCwBam", "reviewer": "Reviewer_7EnS", "summary": "This paper presents evidence for the “asymmetric bias” introduced by the convolution in Mamba. In particular, the authors focus on two setups: composite function and inverse sequence matching. They compare Mamba with Transformers and show that Mamba learns qualitatively different solutions due to the convolution prior to the SSM part. To further show this, the authors also modify Mamba by setting all convolution weights to one or adding residual connection, both interventions mitigate such bias.", "questions": "Q1 Why is the bias introduced by the convolution bad? The authors only report that it hurts performance in the inverse sequence matching task. Additional evidence or discussion, possibly in a more realistic setup, can strengthen the paper.\n\nQ2 Introducing residual connections seems to greatly reduce the convolutional bias and allows Mamba to learn the inverse sequence matching task. What is its effect in the composite function task? Since this modification is easy, would you recommend it for future Mamba-like models? An experiment showing e.g. that the residual connection does not hurt pretraining could strengthen the paper.\n\nQ3. Can the author clarify the information blocking setup? How is the SSM matrix defined? How is the blocking implemented? I feel this is not clear from the text and Figure 5.\n\n**Comments on the clarity and minor comments:**\n\nC1 Missing word in line 106\n\nC2 Sections 3.1 and 3.2  contain many typos, e.g. the input to the SSM should contain $A$ or $\\Delta_t$ and not $H$, also including (at least comments on) the dimensionality of the input could help readers.\n\nC3 The Receptive field of a 2 layer mamba is said to be 6 in line 157. I assume this is because the size of the convolution kernel is 3. However this is not true since the second convolution comes after the SSM part so its receptive field is potentially all the previous tokens.\n\nC4 Figure 5: Layer 1 output is written in natural numbers as the input but shouldn’t it be real vectors in this case?\n\nC5 Line 178: SSM matrix definition should be included or better explained. I don't think the expression in figure 5 (referred to as flow) is correct since the effect of A should also be included.\n\nC6 Figure 6 could be greatly reduced in size and the y axis could e.g. show the interval 0.95-1.\n\nC7 Line 217: “Previous study has shown”, a citation could help here.\n\nC8 Line 218-219 are unclear: the transformers seem to have the same behaviour as mamba when equipped with the convolution, the text should reflect that.\n\nC9 I think there is a typo in Figure 7: standard mamba should be $\\gamma=0.5$, not $\\gamma = 1$.", "limitations": "The authors include a limitation section after the conclusion discussing the lack of real-world experiments and experiments at a larger scale.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nS1 Experimental evidence supports the proposed hypothesis: all experiments performed strongly suggest that the convolution indeed biases Mamba’s solution.\n\nS2 Most new architectures based on SMMs include a 1d convolution, while standard Transformers do not. Hence,  this study is important since it sheds light on a possible problem of using the convolution.\n\n**Weaknesses**\n\nW1 The paper is quite difficult to read and contains many typos. Some sections could be expanded to explain things more clearly.\n\nW2 Experimental setup is limited and the paper lacks experiments on natural language on a realistic task to show the hurting effect of the convolution.", "was_revised": true, "final_justification": "The authors addressed my concerns about clarity and provided an experiment with commercial-grade LLMs, albeit not so realistic. Nonetheless, the work has identified an interesting bias of the causal convolution and an effective way to mitigate it, therefore I recommend acceptance.", "rebuttal": "Through revisions and experiments based on each of your suggestions, our work has been substantially improved. We sincerely thank you for your valuable guidance.\n\n**W1:**\n\nIn line with your suggestion, we will revise Section 3.1 as follows: as per your suggestion, we will replace $H$ with $X$ to facilitate comparison. In the previously submitted version, we omitted $t$ and $A$ in the equations to emphasize the input to the SSM, but we will now include them in the updated version for completeness. In addition, we have provided explicit dimension annotations to aid clarity.\n\n**Section3.1:** The Mamba block can be divided into three parts: the widely known SSM (State Space Model) component, the pre-SSM part, and the post-SSM part. Omitting trivial dimension transformations and setting the batch size to 1 to omit the batch dimension, for a given input \n$U$ to the block, the internal computation process to obtain the output \n$O$ can be described as follows:\n\nPre-SSM\n\n$( \\tilde{U}, Z, dt) = \\textbf{Linear}(U),~~~ U\\in R^{(s,d)}, \\tilde{U}\\in R^{(s,2d+2h)},Z\\in R^{(s,2d)}, dt\\in R^{(s,N_h)},$\n\n$(B, C, X) = \\sigma(\\textbf{Conv1d}(\\tilde{U})),~~~ B\\in R^{(s,h)}, C\\in R^{(s,h)}, X\\in R^{(s,2d)},$\n\nSSM\n\n$Mask = \\textbf{F}(A_0, dt),~~~ A_0\\in R^{N_h}, Mask\\in R^{(N_h, s, s)},$\n\n$I = \\textbf{Repeat}(C B^{\\top},N_h)),~~~  I\\in R^{(N_h, s, s)},$\n\n$S = Mask\\circ I,~~~  S\\in R^{(N_h, s, s)},$\n\n$Y = SX + X,~~~ Y\\in R^{(N_h, s, 2d/N_h)},$\n\nPost-SSM\n\n$Y_{Norm} = \\textbf{RMS}(Y\\circ\\sigma(Z)),~~~ Y_{Norm}\\in R^{(s, 2d)},$\n\n$O = \\textbf{Linear}(Y_{Norm}),~~~ O \\in R^{(s, 2d)},$\n\nwhere \n> - $S$ : SSM matrix,\n> - $s$ : sequence length,\n> - $d$ : hidden‑state dimension,\n> - $h$ : SSM hidden dimension,\n> - $N_h$ : number of SSM heads,\n> - $\\textbf{Linear}$ : linear transformation,\n> - $\\textbf{Conv1d}$ : one‑dimensional convolution,\n> - $\\sigma$ : nonlinear activation function,\n> - $\\textbf{F}$ : function generates the $Mask$,\n> - $\\textbf{Repeat}$ : dimension‑replication operation,\n> - $\\circ$ : pointwise multiplication, \n> - $\\textbf{RMS}$ : RMS normalization.\n\nIn response to your suggestion, we have revised and expanded Section 3.3, incorporating real-world examples to make the composite function task and its underlying symmetry more intuitive and accessible to readers. Also, we have expanded the Section3.4 and added real-world analogues of the task to help readers better understand the inverse sequence matching task and its fundamental nature. Owing to space constraints, we omit the detailed enumeration here.\n\n**W2:**\n\nWe sincerely value your guidance, and to verify the real‑world relevance of our findings, we converted the inverse sequence matching task proposed in the paper into a natural‑language version and evaluated it under the most practical setting. Specifically, we ran the test 3 times on two commercial LLMs: Hunyuan‑TurboS and DeepSeek‑V3. Hunyuan‑TurboS contains 57 stacked Mamba blocks, while DeepSeek‑V3 consists of Transformers and has a comparable parameter count and overall benchmark performance. Each run contained 100 randomly generated inverse sequence matching problems.\n\nAcross the three runs, Hunyuan‑TurboS achieved an average accuracy of **17.6\\%**, far below DeepSeek‑V3’s average of **52.3\\%**. These results mirror the trends we observed on smaller models in the paper, clearly demonstrates the impact of the asymmetry bias introduced by nonlinear convolution on the practical performance of Mamba in large-scale models and further reinforcing our conclusions and their practical significance. \n\nA real testing example can be found in our response to Reviewer  UvAr **Q1**.\n\n**Q1:**\n\nWe sincerely appreciate the thoughtful suggestions you provided. Through the composite function task, we revealed Mamba’s intrinsic bias for asymmetry—a fundamental property introduced by its convolutional design. In our view, no property is inherently “bad”; rather, some are simply ill‑suited to particular applications.\n\nThis bias becomes a shortcoming when the task demands understanding or handling symmetry—as in the inverse sequence matching task. However, in tasks where symmetry is unnecessary, or even needs to be avoided, Mamba’s bias may turn into an advantage. For instance, “finding Jack’s mother’s phone number” constitutes an asymmetric composite function task, whereas its symmetric inversion—“finding the phone number’s mother of Jack”—is both meaningless and distinct from the original. In such tasks, Mamba’s asymmetric bias ceases to be a drawback; other biases or features will instead govern its differences from the Transformer.\n\nAs further evidence, we have validated these observations by implementing the inverse sequence matching task in a commercial scale large language model using natural‑language prompts. The full details are provided in Section W2.\n\n**Q2:**\n\nWe are deeply grateful for your question.\n\n**Res needs to work with SSM:** After introducing a residual connection, the SSM can draw on two information sources in its input: (i) the fused, inherently asymmetric representation produced by the convolution, and (ii) the un‑fused, non‑linearly untransformed signal carried through the residual path. The advantage is that, in tasks such as inverse sequence matching task—which require the SSM to match two symmetrical signals—the model can lean on the residual connection, thereby mitigating the asymmetry introduced by the convolution.\n\n**SSM's non-functioning $\\rightarrow$ Res do not work on composite function task:** We applied this residual‑augmented Mamba to an extensive set of composite function experiments. The results show that, compared with the standard Mamba, the modified Mamba attains composite solution over a larger region of settings, yet it still almost never converges to symmetric solutions. The reason is that, in the composite function task, Mamba can extract the necessary information directly through the convolution, whereas in the inverse sequence matching task the convolutional path is blocked by intervening tokens, forcing the model to rely solely on the SSM. When Mamba can acquire information directly through its convolution—as our experiments demonstrate—it may rely exclusively on that convolutional path, bypassing the SSM entirely. Consequently, the residual pathway we introduced to support the SSM no longer serves a meaningful function in this scenario. Thus, the model’s asymmetric bias remains unchanged. \n\nWe will recommend adding a residual connection when employing Mamba. For longer‑sequence problems like inverse sequence matching task (and many real‑world tasks are far longer), Mamba cannot rely on the convolution alone; the residual then becomes a critical alternative pathway that helps the model gather and align information. We are eager to validate our findings with large scale pre‑training in the future, but in the near term we may not have sufficient computational resources to do so and hope for your understanding.\n\n**Q3 & C5:**\n\nAs you kindly suggested, we will revise Section 3.1 and the caption of Figure 5, and we will add an appendix subsection to explain our blocking mechanism in greater depth so that readers can follow the details more easily.\n\nOur blocking strategy is built on Mamba’s SSM matrix, defined as\n\n$S = Mask\\circ (CB^{\\top})$ ,\n\nand the output of SSM block is \n$SX$.\n\nThe matrix $S$ describes information flow within the SSM. After omitting the head dimension, its shape is $(s,s)$, where $s$ is the sequence length. The element $S_{ij}$ denotes the flow of information from token $j$ to token $i$; if $S_{ij} = 0$, no information passes from token j to token i through the SSM. Our blocking method simply zeroes the entries in S corresponding to flows we wish to block.\n\n**C1:**\n\n**line 106:** The design of our composite function task directly follows the approach used in Zhang et al. \n\n**C2:**\n\nWe will correct these points in Section 3.1. A revised version—including detailed dimension descriptions—has already been provided in our response to W1.\n\n**C3:**\n\nWe sincerely apologize for the confusion our oversight has caused. We mistakenly used “receptive field” when we should have written “pure convolutional receptive field.” Here, the term “convolutional receptive field” refers exclusively to how many tokens can be accessed through convolution alone, without invoking the SSM. We wish to prevent Mamba from obtaining information purely via convolution rather than through the SSM. We will replace all incorrect expressions in the original manuscript with \"pure convolutional receptive field.\"\n\n**C4:**\n\nThe reason the output of layer1 appears as a numeral is that we pass the output of laye 0 through the model’s final linear layer and then take the arg‑max of the resulting logits to obtain the corresponding digit. This approach enables us to better interpret the model’s intermediate representations and facilitates visualization. We will briefly explain this procedure in the caption of Figure5 and add a dedicated subsection in the appendix for a more detailed description.\n\n**C6:**\n\nWe will reduce the figure size and adjust the y‑axis to the range 0.95–1.\n\n**C7:**\n\nWe will add the appropriate reference in the relevant section.\n\n**C8:**\n\n**line 219:**  In this case, the Transformer exhibits a similar preference for composite function solutions as Mamba and struggles to learn symmetric solutions.\n\n**C9:**\n\nWe apologize for the imprecise wording in our earlier explanation. Here, we are referring to the standard Mamba architecture initialized to 1. With the default initialization of $\\gamma = 0.5$, Mamba is unable to learn either the composite or the symmetric solution, making it impossible to observe any bias. By setting the $\\gamma$ to 1, Mamba learns the composite solution much more reliably. Therefore, we employ an initialization of $\\gamma = 1$. We will revise the caption of Figure 7 accordingly."}]

Author rebuttal:

Meta-review: The paper probes architectural differences between Mamba (SSM-based model) and Transformers using controlled synthetic tasks—composite function learning and inverse sequence matching. By experiments, the authors have interesting findings revealing fundamental differences between Mamba and Transformer models, particularly in handling symmetrical patterns and relationships. They propose simple mitigations (e.g., residual paths / gating around the convolution) and shows good potential of improving SSM models.

In rebuttal, the authors also add large-model experiments (e.g., Hunyuan-TurboS vs. DeepSeek-V3) indicating the asymmetry effect persists in scaled settings (17.6% vs. 52.3% on the Passkey task).

Strengths
Multiple reviewers highlight that the paper convincingly traces the failure mode to the nonlinear convolution via targeted interventions and weight modifications.
The synthetic tasks are minimal but effective to test compositional vs. symmetric solutions.
Proposed residual/gating fixes are simple but useful

Weaknesses
The only concern after rebuttal is "Why the bias is bad”. But the authors give good explanation by experimenting on large models.
Decision: Accept (spotlight)