Hybrid Mamba–Transformer Decoder for Error-Correcting Codes

## Abstract

Abstract We introduce a novel deep learning method for decoding error correction codes based on the Mamba architecture, enhanced with Transformer layers. Our approach proposes a hybrid decoder that leverages Mamba’s efficient sequential modeling while maintaining the global context capabilities of Transformers. To further improve performance, we design a novel layer-wise masking strategy applied to each Mamba layer, allowing selective attention to relevant code features at different depths. Additionally, we introduce a progressive layer-wise loss, supervising the network at intermediate stages and promoting robust feature extraction throughout the decoding process. Comprehensive experiments across a range of linear codes demonstrate that our method significantly outperforms Transformer-only decoders and standard Mamba models.

## 1 Introduction

Deep learning–based decoders have achieved remarkable success in decoding error-correcting codes in recent years. Notable examples include Neural Belief Propagation [31], which learns weights of the message-passing algorithm; Neural Min-Sum [29, 12], which approximates the classical min-sum decoder with trainable parameters; Neural RNN decoder [23] for convolutional and turbo error correcting codes. Recently, diffusion-based decoders [7], which model channel noise as a diffusion process that can be reversed; and Transformer-based decoders [8, 10, 34, 36], which exploit self-attention to capture the code structure, reached state-of-the-art performance in neural decoding. However, despite their individual strengths, these methods either incur a high computational cost, compared to classical decoders, or fail to achieve state-of-the-art performance on some codes.

In this work, we propose a novel hybrid decoder that combines the Mamba architecture [17] - known for its highly efficient sequential modeling and low runtime latency - with transformer layers [35] that provide global receptive fields throughout the codeword. Concretely, we introduce a layer-wise masking strategy within each Mamba block, enabling the model to selectively focus on the most informative subsets of bits at varying depths. To further bolster the training dynamics, we propose a layer-wise loss that provides intermediate supervision at each decoding stage. This auxiliary loss not only promotes better gradient propagation through deep networks but also encourages the extraction of the decoding codeword at each stage enabling intermediate estimation of the decode codeword.

Extensive experiments on a diverse suite of binary linear block codes, including BCH, Polar, and LDPC codes, demonstrate that our Mamba–Transformer decoder consistently surpasses both Transformer-only decoders and conventional Mamba implementations. We report relative improvements of up to $18\%$ in BER for BCH and Polar codes, and on-par on LDPC codes, while improving inference speed compared to previous works.

## 2 Related works

### 2.1 Neural Decoders

In recent years, the study of deep learning-based decoders for error correction codes has emerged as a vibrant and rapidly evolving research area [16]. Two broad paradigms have been pursued: model-based architectures, which embed the structure of classical decoding algorithms into neural networks, and model-free architectures, which treat decoding as a purely data-driven mapping.

##### Model-based neural decoders

In model-based approaches, the computational graph of a traditional message-passing decoder is reinterpreted as a deep network with trainable parameters. Neural Belief Propagation (NBP) first demonstrated this idea by assigning learnable weights to the edges and messages of the belief propagation algorithm, resulting in a decoder that jointly optimizes its update rules through gradient-based training [30]. Building on NBP, the Neural Min-Sum decoder approximates the classical Min-Sum algorithm by introducing scalar and vector weight parameters into its summation and normalization steps [29, 13, 25, 26]. This parameterization retains the low-complexity structure of Min-Sum while achieving performance on par with more expensive decoders. To further reduce inference cost, pruning techniques have been applied to compress these networks, systematically removing redundant connections and yielding lightweight variants without significant performance degradation [5].

##### Model-free neural decoders

In contrast, model-free decoders rely solely on the representational power of generic neural architectures. Early work employed fully-connected networks to directly map noisy codewords to their nearest valid codewords, demonstrating feasibility on short block codes [6]. Subsequent studies showed that such networks can scale to moderate block lengths without overfitting [3]. More advanced generative frameworks have also been introduced: diffusion-based decoders model the channel corruption as a forward stochastic process and learn to reverse it via a sequence of denoising steps, achieving impressive gains under various noise conditions [7]. Meanwhile, Transformer-based decoders exploit self-attention to capture long-range code constraints; notable examples include the Error Correction Code Transformer with its extensions [8, 9, 10, 11] and recent variants employing layer-wise masking and cross-message-passing modules to enhance both expressivity and decoding speed [33, 34].

### 2.2 Mamba Architecture

In recent years, State-Space Models (SSMs) have attracted considerable attention as an alternative to purely attention-based architectures for sequence modeling, due to their ability to capture long-range dependencies with favorable computational and memory efficiency [19, 20]. A landmark contribution in this domain is the Structured State Space Sequence (S4) model, which leverages parameterized linear dynamical systems and the HiPPO framework [18] to achieve expressive, convolutional representations of sequential data. Building upon S4, subsequent work proposed the Mamba architecture, wherein the SSM’s convolutional kernels are dynamically generated as functions of the input sequence [17, 14]. Empirical evaluations demonstrate that Mamba attains inference speeds up to five times faster than comparable Transformer models while scaling seamlessly to input lengths on the order of millions of elements. Moreover, when Mamba is integrated with Transformer layers in a hybrid configuration, the resulting model consistently surpasses both standalone Transformer and S4 architectures in a range of language and time-series benchmarks.

## 3 Background

In this section, we formalize the decoding setup for binary linear block codes using the notation of [8]. Let $C\subseteq\mathbb{F}_{2}^{n}$ be a binary linear block code of length $n$ and dimension $k$, defined by its parity-check matrix $H\in\mathbb{F}_{2}^{(n-k)\times n}$. A vector $x\in\mathbb{F}_{2}^{n}$ is a valid codeword if and only if $H\,x\;=\;0\,$.
Transmission occurs over an Additive White Gaussian Noise (AWGN) channel with Binary Phase-Shift Keying (BPSK) modulation. Under this model, the codeword $x\in\{0,1\}^{n}$ is mapped to $x_{s}\in\{\pm 1\}^{n}\subset\mathbb{R}^{n}$ and corrupted by Gaussian noise $z\sim\mathcal{N}(0,\sigma^{2}I_{n})$, yielding the received vector $y\;=\;x_{s}\;+\;z\,$. To enforce invariance to the transmitted codeword and mitigate overfitting, we construct the decoder input from the magnitude of the channel output and its syndrome as in [3]. First, we obtain the hard-decision vector $y_{b}\;=\;\tfrac{1-\operatorname{sign}(y)}{2}\;\in\;\{0,1\}^{n},$ and then compute the syndrome $s\;=\;H\,y_{b}\;\in\;\mathbb{F}_{2}^{\,n-k}.$ Finally, we concatenate the amplitude $|y|\in\mathbb{R}^{n}$ with the syndrome $s$ to form the decoder input $y_{\mathrm{in}}\;=\;\bigl{[}\,|y|;\,s\,\bigr{]}\;\in\;\mathbb{R}^{\,n+(n-k)}\,,$ which is provided to the proposed Mamba–Transformer decoder.

## 4 Method - Mamba-Transformer Decoder

Figure: (a) Model Blocks
Refer to caption: extracted/6471440/Figures/model-high-level-layers.png

The ECCM model is composed from $N_{layers}$ layers, the layers are alternating between mamba layers and attention layers, starting with a mamba layer. The layers $l_{i}$ where $i\in{1,3,...}$ are Mamba layers and the layers $l_{i}$ where $i\in{2,4,...}$ are attention blocks.
The input to the $l_{i}$ layer is denoted $y^{i-1}$, with

$$ $y^{0}=y_{in}\odot W_{emb}$ (1) $$

where $W^{i}_{emb}\in\mathbb{R}^{L\times D}$, where $D$ is the model hidden dimension, $L=2n-k$, and $\odot$ represents the element-wise multiplication operator. Note that $y^{i}$ for $i>0$, will be the output of the $i$-th layer.
The architecture uses two masks that are produced from the parity check matrix, $f(H)\in\mathbb{Z}_{2}^{(n-k)\times(2n-k)}$ and $g(H)\in\mathbb{Z}_{2}^{(2n-k)\times(2n-k)}$ which are used by the mamba layers and attention layers - respectively. The $g(H)$ mask is taken from [8]:

$$ $\displaystyle g(H)=\begin{bmatrix}Graph(H)&H^{T}\\ H&I_{n-k}\end{bmatrix}$ (2) $$

and $Graph(H)\in\mathbb{R}^{n\times n}$

$$ $\mathrm{Graph}(H)[i,j]=\begin{cases}1,&\exists\,m\in[1,n-k]\text{ such that }H [m,i]=1\text{ and }H[m,j]=1\\ 0,&\text{otherwise}\end{cases}$ (3) $$

The $g(H)$ mask, ensures that only pairs that are on the same parity check line are computed, which reduces the complexity of the attention operation and induces knowledge of the code into the model.

The proposed $f(H)$ mask, is devised to have the same effect on the SSM process. By applying it to the matrices of the operation, the effect of the input in a specific position only changes the state for bits that are on the same parity check line. Ensuring that interactions only happens along the parity check lines

$$ $f(H)=[H;I_{n-k}]$ (4) $$

#### 4.0.1 Mamba block

Each Mamba block contains the following operations:
First $y^{i-1}$ is projected with two learnable matrices $W^{i}_{u},W^{i}_{z}\in\mathbb{R}^{D\times D}$:

$$ $u^{i}=W_{u}^{i}y^{i-1}$ (5) $$

$$ $z^{i}=SiLU(W_{z}^{i}y^{i-1})$ (6) $$

where $u^{i},z^{i}\in\mathbb{R}^{L\times D}$.
and SiLU is the activation function [22]. Then apply 1D-Convolution layer $Conv^{i}$ to $u^{i}$ over the sequence length:

$$ $u^{i}_{conv}={Conv}^{i}\left({u^{i}}\right)$ (7) $$

where $u^{i}_{conv}\in\mathbb{R}^{L\times D}$. Then we apply the Selective-State-Space Model [17] with modification to the error-correcting code scenario. First, $u^{i}_{conv}$ projected to $B^{i},C^{i}\in\mathbb{R}^{L\times S}$ where $S$ is the dimension of the state.

$$ $B^{i}=u^{i}_{conv}{W^{i}_{b}}^{T}$ (8) $$

$$ $C^{i}=u^{i}_{conv}{W^{i}_{c}}^{T}$ (9) $$

Where $W^{i}_{b},W^{i}_{c}\in\mathbb{R}^{S\times D}$ are learnable matrices. Then we apply the discretization process (as described in [17]) on matrices $B^{i}$, and $A^{i}$ where $A^{i}\in\mathbb{R}^{D\times S}$ which is a learnable matrix. First, generate the $\Delta^{i}\in\mathbb{R}^{L\times D}$ matrix:

$$ $\Delta^{i}=W_{\Delta}^{i}u^{i}_{conv}$ (10) $$

where $W_{\Delta}^{i}\in\mathbb{R}^{L\times D}$ is a learnable matrix.
Second, initialize the tensors $\bar{A}^{i},\bar{B}^{i}\in\mathbb{R}^{L\times D\times S}$,

$$ $\displaystyle\bar{A}^{i}[l,s,d]=exp(A^{i}[l,s]\Delta^{i}[l,d])$ (11) $\displaystyle\bar{B}^{i}[l,s,d]=B^{i}[l,s]\Delta^{i}[l,d]$ (12) $$

where $l\in[1,L],s\in[1,S],d\in[1,D]$. Here the error-code specific modification is inserted, using a mask. Generate the mask matrix $f(H)=[H;I_{K}]$ where $I_{K}\in\mathbb{R}^{K\times K}$ is the identity matrix. Then apply the mask to the matrices $\bar{B}^{i}$ and $C^{i}$ creating the matrices $\bar{B}^{i}_{M}$ and $C^{i}_{M}$, respectively.

$$ $\bar{B}^{i}_{M}[l,d,s]=\begin{cases}f(H)[l,d]\bar{B}^{i}[l,d,s]&d<(n-k)\\ 0&otherwise\end{cases}$ (13) $$

where $l\in[1,L],s\in[1,S],d\in[1,D]$

$$ ${C}^{i}_{M}[l,s]=\begin{cases}f(H)[l,s]{C}^{i}[l,s]&s<(n-k)\\ 0&otherwise\end{cases}$ (14) $$

Apply the SSM process in which a series of states $h_{l}\in\mathbb{R}^{D\times S}$ are calculated:

$$ $h_{l}[d,s]=\bar{A}[l,d,s]h_{l-1}[d,s]+\bar{B}_{M}^{i}[l,d,s]u^{i}_{conv}[l,d]$ (15) $$

$$ $u^{i}_{ssm}[l,d]=\sum_{i=1}^{S}{h_{l}[d,i]C[l,i]}+R[d]u^{i}_{conv}[l,d]$ $$

where $R\in\mathbb{R}^{F}$ is a learnable vector, And initialize $h_{0}$ to a vector of zeros.

Then we apply the gating from Eq.[6](https://arxiv.org/html/2505.17834v1#S4.E6):

$$ $u^{i}_{fwd}=z^{i}\odot u^{i}_{ssm}$ (16) $$

Up to this point, the description was of the processing in the causal direction.
Now apply the reverse direction processing in order to achieve a Bi-Directional Mamba.
Meaning, substitute $y^{i-1}$,$H$ and $u^{i}_{fwd}$ to
$\overleftarrow{y}^{i-1}$, $\overleftarrow{f(H)}$ and $\overleftarrow{u}^{i}_{fwd}$.
And apply Eq.[5](https://arxiv.org/html/2505.17834v1#S4.E5) through Eq.[16](https://arxiv.org/html/2505.17834v1#S4.E16)
where:

$$ $\overleftarrow{y}^{i-1}[l,d]=y^{i-1}[L-l,d]$ (17) $$

$$ $\overleftarrow{f(H)}[l,d]=f(H)[L-l,d]$ (18) $$

and $\overleftarrow{u}^{i}_{fwd}$ is the output of the process.
Then calculate $u^{i}_{bwd}\in\mathbb{R}^{L\times D}$

$$ $u^{i}_{bwd}[l,d]=\overleftarrow{u}^{i}_{fwd}[L-l,d]$ (19) $$

and then the output of the block $y^{i}\in\mathbb{R}^{L\times D}$ is calculated.

$$ $y^{i}=u^{i}_{fwd}+u^{i}_{bwd}$ (20) $$

### 4.1 Transformer Block

First compute the queries, keys, and values $Q^{i},K^{i},V^{i}\in\mathbb{R}^{L\times D}$
using the the input $y^{i-1}$:

$$ $\displaystyle Q^{i}$ $\displaystyle=y^{i-1}W_{Q}^{i}+b_{Q}^{i}$ (21) $\displaystyle K^{i}$ $\displaystyle=y^{i-1}W_{K}^{i}+b_{K}^{i}$ $\displaystyle V^{i}$ $\displaystyle=y^{i-1}W_{V}^{i}+b_{V}^{i}$ $$

where $W_{Q}^{i},W_{K}^{i},W_{V}^{i}\in\mathbb{R}^{D\times D}$ and $b_{Q}^{i},b_{K}^{i},b_{V}^{i}\in\mathbb{R}^{D}$ are learned parameters. These are reshaped into $h$ attention heads with per-head dimension $d_{k}=D/h$:

$$ $\tilde{Q}^{i},\tilde{K}^{i},\tilde{V}^{i}\in\mathbb{R}^{h\times L\times d_{k}}.$ (22) $$

Then apply the HPSA mechanism as described in [27]:

$$ $\tilde{O}^{i},\alpha^{i}=\mathrm{HPSA}(\tilde{Q}^{i},\tilde{K}^{i},\tilde{V}^{ i},\text{g(H)}),$ (23) $$

with $\tilde{O}^{i}\in\mathbb{R}^{h\times L\times d_{k}}$ and $\alpha^{i}\in\mathbb{R}^{h\times L\times L}$. The outputs from all heads are concatenated $O^{i}\in\mathbb{R}^{L\times D}$:

$$ $O^{i}=\mathrm{concat}(\tilde{O}^{i}).$ (24) $$

Finally, the output of the attention block is computed as $y^{i}_{a}\in\mathbb{R}^{L\times D}$:

$$ $y_{a}^{i}=O^{i}W_{O}^{i}+b_{O}^{i}$ (25) $$

where $W_{O}^{i}\in\mathbb{R}^{D\times D}$ and $b_{O}^{i}\in\mathbb{R}^{D}$. Then apply layer norm [2] to calculate $\tilde{y}^{i}_{a}\in\mathbb{R}^{L\times D}$

$$ $\tilde{y}^{i}_{a}=LayerNorm(y_{a}^{i})$ (26) $$

then $y^{i}\in\mathbb{R}^{L\times D}$ is calculated:

$$ $y^{i}=ReLU(\tilde{y}^{i}_{a}W^{i}_{1}+b_{1})W_{2}+b_{2}$ (27) $$

where $W_{1}\in\mathbb{R}^{D\times 4D},\ b_{1}\in\mathbb{R}^{4D},\ W_{2}\in\mathbb{R}
^{4D\times D},\ b_{2}\in\mathbb{R}^{D}$ are learnable parameters.

### 4.2 Model Output

After each layer, $y^{i}$ is project down to $o^{i}\in\mathbb{R}^{N}$ using $w_{r}\in\mathbb{R}^{D}$,$b_{r}\in\mathbb{R}^{L}$,$W_{s}\in\mathbb{R}^{N\times L}$,$b_{s}\in\mathbb{R}^{N}$ which are learnable parameters:

$$ $o^{i}=\sigma(W_{s}(y^{i}w_{r}+b_{r})+b_{s})$ (28) $$

where $\sigma$ is the sigmoid function. The syndrome is calculated:

$$ $s^{i}=H(o^{i}>0.5)$ (29) $$

if $s^{i}=s$ the processing is stopped - and set $i_{last}=i$,
if $s^{i}\ \neq\ s\ \forall\ i\ \in\ [1,N_{layers}]$ set $i_{last}=N_{layers}$

Note that the model output is an estimate for the input’s multiplicative noise,
therefore in order to calculate the estimated code-word:

$$ $\hat{c}^{i}[l]=\frac{1-sign((1-2o^{i}[l])y_{in}[l])}{2}$ (30) $$

### 4.3 Loss Function

In order to calculate the loss, first calculate in which bit an error occured as in [3]

$$ $z[l]=\frac{1-sign((1-2c[l])x_{in}[l])}{2}$ (31) $$

Then calculate the Binary Cross Entropy (BCE) between $o^{i}$ and $z$, and sum over all the outputs.

$$ $Loss=\sum_{i}^{N_{layers}}BCE(o^{i},z)$ (32) $$

Note that if for some $i_{0}<N_{blocks}$, $Ho^{i_{0}}=0$, the outputs for $i>i_{0}$ are not calculated and therefore not summed in the loss.

## 5 Experiments

To evaluate the proposed decoder, we train it on four classes of linear block codes: Bose–Chaudhuri–Hocquenghem (BCH) codes [4], Low-Density Parity-Check (LDPC) codes [15], Polar codes [1], and MacKay codes. The corresponding parity-check matrices are obtained from [21]. Training data are generated at six signal-to-noise ratio (SNR) levels, $\mathrm{SNR}\in\{2,\dots,7\}$ dB, then added to the generated message to simulate a AWGN channel. We use the zero-codeword in the training process in order to verify that the model doesn’t overfit the codewords it sees, by simply changing to random codewords on model evaluation. The Adam [24] optimizer was configured with a learning rate of $2.5\times 10^{-4}$ and decays to $10^{-10}$ following a cosine [28] schedule. The training was done with batch size of 128 and 1000 batches per epoch.
In all the experiments we set $D=128,N_{blocks}=8,h=8,S=128$, where $D$ is the embedding size, $S$ is the Mamba block’s state size, $h$ is the number of attention heads, and $N_{blocks}$ is the number of blocks, meaning there are 4 Mamba blocks and 4 attention blocks, the resulting model has similar number of parameters to previous methods at approx $1.2M$.
For evaluation, we simulate test examples at SNR levels of 4dB, 5dB, and 6dB, and report the negative natural logarithm of the bit error rate, $-\ln\bigl{(}\mathrm{BER}\bigr{)}.$ Each evaluation run is continued until a fixed number of decoding errors, $500$, has been observed similar to [34].

## 6 Results and Discussions

In Tab.[1](https://arxiv.org/html/2505.17834v1#S6.T1), the results are the presented compared to previous methods, for each code 6 methods are presented: BP, ARBP, ECCT, AECCT, CrossMPT, and our own method ECCM. For each, the table shows the negative natural log of the BER at SNR levels 4dB, 5dB, and 6dB. The best method is marked in bold, in places where reported results were not available the "-" mark was used. The table shows that ECCM consistently outperforms all the other methods across all BCH codes, and SNR levels. Notably outperforming CrossMPT - with a significant improvement in the decoding of BCH(63,45) code, achieving over $18\%$ in terms of negative natural logarithm of BER, $-ln(\text{BER})$, ECCM shows comparable performance to CrossMPT in the decoding of the Polar(64,48) code, and shows notable improvements in longer Polar codes - achieveing up to $7.2\%$ gain in the Polar(128,86) code. While CrossMPT achieves better results in some of the LDPC codes the improvement are modest typically around $4\%$, ECCM achieves better performance in decoding LDPC(49,24) and - comparable to increased - performance on LDPC(121,80). It also outperforms all other models in the MacKay Code, slightly outperforming CrossMPT, which indicates the model is capable in learning very sparse parity-check matrices. Fig [3](https://arxiv.org/html/2505.17834v1#S7.F3) shows the performence in terms of BER as a function of SNR for the above methods. It is important to note that integrating ECCM and CrossMPT is possible, which may close the gap in LDPC codes decoding.

**Table 1: Comparison of decoding performance at three SNR values (4, 5, 6) for BP, ARBP [32], ECCT [8], AECCT [27], CrossMPT [34], and ECCM. The results are measured by the negative natural logarithm of BER ($-\ln(\mathrm{BER})$). The best results are highlighted in bold. Higher is better.**
| Codes | $(N,K)$ | BP | AR BP  [32] | ECCT $1.2M$  [8] | AECCT $1.2M$ [27] | CrossMPT $1.2M$ [34] | ECCM $1.2M$ (ours) |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 |  |  |
| BCH | (31,16) | 4.63<br>– | 5.88<br>– | 7.60<br>– | 5.48 | 7.37 | 9.60 | 6.39 | 8.29 | 10.66 | 7.01 | 9.33 | 12.27 | 6.98 | 9.25 | 12.48 | 7.26 | 9.71 | 12.66 |
| (63,36) | 3.72<br>4.03 | 4.65<br>5.42 | 5.66<br>7.26 | 4.57 | 6.39 | 8.92 | 4.68 | 6.65 | 9.10 | 5.19 | 6.95 | 9.33 | 5.03 | 6.91 | 9.37 | 5.49 | 7.52 | 10.23 |  |
| (63,45) | 4.08<br>4.36 | 4.96<br>5.55 | 6.07<br>7.26 | 4.97 | 6.90 | 9.41 | 5.60 | 7.79 | 10.93 | 5.90 | 8.24 | 11.46 | 5.90 | 8.20 | 11.62 | 7.01 | 10.12 | 14.26 |  |
| (63,51) | 4.34<br>4.50 | 5.29<br>5.82 | 6.35<br>7.42 | 5.17 | 7.16 | 9.53 | 5.66 | 7.89 | 11.01 | 5.72 | 8.01 | 11.24 | 5.78 | 8.08 | 11.41 | 6.10 | 8.77 | 12.22 |  |
| Polar | (64,48) | 3.52<br>4.26 | 4.04<br>5.38 | 4.48<br>6.50 | 5.41 | 7.19 | 9.30 | 6.36 | 8.46 | 11.09 | 6.43 | 8.54 | 11.12 | 6.51 | 8.70 | 11.31 | 6.61 | 8.61 | 11.20 |
| (128,86) | 3.80<br>4.49 | 4.19<br>5.65 | 4.62<br>6.97 | 5.39 | 7.37 | 10.13 | 6.31 | 9.01 | 12.45 | 6.04 | 8.56 | 11.81 | 7.51 | 10.83 | 15.24 | 8.05 | 11.55 | 15.65 |  |
| (128,96) | 3.99<br>4.61 | 4.41<br>5.79 | 4.78<br>7.08 | 5.27 | 7.44 | 10.20 | 6.31 | 9.12 | 12.47 | 6.11 | 8.81 | 12.15 | 7.15 | 10.15 | 13.13 | 7.49 | 10.45 | 13.27 |  |
| LDPC | (49,24) | 5.30<br>6.23 | 7.28<br>8.19 | 9.88<br>11.72 | 6.58 | 9.39 | 12.39 | 5.79 | 8.13 | 11.40 | 6.10 | 8.65 | 12.34 | 6.68 | 9.52 | 13.19 | 6.71 | 9.55 | 13.25 |
| (121,60) | 4.82<br>– | 7.21<br>– | 10.87<br>– | 5.22 | 8.31 | 13.07 | 5.01 | 7.99 | 12.78 | 5.17 | 8.32 | 13.40 | 5.74 | 9.26 | 14.78 | 5.49 | 8.87 | 14.23 |  |
| (121,80) | 6.66<br>– | 9.82<br>– | 13.98<br>– | 7.22 | 11.03 | 15.90 | – | – | – | – | – | – | 7.99 | 12.75 | 18.15 | 7.81 | 12.34 | 18.35 |  |
| MacKay | (96,48) | –<br>– | –<br>– | –<br>– | 7.43 | 10.65 | 14.65 | – | – | – | – | – | – | 7.97 | 11.77 | 15.52 | 7.98 | 11.84 | 15.70 |

## 7 Model Analysis

### 7.1 Ablation Analysis

To analyze the contribution of each of the following proposed modifications: combining Mamba and Transformer, using loss from every layer, and the proposed mask for the mamba layers, variants of the proposed method were trained - removing one modification at a time. The variants were all trained on the BCH(63,45) code, with the same hyper-parameters as discussed above. The same performance evaluation process was used in this study as well. Table [2](https://arxiv.org/html/2505.17834v1#S7.T2) demonstrates that each of the proposed modifications contributes to the performance of the final model. The top row shows the full method, and in each subsequent row, one modification is removed to isolate its effect. The "Mamba Mask" indicates which mask was used in the experiment in the mamba layers, either the proposed mask $f(H)$, or the baseline mask $g(H)$ from ECCT [8]. In experiment (iii) no Mamba layers are used. The "Model Layout" column indicates whether in the experiment Mamba and Transformer layers were used, or only Transformer layers. The "Multi-loss" column is "True" if the loss was computed using the output from each layer, and "False" where it was computed only on the output of the last layer. Experiment (i) shows that using the proposed mask $f(H)$ yields better results than using $g(H)$. Experiment (ii) demonstrates that using the loss from each layer is contributing significantly to the proposed model’s performance. Experiment (iii) shows that combining the mamba and transformer layers yield better results, compared to both experiment (i) and the proposed model, confirming that it the modification is an improvement regardless of the used mask.

**Table 2: Ablation analysis: negative natural logarithm of bit error rate (BER) for our complete method compared with its partial components. Higher values indicate better performance, Highest value is marked in bold.**
| Experiment | Mamba Mask | Model Layout | Multi-Loss | SNR (dB) |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 4 | 5 | 6 |  |  |  |  |
| Full Method | $f(h)$ | Transformer & Mamba | True | 7.01 | 10.12 | 14.26 |
| (i) | $g(H)$ | Transformer & Mamba | True | 6.86 | 9.88 | 13.76 |
| (ii) | $f(H)$ | Transformer & Mamba | False | 5.80 | 8.18 | 11.60 |
| (iii) | N/A | Transformer only | True | 6.66 | 9.45 | 13.31 |

### 7.2 Complexity Analysis

The complexity of the proposed method can be separated into the complexity of the the transformer block and the complexity of the Mamba block. The complexity of the transformer block is $O((2n-k)D^{2}+(2n-k)^{2}D\rho(G(H)))$ where $\rho(H)$ is the sparsity of the mask matrix. Moreover, complexity of the Mamba blocks is $O((2n-k)DS)$, the total complexity of the model is $O(N_{blocks}(2n-k)D(N_{Mamba}S+N_{transformer}(D+(2n-k)\rho(G(H))))$, since the $N_{Mamba}=N_{transformer}=\frac{1}{2}N_{blocks}$, we expect a significant speedup relative to AECCT and CrossMPT which is $O((2n-k)D^{2}+n(n-k)(\rho(H))$ [34].

**Table 3: Average inference speed in $\mu s$ for ECCT, AECCT, ECCM, and CrossMPT. Lower is better.**
| Code | ECCT | AECCT | CrossMPT | ECCM |
| --- | --- | --- | --- | --- |
| LDPC (49,24) | 237.53 $\mu s$ | 229.21 $\mu s$ | 222.57 $\mu s$ | 72.08 $\mu s$ |
| LDPC (121,60) | 332.23 $\mu s$ | 358.81 $\mu s$ | 289.11 $\mu s$ | 260.42 $\mu s$ |
| BCH (31,16) | 222.57 $\mu s$ | 232.56 $\mu s$ | 217.25 $\mu s$ | 58.78 $\mu s$ |
| BCH (63,45) | 239.18 $\mu s$ | 251.39 $\mu s$ | 227.28 $\mu s$ | 75.42 $\mu s$ |
| Polar (64,48) | 239.18 $\mu s$ | 249.63 $\mu s$ | 227.48 $\mu s$ | 72.34 $\mu s$ |
| Polar (128,96) | 315.56 $\mu s$ | 332.23 $\mu s$ | 272.41 $\mu s$ | 221.98 $\mu s$ |

Validating the claim that the suggested method is more efficient, the following experiment was performed. For each of the methods ECCT, AECCT, CrossMPT, and ECCM - on the same machine apply inference on batches of codewords, with the same number of batches, measure the time that the process took and divide by the number of codewords generated. The process was performed with a batch size of 512 and 200 batches, on a machine with a NVIDIA GeForce RTX 4090 GPU and a Intel i9-14400K. For this test the Early Stopping feature of the model was disabled.

**Table 4: Inference speed at diferent SNRs. Lower is better, the best time is marked in bold.**
| Code | No Early Stopping | 4 [dB] | 5 [dB] | 6 [dB] |
| --- | --- | --- | --- | --- |
| LDPC (49,24) | 72.08 $\mu s$ | 70.26 $\mu s$ | 60.85 $\mu s$ | 59.74 $\mu s$ |
| LDPC (121,60) | 260.42 $\mu s$ | 260.42 $\mu s$ | 253.68 $\mu s$ | 145.78 $\mu s$ |
| BCH (31,16) | 58.78 $\mu s$ | 57.79 $\mu s$ | 57.79 $\mu s$ | 57.79 $\mu s$ |
| BCH (63,45) | 75.42 $\mu s$ | 72.89 $\mu s$ | 62.61 $\mu s$ | 62.61 $\mu s$ |
| Polar (64,48) | 72.34 $\mu s$ | 70.01 $\mu s$ | 62.31 $\mu s$ | 61.62 $\mu s$ |
| Polar (128,96) | 221.98 $\mu s$ | 219.5 $\mu s$ | 197.32 $\mu s$ | 119.11 $\mu s$ |

On the same machine, a similar test was performed but instead of testing on the different methods, the test was performed only on ECCM and at different SNRs [dB]: 4, 5, 6. The intention of this test is to show that the early-stopping feature is meaningful for inference runtime. The test results (Table [4](https://arxiv.org/html/2505.17834v1#S7.T4) indicates that a partial model can still be used without major performance degradation for SNRs 5[dB] and 6[dB], since the speedup in these cases is coming from layers of the model not being used.

### 7.3 Attention Score Comparison

Table: (a) No error

In order to compare our model’s behavior with ECCT [8], we will examine the internal attention scores of the model’s layers in two cases, one where there is no error in the input, and the other where there is a single error in the input. Using this method will show how the attention reacts to error in the input. To visualize the attention across the model, compute the full forward pass of the model with the two inputs, and sum the attention scores across the transformer blocks of the model. For this experiment, evaluate all the layers regardless if the syndrome condition is met in Eq. [29](https://arxiv.org/html/2505.17834v1#S4.E29). Examining the attention maps Fig [5](https://arxiv.org/html/2505.17834v1#S7.T5), we can identify four distinct regions corresponding to the structure of the $g(H)$ mask: *magnitude $\rightarrow$ magnitude* (top-left), *magnitude $\rightarrow$ syndrome* (top-right), *syndrome $\rightarrow$ magnitude* (bottom-left), and *syndrome $\rightarrow$ syndrome* (bottom-right). Each of these regions exhibits different behaviors. Notably, the *syndrome $\rightarrow$ syndrome* attention is consistently strong, indicating that the model relies heavily on the syndrome, in addition, the *magnitude $\rightarrow$ syndrome* attention also remains relatively unchanged regardless of the presence of errors, suggesting that the model treats the syndrome as a reference for interpreting the magnitudes, rather than vice versa. Furthermore, when no error is present, the attention in both the *magnitude $\rightarrow$ magnitude* and *syndrome $\rightarrow$ magnitude* regions is low. This implies that the model has learned to infer the presence or absence of errors primarily from the syndrome. However, when an error is present, there is a clear increase in attention across the corresponding column, indicating that the model has learned to examine the entire parity-check line to locate and assess potential errors. In previous analysis on ECCT [34] the *magnitude$\rightarrow$magnitude* and *syndrome$\rightarrow$syndrome* relations were less significant leading to the design of the mask in CrossMPT. This analysis shows that ECCM is able to leverage those relation in contrast with previous works.

Figure: (a) BCH Code N=63 K=45
Refer to caption: extracted/6471440/Figures/BER_SNR_BCH_63_45.png

## 8 Limitations and Broader Impacts

Limitations: While the proposed ECCM decoder demonstrates strong empirical performance and competitive inference efficiency, several limitations should be noted. First, the model architecture, although designed to generalize across code families, was primarily tested on standard benchmarks with moderate block lengths. Its generalization to very long block codes or non-binary codes remains unverified and may require architectural scaling or retraining. Second, while the hybrid Mamba–Transformer structure improves efficiency over attention-only models, the total model complexity remains non-trivial, and resource-constrained environments (e.g., edge devices) may still face deployment challenges. Broader Impacts: Error correction codes are foundational to reliable communication and data storage. The proposed ECCM method improves both the speed and accuracy of decoding. Accuracy improvements can benefit a wide range of technologies, with deep-space transmissions being a notable example, while speed gains may enable learned decoders in real-time systems. However, the black-box nature of learned decoders like ECCM may pose challenges in safety-critical applications where certifiability and interpretability are essential.

## 9 Conclusions

We introduced ECCM, a hybrid Mamba–Transformer decoder for linear error correction codes. By combining Mamba’s efficient sequential modeling with the global context modeling of Transformers, and incorporating parity-check-aware masking and progressive supervision, ECCM achieves state-of-the-art accuracy while maintaining low and improving inference speed. Experimental results across multiple code families demonstrate consistent improvements over existing neural decoders. These findings highlight the potential of hybrid architectures for real-time, high-accuracy decoding, and open the door to further exploration of structured neural models in communication systems.

## References

- [1]
E. Arikan.
Channel polarization: A method for constructing capacity-achieving codes for symmetric binary-input memoryless channels.
IEEE Transactions on Information Theory, 55(7):3051–3073, 2009.
- [2]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.
Layer normalization, 2016.
- [3]
Amir Bennatan, Yoni Choukroun, and Pavel Kisilev.
Deep learning for decoding of linear codes-a syndrome-based approach.
In 2018 IEEE International Symposium on Information Theory (ISIT), pages 1595–1599. IEEE, 2018.
- [4]
Raj Chandra Bose and Dwijendra K Ray-Chaudhuri.
On a class of error correcting binary group codes.
Information and control, 3(1):68–79, 1960.
- [5]
Andreas Buchberger, Christian Häger, Henry D Pfister, Laurent Schmalen, and Alexandre Graell i Amat.
Pruning and quantizing neural belief propagation decoders.
IEEE Journal on Selected Areas in Communications, 39(7):1957–1966, 2020.
- [6]
Sebastian Cammerer, Tobias Gruber, Jakob Hoydis, and Stephan Ten Brink.
Scaling deep learning-based decoding of polar codes via partitioning.
In GLOBECOM 2017-2017 IEEE global communications conference, pages 1–6. IEEE, 2017.
- [7]
Yoni Choukroun and Lior Wolf.
Denoising diffusion error correction codes, 2022.
- [8]
Yoni Choukroun and Lior Wolf.
Error correction code transformer, 2022.
- [9]
Yoni Choukroun and Lior Wolf.
Deep quantum error correction.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 64–72, 2024.
- [10]
Yoni Choukroun and Lior Wolf.
A foundation model for error correction codes.
In The Twelfth International Conference on Learning Representations, 2024.
- [11]
Yoni Choukroun and Lior Wolf.
Learning linear block error correction codes.
arXiv preprint arXiv:2405.04050, 2024.
- [12]
Jincheng Dai, Kailin Tan, Zhongwei Si, Kai Niu, Mingzhe Chen, H. Vincent Poor, and Shuguang Cui.
Learning to decode protograph ldpc codes.
IEEE Journal on Selected Areas in Communications, 39(7):1983–1999, 2021.
- [13]
Jincheng Dai, Kailin Tan, Zhongwei Si, Kai Niu, Mingzhe Chen, H Vincent Poor, and Shuguang Cui.
Learning to decode protograph ldpc codes.
IEEE Journal on Selected Areas in Communications, 39(7):1983–1999, 2021.
- [14]
Tri Dao and Albert Gu.
Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.
arXiv preprint arXiv:2405.21060, 2024.
- [15]
Robert Gallager.
Low-density parity-check codes.
IRE Transactions on information theory, 8(1):21–28, 2003.
- [16]
Tobias Gruber, Sebastian Cammerer, Jakob Hoydis, and Stephan Ten Brink.
On deep learning-based channel decoding.
In 2017 51st annual conference on information sciences and systems (CISS), pages 1–6. IEEE, 2017.
- [17]
Albert Gu and Tri Dao.
Mamba: Linear-time sequence modeling with selective state spaces.
arXiv preprint arXiv:2312.00752, 2023.
- [18]
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré.
Hippo: Recurrent memory with optimal polynomial projections.
Advances in neural information processing systems, 33:1474–1487, 2020.
- [19]
Albert Gu, Karan Goel, and Christopher Ré.
Efficiently modeling long sequences with structured state spaces.
arXiv preprint arXiv:2111.00396, 2021.
- [20]
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré.
Combining recurrent, convolutional, and continuous-time models with linear state space layers.
Advances in neural information processing systems, 34:572–585, 2021.
- [21]
Michael Helmling and Stefan Scholl.
Database of channel codes and ml simulation results.
[www.uni-kl.de/channel-codes](www.uni-kl.de/channel-codes), 2016.
- [22]
Dan Hendrycks and Kevin Gimpel.
Gaussian error linear units (gelus), 2023.
- [23]
Hyeji Kim, Yihan Jiang, Ranvir Rana, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath.
Communication algorithms via deep learning, 2018.
- [24]
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization, 2017.
- [25]
Hee-Youl Kwak, Jae-Won Kim, Yongjune Kim, Sang-Hyo Kim, and Jong-Seon No.
Neural min-sum decoding for generalized ldpc codes.
IEEE Communications Letters, 26(12):2841–2845, 2022.
- [26]
Hee-Youl Kwak, Dae-Young Yun, Yongjune Kim, Sang-Hyo Kim, and Jong-Seon No.
Boosting learning for ldpc codes to improve the error-floor performance.
Advances in Neural Information Processing Systems, 36:22115–22131, 2023.
- [27]
Matan Levy, Yoni Choukroun, and Lior Wolf.
Accelerating error correction code transformers, 2025.
- [28]
Ilya Loshchilov and Frank Hutter.
Sgdr: Stochastic gradient descent with warm restarts, 2017.
- [29]
Loren Lugosch and Warren J. Gross.
Neural offset min-sum decoding.
In 2017 IEEE International Symposium on Information Theory (ISIT). IEEE, June 2017.
- [30]
Eliya Nachmani, Yair Be’ery, and David Burshtein.
Learning to decode linear codes using deep learning.
In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 341–346. IEEE, 2016.
- [31]
Eliya Nachmani, Elad Marciano, Loren Lugosch, Warren J Gross, David Burshtein, and Yair Be’ery.
Deep learning methods for improved decoding of linear codes.
IEEE Journal of Selected Topics in Signal Processing, 12(1):119–131, 2018.
- [32]
Eliya Nachmani and Lior Wolf.
Autoregressive belief propagation for decoding block codes.
arXiv preprint arXiv:2103.11780, 2021.
- [33]
Seong-Joon Park, Hee-Youl Kwak, Sang-Hyo Kim, Sunghwan Kim, Yongjune Kim, and Jong-Seon No.
How to mask in error correction code transformer: Systematic and double masking.
arXiv preprint arXiv:2308.08128, 2023.
- [34]
Seong-Joon Park, Hee-Youl Kwak, Sang-Hyo Kim, Yongjune Kim, and Jong-Seon No.
Crossmpt: Cross-attention message-passing transformer for error correcting codes.
arXiv preprint arXiv:2405.01033, 2024.
- [35]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.
- [36]
Ziyan Zheng, Chin Wa Lau, Nian Guo, Xiang Shi, and Shao-Lun Huang.
White-box error correction code transformer.
In The Second Conference on Parsimony and Learning (Proceedings Track).

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 2
presentation: 2
questions: I do not have any questions. The paper is technically correct, but in my opinion, the contribution is too limited to warrant acceptance.
soundness: 2
strengths: The combination of Mamba layers with Transformer layers is novel in the context of ECC decoding. This hybrid architecture appears to improve both decoding performance and computational efficiency relative to previous machine learning-based approaches.
summary: This paper proposes a hybrid deep learning-based decoder for error-correcting codes, combining the Mamba architecture with Transformer layers. The proposed model aims to leverage Mamba's efficient sequential modeling while maintaining the global contextual modeling of Transformers. The method is evaluated on a range of binary linear block codes, including BCH, Polar, LDPC, and MacKay codes. The authors report improved decoding performance and inference speed relative to previous neural network-based decoders, such as Transformer-only architectures and the CrossMPT model.
weaknesses: I also reviewed this paper for NeurIPS (Reviewer ID: u6gT). This submission appears essentially identical to the NeurIPS version, which was, in my view, justifiably rejected.

While the paper is technically correct (though it could have been written more clearly) and the proposed hybrid architecture is interesting, its contribution remains limited in both novelty and impact. The literature on machine learning-based decoders for error-correcting codes is now very extensive, and this work does not meaningfully advance the state-of-the-art, either in methodology or insight.

Overall, the paper is solid but rather incremental. It would be more appropriate for a specialized or mid-tier venue rather than a top-tier conference like ICLR, which typically expects stronger conceptual contributions or more significant performance improvements (the proposed decoder does not beat state-of-the-art decoders).

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 1
questions: 1. **Runtime evidence:** Were empirical measurements (e.g., samples/sec, latency, GPU memory) taken to support the “*low and improving inference speed*” claim? If so, please include them or provide a runtime benchmark appendix.
2. **Clarification on the SCL Baseline for Polar Code Comparison:** In Appendix 12.2 and Table 4, the paper compares ECCM and CrossMPT against a classical "SCL (L=32)" baseline. Based on this, the authors conclude that ECCM "meaningfully closes the distance to the strong SCL baseline". However, the baseline is not specified as being CRC-Aided (CA-SCL), which is the standard high-performance SCL decoder configuration in modern literature. Could the authors clarify whether the "SCL (L=32)" baseline used in Table 4 includes a CRC? If it does not, how can the claim of closing the gap against a "strong SCL baseline" be justified, given that a ‘plain’ SCL is a significantly weaker baseline than the standard CA-SCL?
soundness: 3
strengths: 1. **Novel and Timely Architecture:** The integration of Mamba, a recent and efficient SSM, into the ECC decoding problem is a new architecture for model-free channel decoders. The hybrid approach achieves strong decoding performance. 
2. **Strong Empirical Performance:** The model achieves highly competitive, and in the case of several BCH codes, state-of-the-art performance compared to strong Transformer-based baselines like AECCT and CrossMPT. This validates the practical efficacy of the paper’s approach.
summary: This paper proposes the Error-Correcting Code Mamba-Transformer (ECCM), a hybrid neural decoder architecture for linear block codes. The design alternates between Mamba (State-Space Model) layers and standard Transformer attention layers, aiming to leverage the linear-time complexity of Mamba for sequential modeling while retaining the global contextual modeling capabilities of the Transformer.

The authors claim three primary contributions: 1) The **hybrid Mamba-Transformer architecture** itself, 2) a new **layer-wise mask for Mamba layers,** **$f(H)$,** derived from parity check matrix, 3) a **progressive layer-wise loss** **function** combined with a syndrome-based early-exit mechanism to stabilize training and reduce inference latency.

Experiments are conducted on BCH, Polar, and LDPC codes. The results show that the proposed ECCM model achieves performance that is on par with, or in some cases (notably BCH) superior to, existing Transformer-only decoders, while offering improved computational complexity.
weaknesses: 1. **Inconsistencies, Typos, and Presentation Issues:** The paper suffers from many typographical errors and presentation issues that reduce clarity and undermine its professional quality.
    - **Formulation and Notation Errors**
        - Eq (30) & (31): The calculation of error vector $z$ and the estimated codeword $\hat{c}^{i}$ should reference the original received vector $y[]$, not the model input $y_{in}[]$ since $y_{in}=[|y|, s]$.
        - Eq (28): The output dimension $N$ of $o^{i}$ should be $n$, given its relationship to the error vector $z$.
        - Eq (4): The notation uses a semicolon ($;$), which conventionally denotes vertical concatenation, whereas the operation required by the stated dimensions is horizontal concatenation.
        - Eq (31): The iterator should use a different variable than $n$ (e.g., $m$), and the upper value should be $n$ (code length).
        - Eq (28) and (29): The paper used the following rule for hard-decision vector: positive y → 0 (binary) and negative y → 1 (on page 3). Because of this rule, it seems that (28) should be $s^i = H \cdot (o^i < 0.5)$ instead of $s^i = H \cdot (o^i > 0.5)$.
    - **Figure-Text Inconsistency**
        
        In Figure 1(b), the Mamba block diagram is misleading. It correctly shows a SiLU activation for the $z$ path, but omits the SiLU activation for the $u$ path after the Conv block. This contradicts the text, as Eq (7) explicitly states $u_{conv}^{i}=SiLU(Conv^{i}(u^{i}))$.
        
    - **Poor Presentation**
    In Figure 1(b), the final output label ($y^i$) is poorly rendered and overlaps with the output arrow, demonstrating a lack of attention to detail and reducing the diagram's readability.
Definition discrepancy in Mamba mask $f(H)$: The mathematical formulation in Eq (13) and (14) is misleading. The paper’s Mamba mask $f(H)$ is defined with dimensions $(n-k) \times (2n-k)$, yet the paper applies it using indices $d $ and $s $ (model/state dimensions), as seen in $f(H)[l, d]$ and $f(H)[l, s]$. This indexing is dimensionally inconsistent with the mask's definition.
3. **Lack of empirical runtime metrics:** The conclusion states that ECCM achieves *“low and improving inference speed”*. However, the main text provides only the big-O complexity (Section 7.2) without any empirical latency, throughput, or FLOPs.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 1
questions: 1. In Sec: 4.2 and 4.3 the authors state that if s_i=s the sample stops, and later outputs are “not calculated” and “not summed” in the loss. Could you add a short pseudocode block showing how this is implemented per sample within a batch during training and testing? And, could you please add a brief layer-usage summary showing where samples stop (i_"last" ) in particular, how many samples terminate before the final block?
2. For Table 2, could you explicitly state the number of blocks used in the ‘Transformer only’ and ‘Mamba only’ settings?
3. The authors train the Hybrid model with Adam at 2.5×10^-4and cosine annealing down to  10^-10(1000 batches/epoch). Could you briefly justify diverging from ECCT/CrossMPT LR/schedules? For Table 2, could you confirm that the ‘Transformer-only’ used the same LR/schedule? For comparability, please add a control run using the ECCT/CrossMPT learning rate and schedule.
soundness: 3
strengths: The paper is novel as it introduces the first use of Mamba architecture for ECC decoding. The approach shows consistent improvements across various code families, with experiments clearly validating the impact of each component including the Mamba architecture, masking strategy, and layer-wise loss, supporting the authors’ design choices.
summary: This paper presents ECCM, a hybrid decoder that integrates Mamba and Transformer architectures for error correction code (ECC) decoding, marking the first use of Mamba’s state-space model in this domain. By combining Mamba’s efficient sequential processing with Transformer’s global attention, the method improves computational efficiency and captures long-range bit dependencies. The authors introduce a new masking strategy for Mamba layers, alternate Mamba and Transformer blocks, and apply progressive layer-wise loss, showing improved performance.
weaknesses: Despite the interesting idea, the paper has some clarity issues that make the method difficult to understand and implement correctly. These issues are detailed further in the questions section.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 4
questions: Would the authors say that this method remains unproven for real applications, given the complexity of transformers? Where do they see these approaches being used for real world deployments? Server-side decoder deployments may be unrealistic given latency and transmission requirements for wireless systems, is this intended for the edge?
soundness: 3
strengths: The paper's approach outperforms the state of the art and is shown to be the optimal design given the components the authors have used - ablations demonstrate that removal of components from this system degrades performance. The bit-error based binary cross entropy loss ensures decoder training remains stable.
summary: The paper presents a hybrid Mamba-Transformer architecture for decoding error correcting codes in wireless transmissions. This scheme is ablation tested by removing components - only the Mamba, the transformer are tested against the use of both. The unique characteristics of SSMs allowing retention of very long context complements the transformer's ability to fit to prior training data in a shorter context window with high fidelity, allowing high decode fidelity here. The approach is contrasted against prior art using machine learning methods for error correcting code decoding.
weaknesses: The given architecture seems much more complex than prior art, and given that channel noise decoding and wireless systems are real-time systems, I would ask what the latency is and whether this high-performance algorithm is suitable for the applications the authors have envisioned. 

I also believe the paper would benefit from a hyperparameter sweep of the network, as it is the experiments seem to have been conducted for a given parameter set but we do not know if that is the optimal one. A balance of latency and quality would be necessary for these applications, and a Pareto plot of architecture/parameter count normalized by latency plotted against quality may be illustrative.


Decision: Reject