CrossMPT: Cross-attention Message-passing Transformer for Error Correcting Codes

## Abstract

Abstract Error correcting codes (ECCs) are indispensable for reliable transmission in communication systems. The recent advancements in deep learning have catalyzed the exploration of ECC decoders based on neural networks. Among these, transformer-based neural decoders have achieved state-of-the-art decoding performance.
In this paper, we propose a novel Cross-attention Message-Passing Transformer (CrossMPT), which shares key operational principles with conventional message-passing decoders.
While conventional transformer-based decoders employ self-attention mechanism without distinguishing between the types of input vectors (i.e., magnitude and syndrome vectors), CrossMPT updates the two types of input vectors separately and iteratively using two masked cross-attention blocks. The mask matrices are determined by the code’s parity-check matrix, which explicitly captures the irrelevant relationship between two input vectors.
Our experimental results show that CrossMPT significantly outperforms existing neural network-based decoders for various code classes.
Notably, CrossMPT achieves this decoding performance improvement, while significantly reducing the memory usage, complexity, inference time, and training time.

## 1 Introduction

The fundamental objective of digital communication systems is to reliably transmit information from source to destination through noisy channels.
Error correcting codes (ECCs) are crucial for ensuring the integrity of transmitted data in digital communication systems.
The advancements in deep learning across diverse tasks, such as natural language processing (NLP), image classification, or object detection (Devlin et al., ; He et al., ; Girshick et al., ; Carion et al., ), have motivated the application of deep learning techniques to ECC decoders.
This has led to the development of neural decoders (Kim et al., ; ; Nachmani et al., ; ; Dai et al., ; Lugosch & Gross, ).
The key aim of these neural decoders is to improve decoding performance by overcoming limitations of the conventional decoders such as belief propagation (BP) (Richardson & Urbanke, ) or min-sum (MS) (Fossorier et al., ) decoders.

Among neural decoders, model-free neural decoders employ an arbitrary neural network architecture (e.g., deep neural networks (Gruber et al., ), recurrent neural networks (Bennatan et al., ) and transformers (Choukroun & Wolf, ; ; Park et al., ; Choukroun & Wolf, ; )) as the ECC decoder, without relying on prior knowledge of specific decoding algorithms.
Since model-free neural decoders are not based on specific decoding algorithms, their training is prone to overfitting, largely due to the exponentially large number of codewords (Bennatan et al., ).
To circumvent overfitting, these neural decoders incorporate a preprocessing step where the magnitude and syndrome vectors from the received codeword are concatenated and used as inputs.
The preprocessing step is essential for integrating an effective network architecture for the ECC decoder without an overfitting issue (Bennatan et al., ).
For example, transformer-based ECC decoders (Choukroun & Wolf, ; ; Park et al., ; Choukroun & Wolf, ; ) achieve state-of-the-art decoding performance.
However, two important questions have not been addressed: 1) how to effectively manage the two distinct input vectors (magnitude and syndrome), and 2) how to design an efficient transformer-based decoder architecture.

Conventional transformer-based ECC decoders, initially proposed as Error Correction Code Transformer (ECCT) (Choukroun & Wolf, ; ; Park et al., ; Choukroun & Wolf, ; ), receive the concatenated magnitude and syndrome embeddings as a single input and utilize self-attention blocks, without a distinct process for handling the two different types of vectors.
In contrast, our approach treats the magnitude and syndrome as *multimodal data*, recognizing their distinct informational characteristics.
The *real-valued* magnitude vector contains the reliabilities of all bit positions, while the *binary* syndrome vector conveys the information of erroneous bit positions.
This deliberate separation necessitates the development of a novel architecture, specifically designed to effectively update these separated magnitude and syndrome embeddings, thereby significantly improving decoding performance.

In this paper, we introduce a novel Cross-attention Message-Passing Transformer (CrossMPT) for ECC decoding.
CrossMPT processes the magnitude and syndrome separately to effectively utilize their distinct informational properties.
It employs two *cross-attention blocks* to iteratively update the magnitude and syndrome embeddings.
Initially, the magnitude embedding is encoded into the *query*, while the syndrome embedding is encoded into *key* and *value*.
The first cross-attention block utilizes this configuration in its attention mechanism to update the magnitude embedding component.
This procedure is reciprocated for the syndrome embedding, which is encoded into the query, while the magnitude embedding is encoded into the key and value.
This configuration enables the second cross-attention block to update the syndrome vector component.
These two masked cross-attention blocks iteratively collaborate to refine the magnitude and syndrome embeddings as in the message-passing algorithm (Richardson & Urbanke, ).

To facilitate training, CrossMPT employs a mask matrix for each cross-attention block.
The first cross-attention block uses the transpose of the parity check matrix (PCM) $H^{T}$ as its mask matrix with the magnitude embedding as the query.
In the second cross-attention block, the PCM $H^{\top}$ itself is applied as the mask matrix, with the syndrome embedding acting as the query.
This strategy leverages the PCM’s inherent representation of the ‘magnitude-syndrome’ relationship, effectively aligning with the architecture’s objectives.
Moreover, the combined size of the two attention maps of CrossMPT is at most half that of the attention map of the conventional transformer-decoder, leading to significantly reduced memory usage.
This reduction enables efficient learning and decoding of longer codes, which previous approaches (concatenating magnitude and syndrome embeddings) are unable to achieve due to high memory usage and computational complexity.
To our knowledge, CrossMPT is the first architecture to integrate an iterative message-passing framework with a cross-attention-based transformer architecture.

Experimental results show that CrossMPT consistently outperforms the original ECCT across various code classes.
Leveraging its shared operational principles with the message-passing algorithm, CrossMPT demonstrates particularly improved decoding performance, especially in low-density parity-check (LDPC) codes.
Notably, we also demonstrate that CrossMPT closely approaches the maximum likelihood decoding performance on short codes.
In addition to its enhanced decoding performance, CrossMPT significantly reduces the computational complexity (e.g., floating point operations (FLOPs), training time, and inference time) of the decoder layer compared to the original ECCT.
Given that the decoder layer constitutes a substantial portion of the total computational cost, this reduction leads to a significant decrease in overall computational complexity.

## 2 Related Works

In the field of neural network-based ECC decoders, there are two primary categories: the model-based decoder and the model-free decoder.
First, model-based decoders are constructed based on the conventional decoding methods (e.g., BP decoder and MS decoder).
They map the iterative decoding process of the conventional decoding methods into neural networks and train the network weights accordingly.
To improve performance over the standard BP decoder, the recurrent neural network was employed for the decoding of BCH codes (Nachmani et al., ).
Several recent studies showed that neural network-based BP and MS decoders outperform the conventional decoding algorithms over various code types (Dai et al., ; Kwak et al., ; Lugosch & Gross, ; Nachmani & Wolf, ; ; Kwak et al., ; Buchberger et al., ).
However, model-based neural decoders may encounter performance limitations due to their restrictive model architectures, which are closely tied to underlying decoding methods.

Unlike model-based decoders, model-free neural decoders employ arbitrary neural network architectures to learn the decoding without relying on specific decoding algorithms.
Model-based neural decoders are constrained by the inherent limitations of the underlying decoding method (e.g., BP).
Consequently, they are unlikely to significantly exceed the performance limitations of traditional decoding algorithms.
On the other hand, model-free neural decoders can leverage state-of-the-art neural network architectures without such constraints.
Previous approaches (Gruber et al., ; Cammerer et al., ; Kim et al., ) implemented fully-connected network to decode codes but faced challenges in training due to overfitting.
Subsequently, the introduction of a preprocessing step utilizing the magnitude and syndrome vectors of the received codeword to learn multiplicative noise has been pivotal in enabling model-free decoders to address the overfitting issue (Bennatan et al., ).
Then, ECCT (Choukroun & Wolf, ) first employed the transformer architecture using the same preprocessing step and demonstrated that the transformer-based decoder outperforms existing decoders including model-based neural decoders.
Building on the ECCT framework, denoising diffusion error correction codes (Choukroun & Wolf, ) interpreted the decoding process as a diffusion process and incorporated a diffusion model to train the original ECCT.
Recently, double-masked ECCT (Park et al., ) utilized two different PCMs for the same linear code to capture the diverse multilateral relationships of the magnitude and syndrome bits and improve the decoding performance.
Notably, transformer-based decoders outperform model-based neural decoders and serve as universal decoders capable of decoding arbitrary code classes with a unified architecture.

## 3 Background

### 3.1 Error Correcting Codes

Let $C$ be a linear block code, which is defined by a generator matrix $G$ of size $k\times n$ and a parity check matrix $H$ of size $(n-k)\times n$.
They satisfy $GH^{\top}=0$ over $\{0,1\}$ with modulo $2$ addition.
A codeword $x\in C\subset\{0,1\}^{n}$ is encoded by multiplying message $m$ with the generator matrix $G$ (i.e., $x=mG$).
Let $x_{s}$ be the binary phase shift keying (BPSK) modulated signal of $x$ and let $y$ be the output of a noisy channel for input $x_{s}$. We assume the additive white Gaussian noise (AWGN) channel and the channel output can be represented by $y=x_{s}+z$, where $z\sim N(0,\sigma^{2})$.
The objective of the decoder ($f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$) is to recover the transmitted codeword $x$ by correcting errors.
When $y$ is received, the decoder first determines whether the received signal is corrupted or not by checking the syndrome $s(y)=Hy_{b}$, where $y_{b}=\text{bin}(\text{sign}(y))$ is the demodulated signal of $y$. Here, $\text{sign}(a)$ represents $+1$ if $a\geq 0$ and $-1$ otherwise and $\text{bin}(-1)=1$, $\text{bin}(+1)=0$.
If $s(y)$ is a non-zero vector, it is detected that $y$ is corrupted during the transmission, and the decoder initiates the error correction process.

Figure: (a) ECCT (Choukroun & Wolf, )
Refer to caption: x1.png

Figure: Figure 2: Architecture of CrossMPT.
Refer to caption: x3.png

### 3.2 Error Correction Code Transformer

ECCT is the first approach to present a model-free decoder with the transformer architecture.
ECCT outperforms other neural BP-based decoders by employing the masked self-attention mechanism, whose mask matrix is determined by the code’s PCM (Choukroun & Wolf, ).
The first issue that needs to be addressed when training transformer-based decoders is the overfitting.
In (Bennatan et al., ), the overfitting issue in model-free neural decoders is described as a poor generalization to untrained new codewords due to the exponentially large number of codewords.
However, it has been resolved by a preprocessing technique that facilitates a syndrome-based decoding (Bennatan et al., ).
It has been theoretically proven that, with this preprocessing step, the decoder’s performance is invariant to the specific codewords in the training set (Bennatan et al., ).

As in (Bennatan et al., ), the preprocessing step of ECCT utilizes the magnitude and syndrome vectors to train multiplicative noise $\tilde{z}_{s}$, which is defined by

$$ $y=x_{s}+z=x_{s}\tilde{z}_{s}.$ (1) $$

ECCT aims to estimate the multiplicative noise in (1), i.e., $f(y)=\hat{z}_{s}$.
Then, the estimation of $x$ is
$\hat{x}=\text{bin}(\text{sign}(yf(y)))$.
If the multiplicative noise is correctly estimated such that $\text{sign}(\tilde{z}_{s})=\text{sign}(\hat{z}_{s})$, then $\hat{x}$ can be computed as:

$$ $\hat{x}=\text{bin}(\text{sign}(yf(y)))=\text{bin}(\text{sign}(x_{s}\tilde{z}_{ s}\hat{z}_{s}))=\text{bin}(\text{sign}(x_{s}))=x.$ $$

ECCT employs a masked self-attention module to train the transformer architecture, where the input embedding is the concatenation of the embedded magnitude and syndrome vector of $y$.
As shown in Figure 1, the mask matrice of ECCT should clearly distinguish between necessary (unmasked) and unnecessary (masked) pairwise relationships among magnitude-magnitude, magnitude-syndrome, and syndrome-syndrome bit relations.
In ECCT, the syndrome-syndrome part was only unmasked for self-relations, while the magnitude-syndrome part was unmasked based on the connections defined by the parity-check matrix (PCM).
The magnitude-magnitude part, however, was unmasked for bit pairs connected at depth 2 (see Algorithm 1 in Choukroun & Wolf ()).
While the masking of magnitude-syndrome relations is intuitive, as it directly uses PCM, determining the relationships among magnitude themselves is not directly derivable from the PCM.
Therefore, the algorithm for masking magnitude-magnitude part is neither straightforward nor unique.
In Figure 1, the white areas indicate unmasked positions (require the attention calculation) whereas the blue areas represent masked positions (omit the attention calculation).
As the proportion of blue increases, the attention matrix becomes sparser and more cost-efficient.

## 4 Cross-attention Message-Passing Transformer

In this section, we present the operational mechanism and architecture of CrossMPT.
CrossMPT handles the magnitude and syndrome embeddings separately, applying a cross-attention mechanism to effectively capture their distinct information.
It shares its core principles with message passing decoding algorithm for decoding linear codes, where the magnitude and syndrome embeddings of the received codewords are iteratively updated.
The overall architecture is illustrated in Figure 2.

### 4.1 Cross-attention Message-passing Transformer

One cross-attention block updates the magnitude embedding by using it as a query and updates them with key and value, generated from the syndrome embedding.
Given this configuration, the attention map has the size $n\times(n-k)$, effectively representing the ‘magnitude-syndrome’ relation.
To reflect this relationship, we employ the transpose of the PCM $H^{\top}$ as the mask matrix.
This is because the $n$ rows of $H^{\top}$ correspond to the $n$ bit positions, and its $n-k$ columns are associated with the parity check equations, directly linking to $|y|$ and $s(y)$, respectively.
The other cross-attention block similarly use $s(y)$ is used as the query, while $|y|$ serves as both key and value.
For this operation, we utilize the PCM $H$ as the mask matrix.

This configuration of separately handling two distinct informational properties resembles the message-passing decoding algorithm for decoding linear codes.
Message-passing algorithms such as the sum-product algorithm (Richardson & Urbanke, ) are widely used for decoding ECCs due to their outstanding decoding performance with low complexity.
The message-passing algorithm operates by exchanging messages between variable nodes (VNs) and check nodes (CNs) over a Tanner (bipartite) graph (Richardson & Urbanke, ).
In the Tanner graph, VNs convey information about the reliability of the received codeword, while CNs indicate the parity check equations.
The edges between VNs and CNs represent the connections (relationships) between them.
The message-passing decoder operates by exchanging messages between VNs and CNs via edges.
The output messages of VNs and CNs are updated in an iterative manner.

Similar to the principles of message-passing algorithms, CrossMPT updates $|y|$ and $s(y)$ by allowing them to exchange messages with each other.
Initially, we update $|y|$ by the masked cross-attention block, with $|y|$ as the query and $s(y)$ as both the key and value.
The syndrome embedding is updated in the subsequent masked cross-attention block, utilizing the previously updated magnitude embedding.
In this block, the syndrome embedding is used as the query, while the updated magnitude embedding serves as the key and value.
The resulting output from this cross-attention block is the updated syndrome.
CrossMPT iteratively updates both the magnitude and syndrome embeddings to identify the multiplicative noise accurately.

As a representative of the message-passing algorithm, Figure 3 depicts the sum-product algorithm and the cross-attention message-passing algorithm.
In the sum-product algorithm, the VN output and CN output messages are iteratively updated using the sum and product operations.
Similar to the sum-product algorithm, the magnitude and syndrome embeddings are iteratively updated using the masked cross-attention (Masked CA in the figure) blocks in CrossMPT.
Note that $H$ and $H^{\top}$ are utilized for the mask matrices for these cross-attention blocks, and ${Q}$, ${K}$, and ${V}$ represent the query, key, and value of the cross-attention mechanism.

Figure: Figure 3: Conceptual comparison of the sum-product message-passing algorithm and the proposed cross-attention (CA) message-passing algorithm.
Refer to caption: x4.png

### 4.2 Model Architecture

In the initial embedding layer, we generate $|y|=(|y_{1}|,\ldots,|y_{n}|)$ and $s(y)=(s(y)_{1},\ldots,s(y)_{n-k})$ from the received codeword, and project each element $y_{i}$ and $s(y)_{i}$ into $d$ dimension embedding row vectors $M_{i}$ and $S_{i}$, respectively, as follows:

$$ $\displaystyle M_{i}=|y_{i}|W_{i},\qquad~{}~{}\text{ for }i=1,\ldots,n,$ $\displaystyle S_{i}=s(y)_{i}W_{i+n},\quad\text{ for }i=1,\ldots,n-k,$ $$

where $W_{i}\in\mathbb{R}^{1\times d}$ for $i=1,\ldots,2n-k$ denote the trainable positional encoding vector.

These two embedded vectors are processed as separate input vectors in the following $N$ decoding layers.
Each decoding layer contains two cross-attention blocks, each consisting of a cross-attention module, a feed-forward neural network (FFNN), and a normalization layer.

In the first cross-attention module, the attention module updates the ‘magnitude’ embedding by using the syndrome.
The query vector $Q_{1}$, key vector $K_{1}$, and value vector $V_{1}$ are assigned as follows:

$$ $Q_{1}=MW_{Q},K_{1}=SW_{K},V_{1}=SW_{V},$ $$

where $M=[M_{1};\cdots;M_{n}]\in\mathbb{R}^{n\times d}$ and $S=[S_{1};\cdots;S_{n-k}]\in\mathbb{R}^{(n-k)\times d}$ denote the magnitude embedding and the syndrome embedding, respectively, and $W_{Q},W_{K},W_{V}$ denote the weight matrices of query, key, and value, respectively.
This architecture is termed the ‘cross-attention’ message-passing transformer since the query corresponds to the magnitude embedding, while the key and value correspond to the syndrome embedding. Then, we employ the following scaled dot-product attention:

$$ ${\rm Attention}(Q_{1},K_{1},V_{1})={\rm softmax}\left(\dfrac{Q_{1}K_{1}^{\top} +g(H^{\top})}{\sqrt{d}}\right)V_{1},$ $$

where $g(H^{\top})$ is the mask matrix, and the function $g$ is defined as

$$ $g(A)_{i,j}=\begin{cases}0\qquad~{}{\rm if}~{}A_{i,j}=1,\\ -\infty\quad{\rm if}~{}A_{i,j}=0.\end{cases}$ (2) $$

This configuration results in an attention map of size $n\times(n-k)$, representing the ‘magnitude-syndrome’ relationship.
Therefore, we use the transpose of the PCM $H^{\top}$ as a mask matrix since the $n$ rows of $H^{\top}$ correspond to the $n$ bit positions and the $n-k$ columns of $H^{\top}$ to the parity check equations, which are closely related to $|y|$ and $s(y)$, respectively.
Finally, the output vector embodies a newly updated magnitude embedding $M^{\prime}$ by using the syndrome.

In the second cross-attention module, we update the ‘syndrome’ embedding with the updated $M’$ corresponding to the magnitude. In other words, the input of the query becomes syndrome and the input of key and value becomes $M’$. We use the shared weight vectors $W_{Q},W_{K},W_{V}$ as in the first cross-attention module, and query vector $Q_{2}$, key vector $K_{2}$, and value vector $V_{2}$ are defined as follows:

$$ $Q_{2}=SW_{Q},K_{2}=M^{\prime}W_{K},V_{2}=M^{\prime}W_{V}.$ $$

Here, the syndrome and magnitude correspond the row and column of the attention map, respectively.
Thus, we employ the mask matrix $g(H)$, whose masking positions are zeros in $H$. Then we apply the scaled dot-product attention and the resulting output vector conveys the updated syndrome.
This output vector is utilized to further refine the magnitude embedding and this process is iteratively repeated across the $N$ decoder layers.

Finally, these output vectors of the last decoder layer are concatenated and pass through a normalization layer and two fully connected (FC) layers.
The first FC layer reduces the $(2n-k)\times d$ dimension embedding to a one-dimensional $2n-k$ vector, and the second FC layer further reduces the dimension from $2n-k$ into $n$.
The final output provides an estimation of $\tilde{z}_{s}$.
Since two cross-attention blocks of CrossMPT share the same weight matrices $W_{Q},W_{K},W_{V}$ and all other layers, CrossMPT has the same number of parameters as the original ECCT.

### 4.3 Training

The objective of the proposed decoder is to learn the multiplicative noise $\tilde{z}_{s}$ in (1) and reconstruct the original transmitted signal $x$.
We can obtain the multiplicative noise by $\tilde{z}_{s}=\tilde{z}_{s}x^{2}_{s}=yx_{s}$.
Then, the target multiplicative noise for binary cross-entropy loss function is defined by $\tilde{z}=\text{bin}(\text{sign}(yx_{s}))$.
Finally, the cross-entropy loss function for a received codeword $y$ is defined by

$$ $\mathcal{L}=-\sum^{n}_{i=1}\left\{\tilde{z}_{i}\log(\sigma(f(y)))+(1-\tilde{z} _{i})\log(1-\sigma(f(y)))\right\}.$ $$

To ensure a fair comparison between CrossMPT and ECCT, we adopt the same training setup used in the previous work (Choukroun & Wolf, ).
We use the Adam optimizer (Kingma & Ba, ) and conduct 1000 epochs.
Each epoch consists of 1000 minibatches, where each minibatch is composed of 128 samples.
All simulations were conducted using NVIDIA GeForce RTX 3090 GPU and AMD Ryzen 9 5950X 16-Core Processor CPU.
The training sample $y$ is generated by $y=x_{s}+z$, where $x_{s}$ is the all-zero codeword and the AWGN channel noise $z$ is from an SNR ($E_{b}/N_{0}$) range of $3$ dB to $7$ dB.
The learning rate is initially set to $10^{-4}$ and gradually reduced to $5\times 10^{-7}$ following a cosine decay scheduler.

**Table 1: Comparison of decoding performance at three different SNR values (4 dB, 5 dB, 6 dB) for BP decoder, Hyper BP decoder (Nachmani & Wolf, ), AR BP decoder (Nachmani & Wolf, ), ECCT (Choukroun & Wolf, ), and the proposed CrossMPT. The results are measured by the negative natural logarithm of BER. The best results are highlighted in bold. Higher is better.**
| Architecture | BP-based decoders | Model-free decoders |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Codes | Parameter | BP | Hyp BP | AR BP | ECCT | CrossMPT |  |  |  |  |  |  |  |  |  |  |
| 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 |  |  |
| BCH | (31,16) | 4.63 | 5.88 | 7.60 | 5.05 | 6.64 | 8.80 | 5.48 | 7.37 | 9.60 | 6.39 | 8.29 | 10.66 | 6.98 | 9.25 | 12.48 |
| (63,36) | 4.03 | 5.42 | 7.26 | 4.29 | 5.91 | 8.01 | 4.57 | 6.39 | 8.92 | 4.86 | 6.65 | 9.10 | 5.03 | 6.91 | 9.37 |  |
| (63,45) | 4.36 | 5.55 | 7.26 | 4.64 | 6.27 | 8.51 | 4.97 | 6.90 | 9.41 | 5.60 | 7.79 | 10.93 | 5.90 | 8.20 | 11.62 |  |
|  | (63,51) | 4.5 | 5.82 | 7.42 | 4.8 | 6.44 | 8.58 | 5.17 | 7.16 | 9.53 | 5.66 | 7.89 | 11.01 | 5.78 | 8.08 | 11.41 |
| Polar | (64,32) | 4.26 | 5.38 | 6.50 | 4.59 | 6.10 | 7.69 | 5.57 | 7.43 | 9.82 | 6.99 | 9.44 | 12.32 | 7.50 | 9.97 | 13.31 |
| (64,48) | 4.74 | 5.94 | 7.42 | 4.92 | 6.44 | 8.39 | 5.41 | 7.19 | 9.30 | 6.36 | 8.46 | 11.09 | 6.51 | 8.70 | 11.31 |  |
| (128,64) | 4.1 | 5.11 | 6.15 | 4.52 | 6.12 | 8.25 | 4.84 | 6.78 | 9.3 | 5.92 | 8.64 | 12.18 | 7.52 | 11.21 | 14.76 |  |
| (128,86) | 4.49 | 5.65 | 6.97 | 4.95 | 6.84 | 9.28 | 5.39 | 7.37 | 10.13 | 6.31 | 9.01 | 12.45 | 7.51 | 10.83 | 15.24 |  |
| (128,96) | 4.61 | 5.79 | 7.08 | 4.94 | 6.76 | 9.09 | 5.27 | 7.44 | 10.2 | 6.31 | 9.12 | 12.47 | 7.15 | 10.15 | 13.13 |  |
| LDPC | (49,24) | 6.23 | 8.19 | 11.72 | 6.23 | 8.54 | 11.95 | 6.58 | 9.39 | 12.39 | 6.13 | 8.71 | 12.10 | 6.68 | 9.52 | 13.19 |
| (121,60) | 4.82 | 7.21 | 10.87 | 5.22 | 8.29 | 13.00 | 5.22 | 8.31 | 13.07 | 5.17 | 8.31 | 13.30 | 5.74 | 9.26 | 14.78 |  |
| (121,70) | 5.88 | 8.76 | 13.04 | 6.39 | 9.81 | 14.04 | 6.45 | 10.01 | 14.77 | 6.40 | 10.21 | 16.11 | 7.06 | 11.39 | 17.52 |  |
| (121,80) | 6.66 | 9.82 | 13.98 | 6.95 | 10.68 | 15.80 | 7.22 | 11.03 | 15.90 | 7.41 | 11.51 | 16.44 | 7.99 | 12.75 | 18.15 |  |
| MacKay | (96,48) | 6.84 | 9.40 | 12.57 | 7.19 | 10.02 | 13.16 | 7.43 | 10.65 | 14.65 | 7.38 | 10.72 | 14.83 | 7.97 | 11.77 | 15.52 |
| CCSDS | (128,64) | 6.55 | 9.65 | 13.78 | 6.99 | 10.57 | 15.27 | 7.25 | 10.99 | 16.36 | 6.88 | 10.90 | 15.90 | 7.68 | 11.88 | 17.50 |
| Turbo | (132,40) | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 4.74 | 6.54 | 9.06 | 5.55 | 7.92 | 10.94 |

Figure: (a) $(31,16)$ BCH code
Refer to caption: x5.png

## 5 Experimental Results

In this section, we compare the proposed CrossMPT with the original ECCT across various code classes.
Our experimental results do not include a comparison with the works of (Choukroun & Wolf, ; ), as they have different objectives, such as generalizing the decoder to unseen codes (Choukroun & Wolf, ) or jointly training the encoder and decoder (Choukroun & Wolf, ).
It is worth mentioning that our cross-attention architecture and the schemes of (Choukroun & Wolf, ; ) are orthogonal methods, and combining them could present a promising direction for future research.

To verify the efficacy of CrossMPT, we train it for BCH codes, polar codes, turbo codes, and LDPC codes (including MacKay and CCSDS codes) and evaluate the bit error rate (BER) performance.
All PCMs are taken from (Helmling et al., ).
The implementation of the original ECCT is taken from (Choukroun & Wolf, ).
For the testing, we collect at least 500 frame errors at each signal-to-noise ratio (SNR) value with random codewords.
Table 1 compares the decoding performance of CrossMPT with the BP decoder, BP-based neural decoders (Nachmani & Wolf, ; ), and ECCT (Choukroun & Wolf, ).
The results of the BP-based decoders in Table 1 are obtained for 50 iterations.
The results for both the proposed CrossMPT and ECCT, which are model-free decoders, are obtained with $N=6$ and $d=128$.
For all types of codes, CrossMPT outperforms the conventional ECCT and all the other BP-based neural decoders.
This improvement of CrossMPT is particularly notable in the case of LDPC codes.
To provide more visual information, we plot the BER graphs for several codes in Figure 4.

An important aspect of our research is CrossMPT’s capability to decode long codes (Appendix A), which remain beyond the reach of ECCT due to its high memory requirements, resulting from large attention maps.
These results demonstrate the practical significance and architectural advantages of CrossMPT, proving its value in scenarios where ECCT encounters limitations.
Especially, it achieves superior decoding performance for LDPC codes, outperforming the BP decoder with the maximum iteration of 100 (provided in Appendix B).
Also, we demonstrate that for short codes, CrossMPT closely approaches the optimal maximum likelihood (ML) decoding performance (provided in Appendix C).
Additional experimental results for comparison with successive cancellation list polar decoder, denoising diffusion ECCT (DDECCT), and the decoding performance for the Rayleigh channel are provided in Appendices D, E, and F, respectively.

Figure: Figure 5: The average attention scores of all $N=6$ layers for ECCT and CrossMPT.
Refer to caption: x8.png

## 6 Ablation Studies and Analysis

### 6.1 Analysis of Attention Mechanisms in ECCT and CrossMPT

We provide a comparative analysis of the attention scores in ECCT and CrossMPT.
Figure 5 shows the average attention scores across $N=6$ layers for both ECCT and CrossMPT for $(32,16)$ LDPC code (Abu-Surra et al., ).
As shown in Figure 5, the attention score map of ECCT reveals different importance among the relationships: magnitude-magnitude, syndrome-syndrome, and magnitude-syndrome.
One key observation is that the magnitude-magnitude and syndrome-syndrome relations exhibit relatively low attention scores compared to the magnitude-syndrome relation in Figure 5.
This suggests that the magnitude-syndrome relationship is more significant than the others.
An ablation study, in which we masked the magnitude-magnitude and syndrome-syndrome relationships, revealed no significant performance difference compared to when these relationships were not masked (see Appendix G).
This ablation study shows that the conventional ECCT could be enhanced by focusing on the more critical relationships.
However, as shown in Figure 5, CrossMPT eliminates the two relations with low attention scores and focuses only on the magnitude-syndrome relation.
Therefore, we can claim that CrossMPT more efficiently targets the crucial aspect (i.e., magnitude-syndrome relation) compared to ECCT.

Figure: (a) Attention scores with a single bit error
Refer to caption: extracted/5918250/figs/fig_layer_2.png

### 6.2 Visualization of Cross-attention Map

To further examine how CrossMPT operates, we intentionally corrupt a pre-determined bit of the $(32,16)$ LDPC code and analyze the resulting attention maps.
Figure 6 shows the attention scores for the first two layers and the summation of their attention scores when the first bit is corrupted.
The summation is carried out vertically to demonstrate the attention score for each bit.
As shown in Figure 6(b), the attention score of the first bit (or first column) is relatively higher than the others.
However, once the error is corrected, CrossMPT no longer assigns high scores to that position (see Figure 11 in Appendix H).
Figures 6(c) and 6(d) depict the attention scores and the summation of attention scores without an error.
Compared to the previous case, the attention scores are more uniformly distributed across all positions.

### 6.3 Number of Parameters

Two cross-attention blocks of CrossMPT share the same parameters for all decoder layers.
They use the same weight matrices $W_{Q},W_{K},W_{V}$ for two cross-attention modules since the performance remains nearly identical even when the parameters are trained separately.
Also, they share the parameters for the normalization layer and the FFNN layer.
Thus, CrossMPT has the same number of parameters as the original ECCT.

### 6.4 Complexity Analysis

Figure 1 illustrates the mask matrices of ECCT and CrossMPT.
In the original ECCT, Figure 1(a) shows that a significant portion of the upper $n\times n$ submatrix is depicted in white, indicating that the most positions are unmasked.
This $n\times n$ submatrix represents depth-2 connections in the Tanner graph (Choukroun & Wolf, ), which results in an increase in the number of unmasked positions, thereby leading to a higher computational required.
On the other hand, the lower $(n-k)\times n$ submatrix and the right $(n-k)\times n$ submatrix, which serve as the masking matrices for CrossMPT, are predominantly shown in blue, indicating that their attention matrices are sparser.
Figure 7 compares the mask matrix density of CrossMPT and ECCT.
For all codes, the mask matrix of CrossMPT is sparser than ECCT, which implies that CrossMPT can achieve lower computational complexity compared to the original ECCT.

The complexity of the self-attention mechanism of ECCT, without considering the masking is, $\mathcal{O}(N(d^{2}(2n-k)+(2n-k)^{2}d))$.
When taking masking into account, the complexity can be reduced to $\mathcal{O}(N(d^{2}(2n-k)+hd))$ (Choukroun & Wolf, ), where $h=\rho_{1}(2n-k)^{2}$ denotes the fixed number of computations of the self-attention module and $\rho_{1}$ denotes the density of the mask matrix in ECCT.
Similarly, the complexity of the two cross-attention modules of CrossMPT, without considering the masking, is $\mathcal{O}(N(d^{2}(2n-k)+2n(n-k)d))$.
When masking is taken into account, the complexity can be reduced to $\mathcal{O}(N(d^{2}(2n-k)+(h_{1}+h_{2})d))$, where $h_{1}=\rho_{2}n(n-k)$ denotes the number of computations of the first cross-attention module, $h_{2}=\rho_{2}(n-k)n$ denotes the number of computations of the second cross-attention module, and $\rho_{2}$ denotes the density of the mask matrix in CrossMPT.
Furthermore, since $\rho_{1}>\rho_{2}$ as shown in Figure 7, we conclude that $h>h_{1}+h_{2}$, which indicates that CrossMPT achieves a reduction in computational complexity compared the original ECCT.

Table 2 compares the FLOPs, inference time, and training time between ECCT and CrossMPT.
The inference time refers to the duration required to decode a single codeword and the training time measures the duration to complete one epoch of training.
All results are obtained for $N=6$ and $d=128$, except for (255,223) BCH code and (384,320) WRAN LDPC code, which are obtained for $N=6$ and $d=32$.
For all three metrics, CrossMPT outperforms ECCT.
Since the inference time and the training time are closely related to the FLOPs, a reduction in FLOPs directly leads to shorter inference and training times.
Notably, for long codes, CrossMPT achieves a significant reduction compared to ECCT.
The results in Tables 1 and 2 demonstrate that the proposed CrossMPT not only improves the decoding performance but also significantly reduces FLOPs, inference time, and training time compared to the original ECCT.
Additional analysis on training convergence and throughput of CrossMPT is provided in Appendix I and J, respectively.

Figure: Figure 7: Comparison of the mask matrix density between ECCT and CrossMPT.
Refer to caption: x11.png

**Table 2: Comparison of FLOPs, inference time, and training time between ECCT and CrossMPT for various codes. Inference time is measured for decoding a single codeword and training time is measured for a single epoch.**
| Codes | Parameter | FLOPs | Inference (codeword) | Training (epoch) | Mask density |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CrossMPT | ECCT | CrossMPT | ECCT | CrossMPT | ECCT | CrossMPT | ECCT |  |  |
| BCH | (63,45) | 11.8 M | 14.0 M | 326 $\mu$s | 328 $\mu$s | 29 s | 29 s | 32.45% | 53.09% |
| LDPC | (121,70) | 28.8 M | 37.7 M | 400 $\mu$s | 450 $\mu$s | 58 s | 80 s | 9.09% | 24.01% |
| (121,80) | 26.3 M | 34.6 M | 391 $\mu$s | 436 $\mu$s | 53 s | 76 s | 9.09% | 21.94% |  |
| Turbo | (132,40) | 41.8 M | 55.0 M | 459 $\mu$s | 511 $\mu$s | 83 s | 110 s | 11.43% | 14.25% |
| BCH | (255,223) | 4.43 M | 12.9 M | 747 $\mu$s | 859 $\mu$s | 56 s | 145 s | 48.63% | 78.21% |
| WRAN | (384,320) | 10.0 M | 29.3 M | 1295 $\mu$s | 1638 $\mu$s | 104 s | 305 s | 5.21% | 13.25% |

## 7 Conclusion

We developed a novel transformer architecture for ECC decoding called CrossMPT, which improves both decoding performance and computational complexity.
CrossMPT achieves this by adopting a more effective architecture that processes magnitude and syndrome through the cross-attention mechanism.
This approach leverages the clear and compact representation of codeword bit relationships in the PCM, enabling the model to accurately learn these relationships while also reducing memory usage, FLOPs, inference time, and training time.
Most existing research on transformer-based decoders has primarily focused on short codes due to challenges in training long codes, caused by high memory usage and complexity.
However, CrossMPT offers a potential breakthrough, paving the way for transformer-based decoders to be effectively applied to long codes.

## Appendix A Performance on Longer Codes

We present the BER performance for three longer codes in Figures 8(a), 8(b), and 8(c) for ECCT and CrossMPT $N=6$, $d=32$.
For all three codes ((a) (529,440) LDPC code, (b) (384,320) wireless regional area network (WRAN) LDPC code, (c) (512,384) polar code), the proposed CrossMPT outperforms the original ECCT.
Despite its reduced complexity, CrossMPT significantly enhances the decoding performance compared to ECCT, not only for short-length codes but also for longer codes.
Also, Figures 8(d) and 8(e) show the decoding performance of CrossMPT for much longer codes.
The BER performances of the (648,540) IEEE802.11n LDPC code ($N=10$, $d=128$) and (1056,880) WiMAX LDPC code ($N=6$, $d=32$) demonstrate that CrossMPT efficiently trains how to decode the codeword even for large $N$ and $d$ and performs well for longer codes.
Again, we emphasize CrossMPT’s capability to decode long codes where ECCT struggles due to high memory allocation (large attention map).
The structure of CrossMPT demonstrates its efficiency in learning long codes, surpassing the limitations of short or moderate codelengths of transformer-based decoders.

Figure: (a) (529,440) LDPC code
Refer to caption: x12.png

## Appendix B Comparison with the BP decoder

Figure 9 shows the decoding performance between the traditional BP decoder with a maximum number of iterations of 20, 50, 100, and 200 and CrossMPT for both
short and long LDPC codes.
Figures 9(a) and 9(b) compare the BER performance for (121,80) LDPC codes ($N=6$, $d=128$) and (648,540) IEEE 802.11n LDPC code ($N=10$, $d=128$), respectively.
Notably, the proposed CrossMPT can outperform the BP decoder for both short and long LDPC codes.
These results highlight that CrossMPT efficiently trains how to decode the codeword across a wide range of code lengths.

Figure: (a) (121,80) LDPC code
Refer to caption: x17.png

## Appendix C Comparison with the ML decoder

We compare ECCT and CrossMPT with the ML decoder for short BCH codes.
Figure 9 demonstrates the BER performance of $(31,16)$ BCH code and $(31,21)$ BCH code.
Especially, these results show that CrossMPT closely approaches the optimal ML performance for short codes.

Figure: (a) (31,16) BCH code
Refer to caption: x19.png

## Appendix D Comparison with Successive Cancellation List Polar Decoder

We compare the BER performance of the SCL decoder, ECCT, and CrossMPT in Table 3.
The performance of the SCL decoder is from (Choukroun & Wolf, ).
Although the contribution of $L$ is significant in long codes, the SCL decoder achieves a great performance with small $L$, such as $L=4$.
As reported in (Choukroun & Wolf, ; ), the SCL decoder outperforms ECCT.
This is because the SCL decoder is a decoder specialized for Polar codes and is a state-of-the-art algorithm that has undergone extensive development over a long period.
CrossMPT has made significant improvements from ECCT and even outperforms the SCL decoder for (64,48) polar code.

**Table 3: Comparison of decoding performance at three different SNR values (4 dB, 5 dB, 6 dB) for SCL decoder, ECCT, and CrossMPT. The results are measured by the negative natural logarithm of BER. The best results are highlighted in bold and the second best is underlined. Higher is better.**
| Method | SCL ($L=1$) | SCL ($L=4$) | ECCT | CrossMPT |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 |
| (64,32) | 7.30 | 9.67 | 13.18 | 8.11 | 10.70 | 14.04 | 6.99 | 9.44 | 12.32 | 7.50 | 9.97 | 13.31 |
| (64,48) | 6.19 | 8.41 | 10.97 | 6.69 | 8.63 | 11.24 | 6.36 | 8.46 | 11.09 | 6.51 | 8.70 | 11.31 |
| (128,64) | 8.37 | 11.69 | 13.70 | 9.60 | 13.16 | 17.42 | 5.92 | 8.64 | 12.18 | 7.52 | 11.21 | 14.76 |
| (128,86) | 7.54 | 10.74 | 15.14 | 9.26 | 13.04 | 17.13 | 6.31 | 9.01 | 12.45 | 7.86 | 11.45 | 15.47 |
| (128,96) | 6.74 | 9.53 | 13.53 | 8.02 | 11.60 | 18.16 | 6.31 | 9.12 | 12.47 | 7.15 | 10.15 | 13.13 |

## Appendix E Comparison with DDECCT

For a fair comparison with DDECCT, we also apply the denoising diffusion training technique to CrossMPT.
Table 4 compares the BER performance of ECCT (Choukroun & Wolf, ), CrossMPT, DDECCT, and CrossMPT applying the denoising diffusion model.
All four decoders are model-free decoders using the transformer architecture, and simulations are taken for $N=6$, $d=128$.
We conduct simulations for codes where DDECC performs better than CrossMPT.
For the rest of the codes, CrossMPT outperforms DDECC.
The proposed CrossMPT shows superior decoding performance compared to the original ECCT.
Compared to DDECC, CrossMPT demonstrates similar BER performance for polar codes, but it even outperforms DDECC for BCH and LDPC codes.
When the denoising diffusion technique is applied to CrossMPT, it achieves the best performance among others, where DDECC, CrossMPT, and ECCT follow.
This proves that the CrossMPT architecture provides separate gain from the denoising diffusion algorithm for transformer-based decoders.

**Table 4: Comparison of decoding performance at three different SNR values (4 dB, 5 dB, 6 dB) for ECCT (Choukroun & Wolf, ), CrossMPT, and DDECC (Choukroun & Wolf, ). The results are measured by the negative natural logarithm of BER. The best results are highlighted in bold and the second best is underlined. Higher is better.**
| Architecture | Without denoising diffusion | With denoising diffusion |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Codes | Parameter | ECCT | CrossMPT | ECCT | CrossMPT |  |  |  |  |  |  |  |  |
| 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 |  |  |
| BCH | (63,36) | 4.86 | 6.65 | 9.10 | 5.03 | 6.91 | 9.37 | 5.11 | 7.09 | 9.82 | 5.23 | 7.20 | 10.01 |
| Polar | (128,64) | 5.92 | 8.64 | 12.18 | 7.52 | 11.21 | 14.76 | 9.11 | 12.9 | 16.30 | 10.21 | 13.63 | 17.28 |
| (128,86) | 6.31 | 9.01 | 12.45 | 7.51 | 10.83 | 15.24 | 7.60 | 10.81 | 15.17 | 8.56 | 12.04 | 15.37 |  |
| (128,96) | 6.31 | 9.12 | 12.47 | 7.15 | 10.15 | 13.13 | 7.16 | 10.3 | 13.19 | 7.57 | 10.61 | 13.33 |  |
| MacKay | (96,48) | 7.38 | 10.72 | 14.83 | 7.97 | 11.77 | 15.52 | 8.12 | 11.88 | 15.93 | 8.85 | 12.58 | 17.69 |

## Appendix F Decoding Performance for Rayleigh Fading Channel

The original ECCT architecture shows robustness to non-Gaussian channels (e.g., Rayleigh fading channel) (Choukroun & Wolf, , Supplementary).
We also measured the decoding performance of CrossMPT in Rayleigh fading channels. To compare with ECCT, we use the same fading channel as in (Choukroun & Wolf, ). The received codeword is given as $y=hx+z$, where $h$ is an $n$-dimensional i.i.d. Rayleigh distributed vector with a scale parameter $\alpha=1$ and $z\sim N(0,\sigma^{2})$. The following table demonstrates the BER performance of ECCT and CrossMPT in Rayleigh fading channel and CrossMPT still outperforms the original ECCT architecture for all types of codes.

| Codes | (31,16) BCH | (64,32) Polar | (128,64) Polar | (128,86) Polar | (121,70) LDPC | (128,64) CCSDS |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 |
| ECCT | 5.18 | 6.04 | 6.92 | 5.53 | 6.62 | 7.80 | 4.31 | 5.37 | 6.63 | 4.02 | 4.81 | 5.70 | 3.91 | 4.97 | 6.31 | 2.46 | 3.97 | 5.79 |
| CrossMPT | 5.53 | 6.55 | 7.61 | 5.91 | 7.17 | 8.48 | 4.70 | 5.93 | 7.34 | 4.41 | 5.38 | 6.46 | 4.25 | 5.53 | 7.11 | 5.25 | 6.94 | 8.92 |

## Appendix G Ablation Study with Additional Masking

To understand the impact of magnitude-magnitude and syndrome-syndrome relationships, we demonstrate the decoding performance of ECCT with additional masking of these relationships.
Table 5 compares the decoding performance of ECCT with this additional masking, standard ECCT, and CrossMPT.
The results show no significant performance degradation with the additional masking, indicating that the magnitude-magnitude and syndrome-syndrome relationships are not critical to decoding performance.

**Table 5: Comparison of decoding performance at three different SNR values (4 dB, 5 dB, 6 dB) for ECCT with this additional masking, standard ECCT, and CrossMPT. The results are measured by the negative natural logarithm of BER. The best results are highlighted in bold. Higher is better.**
| Method | ECCT + Masking | ECCT | CrossMPT |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Parameter | 4 | 5 | 6 | 4 | 5 | 6 | 4 | 5 | 6 |
| $(31,16)$ BCH | 6.52 | 8.55 | 11.42 | 6.39 | 8.29 | 10.66 | 6.98 | 9.25 | 12.48 |
| $(63,45)$ BCH | 5.53 | 7.74 | 10.88 | 5.60 | 7.79 | 10.93 | 5.90 | 8.20 | 11.62 |
| $(64,48)$ Polar | 6.25 | 8.26 | 10.93 | 6.36 | 8.46 | 11.09 | 6.51 | 8.70 | 11.31 |
| $(121,60)$ LDPC | 4.98 | 7.91 | 12.61 | 5.17 | 8.31 | 13.30 | 5.74 | 9.26 | 14.78 |

## Appendix H Visualization of Cross-attention Map

Figure 11 illustrates the attention scores for all $N=6$ layers with a single bit error (bit error in the first position).
The first three layers have relatively high attention score at the error position (first bit).
Then, when the error is corrected, the attention score becomes lower at the last three layers.

Figure: Figure 11: Attention scores of $N=6$ layers with a single bit error.
Refer to caption: extracted/5918250/figs/fig_layer_full.png

## Appendix I Impact of the Proposed Mask Matrix and Training Convergence

The mask matrix in transformer-based decoders enables the model to efficiently learn the relevance between input bits.
In the masked cross-attention module in CrossMPT, the mask matrix is the PCM itself, utilizing only the essential information of codeword bits.
To demonstrate the impact of the CrossMPT architecture, we observe the behavior of the loss at each epoch.
Fig. 12 compares the training convergence between ECCT and CrossMPT.
The training of CrossMPT is much faster than ECCT, demonstrating CrossMPT’s efficiency in achieving superior decoding performance with limited training epochs.

Figure: (a) $(128,86)$ polar code
Refer to caption: x21.png

## Appendix J Throughput Analysis

If the decoder is implemented with specific hardware, pipelining approach can be used to maximize the throughput.
Employing pipelining, CrossMPT achieves a decoding throughput comparable to that of ECCT.
By unrolling $N$ layers and adapting the decoder for a fully parallel hardware architecture, CrossMPT can process two consecutive codewords simultaneously across two cross-attention blocks within the same layer.
This means that while the second cross-attention block is processing the first codeword, the first cross-attention block can concurrently decode the subsequent codeword. This pipelining strategy ensures that throughput levels remain comparable to that of ECCT.
Figure 13 illustrates the example of decoding multiple codewords in CrossMPT with $N=2$:

In wireless communications, the decoder’s throughput is often a more critical concern than latency.
This is because the latency from communication protocols and signal processing in preceding receiver blocks would be longer than the latency introduced by the channel decoder.
Throughput becomes especially important when supporting very high data rates in wireless communication as the channel decoder can be a bottleneck.

Furthermore, in wireless communication scenarios, a sequential algorithm may be preferred for its enhanced performance or reduced complexity.
The layered decoding algorithm for LDPC codes has been widely adopted as a de facto standard (Bae et al., ; Hailes et al., ; Li et al., ), despite its sequential nature and limitation on parallelism, exemplifying the preference for sequential algorithms.
The layered decoding algorithm is favored for its superior decoding performance compared to fully parallel sum-product decoding at equivalent computational complexities.

Figure: Figure 13: Example of decoding multiple codewords in CrossMPT with $N=2$.
Refer to caption: x23.png

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. FER Results: In [1], the authors present their results using the Frame Error Rate (FER) metric, while your results are reported using Bit Error Rate (BER). Could you please explain why you chose to report BER over FER in your evaluation? Are there specific challenges or reasons that prevent you from providing FER results for your approach? Including FER results would be helpful for assessing the practical decoding performance and for making a direct comparison with the results in [1].

2. Training Convergence and Speedup: In Appendix I, authors mention that CrossMPT achieves faster training convergence compared to ECCT. However, the figure provided does not clearly demonstrate a significant speed up. Could you please provide specific data quantifying how much faster CrossMPT converges during training? For instance, you could report metrics such as: wall-clock training time taken by each model to achieve comparable performance levels and percentage improvement in convergence speed or reduction in epochs/time needed.

3. Reproducibility: Do you plan to release the source code or detailed experimental records? Reproducibility is essential for validating the results and facilitating future research based on your work.

[1] A Foundation Model for Error Correction Codes
soundness: 4
strengths: 1. Innovative architecture: introduced the cross-attention mechanisms to separately handle magnitude and syndrome vectors is a notable advancement, bridging concepts from traditional message-passing algorithms and modern transformer architectures.
2. Performance Gains Significantly: in the experiments show significant improvements to existing decoders across various code classes, including BCH, polar, LDPC, and turbo codes.
3. Efficiency Improvements: By reducing memory usage and computational complexity, CrossMPT addresses practical concerns associated with deploying transformer-based decoders, especially for longer codes. 
4. Use of PCM-derived Masks: Aligning the mask matrices with the PCM directly incorporates code-specific structural information into the model, enhancing its ability to learn relevant relationships.
summary: This paper introduces a novel transformer-based architecture called CrossMPT(Cross-attention Message-Passing Transformer) for decoding error-correcting codes(ECCs). The key innovation lies in the use of cross-attention mechanisms to separately and iteratively update magnitude and syndrome vectors, in contrast to conventional transformer-based decodes that employ self-attention without distinguishing between different types of input vectors. The mask matrices in CrossMPT are derived from the code’s parity-check matrix (PCM), explicitly capturing the relationships between magnitude and syndrome vectors. Results demonstrate that CrossMPT significantly outperforms existing neural network-based decoders across various code classes, while also reducing memory usage, computational complexity, inference time, and training time.
weaknesses: 1. Availability of Code or Experimental Details: The paper does not mention whether the code or experimental records will be made available. Open-sourcing the code or providing detailed experimental records are crucial for reproducibility and for the community to gain more benefit from this work.

2. Code length: In the appendix H of "A Foundation Model for Error Correction Codes," the authors compare Maximum Likelihood decoders on the BCH(1023,1013) code. In this paper, authors highlight the significant reductions in memory usage. It would be valuable for the authors to extend this analysis by demonstrating performance on even longer codes or comparing their results against other decoding methods for these extended lengths. This could provide clearer insights into the advantages of their approach.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - The performance measure presented in the paper is BER; however, what really matters for a channel code is the block error probability (BLER). Therefore, I would ask the authors to include BLER results to clearly show that the proposed approach does actually improve the performance with respect to ECCT or other baseline decoding algorithms. Improvements in BER are still important, but correct decoding of the message bits would still require the concatenation of the code with an outer code, which would further increase the decoding complexity, and would diminish the gains of the proposed approach. I believe, not presenting the BLER results at all is not fair, and does not show the real performance of the approach. 

- How does the complexity compare with the more conventional BP decoders and neural BP (NBP) decoders? Are the numerical results presented in Fig. 4 for the same complexity decoders, or for the same number of decoder iterations? If it's the latter, the comparison is not fair. If each iteration of the NBP decoder has half the complexity of the proposed approach, it can run twice the number of iterations, and potentially achieve lower BER. 

- Why is the double-masked ECCT not included in the comparisons?
soundness: 3
strengths: The decoding of error correction codes is an important problem with significant potential impact in practice. Using neural network based decoders is a promising approach that has been receiving significant attention recently. The paper improves the state of the art transformer-based architecture with a simple yet effective enhancement in the architecture. Given that the magnitude and the syndrome provide different types of information on the transmitted symbols, it makes sense to process them separately, and learn different cross-attention modules. Despite the fact that the architectural modification is simple, the improvement in BER performance is significant. The authors also provide a good explanation of their motivation and the impact of their architecture on the decoding process (e.g., attention scores). Overall, it is a well-written paper with clearly highlighted contributions.
summary: This paper studies the neural network based decoding of conventional error correction codes. In essence, decoding of an error correction code is a classification problem. Conventional belief propagation based decoders rely on the code structure to iteratively improve their beliefs on the message bits. An alternative approach is to parameterize the decoder as a neural network, and train it on various codewords. Given the large number of codewords, it is challenging to design such a decoder without exploiting the code structure. An obvious approach is to use the code structure and the basic steps of the belief propagation decoder, but unroll the iterations on the layers of a neural network. This work follows the recently introduced ECCT decoder in this direction, and exploits a transformer architecture for the decoder. In particular, by transforming the decoding operation into the estimation of the sign of the multiplicative noise component, and by separately representing the magnitude of the received signal and its syndrome as two separate pieces of information, the authors train separate masked cross-attention blocks to update the embeddings of these two blocks before reaching the final decision. Through experiments, they show that the resultant decoder can achieve a lower bit error rate (BER) compared to ECCT in decoding a number of different channel codes, with a slightly lower computational complexity.
weaknesses: The architectural modification proposed in the paper with respect to ECCT is rather trivial. In that sense, it does not have significant technical novelty, on the other hand, given its impact on the performance, I believe it is a worthy contribution. 

The paper presents only BER performance results. It is not clear if the proposed approach actually provides any gains in terms of the block error probability, which would really matter for the code.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why was the magnitude chosen instead of another quantity, such as LLR? What is the intuition behind using the magnitude?
2. What happens if the syndrome information is placed as the query in the first attention layer? Does changing the order make any difference?
3. Is the inference time in Table 2 evaluated on the GPU?
4. What is the peak memory requirement? 
5. Was the inference runtime evaluation conducted after code compilation (e.g., using `torch.compile()`)?
soundness: 3
strengths: 1. The update in the attention layer built upon ECCT is well-motivated and shows meaningful performance improvement. Although cross-attention is a well-known technique in the community, it is placed at the right position to improve the performance and efficiency of the previous self-attention-based approach.
2. The paper is easy to follow and provides sufficient details.
3. The experimental results effectively support the claims.
4. The in-depth analysis of the proposed method helps readers understand the paper better.
summary: The paper proposes cross-attention-based message-passing transformers (CrossMPT), which solve the ambiguous masking problem of the naïve self-attention-based approach (ECCT). The magnitude and syndrome information are processed using two cross-attention layers, similar to message-passing algorithms, incorporating a mask defined by the parity check matrix. The proposed scheme achieves superior performance on various codes and SNRs consistently over ECCT and other baselines. The cross-attention map provides some degree of interpretability, while cross-attention itself enjoys reduced complexity compared to full self-attention, enabling the processing of long codes.
weaknesses: Although I think the paper is strong, I found a couple of minor weaknesses.

1. The actual inference time reduction from ECCT is not as great as the FLOPs improvement. Also, runtime evaluation and complexity analysis need more details about the environment. See Questions 3-5.
2. Related works on cross attention in other domains are missing, e.g., language (Vaswani et al., 2017), vision (Chen et al., 2021), text-based image generation (Rombach et al., 2022), etc.

(Vaswani et al., 2017) Vaswani, A. "Attention is all you need." *Advances in Neural Information Processing Systems* (2017).

(Chen et al., 2021) Chen, Chun-Fu Richard, Quanfu Fan, and Rameswar Panda. "Crossvit: Cross-attention multi-scale vision transformer for image classification." *Proceedings of the IEEE/CVF international conference on computer vision*. 2021.

(Rombach et al., 2022) Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2022.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Included in the weaknesses
soundness: 4
strengths: 1. The idea of treating syndrome and magnitude as two separate entities and using cross-attention to capture the relation is novel and interesting. From a error correction point of view, this is more intuitive and understandable than existing approches. 

2. The performance improvement over ECCT is compelling and non-trivial, while maintaining similar or lower complexity.

3. Overall well written and self-contained. Easy to understand and follow the ideas, even for reader unfamiliar with prior work.
summary: The paper proposes a new cross-attention based decoding schemes for linear block codes using transformer architecture. The masking and the cross attention mechanism depend on the parity check matrix of the block code, thus introducing the domain knowledge about the code into the decoding architecture. This work seems to be built on Error Correction Code Trabsformer (ECCT), NeurIPS 2022.
weaknesses: 1. One common troubling trend in comparing the performance of neural decoders is the choice of weak classical baselines. While I understand the difficulty of a common decoder architecture outperforming the highly specialised decoders for each of the class of codes, the comparison should neverthless be done for completeness. For instance, in Fig.4, only BP classical baseline is considered. It is known that BP is not optimal for many codes. For instance, when comparing with BCH codes, Berlekamp-Massey decoding algorithm should be used as the classical non-learning baseline and similarly for Polar codes, successive cancellation list decoding should be used. I strongly suggest authors to include the best classical baselines for each code to clearly show the gap with respect to the SOTA neural decoder, whether it is positive or negative.

2. I beleive authors chose the format of the negative log BER for presenting the results based on the original ECCT paper. But this format is not very informative and very uncommon in information and coding theory literature. While it is easy to identify which scheme is doing better, the gap between the performances is hard to interpret as the scale is not very intuitive. Rather, the standard format of SNR gap in dB for a target BER/BLER (ex. BLER target of $10^{-5}$) is more informative. 

3. While the idea and results are interesting, the changes in the algorithm/architecture are only incremental compared to the original ECCT paper, which limits the novelty to some extent.

4. As with the case of other model-free neural decoders, there is a lack of interpretation for the neural decoder presented. Since the algorithmic contributions are limited, I would suggest atleast adding a discussion on interpretation and analysis, similar to the analysis and interpretation of neural belief propagation codes. A couple of references I can think of are: [1], [2]

[1] "Neural- network-optimized degree-specific weights for ldpc minsum decoding", Wang et. al. ISTC 2021. 

[2] "Interpreting Neural Min-Sum Decoders", Ankireddy et. al, ICC 2023.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1 - The authors should integrate clearer information regarding the complexity as explained in the Weakness section.

2 - There is a large gap remaining with SCL decoding. The authors should explain and investigate why the (cross) ECCT and its variants struggle with it.

3 - The authors should provide comparisons between the methods for various dimensions/layers, as done with the ECCT and its extensions.
soundness: 3
strengths: The paper is well-written and clear.

The paper is well-presented and has extensive ablation studies.
 
The method is interesting and sounds natural given classical message-passing decoders.

The method improves over the original ECCT and allows more memory efficient training on GPUs.
summary: The paper presents an extension of the existing ECCT by splitting, as with classical message-passing decoding (BP), the self-attention module between variable and check nodes. This is done similarly to classical Transformers decoders via the additional cross-attention while the original ECCT has only self-attention layers.
weaknesses: I'm concerned about the complexity analysis and thus the claim of superiority of the method over ECCT.

Cross-MPT adopts a **sequential** processing approach that appears **after** the self-attention, which is the only module in ECCT.
This sequential processing (1 self-attention + 1 cross-attention) is then necessarily slower than ECCT's single parallel processing (1 self-attention only) on modern parallel computing HW.

It obviously allows better decoding performance, making the comparison unfair, even if the parameters are shared.

Meta-review: The paper builds on the existing ECCT approach for error correction via transformers, and it proposes a novel method dubbed cross-attention-based message-passing transformers (CrossMPT). The idea is to process magnitude and syndrome information using two cross-attention layers, incorporating a mask defined by the parity check matrix. The superiority over ECCT is demonstrated in a series of rather thorough experiments on various codes and SNRs.

The reviewers praised the innovative architecture, which leads to performance and efficiency improvements. Most of the issues raised in the initial reviews have been successfulIy addressed in the rebuttal and discussion. One problem that remains is that the proposed approach is still not competitive w.r.t. state-of-the-art non-neural alternatives. However, this problem is common to virtually all transformer-based decoders and the paper does take a step in the right direction providing novel ideas and improvements. Therefore, I recommend acceptance of the paper.

Decision: Accept (Poster)