CrossMPT: Cross-attention Message-passing Transformer for Error Correcting Codes
Authors: Seong-Joon Park
Hee-Youl Kwak
Sang-Hyo Kim
Yongjune Kim
Jong-Seon No
Venue: ICLR 2025 Poster (2025)
Keywords: Cross-attention
Error correcting codes
Message-passing decoder
Neural decoder
Transformer
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a novel SOTA message-passing decoder for error correcting codes based on the cross-attention of transformer architecture

Abstract: Error correcting codes (ECCs) are indispensable for reliable transmission in communication systems. Recent advancements in deep learning have catalyzed the exploration of ECC decoders based on neural networks. Among these, transformer-based neural decoders have achieved state-of-the-art decoding performance. In this paper, we propose a novel Cross-Attention Message-Passing Transformer (CrossMPT), which shares key operational principles with conventional message-passing decoders. While conventional transformer-based decoders employ a self-attention mechanism without distinguishing between magnitude and syndrome embeddings, CrossMPT updates these two types of embeddings separately and iteratively via two masked cross-attention blocks. The mask matrices are determined by the code's parity-check matrix, which explicitly captures and removes irrelevant relationships between the magnitude and syndrome embeddings. Our experimental results show that CrossMPT significantly outperforms existing neural network-based decoders for various code classes. Notably, CrossMPT achieves this decoding performance improvement while significantly reducing memory usage, computational complexity, inference time, and training time.

Published as a conference paper at ICLR 2025
CROSSMPT: CROSS-ATTENTION MESSAGE-PASSING TRANSFORMER FOR ERROR CORRECTING CODES
Seong-Joon Park Hee-Youl Kwak Sang-Hyo Kim Yongjune Kim Jong-Seon No 'POSTECH, 'University of Ulsan, 'Sungkyunkwan University, "Seoul National University joonpark22478gm com, yongjune@postech. ac.kr, jano snu ac.kr
com, amssh ime s ku....... edu,
ABSTRACT
Error correcting codes (ECCs) are indispensable for reliable transmission in com- munication systems. Recent advancements in deep learning have catalyzed the ex- ploration of ECC decoders based neural networks. Among these, transformer- based neural decoders have achieved state-of-the-art decoding performance. this paper, propose novel Cross-Attention Peessage-Passingg Transformer (CrossMPT), which shares key operational principles with conventional message- passing decoders. While conventional transformer-based decoders employ self- attention mechanism without distinguishing between magnitude and syndrome embeddings, CrossMPT updates these two types of embeddings separately and iteratively via two masked cross-attention blocks. The mask matrices are deter- mined by the code's parity-check matrix, which explicitly captures and removes irrelevant relationships between the magnitude and syndrome embeddings. Our experimental results show that CrossMPT significantly outperforms existing neu- ral network-bard -raser decoders for various code classes. Notably, CrossMPT achieves this decoding performance improvement while significantly reducing memory us- age, computational complexity, inference time, and training time.
INTRODUCTION
The fundamental objective of digital communication systems is to reliably transmit information from source destination through noisy channels. Error correcting codes (FCCs) crucial for ensuring the integrity of transmitted data in digital communication systems. The advancements in deep learning across diverse tasks, such natural language processing (NLP), image classification,
and object detection (Devlin et al. 2019 He et al. 2016 Girshick et al. 2014: Carion et al. 2020), have motivated the application of deep learning techniques to ECC decoders. This has led to the development of neural decoders (Kim et al.] 2018: 2020 Nachmani et al.) [016)] [018 Dai et al.) 2021 Lugosch Gross) 2017), which aims to improve decoding performance by overcoming limi- tations of the conventional decoders such as belief propagation (BP) (Richardson & Urbanke 2001)
and min-sum (MS) (Fossorier et al., 1999) decoders.
Among neural decoders, model-free neural decoders employ arbitrary neural network architec- tures (c.g., deep neural networks (Gruber et al. (017). recurrent neural networks (Bennatan et al. 2018) and transformers (Choukroun & Wolf 2022al 2023 Park al. 2025 Choukroun & Wolf, 2024a b)) as the ECC decoder, without relying on prior knowledge specific decoding algorithms. Since model-free neural decoders not based on specific decoding algorithms,, their training is susceptible to overfitting, largely due to the exponentially large number of possible codewords (Ben- natan al. 2018). To circumvent overfitting, these neural decoders incorporate preprocessing step where the magnitude and syndrome vectors from the received codewords are concatenated and used as inputs. The preprocessing step crucial for integrating an effective network architecture for the ECC decoder without an overfitting issue (Bennatan et al. 2018). For instance, transformer-baat ECC decoders incorporating this preprocessing (Choukroun Wolf 2022a| 2023 Park al. 2025] Choukroun & Wolf, 2024a b) achieve sfte--of-hhean decoding performance for short block codes.
"The source code is available at Ittttt::///// http://gtthhbbbccco cttp://iaoppttccccrooosso
Correspnnning authors

---

Published as a conference paper at ICLR 2025
However, two key questions remain unaddressed: 1) how to effectively process the two distinct input vectors (magnitude and syndrome), and 2) how to design a more efficient transformer- based decoder architecture.
Conventional transformer- basedd ECC decoders, initially proposed as Error Correction Code Trans-
former (ECCT) (Choukroun& & Wolf 2022a 2023 Park al. 2025 Choukroun & Wolf 2024a
process the concatenated magnitude and syndrome vectors as a single input, applying self-attention blocks without distinguishing between them. In contrast, our approach treats the magnitude and syn- drome as multimodal data, recognizing their distinct informational characteristics. The real-valued magnitude vector represents bit reliabilities, while the binary syndrome vector conveys the informa- tion of erroneous bit positions. This deliberate separation necessitates the development of novel architecture, specifically designed to effectively update these separated magnitude and syndrome vectors, thereby significantly improving decoding performance.
In this paper, we introduce a novel Cross-Attention Pesssage-Passing Transformer (CrossMPT) for ECC decoding. CrossMPT processes the magnitude and syndrome embeddings separately to lever- age their distinct informational properties. employs two cruss-attention blocks to iteratively update the magnitude and syndrome embeddings. Initially, the magnitude embedding is encoded into the query, while the syndrome embedding is encoded into key and value. The first cross-attention block utilizes this configuration in its attention mechanism to update the magnitude embedding. This procedure is reciprocated for the syndrome embedding, which is encoded into the query. while the updated magnitude embedding is encoded into the key and value. This configuration enables the second cross-attention block to update the syndrome embedding. These two masked cross-attention blocks iteratively collaborate to refine the magnitude and syndrome embeddings as in the message- passing algorithm (Richardson & Urbanke, (000)).
To facilitate training, CrossMPT employs a mask matrix for each cross-attention block. The first cross-attention block uses the transpose of the parity check matrix (PCM) H" as its mask matrix with the magnitude embedding as the query. In the second cross-attention block, the PCM H itself is applied as the mask matrix, with the syndrome embedding as the query. This strategy leverages the PCM's inherent representation of the maggitude-syndromm tecture's 's objectives. Moreover, the combined size of the two attention maps of CrossMPT is at most half that of conventional transformer-decoders significantly reducing memory usage. This reduc- tion allows CrossMPT to train and decode longer codes, which previous approaches (concatenating magnitude and syndrome embeddings) are unable to achieve due to high memory usage and com- putational complexity. To our knowledge, CrossMPT is the first architecture to integrate iterative message-passing framework with a cross-attention-basee transformer decoder.
relationship, aligning with the archi-
Experimental results show that CrossMPT consistently outperforms the original ECCT across vari- ous code classes. Leveraging its shared operational principles with the pessagg-passing algorithm, CrossMPT demonstrates particularly improved decoding performance, especially for low-density parity-check (LDPC) codes. Notably, we also demonstrate that CrossMPT closely approaches the maximum likelihood decoding performance on short codes. Beyond its enhanced decoding performance, CrossMPT sigmificantly reduces computational costs, including floating point opera- tions (FI.OPs), training time, and inference time, compared to the original ECCT. Since the decoder layer constitutes substantial portion of the total computational cost, this reduction leads to a sig- nificant decrease in overall computational complexity.
RELATED WORKS
In the field of neural network-based ECC decoders. there are two primary categories: the model- based decoder and the model-free decoder. First, model-based decoders are constructed based on the conventional decoding methods (e.g., BP decoder and MS decoder). They map the iterative decoding process of the conventional decoding methods into neural networks and train the network weights accordingly. To improve performance over the standard BP decoder, the recurrent neural network was employed for the decoding of BCH codes (Nachmani et al. 2018). Several recent studies showed that neural network- based BP and MS decoders outperform the conventional decod- ing algorithms over various code types (Dai et al. 2021] Kwak et al.) 2022 2023) 2025 Lugosch
& Gross 2017 Nachmani & Wolf 2019 2021
Buchberger al. 2021). However, model-based
2

---

Published as a conference paper at ICLR 2025
Mag syn
Syn mag (9)
Parlly chock matrix
Mask matrix of ECCT
Parity chenk matrix
Mask matrix of CresMPPT
Masked
Masked
Unmaskod
(a) Conventional ECCT (Choukroun & Woli 2022a
(b) CrossMPT
Figure The PCMs and the mask matrices of ECCT and CrossMPT.
neural decoders may encounter performance limitations due to their restrictive model architectures, which are closely tied to underlying decoding methods.
Unlike model-based decoders, which are constrained by the limitations of their underlying algo- rithms (e.g., BP), model-free neural decoders use arbitrary architectures to learn decoding without such restrictions. While early approaches (Gruber et al. 2017] Cammerer et al. 2017;; Kim al.] 2018) employed fully connected networks, they faced overfitting challenges during training. Sub- sequently, the introduction of a preprocessing step utilizing the magnitude and syndrome vectors of the received codeword to learn multiplicative noise has been pivotal in enabling model-free decoders to address the overfitting issue (Bennatan et al. (018). Then, ECCT (Choukroun & Wolf 2022a) first employed the transformer architecture using the the transformer-basee decoder outperforms existing neural decoders including model-based neural decoders. Building the ECCT framework, denoising diffusion error correction codes (Choukroun & Wolf 2023) interpreted the iterative decoding process as diffusion process and incorporated diffusion model to train the original ECCT. Recently, multiple-masks ECCT (Park al. 20255 uti- lized different PCMs for the same linear code to capture the diverse multilateral relationships of the magnitude and syndrome bits and improve the decoding performance. Notably, transformer- based decoders outperform model-basee neural decoders and serve as universal decoders capable of decoding arbitrary code classes with a unified architecture.
preprocessing step and demonstrated that
Furthermore, employing cross-attention mechanisms in architecture has enhanced performance across various domains. In NLP, the transformer decoder (Vaswani al. (017) adopted the cross- attention modules. In vision, CiossViT (Chen et al.) [021] utilized cross- cotss-attentinn for improved im- age classification. For text-based image generation, latent diffusion models (Rombach et al. 2022) integrated cross-attention layers into the model architecture, enabling diffusion models to become powerful and flexible generators. These works demonstrate the versatility cross attention, inspir- ing its application ECC decoding.
3 BACKGROUND
3.1 ERROR CORRECTING CODES
Let C' be linear block code, which defined by a generator matrix G of size & x n and parity check matrix H of size (n - k) x n. They satisfy GH = over {0, 1} with modulo 2 addition. A codeword x C' C. {0, 133 is encoded by multiplying message m with the generator matrix G (i.e., = =mG). Let I, be the binary phase shift keying (BPSK) modulated signal of æ and let be the output of a noisy channel for input T.. We assume the additive white Gaussian noise (AWGN) channel and the channel output can be represented by y F, + where z ~ (O0 g²). The objective the decoder is to recover the transmitted codeword x by correcting errors. When y is received, the decoder first determines whether the received signal is corrupted or not by checking the syndrome s(y) = Hyg, where y = bin (iign())) ) is the demodulated signal of y. Here, sign(a) represents +1 if 2 0 and -1 otherwise and bin(-1) 1, bin(+1) 0. If s(y) is non-zero vector, it is detected that y is corrupted during the transmission, and the decoder initiates the error correction process.
3

---

Published as a conference paper at ICLR 2025
Decoder layer
FFNN
Norm
Wy
Missked ETVR aeteeaatttiinnn
axd
x
PRNN
Mig
Embooding
Nonn
FC
C20 -xx
x
bolood
Wys
CIUSO cruseallention
W
Norm
W
Initial embedding layer
Output layer
Figure 2: Architecture of CrossMPT.
3.2 ERROR CORRECTION CODE TRANSFORMER
ECCT the first approach present a model-free decoder with the transformer architecture. ECCT outperforms other neural decoders by employing the masked self-attention mechanism, whose mask matrix determined by the code's PCM (Choukroun Wolf 2022a). A primary challenge in train- ing transformer- based decoders is the issue of overfitting. In (Bennatan et al.) 2018), the overfitting issue in model-free neural decoders described as poor generalization to untrained codewords due to the exponentially large number of possible codewords. However, it has been resolved by pre- processing technique that facilitates a syndrome- based decoding (Bennatan et al. 2018). has been theoretically proven that, with this preprocessing step, the decoder performance remains invariant to the specific codewords used in the training set (Bennatan et al. 2018).
As in (Bennatan al. 2018), the preprocessing step of ECCT utilizes the magnitude and syndrome vectors to train multiplicative noise Es, which is defined by
= Fg Z
ECCT aims to estimate the multiplicative noise in ( i.c. f(y) = 2g. Then, the estima- tion of x is bin bin(signyyf())), If the multiplicative noise is correctly estimated such that sign(2,) (2.) sign(E.), (2x), then å can be computed as:
a = bingnig (y)) = bin(sign(r,ã,,_) = bin(sig((r.) =
ECCT employs a masked self-attention module to train the transformer architecture, where the input embedding constructed by concatenating the magnitude and syndrome embeddings. As shown in Figure the mask matrices of ECCT should clearly distinguish between necessary (unmasked) and unnecessary (masked) pairwise relationships among and
prgnitude-syndrome
bit relations. ECCT, the syndrome-syndrome part was only unmasked for self-relations, while the magnitude-syndrome part unmasked based the connections de- fined by the PCM. The gegnitudd-maannnunee part, however, was unmasked for bit pairs connected depth 2 (see Algorithm 1 in Choukroun Wolf (2022a)). While the masking of magnitude- syndrome relations is intuitive, as it directly uses PCM, determining the relationships among mag- nitude themselves is not directly derivable from the PCM. Therefore, the algorithm for masking magnitude-mannitude
magnitude- magnitude part is neither straightforward nor unique. In Figure [ the white areas indi- cate unmasked positions that require attention calculations, whereas the blue areas represent masked positions where attention calculations can be omitted. As the proportion of blue increases, the at- tention matrix becomes sparser, enhancing computational efficiency.
CROSS-ATTENTION MESSAGE- TRANSFORMER
In this section, present the operational mechanism and architecture of CrossMPT. CrossMPT processes the magnitude and syndrome embeddings separately, applying a cross-attention mecha- nism to effectively capture their distinct information. It shares principles with message passing

---

Published as a conference paper at ICLR 2025
Product operation
Masked CA
V
VN output message
CN output message
Magnitude
Update
Syndrome
K.
Sum operation
Masked
CA
Cross-attention
M M algorithm
message-passing algorithm
Figure 3: Conceptual comparison of the sum-product message-passing algorithm and the proposed cross-attention (CA) message-passing algorithm.
algorithm for decoding linear codes, iteratively updating the magnitude and syndrome embeddings of the received codewords. The overall architecture depicted in Figure
4.1 CROSS-ATTENTION MESSAGE-PASSING TRANSFORMER
One cross-attention block updates the magnitude embedding by using it as the query while gener- ating the key and value from the syndrome embedding. Given this configuration, the attention map has the size n x (n-k), effectively representing the imagnitde-syndroom relationship,
relation. To reflect this employ the transpose of the PCM H the mask matrix. This is because the rows of H correspond to the n bit positions, and its n k columns are associated with the parity check equations, directly linking yy| and syy), respectively. The other cross-attention block simi- larly uses the syndrome embedding for the query, while the magnitude embedding generates the key and value. For this operation, we utilize the PCM H as the mask matrix.
This configuration of separately processing two distinct informational properties resembles the pressagepasssngg decoding algorithm for decoding linear codes. Message-passing algorithms such as the sum-product algorithm (Richardson & Urbanke| [001] are widely used for decoding LDPC codes. The pesssage-passingg algorithm operates by exchanging messages between variable nodes (VNs) and check nodes (CNs) over a Tanner (bipartite) graph (Richardson & Urbanke 2001) In the Tanner graph, VNs convey information about the reliability of the received codeword. while CNs indicate the parity check equations. The edges between VNs and CNs represent the connec- tions (relationships) between them. The message- message-passing passing decoder operates by exchanging messages between VNs and CNs via these edges. The output messages of VNs and CNs updated in an iterative manner.
Similar the principles of message-passing algorithms, CrossMPT iteratively updates the mag- nitude and syndrome embeddings. First, the magnitude embedding is updated using the masked cross-attention block, where the magnitude embedding for the query and syndrome embedding for the key and value. The syndrome embedding is updated in the subsequent masked cross-altention block, utilizing the previously updated magnitude embedding. In this block, the syndrome embed- ding is used for the query, while the updated magnitude embedding generates the key and value. The resulting output from this eross-atted aessettttntion block is the updated syndrome embedding. CrossMPT it- eratively updates both the magnitude and syndrome embeddings to estimate the multiplicative noise accurately.
As representative of the cassage-passsngg algorithm, Figure depicts the sum-product algorithm and the cross-attention message-passing algorithm. In the sum-product algorithm, the VN output and CN output messages are iteratively updated using the and product operations. Similar to the sum cum-product algorithm, the magnitude and syndrome embeddings are iteratively updated using the masked cross-attention (Masked CA in the figure) blocks in CrossMPT. Note that H and H" serve as the mask matrices for these cross- tross-ateention blocks, while Q, K, and V denote the query, key, and value of the cross-attention mechanism.
5

---

Published as a conference paper at ICLR 2025
4.2 MODEL ARCHITECTURE
In the initial embedding layer, we generate lul I...,,.,, (al)) and s(y) (s(y) (...),...., from the received codeword, and project each element yi and s(y): into d dimension embedding row vectors M, and S, respectively, as follows:
,...., ,..,,,8())) s(y)n-1)
M, y/ W,
for i 1, .....,. n2,
S, = s WWy
for i = ,,..., k,
where W; E RIxd i- 1..... 2m k denotes the trainable positional encoding vector.
These magnitude and syndrome embeddings are processed as separate inputs in the subsequent N decoding layers. Each decoding layer contains two cross-attention blocks, cach consisting of cross-attention module, a feed-forward neural network (FFNN), and normalization layer.
In the first cross-attention module, the attention module updates the 'magnitude" embedding by using the syndrome embedding. The query Q1 key K,, and value Vi are assigned follows:
Q MWQ: K SW K, V1 SSW,,
(((=-)xxd denote the magnitude where M [M :.....-;MM M,) E nad and S [S ( [n_kk]
and syndrome embeddings, respectively. Here, Wq. WK, Wy denote the weight matrices for the query, key, and value, respectively. This architecture is referred as the cross-attention message- passing transformer since the query corresponds to the magnitude embedding, while the key and value correspond to the syndrome embedding. Then, employ the following scaled dot-product attention:
(QIK) + V1,
Attention(Q2. K1, Vi) softmax
Vd
where g( is the mask matrix, and the function g is defined as
10 if Ar, g(A)ss {- -00 if Ai,j =
This configuration results in an attention map of size n x (n k)), representing the 'magnitude- syndrome" relationship. Therefore, we use the transpose of the PCM H as the mask matrix. Here, the n rows of H correspond to the n bit positions and the n - k columns H T to the parity check equations, which are closely related to ly| and s(y). respectively. Finally, the output of this cross-attention module yields the updated magnitude embedding M'.
In the second cross-attention module, we update the 'syndrome" embedding using the updated mag- nitude embedding M'. In other words, the syndrome embedding serves as the query input, while M' is used for both the key and value inputs. We the shared weight matrices Wo, WK. Wy from the first cross-attention module, and query Q2, key K2, and value 1/2 are defined as follows:
Q2 SWe, K2 M'WK, V/2 M'Wv.
Here, the syndrome and magnitude embeddings correspond the rows and columns of the attention map. respectively. Thus, we employ the mask matrix g(H), whose masking positions are zeros in Н. Then, we apply the scaled dot-product attention and the resulting output provides the updated syndrome embedding S'. This updated syndrome embedding is utilized to further refine the magni- tude embedding in the subsequent decoder layer, and this process is iteratively repeated across the N decoder layers.
Finally, the updated magnitude and syndrome embeddings from the last decoder layer are concate- nated and passed through a normalization layer and two fully connected (FC) layers. The first layer reduces the (2n - k) x d dimension embedding second FC layer further reduces the dimension from 2n - k into n. The final output provides an estimation of Σ,. Since two cross-attention blocks of CrossMPT share the same weight matrices Wo. WK, Wv and all other layers, CrossMPT has the same number of parameters as the original ECCT.
one-dimensional 2n k vector, and the
The objective of the proposed decoder is to learn the multiplicative noise ž, in [T) and reconstruct the original transmitted signal v. We can obtain the multiplicative noise by i, 2, ye.. Then, the

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of decoding performance at three different Pa/No, (4 dB, dB, dB) for BP decoder. Hyper BP decoder (Nachmani & Wolf 2019), AR BP decoder (Nachmani & Wolf (02I), ECCT (Choukroun & Wolf 2022a), and the proposed CrossMPT. The results are measured by the negative natural logarithm of BER. The best results are highlighted in bold. Higher is better.
Architecture
Bi-based decoders
Modkl-free decoders
Codes
Parameter
BP
Hyp BP
AR BP
ECCT
CRosMPT
6
4
5
(31.16)
4.63
5.88
7.60
5.05
6.64
8.80
5.48
7.37
9.60
6.39
8.29
10.66
6.98
9.25
12.48
BCH
(63,36)
4.03
5.42
7.26
4.29
5:91
8.01
4.57
6.39
8.92
4.86
6.65
9.10
5.03
6.91
9.37
1.26
8.51
1.97
(63,51)
4.5
5.82
7.42
4.8
6.44
8.58
5.17
7,16
9.53
5.66
7.89
11.01
5.78
8.08
11.41
(64.32)
4.26
5.38
6.50
4.59
1.10
T.69
5.57
7:43
9.82
6.99
9.44
12.32
7,50
9.97
13.31
7.42
8.39
5.41
7.19
Polar
(128,654
4.1
5.11
6.155
4.52
6.12
8.25
4.84
6.78
9.3
5.92
8.64
12.18
7.52
11.21
14.76
(128,86)
4.49
5.65
6.97
4.95
1884
99.28
5.39
7,37
10.13
6,31
9:00
12:45
7.51
10.83
15.24
(128,96)
9.09
5.27
7.44
(19.24)
6.23
8.19
11.72
6.23
8.54
11.95
6.58
9.39
12.39
6.13
8.71
12.10
6.68
9,52
13.19
LDPC
(121.60)
4.82
7.21
10.87
5.22
8.29
13.00
5.22
8.31
13.07
5.17
8.31
13.30
5.74
9.26
14.78
(121,70)
8.76
13.04
6.39
9.81
14.04
6.45
10:01
10.21
16.11
7.06
11.39
17.52
(121,80)
6.66
9.82
13.98
6.95
10.68
15.
7.22
11.03
15,90
7.41
11.51
16:44
7.99
12.75
18.15
MacKay
(96.48)
6.84
9.40
12:57
7.19
10.02
13.16
7.43
10.65
14.65
7.38
10.72
14.83
7.97
11.77
15.52
OCSDS
(128,64)
6.55
9.65
13.78
6.99
10.57
15:27
7.25
10.99
16.36
6.88
10.90
15.90
7.68
11.88
17.50
Turho
(132,40)
NIA
N/A
N/A
NA
N/A
N/A
NIA
N/A
N/A
4.74
6.54
9.05
5.55
7.92
10.94
109
10P
2074
10"1
30-1
10'1
HapBP
10th
H2p
18"
13p
BOTT
BEET
BOST
ChosMPT
ChosMPT
18-
CROMMPT
16-1
(NR()
(N
18"
52
(a) (31, 16) BCH code
(b) (128, 86) polar code
(c) (128, 64) CCSDS
Figure 4: The BER performance of various decoders (BP, Hyp BP, AR BP, ECCT) and CrossMPT.
target multiplicative noise for binary cross-entropy loss function is defined by غ bin(sign(yz.)) Finally, the cross-entropy loss function for a received codeword y is defined by
Σ (ä, log(1 - (((()))) + (1 ē4) log(o
To ensure fair comparison between CrossMPT and ECCT, we adopt the same training setup used in the previous work (Choukroun & Wolf, 2022a). We use the Adam optimizer (Kingma & Ba 2015) and conduct 1000 epochs. Each epoch consists of 1000 minibatches, where each minibatch is composed of 128 samples. All simulations were conducted using NVIDIA GeForce RTX 3090 GPU and AMD Ryzen 9 5950X 16-Core Processor CPU. The training sample y is generated by y I, + z, where x, is the modulated signal corresponding to the all-zero codeword, and a represents the AWGN channel noise, sampled from normalized signal-to-noise ratio (Eg/No range of 3 dB to 7 dB. The learning rate is initially set to 10-4 and gradually reduced to 5 x 10-7 following a cosine decay scheduler.
EXPERIMENTAL RESULTS
In this section, we compare the proposed CrossMPT with the original ECCT across various code classes. Our experimental results do not include a comparison with the works of (Choukroun &
Wolf 2024a b). as they have different objectives, such as generalizing the decoder to unseen codes Choukroun & Wolf 2024a) or jointly training the encoder and decoder (Choukroun & Wolf (olf|2024b) (024b). It is worth mentioning that cross-attention architecture and the schemes of (Choukroun & Wolf
7

---

Published as a conference paper at ICLR 2025
0.0
0.00
ECCT
CiosSMPT
Figure 5: The average attention scores of all N 6 layers for ECCT and CrossMPT.
2024a b) are orthogonal methods, and combining them could present a promising direction for future research.
To verify the efficacy of CrossMPT,, we train it for BCH codes, polar codes, turbo codes, and LDPC codes (including MacKay and CCSDS codes) and evaluate the bit error rate (BER) performance. All PCMs are taken from (Helmling et al. 2019). The implementation of the original ECCT is obtained from (Choukroun & Wolf 20226). For the testing, we collect at least 500 frame errors at each E No with random codewords. Table compares the decoding performance of CrossMPT with the BP decoder, BP-based neural decoders (Nachmani & Wolf 2019 2021), and ECCT (Choukroun) & Wolf, 2022a). The results of the BP-based decoders in Table I are obtained for 50 iterations. The results for both the proposed CrossMPT and ECCT, which are model-free decoders, are obtained with N 6 and d = 128. For all types of codes, CrossMPT outperforms the conventional ECCT and all the other BP-based neural decoders. This improvement of CrossMPT is particularly notable in the case of LDPC codes. To provide more visual information, plot the BER graphs for several codes in Figure 4
An important aspect of our research is CrossMPT's capability decode long codes (Appendix (A). which remain beyond the reach of ECCT due to its high memory requirements, resulting from large attention maps. These results demonstrate the practical significance and architectural advan- tages of CrossMPT, proving its value in scenarios where ECCT encounters limitations. Especially, it achieves superior decoding performance for LDPC codes, outperforming the BP decoder with the maximum iteration of 100 (provided in Appendix B). Additionally, for short codes, CrossMPT closely approaches the optimal maximum likelihood (MI.) decoding performance (provided in Ap- pendix C). Additional experimental results for block error rate (BLER), comparison with successive cancellation list polar decoder, denoising diffusion ECCT (DDECCT), and the decoding perfor- mance for the Rayleigh channel, are provided in Appendices D and respectively.
ABLATION STUDIES AND ANALYSIS
6.1 ANALYSIS OF ATTENTION MECHANISMS IN ECCT AND CROSSMPT
We provide an analysis of the attention scores in ECCT and CrossMPT. Figure 5 shows the aver- age attention scores across N 6 layers for both ECCT and CrossMPT on the (3) LDPC code (Abu-Surra et al. 2010). As shown in Figure 5 the attention map of ECCT reveals different importance among the relationships: syndrome. One key observation is that the magnitude- magniuud-mmanntuud and syndrome-syndrom relations exhibit relatively low attention scores compared to the magnitude-syndrome relation, which sug- gests that the magnitude-syndron which we mask the seggiitud-mmagnituee and syndrome-syndrome relationships, reveal no signif- icant performance difference compared to when these relationships are not masked. This demon- strates that the conventional ECCT could be enhanced by focusing on the more critical relationships, CPossMPT achieves this by eliminating the two relations with low attention scores and concen- trating solely on the magnitude-syndronn relation. Therefore, we can claim that CrossMPT more efficiently focuses the crucial aspect (i.e., magnitude-syndrom relation) compared to ECCT.
siemitude-magnittue
syndrome-syndronm
and magnitude-
relationship is more significant than the others. Appendix [H] in
8

---

Published as a conference paper at ICLR 2025
Layer1
Laye-2
Error position
(a) Attention scores with a single bit error
(b) Summation of attention
scores with single bit error
1.5
Layer
0.5
(c) Attention scores without an error
(d) Summation of attention
Scores without an error
Figure 6: The attention scores (a), (c) with single bit error in the first bit position and without an error The attention scores (b), (d) is carried out in the vertical direction.
6.2 VISUALIZATION OF CROSS-ATTENTION MAP
To further examine how CrossMPT operates, we intentionally corrupt a pre-determined bit of the (32, 16) LDPC code and analyze the resulting attention maps. Figure 6 shows the attention scores for the first two layers and the summation of their attention scores when the first bit is corrupted. The summation is carried out vertically to demonstrate the attention score for each bit. As shown in Figure b)), the attention score of the first bit (or first column) is relatively higher than the others. However, once the error is corrected, CrossMPT no longer assigns high attention scores to that position (see Figure 12 in Appendix D. Figures [6(c) and [(d) depict the attention scores and the summation of attention scores when no errors are present. Compared to the previous case, the attention scores are more uniformly distributed across all bit positions.
6.3 COMPLEXITY ANALYSIS
Two cross- cotss-attnntoon blocks of CrossMPT share the same parameters for all decoder layers. They use the same weight matrices Wo. WK, Wv for two cross-attention modules since the performance remains nearly identical even when the parameters are trained separately. Also, they share the parameters for the normalization layers and the FFNN layers. Thus, CrossMPT maintains the same number of parameters as the original ECCT.
Figure [ illustrates the mask matrices of ECCT and CrossMPT, In the original ECCT, Figure aa) shows that significant portion of the upper X submatrix is depicted in white, indicating that the most positions are unmasked. This n X n submatrix represents depth-2 connections in the Tanner graph (Choukroun & Wolf 2022a). which results in an increase in the number of unmasked posi- tions, thereby leading to a higher computational required. On the other hand, the lower (n k) x n submatrix and the right (n k) x n submatrix, which serve the masking matrices for CrossMPT, are predominantly shown in blue, indicating that their attention matrices are sparser. Figure 7 com- pares the mask matrix density CrossMPT and ECCT. For all codes, the mask matrix of CrossMPT is sparser than ECCT, which implies that CrossMPT can achieve lower computational complexity compared to the original ECCT.
The complexity of the self-attention mechanism of ECCT, without considering masking is, ONN (2n k) + (2n d)) When masking is taken into account, the complexity can be reduced to (d the fixed number of computations of the self-attention module and Pi denotes the density of the mask matrix in ECCT. Similarly, the complexity of the two cross-attention modules of CrossMPT, without considering the masking, is ON (2n-k) + 2n(n - k)d)). When masking is taken into account, the complexity can be reduced to (d k) + (2h)d)). where h = P2n(n (n k) denotes the number of computations of single cross-attention module and P2 denotes the density
k) + hd)) (Choukroun & Wolf 2022a), where h P1 (2m k)2 denotes

---

Published as a conference paper at ICLR 2025
100
(OC
CosMPP (rz)
DCH (31,16)
DCH
BCH (63,40)
BCH
Poler
Polar
Polar
Poler
Polrr
LDJC
LURC
LDNC 12
LDSC
Maday OCEDS Tubo
(03
(63,51)
(64,22]
(64,
(128
(222
(48.24)
(38,45) (12)
Figure 7: Comparison of the mask matrix density between ECCT and CrossMPT.
Table 2: Comparison of FLOPs, inference time, and training time between ECCT and CrossMPT for various codes. Inference time is measured for decoding single codeword and training time is measured for a single epoch.
Codes
Parameter
FLOPs
Inference (codeword)
Training (epoch)
Mask density
Memory usage
CrossMPT
ECCT
CrossMPT
ECCT
CrossMPT
ECCT
CrossPPT
ECCT
CrossMPT
ECCT
BCH
(63,45)
106.4
326 ps
328 js
29
32.45%
53.09%
LDPC
(121,70)
(121,80)
229.7 M
212.5 M
238.0 M
256.8 M
400 ps
391 ps
436 jcs
450 js
53
58
80%
76
9.09%
9.09%
24.01%
21.94%
1936 MiB
1980 MiB
3602 MiB
3926
Turbo
(132,00)
303.6 M
343.4 M
459
511
83
110s
11.43%
14.25%
2362 MiB
5580 MiB
RCH
(255,223)
747 ps
js
48.63%
78.21%
1036 MiB
7318
WRAN
(384,320)
53.1
111.3M
1295 ps
1638 ym
104
305
5.21%
13.25%
3270 MiB
18192 MiB
of the mask matrix in CrossMPT. Furthermore, since Pi > P2 as shown in Figure 7 that > 2h, which indicates that CrossMPT achieves reduction in computational complexity compared the original ECCT.
we conclude
compares the total FLOPs, inference time, training time, and memory usage between ECCT and CrossMPT. The inference time refers to the duration required to decode single codeword and the training time measures the duration to complete one epoch of training. All results are obtained for N = 6 and d = 128, except for (255,223) BCH code and (384,320) WRAN LDPC code, which are obtained for N = 6 and = 32. For all three metrics, CrossMPT outperforms ECCT. Since inference and training times are closely related to the FLOPs, reduction in FLOPSs directly leads to shorter inference and training times. Notably, CrossMPT significantly reduces memory usage compared ECCT, especially for long codes. This improvement arises from the reduced size of the attention map; 2n(n - k) for CrossMPT and (2n 2)² for ECCT. The results in Tables [ and 2 demonstrate that the proposed CrossMPT not only improves the decoding performance but also significantly reduces FLOPs, inference time, training time, and memory usage compared to the original ECCT. Additional analysis the training time required to achieve the target loss is provided in Appendix[]
CrossMPT's sequential decoding approach may limit its throughput in certain scenarios. How- ever, pipelining (Li et al. 2021b] Rowshan et al. (024) enables CrossMPT to effectively increase its throughput when decoding multiple codewords (see Appendix[K]
CONCLUSION
We developed a novel transformer architecture for ECC decoding called CrossMPT, which improves both decoding performance and computational efficiency. CrossMPT achieves this by adopting a more effective architecture that processes magnitude and syndrome embeddings through the cross- attention mechanism. This approach leverages the structured representation of codeword bit rela- tionships in the PCM, enabling the model to accurately learn these relationships while significantly reducing memory usage, FLOPs, inference time, and training time. Most existing transformer based decoders have been limited to short codes due to challenges in training long codes, primarily caused by high memory usage and computational complexity. However, CrossMPT effectively addresses these challenges, improving the viability of transformer-based decoders for long codes.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean Government (MSIT) (RS-2024-00398449, Network Research Center: Advanced Channel Coding and Channel Estimation Technologies for Wireless Communication Evolution) and the National Research Foundation of Korea (NRF) grant funded by
the Korean Government (MSIT) (No. RS-2023-00212103)
REFERENCES
A. Abu-Surra, D. DeClercg, D. Divsalar, and W. E. Ryan. Trapping set enumerators for specific LDPC codes. In Information Theory and Applications (ITA) Workshop, 2010.
S. K. Ankireddy and H. Kim. Interpreting neural min-sum decoders. In IEEE International Confer- ence on Communications (ICC), 2023.
H. Bae, A. Abotabl, H. P. Lin, KB. Song, and J. Lee. An overview of channel coding for 5G NR cellular communications. APSIPA Transactions Signal and Information Processing, 8(1):e17, 2019.
A. Bennatan, Y. Choukroun, and P. Kisilev. Deep learning for decoding of linear codes-a syndrome- based approach. In IEEE International Symposium on Information Theory (ISIT), pp. 1595-1599, 2018.
A. Buchherger, C. Hager, H. D. Pfister, L. Schmalen, and A. G. I. Amat. Pruning and quantizing neural belief propagation decoders. IEEE Journal on Selected Areas in Communications, 39(7): 1957-1966, 2021.
S. Cammerer, T. Gruber, J. Hoydis, and S. ten Brink. Scaling deep learning- -based decoding of polar codes via partitioning. In IEEE Global Communications Conference (GLOBECOM)) 2017.
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020.
C.-F. Chen, Q. Fan, and R. Panda. CiossViT: Cross-attention multi-scale vision transformer for image classification. JEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Y. Choukroun and I.. Wolf. Error correction code transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Y. Choukroun and L. Wolf. Error correction code transformer. LoriTC/ECCT] 2022b. Accessed: 2023-05-22.
https:://iithi: com/
Y. Choukroun and L. Wolf. Denoising diffusion enror correction codes. In International Conference on Learning Representations (ICI.R, 2023.
Y. Choukroun and L. Wolf. A foundation model for error correction codes. In International Con- ference on Learning Representations (ICLR), 2024a.
Y. Choukroun and L.. Wolf. Learning linear block error correction codes. In International Conference on Machine Learning (ICML), 2024b.
J. Dai, K. Tan, Z. Si, K. Niu, M. Chen, H. V. Poor, and S. Cui. Learning to decode protograph LDPC codes. IEEE Journal on Selected Areas in Communications, 39(7): 1983))99331199 2021.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. In North American Chapter the Association for Compu- tational Linguistics (NAACL), 2019.
M. P. C. Fossorier, M. Mihaljevic, and H. Imai. Reduced complexity iterative decoding of low- density parity check codes based on belief propagation. IEEE Transactions on Communications, 47(5):673-680, 1999.
11

---

Published as a conference paper at ICLR 2025
R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
T. Gruber, S. Cammerer, Hoydis, and T. Brink. On deep learning-basee channel decoding. In 51st Annual Conference on Information Sciences and Systems (CISS), pp. 1 1-6, 2017.
P. Hailes, L. Xu, R. G. Maunder, B. M. Al-Hashimi, and L. Hanzo. A survey of FPGA-basse LDPC decoders. IEEE Communications Surveys Tutorials, 18(2): 1892)1098-1111 122, 2015.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In ЛEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2016.
M. Helmling, Scholl, F. Gensheimer, Dietz, D. Kraft, S. Ruzika, and N. Wehn. Database of Channel Codes and ML Simulation Results. In hhttts::/rrppp 2019.
htnp/hhanel-cooee
H. Kim, Y. Jiang, R. Rana, Kannan, S. Oh, and P. Viswanath. Communication algorithms via deep learning. In International Conference Learning Representations (ICLR), 2018.
H. Kim, S. Oh, and P. Viswanath. Physical layer communication via deep learning. IEEE Journal on Selected Areas in Information Theory, 1(1):--18, 2020.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
H.-Y. Kwak, J.-W. Kim, Y. Kim, S.-H. Kim, and J.-S. No. Neural min-sum decoding for generalized LDPC codes. IEEE Communications Letters, 26(12)22841 2845, 2022.
H.-Y. Kwak, D.-Y. Yun, Y. Kim, S.-H. Kim, and J.-S. No. Boosting learning for LDPC codes to improve the error-floor performance. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2023.
H.-Y. Kwak, D.-Y. Yun, Y. Kim, S.-H. Kim, and J.-S. No. Boosted neural decoders: Achieving extreme reliability of LDPC codes for 6G networks. IEEE Journal on Selected Areas in Commu- nications, 2025.
F. Li, C. Zhang, K. Peng, A. E. Krylov, A. A. Katyushnyj, A. V. Rashich, D. A. Tkachenko, S. B. Makarov, and J. Song. Review 5G NR LDPC code: Recommendations for DTTB system. IEEE Access, 9:155413-155424, 2021a.
M. Li, V. Derudder, K. Bertrand, C. Desset, and A. Bourdoux. High-speed LDPC decoders towards Tb/s. IEEE Transactions on Circuits and Systems Regular Pupers, 68(5):2224-2233 2021b.
L. Lugosch and W. Gross. Neural offset min-sum decoding. In IEEE International Symposium on Information Theory (ISIT), pp. 1316-1365, 2017.
E. Nachmani and L. Wolf. Hyper-graph-network decoders for block codes. In Advances in Neural Information Processing Systems (NeurIPS), pp. 2326-2366 2336, 2019.
E. Nachmani and L. Wolf. Autoregressive belief propagation for decoding block codes. arXiv preprint arXiv:2103. 117.00 2021.
E. Nachmani, Y. Beery, and D. Burshtein. Learning to decode linear codes using deep learning- In 54th Annual Allerton Conference Communications, Control, and Computing, pp. 341-346, 2016.
E. Nachmani, E. Marciano, L. Lugosch, W. J. Gross, D. Burshtein, and Y. Beery. Deep learning methods for improved decoding of linear codes. IEEE Journal of Selected Topics Signal Pro- cessing,
131,911,,,1100 2018.
S.JJ. Park, H.-Y. Kwak, S.-H. Kim, S. Kim, Y. Kim, and J.-S. No. Multiple-man masks error correction code transformer for short block codes. IEEE Journal on Selected Areas in Communications, 2025.
12

---

Published as a conference paper at ICLR 2025
T. Richardson and R. Urbanke. The capacity of low-density parity check codes under message- passing decoding. IEEE Transactions on Information Theory, 47(2):599-611 -18, 2001.
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2022.
M. Rowshan, M. Qiu, Y. Xie, X. Gu, and J. Yuan. Channel coding toward 6G: Technical overview and outlook. IEEE Open Journal of the Communications Society, 5:2585-2688 2024.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2017.
L. Wang, S. Chen, J. Nguyen, D. Dariush, and R. Wesel. Neural-network-oppiinnz degree-specific weights for LDPC minsum decoding. In IEEE International Symposium on Topics in Cod- ing (ISTC), 2021.
A PERFORMANCE ON LONGER CODES
We present the BER performance for three longer codes in Figures 8(a) 8(b) and 8(c) for ECCT and CrossMPT N 6, 32. For all three codes ((a) (529,440) LDPC code, (b) (384,320) wireless regional area network (WRAN) LDPC code, (c) (5 (512.384) polar code), the proposed CrossMPT outperforms the original ECCT. Despite its reduced complexity, CrossMPT significantly enhances the decoding performance compared ECCT, not only for short-length codes but also for longer codes. Also, Figures [(d) and S(e) show the decoding performance of CrossMPT for much longer codes. The BER performances of the (648,540) IEEE 802. In LDPC code (N 10, d 128) and (1056,880) WiMAX LDPC code (N = 6, d = 32) demonstrate that CrossMPT efficiently trains how to decode the codeword even for large N and d and performs well for longer codes. Again, we emphasize CrossMPT's capability to decode long codes where ECCT struggles due to high memory allocation (large attention map). The structure of CrossMPT demonstrates its efficiency in learning long codes, surpassing the limitations of short or moderate codelengths of transformer- based decoders.
19.7
11a
10a9
23n0
12
EXCT
DusokP
10
EXCT
QOskET
EOCT
13
SNR(E./%)
SNR(E,1R))
SNIVEL
(a) (529,440) LDPC code
(b) (384,320) WRAN LDPC code
(c) (512,384) polar code
10"1
78"
10'4
1074
10
325
35
425
4.7
38"1
325
35
435
(d) (648,540) IEEE 802.11n
(e) (1056,880) WIMAX
Figure 8: The decoding performance of long codes.
13

---

Published as a conference paper at ICLR 2025
B COMPARISON WITH THE BP DECODER
Figure 9 shows the decoding performance between the traditional BP decoder with maximum num- ber of ieerations of 20, 50, and 100 and CrossMPT for both short and long LDPC codes. Figures ((a) and 9(b) compare the BER performance for (121,80) LDPC codes (N = 6, 128) and (648,540) IEEE 802. LDPC code (N 10, d = 128), respectively. Notably, can outperform the BP decoder for both short and long LDPC codes. These results highlight that CrossMPT efficiently trains how to decode the codeword across a wide range of code lengths.
proposed CrossMPT
10
10
10-4
10-a
BER
10
10
10"4
RP,
10"6
BE, 50
BP 20
10
BP, 100
10
BP 30
10"7
RCCT
10
BP 100
CrosMPT
CrosMPP
10
10"*
Fg/No
Eg/No
(a) (121,80) LDPC code
(b) (648,540) IEEE 802.1In
Figure 9: Performance comparison between BP decoder (iteration 20, 50, and 100) and CrossMPT.
C COMPARISON WITH THE ML DECODER
We compare ECCT and CrossMPT with the ML decoder for short BCH codes. Figure TO demon- strates the BER performance of (31, 16) BCH code and (31, 21) BCH code. Especially, these results show that CrossMPT closely approaches the optimal ML performance for short codes.
10"
10-1
10"4
ML decoding
ML decoding
10-2 BOCT
10-2
BCCT
CrosMPP
CresMIT
10
10
Eg/No
Eg/No
(a) (31,16) BCH code
(b) (31.21) BCH code
Figure 10: The decoding performance comparison between MI. decoder, FCCT, and CrossMPT.
D BLOCK ERROR RATE PERFORMANCE
Table 3 demonstrates BLER results for various code classes (N 6, = 128). Also, Fig-
ure shows the BLER performance of (31,16) BCH code, (63,51) BCH code, and (648,540) IEEE 8n LDPC code. For BCH codes, compare the decoding performance of CrossMPT with the traditional Mameekamp-assseee (BM) decoder, maximum likelihood (ML) decoding algorithm, ECCT. For LDPC codes, compare the decoding performance of CrossMPT with the traditional
14

---

Published as a conference paper at ICLR 2025
BP decoder with maximum number of iterations of 20, 50, and 100. As shown in the table, CrossMPT outperforms ECCT in the BLER performance and also has a comparable BLER results compared to the traditional decoding algorithms.
In addition, the traditional decoders are code-specific decoders, tailored to each class codes. For example, LDPC codes effectively decoded by the BP decoder, BCH codes by the BM decoder, and polar codes by the SCL decoder. However, unlike the traditional decoders, a key advantage of CrossMPT is its versatility. While conventional decoders are good and valid only for respective code classes, CrossMPT performs effectively across a wide range of code classes. This universality highlights the broader applicability and potential of CrossMPT in various decoding scenarios and future communication paradigm such and semantic communication.
The complexity of transformer-basee decoders is relatively high compared code-specific de- coders. Reducing their computational requirements will be an important focus for future work.
Table 3: The BLER results for ECCT and CrossMPT. The results are measured by the negative
natural logarithm of BLER.
Method
ECCT
CrosMPT
Pa/No
(31.6)) BCH
4.19
5.98
8.16
5.12
7.24
10.31
(63,36) BCH
2.43
4.10)
6.40
2.50
4.23
6.61
(63,45) BCH
2.75
4.75
7.67
3.18
5.33
8.66
(63.51)
2.72
2.94
8.36
(64,32) Polar
4.18
6.47
9.07
4.83
7.26
10.41
(64, Polur
3.08
5.06
7.60
3.56
5.68
8.35
(49.24)
4.47
10.68
(121,70) LDPC
3.21
6.69
11.91
4.28
8.42
13.35
16a
1274
Midcoday dooder
Midcoday BM derdder
BP 100
OreMiT
5/7%
5,7%
5.7%
(a) (31,16) BCH code
(b) (63,51) BCH code
(c) (648,540) IEEE 802.
Figure 11: The BLER performance comparison between the traditional decoders and CrossMPT.
E COMPARISON WITH SUCCESSIVE CANCELLATION LIST POLAR DECODER
We compare the BER performance of the SCL decoder, ECCT, and CrossMPT in Table The performance of the SCL decoder is from (Choukroun & Wolf 2022a). Although the contribution of L. is significant in long codes, the SCL decoder achieves great performance with small L. such as L = 4. As reported in (Choukroun & Wolf 2022a 2023), the SCL decoder outperforms ECCT. This because the SCL decoder is a decoder specialized for Polar codes and state-of-the- art algorithm that has undergone extensive development over a long period. CrossMPT has made significant improvements from ECCT and even outperforms the SCL decoder for (64,48) polar code.
F COMPARISON WITH DDECCT
For a fair comparison with DDECCT, we also apply the denoising diffussinn training technique to CrossMPT. Table compares the BER performance of ECCT (Choukroun & Wolf 2022a), CrossMPT, DDECCT, and CrossMPT applying the denoising diffusion model. All four decoders model-free decoders using the transformer architecture, and simulations taken for N 6.
15

---

Published as a conference paper at ICLR 2025
Table Comparison of decoding performance three different Fg/No (4 dB, 5 dB, dB) for SCL. decoder, ECCT, and CrossMPT, The results are measured by the negative natural logarithm of BER. The best results are highlighted in bold and the second best is underlined. Higher is better.
Method
SCL (L
SCL (L 4)
ECCT
CrossMPT
Parameter
4
5
6
4
5
6
4
6
4
5
(64,32)
7.30
9.67
13.18
8.11
10.70
14.04
6.99
9.44
12.32
7.50
9.97
13.31
(64,48)
6.19
8.41
10.97
6.69
8.63
11.24
6.36
8.46
11.09
6.51
8.70
11.31
(128,64)
8.37
11.69
13.70
9.60
13.16
17.42
5.92
8.64
12.18
7.52
11.21
14.76
(128,86)
7.54
10.74
15.14
9.26
13.04
17.13
6.31
9.01
12.45
7.86
11.45
15.47
(128,96)
6.74
9.53
13.53
8.02
11.60
18.16
6.31
9.12
12.47
7.15
10.15
13.13
d = 128. We conduct simulations for codes where DDECC performs better than CrossMPT. For the rest of the codes, CrossMPT outperforms DDECC. The proposed CrossMPT shows superior decod- ing performance compared to the original ECCT. Compared to DDECC, CrossMPT demonstrates similar BER performance for polar codes, codes. When the denoising diffusion technique is applied to CrossMPT, it achieves the best perfor- mance among others, where DDECC, CrossMPT, and ECCT follow. This proves that the CrossMPT architecture provides separate gain from the denoising diffusion algorithm for transformer-bii de- coders.
even outperforms DDECC for BCH and LDPC
Table 5: Comparison of decoding performance at three different Eg/No (4 dB, 5 dB, dB) for ECCT (Choukroun & Wolf, 2022a), CrossMPT, and DDECC (Choukroun & Wolf 2023). The results are measured by the negative natural logarithm of BER. The best results are highlighted in bold and the second best underlined. Higher better.
Architecture
Without denoising diffusion
With denoising diffusion
ECCT
CrossMPT
ECCT
CrossMPT
Codes
Parameter
5
6
4
5
6
4
4
5
6
BCII
(63,36)
4,86
6.65
9.10
5.03
6.91
9.37
5.11
7.09
9.82
5.23
7.20
10.01
(128.64)
5.92
8.64
12.18
7.52
11.21
14.76
9.11
12.9
16,30
10.21
13.63
17.28
Polar
(128,86)
6.31
9.01
12.45
7.51
10.83
15.24
7.60
10.81
15.17
8.56
12.04
15.37
(128,96)
6.31
9.12
12.47
7.15
10.15
13.13
7.16
10.3
13.19
7.57
10.61
13.33
MacKay
(96,48)
7.38
10.72
14.83
7.97
11.77
15.52
8.12
11.88
15.93
8.85
12.58
17.69
G DECODING PERFORMANCE FOR RAYLEIGH FADING CHANNEL
The original ECCT architecture shows robustness non-Gaussian channels (e.g., Rayleigh fading channel) Choukroun & Wolf 2022a Supplementary). We also measured the decoding performance of CrossMPT in Rayleigh fading channels. To compare with ECCT, we use the same fading channel as in (Choukroun & Wolf 2022a). The received codeword is given n-dimensional i.i.d. Rayleigh distributed vector with a scale parameter a and = (O0, o²). The following table demonstrates the BER performance of ECCT and CrossMPT in Rayleigh fading channel and CrossMPT still outperforms the original ECCT architecture for all types of codes.
= hæ + where h is an
Codes
(31.16) BCH
(64,32) Polar
(128.64) Polar
(128,86) Polar
(121,70) LDPC
(128.64 CCSDS
Methods
5
4
6
5
5
4
6
BOCT
5.18
6.04
6.92
5.53
7,80
4.31
5.37
6.63
4.02
4.81
5.70
3.91
4.97
6.31
2.46
3.97
5.79
CPssMPT
5.53
6.55
7.61
5.91
7.17
8.48
4.70
5.93
7.34
4.41
5.38
6.46
4.25
5.53
7.11
5.25
6.94
8.92
16

---

Published as a conference paper at ICLR 2025
H ABLATION STUDY WITH ADDITIONAL MASKING
To understand the impact of -aggituue and syndrome-syndrome relationships, we ex- ammine the decoding performance of ECCT with additional masking applied to all positions responding to these relationships. Table 6 compares the decoding performance of ECCT with this additional masking, standard ECCT, and CrossMPT. The results show no significant performance degradation with the additional masking, indicating that the magnitude-magniuee and syndrome- syndrome relationships are not critical decoding performance.
Table 6: Comparison of decoding performance at three different Es/No (4 dB, 5 dB, dB) for ECCT with additional masking, standard ECCT, and CrossMPT. The results are measured by the negative natural logarithm BER. The best results are highlighted in bold. Higher better.
Method
ECCT Masking
ECCT
MPo
Parameter
5
5
5
(31, 16) BCH
6.52
8.55
11,42
6.39
8.29
10,66
6.98
9.25
12.48
(63, 45) BCH
5.53
7.74
10.88
5.60
7.79
10.93
5.90
8.20
11.62
(64, 48) Polar
6.25
8.26
10.93
6.36
8.46
11.09
6.51
8,70
11.31
(121, 60) LDPC
4.98
7.91
12.61
5.17
8.31
13.30
5.74
9.26
14.78
I VISUALIZATION OF CROSS- ATTENTION MAP
Figure 12 illustrates the attention scores for all N - 6 layers with a single bit error (bit error in the first position). The first three layers have relatively high attention score the error position (first bit). Then, when the етrог is corrected, the attention score becomes lower at the last three layers.
For model-based neural decoders, interpretation and analysis terms of graphs are feasible be- cause their structural architecture is inherently based on conventional graph-based decoding al- gorithms (Wang et al., 2021; Ankireddy Kim 2023). However, in the case of model-free ap- proaches, remains challenging to determine how attention scores weights are assigned to spe- cific nodes and how these assignments are influenced by graph properties such as node degree.
Although we analyzed how the attention scores change depending where the error occurs in Figure 12] we have not yet achieved rigorous analysis beyond this level. This remains a critical issue in model-free approaches and represents problem that needs to be addressed in future work.
Layer
Layer?
Layer
Layer
Layer'
Layar 6
Figure 12: Attention scores of 6 layers with single bit error.
J TRAINING TIME TO ACHIEVE THE TARGET LOSS
In Table 7 we compare the training time required for ECCT and CrossMPT to achieve the target loss. The target loss is set minimum loss of ECCT during the training. For (128, 86) polar code, the minimum loss ECCT during the 1000 epochs is 2.28 x 10-2, 2 To achieve the loss 2.28 x 10-2 CrossMPT takes 6912 s, while ECCT requires 72917 Similarly, for (128, 64) CCSDS code,
17

---

Published as a conference paper at ICLR 2025
the minimum loss of ECCT is 2.79 10-2 and CrossMPT requires 2356 while ECCT requires 85770 s. These results demonstrate that CrossMPT achieves the target loss significantly faster than ECCT, highlighting its efficiency in terms of training time.
Table 7: Comparison of training time to achieve the target loss for ECCT and CrossMPT.
Codes
Methods
Target loss
Time
ECCT
2.28 10
72917
(128,86) Polar
CrossMPT
2.28 10
6912
ECCT
2.79 10-2
86770 s
(128,64) CCSDS
CrossMPT
2.79 x 10-2
2356
K THROUGHPUT ANALYSIS
Table@edemosstratess that CrossMPT outperforms ECCT in terms of inference time. However, while a fully parallel processor can accelerate ECCT, the sequential decoding architecture of CrossMPT limits its potential for throughput improvement. To address this, pipelining approach-commonly employed in various ECC decoders (Li et al.) 202Tbb Rowshan et al.) 2024 can be applied max- imize CrossMPT's decoding throughput (see Figure in (Li et al. 202Ib)). By unrolling two cross-attention blocks, CrossMPT can simultaneously process two consecutive codewords across two cross-ates attention blocks within the same layer. This means that while the second eross attentioo block processes the first codeword, the first cross-attention block concurrently decodes the subse- quent codeword. This pipelining strategy ensures that CrossMPT maintains speed advantages ECCT, in fully parallel scenarios. Figure 13 provides an example of decoding multiple codewords CrossMPT with 2.
In wireless communications, the decoder's throughput is often a more critical concern than latency. This is because the latency from communication protocols and signal processing in preceding re- ceiver blocks would be longer than the latency introduced by the channel decoder. Throughput becomes especially important when supporting very high data rates in wireless communication as the channel decoder can be a bottleneck.
Furthermore, wireless communication scenarios, sequential algorithm may be preferred for its enhanced performance or reduced complexity. The layered decoding algorithm for LDPC codes has been widely adopted as de facto standard (Bae et al.) 2019 Hailes et al., [015 Li al., (021a), despite its sequential nature and limitation parallelism, exemplifying the preference for sequen- tial algorithms. The layered decoding algorithm is favored for its superior decoding performance compared to fully parallel sum- product decoding equivalent computational complexities.
18

---

Published as a conference paper at ICLR 2025
C1-1
Decoded codewords: Co Ci-2
N
Second
decoder layer
Cross attss-atention Block 2
Cross attention Block
C++1
First decoder layer
Cross attention Block 2
C1+2
Cross-athention Block
C143
Ci44
CL
Figure 13: Example of decoding multiple codewords in CrossMPT with N = 2.
19


Reviews:
[{"review_id": "70JcV8eUbK", "reviewer": "Reviewer_caLP", "summary": "This paper introduces a novel transformer-based architecture called CrossMPT(Cross-attention Message-Passing Transformer) for decoding error-correcting codes(ECCs). The key innovation lies in the use of cross-attention mechanisms to separately and iteratively update magnitude and syndrome vectors, in contrast to conventional transformer-based decodes that employ self-attention without distinguishing between different types of input vectors. The mask matrices in CrossMPT are derived from the code’s parity-check matrix (PCM), explicitly capturing the relationships between magnitude and syndrome vectors. Results demonstrate that CrossMPT significantly outperforms existing neural network-based decoders across various code classes, while also reducing memory usage, computational complexity, inference time, and training time.", "questions": "1. FER Results: In [1], the authors present their results using the Frame Error Rate (FER) metric, while your results are reported using Bit Error Rate (BER). Could you please explain why you chose to report BER over FER in your evaluation? Are there specific challenges or reasons that prevent you from providing FER results for your approach? Including FER results would be helpful for assessing the practical decoding performance and for making a direct comparison with the results in [1].\n\n2. Training Convergence and Speedup: In Appendix I, authors mention that CrossMPT achieves faster training convergence compared to ECCT. However, the figure provided does not clearly demonstrate a significant speed up. Could you please provide specific data quantifying how much faster CrossMPT converges during training? For instance, you could report metrics such as: wall-clock training time taken by each model to achieve comparable performance levels and percentage improvement in convergence speed or reduction in epochs/time needed.\n\n3. Reproducibility: Do you plan to release the source code or detailed experimental records? Reproducibility is essential for validating the results and facilitating future research based on your work.\n\n[1] A Foundation Model for Error Correction Codes", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Innovative architecture: introduced the cross-attention mechanisms to separately handle magnitude and syndrome vectors is a notable advancement, bridging concepts from traditional message-passing algorithms and modern transformer architectures.\n2. Performance Gains Significantly: in the experiments show significant improvements to existing decoders across various code classes, including BCH, polar, LDPC, and turbo codes.\n3. Efficiency Improvements: By reducing memory usage and computational complexity, CrossMPT addresses practical concerns associated with deploying transformer-based decoders, especially for longer codes. \n4. Use of PCM-derived Masks: Aligning the mask matrices with the PCM directly incorporates code-specific structural information into the model, enhancing its ability to learn relevant relationships.", "weaknesses": "1. Availability of Code or Experimental Details: The paper does not mention whether the code or experimental records will be made available. Open-sourcing the code or providing detailed experimental records are crucial for reproducibility and for the community to gain more benefit from this work.\n\n2. Code length: In the appendix H of \"A Foundation Model for Error Correction Codes,\" the authors compare Maximum Likelihood decoders on the BCH(1023,1013) code. In this paper, authors highlight the significant reductions in memory usage. It would be valuable for the authors to extend this analysis by demonstrating performance on even longer codes or comparing their results against other decoding methods for these extended lengths. This could provide clearer insights into the advantages of their approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "22b3ZmFLMG", "reviewer": "Reviewer_VWDs", "summary": "This paper studies the neural network based decoding of conventional error correction codes. In essence, decoding of an error correction code is a classification problem. Conventional belief propagation based decoders rely on the code structure to iteratively improve their beliefs on the message bits. An alternative approach is to parameterize the decoder as a neural network, and train it on various codewords. Given the large number of codewords, it is challenging to design such a decoder without exploiting the code structure. An obvious approach is to use the code structure and the basic steps of the belief propagation decoder, but unroll the iterations on the layers of a neural network. This work follows the recently introduced ECCT decoder in this direction, and exploits a transformer architecture for the decoder. In particular, by transforming the decoding operation into the estimation of the sign of the multiplicative noise component, and by separately representing the magnitude of the received signal and its syndrome as two separate pieces of information, the authors train separate masked cross-attention blocks to update the embeddings of these two blocks before reaching the final decision. Through experiments, they show that the resultant decoder can achieve a lower bit error rate (BER) compared to ECCT in decoding a number of different channel codes, with a slightly lower computational complexity.", "questions": "- The performance measure presented in the paper is BER; however, what really matters for a channel code is the block error probability (BLER). Therefore, I would ask the authors to include BLER results to clearly show that the proposed approach does actually improve the performance with respect to ECCT or other baseline decoding algorithms. Improvements in BER are still important, but correct decoding of the message bits would still require the concatenation of the code with an outer code, which would further increase the decoding complexity, and would diminish the gains of the proposed approach. I believe, not presenting the BLER results at all is not fair, and does not show the real performance of the approach. \n\n- How does the complexity compare with the more conventional BP decoders and neural BP (NBP) decoders? Are the numerical results presented in Fig. 4 for the same complexity decoders, or for the same number of decoder iterations? If it's the latter, the comparison is not fair. If each iteration of the NBP decoder has half the complexity of the proposed approach, it can run twice the number of iterations, and potentially achieve lower BER. \n\n- Why is the double-masked ECCT not included in the comparisons?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The decoding of error correction codes is an important problem with significant potential impact in practice. Using neural network based decoders is a promising approach that has been receiving significant attention recently. The paper improves the state of the art transformer-based architecture with a simple yet effective enhancement in the architecture. Given that the magnitude and the syndrome provide different types of information on the transmitted symbols, it makes sense to process them separately, and learn different cross-attention modules. Despite the fact that the architectural modification is simple, the improvement in BER performance is significant. The authors also provide a good explanation of their motivation and the impact of their architecture on the decoding process (e.g., attention scores). Overall, it is a well-written paper with clearly highlighted contributions.", "weaknesses": "The architectural modification proposed in the paper with respect to ECCT is rather trivial. In that sense, it does not have significant technical novelty, on the other hand, given its impact on the performance, I believe it is a worthy contribution. \n\nThe paper presents only BER performance results. It is not clear if the proposed approach actually provides any gains in terms of the block error probability, which would really matter for the code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vDCCWKcsBA", "reviewer": "Reviewer_kBMv", "summary": "The paper proposes cross-attention-based message-passing transformers (CrossMPT), which solve the ambiguous masking problem of the naïve self-attention-based approach (ECCT). The magnitude and syndrome information are processed using two cross-attention layers, similar to message-passing algorithms, incorporating a mask defined by the parity check matrix. The proposed scheme achieves superior performance on various codes and SNRs consistently over ECCT and other baselines. The cross-attention map provides some degree of interpretability, while cross-attention itself enjoys reduced complexity compared to full self-attention, enabling the processing of long codes.", "questions": "1. Why was the magnitude chosen instead of another quantity, such as LLR? What is the intuition behind using the magnitude?\n2. What happens if the syndrome information is placed as the query in the first attention layer? Does changing the order make any difference?\n3. Is the inference time in Table 2 evaluated on the GPU?\n4. What is the peak memory requirement? \n5. Was the inference runtime evaluation conducted after code compilation (e.g., using `torch.compile()`)?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The update in the attention layer built upon ECCT is well-motivated and shows meaningful performance improvement. Although cross-attention is a well-known technique in the community, it is placed at the right position to improve the performance and efficiency of the previous self-attention-based approach.\n2. The paper is easy to follow and provides sufficient details.\n3. The experimental results effectively support the claims.\n4. The in-depth analysis of the proposed method helps readers understand the paper better.", "weaknesses": "Although I think the paper is strong, I found a couple of minor weaknesses.\n\n1. The actual inference time reduction from ECCT is not as great as the FLOPs improvement. Also, runtime evaluation and complexity analysis need more details about the environment. See Questions 3-5.\n2. Related works on cross attention in other domains are missing, e.g., language (Vaswani et al., 2017), vision (Chen et al., 2021), text-based image generation (Rombach et al., 2022), etc.\n\n(Vaswani et al., 2017) Vaswani, A. \"Attention is all you need.\" *Advances in Neural Information Processing Systems* (2017).\n\n(Chen et al., 2021) Chen, Chun-Fu Richard, Quanfu Fan, and Rameswar Panda. \"Crossvit: Cross-attention multi-scale vision transformer for image classification.\" *Proceedings of the IEEE/CVF international conference on computer vision*. 2021.\n\n(Rombach et al., 2022) Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZZMSB2xIig", "reviewer": "Reviewer_UA4Y", "summary": "The paper proposes a new cross-attention based decoding schemes for linear block codes using transformer architecture. The masking and the cross attention mechanism depend on the parity check matrix of the block code, thus introducing the domain knowledge about the code into the decoding architecture. This work seems to be built on Error Correction Code Trabsformer (ECCT), NeurIPS 2022.", "questions": "Included in the weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The idea of treating syndrome and magnitude as two separate entities and using cross-attention to capture the relation is novel and interesting. From a error correction point of view, this is more intuitive and understandable than existing approches. \n\n2. The performance improvement over ECCT is compelling and non-trivial, while maintaining similar or lower complexity.\n\n3. Overall well written and self-contained. Easy to understand and follow the ideas, even for reader unfamiliar with prior work.", "weaknesses": "1. One common troubling trend in comparing the performance of neural decoders is the choice of weak classical baselines. While I understand the difficulty of a common decoder architecture outperforming the highly specialised decoders for each of the class of codes, the comparison should neverthless be done for completeness. For instance, in Fig.4, only BP classical baseline is considered. It is known that BP is not optimal for many codes. For instance, when comparing with BCH codes, Berlekamp-Massey decoding algorithm should be used as the classical non-learning baseline and similarly for Polar codes, successive cancellation list decoding should be used. I strongly suggest authors to include the best classical baselines for each code to clearly show the gap with respect to the SOTA neural decoder, whether it is positive or negative.\n\n2. I beleive authors chose the format of the negative log BER for presenting the results based on the original ECCT paper. But this format is not very informative and very uncommon in information and coding theory literature. While it is easy to identify which scheme is doing better, the gap between the performances is hard to interpret as the scale is not very intuitive. Rather, the standard format of SNR gap in dB for a target BER/BLER (ex. BLER target of $10^{-5}$) is more informative. \n\n3. While the idea and results are interesting, the changes in the algorithm/architecture are only incremental compared to the original ECCT paper, which limits the novelty to some extent.\n\n4. As with the case of other model-free neural decoders, there is a lack of interpretation for the neural decoder presented. Since the algorithmic contributions are limited, I would suggest atleast adding a discussion on interpretation and analysis, similar to the analysis and interpretation of neural belief propagation codes. A couple of references I can think of are: [1], [2]\n\n[1] \"Neural- network-optimized degree-specific weights for ldpc minsum decoding\", Wang et. al. ISTC 2021. \n\n[2] \"Interpreting Neural Min-Sum Decoders\", Ankireddy et. al, ICC 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0H5X99YcDV", "reviewer": "Reviewer_HjdF", "summary": "The paper presents an extension of the existing ECCT by splitting, as with classical message-passing decoding (BP), the self-attention module between variable and check nodes. This is done similarly to classical Transformers decoders via the additional cross-attention while the original ECCT has only self-attention layers.", "questions": "1 - The authors should integrate clearer information regarding the complexity as explained in the Weakness section.\n\n2 - There is a large gap remaining with SCL decoding. The authors should explain and investigate why the (cross) ECCT and its variants struggle with it.\n\n3 - The authors should provide comparisons between the methods for various dimensions/layers, as done with the ECCT and its extensions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and clear.\n\nThe paper is well-presented and has extensive ablation studies.\n \nThe method is interesting and sounds natural given classical message-passing decoders.\n\nThe method improves over the original ECCT and allows more memory efficient training on GPUs.", "weaknesses": "I'm concerned about the complexity analysis and thus the claim of superiority of the method over ECCT.\n\nCross-MPT adopts a **sequential** processing approach that appears **after** the self-attention, which is the only module in ECCT.\nThis sequential processing (1 self-attention + 1 cross-attention) is then necessarily slower than ECCT's single parallel processing (1 self-attention only) on modern parallel computing HW.\n\nIt obviously allows better decoding performance, making the comparison unfair, even if the parameters are shared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)