Efficient Message-Passing Transformer for Error Correcting Codes
Authors: Seong-Joon Park
Taewoo Park
Hee-Youl Kwak
Sang-Hyo Kim
Yongjune Kim
Jong-Seon No
Venue: ICLR 2026 Poster (2026)
Keywords: Channel coding
Error correcting codes
Transformer-based decoder
Message-passing decoder
Neural decoder
Transformer
Efficient attention module
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a novel efficient message-passing decoder for error correcting codes based on the proposed efficient error correcting attention module.

Abstract: Error correcting codes (ECCs) are a fundamental technique for ensuring reliable communication over noisy channels. Recent advances in deep learning have enabled transformer-based decoders to achieve state-of-the-art performance on short codes; however, their computational complexity remains significantly higher than that of classical decoders due to the attention mechanism. To address this challenge, we propose EfficientMPT, an efficient message-passing transformer that significantly reduces computational complexity while preserving decoding performance. A key feature of EfficientMPT is the Efficient Error Correcting (EEC) attention mechanism, which replaces expensive matrix multiplications with lightweight vector-based element-wise operations. Unlike standard attention, EEC attention relies only on query-key interaction using global query vector, efficiently encode global contextual information for ECC decoding. Furthermore, EfficientMPT can serve as a foundation model, capable of decoding various code classes and long codes by fine-tuning. In particular, EfficientMPT achieves 85% and 91% of significant memory reduction and 47% and 57% of FLOPs reduction compared to ECCT for $(648,540)$ and $(1056,880)$ standard LDPC code, respectively.

Published as a conference paper at ICLR 2026
EFFICIENT MESSAGE-PASSING TRANSFORMER FOR ERROR CORRECTING CODES
Seong-Joon Park Taewoo Park! Hee-Youl Kwak
Sang-Hyo Kim² Yongjune Kim Jong-Seon No
'PSTTECH, PUniversity of Ulsan, *Sungkyunkwan University, "Scoul National University joonpark22476 gma com, parktaewoo@posteree ac kr, ghy1228 gma com, iamshkime skku edu, yong jongjjneeposteeee ac. kr, jsno@snu.ac.
ABSTRACT
Error correcting codes (ECCs) are fundamental technique for ensuring reli- able communication over noisy channels. Recent advances in deep learning have enabled transformer-based decoders to achieve state-of-the-ar performance on short codes; however, their computational complexity remains significantly higher than that of classical decoders due to the attention mechanism. To ad- dress this challenge, we propose EfficientMPT, an efficient former that significantly reduces computational complexity while preserving de- coding performance. key feature of EfficientMPP is the Efficcent Error Correct- ing (EEC) attention mechanism, which replaces expensive matrix multiplications with lightweight vector-base dlement-wise operations. Unlike standard attention, EEC attention relies only query-key interaction using global query vector, ef- ficiently encode global contextual information for ECC decoding. Furthermore, EfficientMPT can serve foundation model, capable of decoding various code classes and long codes by fine-tuning. In particular, EfficientMPT achieves 85% and 91 % of signifieant memory reduction and 47% and 57% of FLOPs reduction compared to ECCT for (648, 540) and (1056, 880) standard LDPC codes, respec- tively.
cesssage-passingg trans-
INTRODUCTION
In modern digital communication. reliable data transmission over noisy channels is a primary ob- jective. A fundamental approach achieving this objective is the careful design of error correcting codes (ECCs) to correct noisy errors caused by channel impairments. ECC decoders have benefited from advances in deep learning, achieving improvements over conventional decoding algorithms for various code classes. Among them, neural network-based decoders using the transformer archi- tecture (Vaswani et al. 2017) have achieved state-of-the-arr decoding performance for short-length codes, since the transformer is one of the most powerful neural network structures (Choukroun & Wolf 2022]] 2023; Park et al. 20233 Choukroun & Wolf 2024a ()) The first transformer based decoder. known the Error Correction Code Transformer (ECCT) (Choukroun & Wolf 2022). employs masked self attention module. Recently, a more efficient transformer-based decoder, the Cross-Attention Message-Passing Transformer (CrossMPT) (Park et al.,, al., (225)) has been introduced, utilizing a masked cross-attention module, which is more effective than self-attention for error cor- rection.
However, transformer-based decoders for ECCs face significant challenges due to the high com- putational complexity of the attention module (Choukroun & Wolf 2022; 2023 Park et al. 2023 Choukroun & Wolf 2024a b). Specifically, the attention module exhibits a quadratic complexity of Onn)) /Vaswani al. 2001 Chang al. (202)), where n denotes the number of tokens the code length in transformer- basedd decoders. This excessive complexity increases memory usage and com- putational complexity restricts the practical application of transformer-be based decoders. Designing an efficient attention module is critical for reducing the decoding complexity of transformer-based
TThe source code is available at lttptt:///// httpt/gtttt////////////te com/ iil wwwpppp//tfttieetttteeepe
Corresponding authors

---

Published as a conference paper at ICLR 2026
ECC decoders and allowing longer codes to benefit from the advantages of transformer- based de- coders.
In this work, propose a transformer-based decoder called Efficient Message-Pussing Trans- former (EfficientMPT)) EfficientMPT significantly reduces the number of parameters, memory us- age, and computational complexity (i.e., floating point operations (FLOPs)) while maintaining supe- rior decoding performance. Notably, these reductions become more pronounced as the code length increases, enabling our model to effectively decode long codes.
EfficientMPT consists of two types of EfficientMPT blocks, which iteratively update magnitude and syndrome embeddings, similar to peessage-passing algorithms. A key feature of EfficientMPT block is the Efficient Error-Cor CorrecCore (EEC) attention, which relies only on query-key interaction using the parity-check matrix (PCM). During the query-key interaction, global query vector is generated to effectively capture global contextual information across all syndrome and magnitude elements. This global query vector aggregates information from all positions of the syndrome and magnitude embeddings, providing a condensed representation of their overall structure. Leveraging the global query vector, EEC attention replaces the costly matrix multiplications typically used in standard attention mechanisms. Instead, it employs efficient element-wise vector-based operations, signifi- cantly reducing memory usage and computational complexity. Combined with this approach, EEC attention integrates the PCM into the attention module, enabling the model to embed the code struc- ture. EfficientMPT updates magnitude and syndrome embeddings by simply adding the embedding with the attention output without any other complicated operations. This integration facilitates the training efficiency of EfficientMPT.
Furthermore, EfficientMPT features bit position-invariant and code length-invariant architecture, making it a foundation model for ECC decoding. EfficientMPT, trained across several codes simulta- neously, achieves notable decoding performance on trained codes. For unseen codes, the fine-tuning technique enables EfficientMPT to decode without the need to train the decoder from scratch.
Our EfficientMPT significantly reduces i) GPU memory usage, ii) FLOPs, and iii) the number of parameters while showing the state-of-the-art decoding performance for transformer-based de- coders. The complexity reduction scales favorably with increasing code length. We believe that this represents an important step toward enabling efficient decoding for a wide range of lengths with transformer-based decoders.
2 RELATED WORK
Transformer- based decoders are a subelass of model-free neural decoders. Model-free neural de- coders adopt general neural network anchitectrrs-fflly-coneeeee
network (Gruber al., 2017 Kim et al.) 2018). recurrent neural network (Bennatan et al. ) for ECC decoding, without re- lying existing decoding algorithms (Dai 2021 Buchberger et al. [021] Kwak al. (023)) Model-free decoders using transformer architectures are referred to as transformer-based decoders.
The first transformer-basee decoder, ECCT (Choukroun & Wolf, 2022). employs masked self- attention module to enhance training efficiency and decoding performance. The mask matrix, de- rived from a PCM, designed embed the code structure and capture relationships between bit positions. Building on this approach, several transformer-base decoders have been proposed. An extension of ECCT with multiple masks (Park al. 2023) was introduced to improve its decoding performance by increasing the diversity of the mask matrices. Furthermore, foundation model for ECCT (FECCT) Choukroun & Wolf (2024a) was proposed, enabling the decoding of various code classes by training single model. This approach demonstrates that transformer- decoders can indeed function as foundation decoders. Also, an end-to-end learning framework for transformer- based decoding was proposed (Choukroun & Wolf 2024b).
However, all these methods use the concatenation ÿ. which combines the magnitude ly| and syn- drome syy) of the received vector y (i.e., y ||,] s(y)) )) for the masked self-attention module. This results in a large attention map with high computational complexity and memory usage. The atten- tion map of the original ECCT [Choukroun & Wolf 2022) is sparse, the mask matrix discards unrelated positions entirely. However, the approaches in (Choukroun & Wolf 2024a b) introduce dense matrix that is added element-wise the attention map. Instead of enforcing sparsity by
2

---

Published as a conference paper at ICLR 2026
masking out connections, this matrix applies a continuous weighting to the attention scores, thereby leading to dense attention maps.
In contrast, CrossMPT (Park et al. 2025) employs masked cross-attention module and processes magnitude lyl and syndrome s(y) separately, as they exhibit distinct characteristics. CrossMPT iteratively updates their embeddings through two cross-attention modules, which improves decoding performance. This approach also reduces decoding complexity by decreasing the attention map size and increasing the sparsity of the mask matrix. Similarly, the proposed BBC attention module processes the magnitude and syndrome iteratively, with a further simplified attention mechanism. As result, it significantly reduces the number of parameters, memory usage, and FLOPs, without compromising the decoding performance achieved by CrossMPT.
3 BACKGROUND
3.1 ERROR CORRECTING CODES
Consider a linear block code C, defined by a generator matrix E F and a PCM H e over {0, with modulo 2 addition. A codeword I E C S [0.1]" is encoded by multiplying the message m of size l with G (i.e., x = mG). Le be the binary phase shift keying (BPSK) modulated signal of x, where Vg be the noisy channel output when 2, is transmitted. In the additive white Gaussian noise (AWGN) N(00a²)) The decoder's objective is to
2 fn=n)) where they satisfy GH
2x, and let y
channel, this can be modeled as y 2s + z, where recover the original codeword x by correcting errors caused by noise. Upon receiving the decoder computes the syndrome s(y) Hy.. where yo binisig(yy). Here, sign(a) returns +1 if and -1 otherwise, while bin( (1) = 1 and bin(+1) 0.
3.2 TRANSFORMER BASED DECODERS
The transformer-based decoders employ syndrome-based preprocessing method to address the overfitting problem (Bennatan et al. 2018 Choukroun & Woll 2022). In this preprocess- ing step. decoders generate the magnitude vector 1ı] = (Ivil...., (or and the syndrome vector s(y) (s(y)1.-. 1..... nn)) from the received vector y. which are then used as input to the trans- former. The magnitude and syndrome vectors are used to estimate the multiplicative noise Σ, de- fined as: y = Fs + = xr E. The goal is to estimate y from the received vector y. The decoder function f outputs 2g. and the estimated codeword & is computed as a bin (y))( If the multiplicative noise is accurately estimated, then sign (2.) = sign(2.) and sign (2,8,)
All prior transformer-basee decoders incorporate mask matrix within the attention mechanism to effectively learn relationships among codeword bits (Choukroun & Wolf, 2022 Park et al.) 2023 Choukroun & Wolf, 2024a bl Park et al. 2025). The decoding performance depends on the choice of the mask matrix since it discards less important relationships and focuses on critical ones to facilitate learning. The mask matrix is derived from the PCM H, which explicitly defines direct relationships between codeword bits based on parity check equations.
3.3 ATTENTION MODULE
The attention module can be described by three components: Query (Q), key (K), and value (V). Let X,X' Raxd be the input embeddings, where is the input vector size and d is the embedding dimension. The input embedding X is projected to the query using the trainable weight matrix Wq E Rdxd by = XWQ and the input embedding X' is projected to the key and value using the trainable weight matrices WK, Wy Rddd by K = X'WK, and V X X'Wv. v. To enable multi-head attention, Q K, and V are split into h heads, where each head has reduced dimensionality d, d/h such that Q [Q...... Q*), K [K],... K*], V [V'.... ...,, J*J. where [ denotes concatenation. Finally, the attention output Y can be computed as: Y = [Attn(Q', K! ..t....,Attff Q", K v*]] Wo, where Attn Q K, V) = softmax (QK"/v/ /Vda) V and Wo E Raddd denotes the output weight matrix. This mechanism is also known as scaled dot-producc attention. If X = X', the operation is referred to as self self-attention; if X o X', it is called cross-attention.
3

---

Published as a conference paper at ICLR 2026
Matrix multiliication
@
Element
wise addition @ Broadcasted element-wise multipiiation
Emtee
N
C
Feed Formerd
S
Layer Norm
PAS
Lave Nom
S
EBC attention
Linear
Layer/oom Norm
Layer
2
M
-ch AM
M'
Layer Norm
Embed
Feed Forward
-0 M
Figure 1: Architecture of EfficientMPT.
EFFICIENTMPT
In this section, we introduce novel transformer-based decoder called EfficientMPT. Our proposed EfficientMPT (presented in Figure incorporating EEC attention module (presented in Figure 2(c) has the following three key distinctive characteristics.
Query-key interaction with global context. The EEC attention module focuses solely query- key interaction with global query vector. The global query vector encapsulates comprehensive contextual information from all syndrome magnitude elements. This approach eliminates the use of the value vector V, simplifying the process and reducing computational complexity.
Efficient vector-based attention. Standard attention mechanisms exhibit quadratic complexity Onn²) due to the matrix multiplication involved. In contrast, EEC attention generates a global query vector through row-wise summation and combines it with the key matrix using broadcasted element-wise multiplication. This design eliminates the quadratic matrix multiplication, resulting in linear-time complexity (n) for the attention operation. Consequently, this approach significantly reduces both memory usage and computational complexity.
ECC specialized attention with PCM. In the architecture of EfficientMPT, incorporate the PCM to interchangeably convert between the magnitude and syndrome domains. This effectively embeds the code structure into the transformer, as the PCM defines the constraints that all valid code- words must satisfy. This approach is distinct from standard attention mechanisms, where the PCM indirectly used to construct the mask matrix, whereas EfficientMPPT utilizes the PCM directly. In other words, we develop new attention mechanism specialized for ECC decoding.
Efficient model for foundation ECC decoder. The entire process in EfficientMPT is invariant to bit positions and code lengths, sharing the parameters across various code classes. Trained on multiple codes simultaneously, a single EfficientMPT model can achieve superior decoding performance across several code classes and even generalize to unseen codes with minimal fine-tuning.
4.1 ARCHITECTURE OF EFFICIENTMPT
Figure T presents the architecture of the proposed EfficientMPT. We first generate the magnitude embedding M = [M :M: M,, E guxd where M, = ly, WM for i I...., n. Similarly, we generate the syndrome embedding S = [S..... - S Sn_*] E ( where S; = sy)) for k. Here, WM R Rlxd and Ws E R 1xd are trainable parameters. To establish Effi- cientMPT as position-inveee invariant and the shared Wy for magnitude embeddings and the shared Wg for syndrome embeddings (Choukroun
=
foundation model for ECC decoding, we use
& Wolf 2024a).
Two gray boxed blocks in Figure 1 are referred to EfficientMPT blocks. The first EfficientMPT block, located on the left, contains an ECC attention module that updates the magnitude embedding M to M' using the syndrome embedding S. The query Ql and key K are computed as follows:
Q1 SW@ K, SWW
4

---

Published as a conference paper at ICLR 2026
Broadcasted element-wise multiplication
Σ
Row-wise sum
Matrix multiplication
Magnitude: M
Syndrome: S E (n k) x
L Linear
Linear
Linear
Linear
Linear
d
Linear
Linear
Linear
Q
V
Jofmmax
G lobal
query
2m
s0ftmax
softmax
Linear
Lineer
S AS
(2n k) x
(i k)
(n 14-A6 k)
(a) Masked self-attention
(b) Masked cross-attention
(c) EEC attention
Figure 2: Comparison of attention modules for transformer-based ECC decoders. (a) Masked self-
attention module (Choukroun & Wolf 2022), (b) masked cross-attention module (Park et al. 2025
and (c) proposed EEC attention module.
By applying the proposed EEC attention, which will be explained in the next subsection, obtain the attention output AM, which is added to the magnitude embedding M. The result then passed through normalization layer, followed by feedforward layer with residual connection, obtain the updated magnitude embedding M'. Note that the query and key are derived from the same embedding type, which differs from the configuration used in CrossMPT.
The second EfficientMPP block updates the syndrome embedding using the updated magnitude embedding M'. The query Q2 and key K2 computed follows:
Q2 = M'Wo, = M'WK.
Similar to updating the magnitude embedding, adding the attention output AS to the syndrome embedding S efficiently produces the updated S' which is then utilized to refine the magnitude embedding in the next EfficientPPT block. This process is repeated N times.
Finally, output embeddings (magnitude and syndrome embeddings) from the last EfficientMPT block pass through a normalization layer. The magnitude embedding is then added to the resized d by multiplying the transpose
syndrome embedding, which resized from (n k) x d of the binary PCM H. The combined output embedding then passes through a fully connected layer which reduces the n embedding to one dimensional vector. We want note that all trainable parameters in EfficientMPT are position-invariant and length-invariand which indicates that EfficientMPT can be utilized as a foundation decoder for ECCs.
4.2 EFFICIENT ERROR CORRECTING (EEC) ATTENTION
Figure illustrates three different attention modules employed in cunsformer-based decoders. As
shown in Figure 2(a) the masked self-attention module used in (Choukroun Wolf 2022. Park
et al. 2023 concatenates magnitude and syndrome embeddings as input, resulting in size of (2n
k) x d. Thus, the attention- map has a size of (2n A) x (2n - k) for each header. Figure 2(b)
shows the masked cross-attention module employed in CrossMPT (Park et al.] 2025) for updating the syndrome embedding. As previously noted, CrossMPT utilizes two distinct masked cross-attention modules to separately update magnitude and syndrome embeddings. Each cross-atte coss-attention has an attention map of size (n k) x n. Since CrossMPT employs two cross-attention modules, the total size of the attention maps is 2n(n k).
5

---

Published as a conference paper at ICLR 2026
Figure 2 shows the proposed ECC attention module used to update the syndrome embedding, which is located in the second EfficientMPT block in Figure [ Unlike standard attention mech- anisms that rely computationally expensive matrix multiplications, the ECC attention module employs row-wise summation and broadcasted element-wise multiplication as shown in Figure This design choice is a key factor in reducing computational complexity.
Without loss of generality, we describe the EEC attention module that updates the syndrome embed- ding using information from the magnitude embedding. For the other attention module that updates the magnitude embedding, the roles are simply reversed, with the syndrome embedding used to up- date the magnitude embedding. The magnitude embedding M Rexd is projected to the query Q, key K by weight matrices WQ. Wk E Rdxd, respectively. Next, Q and K are split into attention heads, generating Q K' E Raxdn for i = h.
For the query, we compute the global query vector as follows:
n Lloball softmax Σ [
ERIxdd
where Q'() denotes the jthh row vector of Q' (i.e. Q = Q vector is a critical component of the EEC attention mechanism, as it efficiently captures the global contextual information of the magnitude domain. By condensing the magnitude information across all elements into single vector, the glohal query vector acts high-level representation that can be applied uniformly across the syndrome domain, allowing the model to propagate magnitude information highly efficient manner.
This global query
The next step is to project the magnitude information into the syn- drome domain using the PCM H. Specifically, the key matrix K' E R"xdy is transformed into Ki HK' ((n-k)xd,, This transformation is essential because the PCM H inherently encodes the code structure, representing the relationships between the mag- nitude and syndrome elements. By applying H. the model effec- tively maps the magnitude information into a representation within the syndrome domain. This allows the global query vector, which contains magnitude information, to be distributed across a matrix the syndrome space, enhancing its effectiveness for error correction.
Σ
(a)
(b)
Figure 3: (a) Row-wise sum-
mation and (b) broadcasted element-wise multiplication.
The global query vector Gotobal is then broadcasted and element- wise multiplied with Ki
AS Aglobal & - Liiobal ① KA Wo ((nn-)xd
where @ denotes the broadcasted element-wise multiplication. This operation allows the global context information of the magnitude, captured in the global query vector, to be propagated across all rows of the matrix Ki in the syndrome domain. Essentially, the global query vector broadcasts this global context information directly into the syndrome space, enabling efficient context sharing.
The output of EEC attention AS represents the update for the syndrome embedding, learned from the magnitude embedding. This update is applied to the original syndrome embedding through simple addition: S + S + ÁS. Then, the updated syndrome embedding pass through the normalization layer and the fully-connected layer to obtain the updated syndrome embedding. For the second attention module, the same process applied to the magnitude embedding.
EXPERIMENTAL RESULTS
We adopt the same training setup as ECCT and CrossMPT: 1000 epochs, 1000 minibatches per epoch, and 128 samples per minibatch, using the Adam optimizer (Kingma& & Bal 2014). The learn- ing rate is initialized at 10 and gradually decreases to x 10 using a cosine decay scheduler. The model is trained on all-zero codewords over E from 3 dB to 7 dB, and tested on randomly generated codewords. All simulations are performed on NVIDIA GeForce RTX A5000 GPUs and AMD EPYC 7763 CPU.

---

Published as a conference paper at ICLR 2026
10
10
BER
124
ECCT CrossMPP
ECCT
ECCT FRioerkPPP
GresMPT
CressMPT
1% (dR)
(a) (31,16) BCH
(b) (121,70) LDPC
(c) (128,64) polar
10
10
10
2-21-20
10-5
-B-BP 20
BP 50
-0-EP20
ETCRNEMPT
4.5
E (0E)
ENN (2B)
(d) (576,432) WiMAX LDPC (c) (648,540) IEEE 802. 1nn LDPC
(f) (1056,880) WiMAX LDPC
Figure 4: BER comparison for different code classes.
5.1 DECODING PERFORMANCE
Figures 4(a) 4(b) and 4(c) compare the BER performance of EfficientMPPT CrossMPT, and ECCT for short codes. All simulations are conducted with h = 8, N = 6, and d = 128. Across dif- ferent code classes, EfficientMPT outperforms the original ECCT and achieves decoding perfor- mance comparable to CrossMPT. Additional results for various code parameters are provided in Appendix A
4(e) and 4(f compare the BER performance of three standard LDPC codes: (576, 432) Figures 4(d)
WiMAX. (648, 540) IEEE 802. In, and (1056, 880) WiMAX LDPC codes. All simulations are per- formed with / = 8, N = 10, and d = 128. Additionally, we include the performance of the belief propagation (BP) decoder. Both CrossMPT and EfficientMPT outperform the BP decoder with maximum iterations of 20 and 50. We note that ECCT for all these long codes and CrossMPT for (1056, 880) LDPC codes could not trained due to memory limitations our simulation envi- ronment. However, even for lengths exceeding 1000, EfficientMPT remains trainable and achieves performance gains over the BP decoder. More results are presented in Appendix A B C and
5.2 COMPLEXITY ANALYSIS
caopre[s GPU memory usage, computational complexity (FLOPs), and the number of train-
able parameters across EfficientMPT, CrossMPT, and ECCT for various code classes. All results are obtained under the condition N 6 and d 128. For GPU memory usage, we measure the peak memory usage during the training of a single batch.
The table clearly demonstrates the GPU memory efficiency of EfficientMPT, which consistently requires less GPU memory across all code types. As the code length increases, the memory us- age gap between EfficienMPPT and other methods becomes more pronounced. For instance, for the (3328.640) 5G NR LDPC code, EfficientMPT requires only 0.35 GB, whereas CrossMPT and ECCT require 8.42 GB and 17.98 GB, respectively. In other words, CrossMPT and ECCT require nearly 20x and 50x more memory than EfficienLMPT. This substantial improvement is attributed the proposed EEC attention module in EfficientMPT, which performs vector-based element-wise operations instead of matrix multiplications on large attention maps, such as (2n k)a for ECCT and 2n(n k) for CrossMPT.
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of GPU memory usage, FLOPS, and the number of parameters between Effi-
cientMPT, CrossMPT, and ECCT.
Memory usage
FLOPs
# of parameters
Cides
Parmeeee
EMliienMPP
CrosMPT
BCCT
EiiieieMMP
CresMPP
ECCT
EllicceMPP
CrosMPT
ECCT
WIMAX LDPC
(576, M32)
(16%)
(42%)
(100%)
GR
B.92
(56%)
L11G
(67%)
(1008E)
1,65
(64%)
(1008)
1.70M
802. LDPC
(648,640)
0.05 GB
(15%)
0.13 GB
(38%)
0.34 GB
(1005)
0.94 G
(53%)
L11G
(625)
(100%)
1.78
1.09 M
(61%)
(1005)
1.78 M
(1056,880)
(9%)
0.26
(32%)
(1005)
GB
L65
(545)
(10055)
3.80
(41%)
266
(100%)
WIMAX LDPC
(2304, 1152)
0.18 GB
(3%)
2.63 GB
(441))
6.02 GB
(100%)
8.18 G
(36%)
G
(551%)
22.46
(1005)
1.09 M
(11%)
(100%)
9.50 M
5G NR LDPC
(3328,
0.31 GB
(2%)
8.42 GH
17.98 GB
(100%)
21.44
(34%)
34,65
(5586)
62.76 G
(100%6)
1.09 M
(5%)
21.98 M
(100%)
(31,16)
56
57.
(94%)
(998)
(1005)
(88%)
(978)
(1005)
(915)
(1005)
BCH
(63,46)
36.19 MBB
(92%)
38.54 MB
(98%))
59.52 MB
(100%)
90.2 M
(85%)
99.9 M
(94%)
(100%))
1064M
1,09 M
(90%)
(100%)
1.21 M
Polar
(64,32)
36,19 MB
(900))
38,76 MB
40.05 MB
(100%)
0.11 G
(15R)
0.12
(100%)
0.13
1,09 M
(90%)
121 M (1008)
Polar
[128, 64)
(76%)
MR (85%)
(100%)
(79%)
6.22
0
(89%)
(1008)
0.28 G
1,09
(90%)
(100%)
1:21
LDPC
(121, 20)
36.21 MIB
40.07 MB
44.43 MB
0,20 G
0.23
0.26 G
1.0 M
1.23 M
(82%)
(905)
(1005)
(77%)
(88%)
(100%)
(89%)
(1008)
Furthermore, EfficientMPT effectively reduces FLOPs compared to other methods, with the reduction becom- ing more pronounced as the code length increases. For the (3328, 640) 5G NR LDPC code, Efficien(MPT re- quires only 21.48 G, whereas CrossMPT and ECCT re- quire 34.65 G and 62.76 G, respectively-reprrsentie 38% reduction.
ECOT GreestPT
PiidersPPP
1000
3000
To further investigate the effectiveness of EfficientMPT, we additionally compare FLOOSs in a graph, supple- menting the results in Table the FLOPs of 5G NR LDPC codes based on a base
lengt
Figure 5: Comparison of FLOPs for Ef- ficientMPT, CrossMPT, and ECCT for various SG NR LDPC codes.
Figure 5 presents
graph of size 10 52 with various lifting factors Z
1,2,4, 8, 16, 32, 64 (Richardson & Kudekar 2018). The lifting process generates LDPC codes with code length = 52 x Z and dimension = 10 x Z. Since all 5G LDPC codes in the figure are derived from the same base graph, they share the same code structure regardless of the lifting factor. This setup ensures fair comparison, focusing on the impact of code length while preserving the code structure. As shown the figure, EfficientMPT exhibits nearly linear FLOPs, unlike other methods. This is because the only operation with quadratic complexity in Figure 2(c) is the multiplication with PCM, which is not a dominant computation compared to other operations.
The last column in Table presents the number of trainable parameters. Note that CrossMPT and ECCT have the same number of parameters. Since all parameters in EfficientMPT are code- invariant, the number of parameters remains constant across different code classes as long as N and and = 128, EfficientMPT consistently maintains 1.09 M param-
are the same. For N eters (specifically, 1,097,649), whereas the number of parameters in CrossMPT and ECCT grows rapidly with increasing code length. These results demonstrate that EfficientMPT achieves superior efficiency in all aspects of computational complexity compared to CrossMPT and ECCT.
5.3 FOUNDATION EFFICIENTMPT
To train EfficientMPT as a foundation ECC decoder, we train EfficientMPT on four different codes: (64, 32) LDPC, (121, 60) LDPC, (121, 70) LDPC, and (121, 80) LDPC codes. We refer to this model as foundation EfficientMPT (FEfficientMPTT) In Figure 6 we compare FEfficientMPT with EfficientMPT, ECCT, and foundation ECCT (FECCT) (Choukroun & Wolf 2024a). The decoders trained on a single code (EfficientMPT, ECCT) are trained for 1000 epochs, whereas the foundation models (FEfficientPPT, FECCT) trained for 4000 epochs since they are trained four differ-
8

---

Published as a conference paper at ICLR 2026
10
10
10-0
1240
ECCT -AFFRCerRETTO
ECCT
PEOOT
25
(OR)
EN (68)
(a) (121,70) LDPC
(b) (204,102) MacKay LDPC
(c) (1920,1600) WIMAX LDPC
Figure 6: Decoding performance of (a) (121,70) LDPC code, (b) (204,102) MacKay LDPC code, and (c) (1920,1600) WiMAX LDPC code.
ent codes. Figure 6(a) shows the performance for the (121, 70) LDPC code, which is one of the trained codes, while Figures 6(b) and 6(c) present the performance for the (204, 102) MacKay and (1920, 1600) WiMAX LDPC codes, which are unseen codes. The number following FEfficientMPT and FECCT denotes the number of fine-tuning epochs. For example, 'FEfficientMPT-O" denotes foundation EfficientMPT without fine- fine-tuning and "FEfficientPPT-300 denotes the foundation Effi- cientMPT fine-tuned for 300 epochs.
In Figure 6(a) EfficientMPP and FEfficientMPP exhibit nearly identical performance, which means that the EfficientMPT model can be trained as a foundation ECC decoder without performance degradation. Figure [6b] demonstrates that FEfficientMPT with fine-tuning (FEffficienMMPT-300)) outperforms ECCT and achieves decoding performance comparable to EfficientMPT, even for un- seen codes. Notably, Figure b(c) shows that although FEfficientMPT initially struggles to decode unseen codes, its performance steadily improves with fine-tuning and eventually surpasses the con- ventional BP decoder for the long WiMAX LDPC code. This result is practically meaningful be- cause it demonstrates that FEfficientMPT model can be adapted for longer codes using pretrained foundation model trained on short codes, eliminating the need for expensive full-scale training of long codes from scratch.
5.4 TRAINING MATRIX
In the EfficientMPT architecture, we utilize (or multiply) the PCM to embed the code struc- ture. For further analysis, we replace the PCM H with a trainable matrix. The trainable ma- trix is randomly initialized and optimized along with other parameters during training. Figure 7 shows the trainable matrix before and after the training the (64, 32) LDPC code. The darker the tone of each element, the larger its value. Initially, shown in Figure aa), the train- able matrix is random matrix. However, af- ter training, Figure ( reveals that the trained matrix develops a structure closely resembling the original PCM. In Appendix F comparison between the trained PCM and the original PCM. This observation further supports the use of the PCM in EfficientMPT.
(a) Initial random PCM
(h) Trained PCM
Figure 7: The values of trainable matrix (a) before the training and (b) after the training.
we illustrate a
CONCLUSION
In this work, we proposed EfficientMPT, a transformer- decoder that overcomes the compu- tational limitations of standard attention mechanisms in ECC decoding. EEC attention simplifies the standard attention by employing broadcasted element-wing wise operations and effectively learns the code structure by incorporating the PCM into the attention mechanism. Leveraging the proposed

---

Published as a conference paper at ICLR 2026
EEC attention, we develop EfficientMPT, which significantly reduces the computational complexity while maintaining the superior decoding performance of CrossMPT. In addition, EfficientMPT can also serve universal and foundation-level decoder. A single model achieve notable decod- ing performance across several code classes and even on unseen codes. The reduced computational complexity of EfficientMPT enables support for longer code lengths, allowing more practical codes to benefit from the advantages of transformer-basee decoders and enhancing the feasibility of their application long codes.
ACKNOWLEDGMENTS
This work was supported by Institute of Information Communications Technology Planning & Evaluation (ITTP) grant funded by the Korean Government (MSIT) (RS-2024-00398444, Net- work Research Center: Advanced Channel Coding and Channel Estimation Technologies for Wire- less Communication Evolution) and IITP grant funded by the Korean Govemment (MSIT) (RS- 2023-00229849, MMPCCM/eeCoeetttiitiiiiiiiy SoC Solution for IoT Intelligence with Foundry-Base eFLASH).
REFERENCES
A. Bennatan, Choukroun, and P. Kisilev. Deep learning for decoding of linear codes-a syndrome- based approach. In 2018 IEEE International Symposium on Information Theory (ISIT), pp. 1595- 1599. IEEE, 2018.
A. Buchberger, C. Hager, H. D. Pfister, L. Schmalen, and A. G. I. Amat. Pruning and quantizing neural belief propagation decoders. IEEE Journal of Selected Areas in Communications, 39(7): 1957-1966, 2021.
Chang. P. Wang. M. Lin, F. Wang, D. J. Zhang, R. Jin, and M. Z. Shou. Making vision transform- efficient from token sparsification view. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Y. Choukroun and L. Wolf. Error correction code transformer. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Y. Choukroun and L. Wolf. Denoising diffusion error correction codes. In International Conference on Learning Representations (ICLR), 2023.
Y. Choukroun and L. Wolf. A foundation model for error correction codes. In International Con- ference on Learning Representations (ICLR), 2024a.
Y. Choukroun and L. Wolf. Learning linear block error correction codes. In International Conference on Machine Learning (ICML), 2024b.
J. Dai, K. Tan, Z. Si, K. Niu, M. Chen, H. V. Poor, and S. Cui. Learning to decode protograph ldpe codes. IEEE Journal of Selected Areas in Communications, 39(7):1983-1999 2021.
T. Gruber, S. Cammerer, J. Hoydis, and T. Brink. On deep learning-base channel decoding. In 2017 51st Annual Conference on Information Sciences and Systems (CISS), pp. 1-6. IEEE, 2017.
H. Kim, Y. Jiang, R. Rana, S. Kannan, S. Oh, and P. Viswanath. Communication algorithms via deep learning. In International Conference on Learning Representations (ICLR), 2018
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In arXiv preprint arXiv:1412.6900 2014.
H.-Y. Kwak, Y. Yun, D.-Y.nd Kim, S.-H. Kim, and J.SS. No. Boosting Learning for LDPC Codes to Improve the Enror-Floor Performance. In Advances Neural Information Processing Sys- tems (NeurIPS), 2023.
S.-J. Park, H.- Y. Kwak, S.-H. Kim, Kim. Y. Kim, and J.-S. No. How to mask in Error Correction Code Transformer: Systematic and double masking. In arXiv preprint arXiv:2308. 08/28, 2023.
10

---

Published as a conference paper at ICLR 2026
S.-J. Park, H.-Y. Kwak, S.-H. Kim, Y. Kim, and J.-S. No. CrossMPT: Cross Cross-attention attention message- passing transformer for error correcting codes. In International Conference on Learning Repre- sentations (ICLR), 2025,
net net/forung 1
T. J. Richardson and S. Kudekar. Design of low-density parity check codes for 5g new radio. IEEE Communications Magazine, (6(3):28-344 2018.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2017.
A ADDITIONAL RESULTS FOR VARIOUS CODE PARAMETERS
Table 2 demonstrates the decoding performance for various code classes and parameters for h 8, N 6, and = 128. Across different code classes, EfficientMPT outperforms the original ECCT and achieves decoding performance comparable CrossMPT.
Table 2: Comparison of the BER performance at three different E, Noo values (4 dB, 5 dB, 6 dB) for EfficientMPT, CrossMPT, and ECCT (Choukroun Wolf 2022).
EfficientMPT
CrossMPT
ECCT
Codes
Method
4 dB
dB
6 dB
dB
d
6 dB
dB
5 dB
6 dB
(31,16)
9.11e-4
8.44e-5
3.58e-6
9.26e-4
9.63e-5
3.79e-6
1.68e-3
2.51e-4
2.35e-5
BCH
(63,36)
7.44e-3
1.31e-3
9.85e-5
6.53e-3
9.98e-4
8.49e-5
7.75e-3
1.29e-3
1.12e-4
(63,45)
3.26c-3
3.20c-4
1.50c-5
2.74c-3
2.74c-4
9.10c-6
3.70c-3
4.14c-4
1.79c-5
(64.32)
5.86e-4
4.10e-5
1.56e-6
5.51e-4
4.70e-5
1.66e-6
9.21e-4
7.95e-5
4.46e-6
Polar
(64,48)
1.73e-3
1.88c-4
1.63e-5
1.49e-3
1.67e-4
1.22e-5
1.73e-3
2.12c-4
1.53e-5
(128,64)
8.68e-4
2.27e-5
3.25e-7
5.40e-4
1.35e-5
3.88e-7
2.69e-3
1.77e-4
5.13e-6
(121.60)
3.63e-3
1.18c-4
3.99c-7
3.23e-3
9.47c-5
3.83e-7
5.68e-3
2,46c-4
1.67c-6
LDPC
(121,70)
9.50c-4
1,23e-5
4.26c-8
8.59c-4
1.13e-5
2.46e-8
1.66e-3
3.68e-5
1.10e-7
(121.80)
3,51e-4
4.11e-6
1.68e-8
3.38e-4
2.89c-6
1.31e-8
6.5e-4
1.0e-5
7.25e-8
B COMPARISON WITH DC-ECCT AND E2E DC-ECCT
Table BER comparison between DC-ECCT, E2E ECCT, and EfficientMPPT
Method
DC-ECCT
E2E ECCT
EfficientMPT
Parameter
4 JB
5 JB
6 dB
4 JB
d
dB
dB
5 dB
dB
(61,16) BCH
8.50c-4
6.19e-5
3.58e-6
7.54e-4
1.14e-4
2.65e-6
7.13e-4
5.49e-5
2.20e-6
(64.32) Polar
5.81e-4
2.78e-5
1.08e-6
5.05e-4
3.10e-5
2.07e-6
4.33e-4
2.83e-5
1.02e-6
(49,24) LDPC
3.81e-3
1.43e-3
7.39c-4
1.87e-3
1.55e-4
4.42e-6
1.14e-3
6.15e-5
1.14e-6
3 compares the BER performance of EfficientMPT with DC-ECCT, E2E DC- ECCT (Choukroun & Wolf (024b),, which are improved architectures of ECCT. All results are obtained by training with 1024 samples per minibatch, for 1000 epochs, with 1000 minibatches per epoch. The decoding performances of DC-ECCT and E2E DC-ECCT obtained from Choukroun Wolf (2024b). For all BCH code, polar code, and LDPC code, EfficcintMPPT outperforms all DC-ECCT and E2E DC-ECCT.
Table
C COMPARISON WITH SUCCESSIVE CANCELLATION LIST DECODER
We compare the decoding performance of the successive cancellation list (SCL) decoder with ECCT, CrossMPT, and EfficientMPT for polar codes. The performance of the SCL decoder is taken from

---

Published as a conference paper at ICLR 2026
Table 4: Comparison with SCL decoder for polar codes
Method
SCL(L-I)
SCL(L-4)
DOCT
CrossMPT
CfficienMPPT
Paramiter
4:88
5d8
4B
4:8
5dB
6d8
4.8
5:88
6d8
40B
5dR
6d88
4.8
5
6:88
(64.37)
6.76o-4
6.30p-
1.866
3.01o4
2.256-5
7.9k-7
9.3ce4
7.55-5
4460-6
5.5c-4
4.68x55
1666-6
5.86-4
4.10cs5
1.56o-6
(64.4) 48)
2.05p-3
2.23p4
1.7365
1.3433
1.70c4
1.3155
17503
2.9.E-37
1.5355
140c3
16764
1.22c-5
1.736x3
1.8cc
1,6665
(128.64)
2.32o-4
8.88-6
1.1266
6.766
193c6
272c-
269-3
1.7744
511c46
5-42-4
135--5
389c-
8.80-4
2.76655
3.256-7
Choukroun & Wolf (2022) and Park et al. 20255. Since the SCL decoder for polar codes has been extensively studied over time, while transformer-base decoders are still the early stages of research, it remains challenging for transformer-based methods outperform the SCL decoder. Nevertheless, both CrossMPP and EfficientMPT demonstrate significantly better performance than the original ECCT for polar codes, and achieve comparable performance to the SCL decoder for the (64, 48) polar code. In other words, we can improve decoding performance over the original transformer-based decoder while greatly reducing decoding complexity. These findings suggest that continued research in this direction can lead to further meaningful advancements in transformer- based ECC decoding.
D DECODING RESULTS FOR LONGER CODES
10
801
BEA
10-1
13-6P.00
8P.53
-6P.0
10
(a) (1824, 1520) MIMAXLDPC (b) (2304, 1920) WIMAX LDPC
Figure 8: Decoding performance of (a) (1824, 1520) WIMAX LDPC code and (b) (2304, 1920)
WiMAX LDPC code for the proposed method and BP decoders
Figure 8 compares the decoding performance of the proposed EfficientMPT with that of conven- tional belicf propagation (BP) decoders for long codes. For the (1824, 1520) WiMAX LDPC code, EfficientMPT is trained from seratch and FEfficientMPP is fine-tuned with the pre-trained Effi cientMPT model. The results demonstrate that with only 10 epochs of fine-tuning (FEfffcientMPT- 10), the model already achieves performance comparable the BP decoder. Furthermore, when fine-tuned for 200 epochs (FEIticienMMPT-200)), the performance closely matches that of the Effi- cientMPT trained from scratch and surpasses the BP decoder. addition, FEfficientMPT with fine- tuning also outperforms the BP decoder with maximum iterations of 20 and 50 for the (2304, 1920) WiMAX LDPC code.
This range of code lengths is particularly challenging for conventional transformer- based decoders. EfficientMPT, however, enables broader range of code lengths to benefit from the advantages of hansformer-based decoding. Rather than relying code-specific models, single EfficientMPT model achieves strong decoding performance across various code types, demonstrating the univer- sality of foundation decoders. When deployed as foundation decoder, such a model can signifi- cantly reduce hardware complexity and power consumption by supporting multiple generations of codes within unified architecture ecchiecture-efffctivvelll functioning a "multiple-in-oné decoder. The ef- fectiveness of EfficientMPT also extends the applicability of transformer based decoders to wider range of practical scenarios.
12

---

Published as a conference paper at ICLR 2026
E FEFFICIENTPPT WITH VARIOUS FINE-TUNING SETTINGS FOR UNSEEN CODES
10
10
10.5
PERici IMPTO
FE 20
TEROONTT WOO 12200
10
10
E,N,
E,N,
(a) (204, 102) MacKay LDPC
(1920, 1600) WiMAX
Figure 9: Decoding performance of (a) (204,70) MacKay LDPC code and (b) (1920,1600) WiMAX LDPC code for various fine-tuning epochs.
Figure 9 presents the decoding performance of FEfficientMPP under various unseen codes. The pretrained FEfficienMMPT model was initially trained on four different LDPC codes-(64, 32), (121, 60), (121, 70), and (121, 80) 4000 epochs.
settings for
For the (204, 102) MacKay LDPC code, only few epochs of fine-tuning are sufficient to achieve su- perior decoding performance. In contrast, for the (1920, 1600) WiMAC LDPC code, FEfficientMPP initially struggles with decoding unseen codes. However, its performance gradually improves with fine-tuning.
These results demonstrate that leveraging a pretrained foundation model trained on short codes remains effective for decoding long codes when fine-tuning technique is applied.
F COMPARISON OF TRAINED AND ORIGINAL PCM
(a) Trained PCM
(b) Original PCM
Figure 10: Comparison of the trained PCM and the original PCM. The larger its value, the darker the tone of each element.
Instead of utilizing the PCM the EfficientMPT architecture, we replace the PCM with a train- able matrix for (64,32) LDPC code. After the training, we obtain the trained PCM as shown in Figure I0(a) whose structure closely related to the original PCM in Figure [0(b) This observation further reinforces the justification of incorporating the PCM in EfficientMPT.
G COMPARISON WITH BM DECODER
We compare performance comparison between the BM decoder, ECCT, CrossMPT, and EfficientMPT. The sults demonstrate that transformer- decoders outperform the BM decoder and EfficientMPT shows comparable decoding performance with CPoS Additionally, thanks to the novel EEC
(BM) decoder with transformer-basee decoders. The table shows
13

---

Published as a conference paper at ICLR 2026
attention module, EfficientMPT achieves this powerful decoding performance with significantly duced computational complexity.
The results in this table highlight key advantage of approach: EfficientMPP not only serves as a universal decoder capable of handling various code classes, but also surpasses the performance of code-specific classical decoders like the BM algorithm.
Table 5: BER comparison between BM, ECCT, CrossMPT, and Proposed methods for BCH codes
Code (n.k)
SNR
BM
ECCT
CrossMPT
EfficientMPT
dB
1.16e-2
1.68e-3
9.26e-4
9.11e-4
(31,16) BCH
dB
3.14e-3
2.51e-4
9.63e-5
8.44e-5
6 dB
5.49e-4
2.35e-5
3.79e-6
3.58e-6
dB 6.66e-3
7.75e-3
6.53e-3
7.44e-3
(63,36) BCH
dB 9.17e-4
1.29e-3
9,98e-4
1.31e-3
dB
5.91e-5
1.12e-4
8.49e-5
9.85e-5
dB 7.80e-3
3.70e-3
2.74e-3
3.26e-3
(63,45) BCH
dB
1,46e-3
4.14e-4
2.74e-4
3.20e-4
dB
1,42e-4
1.79e-5
9.01e-6
1.50e-5
H COMPARISON IN RAYLEIGH FADING CHANNEL
We evaluate the performance of ECCT, CrossMPT, and EfficientMPP on a Rayleigh fading channel. In previous papers, ECCT and CrossMPT are known to be robust to the non non-Gaussian channels, such as Rayleigh fading channel. To compare with previous two works, we use the same fading channel as in ECCT and CrossMPT. The received codeword is given as = ha + z, where an n--iieensioal i.i.d. Rayleigh distributed vector with a scale parameter 1 and z N(0, oO), The following results demonstrate the BER performance ECCT, CrossMPT, and EfficientMPT:
Table 6: BER comparison between BCH (31, 16) and LDPC (121, 70) codes
Codes
BCH (31,16)
LDPC (121,70)
Method
4 dB
5 dB
6 dB
4 dB
5 dB
6 dB
ECCT
5.61e-3
2.38e-3
9.85e-4
2.01e-2
6.93e-3
1.82e-3
CrossMPT
3.95e-3
1.43e-3
4.95e-4
1.42e-2
3.98e-3
8.17e-4
EfficientMPT
4.29e-3
1.65e-3
5.06e-4
1.45e-2
4.13e-3
8.42e-4
As the results demonstrate, EfficientMPT also maintains robust decoding performance even this non-AWGN channel.
I PIPELINING OF EFFICIENTMPT
Although EfficientMPT may appear to be limited in throughput and latency due to its serial atten- tion blocks, pipelining
employed various ECC decoders-can can be applied to significantly enhance its decoding throughput (as illustrated in Figure 13, Appendix K in (Park) ctt al. 2025)). By unrolling the two EfficientMPT blocks within each layer for parallel hardware implementation, the model can process two consecutive codewords simultaneously. Specifically, while the second EfficientMPT block (responsible for syndrome updates) operates on the first code- word, the first block (responsible for magnitude updates) can begin decoding the next codeword. This pipelining approach allows EfficientMPT to maintain high and competitive throughput, even in fully parallel processing environments.
14

---

Published as a conference paper at ICLR 2026
J COMPARISON WITH NEURAL-BP BASELINE
Figure 11 compares the BER performance of EfficientMPT with a Neural-BP (NBP) decoder, using the (576, 432) WiMAX and (648, 540) IEEE 802.1 LDPC codes. The NBP decoder is evaluated with a maximum of 20 iterations. As the results demonstrate, our proposed EfficientMPT con- sistently outperforms the Neural-BP decoder across various signal-to-noise ratios for both LDPC codes.
12a
BER
10
ERiceRUPT So--euu
35
5 (88)
(a) (576,432) LDPC (b) (648,540) IEEE 802. LDPC
Figure 11: Decoding performance of (a) (576, 432) WiMAX LDPC code and (b) (648, 540) IEEE
802. In LDPC code for the proposed method and NBP decoders.
K PERFORMANCE ON 5G NR LDPC CODES
Figure 12 compares the BER performance of EfficientMPT, CrossMPT, and BP decoders for Base Graph (130, 110) 5G NR LDPC code. EfficientMPT achieves BER performance that is nearly identical to CrossMPT. Compared to the BP decoder with a maximum of 20 and 50 iterations, the results show that both CrossMPT and EfficientMPT outperform the BP decoder. In terms of complexity, traditional BP decoders generally exhibit lower complexity than neural network-based models. However, EfficientMPT achieves linear computational complexity, marking substantial reduction compared to the quadratic complexity of conventional transformer-base decoders (c.g., ECCT, CrossMPT). This efficiency gain of EfficientPPT represents major contribution towards making transformer based decoders practical for real-world applications.
10
E.N, 1981
Figure 12: Decoding performance of Base Graph 1 (130, 110) 5G NR LDPC code.
15


Reviews:
[{"review_id": "Xlx4AtNZWk", "reviewer": "Reviewer_v8pc", "summary": "This work introduces EfficientMPT, a transformer-based decoder for error-correcting codes that targets computational and memory bottlenecks in existing transformer-based decoders. The method introduces an efficient attention mechanism, replacing standard matrix multiplications with vector-based element-wise operations and integrating the parity-check matrix (PCM) directly into the attention module. EfficientMPT is designed to be position- and length-invariant, enabling it to serve as a foundation model for ECC decoding. Experimental results show significant reductions in GPU memory, FLOPs, and parameter count, while maintaining bit error rate (BER) performance comparable to existing transformer-based decoders.", "questions": "- The sentence “The magnitude embedding is then added to the resized syndrome embedding, which is resized from (n−k)×d to n×d by multiplying the PCM H” is unclear upon first reading—specifically, whether H is in binary or BPSK form. While later sections resolve this, earlier clarification would be helpful.\n- Is any sparsity-inducing regularization applied to the trainable PCM in Figure 7(b)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Effective handling of longer codes, overcoming a key constraint of earlier transformer-based ECC decoders.\n- Position and length invariance, supporting its use as a foundation model.\n- Thorough assessment across multiple code types.", "weaknesses": "- Limited interpretability analysis; it remains unclear how decoding decisions are made compared to traditional belief propagation.\n- Missing comparisons with neural Tanner graph-based decoders.\n- The analysis of polar codes would be strengthened by a more prominent and direct comparison with SCL decoders, particularly systematic SCL decoders when evaluating BER.\nFurther Suggestions\n- Please verify the dimensions of magnitude and syndrome embedding matrices after multiplication by H and Hᵀ in Figure 2(c) (green multi-head patches).\n- The sentence “The lifting process generates LDPC codes of size (52×Z,10×Z)” appears to describe the PCM dimensions rather than the code parameters (n, k). Please clarify.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cgoVfirFi5", "reviewer": "Reviewer_2sfM", "summary": "The paper proposed a transformer based decoder for error correcting codes. The proposed decoder use a lightweight attention module that replace the costly attention mechanism, in the Masked self attention module and masked cross-attention module.\nThe proposed EEC decoder run over the magnitude and syndrome separately with the proposed simpler attention module. \n\nThey get very good results that reduce the number of parameters, memory usage, and FLOPs, without compromising the decoding performance achieved by CrossMPT.", "questions": "1. Can you check the results of the proposed decoder on non-AWGN channel?\n\n2. Can you suggest ways to reduce the gap to the SCL results? maybe some changes in the architectural or at loss level?\n\n3. Please run the simulation on all codes at it appears in the paper of [Choukroun & Wolf (2022)], for example in Table 1 you only test two BCH code, instead of 4 codes and larger ones", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The proposed decoder is very efficient with respect to previous methods in terms of memory usage and FLOPs\n\n2. The proposed EfficientMPT decoder get competitive results compare to previous methods such as ECCT and CrossMPT\n\n3. The EfficientMPT algorithm can also serve as foundation model, a single model can be generalizes to unseen codes with fine-tuning, and get better results than BP on long LDPC codes.\n\n4. The proposed lightweight attention in the decoder using a global query vector and embedding the parity-check matrix, yields a simpler decoder.", "weaknesses": "1. The proposed algorithm evaluate only on simple AWGN channel, and not on non-Gaussian channels such as Rayleigh channel\n\n2. The proposed foundation needs fine-tuning in order to operate well on larger codes.\n\n3. For Polar codes, the classic SCL decoder still get better results than the proposed EfficientMPT decoder,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2J4NXrR8U5", "reviewer": "Reviewer_Cs5s", "summary": "This paper proposes an improved transformer-based decoder for error correction codes. The key novelty is the replacement of matrix multiplications in the attention mechanism with a vector-wise operations using a global query vector. EfficientMPT maintains decoding performance comparable to CrossMPT, while significantly reducing the parameter-count, FLOPs, and GPU usage. This is especially evident at large blocklengths - the efficiency improvements allow training at these lengths, without super-large memory usage.", "questions": "1. How does training from scratch compare to finetuning the foundation model? Specifically, can you add FEfficient-MPT results for LDPC(1824,1520) in Figure 8a? (and/or train-from-scratch in 6c)\n\n2. The BER performance for 5G-LDPC codes is missing. Can you please add this for completeness. I'd expect a similar performance as CrossMPT? What is the gap in performance/complexity to the decoders currently employed in 5GNR?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. EEC attention is well-motivated, and is a natural extension of the ideas in CrossMPT: the global query vector effectively condenses magnitude vector into one vector, rather than relying on a query matrix. The resulting updates resembles message-passing decoding between syndrome info and magnitude info, with simple updates using information from the other modality.\n\n2. The empirical results are very strong. While maintaining the same performance as CrossMPT, efficientMPT achieves significant reductions in memory and computational complexity with a simple architectural change.\n\n3. The main practical contribution is that it allows scaling transformer-based decoding to long blocklengths, which was infeasible via previous methods. FLOPs scale linearly with n, unlike other methods (quadratic dependence?)\n\n4. EfficientMPT can act as a foundation model - unseen codes can be handled via lightweight finetuning.\n\n5. I like the experiment in Figure 7, supports the understanding that a very string inductive-bias based on the PCM has been imposed on the attention mechanism.\n\n6. The evaluation is thorough - compares with good decoders for canonical codes (SCL for polar rather than BP, etc)", "weaknesses": "No major weaknesses.\nWhile transformer-based decoders (including this paper) are still sub-optimal to classical codes/decoders used in practice, this paper improves efficiency of ECCT - which is a major step in the right direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1fXgFb7rBI", "reviewer": "Reviewer_GA3B", "summary": "The paper proposes an efficient variant of a transformer-based ECC decoder. The main difference from the previous methods is an efficient attention module, which is based on query and key components without a value component.", "questions": "Can the proposed attention module architecture be useful to transformer applications other than ECC?\n\nIn line 181 it is said that standard methods have complexity o(n^2).  What is the complexity of the proposed method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper conducted extensive experiments and shows improvement in memory size and computational complexity compared to previous works.", "weaknesses": "The contribution of the present paper, in comparison with the recent works on Transformer-based ECC decoding by Choukroun and Park, seems incremental, as it largely reproduces the same elegant algorithm with  a minor technical adjustment.\n\nThe paper is not clearly written. It is focused on the many low-level technical details and doesn't explain the intuition and motivation \nof the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)