A Foundation Model for Error Correction Codes
Authors: Yoni Choukroun
Lior Wolf
Venue: ICLR 2024 poster (2024)
Keywords: Error Correction Codes
Foundation Model
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose for the first time a SOTA foundation Neural error correction decoder capable to decode arbitrary code at any length.

Abstract: In recent years, Artificial Intelligence has undergone a paradigm shift with the rise of foundation models, which are trained on large amounts of data, typically in a self-supervised way, and can then be adapted to a wide range of downstream tasks. In this work, we propose the first foundation model for Error Correction Codes. This model is trained on multiple codes and can then be applied to an unseen code. To enable this, we extend the Transformer architecture in multiple ways: (1) a code-invariant initial embedding, which is also position- and length-invariant, (2) a learned modulation of the attention maps that is conditioned on the Tanner graph, and (3) a length-invariant code-aware noise prediction module that is based on the parity-check matrix. The proposed architecture is trained on multiple short- and medium-length codes and is able to generalize to unseen codes. Its performance on these codes matches and even outperforms the state of the art, despite having a smaller capacity than the leading code-specific transformers. The suggested framework therefore demonstrates, for the first time, the benefits of learning a universal decoder rather than a neural decoder optimized for a given code.

Published as a conference paper at ICLR 2024
A FOUNDATION MODEL FOR ERROR CORRECTION CODES
Yoni Choukroun
Lior Wolf
The Blavatnik School of Computer Science Tel Aviv University
The Blavatnik School of Computer Science Tel Aviv University wol cscs tau. ac.
choukroun. yon @gma i 1.. com
ABSTRACT
In recent years, Artificial Intelligence has undergone a paradigm shift with the rise of foundation models, which are trained on large amounts of data, typically in a self-supervised way, and can then be adapted to a wide range of downstream tasks. In this work, we propose the first foundation model for Error Correction Codes. This model is trained on multiple codes and can then be applied to unseen code. To enable this, we extend the Transformer architecture in multiple ways: (1) code-invariant initial embedding, which is also position- and length- invariant, (2) a learned modulation of the attention maps that is conditioned on the Tanner graph, and (3) length-invariant code-aware noise prediction module that is based on the parity-check matrix. The proposed architecture is trained on multiple short- and medium-lengtt codes and is able to generalize to unseen codes. Its performance on these codes matches and even outperforms the state of the art, despite having smaller capacity than the leading code-specific transformers. The suggested framework therefore demonstrates, for the first time, the benefits of learning universal decoder rather than a decoder optimized for given code.
INTRODUCTION
Reliable digital communication relies on the design of codes that can be accurately decoded when transmitted over noisy channels. The optimal decoding is defined by the NP-hard maximum likeli- hood rule, and the efficient decoding of codes remains open problem.
Recently, powerful learning-based decoders have been introduced, borrowing from the architecture of well-proven deep models. For example, Transformer-based decoder that incorporates the ror correction code (ECC) into the architecture has been recently proposed by [Chuurroun & Wolf (202aa), outperforming existing methods by sizable margins and at fraction of their time complex- ity. This architecture has been subsequently integrated into a denoising diffusion models paradigm, further improving results (Choukroun & Wolf 2022b).
A major drawback of current ECC neural decoders is the dedicated adaptation of the architecture and optimization of the model with respect each single code of interest. This major challenge prevents a trained neural decoder from being employed for the decoding or fine-tuning of other codes or settings, even for the same family of codes and settings but with different lengths.
Large models known as foundation models (Bommasani et al. 2021) have revolutionized deep learn- ing by addressing multiple complex downstream tasks that pose challenge alternative methods. These models are typically initiated through training extensive unlabeled datasets, primarily by employing self-supervised pretext tasks. Subsequently, transfer learning to new tasks utilizes strate- gies such zero-shot learning, prompt-engineering, or fine-tuning (Brown et al. (020)).
In this work, we consider the design and training of a foundation neural decoder for error correction codes based on the Transformer architecture of Vaswani et al. 2017), which is capable of adapting and generalizing to any code. As far as we can ascertain, this is the first time a universal neural decoder has been presented. Beyond the conceptual novelty, (i) we adapt the Transformer input embedding in order to remain invariant to code length (ii) the positional embedding, as well the code, added as relative positional encoding that is integrated
make three technical contributions:

---

Published as a conference paper at ICLR 2024
into the self-attention via a learned mapping of the node distances in the Tanner graph, and (iii) size-invariant prediction module that is conditioned on the parity-check matrix and is based on a learned aggregation function. Applied to wide variety of codes, method is able reach and even surpass the performance of state-of-the-art per-code learning-based solutions on the zero-shot, pretrained, and fine-tuned settings. This is obtained despite remaining entirely code-invariant and employing an extremely shallow architecture.
RELATED WORKS
Previous work neural decoders divided by Raviv et al. 20200) into two main classes: model- based or model-free. Model-based decoders implement parameterized versions of classical Belief Propagation (BP) decoders, where the Tanner graph is unfolded into neural network in which weights are assigned to each variable edge. This results in an improvement in comparison to the baseline BP method for short codes (Nachmani al. 2016 Nachmani & Wolff 2019 2021). Model- based decoders benefit from strong theoretical background, but the architecture is overly restrictive.
Model-free decoders employ general types of neural network architectures. Earlier approaches [ammmeerr et al. 2017: Gruber et al.) 2017;; Kim et al. 2018) employed stacked fully connected networks or recurrent neural networks that have difficulties in learning the code. However, some of the training approaches are still in use. For example, our work employs the preprocessing of Bennatan et al. (2018), which transforms the channel output such that the decoder remains prov- ably invariant to the codeword. This allows the training of generic neural decoders without the risk of codeword overfitting. Choukroun & Wolf (2022a) introduced the Error Correction Code Transformer (ECCT), obtaining SOTA performance. The model embeds the signal elements into high- high-dimensional space where the analysis is more efficient, while the information about the code integrated via masked self-attention mechanism. Subsequently, Choukroun & Wolf (2022b) ex- tended the denoising diffusion paradigm to ECC, iteratively applying an ECCT, further improving the SOTA by large margin.
We note that neural decoder contributions generally focus on short and moderate-length codes for two main reasons: (i) classical decoders are proven reach the capacity of the channel for large codes, preventing any potential enhancement, and (ii) the emergence of applications driven by the Internet of Things created the requirement for optimal decoders of short moderate codes. For example, 5G Polar codes have code lengths of 32 to 1024 (ESTI 2021).
Recently, the ML community has been focusing intensively on large foundation models trained on internet-scale datasets, which achieve state-of-the-a performance on diverse range of learning tasks. Rather than learning -ask-specific models from scratch, the foundation models are adapted via fine-tuning few-shot/zero-shot learning strategies and subsequently deployed on wide range of domains (Brown et al. 2020; Radford et al. 202I). Such foundation models enable the transfer and sharing of knowledge across domains, mitigate the need for task-specific training data, and have been applied to several fields, such as Natural Language Processing (LLM) Wei et al.] 2022 Brown et al.) 2020), computer vision (Saharia et al.
Touvron al. 2023
2022 Rombach et al.
2022
Ramesh et al.
2022), multimodal analysis (Radford et al.] [021]
Li et al.
2022 Awadalla
al.
2023 OpenAT
2023), and even reinforcement learning (Reed al. 2022)
3 PROBLEM SETTING AND BACKGROUND
In this work, we assume a standard transmission protocol using a linear code C. The code is defined by a generator matrix G (0, 1xx= and the parity check matrix H E (0, x))) defined such that GHT 0 over the order 2 Galois field G2(2). The parity check matrix H entails what is known as a Tanner graph, which consists of n variable nodes and (n k) check nodes. The edges of this graph correspond to the on-bits in each column of the matrix H.
The input message m E {0, 11*h is encoded by G to a codeword x E C C f0,13" satisfying Hz = 0 and transmitted via Binary-Input Symmetric-Outpur channel, e.g., AWGN channel. Let y denote the channel output represented as = x. + E, where x, denotes the Binary Phase Shift Keying (BPSK) modulation of x (i.e., over (+1)), and is random noise independent of
2

---

Published as a conference paper at ICLR 2024
the transmitted 2. The main goal of the decoder f R" R" is to provide soft approximation = f(y) of the codeword.
We follow the preprocessing of Bennatan et al. (2018); Choukroun & Wolf (2022a), in order to remain provably invariant to the transmitted codeword and to avoid overfitting. The preprocessing transforms y to 2n dimensional codeword invariant vector defined as y = h(y) [ly|, s(v)], where, - denotes vector concatenation, [y] denotes the absolute value (magnitude) of and s(y) E (0, 1|=>> denotes the binary code syndrome. The syndrome is obtained via the GF(2) multiplication of the binary mapping of y with the parity check matrix such that
sty) yb == Hbin(y) H (0.5(1 sign(y))).
The induced parameterized decoder fo RR= R with parameters 0 aims to predict the multi- plicative noise denoted É and defined such that y 2, @ é. The prediction of the multiplicative noise instead of the additive physical one used in order to remain invariant the transmitted code- word, since the syndrome is codeword-invarian code overfitting, as described by |Bennatan et al. 2018) and the proof of lemma of (Richardson & Urbanke 2001 The final prediction takes the form क. sign(y fllly,, Hy.)). An illustration of the coding procedure under the foundation model setting is given in Appendix
and y |a. |E], thereby avoiding the risk of
3.1 ERROR CORRECTION CODE TRANSFORMER
The state-of-the-an ECCT (Choukroun & Wolf 2022a) has been recently proposed for neural error decoding. It consists of a transformer architecture (Vaswani et al. (201)) with several modifications. Following Bennatan et al. (2018), the model's input h(y) is defined by the concatenation of the codeword-indepeedden magnitude and syndrome, such that h(y) == [[y], 2s(w)] E IR2n-k, Each element then embedded into a high-dimensional space for more expressivity, such that the initial positional embedding & (((2n-k)xd given by (h(y) 12) @ W where W Q(2n-k)xd is the learnable embedding matrix and @ is the Hadamard product.
The interaction between the bits is performed naturally via the self- self-attention modules coupled with binary mask derived from the parity-check matrix in order to integrate information about the code
Aμ (Q, K, V) Softmax(d "*QK" H(H))
(2)
where g(H) is a fixed binary masking function designed according to the parity-chead matrix H, and Q K,Y V are the classical self-attentio projection matrices.
The masking g(H): {0, (_(=)))>> - (-oo, 03 (2n (2(m is defined by the adjacency matrix of the Tanner graph extended to two-ring connectivity: the mask's on-bits exist between adjacent nodes (distance onc) and secondary neighbors (distance two).
Finally, the transformed embedding is projected onto a one-dimensional vector for the noise predic- tion with linear layer, such that with Wo (R(2n-)xx x)xn and W E Rdx1 have
E= (W((W++11)
(3)
4 THE ECC FOUNDATION MODEL
We present the elements of the proposed Foundation Error Correction Code Transformer (FECCT) decoder, the complete architecture, and the training procedure
4.1 CODE-INVARIANT INITIAL EMBEDDING
Traditional Transformers encode each token according to the model's vocabulary embedding; the positional embedding is added at later stage. In ECCT, a unique model crafted for every code and length. The initial embedding is designed such that each input bit possesses its distinct embedding vector, providing, as a byproduct, a learned positional encoding.
In our ievvaiainv context, we propose new code-invariant embedding, where single embed- ding is given for all magnitude elements, and two embeddings are given for every element of the binary syndrome.
3

---

Published as a conference paper at ICLR 2024
DCCT Binary Maok
Tonnre Graph Distance
1
(h)
Figure 1: For the Hamming(774) code: (a) the parity-check matrix, (b) the induced Tanner graph, (c) the ECCT binary masking, (d) the distance matrix G (HH).
Formally, by considering each dimension of { separately, we define the projection of each element to high, d dimensional embedding {or [
2n i=1
2n-k
E Rd as:
I WM, WS
i
Φ:
(4)
otherwise
(s(v))--+11
where (WM, W W E Rd denote the magnitude encoding and one-hot encoding of the binary syndrome elements, respectively, and (s(y)) denotes the j element of the syndrome vector. Note that unlike ECCT, in which 2n different embeddings learned (for specific code), FECCT learns only three embedding vectors (which are used for all codes). This substantially reduces the number learned parameters.
Since ECCT has a different embedding for each bit, this fixed embedding is capable of capturing the bit position. In contrast, our method enables the embedding of arbitrary codes of any length, the price of losing the positional encoding. This is addressed next, using code-conditionsss modulation of the self-attention maps.
4.2 TANNER GRAPH DISTANCE MASKING AS CODE AND POSITIONAL ENCODING
For a given algebraic code defined by the matrix H, ECCT integrates its information by designing a binary masking function, thresholding the attention between nodes with distances greater than two on the Tanner graph.
In FECCT, the masking serves two purposes. First, similar to ECCT, integrates the code structure into the transformer. Second, since, unlike ECCT, the initial embedding is position-invariann the learned masking adds the relative position information to the processed elements.
Based on the notion that the Tanner graph captures the relations between every two bits in the code, we consider the distance matrix G ((2n-k)((2n-)) which is constructed from Tanner graph of the parity-check matrix H. Specifically, each element (i, in this matrix is defined the length of the shortest path in the Tanner graph between node and node j. An illustration of ((H) on the Hamming(7,44 code is presented in Figure dd).
In order to incorporate the positional information into the self-attention mechanism, we learn a parameterized mapping IN > R from the elements of the distance matrix G(H) to value that is used to modulate the self-attention maps:
QKT
Ay As K, (, = Soffmax
c + (((H)))
(5)
Vd
This H-dependent attention mechanism generalizes that of Eq. 2 where the latter may be approx- imated as special case in which v(G((H),)) 1(ggH)) applied in non-differentiable manner. The general form of modulation in Eq. captures more information from the Tanner graph, and, therefore, about the code. This allows us, for example, to reconstruct the entire graph from the values used to modulate the self- self-attention maps, whereas ECCT's binary thresholding function is often not informative enough for reconstruction, shown in Appendix B
4

---

Published as a conference paper at ICLR 2024
BCCT Mask
PC Marrix
Output PC Layer
31-
30
-05
30-
131 16
100
150
141
(a)
(b)
(c)
Figure 2: For the LDPC(96,48) code: (a) the parity-check matrix, (b) the cropped ECCT binary masking, (c) the learned fully connected output layer. One can observe that the small dynamic range reduces the fully connected dot- hot-product to sum of few signed elements
4.3 PARITY-CHECK AWARE PREDICTION
After the initial encoding, the (2n-k) k) embeddings are propagated through multiple normalized self- attention and feed-forward blocks towards the output module to provide the final noise prediction.
To obtain the prediction, ECCT makes of two fully-connected layers, described in Eq. The first layer Wd-1 shrinks each of the (2n - k) embedding vectors from to and the second W. reduces the output from ij's dimensions (2n k)) y's dimensions (n).
As can be seen in Fig. c), the learned W, is sparse and each bit of the prediction is a linear combination of the corresponding bits in (the diagonal line in the figure) and of its related elements according to the parity-check matrix or induced mask (see the (n k) rightmost columns). This behavior enforced; it emerges organically during training.
Motivated by this phenomenon, in FECCT, we explicitly enforce a similar dependency structure. Since the FECCT is required to be code- and length-invariant, it cannot use a fully connected layer, and this structure is obtained by directly considering the parity check matrix as follows.
We first transform the magnitude and the syndrome parts of the learned embedding separately. The syndrome part is resized from (n k) embeddings to embeddings according to an aggregation in- duced by the phrity-check matrix. Then, the n magnitude embeddings are added to the transformed syndrome elements. Finally, the d-dimensional embedding is shrunk to one dimension for the pre- diction. This way, we are able incorporate the residual information of the syndrome embeddings into the final prediction bits.
Specifically, let us denote the final Transformer's embedding do E R(dx=k)xd, and Ws, WMr E Rdxd two learnable affine transforms matrices. We define by Pa[1:n] = Φo.M E Rexdd and 1:2nn = Φo.s E x(n-k)xd the magnitude and syndrome parts of the embedding, respectively. The output module performs the following projections
= (do WM + HTT (do,
(6)
where E denotes the predicted noise, and d-++1 denotes the final embedding shrinkage. The pro- posed aggregation provides both (i) code-awareness in the sense that the aggregation is induced by the parity-check matrix, and (ii) code-invariance in the sense that the aggregation is invariant code size and can be performed with any code.
4.4 ARCHITECTURE AND TRAINING
An illustration the entire model is given in Figure 3 The initial encoding is performed with a dimensional one-hot encoding for the syndrome part and single d-dimensional vector for the mag- nitude part, for a total three d-dimensional parameters. The decoder is defined as a concatenation of N 6 decoding layers composed of self-attention and malization layers with = 128. The distance embedding is a learned integer-to -c-cllr mapping, for mapping values 1, 1,..., 10, where 10 is the maximal distance encountered in Tanner graph. See
layers interleaved with nor-
5

---

Published as a conference paper at ICLR 2024
bin (kgn X)
Initial Embedding
Decoding
MH-5A
Outpur
+
Megnillude Embeddciig
Feed Fard
Matrrll
Norm
+
||
Marrul
Syndromn
Syndrome Onn has
MH-SA
Distanoe Encoding
FCr
d
FCu
Norm
Norm
Figure Illustration of the proposed Foundation Transformer architecture.
Appendix C for the histogram of distances in the training dataset. Note that self-attention is turned off by mapping the diagonal (distance of zero) to infinity.
The dimension of the ferwafo network of the transformer is four times that of the embedding d, following Vaswani et al. (2017), and is composed of GEGLU layers (Shazeer (020), with layer normalization set the pre-layer setting, as Klein et al. (2017): Xiong al. at. (2020). An eight-head self-attention module is used in all experiments. We note that while larger architectures and longer training times would enable better performance, deepening the accuracy gap from other methods (e.g., GPT-3 (Brown al. 2020) operates successfully 2K inputs with similar Transformer model, but with N 96, d = 12K), error correction requires rather light and shallow models that can be deployed edge devices. The output module is described fully in the previous section.
The training objective is the cross-entropy function, with the goal of learning to predict the mul- tiplicative noise ج (Bennatan al.) 2018). Denoting the soft multiplicative noise é,, such that I, &e,, we obtain & = E. Qas you, Thus, the binary multiplicative noise to be predicted defined by bin r.). such that the loss computed for a single received word y is
C e, log(Sow)) (1 log(1 1 fo()))
The estimated hard-decoded codeword is straightforwardly obtained as è. = (signn (f(() @ v)).
The Adam optimizer (Kingma & Ba 2014) is used with 512 samples per minibatch, for 3000 epochs, with 1000 minibatches per epoch. We note that while using more epochs can improve performance as demonstrated with large foundation models, the current shallow setting already reaches SOTA performance. We initialized the learning rate to 10-4 coupled with a cosine decay scheduler down 10-6 the end of the training. No warmup was employed (Xiong et al. 2020).
The code database from Helmling et al. (2019) was web-scraped in order to extract all possible binary codes. Due our modest computational resources, we limited our training to codes with lengths no longer than 150. A full description of the codes used for training and testing as well as general statistics is given in Appendix C It is important to note that the training set is highly unbalanced between code families.
Due to the construction of the model and its input preprocessing, the zero codeword is sufficient for training for every code. The additive Gaussian noise is sampled randomly per batch in the {2...., 8) normalized SNR (i.e. E,/N)) range. Other types of noise can also be used increase the generalization to noise (e.g., Rayleigh channel), but this is beyond the scope of this paper. Each training batch is constructed as the concatenation of randomly sampled code and noise, with adapted masking of the different clements (c.g., y, self-attention mask, distances, etc.).
Training time is around 12 days, without any coding optimization of the self-attention mechanism. The acceleration of the proposed method (e.g. pruning, quantization, distillation, low-rank approx- imation) (Wang et al. 2020 Lin et al.) 2021) is beyond the scope of this paper and is left for future work. Training and experiments are performed on four 12GB GeForee RTX 2080 Ti GPUs. The training time is approximately 320 seconds per epoch. Testing time depends on the code, rang- ing from 10ms to 30ms per sample using one GPU. While the complexity is similar to the ECCT

---

Published as a conference paper at ICLR 2024
Table I: comparison of our method to the baselines on codes seen during pretraining, Higher is better. BP and learned BP results are provided for either 5 (50) iterations in the first (second) line.
Method
BP
Hyp BP
ARBP
ECCT
Ours
Eg/No
4
5
4
5
6
4
6
5 6
3.72 4.65 5.66
3.96 5.35
7.20
4.33
5.94 8.21
BCH(63,36)
4.56 6.37
8.85
4.53 6.38
9.10
4,03 5.42 7.26
4.29
5.91 8.01
4.57 6.39 8.92
Reed Solomon(21, 15))) BCH(127,
NA NA NA
NA NA NA
NA NA NA NA NA NA
4.70 6.37 8.95
4.62 6.33 8.95
NA
NA NA NA
NA NA NA NA NA 4.25 5.49 7.02 4.59 60 7.69
5.71 7.42
9.11
5.71 7.28
9.12
Reed $olomon(60,52)
NA NA
NA NA
NA NA NA
5.53 7.54
9.98
5.47
7.59
10.21
POLAR(32,16)
NA NA
NA
NA NA
NA NA 4.77 6,30 8.19 5.57 7.43 9.82
NA
6.57 8.94
11.91
6,36
8.36
11.49
3.52 4.04
4.48 4.26 5.38 6.50
P0LAR(6)1,48)
6.21 8.32 10.71
6.06 8.21 10.90
(d kk + ndd))). the number of parameters of the FECCT is much smaller, since is independent of the code's length, contrary the ECCT, which requires ((2n k)dd + n2nn k)) more parameters for the input and output encodings.
EXPERIMENTS
To evaluate our method, we train one model of the proposed architecture with four classes of linear
codes: now-Density Parity Check (LDPC) codes (Gallager 1962). Polar codes (Arikan 2008). Reed
Solomon codes (Reed & Solomon 1960) and Bose Chaudhuri Hocquenghenm (BCH) codes (Bose)
Ray-Chaudhuri] 1960). All parity check matrices are taken from Helmling et al. (2019).
We compare our method with the BP algorithm (Pearl 1988), the recent Autoregressive hyper- network BP of (Nachmani & Wolf (021) (AR BP), and the SOTA ECCT (Choukroun & Wolf 2022a) with the same number of layers. Note that LDPC codes are specifically designed for BP- based decoding (Richardson et al. (200T).
The results are reported as negative natural logarithm bit enror rates (BER) for three different normal- ized SNR values (Eb/ No), following the conventional testing benchmark, e.g., Nachmani & Wolf 2019); Choukroun & Wolf (2022a). BP-based results are obtained after L 5 BP iterations in the first row (i.c. 10-layer neural network) and at convergence results in the second row are obtained after L 50 BP iterations (i.e., 100-layer neural network). During testing, least 10 random codewords are decoded, to obtain at least 50 frames with errors at each SNR value. We trained and tested all reported ECCT results to ensure that the models were trained on the same parity-check matrices. All the other baseline results were obtained from the corresponding papers, omitting the codes that have not been tested by these baselines (NA). We refer the reader to Choukroun & Wolf 2202a b) for complexity and performance comparisons with specialized decoders such as BP and SCL Tal Vardy (2015).
In Table we present the performance of our model on codes seen during training. As can be seen, our method can outperform the state the art or remain very close to it.
Table depicts the zero-shot performance of our model, i.e., the performance on unseen codes. Evidently, our method outperforms the state of the art even on some of the unseen codes. However, for codes from families that are underrepresented in the training set, such as Polar codes, per-code transformer may outperform our zero-shot performance. Reassuringly, the results for BCH((55,) 163) show that our model is able to generalize to unseen length that exceeds the length of the longest training code by a factor of two. We also provide zero-shot generalization performance on the BCH(1023, 1013) code (seven times longer than the larger code in the training set) in Appendix
Finally, in Figure 4 we present the performance improvement when we tin-ttne given code, i.e., train the code of interest for 500 epochs. Other fine-tur fine-tuning strategies can also be more beneficial. As can be seen, the proposed fine-tuning enables us to further outperform or close the gap with the state of the art, where needed. We provide BER curves several codes in Appendix

---

Published as a conference paper at ICLR 2024
Table 2: comparison of our method with literature baselines on zero-shot learning, i.c., the per- formance of the FECCT is provided on new codes only. Higher is better. BP and learned BP results are provided for either 5 (50) iterations in the first (second) line.
Supervision
Unlearned
Fully supervised
Zero-Shot
Method
BP
Hyp BP
ARBP
ECCT
Ours
Eo/No
4
4
4
4.08 4.96
6.07
4.48
6.07
8.45
4.80
6.43
8.69
BCH(63,45)
5.18
7.24
10.20
5.18 7.32 10.31
4.36
5.55
7.26
4.64
6.27
8.51
4.97
6.90
9.41
4.34
6.35
4.64
6.08
8.16
4.95
6.69
9.18
BCH(63,51)
5.63 7.96
11.22
5.71 8.07 11.31
5.80
7,42
4,80
6,44
8.58
5.17 7.16
9.53
BCH(127,92)
NA
NA
NA
NA
NA
NA
NA
NA
NA
4.10
5.71
8.38
4.11 5.84 8.79
BCH(255,163)
NA NA
NA 13.78 NA
NA
NA 10.57
NA
NA
NA
NA
3.34
4.13
5.80
3.34 4.13 5.76
6,55 9.65
6,99
15.27
7.25
10.99
16.36
CCSDS(128.64)
10.51
15.90
6.52 9.67 15.01
CCSDS(32.16)
NA NA
NA
NA
NA
NA
NA
NA
5.93 7.77 10.02
5.23 7.00 9.21
3.80 4.19 4.62
4.57
6.18
8.27
4.81
6.57
9.04
P0LAR(128,86)
6.39 9.08 12.70
5.53 7.90 11.29
4.49 5.65 6.97
4,95
6,84
9.28
5.39
7.37
3.52 4.04 4.48
4.25
5.49
7.02
4.77
6.30
8.19
POLAR(61,32)
6.91 9.18 12.34
5.88 7.91 10.76
4.26 5.38 6.50
4.59
6.10
7.69
5.57
7.43
9.82
14
12
(n(BRR)
6
5
4
5
6
5
4
6
BCH((3,)6))
BCH(66,,51)
RS(21,15)
CCSDS332,16
POLAR(32,16) P0LAR(64,48)
Ours
Figure 4: A comparison of method with the SOTA ECCT, the pretrained FECCT and the fine-
tuned FECCT (Ours-FT) on the code of interest. Higher better.
5.1 ABLATION STUDY
we present in TableBthe impact of the multiple architectural proposi- tions on performance. Besides the method's invariance code and length, these results demonstrate the beneficial impact of each of the contributions on the obtained accuracy. Most crucially, without the suggested distanced-bel based masking (using ECCT's mask on bits for distances smaller than 3), the performance drops when both the input and output are invariant, since there is no position encoding. Interestingly, the invariant initial embedding inductive bias provides better accuracy while having less parameters. Finally, of the graph distance. It is interesting to observe that the model seems to assign the most impact- ful mapping for the elements distanced by and two nodes, remarkably matching the ECCT's two-ring heuristic. Illustrations of typical self-attention maps are given in Appendix
Architectural contributions:
provide in Figure the illustration of the fine- fine-grained learned mapping
Generalization contribution: We provide generalization analysis of the framework in Appendix where we show the advantage of the proposed training strategy for generalization.
DISCUSSION AND LIMITATIONS
The proposed framework permits the efficient integration of a single inde-invariant neural decoder into base stations and error correction system chips, correcting multiple code families different rates. This can potentially save die space as well as storage and memory usage while matching or outperforming the state-of-the-art neural decoders. Also, the development of universal neural
8

---

Published as a conference paper at ICLR 2024
Layer
Layer
Layer
13-
1.7
062
012
0.4
005
0.2
0.0
Distanc
Distatoe
Distanoe
(a)
(b)
(c)
Figure 5: Absolute values of the learned mapping of the proposed FECCT with respect to distance
for three layers of the model among the six. The mapping of all the layers is given in Appendix D
Table 3: A comparison of the impact of each one of the presented contributions on the performance. All the models have a N 1000 epochs. DM refers to the proposed Distanced-basse code Invariant Initial embedding, and refers the suggested Invariant Output parity-check aware aggregation scheme. We note the ECCT has 15% more parameters than the FECCT.
32 architecture and follow the same ECCT's training setting of
Masking strategy, I/ refers to the proposed
Method
POLAR(64..2)
63H(633
4
5
ECCT
4.12
5.22
6.67
4.45
5.81
7.65
ECCT +
4.27
5.54
7.14
4.52
5.98
7.92
ECCT + I0
4.44
5.73
7.40
4.41
5.76
7.62
ECCT + 11+10
4.09
5.26
6.80
4.31
5.62
7.41
ECCT + DM
4.44
5.73
7.37
4.74
6.34
8.53
ECCT + DM
4.44
5.73
7.37
5.17
7.07
9.59
ECCT + DM+
4.36
5.64
7.32
4.53
6.01
8.03
FECCT: ECCT + DM + 11 10
4.36
5.64
7.32
4.52
5.98
8.05
decoders allows for the differentiable optimization of codes. For example, existing or new codes can be improved iteratively to enable more effective decoding in higher SNRs.
It is important note that the proposed universal decoding architecture can be improved further by applying together with other neural decoding algorithms, such as the denoising diffusion ECC paradigm (Choukroun & Wolf, 2022b), which was evaluated with the ECCT architecture backbone.
As our experiments reveal, dataset diversity is important for generalization and performance. Cur- rently, the framework we trained codes obtained from the dataset of Helmling al. (2019) which are highly unbalanced, with an overrepresentation BCH codes. We also note that the parity check matrices of the Polar codes have been standardized following Choukroun & Wolf (2022a) which, besides the under-represee reteesttprssnnttioonn of Polar codes, may be why the FECCT is less performant than ECCT on zero-shot tasks for these specific codes.
CONCLUSION
We present a novel foundation model based the Transformer architecture for the decoding of algebraic block codes. The proposed model allows effective representation of interactions between the code's elements in an invariant fashion, and the processing of any code of any length. The proposed method reaches and even outperforms the state of the art, while having fewer parameters and being code- and size-invariant, enabling the potential efficient deployment of single universal decoder for ECC. The next steps should focus on the deployment of the method existing error correction embedded systems via the utilization of Transformer acceleration methods along with extensive tuning and training of the codes and channels of interest. Additionally, the definition of universal differentiable neural decoder may open the door to the optimization of codes and even to the learning of new families of codes.

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENTS
This project has received funding from the Tel Aviv University Center for Al and Data Science (TAD) and the Blavatnik Computer Science Research Fund. The contribution of the first author is part of PhD thesis research conducted at Tel Aviv University.
REFERENCES
Erdal Arikan. Channel polarization: A method for constructing capacity-achieving codes. In 2008
IEEE International Symposium on Information Theory, pp. 1173-1177. IEBE, 2008
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo, March 2023. URL https:
528.//eodd. zenodo. 7733589
Amir Bennatan, Yoni Choukroun, and Pavel Kisilev. Deep learning for decoding of linear codes-a syndrome- based approach. In 2018 IEEE International Symposium on Information Theory (ISIT), pp. 1595-1599. IEEE, 2018.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.072 2021.
Raj Chandra Bose and Dwijendra K -haudhuau..... On class of error correcting binary group codes. Information and control, 3(1):68-79, 1960.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sebastian Cammerer, Tobias Gruber, Jakob Hoydis, and Stephan ten Brink. Scaling deep learning- based decoding of polar codes via partitioning. In GLOBECOM 2017-2017 IEEE Global Com- munications Conference, pp. 1-6. IEEE, 2017.
Yoni Choukroun and Lior Wolf. Error correction code transformer. Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Yoni Choukroun and Lior Wolf. Denoising diffusion error correction codes. In The Eleventh Inter- national Conference Learning Representations, 2022b.
ESTI. 5g multiplexing and channel coding. etsi 3gpp 38.212.
https://wwk
02.
132122v
2021.
Robert Gallager. Low-density parity-check codes. IRE Transactions on information theory, 8(1): 21-28, 1962.
Tobias Gruber, Sebastian Cammerer, Jakob Hoydis, and Stephan ten Brink. On deep learning-base channel decoding, In 2017 51st Annual Conference on Information Sciences and Systems (CISS), pp. 1-6. IEEE, 2017.
Michael Helmling and Stefan Scholl. MI simulation results. channel ott.://wwwo.......aaaaaaaeeaaaea pid=4| 2016.
https://rptt: de/
Michael Helmling, Stefan Scholl, Florian Gensheimer, Tobias Dietz, Kira Kraft, Stefan Ruzika, and Norbert Wehn. Database of Channel Codes and ML Simulation Results. channel-codes 2019.
www. uni- kl. de/
Hyeji Kim, Yihan Jiang, Ranvir Rana, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Communication algorithms via deep learning. In Sixth International Conference on Learning Representations (ICLR), 2018.
10

---

Published as a conference paper at ICLR 2024
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. Opennmt: Open-source toolkit for neural machine translation. In Pruc. ACL, 2017. doi: 10. 18.53v//v P17-4012. URL https::t///// :/doi.. wrg/10...... 06536v33
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning. pp. 12888-12900 PMLR, 2022.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. arXiv preprint 1rXiv:2106.04555 2021.
Eliya Nachmani and Lior Wolf. Hyper-graph-netwoon decoders for block codes. In Advances in Neural Information Processing Systems, pp. 2326-2336, 2019.
Eliya Nachmani and Lior Wolf. Autoregressive belief propagation for decoding block codes. arXiv preprint 1rXiv:2103.11720 2021.
Eliya Nachmani, Yair Be' ery, and David Burshtein. Learning to decode linear codes using deep learning. In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 341-346. IEEE, 2016.
OpenAl. Gpt-4 technical report. arXiv preprint wrXiv:2303.08774 2023.
Judea Pearl. Probabilistic reasoning in intelligent systems: networks of plausible inference. Morgan kaufmann, 1988.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Nir Raviv, Avi Caciularu, Tomer Raviv, Jacob Goldberger, and Yair Be' Be'ery. perm2vec: Graph permutation selection for decoding of error correction codes using self-attention. arXiv preprint arXiv:2002.023 15, 2020.
Irving S Reed and Gustave Solomon. Polynomial codes over certain finite fields. Journal of the society for industrial and applied mathematics, 8(2):300-304, 1960.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint vrXiv:2205.06/75 2022.
Thomas J Richardson and Rüdiger L Urbanke. The capacity of low-density parity-check codes under message-passing decoding. IEEE Transactions on information theory, 47(2):599 618, 2001.
Thomas J Richardson, Mohammad Amin Shokrollahi, and Rüdiger L Urbanke. Design of capacity- approaching irregular low-density parity-check codes. IEEE transactions on information theory, (7(2):6196637 2001.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Con- ference Computer Vision and Patlern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-36494 2022.

---

Published as a conference paper at ICLR 2024
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXIv:2002.05222, 2020.
Ido Tal and Alexander Vardy. List decoding of polar codes. Theory, 61(5):2213-2226, 2015.
IEEE Transactions Information
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-5999889 2017.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint 0rXiv:2006.04768, 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837 2022.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing. Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533, PMLR, 2020.
12

---

Published as a conference paper at ICLR 2024
A ILLUSTRATION OF THE COMMUNICATION SYSTEM
We provide in Figure 6 an illustration of the proposed communication and decoding setting.
t= 1,... D
Xy (0, 1
xxd
1+1p
(yeh CR"
my E (0, 10
BPSK
Xxt + 2
fen
(&)) c
Encoders
2, (0, In
Universal Decoder
Figure 6: Illustration the communication system. the suggested foundation model setting, single decoder fo is proposed, capable of correcting any code at any length, given the code's 's parity- check matrix. Here, D denotes the number of different samples and noises, indexed by i, to be processed by the decoder.
B LOSS OF ADJACENCY INFORMATION IN ECCT
To illustrate the loss of connectivity information from the ECCT's mask we provide Figure 7 description of the graph reconstruction from the ECCT's mask for the (3, 1) repetition code. As can be seen, the Tanner graph reconstructed from the mask cannot describe graph connectivity ac- curately because of the hard distance (one and two-ring) thresholding. For this code, assuming connectivity between the variable nodes prior would allow faithful reconstruction.
ECCT Mask
%
X
C1
(a)
(b)
(c)
Figure 7: For the (3, 1) repetition code: (a) the original Tanner graph, (b) the induced ECCT's mask, (c) the "Tanner" graph derived from the mask.
C DESCRIPTION OF THE CODES
We present Figure the statistics of the codes used in this work. The codes were randomly sampled for the creation of the datasets.
TRAIN: BCHL1127.10633ssripp
TRAIN: BC
TRAIN: BCH_127_113.2.stipp
TRAIN: BCH.31_16...stipp.t
TRAIN: BCH_127_120_1_stig
TRAIN: BCH_31.21.2.strppp..
TRAIN: BCHL127 127_64_10
64_10.strrpp..
TRAIN: BCH 63.30 30.66,ttrpp..x
TRAIN: 1T..112777.9.ttrip..
TRAIN: BCH.63.36 63.66..55.sstrpp.
TRAIN: BCH.117.78.7_strppp
TRAIN: BCHL63_39_4.stripp
TRAIN: BCH_127_85 61778566_strip.xxt
TRAIN: BCH_63_51 2_strippst
TRAIN: JCH_127.99944sttt
TRAIN: BCH.7.4.1 str.pptttpp.xttt
TRAIN: BCH 15 11 strip.txt
TRAIN: Hopt BCH 127 64 10 140ones
13

---

Published as a conference paper at ICLR 2024
Code Length
Coree Rate
25
21.0
17.5
20
15.0
12.5
10.0
10
7,5
5.0
2.5
0.0
20
30
00
80
100
120
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Length
Rate
(a)
(b)
Tarner Graph Distance Histogram
Code Types
203000
175000
125000
100000
75000
50000)
21000
5
2
10
BCH
CCSDS
RS
LDPC
LTE
PolarCade
Distance
Codus (d)
(c)
Figure 8: We present statistics regarding the codes used in this work: the codes' lengths (a), the codes' rates (b), the Tanner graph distances (c), and the distribution of the code types.
TRAIN:
TRAIN: Www.Cooo.eN228.K6....
TRAIN: hww.Hoog..BCH..........7222....
TRAIN: DolarrCnde.N32,K111.xt...
TRAIN: PolarCnde.N332_KK66...bbb
TRAIN: PolarCode.N332.K22..xtbb.
TRAIN: HAAAAN,HoopBBCCLL..3333334400oo
TRAIN; PolarCode.NN32.K244xxbbx
TRAIN: WWW.HHHp...CHHH.3924433660oo.oo
TRAIN:
TRAIN: PodarCnde_N644.222.MMbbb.
TRAIN: Bop.
TRAIN: -0
TRAIN: PolarCode.NN4..433.xtbb.2
TRAIN: RS
TRAIN: PolarCode N64_ N64.KK88.
TRAIN: Hopt RS 15 6 208ones.alist
TEST: BCH 127 92 5 strip.txt
TRAIN: Hopt.RS RS. 15..555.222onss.ale
TEST: ICHL.15...2.srrppp
TRAIN: 100pp_RR.1157..28855.
alist
TEST: BCH-3 11..3311111__ppp.....
TRAIN: Hopt.RS Swi.RSS.5599 $5.993.3388nes....
TEST: BCH.68
TRAIN: Hopt RS 54ones.alist
TEST: BCH 63 57 strip.txt
TRAIN: Hopt.RS..
7.5.11 RS....R..7....
a8osess.aiis
TEST: CCSDS_Idpc.nll k64.alist
TRAIN: WwI.NI28KK-GG.....U
bi.list
TEST:
TRAIN: LDPC.NI28 K64.GF256.bi... bba
TEST: Hopt_BCH. CCCCH_12_9925_...
12..5.1148nne.allin
TRAIN: I.PPC N96 K48 GF256 d1 bialisst
TEST: Hopt BCH 63 30 alis
TRAIN: LDPC.N96.E 1148.GG644BBL...
TEST: Hopt.BCHL.3.
2.2288nne..alin
TRAIN: LDPC.N96.KA N.C..N96KK8.GF644... K48.GF64.bi...
TEST: PolarCode_NI
.t..eeN12.866.ee... 164.bxt.b22
TRAIN: LDPC_N99.KG (066K48_PPP 1.n0.dmin10..c
TEST: PolarCode N128.K86.1e K86.txt.b22
TRAIN: PolarCode N128 b43.
TEST: eBCH 128 64 strip.txt
14

---

Published as a conference paper at ICLR 2024
D LEARNED MAPPINGS VISUALIZATIONS
We present in Figure the values of the learned mapping of the Tanner graph distances for all the layers the proposed FECCT. In all layers, we see prominence to the mapping distance or distance 2, with layer being the only case in which these two do not have the highest values. This reminiscent of the fixed map in ECCT, where the first two values are mapped to one, and the rest to zero.
Layer
Layer
Layer
20
1.5
5-111
0.8
Distanoe
Distance
Distanoe
Layer
Layer
Laper 6
0.5
0.5
1.3
0.2
0.0
0.0
0.0
5
Distance
Distazoe
Distanoe
Figure 9: Absolute values of the learned mapping of the proposed FECCT with respect to distance for the six layers of the model.
E GENERALIZATION ANALYSIS
To illustrate the importance of dataset diversity, we provide in Table 4 comparison with a FECCT trained on single code only (POLAR(64,48)). The lack of generalization capacity of the model compared to the proposed training strategy evident. The performance on the code interest is only slightly better for the single code version.
Table 4: A comparison between the proposed fully trained FECCT and FECCT trained on the PO- LAR(64,48)) code only (FECCT-single), demonstrating the generalization benefits the proposed method. Zero-shot codes marked by star. Higher is better.
Method
FECCT
single
FECCT
4
4
6
P0LAR(64.48)
6.35 8.50 11.12
6,06 8.21 10.96
P0L,R(128,86)*
3.90 5.36
7.57
7.90
11.29
BCH(63,36)
4.01 5.42
7.30
4.53
6.38 9.10
BCH(66,51)*
4.65 6.35
8.73
5.71 8.07 11.31
Reed Volomon(21,155
4.254.63
4.97
4.56 6.83 10.51
Reed Solomon(60,57 (60,52)
3.68 3.81 3.77
7.47 10.24
CCSDS(128.64)
3.42
4.30
6.52 9.67 15.01
CCSDS
4.10
4.43
5.23 7.00 9.21
15

---

Published as a conference paper at ICLR 2024
F BER CURVES
We depict in Figure 10 the BER curves of the ECCT, the zero-shot FECCT, and the fine-tuned
FECCT.
BCH[63,
ECC
ECCT
10
Zero-Shot Fine luned
10
Zero-She Shot Fin lunia
10-1
10
BER
14
BER
10
10
10
10-1
10
3.0
5.0
Ea/No
Ea/No
R5(63, 52)
CCSDS(32, 16)
EOCT
ECCT
70
Zhro-Shot
Fine Tuned
10
Shot
10-3
BER
10
10-1
10-5
10-4
Collio
Co/No
Figure 10: BER curves of the ECCT, the zero-shot FECCT, and the fine-tuned FECCT for four different codes.
G SELF ATTENTION MAPS
We depict in Figure 12 the self-attention maps at the different layers of the model for different Es/No values averaged over 2048 sample batch. At each layer, we present the original self celf-attention map as well as the distance-filteeed filtered one as described in Eq (7). In Figure II] we present the sefl-attention maps for a given sample. We can observe that the attention is mostly focused on the syndrome values in order to perform the decoding, while in the final layer, the focus is also transferred to the information bits.
H EXPERIMENT ON LARGE CODE
We provide Figure BCH(1023,1013) code. This code is seven times the size of the largest code in training set, and 13 times the mean code length in the training set. The baseline taken from Helmling & Scholl (2016). Evidently, the zero-shot decoding is approximately one dB worse than maximum-likellinn likelihood decoding and better than the Hard Decision Decoding baseline.
the zero-shot performance of the pretrained FECCT on the
16

---

Published as a conference paper at ICLR 2024
Es/S0
Lowe
Eitrend lavee
E0/D0
544 ALL Leses
Filmil Bif Att Taver
S:f Amt Taoas
Filmel Solf ALL Tayer
Sef Art Layar
Fitern Self All Tager
Lave
Fileral delf Art Tayte
Sdf Art
Filrrel Geff
Self
Filtend
Ex/S0-6
El/00-4
El/20-5
Filroned
Figure 11: Absolute values of the self-attention maps the six different layers of the model for two different F No values for the BCH(63,51) code. At each layer, we provide the self-attention before and after the Tanner graph distance filtering. The self-attention tensor is averaged over the batch and head dimensions.
17

---

Published as a conference paper at ICLR 2024
Byy Momd
DySite
Eu/D0
No.. Laps
Filuaal 8-ff Adt. Laves
Sdf Alt Looa
Filized Self ALL Laass
Self All Laous
Filtued All. Baase
ScE
8dF
Filisaed Salf
Self
Filtoed
EB /30m6
E5/00w
Absolute values of the channel output values y followed by the corresponding absolute values of the self-attention maps at the six different layers of the model for two different Eb/Ny values for the BCH(63,51) code. At each layer, we provide the self-attention before and after the Tanner graph distance filtering. The self-attention tensor is averaged over the head dimension.
Figure 12:
18

---

Published as a conference paper at ICLR 2024
BCH(1023, 1013)
100
10 1
10 -
FER
10-3
10
Hard Decision Decoding
Zero-Shot FECCT
10-5 -
Maximum Likelihood
5.0
5.5
6.0
6.5
7.0
7.5
8.0
8.5
9.0
Eb/No
Figure 13: Zero-shot FER performance of the pretrained FECCT, Hard Decision Decoder and Moximum-likelihon decoders on the 1CH(1023,1013) code
19


Reviews:
[{"review_id": "1Oob4Qm7cH", "reviewer": "Reviewer_1rnF", "summary": "The authors proposed a model and is trained on multiple codes and can then be applied to an\nunseen code.\nTransformer architecture in multipleways: \n(1) a code-invariant initial embedding, which is also position- and lengthinvariant,\n(2) a learned modulation of the attention maps that is conditioned on the Tanner graph\n(3) a length-invariant code-aware noise prediction module that is based on the parity-check matrix", "questions": "1.Conclusion should be rewritten based on the results presented mentioning the future scope.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.Error control coding implemtation on the deep learning technique is highly encourgaing.\n2.Authors got the optimized Results in terms of BER.", "weaknesses": "1.Conclusion should be rewritten based on the results presented mentioning the future scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YdrpK0fTYU", "reviewer": "Reviewer_dPJy", "summary": "This work embarks on a very ambitious journey towards creating a foundation code for all downstream ECCs. The suggested structure here largely depends on a prior NeurIPS paper ([NeurIPS’22] Choukroun et al., Error Correction Code Transformer, presumably by the same authors) in that it is a specialized Transformer with bitwise embedding and parity-check-matrix-dependent masking, as appropriate for ECC. Albeit similar to prior work, the present proposal contains enough new materials and gives a highly convincing architecture based on code-aware aggregation that depends on the parity-check matrix as well as code-invariant bitwise embedding.", "questions": "Please respond to the mentioned weaknesses above.\n\nIn Tables 1 and 2, the proposed method seem noticeably worse than ECCT on larger codes. Also, In Table 3, ECCT+DM+II gives the best results. Explanations would be  good.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed idea is very ambitious, and is based on a highly innovative specialization of the Transformer to the classical problem of decoding received codewords of linear codes. The proposed ideas/strategies are very interesting and convincing (such as bitwise embedding independent of particular codes and incorporation of the parity check matrix in the embedding function). The impact on the field of digital communication and data storage could be large.", "weaknesses": "The main issue is that the training of the model is done using codes with lengths up to 150 only, hardly a sufficient length to reflect many modern codes of important applications (testing is also done on relatively small codes, with the largest being a 255 bit BCH). The popular LDPC codes are also curiously missing in the training as well as in the performance evaluation. Likewise for meaningfully long Polar codes. In this sense, I am not sure if the term “foundation model” is justified here. In sum, the idea seems very good, but the validation comes short of a reasonable expectation. I do not feel just saying \"we have limited computing resources\" would be a good enough excuse for such an ambitious title.\n\nWritings on various parts seem direct copies from [NeurIPS’22] Choukroun et al., Error Correction Code Transformer. Try to differentiate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8bY4TIg8Rt", "reviewer": "Reviewer_w8i1", "summary": "The paper attempts to develop foundation model for Error Correction Codes that is trained on large data so that it can be used later for any downstream task. Specifically, authors aim to adapt the Transformer input embedding for robustness to code length variations. To learn the code structure, they use the positional embedding, that is integrated into the self-attention via a learned mapping of the node distances in the Tanner graph. Moreover, for code awareness and channel noise prediction, the paper employs a size-invariant prediction module that is conditioned on the parity-check matrix. In simulations, they tested on codes that are unseen during training. They showed that the proposed FECCT method matches or sometimes perform better than the\nstate of art.", "questions": "please compare the proposed generalist decoder at short lengths (less than 200 bits) with that of specialized decoders at those lengths. Because as I pointed out this is where these foundational decoders would show value if any.\n\n-please plot error rate plots rather than the reporting used in the paper which is not insightful.\n\n-it would be helpful to compare your proposed work with that of\nChoukroun & Wolf (2022b) which extends and enhances ECCT via Denoising diffusion in error correction codes and have far superior performance than ECCT.\n\n-the paper novelty is arguable in light of ECCT design architecture. For the most part following similar development as ECCT. Can the authors elaborate on the novelty relative to ECCT.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-the paper takes a foundational approach to the decoding problem in error correcting codes, which is intellectually interesting. Clearly being able to decode any type of code is an interesting intriguing  exercise.\n- the paper advanced the design of generalist decoders relative to existing generalist decoders by using new embedding and grounding techniques.", "weaknesses": "-Although the design of foundational decoders are very interesting intellectual exercise, the real world impact of it is close to none if not zero.  The reason is that error correcting codes are designed and deployed once and their training is not a big deal even if someone takes deep neural network decoders as opposed to classical BP methods. But more importantly, there is another argument against the value of these generalist decoders: The important thing to recall is that capacity achieving codes exists for long codes and their BP decoders are close to ML performance, I.e., optimal decoders. So there is no gain of these deep neural decoders in long codes. The focus should be short codes for which we do not have good BP decoders. However, for short codes, we really do not need foundational decoders as one can design and easily train specialized neural decoders for the short codes that will very likely beat the performance of generalist decoders for all lengths. It is clear to believe that a generalist decoder will not be able to perform a specialized deep neural decoder for short lengths, unless the authors can show their generalist decoder can beat the performance of state of art short length (less that 200 bits) code decoders, specialized for that specific code length.\n\n-what is the performance in the tables? I see that 3 different values of Eb/N0 is used as the channel input signal (bit) power to noise ratio but What about reported numbers as performance. What are they? I like to see how these reported numbers translate to the error rate performance, as it is the only thing that matters in communication. It does not look like that the authors picked a particular error rate and report the corresponding Required Eb/N0 to achieve such an error rate. Because in that case the lower number is associated with the better scheme not the higher (as the authors stated in the paper).\n\n-the authors need to compare their scheme with Choukroun & Wolf (2022b) which is shown to be superior to ECCT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wzgl5HEZwI", "reviewer": "Reviewer_jdxb", "summary": "The paper builds on top of Error Correction Code Transformer(ECCT), made a few structural change to build a \"foundational\" ECCT (FECCT) model for block code decoding. \n\nFECCT is a generalized version of ECCT:\n\n(1) FECCT is not length-dependent, H-dependent, and not even code-dependent, thus can be trained once can be used for wide range of block codes, with good performance (matching/beating ECCT mostly, and beat BP by a large margin.). Some finetuning can even lead to better performance.\n\n(2) FECCT could an important milestone of deploying neural decoders to real world given its potential, still a lot of hard work is required till that day.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(1) Overall, FECCT is very plausible method, since FECCT is more like an \"decoding algorithm\" rather than simply \"neural decoder\". My definition of \"decoding algorithm\" means the input is H matrix  and received codeword  (just like BP algorithm), and the output should be decoded message. While other neural decoder has dependency on code/length/H/etc. As of my understanding, FECCT has learned some interesting advanced BP-like algorithm, that can be beneficial for a wide family of block codes.\n\n(2) FECCT is built on top of ECCT, the generalization performance on non-zero codewords are preserved, which makes training feasible. FECCT's H-dependent attention is a generalized version of ECCT's attention, which lead to better decoding capability. The proposed neural structure makes sense, and lead to good generalization performance.\n\n(3) The experiment on unseen code with different code family and block length are interesting, which make (1)'s claim stronger that FECCT is more of an \"decoding algorithm\".\n\nOverall, this becomes an interesting work, at least for neural decoder research, first time shows that a \"decoding algorithm\" rather than a complicated mapping can be learned.", "weaknesses": "1. The experiments are mostly built on short block codes (<128, test unseen for 255 at most), while typical capacity-approaching codes such as QC-LDPC has much longer block length. Performance on long block length is going to make this paper stronger, due to long block code's capacity-approaching performance.\n\n2. Interpretability: FECCT should have been learned some interesting algorithm, that can be interpreted as an advanced version of BP. We do see some part of interpretations in the appendix, but not solid enough to get insight on what FECCT's algorithm means.\n\n3. Complexity of network: attention-based neural network are very complex. Deploying the FEECT to any real world production requires some hard work on complexity reduction. In its current form, I am not seeing FECCT can be deployed to modem in short time.  Note that channel coding are heavily used in all modern wireless communication systems, which requires minimal latency, high throughput, and low cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)