SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and in the next stage, the weights of the subsequent  $\Delta_{1}$  iterations are trained. At this point, the weight values of previous  $\Delta_{2}$  iterations, which are already trained in the first stage, are further trained by taking the result of the first stage training as its initial state. This retraining, which is not used in the iter-by-iter training schedule [26], assists in preventing the learning process from falling into a local minimum. Note that the method of one-shot training [5] corresponds to the case of  $\Delta_{1} = \ell_{2}, \Delta_{2} = 0$ , and the iter-by-iter training schedule [26] is equivalent to the case of  $\Delta_{1} = 1, \Delta_{2} = 0$ .
We employ a greedy approach to determine the optimal values for  $\Delta_{1}$  and  $\Delta_{2}$ , resulting that  $\Delta_{1} = 5$ ,  $\Delta_{2} = 10$  offers superior performance in terms of the test FER. Fig. 4(b) shows the evolution of test FER as a function of the iteration number. In the case of one-shot training [5], the vanishing gradient problem hinders training the weights of earlier iterations, so the test FER stays roughly the same until iteration 40 and starts to fall thereafter. The same behavior is observed for  $\bar{\ell} = 40$ . Thus, the test FERs of  $\bar{\ell} = 40$  and  $\bar{\ell} = 50$  are almost the same. Next, since the iter-by-iter training schedule [26] finds the optimal weight for each individual iteration, the test FER falls even at the first iteration of the post decoder (i.e.,  $\ell = 21$ ) without encountering the vanishing gradient problem. However, this local optimization leads to a degraded local minimum, and consequently, the test FER gradually and slowly decreases with each iteration. Likewise, the multi-loss method [5] shows a similar result. In contrast, the block-wise training schedule with  $\Delta_{1} = 5$ ,  $\Delta_{2} = 0$  shows a superior test FER at iteration 25 compared to the other training schedules because it results in a better solution by training the weights of multiple iterations simultaneously. Moreover, the schedule with retraining (i.e.,  $\Delta_{1} = 5$ ,  $\Delta_{2} = 10$ ) outperforms the schedule without retraining (i.e.,  $\Delta_{1} = 5$ ,  $\Delta_{2} = 0$ ) at iteration 30 though it shows a worse result at iteration 25. This implies that through retraining, the weights of intermediate iterations have been adjusted to preprocess error patterns thereby leading to stronger correction capabilities in the final iteration. As a result, at the maximum of 50 iterations, the proposed training schedule with  $\Delta_{1} = 5$ ,  $\Delta_{2} = 10$  provides the better test FER value compared to the other training schedules as shown in Fig. 4(b): 0.11 for the block-wise, 0.16 for the multi-loss, 0.18 for the one-shot, 0.37 for the iter-by-iter, .
3.3 Weight sharing technique using UCN weights
Assuming the techniques proposed thus far (in detail, uncorrected words at  $\mathrm{E_b / N_0} = 4.5$  dB and training schedule with  $\Delta_{1} = 5,\Delta_{2} = 10$ ) are used, we compare the weight sharing techniques
in Fig. 5. Compared to the full diversity weights, the spatial and temporal sharing techniques significantly reduce the number of distinct weights, but cause performance degradation. In contrast, the proposed sharing technique that introduces a new weight type called UCN weight shows almost identical performance while using only about  $2.6\%$  of the weights compared to the full diversity weights. The proposed sharing technique assigns different weights to UCNs and SCNs as shown in Fig. 5. This is feasible because the decoder knows whether a CN satisfies the check equation or not. Using the spatial sharing technique and distinguishing between SCN weight  $w^{(\ell)}$  and UCN weight  $\hat{w}^{(\ell)}$ , the proposed sharing technique can be represented as  $\{\overline{w}^{(\ell)}, w^{(\ell)}, \hat{w}^{(\ell)}\}_{\ell}$  for iteration  $\ell$  and the total number of distinct weights becomes  $3\ell_{2}$ .
Techniques using different weights for UCNs and SCNs have also been proposed in [33, 41]. However, the work [41] uses only one suppression factor  $\rho$  to represent the UCN weight (i.e.,  $\hat{w}^{(\ell)} = (1 + \rho)w^{(\ell)})$ , making the UCN weight dependent on the CN weight. As a result, due to the limited degree of freedom for the UCN weight, it is difficult to obtain the decoding diversity for effectively removing various types of error patterns. Moreover, in [33], if at least one of the CNs belonging to a single proto CN is unsatisfied, all  $z$  CNs from the proto CN are weighted by the UCN weight. This approach, which applies the same weight to a large number of CNs tied together at the proto level, is not suitable for correcting words with a small number of UCNs, because it does not separately handle individual CNs like the proposed method.
4 Performance evaluation
Figure 6: FER performances of (a): WiMAX LDPC (length 576, rate 3/4), (b): IEEE802.11n LDPC (length 648, rate 5/6), (c): 5G LDPC (length 552, rate 1/2) codes.
In this section, we compare the proposed and other conventional decoding schemes in terms of the decoding performance. All simulations are performed by NVIDIA GeForce RTX 3090 GPU and AMD Ryzen 9 5950X 16-Core Processor CPU. For training, the weights are trained by the Adam optimizer [42] with a learning rate of 0.001. We evaluate the decoding performance using Monte Carlo methods with at least 500 uncorrected words for each FER point. Fig. 6(a) shows the FER performances of the proposed scheme for the WiMAX LDPC code. The proposed scheme incorporates the i) boosting learning with uncorrected words, ii) block-wise training schedule with retraining, and iii) spatial weight sharing with UCN weights. The performance is compared with MS decoding, WMS decoding, and existing NMS decoding schemes in [5, 26]. Among the neural decoder studies listed in Table 1, we exclude comparison with the studies that use FAID and layered decoding, or that require enumerating trapping sets and absorbing sets. In addition, we choose not to compare with augmented neural networks [6, 34] since our approach does not increase model complexity to deal with the low-error-rate region of long codes. A comparative analysis for short codes in the waterfall region can be found in the appendix.
For the NMS decoding schemes in [5, 26], the base decoder is used for iterations from 1 to 20 like the proposed scheme, and the training methods introduced in [5, 26] are employed for the post stage. The full diversity weights are used for the schemes in [5, 26] to observe their best performance. For the scheme in [5], received words in the waterfall region  $(\mathrm{E_b / N_0}~2 - 4~\mathrm{dB})$  are used as training samples, and the weights for the post stage are trained all at once without a training schedule. For the scheme
Table 2: Complexity comparison for the WiMAX LDPC of  $(N,M,E,z,\alpha) = (24,6,88,24,15.6)$ , where  $\alpha$  is the comparison count for each CN (i.e.,  $\alpha = d_{c} + \lceil \log d_{c} \rceil - 2$  for the CN degree  $d_{c}$  [13]).
Complexity per iteration	Total complexity (A + 2C + M) ℓ	Memory for weights
Addition A	Comparison C	Multiplication M
MS, ℓ = 50	2Ez = 4224	αMz = 2256	Ez = 2112	542400	0
NMS [5, 26], ℓ = 50	(2E + N)z = 4800	676800	3760
Proposed NMS, ℓ = 50	130
Post processing [22] ℓ = 150	Ez = 2112	1627200	0
in [26], received words from the  $\mathrm{E_b / N_0}$  points where the MS decoder achieves bit error rate of  $10^{-3}$  are used as training samples, and the iter-by-iter training schedule is employed. The remaining hyper-parameters are set in the same way as in the proposed scheme. As shown in Fig. 6(a), the conventional NMS decoders in [5, 26] show good performance in the waterfall region  $(\mathrm{E_b / N_0}2 - 4$  dB), but the error-floor occurs from  $4\mathrm{dB}$ . This is because the training samples are composed of received words without filtering. In contrast, the proposed scheme shows excellent performance in both the waterfall and error-floor regions, and the error-floor phenomenon is barely noticeable down to FER of  $10^{-7}$ . In particular, comparing the results of  $\bar{\ell} = 20$  and  $\bar{\ell} = 50$ , it is confirmed that the post decoder successfully removes the error-floor.
In addition, we compare the proposed scheme with the state-of-the-art post processing scheme in [22]. We directly reference the simulation results from [22]. As shown in Fig. 6(a), the scheme in [22] shows a similar or worse performance to the proposed scheme, but it has disadvantages of having very high decoding complexity and latency since it consumes a large number of iterations  $\overline{\ell} = 150$ . Table 2 compares the schemes in terms of the decoding complexity. The NMS decoder has more multiplications than the MS decoder by  $(E + N)z$  due to the weighting operation. The number of other operations is the same as in the MS decoder. Total complexity is evaluated with assumption that the comparison  $C$  is as twice as complex than the addition  $A$  and multiplication  $M$  [43]. The additional memory for storing the weights of the proposed scheme is  $3\overline{\ell}$  which is much lower than those of [5, 26] which exploit full weight diversity. Since the scheme in [22] does not use weighting, the complexity per iteration is lower than the proposed NMS scheme, but the total complexity is more than twice as high as the proposed NMS scheme due to the higher number of iterations. Moreover, additional complexity is required for the error path detector [22]. In Fig. 6(b), (c), similar results are observed for the IEEE802.11n LDPC and 5G LDPC codes, where the proposed scheme outperforms the other schemes and achieves an FER of  $10^{-7}$  without a severe error-floor.
This paper proposed training methods for the NMS decoder of LDPC codes to enhance the error-floor performance. Using uncorrected words from the base decoder, we trained the post decoder to be specialized for error patterns causing the error-floor, promoting decoding diversity in the cascaded base and post decoders. We also proposed a training schedule to circumvent the vanishing gradient and local minimum problems, and a weight sharing technique that significantly reduces the number of distinct weights without sacrificing performance. The proposed NMS decoder using the trained weights showed the excellent waterfall and error-floor performances for several standard LDPC codes. Along with the performance improvement, the proposed training scheme has the advantage of being flexibly applicable regardless of the types of channel, code, and decoding algorithm. This scheme can also be implemented directly on hardware architectures without additional costs, and can be directly utilized with no prior analysis of the target code and decoding algorithm.
6 Acknowledgments
This work was supported by Samsung Electronics Co., Ltd (IO230411-05859-01), by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government [2021-0-00746, Development of Tbps wireless communication technology], by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00212103),
and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00247197).
[1] Jeshua Bruck and Mario Blaum. Neural networks, error-correcting codes, and polynomials over the binary n-cube. IEEE Transactions on Information Theory, 35(5):976-987, 1989.
[2] William R Caid and Robert W Means. Neural network error correcting decoders for block and convolutional codes. In [Proceedings] GLOBECOM'90: IEEE Global Telecommunications Conference and Exhibition, pages 1028-1031. IEEE, 1990.
[3] LG Tallini and P Cull. Neural nets for decoding error-correcting codes. In IEEE Technical Applications Conference and Workshops. Northcon/95. Conference record, page 89. IEEE, 1995.
[4] Tobias Gruber, Sebastian Cammerer, Jakob Hoydis, and Stephan ten Brink. On deep learning-based channel decoding. In 2017 51st Annual Conference on Information Sciences and Systems (CISS), pages 1-6. IEEE, 2017.
[5] Eliya Nachmani, Elad Marciano, Loren Lugosch, Warren J Gross, David Burshtein, and Yair Be'ery. Deep learning methods for improved decoding of linear codes. IEEE Journal of Selected Topics in Signal Processing, 12(1):119-131, 2018.
[6] Yoni Choukroun and Lior Wolf. Error correction code transformer. In Advances in Neural Information Processing Systems.
[7] Shuai Shao, Peter Hailes, Tsang-Yi Wang, Jwo-Yuh Wu, Robert G Maunder, Bashir M Al-Hashimi, and Lajos Hanzo. Survey of turbo, LDPC, and polar decoder ASIC implementations. IEEE Communications Surveys & Tutorials, 21(3):2309-2333, 2019.
[8] IEEE standard for local and metropolitan area networks-part 16: Air interface for fixed and mobile broadband wireless access systems amendment 2: Physical and medium access control layers for combined fixed and mobile operation in licensed bands and corrigendum 1. IEEEStandard 802.16e, 2020.
[9] Technical specification group radio access network; NR; multiplexing and channel coding (release 17), document ts 38.212 v17.5.0. 3rd Generation Partnership Project, 2023.
[10] Tom Richardson and Shrinivas Kudekar. Design of low-density parity check codes for 5G new radio. IEEE Communications Magazine, 56(3):28-34, 2018.
[11] Tom Richardson and Ruediger Urbanke. Modern Coding Theory. Cambridge University Press, 2008.
[12] Een-Kee Hong, Inkyu Lee, Byonghyo Shim, Young-Chai Ko, Sang-Hyo Kim, Sangheon Pack, Kyunghan Lee, Sunwoo Kim, Jae-Hyun Kim, Yoan Shin, et al. 6G R&D vision: Requirements and candidate technologies. Journal of Communications and Networks, 24(2):232-245, 2022.
[13] William Ryan and Shu Lin. Channel Codes: Classical and Modern. Cambridge University Press, 2009.
[14] Guiqiang Dong, Ningde Xie, and Tong Zhang. On the use of soft-decision error-correction codes in NAND flash memory. IEEE Transactions on Circuits and Systems I: Regular Papers, 58(2):429-439, 2010.
[15] Shubham Chandak, Kedar Tatwawadi, Billy Lau, Jay Mardia, Matthew Kubit, Joachim Neu, Peter Griffin, Mary Wootters, Tsachy Weissman, and Hanlee Ji. Improved read/write cost tradeoff in DNA-based data storage using LDPC codes. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 147-156. IEEE, 2019.
[32] Xin Xiao, Nithin Raveendran, Bane Vasic, Shu Lin, and Ravi Tandon. FAID diversity via neural networks. In 2021 11th International Symposium on Topics in Coding (ISTC), pages 1-5. IEEE, 2021.
[33] Nemin Shah and Yash Vasavada. Neural layered decoding of 5G LDPC codes. IEEE Communications Letters, 25(11):3590-3593, 2021.
[34] Eliya Nachmani and Lior Wolf. Hyper-graph-network decoders for block codes. Advances in Neural Information Processing Systems, 32, 2019.
[35] Sebastian Cammerer, Jakob Hoydis, Fayçal Aït Aoudia, and Alexander Keller. Graph neural networks for channel decoding. In 2022 IEEE Globecom Workshops (GC Wkshps), pages 486-491. IEEE, 2022.
[36] Eliya Nachmani and Lior Wolf. Autoregressive belief propagation for decoding block codes. arXiv preprint arXiv:2103.11780, 2021.
[37] Marc PC Fossorier. Quasicyclic low-density parity-check codes from circulant permutation matrices. IEEE transactions on information theory, 50(8):1788-1793, 2004.
[38] Jeremy Thorpe. Low-density parity-check (LDPC) codes constructed from protographs. IPN progress report, 42(154):42-154, 2003.
[39] Marc PC Fossorier, Miodrag Mihaljevic, and Hideki Imai. Reduced complexity iterative decoding of low-density parity check codes based on belief propagation. IEEE Transactions on Communications, 47(5):673-680, 1999.
[40] Jinghu Chen, Ajay Dholakia, Evangelos Eleftheriou, Marc PC Fossorier, and Xiao-Yu Hu. Reduced-complexity decoding of LDPC codes. IEEE Transactions on Communications, 53(8):1288-1299, 2005.
[41] Xiaofu Wu, Yue Song, Ming Jiang, and Chunming Zhao. Adaptive-normalized/offset min-sum algorithm. IEEE Communications Letters, 14(7):667-669, 2010.
[42] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[43] Heshani Gamage, Nandana Rajatheva, and Matti Latva-Aho. Channel coding for enhanced mobile broadband communication in 5G systems. In 2017 European Conference on Networks and Communications (EuCNC), pages 1-6. IEEE, 2017.
[44] Yury Polyanskiy, H Vincent Poor, and Sergio Verdu. Channel coding rate in the finite blocklength regime. IEEE Transactions on Information Theory, 56(5):2307-2359, 2010.
[45] Shaohua Yang, Yang Han, Xuebin Wu, Roger Wood, and Rick Galbraith. A soft decodable concatenated LDPC code. IEEE Transactions on Magnetics, 51(11):1-4, 2015.
[46] Jieun Oh, Jeongseok Ha, Hyegyeong Park, and Jaekyun Moon. RS-LDPC concatenated coding for the modern tape storage channel. IEEE Transactions on Communications, 64(1):59-69, 2015.
[47] Hyegyeong Park and Jaekyun Moon. Improving SSD read latency via coding. IEEE Transactions on Computers, 69(12):1809-1822, 2020.
7.1 A better trade-off between the waterfall and error-floor performances
(b)
Figure 7: (a): Decoding performance with optimized/non-optimized codes and floating/quantized decoders (b): Comparison between the two networks and single large network.
Typically, LDPC code design assumes the availability of a floating-point decoder and optimizes the code to maximize waterfall performance. In Fig. 7(a), it's evident that the optimized WiMAX LDPC code outperforms the non-optimized regular LDPC code in terms of the waterfall performance when the floating-point MS decoder is employed. Additionally, the WMS decoder enhances the performance by assigning weights. As shown in Fig. 7(a), WMS decoding of the WiMAX LDPC code exhibits the best waterfall performance when floating-point operations are available. However, for practical hardware implementation, quantization of decoding messages is essential, but it leads to performance degradation. Notably, WMS decoding suffers substantial performance degradation with an early onset of the error-floor. A side-by-side comparison of the quantized WMS decoding performances between the non-optimized and WiMAX LDPC codes reveals that the performance loss due to quantization is more pronounced in optimized LDPC codes. Consequently, there's a need for a method that ensures high performance in both the waterfall and error-floor regions with low complexity. This work accomplishes this goal by leveraging machine learning techniques. In Fig. 7(a), the proposed quantized NMS decoder achieves the floating WMS decoding performance for the WiMAX LDPC code in both waterfall and error-floor regions.
7.2 Boosting learning of two networks v.s. conventional learning of a single large network
In Fig. 7(b), we compare the proposed method with the method that uses a single decoder, which is trained by a large single network at once. For the proposed method, the boosting learning is employed with two networks. Both have the same model size. Fig. 7(b) shows that simply increasing the model size doesn't significantly mitigate the error-floor. In contrast, the two-stage network using boosting learning achieves greater decoding diversity and demonstrates superior efficiency in alleviating the error-floor.
7.3 Optimization of parameters of the block-wise training schedule
Table 3: Test FER values of the block-wise training with various  ${\Delta }_{1}$  and  ${\Delta }_{2}$  .
Δ1\ Δ2	0	5	10	15	30
1	0.376	0.144	0.123	0.122	0.150
5	0.193	0.134	0.112	0.127	0.137
10	0.142	0.142	0.13	0.128	0.13
30	0.179
To fine-tune the parameters  $\Delta_{1}$  and  $\Delta_{2}$  used in the block-wise training schedule, we compare the test FER values across a range of  $\Delta_{1}$  and  $\Delta_{2}$ . The comparison is done for the post decoder training
with  $\ell_2 = 30$  and is summarized in Table 3. For the case where  $\Delta_2 = 0$  (i.e., without retraining), the performance improves as  $\Delta_1$  increases from 1 to 10. This can be attributed to the training of more weights simultaneously, aiding in escaping from local minima. However, if  $\Delta_1$  becomes too large, the weights in front are not adequately trained due to the vanishing gradient problem, causing performance degradation. In addition, as the retraining iteration number  $\Delta_2$  grows for a given  $\Delta_1$  value, a similar trend is noted: performance enhances up to a certain point and then plateaus. According to Table 3, the best performance is achieved with  $\Delta_1 = 5$  and  $\Delta_2 = 10$ .
7.4 Discussion on the trained weights
Table 4: Trained weights
\( \ell \)	Weights
\( \mathrm{VW}\overline{w}^{(\ell)} \)	1 ~ 10	0.74	0.98	0.96	1.10	1.20	1.15	1.22	1.19	1.17	1.09
11 ~ 20	1.15	1.09	1.07	1.04	1.05	1.07	1.00	0.98	0.90	0.81
21 ~ 30	1.14	0.97	0.93	0.69	0.66	0.58	0.57	0.51	0.49	0.45
31 ~ 40	0.48	0.41	0.37	0.38	0.38	0.39	0.32	0.33	0.32	0.34
41 ~ 50	0.39	0.37	0.31	0.31	0.34	0.45	0.39	0.37	0.42	0.42
\( CW w^{(\ell)} \)	1 ~ 10	0.74	0.71	0.66	0.69	0.67	0.70	0.75	0.71	0.73	0.76
11 ~ 20	0.73	0.70	0.77	0.79	0.84	0.84	0.86	0.82	0.81	0.97
21 ~ 30	0.19	0.30	0.54	0.58	0.62	0.59	0.73	0.73	0.74	0.78
31 ~ 40	0.72	0.78	0.79	0.81	0.88	0.82	0.88	0.85	0.97	0.79
41 ~ 50	0.81	0.82	0.85	0.91	1.19	0.86	0.96	1.10	1.18	1.62
\( UCW\hat{w}^{(\ell)} \)	1 ~ 10	0.74	0.71	0.66	0.69	0.67	0.70	0.75	0.71	0.73	0.76
11 ~ 20	0.73	0.70	0.77	0.79	0.84	0,84	0.86	0.82	0.81	0.97
21 ~ 30	0.58	0.80	0.83	0.73	0.81	0.86	0.72	0.78	0.73	0.73
31 ~ 40	0.59	0.71	0.64	0.67	0.67	0.72	0.65	0.65	0.66	0.70
41 ~ 50	0.77	0.70	0.72	0.81	0.92	0.58	0.71	0.76	0.87	1.57
(b)
Figure 8: Evolution of the weights and the number of samples as a function of iteration.
Table 4 shows the resulting weights trained using the proposed training methods. Owing to the weight sharing technique, only three types of weights, VW  $\overline{w}^{(\ell)}$ , CW  $w^{(\ell)}$ , and UCW  $\hat{w}^{(\ell)}$ , exist for each iteration. Thus, weight variations across iterations can be depicted in two dimensions, as shown in Fig. 8(a), enabling analysis of the trained results. Without the weight sharing technique, there would be  $(N + E)$  weights per iteration, making a 2D graphical representation unfeasible. Fig. 8(a) shows that the weights for the base decoding stage, where the CW and VW are roughly 0.75 and 1, are similar to the weights of conventional WMS decoding. As a result, the waterfall performance of WMS decoding and NMS decoding are nearly indistinguishable. The NMS decoding technique offers minimal advantage in the waterfall region.
On the other hand, for the post decoding stage, the weights undergo significant changes. At iteration 21, the VW increases while the CW decreases substantially. This means that the channel LLR values are given more weight when performing the sum operation at each VN, while the messages coming from CNs are attenuated. As a result, the decoding process reverts to the initial decoding state (with
a high error rate). This leads to an increase in the number of samples having more than 6 errors (denoted by  $e > 6$ ), as shown in Fig. 8(b), and a decrease in the number of samples with  $e \leq 6$ . This intentional regression aims to break free from trapped error patterns. From iteration 22 onward, the CW gradually increases, the VW decreases, and the count of  $e \leq 6$  samples starts to increase again. After iteration 27, the CW and VW show less variations. We term the period from iteration 21 to 26 as Phase 1 and the period following iterations as Phase2. At iteration 20, the numbers of samples with  $e \leq 6$  and  $e > 6$  are roughly equal, but through Phase 1, samples with  $e > 6$  transform into samples with  $e \leq 6$ . Consequently, by the end of Phase 1, there are more samples with  $e \leq 6$  than samples with  $e > 6$ . The small-sized error samples diminished steadily in Phase 2.
The result without Phase 1 is also shown in Fig. 8(b). Without Phase 1, the process of transforming into small-sized error samples is omitted. Consequently, the decoding proceeds in a situation where the number of samples with  $e \leq 6$  and  $e > 6$  is similar. Although Phase 2 effectively correct most of the  $e \leq 6$  samples, the  $e > 6$  samples remain in significant numbers as they are initially abundant. This indicates the need for the pre-processing of Phase 1.
7.5 Multiple stage decoders