SlowGuess/ACL_2020_OCR huggingface dataset:
As discussed in Section 3.2.1, the dynamic termination mechanism is used to train our RecoverSAT model to learn to determine segment length dynamically conditioned on target-side context such that it is recoverable from multi-modality related errors. In this section, we investigate the effect of this mechanism and the results are shown in Table 3.
As multi-modality related errors generally manifest as repetitive or missing tokens in the translation, we propose two quantitative metrics "Rep" and "Mis" to measure these two phenomenons respectively. "Rep" is defined as the relative increment of repetitive token ratio w.r.t. to a reference AT model. And "Mis" is defined as the relative increment of missing token ratio given the references w.r.t. to a reference AT model. Formally, given the translations  $\hat{\mathbf{Y}} = \{\hat{\mathbf{y}}^1\cdots\hat{\mathbf{y}}^k\cdots\}$  produced by the model to be evaluated and the translations  $\hat{\mathbf{Y}}_{auto} = \{\hat{\mathbf{y}}_{auto}^1\cdots\hat{\mathbf{y}}_{auto}^k\cdots\}$  produced by the reference AT model, "Rep" is defined as
$$
\operatorname {R e p} = \frac {r (\hat {\mathbf {Y}}) - r (\hat {\mathbf {Y}} _ {\text {a u t o}})}{r (\hat {\mathbf {Y}} _ {\text {a u t o}})}, \tag {7}
$$
$$
r (\mathbf {Y}) = \frac {\sum_ {k} \sum_ {j = 2} ^ {| \mathbf {y} ^ {k} |} \mathbb {1} \left(\sum_ {i = 1} ^ {9} \mathbb {1} \left(\mathbf {y} _ {j} ^ {k} = \mathbf {y} _ {j - i} ^ {k}\right) \geq 1\right)}{\sum_ {k} | \mathbf {y} ^ {k} |}, \tag {8}
$$
where  $\mathbb{1}(\text{cond}) = 1$  if the condition cond holds otherwise  $0$ , and  $\mathbf{y}_j^k$  is the  $j$ -th token of the translation sentence  $\mathbf{y}^k$ .
Given  $\hat{\mathbf{Y}}$ ,  $\hat{\mathbf{Y}}_{auto}$  and references  $\bar{\mathbf{Y}} = \{\bar{\mathbf{y}}^1 \cdots \bar{\mathbf{y}}^k \cdots\}$ , "Mis" is defined as
$$
\operatorname {M i s} = \frac {m (\hat {\mathbf {Y}} , \bar {\mathbf {Y}}) - m (\hat {\mathbf {Y}} _ {\text {a u t o}} , \bar {\mathbf {Y}})}{m (\hat {\mathbf {Y}} _ {\text {a u t o}} , \bar {\mathbf {Y}})}, \tag {9}
$$
p	BLEU	Rep	Mis	Step
NAT	24.57	50.09	9.09	1
RecoverSAT(K=10)	0.0	27.09	22.05	6.95	4.2
0.5	29.80	12.69	3.96	5.5
1.0	29.89	13.00	4.75	7.2
1→0	29.90	7.09	3.56	5.1
where  $m(\cdot ,\cdot)$  computes the missing token ratio and is defined as follows:
$$
\begin{array}{l} c _ {w} (\mathbf {y} ^ {k}, \bar {\mathbf {y}} ^ {k}) = \max  \left(c (\bar {\mathbf {y}} ^ {k}, w) - c (\mathbf {y} ^ {k}, w), 0\right), \\ m (\mathbf {Y}, \bar {\mathbf {Y}}) = \frac {\sum_ {k} \sum_ {w \in \bar {\mathbf {y}} ^ {k}} c _ {w} \left(\mathbf {y} ^ {k} , \bar {\mathbf {y}} ^ {k}\right)}{\sum_ {k} | \bar {\mathbf {y}} ^ {k} |}, \tag {10} \\ \end{array}
$$
where  $c(\mathbf{y},w)$  is the occurrence number of a token  $w$  in the sentence  $\mathbf{y}$ .
From Table 3, we can observe that: (1) By using the dynamic termination mechanism  $(p = 0.5$ ,  $1.0, 1 \to 0$ , where  $p$  is the parameter of Bernoulli distribution (Eq. 5)), both repetitive and missing token errors are reduced ("Rep" & "Mis"), and the BLEU scores are increased, indicating the effectiveness of the mechanism; (2) As  $p$  grows larger, the average number of decoding steps ("Step") increases significantly. The reason is that more target sentences are divided into segments equally with smaller  $p$  during training and the model is biased to generate segments with similar lengths. However, if the model is not exposed to randomly divided segments  $(p = 0.0)$ , it fails to learn to recover from multi-modality related errors and the BLEU score drops significantly. (3) By using the annealing dividing strategy  $(p = 1 \to 0$ , see Section 3.2.1), we achieve a good balance between decoding speed and translation quality. Therefore, we use it as the default setting in this paper.
4.6 Effect of Segment Deletion Mechanism
In this section, we investigate the effect of the segment deletion mechanism and the results are shown in Table 4, where  $q$  is the probability of injecting pseudo repetitive segments to each training instance. From the results we can observe that: (1) Without using the segment deletion mechanism
Table 3: Effect of the dynamic termination mechanism. The results are evaluated on the IWSLT16 En-De validation set.  $p$  is the parameter of Bernoulli distribution in Eq. 5. "Rep" and "Mis" measure the relative increment (\%) of repetitive and missing token ratios (see Section 4.5), the smaller the better. "Step" denotes the average number of decoding steps. And "1→0" denotes annealing  $p$  from 1 to 0 linearly.
q	BLEU	Rep	Step
NAT	24.57	50.09	1
RecoverSAT(K=10)	0.0	28.56	26.24	4.4
0.1	29.73	5.11	4.7
0.3	29.61	7.71	5.1
0.5	29.90	7.09	5.1
0.7	29.76	11.47	5.2
0.9	29.25	21.38	5.3
1.0	29.13	20.55	5.2
Table 4: Effect of segment deletion mechanism. The results are evaluated on the IWSLT16 En-De validation set.  $q$  is the probability of injecting pseudo repetitive segments to each training instance (see Section 3.2.2).
Figure 2: Translation quality on the IWSLT16 En-De validation set over sentences in different length.
$(q = 0)$ , the BLEU score drops significantly and the repetitive token errors ("Rep") increase drastically, indicating that the mechanism is effective for recovering from repetitive token errors. (2) As  $q$  grows larger, the average number of decoding steps ("Step") increases steadily because the model is misled that to generate then delete a repetitive segment is expected. Thus,  $q$  should not be too large. (3) The repetitive token errors ("Rep") increase drastically when  $q > 0.7$ . We believe that the reason is that the pseudo repetitive segments are constructed randomly, making it hard to learn the underlying mapping. (4) The model achieves the best performance with  $q = 0.5$ . Therefore, we set  $q = 0.5$  in our experiments.
4.7 Performance over Sentence Lengths
Figure 2 shows the translation quality of the Transformer, our RecoverSAT model with  $K = 10$  and NAT on the IWSLT16 En-De validation set bucketed by different source sentence lengths. From the figure, we can observe that RecoverSAT surpasses NAT significantly and achieves comparable performance to the Transformer on all length buckets, which indicates the effectiveness of our model.
Source	die er-greif_endste Abteilung ist das Denk_mal für die Kinder , das zum Ged_enken an die 1,5 Millionen Kinder , die in den Konzent_rations_lagern und Gas_k_ammern vernichtet wurden , erbaut wurde .
Reference	the most tragic section is the children's mem_orial , built in memory of 1.5 million children killed in concentration camps and gas cham_bers .
NAT	Translation	the most tangible department department the monument monument the children , which was built commem_commem_orate 1.5 1.5 million children were destroyed in the concentration camps and gas cham_bers .
RecoverSAT (K = 10)	Translation	A: [1] the EOS [2] most tangible department is the EOS [3] monument for children EOS [4] built to EOS [5] commem_orate the 1.5 EOS [6] million children destroyed EOS [7] in the concentration camps and EOS [8] in DEL [9] gas EOS [10] cham_bers . EOS
Forced Translation	B: [1] the EOS [2] most tangible department is the EOS [3] monument for children EOS [4] built to EOS [5] commem_orate EOS [6] the 1.5 million children destroyed EOS [7] in the concentration camps and EOS [8] in DEL [9] gas EOS [10] cham_bers . EOS
C: [1] the EOS [2] most tangible department is the EOS [3] monument for children EOS [4] built to EOS [5] commem_ate the 1.5 million children EOS [6] destroyed EOS [7] in concentration camps and EOS [8] in DEL [9] gas EOS [10] cham_bers . EOS
D: [1] the EOS [2] most tangible department is the EOS [3] monument for children EOS [4] built to EOS [5] commem_ate the 1.5 million children destroyed EOS [6] in the concentration camps and EOS [7] in the DEL [8] in DEL [9] gas EOS [10] cham_bers . EOS
Table 5: Translation examples of NAT and RecoverSAT. “Forced Translation” denotes the generated sentence when we manually force the model to generate a certain token (colored green) at a certain position. We use yellow color to label repetitive tokens, red color to label missing tokens, and gray color to label the segments to be deleted. We use “_” to concatenate sub-words and subscript numbers (e.g., [1]) to mark the beginning of each segment.
4.8 Case Study
We present translation examples of NAT and our RecoverSAT model on the WMT14 De  $\rightarrow$  En validation set in Table 5. From the table, we can observe that: (1) The multi-modality problem (repetitive and missing tokens) is severe in the sentence generated by NAT, while it is effectively alleviated by RecoverSAT (see translations A to D); (2) RecoverSAT can leverage target contexts to dynamically determine the segment length to reduce repetitive token errors (see translation B) or recover from missing token errors (see translations C and D); (3) RecoverSAT is capable of detecting and deleting the repetitive segments, even if there are multiple such segments (see translation D).
There has been various work investigating to accelerate the decoding process of sequence generation models (Kalchbrenner et al., 2018; Gu et al., 2018). In the field of neural machine translation, which is the focus of this work, Gu et al. (2018) first propose non-autoregressive machine translation (NAT), which generates all target tokens simultaneously. Although accelerating the decoding process significantly, NAT suffers from the multimodality problem (Gu et al., 2018) which generally
manifests as repetitive or missing tokens in translation. Therefore, intensive efforts have been devoted to alleviate the multi-modality problem in NAT. Wang et al. (2019) regularize the decoder hidden states of neighboring tokens to reduce repetitive tokens; Sun et al. (2019) utilize conditional random field to model target-side positional contexts; Shao et al. (2019a) and Shao et al. (2019b) introduce target-side information via specially designed training loss while Guo et al. (2019a) enhance the input of the decoder with target-side information; Kaiser et al. (2018), Akoury et al. (2019), Shu et al. (2019) and Ma et al. (2019) incorporate latent variables to guide generation; Li et al. (2019), Wei et al. (2019) and Guo et al. (2019b) use autoregressive models to guide the training process of NAT; Ran et al. (2019) and Bao et al. (2019) consider the reordering information in decoding. Wang et al. (2018) further propose a semi-autoregressive Transformer method, which generates segments autoregressively and predicts the tokens in a segment non-autoregressively. However, none of the above methods explicitly consider recovering from multi-modality related errors.
Recently, multi-step NAT models have also been investigated to address this issue. Lee et al. (2018) and Ghazvininejad et al. (2019) adopt an iterative decoding methods which have the potential to re
cover from generation errors. Besides, Stern et al. and Gu et al. (2019) also propose to use dynamic insertion/deletion to alleviate the generation repetition/missing. Different from these work, our model changes one-step NAT to a semi-autoregressive form, which maintains considerable speedup and enables the model to see the local history and future to avoid repetitive/missing words in decoding. Our work can further replace the one-step NAT to improve its performance.
In this work, we propose a novel semi-autoregressive model RecoverSAT to alleviate the multi-modality problem, which performs translation by generating segments non-autoregressively and predicts the tokens in a segment autoregressively. By determining segment length dynamically, RecoverSAT is capable of recovering from missing token errors and reducing repetitive token errors. By explicitly detecting and deleting repetitive segments, RecoverSAT is able to recover from repetitive token errors. Experiments on three widely-used benchmark datasets show that our RecoverSAT model maintains comparable performance with more than  $4\times$  decoding speedup compared with the AT model.
We would like to thank all anonymous reviewers for their insightful comments.
Nader Akoury, Kalpesh Krishna, and Mohit Iyyer. 2019. Syntactically supervised transformers for faster neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1269-1281.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.
Yu Bao, Hao Zhou, Jiangtao Feng, Mingxuan Wang, Shujian Huang, Jiajun Chen, and Lei Li. 2019. Non-autoregressive transformer by position learning. arXiv preprint arXiv:1911.10677.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of
the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1724-1734.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6114-6123.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In Proceedings of International Conference on Learning Representations.
Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In Proceedings of Advances in Neural Information Processing Systems 32, pages 11181-11191.
Junliang Guo, Xu Tan, Di He, Tao Qin, Linli Xu, and Tie-Yan Liu. 2019a. Non-autoregressive neural machine translation with enhanced decoder input. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, volume 33, pages 3723-3730.
Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Yan Liu. 2019b. Fine-tuning by curriculum learning for non-autoregressive neural machine translation. arXiv preprint arXiv:1911.08717.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. 2018. Fast decoding in sequence models using discrete latent variables. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 2390-2399.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. 2018. Efficient neural audio synthesis. In Proceedings of the 35th International Conference on Machine Learning, pages 2410-2419.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173-1182.
Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Hint-based training for non-autoregressive machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5712-5717.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. FlowSeq: Nonautoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4273-4283.
Qiu Ran, Yankai Lin, Peng Li, and Jie Zhou. 2019. Guiding non-autoregressive neural machine translation decoding with reordering information. arXiv preprint arXiv:1911.02215.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725.
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, Xilin Chen, and Jie Zhou. 2019a. Retrieving sequential information for non-autoregressive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3013-3024.
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. 2019b. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. arXiv preprint arXiv:1911.09320.
Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. 2019. Latent-variable nonautoregressive neural machine translation with deterministic inference using a delta posterior. arXiv preprint arXiv:1908.07181.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In Proceedings of the 36th International Conference on Machine Learning, pages 5976-5985.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. 2019. Fast structured decoding for sequence models. In Advances in Neural Information Processing Systems 32, pages 3011-3020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008.
Chunqi Wang, Ji Zhang, and Haiqing Chen. 2018. Semi-autoregressive neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 479-488.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019. Non-autoregressive machine translation with auxiliary regularization. In
Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, volume 33, pages 5377-5384.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019. Imitation learning for non-autoregressive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1304-1312.
A Positional Encoding
Our RecoverSAT model utilizes the positional encoding method in Vaswani et al. (2017) to encode the information about the positions of source tokens. The positional embedding is defined as:
$$
\mathbf {P} \mathbf {E} _ {p o s} [ 2 i ] = \sin \left(\frac {p o s}{1 0 0 0 0 ^ {2 i / d}}\right), \tag {11}
$$
$$
\mathbf {P} \mathbf {E} _ {p o s} [ 2 i + 1 ] = \cos \left(\frac {p o s}{1 0 0 0 0 ^ {2 i / d}}\right), \tag {12}
$$
where  $\mathbf{PE}_{pos}[i]$  is the  $i$ -th element of the positional embedding vector  $\mathbf{PE}_{pos}$  for the position  $pos$ , and  $d$  is the dimension of the positional embedding vector. Then we can compute the input vector of the encoder for the  $m$ -th source token  $w$  as:
$$
\mathbf {E} _ {w} = \mathbf {E} _ {w} ^ {\text {t o k e n}} + \mathbf {P E} _ {m}, \tag {13}
$$
where  $\mathbf{E}_w^{token}$  is the token embedding vector of  $w$ .
However, we can not apply this method to target tokens directly. Since lengths of segments are dynamically determined, the positions of the tokens in the target sentence, except those in the first segment, are not available during generation. To solve the problem, we use the aforementioned method to independently encode the position in the corresponding segment of each token instead and adopt an absolute segment embedding method, which uses a distinct trainable vector to represent the position of each segment. Formally, the input vector of the decoder for the  $n$ -th target token  $v$  of the  $j$ -th segment is computed as:
$$
\mathbf {E} _ {v} = \mathbf {E} _ {v} ^ {\text {t o k e n}} + \mathbf {P} \mathbf {E} _ {n} + \mathbf {E} _ {j} ^ {\text {s e g}}, \tag {14}
$$
where  $\mathbf{E}_j^{seg}$  is the segment embedding vector for the segment position  $j$ .
Model	Iterative Decoding	WMT14 En-De	WMT16 En-Ro	IWSLT16 En-De
En→	De→	Speedup	En→	Ro→	Speedup	En→	Speedup
Transformer	27.17	31.95	1.00×	32.86	32.60	1.00×	31.18	1.00×
NAT-FT	17.69	21.47	-	27.29	29.06	-	26.52	15.6×
NAT-FT+NPD (n=10)	18.66	22.41	-	29.02	30.76	-	27.44	7.68×
NAT-FT+NPD (n=100)	19.17	23.20	-	29.79	31.44	-	28.16	2.36×
SynST	20.74	25.50	4.86×	-	-	-	23.82	3.78×
NAT-IR (iter=1)	✓	13.91	16.77	11.39×	24.45	25.73	16.03×	22.20	8.98×
NAT-IR (iter=10)	✓	21.61	25.48	2.01×	29.32	30.19	2.15×	27.11	1.55×
NAT-FS	22.27	27.25	3.75×	30.57	30.83	3.70×	27.78	3.38×
imitate-NAT	22.44	25.67	-	28.61	28.90	-	28.41	18.6×
imitate-NAT+LPD (n=7)	24.15	27.28	-	31.45	31.81	-	30.68	9.70×
PNAT	23.05	27.18	-	-	-	-	-	-
PNAT+LPD (n=9)	24.48	29.16	-	-	-	-	-	-
NAT-REG	20.65	24.77	-	-	-	-	23.14	-
NAT-REG+LPD (n=9)	24.61	28.90	-	-	-	-	27.02	-
LV NAR	25.10	-	6.8×	-	-	-	-	-
NART	21.11	25.24	30.2×	-	-	-	-	-
NART+LPD (n=9)	25.20	29.52	17.8×	-	-	-	-	-
FlowSeq-base	21.45	26.16	<1.5×	29.34	30.44	-	-	-
FlowSeq-base+NPD (n=30)	23.48	28.40	<1.5×	31.75	32.49	-	-	-
FlowSeq-large	23.72	28.39	<1.5×	29.73	30.72	-	-	-
FlowSeq-large+NPD (n=30)	25.31	30.68	<1.5×	32.20	32.84	-	-	-
FCL-NAT	21.70	25.32	28.9×	-	-	-	-	-
FCL-NAT+NPD (n=9)	25.75	29.50	16.0×	-	-	-	-	-
ReorderNAT	26.51	31.13	-	31.70	31.99	-	30.26	5.96×
NART-DCRF	23.44	27.22	10.4×	-	-	-	-	-
NART-DCRF+LPD (n=19)	26.80	30.04	4.39×	-	-	-	-	-
SAT (K=2)	26.90	-	1.51×	-	-	-	-	-
SAT (K=6)	24.83	-	2.98×	-	-	-	-	-
CMLM-small (iter=1)	✓	15.06	19.26	-	20.12	20.36	-	-	-
CMLM-small (iter=10)	✓	25.51	29.47	-	31.65	32.27	-	-	-
CMLM-base (iter=1)	✓	18.05	21.83	-	27.32	28.20	-	-	-
CMLM-base (iter=10)	✓	27.03	30.53	<1.5×	33.08	33.31	-	-	-
RecoverSAT (K=2)	27.11	31.67	2.16×	32.92	33.19	2.02×	30.78	2.06×
RecoverSAT (K=5)	26.91	31.22	3.17×	32.81	32.80	3.16×	30.55	3.28×
RecoverSAT (K=10)	26.32	30.46	4.31×	32.59	32.29	4.31×	29.90	4.68×
Table 6: Performance (BLEU) of Transformer and the NAT/semi-autoregressive models on three widely-used machine translation benchmark datasets. NPD denotes the noisy parallel decoding technique (Gu et al., 2018) and LPD denotes the length parallel decoding technique (Wei et al., 2019).  $n$  denotes the sample size of NPD or LPD. iter denotes the refinement number of the iterative decoding method.