Token-Level Self-Evolution Training for Sequence-to-Sequence Learning
Year: 2023

Abstract:
Adaptive training approaches, widely used in sequence-to-sequence models, commonly reweigh the losses of different target tokens based on priors, e.g. word frequency. However, most of them do not consider the variation of learning difficulty in different training steps, and overly emphasize the learning of difficult one-hot labels, making the learning deterministic and sub-optimal. In response, we present Token-Level Self-Evolution Training (SE), a simple and effective dynamic training method to fully and wisely exploit the knowledge from data. SE focuses on dynamically learning the under-explored tokens for each forward pass and adaptively regularizes the training by introducing a novel token-specific label smoothing approach. Empirically, SE yields consistent and significant improvements in three tasks, i.e. machine translation, summarization, and grammatical error correction. Encouragingly, we achieve averaging +0.93 BLEU improvement on three machine translation tasks. Analyses confirm that, besides improving lexical accuracy, SE enhances generation diversity and model generalization.

heading: 1 Introduction
text: Sequence-to-sequence learning (Seq2Seq) with neural networks (Sutskever et al., 2014) has advanced the state-of-the-art in various NLP tasks, e.g. translation (Bahdanau et al., 2015; Vaswani et al., 2017), summarization (Cheng and Lapata, 2016), and grammatical error correction (Yuan and Briscoe, 2016). Generally, Seq2Seq models are trained with the cross-entropy loss, which equally weighs the training losses of different target tokens.
However, due to the token imbalance nature (Piantadosi, 2014) and the truth that different tokens contribute differently to the sentence meaning (Church and Hanks, 1990; Chen et al., 2020),
∗Keqin and Liang contributed equally. †Corresponding Author.
several works are developed to reweigh the tokenlevel training loss according to explicit (e.g. frequency) or implicit (uncertainty estimated by offthe-shelf language models) priors (Gu et al., 2020; Xu et al., 2021; Zhang et al., 2022a). For example, Gu et al. (2020) proposed two heuristic criteria based on word frequency to encourage the model to learn from larger-weight low-frequency tokens. Zhang et al. (2022a) introduce target-context-aware metric based on an additional target-side language model to adjust the weight of each target token.
Despite some success, there are still limitations in these adaptive training approaches. First, most of them predetermine the difficult tokens and fix such prior to guiding the training. However, in our preliminary study, we find the hard-to-learn tokens are dynamically changing during training, rather than statically fixed. As shown in Figure 1, as the training progress goes, although the sentence-level loss is nicely converging, the difficult token is changing from “abschließen” to “Sache” in terms of the token-level loss. Second, these adaptive training methods overly emphasize fitting the difficult tokens’ one-hot labels by reweighing the loss, which empirically may cause overfitting and limit the generalization (Norouzi et al., 2016; Szegedy et al., 2016; Xiao et al., 2019; Miao et al., 2021). Also, a more recent study (Zhai et al., 2023) provides
841
theoretical evidence to support that reweighting is not that effective to improve the generalization.
Correspondingly, we design a simple and effective Token-Level Self-Evolution Training (SE) strategy to encourage Seq2Seq models to learn from difficult words that are dynamically selected by the model itself. Specifically, SE contains two stages: ❶self-questioning and ❷self-evolution training. In the first stage, the Seq2Seq models dynamically select the hard-to-learn tokens based on the tokenlevel losses, then we encourage the Seq2Seq models to learn from them in the second stage, where, rather than adopting reweighing, we introduce a novel token-specific label smoothing approach to generate easily digestible soft label, which considers both the ground truth and model’s prediction.
Experiments across tasks, language pairs, data scales, and model sizes show that SE consistently and significantly outperforms both the vanilla Seq2Seq model and the re-implemented advanced baselines. Analyses confirm that besides improved lexical accuracy, SE generates diverse and humanlike generations with better model generalization.

heading: 2 Methodology
text: Preliminary Sequence-to-sequence (Seq2Seq) learning aims to maximize the cross-entropy (CE) loss of the log-likelihood of each target word in y = {y1, . . . , yN}, conditioned on source x, where the optimization treats all tokens equally:
LCE(θ) = − N∑
j=1
log p(yj |y<j ,x; θ) (1)
However, due to the different learning difficulties of each token, it is sub-optimal to treat all tokens equally (Gu et al., 2020). To address this limitation, a series of token-level adaptive training objectives were adopted to re-weight the losses of different target tokens (Xu et al., 2021; Zhang et al., 2022a). The common goal of these methods is to facilitate the model training by fully exploiting the informative but underexplored tokens.
However, our preliminary study shows that the hard tokens are dynamically changing (see Figure 1) in different training steps (or model structures), thus it is sub-optimal to employ static token priors (e.g. frequency) during training. Also, recent studies (Zhai et al., 2023) in the ML community theoretically show that reweighting is not that effective to improve the generalization. Based on the above
evidence, we present the self-evolution learning (SE) mechanism to encourage the model to adaptively and wisely learn from the informative yet under-explored tokens dynamically determined by the model itself (Stage❶ in §2.1), with an easy-tolearn label distribution (Stage❷ in §2.1). A similar work to ours is Hahn and Choi (2019). However, their method mainly considers the situation where the predicted answer is incorrect but close to the golden answer, while our method focuses on all dynamic hard tokens.

heading: 2.1 Token-Level Self-Evolution Learning
text: ❶ Self-questioning Stage. The goal is to select the hard-to-learn tokens that are questioned by the Seq2Seq model itself during training dynamics. Previously, these difficult tokens are predetermined by external models or specific statistical metrics. However, inspired by the finding of dynamic change of difficult tokens during the training stage as shown in Figure 1 and the finding that the trained model contains useful information (Li and Lu, 2021), e.g. synonym, we propose to straightforwardly leverage the behavior of the model to dynamically select target tokens. In practice, we first calculate the token-level CE loss, denoted as {l1, l2, ..., ln}, for each token for each forward pass. Then we set a loss threshold Γ and select the tokens whose losses exceed Γ as the target tokens, i.e., D = {ti|li > Γ} where i ∈ N = {1, 2, ..., n}.
❷ Self-evolution Training Stage. After selecting the difficult tokens, we encourage the model to carefully learn from them. Given the theoretical shortage (Zhai et al., 2023) and potentially caused overfitting or overconfidence problem (Miao et al., 2021) of reweighting and deliberately learning from difficult tokens, we propose to strengthen the learning from these tokens with a newly designed Token-specific Label Smoothing (TLS) approach. Specifically, motivated by the effect of label smoothing (LS) regularization (Szegedy et al., 2016), we combine the ground truth pi and the model’s prediction p̂i to form a new soft label p̃i for the i-th token. Then we use p̃ to guide the difficult tokens D, while leaving label-smoothing CE loss for the other tokens. It is worth noting that we also apply the traditional label smoothing technique to p̂i to activate the information in the predicted distribution. Analogous to human learning, it is often easier for humans to grasp new things described by their familiar knowledge (Reder et al., 2016),
therefore the new soft label fused both accurate ground truth and model’s self-distribution is easily digestible. Mathematically, for difficult tokens ti, p̃i is formulated as:
p̃i = (pi + p̂i)/2. (2)
Then we calculate the losses of difficult tokens and the others, and combine the two losses:
L = −( ∑
i
p̃i · log(p̂i) + ∑
j
pj · log(p̂j)), (3)
where i ∈ D and j ∈ N \D.

heading: 3 Evaluation
text: Machine Translation on three widely-used benchmarks (Ding et al., 2020, 2021c, 2022): smallscale WMT16 English-Romanian (En-Ro; 0.6M), medium-scale WMT14 English-German (En-De; 4.5M), and large-scale WMT14 English-French (En-Fr; 36.0M). We implement the baselines and our approach under Transformer-base settings. We follow the previous adaptive training approach (Gu et al., 2020) to pretrain with the cross-entropy loss with N steps, and further finetune the same steps with different adaptive training objectives, including Freq-Exponential (Gu et al., 2020), Freq-ChiSquare (Gu et al., 2020), D2GPo (Li et al., 2020),
BMI-adaptive (Xu et al., 2021), MixCrossEntropy (Li and Lu, 2021), CBMI-adaptive (Zhang et al., 2022a), and SPL (Wan et al., 2020). For N , we adopt 100K and 30K for larger datasets, e.g. En-De and En-Fr, and small dataset, i.e. En-Ro, respectively. We empirically adopt 32K tokens per batch for large datasets, the learning rate warms up to 1e-7 for 10K steps, and then decays 90K, while for small dataset En-Ro, The learning rate warms up to 1e-7 for 4K steps, and then decays 26K steps. All the experiments are conducted on 4 NVIDIA Tesla A100 GPUs. The SacreBLEU (Post, 2018) was used for evaluation. Besides translation, we also follow previous works (Liu et al., 2021b; Zhong et al., 2022; Zhang et al., 2022b) to validate the universality of our method on more sequenceto-sequence learning tasks, e.g., summarization and grammatical error correction. Text Summarization on XSUM corpus (0.2M). We follow fairseq (Ott et al., 2019) to preprocess the data and train the model, then finetune them for the same steps. We evaluated with the ROUGE (Lin, 2004), i.e. R-1, R-2, and R-L. Grammatical Error Correction on CoNLL14 (1.4M). We follow Chollampatt and Ng (2018) to preprocess the data and train the model, then finetune them for the same steps. The MaxMatch (M2) scores (Dahlmeier and Ng, 2012) were used for evaluation with precision, recall, and F0.5 values.

heading: 3.1 Main Results

heading: SE brings gains across language pairs and scales.
text: Results on machine translation across different data sizes ranging from 0.6M to 36M in Table 1 show that our SE-equipped Transformer “+ SelfEvolution (ours)” 1) considerably improves the performance by averaging +0.92 BLEU points; 2) out-
scales. Shadowed areas mean accurate token prediction estimated with lower cross-entropy loss, i.e. “0-1”.
performs previous competitive method “+ CBMIadaptive” by up to +0.47 BLEU points on large dataset WMT14 En-Fr. These results demonstrate the effectiveness and universality of our SE.

heading: SE brings gains across tasks and backbone sizes.
text: Table 2 lists the performance on more tasks, including translation, summarization, and grammar error correction, upon large pretrained backbone - BART (Lewis et al., 2020), which has above 600M parameters. Compared to a stronger baseline, our SE significantly and incrementally improves the generation quality in all tasks, i.e. +0.4 BLEU, + 0.7 RG-L, and + 1.9 F0.5, respectively, showing our SE is robustly applicable to general scenarios.

heading: SE works well on extremely large dataset. To
text: further verify the effectiveness of SE on extremely large dataset, we conducted an experiment on WMT22 De-En processed by Zan et al. (2022b), which contains 236M training examples. The results in Table 4 show that our method can achieve +0.4 and +1.2 improvement in BLEU and COMET respectively, which proves that our SE also works on extremely large datasets.

heading: 3.2 Analysis
text: We provide some insights to better understand the effectiveness of our approach. The ablation of important modules and parameters is in Appendix A.
SE learns better token representation. To verify whether our method helps learn better tokens representation, we conduct analysis on WMT14 EnDe from learning loss and fine-grained generation
perspectives, respectively. First, we count the token ratios distributed in different cross-entropy loss scales in Table 3 following Zan et al. (2022a). Cross-entropy is a good indicator to quantify the distance between the predicted distribution and the ground truth in the valid dataset, and a lower value means a more similar distribution. As shown, our method improves the low-loss token ratios by +2.3%, indicating SE helps the model learn better token representations by reducing the token uncertainty. In addition, we follow Ding et al. (2021a); Liu et al. (2021a) to break the translation down into different granularities and measure their fined-grained performance. In particular, we calculate1 the F-measure of words by different frequency buckets and BLEU scores of buckets of different lengths in Figure 2. We see SE achieves better performance in all frequencies and sentence buckets, demonstrating our method can improve the performance of different granularities.
SE encourages diverse generations. Lacking generation diversity is a notorious problem for Seq2Seq learning tasks (Sun et al., 2020; Lin et al., 2022). Benefiting from better exploring the model’s prediction with corrected soft labels, SE is expected to improve generation diversity. We follow Wang et al. (2022) to examine this by analyzing the performance in an additional multiplereference test of WMT’14 En-De (Ott et al., 2018). We choose additional references for each of the 500 test sentences taken from the original test. Table 5 shows SE consistently outperforms the baseline with the average improvement being 0.9/1.0 BLEU, which indicates that our SE can effectively generate diverse results.

heading: SE enhances model generalization. Benefiting
text: from better hard token exploration, SE-equipped Transformers are expected to own better generalizations. We examine it by testing on domain shift
1Using compare-mt (Neubig et al., 2019).
scenarios following Ding et al. (2021b). In particular, we evaluate WMT14 En-De models over four out-of-domain test sets (Müller et al., 2020) in Table 6 and find that SE improves the translation by averaging +0.9 BLEU points, showing a better lexical generalization ability.
SE encourages human-like generations. We design two types of evaluation on WMT14 En-Fr: 1) AUTOMATIC EVALUATION with COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020), which have a high-level correlation with human judgments. 2) HUMAN EVALUATION with three near-native French annotators who hold DALF C2 certificate2. Specifically, for human evaluation, we randomly sample 50 sentences from the test set to evaluate the translation adequacy and fluency, scoring 1∼5. For adequacy, 1 represents irrelevant to the source while 5 means semantically equal. For fluency, 1 means unintelligible while 5 means fluent and native. Table 7 shows the automatic and human evaluation results, where we find that our SE indeed achieves human-like translation.

heading: 4 Conclusion
text: In this paper, we propose a self-evolution learning mechanism to improve seq2seq learning, by exploiting the informative-yet-underexplored tokens dynamically. SE follows two stages, i.e. selfquestioning and self-evolution training, and can be used to evolve any pretrained models with a sim-
2http://www.delfdalf.fr/dalf-c2-en.html
ple recipe: continue train with SE. We empirically demonstrated the effectiveness and universality of SE on a series of widely-used benchmarks, covering low, medium, high, and extremely-high data volumes.
In the future, besides generation tasks, we would like to verify the effectiveness of SE on language understanding tasks (Wu et al., 2020; Zhong et al., 2023). Also, it will be interesting to design SEinspired instruction tuning or prompting strategy like Lu et al. (2023) to enhance the performance of large language models, e.g. ChatGPT3, which after all have already been fully validated on lots of conditional generation tasks (Hendy et al., 2023; Jiao et al., 2023; Peng et al., 2023; Wu et al., 2023).
Limitations
Our work has several potential limitations. First, we determine the threshold Γ by manual selection, which may limit the performance of Seq2Seq models, it will make our work more effective and elegant if we dynamically select the threshold. Second, besides the improvement on three widely used tasks, we believe that there are still other abilities, like code generation, of Seq2Seq models that can be improved by our method, which are not fully explored in this work.
Ethics Statement
We take ethical considerations very seriously and strictly adhere to the ACL Ethics Policy. This paper focuses on effective training for sequence-tosequence learning. The datasets used in this paper are publicly available and have been widely adopted by researchers. We ensure that the findings and conclusions of this paper are reported accurately and objectively.

heading: Acknowledgement
text: We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions.
3https://chat.openai.com/

heading: A Appendix
text: Parameter Analysis on Γ As stated in §2.1, we use the loss threshold Γ to dynamically select the hard-to-learn tokens. Here, we analyze the influence of different Γ in detail. In practice, we train the Transformer models with different Γ (in {3,4,5,6}) and evaluate the performance of the WMT14 En-De test set. Table 8 lists the performance of different Γ. The results of Table 8 show that SE is stable and insensitive to Γ within a certain range. Noting that we select Γ = 5 for all experiment settings based on the results in Table 8.
Ablation Study
Metric. In this work, we use the loss-based metric to dynamically select the hard-to-learn tokens. To validate the effectiveness of the metric, we use a simple adaptive training method (“+ ADD”) that adds 1 to the weighting term of loss of the hard-to-learn tokens. The results on WMT16 EnRo are shown in Table 9, the simple Add method can achieve +0.3 BLEU improvement compared to the baseline model, which proves that our proposed self-questioning stage indeed mines informative difficult tokens. Also, we can observe that learning these dynamic difficult tokens with our SE framework (“+ SE”) could outperform “+ ADD” by +0.6 BLUE points, demonstrating the superiority of our token-specific label smoothing approach.
Learning objective. As stated in §2.1, our learning objective is the combination of the ground truth and the model’s prediction. To validate the effectiveness of predicted distribution, we conduct ablation experiments on WMT16 En-Ro and WMT14 En-De. The results in Table 10 show that adding the predicted distribution will consistently improve the model’s performance, which proves the effectiveness of the predicted distribution.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The last section of the paper.
7 A2. Did you discuss any potential risks of your work? Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: The abstract and the introduction section.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.2
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: In the future, besides generation tasks, we would like to verify the effectiveness of SE on language understanding tasks (Wu et al. , 2020; Zhong et al. , 2023). Also, it will be interesting to design SEinspired instruction tuning or prompting strategy like Lu et al. (2023) to enhance the performance of large language models, e. g. ChatGPT3, which after all have already been fully validated on lots of conditional generation tasks (Hendy et al. , 2023; Jiao et al. , 2023; Peng et al. , 2023; Wu et al. , 2023). Limitations
Our work has several potential limitations. First, we determine the threshold Γ by manual selection, which may limit the performance of Seq2Seq models, it will make our work more effective and elegant if we dynamically select the threshold. Second, besides the improvement on three widely used tasks, we believe that there are still other abilities, like code generation, of Seq2Seq models that can be improved by our method, which are not fully explored in this work