Token-level Fitting Issues of Seq2seq Models
Year: 2023

Abstract:
Sequence-to-sequence (seq2seq) models have been widely used for natural language processing, computer vision, and other deep learning tasks. We find that seq2seq models trained with early-stopping suffer from issues at the token level. In particular, while some tokens in the vocabulary demonstrate overfitting, others underfit when training is stopped. Experiments show that the phenomena are pervasive in different models, even in fine-tuned large pretrained-models. We identify three major factors that influence token-level fitting, which include token frequency, parts-of-speech, and prediction discrepancy. Further, we find that external factors such as language, model size, domain, data scale, and pretraining can also influence the fitting of tokens.

heading: 1 Introduction
text: Deep learning models tend to overfit on relatively small datasets because of their strong capacity and a massive number of parameters (Brownlee, 2018; Li et al., 2019; Rice et al., 2020; Bejani and Ghatee, 2021). Studies suggest regularization and early stopping to control the generalization error caused by overfitting (Hastie et al., 2009; Zhang et al., 2017; Chatterjee and Zielinski, 2022). Previous studies mainly analyze the generalization issue on image classification task (Arpit et al., 2017; Zhang et al., 2021), where the learning target is relatively simple. In contrast, NLP task such as machine translation (Zhang et al., 2015; Singh et al., 2017) is more complex with regard to the learning targets, which involve a sequence of tokens.
Natural languages exhibit a long-tailed distribution of tokens (Powers, 1998). The long-tail phenomena have been associated with performance degradation of NLP tasks (Gong et al., 2018; Raunak et al., 2020; Yu et al., 2022), where the rare
∗* Corresponding author.
(low frequency) tokens are ascribed as hard learning targets and popular (high frequency) tokens as easy learning targets. These criteria of easiness of learning targets are intuitive but coarse-grained, which are not associated with the training dynamics. In this paper, we study the easiness of tokens as learning targets from the perspective of overfitting and underfitting. Intuitively, the learning on hard tokens will be slower than that on easy tokens, which may result in underfitting on hard tokens and overfitting on easy tokens, as illustrated by Figure 1. We propose two measures to quantify fitting – fittingoffset and potential-gain. Fitting-offset measures the offset of the best fit from the early-stopping point, which reflects the degree of overfitting or underfitting. Potential-gain measures the accuracy gap between the early-stopping checkpoint and the best fit, which also estimates the accuracy decrease caused by overfitting or underfitting.
We use machine translation as our test bed, training models on English-German benchmark datasets, including News and Europarl domains. Our extensive experiments reveal novel findings: 1) Both overfitting and underfitting occur in a trained seq2seq model. 2) High-frequency tokens are expected to overfit, but some are found underfitted, and low-frequency tokens are expected to underfit, but some are found overfitted. 3) Large pretrained models reduce underfitting effectively during finetuning but are less effective on overfitting. Besides, we propose a direct indicator of easiness – prediction discrepancy, using the probability difference of predictions made by full context and local context as a criterion to group tokens.
In addition to tokens, sentences have also been considered as learning targets, where curriculum learning methods distinguish sentences as easy or hard (Kocmi and Bojar, 2017; Platanios et al., 2019; Xu et al., 2020). For example, the length of a sentence is used as a criterion to identify easy sentences from hard ones, where the easy (short) sen-
276
Early-stopping point
Train
Valid
Lo ss
Epoch
(a) Idealized training and validation loss curves, where the model is selected by early stopping.
easy
hard
Lo ss
Epoch
(b) Easy tokens overfit at the earlystopping point, while hard tokens underfit.
Figure 1: Seq2seq models trained with early stopping may suffer from overfitting or underfitting.
tences are first learned and then the hard (long) sentences. Using our metrics, we find that the length of a sentence is not a good indicator of easy or hard sentences from the perspective of overfitting and underfitting.

heading: 2 Related Work
text: Previous studies compare the fitting of real data and noise data, demonstrating that real data is easier to learn and has a faster convergence speed than noise data (Zhang et al., 2017; Arpit et al., 2017; Chatterjee and Zielinski, 2022). The different convergence speeds also provide an explanation of how early stopping prevents the memorization of noise data. However, these works do not compare the fitting among different learning targets inside the real data, neither between different samples nor between different parts of each sample. In this paper, we conduct experiments on a more complex seq2seq task instead of simple classification. We study the fitting of token-level learning targets, demonstrating that both overfitting and underfitting occur when training seq2seq models.
There are few works studying overfitting and underfitting in NLP. Sun et al. (2017) report that complex structure leads to overfitting in structured prediction. Wolfe and Caliskan (2021) demonstrate that low-frequency names exhibit bias and overfitting in the language model. Varis and Bojar (2021) illustrate that machine translation models generalize poorly on the test set with unseen sentence length. These works discuss overfitting issues on specific conditions, such as complex structure, frequent names, and unseen length. In comparison, we conduct a systematic analysis of the general phenomena of overfitting and underfitting in language. Specifically, we propose quantitative measures, identify major factors, and conduct statistical
hypothesis testing on the phenomena.

heading: 3 Experimental Settings

heading: 3.1 Datasets
text: We use the News corpus as a major dataset for our experiments and analysis, and we further use the Europarl corpus for the comparison of different domains and data scales.
News We use News Commentary v11 for training, newstest2015 for validation, and newstest2016 for testing. The English-German machine translation dataset contains 236, 287 sentence pairs for training, 2, 169 pairs for validation, and 2, 999 pairs for testing.
Europarl We use English-German Europarl v7, following Maruf et al. (2019) to split the train, validation, and test sets. The dataset contains 1, 666, 904 sentence pairs for training, 3, 587 pairs for validation, and 5, 134 pairs for testing.
We tokenize the sentences using MOSES (Koehn et al., 2007), followed by truecase and a BPE (Sennrich et al., 2015) with 30, 000 merging operations. We use separate embedding tables for source and target languages in the model.

heading: 3.2 Model Configurations
text: We study the overfitting and underfitting issues on three model configurations. We use the base model in section 4 and 5, the pre-trained model in section 6, and the big model for the model-size evaluation in section 7.
Base Model We use the standard Transformer base model (Vaswani et al., 2017), which has 6 layers, 8 heads, 512 output dimensions, and 2048 hidden dimensions. We train the model with a learning rate of 5× 10−4, a dropout of 0.3, a label smoothing of 0.1, and an Adam optimizer (Kingma and Ba, 2014).
Big Model Following the standard Transformer big model (Vaswani et al., 2017), we use 6 layers, 16 heads, 1024 output dimensions, and 4096 hidden dimensions. We train the model with a learning rate of 3× 10−4, a dropout of 0.3, a label smoothing of 0.1, and an Adam optimizer.
Pretrained Large Model We use mBART25 (Liu et al., 2020), which has the similar setting as BART large model (Lewis et al., 2020), using 12 layers, 16 heads, 1024 output dimensions, and 4096 hidden dimensions. We fine-tune the model with a learning rate of 3× 10−5, a dropout of 0.3, an attention-dropout of 0.1, a label smoothing of 0.2, and an Adam optimizer.
For each experiment, we train 40 models using random seeds from 1 to 40, obtaining 40 samples for the significance test. During the training of the base or big model, we keep the last 20 checkpoints for analysis, where the checkpoint of earlystopping is the 10-th of the 20 checkpoints. For mBART25, we keep the last 10 checkpoints, and the early-stopping checkpoint is at the 5-th of the checkpoints.

heading: 3.3 Evaluation Metric
text: Measures We propose two measures: fitting-offset and potential-gain.
Fitting-offset represents how far (i.e., number of epochs) the best fit of a group of tokens diverges from the point of early stopping. In this paper, we use epoch as its unit because we evaluate the model using the validation set at the end of each training epoch. As Figure 2 shows, for the easy tokens, the fitting-offset is negative, denoting overfitting, where the best fit is before the early-stopping epoch. For the hard tokens, the fitting-offset is positive, denoting an underfitting, where the best fit is after the early-stopping epoch. Using fitting-offset, we can quantify the degree of overfitting and underfitting.
Potential-gain represents the potential accuracy increase if we move the best fit to the earlystopping epoch. We calculate the measure by subtracting the accuracy of the early-stopping checkpoint from the accuracy of the best fit. Using this measure, we can quantitatively estimate the potential benefits by fixing the overfitting or underfitting issue.
Significance Test Since the distribution of fitting-offset is unknown, we use a non-parametric sign-test (Dixon and Mood, 1946; Hodges, 1955) to test our hypothesis. We train the model N times
to obtain N observations on the fitting-offset. The hypothesis about the overfitting and underfitting can be expressed by
{ H0 : fitting-offset = 0, no fitting issue, H1 : fitting-offset ̸= 0, has fitting issue. (1) If H0 is true, the N observations are expected to be half positive and half negative. The total number of positive observations N+ follows a binomial distribution, through which we decide the rejection region according to a significance level α.
Grouping Ideally, we can calculate the two measures on each token to tell which tokens are overfitted and which tokens are underfitted at the earlystopping epoch. However, direct observation of each token is noisy and does not show obvious patterns. We group the tokens and average the valid losses to reduce the noise, through which the pattern emerges, and we obtain stable measures.

heading: 4 Token-level Results

heading: 4.1 Fitting of Rare Tokens in Seq2seq Model Training
text: Previous studies suggest that long-tail token distribution affects the performance of NLP tasks (Gong et al., 2018; Raunak et al., 2020; Yu et al., 2022). We hypothesize that the low-frequency tokens underfit during training and conduct verification experiments as follows.
Settings We experiment on the News dataset, using a Transformer base model (Vaswani et al., 2017). We categorize the target tokens into high/medium/low-frequency according to their distribution in the training set, with balanced probability mass on the three buckets.
Hypothesis Testing For each group of high/medium/low-frequency tokens, we measure
Figure 4: Fitting-offset of tokens grouped by parts-ofspeech.
the fitting-offset using the checkpoints of each model, obtaining 40 samples for each group. We test our hypothesis on each group using a sign-test (Eq. 1). As a result, we obtain a p-value of 1.9 × 10−5 for high-frequency and 7.5 × 10−11 for low-frequency, which strongly supports the hypothesis that the high/low-frequency tokens either overfit or underfit.
Further as Figure 3 shows, the average fittingoffset for high-frequency tokens is −3.7 with a standard deviation of 3.4. The negative value of the fitting-offset indicates that the high-frequency tokens overfit, where the best fit happens at an average of 3.7 epochs, before the early-stopping point. The average fitting-offset for low-frequency tokens is 5.8 with a standard deviation of 3.3. The positive value of the fitting-offset indicates underfitting, where the best fit happens at 5.8 epochs, after the early-stopping point on average. Based on this evidence, we conclude that Both overfitting and underfitting at the token level occur when training seq2seq models.
Analysis The significant divergence of fittingoffset between the high/low-frequency tokens suggests that the frequency of tokens has a significant influence on their fitting. We quantify the influence using the potential-gain. In particular, take the lowfrequency tokens as an example. The potential-gain is 0.73, which means that the average accuracy is expected to be increased from 45.61 to 46.34 if we move the best fit to the early-stopping epoch. The potential-gain of the high-frequency tokens is 0.05, and that of the medium-frequency tokens is 0.23, which is relatively smaller than that of the low-frequency tokens, suggesting underfitting of the low-frequency tokens is the major issue.

heading: 4.2 Linguistic Factors to Token-level Fitting
text: In section 4.1, we find that the high-frequency tokens tend to overfit and the low-frequency tokens tend to underfit in the seq2seq model as a group. In order to further understand a fine-grained correlation between the frequency and the fitting of a token, we further split the high/low-frequency tokens into smaller groups and conduct experiments on the specific categories. Linguistic factors are considered in the detailed experiments.
Parts-of-speech (POS) We speculate that partsof-speech, as an important linguistic feature, may provide a different perspective to study the overfitting and underfitting issues. We group tokens according to their parts-of-speech as listed in Table 1. Specifically, we first obtain POS tagging on each word using spaCy 1. Then we map the POS of words to tokens by labeling all the tokens of a word with the same POS. Last, we group these tokens according to their POS. Take the group Noun as an example. We group tokens with the POS of NOUN, PRON, and PROPN into one category, naming Noun. We aggregate the major parts of speech into six groups according to their functional similarity, as shown in the Table.
As Figure 4 shows, parts-of-speech has a significant influence on the fitting of tokens. The function words are most likely to overfit, which is likely because they are close-set and easier to learn from the linguistic perspective. On the contrary, nouns are most likely to underfit, which can be due to the openness of the set and the challenging context dependencies.
The potential-gain of nouns is 0.69, increasing the accuracy from 52.38 to 53.07. The potential-
1https://spacy.io/
gains of numbers, symbols, verbs, and adj/adv words are 1.09, 0.58, 0.26, and 0.22, respectively. Surprisingly, the potential-gain of function words is negligible, even though they obviously overfit. We attribute it to the overall high frequency of function words because sufficient training samples reduce the negative impact of overfitting. It is confirmed by the detailed potential-gains shown in Table 2, where the function words with low frequency have a much higher potential-gain of 1.38.
It is worth noting that we use sub-word tokens in the experiments, where we assign the POS of a word to all its sub-words. We also tried words as tokens, which give a similar distribution of fitting offset, and the conclusions hold.
Frequency and Parts-of-speech We combine frequency and POS to make a detailed analysis of the high/low-frequency tokens. As Figure 5 shows, frequency and POS work independently. Among the high-frequency tokens, the function words tend to overfit, while adjectives and nouns tend to underfit. Among the low-frequency tokens, the symbols tend to overfit, while the adjectives and nouns tend to underfit. Based on this evidence, we arrive at a counter-intuitive conclusion that in a seq2seq model, the high-frequency tokens (popular tokens) mostly overfit but can also underfit, and the low-frequency tokens (rare tokens) mostly underfit but can also overfit.
When we look into the potential-gains, as shown in Table 2, we see higher potential-gains than in the previous section. The potential-gain of
low-frequency function words, symbols, and numbers are 1.38, 3.55, and 1.20, respectively. The potential-gains on med-frequency symbols and numbers are 3.41 and 1.50, respectively. Overall the high-frequency tokens have low potential-gains, and the verbs and adjectives have potential-gains of 1.09 and 1.67, respectively. These results demonstrate that combining the frequency and linguistic factors reveals stronger overfitting and underfitting, forecasting higher potential-gains in specific categories.

heading: 4.3 Prediction Discrepancy as A Factor to Token-level Fitting
text: Given that neither frequency nor parts of speech are decisive factors, we propose a new factor – prediction discrepancy, which measures the degree of dependence on long context for predicting each token in a text sequence. The intuition behind this factor is that some tokens may require more infor-
mation from the surrounding context to be correctly predicted, while others may be more independent and rely more on their local context. Specifically, consider a sentence, “I was quoted about 12 months .” the word “months” could most likely be told according to its local context “12”. In comparison, in a German sentence “es war schwer , ihn zu kennen .”, where the word-by-word translation is “it was hard , him to know .” the word “kennen” requires much longer context to predict.
The prediction discrepancy is calculated as
Dj = |P (Yj |Y<j , X)− P (Yj |Yj−1, X)|, (2)
where X is the source sequence, and Y is the target. For each token Yj , we predict it using its full context Y<j and its local context Yj . We use the discrepancy between these two predictions to indicate its dependence on long context. We train an altered Transformer model to do the predictions using two decoders, where one decoder uses the full context of a target while another decoder uses the local context. The two decoders share the same token embedding table and encoder. According to the value of discrepancy, we categorize the tokens into three groups, with big, medium, and small discrepancy, respectively.
Results As Figure 6 shows, the big-discrepancy tokens have an average fitting-offset of −2.7 with a standard deviation of 3.0. The mediumdiscrepancy tokens have an average fitting-offset of 1.0 with a standard deviation of 2.1. The smalldiscrepancy tokens have an average fitting-offset of 8.2 with a standard deviation of 1.8, showing a trend to exceed the boundary of 10. In comparison with frequency, the bigger range of the average fitting-offsets and the smaller standard deviations suggest that discrepancy is a better indicator than frequency. This indicates that the discrepancy is a good indicator of overfitting and underfitting.
The potential-gain of the small-discrepancy tokens is 0.63, increasing the average accuracy of the tokens from 75.85 to 76.48. In comparison with the
potential-gain of 0.75 for the low-frequency tokens, which increases the average accuracy from 45.61 to 46.34, the baseline accuracy of small-discrepancy is much higher, suggesting the effectiveness of discrepancy in discovering fitting issues among high accuracy predictions.
Discrepancy and Frequency Intuitively, discrepancy and frequency are two independent factors, given that discrepancy relies on context and frequency relies on the token itself. As Figure 7 shows, the most significant difference between high-frequency and med/low-frequency tokens is that med/small discrepancy tokens with high frequencies tend to overfit, while the med/small discrepancy tokens with med/low frequencies tend to underfit. In addition, as shown in Table 3, when frequency and discrepancy are combined to predict the overfitting and underfitting, the biggest potential-gain of low-frequency tokens increases from 0.73 to 0.86, suggesting that frequency and discrepancy are two independent factors.
Discrepancy and Parts-of-speech (POS) As Figure 8 shows, discrepancy and POS also work orthogonally. Overall, tokens with a smaller discrepancy have a larger fitting-offset, which consistently appears on numbers, verbs, adjectives, and nouns. Function words and symbols show a different pattern that the med-discrepancy tokens tend to have smaller fitting-offset than high-discrepancy.
As shown in Table 4, small-discrepancy tokens have potential gains of 1.63 and 1.15 on symbols and numbers, respectively. The potential gain on numbers with big/med-discrepancy are 2.72 and 2.02, respectively, suggesting the effectiveness of combining the two factors.
Summary We have identified three independent factors that affect token-level fitting in seq2seq model training, including frequency, parts-ofspeech, and discrepancy. While the former two are internal to the token, the third is external and context-dependent. These indicate that the fitting of tokens results from interestingly complex factors.

heading: 5 Easy Sentences vs Hard Sentences
text: Curriculum learning starts with easy sentences and then with hard sentences (Kocmi and Bojar, 2017; Zhang et al., 2018; Platanios et al., 2019; Xu et al., 2020; Zhou et al., 2020), whereas different criteria are used to measure the difficulty of sentences. Among these criteria, the length-of-sentence is the simplest and most popular one, which hypothesizes that short sentences will be easy to learn and long sentences will be hard to learn. We test the hypothesis by evaluating whether short sentences overfit and long sentences underfit in a trained seq2seq model.
Length-of-sentence We categorize sentences into short/medium/long sentences according to the sentence length that each bucket is allocated with almost the same number of sentences. On the News dataset, the length of short sentences is between 1 and 18 tokens, the length of medium sentences between 19 and 31, and the length of long sentences between 32 and 792.
Hypothesis Testing. We test our hypothesis us-
ing sign-test on News English-German dataset, obtaining a p-value of 3.6× 10−5 for short-sentence, 2.1× 10−5 for medium-sentence, and 2.6× 10−3 for long-sentence, which indicates overfitting or underfitting. The fitting-offset has an average of 1.95, 2.0, and 1.38 for short/medium/long-sentences, respectively. The positive fitting-offsets suggest that they overfit in the trained models. However, as Figure 9 shows, the degree of overfitting and underfitting is less than that of frequency (Figure 3) and discrepancy (Figure 6).
Summary The above experiments suggest that although the length-of-sentence can differentiate easy sentences from hard sentences, its effectiveness may not be as significant as other factors such as frequency, discrepancy, and parts-of-speech. More surprisingly, short sentences are more likely to underfit than long sentences, which is also confirmed by experiments on pretraining settings in section 6, suggesting that we could not simply judge the short-sentences as easy and longsentences as hard.

heading: 6 Fine-tuning of Pretrained Language Models
text: Fine-tuning on a large pretrained model has become the dominant setting for NLP tasks (Kenton and Toutanova, 2019; Lewis et al., 2020; Brown et al., 2020; Liu et al., 2020). We investigate the overfitting and underfitting issues, particularly in the pretraining setting.
Hypothesis Testing We first test whether the overfitting and underfitting issues exist under the pretraining setting. We experiment by fine-tuning
5 3 1 1 3 5 Fitting-offset (epoch)
High
Med
Low
Fr eq
ue nc
y
(a) Fitting-offset of tokens grouped by frequency.
5 3 1 1 3 5 Fitting-offset (epoch)
Long
Med
Short Le ng
th -o
f-s en
te nc
e
(b) Fitting-offset of tokens grouped by length-of-sentence.
5 3 1 1 3 5 Fitting-offset (epoch)
Func
Symb
Num
Verb
Adjv
Noun
Pa rt-
of -s
pe ec
h
(c) Fitting-offset of tokens grouped by parts-of-speech.
Figure 10: The distribution of fitting-offset on pretraining setting.
mBART25 on the News English-German dataset. First, we evaluate frequency as an indicator of overfitting and underfitting. As Figure 10a shows, we obtain a p-value of 1.2 × 10−3 and an average fitting-offset of −1.48 on high-frequency tokens, suggesting the tendency of overfitting on high-frequency tokens. Results on medium/lowfrequency tokens do not show significance, although the average fitting-offsets of 0.08 on medium-frequency tokens suggest slight underfitting.
Next, we consider length-of-sentence. As Figure 10b shows, the fine-tuning tends to overfit for longsentence and we obtain a p-value of 1.1×10−2 and an average fitting-offset of −1.2 on long-sentences. It suggests obvious overfitting of long sentences, which is counter-intuitive because it is widely assumed that long-sentence is harder to learn than short-sentence.
Last, we examine parts of speech. As Figure 10c shows, fine-tuning on function words tends to overfit and that on nouns underfit, which is consistent with the result on non-pretraining setting (Figure 4). A difference is that the fine-tuning of verbs shows underfitting, which is not significantly observed in the non-pretraining settings. We obtain a p-value of 5.8× 10−2 and an average fitting-offset of 0.8 on verbs, suggesting an observable underfitting on verbs. The issue on function words is more significant than on verbs, on which we obtain a pvalue of 4.3× 10−6 and an average fitting-offset of −2.3, suggesting significant overfitting on function words.
The experiments above confirm that the overfitting and underfitting issues exist in the pretraining setting, although it is not as significant as that in the non-pretraining settings. In addition, it shows that overfitting is the major issue in comparison with
underfitting. We attribute it to the effectiveness of large pretraining to prevent underfitting.
It is worth noting that the vocabulary used by the pre-trained model is different from other nonpretraining settings, which may affect the statistical figures but the conclusions hold.

heading: 7 Additional Factors to the Fitting Issue
text: Most of the figures and tables for this section are placed in Appendix A.
The Language As a comparison of languages, we study the issues on the News dataset but with a reversed direction of languages, translating German to English instead of English to German. As Figure 11 shows, our observations on target English tokens are consistent with previous observations on target German tokens. First, the high-frequency tokens tend to overfit, while the low-frequency tokens tend to underfit. Second, the big-discrepancy tokens tend to overfit, while the small-discrepancy tokens tend to underfit. Third, the function words tend to overfit, while the nouns tend to underfit. Last, we also obtain bigger potential-gains by combining the factors and the most significant potential-gains happen on the consistent categories, such as lowfrequency symbols, high-frequency adj/advs, and big/med-discrepancy numbers, as the bold numbers in Table 5, 6, and 7 show.
The Model Size To evaluate the influence of model size, we conduct experiments with a big model on the News English-German dataset. As Figure 15 shows, the distribution of fitting-offset on each category is very close to that of the base model but with a smaller range. We attribute it to the faster convergence of bigger model (Li et al., 2020). One significant difference between the big model and the base model is that the fitting-offset of symbols moves toward the negative region, sug-
gesting overfitting for symbols in the big model. We attribute it to the stronger memorization ability of the bigger model.
The Domain Previous experiments are done on the News dataset. To justify that the phenomena are not domain specific, we conduct the same experiments on the Europarl English-German dataset, which is from the Europarl domain. We randomly sample 250, 000 sentence pairs from the Europarl training set for a fair comparison with the News dataset, which contains 236, 287 samples.
As Figure 16 shows, the observations described in “The Language” block hold but with slight differences in the distribution of fitting-offset of tokens grouped by parts-of-speech, in comparison with Figure 4 evaluated on News dataset. The major difference happens in verbs, adjectives, and nouns, reflecting a different distribution of topics of Europarl in comparison with News.
The Data Scale Intuitively, overfitting and underfitting should be more severe on small datasets. For comparison, we test on a bigger dataset from Europarl. We sample 500, 000 sentence pairs from the Europarl training set in comparison with the model trained using 250, 000 samples.
As Figure 17 shows, the observations described in “The Language” block still hold but the range of the distribution increases. Looking into the potential-gains, we see that they decrease by about 1/4 compared to that of the experiments with 250, 000 samples. The results suggest that the fitting-offset is more challenging to measure, and the potential-gain decreases when the model is trained on a larger dataset, which is expected due to the larger dataset reducing overfitting and underfitting.

heading: 8 Discussion on Possible Solutions
text: Our observation and insights are based on general language phenomena: long-tail distribution, uneven context dependencies, and different parts of speech, which are general among different NLP tasks, and may deserve further study on possible solutions.
Intuitively, the model training or fine-tuning process could be improved by reducing learning on overfitting tokens and increasing learning on underfitting tokens. One example is the previous work on abstractive summarization (Shi et al., 2018), which uses focal loss to give high-freq tokens smaller weights and low-freq tokens larger weights,
achieved 1.5 points improvement on ROUGE-2 on the abstractive summarization task. Such dynamic token-specific weights can partially address the underfitting and overfitting from the perspective of the frequency factor. However, the issue is far from being solved, given that a combination of the frequency and discrepancy factors (Figure 7) reveals more severe fitting issues than the frequency only (Figure 3). We could expect more gains by designing more appropriate methods to consider the combined factors.

heading: 9 Conclusion
text: We study overfitting and underfitting issues of learning targets in the context of neural machine translation. Our experiments demonstrate that overall rare tokens tend to underfit and frequent tokens overfit. We explored detailed factors related to the overfitting and underfitting issues and identified three major influencing factors, which include frequency, parts-of-speech, and discrepancy. This shows that fitting is the result of a complex interaction between multiple factors. Further experiments demonstrate that the issues exist as a general problem for both non-pretraining and pretraining settings. Future work includes the investigation of strategies to alleviate the overfitting and underfitting issues.
Limitations
This paper hypothesizes that seq2seq models have token-level overfitting and underfitting issues, and provides direct evidence to support the hypothesis in various settings, raising a valuable problem for NLP modeling. However, this paper does not provide a solution to the problem due to the theoretical and practical challenges of measuring the convergence speed of each token. We leave the exploration of this topic to future work.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their valuable feedback. This work is funded by the China Strategic Scientific and Technological Innovation Cooperation Project (grant No. SQ2022YFE020038) and the National Natural Science Foundation of China (grant NSFC No. 62161160339). Zhiyang Teng is partially supported by CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2021-046A).

heading: A Appendix
text: We present the experimental results of the additional factors here, prefixing the name of each factor on the caption.
10 8 6 4 2 0 2 4 6 8 10 Fitting-offset (epoch)
High
Med
Low
Fr eq
ue nc
y
(a) Fitting-offset of tokens grouped by frequency.
10 8 6 4 2 0 2 4 6 8 10 Fitting-offset (epoch)
Big
Med
Small
Di sc
re pa
nc y
(b) Fitting-offset of tokens grouped by discrepancy.
10 8 6 4 2 0 2 4 6 8 10 Fitting-offset (epoch)
Func
Symb
Num
Verb
Adjv
Noun
Pa rt-
of -s
pe ec
h
(c) Fitting-offset of tokens grouped by parts-of-speech.
Figure 11: The Language: Fitting-offset of English tokens evaluated on News German-English dataset.
Big Med Small Discrepancy
10 8 6 4 2 0 2 4 6 8
10
Fit tin
gof
fs et
(e po
ch )
Frequency High Med Low
Figure 12: The Language: Fitting-offset of tokens grouped by discrepancy and frequency.
Discrepancy FrequencyHigh Med Low Big 67.14 +0.21 27.14 -0.27 9.90 +0.20 Med 69.91 +0.07 47.08 -0.08 14.67 +0.78 Small 62.99 +0.33 81.85 +0.33 66.38 +1.35
Table 5: The Language: Potential-gain for each category grouped by frequency and discrepancy.
10 8 6 4 2 0 2 4 6 8 10 Fitting-offset (epoch)
High
Med
Low
Fr eq
ue nc
y
(a) Fitting-offset of tokens grouped by frequency.
10 8 6 4 2 0 2 4 6 8 10 Fitting-offset (epoch)
Big
Med
Small
Di sc
re pa
nc y
(b) Fitting-offset of tokens grouped by discrepancy.
10 8 6 4 2 0 2 4 6 8 10 Fitting-offset (epoch)
Func
Symb
Num
Verb
Adjv
Noun
Pa rt-
of -s
pe ec
h
(c) Fitting-offset of tokens grouped by parts-of-speech.
(a) Fitting-offset of tokens grouped by frequency.
(b) Fitting-offset of tokens grouped by discrepancy.
(c) Fitting-offset of tokens grouped by parts-of-speech.
(a) Fitting-offset of tokens grouped by frequency.
(b) Fitting-offset of tokens grouped by discrepancy.
(c) Fitting-offset of tokens grouped by parts-of-speech.
Figure 17: The Data Scale: Fitting-offset of tokens evaluated on Europarl English-German (500, 000 samples)..

heading: Future Work
text: Future work includes the investigation of strategies to alleviate the overfitting and underfitting issues. Limitations
This paper hypothesizes that seq2seq models have token-level overfitting and underfitting issues, and provides direct evidence to support the hypothesis in various settings, raising a valuable problem for NLP modeling. However, this paper does not provide a solution to the problem due to the theoretical and practical challenges of measuring the convergence speed of each token. We leave the exploration of this topic to future work