SlowGuess/NAACL_2022_OCR huggingface dataset:
Don’t Take It Literally: An Edit-Invariant Sequence Loss for Text Generation
Guangyi Liu<sup>1</sup>, Zichao Yang<sup>2</sup>, Tianhua Tao<sup>3</sup>, Xiaodan Liang<sup>4</sup>, Junwei Bao<sup>5</sup>, Zhen Li<sup>1</sup>, Xiaodong He<sup>5</sup>, Shuguang Cui<sup>1</sup>, Zhiting Hu<sup>6</sup>  
<sup>1</sup>Chinese University of Hong Kong, Shenzhen, <sup>2</sup>Carnegie Mellon University, Tsinghua University, <sup>4</sup>Sun Yat-Sen University, <sup>5</sup>JD AI Research, <sup>6</sup>UC San Diego guangyiliu@link.cuhk.edu.cn, zhh019@ucsd.edu
Neural text generation models are typically trained by maximizing log-likelihood with the sequence cross entropy (CE) loss, which encourages an exact token-by-token match between a target sequence with a generated sequence. Such training objective is sub-optimal when the target sequence is not perfect, e.g., when the target sequence is corrupted with noises, or when only weak sequence supervision is available. To address the challenge, we propose a novel Edit-Invariant Sequence Loss (EISL), which computes the matching loss of a target  $n$ -gram with all  $n$ -grams in the generated sequence. EISL is designed to be robust to various noises and edits in the target sequences. Moreover, the EISL computation is essentially an approximate convolution operation with target  $n$ -grams as kernels, which is easy to implement and efficient to compute with existing libraries. To demonstrate the effectiveness of EISL, we conduct experiments on a wide range of tasks, including machine translation with noisy target sequences, unsupervised text style transfer with only weak training signals, and non-autoregressive generation with non-predefined generation order. Experimental results show our method significantly outperforms the common CE loss and other strong baselines on all the tasks. EISL has a simple API that can be used as a drop-in replacement of the CE loss. $^1$
Neural text generation models have ubiquitous applications in natural language processing, including machine translation (Bahdanau et al., 2015, Sutskever et al., 2014, Wu et al., 2016, Vaswani et al., 2017), summarizations (Nallapati et al., 2016, See et al., 2017), dialogue systems (Li et al., 2016), etc. They are typically trained by maximizing the log-likelihood of the output sequence conditioning on the inputs with the cross entropy (CE) loss. The
Figure 1: Invariance exists in both image and text, e.g., image is invariant to translation (top), and text is robust to many forms of edits (bottom).
CE loss can be easily factorized into individual loss terms and can be optimized efficiently with stochastic gradient descent. Due to its computational efficiency and ease to implement, the training paradigm has played an important role in building successful large text generation models (Lewis et al., 2020, Radford et al., 2019). However, the CE loss minimizes the negative log-likelihood of only the reference output sequence, while all other sequences are equally penalized through normalization. This is over-restrictive since for a given reference target sentence, many possible paraphrases are semantically close, hence should not completely be treated as negative samples. For example, as shown in Figure 1, a cat is on the red blanket should be treated equally with on the red blanket there is a cat. A model trained with CE loss falls short of modeling such type of invariance for text.
The problem is even exaggerated when the supervision from a target sequence is not perfect (Pinnis, 2018). On one hand, there could be noises in the reference sequence which makes itself not a valid sentence. As in the last example shown in Figure 1, there is a repetition error in the target sequence, which is common in human generated text. With
$^{1}$ Code: https://github.com/guangyliu/EISL
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2055 - 2078 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 2: Sensitivity of CE and EISL loss w.r.t different types of text edits as the amount of edits increases (x-axis). We use a fixed machine translation model, synthesize different types of edits on target text, and measure the CE and EISL losses, respectively. The edit types include shuffle (changing the word order), repetition (words being selected are repeated), and word blank (words being replaced with a blank token). CE loss tends to increase drastically once a small amount of edits is applied. In contrast, EISL loss increases much more slowly, showing its robustness.
the CE loss, the model is forced to copy all tokens including the error, and assign a high loss for the grammatically correct sequence. The exact tokens matching renders the CE loss sensitive to noises in the target, as shown in Figure 2. On the other hand, there are many problems with only weak supervision for target sequences (Tan et al., 2020, Wang et al., 2021, Lin et al., 2020). For example, in tasks of unsupervised text style transfer (Jin et al., 2022) aiming to rewrite a sentence from one style to another, the original sentence offers weak supervision for the content (rather than the style). Yet using a CE loss here is problematic since it encourages the model to copy every original token.
Prior works have tried to address this problem using reinforcement learning (RL) (Guo et al., 2021, O'Neill and Bollegala, 2019, Wieting et al., 2019). For example, policy gradient was used to optimize sequence rewards such as BLEU metric (Ranzato et al., 2016, Liu et al., 2017). Such algorithms assign high rewards to sentences that are close to the target sentence. Though it is a valid objective to optimize, policy optimization faces significant challenges in practice. The high variance of gradient estimate makes the training extremely difficult, and almost all previous attempts rely on fine-tuning from models trained with CE loss, often with unclear improvement (Wu et al., 2018).
In this paper, we propose an alternative loss to overcome the above weakness of CE loss, but reserve all nice properties such as being end-to-end differentiable, easy to implement, and efficient to compute, which hence can be used as a drop-in replacement or combined with CE. The loss is based on the observation that a viable candidate sequence shares many sub-sequences with the target. Our loss, called edit-invariant sequence loss (EISL), models the matching of each reference  $n$ -gram across all  $n$ -grams in a candidate sequence. The
design is motivated by the translation invariance properties of ConvNets on images (see Figure 3), and captures the edit invariance properties of text  $n$ -grams in calculating the loss. Figure 2 shows the invariance property of EISL in comparison with CE. Appealingly, we show the conventional CE loss is a special case of EISL—when  $n$  equals to the sequence length, EISL calculates the exact sequence matching loss and reduces to CE. Moreover, the computations of EISL is essentially a convolution operation of candidate sequence using target  $n$ -grams as kernels, which is very easy to implement with existing deep learning libraries.
To demonstrate the effectiveness of EISL loss, we conduct experiments on three representative tasks: machine translation with noisy training target, unsupervised text style transfer (only weak references are available), and non-autoregressive generation with flexible generation order. Experiments demonstrate EISL loss can be easily incorporated with a series of sequence models and outperforms CE and other popular baselines across the board.
Deep neural sequence models such as recurrent neural networks (Sutskever et al., 2014, Mikolov et al., 2010) and transformers (Vaswani et al., 2017) have achieved great progress in many text generation tasks like machine translation (Bahdanau et al., 2015, Vaswani et al., 2017). These models are typically trained with the maximum-likelihood objective, which can lead to sub-optimal performance due to CE's exact sequence matching assumption. There are lots of works trying to overcome this weakness. For examples, some works (Ranzato et al., 2016, Rennie et al., 2017, Liu et al., 2017, Shen et al., 2016, Smith and Eisner, 2006) proposed to use policy gradient or minimum risk training to optimize the expected BLEU metric (Papineni
et al., 2002a). Due to the high variance and unstableness in RL training, a variety of training tricks are used in practice. Wieting et al. (2019) developed a new reward function based on semantic similarity for translation. Guo et al. (2021) introduced soft Q-learning for more efficient RL training. On the other hand, Zhukov and Kretov (2017), Casas et al. (2018) made the initial attempts to develop differentiable BLEU objectives, making soft approximations to the count of  $n$ -gram matching in the original BLEU formulation. Shao et al. (2018, 2021, 2020) minimized the  $n$ -gram difference between the model outputs and targets in non-autoregressive generation.
Another line of research that is relevant to our work is learning with noisy labels in classification (Zhang and Sabuncu, 2018, Xu et al., 2019, Wang et al., 2019b, Hu et al., 2019). For text generation, Nicolai and Silfverberg (2020) proposed student forcing to substitute teacher forcing, which can alleviate the influence of noise in the target sequence during decoding. Kang and Hashimoto (2020) proposed loss truncation, which adaptively removes high-loss examples considered as invalid data. Our empirical study shows substantial improvement of our approach over the previous ones.
3 Edit-Invariant Sequence Loss
In this section, we first review the conventional cross-entropy (CE) loss for sequence learning, and point out its weakness, especially when the target sequence is edited. We then introduce the EISL loss which gives a model the flexibility to learn from sub-sequences in a target sequence.
We first establish notations for the sequence generation setting. Let  $(\pmb{x},\pmb{y}^{*})$  be a paired data sample where  $\pmb{x}$  is the input and  $\pmb{y}^{*} = (y_{1}^{*},\dots,y_{T^{*}}^{*})$  is the reference target sequence. Define  $\pmb{y} = (y_{1},\dots,y_{T})$  as a candidate sentence. Our goal is to build a model  $p_{\theta}(\pmb{y}|\pmb{x})$  that scores a candidate sequence  $\pmb{y}$  with parameter  $\pmb{\theta}$ . In the sequel, we omit the condition  $\pmb{x}$  and the subscript  $\pmb{\theta}$  for simplicity.
3.1 The Difficulty of Cross Entropy Loss
The standard approach to learn the sequence model is to minimize the negative log-likelihood (NLL) of the target sequence, i.e., minimizing the CE loss  $\mathcal{L}^{\mathrm{CE}}(\pmb {\theta}) = -\log p(\pmb {y}^{*})$  . The CE loss assumes exact matching of a candidate sequence  $\pmb{y}$  with the target sequence  $y^{*}$  . In other words, it maximizes the probability of only the target sequence  $y^{*}$  while penalizing all other possible sequence outputs that
might be close but different with  $\pmb{y}^{*}$
The assumption can be problematic in many practical scenarios: (1) For a given target sentence, there could be many ways of paraphrasing the sentence such as word reordering, synonyms replacement, active to passive rewriting, etc. Many of the paraphrases are viable candidate sequences, and/or share many sub-sequences with the reference sentence, and thus should not be treated completely as negative samples. Similar to the translation invariance which is shown to be effective in image modeling, a sequence loss that is robust to the shift and edits of sub-sequences in the reference sequence is preferred in order to model the rich variations of sequences; (2) The edit-invariance property is particularly desirable when the reference target sequence is corrupted with noise or is only weak sequence supervision. For instance, in Figure 3, the word is is repeated twice, which is one of the common errors in typing. Using CE loss in the noisy target setting forces the model to learn the data errors as well. In contrast, a sequence loss robust or invariant to the shift of sub-sequences assigns a high probability to the correct sentence even though it does not match the noisy target exactly. The loss thus offers flexibility for the model to select right information for learning.
3.2 EISL: Edit-Invariant Sequence Loss
Motivated by the above discussion, in this section, we draw inspirations from the convolution operation that enables translation invariance in image modeling (Figure 3, left), and propose an edit-invariant sequence loss (EISL) as illustrated in Figure 3 (right). Intuitively, for instance, given a 4-gram on the red blanket, because there is no extra knowledge to determine the position of the 4-gram in the noisy target sequence, we compute the losses across all positions in the noisy target sequence and aggregate. This is essentially a convolution over the target noisy sequence with the given  $n$ -gram as a convolution kernel.
We now derive the EISL loss in more details. Let  $\mathbf{y}_{a:b} = (y_a, \dots, y_{b-1})$  denote a sub-sequence of  $\mathbf{y}$  that starts from index  $a$  and ends at index  $b-1$ , which is of length  $b-a$ . Thus  $\mathbf{y}_{i:i+n}^*$  denotes the  $i$ -th  $n$ -gram in the reference  $\mathbf{y}^*$ . Denote  $C(\mathbf{y}_{i:i+n}^*, \mathbf{y})$  as the number of times this  $n$ -gram occurs in  $\mathbf{y}$ :
$$
C \left(\boldsymbol {y} _ {i: i + n} ^ {*}, \boldsymbol {y}\right) = \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} \mathbb {1} \left(\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*}\right), \tag {1}
$$
where  $\mathbb{1}(\cdot)$  is the indicator function that takes value
Figure 3: Inspired by the ConvNet convolution which applies a convolution kernel to different positions in an image and aggregate (left), we devise similar  $n$ -gram matching and convolution, which is robust to sequence edits (noises, shuffle, repetition, etc) (right).
1 if the  $n$ -grams match, and 0 otherwise. Intuitively, for a text generation model, we would like to maximize the occurrence of an  $n$ -gram from the reference in the target sequence. For a given probabilistic model  $p_{\theta}(\boldsymbol{y})$  (we omit the parameter  $\theta$  wherever the meaning is clear), the expected value of  $C(\boldsymbol{y}_{i:i+n}^{*}, \boldsymbol{y})$  can be computed as follow:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {y} \sim p (\boldsymbol {y})} [ C (\boldsymbol {y} _ {i: i + n} ^ {*}, \boldsymbol {y}) ] \\ = \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} \mathbb {E} _ {p \left(\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n}\right)} \left[ \mathbb {1} \left(\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*}\right) \right] \tag {2} \\ = \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} p \left(\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*}\right). \\ \end{array}
$$
Thus, for each  $i$ -th  $n$ -gram in the reference, a straightforward way to define the learning objective is to minimize the negative log value of its expected occurrence, i.e.,  $-\log \mathbb{E}_{\pmb{y}\sim p(\pmb{y})}[C(\pmb{y}_{i:i + n}^{*},\pmb{y})]$ .
The above loss requires computation of the marginal probability  $p(\pmb{y}_{i':i' + n} = \pmb{y}_{i:i + n}^{*})$  of an  $n$ -gram, which is intractable in practice. We therefore derive an upper bound of the loss and use it as the surrogate to minimize in training. We denote the upper bound surrogate as our EISL loss. Specifically, since for a given  $i'$ ,  $p(\pmb{y}_{i':i' + n} = \pmb{y}_{i:i + n}^{*}) = \sum_{\pmb{y}} p(\pmb{y}_{< i'}) p(\pmb{y}_{i':i' + n} = \pmb{y}_{i:i + n}^{*} | \pmb{y}_{< i'})$ , then:
$$
\begin{array}{l} - \log \mathbb {E} _ {\boldsymbol {y} \sim p (\boldsymbol {y})} [ C (\boldsymbol {y} _ {i: i + n} ^ {*}, \boldsymbol {y}) ] \\ = - \log \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} p \left(\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*}\right), \\ \leq \frac {- \mathbb {E} _ {\boldsymbol {y} \sim p (\boldsymbol {y})} \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} \log p \left(\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*} \mid \boldsymbol {y} _ {<   i ^ {\prime}}\right)}{T - n + 1} \\ := \mathcal {L} _ {n, i} ^ {\text {E I S L}} (\boldsymbol {\theta}). \tag {3} \\ \end{array}
$$
The detailed derivation is attached in Appendix A.1. Notice that the EISL loss involves only the conditional distribution  $p(\pmb{y}_{i':i' + n} = \pmb{y}_{i:i + n}^{*}|\pmb{y}_{< i'})$  which is convenient to compute—we first sample tokens from the model up to the  $i'$  position, then compute NLL of the reference  $n$ -gram  $\pmb{y}_{i:i + n}^{*}$  occurring at
position  $i'$  under the model distribution. The full  $n$ -gram EISL loss is then defined by averaging across all  $n$ -gram positions in the reference:
$$
\mathcal {L} _ {n} ^ {\mathrm {E I S L}} (\boldsymbol {\theta}) = \frac {1}{T ^ {*} - n + 1} \sum_ {i = 1} ^ {T ^ {*} - n + 1} \mathcal {L} _ {n, i} ^ {\mathrm {E I S L}} (\boldsymbol {\theta}). \tag {4}
$$
In practice, inspired by the standard BLEU metric (more in section 3.3), we could also straightforwardly combine different  $n$ -gram losses depending on tasks:
$$
\mathcal {L} ^ {\mathrm {E I S L}} (\boldsymbol {\theta}) = \sum_ {n} w _ {n} \cdot \mathcal {L} _ {n} ^ {\mathrm {E I S L}} (\boldsymbol {\theta}), \tag {5}
$$
where  $w_{n}$  is the weight of the  $n$ -gram loss. The rule of thumb is that a  $n$ -gram EISL loss with lower  $n$  is more robust to noises, as shown in our experiments. Following BLEU, we found that simply using equal weights for different  $n$ -grams up to  $n = 4$  often produces good performance.
As discussed shortly, it is appealing that the  $n$ -gram EISL loss is indeed a direct generalization of the CE loss on the  $n$ -gram level: we sum the CE loss of an  $n$ -gram over all candidate sequence positions by conditioning on samples from the model. Besides, the derivation of the upper bound makes no assumption on the probability function  $p(\pmb{y})$ , hence holds for both autoregressive and non-autoregressive sequence models as demonstrated in our experiments.
Position Selection Minimizing the gram matching loss over all positions can make the model assign equal probabilities at all positions, which causes the training to collapse. We further adapt the loss to enable the model to automatically learn the positions of reference  $n$ -grams. For notation simplicity, let  $g_{i,i'}^n$  denote the conditional probability  $p(\pmb{y}_{i':i' + n} = \pmb{y}_{i:i + n}^* | \pmb{y}_{< i'}^)$  involved above (Eq.3). We can vectorize the probability to get  $\pmb{g}_i^n = [g_{i,1}^n, \dots, g_{i,T - n + 1}^n]^T$ , spanning all potential positions in the candidate sequence. We then normalize the probability vector  $\pmb{g}_i^n$  by Gumbel
Figure 4: As convolution is a common operation for translation invariance in image, we adopt a convolution to achieve the translation invariance in text. The input is the distribution from the model output in log domain, kernel represents the convolution kernel and * is the convolution operation. In this 3-gram example, there are 5 kernels, which correspond to the 5 rows on the right.
softmax (Jang et al., 2017), denoted as  $\pmb{q}_i^n =$  Gumbel softmax  $(\pmb{g}_i^n)$ , which we use as the weight for every  $n$ -gram positions. We multiply the weight with the original log probability to get the new adjusted loss:
$$
\mathcal {L} _ {n, i} ^ {\mathrm {E I S L}} (\boldsymbol {\theta}) \approx - \boldsymbol {q} _ {i} ^ {n} \cdot \log \boldsymbol {g} _ {i} ^ {n}. \tag {6}
$$
The loss can roughly be viewed as the "entropy" of the unnormalized probabilities  $g_{i}^{n}$ , which has minimal value if the mass of the probability is assigned to one location only. Intuitively, if an  $g_{i,i'}^{n}$  is large, then it is likely  $i'$  is the correct position for the reference  $n$ -gram, hence the weight for this position should also be large. This is like the greedy exploitation in reinforcement learning (Mnih et al., 2015). On the other hand, to overcome over-exploitation, the Gumbel softmax introduces randomness in the weight assignment, which helps balance the exploitation-exploration trade-off in position selection for the model.
Efficient Approximate Computation: EISL as Convolution We show the EISL loss can be computed efficiently using the common convolution operator, with very little additional cost compared with the CE loss. The computation involves moderate approximation if the generation model is an autoregressive model, and is exact in the case of a non-autoregressive model (e.g., as in section 4.3). We first discuss the easy case when the model is a non-autoregressive model, where we have  $g_{i,i'}^n = p(\pmb{y}_{i':i' + n} = \pmb{y}_{i:i + n}^* |\pmb{y}_{< i'}) = \prod_{j=1}^n p(y_{i' + j - 1} = y_{i+j - 1}^*)$ . Denote  $V$  as the vocabulary size. Let  $P = [p_1, p_2, \dots, p_T]$  be the probability output by the model across positions, where  $p_{i'} \in \mathbb{R}^V$  is the probability output after softmax at  $i'$ -th position, and each  $p_{i'}$  is independent with
each other. On this basis, we compute the key quantity  $\log g_i^n$  in Eq. 6 as the direct output of the convolution operator. As shown in Figure 4, we can get  $\log g_i^n$  by applying convolution on  $\log P$ , with  $y_{i:i+n}$  as the kernels:
$$
\log \boldsymbol {g} _ {i} ^ {n} = \operatorname {C o n v} \left(\log \boldsymbol {P}, \text {O n e h o t} \left(\boldsymbol {y} _ {i: i + n} ^ {*}\right)\right), \tag {7}
$$
where  $\operatorname{Onehot}(\cdot)$  maps each token to its corresponding one-hot representation and  $\operatorname{Conv}(\cdot, \cdot)$  is the convolution operation with the first argument as input and the second as the kernel. We transform  $\pmb{P}$  into log domain to turn the probability multiplication into log probability summations, where  $\operatorname{Conv}$  can be directly applied. As shown in Figure 4,  $\log P$  is of shape  $V \times T$  and  $\operatorname{Onehot}(\pmb{y}_{i:i+n}^{*})$  is of shape  $V \times n$ , so  $\operatorname{Conv}(\log P, \operatorname{Onehot}(\pmb{y}_{i:i+n}^{*}))$  is an one-dimensional convolution on the sequence axis. Formally, the  $i'$ -th convolutional output is:
$$
\begin{array}{l} \log g _ {i, i ^ {\prime}} ^ {n} = \sum_ {j = 1} ^ {n} \log \boldsymbol {p} _ {i ^ {\prime} + j - 1} \cdot \text {O n e h o t} \left(y _ {i + j - 1} ^ {*}\right) \tag {8} \\ = \sum_ {j = 1} ^ {n} \log p \left(y _ {i ^ {\prime} + j - 1} = y _ {i + j - 1} ^ {*} \mid \mathbf {y} _ {<   i ^ {\prime} + j - 1}\right) \\ \end{array}
$$
After obtaining  $g_{i}^{n}$  by convolution, the EISL loss in Eq. 6 can be easily calculated. We now discuss the case of autoregressive model, where by definition we have  $g_{i,i'}^{n} = \prod_{j=1}^{n} p(y_{i' + j - 1} = y_{i + j - 1}^{*} | y_{<i'}, y_{i:i + j - 1}^{*})$ . The dependence on both  $y_{<i'}$  and  $y_{i:i + j - 1}^{*}$  in each conditional makes exact estimation of  $\log g_{i}^{n}$  very complicated and costly. We thus introduce the approximation where we approximate  $g_{i,i'}^{n}$  as  $\widetilde{g}_{i,i'}^{n} = \prod_{j=1}^{n} p(y_{i' + j - 1} = y_{i + j - 1}^{*} | y_{<i' + j - 1})$ . That is, instead of conditioning on  $y_{i:i + j - 1}^{*}$ , we use the model-generated tokens  $y_{i':i' + j - 1}$  as the condition. This simple approximation enables us to define the probability output
Figure 5: Results of Translation with Noisy Target on German-to-English(de-en) from Multi30k. BLEU scores are computed against clean test data. The  $x$ -axis of all figures denotes the level of noise we injected to target sequences in training. (a) Shuffle: selected tokens are shuffled; (b) Repetition: selected tokens are repeated; (c) Blank: selected tokens are substituted with a special blank token; (d) Synthetical noise: the combination of all three noises ( $x = x_0$  stands for the combination of  $5x_0\%$  of all kinds of noises); (e) Ablation study of  $n$ -grams for EISL on synthetical noise. BLEURT results are shown in Appendix A.3.