Venue: NAACL 2022

# Don't Take It Literally:

An Edit-Invariant Sequence Loss for Text Generation

 Guangyi Liu\({}^{1}\),  Zichao Yang\({}^{2}\), Tianhua Tao\({}^{3}\), Xiaodan Liang\({}^{4}\), Junwei Bao\({}^{5}\), Zhen Li\({}^{1}\), Xiaodong He\({}^{5}\), Shuguang Cui\({}^{1}\), Zhiting Hu\({}^{6}\)

###### Abstract

Neural text generation models are typically trained by maximizing log-likelihood with the sequence cross entropy (CE) loss, which encourages an _exact_ token-by-token match between a target sequence with a generated sequence. Such training objective is sub-optimal when the target sequence is not perfect, e.g., when the target sequence is corrupted with noises, or when only weak sequence supervision is available. To address the challenge, we propose a novel Edit-Invariant Sequence Loss (EISL), which computes the matching loss of a target \(n\)-gram with all \(n\)-grams in the generated sequence. EISL is designed to be robust to various noises and edits in the target sequences. Moreover, the EISL computation is essentially an approximate convolution operation with target \(n\)-grams as kernels, which is easy to implement and efficient to compute with existing libraries. To demonstrate the effectiveness of EISL, we conduct experiments on a wide range of tasks, including machine translation with noisy target sequences, unsupervised text style transfer with only weak training signals, and non-autoregressive generation with non-predefined generation order. Experimental results show our method significantly outperforms the common CE loss and other strong baselines on all the tasks. EISL has a simple API that can be used as a drop-in replacement of the CE loss.1

Footnote 1: Code: [https://github.com/guangyliu/EISL](https://github.com/guangyliu/EISL)

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2055 - 2078

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Neural text generation models have ubiquitous applications in natural language processing, including machine translation Bahdanau et al. (2015); Sutskever et al. (2014); Wu et al. (2016); Vaswani et al. (2017), summarizations Nallapati et al. (2016); See et al. (2017), dialogue systems Li et al. (2016), etc. They are typically trained by maximizing the log-likelihood of the output sequence conditioning on the inputs with the cross entropy (CE) loss. The CE loss can be easily factorized into individual loss terms and can be optimized efficiently with stochastic gradient descent. Due to its computational efficiency and ease to implement, the training paradigm has played an important role in building successful large text generation models Lewis et al. (2020); Radford et al. (2019). However, the CE loss minimizes the negative log-likelihood of only the reference output sequence, while all other sequences are equally penalized through normalization. This is over-restrictive since for a given reference target sentence, many possible paraphrases are semantically close, hence should not completely be treated as negative samples. For example, as shown in Figure 1, a cat is on the red blanket should be treated equally with on the red blanket there is a cat. A model trained with CE loss falls short of modeling such type of invariance for text.

The problem is even exaggerated when the supervision from a target sequence is not perfect Pinnis (2018). On one hand, there could be _noises_ in the reference sequence which makes itself not a valid sentence. As in the last example shown in Figure 1, there is a repetition error in the target sequence, which is common in human generated text. With

Figure 1: Invariance exists in both image and text, e.g., image is invariant to translation (top), and text is robust to many forms of edits (bottom).



the CE loss, the model is forced to copy all tokens including the error, and assign a high loss for the grammatically correct sequence. The exact tokens matching renders the CE loss sensitive to noises in the target, as shown in Figure 2. On the other hand, there are many problems with only _weak_ supervision for target sequences Tan et al. (2020); Wang et al. (2021); Lin et al. (2020). For example, in tasks of unsupervised text style transfer Jin et al. (2022) aiming to rewrite a sentence from one style to another, the original sentence offers weak supervision for the content (rather than the style). Yet using a CE loss here is problematic since it encourages the model to copy every original token.

Prior works have tried to address this problem using reinforcement learning (RL) Guo et al. (2021); O'Neill and Bollegala (2019); Wieting et al. (2019). For example, policy gradient was used to optimize sequence rewards such as BLEU metric Ranzato et al. (2016); Liu et al. (2017). Such algorithms assign high rewards to sentences that are close to the target sentence. Though it is a valid objective to optimize, policy optimization faces significant challenges in practice. The high variance of gradient estimate makes the training extremely difficult, and almost all previous attempts rely on fine-tuning from models trained with CE loss, often with unclear improvement Wu et al. (2018).

In this paper, we propose an alternative loss to overcome the above weakness of CE loss, but reserve all nice properties such as being end-to-end differentiable, easy to implement, and efficient to compute, which hence can be used as a drop-in replacement or combined with CE. The loss is based on the observation that a viable candidate sequence shares many sub-sequences with the target. Our loss, called _edit-invariant sequence loss_ (EISL), models the matching of each reference \(n\)-gram across all \(n\)-grams in a candidate sequence. The design is motivated by the translation invariance properties of ConvNets on images (see Figure 3), and captures the edit invariance properties of text \(n\)-grams in calculating the loss. Figure 2 shows the invariance property of EISL in comparison with CE. Appealingly, we show the conventional CE loss is a special case of EISL--when \(n\) equals to the sequence length, EISL calculates the exact sequence matching loss and reduces to CE. Moreover, the computations of EISL is essentially a convolution operation of candidate sequence using target \(n\)-grams as kernels, which is very easy to implement with existing deep learning libraries.

To demonstrate the effectiveness of EISL loss, we conduct experiments on three representative tasks: machine translation with _noisy_ training target, unsupervised text style transfer (only _weak_ references are available), and non-autoregressive generation with _flexible generation order_. Experiments demonstrate EISL loss can be easily incorporated with a series of sequence models and outperforms CE and other popular baselines across the board.

## 2 Related Work

Deep neural sequence models such as recurrent neural networks Sutskever et al. (2014); Mikolov et al. (2010) and transformers Vaswani et al. (2017) have achieved great progress in many text generation tasks like machine translation Bahdanau et al. (2015); Vaswani et al. (2017). These models are typically trained with the maximum-likelihood objective, which can lead to sub-optimal performance due to CE's exact sequence matching assumption. There are lots of works trying to overcome this weakness. For examples, some works Ranzato et al. (2016); Ronnie et al. (2017); Liu et al. (2017); Shen et al. (2016); Smith and Eisner (2006) proposed to use policy gradient or minimum risk training to optimize the expected BLEU metric Papineni

Figure 2: Sensitivity of CE and EISL loss w.r.t different types of text edits as the amount of edits increases (x-axis). We use a fixed machine translation model, synthesize different types of edits on target text, and measure the CE and EISL losses, respectively. The edit types include shuffle (changing the word order), repetition (words being selected are repeated), and word blank (words being replaced with a blank token). CE loss tends to increase drastically once a small amount of edits is applied. In contrast, EISL loss increases much more slowly, showing its robustness.

 

et al., 2002a). Due to the high variance and unstableness in RL training, a variety of training tricks are used in practice. Wieting et al. (2019) developed a new reward function based on semantic similarity for translation. Guo et al. (2021) introduced soft Q-learning for more efficient RL training. On the other hand, Zhukov and Kretov (2017), Casas et al. (2018) made the initial attempts to develop differentiable BLEU objectives, making soft approximations to the count of \(n\)-gram matching in the original BLEU formulation. Shao et al. (2018, 2021, 2020) minimized the \(n\)-gram difference between the model outputs and targets in non-autoregressive generation.

Another line of research that is relevant to our work is learning with noisy labels in classification (Zhang and Sabuncu, 2018; Xu et al., 2019; Wang et al., 2019b; Hu et al., 2019). For text generation, Nicolai and Silverberg (2020) proposed student forcing to substitute teacher forcing, which can alleviate the influence of noise in the target sequence during decoding. Kang and Hashimoto (2020) proposed loss truncation, which adaptively removes high-loss examples considered as invalid data. Our empirical study shows substantial improvement of our approach over the previous ones.

## 3 Edit-Invariant Sequence Loss

In this section, we first review the conventional cross-entropy (CE) loss for sequence learning, and point out its weakness, especially when the target sequence is edited. We then introduce the EISL loss which gives a model the flexibility to learn from sub-sequences in a target sequence.

We first establish notations for the sequence generation setting. Let \(({\mathbf{x}},{\mathbf{y}}^{*})\) be a paired data sample where \({\mathbf{x}}\) is the input and \({\mathbf{y}}^{*}=(y_{1}^{*},...,y_{T^{*}}^{*})\) is the reference target sequence. Define \({\mathbf{y}}=(y_{1},...,y_{T})\) as a candidate sentence. Our goal is to build a model \(p_{\boldsymbol{\theta}}({\mathbf{y}}|{\mathbf{x}})\) that scores a candidate sequence \({\mathbf{y}}\) with parameter \({\boldsymbol{\theta}}\). In the sequel, we omit the condition \({\mathbf{x}}\) and the subscript \({\boldsymbol{\theta}}\) for simplicity.

### The Difficulty of Cross Entropy Loss

The standard approach to learn the sequence model is to minimize the negative log-likelihood (NLL) of the target sequence, i.e., minimizing the CE loss \(\mathcal{L}^{\text{CE}}({\boldsymbol{\theta}})=-\log p({\mathbf{y}}^{*})\). The CE loss assumes _exact_ matching of a candidate sequence \({\mathbf{y}}\) with the target sequence \({\mathbf{y}}^{*}\). In other words, it maximizes the probability of only the target sequence \({\mathbf{y}}^{*}\) while penalizing all other possible sequence outputs that might be close but different with \({\mathbf{y}}^{*}\).

The assumption can be problematic in many practical scenarios: **(1)** For a given target sentence, there could be many ways of paraphrasing the sentence such as word reordering, synonyms replacement, active to passive rewriting, etc. Many of the paraphrases are viable candidate sequences, and/or share many sub-sequences with the reference sentence, and thus should not be treated completely as negative samples. Similar to the translation invariance which is shown to be effective in image modeling, a sequence loss that is _robust_ to the shift and edits of sub-sequences in the reference sequence is preferred in order to model the rich variations of sequences; **(2)** The edit-invariance property is particularly desirable when the reference target sequence is corrupted with noise or is only weak sequence supervision. For instance, in Figure 3, the word is repeated twice, which is one of the common errors in typing. Using CE loss in the noisy target setting forces the model to learn the data errors as well. In contrast, a sequence loss robust or invariant to the shift of sub-sequences assigns a high probability to the correct sentence even though it does not match the noisy target exactly. The loss thus offers flexibility for the model to select right information for learning.

### EISL: Edit-Invariant Sequence Loss

Motivated by the above discussion, in this section, we draw inspirations from the convolution operation that enables translation invariance in image modeling (Figure 3, left), and propose an edit-invariant sequence loss (EISL) as illustrated in Figure 3 (right). Intuitively, for instance, given a 4-gram on the red blanket, because there is no extra knowledge to determine the position of the 4-gram in the noisy target sequence, we compute the losses across all positions in the noisy target sequence and aggregate. This is essentially a convolution over the target noisy sequence with the given \(n\)-gram as a convolution kernel.

We now derive the EISL loss in more details. Let \({\mathbf{y}}_{a:b}=(y_{a},...,y_{b-1})\) denote a sub-sequence of \({\mathbf{y}}\) that starts from index \(a\) and ends at index \(b-1\), which is of length \(b-a\). Thus \({\mathbf{y}}_{i:i+n}^{*}\) denotes the \(i\)-th \(n\)-gram in the reference \({\mathbf{y}}^{*}\). Denote \(C({\mathbf{y}}_{i:i+n}^{*},{\mathbf{y}})\) as the number of times this \(n\)-gram occurs in \({\mathbf{y}}\):

\[C({\mathbf{y}}_{i:i+n}^{*},{\mathbf{y}})=\sum_{i^{\prime}=1}^{T-n+1}\mathbb{1}({\mathbf{ y}}_{i^{\prime}:i^{\prime}+n}={\mathbf{y}}_{i:i+n}^{*}), \tag{1}\]

where \(\mathbbm{1}(\cdot)\) is the indicator function that takes value 

\(1\) if the \(n\)-grams match, and \(0\) otherwise. Intuitively, for a text generation model, we would like to maximize the occurrence of an \(n\)-gram from the reference in the target sequence. For a given probabilistic model \(p_{\mathbf{\theta}}(\mathbf{y})\) (we omit the parameter \(\mathbf{\theta}\) wherever the meaning is clear), the expected value of \(C(\mathbf{y}^{*}_{i:i+n},\mathbf{y})\) can be computed as follow:

\[\begin{array}{l}\mathbb{E}_{\mathbf{y}\sim p(\mathbf{y})}[C(\mathbf{y}^{*}_{i:i+n},\mathbf{ y})]\\ =\sum\limits_{i=1}^{T-n+1}\mathbb{E}_{p(\mathbf{y}_{i^{\prime};i^{\prime}+n})}\left[ \mathbbm{1}(\mathbf{y}_{i^{\prime};i^{\prime}+n}=\mathbf{y}^{*}_{i:i+n})\right]\\ =\sum\limits_{i^{\prime}=1}^{T-n+1}p(\mathbf{y}_{j^{\prime};i^{\prime}+n}=\mathbf{y}^{ *}_{i:i+n}).\end{array} \tag{2}\]

Thus, for each \(i\)-th \(n\)-gram in the reference, a straightforward way to define the learning objective is to minimize the negative log value of its expected occurrence, i.e., \(-\log\mathbb{E}_{\mathbf{y}\sim p(\mathbf{y})}[C(\mathbf{y}^{*}_{i:i+n},\mathbf{y})]\).

The above loss requires computation of the marginal probability \(p(\mathbf{y}_{i^{\prime}:i^{\prime}+n}=\mathbf{y}^{*}_{i:i+n})\) of an \(n\)-gram, which is intractable in practice. We therefore derive an upper bound of the loss and use it as the surrogate to minimize in training. We denote the upper bound surrogate as our EISL loss. Specifically, since for a given \(i^{\prime}\), \(p(\mathbf{y}_{i^{\prime}:i^{\prime}+n}=\mathbf{y}^{*}_{i:i+n})=\sum_{\mathbf{y}}p(\mathbf{y}_ {i^{\prime}:i^{\prime}+n}=\mathbf{y}^{*}_{i:i+n}|\mathbf{y}_{<i^{\prime}})\), then:

\[\begin{split}&-\log\mathbb{E}_{\mathbf{y}\sim p(\mathbf{y})}[C(\mathbf{y}^{*}_ {i:i+n},\mathbf{y})]\\ &=-\log\sum\limits_{i^{\prime}=1}^{T-n+1}p(\mathbf{y}_{i^{\prime}:i^ {\prime}+n}=\mathbf{y}^{*}_{i:i+n}),\\ &\leq\frac{-\mathbb{E}_{\mathbf{y}\sim p(\mathbf{y})}\sum\nolimits_{i^{ \prime}=1}^{T-n+1}\log p(\mathbf{y}_{i^{\prime}:i^{\prime}+n}=\mathbf{y}^{*}_{i:i+n}| \mathbf{y}_{<i^{\prime}})}{T-n+1}\\ &:=\mathcal{L}^{\text{EISL}}_{n,i}(\mathbf{\theta}).\end{split} \tag{3}\]

The detailed derivation is attached in Appendix A.1. Notice that the EISL loss involves only the conditional distribution \(p(\mathbf{y}_{i^{\prime}:i^{\prime}+n}=\mathbf{y}^{*}_{i:i+n}|\mathbf{y}_{<i^{\prime}})\) which is convenient to compute--we first sample tokens from the model up to the \(i^{\prime}\) position, then compute NLL of the reference \(n\)-gram \(\mathbf{y}^{*}_{i:i+n}\) occurring at position \(i^{\prime}\) under the model distribution. The full \(n\)-gram EISL loss is then defined by averaging across all \(n\)-gram positions in the reference:

\[\mathcal{L}^{\text{EISL}}_{n}(\mathbf{\theta})=\frac{1}{T^{*}-n+1}\sum\limits_{i= 1}^{T^{*}-n+1}\mathcal{L}^{\text{EISL}}_{n,i}(\mathbf{\theta}). \tag{4}\]

In practice, inspired by the standard BLEU metric (more in section 3.3), we could also straightforwardly combine different \(n\)-gram losses depending on tasks:

\[\mathcal{L}^{\text{EISL}}(\mathbf{\theta})=\sum\nolimits_{n}w_{n}\cdot\mathcal{ L}^{\text{EISL}}_{n}(\mathbf{\theta}), \tag{5}\]

where \(w_{n}\) is the weight of the \(n\)-gram loss. The rule of thumb is that a \(n\)-gram EISL loss with lower \(n\) is more robust to noises, as shown in our experiments. Following BLEU, we found that simply using equal weights for different \(n\)-grams up to \(n=4\) often produces good performance.

As discussed shortly, it is appealing that the \(n\)-gram EISL loss is indeed a direct generalization of the CE loss on the \(n\)-gram level: we sum the CE loss of an \(n\)-gram over all candidate sequence positions by conditioning on samples from the model. Besides, the derivation of the upper bound makes no assumption on the probability function \(p(\mathbf{y})\), hence holds for both autogressive and non-autoregressive sequence models as demonstrated in our experiments.

**Position Selection** Minimizing the gram matching loss over all positions can make the model assign equal probabilities at all positions, which causes the training to collapse. We further adapt the loss to enable the model to automatically learn the positions of reference \(n\)-grams. For notation simplicity, let \(g^{n}_{i,i^{\prime}}\) denote the conditional probability \(p(\mathbf{y}_{i^{\prime}:i^{\prime}+n}=\mathbf{y}^{*}_{i:i+n}|\mathbf{y}_{<i^{\prime}})\) involved above (Eq.3). We can vectorize the probability to get \(\mathbf{g}^{n}_{i}=[g^{n}_{i,1},...,g^{n}_{i,T-n+1}]^{T}\), spanning all potential positions in the candidate sequence. We then normalize the probability vector \(\mathbf{g}^{n}_{i}\) by Gumbel

Figure 3: Inspired by the ConvNet convolution which applies a convolution kernel to different positions in an image and aggregate (**left**), we devise similar \(n\)-gram matching and convolution, which is robust to sequence edits (noises, shuffle, repetition, etc) (**right**).

 

softmax (Jang et al., 2017), denoted as \({\mathbf{q}_{i}^{n}=\text{Gumbel${}_{\text{softmax}}$}({\mathbf{g}_{i}^{n}})}\), which we use as the weight for every \(n\)-gram positions. We multiply the weight with the original log probability to get the new adjusted loss:

\[\mathcal{L}_{n,i}^{\text{EISL}}({\mathbf{\theta}})\approx-{\mathbf{q}_{i}^{n}}\cdot \log g_{i}^{n}. \tag{6}\]

The loss can roughly be viewed as the "entropy" of the unnormalized probabilities \({\mathbf{g}_{i}^{n}}\), which has minimal value if the mass of the probability is assigned to one location only. Intuitively, if an \(g_{i;i^{\prime}}^{n}\) is large, then it is likely \(i^{\prime}\) is the correct position for the reference \(n\)-gram, hence the weight for this position should also be large. This is like the greedy exploitation in reinforcement learning (Mnih et al., 2015). On the other hand, to overcome over-exploitation, the Gumbel softmax introduces randomness in the weight assignment, which helps balance the exploitation-exploration trade-off in position selection for the model.

**Efficient Approximate Computation: EISL as Convolution** We show the EISL loss can be computed efficiently using the common convolution operator, with very little additional cost compared with the CE loss. The computation involves moderate approximation if the generation model is an autoregressive model, and is exact in the case of a non-autoregressive model (e.g., as in section 4.3). We first discuss the easy case when the model is a non-autoregressive model, where we have \(g_{i,i^{\prime}}^{n}=p({\mathbf{y}_{t^{\prime}i^{\prime}+n}}={\mathbf{y}_{i::n}^{*}}| {\mathbf{y}_{<i^{\prime}}})=\prod_{j=1}^{n}p(y_{i^{\prime}+j-1}=y_{i+j-1}^{*})\). Denote \(V\) as the vocabulary size. Let \({\mathbf{P}}=[{\mathbf{p}_{1}},{\mathbf{p}_{2}},...{\mathbf{p}_{T}}]\) be the probability output by the model across positions, where \({\mathbf{p}_{i^{\prime}}}\in\mathbb{R}^{V}\) is the probability output after softmax at \(i^{\prime}\)-th position, and each \({\mathbf{p}_{i^{\prime}}}\) is independent with each other. On this basis, we compute the key quantity \(\log{\mathbf{g}_{i}^{n}}\) in Eq. 6 as the direct output of the convolution operator. As shown in Figure 4, we can get \(\log{\mathbf{g}_{i}^{n}}\) by applying convolution on \(\log{\mathbf{P}}\), with \({\mathbf{y}_{i::n}}\) as the kernels:

\[\log{\mathbf{g}_{i}^{n}}=\texttt{Conv}(\log{\mathbf{P}},\texttt{Onehot}({\mathbf{y}_{i:: n}^{*}})), \tag{7}\]

where Onehot\((\cdot)\) maps each token to its corresponding one-hot representation and Conv\((\cdot,\cdot)\) is the convolution operation with the first argument as input and the second as the kernel. We transform \({\mathbf{P}}\) into log domain to turn the probability multiplication into log probability summations, where Conv can be directly applied. As shown in Figure 4, \(\log{\mathbf{P}}\) is of shape \(V\times T\) and Onehot\(({\mathbf{y}_{i::n}^{*}})\) is of shape \(V\times n\), so \(\texttt{Conv}(\log{\mathbf{P}},\texttt{Onehot}({\mathbf{y}_{i::n}^{*}}))\) is an one-dimensional convolution on the sequence axis. Formally, the \(i^{\prime}\)-th convolutional output is:

\[\begin{split}\log g_{i,i^{\prime}}^{n}=&\sum_{j=1} ^{n}\log{\mathbf{p}_{i^{\prime}+j-1}}\cdot\texttt{Onehot}(y_{i+j-1}^{*})\\ =&\sum_{j=1}^{n}\log p(y_{i^{\prime}+j-1}=y_{i+j-1}^ {*}|{\mathbf{y}_{<i^{\prime}+j-1}})\end{split} \tag{8}\]

After obtaining \({\mathbf{g}_{i}^{n}}\) by convolution, the EISL loss in Eq. 6 can be easily calculated. We now discuss the case of autoregressive model, where by definition we have \(g_{i,i^{\prime}}^{n}=\prod_{j=1}^{n}p(y_{i^{\prime}+j-1}=y_{i+j-1}^{*}|{\mathbf{ y}_{<i^{\prime}}},{\mathbf{y}_{i::j-1}^{*}})\). The dependence on both \({\mathbf{y}_{<i^{\prime}}}\) and \({\mathbf{y}_{i::i+j-1}^{*}}\) in each conditional makes exact estimation of \(\log{\mathbf{g}_{i}^{n}}\) very complicated and costly. We thus introduce the approximation where we approximate \(g_{i,i^{\prime}}^{n}\) as \(\widetilde{g}_{i,i^{\prime}}^{n}=\prod_{j=1}^{n}p(y_{i^{\prime}+j-1}=y_{i+j-1} ^{*}|{\mathbf{y}_{<i^{\prime}+j-1}})\). That is, instead of conditioning on \({\mathbf{y}_{i::j-1}^{*}}\), we use the model-generated tokens \({\mathbf{y}_{i::i^{\prime}+j-1}}\) as the condition. This simple approximation enables us to define the probability output

Figure 4: As convolution is a common operation for translation invariance in image, we adopt a convolution to achieve the translation invariance in text. The input is the distribution from the model output in log domain, kernel represents the convolution kernel and \(*\) is the convolution operation. In this \(3\)-gram example, there are 5 kernels, which correspond to the 5 rows on the right.

 

\(P\) as in the non-autoregressive case, by just performing a forward pass of the model (i.e., sampling a token \({\mathbf{y}}^{*}_{i}\) for each position \(i^{\prime}\) and feeding it to the next step to get \({\mathbf{p}}_{i^{\prime}+1}\)). We can then apply the same convolution operator to approximately obtain \(\log{\mathbf{g}}^{n}_{i}\) as in Eq. 7. Besides the great gain of computational efficiency, we note that the approximation is also effective, especially due to the _position selection_ discussed above. Specifically, for each reference \(n\)-gram \({\mathbf{y}}^{*}_{i:i+n}\), the position selection in effect (softly) picks those large-value \(g^{n}_{i,i^{\prime}}\) (while dropping other low-value ones) to evaluate the loss. A large \(g^{n}_{i,i^{\prime}}\) value indicates the candidate \({\mathbf{y}}_{i^{\prime}:i^{\prime}+n}\) is highly likely to match the reference \({\mathbf{y}}^{*}_{i:i+n}\), meaning that using \({\mathbf{y}}_{i^{\prime}:i^{\prime}+n}\) in replacement of \({\mathbf{y}}^{*}_{i:i+n}\) is a reasonable approximation for evaluating the above conditionals. We provide empirical analysis of the approximation in Appendix A.8, where we show the efficient approximate EISL loss values are very close to the exact EISL values.

### Connections with Common Techniques

CE is a special case of EISLA nice property of EISL is that it subsumes the standard CE loss as a special case. To see this, set \(n=T^{*}\) (the target sequence length), and we have:

\[\mathcal{L}^{\text{EISL}}_{T^{*}}=\mathcal{L}^{\text{EISL}}_{T^{*},1}=-\log{ \mathbf{g}}^{T^{*}}_{1}=-\log p({\mathbf{y}}={\mathbf{y}}^{*})=\mathcal{L}^{\text{CE}}.\]

The connection shows the generality of EISL. As a generalization of CE, it enables learning at arbitrary \(n\)-gram granularity.

Connections between BLEU and EISLR Both our method and the popular BLEU Papineni et al. (2002b) metric use \(n\)-grams as the basis in formulation. Here we articulate the connections and difference between the two. Let us first take a review of the BLEU metric. Specifically, BLEU is defined as a weighted geometric mean of \(n\)-gram precisions:

BLEU \[=\text{BP}\cdot\exp\left(\sum_{n=1}^{N}w_{n}\log\text{prec}_{n}\right)\] \[\text{prec}_{n} =\frac{\sum_{s\in\text{gen}_{m}({\mathbf{y}})}\min(C({\mathbf{s}},{\mathbf{ y}}),C({\mathbf{s}},{\mathbf{y}}^{*}))}{\sum_{s\in\text{gen}_{m}({\mathbf{y}})}C({\mathbf{ s}},{\mathbf{y}})},\]

where BP is a brevity penalty depending on the lengths of \({\mathbf{y}}\) and \({\mathbf{y}}^{*}\); \(N\) is the maximum \(n\)-gram order (typically \(N=4\)); \(\{w_{n}\}\) are the weights which usually take \(1/N\); \(\text{prec}_{n}\) is the \(n\)-gram precision, \(\text{gram}_{n}({\mathbf{y}})\) is the set of unique \(n\)-gram subsequences of \({\mathbf{y}}\); and \(C({\mathbf{s}},{\mathbf{y}})\) is the number of times a gram \(s\) occurs in \({\mathbf{y}}\) as defined in Eq. 1. The conventional formulation above enumerates over unique \(n\)-grams in \({\mathbf{y}}\). In contrast, we enumerate over token indexes in calculating the \(n\)-gram matching loss. BLEU considers the \(n\)-gram precisions and has a penalty term while EISL simply maximizes the log probability of \(n\)-gram matchings.

The non-differentiability of BLEU makes it hard to optimize directly, hence most prior attempts resort to reinforcement learning algorithms and use BLEU as the reward Ranzato et al. (2016); Liu et al. (2017). There are also some works trying to

Figure 5: Results of Translation with Noisy Target on German-to-English(de-en) from Multi30k. BLEU scores are computed against clean test data. The \(x\)-axis of all figures denotes the level of noise we injected to target sequences in training. (a) Shuffle: selected tokens are shuffled; (b) Repetition: selected tokens are repeated; (c) Blank: selected tokens are substituted with a special blank token; (d) Synthetical noise: the combination of all three noises (\(x=x_{0}\) stands for the combination of \(5x_{0}\%\) of all kinds of noises); (e) Ablation study of \(n\)-grams for EISL on synthetical noise. BLEURT results are shown in Appendix A.3.

 

introduce differentiable BLEU metric using approximation like Zhukov and Kretov (2017). However, such losses are often too complicated and have not yet demonstrated to perform well in practice.

## 4 Experiments

In this section, we present the experimental results on three text generation settings to test EISL's effectiveness, including learning from noisy text, learning from weak sequence supervision, and non-autoregressive generation models that require flexibility in generation orders. More details of the experimental setting are provided in Appendix A.2.

### Learning from Noisy Text

To test the robustness to noise, we evaluate on the task of machine translation with noisy training target, in which we train the models with noisy sequence targets and evaluate with clean test data.

**Setup** We test EISL loss on Multi30k and WMT18 raw corpus. We use German-to-English (de-en) dataset from Multi30k Elliott et al. (2016), which contains 29k training instances. As inspired by Shen et al. (2019), to simulate various noises in the real data, we introduce four types of noises: shuffle, repetition, blank, and the synthetical noise, i.e., the combination of the aforementioned three types of noise. The noises are only added to the training target sequences. To verify the validity of EISL on real noisy data, we also use German-to-English (de-en) dataset from WMT18 raw corpus, which is a very noisy de-en corpus crawled from the web. We randomly select different number of training samples to test the influence of the data scale. We use a Transformer-based pretrained model BART-base Lewis et al. (2020) and adopt greedy decoding in training and beam search (beam size \(=5\)) in evaluation. We compare EISL loss with CE loss, Policy Gradient (PG), and Loss Truncation (LT). We also conduct ablation experiments to explore the effect of different \(n\)-grams in EISL loss. We use both BLEU Papineni et al. (2002b) and BLEURT, an advanced model-based metric Sellam et al. (2020), as the automatic metrics for evaluation. Due to space limit, we report BLEU results in the main paper, and defer BLEURT results in the appendix, where we can see BLEURT leads to the same conclusion as BLEU.

 

considers whole-sentence matching could struggle on noisy data. In Appendix A.3, we provide more results (e.g., comparison with loss truncation Kang and Hashimoto (2020)) and case studies.

### Learning from Weak Supervisions: Style Transfer

We experiment on transferring two types of text styles Jin et al. (2022), namely sentiment and political slant, to verify EISL can learn from weak sequence supervisions.

**Setup** We use the Yelp review dataset and political dataset. Yelp contains almost 250k negative sentences and 380K positive sentences, of which the ratio of training, valid and test is \(7:1:2\). Li et al. (2018) annotated 1000 sentences as ground truth for better evaluation. The political dataset is comprised of top-level comments on Facebook posts from all 412 members of the United States Senate and House who have public Facebook pages Voigt et al. (2018). The data set contains 270K democratic sentences and 270K republican sentences. And there exists no ground truth for evaluation. The data preprocessing follows Tian et al. (2018). The structured content preserving model Tian et al. (2018) is adopted as the base model.

Following previous work, we compute automatic evaluation metrics: accuracy, BLEU score, perplexity (PPL) and POS distance. We also perform human evaluations on Yelp data to further test the transfer quality.

**Results** As sentiment results are shown in Table 1, the BLEU gets improved from 65.71 to 68.51 with EISL loss. On the premise of the correctness of sentiment transfer, EISL loss plays a critical role to guarantee lexical preservation. In the meanwhile, all of BLEU(human), PPL, and POS distance get improved. It is not surprising that EISL loss helps generate sentences more fluently and select the more appropriate words conditions on the content information. As the human evaluation results are shown in Table 1, the model with EISL loss performs better, in accord with the automatic metrics. After analyzing the generated samples, we found EISL loss could drive the model to adopt the words which fit the scene better and could understand more semantics but not just replace some keywords. See some examples in the Appendix A.4.1.

We report the results of political data in Appendix A.4.2. Our method outperforms all models on BLEU, PPL, and POS distance with comparable accuracy. For a more fair comparison with the base model, our EISL loss improves the base model on all four metrics, including the accuracy.

The results demonstrate the effectiveness of EISL for weak supervision task, improving both transfer accuracy fluency and content preservation.

### Learning Non-Autoregressive Generation

Non-autoregressive neural machine translation NAT Gu et al. (2018) is proposed to predict tokens simultaneously in a single decoding step, which aims at reducing the inference latency. The non-autoregressive nature makes it extremely hard for models to keep the order of words in the sentences, hence CE often struggles with NAT problems. In experiments, we show EISL is superior to CE in NAT which requires modeling flexible generation order of the text.

**Setup** We use English-to-German dataset from WMT14 Luong et al. (2015), which contains 4.5M training instances. We apply our proposed EISL loss on both fully NAT models Gu et al. (2018); Sun et al. (2019) and iterative NAT models Lee et al. (2018); Gu et al. (2019); Ghazvininejad et al. (2019), showing its general applicability and superiority, and we also compare with a wide range of recent methods Shao et al. (2020); Wang et al. (2019); Li et al. (2019); Ghazvininejad et al. (2020). We evaluate with both BLEU and BLEU metrics.

**Results** We first summarize the comparison of BLEU between EISL loss and CE loss in Table 2 (comparison of BLEURT is in Appendix A.5.2). The proposed EISL improves the model performance on both the KD and original datasets. More specifically, for fully NAT models VanillaNAT and NAT-CRF), EISL gives strong improvement. For iterative NAT models (iNAT, LevT, and CMLM), EISL also significantly outperforms the baselines when the iteration step is restricted to a small level as suggested by Kasai et al. (2020). (We show in Appendix A.5.1 that, with increasing iteration steps, the difference fades away. However, as studied in Kasai et al. (2020), iterative NAT models with many iteration steps do not hold the intrinsic advantage of speed since Transformer baselines with a shallow decoder can achieve comparable speedup and only at the sacrifice of minor performance drop.) Table 3 provides more comparison of with recent strong baselines. Specifically, we apply our EISL on the CMLM base model Ghazvininejad et al. (2019) which shows strong superiority. We 

provide qualitative analysis in Appendix A.5.3.

## 5 Conclusions

We have developed Edit-Invariant Sequence Loss (EISL) for end-to-end training of neural text generation models. The proposed method is insensitive to the shift of \(n\)-grams in target sequences, hence suitable for training with noisy data and weak supervisions, where CE loss fails easily. We show CE loss is a special case of EISL and build the connection of EISL with BLEU metric and convolution operation, which both have the invariant property. Experiments on translation with noisy target, text style transfer, and non-autoregressive neural machine translation demonstrate the superiority of our method. The more general applications and superiority of EISL on other diverse text generation problems as well as fundamental challenges, such as compositional generalization Andreas et al. (2019) and causal invariance Hu and Li (2021) in language, remain to be explored further, which we are excited to study in the future.

## References

* J. Andreas, M. Baroni, A. Conneau, D. Kiela, H. Schwenk, L. Barrault, A. Bordes, J. Devlin, A. Fyshe, E. Webbe, et al. (2019)Measuring compositionality in representation learning. In International Conference on Learning Representations, Vol. 375, pp. 2227-2237. Cited by: SS4.2.
* D. Bahdanau, K. Cho, and Y. Bengio (2015)Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, External Links: Link Cited by: SS4.2.
* N. Casas, J. A. R. Fonollosa, and M. R. Costajussa (2018)A differentiable BLEU loss. Note: analysis and first results External Links: 1611.0177 Cited by: SS4.2.
* N. Dai, J. Liang, X. Qiu, and X. Huang (2019)Style transformer: unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 5997-6007. External Links: Document, Link Cited by: SS4.2.
* Desmond Elliott, S. Frank, K. Sima'an, and L. Specia (2016)Multi30K: multilingual english-German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, Berlin, Germany, pp. 70-74. External Links: ISBN 978-3-31 

Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 3515-3523. PMLR.
* Ghazvininejad et al. (2019) Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6112-6121, Hong Kong, China. Association for Computational Linguistics.
* May 3, 2018, Conference Track Proceedings_. OpenReview.net.
* Gu et al. (2019) Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 11179-11189.
* Guo et al. (2021) Han Guo, Bowen Tan, Zhengzhong Liu, Eric P Xing, and Zhiting Hu. 2021. Text generation with efficient (soft) Q-learning. _arXiv preprint arXiv:2106.07704_.
* He et al. (2020)Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A probabilistic formulation of unsupervised text style transfer. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Hu and Li (2021) Zhiting Hu and Li Erran Li. 2021. A causal lens for controllable text generation. _Advances in Neural Information Processing Systems_, 34.
* Hu et al. (2019) Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. 2019. Learning data manipulation for augmentation and weighting. _Advances in Neural Information Processing Systems_, 32.
* Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_, volume 70 of _Proceedings of Machine Learning Research_, pages 1587-1596. PMLR.
* Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* Jin et al. (2022) Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep learning for text style transfer: A survey. _Computational Linguistics_, 48(1):155-205.
* Kang and Hashimoto (2020) Daniel Kang and Tatsunori B. Hashimoto. 2020. Improved natural language generation via loss truncation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 718-731, Online. Association for Computational Linguistics.
* Kasai et al. (2020) Jungo Kasai, Nikolaos Pappas, Hao Peng, J. Cross, and Noah A. Smith. 2020. Deep encoder, shallow decoder: Reevaluating the speed-quality tradeoff in machine translation. _ArXiv preprint_, abs/2006.10369.
* Lee et al. (2018) Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1173-1182, Brussels, Belgium. Association for Computational Linguistics.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1192-1202, Austin, Texas. Association for Computational Linguistics.
* Li et al. (2018) Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
* Li et al. (2019) Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Hint-based training for non-autoregressive machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5708-5713, Hong Kong, China. Association for Computational Linguistics.
* Lin et al. (2020) Shuai Lin, Wentao Wang, Zichao Yang, Xiaodan Liang, Frank F Xu, Eric Xing, and Zhiting Hu. 2020. Data-to-text generation with style imitation. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1589-1598.
* Liu et al. (2017) Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. 2017. Improved image captioning via policy gradient optimization of spider. In 

_IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_, pages 873-881. IEEE Computer Society.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
* Mikolov et al. (2010) Tomas Mikolov, Martin Karafat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010_, pages 1045-1048. ISCA.
* Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. _Nat._, 518(7540):529-533.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016_, pages 280-290. ACL.
* Nicolai and Silfverberg (2020) Garrett Nicolai and Miikka Silfverberg. 2020. Noise isn't always negative: Countering exposure bias in sequence-to-sequence inflection models. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 2837-2846, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* O'Neill and Bollegala (2019) James O'Neill and Danushka Bollegala. 2019. Transfer reward learning for policy gradient-based text generation. _ArXiv preprint_, abs/1909.03622.
* Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
* Papineni et al. (2002a) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002a. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_, pages 311-318.
* Papineni et al. (2002b) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002b. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Pinnis (2018) Marcis Pinnis. 2018. Tilde's parallel corpus filtering methods for WMT 2018. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_, pages 939-945, Belgium, Brussels. Association for Computational Linguistics.
* Prabhumoye et al. (2018) Shirmail Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Ranzato et al. (2016) Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
* Rennie et al. (2017) Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In _2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017_, pages 1179-1195. IEEE Computer Society.
* See et al. (2017)bigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Shao et al. (2018) Chenze Shao, Xilin Chen, and Yang Feng. 2018. Greedy search with probabilistic n-gram matching for neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4778-4784, Brussels, Belgium. Association for Computational Linguistics.
* Shao et al. (2021) Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, and Jie Zhou. 2021. Sequence-level training 

for non-autoregressive neural machine translation. _Comput. Linguistics_, 47(4):891-925.
* Shao et al. (2020) Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. 2020. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In _Proceedings of the AAAI conference on artificial intelligence_, pages 198-205.
* Shen et al. (2016) Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1683-1692, Berlin, Germany. Association for Computational Linguistics.
* Shen et al. (2017) Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 6830-6841.
* Shen et al. (2019) Tianxiao Shen, Jonas Mueller, Regina Barzilay, and Tommi S. Jaakkola. 2019. Latent space secrets of denoising text-autoencoders. _ArXiv preprint_, abs/1905.12777.
* Smith and Eisner (2006) David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In _Proceedings of the COLING/ACL 2006 Main Conference Poster Seessions_, pages 787-794, Sydney, Australia. Association for Computational Linguistics.
* Sun et al. (2019) Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and ZhiHong Deng. 2019. Fast structured decoding for sequence models. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 3011-3020.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_, pages 3104-3112.
* Tan et al. (2020) Bowen Tan, Lianhui Qin, Eric Xing, and Zhiting Hu. 2020. Summarizing text on any aspects: A knowledge-informed weakly-supervised approach. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6301-6309.
* Tian et al. (2018) Youzhi Tian, Zhiting Hu, and Zhou Yu. 2018. Structured content preservation for unsupervised text style transfer. _ArXiv preprint_, abs/1810.06526.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. _Advances in neural information processing systems_, 30.
* Voigt et al. (2018) Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender: A corpus for studying differential responses to gender. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Wang et al. (2019a) Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019a. Non-autoregressive machine translation with auxiliary regularization. In _Proceedings of the AAAI conference on artificial intelligence_, volume 33, pages 5377-5384.
* November 2, 2019_, pages 322-330. IEEE.
* Wang et al. (2021) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak supervision. _arXiv preprint arXiv:2108.10904_.
* Wieting et al. (2019) John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEU:training neural machine translation with semantic similarity. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4344-4355, Florence, Italy. Association for Computational Linguistics.
* Wu et al. (2018) Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018. A study of reinforcement learning for neural machine translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3612-3621, Brussels, Belgium. Association for Computational Linguistics.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikum, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _ArXiv preprint_, abs/1609.08144.
* Xu et al. (2019) Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. 2019. L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_, pages 6222-6233.
* Zhang and Sabuncu (2018) Zhilu Zhang and Mert Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. _Advances in neural information processing systems_, 31.
* Zhukov and Kretov (2017) Vlad Zhukov and Maksim Kretov. 2017. Differentiable lower bound for expected BLEU score. _ArXiv preprint_, abs/1712.04708 

Appendix

### Additional Derivation

For a given \(i^{\prime}\),

\[p(\mathbf{y}_{i^{\prime}:i^{\prime}+n}=\mathbf{y}_{i:i+n}^{*})\\ =\sum_{\mathbf{y}}p(\mathbf{y}_{<i^{\prime}})p(\mathbf{y}_{i^{\prime}:i^{ \prime}+n}=\mathbf{y}_{i:i+n}^{*}|\mathbf{y}_{<i^{\prime}}),\]

then we derive the detail of Eq. 3 in Eq. 9, where the first inequality holds since \(T-n+1\geq 0\); and the second inequality holds by Jensen's inequality.

### Detailed Experimental Setup

#### a.2.1 Learning from Noisy Text

We use a Transformer-based pretrained model BART-base Lewis et al. (2020), containing 6 layers in the encoder and decoder. We train the model using the Adam optimizer with learning rate \(3\times 10^{-5}\) with polynomial decay and the maximum number of tokens is 6000 in one step. The models are trained on one Tesla V100 DGXS with 32GB memory. We start with CE training using teacher forcing for fast initialization. We then switch to combined \(1\)- and \(2\)-gram EISL with weight \(0.8:0.2\), which we select using the validation set. We adopt greedy decoding in training and beam search (beam size \(=5\)) in evaluation. We use fairseq4Ott et al. (2019) to conduct the experiments. We compare EISL loss with CE loss and Policy Gradient (PG), where PG is used to finetune the best CE model. Teacher forcing is employed in CE training.

Footnote 4: Fairseq(-py) is MIT-licensed.

#### a.2.2 Learning from Weak Supervisions: Style Transfer

We use the Adam optimizer with learning rate \(5\times 10^{-4}\), the batch size is \(128\) and the model is trained on one Tesla V100 DGXS 32GB. We compare the results between the base model and the model with EISL. Specifically, on top of the base model, we add the EISL loss (a combination of \(2\), \(3\) and \(4\)-gram with the same weights \(1/3\)) to reduce the discrepancy between the transferred sentence generated by language model and the original sentence. We assign EISL loss with weight \(0.5\).

Following previous work, we compute automatic evaluation metrics: accuracy, BLEU score, perplexity (PPL) and POS distance. For accuracy, we adopt a CNN-based classifier, trained on the same training data, to evaluate whether the generated sentence possesses the target style. Then we measure BLEU score and BLEU(human) score of transferred sentences against the original sentences and ground truth, respectively. PPL metric is evaluated by GPT-2 Radford et al. (2019) base model after finetuning on the corresponding dataset, with the goal to assess the fluency of the generated sentence. POS distance is used to measure the model's semantics preserving ability Tian et al. (2018).

We also perform human evaluations on Yelp data to further test the transfer quality. We first randomly select 100 sentences from the test set, use these sentences as input and generate sentences from the base model Tian et al. (2018) and our model. Then for each original sentence, we present the outputs of the base model and ours in random order. The three annotators are asked to evaluate which sentence is preferred as the transferred sentence of the original sentence, in terms of content preservation and sentiment transfer. They can choose either output or select the same quality. We measure the percentage of times each model outperforms the other.

#### a.2.3 Learning Non-Autoregressive Generation

We use the Adam optimizer with learning rate \(5\times 10^{-4}\) with inverse square root scheduler. We apply sequence-level knowledge distillation to the dataset, which can reduce the complexity of the dataset, making it easier for the model to learn and improving the performance. The models are first trained by CE loss for fast initialization, then focus on \(2\)-gram, \(3\)-gram, and \(4\)-gram with the same weights. Faireseq Ott et al. (2019) is adopted to conduct the experiments. We average the last 5 checkpoints as the final model.

### Additional Results of Learning from Noisy Text

#### a.3.1 Results of BLEU Metric

In this section, we evaluate the results of CE, PG and EISL on BLEURT Sellam et al. (2020) metric. We use the recommended BLEURT-20 checkpoint. It gives a score for every sentence pair, and we averaged the scores to get the final score. The results are shown in Figure 7. Both BLEU metric and BLEURT metric show the superiority of our proposed EISL loss.

#### a.3.2 Comparison with Loss Truncation

The Loss Truncation LT Kang and Hashimoto (2020)), method adaptively removes high log loss 

examples as a way to optimize for distinguishability. In this section, We'd like to show the comparisons with Loss Truncation. We evaluated two variants of LT: (1) LT_Pre which first trains the model with CE loss and then adds LT for further training, and (2) LT which directly trains the model with CE loss and LT together. Hyperparameters were selected on the validation set. For simplicity, we remove the PG curves (Figure 5), and the comparison results with LT are shown in Figure 8.

We can see Loss Truncation can sometimes slightly improve over CE, especially when the data is clean or with low/moderate noise. However, by simply ignoring high-loss data, LT is not good at handling data with high noise (which often leads to high loss). In comparison, our proposed EISL achieves a substantial improvement in the presence of high noise.

#### a.3.3 Reasons of Better Performance with Lower-gram EISL

In this section, we discuss the reason of why the performance of using lower grams is better than higher-gram EISL in Figure 5(e).

Lower-gram EISL is less sensitive to noise. For example, 1-gram EISL focuses mostly on matching individual tokens without caring much about the order of tokens; while a high-gram EISL (e.g., consider the extreme case of \(T^{*}\)-gram where \(T^{*}\) is the target length) reduces to CE (as discussed in Sec 3.3) and is highly sensitive to noise. Thus, in the presence of high data noise, lower-gram EISL would be more robust and perform better.

Besides, on low-noise data (e.g., noise-level = 0 or 1), lower-gram EISL performs comparably with higher-gram EISL, both close to the CE performance. This is because we pretrained the model with CE (as mentioned in the experimental setup), and finetuning with EISL (either with lower- or higher-grams) would not change the performance a lot given the low-noise data.

#### a.3.4 Cases Study

As shown in Table 8, 9, 10, 11 and 12, we randomly sample some examples from generated sentences of the models trained with different types of noise on Multi30k dataset. For the sake of convenience, we use abbreviations in the tables, i.e., SC, RR, BR and NL are short for Shuffle Count, Repetition Ratio, Blank Ratio and Noise Level (for Synthetical Noise), respectively.

Shuffle NoiseWhen there exist a few shuffle noises, e.g., \(\text{SC}=3\), CE loss may lead word redpplicated (Example 1 and Example 2) and slightly wrong word order (Example 4 and Example 5), and there are some information mistranslated (_beautiful_ in Example 4) or extra irrelevant information added (_black_ in Example 5). As shuffle count increases, the aforementioned problems are increasingly severe, resulting the generated sentences meaningless. Especially, there are some words untranslated in PG examples (_einezezunten_ in Example 1, _rigendwo_ in Example 2, _haben_ in Example 5, ). But EISL loss could keep the content consistency and grammatical correctness as far as possible 

Repetition NoiseThe main problem of the models trained by CE and PG with repetition noises is that the models can't filter the repetition noise out in training samples, and try to learn the wrong distribution, leading to generate reduplicated words frequently (Example 1-5). Specifically, the examples of CE and PG in RR\(=50\%\) are very representative. However, it's amazing that EISL can almost avoid such a problem even the repetition ratio achieves \(50\%\). Meanwhile, the main semantics is preserved and the grammar is correct.

Blank NoiseWhen adding blank noise, some tokens in targets will be substituted as _unk_ so the targets will lose some information. We could measure from two aspects: one is the term frequency of meaningless token _unk_ in generated sentences, and the other is the meaningful contents preserved by the models. Obviously, EISL loss handles better than CE loss on both aspects. Especially, when BR \(=20\%\), unlike models with CE, models with PG and EISL barely generate the _unk_ token, and could translate the core content (Example 1-5). As BR increases, EISL could preserve more key information and produce less _unk_ than CE and PG. Moreover, PG performs rather poor when BR is high (like BR \(=45\%\)), and it almost loses all information (Example 1-5) and generates some confusing words (_teil_ in Example 1, _afroamerikanischer_ and _irgendwo_ in Example 3, _beachaufsichgebude_ in Example 4, and _holzstuck_ in Example 5).

Synthetical NoiseWe then evaluate the results of models trained by synthetical noise. Such a

Figure 8: Comparison results with Loss Truncation(LT) of Translation with Noisy Target on German-to-English(de-en) from Multi30k. BLEU scores are computed against clean test data.

Figure 7: Results of Translation with Noisy Target on German-to-English(de-en) from Multi30k. BLEURT scores are computed against clean test data.

 

situation combines aforementioned three types of noises. One most highlighted advantage of EISL is that the generated sentences are almost grammatically correct and include main content as far as possible. However, CE can only stiffly joint some words, and can't guarantee the grammatical correctness (word order, word repetition and so on). PG performs worst, involving all the problems in CE cases and the meaningless word generation problem (Example 1-5).

### Additional Results of Text Style Transfer

#### a.4.1 Examples on Yelp dataset

Some examples of generated sentences are given in Table 4. The model with EISL can select more appropriate adjective and improve the quality of the sentences. In the first example, the model should transfer the negative adjectives _cold_ and _watery_ to some positive adjectives that describe food. Obviously, the _delicious_ is more appropriate than _excellent_. In the second example, the base model reverses both _not_ and _stop_, leading to wrong sentiment and inconsistent content. While the model with EISL could avoid such a situation and generate more suitable sentence.

#### a.4.2 Results on Political dataset

Since the instances from democratic data and republican data are quite different, names of politicians have high correlation with the political slant. Therefore the BLEU score and POS distance have a big gap with the sentiment results. The results are shown in Table 5.

### Additional Results of Non-Autoregressive Generation

#### a.5.1 Results of Iterative NAT Models

As shown in Figure 9, with the increasing of iteration steps, the difference fades away.

#### a.5.2 Results of BLEURT Metric

To show the superiority of our method, We also evaluate on recent text generation metric, BLEURT [Sellam et al., 2020]. BLEURT is an evaluation metric for Natural Language Generation. It takes a pair of sentences as input, a reference and a candidate, and it returns a score that indicates to what extent the candidate is fluent and conveys the meraning of the reference. We use the recommended BLEURT-20 checkpoint. It gives a score for every sentence pair, and we averaged the scores to get the final score. The results are shown in Table 6.

#### a.5.3 Qualitative Analysis on NAT Experiments

Given the non-autoregressive nature (i.e., all tokens are generated simultaneously), the one-to-one matching of CE loss can lead to severe mismatching. We consider the example: the predicted sentence is a cat is onthered blanket and the target sentence is a cat is sitting on the red blanket. The "on the red blanket" part of the prediction will be corrected to match the target positions, and this may lead to overcorrection (e.g., "on the red red blanket ."). Repetition is often a sign of overcorrection. However, with EISL, this situation will not happen because the phrase will be matched to appropriate

\begin{table}
\begin{tabular}{l c c c} \hline \hline Source & my “ hot ” sub was _cold_ and the meat was _watery_ . \\ Base Model & my “ hot ” sub was _excellent_ and the meat was _excellent_ . \\ _with_ EISL & my “ hot ” sub was _deficious_ and the meat was _delicious_ . \\ \hline Source & the man did _not stop_ her . \\ Base Model & the man did _definitely right_ her . \\ _with_ EISL & the man did _definitely stop_ her . \\ \hline \hline \end{tabular}
\end{table}
Table 4: Examples of the generated sentences.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Model** & **Accuracy(\%)** & **BLEU** & **PPL** & **POS distance** \\ \hline Prabhumoye et al. (2018) & 86.5 & 7.38 & - & 7.298 \\ Hu et al. (2017) & **90.7** & 47.50 & - & 3.524 \\ \hline Tian et al. (2018) & 88.0 & 59.63 & 28.46 & 2.348 \\ _with_ EISL & 89.2 & **60.26** & **27.85** & **2.191** \\ \hline \hline \end{tabular}
\end{table}
Table 5: The results on the political dataset. The first two results are reported by [Tian et al., 2018].

 

target tokens. Let's have a look at a real example in Figure 10.

Take the non-autoregressive model CMLM Ghazvininejad et al. (2019) for example, we evaluate the translation of CMLM models trained by CE and EISL. As shown in Figure 11, our proposed EISL can reduce repetition to a large extent.

### Efficiency Analysis

Complexity analysisGiven \(T^{*}\) tokens, the time complexity of CE loss is \(\mathbf{O}(T^{*})\), while the complexity of \(n\)-gram EISL loss is \(\mathbf{O}(n(T^{*}-n+1)^{2})\approx\mathbf{O}(T^{*2})\), assuming small \(n\) is used in practice (e.g., \(n\in\left\{1,2,3,4\right\}\)). However, in practice, the computation cost of the loss (either CE or EISL) is **negligible** compared to the cost of model forward and backward during training. Thus, the extra cost introduced by the EISL loss is rather minor.

Empirical comparison of time costTo quantify the computational cost of different methods, we adopt CE and EISL on top of the same model and setting, and evaluate the consumed time for 1 training epoch. For comparison on both small and large dataset, we evaluate on Multi30k (29k training data, 1k test data) and 1M scale WMT-18 raw corpus (1M training data, 3k test data). The models are tested on one Tesla V100 DGXS with 32 GB memory, the batch size is 128, max number of tokens is 6000 and update frequency is 4. For each method, we test 6 times and average the results as final time. The results are shown in Figure 12.

Empirical total time cost of EISL trainingAs discussed in the experiments in the paper, we first pretrain the model with the CE loss until convergence, and then finetune with the EISL loss. Here we report the total time cost of each stage, based on the WMT-18 translation setting as described in Section 4.1. The results are shown in Table 7. As

Figure 11: The percentage of repeated tokens under different iteration steps.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline  & \multicolumn{2}{c}{**WMT14 en-de KD**} & \multicolumn{2}{c}{**WMT14 en-de**} \\ \cline{2-5} Model & CE & EISL & CE & EISL \\ \hline Vanilla-NAT Gu et al. (2018) & 0.346 & **0.416** & 0.194 & **0.277** \\ NAT-CRF Sun et al. (2019) & 0.441 & **0.464** & - & - \\ iNAT Lee et al. (2018) & 0.332 & **0.437** & - & - \\ LeVT Gu et al. (2019) & 0.355 & **0.458** & 0.214 & **0.333** \\ CMLM Ghazvininejad et al. (2019) & 0.345 & **0.450** & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 6: The results (test set BLEURT) of EISL loss and CE loss applied to non-autoregressive models.

Figure 10: Examples of the generated sentences.

 

the data size increases, the convergence time of both pretraining and finetuning grows. The time cost of the finetuning stage is less than half of that of the pretraining stage.

### Hyperparameters

Regarding which \(n\)-grams to use and their weights \(w_{n}\) in the EISL loss, we found in our experiments that the default values _largely_ following the standard BLEU metric (i.e., maximum \(n=4\) with equal weights) work well. Specifically, we use \(n\in\{2,3,4\}\) and equal weights \(w_{n}=1/3\) as our default values. Most of our experiments adopt the default values which achieve consistent substantial improvement over CE and other rich baselines as shown in our experiments. (except for the synthetic experiment where we show the effect of different \(n\)-grams including those selected using the validation set).

Besides, in our experiments, we first pretrain the model with the CE loss (i.e., EISL with \(n=T^{*}\) and teacher forcing, see Section 3.3) and then finetune with the EISL loss. We simply do the CE pretraining _until convergence_ before switching to the EISL finetuning. Therefore, there is no need of tuning for the training iterations of pretraining.

### Analysis of Efficient Implementation

In order to validate the efficiency and accuracy of our approximation (for autoregressive models) discussed in Section 3.2, we conduct the analysis experiments, showing that the approximate (and efficient) EISL loss values are very close to exact (but expensive) EISL value. We use the same setting as section 4.1, and finetune the model with our efficient approximate EISL loss on Multi30k. Throughout the course of training, we record the loss values of both the exact implementation and our approximate implementation. As shown in Figure 13(a) and (b), the tendency of two losses is very close to each other. We also plot the absolute difference of the two losses as shown in Figure 13(c). We can see the difference decreases as training proceeds. The observations validate the effectiveness of our approximate implementation.

We note that training the model with the exact loss is costly, which necessitates our approximation. Specifically, for \(n\)-gram loss, we need to run the forward pass of the decoder \((T-n)^{2}\) times, and keep the whole computation graph for backpropagation, which will consume much more time and memory. Even for only loss evaluation (without the backward pass), we found the runtime of the exact loss is about 15 times longer than that of the efficient approximate implementation based on convolution operator.

Figure 12: Results of training and inference time. EISL-\(n\) represents \(n\)-gram EISL loss and EISL-\(12\) represents the combination of 1-gram and 2-gram EISL loss.

 



\begin{table}
\begin{tabular}{c c c} \hline \hline Data Size & PreTraining Time (CE) & Finetuning Time (EISL) \\ \hline
1M & 1h 40min 57s & 49min 33s \\
2M & 5h 56min 57s & 1h 35min 10s \\
4M & 8h 55min 18s & 3h 57min 44s \\ \hline \hline \end{tabular}
\end{table}
Table 7: Convergence time of pretraining and finetuning stages.

Figure 13: The change of loss values during training. The x-axis represents the training step. a) gives the loss curve of exact implementation; b) gives the loss curve of efficient approximate implementation as we discussed in section 3.2; and c) gives the absolute difference between the two implementations.





\begin{table}
\begin{tabular}{c l p{284.5pt}} \hline \hline Source (de) & \multicolumn{1}{l}{ein junger man nimmt an eiem lauf teil und derjenige , der dies aufzeichnet ,} \\  & & lichelt . \\ \hline \multicolumn{3}{c}{Target (en)} & **a young man participates in a career while the subject who records it smiles .** \\ \hline \multirow{3}{*}{SC = \(3\)} & CE & young man is running on a a and the other man is smiling . \\  & PG & young man is running on a track and the other man is smiling . \\  & EISL & young man is running in a dirt course and the other is smiling . \\ \hline \multirow{3}{*}{SC = \(6\)} & CE & young man is running a a race and the other is smiling . \\  & PG & young man taking a race and the other smiling . a \\  & EISL & young man is running a race and the other guy is smiling . \\ \hline \multirow{3}{*}{SC = \(9\)} & CE & young man . a at the is running up and up hill smiling taking \\  & PG & young man takes on a slope and thejenige , the the smiles . a \\  & EISL & young man is on a hillside smiling and the others , who is smiling . \\ \hline \multirow{3}{*}{RR = \(15\%\)} & CE & young man is running on a track and the other is smiling . \\  & PG & young man is running on a track and the other is smiling . \\  & EISL & young man is running in a race and the runner is smiling . \\ \hline \multirow{3}{*}{RR = \(30\%\)} & CE & young man man is is running on a track track and the the other is is smiling \\  & PG & young man man is is running on a track track and the other man man who is is is smiling . \\  & EISL & young man is running in a race and the other is smiling at him . \\ \hline \multirow{3}{*}{RR = \(50\%\)} & CE & a young young man man is is smiling smiling at at a a window window while another smiles smiles at him him . \\  & PG & a young man man is is snapping napping on on a a grassy grassy field field and and some people people are are smiling smiling . \\  & EISL & young man running in a race and the other is smiling at the action . \\ \hline \multirow{3}{*}{BR = \(20\%\)} & CE & young man unk u a run and the unk is smiling . \\  & PG & young man is running in a race and the one who is looking at him is smiling . \\  & EISL & young man is running in a race with the runner who is up . \\ \hline \multirow{3}{*}{BR = \(35\%\)} & CE & young man unk u a unk , and the unk is smiling unk \\  & PG & young man unk urck turk outhers unk . \\  & EISL & young man unk is un in a race and the other un is un at the finish . \\ \hline \multirow{3}{*}{BR = \(45\%\)} & CE & young unk is unk on a unk and the unk smiles unk \\  & PG & young man unk a unk teil unk unk . \\  & EISL & young unk un is un in a race , the other is smiling back . \\ \hline \multirow{3}{*}{NL = \(5\)} & CE & young man is running a race and the one who is running is smiling . \\  & PG & young man is running a race and the one scoring is smiling . \\  & EISL & young man is running a race and one of the runners is up to him . \\ \hline \multirow{3}{*}{NL = \(15\)} & CE & young man is unk u nk and the other man is smiling . \\  & PG & young man is on a unk smiling at théjenige . \\  & EISL & young man is in a race , the other smiling . \\ \hline \multirow{3}{*}{NL = \(20\)} & CE & a young man is unk u nk and unk is smiling at him . \\  & PG & young smiles on in ai al and the jerangle smile on . . \\ \cline{1-1}  & EISL & young man unk u lacker and unk , who is unk smiling . \\ \hline \end{tabular}
\end{table}
Table 8: Example 1.





\begin{table}
\begin{tabular}{c l l} \hline \multicolumn{2}{l}{Source (de)} & 15 große hunde spielen auf einem eingezäunten grundstück neben einemhaus . \\ \hline \multicolumn{2}{l}{Target (en)} & **15 large dogs playing in a fenced yard beside a house .** \\ \hline \multirow{3}{*}{SC = \(3\)} & CE & large dogs play on a a dirt path next to a house . \\  & PG & 15 large dogs play on an carthent platform next to a house . \\  & EISL & large dogs are playing on a dirt path next to a house . \\ \hline \multirow{3}{*}{SC = \(6\)} & CE & large dogs play on a a play area next to abandoned house . \\  & PG & 15 large dogs playing on a eingezäunten group stage next to a house . \\  & EISL & group of dogs play on a abandoned path next to a house . \\ \hline \multirow{3}{*}{SC = \(9\)} & CE & large dogs play a . on a field next to abandoned house \\  & PG & dogs play on a snowy grundstück next to a house .15 large \\  & EISL & . 15 large dogs play on an abandoned hillside next to a house . \\ \hline \multirow{3}{*}{RR = \(15\%\)} & CE & large dogs are playing on a fenced in area next to a house . \\  & PG & large dogs are playing on a fenced in area next to a house . \\  & EISL & large dogs are playing on a fenced track next to a house . \\ \hline \multirow{3}{*}{RR = \(30\%\)} & CE & large dogs dogs play on on a a dirt track near a house house . \\  & PG & large dogs dogs play on a fenced-in area area next to a house . \\  & EISL & large dogs play on a fenced walkway next to a house . \\ \hline \multirow{3}{*}{RR = \(50\%\)} & CE & small dogs dogs play on on a a grassy grassy field field next text to to a house house . \\  & CG & huge . \\  & GISL & 15 large dogs dogs are are playing playing on on a a grassy grassy field field \\  & GE & new next to to a house house . \\  & EISL & 15 large dogs playing on a fenced terrain next to a house . \\ \hline \multirow{3}{*}{BR = \(20\%\)} & CE & large dogs play in a fenced yard next to a house . \\  & PG & large dogs are playing on an overcast walk next to a house . \\  & EISL & large dogs are playing in a fenced area near to a house . \\ \hline \multirow{3}{*}{BR = \(35\%\)} & CE & unk dogs play unk a unk unk by a house . \\  & PG & large dogs unk a unk path unk house . \\  & EISL & large dogs unk play in a fenced area next to a house . \\ \hline \multirow{3}{*}{BR = \(45\%\)} & CE & unk dogs unk on a unk unk next to unk house . \\  & PG & large dogs unk aunk unk . \\  & EISL & large unk un are un in a fenced-out game next to a house . \\ \hline \multirow{3}{*}{NL = \(5\)} & CE & large dogs are playing on a fenced in area next to a house . \\  & PG & large dogs are playing on a fenced in area next to a house . \\  & EISL & large dogs are playing on a fenced backwalk next to a house . \\ \hline \multirow{3}{*}{NL = \(15\)} & CE & large dogs are playing on a unk grassy field next to a house . \\  & PG & large dogs playing on a unk next to a house . \\  & EISL & large dogs play on a covered piece of furniture next to a house . \\ \hline \multirow{3}{*}{NL = \(20\)} & CE & large dogs are playing on a a a grassy grassy field next to a house . \\  & PG & large play play in autneck in a house . \\  & EISL & large dogs play on a unk unk next to a house . \\ \hline \end{tabular}
\end{table}
Table 9: Example 2.





\begin{table}
\begin{tabular}{c l p{284.5pt}} \hline \multicolumn{1}{c}{Source (de)} & ein afroamerikanischer mann spielt irgendwo in der statd gitarre und singt \\ \hline Target (en) & **an african american man playing guitar and singing in an urban setting .** \\ \hline SC = 3 & CE & african american man is playing the guitar and singing in the city . arican american man is playing the guitar in the city and singing . arican american man is playing the guitar in the city and singing . \\ \hline SC = 6 & CE & african-american man is playing guitar in the a and singing city . arican american man playing irgendwo in the city guitar singing \\  & EISL & african american man is playing the guitar in the city \\ \hline SC = 9 & CE & african-american man playing guitar in the a and singing city \\  & PG & african americanscher man plays irgendwo in the city guitar singing . a African american man is playing the guitar in the city and singing . aAfrican american man is playing the guitar in the city and singing . \\ \hline RR = \(15\%\) & CE & african american american man plays guitar guitar in the city city . arican american man is playing guitar in the city and singing . arican american man is playing guitar in the city and singing . \\ \hline RR = \(30\%\) & CE & african american man plays guitar in in the city city while singing . arican american man man plays guitar in the city city and sings . an african american man playing guitar in the city and singing . \\ \hline RR = \(50\%\) & CE & african african american american man playing guitar guitar in in the the city city and singing singing . arican american american man man is is playing playing guitar guitar in in the the city city . arican american man playing guitar in the city and singing . \\ \hline BR = \(20\%\) & CE & african american man plays guitar unk sings unk \\  & PG & african american man is playing guitar and singing in the city . arican american man is playing the guitar and singing . \\ \hline BR = \(35\%\) & CE & african american man unk unk suit guitar unk singing unk \\  & EISL & african american man unk guitarist unk singing unk \\  & EISL & african american unk is un a guitar and singing in the city . \\ \hline BR = \(45\%\) & CE & african american unk unk playing unk guitar in unk city unk \\  & PG & afroamerikanischer man unk irgendwo unk unk \\  & EISL & af unk un playing some sort of guitar in the city and singing . \\ \hline NL = \(5\) & CE & african american man plays guitar and songs somewhere in the city . arican american man is playing guitar and singing in the city . arican american man is playing guitar and singing in the city . arican american man is playing guitar and singing somewhere in the city . \\ \hline NL = \(15\) & PE & african american man is playing the guitar in the city and singing . arosmerikanischer man nis rigendow in the city guitarre . arican american man playing some sort of guitar in the city and singing . \\ \hline NL = \(20\) & CE & african american american man is playing the guitar in the the city unk \\  & PG & afroamerikanischer sngt in the city guitarre singt . arosmerikanischer man plays unk unk in the city unk \\ \hline \end{tabular}
\end{table}
Table 10: Example 3.





\begin{table}
\begin{tabular}{c l l} \hline \hline Source (de) & ein strandaufsichtgebaude steht im sand , es ist ein bewölkter tag . \\ \hline Target (en) & **a lifeguard building is on the sand on a cloudy day .** \\ \hline SC = 3 & CE & beach a is standing in the sand on a beautiful day . \\  & PG & beachfront building is standing in the sand on a beautiful day . \\  & EISL & beach view building is standing in the sand on a cloudy day . \\ \hline SC = 6 & CE & beach a is in the sand building on a beautiful day . \\  & PG & beach viewgeb building standing in sand on a beautiful day . \\  & EISL & beach view building is standing in the sand on a beautiful day . \\ \hline SC = 9 & CE & beach a in the sand . a cloudy day stands beach \\  & PG & beachtafischtge building stands in sand , the is a beautiful day . a \\  & EISL & a beachfront building standing in the sand is a beautiful day . \\ \hline RR = 15\% & CE & beachfront building is standing in the sand on a cloudy day . \\  & PG & beachfront building is standing in the sand , it is a cloudy day . \\ \hline RR = 30\% & CE & beachfront beachfront building building is is standing standing in the sand \\  & PG & sandwichent beachfront building building is standing in sand and on a cloudy day . \\  & EISL & beachfront building is standing in the sand , it is a cloudy day . \\ \hline RR = 50\% & CE & beachfront building is standing in sand on a cloudy day . \\  & PG & beachfront building building is is standing standing in in the sand , it is a cloudy day . \\  & EISL & beach view building is standing in the sand , it is a cloudy day . \\ \hline BR = 20\% & CE & beachfront building is standing in sand on a cloudy day unk \\  & PG & beachfront building is standing in sand on a cloudy day . \\  & EISL & beach view building is standing in the sand , it is a cloudy day . \\ \hline NR = 35\% & CE & beach unk tank standing in sand on a cloudy day unk \\  & PG & beach unloading unk unk sand unk a cloudy day . \\  & EISL & beach building unk is un in the sand on a cloudy day . \\ \hline BR = 45\% & CE & unk unk is standing unk the sand unk it is a beautiful day unk \\  & PG & beachtafischtgebaude unk unk sand unk . \\  & EISL & beach unk us in un the sand , this is a cloudy day . \\ \hline NL = 5 & CE & beachfront view building is standing in the sand on a cloudy day . \\  & PG & beachfront view building is standing in sand on a cloudy day . \\  & EISL & beachfront building is standing in the sand , it is a cloudy day . \\ \hline NL = 15 & CE & beach unk unk is standing in the sand unk it is a sunny day . \\  & PG & beachtafischtgebäude tehst in sand , es is a day . \\  & EISL & beach taknds in sand unk it is a sunny day . \\ \hline NL = 20 & CE & beach



\begin{table}
\begin{tabular}{c l l} \hline \hline Source (de) & zwei hunde haben beim spielen dasselbe holzstück im maul . \\ \hline Target (en) & **two dog is playing with a same chump on their mouth .** \\ \hline \multirow{3}{*}{\(\mathrm{SC}=3\)} & CE & dogs are two playing with . pieces of wood in their mouths two \\  & PG & dogs are playing with pieces of black wood in their mouths . \\  & EISL & two dogs are playing with pieces of wood in their mouths . \\ \hline \multirow{2}{*}{\(\mathrm{SC}=6\)} & CE & dogs are two . playing with sticks in their mouths two \\  & PG & dogs have been playing with pieces of wood in their mouths . two \\  & EISL & two dogs are playing with pieces of wood in their mouths . \\ \hline \multirow{3}{*}{\(\mathrm{SC}=9\)} & CE & two dogs their . are playing with sticks in muzzled \\  & PG & dogs haben beim play pieces in their mouth . two \\  & EISL & . two dogs have been playing with sticks in their mouth . \\ \hline \multirow{3}{*}{\(\mathrm{RR}=15\%\)} & CE & two dogs are are playing with a a piece piece of wood in their mouth . \\  & PG & dogs are playing with white wooden blocks in their mouth . \\  & EISL & two dogs are playing with some pieces of wood in their mouths . \\ \hline \multirow{3}{*}{\(\mathrm{RR}=30\%\)} & CE & two dogs dogs are are playing with a a piece piece of of wood in their mouths . \\  & PG & dogs dogs are are playing with white wooden blocks blocks in their mouth . \\  & EISL & two dogs are playing with pieces of wood in their mouths . \\ \hline \multirow{3}{*}{\(\mathrm{RR}=50\%\)} & CE & two dogs dogs are are playing playing with with plastic plastic sticks sticks in in their their mouth mouth . \\  & PG & two dogs dogs are are playing playing with with plastic holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders' holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders’ holders holders’ holders’ holders holders’ holders holders’ holders holders’ holders hold