SlowGuess/NAACL_2022_OCR huggingface dataset:
$P$  as in the non-autoregressive case, by just performing a forward pass of the model (i.e., sampling a token  $\pmb{y}_i^{\prime}$  for each position  $i^{\prime}$  and feeding it to the next step to get  $\pmb{p}_{i^{\prime} + 1}$ ). We can then apply the same convolution operator to approximately obtain  $\log g_i^n$  as in Eq. 7. Besides the great gain of computational efficiency, we note that the approximation is also effective, especially due to the position selection discussed above. Specifically, for each reference  $n$ -gram  $\pmb{y}_{i:i + n}^{*}$ , the position selection in effect (softly) picks those large-value  $g_{i,i'}^n$  (while dropping other low-value ones) to evaluate the loss. A large  $g_{i,i'}^n$  value indicates the candidate  $\pmb{y}_{i':i' + n}$  is highly likely to match the reference  $\pmb{y}_{i:i + n}^{*}$ , meaning that using  $\pmb{y}_{i':i' + n}$  in replacement of  $\pmb{y}_{i:i + n}^{*}$  is a reasonable approximation for evaluating the above conditionals. We provide empirical analysis of the approximation in Appendix A.8, where we show the efficient approximate EISL loss values are very close to the exact EISL values.
3.3 Connections with Common Techniques
CE is a special case of EISL A nice property of EISL is that it subsumes the standard CE loss as a special case. To see this, set  $n = T^{*}$  (the target sequence length), and we have:
$$
\mathcal {L} _ {T ^ {*}} ^ {\mathrm {E I S L}} = \mathcal {L} _ {T ^ {*}, 1} ^ {\mathrm {E I S L}} = - \log \boldsymbol {g} _ {1} ^ {T ^ {*}} = - \log p (\boldsymbol {y} = \boldsymbol {y} ^ {*}) = \mathcal {L} ^ {\mathrm {C E}}.
$$
The connection shows the generality of EISL. As a generalization of CE, it enables learning at arbitrary  $n$ -gram granularity.
Connections between BLEU and EISL Both our method and the popular BLEU (Papineni et al., 2002b) metric use  $n$ -grams as the basis in formulation. Here we articulate the connections and difference between the two. Let us first take a review of the BLEU metric. Specifically, BLEU is defined as a weighted geometric mean of  $n$ -gram precisions:
$$
\mathrm {B L E U} = \mathrm {B P} \cdot \exp \left(\sum_ {n = 1} ^ {N} w _ {n} \log \operatorname {p r e c} _ {n}\right)
$$
$$
\operatorname {p r e c} _ {n} = \frac {\sum_ {s \in \operatorname {g r a m} _ {n} (\boldsymbol {y})} \min  (C (\boldsymbol {s} , \boldsymbol {y}) , C (\boldsymbol {s} , \boldsymbol {y} ^ {*}))}{\sum_ {s \in \operatorname {g r a m} _ {n} (\boldsymbol {y})} C (\boldsymbol {s} , \boldsymbol {y})},
$$
where BP is a brevity penalty depending on the lengths of  $\pmb{y}$  and  $\pmb{y}^*$ ;  $N$  is the maximum  $n$ -gram order (typically  $N = 4$ );  $\{w_n\}$  are the weights which usually take  $1/N$ ;  $\operatorname{prec}_n$  is the  $n$ -gram precision,  $\operatorname{gram}_n(\pmb{y})$  is the set of unique  $n$ -gram subsequences of  $\pmb{y}$ ; and  $C(\pmb{s}, \pmb{y})$  is the number of times a gram  $s$  occurs in  $\pmb{y}$  as defined in Eq. 1. The conventional formulation above enumerates over unique  $n$ -grams in  $\pmb{y}$ . In contrast, we enumerate over token indexes in calculating the  $n$ -gram matching loss. BLEU considers the  $n$ -gram precisions and has a penalty term while EISL simply maximizes the log probability of  $n$ -gram matchings.
The non-differentiability of BLEU makes it hard to optimize directly, hence most prior attempts resort to reinforcement learning algorithms and use BLEU as the reward (Ranzato et al., 2016, Liu et al., 2017). There are also some works trying to
introduce differentiable BLEU metric using approximation like (Zhukov and Kretov, 2017). However, such losses are often too complicated and have not yet demonstrated to perform well in practice.
In this section, we present the experimental results on three text generation settings to test EISL's effectiveness, including learning from noisy text, learning from weak sequence supervision, and non-autoregressive generation models that require flexibility in generation orders. More details of the experimental setting are provided in Appendix A.2.
4.1 Learning from Noisy Text
To test the robustness to noise, we evaluate on the task of machine translation with noisy training target, in which we train the models with noisy sequence targets and evaluate with clean test data.
Setup We test EISL loss on Multi30k and WMT18 raw corpus. We use German-to-English (de-en) dataset from Multi30k (Elliott et al., 2016), which contains 29k training instances. As inspired by Shen et al. (2019), to simulate various noises in the real data, we introduce four types of noises: shuffle, repetition, blank, and the synthetical noise, i.e., the combination of the aforementioned three types of noise. The noises are only added to the training target sequences. To verify the validity of EISL on real noisy data, we also use German-to-English (de-en) dataset from WMT18 raw corpus, which is a very noisy de-en corpus crawled from the web. We randomly select different number of training samples to test the influence of the data scale. We use a Transformer-based pretrained model BART-base (Lewis et al., 2020) and adopt greedy decoding in training and beam search (beam size = 5) in evaluation. We compare EISL loss with CE loss, Policy Gradient (PG), and Loss Truncation (LT). We also conduct ablation experiments to explore the effect of different  $n$ -grams in EISL loss. We use both BLEU (Papineni et al., 2002b) and BLEURT, an advanced model-based metric (Sellam et al., 2020), as the automatic metrics for evaluation. Due to space limit, we report BLEU results in the main paper, and defer BLEURT results in the appendix, where we can see BLEURT leads to the same conclusion as BLEU.
Results The results on noisy Multi30k are presented in Figure 5. The proposed EISL loss provides significantly better performance than CE loss
Figure 6: Results of German-to-English(de-en) Translation on WMT18 raw corpus. BLEU scores are computed against clean parallel test data. On x-axis, 0k denotes the performance of the pretrained model. BLEURT results are similar as shown in Appendix A.3.
Model	Acc (%)	BLEU	BLEU (Human)	PPL	POS Distance
Hu et al. (2017)	86.7	58.4	-	177.7	-
Shen et al. (2017)	73.9	20.7	7.8	72.0	-
He et al. (2020)	87.9	48.4	18.7	31.7	-
Dai et al. (2019)	87.7	54.9	20.3	73.0	-
Tian et al. (2018) with EISL (Ours)	88.8	65.71	22.56	42.07	0.352
88.8	68.51	23.17	41.56	0.275
Tian et al. (2018) (%)	with EISL (Ours) (%)	equal (%)
22.0	30.7	47.3
Table 1: Top: automatic evaluations on the Yelp review dataset. The BLEU (human) is calculated using the 1000 human annotated sentences as ground truth from Li et al. (2018). The first four results are from the original papers. Bottom: human evaluation statistics of base model vs. with EISL. The results denotes the percentages of inputs for which the model has better transferred sentences than other model.
and PG on all the noise types, especially on the high-level noise end. For synthetical noise as shown in Figure 5(d), it's interesting to see that CE and PG completely fail when the noise level is beyond 6, but model trained with EISL has high BLEU score, demonstrating EISL can select useful information to learn despite high noise. This validates that the proposed EISL is much less sensitive to the noise than the traditional CE loss and policy gradient training method. The results of different  $n$ -gram are shown in Figure 5(e). As the noise increases, the importance of lower grams, e.g., 1-gram, is more obvious. The results on real noisy data, WMT18 raw data, are shown in Figure 6. EISL loss achieves better performance than CE loss and PG, and the difference is getting larger when the training data scale increases. This again demonstrates EISL could learn more valid information in rather noisy data, while CE loss which only
considers whole-sentence matching could struggle on noisy data. In Appendix A.3, we provide more results (e.g., comparison with loss truncation (Kang and Hashimoto, 2020)) and case studies.
4.2 Learning from Weak Supervisions: Style Transfer
We experiment on transferring two types of text styles (Jin et al., 2022), namely sentiment and political slant, to verify EISL can learn from weak sequence supervisions.
Setup We use the Yelp review dataset and political dataset. Yelp contains almost 250k negative sentences and 380K positive sentences, of which the ratio of training, valid and test is  $7:1:2$ . Li et al. (2018) annotated 1000 sentences as ground truth for better evaluation. The political dataset is comprised of top-level comments on Facebook posts from all 412 members of the United States Senate and House who have public Facebook pages (Voigt et al., 2018). The data set contains 270K democratic sentences and 270K republican sentences. And there exists no ground truth for evaluation. The data preprocessing follows Tian et al. (2018). The structured content preserving model (Tian et al., 2018) is adopted as the base model.
Following previous work, we compute automatic evaluation metrics: accuracy, BLEU score, perplexity (PPL) and POS distance. We also perform human evaluations on Yelp data to further test the transfer quality.
Results As sentiment results are shown in Table 1, the BLEU gets improved from 65.71 to 68.51 with EISL loss. On the premise of the correctness of sentiment transfer, EISL loss plays a critical role to guarantee lexical preservation. In the meanwhile, all of BLEU(human), PPL, and POS distance get improved. It is not surprising that EISL loss helps generate sentences more fluently and select the more appropriate words conditions on the content information. As the human evaluation results are shown in Table 1, the model with EISL loss performs better, in accord with the automatic metrics. After analyzing the generated samples, we found EISL loss could drive the model to adopt the words which fit the scene better and could understand more semantics but not just replace some keywords. See some examples in the Appendix A.4.1.
We report the results of political data in Appendix A.4.2. Our method outperforms all models on BLEU, PPL, and POS distance with comparable
accuracy. For a more fair comparison with the base model, our EISL loss improves the base model on all four metrics, including the accuracy.
The results demonstrate the effectiveness of EISL for weak supervision task, improving both transfer accuracy fluency and content preservation.
4.3 Learning Non-Autoregressive Generation
Non-autoregressive neural machine translation (NAT, (Gu et al., 2018)) is proposed to predict tokens simultaneously in a single decoding step, which aims at reducing the inference latency. The non-autoregressive nature makes it extremely hard for models to keep the order of words in the sentences, hence CE often struggles with NAT problems. In experiments, we show EISL is superior to CE in NAT which requires modeling flexible generation order of the text.
Setup We use English-to-German dataset from WMT14 (Luong et al., 2015), which contains 4.5M training instances. We apply our proposed EISL loss on both fully NAT models (Gu et al., 2018, Sun et al., 2019) and iterative NAT models (Lee et al., 2018, Gu et al., 2019, Ghazvininejad et al., 2019), showing its general applicability and superiority, and we also compare with a wide range of recent methods (Shao et al., 2020, Wang et al., 2019a, Li et al., 2019, Ghazvininejad et al., 2020). We evaluate with both BLEU and BLEURT metrics.
Results We first summarize the comparison of BLEU between EISL loss and CE loss in Table 2 (comparison of BLEURT is in Appendix A.5.2). The proposed EISL improves the model performance on both the KD and original datasets. More specifically, for fully NAT models (Vanilla-NAT and NAT-CRF), EISL gives strong improvement. For iterative NAT models (iNAT, LevT, and CMLM), EISL also significantly outperforms the baselines when the iteration step is restricted to a small level as suggested by Kasai et al. (2020). (We show in Appendix A.5.1 that, with increasing iteration steps, the difference fades away. However, as studied in Kasai et al. (2020), iterative NAT models with many iteration steps do not hold the intrinsic advantage of speed since Transformer baselines with a shallow decoder can achieve comparable speedup and only at the sacrifice of minor performance drop.) Table 3 provides more comparison of with recent strong baselines. Specifically, we apply our EISL on the CMLM base model (Ghazvininejad et al., 2019) which shows strong superiority. We
Decoding method	Model	WMT14 en-de KD	WMT14 en-de
CE	EISL	CE	EISL
Autoregressive	Transformer base (Vaswani et al., 2017)	27.48
Non-Autoregressive	Vanilla-NAT (Gu et al., 2018)	17.9	22.2	9.12	15.46
NAT-CRF (Sun et al., 2019)	21.88	22.43	-	-
iNAT (Lee et al., 2018)	16.67	22.59	-	-
LevT (Gu et al., 2019)	17.84	23.61	9.91	18.47
CMLM (Ghazvininejad et al., 2019)	17.12	23.05	-	-
Table 2: The test-set BLEU of EISL loss and CE loss applied to non-autoregressive models. "KD" refers to the standard "knowledge distillation" setting in NAT (Gu et al., 2018). iNAT, LevT and CMLM are iterative non-autoregressive models, that could run in multiple decoding iterations. However, the first decoding iteration of these models is fully non-autoregressive, which is what we use as our baselines.
Fully Non-Autoregressive model	WMT14 en-de KD
CMLM with CE (Ghazvininejad et al., 2019)	17.12
Auxiliary Regularization (Wang et al., 2019a)	20.65
Bag-of-ngrams Loss (Shao et al., 2020)	20.90
Hint-based Training (Li et al., 2019)	21.11
CMLM with AXE (Ghazvininejad et al., 2020)	23.53
CMLM with EISL (Ours)	24.17
Table 3: The test-set BLEU of CMLM trained with our EISL, compared to other recent fully non-autoregressive methods. The baseline results are from (Ghazvininejad et al., 2020), where CMLM-with-AXE generates 5 candidates and ranks with loss. Our method follows the same generation configuration as CMLM-with-AXE.
provide qualitative analysis in Appendix A.5.3.
We have developed Edit-Invariant Sequence Loss (EISL) for end-to-end training of neural text generation models. The proposed method is insensitive to the shift of  $n$ -grams in target sequences, hence suitable for training with noisy data and weak supervisions, where CE loss fails easily. We show CE loss is a special case of EISL and build the connection of EISL with BLEU metric and convolution operation, which both have the invariant property. Experiments on translation with noisy target, text style transfer, and non-autoregressive neural machine translation demonstrate the superiority of our method. The more general applications and superiority of EISL on other diverse text generation problems as well as fundamental challenges, such as compositional generalization (Andreas et al., 2019) and causal invariance (Hu and Li, 2021) in language, remain to be explored further, which we are excited to study in the future.
Jacob Andreas, Marco Baroni, Alexis Conneau, Douwe Kiela, Holger Schwenk, Ioic Barrault, Antoine Bordes, Jacob Devlin, Alona Fyshe, Leila Wehbe, et al. 2019. Measuring compositionality in representation learning. In International Conference on Learning
Representations, volume 375, pages 2227-2237. Association for Computational Linguistics.
Noe Casas, José A. R. Fonollosa, and Marta R. Costajussa. 2018. A differentiable BLEU loss: analysis and first results. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Workshop Track Proceedings. OpenReview.net.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5997-6007, Florence, Italy. Association for Computational Linguistics.
Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia. 2016. Multi30K: Multilingual English-German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70-74, Berlin, Germany. Association for Computational Linguistics.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Aligned cross entropy for non-autoregressive machine translation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020,
Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3515-3523. PMLR.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6121, Hong Kong, China. Association for Computational Linguistics.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Jiatao Gu, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11179-11189.
Han Guo, Bowen Tan, Zhengzhong Liu, Eric P Xing, and Zhiting Hu. 2021. Text generation with efficient (soft) Q-learning. arXiv preprint arXiv:2106.07704.
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A probabilistic formulation of unsupervised text style transfer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Zhiting Hu and Li Erran Li. 2021. A causal lens for controllable text generation. Advances in Neural Information Processing Systems, 34.
Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. 2019. Learning data manipulation for augmentation and weighting. Advances in Neural Information Processing Systems, 32.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep learning for text style transfer: A survey. Computational Linguistics, 48(1):155-205.
Daniel Kang and Tatsunori B. Hashimoto. 2020. Improved natural language generation via loss truncation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 718-731, Online. Association for Computational Linguistics.
Jungo Kasai, Nikolaos Pappas, Hao Peng, J. Cross, and Noah A. Smith. 2020. Deep encoder, shallow decoder: Reevaluating the speed-quality tradeoff in machine translation. ArXiv preprint, abs/2006.10369.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173-1182, Brussels, Belgium. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Austin, Texas. Association for Computational Linguistics.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Hint-based training for non-autoregressive machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5708-5713, Hong Kong, China. Association for Computational Linguistics.
Shuai Lin, Wentao Wang, Zichao Yang, Xiaodan Liang, Frank F Xu, Eric Xing, and Zhiting Hu. 2020. Data-to-text generation with style imitation. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1589-1598.
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. 2017. Improved image captioning via policy gradient optimization of spider. In
IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 873-881. IEEE Computer Society.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Tomás Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, September 26-30, 2010, pages 1045-1048. ISCA.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nat., 518(7540):529-533.
Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Caglar Gulgêhre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 280-290. ACL.
Garrett Nicolai and Miikka Silfverberg. 2020. Noise isn't always negative: Countering exposure bias in sequence-to-sequence inflection models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2837-2846, Barcelona, Spain (Online). International Committee on Computational Linguistics.
James O'Neill and Danushka Bollegala. 2019. Transfer reward learning for policy gradient-based text generation. ArXiv preprint, abs/1909.03622.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics* (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002a. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.