SlowGuess/NAACL_2022_OCR huggingface dataset:
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002b. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Marcis Pinnis. 2018. Tilde's parallel corpus filtering methods for WMT 2018. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 939-945, Belgium, Brussels. Association for Computational Linguistics.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1179-1195. IEEE Computer Society.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Chenze Shao, Xilin Chen, and Yang Feng. 2018. Greedy search with probabilistic n-gram matching for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4778-4784, Brussels, Belgium. Association for Computational Linguistics.
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, and Jie Zhou. 2021. Sequence-level training
for non-autoregressive neural machine translation. Comput. Linguistics, 47(4):891-925.
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. 2020. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In Proceedings of the AAAI conference on artificial intelligence, pages 198-205.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683-1692, Berlin, Germany. Association for Computational Linguistics.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6830-6841.
Tianxiao Shen, Jonas Mueller, Regina Barzilay, and Tommi S. Jaakkola. 2019. Latent space secrets of denoising text-autoencoders. ArXiv preprint, abs/1905.12777.
David A. Smith and Jason Eisner. 2006. Minimum risk annealing for training log-linear models. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 787-794, Sydney, Australia. Association for Computational Linguistics.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and ZhiHong Deng. 2019. Fast structured decoding for sequence models. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3011-3020.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
Bowen Tan, Lianhui Qin, Eric Xing, and Zhiting Hu. 2020. Summarizing text on any aspects: A knowledge-informed weakly-supervised approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6301-6309.
Youzhi Tian, Zhiting Hu, and Zhou Yu. 2018. Structured content preservation for unsupervised text style transfer. ArXiv preprint, abs/1810.06526.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender: A corpus for studying differential responses to gender. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019a. Non-autoregressive machine translation with auxiliary regularization. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5377-5384.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. 2019b. Symmetric cross entropy for robust learning with noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 322-330. IEEE.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904.
John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEU:training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344-4355, Florence, Italy. Association for Computational Linguistics.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and TieYan Liu. 2018. A study of reinforcement learning for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3612-3621, Brussels, Belgium. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv preprint, abs/1609.08144.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. 2019. L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 6222-6233.
Zhilu Zhang and Mert Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31.
Vlad Zhukov and Maksim Kretov. 2017. Differentiable lower bound for expected BLEU score. ArXiv preprint, abs/1712.04708.
A.1 Additional Derivation
For a given  $i'$ ,
$$
\begin{array}{l} p \left(\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*}\right) \\ = \sum_ {\boldsymbol {y}} p (\boldsymbol {y} _ {<   i ^ {\prime}}) p (\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*} | \boldsymbol {y} _ {<   i ^ {\prime}}), \\ \end{array}
$$
then we derive the detail of Eq. 3 in Eq. 9, where the first inequality holds since  $T - n + 1 \geq 0$ ; and the second inequality holds by Jensen's inequality.
A.2 Detailed Experimental Setup
A.2.1 Learning from Noisy Text
We use a Transformer-based pretrained model BART-base (Lewis et al., 2020), containing 6 layers in the encoder and decoder. We train the model using the Adam optimizer with learning rate  $3 \times 10^{-5}$  with polynomial decay and the maximum number of tokens is 6000 in one step. The models are trained on one Tesla V100 DGXS with 32GB memory. We start with CE training using teacher forcing for fast initialization. We then switch to combined 1- and 2-gram EISL with weight  $0.8:0.2$ , which we select using the validation set. We adopt greedy decoding in training and beam search (beam size  $= 5$ ) in evaluation. We use fairseq² (Ott et al., 2019) to conduct the experiments. We compare EISL loss with CE loss and Policy Gradient (PG), where PG is used to finetune the best CE model. Teacher forcing is employed in CE training.
A.2.2 Learning from Weak Supervisions: Style Transfer
We use the Adam optimizer with learning rate  $5 \times 10^{-4}$ , the batch size is 128 and the model is trained on one Tesla V100 DGXS 32GB. We compare the results between the base model and the model with EISL. Specifically, on top of the base model, we add the EISL loss (a combination of 2, 3 and 4-gram with the same weights  $1/3$ ) to reduce the discrepancy between the transferred sentence generated by language model and the original sentence. We assign EISL loss with weight 0.5.
Following previous work, we compute automatic evaluation metrics: accuracy, BLEU score, perplexity (PPL) and POS distance. For accuracy, we adopt a CNN-based classifier, trained on the same training data, to evaluate whether the generated sentence possesses the target style. Then we measure BLEU
score and BLEU(human) score of transferred sentences against the original sentences and ground truth, respectively. PPL metric is evaluated by GPT-2 (Radford et al., 2019) base model after finetuning on the corresponding dataset, with the goal to assess the fluency of the generated sentence. POS distance is used to measure the model's semantics preserving ability (Tian et al., 2018).
We also perform human evaluations on Yelp data to further test the transfer quality. We first randomly select 100 sentences from the test set, use these sentences as input and generate sentences from the base model (Tian et al., 2018) and our model. Then for each original sentence, we present the outputs of the base model and ours in random order. The three annotators are asked to evaluate which sentence is preferred as the transferred sentence of the original sentence, in terms of content preservation and sentiment transfer. They can choose either output or select the same quality. We measure the percentage of times each model outperforms the other.
A.2.3 Learning Non-Autoregressive Generation
We use the Adam optimizer with learning rate  $5 \times 10^{-4}$  with inverse square root scheduler. We apply sequence-level knowledge distillation to the dataset, which can reduce the complexity of the dataset, making it easier for the model to learn and improving the performance. The models are first trained by CE loss for fast initialization, then focus on 2-gram, 3-gram, and 4-gram with the same weights. Fairseq (Ott et al., 2019) is adopted to conduct the experiments. We average the last 5 checkpoints as the final model.
A.3 Additional Results of Learning from Noisy Text
A.3.1 Results of BLEURT Metric
In this section, we evaluate the results of CE, PG and EISL on BLEURT (Sellam et al., 2020) metric. We use the recommended BLEURT-20 checkpoint. It gives a score for every sentence pair, and we averaged the scores to get the final score. The results are shown in Figure 7. Both BLEU metric and BLEURT metric show the superiority of our proposed EISL loss.
A.3.2 Comparison with Loss Truncation
The Loss Truncation (LT (Kang and Hashimoto, 2020)), method adaptively removes high log loss
$^2$ Fairseq(-py) is MIT-licensed.
$$
\begin{array}{l} l _ {n, i} ^ {\operatorname {E I S L}} (\boldsymbol {\theta}) = - \log \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} p \left(\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*}\right), \tag {9} \\ = - \log \frac {1}{T - n + 1} \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} \sum_ {\boldsymbol {y}} p (\boldsymbol {y} _ {<   i ^ {\prime}}) p (\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*} | \boldsymbol {y} _ {<   i ^ {\prime}}) - \log (T - n + 1), \\ \leq - \log \frac {1}{T - n + 1} \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} \sum_ {\boldsymbol {y}} p (\boldsymbol {y} _ {<   i ^ {\prime}}) p (\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*} | \boldsymbol {y} _ {<   i ^ {\prime}}), \\ \leq - \frac {1}{T - n + 1} \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} \sum_ {\boldsymbol {y}} p (\boldsymbol {y} _ {<   i ^ {\prime}}) \log p (\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*} | \boldsymbol {y} _ {<   i ^ {\prime}}), \\ = - \frac {1}{T - n + 1} \mathbb {E} _ {\boldsymbol {y} \sim p (\boldsymbol {y})} \sum_ {i ^ {\prime} = 1} ^ {T - n + 1} \log p \left(\boldsymbol {y} _ {i ^ {\prime}: i ^ {\prime} + n} = \boldsymbol {y} _ {i: i + n} ^ {*} \mid \boldsymbol {y} _ {<   i ^ {\prime}}\right), \\ = \mathcal {L} _ {n, i} ^ {\text {E I S L}} (\boldsymbol {\theta}), \\ \end{array}
$$
examples as a way to optimize for distinguishability. In this section, We'd like to show the comparisons with Loss Truncation. We evaluated two variants of LT: (1) LT_Pre which first trains the model with CE loss and then adds LT for further training, and (2) LT which directly trains the model with CE loss and LT together. Hyperparameters were selected on the validation set. For simplicity, we remove the PG curves (Figure 5), and the comparison results with LT are shown in Figure 8.
We can see Loss Truncation can sometimes slightly improve over CE, especially when the data is clean or with low/moderate noise. However, by simply ignoring high-loss data, LT is not good at handling data with high noise (which often leads to high loss). In comparison, our proposed EISL achieves a substantial improvement in the presence of high noise.
A.3.3 Reasons of Better Performance with Lower-gram EISL
In this section, we discuss the reason of why the performance of using lower grams is better than higher-gram EISL in Figure 5(e).
Lower-gram EISL is less sensitive to noise. For example, 1-gram EISL focuses mostly on matching individual tokens without caring much about the order of tokens; while a high-gram EISL (e.g., consider the extreme case of  $T^{*}$ -gram where  $T^{*}$  is the target length) reduces to CE (as discussed in Sec 3.3) and is highly sensitive to noise. Thus, in the presence of high data noise, lower-gram EISL would be more robust and perform better.
Besides, on low-noise data (e.g., noise-level  $= 0$  or 1), lower-gram EISL performs comparably with higher-gram EISL, both close to the CE performance. This is because we pretrained the model with CE (as mentioned in the experimental setup), and finetuning with EISL (either with lower- or higher-grams) would not change the performance a lot given the low-noise data.
A.3.4 Cases Study
As shown in Table 8, 9, 10, 11 and 12, we randomly sample some examples from generated sentences of the models trained with different types of noise on Multi30k dataset. For the sake of convenience, we use abbreviations in the tables, i.e., SC, RR, BR and NL are short for Shuffle Count, Repetition Ratio, Blank Ratio and Noise Level (for Synthetical Noise), respectively.
Shuffle Noise When there exist a few shuffle noises, e.g.,  $\mathrm{SC} = 3$ , CE loss may lead word reduplicated (Example 1 and Example 2) and slightly wrong word order (Example 4 and Example 5), and there are some information mistranslated (beautiful in Example 4) or extra irrelevant information added (black in Example 5). As shuffle count increases, the aforementioned problems are increasingly severe, resulting the generated sentences meaningless. Especially, there are some words untranslated in PG examples (eingezäunten in Example 1, irgendwo in Example 2, haben in Example 5). But EISL loss could keep the content consistency and grammatical correctness as far as possible.
(a) Shuffle Count
(b) Repetition (%)
(c) Blank (%)
(d) Synthetical Noise
Figure 7: Results of Translation with Noisy Target on German-to-English(de-en) from Multi30k. BLEURT scores are computed against clean test data.
Figure 8: Comparison results with Loss Truncation(LT) of Translation with Noisy Target on German-to-English(deen) from Multi30k. BLEU scores are computed against clean test data.
Repetition Noise The main problem of the models trained by CE and PG with repetition noises is that the models can't filter the repetition noise out in training samples, and try to learn the wrong distribution, leading to generate reduplicated words frequently (Example 1-5). Specifically, the examples of CE and PG in  $\mathrm{RR} = 50\%$  are very representative. However, it's amazing that EISL can almost avoid such a problem even the repetition ratio achieves  $50\%$ . Meanwhile, the main semantics is preserved and the grammar is correct.
Blank Noise When adding blank noise, some tokens in targets will be substituted as unk so the targets will lose some information. We could measure from two aspects: one is the term frequency of meaningless token unk in generated sentences,
and the other is the meaningful contents preserved by the models. Obviously, EISL loss handles better than CE loss on both aspects. Especially, when BR  $= 20\%$ , unlike models with CE, models with PG and EISL barely generate the unk token, and could translate the core content (Example 1-5). As BR increases, EISL could preserve more key information and produce less unk than CE and PG. Moreover, PG performs rather poor when BR is high (like BR  $= 45\%$ ), and it almost loses all information (Example 1-5) and generates some confusing words (teil in Example 1, afroamerikanischer and irgendwo in Example 3, beachaufsichtgebäude in Example 4, and holzstück in Example 5).
Synthetical Noise We then evaluate the results of models trained by synthetical noise. Such a
Source	my “ hot ” sub was cold and the meat was watery .
Base Model	my “ hot ” sub was excellent and the meat was excellent .
with EISL	my “ hot ” sub was delicious and the meat was delicious .
Source	the man did not stop her .
Base Model	the man did definitely right her .
with EISL	the man did definitely stop her .
Table 4: Examples of the generated sentences.
Model	Accuracy(%)	BLEU	PPL	POS distance
Prabhumoye et al. (2018)	86.5	7.38	-	7.298
Hu et al. (2017)	90.7	47.50	-	3.524
Tian et al. (2018)	88.0	59.63	28.46	2.348
with EISL	89.2	60.26	27.85	2.191
Table 5: The results on the political dataset. The first two results are reported by (Tian et al., 2018).
situation combines aforementioned three types of noises. One most highlighted advantage of EISL is that the generated sentences are almost grammatically correct and include main content as far as possible. However, CE can only stiffly joint some words, and can't guarantee the grammatical correctness (word order, word repetition and so on). PG performs worst, involving all the problems in CE cases and the meaningless word generation problem (Example 1-5).
A.4 Additional Results of Text Style Transfer
A.4.1 Examples on Yelp dataset
Some examples of generated sentences are given in Table 4. The model with EISL can select more appropriate adjective and improve the quality of the sentences. In the first example, the model should transfer the negative adjectives cold and watery to some positive adjectives that describe food. Obviously, the delicious is more appropriate than excellent. In the second example, the base model reverses both not and stop, leading to wrong sentiment and inconsistent content. While the model with EISL could avoid such a situation and generate more suitable sentence.
A.4.2 Results on Political dataset
Since the instances from democratic data and republican data are quite different, names of politicians have high correlation with the political slant. Therefore the BLEU score and POS distance have a big gap with the sentiment results. The results are shown in Table 5.
A.5 Additional Results of Non-Autoregressive Generation
A.5.1 Results of Iterative NAT Models
As shown in Figure 9, with the increasing of iteration steps, the difference fades away.
A.5.2 Results of BLEURT Metric
To show the superiority of our method, We also evaluate on recent text generation metric, BLEURT (Sellam et al., 2020). BLEURT is an evaluation metric for Natural Language Generation. It takes a pair of sentences as input, a reference and a candidate, and it returns a score that indicates to what extent the candidate is fluent and conveys the mearning of the reference. We use the recommended BLEURT-20 checkpoint. It gives a score for every sentence pair, and we averaged the scores to get the final score. The results are shown in Table 6.
A.5.3 Qualitative Analysis on NAT Experiments
Given the non-autoregressive nature (i.e., all tokens are generated simultaneously), the one-to-one matching of CE loss can lead to severe mismatching. We consider the example: the predicted sentence is a cat is on the red blanket and the target sentence is a cat is sitting on the red blanket. The "on the red blanket" part of the prediction will be corrected to match the target positions, and this may lead to overcorrection (e.g., "on the red red blanket ."). Repetition is often a sign of overcorrection. However, with EISL, this situation will not happen because the phrase will be matched to appropriate
Figure 9: Results of iterative NAT on different decoding iterations.
Model	WMT14 en-de KD	WMT14 en-de
CE	EISL	CE	EISL
Vanilla-NAT (Gu et al., 2018)	0.346	0.416	0.194	0.277
NAT-CRF (Sun et al., 2019)	0.441	0.464	-	-
iNAT (Lee et al., 2018)	0.332	0.437	-	-
LevT (Gu et al., 2019)	0.355	0.458	0.214	0.333
CMLM (Ghazvininejad et al., 2019)	0.345	0.450	-	-
target tokens. Let's have a look at a real example in Figure 10.
Table 6: The results (test set BLEURT) of EISL loss and CE loss applied to non-autoregressive models.
Source	Anja Schlichter managed the tournament
Target	Anja Schlichter leitet das Turnier
CE	Anja Schlichter leitdas Turnier Turnier
EISL	Anja Schlichter leitete das Turnier geleitet
Figure 10: Examples of the generated sentences.
Take the non-autoregressive model CMLM (Ghazvininejad et al., 2019) for example, we evaluate the translation of CMLM models trained by CE and EISL. As shown in Figure 11, our proposed EISL can reduce repetition to a large extent.
Figure 11: The percentage of repeated tokens under different iteration steps.
Complexity analysis Given  $T^{*}$  tokens, the time complexity of CE loss is  $\mathbf{O}(T^{*})$ , while the complexity of  $n$ -gram EISL loss is  $\mathbf{O}(n(T^{*} - n + 1)^{2}) \approx \mathbf{O}(T^{*2})$ , assuming small  $n$  is used in practice (e.g.,  $n \in \{1, 2, 3, 4\}$ ). However, in practice, the computation cost of the loss (either CE or EISL) is negligible compared to the cost of model forward and backward during training. Thus, the extra cost introduced by the EISL loss is rather minor.
Empirical comparison of time cost To quantify the computational cost of different methods, we adopt CE and EISL on top of the same model and setting, and evaluate the consumed time for 1 training epoch. For comparison on both small and large dataset, we evaluate on Multi30k (29k training data, 1k test data) and 1M scale WMT-18 raw corpus (1M training data, 3k test data). The models are tested on one Tesla V100 DGXS with 32 GB memory, the batch size is 128, max number of tokens is 6000 and update frequency is 4. For each method, we test 6 times and average the results as final time. The results are shown in Figure 12.