SlowGuess/ACL_2020_OCR huggingface dataset:
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. In Proceedings of International Conference on Learning Representations (ICLR).
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. 2018. Fast decoding in sequence models using discrete latent variables. In International Conference on Machine Learning, pages 2395-2404.
Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. 2020. Parallel machine translation with disentangled context transformer.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. In Proceedings of International Conference on Learning Representations (ICLR).
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc'Aurelio Ranzato, and Fu-Jie Huang. 2006. A tutorial on energy-based learning. In Predicting Structured Data. MIT Press.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173-1182, Brussels, Belgium. Association for Computational Linguistics.
Jindrich Libovicky and Jindrich Helcl. 2018. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3016-3021, Brussels, Belgium. Association for Computational Linguistics.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. FlowSeq: Nonautoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4281-4291, Hong Kong, China. Association for Computational Linguistics.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pages 1278-1286.
Chitwan Sahara, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Non-autoregressive machine translation with latent alignments. arXiv preprint arXiv:2004.07437.
Chenze Shao, Jinchao Zhang, Yang Feng, Fandong Meng, and Jie Zhou. 2020. Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. In AAAI.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. 2019. Fast structured decoding for sequence models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3016-3026. Curran Associates, Inc.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pages 3104-3112. Curran Associates, Inc.
Lifu Tu and Kevin Gimpel. 2018. Learning approximate inference networks for structured prediction. In Proceedings of International Conference on Learning Representations (ICLR).
Lifu Tu and Kevin Gimpel. 2019. Benchmarking approximate inference methods for neural structured prediction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3313-3324, Minneapolis, Minnesota. Association for Computational Linguistics.
Lifu Tu, Richard Yuanzhe Pang, and Kevin Gimpel. 2019. Improving joint training of inference networks and structured prediction energy networks. arXiv preprint arXiv:1911.02891.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio,
H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019. Imitation learning for non-autoregressive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1304-1312, Florence, Italy. Association for Computational Linguistics.
Chunting Zhou, Jiatao Gu, and Graham Neubig. 2020. Understanding knowledge distillation in non-autoregressive machine translation. In International Conference on Learning Representations (ICLR).
A.1 Training with Distilled Outputs vs. Training with Energy
In order to compare ENGINE with training on distilled outputs, we train BiLSTM models in three ways: "baseline" which is trained with the human-written reference translations, "distill" which is trained with the distilled outputs (generated using the autoregressive models), and "ENGINE", our method which trains the BiLSTM as an inference network to minimize the pretrained seq2seq autoregressive energy. Oracle lengths are used for decoding. Table 5 shows test results for both datasets, showing significant gains of ENGINE over the baseline and distill methods. Although the results shown here are lower than the transformer results, the trend is clearly indicated.
IWSLT14 Energy (↓)	DE-EN BLEU (↑)	WMT16 Energy (↓)	RO-EN BLEU (↑)
baseline	153.54	8.28	175.94	9.47
distill	112.36	14.58	205.71	5.76
ENGINE	51.98	19.55	64.03	21.69
Table 5: Test results of non-autoregressive models when training with the references ("baseline"), distilled outputs ("distill"), and energy ("ENGINE"). Oracle lengths are used for decoding. Here, ENGINE uses BiLSTM inference networks and pretrained seq2seq AR energies. ENGINE outperforms training on both the references and a pseudocorpus.
A.2 Analysis of Translation Results
In Table 6, we present randomly chosen translation outputs from WMT16 RO-EN. For each Romanian sentence, we show the reference from the dataset, the translation from CMLM, and the translation
Source:seful onu a solicitat din nou tuturor partilor, inclusiv consiliului de secureitate onu divizat sa se unifice si sa sustinanegocierile pentru a gasio o solutie politica.Reference :the u.n. chief again urged all parties, including the divided u.n. security council, to unite and support inclusivenegotiations to find a political solution .CMLM :the un chief again again urged all parties, including the divided un security council to unify and support negotiationsin order to find a political solution .ENGINE :the un chief has again urged all parties, including the divided un security council to unify and support negotiations inorder to find a political solution .
Source:adevarul este ca a rupt o racheta atunci cand a pierdut din cauza ca a acuzat crampe in us , insa nu este primul jucatorcare rupe o racheta din frustrare fata de el insusi si il cunosc pe thanasi suficient de bine incat sa stiu ca nu s @-@ armandri cu asta.Reference :he did break a racquet when he lost when he cramped in the us , but he &apos;s not the first player to break a racquetout of frustration with himself , and i know thanasi well enough to know he wouldn &apos;s t be proud of that .CMLM :the truth is that it has broken a rocket when it lost because accused crcrpe in the us , but it is not the first player tobreak rocket rocket rocket frustration frustration himself himself and i know thanthanasi enough enough know knowhewould not be proud of that .ENGINE :the truth is that it broke a rocket when it lost because he accused crpe in the us , but it is not the first player to break areocket from frustration with himself and i know thanasi well well enough to know he would not be proud of it .
Source:realizatoriui studiului mai transmit ca &quot; romaniis simt nevoie de ceva mai multa aventura in viata lor ( 24 % ) ,urmat de afectiene ( 21 % ) , bani ( 21 % ) , siguranta ( 20 % ) , nou ( 19 % ) , sex ( 19 % ) , respect 18 % , incredere17 % , placere 17 % , conectare 17 % , cunoastere 16 % , protectie 14 % , importante 14 % , invatare 12 % , libertate11 % , autocunoastere 10 % si control 7 % &quot; .Reference :the study &apos;s conductors transmit that &quot; romanians feel the need for a little more adventure in their lives (24 % ) , followed by affection ( 21 % ) , money ( 21 % ) , safety ( 20 % ) , new things ( 19 % ) , sex ( 19 % ) respect18 % , confidence 17 % , pleasure 17 % , connection 17 % , knowledge 16 % , protection 14 % , importance 14 % ,learning 12 % , freedom 11 % , self @-@ awareness 10 % and control 7 % . &quot;CMLM :survey survey makers say that &apos; romanians romanians some something adventadventure ure their lives 24 24 %) followed followed by % % % % % , ( 21 % % ) , safety ( % % % ) , new19 % ) , ), 19 % % % , respect 18 % % %% % % % % , , % % % % % % , , % , 14 % , 12 % %ENGINE :realisation of the survey say that &apos; romanians feel a slightly more adventure in their lives ( 24 % ) followed byaff% ( 21 % ) , money ( 21 % ) , safety ( 20 % ) , new 19 % ) , sex ( 19 % ) , respect 18 % , confidence 17 % , 17 % ,connecting 17 % , knowledge % % , 14 % , 14 % , 12 % %
Table 6: Examples of translation outputs from ENGINE and CMLM on WMT16 RO-EN without refinement iterations.
from ENGINE. We observe that without the refinement iterations, CMLM performs well for shorter source sentences. However, it still prefers generating repeated tokens. ENGINE, on the other hand, generates much better translations with fewer repeated tokens.
2826