SlowGuess/NAACL_2021_OCR huggingface dataset:
Antonios Anastasopoulos and David Chiang. 2018. Tied multitask learning for neural speech translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 82-91, New Orleans, Louisiana. Association for Computational Linguistics.
Parnia Bahar, Tobias Bieschke, and Hermann Ney. 2019. A comparative study on end-to-end speech to text translation. In Proceedings of ASRU, pages 792-799.
Sameer Bansal, Herman Kamper, Karen Livescu, Adam Lopez, and Sharon Goldwater. 2019. Pretraining on high-resource speech recognition improves low-resource speech-to-text translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 58-68, Minneapolis, Minnesota. Association for Computational Linguistics.
Rachel Bawden, Biao Zhang, Andre Tattar, and Matt Post. 2020a. ParBLEU: Augmenting metrics with automatic paraphrases for the wmt'20 metrics shared task. In 5th Conference on Machine Translation.
Rachel Bawden, Biao Zhang, Lisa Yankovskaya, Andre Tattar, and Matt Post. 2020b. A study in improving BLEU reference coverage with diverse automatic paraphrasing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 918-932, Online. Association for Computational Linguistics.
Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin. 2018. End-to-end automatic speech translation of audiobooks. In Proceedings of ICASSP, pages 6224-6228.
Alexandre Bérard, Olivier Pietquin, Christophe Servan, and Laurent Besacier. 2016. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In Proceedings of NeurIPS 2016 End-to-end Learning for Speech and Audio Processing Workshop.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In Proceedings of NeurIPS, pages 7059-7069.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2012-2017, Minneapolis, Minnesota. Association for Computational Linguistics.
Liang Ding, Longyue Wang, Xuebo Liu, Derek F. Wong, Dacheng Tao, and Zhaopeng Tu. 2021. Understanding and improving lexical choice in non-autoregressive translation. In Proceedings of ICLR.
Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648, Atlanta, Georgia. Association for Computational Linguistics.
Christian Federmann, Oussama Elachqar, and Chris Quirk. 2019. Multilingual whispers: Generating paraphrases with translation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 17-26, Hong Kong, China. Association for Computational Linguistics.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Semi-autoregressive training improves mask-predict decoding. arXiv preprint arXiv:2001.08785.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6121, Hong Kong, China. Association for Computational Linguistics.
Mitchell A Gordon and Kevin Duh. 2019. Explaining sequence-level knowledge distillation as data-augmentation for neural machine translation. arXiv preprint arXiv:1912.03334.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In Proceedings of ICLR.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented Transformer for speech recognition. In Proceedings of Interspeech, pages 5036-5040.
J Edward Hu, Rachel Rudinger, Matt Post, and Benjamin Van Durme. 2019. ParaBank: Monolingual bitext generation and sentential paraphrasing via
lexically-constrained neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6521-6528.
Hirofumi Inaguma, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. 2019. Multilingual end-to-end speech translation. In Proceedings of ASRU, pages 570-577.
Hirofumi Inaguma, Yosuke Higuchi, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. 2021. Orthros: Non-autoregressive end-to-end speech translation with dual-decoder. In Proceedings of ICASSP.
Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, and Shinji Watanabe. 2020. ESPnet-ST: All-in-one speech translation toolkit. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 302-311, Online. Association for Computational Linguistics.
Sathish Indurthi, Houjeung Han, Nikhil Kumar Lakumarapu, Beomseok Lee, Insoo Chung, Sangha Kim, and Chanwoo Kim. 2020. End-end speech-to-text translation with modality agnostic meta-learning. In Proceedings of ICASSP, pages 7904-7908.
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, et al. 2019. A comparative study on Transformer vs RNN in speech applications. In Proceedings of ASRU, pages 499-456.
Huda Khayrallah, Brian Thompson, Matt Post, and Philipp Koehn. 2020. Simulated multiple reference training improves low-resource machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 82-89, Online. Association for Computational Linguistics.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. Proceedings of ICLR.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015. Audio augmentation for speech recognition. In Proceedings of Interspeech, pages 3586-3589.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In
Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177-180, Prague, Czech Republic. Association for Computational Linguistics.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. 2020. Dual-decoder transformer for joint automatic speech recognition and multilingual speech translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3520-3533, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Yuchen Liu, Hao Xiong, Zhongjun He, Jiajun Zhang, Hua Wu, Haifeng Wang, and Chengqing Zong. 2019. End-to-end speech translation with knowledge distillation. In Proceedings of Interspeech, pages 1128-1132.
Yuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou, Zhongjun He, Hua Wu, Haifeng Wang, and Chengqing Zong. 2020. Synchronous speech recognition and speech-to-text translation with interactive decoding. In Proceedings of AAAI, pages 8417-8424.
Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 881-893, Valencia, Spain. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. SpecAugment: A simple data augmentation method for automatic speech recognition. In Proceedings of Interspeech, pages 2613-2617.
Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011. The kaldi speech recognition toolkit. In Proceedings of ASRU.
Yi Ren, Jinglin Liu, Xu Tan, Zhou Zhao, Sheng Zhao, and Tie-Yan Liu. 2020. A study of non-autoregressive model for sequence generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 149-159, Online. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, volume 200. Cambridge, MA.
Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2019. Attention-passing models for robust and data-efficient end-to-end speech translation. Transactions of the Association for Computational Linguistics, 7:313-325.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proc. of CVPR, pages 2818-2826.
Brian Thompson and Matt Post. 2020. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 90-121, Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, and Juan Pino. 2020a. Fairseq S2T: Fast speech-to-text modeling with fairseq. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations, pages 33-39, Suzhou, China. Association for Computational Linguistics.
Changhan Wang, Anne Wu, and Juan Pino. 2020b. CoVoST 2: A massively multilingual speech-to-text translation corpus. arXiv preprint arXiv:2007.10310.
Chengyi Wang, Yu Wu, Shujie Liu, Zhenglu Yang, and Ming Zhou. 2020c. Bridging the gap between pretraining and fine-tuning for end-to-end speech translation. In Proceedings of AAAI, pages 9161-9168.
Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. 2020d. Curriculum pre-training for end-to-end speech translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3728-3738, Online. Association for Computational Linguistics.
Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. 2017. Hybrid CTC/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 11(8):1240-1253.
Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017. Sequence-to-sequence models can directly translate foreign speech. In Proceedings of Interspeech, pages 2625-2629.
John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Melbourne, Australia. Association for Computational Linguistics.
John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning paraphrastic sentence embeddings from back-translated bitext. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 274-285, Copenhagen, Denmark. Association for Computational Linguistics.
Chunting Zhou, Jiatao Gu, and Graham Neubig. 2019a. Understanding knowledge distillation in non-autoregressive machine translation. In Proceedings of ICLR.
Zhong Zhou, Matthias Sperber, and Alexander Waibel. 2019b. Paraphrases as foreign languages in multilingual neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 113-122, Florence, Italy. Association for Computational Linguistics.
Reference1	She took our order, and then went to the couple in the booth next to us, and she lowered her voice so much, I had to really strain to hear what she was saying.
\( \bar{\text{Paraphrase}}\bar{I} \) (Backward NMT)	She picked up our order, and then went to the pair in the niche next to us and lowered her voice so much that I had to really try to understand them.
Reference2	And she said "Yes, that's former Vice President Al Gore and his wife, Tipper." And the man said, "He's come down a long way, hasn't he?" (Laughter)
\( \bar{\text{Paraphrase}}\bar{2} \) (Backward NMT)	She said, "Yes, that's ex-vice President Al Gore and his wife Tipper." And the man said, "It's a nice gap, what?" (Laughter)
Table 8: Examples of source paraphrases on the Must-C En-De training set
All sentences were tokenized with the tokenizer.perl script in Moses (Koehn et al., 2007). Non-verbal speech labels such as "(Applause)" and "(Laughter)" were removed during evaluation (Di Gangi et al., 2019; Inaguma et al., 2021; Le et al., 2020). We built output vocabularies based on the byte pair encoding (BPE) algorithm (Sennrich et al., 2016) with the Sentencepiece toolkit (Kudo, 2018) $^{6}$ . The joint source and target vocabularies were constructed in the ST and MT tasks, while the vocabularies in the ASR task were constructed with transcriptions only. For autoregressive models, we used 5k for ASR models and 8k for E2E-ST and MT models. We used 16k vocabularies for non-autoregressive E2E-ST models (Inaguma et al., 2021).
For input speech features, we extracted 80-channel log-mel filterbank coefficients computed with a 25-ms window size and shifted every 10ms with 3-dimensional pitch features using Kaldi (Povey et al., 2011). This results in 83-dimensional features for every frame. The features were normalized by the mean and the standard deviation for each training set. To avoid overfitting, training data was augmented by a factor of 3 with speed perturbation (Ko et al., 2015) and SpecAugment (Park et al., 2019). We used  $(m_T, m_F, T, F) = (2, 2, 40, 30)$  for the hyperparameters in SpecAugment.
A.2 Model configuration
We used the Transformer (Vaswani et al., 2017) architecture implemented with the ESPnet-ST toolkit (Inaguma et al., 2020) for all tasks. ASR and E2E-ST models consisted of 12 speech encoder blocks and six decoder blocks. The speech encoders had two CNN blocks with a kernel size of 3 and a channel size of 256 before the first
Transformer encoder layer, which resulted in 4-fold downsampling in the time and frequency axes. The text encoder in the MT models consisted of six Transformer blocks. The dimensions of the self-attention layer  $d_{\mathrm{model}}$  and feed-forward network  $d_{\mathrm{ff}}$  were set to 256 and 2048, respectively, and the number of attention heads  $H$  was set to 4. For a large Transformer model configuration, we increased  $d_{\mathrm{ff}}$  from 256 to 512 and  $H$  from 4 to 8. For a Conformer model configuration, we set  $d_{\mathrm{model}} = 256$ ,  $d_{\mathrm{ff}} = 2048$ , and  $H = 4$ . The kernel size of depthwise separable convolution was set to 15. None of the other training or decoding hyperparameters were modified.
A.3 Initialization
In addition to initializing the encoder parameters of the E2E-ST model by those of the pre-trained ASR model, the auxiliary AR decoder parameters of the NAR models were initialized by those of the corresponding pre-trained AR MT model (Inaguma et al., 2021). The other decoder parameters of both the AR and NAR models were initialized as in BERT (Devlin et al., 2019; Ghazvininejad et al., 2019; Inaguma et al., 2021), where weight parameters were sampled from  $\mathcal{N}(0,0.02)$ , biases were set to zero, and layer normalization parameters were set to  $\beta = 0$ ,  $\gamma = 1$ . Note that we did not use additional data for pre-training.
A.4 Training
The Adam optimizer (Kingma and Ba, 2015) with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$ , and  $\epsilon = 10^{-9}$  was used for training with a Noam learning rate schedule (Vaswani et al., 2017). We used dropout and label smoothing (Szegedy et al., 2016) with a probability of 0.1 and 0.1, respectively. The other training configurations for all tasks are summarized in Table 9. We removed utterances having more than 3000 input speech frames or more than 400 characters due to the GPU memory capacity. The last five best checkpoints based on the validation performance were used for model averaging.
Configuration	ASR	E2E-ST	MT
AR	NAR
Warmup step	25k	25k	25k	8k
Learning rate factor	5.0 (2.0)	2.5	5.0	1.0
Batch size × accum	128	128	256	96
Epoch	45 (30)	30	50	100
Validation metric	Accuracy	BLEU	BLEU	BLEU
Table 9: Summary of training configuration. Numbers inside parentheses correspond to Conformer.
For the training of ASR models used for E2E-ST encoder pre-training, we removed case and punctuation information from transcriptions and then applied a joint CTC/Attention objective (Watanabe et al., 2017). However, we retained this information in the transcriptions and paraphrases used for training the E2E-ST and MT models.
A.5 Case study
We present examples of generated paraphrases on the Must-C En-De training set in Table 8. We observed that most paraphrases kept the original meaning while some words were simplified to alternatives having a similar meaning. We also found that the first conjunction in an utterance was more likely to be omitted via paraphrasing.
A.6 Mathematical formulation of complexity and faithfulness
In this section, we mathematically formulate the corpus-level complexity and faithfulness given  $\mathcal{D} \in \{\mathcal{D}_{\mathrm{st}}, \mathcal{D}_{\mathrm{st}}^{\mathrm{fwd}}, \mathcal{D}_{\mathrm{st}}^{\mathrm{bwd}}, \mathcal{D}_{\mathrm{st}}^{\mathrm{bidir}}\}$ . Our formulation follows (Zhou et al., 2019a), but we also consider the reverse language direction.
Conditional entropy (complexity) The corpus-level complexity of  $\mathcal{D}$  in the forward language direction,  $\mathcal{C}(\overrightarrow{\mathcal{D}})$ , is defined as the conditional entropy  $\mathcal{H}(Y^{\mathrm{t}}|Y^{\mathrm{s}})$  normalized over all samples.  $\mathcal{H}(Y^{\mathrm{t}}|Y^{\mathrm{s}})$  is defined as
$$
\begin{array}{l} \mathcal {H} (Y ^ {\mathrm {t}} | Y ^ {\mathrm {s}}) \\ = - \sum_ {i = 1} ^ {I} p \left(Y _ {i} ^ {\mathrm {t}} \mid Y _ {i} ^ {\mathrm {s}}\right) \cdot \log p \left(Y _ {i} ^ {\mathrm {t}} \mid Y _ {i} ^ {\mathrm {s}}\right) \\ \approx - \sum_ {i = 1} ^ {I} \left(\prod_ {k = 1} ^ {| Y _ {i} ^ {\mathrm {t}} |} p \left(Y _ {i, k} ^ {\mathrm {t}} \mid Y _ {i} ^ {\mathrm {s}}\right)\right) \cdot \sum_ {k = 1} ^ {| Y _ {i} ^ {\mathrm {t}} |} \log p \left(Y _ {i, k} ^ {\mathrm {t}} \mid Y _ {i} ^ {\mathrm {s}}\right) \\ \approx -\sum_{k = 1}^{T^{\mathrm{t}}}\sum_{y_{k}^{\mathrm{t}}\in \mathcal{A}(Y_{i}^{\mathrm{s}})}p(y_{k}^{\mathrm{t}}|\operatorname {Align}(y_{k}^{\mathrm{t}}))\cdot \log p(y_{k}^{\mathrm{t}}|\operatorname {Align}(y_{k}^{\mathrm{t}})) \\ = \sum_ {k = 1} ^ {T ^ {\mathrm {s}}} \mathcal {H} \left(y ^ {\mathrm {t}} \mid Y _ {i, k} ^ {\mathrm {s}}\right), \\ \end{array}
$$
where  $\mathcal{A}$  is an external alignment model, and  $T^{\mathrm{s}}$  and  $T^{\mathrm{t}}$  are the source and target sequence lengths, respectively. We make two assumptions: (1) conditional independence of target tokens given the source text sequence, and (2) the distribution of  $p(y^{\mathrm{t}}|Y^{\mathrm{s}})$  follows the alignment model  $\mathcal{A}$ . Then,  $\mathcal{C}(\overrightarrow{D})$  is calculated as
$$
\mathcal {C} (\overrightarrow {\mathcal {D}}) = \frac {1}{| \mathcal {V} ^ {\mathrm {s}} |} \sum_ {y ^ {\mathrm {s}} \in \mathcal {V} ^ {\mathrm {s}}} \mathcal {H} (y ^ {\mathrm {t}} | y ^ {\mathrm {s}}),
$$
where  $\mathcal{V}^{\mathrm{s}}$  is a set of all words in the source language. Division by  $|\mathcal{V}^{\mathrm{s}}|$  is important to normalize frequent source words.
The corpus-level complexity of  $\mathcal{D}$  in the backward language direction,  $\mathcal{C}(\overleftarrow{\mathcal{D}})$ , is defined similarly as
$$
\mathcal {C} (\overleftarrow {\mathcal {D}}) = \frac {1}{| \mathcal {V} ^ {\mathrm {t}} |} \sum_ {y ^ {\mathrm {t}} \in \mathcal {V} ^ {\mathrm {t}}} \mathcal {H} (y ^ {\mathrm {s}} | y ^ {\mathrm {t}}),
$$
where  $\mathcal{V}^{\mathrm{t}}$  is a set of all words in the target language.
Faithfulness Although the corpus-level conditional entropy can be used to evaluate the complexity of the training data, there are also trivial solutions to generate new data with smaller complexity when target translations are not adequate. Faithfulness is a good measure to assess how close the distilled data distribution is to the real (original) data distribution. The faithfulness of  $\mathcal{D}$  in a forward language direction  $\mathcal{F}(\overrightarrow{\mathcal{D}})$  and a backward language direction  $\mathcal{F}(\overleftarrow{\mathcal{D}})$  is defined as the KL-divergence of the alignment distribution between the real dataset and a distilled dataset, as
$$
\mathcal {F} (\vec {\mathcal {D}}) = \frac {1}{| \mathcal {V} ^ {\mathrm {s}} |} \sum_ {y ^ {\mathrm {s}} \in \mathcal {V} ^ {\mathrm {s}}} \sum_ {y ^ {\mathrm {t}} \in \mathcal {V} ^ {\mathrm {t}}} p _ {\mathrm {r}} (y ^ {\mathrm {t}} | y ^ {\mathrm {s}}) \log \frac {p _ {\mathrm {r}} (y ^ {\mathrm {t}} | y ^ {\mathrm {s}})}{p _ {\mathrm {d}} (y ^ {\mathrm {t}} | y ^ {\mathrm {s}})},
$$
$$
\mathcal {F} (\overleftarrow {\mathcal {D}}) = \frac {1}{| \mathcal {V} ^ {\mathrm {t}} |} \sum_ {y ^ {\mathrm {t}} \in \mathcal {V} ^ {\mathrm {t}}} \sum_ {y ^ {\mathrm {s}} \in \mathcal {V} ^ {\mathrm {s}}} p _ {\mathrm {r}} (y ^ {\mathrm {s}} | y ^ {\mathrm {t}}) \log \frac {p _ {\mathrm {r}} (y ^ {\mathrm {s}} | y ^ {\mathrm {t}})}{p _ {\mathrm {d}} (y ^ {\mathrm {s}} | y ^ {\mathrm {t}})},
$$