## 1 Introduction  
Currently, machine learning (ML) is experiencing a renaissance where deep learning (DL) has been the main driving force. Deep neural networks (DNNs) are extremely powerful machine learning models that can achieve promising performance on challenging problems such as speech recognition [1; 2] and visual object recognition [3; 4; 5; 6]. In particular, due to the capacity of capturing complex linguistic structures, DNNs have enabled great breakthroughs in natural language processing (NLP) [7; 8; 9]. Among the NLP tasks, machine translation (MT) is a successful representative, and its main task is using computer software to translate text or speech from one language to another. It is a common belief that machine translation has experienced three major development waves: rule-based machine translation (RMT) [10], statistical machine translation (SMT) [11], and neural machine translation (NMT) [12]. SMT has been the mainstream driving force during the past two decades. However, this approach may ignore the long dependency beyond the length of phrases and thus cause inconsistencies in translation results such as incorrect gender agreements. It also suffers in separate components such as word aligners, translation rule extractors, and other feature extractors. Compared with the SMTmethod, NMT has a simple architecture, and it is able to capture long dependency in the sentence, which shows a great potential in becoming a new trend of language translation. The dominant NMT approach is the \"Embed-Encode-Attend-Decode\" paradigm. Recurrent neural network (RNN) [13], convolutional neural network (CNN) [14], and self-attention/feed-forward network (SA/FFN) [15] architectures are the most commonly used approaches based on this paradigm. In particular, Google has proposed Transformer, which relies entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or CNNs [15]. The Transformer model aims to deal with long-range dependencies when solving the sequence-to-sequence tasks. This model has outperformed many other models, which is thus the focus of our work. Recent works have focused on large pretrained models, which are mostly built based on the Transformer model, but with much larger capacity [15; 16; 17; 18]. This approach utilizes a combination of pretraining and supervised fine-tuning. The capacity of transformer language models has increased significantly, from 100 million parameters [16], to 1.5 billion parameters [17], and finally 17 billion parameters [9; 18]. Although each increase has brought significant performance improvements in downstream NLP tasks, training such models requires large-scale specialized computing hardware such as Google's TPUs [19]. These computing clusters are typically unaffordable for small/medium-sized enterprises. On the other hand, these models are too complicated and their capacity is too large for us to understand. That is, we know the models perform well, but we do not know the reasons. They work similar to a \"black-box\" and are particularly unsuitable for teaching purposes. The dominant approach to creating machine learning systems is to collect a dataset of training examples demonstrating correct behavior for a desired task, train a system to imitate these behaviors, and then test its performance on independent held-out examples. This approach can provide trained models that work like domain experts, which has been widely accepted. Many prior works have shown that trained models can yield a better prediction accuracy from using a domain-specific bilingual corpus [20; 21; 22]. For example, microblogs are an excellent linguistic resource. [PERSON] et al. have shown that some microblog users post \"self-translated\" messages targeting audiences who speak different languages [21]. Based on this observation, the authors have introduced a method for finding and extracting this naturally occurring Chinese-English parallel segments.