Abstract: In automatic post-editing (APE) it makes sense to condition post-editing (pe) decisions on both the source (src) and the machine translated text (mt) as input. This has led to multi-source encoder based APE approaches. A research challenge now is the search for architectures that best support the capture, preparation and provision of src and mt information and its integration with pe decisions. In this paper we present a new multi-source APE model, called transference. Unlike previous approaches, it (i) uses a transformer encoder block for src, (ii) followed by a decoder block, but without masking for self-attention on mt, which effectively acts as second encoder combining src -> mt, and (iii) feeds this representation into a final decoder block generating pe. Our model outperforms the state-of-the-art by 1 BLEU point on the WMT 2016, 2017, and 2018 English--German APE shared tasks (PBSMT and NMT). We further investigate the importance of our newly introduced second encoder and find that a too small amount of layers does hurt the performance, while reducing the number of layers of the decoder does not matter much.

Introduction: The performance of state-of-the-art MT systems is not perfect, thus, human interventions are still required to correct machine translated texts into publishable quality translations BIBREF0. Automatic post-editing (APE) is a method that aims to automatically correct errors made by MT systems before performing actual human post-editing (PE) BIBREF1, thereby reducing the translators' workload and increasing productivity BIBREF2. APE systems trained on human PE data serve as MT post-processing modules to improve the overall performance. APE can therefore be viewed as a 2nd-stage MT system, translating predictable error patterns in MT output to their corresponding corrections. APE training data minimally involves MT output ($mt$) and the human post-edited ($pe$) version of $mt$, but additionally using the source ($src$) has been shown to provide further benefits BIBREF3, BIBREF4, BIBREF5. To provide awareness of errors in $mt$ originating from $src$, attention mechanisms BIBREF6 allow modeling of non-local dependencies in the input or output sequences, and importantly also global dependencies between them (in our case $src$, $mt$ and $pe$). The transformer architecture BIBREF7 is built solely upon such attention mechanisms completely replacing recurrence and convolutions. The transformer uses positional encoding to encode the input and output sequences, and computes both self- and cross-attention through so-called multi-head attentions, which are facilitated by parallelization. Such multi-head attention allows to jointly attend to information at different positions from different representation subspaces, e.g. utilizing and combining information from $src$, $mt$, and $pe$. In this paper, we present a multi-source neural APE architecture called transference. Our model contains a source encoder which encodes $src$ information, a second encoder ($enc_{src \rightarrow mt}$) which takes the encoded representation from the source encoder ($enc_{src}$), combines this with the self-attention-based encoding of $mt$ ($enc_{mt}$), and prepares a representation for the decoder ($dec_{pe}$) via cross-attention. Our second encoder ($enc_{src \rightarrow mt}$) can also be viewed as a standard transformer decoding block, however, without masking, which acts as an encoder. We thus recombine the different blocks of the transformer architecture and repurpose them for the APE task in a simple yet effective way. The suggested architecture is inspired by the two-step approach professional translators tend to use during post-editing: first, the source segment is compared to the corresponding translation suggestion (similar to what our $enc_{src \rightarrow mt}$ is doing), then corrections to the MT output are applied based on the encountered errors (in the same way that our $dec_{pe}$ uses the encoded representation of $enc_{src \rightarrow mt}$ to produce the final translation). The paper makes the following contributions: (i) we propose a new multi-encoder model for APE that consists only of standard transformer encoding and decoding blocks, (ii) by using a mix of self- and cross-attention we provide a representation of both $src$ and $mt$ for the decoder, allowing it to better capture errors in $mt$ originating from $src$; this advances the state-of-the-art in APE in terms of BLEU and TER, and (iii), we analyze the effect of varying the number of encoder and decoder layers BIBREF8, indicating that the encoders contribute more than decoders in transformer-based neural APE.

Question: What neural machine translation models can learn in terms of transfer learning?
Answer: Multilingual Neural Machine Translation Models