Question: What are the two major categories of context-aware NMT models that we use in the experiments?
Evidence: If the translation of a pronoun requires context beyond the current sentence (see the example in Figure FIGREF1 ), a natural extension of sentence-level NMT models is to condition the model prediction on this necessary context. In the following, we describe a number of existing approaches to making models “aware” of additional context.
The simplest possible extension is to translate units larger than sentences. BIBREF5 concatenate each sentence with the sentence that precedes it, for the source side of the corpus or both sides. All of their models are standard sequence-to-sequence models built with recurrent neural networks (RNNs), since the method does not require any architectural change. BIBREF11 use the same concatenation technique with a Transformer architecture BIBREF2 , and experiment with wider context.
A number of works do propose changes to the NMT architecture. A common technique is to extend a standard encoder-decoder model by additional encoders for the context sentence(s), with a modified attention mechanism BIBREF6 , BIBREF9 , BIBREF8 . One aspect that differs between these works is the architecture of the encoder and attention. While BIBREF6 , BIBREF9 extend an RNN encoder-decoder with a second encoder that the decoder attends to, BIBREF8 extend the Transformer architecture with an encoder that is attended to by the main encoder. BIBREF8 also introduce parameter sharing between the main encoder and the context encoder, but do not empirically demonstrate its importance.
While the number of encoded sentences in the previous work is fixed, BIBREF7 , BIBREF10 explore the integration of variable-size context through a hierarchical architecture, where a first-level RNN reads in words to produce sentence vectors, which are then fed into a second-level RNN to produce a document summary.
Apart from differences in the architectures, related work varies in whether it considers source context, target context, or both (see Table TABREF9 for an overview of language arcs and context types). Some work considers only source context, but for pronoun translation, target-side context is intuitively important for disambiguation, especially if the antecedent itself is ambiguous. In our evaluation, we therefore emphasize models that take into account both source and target context.
Our experiments are based on models from BIBREF9 , who have released their source code. We extend their models with parameter sharing, which was shown to be beneficial by BIBREF8 . Additionally, we consider a concatenative baseline, similar to BIBREF5 , and Transformer-based models BIBREF8 .
This section describes several context-aware NMT models that we use in our experiments. They fall into two major categories: models based on RNNs and models based on the Transformer architecture BIBREF2 . We experiment with additional context on the source side and target side.
This section describes several context-aware NMT models that we use in our experiments. They fall into two major categories: models based on RNNs and models based on the Transformer architecture (Vaswani et al., 2017). We experiment with additional context on the source side and target side.
Answer: They fall into two major categories: models based on RNNs and models based on the Transformer architecture.