Question: Does the Transformerbased concat22 outperform the RNN-based concat22?
Evidence: The BLEU scores in Table TABREF30 show a moderate improvement for most context-aware systems. This suggests that the architectural changes for the context-aware models do not degrade overall translation quality. The contrastive evaluation on our test set on the other hand shows a clear increase in the accuracy of pronoun translation: The best model s-hier-to-2.tied achieves a total of +16 percentage points accuracy on the test set over the baseline, see Table TABREF31 .
Table TABREF32 shows that context-aware models perform better than the baseline when the antecedent is outside the current sentence. In our experiments, all context-aware models consider one preceding sentence as context. The evaluation according to the distance of the antecedent in Table TABREF35 confirms that the subset of sentences with antecedent distance 1 benefits most from the tested context-aware models (up to +20 percentage points accuracy). However, we note two surprising patterns:
The first observation can be explained by the distribution of German pronouns in the test set. The further away the antecedent, the higher the percentage of it INLINEFORM0 es cases, which are the majority class, and thus the class that will be predicted most often if evidence for other classes is lacking. We speculate that this is due to our more permissive extraction heuristics for it INLINEFORM1 es.
We attribute the second observation to the existence of coreference chains where the preceding sentence contains a pronoun that refers to the same nominal antecedent as the pronoun in the current sentence. Consider the example in Table TABREF36 : The nominal antecedent of it in the current sentence is door, Tür in German with feminine gender. The nominal antecedent occurs two sentences before the current sentence, but the German sentence in between contains the pronoun sie, which is a useful signal for the context-aware models, even though they cannot know the nominal antecedent.
Note that only models aware of target-side context can benefit from such circumstances: The s-hier models as well as the Transformer model by BIBREF8 only see source side context, which results in lower accuracy if the distance to the antecedent is INLINEFORM0 1, see Table TABREF35 .
While such coreference chains complicate the interpretation of the results, we note that improvements on inter-sentential anaphora with antecedent distance INLINEFORM0 are relatively small (compared to distance 1), and that performance is still relatively poor (especially for the minority classes er and sie). We encourage evaluation of wider-context models on this subset, which is still large thanks to the size of the full test set.
Regarding the comparison of different context-aware architectures, our results demonstrate the effectiveness of parameter sharing between the main encoder (or decoder) and the contextual encoder. We observe an improvement of 5 percentage points from s-hier-to-2 to s-hier-to-2.tied, and 4 percentage points from s-t-hier to s-t-hier.tied. Context encoders introduce a large number of extra parameters, while inter-sentential context is only relevant for a relatively small number of predictions. We hypothesize that the training signal is thus too weak to train a strong contextual encoder in an end-to-end fashion without parameter sharing. Our results also confirm the finding by BIBREF9 that multi-encoder architectures, specifically s-hier-to-2(.tied), can outperform a simple concatenation system in the translation of coreferential pronouns.
The Transformer-based models perform strongest on pronouns with intra-segmental antecedent, outperforming the recurrent baseline by 9–18 percentage points. This is likely an effect of increased model depth and the self-attentional architecture in this set of experiments. The model by BIBREF8 only uses source context, and outperforms the most comparable RNN system, s-hier.tied. However, the Transformer-based concat22 slightly underperforms the RNN-based concat22, and we consider it future research how to better exploit target context with Transformer-based models.
However, the Transformerbased concat22 slightly underperforms the RNN-based concat22, and we consider it future research how to better exploit target context with Transformer-based models.
Answer: No, it doesn't.