Abstract: Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs.

Introduction: Recent approaches to sequence to sequence learning typically leverage recurrence BIBREF0 , convolution BIBREF1 , BIBREF2 , attention BIBREF3 , or a combination of recurrence and attention BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 as basic building blocks. These approaches incorporate information about the sequential position of elements differently. Recurrent neural networks (RNNs) typically compute a hidden state $h_t$ , as a function of their input at time $t$ and a previous hidden state $h_{t-1}$ , capturing relative and absolute positions along the time dimension directly through their sequential structure. Non-recurrent models do not necessarily consider input elements sequentially and may hence require explicitly encoding position information to be able to use sequence order. One common approach is to use position encodings which are combined with input elements to expose position information to the model. These position encodings can be a deterministic function of position BIBREF8 , BIBREF3 or learned representations. Convolutional neural networks inherently capture relative positions within the kernel size of each convolution. They have been shown to still benefit from position encodings BIBREF1 , however. For the Transformer, which employs neither convolution nor recurrence, incorporating explicit representations of position information is an especially important consideration since the model is otherwise entirely invariant to sequence ordering. Attention-based models have therefore used position encodings or biased attention weights based on distance BIBREF9 . In this work we present an efficient way of incorporating relative position representations in the self-attention mechanism of the Transformer. Even when entirely replacing its absolute position encodings, we demonstrate significant improvements in translation quality on two machine translation tasks. Our approach can be cast as a special case of extending the self-attention mechanism of the Transformer to considering arbitrary relations between any two elements of the input, a direction we plan to explore in future work on modeling labeled, directed graphs.

Question: Can the new position representation be generalized to other tasks?
Answer: Not sure