CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings

1 Introduction . Transformers have been shown to be highly effective on problems involving sequential modeling , such as machine translation ( MT ) [ 48 ] and natural language processing ( NLP ) [ 14 , 41 , 6 ] . Following its success on these tasks , the Transformer architecture raised immediate interest in other domains : automatic speech recognition ( ASR ) [ 15 , 23 ] , music generation [ 26 ] , object detection [ 7 ] , and finally image recognition [ 16 , 47 ] and video understanding [ 5 ] . Two major components of the Transformer are the attention mechanism [ 2 , 48 ] and the positional encoding [ 48 , 44 , 26 , 11 ] . Without the latter , vanilla attention Transformers are invariant with respect to input tokens permutations ( making “ cat eats fish ” and “ fish eats cat ” identical to the model ) . In the original Transformer publication , sinusoidal positional encoding was introduced [ 48 ] . Token positions were encoded in an absolute manner , which was sufficient to achieve state-of-the-art performance in numerous tasks . However , performance issues were later observed when dealing with sequences of length not seen at training time [ 26 , 11 , 59 , 33 , 42 , 18 ] . For most applications relative positions between tokens are more relevant than absolute ones . A number of approaches were thus proposed to encode relative positions in an explicit form [ 44 , 11 , 26 , 42 , 27 ] , leading to improvements in modeling long sequences . However , all these approaches focus on modifying the attention mechanism and suffer from additional computational and memory costs [ 57 ] . Relative positional encoding is also ⇤Currently at Apple . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . notably hard to implement efficiently for multidimensional case , and recent advances in Transformer models for computer vision [ 7 , 16 , 47 ] still rely on learnable absolute positional encoding . Instead of changing the Transformer attention mechanism , we propose to improve absolute sinusoidal positional encodings in two ways : a ) instead of discrete positions , rely on continuous ones , which better match the continuous nature of image , sound or video data ; b ) preserve some information about relative token positions , via a specific augmentation approach for positional embeddings during training . We empirically evaluate our approach , dubbed continuous augmented positional embedding ( CAPE ) , with recent state-of-the-art models in several application domains . We study generalization properties and introduce unique features unlocked by CAPE . The main contributions of this work are : • new augmented continuous positional embedding ( CAPE ) , which encodes some relative position information in a computationally efficient way , and improves generalization performance compared to other positional embeddings across a variety of domains : machine translation , image and speech recognition ; • a single vision Transformer ( UniViT ) trained with CAPE on the mix of different resolutions : it outperforms each single-resolution baseline , generalizes better to unseen resolutions and can naturally process images of any size ; • new CAPE-based adaptive training scheme for ASR that eliminates need for padding . 2 Related Works . Since the appearance of Transformers , many works have investigated ways to encode positional information . A detailed analysis of various positional embeddings is available for BERT architecture [ 49 ] , where authors empirically relate properties of positional embeddings to performance on downstream NLP tasks . A recent study [ 28 ] ( also focuses on BERT ) highlights the negative impact of spurious correlations between word and positional embeddings , and proposes to explicitly disentangle the contribution of positional and content embeddings in the attention mechanism . In contrast , our approach implicitly enforces this disentanglement by leveraging augmentation . Systematic studies of positional embeddings in audio domain are scarce . Several ways to encode relative positions for Transformer-based speech recognition are compared in [ 51 ] . Experiments show that absolute sinusoidal positional embeddings work no better than stacking consecutive frames at each time position ( a particular form of convolution ) . We provide a more thorough evaluation of positional embeddings in ASR , over multiple datasets . We also show that embeddings obtained from a one-layer convolutional frontend benefits from adding positional information . Transformers for computer vision applications are still in their early days , and most works rely on learnable absolute positional embeddings only [ 16 , 47 , 5 , 1 ] . Several recent works complement the Transformer architecture with convolutional layers to induce a spacial relationship between tokens [ 21 , 9 , 53 ] . As we discuss later , this restricts flexibility compared to pure attention-based models . The work [ 21 ] suggests injecting learnable attention biases as an alternative mechanism to positional encoding . Evaluation of several positional encodings and their corresponding generalization has been done in a study [ 9 ] which is in line with our work . Convolutional elements were introduced in the Transformer architecture , leading to better generalization properties . In contrast , our experiments demonstrate that generalization can be achieved without architecture modification or convolutional inductive bias . Concerning video understanding , an evaluation of the impact of positional encoding was done in [ 34 ] : according to the results , positional encoding-free architectures perform best . Other work [ 5 ] reports that adding absolute positional embeddings improves models performance , but contribution of encoding space and time vary between datasets . As a summary , many positional embeddings variants were previously introduced , often modalityspecific . In our cross-modal study we focus on generalization properties of popular and widely used embeddings , and improve on absolute sinusoidal positional embedding , leading to a flexible Transformer architecture , with great generalization properties across a number of different tasks . The closest idea to our work is augmentation of positions in Universal Transformer [ 12 ] where discrete global shifts are applied to synthetic tasks for encoder-decoder Transformer models . In our work , we introduce continuous augmentations , not discrete , which are more natural for continuous modalities like images and speech where CAPE benefits most and which are not discussed in [ 12 ] . In addition to global shifts our augmentations include global scaling and local shifts , and we also synchronize augmentations between encoder and decoder ( Section 5.3 ) . 3 Theoretical Analysis of Absolute Sinusoidal Positional Embeddings . Originally absolute positional K-dimensional embeddings were introduced in [ 48 ] as E2k ( n ) = cos ! kn E2k+1 ( n ) = sin ! kn ! k = 10000 2k/K n 2 Z+ ( 1 ) with k = 1 , 2 , . . . , K/2 enumerating components for a token at position n. For simplicity of analysis , we rewrite Eq . ( 1 ) as a complex-valued embeddings2 with half the number of components : { E ( n ) } k = Ek ( n ) = ei ! kn This definition can be rewritten in a recursive manner , by introducing a unitary operator S : E ( n+ 1 ) = SE ( n ) , with { SX } k = Xkei ! k ( 2 ) Therefore , the embedding at position n contains sufficient information to compute the embedding of the next or previous positions , as applying Sm ( m 2 Z ) performs a relative shift : E ( n + m ) = Sm E ( n ) . Variation in ! i ensures that no positions < 104 are assigned similar embeddings . Before introducing augmentations of positional embeddings , we revisit positions parametrizations for different modalities . Positional encoding for text For natural language processing it is common to split text into words , letters , syllables and other sub-units . Original absolute sinusoidal positional embeddings enumerate these sub-units by their ordinal number n , a common choice that we follow . Positional encoding for images In a framework where patch and image sizes may vary , we find that enumerating patches is not appropriate , as positional embeddings may greatly differ for different scales of an image , leading to generalization issues . In that perspective , we consider scaled coordinates x and y that span interval [ 1 , +1 ] .3 While previous works [ 16 , 47 ] relied on learnable absolute positional embedding , we introduce the following K-dimensional absolute sinusoidal positional embedding defined for each position ( x , y ) 2 R2 : E2k ( x , y ) = cos⇡ ( wk , xx+ wk , yy ) E2k+1 ( x , y ) = sin⇡ ( wk , xx+ wk , yy ) ( 3 ) wk , x = 10 2k/K cos k wk , y = 10 2k/K sin k ( 4 ) Following Eq . ( 2 ) , this corresponds to introducing two commuting unitary operators Sx and Sy , for each unit shift in either direction on the plane . The choice of wk , x and wk , y is kept simple and deterministic , while giving no preference to any direction on the plane and providing angle of “ hatching ” ( all components have different angle and angles uniformly cover possible directions ) , see Figure 9 . Furthermore , embeddings defined by Eq . ( 3-4 ) allow attending to a specific small region around a point on the image without emphasizing points with only same x or only same y while we expect this artifact to contribute in case of concatenation of 1D embeddings for each axis [ 9 , 16 , 3 , 38 ] . Positional encoding for sound We propose to tie positional embeddings to timestamps in seconds . The embedding for a frame centered at t seconds is given by : E2k ( t ) = cos ! kt E2k+1 ( t ) = sin ! kt ! k = 30⇥ 10000 2k/K The choice of ! k corresponds to the scaled version of Eq . ( 1 ) and ensures that queries with 30ms specificity are possible even with minutes-long audio fragments . Two main factors contribute to the choice of base in the exponentiation of wk and wk , x , wk , y frequencies : i ) how precise the location should be for encoding ( e.g . there is no need to encode position in audio much more precisely than one syllable takes ) ; ii ) longest reasonable “ length ” that is expected in the data . CAPE could be parametrized by precision and “ length ” parameters , but we believe that for all practical cases provided choices are reasonable . 2Work [ 50 ] introduces general complex-valued embeddings as continuous word functions over a position . 3Authors of [ 7 ] also perform coordinates scaling but it spans interval [ 0 , 1 ] . 4 Continuous Augmented Positional Embeddings ( CAPE ) . Absolute sinusoidal positional embeddings lack regularization , leading to “ in-domain ” generalization issues as the model may start learning spurious correlations . For applications where inference input sizes may be significantly different than the training ones , “ out-of-domain ” issues may also arise , as positions rarely observed at training may lead to improper inference predictions . To ensure that the model does not learn spurious correlations between content and position , we introduce the following augmentations for absolute sinusoidal positional embeddings at training time , see Figure 1 . Global shift Transform every embedding in a sequence using S operator with a global random shift from uniform zero-mean distribution ⇠ U ( max , max ) : E 0 ( n ) = S E ( n ) { S X } k = Xkei ! k 2 R This modification hides the absolute positional information , but relative relations , see Eq . ( 2 ) , between embeddings still hold . This transformation can be rewritten as augmenting positions by a random shift before encoding with sin and cos : n0i ni + x0i xi + x , y0i yi + y t0i ti + ( 5 ) Local shift To further increase augmentation and prevent capturing spontaneous correlations , we additionally introduce local shifts from uniform zero-mean distribution ✏i ⇠ U ( ✏max , ✏max ) n0i ni + ✏i x0i xi + ✏x , i , y0i yi + ✏y , i t0i ti + ✏i ( 6 ) Global scaling To prevent distances memorization , we also introduce random global scale from log ⇠ U ( log max , log max ) n0i ni x0i xi , y0i yi t0i ti ( 7 ) At training time , computing our continuous augmented positional embedding is performed through four steps : i ) mean-normalization of positions ( extract mean of sequence positions ) , ii ) global shift Eq . ( 5 ) , iii ) local shift Eq . ( 6 ) , and iv ) global scaling Eq . ( 7 ) . At inference time , only meannormalization of positions is performed . The reference implementation can be found in Appendix A .

Summary: This paper proposes an augmentation-based approach (CAPE) for absolute positional embeddings. The empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance. The paper is cleary-written and the experiments are comprehensive.