CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings

1 Introduction . Transformers have been shown to be highly effective on problems involving sequential modeling , such as machine translation ( MT ) [ 48 ] and natural language processing ( NLP ) [ 14 , 41 , 6 ] . Following its success on these tasks , the Transformer architecture raised immediate interest in other domains : automatic speech recognition ( ASR ) [ 15 , 23 ] , music generation [ 26 ] , object detection [ 7 ] , and finally image recognition [ 16 , 47 ] and video understanding [ 5 ] . Two major components of the Transformer are the attention mechanism [ 2 , 48 ] and the positional encoding [ 48 , 44 , 26 , 11 ] . Without the latter , vanilla attention Transformers are invariant with respect to input tokens permutations ( making “ cat eats fish ” and “ fish eats cat ” identical to the model ) . In the original Transformer publication , sinusoidal positional encoding was introduced [ 48 ] . Token positions were encoded in an absolute manner , which was sufficient to achieve state-of-the-art performance in numerous tasks . However , performance issues were later observed when dealing with sequences of length not seen at training time [ 26 , 11 , 59 , 33 , 42 , 18 ] . For most applications relative positions between tokens are more relevant than absolute ones . A number of approaches were thus proposed to encode relative positions in an explicit form [ 44 , 11 , 26 , 42 , 27 ] , leading to improvements in modeling long sequences . However , all these approaches focus on modifying the attention mechanism and suffer from additional computational and memory costs [ 57 ] . Relative positional encoding is also ⇤Currently at Apple . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . notably hard to implement efficiently for multidimensional case , and recent advances in Transformer models for computer vision [ 7 , 16 , 47 ] still rely on learnable absolute positional encoding . Instead of changing the Transformer attention mechanism , we propose to improve absolute sinusoidal positional encodings in two ways : a ) instead of discrete positions , rely on continuous ones , which better match the continuous nature of image , sound or video data ; b ) preserve some information about relative token positions , via a specific augmentation approach for positional embeddings during training . We empirically evaluate our approach , dubbed continuous augmented positional embedding ( CAPE ) , with recent state-of-the-art models in several application domains . We study generalization properties and introduce unique features unlocked by CAPE . The main contributions of this work are : • new augmented continuous positional embedding ( CAPE ) , which encodes some relative position information in a computationally efficient way , and improves generalization performance compared to other positional embeddings across a variety of domains : machine translation , image and speech recognition ; • a single vision Transformer ( UniViT ) trained with CAPE on the mix of different resolutions : it outperforms each single-resolution baseline , generalizes better to unseen resolutions and can naturally process images of any size ; • new CAPE-based adaptive training scheme for ASR that eliminates need for padding . 2 Related Works . Since the appearance of Transformers , many works have investigated ways to encode positional information . A detailed analysis of various positional embeddings is available for BERT architecture [ 49 ] , where authors empirically relate properties of positional embeddings to performance on downstream NLP tasks . A recent study [ 28 ] ( also focuses on BERT ) highlights the negative impact of spurious correlations between word and positional embeddings , and proposes to explicitly disentangle the contribution of positional and content embeddings in the attention mechanism . In contrast , our approach implicitly enforces this disentanglement by leveraging augmentation . Systematic studies of positional embeddings in audio domain are scarce . Several ways to encode relative positions for Transformer-based speech recognition are compared in [ 51 ] . Experiments show that absolute sinusoidal positional embeddings work no better than stacking consecutive frames at each time position ( a particular form of convolution ) . We provide a more thorough evaluation of positional embeddings in ASR , over multiple datasets . We also show that embeddings obtained from a one-layer convolutional frontend benefits from adding positional information . Transformers for computer vision applications are still in their early days , and most works rely on learnable absolute positional embeddings only [ 16 , 47 , 5 , 1 ] . Several recent works complement the Transformer architecture with convolutional layers to induce a spacial relationship between tokens [ 21 , 9 , 53 ] . As we discuss later , this restricts flexibility compared to pure attention-based models . The work [ 21 ] suggests injecting learnable attention biases as an alternative mechanism to positional encoding . Evaluation of several positional encodings and their corresponding generalization has been done in a study [ 9 ] which is in line with our work . Convolutional elements were introduced in the Transformer architecture , leading to better generalization properties . In contrast , our experiments demonstrate that generalization can be achieved without architecture modification or convolutional inductive bias . Concerning video understanding , an evaluation of the impact of positional encoding was done in [ 34 ] : according to the results , positional encoding-free architectures perform best . Other work [ 5 ] reports that adding absolute positional embeddings improves models performance , but contribution of encoding space and time vary between datasets . As a summary , many positional embeddings variants were previously introduced , often modalityspecific . In our cross-modal study we focus on generalization properties of popular and widely used embeddings , and improve on absolute sinusoidal positional embedding , leading to a flexible Transformer architecture , with great generalization properties across a number of different tasks . The closest idea to our work is augmentation of positions in Universal Transformer [ 12 ] where discrete global shifts are applied to synthetic tasks for encoder-decoder Transformer models . In our work , we introduce continuous augmentations , not discrete , which are more natural for continuous modalities like images and speech where CAPE benefits most and which are not discussed in [ 12 ] . In addition to global shifts our augmentations include global scaling and local shifts , and we also synchronize augmentations between encoder and decoder ( Section 5.3 ) . 3 Theoretical Analysis of Absolute Sinusoidal Positional Embeddings . Originally absolute positional K-dimensional embeddings were introduced in [ 48 ] as E2k ( n ) = cos ! kn E2k+1 ( n ) = sin ! kn ! k = 10000 2k/K n 2 Z+ ( 1 ) with k = 1 , 2 , . . . , K/2 enumerating components for a token at position n. For simplicity of analysis , we rewrite Eq . ( 1 ) as a complex-valued embeddings2 with half the number of components : { E ( n ) } k = Ek ( n ) = ei ! kn This definition can be rewritten in a recursive manner , by introducing a unitary operator S : E ( n+ 1 ) = SE ( n ) , with { SX } k = Xkei ! k ( 2 ) Therefore , the embedding at position n contains sufficient information to compute the embedding of the next or previous positions , as applying Sm ( m 2 Z ) performs a relative shift : E ( n + m ) = Sm E ( n ) . Variation in ! i ensures that no positions < 104 are assigned similar embeddings . Before introducing augmentations of positional embeddings , we revisit positions parametrizations for different modalities . Positional encoding for text For natural language processing it is common to split text into words , letters , syllables and other sub-units . Original absolute sinusoidal positional embeddings enumerate these sub-units by their ordinal number n , a common choice that we follow . Positional encoding for images In a framework where patch and image sizes may vary , we find that enumerating patches is not appropriate , as positional embeddings may greatly differ for different scales of an image , leading to generalization issues . In that perspective , we consider scaled coordinates x and y that span interval [ 1 , +1 ] .3 While previous works [ 16 , 47 ] relied on learnable absolute positional embedding , we introduce the following K-dimensional absolute sinusoidal positional embedding defined for each position ( x , y ) 2 R2 : E2k ( x , y ) = cos⇡ ( wk , xx+ wk , yy ) E2k+1 ( x , y ) = sin⇡ ( wk , xx+ wk , yy ) ( 3 ) wk , x = 10 2k/K cos k wk , y = 10 2k/K sin k ( 4 ) Following Eq . ( 2 ) , this corresponds to introducing two commuting unitary operators Sx and Sy , for each unit shift in either direction on the plane . The choice of wk , x and wk , y is kept simple and deterministic , while giving no preference to any direction on the plane and providing angle of “ hatching ” ( all components have different angle and angles uniformly cover possible directions ) , see Figure 9 . Furthermore , embeddings defined by Eq . ( 3-4 ) allow attending to a specific small region around a point on the image without emphasizing points with only same x or only same y while we expect this artifact to contribute in case of concatenation of 1D embeddings for each axis [ 9 , 16 , 3 , 38 ] . Positional encoding for sound We propose to tie positional embeddings to timestamps in seconds . The embedding for a frame centered at t seconds is given by : E2k ( t ) = cos ! kt E2k+1 ( t ) = sin ! kt ! k = 30⇥ 10000 2k/K The choice of ! k corresponds to the scaled version of Eq . ( 1 ) and ensures that queries with 30ms specificity are possible even with minutes-long audio fragments . Two main factors contribute to the choice of base in the exponentiation of wk and wk , x , wk , y frequencies : i ) how precise the location should be for encoding ( e.g . there is no need to encode position in audio much more precisely than one syllable takes ) ; ii ) longest reasonable “ length ” that is expected in the data . CAPE could be parametrized by precision and “ length ” parameters , but we believe that for all practical cases provided choices are reasonable . 2Work [ 50 ] introduces general complex-valued embeddings as continuous word functions over a position . 3Authors of [ 7 ] also perform coordinates scaling but it spans interval [ 0 , 1 ] . 4 Continuous Augmented Positional Embeddings ( CAPE ) . Absolute sinusoidal positional embeddings lack regularization , leading to “ in-domain ” generalization issues as the model may start learning spurious correlations . For applications where inference input sizes may be significantly different than the training ones , “ out-of-domain ” issues may also arise , as positions rarely observed at training may lead to improper inference predictions . To ensure that the model does not learn spurious correlations between content and position , we introduce the following augmentations for absolute sinusoidal positional embeddings at training time , see Figure 1 . Global shift Transform every embedding in a sequence using S operator with a global random shift from uniform zero-mean distribution ⇠ U ( max , max ) : E 0 ( n ) = S E ( n ) { S X } k = Xkei ! k 2 R This modification hides the absolute positional information , but relative relations , see Eq . ( 2 ) , between embeddings still hold . This transformation can be rewritten as augmenting positions by a random shift before encoding with sin and cos : n0i ni + x0i xi + x , y0i yi + y t0i ti + ( 5 ) Local shift To further increase augmentation and prevent capturing spontaneous correlations , we additionally introduce local shifts from uniform zero-mean distribution ✏i ⇠ U ( ✏max , ✏max ) n0i ni + ✏i x0i xi + ✏x , i , y0i yi + ✏y , i t0i ti + ✏i ( 6 ) Global scaling To prevent distances memorization , we also introduce random global scale from log ⇠ U ( log max , log max ) n0i ni x0i xi , y0i yi t0i ti ( 7 ) At training time , computing our continuous augmented positional embedding is performed through four steps : i ) mean-normalization of positions ( extract mean of sequence positions ) , ii ) global shift Eq . ( 5 ) , iii ) local shift Eq . ( 6 ) , and iv ) global scaling Eq . ( 7 ) . At inference time , only meannormalization of positions is performed . The reference implementation can be found in Appendix A .

Summary: This paper proposes “CAPE” (continuous augmented positional embeddings), a novel form of positional embeddings, aimed at making transformer models more robust to changes in the input distribution at inference time (e.g., changes in input size/length/resolution, changes in relative distances/positions between subunits of input, and changes in absolute positions of input subunits).  It accomplishes this by (1) indexing positional embeddings in a continuous way (e.g., for images: indexing by the normalized coordinate $(x,y)  \in [-1,+1]^2$ of a patch; for audio/speech: indexing by the time (in seconds) of a frame), and (2) performing a series of augmentations (global shift, local shift, global scaling) to these “continuous indexes” at each position in the input (these augmentations can be thought of as being similar to performing augmentations on the input).  The paper presents experimental results across image classification (vision), ASR (speech), and machine translation (NLP) tasks, showing that in general, the proposed CAPE embeddings outperform baseline positional embedding methods. In particular, CAPE is compared with (1) _nopos_: no positional embeddings, (2) _abspos_: learnable absolute positional embeddings, (3) _sinpos_: continuous sinusoidal positional embeddings (without the proposed augmentations), and (4) _relpos_: learnable relative positional embeddings (for ASR tasks).   **Detailed summary of experimental results, by domain**  _Image Classification_: The experiments demonstrate that ViT models trained with CAPE embeddings perform better than the baseline embedding methods across test images of different resolutions. Furthermore, the paper proposes “UniViT”, a ViT model trained with CAPE embeddings and randomly resized inputs (128x128 -> 320x320), and shows that UniViT performs extremely well across test image resolutions (160x160 -> 672x672) relative to ViT models trained only on a single resolution.  Because the UniViT model gracefully handles images of different resolutions, this model also provides a way for practitioners to easily trade-off increased throughput for lower accuracy, by lowering the resolution of the input images fed to the model at inference time.  _ASR_: The paper shows that CAPE generally outperforms the other absolute positional embeddings (nopos, sinpos, abspos), while generally being competitive with, or outperforming, the more computationally expensive relative positional embeddings (relpos). It additionally shows that (1) CAPE performs similarly to using SpecAugment as a form of data augmentation during training (and combining CAPE + SpecAugment does best), and (2) CAPE can be used, together with adjusting the STFT hop distance of utterances in a batch, to eliminate the need for padding during ASR training (without degrading performance).  _Machine Translation_:  The paper again demonstrates strong performance for CAPE.