CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
Authors: Tatiana Likhomanenko
Qiantong Xu
Gabriel Synnaeve
Ronan Collobert
Alex Rogozhnikov
Venue: NeurIPS 2021 Poster (2021)
Keywords: transformers
positional encoding
positional embedding
augmentation
image recognition
speech recognition
machine translation

Abstract: Without positional information, attention-based Transformer neural networks are permutation-invariant. Absolute or relative positional embeddings are the most popular ways to feed Transformer models with positional information. Absolute positional embeddings are simple to implement, but suffer from generalization issues when evaluating on sequences longer than seen at training time. Relative positions are more robust to input length change, but are more complex to implement and yield inferior model throughput due to extra computational and memory costs. In this paper, we propose an augmentation-based approach (CAPE) for absolute positional embeddings, which keeps the advantages of both absolute (simplicity and speed) and relative positional embeddings (better generalization). In addition, our empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance as well as increased stability with respect to training hyper-parameters.

CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
Tatiana Likhomanenko" Facebook Al Research
Qiantong Xu Facebook AI Research qiantong@fb.com
Gabriel Synnaeve
Facebook Al Research
tata wnw.rasagaaealllcccmmmm com
gabāfb.com com
Ronan Collobert Facebook AI Research
Alex Rogozhnikov Herophilus, Inc.
locronan@fb.com com
a alex. ccx..rogozhnkkooaaeedeee.rr
Abstract
Without positional information, attention-based Transformer neural networks are intiuutationiivvriinn Absolute or relative positional embeddings are the most popular ways feed Transformer models with positional information. Absolute positional embeddings are simple to implement, but suffer from generalization issues when evaluating sequences longer than seen at training time. Relative po- sitions are more robust to input length change, but are more complex to implement and yield inferior model throughput due to extra computational and memory costs. In this paper, we propose an augmentation baaeentttion-bbsed approach (CAPE) for absolute positional embeddings, which keeps the advantages of both absolute (simplicity and speed) and relative positional embeddings (better generalization). In addition, empirical evaluation state-of-the-ard models in machine translation, im- age and speech recognition demonstrates that CAPE leads to better generalization performance as well increased stability with respect training hyper-parameters.
Introduction
Transformers have been shown to be highly effective problems involving sequential modeling, such as machine translation (MT) and natural language processing (NLP) 14 6 Following
its success these tasks, the Transformer architecture raised immediate interest in other domains: automatic speech recognition (ASR) 15, 23), music generation (26), object detection and finally
image recognition
[16, (7] and video understanding (5].
Two major components the Transformer are the attention mechanism
and the positional II]. Without the latter, vanilla attention Transformers are invariant with respect input tokens permutations (making "cat eats fish" and "fish eats cat" identical the model). the original Transformer publication, sinusoidal positional encoding was introduced [8]. Token positions were encoded an absolute manner, which was sufficient achieve state-of-the-arr performance in numerous tasks. However, performance issues were later observed when dealing with sequences of length not seen at training time (26] I1 between tokens are more relevant than absolute ones. A number of approaches were thus proposed to encode relative positions an explicit form 1 26 42] 27], leading to improvements modeling long sequences. However, all these approaches focus modifying the attention mechanism and suffer from additional computational and memory costs [57]. Relative positional encoding is also
encoding [88 44
26
59 33
42, 18). For most applications relative positions
Currently Apple.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

notably hard to implement efficiently for multidimensional case, and recent advances in Transformer models for computer vision T6 47 still rely on learnable absolute positional encoding.
Instead of changing the Transformer attention mechanism, we propose to improve absolute sinusoidal positional encodings in two ways: a) instead of discrete positions, rely on continuous ones, which better match the continuous nature of image, sound or video data; b) preserve some information about relative token positions, via specific augmentation approach for positional embeddings during training. We empirically evaluate our approach, dubbed continuous augmented positional embedding (CAPE), with recent state-of- the-art models in several application domains. We study generalization properties and introduce unique features unlocked by CAPE. The main contributions of this work are:
new augmented continuous positional embedding (CAPE), which encodes some relative position information in a computationally efficient way, and improves generalization per- formance compared to other positional embeddings across a variety of domains: machine translation, image and speech recognition:
a single vision Transformer (UniViT) trained with CAPE on the mix of different resolutions: it outperforms each single-resolution baseline, generalizes better to unseen resolutions and can naturally process images of any size;
new CAPE-based adaptive training scheme for ASR that eliminates need for padding.
2
Related Works
Since the appearance of Transformers, many works have investigated ways to encode positional information. A detailed analysis of various positional embeddings is available for BERT architec- ture [49]. where authors empirically relate properties of positional embeddings to performance on downstream NLP tasks. recent study (X] (also focuses on BERT) highlights the negative impact of spurious correlations between word and positional embeddings, and proposes to explicitly disentangle the contribution of positional and content embeddings in the attention mechanism. In contrast, our approach implicitly enforces this disentanglement by leveraging augmentation.
Systematic studies of positional embeddings in audio domain scarce. Several ways to encode relative positions for Transformer-based speech recognition are compared in [5T]. Experiments show that absolute sinusoidal positional embeddings work better than stacking consecutive frames at each time position (a particular form of convolution). We provide a more thorough evaluation of positional embeddings in ASR, over multiple datasets. We also show that embeddings obtained from a one-layer convolutional frontend benefits from adding positional information.
Transformers for computer vision applications are still in their early days, and most works rely on learnable absolute positional embeddings only [66
I. Several recent works complement the Transformer architecture with convolutional layers to induce spacial relationship between tokens [21 9 [53]. As we discuss later, this restricts flexibility compared to pure uttention-based models. The work (21) suggests injecting learnable attention biases as alternative mechanism to positional encoding. Evaluation of several positional encodings and their corresponding generalization has been done in study [9] which is in line with our work. Convolutional elements were introduced in the Transformer architecture, leading to better generalization properties. In contrast, our experiments demonstrate that generalization can be achieved without architecture modification convolutional inductive bias. Concerning video understanding, an evaluation of the impact of positional encoding was done in 34 according to the results, positional meoding-ffee architectures perform best. Other work 5 reports that adding absolute positional embeddings improves models performance, but contribution of encoding space and time vary between datasets.
As a summary, many positional embeddings variants were previously introduced, often modality- specific. In our cross-modal study we focus generalization properties of popular and widely used embeddings, and improve on absolute sinusoidal positional embedding, leading to a flexible Transformer architecture, with great generalization properties across a number of different tasks.
The closest idea to our work is augmentation of positions in Universal Transformer [22 where discrete global shifts are applied to synthetic tasks for encoder-decoder Transformer models. In our work, we introduce continuous augmentations, not discrete, which are more natural for continuous modalities like images and speech where CAPE benefits most and which are not discussed in [2].

---

In addition to global shifts our augmentations include global scaling and local shifts, and we also synchronize augmentations between encoder and decoder (Section
3 Theoretical Analysis of Absolute Sinusoidal Positional Embeddings
Originally absolute positional K K-ddimnnsional embeddings were introduced in [88]
as
Ez (n) COS wa
E2x+1 1 (n) sin 47,77
Wi 10000 2
71 E
Z
(1)
with - 1,2.... K/2 enumerating components for a token at position n. For simplicity of analysis, rewrite Eq. 1 as complex-valued embedding with half the number of components:
{B(n)}+ (n) dwar
This definition can be rewritten in recursive manner, by introducing a unitary operator E(n 1) Enn), with {SX)* Xge
Therefore, the embedding at position n contains sufficient information to compute the embed- ding of the next previous positions, applying S (m Z) performs relative shift: E(n + m) = S E(n). Variation in W/ ensures that no positions < 10a are assigned similar embeddings. Before introducing augmentations of positional embeddings, we revisit positions parametrizations for different modalities.
Positional encoding for text For natural language processing it is common to split text into words, letters, syllables and other sub-units. Original absolute sinusoidal positional embeddings enumerate these sub-units by their ordinal number n, common choice that follow.
Positional encoding for images In a framework where patch and image sizes may vary. we find that enumerating patches is not appropriate, as positional embeddings may greatly differ for different scales of an image, leading to generalization issues. that perspective, we consider scaled coordinates and y that span interval -1,+1] While previous works [16 47] relied on learnable absolute positional embedding, we introduce the following K K-dimensional absolute sinusoidal positional embedding defined for each position (I,z) E R²:
E22 v) = cos T [ + wk.yl)
E2 y) = sin π T + wk,y
(3)
Wk. 20 cos Why = 102 sin k
(4)
Following Eq. ( this corresponds to introducing two commuting unitary operators S₂ and Sy. for each unit shift in either direction on the plane. The choice of Wk,r and wk. is kept simple and deterministic, while giving no preference to any direction on the plane and providing angle of "hatching" (all components have different angle and angles uniformly cover possible directions), see Figure [ Furthermore, embeddings defined by Eq.
allow attending to specific small region around point on the image without emphasizing points with only same x or only same while we expect this artifact to contribute in case of concatenation of iD embeddings for each axis [9 16.1 38).
Positional encoding for sound We propose tie positional embeddings timestamps in seconds. The embedding for frame centered at t seconds is given by:
E (t) cos wyt
(2)+1(t) sin wht Wk = 30 10000 10000-26/4 2k/K
The choice of we corresponds to the scaled version of Eq. ( and ensures that queries with 30ms specificity are possible even with minutes-long audio fragments.
Two main factors contribute to the choice of base in the exponentiation of wi and Wk,rr Wkyy frequencies: i) how precise the location should be for encoding (e.g. there is need to encode position in audio much more precisely than one syllable takes); ii) longest reasonable "length" that expected in the data. CAPE could be parametrized by precision and "length" parameters, but believe that for all practical cases provided choices are reasonable.
2 Work [50) introduces general complex-valued embeddings as continuous word functions over position.
Authorr of [ also perform coordinates scaling but spans interval [0, 1].
3

---

global
local
ahitt
scaling
0
0
Figure 1: Example of CAPE's transformations for an image: patches positions are scaled to [-1, 1] x
[-1, 1]; random global shift, local shift, and global scaling are then applied to the grid of positions.
Continuous Augmented Positional Embeddings (CAPE)
Absolute sinusoidal positional embeddings lack regularization, leading to "in-domain" generalization issues as the model may start learning spurious correlations. For applications where inference input sizes may be significantly different than the training ones, "out-of-domain" issues may also arise, as positions rarely observed at training may lead improper inference predictions. To ensure that the model does not learn spurious correlations between content and position, we introduce the following augmentations for absolute sinusoidal positional embeddings at training time, see Figure !
Global shift Transform every embedding in sequence using S operator with a global random shift from uniform zero-mean distribution A ~ (U(-Amaz: Amaz):
E (n) = E())
{s X}= A
A AER
This modification hides the absolute positional information, but relative relations, see Eq. between embeddings still hold. This transformation can be rewritten as augmenting positions by a random shift before encoding with sin and cos:
n + n. +
1 + 24
V
+ yi
Ay
4 4 4
(5)
Local shift additionally introduce local shifts from uniform zero-mean distribution G ~ u1( ((-Cmar: Cmax)
To further increase augmentation and prevent capturing spontaneous correlations, we
n4 ni + Gi
* + 2i Cr.i, V
Cy.i
& + 1
(6)
To prevent distances memorization, also introduce random global scale À from log À ~ U(- log Amar, log Amaz)
Global scaling
n4
An4
Azis
3/4
Ay
At
At training time, computing our continuous augmented positional embedding is performed through four steps: i) mean- moaannorrmaizztioon of positions (extract mean of sequence positions), ii) global shift Eq. ( iii) local shift Eq. (6), and iv) global scaling Eq. (7). At inference time, only mean- normalization of positions is performed. The reference implementation can be found in Appendix A
Experiments
5.1 Image Recognition
We evaluate CAPE embedding empirically with recently proposed Vision Transformer (ViT) [6 47 for image recognition. These works rely on learnable absolute positional embedding (abspos) for both class token and patches, and train ViT models on 224" image with 162 patches. To further improve model quality, [47) performs fine-tuning images of higher resolution 3842, The grid of positional embeddings is then upsampled.
In the following, denote the size and resolution N N as 2

---

103x200
2244222
3344204
6724672
3000102
224/224
6724872
Imageeee Top- 1.60
22
50g05
absoos
CAPE
nOp05
Smppss
30000
301401
SPAR1
186118
abs2014
CAPE
Imageher Top (95)
715
Figure 2: Top-1 accuracy ImageNet and ImageNet-v2 for VIT models trained with different positional embeddings on 224 resolution (solid) and further fine-tuned on 3842 (dashed, "+ft"". Insets focus higher accuracies. The full list of top-1 and top-5 accuracies can be found in Appendix B.1
Tables and
Data and VIT Models All experiments are performed on the ImageNet [13 43 dataset. We report top-1 and top-5 accuracies ImageNet validation set and ImageNet-v2- (a,b.c) 40 test sets. The same convolution-free architecture, identical to the one proposed by [16 (ViT-B) and used by [47] (referred DeiT-B), is chosen for all experiments. A ViT-B/DeiT-B baseline is trained with abspos on 224 images, carefully following Section 6 from (47). The exact same training configuration used for models with other positional embeddings: only positional embedding is changed. We evaluate both proposed absolute sinusoidal positional embedding (sinpos). Eq. 3 and CAPE (Amez 0.5, Emaz = 1/N and Amaz 1.4). As control experiment we also train model without any positional embedding (nopos), which can be interpreted as a 'bag of werds patches', as no patch position information is available. We also train models with different positional embeddings on either 1602 or 3842 images. The whole training configuration remains the same as for training on 2242 images, except for the positional embedding. All models trained on 224 images we additionally fine-tune on images of higher resolution 384 following 47)
Evaluation To study generalization when image sizes vary, we evaluate all models on different resolutions (1602, 224 384" and 672²) by resizing all images in validation and test sets. When evaluating on resolutions different from the training one, bicubic interpolation is applied to abspos embedding: as was justified in [47]. In contrast, sinpos and CAPE approaches can ingest any image resolution, thanks to the continuous nature of their positional embeddings.
5.1.1 Results
In Figure 2 we compare generalization performance of models trained with different positional embeddings on 224 images (solid). Both proposed sinpos and CAPE approaches perform least as well, if not better, than the abspos approach on the same-as-train resolution. When performing inference on resolutions different than the training one, CAPE performs best, notably outperforming abspos on high (6722) and low (1602) resolutions to 25% and 2%, respectively. On 160 and 384 resolutions CAPE trained on 224² resolution performs similar abspos trained on corresponding 160 or 384² inference resolutions (the latter results being reported in Figure 3). This confirms good generalization properties CAPE on image resolutions unseen at training time.
Abspos fine-tuned on a higher resolution (3842) improves in accuracy for both 3842 and 6722 resolutions, while degrading on lower ones (original 224 and lowest 1603), as shown in Figure
'Interpolation is not applied to the class token embedding.

---

ViT Inapeeen Tel- 1.00
05
160x160
305005 384x364
CAPE 160x160
CAPE 224x224
UnIVIT. CAPE
180x280
224424
Teb3
20-
180x362
224-224
SHA.E
672x672
160:160
22Rx224
386c164
6722672
Figure 3: Top-1 accuracy IetgeNet and ImageNet-v2 for ViT models with either abspos CAPE trained on each particular resolution and UniViT model trained on the mixture of resolutions.
Table Test time augmentation (TTA) results on ImageNet when predictions on resolutions 7², +3)) and (r 32)² are combined. Sin- pos and CAPE are trained on 224 resolution.
713
15.0
UnME-- CAPE
100
Our VIT 224?
U-MB-. CAPE
Top-1 1 (5)
Top-5 (96)
Our VIT.6 221
Our VIT 14P322e/186
-TTA
+TTA
-TTA
+ TTA
157
sinpos CAPE
imagis
224
81.01
81,34
95.18
95.49
Figure 4: Accuracy with respect to throughput ImageNet at inference time under variation of image resolution. "-S" and "-Ti" refer to small and tiny architectures []6]. respectively.
UniVIT sinpos
224
80.82
81,34
95.40
95.57
UniVIT CAPE
224
81.26
81.64
95.56
95.71
UniViT, sinpos
384
82.31
82,44
96.04
96.14
VIT, CAPE
384
82.55
82.72
96.18
96.22
(dashed). Sinpos and CAPE fine-tuned on 3842 resolution outperform abspos by 0.3-0.4%, thanks
to better fine-tuning starting point after resolution change. that setting, sinpos and CAPE keep better generalization performance nearby resolutions (2242 and 672 While being seriously impacted on the 160 resolution, CAPE still outperforms others by 12-25%. Comparison of VIT models trained with either abspos CAPE on each particular resolution is shown in Figure 3 CAPE outperforms abspos on specific training resolution, and generalizes better to unseen resolutions.
No Positional Embedding With nopos model, confirm [16]'s 's observation that positional em- bedding does not have a critical importance for ImageNet classification (see Figure 2 Nopos model's simplicity and generalization abilities make it nice baseline for positional embedding study in computer vision. It has generalization accuracy similar to abspos on low 1602 and high 672 resolutions, while CAPE outperforms nopos across the board. It is likely that abspos suffers from the embedding interpolation step on extreme resolutions. In contrast, both simpos and CAPE have the advantage naturally support variation in resolutions.
5.1.2 UniViT: Training Universal Transformer on Different Resolutions
As CAPE-based models can handle any image resolution, we propose single universal ViT model, called UniViT, which is trained on images of different resolutions: during training we resize all images in batch to randomly chosen size, uniformly sampled in the range [128, 320 with step of 32. For experiments with UniViT training configuration remains the same as for ViT. Because of training the images of different resolution VIT training time is only 1.1x longer than ViT trained on 224 resolution. In Figure 3 we compare UniViT model trained with CAPE (A = 1) against different ViT models trained each particular resolution: VIT outperforms Linglerreolutee models for any given resolution and, moreover, generalizes to non-training resolutions well.
Image resolution directly impacts throughput: computational complexity of attention is O(N)) for N N image. In Figure 4 we show ViT with CAPE throughput and accuracy with respect to input image resolution. On 962 resolution Vni with CAPE handles throughput and accuracy

---

similar to "tiny" vanilla VīT, while "small" UniViT with CAPE has 4% higher accuracy (with identical throughput) on resolution 160 and 1.4x higher throughput (with identical accuracy) on resolution 1282. Thus, UniViT unlocks the possibility of dynamically adjusting throughput of a model in a production regime under heavy loads, a practical alternative to improving model throughput at inference time via decreasing model size.
We further improve UniViT with CAPE accuracy by resizing each image to its optimal resolution at inference, as shown in Appendix Table We split ImageNet validation images into 8 bins, according to their size. By selecting an optimal resizing strategy in each bin we are able to improve top-1 accuracy to 92% (in comparison, the model has 81 81.26% on 224² and 82.55% on 384*).
5.1.3 Resizing as Test Time Augmentation (TTA)
As both sinpos- and CAPE-basee models handle well different image resolutions, we propose to perform test time resolution augmentation when evaluating a single model. For TTA we average model's 's logits evaluated on three resolutions for the same image: r (r 32)² and (r + 32)², ? where r is either 224 or 384. As shown in [ ViT and UniViT models trained with either sinpos or CAPE emheddings get 0.2%-0.5% top-1 accuracy boost with this test time augmentation.
5.2 Automatic Speech Recognition (ASR)
Recently it was shown that Transformer 48
architectures are state-of-the-art on different public
benchmarks for ASR 55 33 30
8
Data We evaluate our models on several English speech datasets, both on in-domain data and out-of-domain data. We also analyze our models generalization to long sequences. We consider two standard training benchmarks: Wall Street Journal (WSJ) 20 29 52]. read speech with 81 hh.5h of training data, and TED-LIUM v3 (TL) (24). oratory speech with 452h of training data. Besides these datasets we two other sets for evaluation only: i) LibriSpeech (LS) [6 read speech from audiobook recordings (we use only test sets with clean, test-clean, and noisy, test-other, speech); ii) Robust Video (RV), our in-house English video dataset, which is sampled from public social media videos and aggregated and de-identified before transcription; these videos contain a diverse range of speakers, accents, topics, and acoustic conditions making ASR difficult; the test sets composed of clean and noisy subsets. Details on data and its statistics can be found in Appendix C.1
Evaluation To evaluate our acoustic models on sequence lengths not seen training time split all evaluation utterances by their duration T' into the following bins: T < 10s, T E [10- 15)s, T E [15, 20)s and T > 20s. Our performance metric is word error rate (WER) (no language model is involved), reported for each sequence length bin and for the entire evaluation dataset. For RV data a hand- segmentation is available, allowing to evaluate on the exact same data, but segmented in different ways. More precisely, for RV data we prepare 8 sets where audios have the following respective durations: T 10, 15, 20, 25, 30, 35, 40, 45s.
Acoustic Model (AM) Training All models are trained with Connectionist Temporal Classifica- tion [22] SpecAugment [37 is used as data augmentation in training, and the network architecture follows [30|: the AM encoder is composed of ID convolution (kernel 7, stride 3) with a GLU activation and 36 4-heads Transformer layers [88] finally followed by linear layer which outputs score for each target token. Our token set consists of 26 English alphabet letters, augmented with the apostrophe and word boundary token (further details in Appendix C.2)
Positional Embedding As in vision experiments, we evaluate nopos, sinpos, abspos and CAPE- based models. In addition, we evaluate models trained with relative positional embeddings: in that case, no absolute positions are used, and learnable relative positional embeddings [44] (relpos) are trained in each Transformer layer. We follow [30) to train an AM baseline with relpos. For models with other positional embeddings, the training configuration remains identical. Abspos (E(t) ) is set to cover 13.8s of context. At training/evaluatio
time for the longer sequences we define abspos as E(() = E(L mod N) for / > N. This extrapolation at training time leaves a chance to the acoustic model to generalize to unseen (longer) sequence lengths. Relpos spans large context, 26.8s to the left/right. CAPE's global shift covers 60s, while a local shift set to its maximum to preserve the frames order; Amaz = 1.1 and Xmaz 2 for WSJ and TL, respectively.
7

---

(8271 (50)
12314
1991
10:55
19-20
1030
156
WER (60
Vie
8
15
Figure 5: Word error rate for models trained WSJ with different positional embeddings.
11.0
LS-Owet Sani,
repos
krpos
Hopss (12 (51
10.35
1931
106
333355
15-285
1958
196
136
255
26
mIN 180
14
10:358
100
13325
35a
Figure 6: Word error rate for models trained on TED-LIUM v3 with different positional embeddings.
5.2.1 Results
model trained on WSJ with CAPE outperforms other positional embeddings on both public and RV data across different audio durations, shown in Figure A model trained on TL with CAPE outperforms nopos and sinpos on all data, outperforms abspos and relpos for audio longer than 20s, and behaves similarly shorter durations, see Figure On RV data, CAPE-based models perform uniformly well on different audio durations, including long ones. contrast, other embeddings- based models are seriously impacted when audio duration increases. Finally, CAPE does not have computational or parameters overheads compared to relpos.
No Positional Embedding As expected, nopos models (both WSJ and TL ones) perform similarly WER across different audio durations. However, nopos TL model performs surprisingly well: is competitive to positional embeddings-baas outperforms other models, except CAPE when > 20s. perform ablations in Appendix [C.] to show that key ingredients are CTC loss, sufficient model capacity, and large amount of data for this effect to occur.
models on public data. On RV data, nopos TL model
CAPE Augmentation CAPE can be viewed as a data augmentation performed on input data (positions), which regularizes the trained model. We demonstrate this by training on TL with either sinpos or CAPE and with/without SpecAugment (no other augmentations are used), see Figure 7 Baseline sinpos without any augmentation performs worst with a large gap. Including either CAPE or SpecAugment decreases the WER significantly by 4-5%: SpecAugment is more beneficial due to its domain relevance. Combining together CAPE and SpecAugment further decreases the WER by 2.5-3.5% showing that augmentations complementary each other.
5.2.2 Padding-free ASR with CAPE and Variable STFT Hop Distance
In ASR, when batching audios of different duration, one often relies on padding tokens. propose instead to perform time stretching augmentation on all audios in the batch, such that they will have the

---

Table 2: WMT" 14 BLEU score (3 runs avg.).
+CAE -SDXAA)
Model & Lang.
Embedding
Valid BLEU
Tost BLEU
sinpos
26.88=00005
27. 27.66++0,10
61.61..
abspro
27.36++0.06
relpos
26.81+0.16
27.92+0.07
26.86+-0.13
27.89+00.07
DE
CAPE, A = 50
27.09+0003
27.77+0.16
181.
sinpos
27.09=0.06
28.28+0.28
Figure 7: Validation WER for models trained with different augmentations: "baseline" is a model with sinpos, "+CAPE" adds CAPE's global and local shifts, "+SpecAug" adds SpecAugment.
abspos
27.23++0.02
28. 26-+0.22
DE
CAPE,
27.17=0.11
28.44:00.66
sinpos
47.27+00.3
41.13+0.07
6L-6L
abspos
47.22=0003
0,,21=0,04
relpox
47.12220.03
11.33:1.133
CAPE, A =
47,22=0.03
41.59=00.03
FR
CAPE, 50
47.1420.02
41.48:00.10
same number of frames. We perform this augmentation by tuning the short-time Fourier Transform (STFT) hop distance when computing the audio features. Positions remain tied to the original audio timestamps. These models trained either WSJ or TL show better WER across the board, compared to models trained with a padding approach, as shown in Appendix C.3 Figures 15 and 16 and improvements on RV are quite consistent. As before, CAPE-based models outperform sinpos models.
We found this padding-fire free approach convenient, as it alleviates the implementation of special cases to handle padding tokens in ASR models: c.g. any normalization should be aware which tokens padding, otherwise normalization constants would depend on the amount of padding; any attention module should be aware which tokens it should not attend to. Moreover, for efficient computations and reducing padding tokens audio samples are often packed together via sorting by their duration; this reduces variability in the batches drastically. Our results demonstrate that with CAPE and padding- freding--reeeeee approach we can mix samples of not-too-differem lengths within a batch, providing better randomization and utilize all frames. While UniViT adjusts number of tokens by changing resolution, the STFT hop distance achieves the same for audio. By adjusting the hop distance during inference for padding-free ASR, we can achieve higher throughput with similar recognition quality, see Appendix CC33 Figure [7] However, very low resolution (c.g. 128x 128) in images works well enough while speech recognition far more sensitive token sparsification phoneme can last as little as 30-50ms.
5.3 Machine Translation (MT)
Our MT experiments follow the recent results with Transformers combined with new initialization scheme (ADMIN) 33 (2). This approach allows to train very deep state-of-the-art Transformers for MT. We did not implement back-translation or other domain-specific augmentations.
Data and Models Training Experiments are conducted on standard WMT" 14 English-Frende (FR) and English-Gerne German (DE) benchmarks. For both benchmarks follow 31 32 for FR use 40k subword vocabulary, and evaluate on the provided "valid" file for validation and newstest14 for test. On DE, we consider a 32K subword vocabulary. newstest2013 for validation, and newstes12014 for test. We reproduce results from [31 32 by training a sinpos-based model with 6L-6L, 18L-18L for DE and 61. for FR encoder layers with ADMIN. Training configuration stays the same for other positional embedding- based models, other than positional embeddings being either abspos (covers 1k tokens), relpos (learnable (44) covers max train content: 150 left/right tokens) or CAPE in both encoder and decoder layers. For CAPE to have approximate correspondence in positions of source and target sentences, we first scale positions of source language by a factor so that positions interval is loosely matched between source and target # tokens in source corpus
# tokens target corpus
sentences. For each training sample we then apply the same global shift and scale for source and target positions. Local shifts for source/target positions are independently sampled from U ((-0.5, 0.5). As no absolute positions are provided anymore, we prepend source sentences with a "begin of sentence" token to hint the decoder with first position during both training and evaluation.
Evaluation We select the best checkpoint according to BLEU on the validation set, using a beam size 4 for DE and 5 for FR. Following convention, BLEU is computed by multi- bleu. perl via the

---

sinpes
abspus
CAPE, 4=5
Log Lo0s
CAPE, & 50
DF. 6.-61. 1og
3.0
2.5
2.0
153
522
2344
6944
63-82
189
123-42
1691
199
189
209
229
243 262
289-302
2-21
42-61
62-81
82.101
103
1411
142 15
181
002
182
222
38
282
222
Figure 8: Average negative log loss across consecutive positions measured on test sets with stacked
sentences for models with different positional embeddings: 6L-6L DE (left) and 6L-6L FR (right).
standardized tokenization of the mublilly-accessinn dataset. As WMT' 14 test sets have limited length of sentences being as train artificially stack together sentences in test sets (source and target) to have 300+ tokens in each new target sample (3k original pairs of sentences are transformed into 270 pairs): during stacking dots on junctions stacked sentence are lower-cased to match one-sentence setup. On these test sets we compute log loss for each position: lower value shows that model is more likely to predict a correct word at particular position.
replaced with commas and first letters of the next
Results Comparison between models trained with different positional embeddings WMT 14 benchmarks is shown in Table 2 CAPE outperforms sinpos and abspos on all settings, is similar to relpos DE and outperforms it on FR. Figure & shows that for positions covered in training (<150) all absolute positional embeddings behave similarly and outperform relpos; ii) for positions not seen during training (>200) relpos outperforms others. However, CAPE generalizes better than abspos and sinpos and, moreover, is able to generalize similar to relpos with more data (FR).
Discussion and Conclusion
Encoding positional information is key component of attention-based models. Poor generalization of absolute sinusoidal positional embeddings led to numerous works investigating ways encode rel- evant positional information. Existing solutions are often modality-specific, non-trivial to implement and incur computational overhead. We demonstrated in different domains that existing positional embeddings may generalize poorly in certain conditions. We introduced simple and efficient con- tinuous augmented positional embedding, CAPE, which preserves some information about relative token positions. Thanks to its continuous nature, CAPE allows augmentations which were previously not possible. CAPE makes models more flexible both at training and inference. It generalizes well to input sizes not seen during training across variety of domains. We expect emergence of new training and production pipelines that leverage the adjustable throughput property when tuning the input size: CAPE provides a relatively simple way of producing more efficient Transformer by down-sampling input (e-g for image and audio). Going further, CAPF-based architectures are free from baked-in restrictions on patches positions: these could overlap, or be sparse for example opportunities impossible for convolution- containing Transformers. In contrast to relative positional embeddings which modify attention mechanism, CAPE is ready be used by novel attention mechanisms, such as "linear" Transformers [46]. Finally, CAPE can combined with relative positional embeddings like I and 45 to limit over-fitting to exact relative positions.
Limitations CAPE applies only attention- based models, and no testing was performed outside described modalities. From representation perspective we demonstrated that CAPE is capable of providing relative positioning. however, a model should "learn" it. Thus, we can expect that relative positional embeddings should be beneficial in settings with small amount of data because of more appropriate inductive bias. Proposed Uni model was tested only for image recognition task and further exploration of broader Vni applicability other tasks is subject of future work.
10

---

7 Acknowledgments
We would like to thank Mark Tygert and Edouard Grave for relevant references and helpful discussions, Paden Tomasello for English language editing.
References
[1] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint arXiv:2103. 15691, 2021.
[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.044 2014.
[3] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3286-3295, 2019.
[4] Maxim Berman, Hervé Jégou, Andrea Vedaldi, lasonas Kokkinos, and Matthijs Douze. Multi- grain: a unified image embedding for classes and instances. arXiv preprint WiXi:1102.00550% 2019.
[5] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05095, 2021.
[6] Tom B Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer, 2020.
[8] William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi. Speech- stew: Simply mix all available speech recognition data to train one large neural network. arXiv preprint arXiv:2 104. 02/33, 2021.
[9] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882 2021.
[10] Ekin Dogus Cubuk, Barret Zoph, Jon Shiens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. Advances Neural Information Processing Systems, 33, 2020.
[11] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 9978-2988. 2019.
[12] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. In Intermational Conference on Learning Representations, 2019.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248 255. leee, 2009.
[14] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
[15] Linhao Dong, Shuang Xu, and Bo Xu. treschetransformer:: a no-recurrence sequence-to- sequence model for speech recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5884-5888. IEEE, 2018
11

---

[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[17] J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12, 2011.
[18] Philipp Dufter, Martin Schmitt, and Hinrich Schütze. Position information in transformers: An overview. arXiv preprint arXiv:2102.11090, 2021.
[19] A. Fan, E. Grave, and A. Joulin. Reducing transformer depth on demand with structured dropout. In ICML, 2020.
[20] Garofolo, D. Graff, D. Paul, and D. Pallett. CSR-I (WSJO) complete LDC93S6A. Web Download. Philadelphia: LDC, 1993.
[21] Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint arXiv:2104.01 136, 2021.
[22] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification:
labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006.
[23] Anmol Gulati et al. Conformer: canvolution-augmente transformer for speech recognition. In Interspeech, 2020.
[24] F. Hernandez al. TED-LIUM 3: twice much data and corpus repartition for experiments on speaker adaptation. In SPECOM, 2018.
[25] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8126 8135. IEEE Computer Society, 2020.
[26] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In International Conference Learning Representations, 2018
[27] Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. Improve transformer models with better relative position embeddings. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3327-3335. 2020.
[28] Guolin Ke, Di He, and Tie Yae Liu. Rethinking the positional encoding in language pre-training. arXiv preprint arXiv:2006. 15595, 2020.
[29] LDC and NIST Multimodal Information Group. CSR-II (WSJI) complete LDC94S13A. Web Download. Philadelphia: LDC, 1994.
[30] Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking evaluation in asr: Are our models robust enough? arXiv preprint arXiv:2010. / 1745, 2020.
[31] Bin Liu, Zhirong Wu, Han Hu, and Stephen Lin. Deep metric transfer for label propagation with limited annotated data. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0-0, 2019.
[32] Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. Very deep transformers for neural machine translation. arXiv preprint arXiv:2008.07722 2020.
[33] Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer. Transformers with convolu- tional context for asr. arXiv preprint arXiv: 2019.
12

---

[34] Daniel Neimark, Omri Bar. Maya Zohar, and Dotan Asselmann, Video transformer network. arXiv preprint w0X1v:2102.00719, 2021.
[35] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019 Demonstrations, 2019.
[36] V. Panayotov et al. Librispeech: an ASR corpus based on public domain audio books. In ICASSP, 2015.
[37] D. S. Park et al. SpecAugment: A simple data augmentation method for automatic speech recognition. In Interspeech, 2019.
[38] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018.
[39] D. Povey and other. The Kaldi speech recognition toolkit. In ASRU, 2011.
[40] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[41] Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Out, Kurt Shuster, Eric M Smith, et al. Recipes for building open-domain chatbot. arXiv preprint urXiv:2004. I 13637, 2020.
[42] Jan Rosendahl, Viet Anh Khoa Tran, Weiyue Wang, and Hermann Ney. Analysis of positional encodings for neural machine translation. IWSLT, Hong Kong, China, 2019.
[43] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115())211-255) 2015.
[44] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pages 464-468, 2018
[45] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0988 2021.
[46] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena benchmark for efficient transformers. In International Conference on Learning Representutions, 2021.
[47] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficien image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[48] A. Vaswani al. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[49] Benyou Wang. Lifeng Shang. Christina Lioma, Xin Jiang, Hao Yang, Qun Liu, and Jakob Grue Simonsen. On position embeddings in bert. In International Conference on Learning Represen- lations, volume 2, pages 12-13, 2021.
[50] Benyou Wang. Donghao Zhao Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. In International Conference on Learning Representations, 2020.
[51] Yongqiang Wang, Abdelrahman Mohamed, Due Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, et al. Transformer-based acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6874-6878. IEEF, 2020.
13

---

[52] P. C. Woodland et al. Large vocabulary continuous speech recognition using HTK. In ICASSP. 1994.
[53] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 15808, 2021.
[54] Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE/CVF International Conference Computer Vision (ICCV), pages 6022-6031. IEEE, 2019.
[55] Albert Zeyer, Parnia Bahar, Kazuki Irie, Ralf Schlüter, and Hermann Ney. A comparison of transformer and Istm encoder decoder models for asr. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 8-15. IEEE, 2019.
[56) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17xi:: 12, 2017.
[57] Yu Zhang. James Qin, Daniel Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc Le, and Yonghui Wu. Pushing the limits of semi-supervised learning for automatic speech recognition. arXiv preprint arXiv:2010. 10504, 2020.
[58] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34-07, pages 13001-13008, 2020.
[59] Pan Zhou, Ruchao Fan, Wei Chen, and Jia Jia. Improving generalization of transformer for speech recognition with parallel schedule sampling and relative positional embedding. arXiv preprint arXiv: /91 1.00203, 2019,
14


Reviews:
[{"review_id": "xlT6PP6Ec53", "reviewer": "Reviewer_6kLu", "summary": "This paper proposes an augmentation-based approach (CAPE) for absolute positional embeddings. The empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance. The paper is cleary-written and the experiments are comprehensive.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper explains that \"Relative positions are more complex to implement and yield inferior model throughput\" in the abstract.  It seems that the implementation complexity does not make any sense, since such implementation complexity does not necessarily lead to space or time cost. Such complexity should be clearly stated in space complexity, time complexity, or others that may bring extra cost. It would be better to explain why it affects model throughput. Note that there are some simpler Relative position encoding, see T5 that defined a scalar added to the corresponding logit used for computing the attention weights. This needed furtherly explained, otherwise it may confuse readers.\n\n\n\nWhen discussing complex-valued representation of embeddings in line 81. Please consider the following reference:\nWang et.al. encoding word order in complex embeddings. ICLR 2020. \nwhich also explored the continuous nature of position embeddings.\n\n\nminor issue:\nFigure 1 was not mentioned anywhere in the main text.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "nUt74BFbBFg", "reviewer": "Reviewer_MLDf", "summary": "This paper introduces a data augmentation method that adds random noise to positions for the purpose of computing position embeddings for Transformers in vision, speech, and machine translation.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Model\n\nThe formulation of sinusoidal position encodings as complex numbers is not new; see Wang et al (ICLR 2020), DeBenedetto and Chiang (ICML 2020), Su et al (arXiv:2104.09864).\n\nCan you provide some motivation for your choice of w_{k,x} and w_{k,y}? I’m a little surprised to see a sine/cosine inside another sine/cosine.\n\nLines 111-112: What is mean-normalization?\n\n# Experiments\n\nThe paper presents experiments on three tasks: image recognition, speech recognition, and machine translation.\n\nOn image recognition, two different models are tried. Figure 2 shows that CAPE gives slight improvements except when training on 224x224 and, fine-tuning on 384x384 but testing on 160x160, where CAPE outperforms all other methods by a wide margin.\n\nWhat are the four smaller graphs in Figure 2?\n\nThe paper also presents a related method, UniViT, which doesn’t do global scaling; instead, it resizes the images themselves to random sizes. Across all resolutions, this performs consistently well, better than all other methods including CAPE.\n\nOn speech recognition (Figure 5-6), CAPE generally does well, especially on their own Robust Video dataset.\n\nOn machine translation (WMT14 task), improvements are small (0.27-0.56) and significance testing is not performed.\n\n# Overall\n\nThis is a nice, simple data augmentation method that works on several different modalities and doesn't require the computer to be too \"creative\" (e.g., choosing random words). It seems to help on speech and vision, but not so much on machine translation.\n\nFrom both a presentational and scientific standpoint, it's confusing that a different method (UniVIT) is introduced that outperforms CAPE. Presentationally, because the paper's title is CAPE and the methods section only describes CAPE, and it's in the experiments section that a new method is unexpectedly introduced. Scientifically, because it's not clear why one does better than the other, and in which scenarios one should prefer one or the other.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I'd like to see (1) better treatment of UniVIT, in the form of (a) more organized motivation and presentation, (b) experiments with UniVIT on all tasks where it is applicable; and (2) significance testing.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "aS7jVv1Ui67", "reviewer": "Reviewer_b4uj", "summary": "This paper claimed that the existing positional encoding mechanism (i.g., absolute or relative positional encodings) for the Transformer model has a generalization issue, that is, these position encoding methods have their own advantages and disadvantages, which may prevent the Transformer model from simulating positional information. Through formal analysis of the existing position encoding mechanisms, the authors proposed an augmentation-based approach to integrating the advantages of the existing absolute position encoding and relative position encoding. It is expected that the positional information can be efficiently and simply modeled for the Transformer model. Finally, the proposed method was evaluated on several tasks which adapted to the Transformer model.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. There is a lack of sufficient argument support and analysis about whether this claimed generalization issue really exists. \n\n2. The Transformer model was originally proposed for machine translation tasks. Positional encoding plays an important role. However, the improvements over the baseline on machine translation tasks are marginal over the baseline abspos in Table 2. \n\n3. There is a lack of necessary comparisons with relative positional encoding. I am afraid that CAPE may be inferior to the existing relative positional encoding.\n\n4. There is a lack of necessary ablation experiments to verify that this marginal improvement indeed comes from the alleviation of the claimed generalization issue.\n\n5. If all model parameters in the trained baseline Transformer model are frozen except the positional embeddings, the trained baseline Transformer model is further fine-tuned by one or several Epochs. Or, in the beginning, these positional embeddings are frozen from the beginning and are applied to different position encoding methods to verify their effectiveness.\n\nOverall, I think the motivation of this article is unclear, its novel is limited, its improvement is marginal on the machine translation tasks.\n\nUpdate:\n\nThe author's response addressed part of my main concerns, and I will increase my score from 3 to 4.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "TuhxCDFESXc", "reviewer": "Reviewer_iFjJ", "summary": "This paper proposes “CAPE” (continuous augmented positional embeddings), a novel form of positional embeddings, aimed at making transformer models more robust to changes in the input distribution at inference time (e.g., changes in input size/length/resolution, changes in relative distances/positions between subunits of input, and changes in absolute positions of input subunits).  It accomplishes this by (1) indexing positional embeddings in a continuous way (e.g., for images: indexing by the normalized coordinate $(x,y)  \\in [-1,+1]^2$ of a patch; for audio/speech: indexing by the time (in seconds) of a frame), and (2) performing a series of augmentations (global shift, local shift, global scaling) to these “continuous indexes” at each position in the input (these augmentations can be thought of as being similar to performing augmentations on the input).\n\nThe paper presents experimental results across image classification (vision), ASR (speech), and machine translation (NLP) tasks, showing that in general, the proposed CAPE embeddings outperform baseline positional embedding methods. In particular, CAPE is compared with (1) _nopos_: no positional embeddings, (2) _abspos_: learnable absolute positional embeddings, (3) _sinpos_: continuous sinusoidal positional embeddings (without the proposed augmentations), and (4) _relpos_: learnable relative positional embeddings (for ASR tasks).\n\n\n**Detailed summary of experimental results, by domain**\n\n_Image Classification_: The experiments demonstrate that ViT models trained with CAPE embeddings perform better than the baseline embedding methods across test images of different resolutions. Furthermore, the paper proposes “UniViT”, a ViT model trained with CAPE embeddings and randomly resized inputs (128x128 -> 320x320), and shows that UniViT performs extremely well across test image resolutions (160x160 -> 672x672) relative to ViT models trained only on a single resolution.  Because the UniViT model gracefully handles images of different resolutions, this model also provides a way for practitioners to easily trade-off increased throughput for lower accuracy, by lowering the resolution of the input images fed to the model at inference time.\n\n_ASR_: The paper shows that CAPE generally outperforms the other absolute positional embeddings (nopos, sinpos, abspos), while generally being competitive with, or outperforming, the more computationally expensive relative positional embeddings (relpos). It additionally shows that (1) CAPE performs similarly to using SpecAugment as a form of data augmentation during training (and combining CAPE + SpecAugment does best), and (2) CAPE can be used, together with adjusting the STFT hop distance of utterances in a batch, to eliminate the need for padding during ASR training (without degrading performance).\n\n_Machine Translation_:  The paper again demonstrates strong performance for CAPE.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\nThe work is quite original in my opinion. In particular, the idea of adding a series of augmentations to the positional embeddings, in order to make the model more robust to shifts/distortions in the input at inference time, is a simple/powerful/novel idea. The way in which CAPE embeddings are applied to image classification, to train a model which is robust to different image resolutions, and thus allow for practitioners to perform faster/cheaper inference by downsampling input images, is also quite original.\n\n### Quality\nThe quality of the work is high. The paper presents thorough experiments across three different input domains (images, speech, text), comparing throughout with important baselines.\n\n### Significance\nI believe this paper has the potential to be relatively impactful, for the following reasons.\n1) It offers a domain-agnostic framework for performing augmentation for transformer models, yielding more robust (and higher accuracy) models (e.g., UniViT). For example, I believe it is in many cases simpler to use CAPE than to design domain-specific augmentations (e.g., SpecAugment). CAPE can also be combined with domain-specific augmentations in order to yield even better performance.\n2) The continuous indexing proposed by CAPE allows for training models which are robust to inputs of different lengths, as well as to stretching/compressing inputs.  As a result, CAPE offers an easy way to either (a) make inference cheaper by downsampling inputs, or (b) get rid of padding in ASR training by adjusting STFT hops.\n\n### Clarity\nIn general, the paper is written in a very clear way.  Some questions about specific points meriting further clarification are included at the end of this review.  \n\nThroughout the paper, I think it would be beneficial to be more specific about the performance differences between CAPE and the baseline methods (e.g., CAPE performs XX% absolute better/faster/?? than YY on ZZ)---this would help clarify to the reader the exact gains from CAPE, without them needing to zoom into specific parts of a figure.  These numbers could be included in the abstract/introduction.\n\n## Overall\nI think this paper would make a great contribution to NeurIPS, and I recommend acceptance for this work. \n\n**Main strengths**:\n* I think the accuracy improvements from using CAPE in numerous domains (especially in different-from-training settings), the UniViT model which performs well across image resolutions, the ability to easily navigate the accuracy-computation trade-off by modifying input size (e.g., Figure 4), and the general domain-agnostic framework for data augmentation introduced by CAPE, all have the potential to be impactful.\n\n**Main weaknesses**:\n* I think the main weaknesses of the work are clarity in a few sections (see below), an incomplete empirical comparison with (and related work discussion about) existing methods for dealing with images of different resolutions, and a way-too-brief discussion of the limitations of CAPE.\n\nImportantly, I am not an expert in the latest literature about positional embeddings for transformers, and thus my review is only medium-confidence.\n\nI now provide a longer/more detailed list of questions for the authors.\n\n### Clarifications/Questions:\n1) Choices of equations for positional embeddings:\n* It was unclear to me how the equations for $w_{k,x}$ and $w_{k,y}$ were chosen for the positional embeddings for images.  In particular, why does $w_{k,x}$ include a multiplication by $\\cos(k)$, and $w_{k,y}$ include a multiplication by $sin(k)$?  Why are $w_{k,x} x$ and $w_{k,x} y$, combined additively (for example, as opposed to computing positional embeddings in the x and y dimensions separately, and concatenating)?  Have fixed (as opposed to learned) positional embeddings been used for image classification before, and if so, what functional forms were chosen for those positional embeddings?\n* Across the three domains, how were the different scaling terms for $w_k$ chosen ($10000^{-2k/K}$, $10^{2k/K}$, $30 \\cdot 10^{-2k/K}$)?\n\n2) Image classification experiments:\n* It was not clear to me if when you train (or test) on images of different resolutions, this corresponds to (a) training (or testing) on completely disjoint sets of images, whose original resolutions match the stated resolution (e.g., when you train with 224x224 images, this corresponds to training on all images in the ImageNet dataset whose native resolution is 224x224), or (b) training (or testing) on the same set of images, which have been adjusted (scaled/cropped) to match the stated resolution (e.g., take all the ImageNet images, and rescale them to be 224x224).  Can you please clarify this? For example, is UniViT trained on a larger set of images than the resolution-specific ViT models (or on the same set of images, but with random rescaling)?  On a related note: a discussion about the ImageNet dataset, the resolutions of the images in this dataset, and the way these differences in resolution have been handled by prior work (e.g., via cropping), would be quite helpful.\n* What, in your opinion, is the difference between (1) training a ViT model with CAPE on images that have all been scaled to a certain resolution (e.g., 224x224), (2) training using CAPE on images of different resolutions, and (3) training on randomly resized images (e.g., UniViT)? It seems that all of these approaches would nicely handle images of different sizes (although the ViT models which take as input images of different resolutions would be cheaper on low-resolution images, and more expensive on high-resolution images).  Could comparisons between these different approaches be included in the paper?\n\n3) ASR experiments:\n* Why is the proposed method for getting rid of padding (STFT hop distance tuning), better than simply doing padding? It doesn’t seem that the proposed method is simpler than padding, more computationally efficient than padding, or better performing than padding (dashed and solid lines in figures 15 and 16 seem to perform quite similarly). I think it would be helpful to clarify these points in the main text.\n* Could an equivalent of Figure 4 be produced for ASR?  For example, plot WER of utterances as you increase the STFT hop distance?\n* Could an equivalent of UniViT be trained for ASR?  For example, training a model on randomly chosen STFT hop distance?\n\n4) MT experiments:\n* Could an equivalent of Figure 4 be produced for MT? For example, plotting BLEU as you drop the least meaningful words in the input sentence? At first glance, this seems quite difficult, but I was curious if this accuracy-computation trade-off exists in the NLP domain as well. For example, perhaps you can still generate good translations after dropping stop words? Perhaps this would be easier to demonstrate on sentence classification tasks, in which ignoring certain words in the input might not make a big impact in the ability of the model to classify the sentence correctly (though one would have to be careful with this pruning, since removing certain words (e.g., negations) could completely change the correct answer).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think a broader discussion about the limitations would be helpful (only 1 vacuous sentence was included after the conclusion). For example, discussing when relative positional embeddings would attain better performance than any absolute positional embeddings (e.g., CAPE). From a representation perspective, what can’t be captured by CAPE embeddings?\n\nIn terms of societal impact, the authors could argue that these embeddings provide a relatively simple way of producing more efficient transformer models (by downsampling input, at least for images/audio), and perhaps mention how CAPE could be combined with other efficient transformer innovations (e.g., performer / other “linear attention” transformers).\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although there is some variance in the reviewer assessments about the paper, most reviewers are positive and think the work nicely integrates a set of techniques for positional encoding that can capture relative positional relationships. The work evaluates these techniques in several distinct tasks, including vision, language and speech. As Reviewer MLDf commented: "It (the technique) seems to help on speech and vision, but not so much on machine translation." I would want to thank the authors for providing extensive additional experimental results in the rebuttal, which strengthen the paper. The reviewers have provided many great points for improving the paper, which I hope the authors can include them in the revision of the paper.
Decision: Accept (Poster)