Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes “CAPE” (continuous augmented positional embeddings), a novel form of positional embeddings, aimed at making transformer models more robust to changes in the input distribution at inference time (e.g., changes in input size/length/resolution, changes in relative distances/positions between subunits of input, and changes in absolute positions of input subunits).  It accomplishes this by (1) indexing positional embeddings in a continuous way (e.g., for images: indexing by the normalized coordinate $(x,y)  \in [-1,+1]^2$ of a patch; for audio/speech: indexing by the time (in seconds) of a frame), and (2) performing a series of augmentations (global shift, local shift, global scaling) to these “continuous indexes” at each position in the input (these augmentations can be thought of as being similar to performing augmentations on the input).

The paper presents experimental results across image classification (vision), ASR (speech), and machine translation (NLP) tasks, showing that in general, the proposed CAPE embeddings outperform baseline positional embedding methods. In particular, CAPE is compared with (1) _nopos_: no positional embeddings, (2) _abspos_: learnable absolute positional embeddings, (3) _sinpos_: continuous sinusoidal positional embeddings (without the proposed augmentations), and (4) _relpos_: learnable relative positional embeddings (for ASR tasks).


**Detailed summary of experimental results, by domain**

_Image Classification_: The experiments demonstrate that ViT models trained with CAPE embeddings perform better than the baseline embedding methods across test images of different resolutions. Furthermore, the paper proposes “UniViT”, a ViT model trained with CAPE embeddings and randomly resized inputs (128x128 -> 320x320), and shows that UniViT performs extremely well across test image resolutions (160x160 -> 672x672) relative to ViT models trained only on a single resolution.  Because the UniViT model gracefully handles images of different resolutions, this model also provides a way for practitioners to easily trade-off increased throughput for lower accuracy, by lowering the resolution of the input images fed to the model at inference time.

_ASR_: The paper shows that CAPE generally outperforms the other absolute positional embeddings (nopos, sinpos, abspos), while generally being competitive with, or outperforming, the more computationally expensive relative positional embeddings (relpos). It additionally shows that (1) CAPE performs similarly to using SpecAugment as a form of data augmentation during training (and combining CAPE + SpecAugment does best), and (2) CAPE can be used, together with adjusting the STFT hop distance of utterances in a batch, to eliminate the need for padding during ASR training (without degrading performance).

_Machine Translation_:  The paper again demonstrates strong performance for CAPE.

main_review: ### Originality
The work is quite original in my opinion. In particular, the idea of adding a series of augmentations to the positional embeddings, in order to make the model more robust to shifts/distortions in the input at inference time, is a simple/powerful/novel idea. The way in which CAPE embeddings are applied to image classification, to train a model which is robust to different image resolutions, and thus allow for practitioners to perform faster/cheaper inference by downsampling input images, is also quite original.

### Quality
The quality of the work is high. The paper presents thorough experiments across three different input domains (images, speech, text), comparing throughout with important baselines.

### Significance
I believe this paper has the potential to be relatively impactful, for the following reasons.
1) It offers a domain-agnostic framework for performing augmentation for transformer models, yielding more robust (and higher accuracy) models (e.g., UniViT). For example, I believe it is in many cases simpler to use CAPE than to design domain-specific augmentations (e.g., SpecAugment). CAPE can also be combined with domain-specific augmentations in order to yield even better performance.
2) The continuous indexing proposed by CAPE allows for training models which are robust to inputs of different lengths, as well as to stretching/compressing inputs.  As a result, CAPE offers an easy way to either (a) make inference cheaper by downsampling inputs, or (b) get rid of padding in ASR training by adjusting STFT hops.

### Clarity
In general, the paper is written in a very clear way.  Some questions about specific points meriting further clarification are included at the end of this review.  

Throughout the paper, I think it would be beneficial to be more specific about the performance differences between CAPE and the baseline methods (e.g., CAPE performs XX% absolute better/faster/?? than YY on ZZ)---this would help clarify to the reader the exact gains from CAPE, without them needing to zoom into specific parts of a figure.  These numbers could be included in the abstract/introduction.

## Overall
I think this paper would make a great contribution to NeurIPS, and I recommend acceptance for this work. 

**Main strengths**:
* I think the accuracy improvements from using CAPE in numerous domains (especially in different-from-training settings), the UniViT model which performs well across image resolutions, the ability to easily navigate the accuracy-computation trade-off by modifying input size (e.g., Figure 4), and the general domain-agnostic framework for data augmentation introduced by CAPE, all have the potential to be impactful.

**Main weaknesses**:
* I think the main weaknesses of the work are clarity in a few sections (see below), an incomplete empirical comparison with (and related work discussion about) existing methods for dealing with images of different resolutions, and a way-too-brief discussion of the limitations of CAPE.

Importantly, I am not an expert in the latest literature about positional embeddings for transformers, and thus my review is only medium-confidence.

I now provide a longer/more detailed list of questions for the authors.

### Clarifications/Questions:
1) Choices of equations for positional embeddings:
* It was unclear to me how the equations for $w_{k,x}$ and $w_{k,y}$ were chosen for the positional embeddings for images.  In particular, why does $w_{k,x}$ include a multiplication by $\cos(k)$, and $w_{k,y}$ include a multiplication by $sin(k)$?  Why are $w_{k,x} x$ and $w_{k,x} y$, combined additively (for example, as opposed to computing positional embeddings in the x and y dimensions separately, and concatenating)?  Have fixed (as opposed to learned) positional embeddings been used for image classification before, and if so, what functional forms were chosen for those positional embeddings?
* Across the three domains, how were the different scaling terms for $w_k$ chosen ($10000^{-2k/K}$, $10^{2k/K}$, $30 \cdot 10^{-2k/K}$)?

2) Image classification experiments:
* It was not clear to me if when you train (or test) on images of different resolutions, this corresponds to (a) training (or testing) on completely disjoint sets of images, whose original resolutions match the stated resolution (e.g., when you train with 224x224 images, this corresponds to training on all images in the ImageNet dataset whose native resolution is 224x224), or (b) training (or testing) on the same set of images, which have been adjusted (scaled/cropped) to match the stated resolution (e.g., take all the ImageNet images, and rescale them to be 224x224).  Can you please clarify this? For example, is UniViT trained on a larger set of images than the resolution-specific ViT models (or on the same set of images, but with random rescaling)?  On a related note: a discussion about the ImageNet dataset, the resolutions of the images in this dataset, and the way these differences in resolution have been handled by prior work (e.g., via cropping), would be quite helpful.
* What, in your opinion, is the difference between (1) training a ViT model with CAPE on images that have all been scaled to a certain resolution (e.g., 224x224), (2) training using CAPE on images of different resolutions, and (3) training on randomly resized images (e.g., UniViT)? It seems that all of these approaches would nicely handle images of different sizes (although the ViT models which take as input images of different resolutions would be cheaper on low-resolution images, and more expensive on high-resolution images).  Could comparisons between these different approaches be included in the paper?

3) ASR experiments:
* Why is the proposed method for getting rid of padding (STFT hop distance tuning), better than simply doing padding? It doesn’t seem that the proposed method is simpler than padding, more computationally efficient than padding, or better performing than padding (dashed and solid lines in figures 15 and 16 seem to perform quite similarly). I think it would be helpful to clarify these points in the main text.
* Could an equivalent of Figure 4 be produced for ASR?  For example, plot WER of utterances as you increase the STFT hop distance?
* Could an equivalent of UniViT be trained for ASR?  For example, training a model on randomly chosen STFT hop distance?

4) MT experiments:
* Could an equivalent of Figure 4 be produced for MT? For example, plotting BLEU as you drop the least meaningful words in the input sentence? At first glance, this seems quite difficult, but I was curious if this accuracy-computation trade-off exists in the NLP domain as well. For example, perhaps you can still generate good translations after dropping stop words? Perhaps this would be easier to demonstrate on sentence classification tasks, in which ignoring certain words in the input might not make a big impact in the ability of the model to classify the sentence correctly (though one would have to be careful with this pruning, since removing certain words (e.g., negations) could completely change the correct answer).

limitations_and_societal_impact: I think a broader discussion about the limitations would be helpful (only 1 vacuous sentence was included after the conclusion). For example, discussing when relative positional embeddings would attain better performance than any absolute positional embeddings (e.g., CAPE). From a representation perspective, what can’t be captured by CAPE embeddings?

In terms of societal impact, the authors could argue that these embeddings provide a relatively simple way of producing more efficient transformer models (by downsampling input, at least for images/audio), and perhaps mention how CAPE could be combined with other efficient transformer innovations (e.g., performer / other “linear attention” transformers).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an augmentation-based approach (CAPE) for absolute positional embeddings. The empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance. The paper is cleary-written and the experiments are comprehensive.
main_review: The paper explains that "Relative positions are more complex to implement and yield inferior model throughput" in the abstract.  It seems that the implementation complexity does not make any sense, since such implementation complexity does not necessarily lead to space or time cost. Such complexity should be clearly stated in space complexity, time complexity, or others that may bring extra cost. It would be better to explain why it affects model throughput. Note that there are some simpler Relative position encoding, see T5 that defined a scalar added to the corresponding logit used for computing the attention weights. This needed furtherly explained, otherwise it may confuse readers.



When discussing complex-valued representation of embeddings in line 81. Please consider the following reference:
Wang et.al. encoding word order in complex embeddings. ICLR 2020. 
which also explored the continuous nature of position embeddings.


minor issue:
Figure 1 was not mentioned anywhere in the main text.
limitations_and_societal_impact: None
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces a data augmentation method that adds random noise to positions for the purpose of computing position embeddings for Transformers in vision, speech, and machine translation.

main_review: # Model

The formulation of sinusoidal position encodings as complex numbers is not new; see Wang et al (ICLR 2020), DeBenedetto and Chiang (ICML 2020), Su et al (arXiv:2104.09864).

Can you provide some motivation for your choice of w_{k,x} and w_{k,y}? I’m a little surprised to see a sine/cosine inside another sine/cosine.

Lines 111-112: What is mean-normalization?

# Experiments

The paper presents experiments on three tasks: image recognition, speech recognition, and machine translation.

On image recognition, two different models are tried. Figure 2 shows that CAPE gives slight improvements except when training on 224x224 and, fine-tuning on 384x384 but testing on 160x160, where CAPE outperforms all other methods by a wide margin.

What are the four smaller graphs in Figure 2?

The paper also presents a related method, UniViT, which doesn’t do global scaling; instead, it resizes the images themselves to random sizes. Across all resolutions, this performs consistently well, better than all other methods including CAPE.

On speech recognition (Figure 5-6), CAPE generally does well, especially on their own Robust Video dataset.

On machine translation (WMT14 task), improvements are small (0.27-0.56) and significance testing is not performed.

# Overall

This is a nice, simple data augmentation method that works on several different modalities and doesn't require the computer to be too "creative" (e.g., choosing random words). It seems to help on speech and vision, but not so much on machine translation.

From both a presentational and scientific standpoint, it's confusing that a different method (UniVIT) is introduced that outperforms CAPE. Presentationally, because the paper's title is CAPE and the methods section only describes CAPE, and it's in the experiments section that a new method is unexpectedly introduced. Scientifically, because it's not clear why one does better than the other, and in which scenarios one should prefer one or the other.
limitations_and_societal_impact: I'd like to see (1) better treatment of UniVIT, in the form of (a) more organized motivation and presentation, (b) experiments with UniVIT on all tasks where it is applicable; and (2) significance testing.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper claimed that the existing positional encoding mechanism (i.g., absolute or relative positional encodings) for the Transformer model has a generalization issue, that is, these position encoding methods have their own advantages and disadvantages, which may prevent the Transformer model from simulating positional information. Through formal analysis of the existing position encoding mechanisms, the authors proposed an augmentation-based approach to integrating the advantages of the existing absolute position encoding and relative position encoding. It is expected that the positional information can be efficiently and simply modeled for the Transformer model. Finally, the proposed method was evaluated on several tasks which adapted to the Transformer model.
main_review: 1. There is a lack of sufficient argument support and analysis about whether this claimed generalization issue really exists. 

2. The Transformer model was originally proposed for machine translation tasks. Positional encoding plays an important role. However, the improvements over the baseline on machine translation tasks are marginal over the baseline abspos in Table 2. 

3. There is a lack of necessary comparisons with relative positional encoding. I am afraid that CAPE may be inferior to the existing relative positional encoding.

4. There is a lack of necessary ablation experiments to verify that this marginal improvement indeed comes from the alleviation of the claimed generalization issue.

5. If all model parameters in the trained baseline Transformer model are frozen except the positional embeddings, the trained baseline Transformer model is further fine-tuned by one or several Epochs. Or, in the beginning, these positional embeddings are frozen from the beginning and are applied to different position encoding methods to verify their effectiveness.

Overall, I think the motivation of this article is unclear, its novel is limited, its improvement is marginal on the machine translation tasks.

Update:

The author's response addressed part of my main concerns, and I will increase my score from 3 to 4.
limitations_and_societal_impact: N/A
review_title: null

Meta-review: Although there is some variance in the reviewer assessments about the paper, most reviewers are positive and think the work nicely integrates a set of techniques for positional encoding that can capture relative positional relationships. The work evaluates these techniques in several distinct tasks, including vision, language and speech. As Reviewer MLDf commented: "It (the technique) seems to help on speech and vision, but not so much on machine translation." I would want to thank the authors for providing extensive additional experimental results in the rebuttal, which strengthen the paper. The reviewers have provided many great points for improving the paper, which I hope the authors can include them in the revision of the paper.
Decision: Accept (Poster)