Conditional Positional Encodings for Vision Transformers
Authors: Xiangxiang Chu
Zhi Tian
Bo Zhang
Xinlong Wang
Chunhua Shen
Venue: ICLR 2023 poster (2023)
Keywords: Vision Transformer

Abstract: We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings that are predefined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during the training. Besides, CPE can keep the desired translation equivalence in vision tasks, resulting in improved performance. We implement CPE with a simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework. Built on PEG, we present Conditional Position encoding Vision Transformer (CPVT). We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results.

Published as a conference paper at ICLR 2023
CONDITIONAL POSITIONAL ENCODINGS FOR VISION TRANSFORMERS
Xiangxiang Chu', Zhi Tian', Bo Zhang', Xinlong Wang?, Chunhua Shen I Meituan Inc. 2 Beijing Academy of Al "Zhejiang University, China fchuxiangxiang, tianzhi02, zhangbog7 Jemeituan.com com, xinleng wang968gmaill wtt.://mmaa...m...mm.m chunhua 6me. com
ABSTRACT
We propose conditional positional encoding (CPE) scheme for vision Trans- formers (Dosovitskiy et al., 2021; Touvron et al., 2020). Unlike previous fixed or learnable positional encodings that are predefined and independent of input to- kens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during the training. Besides, CPE can keep the desired translation equivalence in vision tasks, resulting in improved per- formance. We implement CPE with simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework. Built on PEG, present Conditional Position encoding Vision Transformer (CPVT). We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results. Our Code is available at: https: http:/ggttttttt///////
INTRODUCTION
Recently, Transformers (Vaswani et al., 2017) have been viewed strong alternative to Convolu- tional Neural Networks (CNNs) in visual recognition tasks such as classification (Dosovitskiy et al., 2021) and detection (Carion et al., 2020: Zhu al., 2021). Unlike the convolution operation in CNNs, which has a limited receptive field, the self-attention mechanism in the Transformers can capture the long-distance information and dynamically adapt the receptive field according to the image content. Consequently, Transformers are considered more flexible and powerful than CNNs, being promising achieve more progress in visual recognition.
However, the self-attention operation in Transformers is permutation-invarian which discards the order of the tokens in an input sequence. To mitigate this issue, previous works (Vaswani et al., 2017; Dosovitskiy et al., 2021) add the absolute positional encodings to each input token (see Fig- ure la), which enables arder-awareness. The positional encoding can either be learnable or fixed with sinusoidal functions of different frequencies. Despite being effective, these positional encod- ings seriously harm the flexibility of the Transformers, hampering their broader applications. Taking the learnable version as an example, the encodings are often vector of equal length to the input sequence, which are jointly updated with the network weights during training. As a result, the length and the value of the positional encodings fixed once trained. During testing, it causes difficulties of handling the sequences longer than the ones in the training data.
The inability to adapt to longer input sequences during testing greatly limits the range of general- ization. For instance, in vision tasks like object detection, we expect the model can be applied to the images of any size during inference, which might be much larger than the training images. A possi- ble remedy is to use bicubic interpolation to upsample the positional encodings to the target length, but degrades the performance without fine-tuning as later shown in our experiments. For vision For example, the output feature
in general. we expect that the models be ceatlllatoonquvariiaann maps of CNNs shift accordingly as the target objects are moved in the input images. However, the absolute positional encoding scheme might break the translation equivalence because it adds unique
"Corresponding author.

---

Published as a conference paper at ICLR 2023
MLP
MLP
MLP
GAP
L Encoders
(L-1) x Encoders
(I-- I) x Encoders
PEG
PEG
PE
Encoder
Encoder
cls
Embedding
cls
Embedding
Embedding
(a) VIT
(b) CPVT
(c) CPVT-GAP
Figure 1. Vision Transformers: (a) ViT (Dosovitskiy al., 2021) with explicit ID learnable posi- tional encodings (PE) (b) CPVT with conditional positional encoding from the proposed Position Encoding Generator (PEG) plugin, which is the default choice. (c) CPVT-GAP without class token (cls), but with global average pooling (GAP) over all items in the sequence. Note that GAP is a bonus version which has boosted performance.
positional encodings each token (or each image patch). One may overcome the issue with rela- tive positional encodings as in (Shaw et al., 2018). However, relative positional encodings not only come with extra computational costs, but also require modifying the implementation of the standard Transformers. Last but not least, the relative positional encodings cannot work equally well as the absolute ones, because the image recognition task still requires absolute position information (Islam et al., 2020), which the relative positional encodings fail to provide.
In this work, we advocate novel positional encoding (PE) scheme incorporate the position information into Transformers. Unlike the predefined and input-agnostic positional encodings used in previous works (Dosovitskiy et al., 2021; Vaswani et al., 2017; Shaw al., 2018), the proposed PE is dynamically generated and conditioned on the local neighborhood of input tokens. Thus, our positional encodings can change along with the input size and try keep translation equivalence. We demonstrate that the vision transformers (Dosovitskiy et al., 2021; Touvron et al., 2020) with our new PE (i.e. CPVT, see Figure Ic) achieve even better performance. We summarize our contributions as,
We propose a novel positional encoding (PE) scheme, termed conditional position encod- ings (CPE). CPE is dynamically generated with Positional Encoding Generators (PEG) and can be effortlessly implemented by the modern deep learning frameworks (Paszke al., 2019; Abadi et al., 2016; Chen et al., 2015), requiring no changes to the current Trans- former APIs. Through an in-depth analysis and thorough experimentations, we unveil that this design affords both absolute and relative encoding yet it goes above and beyond.
As opposed to widely-used absolute positional encodings, CPE can provide kind of stronger explicit bias towards the translation equivalence which is important to improve the performance of Transformers.
Built on CPE, we propose Conditional Position encoding Vision Transformer (CPVT). achieves better performance than previous vison transformers (Dosovitskiy et al., 2021; Touvron et al., 2020).
CPE can well generalize to arbitrary input resolutions, which are required in many impor- tant downstream tasks such segmentation and detection. Through experiments we show that CPE can boost the segmentation and detection performance for pyramid transformers like (Wang et al., 2021) by a clear margin.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Since self-attention itself is permutation-equivariiee (see A), positional encodings are commonly employed incorporate the order of sequences (Vaswani et al., 2017). The positional encodings can either be fixed or learnable, while either being absolute or relative. Vision transformers follow the same fashion to imbue the network with positional information.
Absolute Positional Encoding. The absolute positional encoding is the most widely used. In the original transformer (Vaswani al., 2017), the encodings are generated with the sinusoidal functions of different frequencies and then they are added to the inputs. Alternatively, the positional encodings can be learnable, where they are implemented with a fixed-dimension matrix/tensor and jointly updated with the model's parameters with SGD.
Relative Positional Encoding. The relative position encoding (Shaw et al., 2018) considers dis- tances between the tokens in the input sequence. Compared the absolute ones, the relative posi- tional encodings can be translation-cquivaraan and can naturally handle the sequences longer than the longest sequences during training (i.e., being inductive). A 2-D relative position encoding is proposed for image classification in (Bello et al., 2019), showing superiority to 2D sinusoidal em- beddings. The relative positional encoding is further improved XLNet (Yang al., 2019b) and DeBERTa (He et al., 2020), showing better performance.
Other forms. Complex-vale embeddings (Wang et al., 2019) are an extension to model global absolute encodings and show improvement. RoFormer (Su et al., 2021) utilizes rotary position em- bedding to encode both absolute and relative position information for text classification. FLOATER (Liu et al., 2020) proposes a novel continuous dynamical model to capture position encodings. It is not limited by the maximum sequence length during training, meanwhile being parameter-efficient.
Similar designs to CPE. Convolutions are used to model local relations in ASR and machine translation (Gulati et al., 2020; Mohamed et al., 2019; Yang et al., 2019a; Yu et al., 2018). However, they are mainly limited 1D signals. We instead process 2D vision images.
VISION TRANSFORMER WITH CONDITIONAL POSITION ENCODINGS
3.1 MOTIVATION
In vision transformers, an input image of size H x W is split into patches with size S x S, the number of patches is N HW1 g The patches are added with the same number of learnable absolute positional encoding vectors. In this work, we argue that the positional encodings used here have two issues. First, it prevents the model from handling the sequences longer than the learnable PE. because unique positional encoding vector
Second, makes the model not translation-equivaaee added to every one patch. The translation equivalence plays an important role in classification because we hope the networks' responses changes accordingly as the object moves in the image.
One may note that the first issue can be remedied by removing the positional encodings since except for the positional encodings, all other components (e.g., MHSA and FFN) of the vision transformer can directly be applied to longer sequences. However, this solution severely deteriorates the perfor- mance. This understandable because the order of the input sequence an important clue and the model has no way to extract the order without the positional encodings. The experiment results on ImageNet are shown in Table 1. By removing the positional encodings, DeiT--iny's 's performance on ImageNet dramatically degrades from 72.2% to 68.2%.
Second, in DeiT (Touvron et al., 2020), they show that we can interpolate the position encodings to make them have the same length of the longer sequences. However, this method requires fine- tuning the model few more epochs, otherwise the performance will remarkably drop, as shown in Table 1. This goes contrary to what would expect. With the higher-resolution inputs, often expect a remarkable performance improvement without any fine-tuning. Finally, the relative position encodings (Shaw et al., 2018; Bello et al., 2019) can cope with both the aforementioned
H and W shall be divisible by S, respectively.
3

---

Published as a conference paper at ICLR 2023
Table Comparison of various positional encoding (PE) strategies tested ImageNet valida- tion set in terms of the top-1 accuracy. Removing the positional encodings greatly damages the performance. The relative positional encodings have inferior performance to the absolute ones
Model
Encoding
Top- I @ Top- @ 384(%)
DeiT-tiny (Touvron et al., 2020)
68.2
68.6
DeiT-tiny (Touvron et al., 2020)
learnable
72.2
71.2
DeiT-tiny (Touvron et al., 2020)
sin-cos
72.3
70.8
DeiT-tiny
2DD RPE (Shaw et al., 2018)
70.5
69.8
issues. However, the relative positional encoding cannot provide absolute position information, which is also important to the classification performance (Islam et al., 2020). As shown in Table 1, the model with relative position encodings has inferior performance (70.5% vs. 72.2%).
3.2 CONDITIONAL POSITIONAL ENCODINGS
We argue that a successful positional encoding for vision tasks should meet these requirements,
Making the input sequence permulatioon-variant and providing stronger explicit bias wards iaastaationequivariinee
Being inductive able to handle sequences longer than ones during training. Having the ability provide the absolute position certain degree. This important to the performance as shown in (Islam et al., 2020).
this work, we find that characterizing the local relationship by positional encodings sufficient to meet all of the above. First, it is permutation-vvriian because the permutation of input sequences also affects the order in some local neighborhoods. However, translation of an object in an input im- age does not change the order in its local neighhorhood, i.e., Second, the model can easily generalize to longer sequences since only the local neighborhoods of a token involved. Besides, if the absolute position of any input token is known, the absolute posi- tion of all the other tokens can be inferred by the mutual relation between input tokens. We will show that the tokens on the borders can be aware of their absolute positions due the commonly-used zero paddings.
(see Section A).
Therefore, we propose positional encoding gen- erators (PEG) to dynam- ically produce the posi- tional encodings condi- tioned on the local neigh- borhood of an input to-
class token
d
reshape
F
reshape
N
H
W
ken.
feature tokens
PEG
position encodings
Positional Encoding Generator. PEG is illustrated in Figure 2. To condition local neighbors, we first reshape the flattened input sequence X Bx of DeiT back to X' RBxHxWxC in the
Figure 2.
Schematic illustration of Positional Encoding Generator (PEG). Note d is the embedding size, N is the number of tokens.
the
2-D image space. Then, function (denoted by F in Figure 2) is repeatedly applied to the local patch X' to produce the conditional positional encodings ER implemented with 2-D convolution with kernel & (k 2 3) and الي zero paddings. Note that the zero paddings here are important to make the model aware of the absolute positions, and F can be of various forms such as various types of convolutions and many others.
PEG can be efficiently
3.3 CONDITIONAL POSITIONAL ENCODING VISION TRANSFORMERS
Built on the conditional positional encodings, we propose our Conditional Positional Encoding Vi- sion Transformers (CPVT)). Except that positional encodings conditional, exactly follow
4

---

Published as a conference paper at ICLR 2023
VIT and DeiT to design vision transformers and we also have three sizes CPVT-Ti, CPVT-S and CPVT-B. Similar to the original positional encodings DeiT, the conditional positional encodings also added to the imput sequence, shown in Figure (b). In CPVT, the position where PEG is applied is also important to the performance, which will he studied the experiments.
In addition, both DeiT and ViT utilize extra learnable class token perform classification (i.e,, cls cls_tooeee ssttoeen shown in Figure (a) and (b)). By design, the class token is not translation-invarian although it can learn to be so. A simple alternative is directly replace it with global average ionissaaionnivvriinn resulting in our CVPT-GAP. Together with
pooling (GAP), which is inherently CPE, CVPT-GAP achieves much better image classification performance.
4 EXPERIMENTS
4.1 SETUP
Datasets. Following DeiT (Touvron et al., 2020), use ILSVRC-2012 ImageNet dataset (Deng et al., 2009) with IK classes and 1.3M images to train all our models. We report the results on the validation with 50K images. Unlike ViT (Dosovitskiy et al., 2021), we do not use the much larger undisclosed JFT-300M dataset (Sun et al., 2017).
Model variants. We have three models with various sizes adapt to various computing scenarios. The detailed settings are shown in Table (see B.1). All experiments in this paper are performed on Tesla V100 machines. Training the tiny model for 300 epochs takes about 1.3 days on single node with 8 V100 GPU cards. CPVT-S and CPVT-B take about 1.6 and 2.5 days, respectively.
Training details All the models (except for CPVT-B) are trained for 300 epochs with global batch size of 2048 on Tesla V100 machines using Adam W optimizer (Loshchilov & Hutter, 2019). We do not tune the hyper-parametert and strictly comply with the settings DeiT (Touvron et al., 2020). The learning rate is scaled with this formula (rsclle 0.0005 BatchSiizeghha//11 The detailed hyperparameters are the B.2.
4.2 GENERALIZATION TO HIGHER RESOLUTIONS
As mentioned before, our proposed PEG can directly generalize to larger image sizes without any fine-tuning. We confirm this here by evaluating the models trained with 224 224 images on the 384 x 384, 448 448, 512 512 images, respectively. The results are shown in Table 2. With the 384 384 input images, the DeiT-tiny with learnable positional encodings degrades from 72.2% to 71.2%. When equipped with sine encoding, the tiny model degrades from 72.2% to 70.8%. In constrat, our CPVT model with the proposed PEG can directly process the larger input images, and CPVT-Ti's performance is boosted from 73.4% to 74.2% when applied to 384 x 384 images. Our CPVT-Ti outperforms DeïT-tiny by 3.0%. This gap continues to increase as the input resolution enlarges.
Table 2. Direct evaluation on other resolutions without fine-tuning. The models are trained on 224. A simple PEG of single layer of 3x3 depth-wise convolution used here
Model
Params
160(%)
224(%)
384(%)
448(%) (%)
512(%)
DeiT-tiny
6M
65.6
72.2
71.2
68.8
65.9
DeiT-tiny (sin)
6M
65.2
72.3
70.8
68.2
65.1
DeiT-tiny (no pos)
6M
62.1
68.2
68.6
68.4
65.0
CPVT-TI
6M
66.8(+1.2)
72.4(+0.2)
73.2(+2.0)
71.8(+3.0)
70.3(+4.4)
CPVT-Ti
6M
67.7 (+2.1)
73.4(+1.2)
74.2(43.0)
72.6(+3.8)
70.8(+4.9)
DeiT-small
22M
75.6
79.9
78.1
75.9
72.6
CPVT-S
22M
76.1(+
79.9
78. 78.6(+2.7) 80.4(+1.5)
76.8(+4.2)
DeiT-base
86M
79.1
81.8
79.7
79.8
78.2
CPVT-B
86M
80.5(+1.4)
81.9(+0.1)
82.3(+2.6)
82 62.4(+2.6) 81.0(+2.8)
Insert one PEG each after the first encoder till the fifth encoder
5

---

Published as a conference paper at ICLR 2023
4.3 CPVT WITH GLOBAL AVERAGE POOLING
By design, the proposed PEG is translation-equivariinn (ignore paddings). Thus, if we further use the translation-invariant global average pooling (GAP) instead of the cls_token before the final classification layer of CPVT. CPVT can be translation-invarian, which should be beneficial to the ImageNet classification task. Note the using GAP here results in even less computation complexity because we do not need to compute the attention interaction between the class token and the image patches. As shown Table 3, using GAP here can boost CPVT by more than 1 1. For example, equipping CPVT-Ti with GAP obtains 74.9% top-1 accuracy on the ImageNet validation dataset, which outperforms DeiT-tiny by a large margin (+2.7%). Moreover, even exceeds DeiT-tiny model with distillation (74.5%). In contrast, DeiT with GAP cannot gain much improvement (only 0.4% as shown in Table 3) because the original learnable absolute PE not translation -qqitvrrian and invaaiiiinnnnniintt.... Given the superior performance, hope
thus GAP with the PE is not model can be a strong PE alternative in vision transformers.
Table 3. Performance comparison of Class Token (CLT) and global average pooling (GAP) on ImageNet. CPVT's can be further boosted with GAP
Model
Head
Params
Top-1 Acc
Top-5 Acc
(%)
(%)
DefT-tiny (Touvron et al., 2020)
CLT
6M
72.2
91.0
DeiT-tiny
GAP
6M
72.6
91.2
CPVT-Ti
CLT
6M
73.4
91.8
CPVT-T $
GAP
6M
74.9
92.6
DeiT-small (Touvron et al., 2020)
CLT
22M
79.9
95.0
DeiT-small
GAP
22M
80.2
95.2
CPVT-S
CLT
23M
80.5
95.2
CPVT-S
GAP
23M
81.5
95.7
1: Insert one PEG each after the first encoder till the fifth encoder
4.4 COMPLEXITY OF PEG
Few Parameters. Given the model dimension d. the extra number of parameters introduced by PEG dxl x 222 if we choose depth-wise convolutions with kernel k. If we use separable convolutions,
this value becomes 1(d2 dd) When & 3 and 1, CPVT-Ti (d = 192) brings about 1, 728 parameters. Note that DeiT-tiny utilizes learnable position encodings with 192 14 x 37632 parameters. Therefore, CPVT-Ti has 35, 904 fewer number of parameters than DeiT-tiny. Even using layers separable convolutions, CPVT-Ti introduces only 38952 37632 = 960 more parameters, which is negelectable compared to the 5.7M model parameters of DeiT--iny.
FLOPs. As for FLOPS, layers of k k depth-wise convolutions possesses 14 14 d x x k FLOPS. Taking the tiny model for example, it involves 196 x 192 x 9 = 0.34M FLOPS for the simple case It 3 and i. 1, which is neglectable because the model has 2. IG FLOPs in total.
4.5 PERFORMANCE COMPARISON
Top-1 Top-1Acccunt Accuracy 1%)
80
We evaluate the performance of CPVT models on the ImageNet validation dataset and report the results in Table 4. Compared with DeiT. CPVT models have much better top-/ accuracy with similar throughputs. Our models can en- joy performance improvement when inputs are upscaled without fine-tuning, while DeiT de- grades as discussed in Table 2, see also Fig- ure for clear comparison. Noticeably, Our model with GAP marked a new saae-of--hhe-at for vision Transformers.
78
DelT
76
CPVT (Singla PEG)
74-
CPVT (0-5)
CPVT @384
CPVT-GAP
20
60
80
Params (M)
Figure 3. Comparison of CPVT and DeiT models under various configurations. Note CPVT @ 384 has improved performance. More PEGs can result in better performance. CPVT-GAP is the best.

---

Published as a conference paper at ICLR 2023
Table 4. Comparison with ConvNets and Transformers on ImageNet and ImageNet Real (Beyer et al., 2020). CPVT have much better performance compared with prior Transformers
Models
Params(M) Input throughput"
ImNet top-1 %
Real
top-1
ResNet-50 (He et al., 2016)
25
224
1226.1
76.2
82.5
ResNet-101 (He et al., 2016)
45
224
753.6
77.4
83.7
ResNet-152 (He et al., 2016)
60
224
526.4
78.3
84.1
RegNetY-4GF (Radosavovic et al., 2020)
21
224
1156.7
80.0
86.4
EfficientNet-BO (Tan & Le, 2019)
224
2694.3
77.1
83.5
EfficienNNet-11 (Tan & Le, 2019)
240
1662.5
79.1
84.9
EfficientNet-22 (Tan & Le, 2019)
9
260
1255.7
80.1
85.9
EfficientNet-B3 (Tan & Le, 2019)
3002
732.1
81.6
86.8
EfficientNet-B4 (Tan & Le, 2019)
19
3802
349.4
82.9
88.0
VIT-B/16 (Dosovitskiy et al., 2021)
86
384-
85.9
77.9
ViT-L/16
307
3842
27.3
76.5
DeiT-tiny w/o PE (Touvron et al., 2020)
224
2536.5
68.2
DeiT-tiny (Touvron et al., 2020)
224
2536.5
72.2
80.1
DeiT-tiny (sine)
224
2536.5
72.3
80.3
CPVT-Ti $
2242
2500.7
73.4
81.3
CPVT-Ti-GAP:
224
2520.1
74.9
82.5
DeiT-tiny (Touvron et al., 2020)
224
2536.5
74.5
82.1
CPVT-Tİa
224
2500.7
75.9
83.0
DeiT-small (Touvron et al., 2020)
22
224
940.4
79.9
85.7
CPVT-S
23
224
930.5
80.5
86.0
CPVT-S-GAP!
23
224
942.3
81.5
86.6
DeiT-base (Touvron al., 2020)
86
224
292.3
81.8
86.7
CPVT-B
88
224
285.5
82.3
87.0
CPVT-B-GAPE
88
224
290.2
82.7
87.7
Measured in img/s 16GB V100 GPU in (Touvron al., 2020).
:: Insert one PEG each after the first encoder fifth encoder trained with hard distillation using RegNetY-160 as the teacher.
We further train CPVT-Ti and DeiT-tiny using the aforementioned training settings plus the hard distillation proposed in (Touvron al., 2020). Specifically, we use RegNetY-160 (Radosavovic et al., 2020) as the teacher. CPVT obtains 75.9%, exceeding DeiT-tiny by 1.4%.
4.6 PEG ON PYRAMID TRANSFORMER ARCHITECTURES
PVT (Wang et al., 2021) is a vision transformer with the multi-stage design like ResNet (He et al., 2016). Swin (Liu al., 2021) is follow-up work and comes with higher performance. We apply our method on both to demonstrate its generalization ability.
ImageNet classification. Specifically, we remove its learnable PE and apply our PEG in position 0 of each stage with a GAP head. We use the same training settings to make fair comparison and show the results Table 13. Our method can significantly boost PVT--iny by 3.1% and Swin-tiny by 1.15% ImageNet (c.f. B.5). We also evaluate the performance of PEG some downstream semantic segmentation and object detection tasks (see B.6). Note these tasks usually handle various input resolutions the training because multi-scale data augmentation is extensively used.
ABLATION STUDY
5.1 POSITIONAL ENCODING OR MERELY A HYBRID?
One might suspect that the PEG's improvement comes from the extra learnable parameters intro- duced by the convolutional layers in PEG, instead of the local relationship retained by PEG. One

---

Published as a conference paper at ICLR 2023
way to test the function of PEG is only adding when calculating and K in the attention layer, so that only the positional information PEG is passed through. We can achieve 71.3% top- 1 ac- curacy ImageNet with DeiT-tiny. This is significantly better than DeiT-tiny w/o PE (68.2%) and similar to the one with PEG on Q, K and V (72.4%), which suggests that PEG mainly serves as positional encoding scheme.
We also design another experiment to remove this concern. By randomly-initaaizzing 3x3 PEG and fixing its weights during the train- ing, we can obtain 71.3% accuracy (Table 5), which is much higher (3.1%*) than DeiT with- out any PE (68.2%). Since the weights of PEG are fixed and the performance improvement can only be due the introduced position informa- tion. On the contrary, when we exhaustively use 12 convolutional layers (kernel size being 1, i.e., not producing local relationship) to re- place the PEG, these layers have much more learnable parameters than PEG. However, it only boosts the performance by 0.4% 68.6%.
Table 5. Positional encoding rather than added pa- rameters gives the most improvement
Kernel
Style
Params
Top-1 Acc
(M)
(%)
none
5.68
68.2
fixed (random init)
5.68
71.3
3
fixed (learned init)
5.68
72.3
1 (12 x)
learnable
6.13
68.6
3
learnable
5.68
72.4
Another interesting finding is that fixing a learned PEG also helps training. When we initialize with learned PEG instead of the random values and train the tiny version of the model from scratch while keeping the PEG fixed, the model can also achieve 72.3% top-1 accuracy on ImageNet. This very close the learnable PEG (72.4%).
5.2 PEG POSITION IN CPVT
We also experiment by varying the position of the PEG in the model. Table (left) presents the ablations for variable positions (denoted as PosIdx) based on the tiny model. We consider the imput of the first encoder by index -l. Therefore, position 0 the output of the first encoder block. PEG shows strong performance (~72.4%% when it is placed at [0, 3].
Note that positioning the PEG can have much better performance than positioning it at -1 (i.e., before the first encoder), as shown in Table 6 (left). We observe that the difference between the two situations they have different receptive fields. Specifically, the former has global field while the latter can only see a local area. Hence, they are supposed to work similarly well if we enlarge the convolution's 's kernel size. To verify our hypothesis, we use a quite large kernel size 27 with a padding size 13 position -1, whose result is reported in Table 6 (right). It achieves similar performance to the one positioning the PEG at (72.5%), which verifies our assumption.
Table 6. Comparison different plugin positions (left) and kernels (right) using DeiT-tiny
Posldx
Top-1 (%)
Top-5 (%)
none
68.2
88.7
-1
70.6
90.2
Posldx -1
kernel
Params 5.7M
Top-1 (%)
Top-5 (%)
72.4
91.2
3x3
70.6 72.5
90.2
3
72.3
91.1
-1
27x27
5.8M
91.3
71.7
90.8
10
69.0
89.1
5.3
COMPARISONS WITH OTHER POSITIONAL ENCODINGS
We compare PEG with other commonly used encodings: absolute positional encoding (e.g. sinu- soidal (Vaswani et al., 2017)), relative positional encoding (RPE) (Shaw et al., 2018) and learnable encoding (LE) (Devlin et al., 2019; Radford et al., 2018), shown in Table 7.
DeiT-tiny obtains 72.2% with the learnable absolute PE. We experiment with the 2-D sinusoidal encodings and it achieves on-par performance. For RPE, we follow (Shaw et al., 2018) and set the
8

---

Published as a conference paper at ICLR 2023
Table 7. Comparison of various positional encoding strategies. LE: learnable positional encoding.
RPE: relative positional encoding
Model
PEG Pos
Encoding
Top-1 (%)
Top-5
(%)
DeiT-tiny (2020)
LE
72.2
91.0
DeiT-tiny
2D sin-cos
72.3
91.0
DeiT-tiny
2D RPE
70.5
90.0
CPVT-Ti
0-1
PEG
72.4
91.2
CPVT-Ti
0-1
PEG LE
72.9
91.4
CPVT-Ti
0-1
4xPEG LE
72.9
91.4
CPVT-Ti
0-5
PEG
73.4
91.8
local range Hyper-parameter K as 8, with which obtain 70.5%. RPE here does not encode any absolute position information, see discussion in D.1 and B.3.
Moreover, combine the learnable absolute PE with single-layer PEG. This boosts the baseline CPVT-Ti (0-1) by 0.5%. we use 4-layer PEG, it can achieve 72.9%. If we add a PEG to each of the first five blocks,
obtain 73.4%, which is better than stacking them within block.
CPE is not a simple combination of APE and RPE. We further compare our method with a baseline with combination of APE and RPE. Specifically, we learnable positional encoding (LE) as DeiT at the beginning of the model and supply 2D RPE for every transformer block. This setting achieves 72.4% top-1 accuracy on ImageNet, which comparable Nevertheless, this experiment does not necessarily indicate that our CPE is a simple combination of APE and RPE. When tested on different resolutions, this baseline cannot scale well compared to ours (Table 8). RPE is not able to adequately mitigate the performance degradation on top of LE. This shall be seen as major difference.
single PEG (72.4%).
Table 8. Direct evaluation on other resolutions without fine-tuning. The models are trained on 224x224. CPE outperforms LE+RPE combination untrained resolutions.
Model
Positional Params
160 %)
224(%%
384(%)
448( %)
512(%)
DeiT-tiny (LE+RPE)
40011
65.6
72.4
70.8
68.4
65.6
DeiT-tiny (PEG at Pos 0)
1920
66.8
72.4
73.2
71.8
70.3
PEG can continuously improve the performance if stacked more. We use LE not only at the beginning but also in the next 5 layers to have a similar thing as 0-5 PEG configuration. This setting achieves 72.7% top-1 accuracy on ImageNet, which 0.7% lower than PEG (0-5). This setting suggests that it is also beneficial to have more of LEs, but not as good as ours. It is expected since we exploit relative information via PEGs at the same time.
CONCLUSION
We introduced CPVT, a novel method to provide the position information in vision transformers, which dynamically generates the position encodings based the local neighbors of each input token. Through extensive experimental studies, we demonstrate that our proposed positional en- codings can achieve stronger performance than the previous positional encodings. The transformer models with our positional encodings can naturally process longer input sequences and keep the desired translation equivalence vision tasks. Moreover, our positional encodings are easy im- plement and come with negligible cost. We look forward to broader application of our method in transformer-driven vision tasks like segmentation and video processing.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large-

---

Published as a conference paper at ICLR 2023
scale machine learning. In 12th {USENIX) symposium operating systems design and imple- mentation ((OSDI) 16), Pp. 065-283, 2016.
Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. In International Conference on Learning Representations, 2021. BtRptttps//openzeeee net forum?
id=xTJEN-q91ID.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3286-33 3295,2995,222 2019.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159 2000..
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 2rX1v:2005.14410 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chun jing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv preprint arXiv:20/2.00364, 2020.
Tianqi Chen, Mu Li, Yutian Li. Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv: 112.02151 2015.
Xiangxiang Chu, Zhi Tian, Yuqing Wang. Bo Zhang. Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurlPS 2021, 2021.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. In International Conference on Learning Representations, 2020.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. Advances in Neural Information Processing Systems, 33, 2020.
Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186. 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: / //opereeview ew. net./formm/ d=Yi cbFdNTTy.
Stéphane d' d'Ascoli, Hugo Touvron, Matthew L Leavitt. Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In Inter- national Conference Machine Learning, pp. 2286-2296. PMLR, 2021.
10

---

Published as a conference paper at ICLR 2023
Anmol Gulati, James Qin, Chung- Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution- Convolution-augmented transformer for speech recognition. arXiv preprint arXiv:2005.08100, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhancee bert with disentangled attention. arXiv preprint 1rXiv:2006.3364, 2020.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Aug- ment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. $129-8138 2020.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, lan Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. In Advances in Neural Processing Systems, 2018.
Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646 666-661.25 Springer, 2016.
Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information do convolutional neural networks encode? In International Conference Learning Representations, 2020.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6399-6408, 2019.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Xuanqing Liu, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Learning to encode position for transformer with continuous dynamical model. In International Conference on Machine Learn- ing, pp. 6327-6335. PMLR, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- Learning Representations, 2019. URL Bkg6RicqY7.
Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer. Transformers with convolutional context for arXiv preprint arXiv:1904.1 1660, 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. PMLR.
Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8026-8037, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl blog, 18))9, 2019.
11

---

Published as a conference paper at ICLR 2023
Iijaa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (0428-10436 2020.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani. Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Ad- vances Neural Information Processing Systems, volume 32, pp. 68-80. Curran Asso- ciates, Inc., 2019. URL https: http:::::///cceeednnggo/ hewpooeeeedngngnneer....... cc/ ccc/paper/2219/ii1e 341 ( -awpeeeaaaaaaeaaa..apppep
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume pp. 464-466 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine 2014.
learning research,
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0900 009644 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. Proceedings of the IEEE international conference on computer vision, pp. 843 852, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 1115-6114. PMLR, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint ar 2xXv:20012.1287, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st In- ternational Conference on Neural Information Processing Systems, pp. 6000-6010, 2017.
Benyou Wang. Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen. Encoding word order in complex embeddings. arXiv preprint arXiv:1912. 123333 2019.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122, 2021.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. arXiv preprint 2rXiv:2011.14503, 2020.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.j 15808, 2021.
Baosong Yang, Longyue Wang, Derek Wong, Lidia S Chao, and Zhaopeng Tu. Convolutional self- attention networks. arXiv preprint arXiv:1904.083 2019a.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv: (aop:/906.08233 2019b.
12

---

Published as a conference paper at ICLR 2023
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoe Le. Qanet: Combining local convolution with global self-attention for reading compre- hension. arXiv preprint arXhv:1804009541, 2018.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond em- pirical risk minimization. In International Conference on Learning Representations, 2018. URL https:/
/tpps//e/peeeeieeeee.e net ffdoooummmmmmrrdpp Rb.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-sequueee perspective with transformers. arXiv preprint arXiv:2012. 15840, 2020.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data aug- mentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp- 13001-13008, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 633 633-641, 2017.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: De- formable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021. URL https: : /owpnreveeww..ee//////// net ttt////rrmmmmiig/eeee 6xe.
13

---

Published as a conference paper at ICLR 2023
A TRANSLATION EQUIVARIANCE
The term translation-cquiiariinee means the output feature maps can be equally translated with the input signal. Imagine there is person the lest-top of an image, if the person is moved to the right-bottom, the output feature maps will change accordingly. This property is very important the success of convolution network. Convolution (ignoring paddings), RPE, and self-attention are all translation-equivaraae operations (regardless of their receptive field). It's nontrivial to make absolute positional encodings like DeïT (using learnable positional encoding) translation-equivarian since different absolute positions will be added if the input signal is translated. Note that our method not strictly translation translatio-eqquivrrann because of the zero padding. Instead, it provides a kind of stronger explicit bias towards the translation-qquieee equivariant property.
B EXPERIMENT DETAILS
B.1 ARCHITECTURE VARIANTS OF CPVT
Table 9. CPVT architecture variants. The larger model, CPVT-B, has the same architecture as ViT- B (Dosovitskiy et al., 2021) and DeiT-B (Touvron et al., 2020). CPVT-S and CPVT-Ti have the same architecture as DeiT-small and DeiT--tny respectively
Model
#channels
#heads
#layers
#params
CPVT-Ti
192
3
12
6M
CPVT-S
384
6
12
22M
CPVT-B
768
12
12
86M
B.2 THE HYPERPARAMETERS OF CPVT
As for the ImageNet classification task, we use exactly the same hyperparameters as DeiT except for the base model because it is not always stably trained using AdamW. The detailed setting is shown in Table 10.
Table 10. Syyper-paammeters for ViT, DeiT and CPVT
Methods
VIT
DeiT
CPVT
Epochs
300
300
300
Batch size
4096
1024
1024
Optimizer
AdamW
AdamW
LAMB
Learning rate decay
cosine
cosine
cosine
Weight decay
0.05
0.05
Warmup epochs
3,4
Label smoothing (Szegedy et al., 2016)
0.1
0.1
Dropout (Srivastava et al., 2014)
0.1
X
X
Stoch. Depth (Huang et al., 2016)
X
0.1
0.1
Repeated Aug (Hoffer et al., 2020)
/
Gradient Clip.
X 9/0.5
Rand Augment (Cubuk et al., 2020)
9/0.5
Mixup prob. (Zhang et al., 2018)
0.8
0.8
Cutmix prob. (Yun et al., 2019)
1.0
1.0
Erasing prob. (Zhong et al., 2020)
0.25
0.25
B.3 IMPORTANCE ZERO PADDINGS
We design experiment verify the importance of the zero paddings, which can help the model infer the absolute positional information. Specifically, we use CPVT-S and simply remove the zero paddings from CPVT while keeping all other settings unchanged. Table 11 shows that this can only obtain 70.5%. which indicates that the zero paddings and absolute positional information play important roles in classifying objects.
14

---

Published as a conference paper at ICLR 2023
Table 11. Ablation study ImageNet performance w/ or w/o zero paddings
Model
Padding
Top- Acc(%)
Top-5 Acc(%)
91.2 72.4
CPVT-Tİ
70.5
89.8
B.4 SINGLE PEG MULTIPLE PEGS
We further evaluate whether or not using multi-position encodings can benefit the performance in Table 12. Notice we denote by i-j the inserted positions of PEG which start from the i-th encoder and end the j 1-th one (inclusion). By inserting PEGs five positions, the top-1 accuracy of the tiny model can achieve 73.4%, which surpasses DeiT-tiny by 1.2%. Similarly, CPVT-S can achieve 80.5% turns out more PEGs do help, but up to a level where more PEGs become incremental (0-5 vs. 0-11).
Table 12. CPVT's sensitivity to number of plugin positions
Positions
Model
Params
Top-
Acc
Top-5 Acc
(M)
(%)
0-1
tiny
5.7
72.4
91.2
0-5
tiny
5,9
73.4
91.8
0-11
tiny
6.1
73.4
91.8
small
22.0
79.9
95.0
0-5
small
22.9
80.5
95.2
0-11
small
23.8
80.6
95.2
B.5 CLASSFICATION EVALUATION OF SWIN WITH PEG
We show the validation curves when training Swin (Liu et al., 2021) equipped with PEG in Figure 4. boost Swin-tiny from 81.10% 82.25% (+1.15%1) on ImageNet.
Top-1 Val Accuracy (%
Swin Tiny
Swin Tiny + CPE
50
100
150 Epochs
200
250
300
Figure 4. CPE boosts Swin Tiny on ImageNet by 1.15% top-1 Acc.
B.6 EVALUATION SEGMENTATION AND DETECTION
Semantic segmentation on ADE20K. We evaluate the performance of PEG on the ADE20K (Zhou et al., 2017) segmentation task. Based on the Semantic FPN framework (Kirillov et al., 2019), PVT achieves much better results than ResNet (He al., 2016) baselines. Under carefully controlled settings, PEG further boosts PVT-tiny by 3.1% 1% mloU.
Object detection on COCO. We also perform controlled experiments with the RetinaNet (Lin et al., 2017) framework the COCO detection task. The results shown in Table 13. In the standard 1 schedule, PEG improves PVT-tiny by 2.0% mAP. PEG brings 2.4% higher mAP under the 3 x schedule.
15

---

Published as a conference paper at ICLR 2023
Table 13. Our method boosts the performance of PVT on ImageNet classification, ADE20K seg-
mentation and COCO detection
ImageNet
Semantic FPN ADE20K
RetinaNet COCO
Backbone
Params
Top-
Params
mloU
Params
MAP
MAP
(M)
(%)
(M)
(9))
(M)
1%, 1x
(%. 3x, +MS)
ResNet-18 (He et al., 2016)
12
69.8
16
32.9
21
31.8
35.4
PVT-tiny (Wang al., 2021)
13
75.0
17
35.7
23
36.7
39.4
PVT-tny+PEG
77.3
23
38.0
PVT-iny+GAP
13
75.9
17
36,0
23
36.9
39.7
PVT-iny-PEG+GAP
13
78.1
17
38,8
23
38.7
41.8
PVT-small (Wang al., 2021)
25
79.8
28
39.8
34
40.4
42.2
PVT-smalI+PEG+GAP
81.2
34
43.0
45.2
PVT-Medium (Wang et al., 2021)
44
81.2
48
41.6
54
41.9
43.2
PVT-Mcdium+PEEGGGAT
44
82.7
48
44,9
54
44.3
46.4
B.7 ABLATION ON OTHER FORMS OF PEG
We explore several forms of PEG based on the tiny model, which change the type of convolution, kernel size and layers. The inserted position is O. The result is shown in Table 14. When use large kernel of x 7 or dense convolution, the performance improvement limited. Stacking more layers of wist convolution doesn't bring significant improvement. Therefore, form as our default implementation. It indicates that this design is enough to provide good position information.
the simplest
Table 14. Other forms of PEG. The simple form of a single depth- xx3 good enough.
Variants
Model
Top- I Acc (%)
Depthwise Conv 3x3
tiny
72.4
Depthwise Conv 7x7
tiny
72.5
(Depthwise Conv x +BN+ReLU)
tiny
72.4
Dense Cony 3x3 tiny
72.3
4 (Dense Conv 3x3-BN-)))
tiny
72.5
C EXAMPLE CODE
C.I PEG
In the simplest form, we use single depth-wise convolution and show its usage in Transformer by the following PyTorch snippet. Through experiments, we find that such simple design (i.e., depth- wise 3x3) readily achieves on par or even better performance than the recent SOTAs. We give the torch implementation example in Alg.
D MORE DISCUSSIONS
D.1 WHY RPE WORKS LESS WELL THAN ABSOLUTE PE?
As mentioned in Section 5.3 (main text), RPE is inferior the absolute positional encoding. It because RPE does not encode any absolute position information. Also discussed in Section B.3 (main text), absolute position information is also important needed to determine which object is at the center of the image. Note that there might be multiple objects in image, and the label of image is the category of the object at the center.
for ImageNet classification
is
Additionally, although RPE becomes popular recently, it is often jointly used with absolute posi- tional encodings (e.g.. in CiT (d' Ascoli et al.. 2021)). the absolute position information is leaked in other ways (e.g., convolution paddings in CoAtNet (Dai et al., 2021)). This further sug- gests absolute position information crucial.
16

---

Published as a conference paper at ICLR 2023
Algorithm PyTorch snippet of PEG.
ingert terch
import torch.nt
claas VieienTranstormer:
def init (layers=1,, dim-192, shead-3, img_size=224, patch_size-16);
celf.pos_block (80(dim)
selt.blesss nn. (TTransformerrhooderrraa tdin, nhead, dimad) for range
]ayers]])
self.patch ferrarrde x):
Patchimbed(img_.ii patch_size, dim+4)
B, C, E, R x.shape
patchlaize self.petchienbeedds
_H, _X // pasch_aize,
patch_aize
x torch, ( (selt.cle_toker
X), dim-l)
for 1, bl in enumerate foaall.bbooo f.blocks):
blk(x)
if == 0:
x= salf.pos self.pos_blockit block (x, _H, _)
x(t,
PBGdnn.oodule!:
def init__self,, dim-2 Ntexttse5e k-3);
self.poz = cn. (dim, din, k, k//2, groupsndim) f Only for dama use, more ertective
comp complicated tunctions
der forward(self, &, H, WI:
B, N, C x.shape
cis_teken, feat_
01, xi:, 1:1
feat_tokens (teat_toenaa
2) C,
x selt.pos [teat,
tteat
x. traataaseese
terch.cat (sls_token, unsqueee x), dimml)
return x
D.2 COMPARISON TO LAMBDA NETWORKS
Our work is also related to Lambda Networks (Bello, 2021) which uses 2D relative positional encod- ings. We evaluate lambda module with an embedding size 128, where we denote its encoding scheme as RPF2DD-112. Noticeably, this configuration has about 5.9M parameters (comparable to DeîT-tiny) but only obtains 68.7%. We attribute its failure the limited ability in capturing the correct positional information. After all, lambda layers are designed with the help of many CNN backbones components such as sown-sanpling to form various stages, to replace ordinary convolu- tions in ResNet (He et al., 2016). In contrast, CPVT is wansformer-basee
D.3 QUALITATIVE ANALYSIS OF CPVT
Thus far, we have shown that PEG can have better performance than the original positional encod- ings. However, because PEG provides the position in implicit way, it is interesting can indeed provide the position information as the original positional encodings. Here we inves- tigate this by visualizing the attention weights of the transformers. Specifically, given 224x224 image (i.e. 14x 14 patches), the score matrix within a single head is 196x 196. We visualize the normalized self-attention score matrix of the second encoder block.
PEG
We first visualize the attention weights of DeiT with the original positional encodings. As shown in Figure (middle), the diagonal element interacts strongly with its local neighbors but weakly with those far-away elements, which suggests that DeiT with the original positional encodings learn to attend the local neighbors of each patch. After the positional encodings are removed (denoted by DeiT w/o PE), all the patches produce similar attention weights and fail to attend to the patches near themselves, see Figure (left).
Finally, show the attention weights of our CPVT model with PEG. As shown in Figure 5 (right), like the original positional encodings, the model with PEG can also learn similar attention pattern, which indicates that the proposed PEG provide the position information well.
We illustrate the attention scores in several encoder blocks of DeiT (Touvron et al., 2020) and CPVT in the Fig. 6. It shows both methods lear similar locality patterns. As attention scores computed over the tokens projected in different subspaces (Q and K), they do not necessarily show a strict diagonal pattern, where some may have slight shift, DeiT' in Fig. 66 and CPVT of Fig. right.
17

---

Published as a conference paper at ICLR 2023
0400
Figure 5. Normalized attention scores (first head) of the second encoder block of DeiT without po- sition encoding (DeiT w/o PE), DeiT (Touvron et al., 2020), and CPVT on the same input sequence. Position encodings are key to developing schema of locality in lower layers of DeiT. Meantime, CPVT profits from conditional encodings and follows a similar locality pattern.
DF7
500
210
The second head (encoder 2)
(b) The third head (encoder 2)
ONT
1xT
CNT
(c) The first head (encoder 3)
The second head (encoder 3)
Figure 6. Normalized attention scores (the second and third head) of the second and third encoder block of DeiT (Touvron et al., 2020), and CPVT on the same input sequence. DeiT and CPVT share similar locality patterns that aligned diagonally (some might shift).
D.4 COMPARISON WITH OTHER APPROACHES
We further compare our method with other approaches such as CvT (Wu et al., 2021), Cin (d' Ascoli et al., 2021) and CoAtNet (Dai et al., 2021) on ImageNet validation set in Table 15. To make fair comparisons, we categorize these methods into two groups: plain and pyramid models. Since our models are primarily for plain models, we adapt our methods on two popular pyramid frameworks PVT and Swin. Our CPVT-S-GAP slightly outperforms ViT- by 0.2% with 4M fewer parameters and 0.8G fewer FLOPs. When equipped with pyramid designs, methods still comparable to CvT and CoAtNet
Comparison with DeiT w/ Convolutional Projection. Note CvT uses depth-wise convolution in q-k-u projection which they call it Convolutional Projection. Instead of using it in all layers, we put only one of such design into DeiT-tiny and train such model from scratch under strictly con- trolled settings. We insert it in the position as in our method. The result is shown in Table 16. This CyT-flavored DeiT achieves 70.6% top-1 I accuracy on ImageNet validation set, which is lower than ours (72.4%). Note that q-k-o projections in CvT utilize three depthwise convolutions, therefore, this setting has more parameters than ours. This attests the difference CvT and CPVT, verifying our advantage by learning better position encodings other than inserting them in all layers to have the ability to capture local context and to remove ambiguity in attention.
18

---

Published as a conference paper at ICLR 2023
Table 15. Performance comparison with other approaches such as CvT (Wu al., 2021), Vin (d' Ascoli et al., 2021) and CoAtNet (Dai et al., 2021) on ImageNet validation set. All the models trained on ImageNet-I6 1 dataset and tested on the validation using x24x224 resolution.
Model
Type
Params
FLOPs
Top-1 Acc (%)
DeiT-small (Touvron al., 2020)
Plain
22M
4.6G
79.9
ConViT-S Ascoli et al., 2021)
Plain
27M
5.4G
81.3
EPVT-S-GAP (ours)
Plain
23M
4.6G
81.5
CoAtNet-O (Dai et al., 2021)
Pyramid
25M
4.2G
81.6
CvT-13 (Wu et al., 2021)
Pyramid
20M
4.5G
81.6
PVT-small (Wang et al., 2021)
Pyramid
25M
3.8G
79.8
PVT-smalI-PEG+GAAP Swin-tin wwin-tiny (Liu et al., 2021)
Pyramid
25M
3.8G
81.2
Pyramid
29M
4.5G
81.3
Swin-Gny+PEG+GAP
Pyramid
29M
4.5G
82.3
Table Comparison with positional encoding in CvT (Wu al., 2021) on ImageNet validation set. All the models trained on ImageNet- Ik dataset and tested the validation set using 224 x 224 resolution.
Model
Params
Insert Position
Top-1 Acc
(%)
CPVT-Ti
5681320
72.4
DeiT+ Convolutional Projection
5685352
0
70.6
19


Reviews:
[{"review_id": "YlzMWFMm5f", "reviewer": "Reviewer_tvtM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a simple modification to the positional encoding mechanism of vision transformers. Instead of using fixed sinusoidal embeddings or learned embeddings, the authors introduce conditional positional encodings (CPE), in which positional embeddings for a given patch are calculated as a learned function of that patch's local neighborhood. CPEs can be easily implemented as a single convolution with padding.\n\nComprehensive experiments using DeiT on Imagenet classification show that CPEs significantly improves accuracy and out perform alternative positional encoding methods. Qualitative visualizations show reasonable positional encoding. And various ablations show CPEs efficacy on additional architectures like Swin and PVT, and rule out the possibility that gains are coming simply from adding additional parameters to the model.", "summary_of_the_review": "The authors propose a novel and effective method to replace absolute and learned positional embeddings. Through numerous experiments and ablations they show that this method is able to improve significantly upon existing methods. While the paper is comprehensive, I have a few questions that I would like addressed (see weaknesses section).", "strength_and_weaknesses": "Strengths:\n\nComprehensive experiments demonstrating that PEG outperforms other positional encoding strategies under a variety of situations.\n\nClever ablations show that the gain is not coming from additional parameters introduced by the CPE.\n\nCareful details of hyperparameters and architecture choices in the appendix.\n\nWeaknesses:\n\nOne concern I have is that the majority of experiments were conducted on the \"tiny\" versions of DeiT/ViT/PVT/Swin. Could the benefits of CPEs only be significant in the \"small model\" regime? This would correspond with conventional wisdom that convolutions provide inductive biases that makes training with low data or small models easier. Additional experiments that illustrate how CPEs perform under model scaling should be conducted.\n\nI also don't completely buy the argument that CPEs allow transformers to have longer input sequences. As shown in section 4.4, positional embeddings account for a very small proportion of parameters in a transformer, and if a longer input sequence is desired one could just add more positional embeddings at very little marginal cost. The real bottleneck is the quadratic complexity of transformers, which CVPT does not address.\n\nIn section 3.2 it is noted that zero-padding allows CPEs to know absolute position. My understanding is that this is because the convolution in the PEG learns that zeros => border. However, this only works for patches that are within kernel size of the border. How do the other patches get absolute location information?\n\nThe sentence at the end of page 8 could be clearer.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RlVyUcNJhM", "reviewer": "Reviewer_eRhn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Positional embedding is an important component in transformers. Moreover, generalization ability to longer sequences with proper positional embedding is also hot topic. Current paper proposes a new positional embedding conditioned on the input (PEG) and not only position itself. This is done vie convolutional layer and tested for ViT on ImageNet data. To preserve structure of the initial image sequence of patches is transformed back to image where conv layer is applied and then output is transformed back to patches sequence. Empirically it is shown that PEG has better generalization ability to other image resolutions and improve results over the baseline positional embeddings.", "summary_of_the_review": "The paper proposes to use conv layer as a relative positional embedding, so that positional embedding depends on the input itself not its position (however zero padding also introduces dependence on the position). The work is interesting overall with some results on improved generalization to other resolutions at test time, improved results overall compared to some baselines. Ablations are also somewhat interesting. The main concern I have is the novelty. Convolution pos. embedding was proposed in many prior works so it is not clear what else this paper brings. Also comparison is done with not so many baselines and conViT for example (rel.pos. in self attention directly) performs better than results from the paper. ", "strength_and_weaknesses": "**Strength**\n- Demonstrating that simple conv-based relative positional embedding (PEG) performs well for ViT models on ImageNet and improves over the baselines.\n- Experiments which demonstrate that proposed embedding provides generalization to other resolutions\n- Ablations on the positioning of PEG and number of occurrences.\n\n**Weaknesses**\n- This paper applying convolution to model relative positional embedding in the context of ViT. But this was done in the prior works, see [1-5], in many domains, especially in speech recognition. The only novelty is the proper applying conv to the image patch sequence: we reshape back to image to apply conv and then return back to patch sequence. However this is very natural way to apply I think. What is then the main difference with all these [1-5] prior works? Especially in the context that it is better to apply PEG in several layers.\n- With current formulations of abs.pos. (or rel.pos.) we can simply use MAE SSL while for the proposed method it is non-trivial as changes the encoding based on the input. A bunch of design choices of positional embeddings remain the main vanilla transformer untouched so many interesting SSL can be applied like MAE or position prediction (see Zhai, S., Jaitly, N., Ramapuram, J., Busbridge, D., Likhomanenko, T., Cheng, J.Y., Talbott, W., Huang, C., Goh, H. and Susskind, J.M., 2022, June. Position Prediction as an Effective Pretraining Strategy. In International Conference on Machine Learning (pp. 26010-26027). PMLR).\n- How it is comparable with other baselines with rel.pos. like Convit [6] or variation of abs.pos. e.g. sin.pos. and CAPE [7]?\n\n\nPrior works and some other relevant works to be cited in the paper:\n- [1] Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y. and Pang, R., 2020. Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech 2020.\n- [2] B. Yang, L. Wang, D. Wong, L. S. Chao, and Z. Tu, “Convolutional self-attention networks,” arXiv preprint arXiv:1904.03107, 2019.\n- [3] A. W. Yu, D. Dohan, M.-T. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. V. Le, “Qanet: Combining local convolution with global self-attention for reading comprehension,” arXiv preprint arXiv:1804.09541, 2018.\n- [4] A. Mohamed, D. Okhonko, and L. Zettlemoyer. Transformers with convolutional context for ASR. arXiv, abs/1904.11660, 2019.\n- [5] Baevski, A., Zhou, Y., Mohamed, A. and Auli, M., 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, pp.12449-12460.\n- [6] d’Ascoli, S., Touvron, H., Leavitt, M.L., Morcos, A.S., Biroli, G. and Sagun, L., 2021, July. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning (pp. 2286-2296). PMLR.\n- [7] Likhomanenko, T., Xu, Q., Synnaeve, G., Collobert, R. and Rogozhnikov, A., 2021. CAPE: Encoding relative positions with continuous augmented positional embeddings. Advances in Neural Information Processing Systems, 34, pp.16079-16092.\n- [8] Li, Y., Si, S., Li, G., Hsieh, C.J. and Bengio, S., 2021. Learnable fourier features for multi-dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34, pp.15816-15829\n- [9] KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation https://arxiv.org/abs/2205.09921 NeurIPS 2022\n- [10] (AliBi) Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.\n- [11] (SHAPE) Kiyono, S., Kobayashi, S., Suzuki, J. and Inui, K., 2021, November. SHAPE: Shifted Absolute Position Embedding for Transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 3309-3321).\n- [12] Dai, Z., Yang, Z., Yang, Y., Carbonell, J.G., Le, Q. and Salakhutdinov, R., 2019, July. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 2978-2988).\n- [13] Su, J., Lu, Y., Pan, S., Wen, B. and Liu, Y., 2021. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PGL_TApUK_e", "reviewer": "Reviewer_oX4R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new alternative to positional encoding for vision Transformers. They suggest desired properties of positional encoding for visual tasks based on a comparison of existing positional encodings, such as absolute or relative positional encodings. The suggested conditional positional encoding (CPE) satisfies the requirements. Existing vision Transformers can readily incorporate CPE without requiring any implementation changes. As a result, CPE consistently improves the performance of ViT on image classification, segmentation, and object detection tasks. Furthermore, global average pooling replacing a class token boosts the performance further.", "summary_of_the_review": "The proposal is simple and effective, however does not provide sufficient novelty and technical contribution warranting the paper acceptance. ", "strength_and_weaknesses": "The motivation for work based on the analysis of existing positional encodings is clear, and the paper is easy to understand. Exhaustive experiments are sufficient to show the superiority of the proposed method. The proposed CPE satisfies all the requirements for the desired positional encodings and is easy to combine with standard Transformers. Moreover, it can generalize to the input sequences with arbitrary lengths.\n\nThe paper argues that modeling local relationships is sufficient to replace positional encoding. However, the term \"conditioned\" is ambiguous in the fact that the practical performance gains result from zero paddings. In other words, the suggested CPE depends on zero paddings that serve as an anchor to offer absolute positional information rather than the local relationship. For instance, PEG without zero padding is inferior to PEG with zero padding, implying that the performance gain of CPE is dependent on zero padding. Furthermore, the fact that convolution with zero padding learns implicit positional information is already well-known and trivial (Islam et al., 2020). Various forms other than convolution with zero paddings should be suggested to demonstrate the effectiveness of CPE.\nAlthough Section 5.1 attempted to clarify the suggested CPE as positional encoding, it should be considered as simply combining the original tokens and additional tokens that aggregate local information among pixels using convolution, in the same way as CvT (Wu et al., 2021) uses convolutional projection to aid in local context modeling.\nBesides positional encodings, either absolute or relative, several recent attempts to incorporate positional information differently, such as ConViT (d’Ascoli et al., 2021) and CoAtNet (Dai et al., 2021). It is better to add and compare the experimental result with these approaches.\n\nMinor comments\nComparison of different plugin positions at 0 and -1 results are shown in Table 6 left, not Table 6 right. (Section 5.2)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BPNtMnsrpQ", "reviewer": "Reviewer_Md5K", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for endowing positional information to vision transformer architectures. The proposed conditional positional encoding (CPE) offers translation-equivariance, and it is able to handle longer sequences than the ones encountered during training. In addition, it also captures absolute position, addressing shortcomings of relative PE.", "summary_of_the_review": "Based on the discussion above, I propose acceptance as the method proposed is novel and offers significant improvements to the classification performance of transformers. Nevertheless, there are some weaknesses that if they were properly addressed, the paper would become much stronger.\n\n## Comments after the rebuttal\nI enjoyed the very constructive discussion that took place during the rebuttal. I think that the paper has become much stronger from the technical point of view, and I continue to support acceptance of this work. Regarding the comments about novelty, I agree that some contributions stem from prior work (e.g. Islam et al., 2020), but I still find the focus of this work somewhat different, and thus useful to the community. \n\nBesides the addition of the new results, I agree with other reviewers that the text also needs to be revised based on the discussions that took place. From my side, I would recommend the following:\n1) focus more on the ability of the method to generalize to resolutions not seen during training,\n2) elaborate on the requirements stated in the beginning of Section 3.2, based on the discussion that took place,\n3) include some comments about comparison of the proposed method with other encoding schemes in Section 2, \n4) revise this sentence \"... and F can be of various forms such as a various types of convolutions and many others\", since all PEG forms considered are based on convolutions (e.g. remove \"and many others\" or add details about specific forms and/or their properties)\n", "strength_and_weaknesses": "## Strengths\nThe proposed conditional positional encoding scheme is able to handle longer sequence than the ones used during training, which is a significant restriction of learnable PE schemes. Moreover, similarly to learnable PE schemes, it offers positional equivariance and absolute positional information. Absolute positional information has been shown to be useful for visual tasks. This is a shortcoming of the relative PE methods. Based on these observations, the proposed method offers an advantage over both learnable and relative PE schemes. The PE is achieved by applying equivariant functions (such as convolutions), called Positional Encoding Generator (PEG), on the reshaped feature tokens.\n\nThe experiments cover both the ability of CPE to handle larger input sequences (higher resolution images), while also showing that the resulting encoding correctly captures positional information. A solution based on global average pooling, instead of the cls_token is also discussed.\n\n## Weaknesses\nThe paper mentions that PEG can have \"various forms\", however in the experimental evaluation a single PEG structure is considered (single layer 3x3 depth-wise convolution). It would be interesting to discuss some alternatives and, possibly, their performance.\n\nAdditionally, a single visual transformer method is considered, DeiT. Although, it is expected that similar improvements would occur for other transformer methods, it would be interesting to include at least one more, e.g. the original ViT, T2T [R1] or XCiT [R2]. \n\nAlso, some additional positional encoding schemes as [R3] and [R4] could be discussed in the related work. Although they mainly address NLP problems, they also offer favourable properties for long sequences.\n\n[R1] Yuan et al., (2021). Tokens-to-Token ViT: Training vision transformers from scratch on imagenet. ICCV\n\n[R2] Ali et al., (2021). XCiT: Cross-covariance image transformers. NeurIPS\n\n[R3] Wang et al., (2020). Encoding word order in complex embeddings. ICLR\n\n[R4] Su et al., (2021). Roformer: Enhanced transformer with rotary position embedding. arXiv", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster