Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new alternative to positional encoding for vision Transformers. They suggest desired properties of positional encoding for visual tasks based on a comparison of existing positional encodings, such as absolute or relative positional encodings. The suggested conditional positional encoding (CPE) satisfies the requirements. Existing vision Transformers can readily incorporate CPE without requiring any implementation changes. As a result, CPE consistently improves the performance of ViT on image classification, segmentation, and object detection tasks. Furthermore, global average pooling replacing a class token boosts the performance further.
strength_and_weaknesses: The motivation for work based on the analysis of existing positional encodings is clear, and the paper is easy to understand. Exhaustive experiments are sufficient to show the superiority of the proposed method. The proposed CPE satisfies all the requirements for the desired positional encodings and is easy to combine with standard Transformers. Moreover, it can generalize to the input sequences with arbitrary lengths.

The paper argues that modeling local relationships is sufficient to replace positional encoding. However, the term "conditioned" is ambiguous in the fact that the practical performance gains result from zero paddings. In other words, the suggested CPE depends on zero paddings that serve as an anchor to offer absolute positional information rather than the local relationship. For instance, PEG without zero padding is inferior to PEG with zero padding, implying that the performance gain of CPE is dependent on zero padding. Furthermore, the fact that convolution with zero padding learns implicit positional information is already well-known and trivial (Islam et al., 2020). Various forms other than convolution with zero paddings should be suggested to demonstrate the effectiveness of CPE.
Although Section 5.1 attempted to clarify the suggested CPE as positional encoding, it should be considered as simply combining the original tokens and additional tokens that aggregate local information among pixels using convolution, in the same way as CvT (Wu et al., 2021) uses convolutional projection to aid in local context modeling.
Besides positional encodings, either absolute or relative, several recent attempts to incorporate positional information differently, such as ConViT (d’Ascoli et al., 2021) and CoAtNet (Dai et al., 2021). It is better to add and compare the experimental result with these approaches.

Minor comments
Comparison of different plugin positions at 0 and -1 results are shown in Table 6 left, not Table 6 right. (Section 5.2)

clarity,_quality,_novelty_and_reproducibility: - Clarity
The paper is written clearly and it is easy to understand.

-Quality
The quality of the paper is good.

-Novelty
The novelty and technical contribution of the approach is incremental. As mentioned in the weakness, the fact that convolution with zero padding learns implicit positional information is trivial (Islam et al., 2020). Despite the use of the name CPE, the actual implementation uses depth-wise convolution to describe the local pattern. It is considered comparable to CvT except for the location of the convolution.

- Reproducibility
All the details of implementations and evaluations are provided sufficiently to reproduce the reported results.

summary_of_the_review: The proposal is simple and effective, however does not provide sufficient novelty and technical contribution warranting the paper acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Positional embedding is an important component in transformers. Moreover, generalization ability to longer sequences with proper positional embedding is also hot topic. Current paper proposes a new positional embedding conditioned on the input (PEG) and not only position itself. This is done vie convolutional layer and tested for ViT on ImageNet data. To preserve structure of the initial image sequence of patches is transformed back to image where conv layer is applied and then output is transformed back to patches sequence. Empirically it is shown that PEG has better generalization ability to other image resolutions and improve results over the baseline positional embeddings.
strength_and_weaknesses: **Strength**
- Demonstrating that simple conv-based relative positional embedding (PEG) performs well for ViT models on ImageNet and improves over the baselines.
- Experiments which demonstrate that proposed embedding provides generalization to other resolutions
- Ablations on the positioning of PEG and number of occurrences.

**Weaknesses**
- This paper applying convolution to model relative positional embedding in the context of ViT. But this was done in the prior works, see [1-5], in many domains, especially in speech recognition. The only novelty is the proper applying conv to the image patch sequence: we reshape back to image to apply conv and then return back to patch sequence. However this is very natural way to apply I think. What is then the main difference with all these [1-5] prior works? Especially in the context that it is better to apply PEG in several layers.
- With current formulations of abs.pos. (or rel.pos.) we can simply use MAE SSL while for the proposed method it is non-trivial as changes the encoding based on the input. A bunch of design choices of positional embeddings remain the main vanilla transformer untouched so many interesting SSL can be applied like MAE or position prediction (see Zhai, S., Jaitly, N., Ramapuram, J., Busbridge, D., Likhomanenko, T., Cheng, J.Y., Talbott, W., Huang, C., Goh, H. and Susskind, J.M., 2022, June. Position Prediction as an Effective Pretraining Strategy. In International Conference on Machine Learning (pp. 26010-26027). PMLR).
- How it is comparable with other baselines with rel.pos. like Convit [6] or variation of abs.pos. e.g. sin.pos. and CAPE [7]?


Prior works and some other relevant works to be cited in the paper:
- [1] Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y. and Pang, R., 2020. Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech 2020.
- [2] B. Yang, L. Wang, D. Wong, L. S. Chao, and Z. Tu, “Convolutional self-attention networks,” arXiv preprint arXiv:1904.03107, 2019.
- [3] A. W. Yu, D. Dohan, M.-T. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. V. Le, “Qanet: Combining local convolution with global self-attention for reading comprehension,” arXiv preprint arXiv:1804.09541, 2018.
- [4] A. Mohamed, D. Okhonko, and L. Zettlemoyer. Transformers with convolutional context for ASR. arXiv, abs/1904.11660, 2019.
- [5] Baevski, A., Zhou, Y., Mohamed, A. and Auli, M., 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, pp.12449-12460.
- [6] d’Ascoli, S., Touvron, H., Leavitt, M.L., Morcos, A.S., Biroli, G. and Sagun, L., 2021, July. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning (pp. 2286-2296). PMLR.
- [7] Likhomanenko, T., Xu, Q., Synnaeve, G., Collobert, R. and Rogozhnikov, A., 2021. CAPE: Encoding relative positions with continuous augmented positional embeddings. Advances in Neural Information Processing Systems, 34, pp.16079-16092.
- [8] Li, Y., Si, S., Li, G., Hsieh, C.J. and Bengio, S., 2021. Learnable fourier features for multi-dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34, pp.15816-15829
- [9] KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation https://arxiv.org/abs/2205.09921 NeurIPS 2022
- [10] (AliBi) Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.
- [11] (SHAPE) Kiyono, S., Kobayashi, S., Suzuki, J. and Inui, K., 2021, November. SHAPE: Shifted Absolute Position Embedding for Transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 3309-3321).
- [12] Dai, Z., Yang, Z., Yang, Y., Carbonell, J.G., Le, Q. and Salakhutdinov, R., 2019, July. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 2978-2988).
- [13] Su, J., Lu, Y., Pan, S., Wen, B. and Liu, Y., 2021. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864.

clarity,_quality,_novelty_and_reproducibility: **Clarity & Quality**
Paper is well written, the idea of the paper presented well. There is some conceptual disagreement with some statement (see below) but apart from that everything is clear. There are a bunch of experiments and ablations are done to show effectiveness of proposed method and investigation its attention maps and where we need to place it as well as generalization to other resolutions at test time.

**Novelty**
I think the work lacks of some novelty. As discussed above there are a bunch of works which use convolution as a relative positional embedding in the past. One of the interesting findings is generalization ability which probably was not considered for convolutional augmented transformer models / positional embedding before. Statement about translation invariance I think is weak as we anyway break it by zero-padding.

**Reproducibility**
- Did authors reimplement DeiT or take its offical code and integrated PEG? 
- Could authors confirm that the only thing varied in the experiments is positional embedding while training pipeline and other parameters are kept the same?
- A bunch of details are given in the main text and in the Appendix.

**Some other general comments**
- "but it degrades the performance without fine-tuning as later shown in our experiments": there is a bunch of works which showed the same trend, e.g. [7]
- sec 3.1 discussion of no pos.emb. : I think it is not "dramatically degradation" of performance. It is very competitive. Also could authors provide no pos.emb. results in Table 2 to know how it is comparable in generalization regime as it should generalize for all resolutions well?
- typo "in A" -> "in Appendix A"
- In Table 1 is 2D RPE actual 2D or it is stacking of x and y coordinates?
- Sec 3.2: points (1) and (3) contradict each other. 
- "CVPT_GAP is uttermost translation-invariant" -> but we have zero padding which breaks translation invariance.
- Sec 4.2 - what is happening with lower resolution? do authors have generalization to lower resolution too?
- There are a bunch of papers proposed not to use CLS token or do some other variant where/how to add it so it is not novel.
- why translation-invariance should be beneficial for ImageNet particularly? We have only centered objects there, maybe it doesn't matter much for this particular dataset.
- Table 2: it is better to have it as graph and put the table into Appendix. Many numbers are hard to parse together.
- Table 3: do authors have results for DeiT-S with GAP? 
- typo: "extra number of parameter introduce" -> "extra number of parameter introduced"
- why in Figure 3 why right plot has shifted diagonal by >10 positions? is it effect of PEG module?
- Figure 4: is 384 train resolution or test one?
- did authors have results for base and small architectures to probe generalization to different resolutions?
- Main text, Append A3: I do not fully agree with statement that absolute positional embedding is not translation equivariant. E.g. sin.pos. embedding is designed in the way that if we learn unitary matrix then we can query neighbors, so we can emulate the rel.pos. embedding in the end. Also pushing on the translation equivariance property a lot and then in the propose method which is also not translation equivariant due to the padding and mentioning that actually abs.pos. embedding is needed too for good performance is very confusing.
- Table 10: could authors clarify how they perform ablation without padding?
- Speed of the proposed embedding is investigated but in the end it is better to insert it in several layers - what is then the slowdown? 
- Table 12: can we have benchmark without GAP part too?
- Appendix D1: I think it should be reformulated in the way that ImageNet contains mainly centered one object and that is why even no pos.emb. works very well (bag of patches). Also it was shown that patches themselves give huge inductive bias and for particular ImageNet we probably do not need relative position information, so abs.pos. works well too. If we look at the 2d relpos in lambda networks it performs on pair with no pos.emb. which support above statement. So more careful formulation and reasoning is needed in the text.

summary_of_the_review: The paper proposes to use conv layer as a relative positional embedding, so that positional embedding depends on the input itself not its position (however zero padding also introduces dependence on the position). The work is interesting overall with some results on improved generalization to other resolutions at test time, improved results overall compared to some baselines. Ablations are also somewhat interesting. The main concern I have is the novelty. Convolution pos. embedding was proposed in many prior works so it is not clear what else this paper brings. Also comparison is done with not so many baselines and conViT for example (rel.pos. in self attention directly) performs better than results from the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a method for endowing positional information to vision transformer architectures. The proposed conditional positional encoding (CPE) offers translation-equivariance, and it is able to handle longer sequences than the ones encountered during training. In addition, it also captures absolute position, addressing shortcomings of relative PE.
strength_and_weaknesses: ## Strengths
The proposed conditional positional encoding scheme is able to handle longer sequence than the ones used during training, which is a significant restriction of learnable PE schemes. Moreover, similarly to learnable PE schemes, it offers positional equivariance and absolute positional information. Absolute positional information has been shown to be useful for visual tasks. This is a shortcoming of the relative PE methods. Based on these observations, the proposed method offers an advantage over both learnable and relative PE schemes. The PE is achieved by applying equivariant functions (such as convolutions), called Positional Encoding Generator (PEG), on the reshaped feature tokens.

The experiments cover both the ability of CPE to handle larger input sequences (higher resolution images), while also showing that the resulting encoding correctly captures positional information. A solution based on global average pooling, instead of the cls_token is also discussed.

## Weaknesses
The paper mentions that PEG can have "various forms", however in the experimental evaluation a single PEG structure is considered (single layer 3x3 depth-wise convolution). It would be interesting to discuss some alternatives and, possibly, their performance.

Additionally, a single visual transformer method is considered, DeiT. Although, it is expected that similar improvements would occur for other transformer methods, it would be interesting to include at least one more, e.g. the original ViT, T2T [R1] or XCiT [R2]. 

Also, some additional positional encoding schemes as [R3] and [R4] could be discussed in the related work. Although they mainly address NLP problems, they also offer favourable properties for long sequences.

[R1] Yuan et al., (2021). Tokens-to-Token ViT: Training vision transformers from scratch on imagenet. ICCV

[R2] Ali et al., (2021). XCiT: Cross-covariance image transformers. NeurIPS

[R3] Wang et al., (2020). Encoding word order in complex embeddings. ICLR

[R4] Su et al., (2021). Roformer: Enhanced transformer with rotary position embedding. arXiv
clarity,_quality,_novelty_and_reproducibility: Based on the discussion above, I find hat the proposed PE scheme has novelty. The paper is well written and easy to follow. Although the method description is quite brief, it is clear as it is based on simple concepts. For the same reason, it should be easy to reproduce the results. Also, a brief code snippet is provided as reference. 
summary_of_the_review: Based on the discussion above, I propose acceptance as the method proposed is novel and offers significant improvements to the classification performance of transformers. Nevertheless, there are some weaknesses that if they were properly addressed, the paper would become much stronger.

## Comments after the rebuttal
I enjoyed the very constructive discussion that took place during the rebuttal. I think that the paper has become much stronger from the technical point of view, and I continue to support acceptance of this work. Regarding the comments about novelty, I agree that some contributions stem from prior work (e.g. Islam et al., 2020), but I still find the focus of this work somewhat different, and thus useful to the community. 

Besides the addition of the new results, I agree with other reviewers that the text also needs to be revised based on the discussions that took place. From my side, I would recommend the following:
1) focus more on the ability of the method to generalize to resolutions not seen during training,
2) elaborate on the requirements stated in the beginning of Section 3.2, based on the discussion that took place,
3) include some comments about comparison of the proposed method with other encoding schemes in Section 2, 
4) revise this sentence "... and F can be of various forms such as a various types of convolutions and many others", since all PEG forms considered are based on convolutions (e.g. remove "and many others" or add details about specific forms and/or their properties)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a simple modification to the positional encoding mechanism of vision transformers. Instead of using fixed sinusoidal embeddings or learned embeddings, the authors introduce conditional positional encodings (CPE), in which positional embeddings for a given patch are calculated as a learned function of that patch's local neighborhood. CPEs can be easily implemented as a single convolution with padding.

Comprehensive experiments using DeiT on Imagenet classification show that CPEs significantly improves accuracy and out perform alternative positional encoding methods. Qualitative visualizations show reasonable positional encoding. And various ablations show CPEs efficacy on additional architectures like Swin and PVT, and rule out the possibility that gains are coming simply from adding additional parameters to the model.
strength_and_weaknesses: Strengths:

Comprehensive experiments demonstrating that PEG outperforms other positional encoding strategies under a variety of situations.

Clever ablations show that the gain is not coming from additional parameters introduced by the CPE.

Careful details of hyperparameters and architecture choices in the appendix.

Weaknesses:

One concern I have is that the majority of experiments were conducted on the "tiny" versions of DeiT/ViT/PVT/Swin. Could the benefits of CPEs only be significant in the "small model" regime? This would correspond with conventional wisdom that convolutions provide inductive biases that makes training with low data or small models easier. Additional experiments that illustrate how CPEs perform under model scaling should be conducted.

I also don't completely buy the argument that CPEs allow transformers to have longer input sequences. As shown in section 4.4, positional embeddings account for a very small proportion of parameters in a transformer, and if a longer input sequence is desired one could just add more positional embeddings at very little marginal cost. The real bottleneck is the quadratic complexity of transformers, which CVPT does not address.

In section 3.2 it is noted that zero-padding allows CPEs to know absolute position. My understanding is that this is because the convolution in the PEG learns that zeros => border. However, this only works for patches that are within kernel size of the border. How do the other patches get absolute location information?

The sentence at the end of page 8 could be clearer.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clear.

Quality: The paper is of high quality.

Novelty: Method is fairly novel, but is somewhat related to architectures that combine convolutional and self-attention layers.

Reproducibility: The method seems reproducible from the information given in the paper. The appendix is particularly well written in this respect.
summary_of_the_review: The authors propose a novel and effective method to replace absolute and learned positional embeddings. Through numerous experiments and ablations they show that this method is able to improve significantly upon existing methods. While the paper is comprehensive, I have a few questions that I would like addressed (see weaknesses section).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This is a borderline paper. there has been lot of back and forth in discussion among reviewers. On one hand, the paper does a good job in comparing the role of convolutions in encoding positional information. On the other hand, claims about novelty are false and many techniques are already established in earlier works. However the reviewers still believe there is value in the experiments presented in the paper and I agree with them. One of the reviewers strongest objections against paper are based on the novelty of the approach, author's acknowledge that in the response to an extent and this should be corrected in the final version with referencing the right earlier works for the method and limit the paper claims to experimentation and comparisons. I suggest acceptance assuming authors will correct this in final version.

Pros
 - Extensive experimentation and comparison with existing approaches.
- Evaluation of different resolution inputs.

Cons 
- Limited novelty.
- Overclaiming contributions/ missing comparisons to some existing works.

I strongly suggest rewriting the final version as per one of the reviewers suggestions - text itself should be rewritten to not claim novelty of the convolution layer with padding as a positional embedding, but rather in-depth analysis of conv block, analysis if it is learning both relpos and abspos, and generalization to other resolutions not used during training.
justification_for_why_not_higher_score: Limited novelty and unclear contributions.
justification_for_why_not_lower_score: Extensive experimentation of convolution as position encoding.
Decision: Accept: poster