Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel metric for measuring and visualizing the encoded positional information arising due to padding in Convolutional Neural Networks. They formally define the encoded information as PPP (Position-information Pattern from Padding) which measures the distance between feature maps arising from an algorithmic padding (such as zero padding) and an optimal notion of padding (i.e. the ground truth pixels that would be around the image). They demonstrate that PPP is a more reliable metric (low variance) than alternate metrics and hence it is useful for measuring the effectiveness of methods debiasing PPP due to padding. They show that PPP is a representation that the network develops as a part of its learning process.
strengths_and_weaknesses: ## Strengths
1. The paper is well-written, and contains sufficient details.
2. The proposed problem of determining the amount of position information encoded due to padding is important as it can be interfering to actual cues and degrade model performance.
3. The proposed metric is motivated well, and the analysis of PPP and prior works is thorough.
4. The idea of using an optimal padding scheme to quantify PPP is sound and reveals consistent patterns in experiments.
​
## Weaknesses
1. The evaluation is conducted on only one dataset of just 480 images. A more thorough investigation can be done by incorporating diverse datasets and observing the consistency of PPP as a metric over PosENet.
2. Fig. 5 and Section 4.3 demonstrate that PPP is a representation that the network favorably develops as a part of its learning process. Can the authors add some discussion on the implications of this? What new insights can be drawn from this experiment?
questions: What can be more quantitative metrics to justify a good PPP metric? Currently, the majority of the focus is on the large standard deviation of PosENet. However, there must be other ways to compare which metric is a better representative of measuring the result of padding.
limitations: The authors already mention an important limitation of PPP being linked to the architecture of the network as it is evaluated on the outputs of a model.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Authors present a novel metric for detecting the presence and quantifying the strength of positional information encoding due to padding. They define PPP as a spatial statistic, i.e., expected absolute difference between algorithmically padded and optimally padded images’ activations, and summary statistics are defined from the spatial ones, i.e., SNR and MAE for PPP. In addition, they propose a new padding strategy, which aims to preserve local variability around a boundary when padding. The proposed metric is applied on pre-trained networks as well as those that are trained from scratch at different epochs. Results suggest that the position information encoding gets stronger with training, and very strong for pre-trained models. Compared to two existing alternatives, the proposed metric has lower variance, and therefore seems to be a more reliable metric. 
strengths_and_weaknesses: Strengths:
1.	The investigated concept is intriguing and potentially affect all application of CNNs. 
2.	The proposed measure is very intuitive and easy to compute. 
3.	Results suggest that the proposed metric shows lower variability compared to alternatives, and show that almost all networks encode position to some extent. 
4.	Results showing emphasized position encoding for pre-trained models is interesting. 

Weaknesses: 
1.	A solution towards removing the position encoding is not discussed. 
2.	Importance of quantifying the strength of PPP is not clear to me. 
3.	Authors state that reliable PPP metrics are important for understanding PPP effects in different tasks. While this point is surely intriguing, such an explanation or understanding is not explicitly given in the article. Can the authors explicitly explain what type of understanding one reaches by looking at the PPP maps? 
4.	The conclusion of the article remains a bit vague. While the proposed metrics have some more desirable attributes, value of these attributes for applications is unclear to me.  How will this actually improve the practice or our understanding? 

questions: 1.	My main suggestion is to better motivate the investigation and the developed metrics for applications of CNNs. How will this technology actually be used? A roadmap towards improving the current state of things using the PPP is unclear. 
2.	What is the value of quantifying the strength of position encoding?  Better explanation towards this end would substantially improve the article. 
3.	It is shown in figure 5, that proposed metrics are different at different layers. In this setting, it may be important for authors to provide some guidelines telling which layer one should look at. More specifically, difference (quantified with PPP) at which layer would cause the biggest problem for prediction accuracy? 

limitations: Yes, they have.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This study investigates position information patterns learned due to the use of padding. A new evaluation method has been proposed by taking the difference between an optimal padding strategy and the “algorithmically-padded features”. The difference pattern becomes more recognizable when more samples are applied. Then signal-to-noise ratio and mean absolute error are proposed to measure the layer-wise difference between the two paddings. This whole system is claimed to be a better measure for the position information.

strengths_and_weaknesses: Strengths: The proposed idea of randn padding is plausible, a window is used to compute the mean and deviation values within. Then we can sample padding values based on the learned normal distribution. Addressing the effect of strided convolution also makes the investigation more comprehensive. The proposed randn padding can deliver less position information than other padding strategies.
Weakness: This study also has some essential problems that need to be addressed, please see the questions below.

questions: The two metrics SNR-PPP and MAE-PPP, proposed in this paper are questionable. The use of signal-to-noise ratio and mean absolute error is not new, the new metric is essentially the proposed “optimal” padding, which can be used to compare against other paddings and compute differences, including SNR and MAE or any other metrics. Then let’s focus on the basic question, what the “optimal” padding is.

This proposed position measure is conducted at feature level, which is more difficult to draw conclusions than those studies at output level. Accuracy on the BHV test from [1] can be considered a measure against human perception, which is a golden standard in CNN understanding, e.g., why the classifier behaves differently than human and what could be the factor behind (absolute position). Similarly, the artificial ground-truth used in [2] can validate CNNs in a human readable way, if the model can output the wanted pattern based on position. This type of validation has also been applied in other CNN understanding papers[3,4].

The measure proposed in this paper is computed at the feature level, which is not human readable, or is still under exploration. Then the defined “optimal” padding becomes a golden standard to compare, which is very questionable. The paper also agrees with the “optimal” padding shown in Eqs. (1) and (2) “In practice, such an optimal-padding scheme is difficult to achieve.” I personally doubted if such padding exists in practice. Without further explanation, Sec2.2(Fig 1., line 81) directly shows the visual difference between the “optimal” and the “algorithmical” padding, this is really important but looks confusing. I can only guess the “optimal” refers to the randn padding from sec 2.4(please correct me if I am wrong), and the large difference to zero-padding is not surprising. The padded value by zero-padding is very different from other content-based, e.g., circular, reflect, replicate, which has been already studied in previous studies[1,5]. (In addition, the finding in Fig 3 and Table 1 that background color matters can already be found in previous study[5].) More importantly, I am still seeking an explanation why this randn padding could be the “optimal” to compute the two metrics, it looks more like another well-designed “algorithmical” padding. The “optimal” golden standard needs to be set and studied thoroughly, then the resulting findings (Table 2 and 3) can be analyzed reliably. Otherwise, it makes less sense no matter how high or low values a metric can achieve.

Therefore, I suggest a more thorough study on how the “optimal” padding can be achieved before computing metrics on top. And based on Table 1, the proposed randn padding behaves similarly to the replicate padding, which can be considered a special case drawn from the same distribution as randn padding.

Based on the questions above, I would suggest this study can be further improved for a better presentation and less confusion, the use of SNR and MAE is not main focus. More importantly, the design of "optimal" padding should be better addressed.

[1] Kayhan, Osman Semih, and Jan C. van Gemert. "On translation invariance in cnns: Convolutional layers can exploit absolute spatial location." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[2] Islam, Md Amirul, Sen Jia, and Neil DB Bruce. "How much position information do convolutional neural networks encode?." arXiv preprint arXiv:2001.08248 (2020).
[3]Geirhos, Robert, et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." arXiv preprint arXiv:1811.12231 (2018).
[4]Geirhos, Robert, Kristof Meding, and Felix A. Wichmann. "Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency." Advances in Neural Information Processing Systems 33 (2020): 13890-13902.
[5]Islam, Md Amirul, et al. "Position, padding and predictions: A deeper look at position information in cnns." arXiv preprint arXiv:2101.12322 (2021).
limitations: The paper can be further polished and re-organized to reduce misleading. The applicability of randn can be further discussed, the previously studied findings can be omitted.
review_title: null

Meta-review: metareview: The three reviewers all leaned towards rejection for this paper. One reviewer was concerned with the relatively small number of images used in the experiment and how valid the conclusions can be from that for PPP as a better metric. Another confusion was over how optimality in padding can be defined. This was important because the MAE and SNR measures were based off of this.
confidence: Certain
Decision: Reject