Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new metric to detect "Position-information Pattern from Padding" which is an artifact resulting from different types of padding applied throughout a convolutional neural network.
strength_and_weaknesses: Strengths: The paper takes a systematic approach to the task. 

Weakness: I don't believe the authors present a convincing argument for why this problem is important and their experiments don't demonstrate the value of solving this problem in terms of model generalization.

clarity,_quality,_novelty_and_reproducibility: The paper makes the statement "padding can implicitly provide a network model with positional information. Such positional information can cause unwanted side-effects by interfering and affecting other sources of position-sensitive cues" and cites a variety of papers. The problems that the border information causes are not obvious to me so I would expect a more solid experimental setup where the proposed metric helps uncover and avoid poor generalization.
summary_of_the_review: The paper presents a new metric but fails to demonstrate the value beyond an assumed importance based on existing published work. The paper is potentially impactful but it is not clear to me what the negative issues are with the model knowing where the border is. I think this work should directly have experiments with that generalization goal in order to demonstrate relevance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the effect of convolutional neural networks encoding position information due to padding. A new metric for measuring and visualizing the encoded positional information named Position-information Pattern from Padding (PPP) was proposed. The paper then discusses some of the disadvantages of the existing metrics based on PosENet and tests in F-Conv and show through experiments PPP has the potential to fix these disadvantages.
strength_and_weaknesses: # Strength

1. Defining a novel metric for quantifying position information due to padding.

2. Systematically discussed the issues with existing metrics

# Weakness

1. PPP metrics are potentially biased by the model architecture and parameters.

2. Only some basically networks were used in analysis so generalizability to more SOTA network architectures still needs exploration.

3. I wonder if the authors can illustrate some of undesired effects of padding on certain real world examples in addition to just showing the edge view of those blue-yellow blocks. 
clarity,_quality,_novelty_and_reproducibility: The paper is written in a clear fashion. Code will be released so I assume the reproducibility is good given that most experimental setups are straightforward.
summary_of_the_review: The paper investigates an interesting topic of position information related to padding and includes a lot of well-thought-out discussions. The experiments seem to indicate some practical value of the proposed PPP metric although the experimental setup might be a little simplified.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the Position-information Pattern from Padding (PPP) encoded in neural networks. PPP is defined as difference between algorithmically and optimally padded activations, then SNR and MAE are computed on the difference to quantify how much PPP exists. The result shows a pre-trained CNN model contains a lot of position information. Comparing with the two previous studies, this paper is claimed to be more robust and reliable for PPP evaluation.

strength_and_weaknesses: Strength: This study investigates a fundamental question that affects all convolutional neural networks used in computer vision. The proposed method seems to have lower variability than the other methods.

Weaknesses:
This study did not discuss how the PPP affects CNNs in practice. The proposed padding did not compare against other commonly used paddings in vision application to show its efficacy, which is not convincing. The padding scheme designed in Section 2.4 is to weaken PPP detected by eq3, which is based on an optimal padding solution as shown in eq2. It is better to discuss why cropping from a bigger picture can be considered a theoretical optimal padding. Having more content(pixels) could (slightly)deliver more semantic information, but I do not think the information can be associated with PPP, and the proposed padding is also algorithmical solution.
clarity,_quality,_novelty_and_reproducibility: The proposed padding strategy is sampling from a normal distribution which is learned within a sliding window, its novelty is limited. More importantly, its efficacy has not been validated in practice comparing to other padding strategy. 
summary_of_the_review: Considering all of the strengths and weaknesses, I would not recommend this paper to be published at this moment.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper argues based on prior work that the encoding of positional information from padding, abbreviated PPP by convolutional neural networks is an undesired property that can have unwanted side-effects. The degree to which PPP is learned as a consequence of different padding techniques is investigated and a new way to quantify it is proposed. In addition a new padding technique is explored, which could help to lower PPP.
strength_and_weaknesses: Strengths
---------
- Investigating PPP makes sense and the method proposed to do this is relatively simple.

- The experiments are thorough and relatively large in scale showing clear results.

- An ability to reliably quantify PPP is a requirement for further understanding of its consequences and possibly ways of mitigating its effects.

Weaknesses
----------
- Some table and figure captions are lacking details and/or are confusing.
  - Figure 1 is placed and referenced so early in the manuscript that it is hard to understand what it contains. E.g. what is the sampling process, how are the two signals generated, how is PP quantified? What is the right column in relation to the other four?
  - I suggest you explain what the numbers indicate in table 1. Also this is a very large table with a lot of information that is only referenced once in text and in a single short sentence. Is all of this information really necessary?
  - Table 2 is also relatively large, and I find it hard to deduce from the text what numbers should be compared and how this relates to the title in bold. Consider helping the reader a bit more.
  - Table 3, here the caption seems a bit misleading. "The results show a consistent and significant increment of PPP is developed through the model training". Somehow this seems more fitting as a description of Figure 5, as table 3 seems to be about pretraining and it only compares pretraining with no pretraining.

- As far as I understand Table 2 it shows the different possitional encoding measurement metrics (SPC, MAE, and PPP-MAE) for the different padding schemes and models. In section 4.2 it is concluded that because the standard deviations are larger than the differences between the padding variants, the PosENet based metrics cannot be used to say much about the benefits of these padding methods. While this may be true, as far as I can deduce, this does not necessarily indicate a problem with the PosENet based metrics, it may also just be the case that PPP varies a lot over training runs.

- An intuitive explanation for the equations given (1, 2, and 3) would help understanding.

 Other smaller annoyances
  - Consider avoiding the abbreviation of method names in the abstract or write what they are.
  - "Padding, one of the most fundamental components in neural network architectures, has received much less attention than other modules in the literature", what do you mean by this? What is a module in this context?

clarity,_quality,_novelty_and_reproducibility: - On clarity, despite the simplicity of the equations and the experiments conducted, I find the work relatively hard to read. More details is given under weaknesses.

- On quality, despite its clarity issues, I like the work. The manuscript could be improved if the method was used to propose a padding technique that actually made a difference in terms of the investigated metrics.

- On novelty, I don't find it particularly surprising that neural networks can and do learn to detect padded values in order to inform about locations. I also think the idea that the proposed randn padding scheme would alleviate this to be a long shot. It seems to me that in order to really be able to really match the distribution in the padded regions more advanced techniques are needed. One could imagine using adversarial learning or other methods for generating realistic image data. Despite this, the work is looking at a previously largely overlooked aspect of neural network architecture design choice and making meaningful contributions to quantifying how well these choices work.

- On reproducibility, code and data collection scripts will be made available on acceptance, which would make it easier to reproduce.



summary_of_the_review: While I like the paper, I feel it needs more work in terms of clarity and presentation to make it acceptable.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a mew metric to detect information about position via padding artifacts.

Strengths: The paper is systematic in its discussion of existing metrics and proposing a new one.

Weaknesses: The paper is challenging to read, and the experimental validation would benefit from illustrating the undesired effects and the method's alleviation of these on real world cases.
justification_for_why_not_higher_score: Three out of four reviewers recommend that the paper is not ready for publication, and I do not have sufficient reason to go against their recommendation.
Decision: Reject