Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides an analysis of the effect of training convolutional neural networks with a particular image size in the generalisation to different image sizes at inference time, as a consequence of downscaling operations within the architecture and specifically the padding necessary for enabling flexible input sizes. The analysis demonstrates that standard methods exhibit drops in performance in a periodic fashion with the input size, as best illustrated in Figure 1, and provides an explanation for the effect. Further, the paper presents simple mechanism to overcome this issue: randomising the location of the padding during training to break the bias due to a systematic location, as is common practice.
strength_and_weaknesses: ### Strengths

The problem tackled by this paper is conceptually simple and therefore easy to follow. The paper is well written, the figures are very informative and the experimental design helps sheds light on the aspects that are analysed. Furthermore, the conclusions are supported by the results. Finally, the paper not only analyses a pitfall in the way convolutional neural networks are typically trained, but it also presents a method to overcome the problem. Therefore, I have an overall good impression of the paper.

### Weaknesses

My main concern is related to the novelty of the results presented in the paper. The paper seems to be an incremental extension of Alsallakh et al. (2021), which analysed spatial biases in CNNs due to padding. This paper sets the focus on the input size and the pooling operation ("mind the pool" vs. "mind the pad", in the titles), but fundamental issue underlying the pitfalls analysed in both papers is, my opinion, the same: padding. The issue with pooling is really padding, not pooling itself, as indeed explained by the authors. In fact, the problem seems to be identical regardless of the type of downscaling (due to pooling or striding), precisely because it is caused by padding. All elements---padding, input size, downscaling---are related, and in fact Alsallakh et al. (2021) already the effect of input size and pointed out at sharp differences when the input size differed by just one pixel. 

Several elements of analysis in the present paper were already discussed in Alsallakh et al. (2021), for instance the asymmetry of the kernels (Figure 4 in "mind the pool" and Figure 6 in "mind the pad") or the shift of the receptive field (Figure 8 and 7 respectively). Furthermore, other previous work, such as Shocher et al. (2020) had set the focus specifically in the downscaling operation, as pointed out here. In this regard, a significant part of the analysis offers little novelty.

Unrelated to the above, one question I have regarding the proposed method to remove the bias towards the training image size is what the influence is of fixing one side for the padding at test time. I expect little influence, but having to stick to just one seems arbitrary and it would be nice to show experimentally whether the choice has no, little or moderate influence in the performance. As a suggestion, the curves in Figure 1 and equivalent figures could display a confidence interval obtained from testing with multiple options for padding at test time.

Finally, as a minor comment, when mentioning fully-convolutional architectures, I would cite Springenberg et al. (2015) (cited later in the paper) as well as Long et al. (2015). (The reviewer is not related to any of these papers).
clarity,_quality,_novelty_and_reproducibility: As mentioned in the previous section, the paper is clearly written and the quality of the figures certainly help in illustrating the analysis and presenting the results. Some minor elements that may be improved are the following:

* In the second paragraph of Section 3.1, note that "specially" is written twice instead of "spatially".
* The third paragraph of Section 3.2 starts with "It is evident that [...]" without referencing any figure or table.

The quality of the paper is good, as discussed before, and I am not aware of any code provided alongside the submission for reproducibility.

I have discussed novelty at length in the previous section.
summary_of_the_review: My overall impression of this paper is positive, as it provides a comprehensive analysis of a pitfall in CNNs trained in the currently standard way, even if typically the input size is fixed as well during test time, as well as a simple enough solution to overcome the problem. Furthermore, the paper is well written. Nonetheless, it is also fair to point out that the novelty is limited in view of previous work such as Alsallakh et al. (2021) or Shocher et al. (2020), for instance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates a problem I would not have thought about -- CNNs overfitting to input size due to the pooling arithmetic involved -- this in itself is an extremely exotic thought! The authors show that due to the unconsumed padding, often the networks trained on specific input sizes will not generalize or extrapolate They proceed to propose a simple solution called Spatially-Balanced Pooling (SBPool) that mitigates this phenomenon, by randomly alternating the location of the unconsumed line of padding. The results are extremely strong for ImageNet classification and Semantic segmentation across architectures. They conclude with a nice analysis and discussion alongside strong baselines which fail to fix the phenomenon.


(The brevity of the review should not be taken negatively, it is due to the clarity of the paper in its ideas, hypothesis and experimentation). 


strength_and_weaknesses: My summary of the paper notes the strengths of paper. It is an extremely simple idea that works very well in practice. Experiments and discussion are through along with the exposition which helped set up the problem in the first place. 

While I do not have technical qualms, I am extremely disappointed by the inaccessible plots and figures. They are of extremely poor quality and would benefit from a revision with every single plot and figure being redone and rendered with vector graphics. 

I strongly urge the authors to fix this and I would be willing to give it a higher score after that and after discussions with other reviewers. 
clarity,_quality,_novelty_and_reproducibility: Highly reproducible. 
summary_of_the_review: See above. I really want the authors to submit a revision with fixed figures. It reflects poorly to have this as a major issue when the idea and the paper itself are extremely well done!
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper looks at how convolutional neural networks can overfit on the input size. This is first shown experimentally (figure 1) and then the authors hypothesize that it is due to downsampling (figure 2). Simply put, there is no way to apply a kernel of size 3 (for example) with a stride of 2 over an input of size 4 in such a way that it is not biased. They validate this hypothesis (figures 3 and 4) experimentally.

In section 3 the authors propose a solution: In cases where the kernel size is not a divisor of the padded input size, they divide the padding between both sides randomly. They show experimentally that this solution reduces sensitivity to input sizes and finish the paper with a detailed discussion of related work.
strength_and_weaknesses: I believe this paper highlights a simple problem, explains the cause, and proposes a simple solution. Although the problem and solution are both simple, the authors did a good job at being very exhaustive in their experiments and discussion of related work, which I believe gives the paper sufficient content to be considered for acceptance.

Most of my concerns are relatively minor. One minor issue I see is that the paper repeatedly talks about "unused padding" or "pooling" as the reasons for overfitting. But the input size overfitting is unrelated to padding, which the authors acknowledge in section 4.1. The term "pooling" on the other hand seems a bit confusing, because that is a word I associated with, e.g., average- or max-pooling, and not with a regular downsampling layer (i.e., a kernel which uses a stride greater than 1). Perhaps the authors could clarify these nuances earlier on.

Minor comments:

* There's a bracket missing in section 2.4 after "(Figure 1"
* In section 2.2 it says that "it is not immediately obvious why the other peaks recur at an interval of 32". I read this statement as "we don't know why this happens", but then in section 2.3 it turns out that the authors do know. Perhaps this sentence can simply be deleted.
* The term "overfitting" is a bit confusing, because the model's best results are often not for input sizes that it was trained on (e.g., in figure 1 the best results are for 256 or 288, not for 244), so it's strange to say that it has overfit to the training input size. I guess the precise statement would be that the model has overfit to the border conditions caused by downsampling? Perhaps the wording should reflect this (e.g., in the "Our contributions" part in section 1).
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and clear. The quality of the work is good.

The novelty of this work is limited (both the problem and solution are quite simple), but this is compensated by an exhaustive set of experiments and thorough discussion of the problem, which still provides value to the community.

As the experiments mostly use off-the-shelf models which can be readily retrained on different input sizes, I imagine the results to be easily reproducible.
summary_of_the_review: A simple solution to the simple problem of downsampling layers in neural networks overfitting to the boundary conditions, but given that this is a well-written paper with an exhaustive set of experiments, I think this paper should be considered for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers a problem of applying a trained convolutional networks to images of different sizes of input than the one they have been trained on. They note the degradation in performance and carefully analyse and provide explanation for this problem as well as solution. The method is simple and easy to understand. Some writing can be improved as pointed out by the reviewers. 
justification_for_why_not_higher_score: The method is nice and useful but not novel enough or of high enough importance to warrant higher score.
justification_for_why_not_lower_score: Well done, with a simple method that works.
Decision: Accept: poster