Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper studies the effect of padding on artefacts in CNN feature maps and performance on image classification and object detection. It convincingly makes the case that these artefacts have a significant detrimental effect on task performance, e.g. leading to blind spots / missed detections of small objects near the image border. It also studies the effect of uneven padding in downsampling layers, where the padding may only affect some sides of the image and not others, depending on the image size. A condition is presented for when this does / does not occur. The effect of different padding methods is also studied from the perspective of foveation by computing the number of paths from an input pixel to the output. A number of practical recommendations are given.

The paper is well written. It contains lots of details that are relevant to CNN architecture design, especially when the appendix is taken into account. Proposed fixes are simple and produce a very significant improvement in performance on imagenet classification and object detection, so are likely to be adopted by practitioners. 

The paper states:
"It is evident that the 1-pixel border variations in the second map are caused by the padding mecha-nism in use. This mechanism pads the output of the previous layer with a 1-pixel 0-valued border inorder to maintain the size of the feature map after applying a 3x3 convolutional kernel. The maps inthe first layer are not impacted because the input we feed is zero valued. Subsequent layers, however,are increasingly impacted by the padding, as preceding bias terms do not warrant 0-valued input."
It would be interesting to know if batchnorm or some other kind of normalization might mitigate this issue, because if the feature map is constant but non-zero, normalization will make it all zero. Of course this will not hold for non-zero (natural) inputs, but it would still be interesting to see a discussion on the effect of (batch) normalization on padding artefacts.


Typos
Section 4: "To serves"
Section 5: RseNet

------ 
UPDATE
I have now read the other reviews, author response and updated paper, and have decided to maintain my rating.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper does empirical analysis on an object detector, esp. how it fails due to slight shift of objects in videos. By using different padding schemes for conv filters, the authors find the padding scheme is the root for such failures.

Update after rebuttal:
The author responses and manuscript revisions have addressed most of my concerns. Now I lean towards acceptance. 

Reasons for score:
1. This paper is not written professionally. The introduction is very short. The following sections are more like experimental notes than a technical paper. 
2. The analysis is mostly based on a single use case, i.e. an SSD object detector for traffic lights. This study is highly insufficient. More use cases and application scenarios should be investigated.
3. Experimental evaluation is also highly insufficient. Only changing the input image size by 1 pixel doesn't reveal much. A lot of ablation studies and changing of padding schemes should be evaluated, preferably on a few datasets of different tasks (classification, detection, segmentation, etc.).

Other comments:
I agree that padding could lead to biases when doing downsampling using stride > 1, as illustrated in Fig. 6. However, this seems to be easily fixed by doing random image flipping as one of the data augmentations? It would even this bias. So this issue may not be practically severe, although it is worth study per se.

Minor issues:
The PDF file of this paper seems to have issues. The printed characters are blurry, and acrobat reports errors when open it. But some other PDF readers can open it.
 
review_title: Quite empirical analysis (rating raised after rebuttal)

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary of Submission:

This submission points out that 0-padding in convolutional networks induces areas where responses for object detections are significantly decreased. This leads to misdetections of small objects. The submission studies the case of traffic lights. Analysis of filter response maps at various layers of the network and analysis of the average filter weights show the effect of the 0-padding. Making the padding symmetric and using mirror or reflection padding are shown to tackle the issue.

Update after Rebuttal:
I'd like to thanks the authors for addressing the comments really thoroughly. The additional experiments using anti aliased networks are indeed very interesting. I agree with AnonReviewer4 that some of the material in this submission is probably known to people already. However, I do strongly believe that many people are unaware of the impact and importance of these effects and therefore I believe publication of this content is important.

Strengths:

- Padding is omni-present in modern CNN architectures and its effect is not often analyzed. This submission contains a thorough analysis, points out the problem with 0-padding and shows how various alternatives show significant improvement.

- Especially the analysis of the asymmetric padding that leads to asymmetric filters  in Section 5 is very interesting.

- The text is well written and easy to follow.


Weaknesses:

- The introduction describes how the issue with the 0-padding leads to drastic differences in object detection between frames in a video. There has been work [39] which also showed that aliasing in CNNs causes these effects which does not seem to be discussed in this manuscript. What would happen to the effects of the 0-padding if an anti-aliased network would be used?

- The experiments are focused on traffic lights. Given the generality of the issue presented in the manuscript I feel that additional experiments with other small objects or related problems could have been included.


Explanation of Rating:

This submission describes a problem with 0-padding that significantly impacts an important problem and shows how to mitigate the situation. Therefore I recommend accepting this submission.


Minor Comment:

Page 5: layer in RseNet -> ResNet
review_title: This submission describes how 0-padding impacts small object detection and presents how to mitigate the problem.

final_score: rating: 7: Good paper, accept
review_content: This paper addresses a commonly neglected factor in CNN design, namely the consequences of typical compromises made in padding the feature maps for convolutions and resamplings. On the downside, much of the material is known in some form in literature or practice, but on the other hand, the paper has value in systematically bringing together and analyzing these effects, and does identify and quantify effects that are apparently novel.

It is a common belief that choices like padding are minor technical details, which, even when sloppy, the network will just somehow learn to tolerate. This is to some extent true and it is remarkable how well CNN's work even when apparently crippled by questionable design. However, there is a cost and the network may need to make sub-optimal compromises to work around these issues. The paper's demonstration of lopsidedness of the learned filters is apparently novel and drives this home in an interesting way: to handle the uneven padding, the network must make a compromise where the filter kernels are slightly distorted so that they can still make a decent job of both the interior while tolerating the drastically broken boundary. However this "decent job" is not necessarily as good as it could be if the boundary defect was absent -- here the paper proposes a very simple change that demonstrably both improves performance and eliminates the kernel distortion.

I do wonder how these findings compare to e.g. the results of the explicit padding paper, which tackles these issues head on in a principle yet more complicated manner. It seems like e.g. the poor performance of standard padding in the Gaussian blur example in that paper could be explained by effects similar to the aforementioned distortion of the filters in the interior.

The paper also identifies the phenomenon of arbitrarily distorted predictions made in vicinity of padding boundaries, leading to incorrect predictions when small objects occupy this space. While unsurprising, apparently this issue has received little attention in literature, perhaps in part because e.g. imagenet data is labeled based on what is at the center of the image. In any case, higher detection accuracy is reported for many tasks when the boundary handling is improved.

In addition to these the paper analyzes the number of connections between output and input pixels when various padding strategies are used and makes some recommendations about the most balanced choices. The foveation patterns of most coarse padding methods do seem unhealthy and it's plausible that it's connected to the arbitrary nonlinear distortion in prediction scores near boundaries. While a thorough connection is not necessarily established, this is at least hinted by Fig 5b.

The paper is clearly written and easily understandable.

Overall, while the findings of the paper are arguably fairly simple and elementary compared to many works in ICLR, bringing attention to this sort of effects is valuable and I did learn something from reading the paper, so I would lean slightly towards accepting it.

_Update after rebuttal_: The authors have been very forthcoming with further analyses in the rebuttals. I believe the findings deserve to be published and that bringing attention to these issues is a good thing, and accordingly I have bumped my rating up a bit.
review_title: Relatively interesting study on elementary but often neglected shortcomings in CNN design

Meta-review: This submission explores how certain common padding choices can induce spatial biases in convolutional networks. It looks into alternative padding schemes which mitigate these issues and demonstrates significant performance improvements in widely used convnets. Reviewers generally agreed that this is an important point that should be more widely understood in the community, and that the proposed changes are relatively simple to adopt, so this work is likely to be impactful. Most reviewers thought the paper was well-written, describing the problem well, and the analysis well-executed. Most reviewers acknowledged that most of the weaknesses described in their initial reviews were well-addressed by the authors' responses and manuscript updates. Given the strength of the analysis and the impact for many practitioners, I recommend the submission be accepted with a spotlight presentation.
Decision: Accept (Spotlight)