Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary of the paper:

This submission studies the effect of zero/one padding in convolutional networks when using images that are pasted on a canvas. The main thesis is that zero padding induces absolute position information and that this leads to better performance in many cases. The study is based on segmentation and classification where the input image is placed on a background canvas.

Update after Rebuttal:

I do appreciate the argument given in appendix A.1 about the relative position that changes with cropping and that boundary information may get lost. However, it is not clear to me if that change in relative position matters because the network could take this into account. However, the issue that the feature maps between the shortcut and residual connection do not align with the bilinear interpolation seems to be much harder take into account for the network. To me it seems an actual example on the ImageNet classification (i.e. Table 2) that shows that the degradation in performance is not due to the resampling/misalignment of the feature maps in the ResNet would be very important. Moreover, a related problem is that it seems plausible that a network could extract position information from the spatially varying misalignment of the feature maps (in the image center there is no misalignment and on the border there is 1px (for 3x3 conv). The amount would reveal the position). Therefore an experiment that shows that this does not happen in practice would also be important. This is the same major concern I share with AR4 even after the rebuttal.


Strengths:

Padding is omi-present in current convolutional and "fully" convolutional architectures but its effect is not studied well.
The manuscript clearly states the hypothesis that are investigated.

Weaknesses:

The motivation for using canvases in the way done within this manuscript is unclear to me. The introduction states this is due to the fact that images need to be rectangular to be processed with a CNN. While in practice mostly rectangular images are used with CNNs it seems to me that convolutions could also be applied to non-rectangular images if required (with custom implementation). However, more importantly the manuscript does not just pad images to be rectangular but instead pastes a much smaller image onto a large canvas. This is not clearly motivated in the current form of the manuscript as far as I can tell.

In Section 3 it is unclear if one network is trained for all grid resolutions or if the networks are trained per grid resolution.

Unclear how no padding is conducted in ResNet. Section 3 indicates that a standard ResNet-18 is modified to a no padding version by removing the padding from the convolutions and using bilinear resize to match the sizes. The way I understand this is that for example an NxN image is convolved without padding and hence the output will be an (N-k) x (N-k) image, now either the output of the convolution or the original NxN image is resized to match the other resolution before adding the two. This seems like a very problematic implementation as in either version the spatial alignment between the two is not preserved and importantly the misalignment is spatially varying while the convolution kernel will not vary spatially. In my mind a no padding implementation of a ResNet would simply crop the original image in the consistent way to how the convolution does not produce outputs for the boundary regions. If my understanding of this implementation is correct I feel that this is a major flaw in this submission.

Do the no padding segmentation networks contain upsampling operations and or u-net type skip connections? How are the no padding versions of these operations implemented? Especially for the city scapes segmentation results it is unclear to me how a full image segmentation was obtained without using any form of padding or related mechanism of retaining the information around the image boundary.

In Table 3 the white padding seems to contain stronger location information than the black padding which contradicts the main thesis of the submission that says black padding contains more location information. However, Table 3 uses different grid sizes for black and white padding. Are the different grid sizes the reason for this discrepancy? What is the motivation for using different grid sizes?

One point I find missing in this submission is that no padding networks will have spatially smaller feature maps which means less information might be stored in the intermediate layers. This could be a reason why the no-padding versions of Table 4 underperform compared to reflection padding.

What happens for networks where the input/output resolution varies, i.e. "fully" convolutional networks? In these cases absolute position information intuitively would prevent generalization to different image sizes.

Reason for score:

This submission discusses an important and relevant topic. However, at this point I believe that there are potentially flaws in the experiments which might significantly influence the findings. Therefore the current rejection rating. If the rebuttal could further discuss the points in the weaknesses section and explain in detail how the no-padding versions were implemented that would be helpful to further assess the correctness of the above assessment. 

review_title: This manuscript investigates the important topic of padding in CNNs. The review points out some potential issues in the experimental evaluation.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The topic of the paper is interesting and important. The padding strategy seems to be a small but largely overlooked aspect in CNN learning. While many papers attempt to reduce/improve the effect of position in images. This paper gives a different perspective on the padding patterns that essentially causes these position sensitivity. 

The paper is easy to follow: the hypothesis at the beginning are important and interesting. And the hypothesis are quite clearly verified with experiments.

The Hypothesis raised in this paper are important factors for training networks for various tasks. especially for H5 and H2. And the paper gives useful conclusions based on experiments.   H4 is a bit unnecessary as it is common knowledge that as layer goes deeper, it has a larger receptive field thus even boarder information will be included in the center regions. 

The current analysis are based on CIFAR, it would be more convincing if these hypothesis are further validated on larger datasets such as imagenet or coco (for segmentation). 

A minor suggestion, it would be more informative if the author in related work/analysis compare with the positioning methods used in transformers. 
review_title: Interesting paper on boarder padding analysis

final_score: rating: 7: Good paper, accept
review_content: This paper studies the effect of padding on the Convolutional Neural Network. The authors try to answer the following questions: 1) what type of padding provides the most position information, 2) does the background value affects model accuracy when processing a patch on a canvas, 3) which part of the image suffers the most from the boundary effect, and 4) whether the position information provided by padding improves or degrades model performance. In order to answer these questions, the authors design multiple tasks and perform extensive experiments. The empirical results show that: 1) zero-padding provides the most location information compared with other common padding methods, 2) the background value of the canvas do affects the accuracy when processing a patch, 3) the boundary effect is not specific to the image boundary---the model is affected by the boundary over the entire image, and 4) the effect of padding on model accuracy depends on the task.

This paper studies an empirically important yet under studied problem and is well written. The target questions / hypothesis are well motivated, and the experiment designs are reasonable and help to answer the target question. The extensive experiments provide valuable insight into the model properties and may benefit both fundamental CNN research and CNN applications in the future. Nevertheless, there are some problems that are not sufficiently addressed in the paper and may be improved.
First of all, an implicit hypothesis in the experiments is that the boundary effect is generic to the architecture. However, my conjecture is that how much padding affects the model strongly depends on the overall model architecture such as the readout architecture (e.g. avg pooling, max pooling, fc-layer), the receptive field of the neurons, etc. The correlation between model architecture and boundary effect is not discussed in the paper, and the implementation details are not clearly described to estimate how much it may affect the conclusion.

Second, as an empirical study with newly defined tasks, it is important to demonstrate whether the results are significant. More specifically, it would be helpful if the authors can 1) include intuitive baselines for reference performance (e.g. a model on (k+1)^2 patches instead of k^2), 2) show the learning curve / variance etc. to demonstrate whether the difference is meaningful, and 3) provide a more complete description about the experiment protocol (eg. how were the meta-parameters determined, whether the results are affected by the meta-parameters).

Third, some of the arguments are based on qualitative results. However, the figures are not clear enough and it is hard to follow the arguments. Also, qualitative results do not provide strong enough evidence for these arguments, and it would be more persuasive if the authors can provide quantitative analysis instead and use qualitative results as a support.
Finally, given that a learned representation can be distributed, I don't think the dimensionality estimation (Table 3) alone provides useful information. The argument implicitly assumes that 1) all other dimensions are independent of location information, 2) the location information cannot be reduced to a lower dimensional space. These assumptions are not verified.

The rebuttal provide valuable information that strengthens the original paper. While some of my concerns are not resolved in the rebuttal, they require additional experiments and may be beyond the scope of rebuttal.
review_title: This paper provides valuable insight about padding in CNNs through extensive experiments and is well written.

final_score: rating: 3: Clear rejection
review_content: The paper seeks to understand how different padding modes and canvas colors affect the performance of a convolutional neural network in classification and semantic segmentation tasks. The question seems somewhat strange - surely a network should be able to counteract a consistent change in padding or background color. If there was a strong effect it would be an interesting finding indeed. Unfortunately, the paper fails to convince that any but the most obvious effects exist.

There is an underlying assumption that knowledge of absolute spatial location should be helpful in some tasks and unhelpful in others. Why should absolute position matter for CIFAR-10 classification or semantic segmentation? Before jumping to this conclusion, this should be measured directly, e.g., by embedding horizontal and vertical positions as constant feature maps at every convolution. 

The findings in Section 2 are not very surprising. Assuming that images are not mostly black, the greatest contrast is found against a constant black background, and partial convolution only enhances this effect - it is equivalent to zero padding but brightening the image artificially when the convolution kernel overlaps it only partially.

Table 2 shows that classification results vary wildly between padding and no padding, when comparing black canvas and zero padding. This makes no sense to me. Especially in large grids, a picture embedded somewhere else than the edge of a black canvas, without padding, should be fundamentally the same as having just the picture and zero padding. ResNet-18 that the paper uses has a global pooling layer at the end, ensuring that global position information is certain to be lost at that point at the latest.

I'm extremely concerned about the paper's decision to use bilinear interpolation to match the resolution with skip connection in a ResNet architecture, when padding is not used. My understanding is that the convolution branch output is upscaled to the skip branch resolution before composition. This introduces a scaling factor to the convolution branch, and therefore moves the contents of feature maps towards the edges. The composite will thus not have the features from the two branches line up with each other  anymore. This shifting effect is largest near the edges and smallest near the center, which matches with the observed performance characteristics. The subsequent convolution layers may be able to undo some of this shifting, but only at the cost of location-dependent kernels that are tailored to fit the offset  caused at different parts of the image. I'm inclined to believe that many of the alleged problems with non-padded convolutions are due to this design. For example, Table 10 suggests that the padding itself is a much more important factor than if padding is with zeros or a reflection, which makes little sense if locating the boundary were the critical aspect.

The paper is mostly clearly written, but some of the terminology is confusing. Section 3.1 talks about location-dependent classification, whereas the goal is to have classification be independent of location.  There are other cases of similar dependence vs independence confusion. What Horizontal and Gaussian mean in Table 1 should be explained. On page 7, formula $\frac{k-1}{k}$ should read $\frac{k^2-1}{k^2}$. In Figure 6, it is unclear how the activations are mapped to grayscale values. Regardless, high or low activations do not mean much with ReLU activation function, as any scaling can be counteracted in the weights of the subsequent convolution kernels, so the relative strengths of activations may not carry much information. In Figure 12, are the w/pad and w/o pad flipped? The effect seems inverse to Figure 1.

In summary, I find the findings questionable and too counterintuitive - e.g., the big difference in classification power between black background vs. zero padding when a small image is embedded in a large black canvas. The paper fails to demonstrate convincingly that these findings are real instead of artifacts of the experimental design.

Pros: mostly clearly written, many experiments.

Cons: dubious choices for network design and experiments, counterintuitive results that are not properly
explained or analyzed.

** UPDATE after reading other reviews, author responses, and revised paper **

I'm unconvinced by the arguments related to the resizing of feature maps between ResNet branches. The added experiment in A.1 does not address the question of misaligning the features between skip branch and convolution branch. 

Figure 8 suggests that the alignment with input is well preserved with resizing (i.e., Fig. 8(a, d) look similar). The illustration is misleading because it uses a kernel with constant weights and *extremely* oversaturates the colors. In other circumstances, we would see that each corner in Fig. 8(c, d) is different. This is because Fig. 8(b) will contain four identical blobs after the convolution, and Fig. 8(c) is by definition a central crop of Fig. 8(b). Thus, the bits we see in Fig. 8(c) are different quadrants of this blob, and they are in fact all different. This leads to the misalignment between skip and convolution branches, so that subsequent processing will see different data at each of the four corners. The figure makes it look like there are only small differences by oversaturating the colors to the point that only the footprint of the convolution can be seen. Using a non-symmetrical convolution kernel would have also highlighted the differences.

The experiment also misses the larger point: The different architectures may cause differences in the results that exceed the effects of the padding itself. The paper claims to measure the latter, but I believe both AR2 and I are concerned that the paper may be measuring the architectural effects instead. As this was my main concern about the experimental design, my confidence in the paper's results is not increased, and my rating is not changed. If anything, I'd be inclined to lower my score because of how misleading the added Fig. 8 is.
review_title: Questionable design and too counterintuitive results

Meta-review: This paper explores the effects of padding in convnets used for various visual recognition tasks (classification, segmentation). This is an important and relevant design choice that is often overlooked, as noted by reviewers. However, I share the concerns of AR2 & AR4 with the evaluation. The design of the ResNet variant used for the "No Pad" baseline seems potentially fatally flawed: the bilinear upsampling used to match the feature map sizes for the residual addition results in a misalignment of the inputs with the outputs, which potentially explains the performance degradations seen throughout most experiments, as opposed to (or perhaps in addition to) the lack of positional information and border effects resulting from the NoPad scheme that is claimed as the reason for the performance drop. It is true that how to do this is an open question, as the authors argue in their added Appendix A.1, but I nonetheless share the reviewers' skepticism that the chosen approach will result in a meaningful comparison of the effect of padding and border effects. In fact, the results in Table 5 on texture recognition seem to suggest that "No Pad" approach may indeed be flawed, given that "No Pad" performs the worst, while "Reflect" padding performs best, even though both methods share the property that the network should have difficulty inferring positional information. Given the reliance on this dubious baseline throughout the results, I can't recommend the submission for acceptance in its current form. However, I still appreciate the direction of this work and hope the authors will consider resubmitting it after revising it in order to make the evaluation more convincing based on the reviewers' feedback.
Decision: Reject