Mind the Pad -- CNNs Can Develop Blind Spots
Authors: Bilal Alsallakh
Narine Kokhlikyan
Vivek Miglani
Jun Yuan
Orion Reblitz-Richardson
Venue: ICLR 2021 Spotlight (2021)
Keywords: CNN
convolution
spatial bias
blind spots
foveation
padding
exposition
debugging
visualization

Abstract: We show how feature maps in convolutional networks are susceptible to spatial bias. Due to a combination of architectural choices, the activation at certain locations is systematically elevated or weakened. The major source of this bias is the padding mechanism. Depending on several aspects of convolution arithmetic, this mechanism can apply the padding unevenly, leading to asymmetries in the learned weights. We demonstrate how such bias can be detrimental to certain tasks such as small object detection: the activation is suppressed if the stimulus lies in the impacted area, leading to blind spots and misdetection. We explore alternative padding methods and propose solutions for analyzing and mitigating spatial bias.


Published as a conference paper at ICLR 2021
MIND THE PAD CNNS CAN DEVELOP BLIND SPOTS
Bilal Alsallakh Facehook Al
Narine Kokhlikyan Facebook Al
Vivek Miglani Facebook Al
Jun Yuan NYU
Orion Reblitz-Richardson Facebook AI
ABSTRACT
We show how feature maps in convolutional networks are susceptible to spatial bias. Due combination of architectural choices, the activation certain loca- tions is systematically elevated or weakened. The major source of this bias is the padding mechanism. Depending on several aspects of convolution arithmetic, this mechanism can apply the padding unevenly, leading to asymmetries in the learned weights. We demonstrate how such bias can be detrimental certain tasks such as small object detection: the activation is suppressed if the stimulus lies in the impacted area, leading blind spots and misdetection. We propose solutions mitigate spatial bias and demonstrate how they can improve model accuracy.
MOTIVATION
Convolutional neural networks (CNNs) serve as feature extractors for wide variety of machine- learning tasks. Little attention has been paid to the spatial distribution of activation in the feature maps CNN computes. Our interest in analyzing this distribution is triggered by mysterious failure cases traffic light detector: The detector successfully detects a small but visible traffic light in road scene. However, it fails completely in detecting the same traffic light in the next frame captured by the ego-vehicle. The major difference between both frames a limited shift along the vertical dimension the vehicle moves forward. Therefore, the drastic difference in object detection is surprising given that CNNs are often assumed to have a high degree of translation invariance [8] 17].
The spatial distribution of activation in feature maps varies with the input. Nevertheless, by closely examining this distribution for large number of samples, we found consistent patterns among them, often in the form of artifacts that do resemble any input features. This work aims analyze the root cause of such artifacts and their impact on CNNs. We show that these artifacts are responsible for the mysterious failure cases mentioned earlier, as they can induce 'blind spots' for the object detection head. Our contributions are:
Demonstrating how the padding mechanism can induce spatial bias in CNNs (Section
Demonstrating how spatial bias can impair downstream tasks (Section S
Identifying uneven application of O-padding as a resolvable source of bias (Section 5
Relating the padding mechanism with the foveation behavior of CNNs (Section 0
Providing recommendations to mitigate spatial bias and demonstrating how this can prevent blind spots and boost model accuracy.
2 THE EMERGENCE OF SPATIAL BIAS CNNS
Our aim is to determine to which extent activation magnitude in CNN feature maps is influenced by location. We demonstrate our analysis on a publicly-aviilable traffic-light detection model [36] This model implements the SSD architecture P6 in TensorFlow M1 using MobileNet-vi [3 as
feature extractor. The model is trained on the BSTLD dataset 4 which annotates traffic lights in road scenes. Figure shows two example scenes from the dataset. For each scene, show two feature maps computed by two filters the th convolutional layer. This layer contains 512 filters whose feature maps used directly by the first box predictor in the SSD detect small objects.

---

Published as a conference paper at ICLR 2021
input
feature maps of layer conv2d pointwise
avg. filter response
Imae
map #1
map #512
averaged over 5 maps
image
avg. response
avg. response avg. response
averaged over the rest inages
Figure 1: Averaging feature maps per input (column marginal) and per filter (row marginal) in the last convolutional layer of a traffic light detector. Color indicates activation strength (the brighter, the higher), revealing line artifacts in the maps. These artifacts are the manifestation of spatial bias.
The bottom row Figure I shows the average response of each of the two aforementioned filters, computed over the test set in BSTLD. The first filter seems to respond mainly to features in the top half of the input, while the two average maps that do not seem to resemble any scene features and are consistently present in the individual feature maps. We analyzed the prevalence of these line artifacts in the feature maps of all 512 filters. The right column in Figure T shows the average of these maps per scene, well as over the entire test set (see supplemental for all 512 maps). The artifacts are largely visible in the average maps, with variations per scene depending on which individual maps are dominant.
second filter responds mainly to street areas. There are visible lines in
A useful way to make the artifacts stand out is to neutralize scene features by computing the feature maps for a zero-valued input. Figure depicts the resulting average map for each convolutional layer after applying ReLU units. The first average map is constant as we expect with 0-valued input. The second map also constant except for 1-pixel boundary where the value lower the left border and higher at the other three borders. We magnify the corners to make these deviations visible. The border deviations increase in thickness and in variance at subsequent layers, creating multiple line artifacts at each border. These artifacts become quite pronounced at ReLU 8 where they start to propagate inwards, resembling the ones Figure
RetU (360, 640, 32)
RetU 1 (360, 640, 64)
RelU (180, 320, 128)
Ret 3 (180, 320, 128)
188
1.9
1.
1.7
80,
1.75
1.6
12
1.70
1.0
1.4
1.8
0.8
12
1.80
ReLU 145, 80, 512)
ReLU (45, 80, 512)
ReLU 10 145, 90, 512)
ReLU 11 145, 80, 512)
1.0
05
01
Figure 2: Activation maps for 0 input, averaged over each layer's filters (title format: H HxWxC),
2

---

Published as a conference paper at ICLR 2021
It is evident that the 1 1-pixel border variations in the second map caused by the padding mecha- nism in use. This mechanism pads the output of the previous layer with a 1-pixel O-valued border in order to maintain the size of the feature map after applying 3x3 convolutional. The maps in the first layer are not impacted because the input we feed is zero valued. Subsequent layers, however, increasingly impacted by the padding, preceding bias terms do not warrant O-valued input.
is noticeable in Figure that the artifacts caused by the padding differ across the four borders. To investigate this asymmetry, analyze the convolutional kernels (often called filters) that produce the feature maps. Figure depicts a per-layer mean of these 3x3 kernels. These mean kernels exhibit different degrees of asymmetry in the spatial distribution of their weights. For example, the kernels in L1 assign (on average) negative weight at the left border, and a positive weight at the bottom. This directly impacts the madding-induee variation at each border. Such asymmetries are related to uneven application of padding we explain in Section S
Lo: 32
L1:32
12:
128
L4:
LS: 256
256
L7:512
512
L9:
L10: 512
L11:512
Figure 3: Mean kernel per convolutional layer. All kernels are 3 x 3, the titles show their counts.
3 IMPLICATIONS OF SPATIAL BIAS
We demonstrate how feature-map artifacts can cause blind spots for the SSD model. Similar issues arise in several sojel- detectors, e.g., for faces and masks, as well as pixel-oriented tasks such as semantic segmentation and image inpainting (see supplemental for examples).
Figure 4 illustrates how the SSD predicts small objects based on the feature maps of the 11-th convolutional layer. The SSD uses the pixel positions in these maps as anchors of object proposals. Each proposal scored by the SSD to represent target category, with "background" being an implicit category that is crucial to exclude irrelevant parts of the input. In addition to these scores, the SSD computes a bounding box to localize the predicted object at each anchor. We examine
45 Feature Maps
Per-Anchor Logits for:
Per-Anchor Bounding Boxes the Image Space
with as CNN input
Background (45 x 80)
computed O-valued inpul by the SSD predictor at 45 80 locations under 1:2 aspect ratio
Traffic Light (45 80)
-15
traffic light extent marked orange
score:
(shilted upwards]
7% (baseline)
downwards)
Figure 4: The formation of blind spots in SSD, illustrated via its box predictor internals with zero-valued input. The predictor uses spatial anchors to detect and localize the target object at 45 x 80 possible locations based on 512 feature maps. Certain anchors background due to feature-may artifacts, as evident in the logit maps. Traffic lights at the corre- sponding location cannot be detected demonstrated with real scene (middle one in the bottom).
predisposed to predict
3

---

Published as a conference paper at ICLR 2021
(a) zero padding
detection score
(b) mirror padding (SYMMETRIC)
Score: 5%
0.8
stimulus
86%
0.2
Figure 5: (a) A map showing via color the detection score the SSD computes for a traffic light when present at various locations. The detection is muted when the stimulus lies in the area impacted by the artifacts. (b) The same map after changing the padding method to SYMMETRIC. The detection scores are rather constant except for periodic variations due to the SSD's ' reliance on anchors.
object proposals computed at 1:2 aspect ratio, as they resemble the shape of most traffic lights in the dataset. We visualize the resulting score maps both for the background category and for traffic lights, when feeding a O-valued input to the SSD. We also visualize the bounding boxes of these proposals in the image space. The SSD predicts the image content to be of background category all anchor locations, as evident from the value range in both score maps. Such predictions are expected with an input that contains traffic lights. However, the line artifacts in the feature maps have strong impact on the score maps. These artifacts elevate the likelihood anchors closer to the top to be classified as background (see the yellow band in the background score map). Conversely, these anchors have significantly lower scores for the traffic light category, compared with other anchors in the feature map. Such difference in the impact on the target categories due to the different weights the SSD assigns to the feature maps for each target. As result, the artifacts lead to potential blind spots in which the scores for certain categories are artificially muted.
To validate whether or not the blind spots hinder object detection, we examine road scenes that contain highly-visible traffic light instances the impacted area. Fiture shows an example of such a scene. The SSD computes a low detection score of 7% when the traffic light lies in the blind spot (see middle image), far below the detection false-positive cutoff. Shifting the scene image upwards or downwards makes the instance detectable with high score as long as it lies outside the blind spot. This explains the failure cases mentioned in Section I To further validate this effect, we run the SSD on baseline images that each contains one traffic light instance at specific location in the input. We store the detection score for each instance. Figure 5h depicts the computed scores in a 2D map. It is evident that the model fails to detect the traffic light instance exactly when it is located within the "blind spot" band. The artifacts further disrupt the localization of the objects as evident in the top-right plot in Figure gl which shows per-anchor object proposals computed for a 0 input.
REMINDER: WHY IS PADDING NEEDED IN CNNs?
Padding is applied at most convolutional layers in CNNs serve two fundamental purposes:
Maintaining feature map size A padding that satisfies this property is often described as SAME or along each dimension.
HALF padding. FULL padding expands the maps by kernel size
VALID padding performs no padding, eroding the maps by the same amount. SAME padding is important to (1) design deep networks that can handle arbitrary input size (a challenge in the presence of gradual erosion), (2) maintain the aspect ratio of non-square input, and (3) concatenate feature maps from different layers as in Inception 39 and ResNet [12] models.
Reducing information bias against the boundary Consider 3x3 kernel applied to a 2D input. An input location at least 2 pixels away from the boundary contributes to nine local convolution operations when computing the feature map. On the other hand, the corner is involved only one time under VALID padding, four times under 1-pixel SAME 0- O-padding, and nine times under 2-pixel FULL O-padding. With SAME O-padding, the cumulative contribution differences among the input pixels grow exponentially over the CNN layers. We refer to such uneven treatment of input pixels as the foveation behavior of the padding mechanism and elaborate on this in Section g
We next explore solutions to the issues that cause padding to induce spatial bias.
4

---

Published as a conference paper at ICLR 2021
64
Input: 12 x
06)
highly asymmetric mean filters (wort. centers)
Paddled Input: 14 x
224x224 imagges
Trainod on
Downsampling pixels the
Reenet-18
Resnet-50
Reenet-101
applled)
imges
Outout:
Thained 2251625
Figure 6: (a) Illustrating the problem of uneven padding when down-sampling at stride of 2. The padding along x-axis is consumed only at the left side. (b) Mean 3 x 3 filters in three ResNet models, trained on ImageNet with two input sizes. Color encodes average weight (green is positive). A size that induces uneven padding (top row) can lead to asymmetries, esp. around down-sampling layers. These asymmetries are mitigated when the input size induces no uneven padding (bottom row).
ELIMINATING UNEVEN APPLICATION OF PADDING
While useful to reduce bias against the boundary. applying padding at down-sampling layers can lead to asymmetry in CNN internals. Figure illustrates the source this asymmetry when strided convolution is used for downsampling: At one side of the feature map, the padding is consumed by the kernel while at the other side is not. To warrant even application of padding throughout the CNN, the following must hold at all d down-sampling layers, where (h4, w;) is the output shape at the i-th layer with h.f x ky" as kernel size, (s/ (.) as strides, and = (F) .P.)) as padding amount (refer appendix A for proof):
Vi E (1.... : :4:1= = . (h, - 1) + k - -2. A W:-1 -, (we + 2 P (1)
The values hg and wi represent the CNN input dimensions. The above constraints are not always satisfied during training inference with arbitrary input dimensions. For example, ImageNet clas- sifiers based on ResNet [12] and MobileNet [3 contain five down-sampling layers (d 5) that
apply 1-pixel O-padding before performing 2-strided convolution. To avoid uneven upplication of padding, the input to these CNNs must satisfy the following, explained in appendix A
ho 01 2d +1 32.a; a, +1 and wo a2 2d +1 32 a2 + where a1, a2 N+ (2)
The traditional and prevalent input size for training ImageNet models is 224x224. This size violates Eq. 2 leading to uneven padding every down-sampling layer in ResNet and MobileNet models where 0-padding is effectively applied only the left and top sides of layer input. This over-represents zeros at the top and left sides of 3 feature-map map patches the filters convolved with during training. The top row of Figure 6 shows per-layer mean filters three ResNet models in PyTorch (33 pre-trained ImageNet with 224x224 images. In all of these models, few of the mean filters, adjacent to down-sampling layers, exhibit stark asymmetry about their centers.
We increase the image size 225 x25 without introducing additional image information This size satisfies Eq. - warranting even application of padding at every downsampling layer in the above models. Retraining the models with this size strongly reduces this asymmetry as evident in the bottom row of Figure 6p. This, in turn, visibly boosts the accuracy in all models experimented with as we report Table The accuracy did not improve further when we retrained two of the models, ResNet- 18 and ResNet-34, 226 x 226 images. This provides evidence that the boost is due to eliminating uneven padding and not merely due to increasing the input size.
This size been used facilitate model comparison on ImageNet, since the inception AlexNet. 2 This is done via constant padding. The side to pad with one pixel is chosen at random to halance out the application of padding both sides over the training set. No additional padding applied further layers.
5

---

Published as conference paper at ICLR 2021
Replacing O-padding with padding method that reuses feature map values can alleviate the asym- metry in the learned filters in the presence of unevenly applied padding. Another possibility is to use rigid downsampling kernel, such max max-pooling instead of learned one. Appendix C demon- strates both possibilities. Finally, antialiasing before downsampling 43 asymmetry elaborate in Section and in Appendix
can strongly reduce the
Table 1: Top-1 (and top-5) accuracy of five ImageNet classifiers trained with different input sizes.
Input Size?
MobileNet
ResNet-18
ResNet-34
ResNet-50
ResNet-101
224x224
68.19 (88.44)
69.93 (89.22)
73.30 (91.42)
75.65 (92.47)
77.37 (93.56)
225x225
68.80 (88.78)
70.27 (89.52)
73.72 (91.58)
76.01 (92.90)
77.67 (93.81)
Even when padding is applied (p¹ 0 p. = 0), an input size that does satisfy Eq. can lead to uneven erosion of feature maps, in turn, reducing the contribution of pixels from the impacted sides (Fig Ze. Satisfying Eq imposes restriction
input size, e.g., values in increments of 2d 32 with the above models (193 x 193, 225 x 225, 257 257, .... Depending on the application domain, this be guaranteed either by resizing an input to the elosest increment, or by padding accordingly with suited values.
PADDING MECHANISM AND FOVEATION
By foveation we mean the unequal involvement of input pixels in convolutional operations through- out the CNN. Padding plays fundamental role in the foveation behavior of CNNs. We visualize this behavior by means of a foveation map that counts for each input pixel the number of convolutional paths through which it can propagate information to the CNN output. We obtain these counts by computing the effective receptive field [2X] for the sum the final convolutional layer after assign- ing all weights in the network (code in supplemental). Neutralizing the weights is essential to obtain per- pixel counts of input-output paths that reflect the foveation behavior.
VALID
Zaro /
Zaro
(c)
2-dilated
2-strided pooling 127x127 Input
Zaro (BAME)
(SAME)
Zero SAMEE)
(b)
Mirror (YMMETRIO)
Mirror (RETECCT)
Replicate fiter))
Circular
PartialConv
Distribution
Figure 7: Foveation behavior of different padding methods applied to VGG-19 37 and illustrated in 512 x 512 input space (unless otherwise stated). Color represents the number of paths to the output for each input pixel. (a) The difference between VALID, FULL, and SAME O-padding. (b) SAME alternatives to O-padding. (c) Dilation amplifies foveation of SAME O-padding. (d) Strides can lead to checkerboard patterns. (e) Foveation effects are more extensive in smaller inputs (relative to input size) and sensitive uneven padding.
Figure ? shows the extensive foveation effect when no padding applied. The diminishing con- tribution of vast areas of the input explains the drastic drop in accuracy recently observed under VALID padding [16]. contrast, FULL O-padding does not incur foveation, however, at the cost of increasing the output size after each layer, making it impractical as explained in Section 4 SAME O-padding incurs moderate foveation at the periphery, whose absolute extent depends on the num- ber of convolutional layers and their filter sizes. Its relative extent depends on the input size: the larger the input, the larger the ratio the constant area in yellow (refer to appendix for a detailed example).

---

Published as a conference paper at ICLR 2021
Figure 7p shows the foveation behavior of alternatives to SAME O-padding that have roots in wavelet analysis 19 and image processing [27]. Mirror padding mirrors pixels at the boundary to fill the padding area. When the border is included (S (YMMETRIC mode in TensorFlow) all input pixels have an equal number of input-output paths F resulting in a uniform foveation map. When the border is not included (REFLECT mode both in Pyyoorch and in [TensorFlow), the map exhibits bias against the border and towards a contour in its proximity. This bias is amplified over multiple layers. Replication padding exhibits the opposite bias when the padding area is wider than pixel. This is because it replicates the outer 1-pixel border multiple times to fill this area 3. The method is equivalent to SYMMETRIC if the padding area is 1-pixel wide. Circular padding wraps opposing borders, enabling the kernels to seamlessly operate on the boundary and resulting in a uniform map. Partial Convolution [22] has been proposed as a padding method that treats pixels outside the original image as missing values and rescales the computed convolutions accordingly 24) Its foveation behavior resembles reflective padding 3. Distribution padding [30] resizes the input to fill the padding area around the original feature map, aiming at preserving the distribution of the map. Its foveation map is largely uniform, except for the corners and edges.
Impact of input size Besides influencing the relative extent of foveation effects, the input size also determines the presence of uneven padding (or uneven reature-map erosion), as we discussed in Section D Figure [ shows the foveation map for VGG-19 with a 127 127 input. This input violates Eq. | at every downsampling layer (appendix A leading to successive feature map erosion at the bottom and right sides which is reflected in the foveation map (see appendix B for detailed example). The bottom-right part of the input hence less involved in the CNN computations.
Impact of dilation We assign a dilation factor of to all VGG-19 convolutional layers. While this exponentially increases the receptive field of the neurons at deeper layers [2), dilation doubles the extent of the non-uniform peripheral areas that emerge with SAME O-padding as evident in Figure 7- SYMMETRIC and circular padding maintain uniform foveation maps regardless of dilation In contrast, dilation increases the complexity of these maps for REFLECT and replication padding.
Impact of strides Whether learned on based on pooling, downsampling layers can amplify the impact of succeeding convolutional layers on foveation behaviour. Furthermore, these layers can cause input pixels to vary in the count of their input-output paths. This can happen when the kernel size is not divisible by the stride, leading to a checkerboard pattern the foveation maps. This manifests in ResNet models as we illustrate in appendix B In VGG-19, all max-pooling layers use a stride of and kernel size of 2. Changing the kernel size to leads to a checkerboard pattern as evident in Figure d.. Such effects were shown to impact pixel-oriented tasks 32)
The padding technique and its foveation behaviour have direct impact on feature-map artifacts (Sec- tion 77 and the ability of CNNs to encode spatial information (Section g+ Understanding the foveation behavior is key to determine how suited a padding method is for a given task. For ex- ample, small object detection is known to be challenging close to the boundary (26). in part due to the foveation behavior of SAME O-padding. In Figure Sp. we change the padding method in the SSD SYMMETRIC. The stimulus is noticeably more detectable at the boundary, compared with O-padding q In contrast, ImageNet classification is less sensitive to foveation effects because the target objects are mostly located away from the periphery. Nevertheless, the padding method was shown to impact classification accuracy 2 because it still affects feature map artifacts.
7 PADDING METHODS AND FEATURE MAP ARTIFACTS
It is also noticeable that the score map in Figure 5p is more uniform than in Figure 5h. In particular, under SYMMETRIC padding the model is able to detect traffic lights placed in the blind spots of the original O-padded model. To verify whether the line artifacts in Figure 2 are mitigated, we inspect the mean feature maps of the adapted model. With a constant input, S SYMEERIC padding warrants constant maps throughout the CNN because it reuses the border to fill the padding area. Instead, we average these maps over 30 samples generated uniformly at random. Figure R depicts the mean maps which are largely uniform, unlike the case with O-padding.
Refer to appendix or to http: http:://minde theiind-theppaae -pad. q thub. 1o for visual illustration and further thcoretical analysis of the fovcation hehavior.
Sinne the input size causes uneven application of padding, the right and bottom borders are still challenging.
7

---

Published as a conference paper at ICLR 2021
RetL 1360. 32)
ReLLJ 1-1300, 64)
RetU -1380, 320,
RetU -1200, 325,
ReLL 2510
2510
(45, BALU0-1445 112]
00. 5:23)
RetU 30 (45, 51.31
Rell 512)
340
Figure 8: The same feature maps in Figure 2 generated under mirror padding and averaged over 30 mundoml--generate input samples. The line artifacts induced by O-padding are largely mitigated.
To further analyze the impact of SYMMETRIC padding, we retrain the adapted model following the original training protocol. This significantly improves the average precision (AP) as reported in Table 2 under different overlap thresholds (matching IoU), confirming that small object detection is particularly sensitive to feature-map artifacts.
Table 2: Performance of the SSD traffic light detector, trained under two different padding schemes.
Average Precision (AP)
APQ.20IOU
AP@.50IOU
AP@.75IOU
AP@ .90IOU
Zero Padding
80.24%
49.58%
3.7%
0.007%
Mirror Padding
83.20%
57%
8.44%
0.02%
Of the padding methods listed in Section 6 mirror padding in both SYMMETRIC and REFLECT modes, PartialConv, and circular padding are generally effective at reducing feature map artifacts that emerge under zero padding, in particular salient line patterns. In contrast, distribution padding can induce significant artifacts. Refer to appendix for comparative examples of artifacts under the aforementioned padding schemes.
Artifact magnitude and propagation While feature-map artifacts are induced by the padding mechanism at the boundary, their magnitude and inward propagation in the maps are impacted by several architectural aspects of CNNs. In particular, certain normalization schemes such as batchnorm [I5] tend to limit the range of variation within feature map and to relatively harmo- nize this range across different maps. This, in turn, impacts how possible artifacts these maps ac- cumulate when they are processed by the next convolutional layer. Similarly, artifacts that manifest after applying ReLU units are of positive sign. These factors were instrumental the formation of potential blind spots described in Section 3 We hence recommend involve non-convolutional layers when inspecting the feature maps. Besides having possible impact on artifact magnitude, several aspects of convolution arithmetic, such filter size and dilation factors, can also impact the spatial propagation of these artifacts.
RELATED FINDINGS AND TAKEAWAYS
Handling the boundary is an inherent challenge when dealing with spatial data 99. Mean padding is known to cause visual artifacts traditional image processing, with alternative methods proposed to mitigate them (2. CNNs have been often assumed to deal with such effects implicitly. Innamorati al 14 propose learning separate sets of filters dedicated to the boundaries to avoid impacting the weights learned by regular filters. A grouped padding strategy, proposed to support 2 x 2 filters [41], offers avenues to mitigate uneven padding and corresponding skewness foveation maps without restrictions on input size (see note in appendix B for cxplanation). Finally, insights from signal and image processing 10 could inspire further CNN padding schemes.
Zero padding has been recently linked to CNNs ability to encode position information 16
18
29 In contrast, circular padding was shown to limit this ability [7] and boost shift invariance 355 The imput sizes in those studies do induce uneven padding. This can be, in part, the underlying
mechanism behind the aforementioned ability. Whether or not this ability is desirable depends on the task, with several methods proposed to explicitly encode spatial information
5 20 25 29 3T
8

---

Published as a conference paper at ICLR 2021
Downsampling using max pax-liooing strided convolution has been shown to impact shift invariance in CNNs by incurring aliasing effects 3
38 4 These effects can manifest in the same symptoms reported in Section L albeit for different reason. Zhang [x demonstrated how blurring the feature maps before subsampling mitigates aliasing effects and improves ImageNet classification ac- curacy of various popular CNNs.. We analyzed the mean filters in antialiased MobileNet and ResNet models pre- pre-trained on ImageNet under O-padding, with 224 x x224 as input size (refer to Appendix ( We found that antialiasing can also mitigate the asymmetry of mean filters that exhibited high asym- metry in the baseline models, especially at deeper layers. This is remarkable given that these models are trained on 224 x 224 images, which incurs one-sided zero padding at every downsampling layer. This could, in part, be attributed to the ability of the BlurPool operator used in antialiased CNN to smoothen the acuity of zero-padded borders, in turn, reducing the value imbalance incurred by one-sided padding. Further analysis is needed to examine the interaction between padding and alias- ing effects in CNNs and to establish possible synergy between antialiasing and climinating uneven application of padding.
Luo et al PS drew connections between effective receptive fields and foveated vision. Our analysis links foveation behavior with the padding scheme and suggests that it might occur implicitly in CNNs when using VALID or SAME O- 0-padding, without the need for explicit mechanisms (2) 21]. Furthermore, it explains the drastic accuracy drop noted by [16] under VALID padding, which is amplified by feature map erosion.
Choosing a padding method SAME O-padding is by far the most widely-used method. Compared with other methods, it can enable as much as 50% faster training and inference. Problem-specific constraints can dictate different choices 34 35 40, In the lack of a universally superior padding method, we recommend considering multiple ones while paying attention to the nature of the data and the task, as well as to the following aspects:
Feature-map statistics: O-padding can alter the value distribution within the feature maps and can shift their mean value the presence of ReLU units. The alternatives presented in Section C tend preserve this distribution, thanks reusing existing values in the maps.
Foveation behavior: O-padding might not be suited for tasks that require high precision at the periphery, unlike circular and SYMMETRIC mirror padding.
Interference with image semantics (esp. with padding amount > 1 pixel): For example, circular padding could introduce border discontinuities unless the input is panoramic B5),
Potential to induce feature map artifacts: All alternatives to O-padding induce relatively fewer artifacts, except for Distribution padding 30 (see appendix Dj.
We also recommend eliminating uneven padding downsampling layers both at training and at inference time, as we illustrated in Section 5 This is especially important when zero padding is applied and the downsampling is learned. The scripts used to generate the visualizations in this paper are available in the supplemental as well as at httt:::://mnnnthe--aae .. tit...
Summary We demonstrated how the padding mechanism can induce spatial bias in CNNs, in the form of skewed kernels and feature-map artifacts. These artifacts can be highly pronounced with the widely-used O-padding when applied unevenly at the four sides of the feature maps. We demonstrated how such uneven padding inherently take place in state-of-the-arr CNNs, and how the artifacts it causes can be detrimental to certain tasks such as small object detection. We provided visualization methods to expose these artifacts and to analyze the implication of various padding schemes on boundary pixels. We further proposed solutions to eliminate uneven padding and to mitigate spatial bias in CNNs. Further work is needed closely examine the implications of spatial bias and foveation in various applications (see supplementary for examples), as well as padding impact on recurrent models and 1-D CNNs.
ACKNOWLEDGEMENT
We thankful to Ross Girshick for providing useful recommendations and experiment ideas, and to Shubham Muttepawar for implementing an interactive tool out of our analysis scripts, guided by front-end specialist Edward Wang and our Al user-experience designer Sara Zhang.

---

Published as a conference paper at ICLR 2021
REFERENCES
[1] M. Abadi, A. Agarwal, Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, al. TensorFlow: Large-scale machine learning on heterogeneous distributed sys- tems. arXiv preprint arXiv 044677, 2016.
[2] E. Akbas and M. P. Eckstein. Object detection through search with foveated visual system. PLoS computational biology, x3( 10057433 2017.
[3] A. Azulay and Y. Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research (JMLR),
2019.
[4] K. Behrendt, L. Novak, and R. Botros. A deep learning approach to traffic lights: Detection, tracking, and classification. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 1370-1377. IEEE, 2017.
[5] C.-A. Brust, S. Sickert, M. Simon, E. Rodner, and J. Denzler. Convolutional patch networks with spatial prior for road detection and urban scene understanding. In International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP),201
[6] G. F. Elsayed, P. Ramachandran, J. Shlens, and S. Kornblith. Revisiting spatial invariance with low-rank local connectivity. In International Conference on Machine Learning (ICML), 2020.
[7] Geiping, H. Bauermeister, H. Dröge and M. Moeller. Inverting gradients-how easy is it break privacy in federated learning? arXiv preprint arXiv:2003, 14053, 2020.
[8] R. Gens and P. M. Domingos. Deep symmetry networks. In Advances in neural information processing systems (NeurIPS). pp. 2537-2545 2014.
[9] D. Griffith and Amrhein. An evaluation of correction techniques for boundary effects in spatial statistical analysis: traditional methods. Geographical Analysis, 15(4):352-36) 1983.
[10] V. Gupta and N. Ramani. A note on convolution and padding for two-dimensional data. Geo- physical Prospecting, 26(1):214-217, 1978.
[11] L. Hamey. A functional approach to border handling in image processing. In International Conference Digital Image Computing: Techniques and Applications, pp. 1-8, 2015.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
[13] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: Nkdv:1704 774.004661 2017.
[14] C. Innamorati, T. Ritschel, T. Weyrich, and N. J. Mitra. Learning on the edge: Investigating boundary filters in CNNs. International Journal of Computer Vision (IJCV), pp. 1-10, 2019.
[15] Ioffe and Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), pp. 448- 456, 2015.
[16] M. A. Islam, S. Jia, and N. D. Bruce. How much position information do convolutional neural networks encode? In International Conference on Learning Representutions (ICLR), 2020.
[17] M. Jaderberg. K. Simonyan, A. Zisserman, al. Spatial transformer networks. In Advances
in neural information processing systems (NeurIPS), pp. 2017-2025, 2015.
[18] O. S. Kayhan and J. C. van Gemert. On translation invariance CNNs: Convolutional layers exploit absolute spatial location. In IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[19] T. L. Kijewski- Correa. Full-scale measurements and system identification: A time- perspective. PhD thesis, University of Notre Dame., 2003.
10

---

Published as a conference paper at ICLR 2021
[20] I. Kim, W. Back, and S. Kim. Spatially attentive output layer for image classification. In IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[21] H. Larochelle and G. E. Hinton. Learning to combine foveal glimpses with third-order boltzmann machine. In Advances in nearal information processing systems (NeurIPS), pp- 1243-1251, 2010.
[22] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro. Image inpainting for irregular holes using partial convolutions. In European Conference on Computer Vision, 2018.
[23] G. Liu, K. J. Shih, T.-.. Wang, F. A. Reda, K. Sapra, Z. Yu, A. Tao, and B. Catanzaro. Partial convolution based padding. In arXiv preprint 1iXiv:18111111 2018.
[24] R. Liu and J. Jia. Reducing boundary artifacts in image deconvolution. In IEEE International Conference on Image Processing (ICIP), pp. 505-508, 2008
[25] R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski. An intriguing failing of convolutional neural networks and the CoordConv solution. Advances in Neural Information Processing Systems (NeurIPS), pp. 9605-9616, 2018.
[26] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. SSD: Single shot multibox detector. In European Conference on Computer Vision, pp. 21-37, 2016.
[27] S. Lou, X. Jiang, and P. J. Scott. Fast algorithm for morphological filters. Journal of Physics: Conference Series, 311(():012001) 2011 1.
[28] W. Luo, Y. Li, R. Urtasun, and R. Zemel. Understanding the effective receptive field in deep convolutional neural networks. In Advances Neural Information Processing Systems (NeurIPS), pp. 4898-4906, 2016.
[29] R. Murase, M. Suganuma, and T. Okatani. How can cnns use image position for segmentation? arXiv preprint arXiv:2005.00 034663, 2020.
[30] A.-D. Nguyen, S. Choi, W. Kim, S. Ahn, J. Kim, and S. Lee. Distribution padding in convo- lutional neural networks. In IEEE International Conference Image Processing (ICIP), pp. 4275-4279, 2019.
[31] D. Novotny, S. Albanie, D. Larlus, and A. Vedaldi. Semi-convolutionaa operators for instance segmentation. In European Conference on Computer Vision (ECCV), pp. 86-102, 2018.
[32] A. Odena, V. Dumoulin, and C. Olah. Deconvolution and checkerboard artifacts. Distill, (10)ee3, 2016.
[33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, et al. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8024-8035, 2019.
[34] P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollár. Learning to refine object segments. In European Conference on Computer Vision (ECCV), pp. 75-91, 2016.
[35] S. Schubert, P. Neubert, J. Pöschmann, and P. Pretzel. Circular convolutional neural networks for panoramic images and laser data. IEEE Intelligent Vehicles Symposium (IV), pp. 653- 660, 2019.
[36] E. Shalnov. BSTI.Ddemo: sample project to train and evaluate model on BSTLD. https: demo 2019.
7/ //qithub.
[37] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recog- nition. In International Conference Learning Representations (ICLR), 2015.
[38] G. Sundaramoorthi and T. E. Wang. 1 2019.
Translation insensitive CNNs. arXiv preprint
11

---

Published as a conference paper at ICLR 2021
[39] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015.
[40] S. Vashishth, S. Sanyal, V. Nitin, N. Agrawal, and P. Talukdar. InteractE: Improving convolution-based knowledge graph embeddings by increasing feature interactions. In AAAI conference on Artifical Intelligence, 2020.
[41] S. Wu, G. Wang. P. Tang, F. Chen, and L. Shi. Convolution with even-sized kemells and symmetric padding. In Advances in Neural Information Processing Systems (NeurIPS), pp- 1192-1203, 2019.
[42] Yu and Koltun. Multi-scale context aggregation by dilated convolutions. In International Conference Learning Representations (ICLR), 2016.
[43] R. Zhang. Making convolutional networks shift-invariant again, International Conference on Machine Learning (ICML), 2019.
12

---

Published as a conference paper at ICLR 2021
A ELIMINATING UNEVEN APPLICATION OF PADDING
Consider a CNN with downsampling layers, L1, Iz,,..., Lng. To simplify the analysis and without loss of generality we assume that the kernels in these layers are of square shape and that all other layers maintain their input size, We denote by 8; and k; the stride and kernel size of layer Li. We denote by h; and w; the dimensions of the feature maps computed by L, We denote by ho and we the size of the CNN input. We examine the conditions to warrant no uneven application of padding along the height dimension. Parallel conditions apply to the width dimension.
We denote by h; the height of the padded input to L,- The effective portion h4 S h; of this amount processed by the convolutional filters in L; is equal to:
h4 5; (h, 1) +
Our goal is to warrant that h2 = h, to prevent information loss and to avoid uneven padding along the vertical dimension when the unconsumed part h; - h4 < is an odd number.
Since the non-dowssampling layers maintain their input size, we can formulate the height of the padded input as follows:
h4 h4-1 + Pp;
where pi is the amount of padding applied the top and at the bottom of the input in L,, Accord- ingly, we can warrant no uneven padding if the following holds:
Vi E [1. h = 8, (h, 1) + k; - 2.
(3)
Example 1: ResNet--18 This network contains five downsampling layers (d 5) all of which use stride of 2. Despite performing downsampling. all of these layers apply padding amount entailed by SAME padding to avoid information bias against the boundary. In four of these layers having 3 x kernels (k; 3), the amount used is p; = 1. For the first layer having 7 x 7 kernels, this amount is equal to 3. both cases, the term k; 2 p: in Eq. 3 is equal to 1. To warrant no uneven padding along the vertical dimension, the heights of the feature maps at downsampling layers should hence satisfy:
Vi E []. h--1 2 (h: 1) 2 Accordingly, the input height should satisfy:
hg 2d ha (2 - 1) = 2 (ha 1) +1
where ha is the height of the final feature map, and can be any natural number larger than to avoid a degenerate case of a x input. The same holds for the input width:
We = 2d (wa 1) + 1
+ 1, yielding even padding in all five 225x225 input satisfies these constraints since 225 2 7 downsampling layers and output feature maps of size 8 x 8.
Example 2: VGG-16 This network contains five max- max-pooling layers (d = 5) all of which use a stride of 2 and kernel size of 2 and apply no padding. To warrant no uneven padding along the vertical dimension, the heights of the feature maps at all of these layers should hence satisfy:
E [1.d]: h4-1 =2 (h+ 1) + 2 2 h:
Accordingly, the input dimensions should satisfy:
hg 2d ha and wp 2d Wa
(4)
A 224 224 input satisfies these constraints since 224 2 7, causing feature-man map erosion any downsampling layer and resulting in output feature maps of size 7 x 7.
13

---

Published as a conference paper at ICLR 2021
B THE EXTENT OF FOVEATION UNDER SAME O-PADDING
We illustrate how the absolute extent of foveation under SAME O-padding depends on the number of convolutional layers, and how its relative extent depends the input size.
In the following maps, color represents the number of paths to the CNN output for each input pixel.
Note: The checkerboard pattern is caused by downsampling layers in ResNet that use 3 kernels and stride of 2.
Figure 9: The foveation maps of two ResNet architectures under 0 padding, illustrated with 225x225 input. Compared with ResNet-50, ResNet-101 has twice the number of convolutional layers with non-unitary filter sizes. Accordingly, the extent of the foveation effect is doubled.
513 513
257 257
129 129
65
33
Figure 10: The foveation maps of ResNet-50 under O padding, illustrated with inputs of different size. The smaller the input, the larger the relative extent foveation.
In the next figure, we illustrate how uneven application of padding impacts the foveation maps. Note: It is possible to rectify the skewness in the 2nd foveation map by alternating the side where one-sided padding is applied between successive downsampling layers. This, however, does not mitigate the skewness in the learned filters (see next Section).
257 x 257 Input
256 x 268 Input
Figure 11: The foveation maps of 50 under 0 padding, illustrated with two input sizes. With a 257 257 input, the padding is evenly applied at all downsampling layers, leading symmetric foveation map. With 256 x 256 input, the padding is applied only to the left and top sides of feature maps at all downsampling layers, which limits the number of convolutional input-output paths for pixels in the bottom and right sides evident in the skewed foveation map.
14

---

Published as a conference paper at ICLR 2021
C
THE IMPACT OF THE PADDING METHOD ON LEARNED WEIGHTS
In the presence of uneven application of padding, O-padding causes skewness the learned weights because the filters are exposed more frequently feature-map patches with zeros their top and left sides. Redundancy methods such as circular or mirror padding mitigate such skewness because they fill the padding areas with values taken from the feature maps. PartialConv also mitigates such skewness because it assumes the pixels in the padding area are missing, and rescales the partial convolutional sum to account for them. Below we show the effectiveness of these alternatives in mitigating the skewness in three ResNet architectures.
O-Padding
Circular Padding
O-Padding
PartialConv
(a) filters of ResNet-18 trained on 224x224 images under two padding methods, reaching 69.93% top-1 accuracy under O-padding and 70.28% top-1 accuracy under circular padding.
(b) Mean filters ResNet-50 trained on x24x images under two padding methods, reaching 76.15% top-1 accuracy under O-padding and 76.61% top-1 accuracy under PartialConv.
Figure 12: Mean filters of two ResNet models trained on ImageNet with 224x224 images. The input size causes uneven application of padding, leading frequent asymmetries the mean filters under padding. We illustrate how two alternatives, circular padding and PartialConv 23] enable learning highly-symmetre mean filters despite the uneven application of padding.
0-Padding
PartialConv
Figure 13:
Mean filters of ResNet-101 trained on ImageNet with 224x224 images under both 0-
padding and PartialConv P3 The input size causes uneven application of padding, leading to frequent asymmetries the mean filters under padding. contrast, PartialConv produces highly symmetric mean filters, thanks for its treatment of pixels outside the feature map missing values.
What if no padding is applied during downsampling? VGG models perform downsampling using pooling layers that do not apply any padding. Accordingly, the mean filters not exhibit significant skewness, even if the input size does not satisfy Eg
O-padding with 224x224 images
D-padding with 225x225 images
PartialConv with 224x224 images
Figure 14: Mean filters of VGG-16 trained on ImageNet under different conditions. Most mean filters exhibit high symmetry when trained with 225xx22222 x25 images where the size violates Eq.
15

---

Published as a conference paper at ICLR 2021
D THE IMPACT OF PADDING METHODS ON FEATURE-MAP ARTIFACTS
We show per-layer mean feature maps in ResNet-18 18 under different padding methods. The mean maps are averaged over 20 input samples generated random.
L3E.3R 128
1311 11..c000 momil -2
MAA.ETT 76
can.onss-- 17.
Figure 15: Feature map artifacts under zero padding. Line artifacts accumulate to become significant and asymmetric at deeper layers.
L1 L..oaan::1 125.30
L10 conrz 156,50
L1:3am2 (t8. 36
ULL.see2 156 te1
12. (28 20
c20 128
L23:an05 (23
12.1 core 128
Figure 16: Circular padding largely preserves the randomness and mitigates line artifacts.
L2 (2.
L20cand--0a 2b
L2. 40-25
13: 120
123 como
Figure 17: SYMMETRIC mirror padding also preserves the randomness and mitigates line artifacts.
L1B
Li -54
Lidcand P9
L100me0: -4,
(XT
13 -K,
L40
Figure 18: REFLECT mirror padding also preserves the randomness and mitigates line artifacts.
13 (34.90)
L1. U111.090-166-90
12.5022-DD -24-200
12 -06 201
Figure 19: PartialConv 23 highly preserves the symmetry of the feature maps. The scaling factors it uses can break the randomness at the boundary.
16

---

Published as a conference paper at ICLR 2021
3-dL. 64, 240, 2401
2-11. 66, 2411
1.22-12.4 044 1240
4-113 320, 0244-324.000 22411
5-11 123 126, 124/
6-CL 120, 60,
7-12,256: 220.2256-32200 (3.02)
0-12,256
9.12.238.08
A0. 12. 11-12
12-11 322 3L
22-12 22.331
14-12. 522.32
15-12
28-12 522
Feature map artifacts of a VGG-19 model under Distribution Padding (interpolation mode) 30 Due to multiple resize operations used to fill the padding area, the artifacts grow from the boundary inwards. We use a saturated constant input to make the effect visible.
Figure 20:
THE IMPACT OF ANTIALIASING ON THE LEARNED WEIGHTS
We demonstrate how antialiasing significantly reduces the asymmetry of mean filters around downsampling layers, even in the presence of unevenly-applied zero padding.
Highly asymmetric mean filters (wrt.. centers)
Mildly asymmetric filters (w.rt. centers)
Figure 21: Mean filters of four models trained on ImageNet with 224x224 images under O-padding both without and with antialiasing.
Highly asymmetric mean filters (w.rt. centers)
Baseline Models
ResNet 24
ResNet-118 102
Andt aal--lllseedddeee Models
Mildly asymmetric mean filters (w.rt. centers)
Figure 22: Mean filters of two models trained on ImageNet with x24x224 images under 0- O-padding both without and with antialiasing.
17

---

Published as a conference paper at ICLR 2021
F FOVEATION ANALYSIS OF PADDING ALGORITHMS
Refer to hht.e::/mind-the-pppp github.ic.... for an interactive and animated visual illustra- tion of padding algorithms and their foveation behavior. This appendix serves print version.
Among the SAME padding algorithms we discussed in the manuscript, two algorithms warrant that each input pixel is involved in equal number of convolutional operations, leading uniform foveation maps: circular padding and SYMMETRIC mirror padding. In contrast, this number varies under zero padding, REFLECT mirror padding, replication padding, and partial convolution.
We illustrate in detail how each padding algorithm treats the input pixels. For this purpose we illustrate step by step how each pixel processed by the convolutional kernel. We choose set of pixels that are sufficient to expose the behavior of the respective algorithm. This set spans an area within two or three pixels from the boundary that encompasses all relevant cases for the analysis and is situated at the top-left corner. The behavior at the other corners is analogous.
All illustrations use stride of Except for VALTD, all configurations warrant SAME padding.
VALID Padding: This algorithm is illustrated 3 3 kernel without dilation. A larger
kernel size or dilation factor will increase the foveation effect.
Zero Padding: This algorithm is illustrated on a x 3 kernel without dilation. A larger
kernel size or dilation factor will increase the foveation effect.
Circular Padding: This algorithm is illustrated on 3 x 3 kernel without dilation. It is straightforward to prove that the algorithm warrants equal treatment of the pixels irre- spective of the kernel size or dilation factor. This is because it effectively applies circular convolution: Once the kernel hits one side, it can seamlessly operate on the pixels of the other side. Circular convolution hence renders the feature map as infinite to the kernel, warranting that edge pixels are treated in the same manner as interior pixels.
Mirror Padding (SYMMETRIC): This algorithm warrants that each pixel is involved in the same number of convolutional operations. It is important to notice that, unlike un- der circular convolution, these operations do not utilize the kernel pixels uniformly as we demonstrate in detail. We illustrate the algorithm behavior under the following settings:
3 x 3 kernel and dilation factor of 1.
5 x 5 kernel and dilation factor of 1.
3 x 3 kernel and dilation factor of 2.
X 2 kernel and dilation factor of 1, along with a grouped padding strategy com-
pensate for uneven padding 41
4 x kernel size and dilation factor of 1, along with grouped padding strategy.
Mirror Padding (REFLECT): This algorithm is illustrated 3 x 3 kernel without dila- tion.
This algorithm is illustrated on 5 kernel without dilation. We choose this kernel size since x 3 kernel under SAME, padding would render the algorithm equivalent to SYMMETRIC mirror padding.
Replication Padding:
Partial Convolution: This algorithm is illustrated on 3 kernel without dilation. Its foveation behavior is analogous to REF LECT mirror padding.
18

---

VALID Padding
Illustrated on a 3x3 kernel
Input
of conv ops each pixel is involved in
2
6
h
3
9
9
9
3
9
9
9
3
Which kernel cells these ops utilize?
Detailed Illustration of how the counts are derived
Convolutions involving (a)
Convolutions involving (b)
Convolutions involving (c)
Q
Convolutions involving (d): rotated version of (b)
Convolutions involving (e)
d
Convolutions involving (f)
4
6
3
Convolutions involving (g): Rotated version of (c)
Convolutions involving (h): Rotated version of
Convolutions involving (): Regular uniform treatment

---

Zero Padding
Illustrated on 3x3 kernel and 1-pixel padding
# of conv ops each pixel involved in
Original Input
Padded Input
8
4
6
d
0
0
6
9
9
9
0
Which kernel cells these ops utilize?
1 1
1
sum 44
sum
sum
sum
unitorm
Detailed Illustration of how the counts are derived
Convolutions involving (a)
0
0 0
0
a b
b
b
a
C
e
e
Convolutions involving (b)
Other border cases translation rotation of (a) (b)

---

Circular Padding
Illustrated on 3x3 kernel and 1-pixel padding
# conv ops each pixel is involved in
Original Input
Padded Input
9
P
o
n
m
a
b
a
b
C
d
9
9
h
9
9
9
9
18
9
U
9
9
9
9
9
9
p
9
Which kernel cells these ops utilize?
1
sum 9
sum
uniform
unitorm
Detailed Illustration of how the counts are derived
Convolutions involving (a)
b
b
b
X
W
Convolutions involving (b)
X
W
Other border cases are translation or rotation of (a) or (b)

---

Mirror Padding (SYMMETRIC)
Illustrated on 3x3 kernel and 1-pixel padding
# of conv ops each pixel is involved in
Original Input
Padded Input
a a
a
a
a
9
9
d
d d
f
9
9
g
h
9
Which kernel cells these ops utilize?
c:
d:
e:
f:
gF
1
2
1
1
1
2
2
9
9
9
sum
9
uniform
uniform
uniform
uniform
Detailed Illustration of how the counts are derived
Convolutions involving (a)
a
b
a
b
a
b
a
e
g
g
g
9
9
Convolutions involving (b)
b
b
b
b
a
b
b
d
e
d
d
d
h
g
h
h
g
Convolutions involving (c)
a
b
a
a
b
a
g
h
g
g g
9

---

Convolutions involving (d): Rotated version of (b)
Convolutions involving (e)
a
C
a
b
b
b
C
a
b
d
0
d
d
6
d
e
h
i
h
h
h
g
h
b
b
b
b
9
g
Convolutions involving (f)
a b
C
a
b
a
a
a
b
a
b
c
b
C
a
b
a
a
a
b
C
a
b
c
d
e
d
d
d
e
e
h
h
h
h
g
g
g
Canvolutions involving (g): Rotated version of (c)
Convolutions involving (h): Rotated version of (f)
Convolutions involving (i): Regular uniform treatment

---

Original Input
Padded Input
# of conv ops for each pixel
Mirror Padding (SYMMETRIC)
Illustrated on 5x5 kernel and 2-pixel padding
23
25
25
Which kernel cells these ops utilize?
d:
E
sum 25
Sum 25
Sum 25
Sum
Sum
Sum
Sum
Sum
Sum
untorm
Detailed Illustration of how the counts are derived
Convolutions involving (a)
Convolutions involving (b)
d
Convolutions involving (c)
ddef

---

Convolutions involving (d): Rotated version of (b)
Convolutions involving (e)
D.D.
ROOKE
Convolutions involving (f)
D.C.
Convolutions involving (g): Rotated version of (c)
Convolutions involving (h): Rotated version of (f)
Convolutions involving (i): Regular uniform treatment

---

Original Input
Padded Input
# of conv ops for each pixel
Mirror Padding (SYMMETRIC)
Illustrated on 3x3 kernel and 1-pixel padding with dilation factor of
Which kernel cells these ops utilize?
C:
6:
g
Sum
Sum
uniform
Detailed Illustration of how the counts are derived
Convolutions involving (a)
Convolutions involving (b)
D
Convolutions involving (c)

---

Convolutions involving (d): Rotated version of (b)
Convolutions involving
Convolutions involving (f)
dent
Convolutions involving (g): Rotated version of (c)
Convolutions involving (h): Rotated version of (f)
Convolutions involving (i): Regular uniform treatment

---

Mirror Padding (SYMMETRIC) with Grouping
Illustrated on 2x2 kernel and 1-pixel padding
A grouped padding strategy is applied to balance uneven padding (Wu et al 2019)
Padded top- left corner
Padded bottom- left corner
Padded top- right corner
Padded at bottom- right corner
Original Input
a
Number of conv ops each pixel is involved in
Padded at top- left corner
Padded at bottom- left corner
Padded at top- right corner
Padded at bottom- right corner
Average (grouped padding strategy)
2
4
4
4
4
4
4
4
4
Which kernel cells these ops utilize?
Padded at top- corner
Padded bottom- left corner
Padded top- right corner
Padded at bottom- right corner
Average (grouped padding strategy)
Convolutions involving (a)
2
0.75
0.5
Convolutions involving (b)
1.5
1.5
sum
sum
sum
sum 2
sum 4
Convolutions involving (c)
1.5
0.5
1.5
sum
sum
sum
sum =
sum
Convolutions involving (d)
uniform
unitorm
uniform
unifom
uniform

---

Detailed Illustration of how the counts are derived
Convolutions involving (a)
Padded at bottom- right corner
b
Padded at top- left corner
Padded at top- right corner
Padded at bottom-
a
a
b
left corner
C
C
d
Convolutions invalving (b)
Padded top- left corner
Padded at top- right corner
Padded at bottom- left corner
Padded bottom- right corner
Convolutions involving (c): Rotated version of (b)
Convolutions involving (d): Rotated version of (a)

---

Mirror Padding (SYMMETRIC) with Grouping
Illustrated on 4x4 kernel and 1-pixel padding
Original Input
Padded at top-left
Padded bottom-
Padded at top-right
Padded bottom- right corner
left corner
Number of conv ops each pixel is involved in
Padded at top-left
Padded at bottom-left
Padded at top-right
Padded at bottom-right
Average (grouped padding strategy)
20
12
16
20
20
18
18
18
20
20
18
18
18
12
12
16
18
38
12
12
18
18
18
18
18
18
18
20
20
18
18
18
20
18
18
12
12
18
18
12
12
18
18
18
18
18
16
Which kernel cells these ops utilize?
Padded at top-left
Padded at bottom-left
Padded at top-right
Padded at bottom-right
Average (grouped padding strategy)
(a)
2as
0.79
cam 25
om 12
rum 35
(b)
235
JUXX
(c)
2
2
2
05
25
23
(d)
rolated version of (b)
(e)
0.75
1.5
6.6
0.25
12
(
0.6
SUSTH 22
SAR 23
SUT IZ
SUST 25
( rotated version (c)
rotated version of (f)
regular treatment

---

Original Input
Padded Input
# of conv ops for each pixel
Replication Padding
Illustrated on 5x5 kernel and 2-pixel padding
a
Which kernel cells these ops utilize?
g
h:
E
SUITI Sum
Sum
sum
SUm
Sum 20
Sum 30
Sum 20
Sum uniform
Detailed Illustration of how the counts are derived
Convolutions involving (a)
Convolutions involving
D
aaabe
ARA
Raabo
Convolutions involving (c)
Maabe

---

Convolutions involving (d): Rotated version of (b)
Convolutions involving (e)
babe
Convolutions involving (f)
boabe
Convolutions involving (g): Rotated version of (c)
Convolutions involving (h): Rotated version of (f)
Convolutions involving (i): Regular uniform treatment

---

Mirror Padding (REFLECT)
Illustrated on 3x3 kernel and 1-pixel padding
# of conv ops each pixel involved in
Original Input
Padded Input
b
C
4
6
8
8
d
8
16
12
12
12
h
h
g
1
6
9
Which kernel cells these ops utilize?
b:
c:
d:
f:
h:
1
2
1
1
1
2
2
2
2 2
4
2 2
1
1
2
1
1
1
1
2
1
1
1
1
1
1
1
1
1
2
1
1
2
uniform
Detailed Illustration of how the counts are derived
Convolutions involving (a)
e
B
d
b
g
g
Convolutions involving (b)
h
h
Convolutions involving (c)
b
b
b
b
b
b
d
e
d
e
d
e
8
g h
h
h
g
h
h
g

---

Convolutions involving (d): Rotated version of (b)
Convolutions involving (e)
e
e
d
b
b
C
b
b
b
b
b
b
a
b
e
e
e
d
h
h
h
h
h
g
h
B
e
b
c
b
b
c
b
h
g
i
h
Convolutions involving (f)
e
e
e
h
g
Convolutions involving (g): Rotated version of (c)
Convolutions involving (h): Rotated version of (f)
Convolutions involving (i): Regular uniform treatment

---

Partial Convolution
Illustrated on a 3x3 kernel
Weighted of conv ops each pixel is involved in
Input
8.79
1245
10.5
10.5
10.
h
7.6
10.5
9
9
9
75
10.
9
9
10s
Which kernel cells these ops utilize?
1.5
1
1
1.5
3.5
1.5
LG 2.25
15
1.5
2.55
15
15
1.5
1.
1
1
15
3.5
1.5
1.5
15
222
1.5
Detailed Illustration of how the counts are derived
Convolutions involving (a)
weight
9/4=
2.25
9/6 =
1.5
9/6
1.5
9/9-
sum 6.25
Convolutions involving (b)
b
weight
9/4
2.25
9/6
1.5
9/6
1.5
9/9
9/6
1.5
9/9-
sum 8.75
Convolutions involving (c)
6
C
b
c
weight
9/6
1.5
9/9 =
9/6
1.5
9/9
9/6
1.5
9/9
sum 7.5

---

Convolutions involving (d): rotated version of (b)
Convolutions involving (e)
weight
=
2.25
9/6
155
9/6
1.5
9/6
1.5
9/9=
1
9/9
0
weight
15
sum 12.25
Convolutions involving (f)
0
0
0
weight
1.5
3
weight
9/66=
1.5
9/9
9/9
sum = 10.5
Convolutions involving (g): Rotated version of (c)
Convolutions involving (h): Rotated version of (f)
Convolutions involving (): Regular uniform treatment


Reviews:
[{"review_id": "AFoBk-NNUyu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper does empirical analysis on an object detector, esp. how it fails due to slight shift of objects in videos. By using different padding schemes for conv filters, the authors find the padding scheme is the root for such failures.\n\nUpdate after rebuttal:\nThe author responses and manuscript revisions have addressed most of my concerns. Now I lean towards acceptance. \n\nReasons for score:\n1. This paper is not written professionally. The introduction is very short. The following sections are more like experimental notes than a technical paper. \n2. The analysis is mostly based on a single use case, i.e. an SSD object detector for traffic lights. This study is highly insufficient. More use cases and application scenarios should be investigated.\n3. Experimental evaluation is also highly insufficient. Only changing the input image size by 1 pixel doesn't reveal much. A lot of ablation studies and changing of padding schemes should be evaluated, preferably on a few datasets of different tasks (classification, detection, segmentation, etc.).\n\nOther comments:\nI agree that padding could lead to biases when doing downsampling using stride > 1, as illustrated in Fig. 6. However, this seems to be easily fixed by doing random image flipping as one of the data augmentations? It would even this bias. So this issue may not be practically severe, although it is worth study per se.\n\nMinor issues:\nThe PDF file of this paper seems to have issues. The printed characters are blurry, and acrobat reports errors when open it. But some other PDF readers can open it.\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "8OL6bH6hXTb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses a commonly neglected factor in CNN design, namely the consequences of typical compromises made in padding the feature maps for convolutions and resamplings. On the downside, much of the material is known in some form in literature or practice, but on the other hand, the paper has value in systematically bringing together and analyzing these effects, and does identify and quantify effects that are apparently novel.\n\nIt is a common belief that choices like padding are minor technical details, which, even when sloppy, the network will just somehow learn to tolerate. This is to some extent true and it is remarkable how well CNN's work even when apparently crippled by questionable design. However, there is a cost and the network may need to make sub-optimal compromises to work around these issues. The paper's demonstration of lopsidedness of the learned filters is apparently novel and drives this home in an interesting way: to handle the uneven padding, the network must make a compromise where the filter kernels are slightly distorted so that they can still make a decent job of both the interior while tolerating the drastically broken boundary. However this \"decent job\" is not necessarily as good as it could be if the boundary defect was absent -- here the paper proposes a very simple change that demonstrably both improves performance and eliminates the kernel distortion.\n\nI do wonder how these findings compare to e.g. the results of the explicit padding paper, which tackles these issues head on in a principle yet more complicated manner. It seems like e.g. the poor performance of standard padding in the Gaussian blur example in that paper could be explained by effects similar to the aforementioned distortion of the filters in the interior.\n\nThe paper also identifies the phenomenon of arbitrarily distorted predictions made in vicinity of padding boundaries, leading to incorrect predictions when small objects occupy this space. While unsurprising, apparently this issue has received little attention in literature, perhaps in part because e.g. imagenet data is labeled based on what is at the center of the image. In any case, higher detection accuracy is reported for many tasks when the boundary handling is improved.\n\nIn addition to these the paper analyzes the number of connections between output and input pixels when various padding strategies are used and makes some recommendations about the most balanced choices. The foveation patterns of most coarse padding methods do seem unhealthy and it's plausible that it's connected to the arbitrary nonlinear distortion in prediction scores near boundaries. While a thorough connection is not necessarily established, this is at least hinted by Fig 5b.\n\nThe paper is clearly written and easily understandable.\n\nOverall, while the findings of the paper are arguably fairly simple and elementary compared to many works in ICLR, bringing attention to this sort of effects is valuable and I did learn something from reading the paper, so I would lean slightly towards accepting it.\n\n_Update after rebuttal_: The authors have been very forthcoming with further analyses in the rebuttals. I believe the findings deserve to be published and that bringing attention to these issues is a good thing, and accordingly I have bumped my rating up a bit.", "extra_scores": {}, "extra_text": {}}, {"review_id": "64eXg3UvNXc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the effect of padding on artefacts in CNN feature maps and performance on image classification and object detection. It convincingly makes the case that these artefacts have a significant detrimental effect on task performance, e.g. leading to blind spots / missed detections of small objects near the image border. It also studies the effect of uneven padding in downsampling layers, where the padding may only affect some sides of the image and not others, depending on the image size. A condition is presented for when this does / does not occur. The effect of different padding methods is also studied from the perspective of foveation by computing the number of paths from an input pixel to the output. A number of practical recommendations are given.\n\nThe paper is well written. It contains lots of details that are relevant to CNN architecture design, especially when the appendix is taken into account. Proposed fixes are simple and produce a very significant improvement in performance on imagenet classification and object detection, so are likely to be adopted by practitioners. \n\nThe paper states:\n\"It is evident that the 1-pixel border variations in the second map are caused by the padding mecha-nism in use. This mechanism pads the output of the previous layer with a 1-pixel 0-valued border inorder to maintain the size of the feature map after applying a 3x3 convolutional kernel. The maps inthe first layer are not impacted because the input we feed is zero valued. Subsequent layers, however,are increasingly impacted by the padding, as preceding bias terms do not warrant 0-valued input.\"\nIt would be interesting to know if batchnorm or some other kind of normalization might mitigate this issue, because if the feature map is constant but non-zero, normalization will make it all zero. Of course this will not hold for non-zero (natural) inputs, but it would still be interesting to see a discussion on the effect of (batch) normalization on padding artefacts.\n\n\nTypos\nSection 4: \"To serves\"\nSection 5: RseNet\n\n------ \nUPDATE\nI have now read the other reviews, author response and updated paper, and have decided to maintain my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4TMA1n5Yj4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of Submission:\n\nThis submission points out that 0-padding in convolutional networks induces areas where responses for object detections are significantly decreased. This leads to misdetections of small objects. The submission studies the case of traffic lights. Analysis of filter response maps at various layers of the network and analysis of the average filter weights show the effect of the 0-padding. Making the padding symmetric and using mirror or reflection padding are shown to tackle the issue.\n\nUpdate after Rebuttal:\nI'd like to thanks the authors for addressing the comments really thoroughly. The additional experiments using anti aliased networks are indeed very interesting. I agree with AnonReviewer4 that some of the material in this submission is probably known to people already. However, I do strongly believe that many people are unaware of the impact and importance of these effects and therefore I believe publication of this content is important.\n\nStrengths:\n\n- Padding is omni-present in modern CNN architectures and its effect is not often analyzed. This submission contains a thorough analysis, points out the problem with 0-padding and shows how various alternatives show significant improvement.\n\n- Especially the analysis of the asymmetric padding that leads to asymmetric filters  in Section 5 is very interesting.\n\n- The text is well written and easy to follow.\n\n\nWeaknesses:\n\n- The introduction describes how the issue with the 0-padding leads to drastic differences in object detection between frames in a video. There has been work [39] which also showed that aliasing in CNNs causes these effects which does not seem to be discussed in this manuscript. What would happen to the effects of the 0-padding if an anti-aliased network would be used?\n\n- The experiments are focused on traffic lights. Given the generality of the issue presented in the manuscript I feel that additional experiments with other small objects or related problems could have been included.\n\n\nExplanation of Rating:\n\nThis submission describes a problem with 0-padding that significantly impacts an important problem and shows how to mitigate the situation. Therefore I recommend accepting this submission.\n\n\nMinor Comment:\n\nPage 5: layer in RseNet -> ResNet", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission explores how certain common padding choices can induce spatial biases in convolutional networks. It looks into alternative padding schemes which mitigate these issues and demonstrates significant performance improvements in widely used convnets. Reviewers generally agreed that this is an important point that should be more widely understood in the community, and that the proposed changes are relatively simple to adopt, so this work is likely to be impactful. Most reviewers thought the paper was well-written, describing the problem well, and the analysis well-executed. Most reviewers acknowledged that most of the weaknesses described in their initial reviews were well-addressed by the authors' responses and manuscript updates. Given the strength of the analysis and the impact for many practitioners, I recommend the submission be accepted with a spotlight presentation.
Decision: Accept (Spotlight)