Boundary Effects in CNNs: Feature or Bug?
Authors: Md Amirul Islam
Matthew Kowal
Sen Jia
Konstantinos G. Derpanis
Neil Bruce
Venue:  (2021)
Keywords: Boundary Effects
Absolute Position Information
Padding
Canvas color
Location Dependent Task

Abstract: Recent studies have shown that the addition of zero padding drives convolutional neural networks (CNNs) to encode a significant amount of absolute position information in their internal representations, while a lack of padding precludes position encoding. Additionally, various studies have used image patches on background canvases (e.g., to accommodate that inputs to CNNs must be rectangular) without consideration that different backgrounds may contain varying levels of position information according to their color. These studies give rise to deeper questions about the role of boundary information in CNNs, that are explored in this paper: (i) What boundary heuristics (e.g., padding type, canvas color) enable optimal encoding of absolute position information for a particular downstream task?; (ii) Where in the latent representations do boundary effects destroy semantic and location information?; (iii) Does encoding position information affect the learning of semantic representations?; (iv) Does encoding position information always improve performance? To provide answers to these questions, we perform the largest case study to date on the role that padding and border heuristics play in CNNs. We first show that zero padding injects optimal position information into CNNs relative to other common padding types. We then design a series of novel tasks which allow us to accurately quantify boundary effects as a function of the distance to the border. A number of semantic objectives reveal the destructive effect of dealing with the border on semantic representations. Further, we demonstrate that the encoding of position information improves separability of learned semantic features. Finally, we demonstrate the implications of these findings on a number of real-world tasks to show that position information can act as a feature or a bug.

Under review as a conference paper at ICLR 2021
BOUNDARY EFFECTS IN CNNS: FEATURE OR BUG?
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent studies have shown that the addition of zero padding drives convolutional neural networks (CNNs) to encode a significant amount of absolute position infor- mation in their internal representations, while lack of padding precludes position encoding. Additionally, various studies have used image patches on background canvases (e.g., to accommodate that inputs to CNNs must rectangular) without consideration that different backgrounds may contain varying levels of position information according to their color. These studies give rise to deeper questions about the role of boundary information in CNNs, that are explored in this paper: (i) What boundary heuristics (e.g., padding type, canvas color) enable optimal encoding of absolute position information for a particular downstream task?; (ii) Where in the latent representations do boundary effects destroy semantic and loca- tion information?: (iii) Does encoding position information affect the learning of semantic representations?: (iv) Does encoding position information always improve performance? To provide answers to these questions, we perform the largest case study to date on the role that padding and border heuristics play in CNNs. We first show that zero padding injects optimal position information into CNNs relative other common padding types. We then design a series of novel tasks which allow to accurately quantify boundary effects as a function of the distance to the border. A number of semantic objectives reveal the destructive effect of dealing with the border on semantic representations. Further, we demonstrate that the encoding of position information improves separability of learned semantic features. Finally, we demonstrate the implications of these findings on a number of real-world tasks to show that position information can act feature bug.
INTRODUCTION
One of the main intuitions behind the success of CNNs for visual tasks such image classifica-
tion (Krizhevsky et al.) 2012, [Simonyan & Zisserman 2015; Szegedy et al.) 2015 Huang et al.)
2017), video classification (Karpathy et al.] [014 Yue- Hei Ng et al. [015 Carreira & Zisserman
2017). object detection (Ren et al.) 2015) Redmon et al. 2016 He et al. (017)), generative image models (Brock et al. 2018), and semantic segmentation (Long et al.) 2015 Noh al. 2015 Chen ct al. 2017| is that convolutions add visual inductive bias to neural networks that objects can appear anywhere in the image. To accommodate the finite domain of images, manual heuristics (e.g.. padding) have been applied to allow the convolutional kernel's 's support to extend beyond the border of an image and reduce the impact of the boundary effects (Wohlberg & Rodriguez [017]] Tang et al.)
2018; Liu et al. 201 Sa, Innamorati et al. 2019 Liu et al.) 2018b). Recent studies (Pérez et al. 2019 Islam et al. 2020 Kayhan & Gemert 2020) have shown that zero padding allows CNNs to encode absolute position information despite the presence of pooling layers in their architecture (e.g., global average pooling). In our work, we argue that the relationship between boundary effects and absolute position information extends beyond zero padding and has major implications in CNN's ability to encode confident and accurate semantic representations (see Fig.
An unexplored area related to boundary effects is the use of canvases (i.e., backgrounds) with image patches (see Fig. []] top row). When using image patches in a deep learning pipeline involving CNNs, the user is required to paste the patch onto background due the constraint that the image must be rectangular. Canvases have been used in a wide variety of domains, such as image generation (Gre- gor et al., 2013 Huang et al., 2019), data augmentation (DeVsies & Taylor 2017). image inpaint- ing (Demir& & Unal 2018 Yu et al. 2018), and interpretable Al (Geirhos et al. 2018 Esser et al. 2020). To the best of our knowle moweledge, this paper contains the first analysis done on canvas value selection.

---

Under review as a conference paper at ICLR 2021
In other works, the canvas value is simply cho- sen based on the authors intuition. Given the pervasiveness of CNNs in multitude of appli- cations, is of paramount importance to fully understand what the internal representations encoding in these networks, as well as isolating the precise reasons that these representations are learned. This comprehension can also allow for the effective design of architectures that over- come recognized shortcomings (e.g.. residual connections (He et al., 2016) for the vanishing gradient problem). As boundary effects and po- sition information in CNNs are still largely not fully understood, we aim to provide answers to the following hypotheses which reveal funda- mental properties of these phenomenon:
Innut
Par
Impact Padding
Impact of Padding
welo
Impactio
Figure I: An illustration of how CNNs use posi- tion information to resolve boundary effects. We place CIFAR-10 images in random locations on a canvas of 0's (black) I's (white). We evaluate if a ResNet-18, 18, trained w/ or w/o padding for seman- tic segmentation,
Hypothesis I: Zero Padding Encodes Max- imal Absolute Position Information: Does zero padding encode maximal position infor- mation compared other padding types? We evaluate the amount of position information in networks trained with different padding types and show zero padding injects more position information than common padding types, e.g., reflection, replicate, and circular.
segment the image region. Surprisingly, performance is improved when either zero padding black canvas is used, implying position information can be exploited from border heuristics reduce the boundary effect. Colormap is 'viridis': yellow is high confidence.
Hypothesis Different Canvas Colors Affect Performance: Do different background values have an effect on performance? If the padding value at the boundary has a substantial effect CNNs performance and position information contained in the network, one should expect that canvas values may also have similar effect.
Hypothesis III: Position information is Correlated with Semantic Information: Does a net- work's ' ability to encode absolute position information affect its ability encode semantic informa- tion? zero padding and certain canvas colors can affect performance on classification tasks due to the increased position information, networks ability to encode semantic information. We demonstrate that encoding position information improves the robustness and separability of semantic features.
expect that the position information is correlated with
Hypothesis IV: Boundary Effects Occur at All Image Locations: Does a CNN trained without padding suffer performance solely the border, or at all image regions? How does the performance change across image locations? Our analysis reveals strong evidence that the border effect impacts CNN's performance all regions the input, contrasting previous assumptions (Tsotsos al. 1995: Innamorati et al., 2019) that border effects exist solely at the image border.
Hypothesis V: Position Encoding Can Act as a Feature or Bug: Does absolute position informa- tion always correlate with improved performance? A CNN's ability to leverage position information from boundary information could hurt performance when task requires conslationinvvarianne e.g.. texture recognition; however, it can also be useful if the task relies position information, e.g.. semantic segmentation.
To give answers to these hypotheses (hereon referred to as H-X), we design series of novel tasks as well as use existing techniques to quantify the location information contained different CNNs with various settings. In particular, we introduce location dependant experiments (see Fig. 2 which use grid-based strategy to allow for a per-location analysis of absolute position encoding and performance semantic tasks. The per-location analysis plays critical role in representing the boundary effects as a function of the distance to the image border. We also estimate the number of dimensions which encode position information in the latent representations of CNNs. Through these experiments show both quantitative and qualitative evidence that boundary effects have a substantial effect on CNNs in surprising ways and then demonstrate the practical implications of these findings on multiple real-world applications. Code will be made available for all experiments.
2

---

Under review as a conference paper at ICLR 2021
Table 1:
Position encoding results w/ metries SPC1: high is better and MAEJ: low is better, w/ different padding types. i denotes zero-padding based methods.
Image Classification
Semantic Segmentation
Input
Prediction
Input
Prediction
deer
CNN
CNN
Horizontal SPCT
Gaussian
3
Classes
3
Padding
MAE]
SPCt
MAE
Figure 2: We consider two location dependant tasks designed to investigate the boundary effects in CNNs. random image is placed on a random grid location and the CNN predicts either C' class logits (a: classification), or C' class logits for each pixel (b: segmentation).
Zero Pad!
.406
.216
.591
.146
Partiall
.424
.213
.604
.144
Circular
.296
.236
.455
.165
Replicate
.218
.241
.396
.173
Reflect
.12
.242
.409
.172
w/o Pud
.204
.243
.429
.168
ABSOLUTE POSITION INFORMATION IN CNNS
What Type of Padding Injects Optimal Location Information? With the ultimate goal of revealing characteristics that determine the impact that boundary effects plays in CNNs with respect to absolute position information, we first determine which commonly used padding type encodes the maximum amount of absolute position information. We evaluate the ability of different padding types (i.e., zero, circular, reflection, and replicate) to encode absolute position information by extending the experiments from (Islam al. (2020), which only considered zero padding. We first train simplified VGG network (Simonyan & Zissermann 2013) with five layers (VGG-5, see Appendix [A.2 for implementation details) on Tiny ImageNet (Le & Yang 2015) for each padding type. We follow the settings in (Islam et al.) 2020): a read-out module, trained using DUT-S (Wang et al. 2017) images, takes the features from a frozen VGG-5 model's s last layer, pre-trained on Tiny ImageNet, and predicts gradient-like like position map (see top row in Table. I]. We experiment with two position maps, which are the same for every image: (i) 'horizontal' and (ii) 'Gaussian', These gradient- like position maps change smoothly from 0 to 1, from the dark-blue to yellow, respectively. For a fair comparison with (Islam et al., 2020), we report results using Spearman Correlation (SPC) and Mean Absolute Error (MAB) with input images from PASCAL-S (Li et al. 2014). From Table it is clear that zero padding delivers the strongest position information, compared with replicate, boundary reflection, and circular padding, supporting H-I. Note that partial convolution (Liu et al. 2018a) still pads with zeros, but re- re-weights the output of the convolution based on how many zeros are padded. Thus, position information is still encoded when partial convolutions are used. Interestingly, circular padding is often the second most capable padding type. We conjecture this is because circular padding takes values from the opposite side of the image where the pixel values are typically less correlated than the directly neighbouring pixels. Thus, circular padding often has a value transition at the border, contrasting reflection and replicate which offer little or no signal the CNN regarding the whereabouts of the image border.
3 LOCATION DEPENDANT TASKS FOR POSITIONAL ANALYSIS
We begin by describing our experimental settings and the implementation details for the proposed location dependant experiments with grid-based inputs. These experiments are used to analyze the border effects with respect to position information encoded in CNNs. These consist of location dependant image classification (Fig. (a) and S...B.11, and segmentation (Fig.ā and Sec. (32)). under different canvas color settings. Our experiments are designed with the goal of determining, for different canvas colors (H-II), where in the input CNNs suffer from the border effect (H-IV), and how the encoding of position information affects the learning of semantic features (H-III).
Experimental Settings and Implementation Details. Our image classification and segmentation experiments use 'location dependant' inputs (see Fig. aabove and Fig. 9 in the appendix for more detailed examples). The input is colored canvas (the colors used are Black (0, O, 0J. White [L, 1,1. and the CIFAR-10 dataset (Krizhevsky et al., 2014) Mean [0.491, 0.482, 0.446) with an image patch randomly placed on a l x le grid. Unless mentioned otherwise, we use CIFAR- 10 for all experiments. Given a 32 x 32 CIFAR-IU 10 training image as the image patch, we randomly choose a grid location, L. and place the CIFAR- 10 training sample in that location. For example, in the case of a k x & grid, the size of the grid canvas is 32k x 32k, where each grid location has a size of 32 x 32 and A2 total locations (see Fig. 9 in the appendix). All experiments are run for & E 53 7, 9, 11, 13, 15). To ensure a fair comparison between grid locations, the evaluation protocol consists of running the entire validation set of CIFAR-10 10 on each individual grid location (i.e., we the validation set 122 times
3

---

Under review as a conference paper at ICLR 2021
Table 2: Location dependant (a) image classification and (b) semantic segmentation accuracy on CIFAR-I0 under zero/no padding settings and canvas colors Black, White, and Mean.
Image Classification
Image Segmentation
Pudding
3x3
5x5
7x7
9x9
11x11
13x13
15x15
3x3
5x5
7x7
9x9
11x11
13x13
15x
Zero Pad
78.4
74,9
74.3
76.2
76,9
75.2
76.9
68.2
67.8
67.8
67.7
64.0
63.4
60.3
w/o Pad
76,4
74.0
67.6
67.5
67.4
60.3
47.2
59.5
53.5
51,4
50.4
48.1
48.3
45.6
Zero Pad
81.2
80.4
80.3
80,3
80,3
80.1
80.0
69.0
64.3
63.2
54,8
49.1
45,9
47.4
w/o Pad
79.8
76.3
41.6
34.2
36,6
24.9
19.0
57.7
32.8
31,5
26.4
23,1
17.2
19.2
Zero Pad
81.3
80,5
78,9
78.9
78,8
80.4
80.2
66.1
63.8
57.5
55.0
51.6
49,5
42.5
w/o Pad
81.1
77.5
68.8
34,8
31,6
32.8
26.2
42.0
35.4
29.6
26.3
24.1
23,9
24.5
for a single validation epoch). We then average the performance over all grid locations to obtain the overall accuracy. We report classification and segmentation accuracy in terms of precision and mean intersection over union (mloU), respectively. We use a ResNet- 18 network trained from scratch, unless stated otherwise. ResNets with no padding are achieved by setting the padding size to zero in the convolution operation. For fair comparison between the padding and no padding baseline, we use bilinear interpolation (see Appendix [ for discussion) to match spatial resolutions between the residual output and the feature map for the no padding case, which was not accounted for in previous work (Kayhan& & Gemert 2020
3.1 LOCATION DEPENDANT IMAGE CLASSIFICATION
We investigate whether CNNs trained with and w/o padding are equally capable of exploiting absolute position information to predict the class label in all image locations, with respect to the distance from the image boundary and for variable grid sizes. The location dependant image classification experiment is multi-class classification problem, where each input has a single class label and the CNN is trained using the multi-class cross entropy loss (see Fig. 2 (a)). Therefore, the network must learn semantic features invariant to the patch location, to reach a correct categorical assignment.
Table (left) shows the location dependant image classification results. For all canvases, the networks trained with padding are more robust to changes in grid sizes. In contrast, models trained w/o padding significantly drop in performance with the increase of grid size, position information is lost and boundary information cannot be exploited. Further, the canvas colors seem to have a noticeable effect on classification performance (H-III as the white and mean canvases have a much more significant performance drop than the black as the grid size increases. The difficulty in separating image semantics from the background signal is due to non-zero canvases creating noisy activations at regions near the image patch border, which is explored further in Section 5
3.2 LOCATION DEPENDANT IMAGE SEGMENTATION
The experiment in this section examines similar properties as the previous location dependant image classification, but for dense labelling scenario. This task is simply a multi-class per-pi per-pixel classification problem, where each pixel is assigned a single class label. We follow the same grid strategy classification to generate training sample. Since CIFAR-10 is classification dataset and does not provide segmentation ground-truth, we generate synthetic ground-truth for each sample by assigning the class label to all the pixels in the grid location where the image belongs to (see Fig. 2 (b)). Following existing work (Chen et al. 2017). we use a per-pixel cross entropy loss to train the network. For evaluation, we compute mloU at per grid location and take the average report results.
Image segmentation results are shown in Table (right). A similar pattern is seen as the classification experiment (Sec. 3.Tp. Networks trained with padding consistently outperform networks trained w/o padding, and the difference grows larger as the grid size increases. Contrasting the classification experiment, the performance of networks with padding decreases slightly as the grid size increases. The reason for this is that the mloU metric is averaged across all categories including the background, so object pixels are equally weighted in the mloU calculation even though the ratio of background pixels to object pixels increases dramatically for larger grid sizes. For the no padding case, we observe similar patterns to the classification experiment as the white and mean canvas scenarios suffer more from a large grid size than the black canvas case. This finding further suggests that, independent of the task, a black canvas injects more location information to a CNN (H-II), regardless of the semantic difficulty, than white or mean colored canvas, which is further explored in Sec. 5
4

---

Under review as a conference paper at ICLR 2021
Image pairs reflecting semantic concepts
Table Dimensionality estimation (%) of two semantic concepts (location and semantic category) under different tasks and settings.
Semantics Locatin
Segmentation
Classification
Canvas
Grid
Padding
Rioeation
RChs
Laacation
PCtass
Zero Pad
15.2%
14.9%
12.7%
12.6%
Black
7x7
No Pad
12.7%
12.8%
12.1%
11.9%
Sample pair Sample pair 2
Zero Pad
12.5%
12.3%
12.2%
12.1%
Figure 3: Sample pair generation reflect-
White
7x7
No Pad
10.9%
10.9%
11.5%
11.6%
ing two semantic concepts.
INTERPRETING REPRESENTATIONS FOR DIMENSIONALITY ESTIMATION
Previous works (Bau et al. 2017 Esser et al. 2020 proposed various mechanisms to interpret different semantic concepts from latent representations by means of quantifying the number of neurons which encode particular semantic factor, k. Given a pretrained CNN encoder EE()) where 2 latent representation and given an image pair (I", 1") p(I", I*) which are similar in the k-th semantic concept, we aim to estimate the dimensionality of the semantic factor, 2g. that represents this concept in the latent representation. A positive mutual information between / and 7o implies similarity of I and 7o in the k-th semantic concept, which will be preserved in the latent representations (T) and TE (T)), only if E encodes the k-th semantic concept. Following (Esser et al. 2020), we approximate the mutual information between E(I*) and IE(1") with the correlation of each dimension in the latent representation (see Sec. A.3 in Appendix for details).
We generate image pairs which share one of two semantic concepts: (i) location (ii) semantic class. For example, the image pair sharing the location factor (see Fig row) differs in the class and canvas color, while the pair the bottom color and location. With this simple generation strategy, we can accurately estimate the number of dimensions in the latent representation which encodes the k-tt semantie factor. Note that the remaining dimensions not captured in either the location or semantic class is allocated to the residual semantic factor, which by definition will capture all other variability in the latent representation, z.
shares the semantic class but differs in canvas
Table 3 shows the estimated dimensionality for the semantic factors location and class. The latent representation used is the last stage output of ResNet- 18 before the global average pooling layer. We used the networks from Sec. 3 which are trained for segmentation (left) and classification (right) with the appropriate background (i.e.. black on the top and white on the bottom row) and grid settings. The results clearly show that networks trained with pero-padding contain more dimensions which encode the semantic factor 'location' (H-I). Further, Table shows that there a positive correlation between the encoding of location and the encoding of semantics, i.c. a larger number of dimensions encoding location implies larger number of neurons encoding semantics, supporting H-lII. More dimensionality estimation results can be found in Sec. A.5.1 in the appendix.
5 PER-LOCATION ANALYSIS
In this section, take advantage of the grid-based learning paradigm and conduct further evaluations on a per-location basis to test H-I, H-II, H-III, and H-IV. In particular, we analyze the relationship between zero padding and the border effect. We then show quantitative and qualitative results which reveal strong evidence that zeros, whether as canvas or padding, inject maximal location bias.
Distance-to-Borda Analysis: What Input Regions Suffer Most from Border Effects? First, we analyze the image classification and segmentation results reported in Secs. 3.1 and 3.2) with respect to the distance from the closest border which will allow us to answer this question. To obtain the accuracy at each distance, we average the accuracies over all grid locations with the same distance to the nearest border (e.g., a distance to a border of zero refers to the average accuracy of the outer-most ring of grid locations). Figure (left) shows the accuracy difference between the padding baseline (the blue horizontal line) and the no padding cases. Interestingly, the accuracy difference is higher at grid locations close to the border and decreases towards the image center. This analysis strongly suggests that zero padding significantly impacts the border effect, and injects position information to the network function of the object location relative the distance of the nearest border. In
5

---

5). Right: Sample Under review as a conference paper at ICLR 2021
Faiding
pobline
pabling Zero Parkdigg
R.1ck Whow Nean
God
1x 13
11 1i
Figure 4: Left: Location dependant image classification (left two) and segmentation (right two). Results show the accuracy difference between padding and no padding, various distances to the border and canvases, Right: Heatmap visualization of 13 x 13 grid per-location segmentation mloU on CIFAR- (Krizhevsky et al., 2014) under black (left), white (middle), and mean (right) background settings. As can be seen, CNNs without padding have difficulties near the border.
contrast, the no padding case fails to deliver any position information at the border locations which leads to significant performance drop. Also note that there is substantial difference in performance at the center of the image, at the farthest distance from the border, supporting H-IV. We also visualize heatmap in Fig. (right) which shows consistency with the results in Fig. 4 (left). As shown in Fig. (right), the networks trained w/ zero padding perform consistently across all grid locations while the networks trained w/o padding perform poorly near the border. Note that of the three canvases for the no padding case, the black canvas yields the lowest drop in relative performance when comparing the center region to locations near the border (H-IT).
the accuracy at all grid locations
Are Border Effects Only at the Border? While intuition might suggest the border effect occurs solely the border, is natural analyze other regions in the input space also suffer from the border effect. Figure 5 (left) compares filter activations with and without zero padding. Note that filter activations are randomly sampled from the feature map for the specific layer. Activations found near the border propagate less information through the network during the forward pass due to the limited amount of connectivity to downstream layers compared to activations at the center, as discussed in (T'sosoos et al., 1995). Further, the convolution cannot fully overlap the border regions without padding and thus will not recognize objects as well. This phenomenon can be seen in Fig. (bottom-left), where the activations for grid location case. Interestingly, for grid location 13 (i.e., center), there is also a visible difference in the activation space. Here, activations found for the no padding case are blurred and noisy which contrasts the tight square shaped activations when zero padding is used. While border effects mainly impact regions near the border, these results show clear evidence that input locations at the center of the image are also impacted with lack of padding which is evidence supporting H-IV. This also explains the large performance drop the center of the grid in Fig- (left and right).
significantly reduced in the padding
Does Encoding Location Enable the Learning of Semantics? In Sec. we provided quantitative evidence that reveals the correlation between the number of neurons encoding position and semantic information (H-III). We further investigate this phenomenon to see how position information by means of zero padding allows richer semantics to learned for the tasks of image classification and semantic segmentation. Figure 5 (right) heatmaps show segmentation predictions for different grid locations, L. a x grid. When no padding is used CNNs have difficulty segmenting images
who Inage
6
Pad
eio Pad
Plad
Wif Pwo
1-1
20
30
L
Figure 5: Left: Filter activation visualization for the classification task on CIFAR-10 with white background and 5 x grid size. is clear that zero padding provides richer information and larger activations downstream, particularly at locations near the boundary (e.g., predictions of semantic segmentation on different locations of 7 x 7 grid under three background settings. Confidence maps are plotted with 'cividis' colormap, where yellow and dark blue indicates higher and lower confidence, respectively.

---

Under review as a conference paper at ICLR 2021
Black
White
Mean
Black
White
Mean
Black
White
Moon
Layer
Layer
Layer 17
Figure 6: Comparison of filter activations for the location dependant segmentation task trained without padding, 5 x 5 grid size, L = 13, and three canvas colors, black, white, and mean. Notice the large activations in the background region for black, contrasting that of white and mean.
near the border (highlighted with circles in Fig. (5) except when black canvas is used. However, for locations near the center of the image, reduced position information due to no padding greatly reduces the network's 's confidence in semantic encodings. In contrast, zero padding is consistent and confident in segmenting objects across all the grid locations and canvas colors. Further, we use t-SNE (Maaten Hinton| 2008) to visualize the classification logits in Fig. 12 of appendix. The separability of the semantic classes is significantly improved when padding is used, and the effect is particularly pronounced at locations near the border (I. = 7). This further supports H-IIT that absolute position information, by means of zero padding, enables CNNs to learn more robust semantic features, which in turn allows for greater separability in the prediction logits.
Why Do Explicit Zeros Inject Location Information? We now explore what enables CNNs to encode positional information when zeros exist at the boundary (i.e., as padding or canvas (H-Il)) by analyzing the activations of network trained for the location dependant segmentation task. For k k grid, the ratio of canvas pixels to total pixels is 62-1 This implies that 3:2 the vast majority of labels will be the background class, and therefore the majority of filters should focus on correctly labelling the canvas. To determine if this is true for all canvases, we visualize randomly sampled filter activations (see Fig. 6) for networks trained without padding for the location dependant segmentation task. The activations are visualized using the 'gray' colormap, where light and dark intensities denote high and low activations, respectively. Note that the activations are taken from the output of the convolutional layer and are normalized to between [0, 1] before plotting. Even at the earlier layers (e.g., layer 7), there is clear difference in the patterns of activations. The majority of filters have low activations for the image region, but high activations for the background region. In contrast, the white and mean canvases have mostly low activations for the canvas but high activations for the image. Interestingly. particularly at layer 17 (the last convolution layer), the activations for the black background are reminiscent of oriented filters (e.g., Gaussian derivative filters) in a number of different orientations and locations, indicating they can capture more diverse input signals compared the white and mean canvases, which consistently activate over the center of the input region. Figure eleurlyy demonstrates that zeros at the boundary, in the form of a black canvas, allows easier learning of semantics and absolute position for CNNs compared to other values supporting H-ll.
Canvas Analysis:
6 APPLICABILITY TO SEMANTIC SEGMENTATION, TEXTURE RECOGNITION, DATA AUGMENTATION, AND ADVERSARIAL ROBUSTNESS
Given the intriguing findings above, it is natural to ask how much the demonstrated phenomenon affects real world tasks with SOTA architectures. More specifically, does encoding position always improve performance or does it cause unwanted effects on certain tasks (H-V)?
Semantic Segmentation. We now measure the impact of zero padding segment objects near the image boundary with a strong semantic segmentation network on an automotive-centri dataset. We use the DeepLabv3 (Chen al. 2017) network and the Cityscapes (Cordts et al. 2016) dataset, trained with different padding types. From Table 4 it is clear that DecpLabv3 with zero padding achieves superior results compared to the model trained without padding or with reflect padding. Additionally, we perform an analysis by computing the mloU for rectangular ring-like regions (see Fig. 7 (top-left)), between X% and Y%, where X and Y are relative distances from the border (e.g., 0% 5% is the outer most region of the image, while 5% - -00% is the neighbouring inner 5% region) to quantify the performance decrease from the boundary effect and lack of positional information. From Table | the performance drop between the total mloU (100%) and the border region (0-5%) is more significant for the no padding case and reflect padding case compared to the zero padding case, which agrees with the results found in Sec. 5 This further demonstrates that the absolute position
7

---

Under review as a conference paper at ICLR 2021
Table 4: Performance comparison of DeepLabv3 w/ and w/o padding for different image regions, Top- left image in Fig. 7 shows outer re- gions used for this analysis.
Evaluation Region mloU(9)
Method
0-5%
5-10%
10-15%
100%
Zero Pad
72.6
72.7
73.8
74.0
WC Fadding
Fadding
Reflect
71.9
72.0
73.7
73.9
Figure 7: Example predictions on the Cityscapes validation set when training w/ and w/o padding. Best viewed zoomed in.
No Pad
63.7
66.4
67.3
69.1
Table 5: (a) Texture recognition results on two datasets with different padding types. (b) Perfor- mance and robustness of DeepLabv3 variants trained with Cutout VDees & Taylor 2017 using two canvas (Black and White).
GTOS-M M
DTD
Padding
Segmentation
Robustness
Res34
Res50
VGG5
Res34
Res50
VGG5
Method
B
W
W
No Pad
71.7
76.3
33.6
57.5
67.0
27.3
Desv--Re50
73.9
74.1
53.7
55.8
Zero Pad
78.7
81.7
39.7
68.6
70.6
32.8
DLv3-Res101
75.5
75.2
51.9
Reflect
80.6
85.0
43.1
68.8
71.2
34.0
(b)
(a)
information due to zero padding improves the performance at all image regions, while reflect padding not as beneficial the image boundaries. Figure shows examples of how DeepLabv3 trained with zero padding generates more accurate predictions. particularly near the border of the image. Note that thin or complex objects near the border regions are particularly affected (e.g., light posts). The reason that performance suffers even with padding, is the lack of semantic and contextual information near the border, which is not the case for grid-based tasks (Sec. 3} since the image patch contains the entire CIFAR- 10 image. Additional results can be found in Sec. A.8 Sec. in the Appendix.
Texture Recognition. We evaluate three models with three padding types on the task of texture recognition. We use ResNet-34, ResNet-50, amd VGG-5 trained with zero, reflect, and padding settings, with the GTOS-Mobll hypothesize that, due to the nature of textures existing in the entire image, position information will not benefit the performance of the CNN. As shown in Table (a), models trained with reflect padding outperform the models trained with zero padding. This result implies that position information is not important for the task of texture recognition. Although no padding has less position information than reflect padding, the CNN suffers from the border effects without padding (see Fig. (left)), which hurts performance significantly (i.e., since the kernel's 's support does not cover the entire image domain).
dataset (Xue et al.) 2018) and DTD (Cimpoi et al. 2014). We
Canvas Analysis: Cutout & Adversarial Robustness. We investigate the impact of different canvas colors terms of performance and robustness using data augmentation strategy, Cutout Vrees & TTylor 2017). which simply places a rectangular black mask over random image regions during training. We evaluate DeepLabv3 with two backbones using the Cutout strategy for semantic segmentation the PASCAL VOC 2012 (Everingham et al., 2010) dataset with black and white masks (see Fig. 17 in the appendix for example inputs). We also evaluate the robustness of each model to show which canvas is more resilient to the GD-UAP adversarial attack (Mopuri al., 2018). Note ResNet1 that the GD-UAP attack is generated based on the image-agnostic DeepLab-ResNetl00 01 backbone. As shown in Table (b), DeepL Labv3 trained with white-mask Cutout is significantly more robust to adversarial examples than the black canvas, without sacrificing segmentation performance.
CONCLUSION
With the goal of answering whether boundary effects are a feature or a bug, we have presented evidence that the heuristies used at the image boundary play a much deeper role in CNN's ability to perform different tasks than one might assume. By designing a series of location dependant experiments, we have performed unique exploration into how this connection reveals itself. We
8

---

Under review as a conference paper at ICLR 2021
showed that zero padding encodes more position information relative common padding types (H-I) and that zero padding causes more dimensions to encode position information and that this correlates with the number of dimensions that encode semantics (H-III). We examined the abilitt of CNNs to perform semantic tasks as a function of the distance to a border. This revealed the capability a black canvas to provide rich position information compared other colors (i.e., White and Mean) (H-IT). We visualized a number of features in CNNs which showed that boundary effects have an impact on all regions of the input (H-IV), and highlighted characteristics of border handling techniques which allow for absolute position information to be encoded. This position encoding enables CNNs to learn more separable semantic features which provide more accurate and confident predictions (H-III). We conducted these experiments with the following question in mind: Are boundary effects feature or bug (H-V)? After teasing out the above underlying properties, we were able to validate the hypothesis that different types of padding, levels of position information, and canvas colors, could be beneficial depending on the task at hand! To be more clear: the boundary effects can be used to improve performance if you know what to look for, but can also be detrimental to CNNs performance if not taken into consideration. We strongly believe these findings will allow future researchers to more efficiently propose and implement improved algorithms which deal with boundary effects.

---

Under review as a conference paper at ICLR 2021
REFERENCES
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. CVPR, 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity
natural image synthesis. arXiv: / 1096, 2018.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. urXiv:/706.0558/ 2017.
Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. TPAMI, 2018.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Ugur Demir and Gozde Unal. Patch-based image inpainting with generative adversarial networks. arXiv preprint
2018.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 2017.
Patrick Esser, Robin Rombach, and Bjöm Ommer. A disentangling invertible interpretation network for explaining latent representations. In CVPR, 2020.
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2010 (VOC2010) Results. http://www.pascal. 2010.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2018.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. DRAW: A recurrent neural network for image generation. In ICML, 2015.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV, 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In CVPR, 2017,
Zhewei Huang, Wen Heng, and Shuchang Zhou. Learning to paint with model-based deep reinforce- ment learning. In ICCV, 2019.
Carlo Innamorati, Tobias Ritschel, Tim Weyrich, and Niloy J Mitra. Learning the edge: Investigat- ing boundary filters in CNNs. 1JCV, 2019.
Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information do convolutional neural networks encode? In /CLR, 2020.
Andrej Karpathy, George Toderici. Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
10

---

Under review as a conference paper at ICLR 2021
Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cans: Convolutional layers can exploit absolute spatial location. In CVPR, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolu- tional neural networks. In NIPS, 2012.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset. online: http://www. cs. toronto. edu/krizfcifar himl, 2014.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
Y. Li, X. Hou, C. Koch, M. Rehg, and A. L. Yuille. The secrets of salient object segmentation. In CVPR, 2014.
Guilin Liu, Kevin Shih, Ting-Chun Wang, Fitsum A Reda, Karan Sapra, Zhiding Yu, Andrew Tao, and Bryan Catanzaro. Partial convolution based padding. arXiv: 2018a.
Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. In NeurlPS, 2018b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. JMLR, 2008.
Konda Reddy Mopuri, Aditya Ganeshan, and R Venkatesh Babu. Generalizable data-free objective for crafting universal adversarial perturbations. TPAMI, 2018.
Hyconwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In ICCV, 2015.
Jorge Pérez, Javier Marinković, and Pablo Barceló. On the turing completeness of modern neural network architectures. ICLR, 2019.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICI.R, 2015.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR. 2015.
Ming Tang, Linyu Zheng, Bin Yu, and Jinqiao Wang. High speed kernelized correlation filters without boundary effect. arXiv:1806.064// 2018.
John K Tsotsos, Scan M Culhane, Winky Yan Kei Wai, Yuzhong Lai, Neal Davis, and Fernando Nuflo. Modeling visual attention via selective tuning. Artificial intelligence, 1995.
Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. In CVPR, 2017.
Brendt Wohlberg and Paul Rodriguez. Convolutional sparse coding: Boundary handling revisited. arXiv: 17070077 18, 2017.
Jia Xue, Hang Zhang, and Kristin Dana. Deep texture manifold for ground terrain recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 558-567, 2018.
11

---

Under review as a conference paper at ICLR 2021
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In CVPR, 2018.
Joe Yue-Hei Ng. Matthew Hausknecht, Sudheendra Vijaanarasimman, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In CVPR, 2015.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 NO PADDING RESNET IMPLEMENTATION USING BILINEAR INTERPOLATION
In this work, we attempt to explore the effect of different padding strategies in addition to networks with no padding. Our goal is to analyze the most standard settings in computer vision and as ResNets are one of the most common CNN architectures used, we thus aim to remove padding from a ResNet. The main issue with this objective is that the residual block of ResNet requires padding for the residual connection to match the same feature resolution as the next layer. This question can therefore be reduced 'how can the input be aligned with the output feature map?".
To the best of our knowledge, how to match the feature map and the input is an open question. Two simple solutions could be applied: (i) crop the centre region from the input to align with the output, or (ii) interpolate the output feature map to the same size as the input. Before choosing an appropriate operation, we must first consider what the feature map encodes. For dense labelling tasks, the intermediate feature map is believed to retain the spatial relationship because convolutional filters are translation -euuiviriinivviiint...
For instance, most of the one-stage object detectors apply location loss for the bounding box on multiple feature maps, both anchor-based and anchor-free. For example, if a dog is located at 0.1 width to the right of input, its corresponding region in the feature map will be the same. If this intermediate embedding is rranlation-equiivriann, it is clear that the centre cropping will break this equivariance. If crop the centre of the input image, the input be aligned with the smaller feature map, the relative position of the dog will be changed. In contrast, interpolation (e.g., bilinear) of the feature map to the same size the input appears to be a better choice because the whole process should keep a more consistent spatial alignment.
We now validate this choice by conducting simple toy experiment. We create black canvas with three channels, (100, 100, 3), the height and width equal 100, (see Fig. aa)). Then, we fill four squares with different colours the four corners where each square has size of 10 x 10. One convolution layer with a kernel size of 21 and a stride of one is applied on the input image with the all weights of the kernel experiment, so the convolutional filter is applied separately on each channel so as to retain the color information.
one. Note that we only care about the spatial equivariance in this
The output of padded convolution is shown in Fig. (() (100 x 100), and the output of the convolution w/o padding shown in Fig- ((c) (80 x 80). Note that the color is enhanced for visualization purposes, the actual output value will decay gradually towards the center of the feature map, and it will spread out even more when padding is applied. As discussed previously, one can also crop the center the input to match the feature map (see Fig. (e)). However, the cropping operation will excludes the content near the border and the translation-equivvriance property no longer holds. Thus, we choose to interpolate the feature map to the input size. As we can see from Fig g d), the resized feature map can better match the content of the input.
(a)
(b)
(c)
(d)
(e)
Figure 8: (a) the input image, (100 100). (b) the output of the convolution with padding, (100 100).
(c) the output of the convolution w/o padding, (80 x 80). (d) the resized feature map of (c), (100 x 100).
(e) the center crop of the input image, (80 x 80).
A.2 IMPLEMENTATION DEATILS OF VGG-5 NETWORK FOR POSITION INFORMATION
We use simplified VGG network (VGG-5) for the position encoding experiments in Sec. 2 and
texture recognition experiments in Sec. 6 The details of the VGG-5 architecture shown in Table
13

---

Under review as a conference paper at ICLR 2021
(in this table we show the VGG-5 network trained the tiny ImageNet dataset, the VGG-5 network trained on texture recognition has a different input size: 224 x 224). Note that the network is trained from scratch. The tiny ImageNet dataset contains 200 classes and each class has 500 images for training and 50 for validation. The size of the input image is 64 x 64, a random crop of 56 x 56 is used for training and center crop is applied for validation. The total training epochs is set 100 with an initial learning rate of 0.01. The learning rate was decayed at the 60th and 80th epochs by multiplying the learning rate by factor of 0.1. A momentum of 0.9 and weight decay of le are applied with the the stochastic gradient descent optimizer. After the pre-training process, simple read-out module applied on the pre-trained frozen backbone for position evaluation, following the training protocol used in Islam et al. (2020). Note that the type of padding strategy is consistent between the pre-training and position evaluation procedures.
Table 6: VGG-5 architecture trained on tiny ImageNet.
RGB image x 56x666x3
Conv2d (3 x 3), Batch Norm, ReLU, MaxPool2d 282288633
Conv2d (3 x 3), Batch Norm, ReLU, MaxPool2d 14x64
Conv2d (3 3), Batch Norm, ReLU, MaxPool2d IR 7x7x
Conv2d (3 x 3), Batch Norm, ReI.U - IR. 7x7x256
Global Average Pooling (GAP) R x
FC > (256, classes)
A.3 GRID-BASED INPUTS FOR POSITIONAL. ANALYSIS
Figure 9 shows examples inputs for the location dependant experiments (Sec. 4 in the main paper), and the ground truth for each of the tasks the right hand side. As previously mentioned, all the experiments were run with three different canvas colors to show the impact of the border effect with regards to backgrounds. For the segmentation ground truth, the ratio of background pixels object pixels grows exponentially as the grid size increases. However, as the evaluation metric is mean intersection over union (mloU), the overall performance is averaged between the object classes and the background class, even though the background class makes up the majority of the ground truth labels.
Black
White
Mean
32
32
32k
32
32
32k
32/
32k
Class Label
Segnentation Label
Figure 9: An illustration of our proposed grid settings (A: = 3) with all three different canvas colors for the location dependant tasks.
14

---

Under review as a conference paper at ICLR 2021
A.4 LOCATION DEPENDANT IMAGE CLASSIFICATION AND SEGMENTATION
In this section, we revisit the location dependant image classification and segmentation experiments from Sec. 3 of the main paper, but present them in visual format (See Fig. (0) enhance the reader's s ability to see patterns in performance as the grid size increases. It is apparent that zero padding provides massive performance boost for location dependant tasks as larger grid sizes are used. Also, it is apparent that zero padding is particularly helpful as the ratio of background pixels image pixels increases in the input. For segmentation, the performance decreases even with zero padding: however, a large performance drop is still observed with lack of padding. Additionally, note the increase performance when a black canvas is used compared with white or mean canvases (see Sec. 5 in the main paper for discussion on black canvases).
Black w/ Padding
White of Padding
Mean wi Padding
Black wio Pidding
White wio Padiing
Miaa wio Fadding
80
Acoracy (%)
Accuracy (%)
60
60
20
3
15
13
15
Grid Steee
Crid Size
Figure 10: Location dependant image classification (left) and semantic segmentation (right). This figure contains the same results as in Table in the main paper, but visualized as plot. Note the performance gains when padding black canvas is used.
Results using VGG-11 Networks. We extend the location dependent segmentation experiment by applying different backbone architecture. We choose VGG-11 network further validate the correlation between network architectures and boundary effects. Table presents the Image segmentation results. We observe similar pattern as segmentation experiment (Sec. 3.2). Networks trained with padding consistently outperform networks trained w/o padding, and the difference grows larger as the grid size increases.
Table 7: Location dependant semantic segmentation accuracy CIFAR-10 under zero/no padding and black canvas settings using VGG-11 network.
Image Segmentation
Padding
3x3
5x5
7x7
9x9
11xll
13x 13
15x
15
Zero Pad
66.8
66.2
64.5
63.2
56.9
56.0
52.6
B
w/o Pad
63.2
55.8
53.3
50.1
48.8
47.2
48.7
Results on reference grid size using ResNet-18 Networks. We extend the location dependent seg- mentation experiment by applying different even grid sizes. Table & presents the Image segmentation results. We observe a similar pattern as other grid sizes segmentation experiment (Sec. 3.2).
Table 8: Location dependant semantic segmentation accuracy CIFAR-10 under zero/no padding and black canvas settings using ResNet18 network.
Segmentation
Padding
4x4
6x6
Zero Pad
68.8
68.1
w/o Pad
58.1
54.9
15

---

Under review as a conference paper at ICLR 2021
A.5 DIMENSIONALITY ESTIMATION OF DIFFERENT SEMANTIC CONCEPTS
For image pair (7o, 1o) p(1", (*)) which are similar in the k-th semantic concept, follow- ing (Esser et al., 2020), we approximate the mutual information with their correlation for each dimension
Cov
Correlations = =p Var (I")), Var (E (T"),)
We assume that the residual factor has maximum dimension of |a| (the total dimension of the latent representation) and use the softmax equation to get the resulting dimension:
exp Cs E exp exp C
[x =
N
where |2x| is the dimension of the semantic factor k. and F is the total number of semantic factors including the residual factor. Note we do not need an estimate of the absolute mutual information for estimating the proportion of location and semantic dimensions. Only the differences between the mutual information for position and semantic class for image pairs are used to quantify the ratio of location and semantic-specific neurons. Therefore, the relative difference is still meaningful and only the absolute numbers might not be.
A.5.1 ADDITIONAL DIMENSIONALITY ESTIMATION RESULTS
Table 9 presents additional results dimensionality estimation for different semantic concepts based on the latent representation (see Table 3 in the main paper). We observe consistent results padding encodes more location and semantic information compared to the no padding case.
Table Dimensionality estimation (%) of semantic concepts (location and semantic category) under different tasks and settings. Remaining dimensions are assigned to residual factor.
Semantic Factors,
Canvas
Grid
Padding
Location
ACass
Canvas
Grid
Padding
Semantic Factors, z
Zero Pad
17.0% %
16.3 %
Location
|cCtas
Black
5x5
No Pad
12.6%
12.8%
Zero Pad
11.2% % 10.9%
11.2 %
Black
9x9
Zero Pad
16.4% %
15.4% %
No Pad
10.9%
White
5x5
No Pad
11.5%
11.6%
(a) Image Classification
(a) Semantic Segmentation
A.6 EXTENDED PER-LOCATION ANALYSIS
We now present additional 'per-location' results. That is, take advantage of the location dependant grid-based input and analyze the performance of CNNs at each location on the grid. This is done to reveal the impact of border effects with respect to the absolute location of the object of interest. We first show class-wise performance for the location dependant semantic segmentation task (Sec. A. Next, we show the performance as function of the distance to the nearest border by averaging the accuracy over all locations which are specified number of grid locations away from the nearest border (Sec. A.6.2). We then provide additional t-SNE (Maaten & Hinton 2008) visualizations to examine the separability of the CNN's learned semantic features (Sec. A.6.3) Finally, we display location dependant semantic segmentation predictions and analyze the impact that border effects have per-pixel predictions for various input locations (Sec. A.7. Note that all experiments are done with the same settings as Sec. the main paper, on the CIFAR-10 (Krizhevsky et al. 2014) dataset.
16

---

Under review as a conference paper at ICLR 2021
Table 10:
Location dependant image segmentation: Category-wise mloU CIFAR 10 (Krizhevsky et al., 2014) for two different locations under w/ and w/o padding settings and Black and Mean canvas color. The grid size for both canvases x
Black
Mean
Categories
L
7
L
25
L 7
L
25
Pad
w/o Pad
w/ Pad
w/o Pad
w/ Pad
w/o Pad
w/ Pad
w/o Pad
Background
0.99
0.99
0.99
0.99
0.99
0.98
0.99
0.99
Plane
0.67
0.34
0.65
0.53
0.50
0.07
0.55
0.31
Car
0.80
0.51
0.76
0.68
0.58
0.08
0.69
0.55
Bird
0.57
0.18
0.57
0.44
0.42
0.01
0.47
0.24
Cat
0.46
0.14
0.43
0.35
0.34
0.01
0.40
0.16
Deer
0.63
0.30
0.62
0.49
0.47
0.01
0.55
0.18
Dog
0.53
0.30
0.53
0.39
0.43
0.01
0.50
0.20
Frog
0.67
0.41
0.64
0.59
0.53
0.03
0.63
0.48
Horse
0.70
0.41
0.70
0.57
0.52
0.02
0.62
0.32
Ship
0.78
0.43
0.74
0.64
0.58
0.03
0.66
0.48
Truck
0.74
0.41
0.71
0.60
0.57
0.04
0.67
0.44
Overall
0.66
0.40
0.67
0.57
0.54
0.12
0.61
0.40
Black No Padding + White No Padding + Mean No Padding
Padding Baseline
Grid
Grid
Grid
15 Grid
Aecraay Dideesnne (80
-00
-30
Disunceto Closss Booke
Distace Choest Boolsr
Chocs
Closss Boods
Figure 11: Location dependant image classification (left two) and segmentation (right two). Results show the accuracy difference between padding (blue horizontal line) and no padding (orange markers), various distances to the border and canvas colors.
A.6.1 PER-LOCATION CATEGORY- MIOU ANALYSIS
Table 10 shows the sategory-wise mloU for the location dependant image segmentation task for 7 x grid with black and canvases. We show the category-wise performance for location the very top right corner (I. = 7) and at the center of the grid (I. = 25), which highlights how the encoding absolute position information affects the learning of semantic representations.
For both locations, the border and the center, zero padding gives large increase in performance for all classes compared lack of padding. This particularly pronounced with mean canvas, demonstrating how the black canvas explicitly injects position information, even without the use of zero padding. For example, comparing the black and mean canvas absolute position information can be in learning distinct semantic representations. The network trained with a mean canvas has difficult time learning to segment images at this location when no padding is used and suffers large drop in performance compared to the black canvas. Some classes even score around 1% mloU, which implies that the network fails to learn to segment certain classes (i.e., Bird, Cat, Deer, and Dog) with these settings. When zero padding is added (i.e., Mean, w/ padding, L = 7), the network achieves performance boost of between 35% - 60%. When black canvas is used inject position information instead (i.e., Black, w/o padding, L 7), the performance gains range from 15% 40%. Clearly, the encoding position information, by means of zero padding black canvas. has stark effect CNN's 's ability learn distinctive semantic
shows how important
17

---

Under review as a conference paper at ICLR 2021
Input
w/ pad
wio pad
7
L 10
15
19
= 34
45
Figure 12: t-SNE Maaten & Hinton 2008) visualization of the CIFAR-10 test set classification logits for a grid. Examples of single input are given in the top row, while the embedding visualizes the entire dataset (bottom two rows).
features. We similar, but not quite drastic, pattern at the center of the image, further showing how the boundary effects impact all locations in an image, and not just at the image border.
just
A.6.2 DISTANCE TO BORDER PERFORMANCE
Figure 11 shows the performance as function of the distance to the closest border for all three canvas colors. The networks with zero padding are represented plotted markers show the difference in performance when no padding is used. Consistent with the results in the main paper, locations near the border are on average, much more difficult for networks to classify and segment, particularly as the grid size increases.
blue horizontal line, where the
A.6.3 T-SNE VISUALIZATIONS
We use t-SNE (Maten & Hinton 2008 to visualize the test set classification logits (from the location dependant classification task) in Fig. I2 Note that the single input examples at the top row are shown merely to highlight the location L. and that the second and third rows show embeddings of the entire test set. The separability of the semantic classes is significantly improved when padding is used, and the effect is particularly pronounced locations hypothesis that absolute position information, by means of zero padding, enables CNNs to learn more robust semantic features, which in turn allows for greater separability in the predicted logits.
the border (I. 7). This further supports the
A.7 LOCATION DEPENDANT IMAGE SEGMENTATION PREDICTIONS
Figure 13 shows predictions of the location dependant image segmentation task for grid size 5. We visualize the predictions as a heatmap. where each pixel is colored according to the confidence that the semantic category appears in that pixel's location. We show predictions with padding (left) and without padding (right) for various grid locations, L. Note how boundary effects significantly impact locations near the border. particular, locations in the corners are most affected, as they suffer from boundary effects originating from two borders (e.g., top and left border for L = 1). Figure [3 shows predictions of the location dependant image segmentation task for a grid size We visualize the predictions as a heatmap, where each pixel is colored according to the confidence that the semantic category appears in that pixel's location. We show predictions with padding (left) and without padding (right) for various grid locations, L. Note how boundary effects significantly impact locations near the border. particular, locations in the comners are most affected, as they suffer from boundary effects originating from borders (e.g., top and left border for I. 1).
18

---

Under review as a conference paper at ICLR 2021
2
L
L
L 9
L
L=11
L
13
1=14 14
L 15
16
L=
L 22
L 23
L= 24
L 25
Figure 13: Sample predictions of image segmentation all the locations of 5 5 grid under the mean canvas setting. Confidence maps are plotted with the 'viridis' colormap, where yellow and dark blue indicates higher and lower confidence, respectively.
Image
GT
w/o Padding
w/ Padding
Figure 14: Example predictions on the Cityscapes validation set when training w/ and w/o padding. Best viewed with zoom.
19

---

Under review as a conference paper at ICLR 2021
Table 11: IoU comparison of DeepLabv3 for semantic segmentation task with three different padding (Zero, Reflect, and No pad) settings.
Eval. Region
Zero Pad
Reflect
No Pad
0% 5%
72.6
71.9
63.7
30%--28%
80%6-854
5%- 10%
72.7
72.0
66.4
10%- 15%
73.8
73.7
67.2
15%- 20%
73.9
74.1
67.9
20%- 25%
74.7
74.8
68.5
Figure 15:
An illustration of the evaluation
25%- 30%
75.3
75.4
69.6
regions used for the analysis in Table 11 and Fig.
30%- 35%
75.1
75.2
69.4
35%- 40%
74.7
75.2
69.3
40%- 45%
74,4
74.8
69.2
45%- 50%
74.2
74.5
69.4
50%- 55%
74.4
74.9
69.8
55%- 60%
74.3
74.8
69.7
60%- 65%
73.8
74.3
69.2
w/o padding
65%- 70%
73.8
74.4
68.8
Reffect pudding
70%- 75%
73.9
74.5
68.9
Zero padding
75%- 80%
73.8
74.4
69.2
80%- 85%
73.5
74.1
68.1
30
100
Region Distribution
85% 90%
71.4
71.9
65.1
90%- 95%
71.3
72.0
64.2
Figure 16:
Performance comparison of DeepLabv3 network with respect various im- age regions and padding settings.
95%- 100%
69.7
70.1
70.2
Overall
74.0
73.9
69.1
A.8 EXTENDED BOUNDARY EFFECT ANALYSIS ON CITYSCAPES DATASET
We continue to investigate the impact that zero padding has on the ability of strong and deep CNN to segment objects near the image boundary. Results shown use the same network and training settings in Sec. of the main paper, the Cityscapes (Cordts et al. 2016) dataset. We first show additional qualitative examples in Fig. 14 which clearly shows large reduction in performance at locations near the border when padding is used, particularly for thin objects (e.g., street lamps column poles).
We present additional results (see Table and Fig. 16 of the analysis presented in Sec. (semantic segmentation) in the main paper. Fig. 15 shows sample evaluation regions used for this analysis. The no padding case has steeper drop-off in performance regions of evaluation get closer to the image boundary. Note how, in all cases, the performance increases from the border to the inner 25%, at which point the performance is somewhat stagnant until it reaches the innermost 80%.
Surprisingly, we also observe a steeper drop off in the middle of the image for the no padding case, supporting our hypothesis that boundary effects play role all regions of the image without the use of padding. We believe the drop in performance at the center regions is due to Cityscapes being an automotive-centris dataset, where pixels the center of the image are often at large distances away from the camera, unless the vehicle collecting the data has an object directly in front of it.
A.9 CANVAS ANALYSIS: CUTOUT & ADVERSARIAL. ROBUSTNESS
Figure 17 shows two training examples of Cutout strategy. Following Cutout, we simply place rectangular mask (black and white) over random region during the training. Note that evaluate on the standard PASCAL VOC 2012 validation images.
20

---

Under review as a conference paper at ICLR 2021
Cutout Black Canvas
Cutout White Canvas
Cutout Black Canvas
Cutout White Canvas
Figure 17: Sample training images generated using Cutout under two different canvases.
21


Reviews:
[{"review_id": "pdQSp8W2shP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the effect of padding on the Convolutional Neural Network. The authors try to answer the following questions: 1) what type of padding provides the most position information, 2) does the background value affects model accuracy when processing a patch on a canvas, 3) which part of the image suffers the most from the boundary effect, and 4) whether the position information provided by padding improves or degrades model performance. In order to answer these questions, the authors design multiple tasks and perform extensive experiments. The empirical results show that: 1) zero-padding provides the most location information compared with other common padding methods, 2) the background value of the canvas do affects the accuracy when processing a patch, 3) the boundary effect is not specific to the image boundary---the model is affected by the boundary over the entire image, and 4) the effect of padding on model accuracy depends on the task.\n\nThis paper studies an empirically important yet under studied problem and is well written. The target questions / hypothesis are well motivated, and the experiment designs are reasonable and help to answer the target question. The extensive experiments provide valuable insight into the model properties and may benefit both fundamental CNN research and CNN applications in the future. Nevertheless, there are some problems that are not sufficiently addressed in the paper and may be improved.\nFirst of all, an implicit hypothesis in the experiments is that the boundary effect is generic to the architecture. However, my conjecture is that how much padding affects the model strongly depends on the overall model architecture such as the readout architecture (e.g. avg pooling, max pooling, fc-layer), the receptive field of the neurons, etc. The correlation between model architecture and boundary effect is not discussed in the paper, and the implementation details are not clearly described to estimate how much it may affect the conclusion.\n\nSecond, as an empirical study with newly defined tasks, it is important to demonstrate whether the results are significant. More specifically, it would be helpful if the authors can 1) include intuitive baselines for reference performance (e.g. a model on (k+1)^2 patches instead of k^2), 2) show the learning curve / variance etc. to demonstrate whether the difference is meaningful, and 3) provide a more complete description about the experiment protocol (eg. how were the meta-parameters determined, whether the results are affected by the meta-parameters).\n\nThird, some of the arguments are based on qualitative results. However, the figures are not clear enough and it is hard to follow the arguments. Also, qualitative results do not provide strong enough evidence for these arguments, and it would be more persuasive if the authors can provide quantitative analysis instead and use qualitative results as a support.\nFinally, given that a learned representation can be distributed, I don't think the dimensionality estimation (Table 3) alone provides useful information. The argument implicitly assumes that 1) all other dimensions are independent of location information, 2) the location information cannot be reduced to a lower dimensional space. These assumptions are not verified.\n\nThe rebuttal provide valuable information that strengthens the original paper. While some of my concerns are not resolved in the rebuttal, they require additional experiments and may be beyond the scope of rebuttal.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q_SdEAzBrg0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\n\nThis submission studies the effect of zero/one padding in convolutional networks when using images that are pasted on a canvas. The main thesis is that zero padding induces absolute position information and that this leads to better performance in many cases. The study is based on segmentation and classification where the input image is placed on a background canvas.\n\nUpdate after Rebuttal:\n\nI do appreciate the argument given in appendix A.1 about the relative position that changes with cropping and that boundary information may get lost. However, it is not clear to me if that change in relative position matters because the network could take this into account. However, the issue that the feature maps between the shortcut and residual connection do not align with the bilinear interpolation seems to be much harder take into account for the network. To me it seems an actual example on the ImageNet classification (i.e. Table 2) that shows that the degradation in performance is not due to the resampling/misalignment of the feature maps in the ResNet would be very important. Moreover, a related problem is that it seems plausible that a network could extract position information from the spatially varying misalignment of the feature maps (in the image center there is no misalignment and on the border there is 1px (for 3x3 conv). The amount would reveal the position). Therefore an experiment that shows that this does not happen in practice would also be important. This is the same major concern I share with AR4 even after the rebuttal.\n\n\nStrengths:\n\nPadding is omi-present in current convolutional and \"fully\" convolutional architectures but its effect is not studied well.\nThe manuscript clearly states the hypothesis that are investigated.\n\nWeaknesses:\n\nThe motivation for using canvases in the way done within this manuscript is unclear to me. The introduction states this is due to the fact that images need to be rectangular to be processed with a CNN. While in practice mostly rectangular images are used with CNNs it seems to me that convolutions could also be applied to non-rectangular images if required (with custom implementation). However, more importantly the manuscript does not just pad images to be rectangular but instead pastes a much smaller image onto a large canvas. This is not clearly motivated in the current form of the manuscript as far as I can tell.\n\nIn Section 3 it is unclear if one network is trained for all grid resolutions or if the networks are trained per grid resolution.\n\nUnclear how no padding is conducted in ResNet. Section 3 indicates that a standard ResNet-18 is modified to a no padding version by removing the padding from the convolutions and using bilinear resize to match the sizes. The way I understand this is that for example an NxN image is convolved without padding and hence the output will be an (N-k) x (N-k) image, now either the output of the convolution or the original NxN image is resized to match the other resolution before adding the two. This seems like a very problematic implementation as in either version the spatial alignment between the two is not preserved and importantly the misalignment is spatially varying while the convolution kernel will not vary spatially. In my mind a no padding implementation of a ResNet would simply crop the original image in the consistent way to how the convolution does not produce outputs for the boundary regions. If my understanding of this implementation is correct I feel that this is a major flaw in this submission.\n\nDo the no padding segmentation networks contain upsampling operations and or u-net type skip connections? How are the no padding versions of these operations implemented? Especially for the city scapes segmentation results it is unclear to me how a full image segmentation was obtained without using any form of padding or related mechanism of retaining the information around the image boundary.\n\nIn Table 3 the white padding seems to contain stronger location information than the black padding which contradicts the main thesis of the submission that says black padding contains more location information. However, Table 3 uses different grid sizes for black and white padding. Are the different grid sizes the reason for this discrepancy? What is the motivation for using different grid sizes?\n\nOne point I find missing in this submission is that no padding networks will have spatially smaller feature maps which means less information might be stored in the intermediate layers. This could be a reason why the no-padding versions of Table 4 underperform compared to reflection padding.\n\nWhat happens for networks where the input/output resolution varies, i.e. \"fully\" convolutional networks? In these cases absolute position information intuitively would prevent generalization to different image sizes.\n\nReason for score:\n\nThis submission discusses an important and relevant topic. However, at this point I believe that there are potentially flaws in the experiments which might significantly influence the findings. Therefore the current rejection rating. If the rebuttal could further discuss the points in the weaknesses section and explain in detail how the no-padding versions were implemented that would be helpful to further assess the correctness of the above assessment. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DoypBAg2P9w", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The topic of the paper is interesting and important. The padding strategy seems to be a small but largely overlooked aspect in CNN learning. While many papers attempt to reduce/improve the effect of position in images. This paper gives a different perspective on the padding patterns that essentially causes these position sensitivity. \n\nThe paper is easy to follow: the hypothesis at the beginning are important and interesting. And the hypothesis are quite clearly verified with experiments.\n\nThe Hypothesis raised in this paper are important factors for training networks for various tasks. especially for H5 and H2. And the paper gives useful conclusions based on experiments.   H4 is a bit unnecessary as it is common knowledge that as layer goes deeper, it has a larger receptive field thus even boarder information will be included in the center regions. \n\nThe current analysis are based on CIFAR, it would be more convincing if these hypothesis are further validated on larger datasets such as imagenet or coco (for segmentation). \n\nA minor suggestion, it would be more informative if the author in related work/analysis compare with the positioning methods used in transformers. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BvN-VoyocB3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper seeks to understand how different padding modes and canvas colors affect the performance of a convolutional neural network in classification and semantic segmentation tasks. The question seems somewhat strange - surely a network should be able to counteract a consistent change in padding or background color. If there was a strong effect it would be an interesting finding indeed. Unfortunately, the paper fails to convince that any but the most obvious effects exist.\n\nThere is an underlying assumption that knowledge of absolute spatial location should be helpful in some tasks and unhelpful in others. Why should absolute position matter for CIFAR-10 classification or semantic segmentation? Before jumping to this conclusion, this should be measured directly, e.g., by embedding horizontal and vertical positions as constant feature maps at every convolution. \n\nThe findings in Section 2 are not very surprising. Assuming that images are not mostly black, the greatest contrast is found against a constant black background, and partial convolution only enhances this effect - it is equivalent to zero padding but brightening the image artificially when the convolution kernel overlaps it only partially.\n\nTable 2 shows that classification results vary wildly between padding and no padding, when comparing black canvas and zero padding. This makes no sense to me. Especially in large grids, a picture embedded somewhere else than the edge of a black canvas, without padding, should be fundamentally the same as having just the picture and zero padding. ResNet-18 that the paper uses has a global pooling layer at the end, ensuring that global position information is certain to be lost at that point at the latest.\n\nI'm extremely concerned about the paper's decision to use bilinear interpolation to match the resolution with skip connection in a ResNet architecture, when padding is not used. My understanding is that the convolution branch output is upscaled to the skip branch resolution before composition. This introduces a scaling factor to the convolution branch, and therefore moves the contents of feature maps towards the edges. The composite will thus not have the features from the two branches line up with each other  anymore. This shifting effect is largest near the edges and smallest near the center, which matches with the observed performance characteristics. The subsequent convolution layers may be able to undo some of this shifting, but only at the cost of location-dependent kernels that are tailored to fit the offset  caused at different parts of the image. I'm inclined to believe that many of the alleged problems with non-padded convolutions are due to this design. For example, Table 10 suggests that the padding itself is a much more important factor than if padding is with zeros or a reflection, which makes little sense if locating the boundary were the critical aspect.\n\nThe paper is mostly clearly written, but some of the terminology is confusing. Section 3.1 talks about location-dependent classification, whereas the goal is to have classification be independent of location.  There are other cases of similar dependence vs independence confusion. What Horizontal and Gaussian mean in Table 1 should be explained. On page 7, formula $\\frac{k-1}{k}$ should read $\\frac{k^2-1}{k^2}$. In Figure 6, it is unclear how the activations are mapped to grayscale values. Regardless, high or low activations do not mean much with ReLU activation function, as any scaling can be counteracted in the weights of the subsequent convolution kernels, so the relative strengths of activations may not carry much information. In Figure 12, are the w/pad and w/o pad flipped? The effect seems inverse to Figure 1.\n\nIn summary, I find the findings questionable and too counterintuitive - e.g., the big difference in classification power between black background vs. zero padding when a small image is embedded in a large black canvas. The paper fails to demonstrate convincingly that these findings are real instead of artifacts of the experimental design.\n\nPros: mostly clearly written, many experiments.\n\nCons: dubious choices for network design and experiments, counterintuitive results that are not properly\nexplained or analyzed.\n\n** UPDATE after reading other reviews, author responses, and revised paper **\n\nI'm unconvinced by the arguments related to the resizing of feature maps between ResNet branches. The added experiment in A.1 does not address the question of misaligning the features between skip branch and convolution branch. \n\nFigure 8 suggests that the alignment with input is well preserved with resizing (i.e., Fig. 8(a, d) look similar). The illustration is misleading because it uses a kernel with constant weights and *extremely* oversaturates the colors. In other circumstances, we would see that each corner in Fig. 8(c, d) is different. This is because Fig. 8(b) will contain four identical blobs after the convolution, and Fig. 8(c) is by definition a central crop of Fig. 8(b). Thus, the bits we see in Fig. 8(c) are different quadrants of this blob, and they are in fact all different. This leads to the misalignment between skip and convolution branches, so that subsequent processing will see different data at each of the four corners. The figure makes it look like there are only small differences by oversaturating the colors to the point that only the footprint of the convolution can be seen. Using a non-symmetrical convolution kernel would have also highlighted the differences.\n\nThe experiment also misses the larger point: The different architectures may cause differences in the results that exceed the effects of the padding itself. The paper claims to measure the latter, but I believe both AR2 and I are concerned that the paper may be measuring the architectural effects instead. As this was my main concern about the experimental design, my confidence in the paper's results is not increased, and my rating is not changed. If anything, I'd be inclined to lower my score because of how misleading the added Fig. 8 is.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the effects of padding in convnets used for various visual recognition tasks (classification, segmentation). This is an important and relevant design choice that is often overlooked, as noted by reviewers. However, I share the concerns of AR2 & AR4 with the evaluation. The design of the ResNet variant used for the "No Pad" baseline seems potentially fatally flawed: the bilinear upsampling used to match the feature map sizes for the residual addition results in a misalignment of the inputs with the outputs, which potentially explains the performance degradations seen throughout most experiments, as opposed to (or perhaps in addition to) the lack of positional information and border effects resulting from the NoPad scheme that is claimed as the reason for the performance drop. It is true that how to do this is an open question, as the authors argue in their added Appendix A.1, but I nonetheless share the reviewers' skepticism that the chosen approach will result in a meaningful comparison of the effect of padding and border effects. In fact, the results in Table 5 on texture recognition seem to suggest that "No Pad" approach may indeed be flawed, given that "No Pad" performs the worst, while "Reflect" padding performs best, even though both methods share the property that the network should have difficulty inferring positional information. Given the reliance on this dubious baseline throughout the results, I can't recommend the submission for acceptance in its current form. However, I still appreciate the direction of this work and hope the authors will consider resubmitting it after revising it in order to make the evaluation more convincing based on the reviewers' feedback.
Decision: Reject