Mind the Pool: Convolutional Neural Networks Can Overfit Input Size
Authors: Bilal Alsallakh
David Yan
Narine Kokhlikyan
Vivek Miglani
Orion Reblitz-Richardson
Pamela Bhattacharya
Venue: ICLR 2023 poster (2023)
Keywords: Convolutional Neural Networks
Pooling
Input Size
Overfitting

Abstract: We demonstrate how convolutional neural networks can overfit the input size: The accuracy drops significantly when using certain sizes, compared with favorable ones. This issue is inherent to pooling arithmetic, with standard downsampling layers playing a major role in favoring certain input sizes and skewing the weights accordingly. We present a solution to this problem by depriving these layers from the arithmetic cues they use to overfit the input size. Through various examples, we show how our proposed spatially-balanced pooling improves the generalization of the network to arbitrary input sizes and its robustness to translational shifts.

Published as a conference paper at ICLR 2023
MIND THE POOL: CONVOLUTIONAL NEURAL NET- WORKS CAN OVERFIT INPUT SIZE
Bilal Alsallakh
David Yan
Narine Kokhlikyan
Vivek Miglani Meta AI
Voxel AI
Meta
Meta Al
Orion Reblitz-Richardson Meta AI
Pamela Bhattacharya Meta
ABSTRACT
We demonstrate how convolutional neural networks can overfit the input size: The accuracy drops significantly when using certain sizes, compared with favorable ones. This issue is inherent to pooling arithmetic, with standard downsampling layers playing major role in favoring certain input sizes and skewing the weights accordingly. We present solution to this problem by depriving these layers from the arithmetic cues they use to overfit the input size. Through various examples, we show how our proposed spatially-balancee pooling improves the generaliza- tion of the network to arbitrary input sizes and its robustness to translational shifts.
INTRODUCTION
Convolutional neural networks (CNNs) versatile models in machine learning. Early CNN architectures used in image classification were restricted to fixed input size. For example, AlexNet (Krizhevsky et al. 2012) was designed to classify 224 x 224 images from ImageNet (Deng jet al. 2009). To facilitate model comparison, this size has been adopted in subsequent ImageNet classifiers such as VGGNet (Simonyan & Zisserman 2015) and ResNet (He al. 2016).
The adoption of fully-convolutional architectures (Long et al. 2015 Springenberg et al. 2015) and global pooling methods (Lin al., 2014; He et al. (015) demonstrated how CNNs process inputs of arbitrary size. Fully convolutional networks climinate the use of fully-connected layers in CNN backbones, preserving 2D feature maps the output of these backbones. Global pooling summarizes feature maps of arbitrary sizes into fixed-size vectors that can be processed using fully- connected classification layers. This ability to process inputs of varying sizes enables CNN-based classifiers to leverage their full resolution and preserving their aspect ratios.
The role of input size in CNNs has been mainly studied with respect computational efficiency, receptive field adequacy, and model performance (Richter et al. [202T]. In this paper, we study the impact input size on the robustness and generalization in analyzing the sensitivity of CNNs with flexible input size variations in this size, as illustrated We demonstrate how the input size(s) used during training can strongly impact this sensitivity, and in turn, the robustness of CNNs to input shifts. We further introduce solution to reduce this sensitivity. Our contributions are:
CNNs. In particular, we are interested
in Figure| [
Demonstrating how CNNs can overfit the boundary conditions dictated by the input size used during training and identifying pooling arithmetic as the culprit (Section 2
Introducing modification to stride-based downsampling layers such maxpooling and strided convolution to mitigate size overfitting (Section 3) and demonstrating how it can improve the accuracy and shift robustness of CNNs in two exemplary tasks.
In Section we discuss the implications of size overfitting and link our observations with relevant findings in the literature.
Work done mainly while at Meta AL.
I We refer to this as size overfitting for brevity.
By pooling we refer any stride-based downsampling such as maxpooling and strided convolution.

---

Published as a conference paper at ICLR 2023
7x7 8x8
9x9
Size of the final feature maps
10x10
Top-1 accuracy on Image/Net
Standard CNN
Spatially- balanced CNN
70
www wwwmmm www
Mm
68
Input size during training: 224x224
66
64
288
Input size at inference time (width = height)
Figure 1: The ImageNet top-1 accuracy of two ResNet- 18 models as function of input size. Both models are trained on 224 x 224 images. The standard CNN represents the baseline available in PyTorch. Our spatially-balances CNN mitigates periodic size overfitting.
2 How Do CNNS OVERFIT INPUT SIZE?
Consider the ResNet-18 implementation available in PyTorch (Paszke et al. 2019). We analyze the sensitivity of the provided pretrained model to variations input size. The model is trained on ImageNet using 224 x 224 images. uses global average pooling to summarize the feature maps of last convolutional layers into scalar values.
2.1 VARYING THE INPUT SIZE
For the purpose of our analysis, we vary the input size from 192 X 192 to 299 x 299. This is done by simultaneously increasing the width and the height by pixel, limiting the input square shape. This simplifies the analysis and preserves the aspect ratio used during training.
We follow the same resizing method used during training when possible. This method first resizes the image so that the smaller dimension is equal to s 256. Then, the method applies a random crop of size 224 224. use the same steps, changing mainly the crop size, and applying centered crop instead of random crop. This maintains the object scale to match the training images. Centered crops are typically used in the validation phase to climinate randomness. A crop smaller than 224 224 incurs a loss of information at the periphery. A crop s' x s' larger than 256 256 would require padding. To avoid padding artifacts, we change the first step to use & = max (s', 2266).
The information loss in crops smaller than 224 224 and the increased object scale in crops larger than 256 x 256 can potentially impact the classification result of certain instances. Nevertheless, the analysis helps identify fundamental impact of input size on CNNs, explain next.
2.2 ANALYZING SENSITIVITY TO INPUT SIZE
For each input size, we compute the accuracy of the pretrained model on the ImageNet validation set after resizing the images described above. Figure Figure[depictt depicts in blue the accuracy function of the input dimension. Both input dimensions are increased simultaneously.
The validation accuracy generally increases with the input size in the range we considered. However, there are remarkable drops in accuracy that occur periodically at an interval 32, immediately after reaching peak. This suggests that the model favors spec ific input sizes that correspond to these peaks, while struggles with inputs that are 1-pixel larger in width and in height. We next demonstrate how these peaks and drops in accuracy byproduct of pooling arithmetic.
2

---

Published as a conference paper at ICLR 2023
convi
maxpool
L2.0.comw2
L3.0.comv2
LA.0.comv2
132 112
Imput size: 224 224
56
29
14xI4
Padding
03,333
Thereol usect
83 3.2, 20
uncersumed 1-pixel line of padding
119
111 fupeee 10 noar the are muay
ieput sine: x25 295
-ugee 12 veeer I mor puin
MITD-B-ppoe Aupooc 12 1000 it ch 101 DALION
Padding
0,3,33
cuppd supputeese feature maps
Thereol uspct
[3.3, 3, 2)
22 apter 1 aua suay
x28
ieput 258
64 64
32 82
Pedding (3, 3,
(3,3, 2)
120
257 957
95
9x9
Padtiigg (,
Thereaf
3,333
used:
Figure 2: Illustrating how input size impacts pooling arithmetic in ResNet-18 18. Each column repre- sents stride-2 downsampling layer. An input of size 224 x 224 or 256 256 leads each layer to leave out --pieel row and column of the applied padding. In contrast, an input of size 225 x 225 or 257 x 257 causes all these maps to have odd dimensions, leaving no padding unconsumed.
2.3 EXPLAINING PERIODIC SIZE OVERFITTING
To understand why the model overfits specific input sizes, analyze how it processes inputs of different sizes. We focus on downsampling layers since the remaining layers in the convolutional backbone do not impact the size. The Py PyTorch implementation of ResNet-18 uses both maxpooling and strided convolution (Springenberg et al.) 2015) for downsampling, with a total of five such layers. All of these layers use odd-sized kernels.
Figure 2 illustrates how four inputs of different sizes are processed by the downsampling layers. In particular, show the output size computed by each of these layers, and whether the layer leaves parts of the padding applied to its input unconsumed. This happens when the dimensions of this input are of a different parity than the odd-sized kernel (Alsallakh et al., 2021b).
An image of size 224 x 224 results in an even-sized input at each layer, leading to unconsumed padding at the right and bottom sides of those inputs. The same happens with input images of size 256 x 256. In contrast, images of size 225 x 225 and 257 x 257 result in odd-sized inputs at each intermediate layer, leaving no padding unconsumed. Input sizes between the above edge cases result in unconsumed padding at subset of the layers. For example, 226 x 226 input impacts only the first downsampling layer while a 227 x 227 input impacts only the second layer. 228 x 228 input impacts the first and the second downsampling layers only.
Since the model is trained on 224 x 224 images, it expects to consume padding only at the left and top sides of the input of each downsampling layer. The peaks of the blue plot in Figure correspond to input sizes that result the same behavior at these layers. The behavior is periodic with respect to input size. It recurs at an interval of Pi where [D] the number of stride-2 downsampling layers and is equal to [D] 5 in model. The sharp drops in Figure | correspond to input sizes that lead to the opposite behavior: The padding is consumed at both sides of every downsampling layer, deviating significantly from the training-tine time behavior. These drops recur at the same interval, e.g. at 193 x 193, 225 x 225 and 257 x 257.
correso[o
Appendix C demonstrates the above issue in various models besides ResNets such Efficient- Net (Tan & Le 2019). MobileNet (Sandler et al. 2018). MNASNet (Tan et al.) 2019), RegNet (Ra- dosavovic al. 2020b, ResNeXt (Xie al. 2017) and VGGNet (Smoonann & Zisserman (2015)
3

---

Published as a conference paper at ICLR 2023
2.4 VALIDATING THE ROLE POOLING ARITHMETIC
To validate observation about periodic size overfitting, retrain the aforementioned ResNet-18 using three different input sizes: 193 x 193, 225 x 225 and 256 x 256. The first two sizes leave no padding unconsumed, while the third size exhibits the opposite behavior Figure illustrates.
Top socuracy on Imagelet
ResNet-18 trained 193x193
Tapa arcurary Imaegettet
ResNet-19 trained 225x225 images
Top-1 Imagetie
ResNet-10 trained 256x256 images
0.72
0.72
mmm
0.52
6.
0.47
0.53
132
132
224 228
25t height)
192x:32
224 x24
258 226 hright)
x
224
25%
input
Input
height)
Figure 3: Sensitivity to input size of ResNet-18 trained using three different input sizes.
Figure R depicts the input-size sensitivity of the resulting models at inference time. It is evident that the model trained on 256 x 256 images exhibits the same behavior as the model trained on 224 224 ones discussed in the previous Section (Figure [T] It favors the same input sizes, and exhibits sharp drops in accuracy at the same unfavorable input sizes, Interestingly, the models trained using 193 193 and 225 225 images do not exhibit sharp drops in accuracy. Their accuracy remains relatively steady within the same interval, with significant jumps between the intervals. These jumps correspond to an increase in the size the final feature maps, e.g. from 7 x 7 to 8 x when the input size changes from 224 x 224 to 225 x 225 (Figure 2
We aim to understand why the first two models do not exhibit periodic drops performance, unlike the third and the original pretrained model. For this purpose, we examine the mean 3 x 3 kernel of each layer as suggested by Alsallakh et al. (022a)). Figure 4 illustrates how these mean kernels computed. Figure 4p depicts the mean kernels of the original model, pretrained on 224 224 images.d d Figure depicts the mean kernels of the model trained on 225 x 225 images. is evident that all mean kernels exhibit high symmetry about their center when trained on 225 225 images. In contrast, the mean kernels of downsampling layers or adjacent layers exhibit a strong asymmetry when trained 224 x 224 images. This asymmetry is due to overexposure zero padding the left and top sides, caused by unconsumed padding as demonstrated by Alsallakh et al. (2021b).
Our analysis confirms the fundamental role of pooling arithmetic in the ability of CNNs to overfit input size. The input size used during training dictates the overfitting behavior. We next introduce an adjustment CNNs that mitigates this behavior.
(a) Aggregating layer weights
(b) ResNet-16 18 trained 224x224 images
(c) ResNet-18 trained 225x225 images
(d) ResNet-18 18 with SBPool trained on 224x224 images
kokk kemne
Figure 4: The impact of unconsumed padding on the learned weights in ResNet-18. (a) Computing the mean kernel in a layer as a means to examine potential spatial skewness. We show the mean 3 x 3 kernels of (b) the baseline model, pretrained on 224 224 images. (c) model trained on 225 x 225 images, and (d) a model trained on 224 x 224 images with our mitigation (SBPool).
4

---

Published as a conference paper at ICLR 2023
3 SPATIALLY-BALANEED POOLING (SBPOOL)
We propose an adjustment to downsampling layers in CNNs that aims to prevent them from overfit- ting the input size during training. Our adjustment is designed to deprive these layers from cues that lead to overfitting, whether the downsampling is based on pooling key insight is that when an input size incurs unconsumed padding at a standard downsampling layer, the latter is always located at the right and bottom sides of the layer's 's input (Fig(pp) In contrast, the padding at the left and top sides is always consumed. This happens over and over again as the CNN processes various training samples of the left and top sides, giving the CNN an artificial cue about that input size.
strided convolution. Our
size. This leads to overexposure padding the
To deprive the CNN from the mentinee cue, we modify downsampling layers so that poten- tially unconsumed padding can occur at any side of the input with equal probability during training (Figure 3p)) With this modification, unconsumed padding is evened on average different sides as the CNN processes various training samples. As a result, unconsumed padding does not incur spatial bias in the learned filters (Figure 4i). We call this modification SBPool.
(a) Standard Pooling
Spatially-balanced Pooling (SBPool)
Unconsumed padding
Randomly select from above variants during training
Figure 5: (a) Standard downsampling: Unconsumed padding is always at the right and bottom sides of the input. (b) Our proposed downsampling: Randomly choose between four sampling grids, to warrant equal likelihood of unconsumed padding at each side.
SBPool is straightforward to implement as a wrapper of standard downsampling layers. The wrapper first computes unconsumed padding (un, uw) as follows:
Un = (h: + Ptop + Pnotton - dn (ki - 1)--1 -1) mod 8h
Uw = (w + Plest + Pright dn (kw 1)
mod 8w
where (h, w) are the spatial dimensions of the layer's input, (Prop. Pieft Poottom Pright) is the padding intended to be applied to that input, (ka, kw)) is the kernel size, and (dh, dw)) is the kernel dilation, and ($h. 8w) are the stride factors. The padding applied is then modified randomize at which side the unconsumed padding (un, Uw) is effectively incurred:
Peop &
PLop randInt (10, unl)
Plest + Pleft - randInt ([0, uw]))
where randint() is function that returns integer within the given range uniformly at random.
The above modification to the padding applied allows the convolutional kernel to randomly assume any of the starting positions and corresponding sampling grids illustrated in Figure Appendix A provides an algorithmic outline of the described modification. Note that when parallel branches perform downsampling in skip connections, ensure the sampling grids they assume aligned (refer to Appendix E for illustration).
At inference time, assign each downsampling layer fixed sampling grid to ensure determin- istic evaluation results. We select different grids for these layers to minimize the distortion of the receptive field as we discuss in Section 4
We demonstrate how SBPool mitigates size overfitting in two exemplary tasks that involve different CNN architectures and datasets. Appendix provides additional examples on other architectures.
Thhe equation adapted from the documentation of MaxPool2d in PyTorch
5

---

Published as a conference paper at ICLR 2023
3.1 IMAGE CLASSIHICATION
We retrain the ImageNet classification model described in Section 2 with SBPool using the same hyperparameters. Both the baseline model and the new model trained 224 224 images, and their corresponding feature maps are hence of equal sizes. Accordingly, both models use the same compute power (FLOPS) during training.
Figure D depicts the top-1 accuracy of both models as a function of the input dimension. For each input size, we resize the images in the validation set as described in Section 2.1 Table [ lists the accuracy for selected sizes. It is evident that the spatially- balanced model does not suffer from sharp drops in accuracy, unlike the baseline model. With sizes favorable for the baseline model, marked with (*), both models have comparable performance. However, at unfavorable sizes the spatially- balanced model significantly outperforms the baseline model.
Table 1:
Top-1 accuracy of ResNet-18 ImageNet for different input sizes the inference phase.
Input size
221 x 22/1"
225 x 255
256 x 256"
257 x 257
288 x 288"
289 x 289
Baseline
69.76
66.53
70.63
68.74
70.83
68.53
CNN with SBPool
69.64
70.10
70.81
70.73
71.54
71.21
Figure Fl depicts the mean kernels of the spatially-balanced model. evident that SBPool miti- gates the asymmetry observed in the baseline model (Figure 4p). This improves the shift robustness of the model, as we demonstrate in Appendix C as MobileNet and VGGNet as we demonstrate Appendix D
The results generalize to other model families such
3.2 SEMANTIC SEGMENTATION
Dilated residual networks Yu et al. 2017) serve as a simple architecture for semantic image seg- mentation. We select model pretrained by the authors M on the Cityscapes dataset (Cordts al., 2016). The model contains three downsampling layers [iD = 3). Its weights are initialized using an equivalent image classification model trained on ImageNet. Subsequently, the model is trained on 896 x 896 image patches randomly sampled from the Cityscapes training set. In the testing phase, the model is applied the entire image, whose size is 2048 x 1024.
To obtain a spatially- balanced segmentation model, we first retrain the classification model used for initialization under SBPool, and then retrain the segmentation model. Figure depiets the perfor- mance of both the baseline model and the spatially-balanced model as functions of input size. For each size, we crop the test set images along with the corresponding ground truth images into cen- tered squares of that size and compute the mean Average Precision (mAP) of both models on those images.
It is evident in Figure 6 that the baseline model exhibits periodic size overfitting at interval of 2| = 8. This behavior is mitigated in the spatially- model whose performance increases steadily with input size, exhibiting minor fluctuation. Moreover, the latter model outperforms the baseline by visible margin. Table the mAP of both models for different input sizes, including the ones used originally in the training phase (896 x 896) and in the validation phase (2048 1024). The performance of the baseline model drops significantly with the unfavorable sizes of 201 201 and 897 x 897.
Table 2: Performance (mAP) of segmentation models on CityScapes with different crop sizes.
Input size (centered crop)
200x200
201 x201
896
897x897
2048x 1024
Baseline
41.180
39.380
66.63
66.19
68.00
CNN with SBPool
41.65
42.28
67.65
67.72
68.37
"This does not hold when training on 225 x 225 images as done by Alsallakh al 2022 since the internal feature maps will be larger than the ones with 224 x 224 input images as illustrated in Figure 'The model name is DRN-D-22 and is available at hthpt::// /go. yf. wt..ic//oe wtt.ii//drcce www.pppyyycppp.... model 1

---

Published as a conference paper at ICLR 2023
0.9
Standard CNN
CNN with SBPool
scale break
MAP on Cityscapes centered crops
0.50
0.48
FCca broak
0:34
6.42
Input size inference time (width height)
Figure 6: The mean Average Precision (mAP) two Cityscapes semantic segmentation models of the same architecture Yu et al. 2017), function of input size. The baseline model is pretrained by the original authors The spatially-balane -halanced model is retrained under SBPool.
DISCUSSION
Size overfitting has been observed in certain language models trained fixed input size (Wang & Chen 22200). We have demonstrated that, unless mitigated, CNNs are prone to overfitting input size, which can skew the learned weights.
Why Does Size Overfitting Matter? Even if the input size is identical at training and inference time, it is worth mitigate size overfitting. This mitigation improves shift invariance as we show in Appendix C Oftentimes, the learned weights used initialize other models, trained tasks and datasets. For example, weights pretrained on ImageNet with 224 x 224 images are widely used for this purpose. even when the downstream task uses a different variable input size. This makes the pretrained weights sub-optimal starting point, as our results demonstrate (Figure 6)
other
Sources of Inspiration SBPool inspired by practical solutions related problems. Notably, to prevent flat-head syndrome in infants, the head orientation should be varied when in supine po- sition (Xia al. 2008). Likewise, spaceships need continuous rolling avoid overheating of the side exposed to the Sun, and automobile tires need to be rotated on regular basis. Wu et al. (2019) employ similar idea to avoid information erosion when using 2 x convolutional kernels. Their solution divides the feature maps of each layer into four groups that correspond to the variations in Figure The key difference our work is that SBPool injects variation across different samples not within the feature maps computed for one sample. Accordingly, it serves different purpose and only needed for downsampling layers during training.
4.1 IS PADDING THE CULPRIT?
Padding is known to impact CNN's spatial invariance (Kayhan & van Gemert 2020 Islam al. 2020). We examine whether is the only cue CNNs can use to overfit input size.
What Other Padding Methods Were Applied? The downsampling layers in the examples presented so far use zero padding. Other padding methods were shown to alleviate feature map artifacts incurred by salient lines zero (Alsallakh et al., 2021b). We analyzed size overfitting in various ResNet models trained ImageNet under different padding methods, including Partial Convolution (Liu et al. 2018). All of these models are trained on 224 x 224 images, which incur unconsumed padding every downsampling layer. Figure two examples, showing how the models overfit the input size. This demonstrates how one-sided unconsumed padding can provide CNNs with sufficient cues about the input size, regardless of the padding method.

---

Published as a conference paper at ICLR 2023
ResNet-18 with circular convolution
ResNet-50 with partial convolution
Top-1 accuracy on ImageNet
Top-1 alccuaccyy on ImageNet
MV www
Mwmammyy mwm
0.76
0.67
0.74
0.64
0.72
input size
height)
Input size
height)
Figure 7: Sensitivity input size of two ResNet models trained under different padding methods:
circular and PartialConv (Liu et al., 2018). Both models are trained on 224 224 images.
What if No Padding Were Applied? Downsampling layers often apply padding to avoid marginalization of the periphery, especially when their kernels are 3 x 3 or larger. When no padding applied,
[ computes unconsumed parts of the layer's input that represent information crosion. Appendix D provides an example with VGGNet which applies padding during downsampling. Nevertheless, VGGNet overfits the input size. Furthermore SBPool mitigates this overfitting by ran- domly selecting which side the information erosion occurs. This suggests that information erosion provides CNNs with cues to overfit the input size. To further investigate such cues, we next analyze how different input sizes impact the receptive field.
4.2 IMPACT ON RECEPTIVE FIELD
Richter et al. 2021) demonstrate how mismatch between input size and the receptive field impacts object recognition in CNNs, focusing on object scale. Jang et al.] (2022) proposed a data-driven approach optimize the receptive field. Here focus how pooling arithmetic impacts the receptive field, making it sensitive to small changes in input size.
Figure depicts the receptive field of ResNet-50 model under various conditions, following the visualization method proposed by Alsallakh et al. (221c). A 225 x 225 input incurs no unconsumed padding, leading to a well-centered receptive field in the imput space. 224 224 input incurs un- consumed padding at every downsampling layer. This in turn shifts the center of the receptive field the baseline model to the top left corner. By applying SBPool, unconsumed padding is balanced opposing sides, recovering the alignment between the receptive field and the input. The misalign- ment in the middle plot of Figure provides a cue which enables CNNs to periodically overfit sizes that exhibit the same misalignment. Shocher et al. (2020) observed similar misalignment between the input and the output in CNNs and proposed a special type of convolutional layers as a mitigation.
225 225 input (standard CNN)
224 x 224 input (standard CNN)
224 224 Input (CNN w. SBPool)
Figure 8: The receptive field of ResNet-50 under various conditions. Without SBPool, 224 224 input causes misalignment of the field's center in the input space, as the middle plot demonstrates.
4.3 POOLING TECHNIQUES IN CNNS
Subsampling, pooling, and strided convolution techniques are ubiquitous in modern CNNs, and have received significant amount of research attention. Boureau et al. (2000) explored mixing max and
8

---

Published as a conference paper at ICLR 2023
ResNet- 18 with fractional pooling
ResNet- 18 with BlurBool
Top-1 accuracy en ImageNet
Top-1 accuracy on ImaggeNet
wwwwww V
0.71
0.68
0.66
ww
0.69
0.6 x 192
0.6
224 224
256 256
192
224 x 224
256 x 256
Input size at inference time (width height)
Input inference time (width height)
Figure 9: Sensitivity to input size under different pooling methods, Fractional Pooling (Graham 2014) and BlurPool (Zhang 2019). Both models are trained on 224 224 images.
average pooling, with extensions proposed include learned weights (Lee et al. (016), stochasticity Yu et al., [0144), or both (Kobayashi] (2019b). These extensions aim to improve on the deficiencies of max and average pooling, namely detail loss, introduction of artifacts, and overfitting. A family of
related methods involve randomly sampling from pooling regions, and tend to confer regularizing effect and to increase shift invariance (Zeiler & Fergus, 2013 Zhai al., 2017 Kobayashi] (2019a). At inference time, such stochastic pooling methods typically use some form of averaging over the probabilities used during training.
Another area of work involves subsampling operators which permit arbitrary input-to-output ratios (Graham] 2014 Jang al., [0222 Shocher al.] (020). Spectral pooling methods (Rippel et al.
[ & Ma 2020 Zhang 2021) cast subsampling as cropping in the frequency domain,
allowing any output size while retaining maximum information. Recently, RRad et al.] (2022) ex- tended spectral pooling allow learnable strides. Decomposing images into frequency components can also be seen in wavelet pooling (Williams Li 2018) and in LiftPool (Zhao & Snock 2021).
A number of pooling techniques have been proposed to improve translation invariance. BlurPool (Zhang 2019) reduces the aliasing incurred by strided operations by applying a Gaussian blur. Hos- sain et al. (2021) uses similar approach, additionally allowing the standard deviation to be learn- able to control the strength of blurring. Xu et al. (202T) and CChamnn & Dokmanic (202T) enforce translation equivariance using sampling techniques tailored for that purpose.
Besides max-pooling, average pooling, and strided convolution, we analyze input-size overfitting under two popular pooling techniques: Fractional Pooling (Graham 2014) and BlurPool. For this purpose, we employ these techniques in two ImageNet classification models, trained on 224 x 224 images. The first model is based on VGGNet, where we replace max-pooling layers with fractional pooling layers having an input-output ratio of 0.5. The second model is ResNet- 18 pretrained by [Zhang] (2019). Figure [ depicts the performance of each model as a function of input size. is evident that the both models exhibit periodic size overfitting. Interestingly, the BlurPool model exhibits a delay in the performance drops, compared with the baseline model in Figure As future work, we interested in understanding potential overfitting behavior of various pooling methods and how applicable SBPool could be with these methods.
SUMMARY
We demonstrated how pooling arithmetic makes CNNs susceptible to overfitting the input size. The overfitting behavior is periodic, favoring sizes that induce similar boundary conditions at pooling layers to the ones encountered during training. We demonstrated how overfitting can skew the learned weights and impact shift invariance CNNs. We presented technique to mitigate overfit- ting by preventing pooling layers from developing specific arithmetic patterns during training. Our technique can be incorporated into standard pooling and downsampling layers. Through various experiments, demonstrate how this technique improves the robustness of CNNs translational shifts and to changes in the input size. This helps make the learned representation more generic for downstream tasks and can confer significant improvement in their accuracy.

---

Published as a conference paper at ICLR 2023
REFERENCES
Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani, Shubham Muttepawar, Edward Wang, Sara Zhang, David Adkins, and Orion Reblitz-Richardonn. Debugging the internals of convolutional networks. In eXplainable Al approaches for debugging and diagnosis NeurlPS Workshop, 2021a.
Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani. Jun Yuan, and Orion Reblit-RRichardsnn Mind the pad cnns can develop blind spots. In International Conference on Learning Representations (ICLR), 2021b.
Bilal Alsallakh. Vivek Miglani. Narine Kokhlikyan, David Adkins, and Orion Seblitz-Richardson Are convolutional networks inherently foveated? In SVRHM 2021 Workshop@ @ NeurlPS, 2021c.
Y-Lan Boureau, Jean Ponce, and Yann LeCun. A theoretical analysis of feature pooling in visual recognition. International conference on machine learning. pp. 111-118, 2010.
Anadi Chaman and Ivan Dokmanic. Truly shift-invariant convolutional neural networks. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 3773-3783, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Be- nenson, al. The cityscapes dataset for semantic urban scene understanding. IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Patern Recognition, pp. 248-255 IEEE, 2009.
Benjamin Graham. Fractional max-pooling. arXiv preprint
2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep con- volutional networks for visual recognition. IEEE transactions on pattern analysis and machine intelligence,37/0 37(9); 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE conference on Computer Vision and Puttern Recognition, pp. 770-778, 2016.
Md Tahmid Hossain, Shyh Wei Teng, Ferdous Sohel, and Guojun Lu. Anti-aliasing deep im- age classifiers using novel depth adaptive blurring and activation function. arXiv preprint arXiv: 10.22/01.000999 2021.
Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information do convolutional neural networks encode? In International Conference Learning Representations (ICLR), 2020.
Dong-Hwan Jang, Sanghyeok Chu, Joonhyuk Kim, and Bohyung Han. Pooling revisited: Your receptive field is suboptimal. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 549-558, 2022.
Osman Semih Kayhan and Jan C van Gemert. On translation invariance in CNNs: Convolutional layers can exploit absolute spatial location. In IEEE conference on Computer Vision and Pattern Recognition, 2020.
Takumi Kobayashi. Gaussian-based pooling for convolutional neural networks. Advances in Neural Information Processing Systems, 32, 2019a.
Takumi Kobayashi. Global feature guided local pooling. In /EEE/CVF International Conference on Computer Vision, pp. -374-3374, 2019b.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pp- 1097-1105, 2012.
Chen- Yu Lee, Patrick W Gallagher, and Zhuowen Tu. Generalizing pooling functions in convo- lutional neural networks: Mixed, gated, and tree. Artificial intelligence and statistics, pp- 464-472. PMLR, 2016.
10

---

Published as a conference paper at ICLR 2023
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In International conference learning representations (ICLR), 2014.
Guilin Liu, Kevin J. Shih, Ting- fing-Chun Wang, Fitsum A. Reda, Karan Sapra, Zhiding Yu, et al. Partial convolution based padding. arXiv preprint arXiv:1811.11718 2018.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, al. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8024-8035, 2019.
lljaa Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In /EEE/CVF conference on computer vision and pattern recognition, pp. 10428- 10436, 2020.
Rachid Riad, Olivier Tehoul, David Grangier, and Neil Zeghidour. Learning strides in convolutional neural networks. International conference on learning representations (ICLR), 2022.
Mats I. Richter, Wolf Byttner, Ulf Krumnack, Anna Wiedenroth, Ludwig Schallner, and Justin Shenk. (input) size matters for cnn classifiers. In International Conference on Artificial Neu- ral Networks, pp. 133-144. Springer, 2021.
Oren Rippel, Jasper Snoek, and Ryan Adams. Spectral representations for convolutional neural networks. In Advances in neural information processing systems, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In IEEE conference computer vision and 2018.
pattern recognition, pp.
Assaf Shocher, Ben Feinstein, Niv Haim, and Michal Irani. From discrete continuous convolution layers. arXiv preprint arXiv:2006. /1tp:20000.//200. /20, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conferencee on Learning Representations (ICLR), 2015.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In International Conference on Learning Representations, 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114 14. PMLR, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform- aware neural architecture search for mobile. In JEEE/CVF Con- ference Computer Vision and Patlern Recognition, pp. 2820-2828, 2019.
Yu-An Wang and Yun-Nung Chen. What do position embeddings learn? an empirical study of pre-trained language model positional encoding. In Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6840-6849, 2020.
Travis Williams and Robert Li. Wavelet pooling for convolutional neural networks. In International Conference on Learning Representations, 2018.
Shuang Wu, Guanrui Wang. Pei Tang, Feng Chen, and Luping Shi. Convolution with even-sized kernlls and symmetric padding. Advances in Neural Information Processing Systems, 32, 2019.
James J Xia, Kathleen A Kennedy, John F Teichgraeber, Kenneth Q Wu, James B Baumgartner, and Jaime Gateno. Nonsurgical treatment of deformational plagiocephaly: a systematic review. Archives of pediatrics & adolescent medicine, 162(8):719-721) 2008

---

Published as a conference paper at ICLR 2023
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In JEEE conference on computer vision and pattern recog- nition, pp. 1492-1500, 2017.
Jin Xu, Hyunjik Kim, Thomas Rainforth, and Yee Teh. Group equivariant subsampling. Advances in Neural Information Processing Systems, 34:5934-5946, 2021.
Dingjun Yu, Hanli Wang, Peiqiu Chen, and Zhihua Wei. Mixed pooling for convolutional neural networks. In International conference on rough sets and knowledge technology. pp. 364-375. Springer, 2014.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In IEEE conference on computer vision and pattern recognition, pp. 472-480, 2017.
Matthew D Zeiler and Rob Fergus, Stochastic pooling for regularization of deep convolutional neural networks. International Conference on Learning Representations (ICLR), 2013.
Shuangfei Zhai, Hui Wu, Abhishek Kumar, Cheng, Yongxi Lu, Zhongfei Zhang, and Rogerio Feris. S3pool: Pooling with stochastic spatial sampling. In IEEE Conference Computer Vision and Puttern Recognition, pp. 4970-4978, 2017.
Hao Zhang and Jianwei Ma. Hartley spectral pooling for deep learning. CSIAM Transactions on Applied Mathematics, 1(3):188 5298,52299-02 2020. ISSN 2708-0579.
Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pp. 7324-7334. PMLR, 2019.
Zhendong Zhang. Frequency pooling: Shift-equivalent and anti-aliasing downsampling. arXiv preprint arXiv:2109. 11839, 2021.
Jiaojiao Zhao and Cees GM Snoek. LiftPool: Bidirectional convnet pooling. In International Conference Learning Representations, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
A SBPOOL ALGORITHM
The following listing demonstrates how SBPool wraps existing downsampling layers to spatially- balance the downsampling results. Refer to Eq [ and Eq 2q[ explanation of the layer parameters involved (denoted by L.voriable) and of the variables introduced (u, and uw.).
Algorithm 1: Spatially- Spatially-Bllanced Pooling
Input: L: The downsampling layer, e.g. maxpooling strided convolution
Input: X: The layer's input: a tensor of shape (#samples, #channels, height, width)
Output: Y: Downsampling result
Un. + (X.height + L.Prop + L.Poostom - L.dn (L.kn 11 - mod L.sk
+ (X.width + I.Pioft + TT.pright L.dr (T.kiu - 1) mod L.sw
L.Prop +- L.Prop randInt (10, unl)
L.Prent T.Ppeet randInt uw]))
if L.Prop 0 then
X < Xlisi, -L.prop : L.Peop + 0 /* balance vertical erosion
if L.Pieft < 0 then
X + - : L.Piect + 0 balance horizontal erosion */
Y I(X)
Reset L.Prop and L.Piert to their original values
12

---

Published as a conference paper at ICLR 2023
B
SIZE OVERHITTING EXAMPLES
Here we demonstrate size overfitting in various models trained on different datasets and tasks.
B. SCENE CLASSIHCATION
We examine two ResNet models trained Places365 (Zhou et al. 2017 and provided by the dataset curators at https::p. : /tt/g/tttiii siitthu.... wttpuu..oooo/oooom wwt.///ASSiiiiii.... hwt.wssio/plllmmm ces366 The images were resized to 224 224 during training. The models have five downsampling layers that use odd-sized kernels. Figure demoosttratss how both models exhibit periodic size overfitting at an interval of 32, similar to the ImageNet example presented in Section
ResNet-50
ResNet-18
Top-1 Accuracy on the Places- 365 validation set
w
wmw
032
Input during training: 224 224
Input size at inference time (width height)
Figure 10: The accuracy of two scene classification models, trained on Places-65, function of input size. Both models are trained on 224 224 images and contain |D] = downsampling layers.
Both models exhibit periodic size overfitting at an interval 2| 32.
B.2 CITYSCAPES SEMANTIC SEGMENTATION
We select two additional Cityscapes models pretrained by the author of dilated residual networks: DRN-D-38 and DRN-D-105, available at htttp://go.y ctt.rr/cttyscpssseeeeea---- modelessmm
Both models contain three downsampling layers. Figure 11 demonstrates how both models exhibit
periodic size overfitting at an interval of 23 8, similar to the example presented in Section 3.2
MAP (96) on the CityScapes
DRN-D-105
DRN-D-38
vall vallccation set
Architecture: Dilated Residual Network (DRN)
Input size during training: 896 x 896
200110
2011211
340330
aitad
284333
Input size at inference time (width height)
Figure 11: The mAP of two semantic segmentation models CityScapes, function of input size. Both models are pretrained by Yu et al. (2017) on 896 x 896 patches and contain [D] = downsampling layers. Both models exhibit periodic size overfitting at an interval of 2 DI
13

---

Published as a conference paper at ICLR 2023
B.3 IMAGENET CLASSIHICATION
We examine five ImageNet classification models available in PyTorch (Paszke et al. 2019) and
pretrained 224 x 224 images. Each model contains five downsampling layers (TD 5) that odd-sized kernels. Figures] and demonstraats how these models exhibit periodic size overfitting an interval of 32. It is noticeable that with MNASNet (Tan al.) 2019). there are intermediate peaks and drops, c.g., when the input size is 200 x 200, 208 x 208, or 216 x 216. These peaks correspond to partial agreement between the unconsumed padding patterns between these sizes and the size used during training (see Figure 2
RegNet-x8-gf
ImageNet validation set
Top-1 accuracy on
ResNext-50_32x40
EfficlentNet-00
Input size during training: 224 x 224
2203
Input size at inference time (width height)
Figure 12: The top-1 accuracy of three ImageNet classifiers as function of input size: Efficient- Net (Tan & Le 2019), RegNet (Radosavovic et al., 2020), and ResNeXt (Xie et al.) (017)) Each model contains five downsampling layers and exhibits periodic size overfitting at an interval of 32.
MobileNet-V2
Top-1 accuracy on validation set
MNASNet
ImageNet
Input size during training: 224 224
25t cast
Input size at inference time (width height)
Figure 13: The top-1 accuracy of two ImageNet classification models, MobileNet (Sandler et al. 2018) and MNASNet (Tan al. 2019). as function of input size. The models contain [ downsampling layer and hence exhibit periodic size overfitting at an interval of 2| 32. Inter- mediate peaks are also visible within each interval. These peaks correspond to partial agreement in unconsumed padding patterns between the respective input size and the one used during training.
14

---

Published as a conference paper at ICLR 2023
C SIZE OVERHITTING AND SHIFT CONSISTENCY
We compute the shift consistency metric proposed by Zhang 2019) for the models discussed in Section 3 with range input sizes. Figure [4 shows the consistency both of the standard ResNet- 18 and of the model trained on ImageNet under SBPool as functions of input size. It is evident that SBPool improves the consistency for input sizes that not match the preferred ones.
85
Shift Consistency
Standard CNN
CNN with SBPool
Input size during training: 224 224
Input size inference (width height)
Figure 14: The shift consistency of ResNet-18 on ImageNet classfiittion as a function of input size, with and without SBPool.
To compute the consistency in case of semantic segmentation, generalize the metric proposed by (Zhang 2019) beyond image classification. For this purpose, we use the segmentation result of the full input size at inference time reference, and compute how consistent these results with the ones computed for random crops of a specific size. Figure [4]shows the consistency of the standard DRN model on CityScapes, compared with the model trained under SBPool as functions of the crop size. It is evident that SBPool improves the consistency for various input sizes by significant margin.
CNN with SBPool
www
Standard CNN
Shift Consistency
Architecture: Dilated Residual Network (DRN)
0.89
Segmentation model: drn-d-22 2222
Input size during training: 896
Input width inference time (height is fixed to 1024)
Figure 15: The shift consistency of the DRN-D-22 segmentation model computed on CitySeapes with and without SBPool, function of input size.
15

---

Published as a conference paper at ICLR 2023
D SBPOOL EXAMPLES
We demonstrate SBPool two additional models, besides the ones in Section
S
D.1 MOBILENET-V2
We retrained the MobileNet-V2 (Sandler et al. 2018) model available in PyTorch for ImageNet classification under SBPool, using the same training recipe. Figure [6 plots the size sensitivity of both the retrained model and the one provided by Py PyTorch. As expected, the retrained model improves the generalization to varying input size.
Standard CNN
Top-1 accuracy on ImageNet validation set
CNN with SBPool
Input size during training: 224 224
Input size at inference time (width height)
The top-1 accuracy of MobileNet-V V2 on ImageNet classification as a function input size, with and without SBPool.
Figure 16:
D.2 RESNET-50
We retrained the ResNet-50 (He et al.) 2016) model available in PyTorch for ImageNet classifica- tion under SBPool, using the same training recipe. Figure plots the size sensitivity of both the retrained model and the provided by PyTorch. As expected, the curve corresponding to the retrained model exhibits no sharp drops in performance and less oscillation, unlike the pretrained model. Moreover, the mean kernels of the retrained model show high symmetry, while certain mean kernels in the pretrained model are highly asymmetric.
Input during training:
Standard CNN
Top-1 acouracy on validation set
Mean kernels
ImageNet
CNN with SBPcol
Input size at inference time (width height)
Figure 17: The top-1 accuracy of ResNet-50 ImageNet classification function of input size, with and without SBPool.
16

---

Published as a conference paper at ICLR 2023
D.3 VGGNET
VGGNet models for ImageNet classification contain five maxpooling layers. These layers use x kernels and apply no padding, unlike ResNet models. Accordingly, input of size 224 224 is evenly processed at each these layers. Moreover, An input of size 223 x 223 leads to uneven erosion at each pooling layer: 1-pixel lines the bottom and right sides are ignored by the layer.
We trained two VGG-11 models on ImageNet with 223 x 223 as input size, once with SBPool and once without SBPool. Figure left shows the sensitivity of both models input size inference time. As expected, without SBPool the CNN overfits the input size: It shows significant drops when the size 224 224 or 256 x 256, as both sizes prevent erosion, unlike the size for training. In contrast, with SBPool the CNN does not overfit the input size, instead showing steady improvement the top-1 accuracy as the input size increases, with minimal oscillation.
Figure right compares the sensitivity of the SBPool model with the pretrained model available in Py PyTorch, which trained on 224 x 224 images. While the latter model does not show sharp drops, it does prefer the size used during training, showing significant jump in accuracy at this size as well as at 256 x 256. Like the training size, the latter size prevents erosion at pooling layers.
VGG-11 trained on 223x223 Images
VGG-11 with SBPool trained on 223x223 images
VGG-11 Trained on 224x224 Images
Top-1 acourscy on ImageNett validation set
3091330
Input size at inference time (width height)
Input size at inference time (width height)
Figure 18: The top-1 accuracy of three VGG-11 models on ImageNet classification as a function of input size. The left plot compares two of these models trained on 223 223 images with and without SBPool. The right plot compares the SBPool model with the baseline model available in PyTorch.
E SBPOOL WITH PARALLEL BRANCHES
77 067
D8
Tdme 64
44 Nuno 536
RC1 10000 EXg
821
952 1000 EX
258
215
Image
7x7 mame
Lool
Kuno
1000 2X
2003 pa
Mern pap
Food and
[c 1000
Figure 19: A diagram of ResNer-18 showing the main branch well skip connections. Down- sampling layers are highlighted in orange. The diamonds represent downsampling operations that take place in skip connections, parallel to their counterparts in the main branch. We that these operations use the same sampling grids SBPool assigns to their counterparts in the main branch, (Di- agram adapted from He et al.)(220 (2016)).
17


Reviews:
[{"review_id": "uzSKPXBMbCg", "reviewer": "Reviewer_L9hd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates a problem I would not have thought about -- CNNs overfitting to input size due to the pooling arithmetic involved -- this in itself is an extremely exotic thought! The authors show that due to the unconsumed padding, often the networks trained on specific input sizes will not generalize or extrapolate They proceed to propose a simple solution called Spatially-Balanced Pooling (SBPool) that mitigates this phenomenon, by randomly alternating the location of the unconsumed line of padding. The results are extremely strong for ImageNet classification and Semantic segmentation across architectures. They conclude with a nice analysis and discussion alongside strong baselines which fail to fix the phenomenon.\n\n\n(The brevity of the review should not be taken negatively, it is due to the clarity of the paper in its ideas, hypothesis and experimentation). \n\n", "summary_of_the_review": "See above. I really want the authors to submit a revision with fixed figures. It reflects poorly to have this as a major issue when the idea and the paper itself are extremely well done!", "strength_and_weaknesses": "My summary of the paper notes the strengths of paper. It is an extremely simple idea that works very well in practice. Experiments and discussion are through along with the exposition which helped set up the problem in the first place. \n\nWhile I do not have technical qualms, I am extremely disappointed by the inaccessible plots and figures. They are of extremely poor quality and would benefit from a revision with every single plot and figure being redone and rendered with vector graphics. \n\nI strongly urge the authors to fix this and I would be willing to give it a higher score after that and after discussions with other reviewers. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "POJlS8Ddc3", "reviewer": "Reviewer_JXFi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides an analysis of the effect of training convolutional neural networks with a particular image size in the generalisation to different image sizes at inference time, as a consequence of downscaling operations within the architecture and specifically the padding necessary for enabling flexible input sizes. The analysis demonstrates that standard methods exhibit drops in performance in a periodic fashion with the input size, as best illustrated in Figure 1, and provides an explanation for the effect. Further, the paper presents simple mechanism to overcome this issue: randomising the location of the padding during training to break the bias due to a systematic location, as is common practice.", "summary_of_the_review": "My overall impression of this paper is positive, as it provides a comprehensive analysis of a pitfall in CNNs trained in the currently standard way, even if typically the input size is fixed as well during test time, as well as a simple enough solution to overcome the problem. Furthermore, the paper is well written. Nonetheless, it is also fair to point out that the novelty is limited in view of previous work such as Alsallakh et al. (2021) or Shocher et al. (2020), for instance.", "strength_and_weaknesses": "### Strengths\n\nThe problem tackled by this paper is conceptually simple and therefore easy to follow. The paper is well written, the figures are very informative and the experimental design helps sheds light on the aspects that are analysed. Furthermore, the conclusions are supported by the results. Finally, the paper not only analyses a pitfall in the way convolutional neural networks are typically trained, but it also presents a method to overcome the problem. Therefore, I have an overall good impression of the paper.\n\n### Weaknesses\n\nMy main concern is related to the novelty of the results presented in the paper. The paper seems to be an incremental extension of Alsallakh et al. (2021), which analysed spatial biases in CNNs due to padding. This paper sets the focus on the input size and the pooling operation (\"mind the pool\" vs. \"mind the pad\", in the titles), but fundamental issue underlying the pitfalls analysed in both papers is, my opinion, the same: padding. The issue with pooling is really padding, not pooling itself, as indeed explained by the authors. In fact, the problem seems to be identical regardless of the type of downscaling (due to pooling or striding), precisely because it is caused by padding. All elements---padding, input size, downscaling---are related, and in fact Alsallakh et al. (2021) already the effect of input size and pointed out at sharp differences when the input size differed by just one pixel. \n\nSeveral elements of analysis in the present paper were already discussed in Alsallakh et al. (2021), for instance the asymmetry of the kernels (Figure 4 in \"mind the pool\" and Figure 6 in \"mind the pad\") or the shift of the receptive field (Figure 8 and 7 respectively). Furthermore, other previous work, such as Shocher et al. (2020) had set the focus specifically in the downscaling operation, as pointed out here. In this regard, a significant part of the analysis offers little novelty.\n\nUnrelated to the above, one question I have regarding the proposed method to remove the bias towards the training image size is what the influence is of fixing one side for the padding at test time. I expect little influence, but having to stick to just one seems arbitrary and it would be nice to show experimentally whether the choice has no, little or moderate influence in the performance. As a suggestion, the curves in Figure 1 and equivalent figures could display a confidence interval obtained from testing with multiple options for padding at test time.\n\nFinally, as a minor comment, when mentioning fully-convolutional architectures, I would cite Springenberg et al. (2015) (cited later in the paper) as well as Long et al. (2015). (The reviewer is not related to any of these papers).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7UYSuKFv7g4", "reviewer": "Reviewer_b1SZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper looks at how convolutional neural networks can overfit on the input size. This is first shown experimentally (figure 1) and then the authors hypothesize that it is due to downsampling (figure 2). Simply put, there is no way to apply a kernel of size 3 (for example) with a stride of 2 over an input of size 4 in such a way that it is not biased. They validate this hypothesis (figures 3 and 4) experimentally.\n\nIn section 3 the authors propose a solution: In cases where the kernel size is not a divisor of the padded input size, they divide the padding between both sides randomly. They show experimentally that this solution reduces sensitivity to input sizes and finish the paper with a detailed discussion of related work.", "summary_of_the_review": "A simple solution to the simple problem of downsampling layers in neural networks overfitting to the boundary conditions, but given that this is a well-written paper with an exhaustive set of experiments, I think this paper should be considered for acceptance.", "strength_and_weaknesses": "I believe this paper highlights a simple problem, explains the cause, and proposes a simple solution. Although the problem and solution are both simple, the authors did a good job at being very exhaustive in their experiments and discussion of related work, which I believe gives the paper sufficient content to be considered for acceptance.\n\nMost of my concerns are relatively minor. One minor issue I see is that the paper repeatedly talks about \"unused padding\" or \"pooling\" as the reasons for overfitting. But the input size overfitting is unrelated to padding, which the authors acknowledge in section 4.1. The term \"pooling\" on the other hand seems a bit confusing, because that is a word I associated with, e.g., average- or max-pooling, and not with a regular downsampling layer (i.e., a kernel which uses a stride greater than 1). Perhaps the authors could clarify these nuances earlier on.\n\nMinor comments:\n\n* There's a bracket missing in section 2.4 after \"(Figure 1\"\n* In section 2.2 it says that \"it is not immediately obvious why the other peaks recur at an interval of 32\". I read this statement as \"we don't know why this happens\", but then in section 2.3 it turns out that the authors do know. Perhaps this sentence can simply be deleted.\n* The term \"overfitting\" is a bit confusing, because the model's best results are often not for input sizes that it was trained on (e.g., in figure 1 the best results are for 256 or 288, not for 244), so it's strange to say that it has overfit to the training input size. I guess the precise statement would be that the model has overfit to the border conditions caused by downsampling? Perhaps the wording should reflect this (e.g., in the \"Our contributions\" part in section 1).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster