Boundary Effects in CNNs: Feature or Bug?

1 INTRODUCTION . One of the main intuitions behind the success of CNNs for visual tasks such as image classification ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; Szegedy et al. , 2015 ; Huang et al. , 2017 ) , video classification ( Karpathy et al. , 2014 ; Yue-Hei Ng et al. , 2015 ; Carreira & Zisserman , 2017 ) , object detection ( Ren et al. , 2015 ; Redmon et al. , 2016 ; He et al. , 2017 ) , generative image models ( Brock et al. , 2018 ) , and semantic segmentation ( Long et al. , 2015 ; Noh et al. , 2015 ; Chen et al. , 2017 ; 2018 ) , is that convolutions add a visual inductive bias to neural networks that objects can appear anywhere in the image . To accommodate the finite domain of images , manual heuristics ( e.g. , padding ) have been applied to allow the convolutional kernel ’ s support to extend beyond the border of an image and reduce the impact of the boundary effects ( Wohlberg & Rodriguez , 2017 ; Tang et al. , 2018 ; Liu et al. , 2018a ; Innamorati et al. , 2019 ; Liu et al. , 2018b ) . Recent studies ( Pérez et al. , 2019 ; Islam et al. , 2020 ; Kayhan & Gemert , 2020 ) have shown that zero padding allows CNNs to encode absolute position information despite the presence of pooling layers in their architecture ( e.g. , global average pooling ) . In our work , we argue that the relationship between boundary effects and absolute position information extends beyond zero padding and has major implications in a CNN ’ s ability to encode confident and accurate semantic representations ( see Fig . 1 ) . An unexplored area related to boundary effects is the use of canvases ( i.e. , backgrounds ) with image patches ( see Fig . 1 , top row ) . When using image patches in a deep learning pipeline involving CNNs , the user is required to paste the patch onto a background due to the constraint that the image must be rectangular . Canvases have been used in a wide variety of domains , such as image generation ( Gregor et al. , 2015 ; Huang et al. , 2019 ) , data augmentation ( DeVries & Taylor , 2017 ) , image inpainting ( Demir & Unal , 2018 ; Yu et al. , 2018 ) , and interpretable AI ( Geirhos et al. , 2018 ; Esser et al. , 2020 ) . To the best of our knowledge , this paper contains the first analysis done on canvas value selection . In other works , the canvas value is simply chosen based on the authors intuition . Given the pervasiveness of CNNs in a multitude of applications , it is of paramount importance to fully understand what the internal representations are encoding in these networks , as well as isolating the precise reasons that these representations are learned . This comprehension can also allow for the effective design of architectures that overcome recognized shortcomings ( e.g. , residual connections ( He et al. , 2016 ) for the vanishing gradient problem ) . As boundary effects and position information in CNNs are still largely not fully understood , we aim to provide answers to the following hypotheses which reveal fundamental properties of these phenomenon : Hypothesis I : Zero Padding Encodes Maximal Absolute Position Information : Does zero padding encode maximal position information compared to other padding types ? We evaluate the amount of position information in networks trained with different padding types and show zero padding injects more position information than common padding types , e.g. , reflection , replicate , and circular . Hypothesis II : Different Canvas Colors Affect Performance : Do different background values have an effect on performance ? If the padding value at the boundary has a substantial effect on a CNNs performance and position information contained in the network , one should expect that canvas values may also have a similar effect . Hypothesis III : Position information is Correlated with Semantic Information : Does a network ’ s ability to encode absolute position information affect its ability to encode semantic information ? If zero padding and certain canvas colors can affect performance on classification tasks due to the increased position information , we expect that the position information is correlated with a networks ability to encode semantic information . We demonstrate that encoding position information improves the robustness and separability of semantic features . Hypothesis IV : Boundary Effects Occur at All Image Locations : Does a CNN trained without padding suffer in performance solely at the border , or at all image regions ? How does the performance change across image locations ? Our analysis reveals strong evidence that the border effect impacts a CNN ’ s performance at all regions in the input , contrasting previous assumptions ( Tsotsos et al. , 1995 ; Innamorati et al. , 2019 ) that border effects exist solely at the image border . Hypothesis V : Position Encoding Can Act as a Feature or a Bug : Does absolute position information always correlate with improved performance ? A CNN ’ s ability to leverage position information from boundary information could hurt performance when a task requires translation-invariance , e.g. , texture recognition ; however , it can also be useful if the task relies on position information , e.g. , semantic segmentation . To give answers to these hypotheses ( hereon referred to as H-X ) , we design a series of novel tasks as well as use existing techniques to quantify the location information contained in different CNNs with various settings . In particular , we introduce location dependant experiments ( see Fig . 2 ) which use a grid-based strategy to allow for a per-location analysis of absolute position encoding and performance on semantic tasks . The per-location analysis plays a critical role in representing the boundary effects as a function of the distance to the image border . We also estimate the number of dimensions which encode position information in the latent representations of CNNs . Through these experiments we show both quantitative and qualitative evidence that boundary effects have a substantial effect on CNNs in surprising ways and then demonstrate the practical implications of these findings on multiple real-world applications . Code will be made available for all experiments . 2 ABSOLUTE POSITION INFORMATION IN CNNS . What Type of Padding Injects Optimal Location Information ? With the ultimate goal of revealing characteristics that determine the impact that boundary effects plays in CNNs with respect to absolute position information , we first determine which commonly used padding type encodes the maximum amount of absolute position information . We evaluate the ability of different padding types ( i.e. , zero , circular , reflection , and replicate ) to encode absolute position information by extending the experiments from ( Islam et al. , 2020 ) , which only considered zero padding . We first train a simplified VGG network ( Simonyan & Zisserman , 2015 ) with five layers ( VGG-5 , see Appendix A.2 for implementation details ) on Tiny ImageNet ( Le & Yang , 2015 ) for each padding type . We follow the settings in ( Islam et al. , 2020 ) : a read-out module , trained using DUT-S ( Wang et al. , 2017 ) images , takes the features from a frozen VGG-5 model ’ s last layer , pre-trained on Tiny ImageNet , and predicts a gradient-like position map ( see top row in Table . 1 ) . We experiment with two position maps , which are the same for every image : ( i ) ‘ horizontal ’ and ( ii ) ‘ Gaussian ’ . These gradient-like position maps change smoothly from 0 to 1 , from the dark-blue to yellow , respectively . For a fair comparison with ( Islam et al. , 2020 ) , we report results using Spearman Correlation ( SPC ) and Mean Absolute Error ( MAE ) with input images from PASCAL-S ( Li et al. , 2014 ) . From Table 1 , it is clear that zero padding delivers the strongest position information , compared with replicate , boundary reflection , and circular padding , supporting H-I . Note that partial convolution ( Liu et al. , 2018a ) still pads with zeros , but re-weights the output of the convolution based on how many zeros are padded . Thus , position information is still encoded when partial convolutions are used . Interestingly , circular padding is often the second most capable padding type . We conjecture this is because circular padding takes values from the opposite side of the image where the pixel values are typically less correlated than the directly neighbouring pixels . Thus , circular padding often has a value transition at the border , contrasting reflection and replicate which offer little or no signal to the CNN regarding the whereabouts of the image border . 3 LOCATION DEPENDANT TASKS FOR POSITIONAL ANALYSIS . We begin by describing our experimental settings and the implementation details for the proposed location dependant experiments with grid-based inputs . These experiments are used to analyze the border effects with respect to position information encoded in CNNs . These consist of location dependant image classification ( Fig . 2 ( a ) and Sec . 3.1 ) , and segmentation ( Fig . 2 ( b ) and Sec . 3.2 ) , under different canvas color settings . Our experiments are designed with the goal of determining , for different canvas colors ( H-II ) , where in the input CNNs suffer from the border effect ( H-IV ) , and how the encoding of position information affects the learning of semantic features ( H-III ) . Experimental Settings and Implementation Details . Our image classification and segmentation experiments use ‘ location dependant ’ inputs ( see Fig . 2 above and Fig . 9 in the appendix for more detailed examples ) . The input is a colored canvas ( the colors used are Black [ 0 , 0 , 0 ] , White [ 1 , 1 , 1 ] , and the CIFAR-10 dataset ( Krizhevsky et al. , 2014 ) Mean [ 0.491 , 0.482 , 0.446 ] ) with an image patch randomly placed on a k × k grid . Unless mentioned otherwise , we use CIFAR-10 for all experiments . Given a 32× 32 CIFAR-10 training image as the image patch , we randomly choose a grid location , L , and place the CIFAR-10 training sample in that location . For example , in the case of a k × k grid , the size of the grid canvas is 32k × 32k , where each grid location has a size of 32× 32 and k2 total locations ( see Fig . 9 in the appendix ) . All experiments are run for k ∈ { 3 , 5 , 7 , 9 , 11 , 13 , 15 } . To ensure a fair comparison between grid locations , the evaluation protocol consists of running the entire validation set of CIFAR-10 on each individual grid location ( i.e. , we run the validation set k2 times for a single validation epoch ) . We then average the performance over all grid locations to obtain the overall accuracy . We report classification and segmentation accuracy in terms of precision and mean intersection over union ( mIoU ) , respectively . We use a ResNet-18 network trained from scratch , unless stated otherwise . ResNets with no padding are achieved by setting the padding size to zero in the convolution operation . For fair comparison between the padding and no padding baseline , we use bilinear interpolation ( see Appendix A.1 for discussion ) to match spatial resolutions between the residual output and the feature map for the no padding case , which was not accounted for in previous work ( Kayhan & Gemert , 2020 ) .

Summary: This paper studies the effect of padding on the Convolutional Neural Network. The authors try to answer the following questions: 1) what type of padding provides the most position information, 2) does the background value affects model accuracy when processing a patch on a canvas, 3) which part of the image suffers the most from the boundary effect, and 4) whether the position information provided by padding improves or degrades model performance. In order to answer these questions, the authors design multiple tasks and perform extensive experiments. The empirical results show that: 1) zero-padding provides the most location information compared with other common padding methods, 2) the background value of the canvas do affects the accuracy when processing a patch, 3) the boundary effect is not specific to the image boundary---the model is affected by the boundary over the entire image, and 4) the effect of padding on model accuracy depends on the task.