Boundary Effects in CNNs: Feature or Bug?

1 INTRODUCTION . One of the main intuitions behind the success of CNNs for visual tasks such as image classification ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2015 ; Szegedy et al. , 2015 ; Huang et al. , 2017 ) , video classification ( Karpathy et al. , 2014 ; Yue-Hei Ng et al. , 2015 ; Carreira & Zisserman , 2017 ) , object detection ( Ren et al. , 2015 ; Redmon et al. , 2016 ; He et al. , 2017 ) , generative image models ( Brock et al. , 2018 ) , and semantic segmentation ( Long et al. , 2015 ; Noh et al. , 2015 ; Chen et al. , 2017 ; 2018 ) , is that convolutions add a visual inductive bias to neural networks that objects can appear anywhere in the image . To accommodate the finite domain of images , manual heuristics ( e.g. , padding ) have been applied to allow the convolutional kernel ’ s support to extend beyond the border of an image and reduce the impact of the boundary effects ( Wohlberg & Rodriguez , 2017 ; Tang et al. , 2018 ; Liu et al. , 2018a ; Innamorati et al. , 2019 ; Liu et al. , 2018b ) . Recent studies ( Pérez et al. , 2019 ; Islam et al. , 2020 ; Kayhan & Gemert , 2020 ) have shown that zero padding allows CNNs to encode absolute position information despite the presence of pooling layers in their architecture ( e.g. , global average pooling ) . In our work , we argue that the relationship between boundary effects and absolute position information extends beyond zero padding and has major implications in a CNN ’ s ability to encode confident and accurate semantic representations ( see Fig . 1 ) . An unexplored area related to boundary effects is the use of canvases ( i.e. , backgrounds ) with image patches ( see Fig . 1 , top row ) . When using image patches in a deep learning pipeline involving CNNs , the user is required to paste the patch onto a background due to the constraint that the image must be rectangular . Canvases have been used in a wide variety of domains , such as image generation ( Gregor et al. , 2015 ; Huang et al. , 2019 ) , data augmentation ( DeVries & Taylor , 2017 ) , image inpainting ( Demir & Unal , 2018 ; Yu et al. , 2018 ) , and interpretable AI ( Geirhos et al. , 2018 ; Esser et al. , 2020 ) . To the best of our knowledge , this paper contains the first analysis done on canvas value selection . In other works , the canvas value is simply chosen based on the authors intuition . Given the pervasiveness of CNNs in a multitude of applications , it is of paramount importance to fully understand what the internal representations are encoding in these networks , as well as isolating the precise reasons that these representations are learned . This comprehension can also allow for the effective design of architectures that overcome recognized shortcomings ( e.g. , residual connections ( He et al. , 2016 ) for the vanishing gradient problem ) . As boundary effects and position information in CNNs are still largely not fully understood , we aim to provide answers to the following hypotheses which reveal fundamental properties of these phenomenon : Hypothesis I : Zero Padding Encodes Maximal Absolute Position Information : Does zero padding encode maximal position information compared to other padding types ? We evaluate the amount of position information in networks trained with different padding types and show zero padding injects more position information than common padding types , e.g. , reflection , replicate , and circular . Hypothesis II : Different Canvas Colors Affect Performance : Do different background values have an effect on performance ? If the padding value at the boundary has a substantial effect on a CNNs performance and position information contained in the network , one should expect that canvas values may also have a similar effect . Hypothesis III : Position information is Correlated with Semantic Information : Does a network ’ s ability to encode absolute position information affect its ability to encode semantic information ? If zero padding and certain canvas colors can affect performance on classification tasks due to the increased position information , we expect that the position information is correlated with a networks ability to encode semantic information . We demonstrate that encoding position information improves the robustness and separability of semantic features . Hypothesis IV : Boundary Effects Occur at All Image Locations : Does a CNN trained without padding suffer in performance solely at the border , or at all image regions ? How does the performance change across image locations ? Our analysis reveals strong evidence that the border effect impacts a CNN ’ s performance at all regions in the input , contrasting previous assumptions ( Tsotsos et al. , 1995 ; Innamorati et al. , 2019 ) that border effects exist solely at the image border . Hypothesis V : Position Encoding Can Act as a Feature or a Bug : Does absolute position information always correlate with improved performance ? A CNN ’ s ability to leverage position information from boundary information could hurt performance when a task requires translation-invariance , e.g. , texture recognition ; however , it can also be useful if the task relies on position information , e.g. , semantic segmentation . To give answers to these hypotheses ( hereon referred to as H-X ) , we design a series of novel tasks as well as use existing techniques to quantify the location information contained in different CNNs with various settings . In particular , we introduce location dependant experiments ( see Fig . 2 ) which use a grid-based strategy to allow for a per-location analysis of absolute position encoding and performance on semantic tasks . The per-location analysis plays a critical role in representing the boundary effects as a function of the distance to the image border . We also estimate the number of dimensions which encode position information in the latent representations of CNNs . Through these experiments we show both quantitative and qualitative evidence that boundary effects have a substantial effect on CNNs in surprising ways and then demonstrate the practical implications of these findings on multiple real-world applications . Code will be made available for all experiments . 2 ABSOLUTE POSITION INFORMATION IN CNNS . What Type of Padding Injects Optimal Location Information ? With the ultimate goal of revealing characteristics that determine the impact that boundary effects plays in CNNs with respect to absolute position information , we first determine which commonly used padding type encodes the maximum amount of absolute position information . We evaluate the ability of different padding types ( i.e. , zero , circular , reflection , and replicate ) to encode absolute position information by extending the experiments from ( Islam et al. , 2020 ) , which only considered zero padding . We first train a simplified VGG network ( Simonyan & Zisserman , 2015 ) with five layers ( VGG-5 , see Appendix A.2 for implementation details ) on Tiny ImageNet ( Le & Yang , 2015 ) for each padding type . We follow the settings in ( Islam et al. , 2020 ) : a read-out module , trained using DUT-S ( Wang et al. , 2017 ) images , takes the features from a frozen VGG-5 model ’ s last layer , pre-trained on Tiny ImageNet , and predicts a gradient-like position map ( see top row in Table . 1 ) . We experiment with two position maps , which are the same for every image : ( i ) ‘ horizontal ’ and ( ii ) ‘ Gaussian ’ . These gradient-like position maps change smoothly from 0 to 1 , from the dark-blue to yellow , respectively . For a fair comparison with ( Islam et al. , 2020 ) , we report results using Spearman Correlation ( SPC ) and Mean Absolute Error ( MAE ) with input images from PASCAL-S ( Li et al. , 2014 ) . From Table 1 , it is clear that zero padding delivers the strongest position information , compared with replicate , boundary reflection , and circular padding , supporting H-I . Note that partial convolution ( Liu et al. , 2018a ) still pads with zeros , but re-weights the output of the convolution based on how many zeros are padded . Thus , position information is still encoded when partial convolutions are used . Interestingly , circular padding is often the second most capable padding type . We conjecture this is because circular padding takes values from the opposite side of the image where the pixel values are typically less correlated than the directly neighbouring pixels . Thus , circular padding often has a value transition at the border , contrasting reflection and replicate which offer little or no signal to the CNN regarding the whereabouts of the image border . 3 LOCATION DEPENDANT TASKS FOR POSITIONAL ANALYSIS . We begin by describing our experimental settings and the implementation details for the proposed location dependant experiments with grid-based inputs . These experiments are used to analyze the border effects with respect to position information encoded in CNNs . These consist of location dependant image classification ( Fig . 2 ( a ) and Sec . 3.1 ) , and segmentation ( Fig . 2 ( b ) and Sec . 3.2 ) , under different canvas color settings . Our experiments are designed with the goal of determining , for different canvas colors ( H-II ) , where in the input CNNs suffer from the border effect ( H-IV ) , and how the encoding of position information affects the learning of semantic features ( H-III ) . Experimental Settings and Implementation Details . Our image classification and segmentation experiments use ‘ location dependant ’ inputs ( see Fig . 2 above and Fig . 9 in the appendix for more detailed examples ) . The input is a colored canvas ( the colors used are Black [ 0 , 0 , 0 ] , White [ 1 , 1 , 1 ] , and the CIFAR-10 dataset ( Krizhevsky et al. , 2014 ) Mean [ 0.491 , 0.482 , 0.446 ] ) with an image patch randomly placed on a k × k grid . Unless mentioned otherwise , we use CIFAR-10 for all experiments . Given a 32× 32 CIFAR-10 training image as the image patch , we randomly choose a grid location , L , and place the CIFAR-10 training sample in that location . For example , in the case of a k × k grid , the size of the grid canvas is 32k × 32k , where each grid location has a size of 32× 32 and k2 total locations ( see Fig . 9 in the appendix ) . All experiments are run for k ∈ { 3 , 5 , 7 , 9 , 11 , 13 , 15 } . To ensure a fair comparison between grid locations , the evaluation protocol consists of running the entire validation set of CIFAR-10 on each individual grid location ( i.e. , we run the validation set k2 times for a single validation epoch ) . We then average the performance over all grid locations to obtain the overall accuracy . We report classification and segmentation accuracy in terms of precision and mean intersection over union ( mIoU ) , respectively . We use a ResNet-18 network trained from scratch , unless stated otherwise . ResNets with no padding are achieved by setting the padding size to zero in the convolution operation . For fair comparison between the padding and no padding baseline , we use bilinear interpolation ( see Appendix A.1 for discussion ) to match spatial resolutions between the residual output and the feature map for the no padding case , which was not accounted for in previous work ( Kayhan & Gemert , 2020 ) .

Summary: The paper seeks to understand how different padding modes and canvas colors affect the performance of a convolutional neural network in classification and semantic segmentation tasks. The question seems somewhat strange - surely a network should be able to counteract a consistent change in padding or background color. If there was a strong effect it would be an interesting finding indeed. Unfortunately, the paper fails to convince that any but the most obvious effects exist.