Do Deep Neural Networks for Segmentation Understand Insideness?

1 INTRODUCTION . Image segmentation is necessary for complete image understanding . A key component of image segmentation is to determine whether a pixel is inside or outside a region , ie . the “ insideness ” problem ( Ullman , 1984 ; 1996 ) . Deep Neural Networks ( DNNs ) have been tremendously successful in image segmentation benchmarks , but it is not well understood whether DNNs represent insideness or how . Insideness has been overlooked in DNNs for segmentation since they have been mainly applied to the modality of “ semantic segmentation ” , ie . labelling each pixel with its object category ( Ronneberger et al. , 2015 ; Yu & Koltun , 2016 ; Visin et al. , 2016 ; Badrinarayanan et al. , 2017 ; Chen et al. , 2018b ; Long et al. , 2015 ; Lateef & Ruichek , 2019 ) . In such cases , insideness is not necessary since a solution can rely only on object recognition . Yet , the recent need to solve more sophisticated visual tasks has fueled the development of DNNs with the ability to segment individual object instances , rather than object categories ( Li et al. , 2016 ; 2017 ; Song et al. , 2018 ; Chen et al. , 2018a ; Hu et al. , 2018 ; Maninis et al. , 2018 ; Liu et al. , 2018b ; He et al. , 2017 ) . In these segmentation modalities , insideness plays a central role , especially when there are few cues besides the boundaries of the objects , e.g . when there is lack of texture and color , and objects are unfamiliar . Thus , insideness is necessary to achieve true generalization in image segmentation . In this paper , we investigate derived and learned insideness-related representations in DNNs for segmentation . We take the reductionist approach by isolating insideness from other components in image segmentation . We analyze the segmentation of closed curves , similar to the methodology in Minsky & Papert ’ s historic book Perceptrons ( Minsky & Papert , 1969 ) . In this way , we distill insideness to a minimum representation by eliminating other components . We analytically demonstrate that two state-of-the-art network architectures , namely , DNNs with dilated convolutions ( Yu & Koltun , 2016 ; Chen et al. , 2018b ) and convolutional LSTMs ( ConvLSTMs ) ( Xingjian et al. , 2015 ) , among other networks , can exactly solve the insideness problem for any given curve with network sizes that are easily implemented in practice . The proofs draw on algorithmic ideas from classical work on visual routines ( Ullman , 1984 ; 1996 ) , namely , the rayintersection method and the coloring method , to derive equivalent neural networks that implement these algorithms . Then , in a series of experiments with synthetically generated closed curves , we evaluate the capabilities of these DNNs to learn the insideness problem . The experiments show that when using standard training strategies , the DNNs do not learn general solutions for insideness , even though these DNNs are sufficiently complex to capture the long-range relationships . The only network that achieves almost full generalization in all tested cases is a recurrent network with a training strategy designed to encourage a specific mechanism for dealing with long-range relationships . These results add to the growing body of works that show that DNNs have problems in learning to solve some elemental visual tasks ( Linsley et al. , 2018 ; Liu et al. , 2018a ; Wu et al. , 2018 ; ShalevShwartz et al. , 2017 ) . Shalev-Shwartz et al . ( 2017 ) introduced several tasks that DNNs can in theory solve , as it was shown mathematically , but the networks were unable to learn , not even for the given dataset , due to difficulties in the optimization with gradient descent . In contrast , the challenges we report for insideness are related to poor generalization rather than optimization , as our experiments show the networks succeed in solving insideness for the given dataset . Linsley et al . ( 2018 ) introduced new architectures that better capture the long-range dependencies in images . Here , we show that the training strategy has a big impact in capturing the long-range dependencies . Even if the DNNs we tested had the capacity to capture such long-range dependencies , they do not learn a general solution with the standard training strategies . 2 THE REDUCTIONIST APPROACH TO INSIDENESS . We now introduce the paradigm that will serve to analyze insideness-related representations in DNNs . Rather than using natural images , we use synthetic stimuli that solely contains a closed curve . In this way , we do not mix the insideness problem with other components of image segmentation found in natural images , e.g . self-similarity of segments at the level of object categories or parts , representation of the hierarchy of segments , etc.These components will be studied separately in future works , and finally put together to improve and understand how DNNs segment images . Let X ∈ { 0 , 1 } N×N be an image or a matrix of size N × N pixels . We use Xi , j and ( X ) i , j , indistinguishably , to denote the value of the image in position ( i , j ) . We use this notation for indexing elements in any of the images and matrices that appear in the rest of the paper . Also , in the figures we use white and black to represent 0 and 1 , respectively . Insideness refers to finding which pixels are in the inside and which ones in the outside of a closed curve . We assume without loss of generality that there is only one closed curve in the image and that it is a digital version of a Jordan curve ( Kong , 2001 ) , ie . a closed curve without self-crosses nor self-touches and containing only horizontal and vertical turns , as shown in Fig . 1 . We further assume that the curve does not contain the border of the image . The curve is the set of pixels equal to 1 and is denoted by FX = { ( i , j ) |Xi , j = 1 } . The pixels in X that are not in FX can be classified into two categories : the inside and the outside of the curve ( Kong , 2001 ) . We define the segmentation of X as S ( X ) ∈ { 0 , 1 } N×N , where S ( X ) i , j = { 0 if Xi , j is inside 1 if Xi , j is outside , ( 1 ) and for the pixels in FX , the value of S ( X ) i , j can be either 0 or 1 . Note that unlike object recognition , the definition of insideness is rigorously and uniquely determined by the input image itself . The number of all digital Jordan curves is enormous even if the image size is relatively small , e.g . it is more than 1047 for the size 32×32 ( App . A ) . In addition , insideness is a global problem ; whether a pixel is inside or outside depends on the entire image , and not just on some local area around the pixel . Therefore , simple pattern matching , ie . memorization , is impossible in practice . 3 CAN DNNS FOR SEGMENTATION SOLVE INSIDENESS ? . The universal approximation theorem ( Cybenko , 1989 ) tells us that even a shallow neural network is able to solve the insideness problem . Yet , it could be that the amount of units is too large to be implementable in practice . In this Section , we introduce two DNN architectures that are able to solve the insideness problem at perfection and they are easily implementable in practice . One architecture is feed-forward with dilated convolutions ( Yu & Koltun , 2016 ; Chen et al. , 2018b ) and the other is recurrent : a ConvLSTM ( Xingjian et al. , 2015 ) . 3.1 FEED-FORWARD ARCHITECTURE WITH DILATED CONVOLUTIONS . Dilated convolutions facilitate capturing long-range dependencies which are key for segmentation ( Yu & Koltun , 2016 ; Chen et al. , 2018b ) . To demonstrate that there are architectures with dilated convolutions that can solve the insideness problem , we borrow insights from the ray-intersection method . The ray-intersection method ( Ullman , 1984 ; 1996 ) , also known as the crossings test or the even-odd test ( Haines , 1994 ) , is built on the following fact : Any ray that goes from a pixel to the border of the image alternates between inside and outside regions every time it crosses the curve . Therefore , the parity of the total number of such crossings determines the region to which the pixel belongs . If the parity is odd then the pixel is inside , otherwise it is outside ( see Fig . 1a ) . The definition of a crossing should take into account cases like the one depicted in Fig . 1b , in which the ray intersects the curve , but does not change region after the intersection . To address these cases , we enumerate all possible intersections of a ray and a curve , and analyze which cases should count as crossings and which ones should not . Without loss of generality , we consider only horizontal rays . As we can see in Fig . 1c , there are only five cases for how a horizontal ray can intersect the curve . The three cases at the top of Fig . 1c , are crosses because the ray goes from one region to the opposite one , while the two cases at the bottom ( like in Fig . 1b ) are not considered crosses because the ray remains in the same region . Let ~X ( i , j ) ∈ { 0 , 1 } 1×N be a horizontal ray starting from pixel ( i , j ) , which we define as ~X ( i , j ) = [ Xi , j , Xi , j+1 , Xi , j+2 , . . . , Xi , N , 0 , . . . , 0 ] , ( 2 ) where zeros are padded to the vector if the ray goes outside the image , such that ~X ( i , j ) is always of dimension N . Let ~X ( i , j ) · ~X ( i+1 , j ) be the inner product of the ray starting from ( i , j ) and the ray starting from the pixel below , ( i+1 , j ) . Note that the contribution to this inner product from the three cases at the top of Fig . 1c ( the crossings ) is odd , whereas the contribution from the other two intersections is even . Thus , the parity of ~X ( i , j ) · ~X ( i + 1 , j ) is the same as the parity of the total number of crosses and determines the insideness of the pixel ( i , j ) , ie . S ( X ) i , j = parity ( ~X ( i , j ) · ~X ( i+ 1 , j ) ) . ( 3 ) Dilated convolutions , also called atrous convolutions , are convolutions with upsampled kernels , which enlarge the receptive fields of the units but preserve the number of parameters of the kernel ( Yu & Koltun , 2016 ; Chen et al. , 2018b ) . In App . B we prove that equation 3 can be easily implemented with a neural network with dilated convolutions . The demonstration is based on implementing the dot product in equation 3 with multiple layers of dilated convolutions , as they facilitate capturing the information across the ray . The number of dilated convolutional layers is equal to the logarithm in base-2 of the image size , N . The dot product can also be implemented with two convolutional layers , but with the drawback of using a long kernel of size 1×N . The multiple dilated convolutions use kernels of size 3 × 3 , and they are equivalent to the long kernel of 1 × N . Finally , the parity function in equation 3 is implemented by adapting the network introduced by Shalev-Shwartz et al . ( 2017 ) , which yields a two layer convolutional network with 1× 1 kernels . Note that the proof introduces the smallest network we could find that solves the insideness problem with dilated convolutions . Larger networks than the one we introduced can also solve the insideness problem , as the network size can be reduced by setting kernels to zero and layers to implement the identity operation .

Summary: This submission introduces a new concept, termed insideness, to study semantic segmentation in deep learning era. The authors raise many interesting questions, such as (1) Does deep neural networks (DNN) understand insideness? (2) What representations do DNNs use to address the long-range relationships of insideness? (3) How do architectural choices affect the learning of these representations? This work adopts two popular networks, dilated DNN and ConvLSTM, to implement solutions for insideness problem in isolation. The results can help future research in semantic segmentation for the models to generalize better.