Zero Pixel Directional Boundary by Vector Transform

1 INTRODUCTION . Boundaries are considered to be one of the important interpretable visual cues that can describe both the low-level image characteristics as well as high-level semantics of the image . Human vision uses occluding contours and boundaries to interpret unseen or seen objects and classes . Because boundaries are an important visual cue for image understanding , they are also exploited as priors in several vision tasks ( Zhu et al. , 2020 ; Kim et al. , 2021 ; Hatamizadeh et al. , 2019 ; Revaud et al. , 2015 ; Cashman & Fitzgibbon , 2012 ) . Some key works on contours ( Cootes et al. , 2001 ; Matthews & Baker , 2004 ; Kass et al. , 1988 ) have greatly impacted early research in computer vision . Although the advent of deep learning and end-to-end learning has somewhat shifted the focus away from interpretable visual cues , boundary discovery still remains an important task in computer vision . Supervised deep learning has greatly transformed problems such as object detection and segmentation ( Redmon et al. , 2016 ; Chen et al. , 2017 ; Cheng et al. , 2020 ) by redefining the problem ( Kirillov et al. , 2019 ) , using high-quality datasets ( Cordts et al. , 2016 ; Neuhold et al. , 2017 ) and better network architectures ( Cheng et al. , 2020 ; 2021 ; Wang et al. , 2021b ) . Boundary detection , however , has seen a rather modest share of such progress . Although , modern deeply learned methods ( Xie & Tu , 2015 ; Liu et al. , 2017 ; Maninis et al. , 2017 ) provide better accuracy and the possibility to learn only the high-level boundaries , a particularly elusive goal in learned boundary detection has been the so-called crisp boundaries ( Isola et al. , 2014 ; Wang et al. , 2018 ; Deng et al. , 2018 ) . The formulation of boundary detection as a binary segmentation task naturally introduces class imbalance , which makes detecting pixel thin sharp boundaries extremely difficult . Arguably a majority of recent methods in boundary detection are proposed in order to tackle the aforementioned issue . Many methods address the lack of ‘ crispness ’ by fusing high-resolution features with the middleand high-level features ( Xie & Tu , 2015 ; Liu et al. , 2017 ) . Such a strategy has been successful in other dense prediction tasks ( Ronneberger et al. , 2015 ) as well . Others propose different loss functions ( Kokkinos , 2016 ; Deng et al. , 2018 ; Kervadec et al. , 2019 ) to address class imbalance . Despite the improvements , we identify two issues regarding crisp boundary detection . The first is that the evaluation protocol ( Martin et al. , 2004 ) does not necessarily encourage crisp detection , as the quantification is done after Non-Maximal Suppression ( NMS ) . Such an evaluation may be misleading when the network outputs need to be used at training time for other high-level tasks , e.g. , segmentation ( Kim et al. , 2021 ) . Second , the current losses ( Kokkinos , 2016 ; Xie & Tu , 2015 ; Ma et al. , 2021 ) push for edges as crisp as the ground-truth rather than as crisp as possible . This is particularly harmful since many boundary detection datasets ( Arbelaez et al. , 2010 ; Silberman et al. , 2012 ) contain ambiguous boundaries or inconsistently thick boundaries . In this paper , we take a different perspective on boundary detection . Boundaries are formed where visual features change , popularly referred to as the differential representation ( Boykov et al. , 2006 ; Kervadec et al. , 2019 ) . Such an assumption treats the boundaries as a set of 1-D surfaces embedded in a continuous 2D space , implying they do not have any thickness . Many previous energyminimization based approaches ( Chan & Zhu , 2005 ; Chan & Vese , 2001 ; Paragios et al. , 2002 ; Boykov et al. , 2006 ; Ma & Manjunath , 2000 ) and a few current methods ( Kervadec et al. , 2019 ) tackle boundaries in a similar way . Level-set methods ( Chan & Zhu , 2005 ; Boykov et al. , 2006 ; Ma & Manjunath , 2000 ) consider boundaries as the level-set of a continuous function of the image . Specifically , ( Ma & Manjunath , 2000 ) defines the energy function related to the distance and direction of the boundary at each pixel and extracts the directional normals at the boundary using such an energy . Inspired by such works and also recent works on 3D implicit functions ( Tancik et al. , 2020 ; Sitzmann et al. , 2020 ; Mildenhall et al. , 2020 ) , we represent boundaries via a field of unit vectors defined at each pixel , pointing towards the closest boundary surface . The proposed vector field representation naturally solves class imbalance . In distance transforms , vector fields are considered incomplete euclidean transforms ( Osher & Sethian , 1988 ) , equal to the Jacobian of the signed distance field . The vector field we use is in fact the Jacobian of the positive distance field . In contrast to distance fields , it provides high sensitivity at the boundaries and is easily localizable . We demonstrate the equivalence of the normal field to the surface contour representation using the level set of the normal field ’ s divergence , providing infinitely sharp boundaries . Owing to the zero-thickness , we refer to our result as the zero-pixel boundary . Our method is virtually hyper-parameter free at training and test time , and can provide zero-pixel thick boundaries at training time . In order to evaluate the boundaries using the surface interpretation , we also advocate the use of surface distances including the average symmetric surface distance ( assd ) metric that is less prone to class imbalance and variable boundary thickness in the ground-truth ( Kervadec et al. , 2019 ) . Such metrics are very popular in biomedical image segmentation ( Yeghiazaryan & Voiculescu , 2018 ) . We show significant improvements in all metrics using our boundary representation when compared to the various combinations of Dice ( Dice , 1945 ) and cross-entropy losses in several large datasets . 2 RELATED WORK . There is a rich history of boundary detection in computer vision . Previous work on boundaries showed a diversity of definitions and approaches . We differentiate them based on two different interpretations of boundaries : i.e. , i ) a boundary is a separation between two or more image regions with different visual features and ii ) a boundary is a thin group of pixels belonging to a specific class . It should be noted that most modern methods fall under the second category . Perhaps the most notable representatives of the first strand are the energy-based segmentation methods ( Chan & Vese , 2001 ; Comaniciu & Meer , 2002 ; Boykov et al. , 2006 ; Grady , 2006 ; Ma & Manjunath , 2000 ) . These methods relied on hand-crafted features and various optimization strategies to compute the low-level boundaries . In particular , Ma & Manjunath ( 2000 ) compute the normal vectors of the low-level boundaries from an energy function , without looking into an equivalent learnable representation . Graph-based segmentation methods ( Shi & Malik , 2000 ; Felzenszwalb & Huttenlocher , 2004 ; Cheng et al. , 2016 ) construct a graph from the image and cut the graph to obtain non-overlapping regions whose boundaries are viewed as image boundaries . A few deep learning methods followed a similar approach ( Wang et al. , 2021a ; Kervadec et al. , 2019 ) . Despite the advantage of the definition , current representations in this category are hard to adapt to generic boundaries and a compact multi-boundary representation with good performance remains lacking . A larger corpus of work utilizes pixel-wise image features to decide whether pixels belong to a ‘ boundary ’ class . They form our category ii ) methods . Early methods utilize various filter operators to detect discontinuities in image intensities or colors ( Canny , 1986 ; Sobel , 1972 ) . Learning based methods substantially boost the edge detection performance by classifying handcrafted features ( Konishi et al. , 2003 ; Martin et al. , 2004 ; Arbelaez et al. , 2010 ; Dollár & Zitnick , 2013 ; Hallman & Fowlkes , 2015 ) . Modern deep neural network ( DNN ) methods have further improved this field by learning powerful feature representations , particularly high-level semantic information ( Shen et al. , 2015 ; Bertasius et al. , 2015 ; Xie & Tu , 2015 ; Liu et al. , 2017 ; Wang et al. , 2018 ; Deng et al. , 2018 ; He et al. , 2019 ) . Yang et al . ( 2016 ) leveraged the powerful deep features to detect only object boundaries . Others try to simultaneously detect edges and predict the semantic class of each edge point , so-called semantic edge detection ( Hariharan et al. , 2011 ; Yu et al. , 2017 ; Liu et al. , 2018 ; Yu et al. , 2018 ) . On the other hand , classifying pixels as boundary class introduces class imbalance during training . A common counter-strategy is to use a weighted cross-entropy loss giving the non-boundary pixels a small weight and the boundary class a large weight ( Xie & Tu , 2015 ; Liu et al. , 2017 ; He et al. , 2019 ) . Yet , despite an improvement over regular cross-entropy , it does not solve the problem . To thin the boundaries , Non-Maximal Suppression ( NMS ) is usually adopted . Such methods may be harmful when directly integrated with higher-level tasks such as segmentation ( Kim et al. , 2021 ) . The Dice loss ( Dice , 1945 ) was thus advocated to generate crisp boundaries before NMS ( Deng et al. , 2018 ) , but it still produces several pixel thick boundaries and suffers more from missed predictions . Variations of the Dice loss ( Shit et al. , 2021 ) have been proposed to counter the missed detections . However , the right approach still depends on the downstream tasks ( Zhu et al. , 2020 ; Kim et al. , 2021 ; Hatamizadeh et al. , 2019 ; Shit et al. , 2021 ) and in either case a careful selection of training as well as testing hyper-parameters is required . We provide an alternative approach , motivated by the class imbalance , while having no sensitive hyper-parameter . 3 BOUNDARY TRANSFORM AND REPRESENTATION . In this section , we first discuss the surface representation of boundaries as a normal vector field transform and prove its relevant properties . Our boundary representation is inspired by recent work on implicit neural 3D surface representations ( Park et al. , 2019 ; Mescheder et al. , 2019 ) , energybased methods on edge detection ( Ma & Manjunath , 2000 ; Boykov et al. , 2006 ) and distance transforms ( Osher & Sethian , 1988 ) . In 3D surface representations , a Signed Distance Function ( SDF ) ( Osher & Sethian , 1988 ) or occupancy map ( Mescheder et al. , 2019 ) is used as representation . We instead propose a unit vector field from every point to the closest boundary . This choice is motivated by the high sensitivity and richer boundary context provided by the unit vector field , as shown in our experimental results in § 5 . Fig . 1 shows our vector field representation of the ground-truth , with predictions using a standard quiver plot on a sub-sampled set of points . We assume a continuous boundary image domain Ω ⊂ R2 with the set of boundary points { x′ } = Π ⊂ Ω . Each point is denoted as a 2-vector x = ( x , y ) ∈ R2 . In order to encode boundary properties on the whole image , we compute a signed x and y distance field separately and finally encode only the direction . The result is a unit vector field that represents the boundary . We can express our boundary representation by the following transform for any point x ∈ Ω : fx ( x ) = inf x′∈Π dx ( x , x ′ ) , fy ( x ) = inf x′∈Π dy ( x , x ′ ) , v ( x ) = f ( x ) ‖f ( x ) ‖2 , if ‖f ( x ) ‖2 6= 0 , otherwise n , n = lim dx→0+ f ( x ) ‖f ( x ) ‖2 . ( 1 ) Equation ( 1 ) defines the transform as a function v ( x ) : Ω→ R2 going from the boundary Π to a field representation . Here , dx and dy are operators that give the signed x and y difference , respectively . Similarly , fx and fy are the x and y components of the field f. Note that we choose the field vector arbitrarily among the two possible values at the boundary using the positive dx distance . We note the following properties of the vector field v. Property 3.1 The vector field v ( x ) is equal to the unit normal field at the boundary . Proof This is a well known result ( Osher & Fedkiw , 2003 ) and can be proved easily ( see equation ( 2.4 ) in the reference ) . The fact that we forcefully choose one normal over its negative directional normal at the boundary points does not affect the statement . Property 3.2 Given a vector field representation v ( x ) of a boundary , one can obtain the binary boundary representation by considering the following transform : g ( x ) = div v ( x ) . ( 2 ) The original boundary set Π can then be found by taking the zero level set of g ( x ) + 2 , i.e. , Π = L0 ( g + 2 ) . ( 3 ) Proof In the infinitesimal neighborhood of the boundary points , using property 3.1 , the vector field is normal to the boundary , provided that good approximate normals can be obtained from equation ( 1 ) . As the neighborhood size approaches zero , the tangential vector components approach zero around a point for a continuous boundary segment . Thus , around such an infinitesimal neighborhood , the normal fields pointing in opposite direction will subtract perfectly , creating a divergence flow of -2 and around 0 or positive away from boundaries . Strictly speaking the result holds only for piece-wise smooth surfaces ( Osher & Fedkiw , 2003 ) , with lower than -2 divergence possible at discontinuous surface points . Property 3.3 The relation is one-to-one between the binary boundary representation and the proposed vector field representation in a continuous domain . Proof This property is the result of equation ( 1 ) , for the forward transform and equation ( 3 ) for the inverse transform , providing a one-to-one relation . Note that the vector field transform as defined in equation ( 1 ) has to correct for two different kinds of indeterminate states . The first is on the boundary , that is solved by using the right hand limit so that one of the two opposite directions is chosen consistently . The second is when the infimum operation in equation ( 1 ) produces two or more closest points , corrected by choosing any one of the points for the infimum . The vector fields around such points flip directions creating a positive divergence as shown in Fig . 2 . More discussions are provided in §5 about the latter , which are in fact helpful for deciding superpixel centers . The above properties and their proofs are crucial for the validity of the proposed boundary representation and also to go from one representation to another for inference and visualization . Vector Transform and the Distance Transform . In essence , the normalized vector field proposed in equation ( 1 ) is another representation of the distance transform . Let φ ( x ) ∈ R+ define the distance transform , then the vector field v ( x ) in equation ( 1 ) can be obtained by the following partial derivatives ( Osher & Sethian , 1988 ; Osher & Fedkiw , 2003 ) : v ( x ) = −∇φ ( x ) . ( 4 ) One can optimize a given network by minimizing the loss on the distance transform ( DT ) or SDF ( Dapogny & Frey , 2012 ; Caliva et al. , 2019 ; Park et al. , 2019 ) instead of using the loss on the normalized vector field . Compared to the binary mask , the Vector transform ( VT ) , DT and SDF have an added advantage that they are sensitive to small topological changes . SDF on the other hand , does not support overlapping and open surfaces and is not easily adaptable to the image boundary problem . However , there are several reasons which make DT unsuitable for learning boundaries . During training , when the distance field is close to 0 i.e. , around the boundaries , any loss on the DT or SDF loses importance . Apart from the convergence problems of DT with gradient descent Osher & Fedkiw ( 2003 ) , DT is also hard to localize by thresholding under noise compared to the SDF and VT . In SDF , localizing the surface amounts to finding the zero crossings and in VT , the divergence measure in equation ( 2 ) provides an extremely sharp contrast making equation ( 3 ) trivial to solve . These differences in the thresholding problem can be seen in Fig . 5 in Appendix B and the experimental results . Additionally , despite reducing the class imbalance compared to binary boundary prediction , DT has an implicit bias to the weighted average of its typical range . On the other hand , a normalized vector field from VT equation ( 1 ) is sensitive to the topology similar to a distance field while also being localizable and sensitive at the boundaries , as shown in Fig . 1 .

Summary: The paper proposes a method to detect boundaries in an image by interpreting boundaries as 1-D surfaces and formulating a one-to-one vector transformation function. The paper provides theoretical justification of the vector transformation representation. Experimental results show the effectiveness on publicly available datasets.