Zero Pixel Directional Boundary by Vector Transform

1 INTRODUCTION . Boundaries are considered to be one of the important interpretable visual cues that can describe both the low-level image characteristics as well as high-level semantics of the image . Human vision uses occluding contours and boundaries to interpret unseen or seen objects and classes . Because boundaries are an important visual cue for image understanding , they are also exploited as priors in several vision tasks ( Zhu et al. , 2020 ; Kim et al. , 2021 ; Hatamizadeh et al. , 2019 ; Revaud et al. , 2015 ; Cashman & Fitzgibbon , 2012 ) . Some key works on contours ( Cootes et al. , 2001 ; Matthews & Baker , 2004 ; Kass et al. , 1988 ) have greatly impacted early research in computer vision . Although the advent of deep learning and end-to-end learning has somewhat shifted the focus away from interpretable visual cues , boundary discovery still remains an important task in computer vision . Supervised deep learning has greatly transformed problems such as object detection and segmentation ( Redmon et al. , 2016 ; Chen et al. , 2017 ; Cheng et al. , 2020 ) by redefining the problem ( Kirillov et al. , 2019 ) , using high-quality datasets ( Cordts et al. , 2016 ; Neuhold et al. , 2017 ) and better network architectures ( Cheng et al. , 2020 ; 2021 ; Wang et al. , 2021b ) . Boundary detection , however , has seen a rather modest share of such progress . Although , modern deeply learned methods ( Xie & Tu , 2015 ; Liu et al. , 2017 ; Maninis et al. , 2017 ) provide better accuracy and the possibility to learn only the high-level boundaries , a particularly elusive goal in learned boundary detection has been the so-called crisp boundaries ( Isola et al. , 2014 ; Wang et al. , 2018 ; Deng et al. , 2018 ) . The formulation of boundary detection as a binary segmentation task naturally introduces class imbalance , which makes detecting pixel thin sharp boundaries extremely difficult . Arguably a majority of recent methods in boundary detection are proposed in order to tackle the aforementioned issue . Many methods address the lack of ‘ crispness ’ by fusing high-resolution features with the middleand high-level features ( Xie & Tu , 2015 ; Liu et al. , 2017 ) . Such a strategy has been successful in other dense prediction tasks ( Ronneberger et al. , 2015 ) as well . Others propose different loss functions ( Kokkinos , 2016 ; Deng et al. , 2018 ; Kervadec et al. , 2019 ) to address class imbalance . Despite the improvements , we identify two issues regarding crisp boundary detection . The first is that the evaluation protocol ( Martin et al. , 2004 ) does not necessarily encourage crisp detection , as the quantification is done after Non-Maximal Suppression ( NMS ) . Such an evaluation may be misleading when the network outputs need to be used at training time for other high-level tasks , e.g. , segmentation ( Kim et al. , 2021 ) . Second , the current losses ( Kokkinos , 2016 ; Xie & Tu , 2015 ; Ma et al. , 2021 ) push for edges as crisp as the ground-truth rather than as crisp as possible . This is particularly harmful since many boundary detection datasets ( Arbelaez et al. , 2010 ; Silberman et al. , 2012 ) contain ambiguous boundaries or inconsistently thick boundaries . In this paper , we take a different perspective on boundary detection . Boundaries are formed where visual features change , popularly referred to as the differential representation ( Boykov et al. , 2006 ; Kervadec et al. , 2019 ) . Such an assumption treats the boundaries as a set of 1-D surfaces embedded in a continuous 2D space , implying they do not have any thickness . Many previous energyminimization based approaches ( Chan & Zhu , 2005 ; Chan & Vese , 2001 ; Paragios et al. , 2002 ; Boykov et al. , 2006 ; Ma & Manjunath , 2000 ) and a few current methods ( Kervadec et al. , 2019 ) tackle boundaries in a similar way . Level-set methods ( Chan & Zhu , 2005 ; Boykov et al. , 2006 ; Ma & Manjunath , 2000 ) consider boundaries as the level-set of a continuous function of the image . Specifically , ( Ma & Manjunath , 2000 ) defines the energy function related to the distance and direction of the boundary at each pixel and extracts the directional normals at the boundary using such an energy . Inspired by such works and also recent works on 3D implicit functions ( Tancik et al. , 2020 ; Sitzmann et al. , 2020 ; Mildenhall et al. , 2020 ) , we represent boundaries via a field of unit vectors defined at each pixel , pointing towards the closest boundary surface . The proposed vector field representation naturally solves class imbalance . In distance transforms , vector fields are considered incomplete euclidean transforms ( Osher & Sethian , 1988 ) , equal to the Jacobian of the signed distance field . The vector field we use is in fact the Jacobian of the positive distance field . In contrast to distance fields , it provides high sensitivity at the boundaries and is easily localizable . We demonstrate the equivalence of the normal field to the surface contour representation using the level set of the normal field ’ s divergence , providing infinitely sharp boundaries . Owing to the zero-thickness , we refer to our result as the zero-pixel boundary . Our method is virtually hyper-parameter free at training and test time , and can provide zero-pixel thick boundaries at training time . In order to evaluate the boundaries using the surface interpretation , we also advocate the use of surface distances including the average symmetric surface distance ( assd ) metric that is less prone to class imbalance and variable boundary thickness in the ground-truth ( Kervadec et al. , 2019 ) . Such metrics are very popular in biomedical image segmentation ( Yeghiazaryan & Voiculescu , 2018 ) . We show significant improvements in all metrics using our boundary representation when compared to the various combinations of Dice ( Dice , 1945 ) and cross-entropy losses in several large datasets . 2 RELATED WORK . There is a rich history of boundary detection in computer vision . Previous work on boundaries showed a diversity of definitions and approaches . We differentiate them based on two different interpretations of boundaries : i.e. , i ) a boundary is a separation between two or more image regions with different visual features and ii ) a boundary is a thin group of pixels belonging to a specific class . It should be noted that most modern methods fall under the second category . Perhaps the most notable representatives of the first strand are the energy-based segmentation methods ( Chan & Vese , 2001 ; Comaniciu & Meer , 2002 ; Boykov et al. , 2006 ; Grady , 2006 ; Ma & Manjunath , 2000 ) . These methods relied on hand-crafted features and various optimization strategies to compute the low-level boundaries . In particular , Ma & Manjunath ( 2000 ) compute the normal vectors of the low-level boundaries from an energy function , without looking into an equivalent learnable representation . Graph-based segmentation methods ( Shi & Malik , 2000 ; Felzenszwalb & Huttenlocher , 2004 ; Cheng et al. , 2016 ) construct a graph from the image and cut the graph to obtain non-overlapping regions whose boundaries are viewed as image boundaries . A few deep learning methods followed a similar approach ( Wang et al. , 2021a ; Kervadec et al. , 2019 ) . Despite the advantage of the definition , current representations in this category are hard to adapt to generic boundaries and a compact multi-boundary representation with good performance remains lacking . A larger corpus of work utilizes pixel-wise image features to decide whether pixels belong to a ‘ boundary ’ class . They form our category ii ) methods . Early methods utilize various filter operators to detect discontinuities in image intensities or colors ( Canny , 1986 ; Sobel , 1972 ) . Learning based methods substantially boost the edge detection performance by classifying handcrafted features ( Konishi et al. , 2003 ; Martin et al. , 2004 ; Arbelaez et al. , 2010 ; Dollár & Zitnick , 2013 ; Hallman & Fowlkes , 2015 ) . Modern deep neural network ( DNN ) methods have further improved this field by learning powerful feature representations , particularly high-level semantic information ( Shen et al. , 2015 ; Bertasius et al. , 2015 ; Xie & Tu , 2015 ; Liu et al. , 2017 ; Wang et al. , 2018 ; Deng et al. , 2018 ; He et al. , 2019 ) . Yang et al . ( 2016 ) leveraged the powerful deep features to detect only object boundaries . Others try to simultaneously detect edges and predict the semantic class of each edge point , so-called semantic edge detection ( Hariharan et al. , 2011 ; Yu et al. , 2017 ; Liu et al. , 2018 ; Yu et al. , 2018 ) . On the other hand , classifying pixels as boundary class introduces class imbalance during training . A common counter-strategy is to use a weighted cross-entropy loss giving the non-boundary pixels a small weight and the boundary class a large weight ( Xie & Tu , 2015 ; Liu et al. , 2017 ; He et al. , 2019 ) . Yet , despite an improvement over regular cross-entropy , it does not solve the problem . To thin the boundaries , Non-Maximal Suppression ( NMS ) is usually adopted . Such methods may be harmful when directly integrated with higher-level tasks such as segmentation ( Kim et al. , 2021 ) . The Dice loss ( Dice , 1945 ) was thus advocated to generate crisp boundaries before NMS ( Deng et al. , 2018 ) , but it still produces several pixel thick boundaries and suffers more from missed predictions . Variations of the Dice loss ( Shit et al. , 2021 ) have been proposed to counter the missed detections . However , the right approach still depends on the downstream tasks ( Zhu et al. , 2020 ; Kim et al. , 2021 ; Hatamizadeh et al. , 2019 ; Shit et al. , 2021 ) and in either case a careful selection of training as well as testing hyper-parameters is required . We provide an alternative approach , motivated by the class imbalance , while having no sensitive hyper-parameter . 3 BOUNDARY TRANSFORM AND REPRESENTATION . In this section , we first discuss the surface representation of boundaries as a normal vector field transform and prove its relevant properties . Our boundary representation is inspired by recent work on implicit neural 3D surface representations ( Park et al. , 2019 ; Mescheder et al. , 2019 ) , energybased methods on edge detection ( Ma & Manjunath , 2000 ; Boykov et al. , 2006 ) and distance transforms ( Osher & Sethian , 1988 ) . In 3D surface representations , a Signed Distance Function ( SDF ) ( Osher & Sethian , 1988 ) or occupancy map ( Mescheder et al. , 2019 ) is used as representation . We instead propose a unit vector field from every point to the closest boundary . This choice is motivated by the high sensitivity and richer boundary context provided by the unit vector field , as shown in our experimental results in § 5 . Fig . 1 shows our vector field representation of the ground-truth , with predictions using a standard quiver plot on a sub-sampled set of points . We assume a continuous boundary image domain Ω ⊂ R2 with the set of boundary points { x′ } = Π ⊂ Ω . Each point is denoted as a 2-vector x = ( x , y ) ∈ R2 . In order to encode boundary properties on the whole image , we compute a signed x and y distance field separately and finally encode only the direction . The result is a unit vector field that represents the boundary . We can express our boundary representation by the following transform for any point x ∈ Ω : fx ( x ) = inf x′∈Π dx ( x , x ′ ) , fy ( x ) = inf x′∈Π dy ( x , x ′ ) , v ( x ) = f ( x ) ‖f ( x ) ‖2 , if ‖f ( x ) ‖2 6= 0 , otherwise n , n = lim dx→0+ f ( x ) ‖f ( x ) ‖2 . ( 1 ) Equation ( 1 ) defines the transform as a function v ( x ) : Ω→ R2 going from the boundary Π to a field representation . Here , dx and dy are operators that give the signed x and y difference , respectively . Similarly , fx and fy are the x and y components of the field f. Note that we choose the field vector arbitrarily among the two possible values at the boundary using the positive dx distance . We note the following properties of the vector field v. Property 3.1 The vector field v ( x ) is equal to the unit normal field at the boundary . Proof This is a well known result ( Osher & Fedkiw , 2003 ) and can be proved easily ( see equation ( 2.4 ) in the reference ) . The fact that we forcefully choose one normal over its negative directional normal at the boundary points does not affect the statement . Property 3.2 Given a vector field representation v ( x ) of a boundary , one can obtain the binary boundary representation by considering the following transform : g ( x ) = div v ( x ) . ( 2 ) The original boundary set Π can then be found by taking the zero level set of g ( x ) + 2 , i.e. , Π = L0 ( g + 2 ) . ( 3 ) Proof In the infinitesimal neighborhood of the boundary points , using property 3.1 , the vector field is normal to the boundary , provided that good approximate normals can be obtained from equation ( 1 ) . As the neighborhood size approaches zero , the tangential vector components approach zero around a point for a continuous boundary segment . Thus , around such an infinitesimal neighborhood , the normal fields pointing in opposite direction will subtract perfectly , creating a divergence flow of -2 and around 0 or positive away from boundaries . Strictly speaking the result holds only for piece-wise smooth surfaces ( Osher & Fedkiw , 2003 ) , with lower than -2 divergence possible at discontinuous surface points . Property 3.3 The relation is one-to-one between the binary boundary representation and the proposed vector field representation in a continuous domain . Proof This property is the result of equation ( 1 ) , for the forward transform and equation ( 3 ) for the inverse transform , providing a one-to-one relation . Note that the vector field transform as defined in equation ( 1 ) has to correct for two different kinds of indeterminate states . The first is on the boundary , that is solved by using the right hand limit so that one of the two opposite directions is chosen consistently . The second is when the infimum operation in equation ( 1 ) produces two or more closest points , corrected by choosing any one of the points for the infimum . The vector fields around such points flip directions creating a positive divergence as shown in Fig . 2 . More discussions are provided in §5 about the latter , which are in fact helpful for deciding superpixel centers . The above properties and their proofs are crucial for the validity of the proposed boundary representation and also to go from one representation to another for inference and visualization . Vector Transform and the Distance Transform . In essence , the normalized vector field proposed in equation ( 1 ) is another representation of the distance transform . Let φ ( x ) ∈ R+ define the distance transform , then the vector field v ( x ) in equation ( 1 ) can be obtained by the following partial derivatives ( Osher & Sethian , 1988 ; Osher & Fedkiw , 2003 ) : v ( x ) = −∇φ ( x ) . ( 4 ) One can optimize a given network by minimizing the loss on the distance transform ( DT ) or SDF ( Dapogny & Frey , 2012 ; Caliva et al. , 2019 ; Park et al. , 2019 ) instead of using the loss on the normalized vector field . Compared to the binary mask , the Vector transform ( VT ) , DT and SDF have an added advantage that they are sensitive to small topological changes . SDF on the other hand , does not support overlapping and open surfaces and is not easily adaptable to the image boundary problem . However , there are several reasons which make DT unsuitable for learning boundaries . During training , when the distance field is close to 0 i.e. , around the boundaries , any loss on the DT or SDF loses importance . Apart from the convergence problems of DT with gradient descent Osher & Fedkiw ( 2003 ) , DT is also hard to localize by thresholding under noise compared to the SDF and VT . In SDF , localizing the surface amounts to finding the zero crossings and in VT , the divergence measure in equation ( 2 ) provides an extremely sharp contrast making equation ( 3 ) trivial to solve . These differences in the thresholding problem can be seen in Fig . 5 in Appendix B and the experimental results . Additionally , despite reducing the class imbalance compared to binary boundary prediction , DT has an implicit bias to the weighted average of its typical range . On the other hand , a normalized vector field from VT equation ( 1 ) is sensitive to the topology similar to a distance field while also being localizable and sensitive at the boundaries , as shown in Fig . 1 .

Summary: This paper proposes treating boundaries as 1D surfaces using vector transformations in Equation (1).  The paper then proves a 1-to-1 relationship between the vector field representation and a binary boundary map.  The paper further learns the vector representation into an NN by using a variant of HR-Net that outputs a dense two-channel image with x and y component of the vector field and train it with Mean Sqaured Error loss.  Since the predicted boundaries using this method can be infinitely thin, so the paper proposes using average surface distances to evalute the performance, and shows improved results of their method comparing against three other losses using the same NN architecture.