High-Frequency guided Curriculum Learning for Class-specific Object Boundary Detection

1 INTRODUCTION . Class-specific object boundary extraction from images is a fundamental problem in Computer Vision ( CV ) . It has been used as a basic module for several applications including object localization [ Yu et al . ( 2018a ) ; Wang et al . ( 2015 ) ] , 3D reconstruction [ Lee et al . ( 2009 ) ; Malik & Maydan ( 1989 ) ; Zhu et al . ( 2018 ) ] , image generation [ Isola et al . ( 2017 ) ; Wang et al . ( 2018 ) ] , multi-modal image alignment Kuse & Shen ( 2016 ) , and organ feature extraction from medical images [ Maninis et al . ( 2016 ) ] . Inspired from the sweeping success of deep neural networks in several CV fields , recent works [ Yu et al . ( 2017 ) ; Acuna et al . ( 2019 ) ; Yu et al . ( 2018b ) ] designed ConvNet-based architectures for object boundary detection and demonstrated impressive results . However , we notice that the results from these methods , as shown in Figure 1b , still suffer from significant false-alarms and misdetections even in regions without any clutter . We hypothesize that although boundary detection is simple at some pixels that are rooted in identifiable high-frequency locations , other pixels pose a higher level of difficulties , for instance , region pixels with an appearance similar to boundarypixels ; or boundary pixels with insignificant edge strengths ( ex : camouflaged regions ) . Therefore , the training process needs to account for different levels of learning complexity around different pixels to achieve better performance levels . In classical CV literature , the different levels of pixel complexities are naturally addressed by decomposing the task into a set of sequential sub-tasks of increasing complexities [ Leu & Chen ( 1988 ) ; Lamdan & Wolfson ( 1988 ) ; Kriegman & Ponce ( 1990 ) ; Ramesh ( 1995 ) ] . Most often , boundary detection problem has been decomposed into three sub-tasks : ( a ) low-level edge detection such as Canny [ Canny ( 1986 ) ] ; ( b ) semantic tagging/labeling of edge pixels [ Prasad et al . ( 2006 ) ] and ( c ) edge linking/refining [ Jevtić et al . ( 2009 ) ] in ambiguous regions . These approaches first solve the problem for simpler pixels ( with sufficient edge strength ) and then reason about harder pixels in the regions with ambiguous or missing evidence . However , with the advent of ConvNets , this classical perspective towards boundary extraction problem has been overlooked . New end-to-end trainable ConvNets have pushed the boundaries of state-of-the-art significantly compared to classical methods . However , we believe that classical multi-stage problem-solving schemes can help to improve the performance of ConvNet models . A parallel machine learning field , Curriculum Learning [ Bengio et al . ( 2009 ) ] also advocates this kind of multi-stage training schemes which train the network with a smoother objective first and later with the target task objective . These schemes are proven to improve the generalization of the models and convergences of training processes in several applications . Motivated by these factors , this work devises a curriculum-learning inspired two-stage training scheme for object boundary extraction that trains the networks for simpler tasks first ( subtasks a and b ) and then , in the second stage , trains to solve the more complex sub-task ( c ) . Our experimental results on a simulated dataset and a real-world aerial image dataset demonstrate that this systematic training indeed results in better performances . As mentioned already , the task of predicting object boundaries is mostly rooted in identifiable higher-frequency image locations . We believe that explicit augmentation of high-frequency contents to ConvNet will improve the convergence of training processes . Hence , this work designs a simple fully convolutional network ( FCN ) that takes in also high-frequency bands of the image along with the RGB input . Here in this work , we choose to use high-frequency coefficients from wavelet decomposition [ Stephane ( 1999 ) ] of the input image and augment them to conv features at different levels . These coefficients encode local features which are vital in representing sharp boundaries . Our empirical results convey that this explicit high-frequency augmentation helps the model to converge faster , especially in the first stage of curriculum learning . In summary , our contributions in this work are the following : • A novel two-stage training scheme ( inspired from curriculum-learning ) to learn classspecific object boundaries . • A novel ConvNet augmented by high-frequency wavelets . • A thorough ablation study on a simulated MNIST digit-contour dataset . • Experiments with a challenging aerial image dataset for road contour extraction • A real-world application of road contour extraction for aligning geo-parcels to aerial im- agery . Related Work : The problem of extracting object boundaries has been extensively studied in both classical and modern literature of CV . Most of the classical methods start with low-level edge detectors and use local/global features to attach semantics to the detected pixels . They , later , use object-level understanding or mid-level Gestalt cues to reason about missing edge links and occluded boundaries . The work by Prasad et al . ( 2006 ) used local texture patterns and a linear SVM classifier to classify edge pixels given by Canny edge detector . The work by Mairal et al . ( 2008 ) reasoned on low-level edges , but learned dictionaries on multiscale RGB patches with sparse coding and used the reconstruction error curves as features for a linear logistic classifier . The work of Hariharan et al . ( 2011 ) proposed a detector that combines low-level edge detections and semantic outputs from pre-trained object detectors to localize class-specific contours . Several recent works Yang et al . ( 2016 ) ; Yu et al . ( 2017 ) adopted fully convolutional networks ( FCN ) for the task of semantic boundary extraction . The work by Yang et al . ( 2016 ) proposed a FCN-based encoder-decoder architecture for object contour detection . The work of Bertasius et al . ( 2015 ) first used VGG-based network to locate binary semantic edges and then used deep semantic segmentation networks to obtain category labels . More recently , the work of Yu et al . ( 2017 ) proposed an FCN architecture , CASENet , with a novel shared concatenation scheme that fuses low-level features with higher conv layer features . The shared concatenation replicates lower layer features to separately concatenate each channel of the class activation map in the final layer . Then , a K-grouped 1 × 1 conv is performed on fused features to generate a semantic boundary map with K channels , in which the k-th channel represents the edge map for the k-th category . A few recent works Acuna et al . ( 2019 ) ; Yu et al . ( 2018b ) integrated an alignment module into the network to account for noise in the contour labels . Most of these networks are trained in an end-to-end manner , using crossentropy-based objective functions . These objectives treat all boundary pixels equally irrespective of the complexity around them . Unlike existing methods , we use explicit high-frequency augmentation to ConvNet and train it in a curriculum learning scheme that accounts for different levels of pixel complexities with two stages . 2 THE PROPOSED CURRICULUM LEARNING SCHEME . Curriculum Learning & Multi-Stage Training : Curriculum learning ( CL ) or multi-stage training schemes are motivated from the observation that humans and animals seem to learn better when trained with a curriculum like strategy : start with easier tasks and gradually increase the difficulty level of the tasks . A pioneering work of Bengio et al . ( 2009 ) introduced CL concepts to machine learning fields . This work proposed a set of CL schemes for the applications of shape recognition and language modeling ; demonstrated better performance and faster convergence . This work established curriculum learning as a continuation method . Continuation methods [ Allgower & Georg ( 2012 ) ] start with a smoothed objective function and gradually move to less smoothed functions . In other terms , these methods consider a class of objective functions that can be expressed as , Cλ ( θ ) = ( 1− λ ) Co ( θ ) + λCt ( θ ) ( 1 ) where Co ( θ ) is smoother or simple objective and Ct ( θ ) is the target objective we wish to optimize . There are several ways to choose Co ; it can either be the same loss function as Ct but solving the task on simpler examples , or be a proxy task simpler than the target task . In general , λ is a ( binary ) variable that takes values zero or one . It is set to zero initially and later increases to one . The epoch where it changes from zero to one is referred to as the switch epoch . Curriculum Learning in CV : CL-inspired training schemes are recently gaining attention in CV fields . Recently some popular CV architectures leveraged CL based schemes to improve model generalization and training stabilities . In FlowNet 2.0 [ Ilg et al . ( 2017 ) ] for optical flow prediction , simpler training data are fed into the network first and then the more difficult dataset . The object detection framework of Zhang et al . ( 2016 ) first trains simpler networks ( proposal , refiner-nets ) and then trains the final output-net in the end . Here we propose a two-stage CL scheme for object boundary detection methods . The proposed CL based training scheme for learning Object Boundaries : ConvNets for classspecific object boundary detection are in general trained with multi-label cross entropy-based objectives [ Yu et al . ( 2017 ) ] . Ct ( θ ) = − ∑ k ∑ p ( βYk ( p ) log Ŷk ( p ; θ ) + ( 1− β ) ( 1− Yk ( p ) ) log ( 1− Ŷk ( p ; θ ) ) ) ( 2 ) where θ denotes the weights of the network ; and p and k represent indices of pixel and class labels respectively . Ŷ and Y represent prediction and groundtruth label maps . β is the percentage of non-edge pixels in the image to account for skewness of sample numbers [ Yu et al . ( 2017 ) ] . This objective treats all the contour pixels equally and does not account for the complexity of the task around them . Here , we consider this as the target objective function , Ct , that we wish to optimize . We start the training , however , with a simpler task Co. We believe the pixels with strong edge strength are easy to localize and semantically identify . Hence , we propose to solve the task around those pixels in the first stage . We take element-wise multiplication between canny edge map of the input and dilated groundtruth label to prepare supervisory signal Z for this stage . Z = EI YD ( 3 ) where EI is the Canny edge map of image I and YD the dilated groundtruth map . This is as shown in Figure 2e . The objective function for this stage becomes ; Co ( θ ) = − ∑ k ∑ p ( βZk ( p ) log Ẑk ( p ; θ ) + ( 1− β ) ( 1− Zk ( p ) ) log ( 1− Ẑk ( p ; θ ) ) ) ( 4 ) Since we use a dilated version of GT in preparing Z , it also contains some non-object contour pixels . However , these might be refined in the second stage of CL , when trained with Y ( Eq 2 ) . Hence , the CL objective function in Eq 1 uses Eq 4 and Eq 2 as initial and target objective functions respectively . In the CL training scheme , we set the switch epoch as T/2 , where T is the total number of training epochs .

Summary: The suggest two improvements to boundary detection models: (1) a curriculum learning approach, and (2) augmenting CNNs with features derived from a wavelet transform. For (1), they train half of the epochs with a target boundary that is the intersection between a Canny edge filter and the dilated groundtruth. The second half of epochs is with the normal groundtruth. For (2), they compute multiscale wavelet transforms, and combine it with each scale of CNN features. They find on a toy MNIST example that the wavelet transform doesn’t impact results very much and curriculum learning seems to provide some gains. On the Aerial Road Contours dataset, they find an improvement of ~15% mAP over the prior baseline (CASENet).