High-Frequency guided Curriculum Learning for Class-specific Object Boundary Detection
Authors: VSR Veeravasarapu
Deepak Mittal
Abhishek Goel
Maneesh Singh
Venue:  (2020)
Keywords: Computer Vision
Object Contour Detection
Curriculum Learning
Wavelets
Aerial Imagery

Abstract: This work addresses class-specific object boundary extraction, i.e., retrieving boundary pixels that belong to a class of objects in the given image. Although recent ConvNet-based approaches demonstrate impressive results, we notice that they produce several false-alarms and misdetections when used in real-world applications. We hypothesize that although boundary detection is simple at some pixels that are rooted in identifiable high-frequency locations, other pixels pose a higher level of difficulties, for instance, region pixels with an appearance similar to the boundaries; or boundary pixels with insignificant edge strengths. Therefore, the training process needs to account for different levels of learning complexity in different regions to overcome false alarms. In this work, we devise a curriculum-learning-based training process for object boundary detection. This multi-stage training process first trains the network at simpler pixels (with sufficient edge strengths) and then at harder pixels in the later stages of the curriculum.  We also propose a novel system for object boundary detection that relies on a fully convolutional neural network (FCN) and wavelet decomposition of image frequencies.  This system uses high-frequency bands from the wavelet pyramid and augments them to conv features from different layers of FCN. Our ablation studies with contourMNIST dataset, a simulated digit contours from MNIST, demonstrate that this explicit high-frequency augmentation helps the model to converge faster. Our model trained by the proposed curriculum scheme outperforms a state-of-the-art object boundary detection method by a significant margin on a challenging aerial image dataset.  


Under review as a conference paper at ICLR 2020
HIGH-FREQUENCY GUIDED CURRICULUM LEARNING FOR CLASS-SPECIFI OBJECT BOUNDARY DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
This work addresses class-specific object boundary extraction, i.., retrieving boundary pixels that belong to a class of objects the given image. Although recent ConvNer-based approaches demonstrate impressive results, we notice that they produce several false-alarms and misdetections when used in real-world ap- plications. We hypothesize that although boundary detection is simple at some pixels that are rooted in identifiable high-frequency locations, other pixels pose a higher level of difficulties, for instance, region pixels with an appearance similar to the boundaries; or boundary pixels with insignificant edge strengths. Therefore, the training process needs account for different levels of learning complexity in different regions to overcome false alarms. In this work, we devise a curriculum- learning-bai based training process for object boundary detection. This multi-stage training process first trains the network at simpler pixels (with sufficient edge strengths) and then at harder pixels the later stages of the curriculum. We also propose novel system for object boundary detection that relies on fully convo- lutional neural network (FCN) and wavelet decomposition of image frequencies. This system uses high-frequency bands from the wavelet pyramid and augments them to conv features from different layers of FCN, Our ablation studies with conTourMNIST dataset, simulated digit contours from MNIST, demonstrate that this explicit high-free sighfrreuency augmentation helps the model to converge faster. Our model trained by the proposed curriculum scheme outperforms state-of-the-art object boundary detection method by a significant margin on a challenging aerial image dataset.
INTRODUCTION
(a) Input image
(b) CASENet output
(c) Ours
(d) Groundtruth
Figure 1: Class-specins object boundary extraction: This work proposes novel ConNNer architecture & two-stage training scheme for class saesi-peeiffic object boundary estimation. This is sample result from the model when applied problem extracting road contours from aerial image tiles.
Class Class-specific object boundary extraction from images is fundamental problem in Computer Vision (CV). It has been used as a basic module for several applications including object localization Yuu et al. (2018a); Wang et al. [,015)], 3D reconstruction [Lee et al. (2009); Malik & Maydan (1989); Zhu et al. (2018)). image generation [Isola et al. (0017): Wang et al. (2018), multi-modal image alignment Kuse & Shen (2016), and organ feature extraction from medical images [Maninis et al. (2016)]. Inspired from the sweeping success deep neural networks in several CV fields, recent
works al, 2017) Acuna al. (0119) Yu (2018b) designed ConvNet-based architectures

---

Under review as a conference paper at ICLR 2020
for object boundary detection and demonstrated impressive results. However, notice that the results from these methods, as shown in Figure [b] still suffer from significant false-alarms and misdetections even in regions without any clutter. We hypothesize that although boundary detection is simple at some pixels that are rooted in identifiable high-frequency locations, other pixels pose higher level of difficulties, for instance, region pixels with an appearance similar to boundary- pixels; or boundary pixels with insignificant edge strengths (ex: camouflaged regions). Therefore, the training process needs to account for different levels of learning complexity around different pixels to achieve better performance levels.
In classical CV literature, the different levels of pixel complexities are naturally addressed by de- composing the task into a set of sequential sub-tasks of increasing complexities [Leu & Chen (1888); Lamdan & Wolfssnn| 1988); Kriegmann & Ponce (1990): Ramesh (1995)]. Most often, boundary de-
tection problem has been decomposed into three sub-tasks: (a) low-level edge detection such as
Canny (Canny [: (b) semantic tagging/laheling of edge pixels [Prasad et al. (2006)) and (c)
edge linking/refining (Jevtić et al. [ in ambiguous regions. These approaches first solve the problem for simpler pixels (with sufficient edge strength) and then reason about harder pixels in the regions with ambiguous or missing evidence. However, with the advent of ConvNets, this classical perspective towards boundary extraction problem has been overlooked. New end-to-end trainable ConvNets have pushed the boundaries of state-of-the-art significantly compared to classical meth- ods. However, we believe that classical multi-stage problem-solving schemes can help to improve the performance of ConvNer models. A parallel machine learning field, Curriculum Learning [Ben- gio et al. (2009h] also advocates this kind of multi-stage training schemes which train the network with a smoother objective first and later with the target task objective. These schemes are proven to improve the generalization of the models and convergences of training processes in several ap- plications. Motivated by these factors, this work devises a curriculum- learning inspired two-stage training scheme for object boundary extraction that trains the networks for simpler tasks first (sub- tasks a and b) and then, the second stage, trains to solve the more complex sub-task (c). Our experimental results on simulated dataset and a real-world acrial image dataset demonstrate that this systematic training indeed results in better performances.
As mentioned already, the task of predicting object boundaries is mostly rooted in identifiable figger-frequency image locations. We believe that explicit augmentation of high-frequency con- tents to ConvNet will improve the convergence of training processes. Hence, this work designs a simple fully convolutional network (FCN) that takes in also high-frequency bands of the image along with the RGB input. Here in this work, we choose to use high-frequency coefficients from wavelet decomposition [Stephane] (1999)] of the input image and augment them to conv features at different levels. These coefficients encode local features which are vital in representing sharp boundaries. Our empirical results convey that this explicit high- tigh-frequency augmentation helps the model to converge faster, especially in the first stage of curriculum learning.
In summary, our contributions in this work are the following:
A novel two-stage training scheme (inspired from curriculum-lcarning) learn class- specific object boundaries.
A novel ConvNet augmented by high- frequeryy wavelets.
A thorough ablation study on a simulated MNIST digit-contour dataset.
Experiments with challenging aerial image dataset for road contour extraction
A real-world application of road contour extraction for aligning geo-parcels to aerial im- agery.
The problem of extracting object boundaries has been extensively studied in both classical and modern literature of CV. Most of the classical methods start with low-level edge de- tectors and use local/global features to attach semanties to the detected pixels. They, later, use object-level understanding or mid-level Gestalt cues to reason about missing edge links and oc- cluded boundaries. The work by Prasad et al. (2006) used local texture patterns and a linear SVM classifier to classify edge pixels given by Canny edge detector. The work by Mairal et al. (2008) reasoned on low-level edges, but learned dictionaries on multiscale RGB patches with sparse cod- ing and used the reconstruction eиror curves as features for linear logistic classifier. The work of Hariharam et al. (20TT) proposed a detector that combines low-level edge detections and semantic outputs from pre- trained object detectors to localize class-specific contours.
Related Work:
2

---

Under review as a conference paper at ICLR 2020
Several recent works Yang et al. (2016); Yu et al. (2017) adopted fully convolutional networks (FCN) for the task of semantic boundary extraction. The work by Yang et al.) (2016) proposed a FCN- based pecoder-decoder architecture for object contour detection. The work of Bertasius et al. (2015) first used VGGi-bsee network to locate binary semantic edges and then used deep semantic segmentation networks obtain category labels. More recently, the work of Yu al. (2017) proposed FCN architecture, CASENet, with a novel shared concatenation scheme that fuses low-level features with higher conv layer features. The shared concatenation replicates lower layer features to separately concatenate each channel of the class activation map in the final layer. Then, K K-grouped 1 x conv is performed on fused features to generate semantic boundary map with K channels, in which the k-th channel represents the edge map for the k-th category. A few recent works Acuna al. 22019; Yu et al. 2018b) integrated an alignment module into the network to account for noise in the contour labels. Most of these networks are trained in an end-to-end manner, using cross- entropy- objective functions. These objectives treat all boundary pixels equally irrespective of the complexity around them. Unlike existing methods, we explicit high- nigh-frequency augmentation to ConvNet and train it in a curriculum learning scheme that accounts for different levels of pixel complexities with two stages.
2 THE PROPOSED CURRICULUM LEARNING SCHEME
Curriculum Learning & Multi-Stage Training: Curriculum learning (CI.) or multi-stage training schemes are motivated from the observation that humans and animals seem to learn better when trained with a curriculum like strategy: start with casier tasks and gradually increase the difficulty level of the tasks. A pioneering work of |engio et al. (2099) introduced CL concepts to machine learning fields. This work proposed a set of CL schemes for the applications of shape recognition and language modeling; demonstrated better performance and faster convergence. This work established curriculum learning continuation method. Continuation methods [Allgower & Georg (2012) start with smoothed objective function and gradually move to less smoothed functions. other terms, these methods consider class of objective functions that can be expressed as,
(x = ()C..0) XC,
where (0) is smoother or simple objective and (0) is the target objective we wish to optimize. There are several ways to choose Co: it can either be the same loss function as Ct but solving the task on simpler examples, or be a proxy task simpler than the target task. In general, À is (binary) variable that takes values zero or one. It is set to zero initially and later increases to one. The epoch where it changes from zero to one is referred to as the switch epoch.
Curriculum Learning in CV: CL.-inspired training schemes are recently gaining attention in CV fields. Recently some popular CV architectures leveraged CL. based schemes to improve model generalization and training stabilities. In FlowNet 2.0 [lg et al. (2017)] for optical flow prediction, simpler training data are fed into the network first and then the more difficult dataset. The object detection framework of ZZann al. (2016) first trains simpler networks (proposal, refiner-nets) and then trains the final output-net in the end. Here we propose a two-stage CL scheme for object boundary detection methods.
The proposed CL based training scheme for learning Object Boundaries: ConvNets for class- specific object boundary detection are in general trained with multi-lahel cross entropy-based objec- tives Yuu et al. (2017)].
-10)=--> (B&(()) log P4 (1- () (e(p)) log(1 Pe(p:0))
(2)
where 0 denotes the weights the network; and p and ls represent indices of pixel and class labels respectively. Y and Y represent prediction and groundtruth label maps. 9 is the percentage of non-edge pixels in the image to account for skewness of sample numbers (Yu et al. (2017)). This objective treats all the contour pixels equally and does not account for the complexity of the task around them. Here, we consider this the target objective function, C., that wish optimize.
We start the training, however, with a simpler task C,,- We believe the pixels with strong edge strength are easy localize and semantically identify. Hence, we propose to solve the task around
3

---

Under review as a conference paper at ICLR 2020
those pixels in the first stage. We take element-wine multiplication between canny edge map of the input and dilated groundtruth label to prepare supervisory signal Z for this stage.
Er Yn
where E is the Canny edge map image I and YD the dilated groundtruth map. This as shown in Figure 2e
(a) Input image
(b) Canny edges (c) Groundtruth Y (So- pervisor for Stage 2)
dilated Y
Supervisor for Stage
Figure 2: Supervisory signal preparation for Stage
The objective function for this stage becomes;
Co(0) -- P
(Zx log (1) -1 Zx(x)) log(1 Zn(p;
Since we use dilated version of GT in preparing also contains some non-object contour pixels. However, these might be refined in the second stage of CL, when trained with Y (Eq 2 Hence, the CL objective function in Eq usee ag Eq initial and target objective functions respectively. In the CL training scheme, set the switch epoch T/2, where T is the total number of training epochs.
HIGH FREQUENCY AUGMENTED CONVNET
The first stage in the proposed CL-based training scheme is about learning to locate class high- frequency pixel locations and recognizing their semantic entity. We believe that explicit augmenta- tion of high-frequency contents of the input image to the network will help learn faster. Hence, we design novel architecture with explicit high-frequency augmentation that can make effective use of our multi-stage training scheme.
Frequency Decomposition via Wavelet Transform (WT): Here we briefly introduce the main concepts of wavelet decomposition of the images (Stephane] (199)]] The multi-resolution wavelet transform provides localized spatial frequency analysis of images. This transform decomposes the given image into different frequency ranges, thus, permits the isolation of the frequency compo- nents introduced by boundaries into certain subbands, mainly in high-frequency subbands. The 2D wavelet decomposition of an image (T E R2Mx2N is performed by applying ID low-pass (d) and high-pass (1)) filters. This operation results in four decomposed subband images at each level, referred to as low-low (W"),, low-high (Wilh), high-low (W* and high-high (Whh) wavelet coefficients. For instance, the single-level WT is defined as follows.
(i,j) Σ Σ / (2i k. (jok)6(11
wl' = Σ Σ k. k
(5)
(1,)) = Σ Σ ke k
whth Σ Σ ((2 + k, (1))))))+)))) &
(k))))())
All the convolutions above are performed with stride 2. yielding a down-ianpling of factor along each spatial dimension. The WT results in four bands (W With Wht, Whil E RMXN at first
4

---

Under review as a conference paper at ICLR 2020
level. A multi-scale wavelet decomposition successively performs Eq low-low frequency efficients {.} from fine to coarse resolution. this sense of resolution, decomposition in multi- resolution wavelet analysis is in analogy to the dawp-iampling steps in ResNet blocks [He et al. worth noting that, while the low-frequency coefficients {.} store local
(2016)] Moreover, averages of the input data, its high-frequency counterparts, namely {-)th, {-}M and (-) hh encode local textures which are vital in recovering sharp boundaries. This motivates us to make use of the high-frequency wavelet coefficients to improve the quality pixel-level boundary extraction. Throughout this paper, we extensively use the Haar wavelet for its simplicity and effectiveness to boost the performances of the underlying ConvNet. In this scenario, the Haar filters used for decom- are given by Φ (0.5, 0.5) and (0.5, 0.5).
position in Eq 5
High Froquancy
Pyneria
wi
wi
Clunteloen
Skip-4
Skip
Figure 3: High-frequency augmented contour detection architecture.
The proposed ConvNet: Our architecture is as shown in Figure 3 We now explain the components of the network.
ResNet-101 Backbone: Several recent ConvNet architectures including CASENet use ResNet-101 as a basic building block. We use similar architecture with simple concatenation as our backbone architecture. ResNet-101 with five blocks that uses 4 down-sampling steps (max-pool or stride 2). This results in 4 tensors of conv features at four scales: {F.)
Wavelets augmentation: Multi-scale wavelet decomposition of the input image with four levels produce a pyramid of frequency bands. The spatial resolutions of these bands are analogous to conv features each level. We select high frequency bands each level of wavelet pyramid and concatenate in channel dimension of the conv features at corresponding level in the ResNet: {Fr + W Whl W=
)==1....1
Fuse blocks: These Fuse module are designed to learn proper fusion schemes to fuse wavelets to conv features. These are based 1 conv blocks that takes in {F, W Whi Whih and produce features to match next block's 's input dimensions.
Skip modules: Similar to CASENet, we design set of skip connections that pass lower layer features to the classification layer. These skip modules uses up-sampling followed by 1 x com layers (with 64 filters). After sammamli these features from different resolutions are brought back to original input resolution. All these features are concatenated in the channel axis.
cls module: This is pixel-level classification module implemented with comv layer with filters, followed by sigmoid layer to produce class-specific object boundary maps from concatenated features from different levels of ResNet. For experiments provided Section we use work with the problems of single class object contour extraction.
4 EXPERIMENTS
This section provides an extensive evaluation of our boundary extraction model and training scheme with a simulated dataset as well as challenging real-world dataset. Our simulated dataset is pre- pared for thorough ablation study without tolerating long training periods. Our real-world dataset collected to train the proposed model for road contour extraction from aerial images, that is an essential component for the system build in Section
5

---

Under review as a conference paper at ICLR 2020
4.1 ABLATION STUDIES WITH MNIST DIGIT-CONTOURS
4.1.1 EXPERIMENTAL SETUP
The MNIST dataset [Deng 2012 is popularly known toy dataset in machine learning, originally designed for image classification algorithms. Lately, has been adapted to work with several tasks such as object detection [multiMNIST Eslami et al. (2016)), color image genera- tion [colorMNIST Metz et al. (2016)), and Spatio-temporal representation learning [moVingMNISTT Srivastava et al. (0155]. These adopted datasets have been used understand the behavior of the models and training processes without tolerating long training periods. Similarly, we also simulate contourMNIST dataset using MNIST digits bbjects-of-interes
Dataset preparation:
The MNIST database contains 70000 gray-scale images of handwritten digits of resolution 28 x 28. The test images (10,000 digits) kept separate from the training images (60,000 digits). For our study, simulate 128 128- sized images with random digits placed on random background sampled from Pascal-VOS dataset [Everingham al. [2013)] (as shown in Figure +. Corresponding pixel-level groundtruth labels for digits contours are also generated for each simulated image. To mimic human labeling noise in the groundtruth of real-world training data, we transform these GT labels (see Figure (ii with randomly generated lps (thin-plate-splines) transformations. In this process, we prepare train set of 5000 images and a validation set of 500 images. Training on this dataset for ten epochs takes just around 17 minutes for all the experiments in this section.
OPONAC AT PU
L
&
(l) CarneeraC vs CASENet
(g) impact
(a) Input
(b) CASENa
(c) Ours
(d) GT
(h) HF impact
(i) HF impact on CL stages
Figure 4: Evaluation results on contourMNIST
Implementation details: We have implemented all our methods using the PyTorch machine learn- ing framework. We used SGD optimizer with learning rate le 4 for all experiments. Training experiments are done using NVIDIA GTX-1080 GPU with 12GB RAM. For wavelets decomposi- tion. we used pytorch-wavelet,/ package. All the models in this section are trained for ten epochs. Switch epoch is set to 5th epoch for CL-based training experiments. We refer our high- sighfrreuencc augmented ConvNet as CnirNet for brevity in this section.
We used average precision (AP) as an evaluation metric. Prediction maps are thresholded with a fixed threshold of 0.5 to compute false and true positives. The usual ground- truth labels (Y) are used for validation. For simplicity, we omitted expensive post-processing steps (such as thinning) and evaluations at -ati-al-data-scale [ODS Acuna et al. [2019)] during training epochs. Each training experiment is run for five times for statistical significance of the results and observations. Validation accuracy, in terms of AP, is computed for every epoch and plotted in Figure Shaded regions in the plot represent standard deviations in the validation accuracies of different models and training schemes. For final trained models, also report AP measures at ODS, i.e. maxAP, that finds maximum AP obtained by different thresholds. These values are shown in the brackets, following the label names in the legends of the plots.
Evaluation criteria:

---

Under review as a conference paper at ICLR 2020
4.1.2 ABLATION STUDIES
How do our proposals (Cntrree++CL) perform compared to the state-of-the-art?
We first start comparing the performance of our proposals with a state-of-the-art method, CASENet Yu al.] (017)] Figure 4e illustrates the validation accuracy plots of CASENet and our Cutr- Ner+CL over training epochs. CutrNer+CL seem to converge faster than the CASENet. Accord- ing to AaxAP scores mentioned the legend of the plot, our network (CnieNet+ CL) outperforms CASENet by approximately 6%. It means that CntrNet+CI. produces less false alarms than the CASENet. This is also quite evident in the qualitative results provided in Figure
Is the proposed CL training necessary?
Here we evaluate how curriculum learning impacts the learning processes the absence of high-frequency (HF) augmentation. Towards this, we experiment with CntrNet without HF augmentation, "rgmeetationnCntrNee"" CutrNer Validation accuracy plots are as shown in Figure 4f when CutrNer is
trained with and without CL-based scheme. The model achieves better generalization performances when trained with CL. It improves validation accuracy by 7.4 % compared to the one trained without CL..
Is the proposed high-frequency augmentation necessary?
We also evaluate how high-frequency augmentation impacts the model learning process and perfor- mance in the absence of curriculum learning. As seen in Figure 44g high-fregg high-frequency augmentation seems help the model to achieve better performances promptly, starting from Stage 1. However, it seems to be helping the model's performance by only small extent (0.6%) in the end.
How does high-frew greu-eryyeeeyy augmentation impact the first stage of CL?
As seen in Figure 4j. the explicit augmentation of high-frequencies seems to be helping stage I to converge faster. This expected because Stage is about identifying the pixels with high frequen- cies and recognize their class.
(a) Input image
(b) CASENet output
Ours
GT
(e) maxAP
Figure 5: Evaluation results on Areal Image Road Contours dataset
4.2 RESULTS ON AERIAL ROAD CONTOURS DATASET
In this section, we evaluate the models real-world aerial image dataset for the task of extracting road contours.
Dataset preparation: Our dataset originally is prepared for geo-parcel alignment task (Section 5 and contains 13,368 aerial image tiles captured over city of Redding, California. Each of these tiles 4864 x 7168 resolution and approximately cover an area of 0.15 x 0.2 square miles. Samples of these tiles are shown in Figures [7 For training and evaluation experiments, we manually label 1l of these aerial image tiles with road contours. This labeling process took approximately hours tile. Ten tiles are used to prepare a training set, and one tile is left for validation. To prepare our train and val sets, we randomly crop several subregions 2000 x 2000 from these tiles and resize them to 256 x 256, We also some data augmentation techniques to amplify the scale of the set. The train and val sets contain 21649 and 522 samples, respectively.
The network's 's implementation details similar to above. Here set batch size as eight and input size is 256 x 256. The models are trained for 20 epochs that take approximately 2 days 18 hours AWS GPU instance with NVIDIA Tesla K80.
Implementation:
We report maxAP scores for both CASENet and our model (CntrNet+CL) in Figure Se Our method performs with accuracy of 89.6%, while CASENet is at 75.3% maxAP points. In
Evaluation:

---

Under review as a conference paper at ICLR 2020
other words, method outperforms CASENet by nearly 15%. As shown in Figure 5 our results are sharper and accurate compared to CASENet. A result at the tile scale can be seen in Figure 6 We use this model in real-world application of geo-parees alignment in the next section.
(a) Input image
(b) CASENct output
(c) Ours
(d) GT
Figure Sample result on a tile from Parcel alignment dataset
APPLICATION TO GEO-PARCEL ALIGNMENT
In this section, discuss application of the proposed model, CntrNet+CIL (trained for road tour extraction), for aligning geo-parcel data with aerial image tiles. Geo-parcel data is generally used identify public and private land property boundaries for tax assessment processes. Parcels are shapefiles in the records maintained by local counties and represent latitude-longiudde GPS co- ordinates of the property boundaries. We project these parcel shapes (using perspective projection) onto a geo-tagged coordinate system of the camera with which aerial imagery was captured. This process results in binary contour images as shown Figure /c] These contours are ideally expected to match visual property entities in the aerial image of the corresponding region (shown in Figure 7ap. However, due to several differences in their collection processes, these two modalities of the data often misalign by large extent, sometimes, in the order of 10 meters. Figure 7d depicts the misalignment of the original (before alignment) parcel contours overlaid on the acrial image in blue color. This misalignment might lead to wrong property assignments individuals, thus, incorrect tax assessments. These data modalities of geographical data need to be aligned well before using to assist the processes of property assignment and tax assessment.
(a) Acrial image
(b) Road contours from (a) (c) Gco-parcels
(d) Aligned parcels over- laid on (a)
Figure 7: Geo-paree alignment with road evidences. Better in enlarged version.
We use our CnirNet+ CL model trained on the aerial road contours dataset prepared in Section Given an aerial image tile, as shown Figure 7a we first divide the tile into 12 non-overlapping patches and pass them batch to the model. The predictions from the model are stitched back to form road contour prediction at tile resolution, as shown Figure 7b One can pose this alignment problem as moving and target images, respectively. We designed an image registration network for the same, and discussions about the registration network is out of the scope of this paper. few samples of final aligned parcels are overlaid with red color in Figure 7d
image registration task by considering geo-parcel and road contour images
6 CONCLUSIONS
In this work, we presented novel ConvNet with explicit high-frequency augmentation and new two-stage curriculum learning scheme for class-specific object boundary extraction. Our ablation studies with simulated MNIST digit-contours dataset demonstrated that this explicit high-frequency augmentation helps the model to converge faster. Our high- gigh-frequency augmented model, when trained with proposed CL based scheme, outperformed CASENet by nearly 15% aerial image dataset. We also demonstrated a use-case developed contour extraction model to align geo-parcel boundaries with roads extracted from aerial imagery.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Acuna, Amlan Kar, and Sanja Fidler. Devil is in the edges: Learning semantic boundaries from noisy annotations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11075-11083, 2019.
Eugene I. Allgower and Kurt Georg. Numerical continuation methods: an introduction, volume 13. Springer Science Business Media, 2012.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, ACM, 2009.
Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. High-for-low and low-for-high: Efficient boundary detection from deep object features and its applications to high-level vision. In Pro- ceedings of the IEEE International Conference on Computer Vision, pp. 504-512, 2015.
John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679-698, 1986.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141-144 2012.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey Hin- ton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233 3233, 2016.
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98-136 2015.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. In 201 International Conference Computer Vision, pp. 991-998. IEFE, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Aleksandar Jevtić, Ignacio Melgar, and Diego Andina. Ant based edge linking algorithm. 2009 35th Annual Conference of ICEE Industrial Electronics, pp. 3353-3358. IEEE, 2009.
David J. Kriegman and Jean Ponce. On recognizing and positioning curved 3-d objects from image contours. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12711111177-113, 1990.
Manohar Kuse and Shaojie Shen. Robust camera motion estimation using direct edge alignment and sub-gradient method. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 573-579. IEEE, 2016.
Yehezkel Lamdan and Haim Wolfson. Geometric hashing: A general and efficient model-based recognition scheme, 1988.
David C Lee, Martial Hebert, and Takeo Kanade. Geometric reasoning for single image structure recovery. In 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 2136-2144 IEFE, 2009.

---

Under review as a conference paper at ICLR 2020
Jia Via-Guu Leu and Limin Chen. Polygonal approximation of 2-d shapes through boundary merging. Pattern Recognition Letters, 7(4):231-238, 1988.
Julien Mairal, Marius Leordeanu, Francis Bach, Martial Hebert, and Jean Ponce. Discriminative sparse image models for class-specific edge detection and image interpretation. In European Conference on Computer Vision, pp. 43-56. Springer, 2008.
Jitendra Malik and Dror Maydan. Recovering three-dimensional shape from single image of curved objects. IEEE Transactions on Puttern Analysis and Machine Intelligence, 11(6):555- 566, 1989.
Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Pablo Arbeláez, and Luc Van Gool. Deep retinal image understanding. In International conference on medical image computing and computer-assisn intervention, pp. 140-148. Springer, 2016.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. CoRR, abs/161 1.02163, 2016. Utthttp:// aww.i.xii www.v..oo/ab//.se.e/.ee wtt.aa/1../...1.111111..1.11....
Mukta Prasad, Andrew Zisserman, Andrew Fitzgibbon, M Pawan Kumar, and Philip HS Torr. Learn- ing class-specific edges for object detection and segmentation. In Computer Vision, Graphics and Image Processing, pp. 94-105. Springer, 2006.
Visvanathan Ramesh. Performance characterization of image understanding algorithms. PhD the- sis, Citeseer, 1995.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pp. 843-852, 2015.
Mallat Stephane. wavelet tour of signal processing. The Sparse Way, 1999.
Shenlong Wang, Sanja Fidler, and Raquel Urtasun. Lost shopping! monocular localization in large indoor spaces. In Proceedings of the IEEE International Conference Computer Vision, pp- 2695-2703, 2015.
Ting- Chug Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8798-8807 2018.
Jimei Yang, Brian Price, Scott Cohen, Honglak Lee, and Ming- Yang. Object contour detec- tion with fully convolutional encoder-decoder network. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 193-202, 2016.
Xin Yu, Sagar Chaturvedi, Chen Feng. Yuichi Taguchi, Teng-Yos Yok Lee, Clinton Fernandes, and Srikumar Ramalingam. Vlase: Vehicle localization by aggregating semantic edges. In 2018 IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3196-3203. IEEE, 2018a.
Zhiding Yu, Chen Feng, Ming- Yu Liu, and Srikumar Ramalingam. Casenet: Deep category- aware semantic edge detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5973, 2017.
Zhiding Yu, Weiyang Liu, Yang Zou, Chen Feng, Srikumar Ramalingam, BVK Vijaya Kumar, and Jan Kautz. Simultaneous edge alignment and learning. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 388-404, 2018b
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, (4910):449-15000 2016.
Dongchen Zhu, Jiamao Li, Xianshun Wang, Jingquan Peng, Wenjun Shi, and Xiaolin Zhang, Seman- tic edge based disparity estimation using adaptive dynamic programming for binocular sensors. Sensors, 18(4):1074, 2018
10


Reviews:
[{"review_id": "rkgtgQXRYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The suggest two improvements to boundary detection models: (1) a curriculum learning approach, and (2) augmenting CNNs with features derived from a wavelet transform. For (1), they train half of the epochs with a target boundary that is the intersection between a Canny edge filter and the dilated groundtruth. The second half of epochs is with the normal groundtruth. For (2), they compute multiscale wavelet transforms, and combine it with each scale of CNN features. They find on a toy MNIST example that the wavelet transform doesn’t impact results very much and curriculum learning seems to provide some gains. On the Aerial Road Contours dataset, they find an improvement of ~15% mAP over the prior baseline (CASENet).\n\nI have several concerns with this work:\n* The idea of using wavelet transforms to augment CNNs has been more thoroughly explored in prior work (e.g., see [1]).\n* No comparison to existing SOTA segmentation models (e.g., [2]). These semantic / instance segmentation models can easily be adapted to the task of boundary detection. I suspect the baseline here is weak.\n* Section 6 is severely unfinished. The explanation is sparse and there are no quantitative results -- just the output of the model overlaid on one example.\n* The choice of curriculum learning task is arbitrary, and there are no ablations explaining why this is a reasonable task. For example, what about random subsets of pixels? At the moment, it offers no insight for practitioners.\n* There are no ablations for the Aerial Road Contours experiments. This seems necessary because it is the only realistic dataset evaluated in this work. The MNIST experimental results appear qualitatively different from the Contours experiment. For example, they show that wavelet features do not make much of a difference, but does it make a difference for Contours?\n\nAltogether, this work unfortunately offers few insights to vision practitioners, let alone general practitioners. Substantial work needs to be devoted to expanding experimental coverage. \n\n[1] Wavelet Convolutional Neural Networks. Shin Fujieda, Kohei Takayama, Toshiya Hachisuka\n[2] TensorMask: A Foundation for Dense Object Segmentation. Xinlei Chen, Ross Girshick, Kaiming He, Piotr Dollár", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1grXbxL5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper shows the efficiency of curriculum learning and using problem specific features for contour detection.\n\nThe authors consider a network trained for class-specific edge detection (e.g. outlining edges of roads in an image). They propose two problem domain tricks to improve the performance:\n- use curriculum learning by training the network to first detect the \"easy\" edges, i.e. edges found also using the Canny edge detector\n- add high frequency wavelet coefficients as additional feature maps to the convnet.\n\nThe two techniques prove important on two tasks:\n- modified MNIST edge detection\n- road boundary detection in aerial imaginery.\n\nMaybe the most important aspect of the paper is that shows that with little data (the real world aerial imaginary dataset had only 11 labeled tiles) manual feature engineering and smart cost function selection are still relevant. \nSince this is a common pattern in many application domains, such as specialized medical image processing where labeled data is scarce, the paper is important. However, it is not clear if the proposed chanes are needed when more labeled data is available and how much do they overfit to the small test set.\n\nThe paper could be strengthened by analysing the impact of the proposed problem-dependent CL and features versus the amount of available training data. Are they still relevant with 100 labeled images?\nThese experiments could be even run on the artificial MNIST set.\n\nMoreover, some analysis of result significance is needed. On the real-world dataset there is only 1 test case!! How much was the network tuned to properly work on it? Maybe the authors can run a  cross-validation to show that the results don't overfit to this one test image?\n\nMinor remarks: You refer to Stephane's Mallat book as Stephane 1999, this is wrong, his first name is Stephane and last is Mallat, please fix the bibliography and use Mallat 1999.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkx3Oy1VqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main idea of the paper is adding a curriculum learning-based extension to CASEnet, a boundary detection method from 2017. In the first phase, the loss emphasizes easier examples with high gradient in the image, and in the second phase, the method is trained on all boundary pixels. This change seems to improve edge detection performance on a toy MNIST and an aerial dataset. \n\nA second innovation claimed by the authors is adding Wavelet decomposition-based processing into the net. Unfortunately, this mostly only speeds up learning, as the ablation does not show meaningful improvements relative to the error bounds in later stages of training. Furthermore, the paper lacks a discussion of related work on incorporating wavelet ideas into neural networks. For example: \n-- Generic Deep Networks with Wavelet Scattering, by Ouyallon et al. \n-- Invariant scattering convolution networks, by Bruna et al \nand multiple more recent ones. Without either clear performance gains or more in-depth discussion of this novelty, it is not clear how to take it into account. \n\nWhen reading the paper, it appears that \"boundary detection\" for the cases that the authors are exploring is very directly related to 2-class semantic segmentation (road / non-road), the only difference being that the edge boundaries are weighted much higher in the cross-entropy loss. As such, there is a lot more recent net architecture work for semantic segmentation that should be directly applicable, and should perform much better than CaseNet when adapted to the task. As a result, the experiments and the significance of this paper are rather marginal. \n\nIn experimental results, the authors threshold prediction with 0.5, which is suboptimal. The resulting metric, which is just \"accuracy\" is called incorrectly \"average precision\". Instead, true definition of average precision should be used, that is not dependent on potentially suboptimal fixed thresholding but on the area under precision-recall curve instead. Finally, it would be helpful to do ablation and confidence bounds also on the main aerial road results, as the 15% gain is significantly more than the gain that appears in the toy dataset. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received all negative reviewers, and the scores were kept after the rebuttal. The authors are encouraged to submit their work to a computer vision conference where this kind of work may be more appreciated. Furthermore, including stronger baselines such as Acuna et al is recommended.
Decision: Reject