CAFENet: Class-Agnostic Few-Shot Edge Detection Network

1 INTRODUCTION . Semantic edge detection aims to identify pixels that belong to boundaries of predefined categories . Boundary information has been shown to be effective for boosting the performance of semantic segmentation ( Bertasius et al. , 2016 ; Chen et al. , 2016 ) and localization ( Yu et al. , 2018a ; Wang et al. , 2015 ) . It also plays a key role in applications such as object reconstruction ( Ferrari et al. , 2007 ; Zhu et al. , 2018 ) , image generation ( Isola et al. , 2017 ; Wang et al. , 2018 ) and medical imaging ( Abbass & Mousa , 2017 ; Mehena , 2019 ) . Early edge detection algorithms interpret the problem as a low-level grouping problem exploiting hand-crafted features and local information ( Canny , 1986 ; Sugihara , 1986 ) . Recently , there have been significant improvements on edge detection thanks to the advances in deep learning . Moreover , beyond previous boundary detection , category-aware semantic edge detection became possible ( Acuna et al. , 2019 ; Hu et al. , 2019 ; Yu et al. , 2018b ) . However , it is impossible to train deep neural networks without massive amounts of annotated data . To overcome the data scarcity issue in image classification , few-shot learning has been actively discussed for recent years ( Finn et al. , 2017 ; Lifchitz et al. , 2019 ) . Few-shot learning algorithms train machines to learn previously unseen classification tasks using only a few relevant labeled examples . More recently , the idea of few-shot learning is applied to computer vision tasks requiring highly laborious and expensive data labeling such as semantic segmentation ( Dong & Xing , 2018 ; Wang et al. , 2019 ) and object detection ( Fu et al. , 2019 ; Karlinsky et al. , 2019 ) . Based on meta-learning across varying tasks , the machines can adapt to unencountered environments and demonstrate robust performance in various computer vision problems . In this paper , we consider a novel few-shot learning challenge , few-shot semantic edge detection , to detect the semantic boundaries using only a few labeled samples . Through experiments , we show that few-shot semantic edge detection can not be simply solved by fine-tuning a pretrained semantic edge detector or utilizing a nonparametric edge detector in a few-shot segmentation setting . To tackle this elusive challenge , we propose a class-agnostic few-shot edge detector ( CAFENet ) and present new datasets for evaluating few-shot semantic edge detection . Fig . 1 shows the architecture of the proposed CAFENet . Since the edge labels do not contain enough semantic information due to the sparsity of labels , performance of the edge detector severely degrades when the training dataset is very small . To overcome this , we adopt the segmentation process in advance of detecting edge with downsized feature and segmentation labels generated from boundaries labels . We utilize a simple metric-based segmentator generating a segmentation mask through pixel-wise feature matching with class prototypes , which are computed by masked average pooling of ( Zhang et al. , 2018 ) . The predicted segmentation mask provides the semantic information to the edge detector . The multi-scale attention maps are generated from the segmentation mask , and applied to corresponding multi-scale features . The edge detector predicts the semantic boundaries using the attended features . Using this attention mechanism , the edge detector can focus on relevant regions while alleviating the noise effect of external details . For meta-training of CAFENet , we introduce a simple yet powerful regularization method , Multi-Split Matching Regularization ( MSMR ) , performing metric learning on multiple low-dimensional embedding sub-spaces during meta-training . The main contributions of this paper are as follows . First , we introduce a few-shot semantic edge detection problem for performing semantic edge detection on previously unseen objects using only a few training examples . Second , we introduce two new datasets of SBD-5i and FSE-1000 for few-shot edge detection . Third , we propose a few-shot edge detector , CAFENet and validate the performance of the proposed method through experiments . 2 RELATED WORK . 2.1 FEW-SHOT LEARNING . To tackle the few-shot learning challenge , many methods have been proposed based on metalearning . Optimization-based methods ( Finn et al. , 2017 ; Ravi & Larochelle , 2016 ) train the metalearner which updates the parameters of the actual learner so that the learner can easily adapt to a new task within a few labeled samples . Metric-based methods ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Yoon et al. , 2019 ) train the feature extractor to assemble features from the same class together on the embedding space while keeping features from different classes far apart . Recent metric-based approaches propose dense classification ( Hou et al. , 2019 ; Kye et al. , 2020 ) . Dense classification trains an instance-wise classifier on pixel-wise classification loss which imposes coherent predictions over the spatial dimension and prevents overfitting as a result . Our model adopts the metric-based method for few-shot learning . Inspired by dense classification , we propose multi-split matching regularization which divides the feature vector into sub-vector splits and performs split-wise classification for regularization in meta-learning . 2.2 FEW-SHOT SEMANTIC SEGMENTATION . The goal of few-shot segmentation is to perform semantic segmentation within a few labeled samples based on meta-learning ( Shaban et al. , 2017 ; Dong & Xing , 2018 ; Wang et al. , 2019 ) . OSLSM of ( Shaban et al. , 2017 ) adopts a two-branch structure : conditioning branch generating element-wise scale and shift factors using the support set and segmentation branch performing segmentation with a fully convolutional network and task-conditioned features . Co-FCN ( Rakelly et al. , 2018 ) also utilizes a two-branch structure . The globally pooled prediction is generated using support set in conditioning branch , and fused with query features to predict the mask in segmentation branch . SGOne of ( Zhang et al. , 2018 ) proposes a masked average pooling to compute prototypes from pixels of support features . The cosine similarity scores are computed between the prototypes and pixels of query feature , and the similarity map guides the segmentation process . CANet of ( Zhang et al. , 2019 ) also adopts masked average pooling to generate the global feature vector , and concatenate it with every location of the query feature for dense comparison in predicting the segmentation mask . PANet of ( Wang et al. , 2019 ) introduces prototype alignment , predicting the segmentation mask of support samples using query prediction results as labels of query samples , for regularization . PMM of Yang et al . ( 2020 ) utilizes multiple prototypes generated using Expectation-Maximization ( EM ) algorithm to effectively leverage the semantic information from the few labeled samples . 2.3 SEMANTIC EDGE DETECTION . Semantic edge detection aims to find the boundaries of objects from image and classify the objects at the same time . The history of semantic edge detection ( Acuna et al. , 2019 ; Hu et al. , 2019 ) dates back to the work of ( Prasad et al. , 2006 ) which adopts the support vector machine as a semantic classifier on top of the traditional canny edge detector . Recently , many semantic edge detection algorithms rely on deep neural network and multi-scale feature fusion . CASENET of ( Yu et al. , 2017 ) addresses the semantic edge detection as a multi-label problem where each boundary pixel is labeled into categories of adjacent objects . Dynamic Feature Fusion ( DFF ) of ( Hu et al. , 2019 ) proposes a novel way to leverage multi-scale features . The multi-scale features are fused by weighted summation with fusion weights generated dynamically for each images and each pixel . Meanwhile , Simultaneous Edge Alignment and Learning ( SEAL ) of ( Yu et al. , 2018b ) deals with severe annotation noise of the existing edge dataset ( Hariharan et al. , 2011 ) . SEAL treats edge labels as latent variables and jointly trains them to align noisy misaligned boundary annotations . Semantically Thinned Edge Alignment Learning ( STEAL ) of ( Acuna et al. , 2019 ) improves the computation efficiency of edge label alignment through a lightweight level set formulation . 3 PROBLEM SETUP . For few-shot semantic edge detection , we use train set Dtrain and test set Dtest consisting of nonoverlapping categories Ctrain and Ctest . The model is trained only using Ctrain , and the test categories Ctest are never seen during the training phase . For meta-training of the model , we adopt episodic training as done in many previous few-shot learning works . Each episode is composed of a support set with a few-labeled samples and a query set . When an episode is given , the model adapts to the given episode using the support set and detect semantic boundaries of the query set . By episodic training , the model is learned so that it adapts to the unseen class using only a few labeled samples and predict semantic edges of query samples . For Nc-way Ns-shot setting , each training episode is constructed by Nc classes sampled from Ctrain . When Nc categories are given , Ns support samples and Nq query samples are randomly chosen from Dtrain for each class . In evaluation , the performance of the model is evaluated using test episodes . The test episodes are constructed in the same way as the training episodes , except Nc classes and corresponding support and query samples are sampled from Ctest and Dtest . In this work , we address Nc-way Ns-shot semantic edge detection . The goal is training the model to generalize to Nc unseen classes given only Ns images and their edge labels . Based on the few labeled support samples , the model should produce edge predictions of query images which belong to Nc unencountered classes . 4 METHOD . We propose a novel algorithm for few-shot semantic edge detection . Fig . 2 illustrates the network architecture of the proposed method . The proposed CAFENet adopts the semantic segmentation module to compensate for the lack of semantic information in edge labels . The predicted segmentation mask is utilized for attention in skip connection . The final edge detection is done using attentive multi-scale features . 4.1 SEMANTIC SEGMENTATOR . Most previous works on semantic edge detection directly predict edges from the given input image . However , direct edge prediction is a hard task when only a few labeled samples are given . To overcome this difficulty in few-shot edge detection , we adopt a semantic segmentation module in advance of edge prediction . With the assistance of the segmentation module , CAFENet can effectively localize the target object and extract semantic features from query samples . For few-shot segmentation , we employ the metric-learning which utilizes prototypes for foreground and background as done in ( Dong & Xing , 2018 ; Wang et al. , 2019 ) . Given the support set S = { xsi , ysi } Ns i=1 , the encoder E extracts features { E ( xsi ) } Ns i=1 from S. Also , for support labels { ysi } Ns i=1 , we generate the dense segmentation mask { Msi } Ns i=1 using a rule-based preprocessor , considering the pixels inside the boundary as foreground pixels in the segmentation label . Using down-sampled segmentation labels { msi } Ns i=1 , the prototype for foreground pixels PFG is computed as PFG = 1 Ns 1 H ×W ∑ i ∑ j Ej ( x s i ) m s i , j ( 1 ) where j indexes the pixel location , Ej ( x ) andmsi , j denote the jth pixel of featureE ( x ) and segmentation mask msi . H , W denote height and width of the images . Likewise , the background prototype PBG is computed as PBG = 1 Ns 1 H ×W ∑ i ∑ j Ej ( x s i ) ( 1−msi , j ) . ( 2 ) Following the prototypical networks of ( Snell et al. , 2017 ) , the probability that pixel j belongs to foreground for the query sample xqi is p ( yqi , j = FG|x q i ; E ) = exp ( −τd ( Ej ( xqi ) , PFG ) ) exp ( −τd ( Ej ( xqi ) , PFG ) ) + exp ( −τd ( Ej ( x q i ) , PBG ) ) ( 3 ) where d ( · , · ) is squared Euclidean distance between two vectors and τ is a learnable temperature parameter . With query samples { xqi } Nq i=1 and the down-sampled segmentation labels for query { mqi } Nq i=1 , the segmentation loss LSeg is calculated as the mean-squared error ( MSE ) loss between predicted probabilities and the down-sized segmentation mask LSeg = 1 Nq 1 H ×W Nq∑ i=1 H×W∑ j=1 { ( p ( yqi , j = FG|x q i ; E ) −m q i , j ) 2 } . ( 4 ) Note that the segmentation mask is generated in down-sized scale so that any pixel near the boundaries can be classified into the foreground to some extent , as well as the background . Therefore , we approach the problem as a regression using MSE loss rather than cross entropy loss .

Summary: This paper introduces a novel problem of "few-shot semantic edge detection" where semantic boundaries are to be learned/detected with a few labeled samples. In order to remedy the issue of label sparsity within the few-shot scenario, the authors have leveraged the use of the segmentation process which provides the semantic information to the edge detector. They have also incorporated a meta-learning approach, namely Multi-Split Matching Regularization (MSMR), to avoid the overfitting when high-dimensional embeddings are used for feature matching.