CAFENet: Class-Agnostic Few-Shot Edge Detection Network

1 INTRODUCTION . Semantic edge detection aims to identify pixels that belong to boundaries of predefined categories . Boundary information has been shown to be effective for boosting the performance of semantic segmentation ( Bertasius et al. , 2016 ; Chen et al. , 2016 ) and localization ( Yu et al. , 2018a ; Wang et al. , 2015 ) . It also plays a key role in applications such as object reconstruction ( Ferrari et al. , 2007 ; Zhu et al. , 2018 ) , image generation ( Isola et al. , 2017 ; Wang et al. , 2018 ) and medical imaging ( Abbass & Mousa , 2017 ; Mehena , 2019 ) . Early edge detection algorithms interpret the problem as a low-level grouping problem exploiting hand-crafted features and local information ( Canny , 1986 ; Sugihara , 1986 ) . Recently , there have been significant improvements on edge detection thanks to the advances in deep learning . Moreover , beyond previous boundary detection , category-aware semantic edge detection became possible ( Acuna et al. , 2019 ; Hu et al. , 2019 ; Yu et al. , 2018b ) . However , it is impossible to train deep neural networks without massive amounts of annotated data . To overcome the data scarcity issue in image classification , few-shot learning has been actively discussed for recent years ( Finn et al. , 2017 ; Lifchitz et al. , 2019 ) . Few-shot learning algorithms train machines to learn previously unseen classification tasks using only a few relevant labeled examples . More recently , the idea of few-shot learning is applied to computer vision tasks requiring highly laborious and expensive data labeling such as semantic segmentation ( Dong & Xing , 2018 ; Wang et al. , 2019 ) and object detection ( Fu et al. , 2019 ; Karlinsky et al. , 2019 ) . Based on meta-learning across varying tasks , the machines can adapt to unencountered environments and demonstrate robust performance in various computer vision problems . In this paper , we consider a novel few-shot learning challenge , few-shot semantic edge detection , to detect the semantic boundaries using only a few labeled samples . Through experiments , we show that few-shot semantic edge detection can not be simply solved by fine-tuning a pretrained semantic edge detector or utilizing a nonparametric edge detector in a few-shot segmentation setting . To tackle this elusive challenge , we propose a class-agnostic few-shot edge detector ( CAFENet ) and present new datasets for evaluating few-shot semantic edge detection . Fig . 1 shows the architecture of the proposed CAFENet . Since the edge labels do not contain enough semantic information due to the sparsity of labels , performance of the edge detector severely degrades when the training dataset is very small . To overcome this , we adopt the segmentation process in advance of detecting edge with downsized feature and segmentation labels generated from boundaries labels . We utilize a simple metric-based segmentator generating a segmentation mask through pixel-wise feature matching with class prototypes , which are computed by masked average pooling of ( Zhang et al. , 2018 ) . The predicted segmentation mask provides the semantic information to the edge detector . The multi-scale attention maps are generated from the segmentation mask , and applied to corresponding multi-scale features . The edge detector predicts the semantic boundaries using the attended features . Using this attention mechanism , the edge detector can focus on relevant regions while alleviating the noise effect of external details . For meta-training of CAFENet , we introduce a simple yet powerful regularization method , Multi-Split Matching Regularization ( MSMR ) , performing metric learning on multiple low-dimensional embedding sub-spaces during meta-training . The main contributions of this paper are as follows . First , we introduce a few-shot semantic edge detection problem for performing semantic edge detection on previously unseen objects using only a few training examples . Second , we introduce two new datasets of SBD-5i and FSE-1000 for few-shot edge detection . Third , we propose a few-shot edge detector , CAFENet and validate the performance of the proposed method through experiments . 2 RELATED WORK . 2.1 FEW-SHOT LEARNING . To tackle the few-shot learning challenge , many methods have been proposed based on metalearning . Optimization-based methods ( Finn et al. , 2017 ; Ravi & Larochelle , 2016 ) train the metalearner which updates the parameters of the actual learner so that the learner can easily adapt to a new task within a few labeled samples . Metric-based methods ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Yoon et al. , 2019 ) train the feature extractor to assemble features from the same class together on the embedding space while keeping features from different classes far apart . Recent metric-based approaches propose dense classification ( Hou et al. , 2019 ; Kye et al. , 2020 ) . Dense classification trains an instance-wise classifier on pixel-wise classification loss which imposes coherent predictions over the spatial dimension and prevents overfitting as a result . Our model adopts the metric-based method for few-shot learning . Inspired by dense classification , we propose multi-split matching regularization which divides the feature vector into sub-vector splits and performs split-wise classification for regularization in meta-learning . 2.2 FEW-SHOT SEMANTIC SEGMENTATION . The goal of few-shot segmentation is to perform semantic segmentation within a few labeled samples based on meta-learning ( Shaban et al. , 2017 ; Dong & Xing , 2018 ; Wang et al. , 2019 ) . OSLSM of ( Shaban et al. , 2017 ) adopts a two-branch structure : conditioning branch generating element-wise scale and shift factors using the support set and segmentation branch performing segmentation with a fully convolutional network and task-conditioned features . Co-FCN ( Rakelly et al. , 2018 ) also utilizes a two-branch structure . The globally pooled prediction is generated using support set in conditioning branch , and fused with query features to predict the mask in segmentation branch . SGOne of ( Zhang et al. , 2018 ) proposes a masked average pooling to compute prototypes from pixels of support features . The cosine similarity scores are computed between the prototypes and pixels of query feature , and the similarity map guides the segmentation process . CANet of ( Zhang et al. , 2019 ) also adopts masked average pooling to generate the global feature vector , and concatenate it with every location of the query feature for dense comparison in predicting the segmentation mask . PANet of ( Wang et al. , 2019 ) introduces prototype alignment , predicting the segmentation mask of support samples using query prediction results as labels of query samples , for regularization . PMM of Yang et al . ( 2020 ) utilizes multiple prototypes generated using Expectation-Maximization ( EM ) algorithm to effectively leverage the semantic information from the few labeled samples . 2.3 SEMANTIC EDGE DETECTION . Semantic edge detection aims to find the boundaries of objects from image and classify the objects at the same time . The history of semantic edge detection ( Acuna et al. , 2019 ; Hu et al. , 2019 ) dates back to the work of ( Prasad et al. , 2006 ) which adopts the support vector machine as a semantic classifier on top of the traditional canny edge detector . Recently , many semantic edge detection algorithms rely on deep neural network and multi-scale feature fusion . CASENET of ( Yu et al. , 2017 ) addresses the semantic edge detection as a multi-label problem where each boundary pixel is labeled into categories of adjacent objects . Dynamic Feature Fusion ( DFF ) of ( Hu et al. , 2019 ) proposes a novel way to leverage multi-scale features . The multi-scale features are fused by weighted summation with fusion weights generated dynamically for each images and each pixel . Meanwhile , Simultaneous Edge Alignment and Learning ( SEAL ) of ( Yu et al. , 2018b ) deals with severe annotation noise of the existing edge dataset ( Hariharan et al. , 2011 ) . SEAL treats edge labels as latent variables and jointly trains them to align noisy misaligned boundary annotations . Semantically Thinned Edge Alignment Learning ( STEAL ) of ( Acuna et al. , 2019 ) improves the computation efficiency of edge label alignment through a lightweight level set formulation . 3 PROBLEM SETUP . For few-shot semantic edge detection , we use train set Dtrain and test set Dtest consisting of nonoverlapping categories Ctrain and Ctest . The model is trained only using Ctrain , and the test categories Ctest are never seen during the training phase . For meta-training of the model , we adopt episodic training as done in many previous few-shot learning works . Each episode is composed of a support set with a few-labeled samples and a query set . When an episode is given , the model adapts to the given episode using the support set and detect semantic boundaries of the query set . By episodic training , the model is learned so that it adapts to the unseen class using only a few labeled samples and predict semantic edges of query samples . For Nc-way Ns-shot setting , each training episode is constructed by Nc classes sampled from Ctrain . When Nc categories are given , Ns support samples and Nq query samples are randomly chosen from Dtrain for each class . In evaluation , the performance of the model is evaluated using test episodes . The test episodes are constructed in the same way as the training episodes , except Nc classes and corresponding support and query samples are sampled from Ctest and Dtest . In this work , we address Nc-way Ns-shot semantic edge detection . The goal is training the model to generalize to Nc unseen classes given only Ns images and their edge labels . Based on the few labeled support samples , the model should produce edge predictions of query images which belong to Nc unencountered classes . 4 METHOD . We propose a novel algorithm for few-shot semantic edge detection . Fig . 2 illustrates the network architecture of the proposed method . The proposed CAFENet adopts the semantic segmentation module to compensate for the lack of semantic information in edge labels . The predicted segmentation mask is utilized for attention in skip connection . The final edge detection is done using attentive multi-scale features . 4.1 SEMANTIC SEGMENTATOR . Most previous works on semantic edge detection directly predict edges from the given input image . However , direct edge prediction is a hard task when only a few labeled samples are given . To overcome this difficulty in few-shot edge detection , we adopt a semantic segmentation module in advance of edge prediction . With the assistance of the segmentation module , CAFENet can effectively localize the target object and extract semantic features from query samples . For few-shot segmentation , we employ the metric-learning which utilizes prototypes for foreground and background as done in ( Dong & Xing , 2018 ; Wang et al. , 2019 ) . Given the support set S = { xsi , ysi } Ns i=1 , the encoder E extracts features { E ( xsi ) } Ns i=1 from S. Also , for support labels { ysi } Ns i=1 , we generate the dense segmentation mask { Msi } Ns i=1 using a rule-based preprocessor , considering the pixels inside the boundary as foreground pixels in the segmentation label . Using down-sampled segmentation labels { msi } Ns i=1 , the prototype for foreground pixels PFG is computed as PFG = 1 Ns 1 H ×W ∑ i ∑ j Ej ( x s i ) m s i , j ( 1 ) where j indexes the pixel location , Ej ( x ) andmsi , j denote the jth pixel of featureE ( x ) and segmentation mask msi . H , W denote height and width of the images . Likewise , the background prototype PBG is computed as PBG = 1 Ns 1 H ×W ∑ i ∑ j Ej ( x s i ) ( 1−msi , j ) . ( 2 ) Following the prototypical networks of ( Snell et al. , 2017 ) , the probability that pixel j belongs to foreground for the query sample xqi is p ( yqi , j = FG|x q i ; E ) = exp ( −τd ( Ej ( xqi ) , PFG ) ) exp ( −τd ( Ej ( xqi ) , PFG ) ) + exp ( −τd ( Ej ( x q i ) , PBG ) ) ( 3 ) where d ( · , · ) is squared Euclidean distance between two vectors and τ is a learnable temperature parameter . With query samples { xqi } Nq i=1 and the down-sampled segmentation labels for query { mqi } Nq i=1 , the segmentation loss LSeg is calculated as the mean-squared error ( MSE ) loss between predicted probabilities and the down-sized segmentation mask LSeg = 1 Nq 1 H ×W Nq∑ i=1 H×W∑ j=1 { ( p ( yqi , j = FG|x q i ; E ) −m q i , j ) 2 } . ( 4 ) Note that the segmentation mask is generated in down-sized scale so that any pixel near the boundaries can be classified into the foreground to some extent , as well as the background . Therefore , we approach the problem as a regression using MSE loss rather than cross entropy loss .

Summary: This paper works on few-shot semantic edge detection. Instead of dealing with the problem in a single stage, the authors decompose the problem into two stages.  First, a few-shot segmentation stage, where the foreground and the background probability are estimated via attention with the foreground and the background prototype (averaged feature vector on the foreground and the background region). Second, the feature maps from the encoder are masked by the attention map and sent to the decoder to generate the final edge-map.