CAFENet: Class-Agnostic Few-Shot Edge Detection Network
Authors: Younghyun Park
Jun Seo
Jaekyun Moon
Venue:  (2021)
Keywords: Few-shot edge detection
Few-shot learning
Semantic edge detection

Abstract: We tackle a novel few-shot learning challenge, few-shot semantic edge detection, aiming to localize boundaries of novel categories using only a few labeled samples. Reliable boundary information has been shown to boost the performance of semantic segmentation and localization, while also playing a key role in its own right in object reconstruction, image generation and medical imaging. Few-shot semantic edge detection allows recovery of accurate boundaries with just a few examples. In this work, we present a Class-Agnostic Few-shot Edge detection Network (CAFENet) based on meta-learning strategy. CAFENet employs a semantic segmentation module in small-scale to compensate for lack of semantic information in edge labels. The predicted segmentation mask is used to generate an attention map to highlight the target object region, and make the decoder module concentrate on that region. We also propose a new regularization method based on multi-split matching. In meta-training, the metric-learning problem with high-dimensional vectors are divided into smaller subproblems with low-dimensional sub-vectors. Since there are no existing datasets for few-shot semantic edge detection, we construct two new datasets, FSE-1000 and SBD-5i, and evaluate the performance of the proposed CAFENet on them. Extensive simulation results confirm that the proposed CAFENet achieves better performance compared to the baseline methods using fine-tuning or few-shot segmentation.

Under review as a conference paper at ICLR 2021
CAFENET: CLASS-AGNOSTIC FEW-SHOT EDGE DETECTION NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
We tackle novel few-shot learning challenge, few-shot semantic edge detection, aiming to localize boundaries of novel categories using only a few labeled sam- ples. Reliable boundary information has been shown boost the performance of semantic segmentation and localization, while also playing a key role in its own right in object reconstruction, image generation and medical imaging. Few-shot semantic edge detection allows recovery of accurate boundaries with just few examples. In this work, we present Class-Agnostic Few-shot Edge detection Network (CAFENet) based meta-learning strategy. CAFENet employs se- mantic segmentation module in small-scale to compensate for lack of semantic information in edge labels. The predicted segmentation mask is used to generate an attention map to highlight the target object region, and make the decoder mod- ule concentrate on that region. We also propose on multi-split matching. In meta-traine meta-training, the metric-learning problem with high- dimensional vectors are divided into smaller subproblems with low- -wwddimesiional sub-vectors. Since there are no existing datasets for few-shot semantic edge de- tection, construct two datasets, FSE-1000 and SBD-5', and evaluate the performance of the proposed CAFENet on them. Extensive simulation results confirm that the proposed CAFENet achieves better performance compared to the baseline methods using fine-tuning or few-shot segmentation.
regularization method based
INTRODUCTION
Semantic edge detection aims to identify pixels that belong to boundaries of predefined categories. Boundary information has been shown to be effective for boosting the performance of semantic segmentation (Bertasius et al. 2016: CCen et al., 2016) and localization (Yu et al. 2018a| Wang et al. 2015). It also plays a key role in applications such object reconstruction (Ferrari et al. Zhu et al. 2018 image generation (Isola et al. 2017 Wang et al., (2018) and medical imaging (Abbass & Mousa 2017: Mehena 2019). Early edge detection algorithms interpret the problem as a lowe-leeel grouping problem exploiting hand-crafted features and local information (Canny) 1986 Sugihara| 1986). Recently, there have been significant improvements on edge detection thanks the advances in deep learning. Moreover, beyond previous boundary detection, category- semantic edge detection became possible (Acuna et al. 2019 Hu et al. 2019 Yu al., 2018b). However, is impossible to train deep neural networks without massive amounts of annotated data.
2007
To overcome the data scarcity issue in image classification, few-shot learning has been actively discussed for recent years (Finn et al. [017]] Lifchitz et al.] (019).. Few-shot learning algorithms train machines to learn previously unseen classification tasks using only few relevant labeled examples. More recently, the idea of few-shot learning is applied to computer vision tasks requiring highly laborious and expensive data labeling such as semantic segmentation (Dong & Xing 2018 Wang et al. 2019) and object detection (Fu et al.) 2019 Karlinsky et al.) 2019). Based on meta-learning across varying tasks, the machines can adapt to unencountered environments and demonstrate robust performance in various computer vision problems. In this paper, we consider a novel few-shot learning challenge, few-shot semantic edge detection, to detect the semantic boundaries using only few labeled samples. Through experiments, we show that few-shot semantic edge detection can not be simply solved by fine-tuning a pretrained semantic edge detector or utilizing a nonparametric edge detector in few-shot segmentation setting. To tackle this elusive challenge, we propose class-agnostic few-shot edge detector (CAFENet) and present new datasets for evaluating few-shot semantic edge detection.

---

Under review as a conference paper at ICLR 2021
Feature Extractor
Segmentator
Edge Detector
Roundayy prodiciion
forsersund
makked avcrage pooling
Rechiek-38
Backgmund
Probxtppss
Support
Shar wriights
Support
Edge detecttor
Foreground
Maltsszale
Bockground Redure mutching
Query fostures
Segrariation oeddctionn
Quary set
Figure 1:
Architecture overview the proposed CAFENel. The feature extractor or encoder extracts feature from the image, the segmentator generates segmentation mask based on metric learning, and the edge detector detects semantic boundaries using the segmentation mask and query features.
Fig. [ shows the architecture of the proposed CAFENet. Since the edge labels do not contain enough semantic information due to the sparsity of labels, performance of the edge detector severely degrades when the training dataset is very small. To overcome this, adopt the segmentation pro- cess in advance of detecting edge with downsized feature and segmentation labels generated from boundaries labels. We utilize simple metrie-based segmentator generating segmentation mask through pixel-wise feature matching with class prototypes, which are computed by masked average pooling of (Zhang al. 2018). The predicted segmentation mask provides the semantic information to the edge detector. The multi-scale attention maps are generated from the segmentation mask, and applied to corresponding multi-scale features. The edge detector predicts the semantic boundaries using the attended features. Using this attention mechanism, the edge detector can focus on rele- vant regions while alleviating the noise effect of external details. For meta-training of CAFENet, we introduce simple yet powerful regularization method, Multi-Split Matching Regularization (MSMR), performing metric learning on multiple low-dimensional embedding sub-spaces during meta-training.
The main contributions of this paper are as follows. First, we introduce a few-shot semantic edge detection problem for performing semantic edge detection on previously unseen objects using only a few training examples. Second, we introduce two new datasets of SBD-5' and FSE- 1000 for few-shot edge detection. Third, propose few-shot edge detector, CAFENet and validate the performance of the proposed method through experiments.
2
RELATED WORK
2.1 FEW-SHOT LEARNING
To tackle the few-shot learning challenge, many methods have been proposed based on meta- learning. Optimization-basa methods (Finn et al., 2017] Ravi & Larochelle] 2016) train the meta- learner which updates the parameters of the actual learner that the learner can easily adapt to a new task within few labeled samples. Metric-based methods (Vinyals et al., 2016 Snell et al.) [201]] Yoon et al. 2019) train the feature extractor to assemble features from the same class together on the embedding space while keeping features from different classes far apart. Recent metric-based ap- proaches propose dense classification (Hou al., 2019 Kye et al. (020). Dense classification trains an instance-wise classifier pixel-wise classification loss which imposes coherent predictions over the spatial dimension and prevents overfitting as a result. Our model adopts the vectrcebasee method for few-shot learning. Inspired by dense classification, propose multi-split matching regulariza- tion which divides the feature vector into vectvr splits and performs split-wise classification for regularization in meta-learning.
2.2
FEW-SHOT SEMANTIC SEGMENTATION
The goal of fewt segmentation is to perform semantic segmentation within a few labeled samples based meta-learning (Shaban et al. 2017 Dong & Xing 2018 Wang et al. 2019). OSLSM of (Shaban et al. 2017) adopts a two-branch structure: conditioning branch generating dlement-wise scale and shift factors using the support set and segmentation branch performing segmentation with a fully convolutional network and condiiondi features. Co-FCN (Rakelly et al.] 2018) also utilizes two-branch structure. The globally pooled prediction is generated using support set in
2

---

Under review as a conference paper at ICLR 2021
conditioning branch, and fused with query features to predict the mask in segmentation branch. SG- One of (Zhang et al. 2018) proposes a masked average pooling to compute prototypes from pixels of support features. The cosine similarity scores are computed between the prototypes and pixels of query feature, and the similarity map guides the segmentation process. CANet of (Zhang et al.) 2019) also adopts masked average pooling to generate the global feature vector, and concatenate with every location of the query feature for dense comparison in predicting the segmentation mask. PANet of (Wang et al., 2019) introduces prototype alignment, predicting the segmentation mask of support samples using query prediction results as labels of query samples, for regularization. PMM of Yang et al. (2020) utilizes multiple prototypes generated using Expectation Expecaaton-Maximizatton (EM) algorithm to effectively leverage the semantic information from the few labeled samples.
2.3 SEMANTIC EDGE DETECTION
Semantic edge detection aims to find the boundaries of objects from image and classify the objects at the same time. The history of semantic edge detection (Acuna et al. 2019 Hu al. 2019) dates back to the work of (Prasad et al. (006) which adopts the support vector machine as semantic classifier on top of the traditional canny edge detector. Recently, many semantic edge detection algorithms rely on deep neural network and multi-scale feature fusion. CASENET of (Yu et al.) 2017) addresses the semantic edge detection as a multi-label problem where each boundary pixel is labeled into categories of adjacent objects. Dynamic Feature Fusion (DFF) of (Hu et al. 2019) proposes novel way to leverage multi-scale features. The multi-scale features are fused by weighted summation with fusion weights generated dynamically for each images and each pixel. Meanwhile, Simultaneous Edge Alignment and Learning (SEAL) of (Yu et al. (0188) deals with severe annotation noise of the existing edge dataset (Hariharan al. 2011). SEAL treats edge labels as latent variables and jointly trains them to align noisy misaligned boundary annotations. Semantically Thinned Edge Aligmment Learning (STEAL) of (Acuna al., (2099) improves the computation efficiency of edge label alignment through a lightweight level set formulation.
3 PROBLEM SETUP
For few-shot semantic edge detection, we use train set Dirain and test set Drest consisting of non- overlapping categories Crrain and Crest- The model is trained only using Crrains and the test cat- egories Crest are never seen during the training phase. For meta-training of the model, we adopt episodic training as done in many previous few-shot learning works. Each episode is composed of support set with few-labeled samples and query set. When episode is given, the model adapts to the given episode using the support set and detect semantic boundaries of the query set. By episodic training, the model is learned so that it adapts to the unseen class using only a few labeled samples and predict semantic edges of query samples.
For No-way Ng-shot setting, each training episode is constructed by No classes sampled from Ctrain. When N₂ categories are given, N, support samples and N₂ query samples are randomly chosen from Deraen for each class. In evaluation, the performance of the model evaluated using test episodes. The test episodes are constructed in the same way as the training episodes, except Ne classes and corresponding support and query samples are sampled from Ctest and D
In this work, we address No-way Ng-shot semantic edge detection. The goal is training the model to generalize to Nº unseen classes given only N₂ images and their edge labels. Based on the few labeled support samples, the model should produce edge predictions of query images which belong to Nº unencountered classes.
4 METHOD
We propose novel algorithm for few-shot semantic edge detection. Fig. illustrates the network architecture of the proposed method. The proposed CAFENet adopts the semantic segmentation module to compensate for the lack of semantic information in edge labels. The predicted segmenta- tion mask is utilized for attention skip connection. The final edge detection is done using attentive multi-scale features.
4.1 SEMANTIC SEGMENTATOR
Most previous works on semantic edge detection directly predict edges from the given input im- age. However, direct edge prediction is a hard task when only few labeled samples are given. To overcome this difficulty in few-shot edge detection, we adopt semantic segmentation module in ad- vance of edge prediction. With the assistance of the segmentation module, CAFENet can effectively
3

---

Under review as a conference paper at ICLR 2021
Query somple
Prediction
((0)
D
320
E(1)
S(1)
D
160 x 160
160 x 160
160 x 160
320 320
E(2)
S(2)
D
x
80 80
80 80
COnV
E
S(9)
D
COIV2
x
40 40
4D 40
COnV3
Support set May
R(4
((4) 20 20
D
c00
20 20
D (4.0
Foregrsura puooiags
Segmentttor
Up-ssupling
Muitphoction
Concatoration
Bacigroans piotetype
Attemtio map
Figure 2: Network architecture overview of proposed CAFENet. ResNet-34 encoder E()) E(4) extracts multi-level semantic features. The segmentator module generates segmentation prediction using query fea- ture from EX4) and prototypes Prg. PBG from support set features. Small bottleneck blocks g g(4) transform the original image and multi-scale features from encoder blocks to be more suitable for edge detec- tion. The attention maps generated from segmentation prediction are applied to multi-scale features localize the semantically related region. Decoder ( D takes attentive multi-scale features to give edge predic- tion.
localize the target object and extract semantic features from query samples. For few-shot segmen- tation, we employ the metric metrie-learning which utilizes prototypes for foreground and background as done in (Dong & Xing 2018 Wang et al. 2019). Given the support set S = {-] M v/ J the en- coder E extracts features {E(+
Ji= from S. Also, for support labels [
Ji=1 we generate the 1==1 using rule-based preprocessor, considering the pixels inside
dense segmentation mask (M;)
the boundary as foreground pixels in the segmentation label. Using down-sampled segmentation
labels {m:]== the prototype for foreground pixels PFG is computed as
1 PFG N. H W Eg
Es(2))mis
where j indexes the pixel location, (x) and mi, denote the jth pixel of feature E(x) and segmen- tation mask m.. H, W denote height and width of the images. Likewise, the background prototype PBG computed as
1 1 ΣΣ PBG N. HXW
(,()))(1-m) miz).
Following the prototypical networks of (nell et al. 2017). the probability that pixel j belongs to foreground for the query sample z! is
cap(
plui, FGIx;: erp(- (x Pro))
(3)
(x PsG))
where -(-.-) is squared Euclidean distance between two vectors and is learnable temperature parameter. With query samples {a
N. and the down-sampled segmentation labels for query {m N. the segmentation loss Lseg is calculated as the mean-squared error (MSE) loss between predicted probabilities and the down-sized segmentation mask
No HXW
1 N HX
Lseg
Σ
= FG]x;; [
- m
Note that the segmentation mask is generated in down-sized scale so that any pixel near the bound- aries be classified into the foreground to extent, well the background. Therefore, approach the problem as regression using MSE loss rather than cross entropy loss.
4.2 MULTI-SPLIT MATCHING REGULARIZATION
The metric-based few-shot segmentation method utilizes distance metrics between the high- dimensional feature vectors and prototypes, in Fig. 3al However, this approach is prone
4

---

Under review as a conference paper at ICLR 2021
Foreground prolotype
Fonegrundd protoype
Pisel-wise
Siltl matching
matching
w
Background
Backgroud polottpee
prototype
Query
Query feature
(a) High-ding ligg--iimensional Matching
(b) Split-wise Matching
Figure 3: Comparison between (a) High-dimensional feature matching used in Dong & Xing 2018
Wang
2019) and (b) split-wise feature matching in MSMR
overfit due the massive number of parameters in feature vectors. To get around this issue, we propose novel regularization method, multi-split matching regularization (MSMR). In MSMR, high-dimensional feature vectors are split into several low-dimensional feature vectors, and the met- ric learning is conducted each vector split Fig. 3b
With the query feature E(x)) E RCXWxH, where C is channel dimension and H, W are spatial dimensions, we divide E(x!) into sub-vectors {=") along channel dimension. Each sub- vector E² (a.) is in R xWxH Likewise, the prototypes PFG and PBG also disassembled into K vectors { and {P | k== along channel dimension where Ppc: Pgg E R
For the 6thh sub-vector query feature E (a?), the probability that the jth pixel belongs to the foreground class is computed as follows:
cry (pTddd(E) (x)), Ptc)) p (w.,, = FG exp(-rd(P)) (o))) PFc)) + eerp( esp(- (opp(-d((F))) (x))) Pb))
(S)
MSMR divides the original metric learning problem into K small sub-problems composed of fewer parameters and acts regularizer for high-dimensional embeddings. The prediction results of K sub-problems are reflected on learning by combining the split-wiss wise segmentation losses to original segmentation loss Eq. The total segmentation loss calculated
N4 HxW Σ
K
T.seg
Σ
((r.s m2,)2²
Σ
(ris m²,
(6)
N₂ H W
where Pij = p(vV, FG E) and ph. d² (wl.) FG|=9;E),
4.3 ATTENTIVE EDGE DETECTOR
As shown in Fig. we adopt the nested encoder structure extract rich hierarchical features. The multi-scale side outputs from encoder E E(4) are post-processed through bottleneck blocks ((2) ~ 5(1), Since ResNet-34 gives side outputs of down-sized scale, we pass the original image through bottleneck block ((()) to extract local details in original scale. In front of 5(3), we employ the Atrous Spatial Pyramid Pooling (ASPP) block of (Chen et al. 2017). We have empirically found that locating ASPP there shows better performance.
In utilizing multi-scale features, we employ the predicted segmentation mask M from the segmen- tator where the 5th pixel of M is the predicted probability from Eq. 3 Note that generate M based on the entire feature vectors and the prototypes instead of utilizing sub-vectors, since the split- wise metric learning is used only for regularizing the segmentation module. For each layer 1, M() denotes the segmentation mask upscaled to the corresponding feature size by bilinear interpolation.
Using segmentation prediction mask (1)1), we generate attention map A as follows. First, the prediction with a value lower than threshold À is rounded down to zero, to ignore activation in re- gions with low confidence. Second, we broaden the attention map using morphological dilation of (Feng et al.) 2019) as second chance, since the segmentation module may not always guarantee fine results. The final attention map of (th layer (()) computed as follows
A 1 1 A)M (1) + (1 > A)
where (M) applied to the multi-scale features of corresponding bottleneck blocks ((0) S(4), We apply the
()
(1) is the rounded value of prediction mask M)) The attention maps
5

---

Under review as a conference paper at ICLR 2021
Iaput image
g00
$99
128
Input image
g+i
ST3)
256
Enooder sideoottput
Pisel-wise semantic attenton
Decoder input
Figure 4: An example of activation map of (Yoiinkii et al. 2015) before and after pixel-wise semantic attention (warmer color has higher value). As seen, the attention mechanism makes encoder side-outputs attend to the regions of target object (horse the figure).
residual attention of (Hou al., 2019), where the initial multi-level side outputs from 5(1) are pixel- wisely weighted by 1 + A to strengthen the activation value of the semantically important region. We visualize the effect of semantic attention in Fig. 4
As shown in Fig- 2] the decoder network is composed of five consecutive convolutional blocks. Each decoder block D(!) contains three 3 3 convolution layers. The outputs of decoder blocks D(1) (44) are bilinearly upsampled
and passed to the next block. Similar to (Feng al. 2019). the up-sampled decoder outputs are then concatenated to the skip connection features from bottleneck blocks 5(0) ~ 5 and previous decoder blocks. Multi-scale semantic information and local details transmitted through skip architectures. The hierarchical decoder network in turn refines the outputs of the previous decoder blocks and finally produces the edge prediction g of query samples z².
Following the work of (Deng et al. 2018), we combine cross-entropy loss and Dice loss to produce crisp boundaries. Given query set Q = {x/ ve J and prediction mask i the cross-entropy loss is computed as
N₂ LCE
Σ
log(??)
Σ
Log(1 #/))
(8)
JEY,
JEY
where Y4 and Y- denote the sets foreground and background pixels. The Dice loss then computed as
(O,,,) +,, +
Lpice
23, VL
where denotes the pixels of a label. The final loss for meta eett-training is given by
Lffint = Lseg + LoE + I.Dice.
(10)
5 EXPERIMENTS
5.1 DATASETS
5.1.1
SBD-5
Based on the SBD dataset of (Hariharan al. 2011) for semantic edge detection, we propose new SBD-5' dataset. With reference to the setting of Pascal-5', 20 classes of the SBD dataset are divided into 4 splits. In the experiment with split i, 5 classes in the ith split are used as test classes Ctest- The remaining 15 classes are utilized as training classes Ctrain- The training set Dtrain constructed with all image-annotation pairs whose annotation include at least one pixel from the classes in Ctrain- For each class, the boundary pixels which do not belong to that class are considered as background. The test set Drest is also constructed in the same way as Deraens using Clest this time. Considering the difficulty of few-shot setting and severe annotation noise of the SBD dataset, we extract thicker edges. We utilize edges extracted from the segmentation mask as ground truth instead of original boundary labels of the SBD dataset, and thickness of extracted edge lies between 3 ~ 4 pixels on average. We conduct experiments with each split of i = 0 ~ 3, and report performance of each split as well as the averaged performance. Note that unlike Pascal-5', we do not consider division of training and test samples of the original SBD dataset. As a result, the images in D Derein might appear in Drest with different annotation from class Ctest-

---

Under review as a conference paper at ICLR 2021
5.1.2 FSE-1000
The datasets used in previous semantic edge detection research such as SBD of (Hartharan et al. 2011] and Cityscapes of (Cordts et al. 2016) are not suitable for few-shot learning as they have only 20 and 30 classes, respectively. We propose new dataset for few-shot edge detection, which we call FSE-1000, based on FSS-1000 of (Wei et al. 2019). FSS-1000
dataset for few-shot segmentation and composed of 1000 classes and 10 images per class with grreroound-backgrourg segmentation annotation. From the images and segmentation masks of FSS- 1000, we build FSE- 1000 by extracting boundary labels from segmentation masks. As done in SBD-5', we extract thick edges of which thickness is around ~ 3 pixels on average in the light of difficulty associated with fho-shot setting. For dataset split, we split 1000 classes into 800 training classes and 200 test classes. We will provide the detailed class configuration in the Supplementary Material.
5.2 EVALUATION SETTINGS
We use two evaluation metrics measure the few-shot semantic edge detection performance of our approach: the Average Precision (AP) and the maximum F-measure (MF) at optimal dataset scale (ODS). In evaluation, we compare the unthinned raw prediction results and the ground truths without Non-Maximum Suppression (NMS) following (Acuna et al. 2019; Yu et al. (20T8)) For the evaluation of edge detection, an important parameter is matching distance tolerance which is an error threshold between the prediction result and the ground truth. Prior works on edge detection such as (Acuna et al.) 2019 Hariharan et al. 2011 Yu et al. 2017 2018b) adopt non-zero distance tolerance to resolve the annotation noise. However, the proposed datasets for few-shot edge detection utilize thicker boundaries to overcome the annotation noise issue instead of adopting distance tolerance. Moreover, evaluation with non-zero distance tolerance requires additional heavy computation. This becomes more problematic under few-shot setting where the performance should be measured on the same test image multiple times due the variation in the support set. For these reasons, set distance tolerance to be 0 for both FSE-1000 and SBD-5'. In addition, we evaluate the positive predictions from the area inside an object and zero-padded region as false positives, which stricter than the evaluation protocol in prior works of (Hariharan et al., 2011
Yu et al. 2017).
5.3 IMPLEMENTATION DETAIL
We implement our framework using Pytorch library and adopt Scikit-learn library construct the precision-recall curve and compute average precision (AP). For the encoder, ResNet-34 pretrained on ImageNet adopted. All parameters except the encoder parameters are learned from scratch. The entire network is trained using the Adam optimizer of (Kingma & Bal 2014) with weight decay regularization of (Loshchilov Hutter 2017). In both experiments on FSE-1000 and SBD-5', we use learning rate of 10 and an 12 weight decay rate of 10-2 For FSE-1000 experiments, the model trained with 40,000 episodes and the learning rate decayed by 0. after training 38,000 episodes. For SBD-5' experiments, 30,000 episodes are used for training. and the learning rate is decayed by 0.1 after training 28,000 episodes. Higher shot training of (Liu et al. 2019) is employed in 1-shot experiments for both datasets.
5.3.1 DATA PREPROCESSING
During training, we adopt data augmentation with random rotation by multiples of 90 degrees for both SBD-5' and FSE-1000. We additionally resize SBD-5' data 320x320, while no such resizing is performed on FSE-1000. During evaluation, images of SBD-5' are zero-padded to 512x512. Again, the original image size is used for 1000.
5.4 EXPERIMENT RESULT
Table shows the experiment results on the SBD-5' dataset. To verify the value of the proposed method, we compare CAFENet with two baselines. The first baseline is fine-tuned edge detection model with only a few labeled samples. Meta-learning strategy is not used for the first baseline. We employ the DFF of (Hu et al., 2019), utilizing the implementation offered by the authors. For each split of SBD-5', we pretrain a 15-way semantic edge detector with training classes and fine- tune the pretrained edge detector with a few labeled samples for new classes in test split. During pretraining, follow the training strategies and hyperparameters of (Hu et al., 2019). In fine- tuning, we randomly initialize some sub-modules that are closely related to final prediction ("side5", "side5-w", and "ada-learner") and train them altogether using the support images.
7

---

Under review as a conference paper at ICLR 2021
The second baseline is constructed by combining con-parametric edge detectors such Canny detector or Sobel detector with a few-shot segmentation algorithm. The semantic edge detection is occasionally interpreted as a dual task of semantic segmentation, but prior work of (Acuna et al. 2019) verifies that the semantic edge detector outperforms the segmentator combined with the Sobel edge detector, and demonstrates the importance of semantic edge detection. In experiments, we combine PANet and PMM, the state-of-the-arr few-shot segmentation method, with the Sobel edge detector. To experiment with existing segmentation methods, we utilize the implementation provided by the authors. For each split of SBD-5', we meta-train the PANet and PMM on training classes using the segmentation labels. In evaluation, we obtain the edge predictions of test classes by applying the Sobel edge detector on the segmentation masks as done in (Acuna et al. 2019), and compare the predictions with the edge labels of test classes.
We utilize the ResNet-34 34 backbone for CAFENet and PANet. For PMM, we utilize ResNet-50 backbone. We also employ higher shot training in 1-shot experiments for both baselines as done in CAFENet experiments. The results in Table [ show that the proposed CAFENet outperforms all baselines in both MF and AP scores by significant margin. The experiment results prove that the few-shot semantic edge detector fine-tuned semantic edge detector. In Table [ the experiment results on the FSE-1000 dataset are shown. For FSE-1000, to train a semantic edge detector with training set of FSE-1000, due to the large number of training classes. We can see that the proposed CAFENct outperforms the baseline even when the dataset contains more diverse classes.
not be simply substituted by few-shot segmentator the
only experiment with the few-shot segmentation baseline since it is hard
Table I: Evaluation results of proposed CAFENet on SBD-5'. 1000 randomly sampled test episodes are used for evaluation. MF and AP scores are measured by %
Metric
Method (5-shot)
SBD-5
SBD-5'
SBD-52
SBD-5
Mean
DFF + Finetune
9.25
8.63
8.08
7.83
8.45
MF (ODS)
PANet Sobel
19.66
23.48
21.10
18.09
20.58
PMM Sobel
31.73
29.99
29.91
26.03
29.42
CAFENet (Ours)
34.71
36.81
32.02
28.37
32.98
DFF Finetune
9.17
6.77
7.04
6.14
7.28
PANet Sobel
11.91
14.13
11.92
9.42
11.85
AP
PMM + Sobel
23.26
20.76
20,38
17.94
20.59
CAFENet (Ours)
30.47
32.40
27.01
23.06
28.24
Metric
Method (I-shot)
SBD-5
SBD-5
SBD-52
SBD-5
Mean
DFF + Finetune
8.87
5.54
4.91
2.95
5.57
MF (ODS)
PANet Sobel
19.33
23.01
20.79
17.62
20.19
PMM + Sobel
31.18
29.23
29.38
25.65
28.86
CAFENet (Ours)
31.54
34.75
29.47
26.68
30.61
DFF + Finetune
7.91
3.71
3.31
1.55
4.12
PANet Sobel
12.32
14.39
12.13
9.76
12.15
AP
PMM Sobel
22.77
20.21
19.85
17.56
20.10
CAFENet (Ours)
26.81
29.08
23.77
20.44
25.03
Table 2: 1-way I-shot and 1-way 5-shot results proposed CAFENet on FSE-1000. 1000 randomly sampled test episodes are used for evaluation. MF and AP scores are measured by %
Metrie
Method
1-way 1-shot
I-way 5-shot
PANet Sohel
38.38
38.78
MF (ODS)
PMM + Sobel
36.31
36.46
CAFENet (Ours)
58.47
60.63
PANet Sobel
27.85
28.11
AP
PMM + Sobel
31.22
31.82
CAFENet (Ours)
60.54
63.92
8

---

Under review as a conference paper at ICLR 2021
6 CONCLUSION
In this paper, we establish the few-shot semantic edge detection problem. We proposed the Class- Agnostic Few-shot Edge detector (CAFENet) based on a skip architecture utilizing multi-scale fea- tures. To compensate the shortage of semantic information in edge labels, CAFENet employs segmentation module in low resolution and utilizes segmentation masks to generate attention maps. The attention maps applied multi-scale skip connection to localize the semantically related region. We also present the MSMR regularization method splitting the feature vectors and pro- totypes into several low-dimension sub-vectors and solving multiple metric-learning sub-problems with the sub-vectors. We built two novel datasets of FSE- and SBD-5' well-suited to few-shot semantic edge detection. Experimental results demonstrate that the proposed method significantly outperforms the baseline approaches relying on fine-tuning or few-shot semantic segmentation.
REFERENCES
Husein Hadi Abbass and Zainab Radhi Mousa. Edge detection of medical images using markov basis. Applied Mathematical Sciences, 11(37)1825-1883, 2017.
David Acuna, Amlan Kar, and Sanja Fidler. Devil is in the edges: Learning semantic boundaries from noisy annotations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 11075-11083, 2019.
Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural fields. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 3602-3610, 2016.
John Canny. A computational approach edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679-698, 1986.
Liang- Chieh Chen, Jonathan T Barron, George Papandreou, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with task-specific edge detection using cnns and a discriminatively trained domain transform. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4545-4554, 2016.
Liang- Chin--Chich Chen, George Papandreou, Iasonas Kokkinos. Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Ruoxi Deng, Chunhua Shen, Shengjun Liu, Huibing Wang, and Xinru Liu. Learning to predict crisp boundaries. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 562-578, 2018
Nanqing Dong and Eric Xing. Few-shot semantic segmentation with prototype learning. In BMVC, volume 3, 2018.
Mengyang Feng, Huchuan Lu, and Errui Ding. Attentive feedback network for boundary-aware salient object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1623-1632, 2019.
Vittorio Ferrari, Loic Fevrier, Frederic Jurie, and Cordelia Schmid. Groups of adjacent contour segments for object detection. IEEE transactions on pattern analysis and machine intelligence, 30(1):36-51, 2007.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.

---

Under review as a conference paper at ICLR 2021
Kun Fu, Tengfei Zhang, Yue Zhang, Menglong Yan, Zhonghan Chang, Zhengyuan Zhang, and Xian Sun. Meta-ssd: Towards fast adaptation for few-shot object detection with meta- leataleerningg. IEEE Access, 7:77597-77606, 2019.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. In 201 International Conference on Computer Vision, pp. 991-998. IEEE, 2011.
Ruibing Hou, Hong Chang, MA Bingpeng, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In Advances in Neural Information Processing Systems, pp. 4005- 4016, 2019.
Yuan Hu, Yunpeng Chen, Xiang Li, and Jiashi Feng. Dynamic feature fusion for semantic edge detection, arXiv preprint arXiv: 0.000099044 2019.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 112--1134, 2017.
Leonid Karlinsky, Joseph Shtok, Sivan Harary, Eli Schwartz, Amit Aides, Rogerio Feris, Raja Giryes, and Alex M Bronstein. Repmet: Representative-based metric learning for classification and few-shot object detection. In Proceedings of the IEEE Conference on Computer Vision and Pullern Recognition, pp. 5197-5206, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1idi::412.69900 2014.
Seong Min Kye, Hae Beom Lee, Hoirin Kim, and Sung Ju Hwang. Transductive few-shot learning with meta-learned confidence, 2020.
Yann Lifchitz, Yannis Avrithis, Sylvaine Picard, and Andrei Bursuc. Dense classification and im- planting for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9258-9266 2019.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sungju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In In- ternational Conference Learning Representations, 2019. URL hittps://openreett
net
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
171150171,.01011, 2017.
J Mehena. Medical image edge detection using modified morphological edge detection approach. 2019.
Mukta Prasad, Andrew Zisserman, Andrew Fitzgibbon, M Pawan Kumar, and Philip HS Torr. Learn- ing spesiicp edges for object detection and segmentation. In Computer Vision, Graphics and Image Processing, pp. 94-105. Springer, 2006.
Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Efros, and Sergey Levine. Conditional networks for few-shot semantic segmentation. 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv: (000:/77903///// 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Kokichi Sugihara. Machine interpretation of line drawings. The Massachusetts Institute of Tech- nology, 1986.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
10

---

Under review as a conference paper at ICLR 2021
Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot im- age semantic segmentation with prototype alignment. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9197-9206, 2019.
Shenlong Wang, Sanja Fidler, and Raquel Urtasun. Lost shopping! monocular localization in large indoor spaces. In Proceedings of the IEEE International Conference Computer Vision, pp- 2695-2703, 2015.
Ting-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8798 -8888,87700000 2018.
Tianhan Wei, Xiang Li, Yau Pun Chen, Yu- Yu-Wing Tai, and Chi- Tang. Fss- 1000: 1000-class dataset for few-shot segmentation. arXiv preprint arXiv: 12347, 2019.
Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for semantic segmentation. arXiv preprint arXiv:2008.03894 2020.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. Tapnet: Neural network augmented with task- adaptive projection for few-shot learning. arXiv preprini 1eXiiv:905.000 2019.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint (rxiiv1506.065777 2015.
Xin Yu, Sagar Chaturvedi, Chen Feng, Yuichi Taguchi, Teng-Yok Lee, Clinton Fernandes, and
Srikumar Ramalingam. Vlase: Vehicle localization by aggregating semantic edges. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3196-3203.
IEEE, 2018a.
Zhiding Yu, Chen Feng, Ming-Yu Liu, and Srikumar Ramalingam. Casenet: Deep category-aware semantic edge detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5964-5973, 2017
Zhiding Yu, Weiyang Liu, Yang Zou, Chen Feng, Srikumar Ramalingam, BVK Vijaya Kumar, and Jan Kautz. Simultancous edge alignment and learning. In Proceedings of the European Confer- ence on Computer Vision (ECCV), pp. 388-404, 2018b.
Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic segmen- tation networks with iterative refinement and attentive few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5217-5226, 2019.
Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. arXiv preprint arXiv: 1810.00991 2018.
Dongchen Zhu, Jiamao Li, Xianshun Wang, Jingquan Peng. Wenjun Shi, and Xiaolin Zhang. Seman- tic edge based disparity estimation using adaptive dynamic programming for binocular sensors. Sensors, 1874):1074, 2018
A ADDITIONAL EXPERIMENTAL SETUP
In this section, we provide detailed information about experimental setup. We adopt the ImageNet pretrained ResNet-34 with 64-125-256-512 channels for each residual block from [pytorch frame- work] as the encoder. To construct the skip architecture, we employ the bottleneck block of ResNet as the post-processing blocks S(1) 5(4), Each bottleneck block consists of two 1xl convolutional layers and one 3x3 convolutional layer with expansion rate of 4. Dropout with the ratio of 0.25 is applied to the end of each bottleneck block. For the ASPP Module in front of S(3), adopt the dilation rate of 1,4,7,11. The segmentation module generates a segmentation prediction with the rounding threshold value À of 0.4. For decoder, each decoder block is composed of three consecu- tive 3x3 convolutional layers, and dropout with the ratio of 0.25 is again located at the end of each layer.
11

---

Under review as a conference paper at ICLR 2021
During meta-training of CAFENet, set the number of query samples in training episodes to be 5 for FSE-100 and 10 for SBD-5', respectively. In evaluation, we employ average perrage-preciioo_ccee score function of Scikit-lcarn library measure the Average Precision (AP) score. We compute the AP score for each image and average them to measure the overall performance. For Maximum F- measure (MF) score, measure true positives (TP), false positives (FP) and false negatives (FN) every 0.01 threshold intervals for each image, and accumulate the values for all images in 1000 test episodes. The MF score is computed using the accumulated TP, FP, and FN values.
B ABLATION STUDIES
In this section, we shot the results of ablation experiments to examine the impact of proposed MSMR and attentive decoder. Table shows the experiment results on the PSE-1000 dataset. The base- line method in Table [B.]] J
conducts edge prediction in low resolution and utilizes the auxiliary loss from low resolution together with the loss from the edge prediction original resolution for meta- training. The edge prediction is done using metric-based method utilizing prototypes which are computed using down-sampled edge labels. The method dubbed as Seg utilizes a segmentation module without MSMR or attentive decoding. Seg directly matches high-dimensional query feature vectors with prototypes both training and evaluation. In Seg, the segmentation module is utilized only provide the segmentation loss that assists for model learning to extract semantic features. Seg + Att employs the predicted segmentation mask for the additional attention process in skip ar- chitecture. Seg MSMR + Att additionally utilizes the MSMR regularization for training. For fair comparison, all methods use the same network architecture and training hyperparameters. For SBD-5' datasets, the ablation experiments are done with same model variations as FSE-1000. The results on SBD-5' are shown in Table B.2
Table B.I: Ablation experiment results proposed CAFENet on FSE-1000. 1000 randomly sampled test episodes are used for evaluation. MF and AP scores are measured by %
Metric
Method
1-way 1-shot
1-way 5-shot
baseline
52.71
53.52
MF (ODS)
Seg Seg + Att
56.89
59.65
58.00
60.14
Seg Att + MSMR
58.47
60.63
baseline
53.66
54.59
Seg
58.80
61.87
AP
Seg + Att
59.81
62.37
Seg All + MSMR 60.54
63.92
Tables B.1 and B.2 demonstrate that the use of the segmentation module in Seg gives significant and SBD-5' datasets. It is also seen that
performance advantages over baseline for both the additional use of attentive decoding, Seg + Att, generally improves the performance over Seg. Finally, adding the effect of MSMR regularization gives substantial extra gains, as seen by the scores associated with Seg + MSMR + Att. Clearly, when compared with baseline, our overall approach Seg + MSMR Att provides large gains. the main paper, we report the results of Seg MSMR Att as the results of CAFENet.
B. ADDITIONAL EXPERIMENTS ON MULTI-SPLIT MATCHING REGULARIZATION
B.1.1 FEATURE MATCHING METHOD FOR SEGMENTATION
In Table B.3 we have compared various feature matching methods between prototypes and query feature vectors for producing segmentation prediction SBD-5'. The method baseline refers to the original method generating segmentation prediction using only the similarity metric between high-dimensional vectors done in Eq. 3 For the method average, segmentation predictions from low-dimensional feature splits (Eq. 55) and original high-dimensional feature vectors (Eq. 5)) are averaged to generate the final prediction mask. The average method can be understood method
12

---

Under review as a conference paper at ICLR 2021
Table Ablation experiment results proposed CAFENet on SBD-5'. episodes are used for evaluation. MF and AP scores are measured by %
1000 randomly sampled test
1=0
acroplane, bard.birdoo...... bottle
busscarseat.chair.com
table.dog. mbike person
sheep, sofa, train.tv
Metric
Method (5-shot)
SBD-5
SBD-5
SBD-5'
SBD-5
Mean
baseline
22.27
19.64
20.41
20.41
20.20
MF (ODS)
Seg Seg + Att
30.61
31.62
28.06
24.97
28.82
31.75
33.41
28.44
26.03
29.91
Seg Att MSMR
34.71
36.81
32,02
28.37
32.98
baseline
18.68
15.57
14.97
14.05
15.82
Seg
26.14
26.78
21.92
18.43
23.32
AP
Seg + Att
27.61
28.39
22.66
20.11
24.69
Seg + Att MSMR
30.47
32.40
27.01
23.06
28.24
Metric
Method (1-shot)
SBD-5
SBD-5
SBD-5
SBD-5
Mean
baseline
21.81
19.49
20.34
18.06
19.93
MF (ODS)
Seg
29.89
31.64
27.89
24.41
28.46
Seg Att
30.72
33.03
28.63
25.04
29.36
Seg + Att MSMR
31.54
34.75
29.47
30.61
baseline
18.11
15.47
14.73
13.89
15.55
Seg
25.16
26.15
21.52
18.52
22.84
Seg Att
26.10
27.21
22.47
18.81
23.65
Seg Alt MSMR
26.81
29.08
23.77
20.44
25.03
utilizing MSMR not only for regularization, but also for inference. In the weighted sum method, above five segmentation masks are combined using a weighted sum with learnable weights. As we can in Table B.3. the MSMR method shows the best performance when employed for regularization.
Table B.3: Comparison of different feature matching method on SBD-5' under I-way 5-shot setting. MF and scores are averaged over splits
Feature matching method
baseline
average
weighted sum
AP
34.61 29.91
31.05 26.20
31.44 26.46
MF(ODS)
B.1.2 NUMBER OF VECTOR SPLITS
MSMR divides the high-dimensional feature into multiple splits. Table B.4 shows the performance of proposed CAFENet with varying numbers of splits K. Comparing the K = case with other cases, we can see that applying MSMR regularization consistently improves performance. We can that K = 4 results in the best AP and MF performance. The performance gain is marginal when we divide the embedding dimension into too small (K 16) or too big (K 2) pieces.
Table B.4: Comparison of different numbers of vector splits on SBD-5' under 1-way 5-shot setting, MF and scores are averaged over splits
Number of splits
K
K 2
K 4
K
8
K 16
AP
24.69
26.48 31.62
29.91 32.30
27.68
23.83
MF(ODS)
29.91
31.58
30.77
13

---

Under review as a conference paper at ICLR 2021
B.1.3 RANDOM PROJECTION FOR VECTOR SPLITS
In MSMR, we divide query feature into K splits along the channel dimension, i.e. we apply deter- ministic splitting. In Table B.5 we compare the performance of different splitting methods. The method dubbed as Deterministic refers to the MSMR method that we utilize in CAFENet. The Random method randomly splits feature vectors into parts in each episode. The Baseline method does not split features at all. Interestingly, Random's 's performance significantly degrades, even below that of Baseline.
Table B.5: Comparison of different splitting methods SBD-5'. MF and AP scores measured by %
Metric
Method (5-shot)
SBD-5
SBD-5'
SBD-5'
SBD-5
Mean
Deterministic
34.71
36.81
32.02
28.37
32.98
Baseline
31.75
33.41
28.44
26.03
29.91
(ODS)
Random
23.67
17.22
21.16
20.51
20.64
Deterministic
30.47
32.40
27.01
23.06
28.24
AP
Baseline
27.61
28.39
22.66
20.11
24.69
Random
20.37
17.51
18.26
15.76
18.07
Metric
Method (1-shot)
SBD-5
SBD-5
SBD-52
SBD-5
Mean
Deterministic
31.54
34.75
29.47
26.68
30.61
MF (ODS)
Baseline
30.72
33.03
28.63
25.04
29.36
Random
23.36
17.04
21.43
20.47
20.58
Deterministic
26.81
29.08
23.77
20.44
25.03
AP
Baseline
26.10
27.21
22.47
18.81
23.65
Random
20.12
17.35
18.57
15.64
17.92
B.2 ADDITIONAL EXPERIMENTS ON SEMANTIC ATTENTION
B.2.1 EXPERIMENTS ON APPLYING THE SEMANTIC ATTENTION
In the proposed CAFENet, we utilize the semantic attention in the attentive decoder. In Table B.6 we compare three methods that utilize the attention in different manners. Attentive Decoding is our proposed CAFENet that applies the semantic attention to the multi-scale features. The second method, Direct Attention, is a method that directly passes the feature to the edge detector and applies the semantic attention to the final edge prediction. The last method, No Attention baseline where the edge detector generates prediction without any attention. For both 1-shot and 5-shot settings, regardless of how the attention is applied, semantic attention considerably improves performance. These results show the effectiveness of semantic attention. The results also show that the proposed approach of Attentive Decoding yields better results compared to Direct Attention.
B.2.2 EXPERIMENTS ON GENERATING THE ATTENTION
In the proposed attentive decoding, we can generate an attention map using various methods. In [..1 compare two different methods to generate the attention map. Segmentation Atten- tion is the method adopted in proposed CAFENet. segmentation module S is utilized as the M equation 7 In Edge Attention, the edge prediction E(S) is generated from segmentation mask by equation [] following (Feng et al. (2019). The generated edge prediction E(S) is then used as the M in equation 7 Experiment results show that utilizing the segmentation mask to generate the attention map performs better than utilizing the edge prediction.
Segmentation Attention, the output of the
E(S) = IS - AvoPool(S))
(1I)
14

---

Under review as a conference paper at ICLR 2021
Table B.6: Comparison different methods to apply attention on SBD-5'.
Metric
Method (5-shot)
SBD-5"
SBD-5'
SBD-5"
SBD-5*
Mean
Attentive Decoding
34.71
36.81
32.02
28.37
32.98
MF (ODS)
Direct Attention
33.47
36.64
31.85
28.28
32.56
No Attention
31.75
33.41
28.44
26.03
29.91
Attentive Decoding
30.47
32.40
27.01
23.06
28.24
AP
Direct Attention
30.31
31.93
26.15
22.59
27.75
No Attention
27.61
28.39
22.66
20.11
24.69
Metric
Method (1-shot)
SBD-5"
SBD-5'
SBD-3"
SBD-5*
Mean
Attentive Decoding
31.54
34.75
29.47
26.68
30.61
MF (ODS)
Direct Attention
31.27
34.53
30.01
26.41
30.56
No Attention
30.72
33.03
28.63
25.04
29.36
Attentive Decoding
26.81
29.08
23.77
20.44
25.03
AP
Direct Attention
26.46
28.61
23.64
20.23
24.74
No Atlention
26.10
27.21
22.47
18.81
23.65
Table B.7: Comparison of different methods generate attention SBD-5'.
Metrie
Method (5-shot)
SBD-5"
SBD-5'
SBD-5*
SBD-5
Mean
MF
Segmentation Attention
34.71
36.81
32.02
28.37
32.98
(ODS)
Edge Attention
33.83
37.01
31.14
27.92
32.48
Segmentation Attention
30.47
32.40
27.01
23.06
28.24
AP
Edge Attention
29.16
32.35
25.56
22.52
27.40
Metric
Method (1-shot)
SBD-5
SBD-5
SBD-52
SBD-5
Mean
MF (ODS)
Segmentation Attention
31.54
34.75
29.47
26.68
30.61
Edge Atlention
30.25
33.58
30.64
26.16
30.16
Segmentation Attention
26.81
29.08
23.77
20.44
25.03
AP
Edge Attention
25.21
27.83
24.87
20.34
24.56
C QUALITATIVE RESULTS
In Figures C.I and C.2 visualize the qualitative results of CAFENet for FSE-1000 and SBD- 5', respectively. We can see that the proposed CAFENet successfully detect the edge of the target class from given images. In Figure C.I we can compare the qualitative results of edge predictions from different methods. We can see that the 'DFF + Finetune' method succeeds in finding the edges of the objects, but lacks the ability learn semantic information and hardly distinguish the boundary of target class from the other boundaries. For "PANet Sobel' method, on the other hand, successfully understands the semantic information and localizes the target object, but fails to refine the correct boundary. The proposed CAFENet, however, is capable of localizing the objects from target class and detecting the correct boundary at the same time.
Figure C.3 visualizes more qualitative results on SBD-5' from ablation experiments. We illustrate and compare the boundary prediction results the baseline, Seg. Seg + Att, and Seg + Att + MSMR methods. For fair comparison, all methods share the same support set. From the results, we can clearly see that the techniques proposed in CAFENet steadily improve the quality of edge prediction.
15

---

Under review as a conference paper at ICLR 2021
image
ground truth
DFF + +inetume
PANet + Sobel
CAFENet
Figure C.I: Qualitative examples of 5-shot edge detection SBD-5' dataset.
D LABEL GENERATOR
D. EDGE LABEL GENERATOR
Algorithm D.I generates the edge labels from the segmentation labels. The edge label generator finds the regions where the pixel value of segmentation label drastically changes, and determine the pixels in the regions as the boundary. Note that the pixels at the border of the image are also determined as the boundary.
16

---

and record the location of foreground pixels and mark the foreground Under review as a conference paper at ICLR 2021
image
ground truth
PANet
PANet Sobel
CAFENet
Figure C.2: Qualitative examples of 5-shot edge detection FSE-1000 dataset.
D.2 SEGMENTATION LABEL GENERATOR
Algorithm D.2 is the segmentation label generator which generates the segmentation label from the edge label. Before the label generation, the pixels are divided into several groups based on boundary labels. We employ the Breadth-Finss Search (BFS) algorithm and divide pixels into groups {G*, G ......, Gm). }. The segmentation label generator of Algorithm D.2 classifies these groups into foreground and background. First, the algorithm sweeps each column and row to count the number of pixel value change in edge label. there are certain numbers of changes, the algorithm again sweeps the column pixels in the column or row in a matrix T.. Based on pixel groups {G' G² G²,... Gm), the marking results in T are then divided into pixel value groups {T1, T ..., T The probability that each
17

---

Under review as a conference paper at ICLR 2021
Card
Gni
a
un
imaaee
ground truth
baseline
Sag
Seg +Att
Seg +Att +MMM
Figure C.3: Qualitative results comparison of ablation experiments
18

---

Under review as a conference paper at ICLR 2021
Algorithm D.1 Edge Label Generation
Input: Segmentation label M of an image
Output: Edge label of an imgg.
<- Ow*# for (h, w) in (1, (I.1)...(HH.W)) do
Initialize zero matrix having same shape with M > H/W is height/width of the image
if MM.w = 1 then
for (a, b) = (+r,-r)....(1:r)) do if tbb = 0 then
radius determines thickness of edge
Uh,w +
D 0/1 means non-edge/edge pixel, respectively
break
else if (/ + < 0) or (h + > H) or (w + b < 0) or (w + > W) then
1/h +
break
return y/
Return label annotation
image
segmentation label
edge label
image
segmentation label
edge label
Figure D.1: Result of edge label generator.
group belongs to the foreground is calculated as the mean of pixel values T The groups with probability above the threshold À determined the foreground groups, and the pixels belongs to foreground groups are marked as foreground pixels. We set the threshold value À to be 20/ 255..
image
edge label
marking result
pixel groups
segmentation label
Figure D.2: Result of segmentation label generator.
E DETAILS ON DATASETS
E. FSE-1000
We build FSE-1000 using existing few-shot segmentation dataset, FSS-1000. We extract the boundary labels from segmentation annotations using Algorithm [.]] The radius value r in Algo- rithm D.I is set to in FSE-1000, 1000 categories in FSE- 1000 are split into 800 train classes and 200 test classes. For the detailed class configuration, the reader may refer to our attachment on class configuration. Figure D.I visualizes the result of our edge extraction algorithm.
E.2 SBD-5'
SBD-5' is constructed based on the existing semantic edge detection dataset (SBD). Due to the noise of boundary annotations in original SBD, we utilize the thicker edge as done in PSE-1000 To extract thicker edge, generate the segmentation labels from the edge labels using Algorithm D.2 D.2
instead of using existing segmentation labels of SBD. Figure [D2 shows the process generating the segmentation label from the edge label. From the generated segmentation labels, extract edge
19

---

Under review as a conference paper at ICLR 2021
Algorithm D.2 Segmentation Label Generation
Input: Edge label of an image, pixel groups G' "'....,'** G"
Output: Segmentation label M of image. Initialize M, T zero matrix having same shape with
M, Ow.# for = I....,Ho do
> H is height of the image
ent, mode f- for I....WWW do
D W is width of the image Accumulate changes of pixel value
yh, = mod + 1,2) then cnt +- ent +1
mode + mod + 1,2)
if mod(cnt, 4) and ent + 0 then ent', mode' +- for w' = W.....WW do
Check if there are FG pixels in the row
> Find location of FG pixels in the row
if yh, = modem + +, then enl' &- end' +1 mode' <- moddmode + 11 if mod(cnt', = 2 then
Th, +1
Record location of FG pixels the row Repeat the same process for every column
for === l...,W do
ent, mode t for h= I.....H do
if Yh mod(m + 1, then end end + mode + mod + 1,2)
if mod 4) and ent * 0 then ent', mode' + for h ..... do if yb w mnod(mode. + 1,2) then ent' +- ent' + mode' + mod(mode + 2, if mod(cnt', 4) 2 then Tw for i 1, 1,..., n do
T + TT Th. if mean(T") 2 A then M (h.wet
Check the probability that G¹ belongs to foreground
means foreground pixel
else
Mn.w((hw)eG +
O means background pixel
return M
Return segmentation annotation
labels using Algorithm D.I with radius value of 4. This process allows us to train the proposed CAFENet using only the edge labels.
While all images in FSE-1000 have the same size, images in SBD-5' have different size. However, constructing training episode as a mini-batch requires images with the same size. Previous works on semantic edge detection typically apply random cropping to deal with this issue. For the few-shot setting, however, random cropping severely degrades informativeness of support set and consequently hinders learning. Alternatively, we utilize the training examples resized to 320 320 to maintain the information of images as much as possible. When resizing the edge labels for training. first generate segmentation labels in original scale using Algorithm D.2 and resize the segmentation labels to 320 x 320. Then, we extract edge labels from resized segmentation labels using Algorithm D.I with radius value 3.
20

---

Under review as a conference paper at ICLR 2021
F ADDITIONAL RESULTS WITH MULTI-ANGLE INPUT TEST
In this section, we report the few-shot semantic edge prediction results with multi-angle input test. In multi-angle input test, the model predicts the edge by averaging the 4 edge prediction results from 4 copies of an input image rotated by multiples of 90 degrees. We have empirically found that multi- angle imput test significantly improves the performance. Table and2222how the evaluation results with multi-ange input test for FSE- 1000 and SBD-5', respectively. We can verify the effectiveness of the multi-angle input test from the results.
Table F.I: 1-way 5-shot results with the multi-angle input FSB-1000. 1000 randomly sampled episodes are used for evaluation, MF and AP scores are measured by
Metric
Method
1-way 5-shot
baseline
55.23
MF (ODS)
Seg Seg + Att
61.17
61.90
Seg Alt MSMR
62.23
baseline
56.12
Seg
63.32
AP
Seg + Att
63.83
Seg Att MSMR
65.81
Table F.2: 1-way 5-shot results with the multi angle input test on SBD-5' are used for evaluation. MF and AP scores are measured by
1000 randomly sampled test episodes
Metric
Method(5-shoo)
SBD-50
SBD-5!
SBD-52
SBD-53
Mean
baseline
24.38
23.78
22.81
20.39
22.84
MF (ODS)
Seg Seg + Att
32.46
33.18
29.63
26.46
30.43
34.29
35.76
32.25
28. 12
32.61
Seg + Att + MSMR
36.13
37.92
34.18
30.21
34.61
bascline
21.03
20.39
17.48
16.31
18.80
Seg
27.92
28.00
23.94
20.06
24.98
AP
Seg Att Seg Att + MSMR
29.94
30.40
24.34
21.38
26.52
31.92
33.51
29.28
24.91
29.91
21


Reviews:
[{"review_id": "llmkKwlfhuo", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper introduces a novel problem of \"few-shot semantic edge detection\" where semantic boundaries are to be learned/detected with a few labeled samples. In order to remedy the issue of label sparsity within the few-shot scenario, the authors have leveraged the use of the segmentation process which provides the semantic information to the edge detector. They have also incorporated a meta-learning approach, namely Multi-Split Matching Regularization (MSMR), to avoid the overfitting when high-dimensional embeddings are used for feature matching.\n\nPros:\n- The motivation was effectively described and the problem was well-defined which made it easy to read. \n- Proposed architecture smartly inherits many previously introduced components (e.g., prototype-based metric learner)and combines them into a well-working framework to train a few-shot edge detector.\n- Quantitative results show reasonable edge detection performance\n\nCons/Questions:\n- The authors constructed the baseline by combining PANet (ICCV'19) with the Sobel edge detector while disregarding some of the better/recent approaches shown below:\n*  Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. (CVPR'19)\n** Prototype Mixture Models for Few-shot Semantic Segmentation (ECCV '20)\n*** Part-aware Prototype Network for Few-shot Semantic Segmentation (ECCV '20)\nWould the proposed approach still be effective than the combined version of Segmentators+Sobel if the segmentors were updated to more recent models?\n\n- Where does the proposed approach fit w.r.t the previous approaches mentioned in Section 2.2 and 2.3?\n- Theoretical or analytical reasoning to support the effectiveness of MSMR would better convey the authors' claim\n\nMinor comments:\n- There might be some incorrect labels in Figure 3. In this figure, the diagram on the left is not supposed to be split-wise matching, right?", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZYNGTl4rM3T", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1 This work is well written and easy to understand.\n2 Extensive ablation experiments are performed to verify the effectiveness of the proposed modules in the CAFENet.\n\nCons:\n1 This new task is very similar to the few-shot segmentation. What are the advantages of this task over few-shot segmentation? The significance of this new research is doubtful. \n\n2 The proposed method was named “class-agnostic”. However,  it seems the basic setting is exactly the same as few-shot segmentation, i.e., if I want to segment the horse in the given query, I need to provide a hose image and its mask. So, I think the method is class-aware rather than class-agnostic.\n\n3 In my opinion, the main technical contribution of this work lies in the split-wise matching, which is employed to produce better segmentation masks. It is hard to find novel/specific designs in the edge detector part. So, I consider that the performance will significantly rely on the predicted segmentation masks, making this new task back to few-shot segmentation. \n\n4 Using PANet + Sobel as a baseline is unfair. Authors should replace the few-shot segmentation branch with PANet and more latest advanced few-shot segmentation methods for comparison. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nn7zQYkbTN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the few-shot edge detection task, which is similar to few-shot segmentation but for the dual task of detecting semantic edges. For the task, the authors construct datasets and experimental settings constructed from existing edge detection dataset (BSD) and a few shot segmentation dataset (FSS). For the proposed method, the authors:\n\n1) Propose a meta-learning strategy based on prototypical learning and using intermediate low-res segmentation prediction to generate attention maps for few shot edge detection\n2) Propose using a multi-split matching which performs prototypical matching (for fg and bg) using many splits of the high dimensional feature\n\nThis paper is the first to propose the task of few shot semantic edge detection (dual task of few shot segmentation, which has been explored before). This is a relevant problem, and is well suited for few shot tasks instead of segmentation since learning a boundary can be easier (modelling high frequency info) as compared to modelling interior pixels (often containing low frequency info) that has to be done in few shot segmentation. The method section is also clearly written. \n\nI have **two major issues** with the paper in its current form:\n\n1. Results of PANet + Sobel filter: \n- The dataset constructed in this paper thicken edges by a certain radius (as explained in the paper and in the supplementary material) to reduce the time complexity of computing ODS. Quote - \"Moreover, evaluation with non-zero distance tolerance requires additional heavy computation. This becomes more problematic under few-shot setting where the performance should be measured on the same test image multiple times due to the variation in the support set\". \n- However, PANet + Sobel filter would only generate thin edges and there is no mention in the paper of the authors thickening its output edges according to the dataset (whereas thick edges are used to train CAFENet, so it has learned to produce thicker edges). Quote - \"For each split of SBD-5i, we meta-train the PANet on training classes using the segmentation labels. In evaluation, we obtain the edge predictions of test classes by applying the Sobel edge detector on the segmentation masks as done in (Acuna et al., 2019), and compare the predictions with the edge labels of test classes\"\n\nThis evaluation mismatch is a serious flaw and I would love to either get an explanation for why this is actually correct or updated results on these.\n\nGenerally, my view on this is that the evaluation should follow previous methods and should be done only on *exact* ground truth edges. Since exact ground truth edges are unavailable, using a error threshold makes sense. Evaluating with thicker edges as ground truth unfairly up-scores methods that generate thicker edges themselves, which are definitely not desirable. Therefore, I don't think the dataset with thickened edges as proposed is viable to benchmark improvements in the future for few-shot edge detection. \n\n2. Multi-Split Matching Regularization - The proposed MSMR technique is well ablated for the different choices of its hyperparameters, but at the end of the day is still an ad-hoc technique. What other design choices could be used? The proposed splitting of the vectors into smaller vectors is basically a fixed set of projections from the original feature space to a lower dimension where the different projections discard a lot of dimensions. Is this the best choice? There is a lot of theory behind random projections -- how would they work here? Instead of using K splits into N/K size, would K random projections into N/K size perform better? I believe this needs to be ablated more, and MSMR needs to be grounded more in existing work. \n\nOverall, I believe 1. is a serious flaw and 2. is something that would make the paper much more attractive to a reader and provide more insight into why MSMR results in additional improvements. I am rating this paper at a 4 now and hoping that my assessment of issue 1 is a mistake. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ZyHmuDF06J", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nA. Summary:\nThis paper works on few-shot semantic edge detection. Instead of dealing with the problem in a single stage, the authors decompose the problem into two stages.  First, a few-shot segmentation stage, where the foreground and the background probability are estimated via attention with the foreground and the background prototype (averaged feature vector on the foreground and the background region). Second, the feature maps from the encoder are masked by the attention map and sent to the decoder to generate the final edge-map.\nIn the experiments, the authors modify two existing datasets for evaluation.\n\nB. Strength:\n1.  The method is clearly described and easy to reproduce.\n2. Enforcing the group-wise similarity in sec.4.2 is a reasonable regularization.\n3. The two-stage decomposition separates the high-level sub-task (semantics) from the low-level sub-task (edge), which makes the training easier. \n\nC. Weakness:\n1. The attention map preserves the internal region of a class. However, this paper is about semantic edge detection, where the internal region does not contain any edge. So the author can try to process the final attention map by distance transform, which only focus on the region around the edge of the attention map and potentially further improve the results.\n2. The comparison with previous methods, especially the PANet+Sobel is unfair. The semantic response map corresponds to the attention map in the proposed method, so the author should also use sobel operator on their attention map to have a fair comparison, which can also serve as an ablation study to verify whether the attention map is one of the major contributions to the edge quality.\n3. Another ablation study worth to be investigated is to apply the attention map directly on the edge map, which will further sperate the high-level sub-task(semantic attention maps) from the low-level sub-task  (edge-detection). \n\nSome minor issues:\n1.  The attention map in Fig.2 (positive is black)is inconsistent with the rest of the paper (positive is white).\n\nD. Justification of the score:\nI vote for the current score because this paper lacks sufficient insights and novelty both algorithmically and conceptually. It also contains some issues in evaluation.\n\nE. Expectation for the rebuttal:\nI hope the authors could address my questions in C.weakness.\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces the new task of few-shot semantic edge detection by adapting existing datasets. It proposes a new method which is compared to a baseline.

Pros:
- Clear writing. 
- Extensive ablation experiments.
- Good architectural choices.

Mixed:
- The value of the new task raises a mix of opinions. For example R1 sees it as a "relevant problem, and is well suited for few shot tasks", but R2 finds is very similar to few-shot segmentation. I think a more interesting version of the problem (that would also create more separation to few-shot segmentation) would be to also consider internal edges, not just "semantic boundaries". For example the original BSDS dataset has pure edge annotations. 
- Besides the task, another novelty of the paper is the proposed multi-split matching technique, but while it is well demonstrated empirically (as backed by additional results given by authors in rebuttal), R3 would like to have seen "theoretical or analytical reasoning" and R1 says it is an "ad-hoc technique".

Cons:
- the PANet+Sobel baseline. All 4 reviewers are unhappy with this baseline: 3 of them find it unfair because of the non-standard edge thickening employed and 2 think there would be more recent and better baselines. The authors provided a rebuttal arguing that their GT edges are "not too thick to be unfair" but two of the reviewers mentioned they remained unconvinced -- R1 hopes "the authors will work on cleaner evaluation of the baseline" and R4 find the baseline "still unconvincing in the revised version".

Overall the paper would benefit from one more iteration focusing on the evaluation procedure to be convincing and impactful.


Decision: Reject