Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
A. Summary:
This paper works on few-shot semantic edge detection. Instead of dealing with the problem in a single stage, the authors decompose the problem into two stages.  First, a few-shot segmentation stage, where the foreground and the background probability are estimated via attention with the foreground and the background prototype (averaged feature vector on the foreground and the background region). Second, the feature maps from the encoder are masked by the attention map and sent to the decoder to generate the final edge-map.
In the experiments, the authors modify two existing datasets for evaluation.

B. Strength:
1.  The method is clearly described and easy to reproduce.
2. Enforcing the group-wise similarity in sec.4.2 is a reasonable regularization.
3. The two-stage decomposition separates the high-level sub-task (semantics) from the low-level sub-task (edge), which makes the training easier. 

C. Weakness:
1. The attention map preserves the internal region of a class. However, this paper is about semantic edge detection, where the internal region does not contain any edge. So the author can try to process the final attention map by distance transform, which only focus on the region around the edge of the attention map and potentially further improve the results.
2. The comparison with previous methods, especially the PANet+Sobel is unfair. The semantic response map corresponds to the attention map in the proposed method, so the author should also use sobel operator on their attention map to have a fair comparison, which can also serve as an ablation study to verify whether the attention map is one of the major contributions to the edge quality.
3. Another ablation study worth to be investigated is to apply the attention map directly on the edge map, which will further sperate the high-level sub-task(semantic attention maps) from the low-level sub-task  (edge-detection). 

Some minor issues:
1.  The attention map in Fig.2 (positive is black)is inconsistent with the rest of the paper (positive is white).

D. Justification of the score:
I vote for the current score because this paper lacks sufficient insights and novelty both algorithmically and conceptually. It also contains some issues in evaluation.

E. Expectation for the rebuttal:
I hope the authors could address my questions in C.weakness.




review_title: A straghitfoward combination of few shot segmentation+edge detection 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Pros:
1 This work is well written and easy to understand.
2 Extensive ablation experiments are performed to verify the effectiveness of the proposed modules in the CAFENet.

Cons:
1 This new task is very similar to the few-shot segmentation. What are the advantages of this task over few-shot segmentation? The significance of this new research is doubtful. 

2 The proposed method was named “class-agnostic”. However,  it seems the basic setting is exactly the same as few-shot segmentation, i.e., if I want to segment the horse in the given query, I need to provide a hose image and its mask. So, I think the method is class-aware rather than class-agnostic.

3 In my opinion, the main technical contribution of this work lies in the split-wise matching, which is employed to produce better segmentation masks. It is hard to find novel/specific designs in the edge detector part. So, I consider that the performance will significantly rely on the predicted segmentation masks, making this new task back to few-shot segmentation. 

4 Using PANet + Sobel as a baseline is unfair. Authors should replace the few-shot segmentation branch with PANet and more latest advanced few-shot segmentation methods for comparison. 


review_title: This work poses a new few-shot learning task, i.e., few-shot semantic edge detection, and proposes a Class-Agnostic Few-shot Edge detection Network to tackle this problem. In addition, two new datasets, FSE-1000 and SBD-5i, were introduced to evaluate the performance.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper introduces a novel problem of "few-shot semantic edge detection" where semantic boundaries are to be learned/detected with a few labeled samples. In order to remedy the issue of label sparsity within the few-shot scenario, the authors have leveraged the use of the segmentation process which provides the semantic information to the edge detector. They have also incorporated a meta-learning approach, namely Multi-Split Matching Regularization (MSMR), to avoid the overfitting when high-dimensional embeddings are used for feature matching.

Pros:
- The motivation was effectively described and the problem was well-defined which made it easy to read. 
- Proposed architecture smartly inherits many previously introduced components (e.g., prototype-based metric learner)and combines them into a well-working framework to train a few-shot edge detector.
- Quantitative results show reasonable edge detection performance

Cons/Questions:
- The authors constructed the baseline by combining PANet (ICCV'19) with the Sobel edge detector while disregarding some of the better/recent approaches shown below:
*  Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. (CVPR'19)
** Prototype Mixture Models for Few-shot Semantic Segmentation (ECCV '20)
*** Part-aware Prototype Network for Few-shot Semantic Segmentation (ECCV '20)
Would the proposed approach still be effective than the combined version of Segmentators+Sobel if the segmentors were updated to more recent models?

- Where does the proposed approach fit w.r.t the previous approaches mentioned in Section 2.2 and 2.3?
- Theoretical or analytical reasoning to support the effectiveness of MSMR would better convey the authors' claim

Minor comments:
- There might be some incorrect labels in Figure 3. In this figure, the diagram on the left is not supposed to be split-wise matching, right?
review_title: a problem of few-shot semantic edge detection introduced

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes the few-shot edge detection task, which is similar to few-shot segmentation but for the dual task of detecting semantic edges. For the task, the authors construct datasets and experimental settings constructed from existing edge detection dataset (BSD) and a few shot segmentation dataset (FSS). For the proposed method, the authors:

1) Propose a meta-learning strategy based on prototypical learning and using intermediate low-res segmentation prediction to generate attention maps for few shot edge detection
2) Propose using a multi-split matching which performs prototypical matching (for fg and bg) using many splits of the high dimensional feature

This paper is the first to propose the task of few shot semantic edge detection (dual task of few shot segmentation, which has been explored before). This is a relevant problem, and is well suited for few shot tasks instead of segmentation since learning a boundary can be easier (modelling high frequency info) as compared to modelling interior pixels (often containing low frequency info) that has to be done in few shot segmentation. The method section is also clearly written. 

I have **two major issues** with the paper in its current form:

1. Results of PANet + Sobel filter: 
- The dataset constructed in this paper thicken edges by a certain radius (as explained in the paper and in the supplementary material) to reduce the time complexity of computing ODS. Quote - "Moreover, evaluation with non-zero distance tolerance requires additional heavy computation. This becomes more problematic under few-shot setting where the performance should be measured on the same test image multiple times due to the variation in the support set". 
- However, PANet + Sobel filter would only generate thin edges and there is no mention in the paper of the authors thickening its output edges according to the dataset (whereas thick edges are used to train CAFENet, so it has learned to produce thicker edges). Quote - "For each split of SBD-5i, we meta-train the PANet on training classes using the segmentation labels. In evaluation, we obtain the edge predictions of test classes by applying the Sobel edge detector on the segmentation masks as done in (Acuna et al., 2019), and compare the predictions with the edge labels of test classes"

This evaluation mismatch is a serious flaw and I would love to either get an explanation for why this is actually correct or updated results on these.

Generally, my view on this is that the evaluation should follow previous methods and should be done only on *exact* ground truth edges. Since exact ground truth edges are unavailable, using a error threshold makes sense. Evaluating with thicker edges as ground truth unfairly up-scores methods that generate thicker edges themselves, which are definitely not desirable. Therefore, I don't think the dataset with thickened edges as proposed is viable to benchmark improvements in the future for few-shot edge detection. 

2. Multi-Split Matching Regularization - The proposed MSMR technique is well ablated for the different choices of its hyperparameters, but at the end of the day is still an ad-hoc technique. What other design choices could be used? The proposed splitting of the vectors into smaller vectors is basically a fixed set of projections from the original feature space to a lower dimension where the different projections discard a lot of dimensions. Is this the best choice? There is a lot of theory behind random projections -- how would they work here? Instead of using K splits into N/K size, would K random projections into N/K size perform better? I believe this needs to be ablated more, and MSMR needs to be grounded more in existing work. 

Overall, I believe 1. is a serious flaw and 2. is something that would make the paper much more attractive to a reader and provide more insight into why MSMR results in additional improvements. I am rating this paper at a 4 now and hoping that my assessment of issue 1 is a mistake. 
review_title: New task of Few-Shot Edge Detection along with a proposed method CAFENet

Meta-review: The paper introduces the new task of few-shot semantic edge detection by adapting existing datasets. It proposes a new method which is compared to a baseline.

Pros:
- Clear writing. 
- Extensive ablation experiments.
- Good architectural choices.

Mixed:
- The value of the new task raises a mix of opinions. For example R1 sees it as a "relevant problem, and is well suited for few shot tasks", but R2 finds is very similar to few-shot segmentation. I think a more interesting version of the problem (that would also create more separation to few-shot segmentation) would be to also consider internal edges, not just "semantic boundaries". For example the original BSDS dataset has pure edge annotations. 
- Besides the task, another novelty of the paper is the proposed multi-split matching technique, but while it is well demonstrated empirically (as backed by additional results given by authors in rebuttal), R3 would like to have seen "theoretical or analytical reasoning" and R1 says it is an "ad-hoc technique".

Cons:
- the PANet+Sobel baseline. All 4 reviewers are unhappy with this baseline: 3 of them find it unfair because of the non-standard edge thickening employed and 2 think there would be more recent and better baselines. The authors provided a rebuttal arguing that their GT edges are "not too thick to be unfair" but two of the reviewers mentioned they remained unconvinced -- R1 hopes "the authors will work on cleaner evaluation of the baseline" and R4 find the baseline "still unconvincing in the revised version".

Overall the paper would benefit from one more iteration focusing on the evaluation procedure to be convincing and impactful.


Decision: Reject