Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tackles the few-shot segmentation task through a proposed mask matching transformer (MM-Former). The MM-Former contains two parts. The first part decomposes query images into multiple segmentation proposals with a class-agnostic segmenter. The second part merges related segment proposals into final masks guided by support images. With the ResNet-50 backbone, the proposed method get competitive results on the popular COCO and PASCAL benchmarks.
strengths_and_weaknesses: ***Strengths***
1. This paper is well-organized and can be easily understood by readers. The technical details are introduced clearly.
2. The authors conducted extensive experiments on multiple benchmarks to investigate the effectiveness of different modules and designs in this paper.

***Weakness***
1. In the potential objects segmenter section, the authors just borrow the ideas from Mask2Former to generate object proposals. But in few-shot learning, the number of meaningful object parts is unknown both in the base training set and the novel test set. How does the network learn these object segmenter without any guidance? If the object segmenter is inaccurate, will it affect the subsequent query-related part merge?
2. The proposed MM-Former exploits transformers as decoders for few-shot semantic segmentation. As is known a transformer may need a lot of learnable parameters. It will be better to list the parameters of different modules in MM-Former to see how the number of learnable parameters of transformers affects the final performance.
3. In Table 2, we can find that the performance of MM-Former on PASCAL is not good as other state-of-the-art algorithms. I want to see a thorough analysis of such a phenomenon. Maybe we can know more about the characteristics of MM-Former.
questions: I am curious about whether the number of meaningful object proposals affects the final performance. There are also many other issues discussed in the weakness section.
limitations: I didn't see any obvious negative societal limitations of this work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a new two-stage framework that decouples the matching and segmentation modules for few-shot segmentation. Extensive experiments and ablation studies on COCO and Pascal datasets also verify the algorithm's effectiveness.
strengths_and_weaknesses: 1. The writing of this paper is redundant and ambiguous, and the logic is not clear.

2. There are lots of grammatical errors which need to be corrected.

3. Lack of innovation and low novelty.

4. Poor performance on the Pascal dataset even compared with the methods proposed years ago. 

5. Proposed two-stage strategy introduces a new perspective focusing on mask-level segmentation.
questions: 
-  In #40,  " Concretely, pixel-level relationships are modeled between the support and query feature maps either by attention mechanism or 4D convolutions." Using "either...or..." is too arbitrary. There are also other methods to deal with the relationship between the query and support sets, like HSNet and PGNet.



- Poor performance on the Pascal dataset even compared with the methods in the community proposed years ago. In #85, the author attributed the reason to the limited scale of the dataset, which does not convince the reviewer.



- Adding more visual illustrations will make it clearer while introducing the network structure in subsection 2.2.


- In the Introduction and Abstract sections, the author emphasizes the advantages of the algorithm in complexity. Relevant comparative experiments and explanations need to be supplemented to support the claims.


-  In #115, " We use the outputs from the last three layers in the following modules", "the last three layers" is ambiguous. Please further elaborate on the details.


- The novelty is marginal since the core idea is borrowed from MaskFormer.


- The impacts of different components brought to the model efficiency should be discussed, including the overall final model efficiency in terms of fps and model size.


- There are quite a lot of grammatical errors and ambiguous expressions. The reviewer did not try to point them all out together. Please check your manuscript carefully and correct all mistakes before submission. Typos include but are not limited to:

a. In #22, "one of a fundamental tasks" -> "one of the fundamental tasks"

b. In #22, “ has achieved grand success” -> " has achieved a grand success"

c. In #29, " segment any objects" remove "any"

d. A large number of articles are missing or used incorrectly. For example, "We refer this kind of method as 'few-to-many' matching approach." -> "We refer this kind of method as a 'few-to-many' matching approach." I won't list all since there are too many.

e. In #37, "While acceptable results obtained" -> "While acceptable results were obtained". Lack of the predicate.

f. In #51, "Rather than matching" -> "Rather than being matched"

g. In #300, ", it is not suitable for the matching problem and impair the matching performance." -> "impairs"
limitations: The stated limitation, i.e., a large gap between the oracle case, indicates room for future improvements, the reviewer still believes that broader negative impacts and limitations should be discussed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a few-shot semantic segmentation (FSSS) network based on Mask2Former [1]. By formulating the semantic segmentation as a mask classification problem like [1], the authors introduced a new matching mechanism in mask-level instead of pixel-level matching in previous FSSS methods. Additionally, they propose a feature-alignment block based on the attention mechanism to align both support and query features individually and cross-align between them. They conducted experiments on two FSSS datasets (COCO-20i and PASCAL-5i) and achieve the SOTA results.
strengths_and_weaknesses: **Strengths**
+ The structure of this paper is well-organized and easy to follow the ideas of the authors. 
+ The proposed method achieves significant improvements on the two datasets compared with other recent methods.

**Weaknesses**
+ The only novel contribution of this paper is the Feature Alignment Block (FAB) in Mask Matching Module and it significantly improves the results, the other blocks are incremental. The authors simply adapt it to Mask2Former architecture. It would be better to investigate more about the FAB in different methods (CyCTR, HSNet) to prove its effectiveness is generalizable enough.
+ The “few-to-few” matching (L53) is not new. Previous approaches ([2], [3]) have applied it to the few-shot instance segmentation task. 
The proposed method (the mask2former architecture) tends to overfit small datasets (L225), then it is not suitable for the few-shot setting.
+ Related work is not well presented. They do not provide any analysis or comparison of previous work.

questions: + Can the author add the running time, training time, and the memory consumed by the model? As the Mask Matching Module is implemented by standard transformer decoder layers, does it requires large memory and time?
+ What is the detailed implementation of the Mask Matching Module?
In Fig. 2, the final mask has the same resolution as the F2 feature map, how to upsample the mask to the original resolution of the input image?
+ Explain the self-alignment module and intuition of this. Can we replace it with the self-attention module (compare the performance and runtime)
+ Explain the use of the MLP in Eq.(4): describe the input, and output range, and how it can correct the cosine similarity vector of N potential masks. Visualize the M and M^ before and after. It helps the reader what is happening inside the block.
Why do not use dice loss in L_m (L173)
+ L180: “Meanwhile, only constrain on Sˆ pos may lead all outputs of Cross Alignment block to tend to be same”: why? explain in more detail?
+ L182: how to find the lowest IoU (ex: 2 potential masks with no overlap with GT)
+ Can the authors add more visualization of the failure cases?
+ In Eq. (4), redundant closing parenthesis ')'.

I will increase my score if the authors address all of my concerns.
limitations: No
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an alternate way of performing FSS by a 2-stage training procedure, where a network is first trained to generate a set of potential masks for the query image, followed by a second stage where another network is trained to generate features for both query and support, which are then pooled using the support masks, and the candidate query masks. From this, they find the best matching candidate masks through correlation followed by an MLP. The authors claim that the decoupling of segmentation and matching is beneficial for learning, and to back it up they show strong experimental results.
strengths_and_weaknesses: ## Major Strengths
* Novel approach
* Simple architecture (although 2-stage training makes it more complicated)

## Minor Strengths
* Good results
* Good ablations
* Code provided

## Major Weaknesses
* Concerns for test-time class leakage into the first stage of training.

## Minor Weaknesses
* Somewhat underwhelming results on PASCAL
* Even without class leakage, I'm somewhat unconvinced of the reasoning behind the improvement from 2-stage training.
* The paper is somewhat difficult to follow
* The code is hard to read and seems to include multiple mistakes

## Conclusion
Without further clarification by the authors I can not recommend the paper in its current form for acceptance, hence I've selected a rating of 4. However, I am open to increasing the rating to a 6 or 7 given that the authors address my concerns and questions.

### EDIT
After the response of the authors I am increasing my rating from a 4->6, and confidence from 3->4.

questions: ## Questions regarding configs and code
Following the authors instructions in the code readme:

> We conduct a 2-stage-training strategy. 
The Potential Objects Segmenter (POS) is trained in stage 1. The config files for PASCAL and COCO are provided at 
"configs/pascal/step1.yaml" and "configs/coco/step1.yaml" 
The Mask Matching Module (MM) is trained in stage 2. The config files are provided at 
"configs/{*datasets*}/step2_{*split*}.yaml" 
Execute this command at the root directory: 
    python train.py --config-file {*model_config*}
Note: We recorded the random_seed during training, using this random seed can fully reproduce the results in the paper.

Notably the authors only provide a single config for the POS network. In the configs it seems like there should be one POS model per fold, so this might have been a mistake.

If the stage 1 training involves a POS model which sees classes belonging to the test set of other folds, there is leakage.

## Question regarding loss for POS during stage 1
In MaskFormer, the loss is divided into classification and segmentation of all K classes. Typically in episodic FSS training the given label is only of the single mask of the target class. Is this also the case in your method, or is your segmentation loss based on all classes that appear in the image? This was not obvious to me from the paper, and I could also not really understand it from reading the code, as the code is quite complex.

## Question regarding generalization of POS
If there is no class leakage between base and novel classes during training of POS, how come the N proposals don't overfit to the base classes? In the paper it is claimed that the frozen backbone keeps the POS from overfitting, but one could easily see the proposals closely matching the base classes in feature space since the queries are learnable. Is there any mechanism to prevent this from happening?

## Question regarding motivation for 2-stage training
In the paper you claim that training end-to-end:
> When training in an end-to-end fashion, POS would be influenced by the training of
the MM module, degrading the quality of mask proposals

I'm not sure I follow the reasoning behind this. Could you provide a more thorough explanation why one would expect the masks to become worse in this case? From what I gather you seem to be saying that focusing only on the segmentation loss is beneficial, since propagating other losses could lead to worse segmentations. However, why not make a seperate mask prediction for the matching and segmentation tasks, but still corresponding to the same proposal?

## Anonymization
The provided code is not sufficiently anonymized, I found references to author home folders in the code. (I did not google the names to respect reviewing anonymity). For future submissions I recommend the authors to more thoroughly anonymize the code.


limitations: The authors' usage of an oracle model to see the performance gap was an interesting and useful way to gauge the current limitations.
review_title: null

Meta-review: metareview: This paper introduced a new matching mechanism in mask-level instead of pixel-level matching in previous few-shot segmentation methods. Additionally, they propose a feature-alignment block based on the attention mechanism to align both support and query features individually and cross-align between them. Reviewers in general agree on the novelty of the proposed approach. Some reviewers were concerned about some details of module design, and the authors have answered those satisfactorily. Finally, AC believes that this approach of producing mask proposals and matching at the mask level is a novel direction for few-shot segmentation, and recommends acceptance of the paper.
confidence: Certain
Decision: Accept