Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Unlike mainstream methods that involve intricate decoder structure, this paper proposes to revive the plain framework of feature extractor + linear classification head for few-shot segmentation (FSS). Concretely, this paper designs a framework called Feature-proxy Transformer (FPTrans), where it utilizes proxy, a vector to represent the foreground as well as the background classes. It makes queries interact with support features and use multiple background proxies through a novel prompting strategy. Extensive experiment results verify that the proposed method achieves competitive performance, which shows that it can serve as a simple yet strong baseline for FSS. 
strengths_and_weaknesses: ##### Strengths #####
1) This paper is well-organized and easy to read. Figure 2 is very clear to illustrate the whole framework. 

2) This paper incorporates prompt learning to few-shot segmentation in a clever way. The framework provides dedicated modifications to make it more suitable for FSS, for instance, multiple background prompts. 

3) The visualization results in the paper are good. 

##### Weaknesses #####
I have some questions about the experiment results. 
1) Network backbone. The comparison is conducted with different network backbones, i.e., the previous methods take ResNet-50 / 101, while the proposed method takes ViT / DeiT. I notice that the author compares PFENet and CyCTR on ViT. However, the experiments on DeiT are not included. And the comparsions on more SOTA methods, e.g., BAM, HSNet are also omitted. In addition, I think it is better to add the parameter size and FLOPS of different backbones for a clearer comparison. 
2) 1-shot performance. I notice that the proposed FPTrans shows inferior performance than BAM when taking ViT as the backbone. Can the authors have more discussions or explanations about this phenomenon? In addition, I am curious about the performance of BAM with ResNet-101, which is also an important baseline. 

Minor issues:
The text in the figures can be larger to make it clearer. 

questions: My questions and suggestions are listed in weaknesses. My main concern lies in the experiment results. 
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes the Feature-Proxy Transformer for few-shot segmentation, which utilizes the ViT as the backbone and interacts the query features with support features through prompts. Besides, this paper introduces to obtain multiple background proxies via Voronoi-based method. Experiments are conducted on Pascal-$5^i$ and COCO-$20^i$ datasets. 
strengths_and_weaknesses: 
**Strengths**
1. This paper is easy to follow and may be easy to implement.
2. This paper demonstrates that the framework with ViT backbone could achieve decent performance on both benchmarks.
3. The proposed operations are effective according to the experiments.
4. Interacting query and support features through prompts is interesting.
5. It shows an excellent transferable performance (from COCO to Pascal).

**Weaknesses**  
It is hard to tell the main contribution of this work.  
First, the idea of extracting multiple proxies (prototypes) from support features is not novel since it has been discussed in [30,52]. The only difference is that this paper applies the Voronoi-based method instead of the EM algorithm [30,52]. 

Besides, although this paper claims that the FPTrans is a plain framework, the prompt generation process is too heavy. It requires to use the entire ViT backbone to produce the prompts, which largely increases computational cost. Specifically, this additionally adds an approximate 50% computational overhead compared with the baseline, which may be unacceptable since the ViT-Base backbone is already a much larger model compared with Resnet-50/101, but the overall framework can only obtain comparable results with previous approaches with ResNet-50 backbone. Besides, this paper misses the ablation of only using the learnable prompts.

questions: In the rebuttal, I would like to see more discussion about the main contribution of this work and the analysis of the prompts. From my perspective, the Pair Loss and Multiple Proxies are minor contributions that are effective but lack novelty. The Prompts for FSS is novel, but the absolute gain (considering the computational cost) is relatively small. It would be better to include the mentioned ablation study, and also the quantitative computational cost comparison (*eg.*, inference time, flops et. al.) in the rebuttal.
limitations: Limitations are disscussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a different approach to FSS, whereby instead of independently processing and matching support and query samples, the support samples are processed to generate prompts, which are then concatenated with both query and support tokens. Importantly these prompts act as message passers between the support and query feature extractors. Finally, proxies are generated and the query tokens classified. The authors claim that this approach has multiple benefits,
1) Architecture is simplified, compared with decoder architectures
2) Performance is on par with or greater than previous approaches
strengths_and_weaknesses: ## Major Strengths
* Novelty, specifically the prompting approach and message passing for conditioning has to my knowledge not previously been done in FSS. I think the approach is interesting and may yield further insights.

## Minor Strengths
* Paper is clearly written and easy to follow.
* Extensive experiments, in particular ablations.

## Major Weaknesses
* It is not easy to judge whether the improvements come from the proposed method, as the backbones have been changed. The authors have provided reimplementations of previous methods using the ViT backbone, however one has to assume that such a change may require finetuning (e.g. learning rates) for those previous methods and hence the comparison is not simple to make. Judging from vision transformers success in other related tasks, such as segmentation and detection, I believe there is a large risk that most (if not all) gains can be attributed to the change of backbone.

## Minor Weaknesses
* While the authors argue that their approach is simpler than previous decoder-based methods, I do not personally agree. To me, the architecture seems quite involved, in particular the need for a first proxy generation step, separate from the rest.
* Code not provided, however, promised to be released.
* Performance gains relative to recent work are not significant, in particular, compared to BAM.

## Conclusions
While there are potential issues with where the performance gains come from, the authors also do not claim this as their main contribution. Instead, it is to be viewed as an alternate way to perform FSS. I think the paper is interesting enough to warrant acceptance, despite my concerns. 
questions: * Is there a particular reason that you chose not to use a decoder? In my experience, using a simple feature + conditioning based decoder on the predictions can increase segmentation accuracy by quite a large amount. I do not agree that refraining from any upsampling is a simplification of the method, but I'm open to hear your reasoning.

limitations: The authors addressed the issue of having to use a separate network for the prompt generation.
review_title: null

Meta-review: metareview: This paper studies the plain segmentation framework (feature extractor + linear classification) for few-shot segmentation. It introduces a prompt based query and support interaction method to enable this framework to work well. All the reviewers recognize the proposed method is novel and the performance is good. Though they have some concerns on the computational cost and the fairness of the experiment comparison (e.g., whether they use the same backbone), the authors address these concerns well in their response. All the reviewers agree with accepting this submission. Although their ratings are not very strong supportive, AC agrees this submission brings some values to the community. It inspire some new thinking of the FSS framework design. The overall framework is still heavy. Hopefully, in the future follow up works, the framework can be further simplified.  
confidence: Less certain
Decision: Accept