Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a model to facilitate the few-shot semantic segmentation task by utilizing pixel-wise relationships between support and query images. The authors use a cycle-consistent attention mechanism to perform cross-attention between features from support and query images. Experiments on the two commonly used benchmarks validate the effectiveness of the proposed method.
main_review: Pros:

* The idea of considering pixel-wise relationships between support and query images is interesting.
* The paper is well-written, easy to follow.
* Extensive experiments with adequate discussions.
* The proposed method achieves competitive results on two benchmarks.

Cons/Questions:

* I'm confused why the model is designed for the pixel-wise relationship. From the network structure in the supplementary, the input of the cycle-consistent transformer is the features combined with the class prototype, which is obtained by the masked average pooling method. The prototype is class-level, right? Why called the cycle-consistent transformer considers the pixel-wise relationship?
* The whole CyCTR module consists of stacking L encoders. What's the setting of L in the reported experiments? 
* The method seems to be computationally a little expensive. Did the authors analyze the efficiency of the proposed method?
* Current benchmark is established on the one-way task, which is a binary mask prediction. Can the proposed method be applied to multi-way semantic segmentation tasks?

typo: L210 "... offeset from each piexl and P represents" should be pixel
limitations_and_societal_impact: Based on my knowledge, this paper poses no negative societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the few-shot semantic segmentation problem with cycle-consistent transformer (CyCTR) that utilizes pixel-wise support information for query prediction. The CytCTR estimate the cross-attention between support and query images with the cross-consistent attention mechanism. The proposed method achieves the start-of-the-art performance on multiple datasets.
main_review: The idea of pixel-level cycle consistency shares similarities with [1], nonetheless, it is still interesting to see it is utilized in the attention map of the transformer modules.

1) From reviewer's understanding, when training, the attention weights of those pixels that are not “cycle-consistent” are added with a minus infinity and can be seen as disabled as they are normalized as zero after the softmax. As I do not see any explicit measures like loss function that guide the pixels to be “consistent”, this reminds me of dropout, which randomly disables a part of the network. Therefore, it is desirable to see more training details, e.g., is the percentage of pixels that are “inconsistent” decreases during the training process and reaches a low level in the late stage of the training? Any visualizations about the attention maps that can illustrate how pixels are “consistent”?

2) There may exist "misplaced mapping". For example, the nearest point of "cow" in query belongs to "horse" in support and then the nearest point of "horse" in support belongs to "cow" in query, which is actually inconsistent but will be viewed as "consistent" in this work.

3) Though the part of “Mask guided sparse sampling” shown effectiveness in the results, it lacks ablation study or comparison with other methods.

4) Some technical details are missing: e.g. when stacking L encoders, is the output of the previous encoder serves as the Z_kv of the next layer? Is X_s is inputted into each layer of the encoders? And some parts of the paper are not easy to follow: e.g. in Fig. 3, same symbols (Z_kv, Z_q) are used for inputs to two blocks (Cross-alignment, Self-alignment), which sounds like they were identical.

5) Another concern is the parameter size. The ResNet-101 is already a big model, and the transformer is also known for its resource requirements. Authors trained this few-shot model with 4 V100s, using much more resources than previous few-shot segmentation methods.

[1] Kang, Guoliang, et al. “Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 3569–3580.

limitations_and_societal_impact: Good for me.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper introduces a new transformer architecture designed to tackle image segmentation in the few-shot setting. The new architecture seems to make it easier for the pre-trained model to generalise to new classes and produce more accurate segmentation masks. The comparison with benchmarks shows incremental improvements on PASCAL-5i and COCO-20i in 1-shot and 5-shot setting, with two different convolutional backbones.

Due to minor yet numerous issues the paper is currently below the acceptance bar, but I am willing to improve my score if these are addressed.
main_review: The proposed method is novel and is targeted at improving the cross-attention mechanism so that inadequate keys are not attended to. Few-shot segmentation is an important problem in machine vision both application- and research-wise as fully labelled data is scarce and quick generalisation to new classes in general remains difficult for machine learning systems.

Details:
1. The proposed cycle-consistent transformer, for a given query image token, attends only to those support tokens that satisfy the cycle-consistency rule: support token i and query token j are in cycle consistency if j has the highest affinity value (QK^T attention matrix) with i among all query tokens and vice versa. This seems to be a 'hard' attention, in contrast to the softmax attention in popular transformers. Authors could potentially provide more intuition on that, for example, what happens if for a given query token there is no support one that satisfies cycle-consistency - in such a case, aren't all attention weights equal to 0, due to additive bias?
2. The method requires a vision-backbone model such as ResNet. It is mentioned that this was pre-trained, but it is unclear if that part was also fine-tuned together with CyCTR, or remained fixed? Also, the scores obtained with ResNet-101 are often worse than those with ResNet-50, in particular for 1-shot segmentation, what could that be attributed to?
3. It is unclear which 'middle-level features' and 'high-level features' (lines 233-234) are used in case of each backbone. Also it is unclear what the 'prior map' refers to.
4.  line 239 is unclear: 'As for the baseline of our method, we use two residual blocks [10] instead of the transformer to encode query feature' -  How residual blocks replace cross-attention? It should be marked in Table 4 that the first line refers to that benchmark model. Also, language errors. 
5. (246) 'during test, 1000 episodes' - what are episodes here? unclear
6. Figure 4 - if the 'query' column shows the ground truth segmentation, maybe it should be marked.
7. What are the different stacked encoders that are mentioned in 4.4.2?
8. There are multiple typos, language errors and unclear formulations:
- 59: to encoder -> to encode
- 105: point -> pointed
- 113: are -> were
- 128: ideology -> idea
- 234: 'high-level feature are used to generate a prior map that concatenated with the middle level feature' - unclear
- 237: restuls -> result
- 238: convolution -> convolutional
- 238: is consisted by -> consists of
- 245: classes is used -> classes are used
- 246: during test -> at test time
- 253 training *for*  50 epochs
- 263: the number of head for all attention layers are set -> the number of heads for all attention layers is set
- 306: one may curious








limitations_and_societal_impact: The paper does not address those issues. Negative societal impact of this work is probably similar to that of any work in machine vision - authors may want to discuss the impact of digital surveillance since improvements in image segmentation might results in greater capacity of those systems.

As for the limitations, it would be interesting to see how the number of training classes affects the scores.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a novel module called Cycle-Consistent Transformer (CyCTR) that aggregates pixel-level support features into query features via cross-attention. Cycle consistency filters out irrelevant pixel-level support features. CyCTR achieves SOTA results on Pascal-5 and COCO-20 by an appreciable margin.
main_review: The paper contains novel and interesting approach that beats existing SOTA approaches for few shot segmentation on Pascal-5 and COCO-20 by an appreciable margin. Parts of the paper have clarity while some lack clarity.

CyCTR consists of Self-alignment block and Cross-alignment block. In the former, Query, Key and Value come from the same embedding. In the latter, the Query vector comes from the query images while Key and Value come from the support image. While CyCTR uses pixel-level support features, some of its competing approaches use semantic level prototypical features from support image or the foreground region in support image to encode query features.

### Recommendation:
Section 3 (specifically its second half) has a lot of typos, poorly written and lacks clarity. Likewise, for the bottom paragraph on page 8. There is a stark difference between other sections. Recommend authors to review and improve clarity. Otherwise, the mathematical expressions are sound.

limitations_and_societal_impact: The paper discusses limitations at the end of the experiments section. It does not include any comments on societal impact, other than the response to check list at the very end of the submission.
review_title: null

Meta-review: This paper proposes to do few-shot segmentation by applying a transformer to the few-shot support examples. The transformer’s attention is constrained so that only pixels that meet the proposed “cycle-consistency” constraint may be used for the prediction task. While the submission isn’t particularly groundbreaking methodologically, the approach seems to be novel overall and the quantitative results are quite strong across two standard benchmarks: for example on COCO, the proposed approach outperforms previous SotA by 40% to 32% for 1-shot, or 45% to 37% for 5-shot. The method is ablated to demonstrate the benefit of various parts of the model, e.g. the use of the cycle-consistency constraint vs. a baseline transformer with no cycle-consistency masking.

Reviewers pointed out a number of clarity and presentation issues with the submission. Please take these suggestions into account and revise the text accordingly for the camera-ready. Overall the text can be a bit difficult to parse at many points, and this caused confusion among reviewers and myself -- please try to revise and clarify wherever possible to phrase things more straightforwardly, especially in the core method section.

Other few-shot segmentation papers reported similar results to those in the submission [1, 2], but as they were roughly concurrent with this submission, the submission can’t be penalized for them. However, the authors are strongly encouraged to include comparisons with these recent works in the camera-ready version, and provide context comparing their proposed approach to these methods.

Given the novelty of the approach and the strength of the results, I recommend accepting the paper to NeurIPS.

[1] https://arxiv.org/abs/2104.01538
[2] https://arxiv.org/abs/2103.15402
Decision: Accept (Poster)