Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: **Problem**: This work addresses the problem of identifying correspondences between a given query feature map and a "support"/reference feature map. The goal for the correspondences is to be able to perform few-shot object segmentation or actor segmentation in videos.

**Solution**: The paper proposes a novel architecture (MMC) that takes as input dense feature similarity tensors and outputs frame-wise segmentation maps for the videos. The input similarity tensors measure the similarity between the support image features and features of the query video frames, at multiple levels of a CNN feature pyramid. The proposed MMC model comprises of a transformer-based encoder that reasons about the similarities using information from all levels of the similarity tensor pyramid and a decoder to output the frame-wise segmentation masks. 

strengths_and_weaknesses: ## Strengths

### Intuitive Architecture 
The proposed architecture is fairly straightforward to follow. The model involves a transformer architecture that uses learned key/value features to compute the attention and output features. These output features are pooled across the feature pyramid using a bidirectional pooling method. Finally the pooled features are passed to a decoder model to output the segmentation maps.

### Results 
The comparison to state-of-the-art approaches demonstrates that the presented model can produce improved segmentation maps. The quantitative results show improved mIoU values across the two tasks: object segmentation and actor segmentation.

---

## Weaknesses 

### Novelty 
While the results are impressive, in the current form of the text, it is difficult to recognize the novelty of the work. This is partly because the novelty or importance of some of the design choices seems overstated in the text. 
- The idea of reasoning across multiple feature levels of a CNN/similarity tensors is not novel (as pointed out by the authors too). This is the key idea of the "Multiscale Multigrid Attention". However, the specific implementation of this idea could be unique/novel. But it is not clear whether this is the case here. Section 2.2 seems to be inflating the contribution here. It is not clear why the stacked idea is presented since it is not the obvious/conventional method of performing feature pyramid pooling (to my best knowledge). And in addition to that, the stacked method actually performs almost equally well (in Table 1). Furthermore, the relationship to V-cycle correction is extremely loose. These explanations make it a bit harder to discern what the contribution is here. 
- One of the contributions is the idea of using a memory module. The memory module is essentially learned keys (and values) which are shared across different levels. This idea seems novel, but as a core contribution this has not been investigated enough experimentally. For example, is it necessary to share this across the different levels? Do the same features need to be used as keys and values? 
- The text seems to be using the term meta-learning (and meta-training, meta-testing) a lot. Is this one of the contributions? It is not clear why this is referred to as meta-learning. As far as I understand, meta-learning is the idea of optimizing through gradient descent update steps (or other optimization processes) to facilitate few-shot transfer learning - possibly introduced here first: 
>"Finn, C., Abbeel, P., & Levine, S. (2017, July). Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning (pp. 1126-1135). PMLR." 


questions: - See comment on "novelty" in weaknesses above. 
- For the task of Few Shot Video Object Segmentation, it looks like the convention is to report two metrics: mIoU and contour accuracy (as in DANet [5]). Is there a reason the contour accuracy is not reported here?
limitations: Discussion on limitations of the presented model is very limited in the paper. 
Discussion on societal impact is not covered in the paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a multiscale multigrid comparator transformer to address the few-shot video segmentation task. Experiments on few-shot video object segmentation and actor/action segmentation show effectiveness.
strengths_and_weaknesses: Pos: A novel comparator for few-shot learning tasks associated with dense predictions in videos is presented.

Neg:
1. Please explain what is static backbone and dynamic backbone.
2. In Equation 1, what are X_q, X_k and X_v respectively?
3. Equation 3 seems to be only a Unidirectional cross-scale information exchange. Where does the bidirectional information exchange mentioned by the author reflect?
4. This paper is hard to follow.
5. Writing needs to be improved.
questions: Please refer to the Weaknesses above for the details.
limitations: Yes. There's no suggestions.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a multi-scale multi-grid comparator transformer for Few-shot video segmentation that sims to overcome limitations of prior Few-Shot segmentation methods that limit feature comparisons to only a single feature layer thus ignoring information that might be valuable. Typically features in the early net work layers aid in precise localization and deeper layer features give higher level semantic information. The work aims to combine both coarse and fine-grained features via multi-scale and multi-grid operations to improve few shot segmentation. In addition, the work also introduces a novel multiscale memory learning in the decoder which helps preserve the details within feature maps across scales 

Contributions of the work are stated as:
+ a meta-learning method for multiscale comparison of query and support features is introduced
+ this comparator is a multiscale, multigrid transformer decoder that allows bidirectional multiscale information flow
+ finally the work also proposes a multiscale memory learning module within the transformer decoder
strengths_and_weaknesses: Strengths
+ well written paper
+ tackles an important problem 
+ proposes improvements and novel architectural additions that improves the prior state-of-the-art results by a large margin

Weakness
+ evaluation can be more rigorous
+ how does this method work on standard image few-shot segmentation tasks? since reference [19] in the paper was referred to, it would be nice to see results on COCO 20i, Pascal 5i and FSS 1000 datasets.



Reference [19] in paper: Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 6941–6952, 2021
questions: Questions:

Q) how does this method work on standard image few-shot segmentation tasks? since reference [19] in the paper was referred to, it would be nice to see results on COCO 20i, Pascal 5i and FSS 1000 datasets.
limitations: The work only talks about limitations in terms of 2D backbone features. But it would be important to examine failure cases and see why that is happening.

Additionally, the impact of pre-trained features can be examined to further understand how important such features are?
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper presents a multi-scale transformer as a comparator in few-shots video segmentation tasks. Compared with prior works that mostly use one feature resolution for comparison, they leverage multi-scale comparison to obtain both coarse and fine associations. A multi-grid mechanism is used for communication between different scales. The authors use a pretrained memory module to perform cross attention with the correlation volume. The resultant method outperforms current state-of-the-art methods in few-shot video object segmentation and few-shot actor segmentation.
strengths_and_weaknesses: Strengths:
- This paper is well-motivated. Having multi-scale correlation is a sensible solution to few-shots video segmentation problems. I also like the idea of keeping the spatial resolution in the transformer decoder which is again important to segmentation tasks. 
- The proposed method achieves good results on benchmark datasets, which further support the previous point.


Weaknesses:
- I think the writing can be improved. The authors rely heavily on abstract notation and procedural manipulation of these symbols without a high-level picture (in section 3). Most symbols are also not semantic so readers have to go back and forth many times to recall the meaning of a symbol. Yet, section 2.2 is too high-level and does not connect with the rest of the paper. The exact mechanism of multigrid bidirectional communication is not well explained. Adding some figures or rewriting sections 2 and 3 might help. This obstructs my understanding of this paper so I am putting a low confidence.
- There should be multiple transformer decoder layers (L134). This is not shown in Figure 1 or 2 (like with a *n). 
- A “memory” module should carry information across different timesteps, iterations or scales. Currently I find that the memory module is meta-learned and is not updated during inference time. If this is the case, it should not be called a memory module as it does not dynamically store information. It is more like a set of queries.
- The features for the support and target query are extracted from the same feature extractor (L223). This means the mask is not an input to the target query and the correlation volume does not carry mask information. I do not find this mask used as input in any other places. This does not make sense as the output should depend on the support mask.

questions: Is there any running time analysis? See also weaknesses.
limitations: The authors mention the use of 2D features as a limitation.
review_title: null

Meta-review: metareview: The paper develops a multigrid variant of the transformer architecture and applies it to video segmentation tasks.  After the author response and discussion phase, one reviewer recommends accept, but three of the four reviewers lean towards rejecting the paper.  In discussion, these three reviewers all acknowledged having read the author rebuttal and chose not to improve their scores.  The common concerns voiced across these reviews center on questionable novelty, clarity of explanation, and incremental experimental impact.  The Area Chair has also taken a detailed look at the paper, reviews, and author responses, and agrees with the concerns raised by these three reviewers.

Reviewer HsqG notes that "the idea of reasoning across multiple feature levels of a CNN/similarity tensors is not novel (as pointed out by the authors too)" and "Section 2.2 seems to be inflating the contribution here."  Section 2.2 and the author response highlights "bidirectional information exchange across scales, inspired by multigrid methods" as a key contribution.  However, reference [14] (Ke et al., Multigrid neural architectures, CVPR'17) explores exactly this idea: bidirectional information flow across scales, within a CNN architecture and even utilizes the same "multigrid" terminology.  From the point of neural network architecture design, the current paper's novelty appears limited to adapting previously established ideas to transformers.  So as not to appear to overclaim, the paper needs a broader discussion of the relationship of the proposed design to [14] as well as other prior work spanning multiscale, multiresolution, and feature pyramid architectures.

Reviewer concerns over experiments include marginal gains over ablated variants of the system (Table 1), and mixed results in comparison to DANet provided in the author response.  Overall, the response left unresolved questions over of contribution novelty, presentation clarity, and practical impact.
confidence: Certain
Decision: Reject