Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces the memory tokens into VisTR. They modified the box-based matching to the mask-based matching.  They also employ instance matching and clip-level tracking. The experiment results on Youtube-VIS2019 and Youtube-VIS2021  demonstrate the effectiveness of the proposed methods. The proposed method also has a high inference speed. The video sequences in the supplementary have high quality.
main_review: The proposed method has achieved promising results regarding accuracy and efficiency. However, I have some concerns regarding the clarity and significance.

1. The novelty is limited. This paper is mainly based on VisTR. However, in both the introduction and the method sections, it is not clear which parts are proposed by this paper and which part is inspired by VisTR. It will be better if the author can summarize the contribution at the end of the introduction.  The main contribution may be the memory tokens and mask-based tracking. 

2. It will be better if the author could include some of the related works regarding the memory tokens.

3. The ablation studies are not clear. It will be better if the author could make more detailed comparisons with VisTR under similar settings. It would be interesting to see how much improvement is brought by the new tracking loss and the memory bank.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on end-to-end solutions for video instance segmentation tasks by resigning transformers. Different from recently proposed per-clip pipelines VisTR, leverages richer information from multiple frames. This paper adopts Inter-frame Communication Transformers(IFC), which reduces overhead by encoding the context with the input clip as a memory token and correlates with other frames through the exchange of memory tokens. Experiments on the Youtube VIS benchmark demonstrate the effectiveness of the proposed method.
main_review: 1. IFC is interesting. However, I am not clear about the order of frame tokens and memory token intersection. Do memory tokens represent the instances? 

2. For the loss, could you mind giving the results replacing cross-entropy loss with focal loss?

3. For transformer encoder and encoder, you use the traditional self-attention or deformable-based self-attention module like in Deformable-DETR.

4. Conditional weight generations have achieved success in instance segmentation, e.g. CondInst and SOLOv2. Could you mind giving the results by using an embedded way?
limitations_and_societal_impact: 1. Could you explain more about memory tokens? Why does this token help compared with total clip-level interaction or sparse-based clip-level interaction, like in video recognition paper?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes Inter-frame Communication Transformers (IFC) for video instance segmentation, which utilizes the memory tokens to pass message across frames and enrich frame representation within a clip. Conditional convolution is used for the mask decoder. IFC is validated on YouTube-VIS with SOTA performance in an offline setting.
main_review: Strengths: 
* The idea of using concise memory tokens for communication between frames is novel and shown to be effective.
* The method achieves a SOTA performance on Youtube VIS with less complexity compared to VisTr. 
* The important modules and parameters settings are evaluated in extensive ablative studies. 
* The paper is well written and well organized.
* Very well-designed method figure helps the understanding.


Weaknesses:

1)	Details regarding the inference and training is somewhat unclear. The Appendix only add some more details. In particular, exactly how the inference pipeline proceeds and how clips are sampled and handled during training is crucial.

2)	The results of most ablative studies and analysis are inconsistent with the state-of-the-art results. In table 2a the Res50 variant achieves AP 41.0. But table 2c, 3a, 3c, and 3d all show different numbers for the corresponding setting (around 39 AP). Please clarify.

3)	The importance of the proposed IFC is much less compared to No Comm (table 3a) for larger clip lengths. This is also seen when comparing M=8 with M=1 in table 3c. Please add a discussion in the paper and clarify in the rebuttal.

4)	I believe that the reason for the effect observed in Q3 is that the cross attention performed in the transformer decoder also allows inter-frame communication. A discussion on this in relation to the proposed IFC is necessary. That is, why is IFC needed when the decoder also allows inter frame communication?

5)	In table 2a and 2b it need to be clarified which methods are online (ie causal) and offline (non-causal).

In summary, this is a strong paper with strong results. The memory tokens is a quite general idea that could be used for other applications. My preliminary rating is therefore positive. The authors however need to address my comments above and update the paper accordingly.

limitations_and_societal_impact: Limitations of the work is not addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose IFC Transformer for video instance segmentation. Compared to VisTR, the proposed method reduces the computational cost within the input clip. On the YouTube-VIS 2019 and 2021 datasets, the proposed method outperforms the other STOA methods in terms of accuracy and speed.
main_review: 1. The paper is clearly written.
2. The proposed method not only archives STOA results but also faster inference speed.
limitations_and_societal_impact: 1. How about the segmentation results on similar objects? As you use the global feature matching in Transformers, I think the proposed method may have difficulty distinguishing similar objects.
2. There are no qualitative comparison results with other methods.
3. Despite STOA results, what is the reason to choose Transformer? What are the advantages of Transformer?
review_title: null

Meta-review: The submission discusses video instance segmentation, specifically an inter-frame communication transformer which significantly reduces the overhead of information that is shared between frames. This helps to attain SOTA results while running at an impressive 89FPS. All reviewers appreciated the submission in their initial review leaning positive while raising concerns about about limited novelty, unclear ablation studies, some questions regarding the method (loss, order of frame tokens), inference and training, a missing qualitative comparison.

The authors provided a rebuttal which the reviewers appreciated. They remained concerned particularly about the issue that the authors chose to report the best result in their comparison to SOTA. This is not good practice and the authors are strongly encouraged to fix this. AC assumes that the authors will address this as the average scores provided in the ablation study still outperform SOTA, albeit less significantly.
Decision: Accept (Poster)