Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a framework for the video instance segmentation task. In order to avoid explicit instance association, which is responsible for computation overhead increment, the author designed both instance query and proposal propagation mechanism.
The framework consists of an intra-query attention module, temporally consistent matching, and an additional loss for removing duplicated predictions.
strengths_and_weaknesses: [Strengths]

1. The proposed method is simple and reasonable.
2. The paper is clearly presented and well-organized.
3. The proposed method outperforms all baselines including current state-of-the-art methods.

[Weaknesses]
Please refer to Questions.
questions: 1. I notice that some of the numbers in Table 1a are different from those in other papers.

For example, according to Table2 in IFC[10] paper, the number of VisTR and IFC are:

Name | AP | AP50 | AP75 | AR1 | AR10

VisTR | 35.6 | 56.8 | 37.0 | 35.2 | 40.2
IFC | 41.2 | 65.1 | 44.6 | 42.3 | 49.6

Although that does not necessarily raise a concern about the result, I wonder what could cause the difference.


2. In L301-313, the author argues the performance difference results from temporal propagation, yet it could simply be caused by the model size difference. For a fair comparison, the author should also provide the model parameter size after the replacement.
limitations: Please refer to Questions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents InsPro, a new method for online video instance segmentation by propagating instance queries across frames. The proposed InsPro, built on SparseR-CNN, defines a fixed set of instance queries and proposals to recognize and segment objects, and then propagate the updated queries and proposals to next frame. A temporal memory bank along with an intra-query attention are adopted to augment the instance queries. The proposed method InsPro achieves good results on Youtube VIS-2019 and Youtube VIS-2021 in terms of online video instance segmentation.
strengths_and_weaknesses: ### Strength

- The proposed method based on Sparse R-CNN adopts query-proposal propagation across frames for video instance segmentation and a temporal memory bank to augment queries with past frames.
- The proposed method is simple and effective, and easy to follow.
- This paper proposes an effective box de-duplication loss to further remove the duplicates.
- The overall performance of the proposed InsPro is good and experiments are abundant.

### Weakness

- The core idea of InsPro about using query propagation across frames has been explored in several works [1][2]. TrackerFormer[1] and EfficientVIS[2] adopt a similar way to propagate queries across frames and also can be reshaped into an online method. I'm concerned about the novelty of the proposed InsPro and the detailed comparisons among InsPro, TrackerFormer, and EfficientVIS.
- Line269, Sec 4.3: As far as I know, IFC can run near-offline(T=2) inference, see [3].


[1] Meinhardt et.al. TrackFormer: Multi-Object Tracking with Transformers.  
[2] Wu et.al. Efficient Video Instance Segmentation via Tracklet Query and Proposal.  
[3] Hwang et.al. Video Instance Segmentation using Inter-Frame Communication Transformers.  
questions: 1. The proposed method is built based on Sparse R-CNN but applies dynamic/conditional convolution to generate instance masks. I'm concerned about the pretraining performance on COCO. QueryInst[1] also extends Sparse R-CNN for (video) instance segmentation, how about the performance difference between the proposed InsPro and QueryInst in terms of COCO instance segmentation?

  [1] Fang et.al. Instances as Queries.

limitations: no
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes InsPro for online video instance segmentation. Instance queries are propagated and updated from the previous frames to current frame with implicit object association. Intra-query attention, temporally consistent matching, and box deduplication loss
are also proposed. Experiments are conducted on the Youtube-VIS 2019 and 2021.

strengths_and_weaknesses: **Strengths**:

1. The paper has a good motivation to solve online VIS problem with implicit query matching. The proposed method achieves both good evaluation performance and inference speed.

2. The paper shows the effect of the proposed query/proposal propagation, intra query attention and BDL with adequate ablation experiments. 

3. The paper is organized well with good writing and figures.

**Weakness**:

1. Missing discussion for related works in query-based VIS methods. In the paragraph for Query-based Methods, both IFC[a] and VisTr [b] are neglected. Also, SeqFormer[c] and Mask2Former[d] are also query-based VIS although they are arXiv but worth mentioning.

2. Missing related works in clip-level VIS Methods, where the online VIS method PCAN [e] also proposes the online object feature propagation and updating for the target tracklet. The tech differences comparison to [e] should be discussed in the related work section as well as the table results comparison on Youtube-VIS. 

[a] Video instance segmentation using inter-frame communication transformers. NeurIPS, 2021.

[b] End-to-end video instance segmentation with transformers. CVPR, 2021.

[c] SeqFormer: a Frustratingly Simple Model for Video Instance Segmentation. arXiv:2112.08275

[d] Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764 (2021)

[e] Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation. NeurIPS, 2021.


3. The tech contribution of InsPro is limited. The idea of query propagation has been adopted in both TransTrack and MOTR. The frame-level InsPro is a straightforward combination between Sparse R-CNN (for query-based object detection) and CondInst(providing mask head). Then, the frame-level InsPro is further extended to video by instance query propagation during online inference.

4. What are the typical failure cases of the methods? How to handle new objects with similar appearance? How to handle/correct the accumulation error during propagation process, for example, an instance query is wrongly matched at an early inference stage?

questions: I will consider raise my rating if the concern in the weakness part can be well addressed. The paper neglects discussion/comparison with related works in query-based VIS and temporal object propagation. The key idea of query propagation has been explored in TransTrack, which influences the paper novelty a lot.
limitations: There is no limitation or potential negative societal impact discussion in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a query-based framework for online video instance segmentation, which designs an instance query and proposal propagation mechanism to perform instance association implicitly. With such a mechanism, they achieve implicit instance association in parallel with segmentation and elegantly take advantage of temporal clues in videos. Experiments show the effectiveness of this method.
strengths_and_weaknesses: Strengths:
1) This paper proposes a novel query&proposal propagation mechanism for the video instance segmentation task, which is proved effective.
2) This paper achieves satisfactory results among the online VIS methods.
3) This paper is well organized.

Weakness:
1) The key idea of query&proposal propagation in this paper is similar to the method proposed in TrackFormer [28] (CVPR 2022), thus the author may claim the relation and difference between them. 
2) It is confusing for the baseline of ablation study in Tab. 2 (a)(c)(d). The AP score of 37.4 in these tables is mismatched with the AP score of 40.2 in Tab. 1 and Tab. 2(b), thus the author may clarify the setting or baseline for these results.
3) It would be better to have an analysis of the variance of InsPro-lite, e.g. the effects of key frame numbers. 

questions: As described in weakness.
limitations: No serious negative societal impact of this work.
review_title: null

Meta-review: metareview: The paper discusses a method for online video instance segmentation. Reviewers appreciated the proposed method but raised concerns regarding difference of reported results to other papers, method being similar to prior work, and limited novelty. The rebuttal addressed most of the concerns prompting reviewers to increase their rating to an accept recommendation. AC doesn't see reasons to overturn an unanimous reviewer recommendation.
confidence: Certain
Decision: Accept