InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation
Authors: Fei He
Haoyang Zhang
Naiyu Gao
Jian Jia
Yanhu Shan
Xin Zhao
Kaiqi Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video Instance Segmentation
Instance Query
Instance Segmentation

Abstract: Video instance segmentation (VIS) aims at segmenting and tracking objects in videos. Prior methods typically generate frame-level or clip-level object instances first and then associate them by either additional tracking heads or complex instance matching algorithms. This explicit instance association approach increases system complexity and fails to fully exploit temporal cues in videos. In this paper, we design a simple, fast and yet effective query-based framework for online VIS. Relying on an instance query and proposal propagation mechanism with several specially developed components, this framework can perform accurate instance association implicitly. Specifically, we generate frame-level object instances based on a set of instance query-proposal pairs propagated from previous frames. This instance query-proposal pair is learned to bind with one specific object across frames through conscientiously developed strategies. When using such a pair to predict an object instance on the current frame, not only the generated instance is automatically associated with its precursors on previous frames, but the model gets a good prior for predicting the same object. In this way, we naturally achieve implicit instance association in parallel with segmentation and elegantly take advantage of temporal clues in videos. To show the effectiveness of our method InsPro, we evaluate it on two popular VIS benchmarks, i.e., YouTube-VIS 2019 and YouTube-VIS 2021. Without bells-and-whistles, our InsPro with ResNet-50 backbone achieves 43.2 AP and 37.6 AP on these two benchmarks respectively, outperforming all other online VIS methods.

InsPro: Propagating Instance Query and Proposal for Online Video Instance Segmentation
Fei He 1.2 Haoyang Zhang', Naiyu Gao", Jian Jia 12, Yanhu Shan', Xin Zhao Kaiqi Huang
CRISE, Institute of Automation, Chinese Academy of Sciences 2 School of Artificial Intelligence, University of Chinese Academy of Sciences
CAS Center for Excellence in Brain Science and Intelligence Technology Horizon Robotics
(hefei2018, 1iaa ac. en, (haoyang zhang, naiyu01.gao gan,, shan)Ohorizon.co ai {xzhao kaihi huang) enlpr .ia. ac.
Abstract
Video instance segmentation (VIS) aims at segmenting and tracking objects videos. Prior methods typically generate frame-level clip-level object instances first and then associate them by either additional tracking heads or complex instance matching algorithms. This explicit instance association approach increases system complexity and fails to fully exploit temporal cues in videos. In this paper, we design simple, fast and yet effective query-based framework for online VIS. Relying on an instance query and proposal propagation mechanism with several specially developed components, this framework can perform accurate instance association implicitly. Specifically, we generate frame-level object instances based on of instance query-propose pairs propagated from previous frames. This instance query-proposal pair is learned to bind with one specific object across frames through conscientiously developed strategies. When using such a pair to predict an object instance on the current frame, not only the generated instance automatically associated with its precursors on previous frames, but the model gets good prior for predicting the same object. In this way, we naturally achieve implicit instance association advantage of temporal clues in videos. To show the effectiveness of our method InsPro, we evaluate on two popular VIS benchmarks, i.e., YouTube- VIS 2019 and YouTube-VIS 2021. Without bells-and-whistles, our InsPro with ResNet-50 backbone achieves 43.2 AP and 37.6 AP on these two benchmarks respectively, outperforming all other online VIS methods.
parallel with segmentation and elegantly take
Introduction
Video instance segmentation (VIS) [1] is a challenging but important computer vision task. It requires not only segmenting object instances on each video frame but also associating them across all frames. Due to its fine-grained object representation form, it has got wide range of applications in various areas such as autonomous driving and video editing.
Existing VIS methods can be categorized into two groups: frame-level methods and clip-level methods. Frame-level methods [1. 2. 4] generally follow a buckking-b-deteetti paradigm. which first generate per-fraud per-frame object instances by existing instance segmentation models [5, 6]. and then associate them across frames additional tracking heads (as shown in Figure (a)). In comparison, clip-level methods [7, 8, 9, 10] take 'elip-matching" paradigm, which divide video
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

query-proposll pair
Our SegNet
SegNet
Ins
Ins
Previous
Ins Ins 2
Previous
temporal propagation
query proposal pair
SegNet
Instance
ID
Ins
tokociation
SegNet
Ins
2
Ins
Ins y
Ins Ins
Current
Current
(a) Previous VIS paradigm
(b) Our InsPro
Figure 1: (a) Previous methods take two-step approach to VIS. They first generate object instances and then perform explicit instance association link them across frames. (b) Our InsPro implements implicit instance association through temporal propagation mechanism, achieving object instance segmentation and tracking in one shot. It generates frame-level object instances based on a set of instance query and proposal pairs propagated from previous frames. Since the instance query-propond pair is learned to represent one specific object across frames, the instance association is naturally achieved in parallel with segmentation and video temporal clues elegantly exploited meanwhile.
into multiple overlapped clips, generate instance predictions for each clip, and then associate these clip-level predictions by some sand-crafted instance matching algorithms. Whether frame-level or clip-level methods, both of them inevitably need an explicit instance association step to fulfill object tracking. This generally requires design a complicated association strategy to achieve good tracking performance, which is not trivial. More importantly, the explicit association step increases model complexity and slows inference speed. Furthermore, this extra step also indicates that the temporal clues intrinsic in videos are not well utilized, as the instance prediction is performed separately on each frame or each clip.
In this work, inspired by the recent success of query-based object detectors [11, 12], we propose a simple, fast and yet effective query-based framework for online VIS. Our system, dubbed InsPro, segments and tracks objects in one shot through an instance query and proposal propagation strategy with carefully designed modules (Figure 1 (b)) which eliminates the explicit instance association step. Specifically, our approach generates frame-level object instances based on a set of instance query-proposal pairs propagated from previous frames. In the learning process, we develop several techniques to make sure that the generated instance object across frames. Thus, when an object instance is generated using such a on the current frame, it is automatically associated with its precursors on all previous frames. In this way, we achieve implicit object association without a linking step. Meanwhile, this instance query-proposal propagation mechanism also enables our VIS system to achieve better prediction accuracy (see Table 1). This benefits from the instance query-proposal pair's encoding one object's temporal and spatial information across all previous frames, which provides very good prior for the model to infer the same object on the current frame. In this sense, our query-based VIS method actually implements an efficient way to exploit the intrinsic temporal clues in videos.
pair corresponds to one specific pair
To fulfill the advantages of our VIS system, learning exclusive and expressive instance query-propont pairs is the key. In this work, we develop several strategies to ensure the learning effectiveness. First, we design a temporally consistent matching mechanism enforce the one-to-one correspondence between the instance query-proposse pair and a specific ground truth object across frames during training. Second, we propose a box deduplication loss to enlarge the distance between instance proposals. This helps suppress duplicate proposals on the same object and increase the exclusivity of the generated instance query-proposal pair. At the same time, the sparsely distributed unoccupied query-proposal pairs serve as candidates in the next frame to detect new objects, allowing our system to achieve new object detection and tracking effortlessly. Third, we propose an intra-query attention module that enhances instance query with its predecessors encoding the same object. This explicitly aggregates long-range object information into the query, augmenting its representation capacity, which helps handle occlusion and motion blur.
To validate the effectiveness and efficiency of our InsPro, we conduct extensive experiments on two popular VIS benchmarks [1], i.e., YouTube--II 2019 and YouTube- VIS 2021. Without bells-and- whistles, our InsPro with ResNet-50 [13] backbone achieves 43.2 AP on YouTube- VIS 2019 and 37.6 AP on YouTube-Ve VIS 2021 respectively, outperforming all other online VIS models. Moreover, our

---

lite variant, InsPro-lite, reaches 38.7 AP on YouTube-VIL 2019 at impressive 45.7 FPS on a Nvidia RTX2080Ti GPU.
In summary, we make the following contributions in this paper. 1) We propose a simple, fast and yet effective query-based framework for online VIS. 2) We develop several techniques make the query-proposal pair propagation mechanism work smoothly. These techniques distinguish our work from other query propagation-base simpler, more elegant and more effective than them. 3) Our VIS system achieves the state-of-the-art performances on two popular VIS benchmarks.
object association methods [14, 15, 16], and make our work
Related Work
paradigm and can run in an online fashion. They first generate instance predictions frame by frame and then perform explicit instance association. MaskTrack R-CNN [1] first proposes the VIS task and simply adds an additional tracking head to Mask R-CNN [5] for instance association. Follow-up works [2, 3, 4, 17] improve either the segmentation or the tracking algorithm to achieve better performance. On the other hand, some works [8, 18. 19, 20, 21, 22] attempt to perform temporal feature fusion to improve instance segmentation and association. For example, PCAN [22] proposes frame- and instance-level prototypical cross-attentinn modules to leverage rich spatio-tempon segmentation. All these methods require additional modules to achieve explicit instance association, which expands model complexity and reduces inference speed. By contrast, our method performs instance association implicitly through an instance query and proposal propagation mechanism, which simpler and naturally exploits the temporal and spatial consistency in videos.
Frame-level VIS Methods mainly adopt a
information to facilitate better
Clip-level VIS Methods take a 'clip-matching' paradigm, which process multiple frames within a clip simultaneously and then perform instance matching between clips to complete VIS. While some methods [7, 23, 9] propagate instance information within clip with well-designed propagation modules to model temporal context, recent works [8, 10] utilize transformer [24] to model temporal context in end-to-end manner. These methods normally need hand-crafted matching algorithms to complete instance association between clips. Although they usually achieve high performance, they can only run in offline mode, which restricts their application to limited areas. In contrast, method achieves comparable performance but can run online.
Query-based Methods have attracted increasing attention in recent years due to their flexibility and simplicity. DETR [11] first uses a set of learned queries interacting with image features to encode objects, and then directly outputs detections by decoding the transformed queries. Following works [25, 26, 27, 28, 29] improve DETR in terms of either training efficiency or detection perfor- mance. Sparse R-CNN [12] builds a query-based detector on top of R-CNN architecture [30, 31]. Its follow-up works [3, 32] extend it to instance segmentation and video object detection. Besides, Max- DeepLab [33] proposes box-free panoptic segmentation method with external query. DAFL [34] uses set of queries to encode pedestrian information for pedestrian attribute recognition.
The success of DETR has also inspired query- based VIS methods. VisTR [8] adapts DETR to the VIS task. It takes a video clip as input and directly outputs the sequence of masks for each instance orderly. IFC [10] proposes inter-frame communication transformers reduce the heavy computation and memory usage of VisTR-like VIS methods. Similar to VisTR, Mask2Former [35] applies masked attention to video clip and directly predicts 3D instance volume. To learn a powerful video-level instance query, SeqFormer [36] aggregates temporal information from each frame to the instance query. These methods work on clips rather than frames, and achieve object association through sharing of the queries within a clip rather than query propagation. Thus, they still need instance matching between clips. Instead, our method applies to frames, and can propagate query-proposas pairs through the entire video and thereby can associate object instances over any frame length.
Query Propagation-based Object Association Methods have been recently explored in several works, such as TransTrack [14), TrackFormer [15], MOTR [16) and EfficientVIS [37], which are also inspired by query-based methods [11, 12]. This shows the effectiveness and potential of such a new object linking approach. The differences between our InsPro and them are as follows.
3

---

Temponi propagation
QUEE
update
Ciss
Rog
SegHead
SegHead
SegHend
Selfoai
Dynamic Cony
Gimeraaie
Weihh
Vtett
Feature
Feature
Feature
Extractor
Extractor
Extmetoor
Params
Muskc PCN Head
RolAlign
Mask Dranhh
f
M
4
(a) Overviow of InsPro
(b) Details SogHead
Figure 2: (a) Overview of our InsPro. It performs VIS by propagating instance query-prop proposal pairs across frames. ginit RxxC and Pinit E RR-x44 initial instance queries and proposals on the first video frame, respectively. They are used in SegHead to predict instance results ro on frame Io, and produce updated qo and Po which are propagated to the frame. By repeating this process, complete the VIS task. (b) Details of SegHead. It is multi-stage network, consisting of a dynamic instance interaction module (DIIM) and an instance segmentation module. The former transforms instance queries with Rol features of corresponding proposals and produces object features, while the latter predicts object instances based on the object features and conditional convolution [39].
First, our InsPro is different from them the way of either tracking seen objects or detecting new objects. TransTrack is basically match detection boxes tracked boxes in each frame, while our InsPro performs implicit association. More importantly, TransTrrack FrackFormer and MOTR adopt a track query subset to track seen objects and an extra object query subset to detect new objects. This requires additional heuristic rules to combine two type queries, and may miss occluded or blurred objects with low scores, which can result in object trajectory break [38]. Our InsPro simply propagates all object queries produced in the previous frame the current frame, and keeps using this set to track
method, because it still needs to explicitly
objects and detect new objects, which is much simpler and more elegant. As for EfficientVIS, our concurrent work, does not consider this new object detection problem, and its performance will probably be impacted greatly if there are new objects in the next clip.
Furthermore, we design more intelligent strategy to suppress duplicates. TransTrack and Track- Former employ score filtering or NMS to reduce duplicate predictions. MOTR builds a temporal aggregation network to learn more discriminative features to address this problem, while EfficientVIS does not discuss this problem. By contrast, we design a Box Deduplication Loss to suppress dupli- cates and an Inter-query attention module to enhance queries with their predecessors. Our solution avoids heuristic rules and post-processing steps, and is more effective according to the experimental results (see Table 2 (e)).
3 InsPro
We aim to design a simple and fast VIS system that performs instance association implicitly and exploits video temporal clues elegantly. To this end, we take a query-based VIS approach that predicts object instances on each frame based on set of instance query-proposal pairs propagated from previous frames. In this section, introduce query and proposal propagation mechanism and an instance segmentation head. Mcanwhile, we also describe those proposed techniques that make propagation mechanism work well.
VIS system, InsPro, including instance
3.1 Instance Query and Proposal Propagation
The instance query and proposal propagation mechanism enables our VIS system to perform object instance association implicitly in parallel with instance segmentation. Since inspired by the recent query-bart based object detector Sparse R-CNN [12), first briefly review Sparse R-CNN.

---

Sparse R-CNN [12] formulates object detection as a set prediction problem and achieves state-of-the art performance. It simplifies the detection pipeline and removes heuristic components like NMS. Specifically, it first initializes a fixed set of learnable instance queries (N x C. N denotes the number of queries and C the query dimension) paired with learnable instance proposals (N x 4) to describe objects in an image. As illustrated in Figure 2 (b), each instance query is convolved with the Rol feature of the corresponding proposal to output a more discriminate feature Or [12]. After multi-stage iterative updating. the instance query encodes more accurate object appearance information while the proposal captures more precise location. Finally, decoding the object feature o produced using the instance query-proposal pairs, we get the detection results.
Inspired by this instance query-proposee representation of an object, we design a query- proposll temporal propagation mechanism (as shown in Figure 2 (a)) to achieve implicit object instance associ- ation and temporal cue utilization in VIS. Our key insight is that there is a one-to-one correspondence between the learned instance query-pre prrepsal pair and specific object. If we manage to preserve this correspondence from the first frame to the one where the object finally disappears, then we realize object tracking and object information propagation spontaneously.
To this end, we first initialize a set of instance queries qivit RNxC and proposals Pina RRxx on the first video frame lg. where qinit and Pinat are learnable parameters and arranged in pairs. After learning, they are able to encode objects on the first frame. Decoding them with the first frame image feature inside the SegHead, we obtain instance results To as well as a new set of updated pairs (go- Po). Then we propagate this pair set (qn- Pa) to the next frame as input to the SegHead. Similarly, we get the instance results 11 and another new set of (q1, P1) on this frame. Among them, the object instance produced on this frame shares the same ID with the one on the previous frame if they are both decoded by the same slice of the instance queries. In this way, we automatically link object instances belonging to an identical object across frames and elegantly make use of object priors from the past. Repeating the above process until the last video frame, we then accomplish the VIS task on this video.
Please note that our InsPro simply propagates all object queries produced in the previous frame to the current frame, and keeps using this set to track seen objects and detect new objects. Instead, recent works [14, 15, 16] that take a similar query-propagatin to track seen objects and a new object query set to detect new objects respectively, which requires additional heuristic rules to combine these two type queries. Moreover, they rely on hand-crafted rules like a score threshold to select a subset of track queries, and occluded objects with low prediction scores are probably filtered out, which results in non-negligible true object missing and fragmented trajectories [38]. In comparison, our method is obviously simpler, more elegant and more effective (see Table 2 (e)).
mechanism use a track query set
Intra-query Attention Since frame-by-frame temporal propagation encodes only short-range temporal cues, the instance query from just the last frame shows limitations in dealing with tough scenarios, e.g., occlusion and motion blur. To boost the representation capacity of instance query, we augment it in practice with instance features from previous T' frames [40, 41]. Specifically, we build a feature bank that caches instance features from previous T frames and perform intra-query attention inside this bank to aggregate long-range temporal cues into the current instance query, as shown in the upper part of Figure (b). Formally, at frame I, instance features o from previous T frames are put together to form feature bank fb (oi-T+1..... o.}. Then, the enhanced instance query is computed as:
Ex- of exp(e(oi_.)) + +
di
Emdhe exp(<(()
where i denotes the i-th query and E(-) is a linear transformation function. The enhanced q, is basically a weighted sum of instance features inside the feature bank, and the weights are learned upon the quality of the queries. Experiments (Table 6 (c)) show that this augmentation improves the query representation capacity greatly.
3.2 Segmentation Head
The segmentation head transforms the instance query and performs instance segmentation on each frame. As illustrated in Figure (b), is a multi-stage network and has two main parts: a dynamic instance interaction module and an instance segmentation module.
5

---

Dynamic Instance Interaction Module transforms the instance query with the proposal Rol feature and yields object features. has M stages and forms an iterative structure. At the first stage, given a pair of instance queries ---1 E IR xC and proposals Pt-1 E R that propagated from the previous frame 1,- first augments the instance queries by self-attention module [24], which models the inter-query relations. At the same time, it extracts the Rol feature of each proposal on the feature map by RoLAlign [5]. Then, each enhanced instance query convolves with its corresponding Rol feature through dynamic convolution [42] to get the object feature or E IR Since the object feature absorbs temporal cues encoded in instance queries, it has better representation ability than the hingle-frame, Rol feature (see Table (d)). The object feature is used to predict object instances in the following instance segmentation module, and the newly generated object boxes together with the object features proceed as input the next stage in the iterative process. At the final stage, the object feature is processed together with its precursors from previous frames through the aforementioned intra-query attention module, to produce instance queries for the next frame.
Instance Segmentation Module decodes the object feature on and produces VIS predictions. has three main heads. While the classification head predicts object classes, the regression one generates object boxes. Another mask head is responsible for producing instance masks through conditional convolution [39] approach. Specifically, it first uses O₁ to generate conditional convolution weights in weight generator. As shown in Figure 2 (b), it inputs of that represents the i-th object instance feature to the weight generator and outputs set of convolution parameters w.. Meanwhile, it produces mask feature maps by transforming FPN [43] feature maps through a mask branch. Note that the output mask feature maps have 8 channels and 1 8 resolution of the input image, and are boosted by concatenating a 2-channel relative coordinates map to it. This relative coordinates map is computed using the center of predicted object boxes and provides strong location cues for predicting instance masks. Finally, we feed the combined feature maps f fmaak E 100x x w and convolution parameters w; to mask FCN head, predicting the i-th object instance mask via conditional convolution:
m CondConv(f! w:).
(2)
For more details about the instance segmentation module, please refer to [39].
3.3 Temporally Consistent Matching
The key to the success of our InsPro to make sure that the evolving instance query-propont pair corresponds to the same object across frames in video. To ensure this, one technique we propose is temporally consistent matching. This technique matches predictions and ground truth during training, assigns each ground truth object a proper prediction, and propagates the matching made on previous frames to subsequent frames.
Specifically, given a training batch consisting of multiple consecutive frames, we first compute the matching cost Lmaatch between predictions and ground truth objects on the first frame:
Laatch Ads= Lets + ALa LT.1
Agina
Egious
where Cets is the focal loss [44] between predicted classifications and ground-truth labels, LL and Egion are LI loss and the generalized IoU loss [45] between predicted boxes and ground-truth boxes, respectively. Acta, d7ι and Àgiau are loss weights and set as 2, 5, and 2, respectively.
We search for the best bipartite matching that minimizes the matching cost Limatch with the Hungarian algorithm [46]. After finding the best matching on the first frame, we propagate this matching to other frames. Concretely, if one ground truth object still exists on subsequent frames, it will be matched to the prediction that is generated by the same instance query on the first frame. If there are new objects emerging, new matching will be made between the new objects and yet unmatched predictions. If a ground truth object disappears, its corresponding predictions will not participate in new matching process. Through this temporally consistent matching mechanism, we bind one ground truth object to single instance query during training.
3.4 Loss Function
Box Deduplication Loss Although the self-attention mechanism between queries has driven the model to generate fewer duplication predictions [11], we still observe multiple overlapped proposal

---

Lime
( wio BDL.
(b) w BDL
Figure 3: (a) Multiple duplicate boxes exist on the same object across frames. (b) After applying the proposed box deduplication loss (BDL) in training, the duplicate predictions are significantyy suppressed along with temporal propagation.
boxes on the same object across many frames, as displayed in Figure (a). We conjecture this is caused by those unmatched queries which cannot be pushed away from those matched queries due lack of supervision. To address this problem, we propose box deduplication loss push away prediction boxes in terms of the center-to-center distance between them. As a result, not only the duplication problem is alleviated, but the sparsely distributed unmatched query-proposal pairs can serve as candidates in the next frame to detect and track new objects (see Figure 7 in Appendix). The loss is defined as:
k
C² D D2(()
deddup
max(B
0),
where b is ground truth box, bneg is negative box that has the i-th largest IoU with b among those unmatched predicted boxes, C(-) is the center distance calculation function, and DD.) is the diagonal length calculation function. is set as 0.1 and I as This loss penalizes the short distance between and 6 beeg and drags all other duplicate boxes away from [47]. With this new loss, our final box loss function is formed as:
Lhox = ALa LI.1 + Agion Legou + Addedup Litedup
where dLι and Agiou have the same values in Equation 3, and Adedup is set as
Overall Loss Given the one-to-one matching results, the final loss on each training frame is sum of classification, box and mask losses:
L Dets Lets + Aboz Lbox + Adice Laiice + A f
where Ldice and Lroeal are dice loss [48] and focal loss [44] for foreground mask prediction, respectively. We set Aboz = 1, Adice = 5 and Afocal = 5. Finally, the losses of all training frames inside batch are summed together and normalized by the number of frames.
4 Experiments
4.1 Datasets and Evaluation Metrics
We evaluate our method on YouTube-VTS 2019 and 2021 benchmarks []]. YouTube-VIS 2019 consists of 2,238 training videos and 302 validation videos. and labels 40 object categories. YouTube VIS 2021 an extended version, which comprises 2,985 training videos and 421 validation videos, and labels improved 40 categories. All videos in these two datasets are annotated every 5 frames with object bounding box, object category, instance mask and instance ID. Following [1], we report video-level average precision (AP) and average recall (AR) on the validation sets as the evaluation metrics, where both accurate instance segmentation and instance association are necessary to achieve high performance.

---

Table 1: Comparison of our InsPro tfat-of-the-aae methods. All methods use ResNet-50 as backbone. C: additionally using COCO train2017 images that contain YouTube- VIS categories for training. The inference speed tested on Nvidia RTX2080Ti GPU. indicates using deformable convolution [52] in backbone. indicates that the FPS is measured by parallel processing of images one clip rather than sequential processing.
YouTube- VIS
2019 Val.
YouTube- VIS 2021 Val.
Method
Online
AP
AP7s
AR,
AR
AP
AP
AP7s
AR,
AR
FPS
STEm-Seg (C)
X
30.6
50.7
33.5
31.6
37.1
4.4
VisTR [8]
35.6
56.8
37.0
35.2
40.2
30.0
Propose-Reduce Reduce [9] (C)
X
63.0
43.8
41.1
49.7
MaskProp" [23]
x
40.0
42.9
< 10
IFC [10]
x
39.0
60.4
42.7
41.7
51.6
35.2
57.2
37.5
46.51
EfficientVIS [37]
X
37.9
59.7
43.0
40.3
46.6
34.0
57.5
37.3
33.8
42.5
36
MaskTrack R-CNN [1]
V
30.3
51.1
32.6
31.0
35.5
28.6
48.9
29.6
26.5
33.8
26.1
SipMask [4]
54.1
35.8
35.4
40.1
31.7
52.5
34.0
30.8
37.8
STMask [20]
33.5
52.1
36.9
31.1
39.2
28.6
SG-Net [2]
34.8
56.1
36.8
35.8
40.8
23.0
PCAN [22]
36.1
54.9
39.4
36.3
41.6
Cross [18]
36.3
56.8
38.9
35.6
40.7
34.2
54.4
37.9
30.4
38.2
25.6
HybridVIS [21] (C)
41.3
61.5
43.5
42.7
47.8
35.8
56.3
39.1
33.6
40.3
<00
InsPro-lite
38.7
60.9
41.7
36.9
43.6
45.7
InsPro
40.2
62.9
43.1
37.6
44.5
36.1
57.6
39.6
30.9
40.4
26.3
InsPro (C)
43.2
65.3
48.0
38.8
49.0
37.6
58.7
40.9
32.7
41.4
26.3
4.2
Implementation Details
We implement our InsPro with Detectron2 [49], and most hyperparameters are set following Sparse R-CNN [12] and CondInst [39] unless otherwise specified. More implementation details can be found Appendix A.I.
Training Details We employ A [50] with an initial learning rate of 2.5 x 10-5 and weight decay 0.0001 as our model optimizer. We initialize our model with parameters pre-trained on COCO [51], and train it for 32k iterations where the learning rate is divided by 10 at iterations 24k and 28k, respectively. The training performed end-to-end 8 Nvidia RTX2080Ti GPUs and each GPU holds one mini-batch which contains three frame images randomly sampled from the same video. Data augmentation includes only random horizontal flip and multi-scale training where the training image is resized so that the length of its shortest side is at least 288 and at most 512. Unless otherwise noted, our InsPro adopts ResNet-50 [13] as backbone and uses 100 instance queries in our experiments.
Inference Details In inference, we resize the frame image size to 640 360, which follows MaskTrack R- R-CNN [1]. The size of the feature bank is set to 18 by default. If the generated proposal box exceeds the frame's boundaries, it will be clipped to corresponding boundaries. No multi-scale testing is adopted in our experiments.
InsPro-lite We also build lite version of method, named InsPro-lite. In this variant, inspired by [32], we divide video frames into key frames and non-key frames, i.e., we select one key frame per K frames in video and treat other frames non-key ones. K 10 by default. On key frames. we conduct the dynamic instance interaction 6 times while only once on non-key frames. This takes advantage the redundancy of videos and helps reduce inference computation time. Our InsPro-lite reaches high inference speed of 45.7 FPS at small accuracy loss (Table 1).
4.3 Main Results
We perform a thorough comparison our InsPro to state-of-the-arr VIS methods on YouTube- VIS 2019 and 2021. Existing VIS methods can be divided into two categories according to whether they run online or offline [53]. Since some methods [7, 9] use 80k transformed COCO images [51] as extra training data to prevent overfitting to YouTube-Vir VIS, for fair comparison, also report our

---

Table 2: Ablation studies on YouTube- VIS 2019.
(a) Effectiveness of instance query and pro- posal propagation, and temporally consis- tent matching (TCM).
(c) Intra-query attention. Tis the
(b) Effectiveness of the proposed length of the feature hank.
box deduplication loss (BDL).
query
proposal
TCM
AP
APso
AP
AP
APsa
APra
FPS
(A)
24.0
41.3
24.2
AP
APso
AP76
FPS
57.7
41.6
(B)
36.3
56.3
38.9
w/a BDL
37.4
57.6
41.1
26.3
T=9
39.7
61.6
42.1
26.3
(C)
w/ BDL
38.4
57.7
41.6
26,3
62.9
(D)
36.7
57.3
39.9
T=27
40.1
62.6
42.2
26.3
(6)
36.6
55.5
40.3
T=36
40.1
62.5
42.2
26.3
(d) Comparison between our temporal propagation paradigm and tracking-by-detectine be-detection" paradigm.
(e) Comparison between our united query and 'track-and-detec" query'.
AP
AP5o
AP7s
Param (M)
FLOPs
FPS
AP
APso
APr5
Tracking by detection
31.5
49.3
34.1
119.9
48.3
25.4
Track and detect query
37.4
56.9
40.3
$7.6
106.1
45.5
26.3
38.4
results with and without extra COCO training data. Table 1 presents all the results obtained with ResNet-50 backbone on a Nvidia RTX2080TI GPU.
YouTube- 2019 Table (left) shows the comparison between our InsPro and other state-of-the- art methods on YouTube- VIS 2019 validation set. We can see that, in the online group, our InsPro outperforms all other popular methods under the same data setting. Specifically, our InsPro achieves 40.2 AP without COCO data and 43.2 AP with COCO data respectively, surpassing other online VIS methods by a large margin. Even our lite version, InsPro-lite performs better than all other online methods trained without COCO data, reaching 38.7 AP at an impressive speed of 45.7 FPS.
YouTube- VIS 2021 Table (right) displays results on YouTube- VIS 2021 validation set. It shows similar comparison pattern to YouTube- ViI 2019 and InsPro achieves the state-of-the-art performance once again.
4.4
Ablation Study
We conduct extensive experiments on YouTube- VIS 2019 to study the effectiveness and individual performance contribution of our proposed modules.
Temporal Propagation and Matching Our InsPro built on the proposed instance query and proposal temporal propagation mechanism. Table (a) shows how this mechanism contributes to our high performance. In this table, method A represents the video instance segmentation baseline, where each frame is processed individually without any temporal propagation, and object instances generated on each frame are linked if they are produced from the same instance query slice. Since pair corresponds to the
this method lacks the mechanism ensure the instance guery-proposs same object across frames, it only achieves 24.0 AP due to inaccurate instance association. By contrast, when we add the temporal propagation (method B), we can easily improve the performance significantly to 36.3 AP. This evidences the importance and effectiveness of the proposed temporal propagation technique
query-based VIS framework. we further adopt the temporally consistent matching strategy during training (method C), we achieve an even better performance of 37.4 AP.
We also analyze the separate performance of propagating only instance query (method D) or instance proposal (method E). The results show that these two settings achieve a similar performance boost (36.7 AP vs 36.6 AP). Applying them together yields 37.4 AP, bringing further performance gain.
Box Deduplication Loss We propose a box deduplication loss to suppress the duplicate proposal boxes on the same object across frames. With the qualitative results shown in Figure we show the quantitative comparison in Table 2 (b). We can see that supervising the learning with this loss during training can lead 1. 1. AP improvement (38.4 4 AP vs 37.4 AP). This performance gain brought by fewer duplicate boxes and fewer missed detections.

---

Intra-query Attention We perform query attention inside feature bank to augment the instance query so that it can capture long-range temporal cues. As we can see in Table (c), this simple method works well and improves the performance considerably. In particular, T = =1 indicates no intra-query attention used and 38.4 AP is achieved. When we increase the volume of the feature bank, the performance rises and saturates at 40.2 AP with T = 18. It is worthwhile to note that this lightweight yet effective intra-query attention module brings almost no speed drop.
Despite the fact that our InsPro does not perform explicit instance association, it still outperforms all other online methods implementing explicit tracking matching. To verify that superior performance comes from the temporal propagation mechanism rather than the instance segmentation model design, we compare our temporal propagation VIS approach to the typical 'tracking-by by-detection' paradigm with the same instance segmentation baseline. We implement a 'trakknn-by-deen by-detection' VIS system by replacing the Mask R-CNN part in MaskTrack R- R-CNN [1] with our instance segmentation model. In this case, the only independent variable is the object tracking method.
Temporal Propagation vs. Tracking-by-Detectiin
As shown in Table 2 (d), our InsPro surpasses the besking-b--deecttonn model by large margin even if our design simpler and faster, which soundly proves the effectiveness of our method. We argue again that this is because the evolving instance query-proposal pair in propagation encodes object temporal and spatial cues intrinsic in videos, whereas generally incapable of exploiting this advantage.
methods are
Our United Query vs. Track-and-Dee medd-ettett Query We further compare our method to those MOT methods that adopt similar query preyaaproaagttion method for object tracking. These methods rely on two different query sets, i.e., a track query set and an object query set, to track seen objects and detect new objects respectively, while we only maintain one united query set. They need heuristic rules to combine these two type queries, Meanwhile, they manually select track queries with high scores from the previous frame to build the track query set. This makes them complex and less effective in tracking since occluded objects with low scores probably have broken trajectories because of the filtering.
To show the superiority of our method, we compare the 'track-and-detect query' paradigm adopted in the most recent MOTR [16] to ours using the same instance segmentation baseline. We follow MOTR [16] exactly to set up the model and experiment settings. To exclude the influence of other factors, we do not use temporal feature aggregation in both methods. Table 2 (c) shows the comparisons on YouTube- VIS 2019. can be seen that our InsPro achieves a higher performance even using simpler query design. We attribute this advantage to our conscientiously designed modules described in Sec 3.
5 Conclusion
In this paper, we propose simple, fast and yet effective query-based framework for online VIS. In this framework, we rely on a novel instance query and proposal propagation mechanism to undertake VIS, where we generate object instances based on a set of evolving instance query-proposse pairs propagated from previous frames. This mechanism enables our model not only to associate object instances implicitly, but to utilize video temporal cues elegantly. To make this propagation mechanism work well, we develop several modules to ensure that the learned instance query-propor pair keeps being bound to one object, These modules include an intra-query attention unit, a temporally consistent matching mechanism and a box deduplication loss. Extensive experiments on YouTube- VIS 2019 and 2021 verify the effectiveness of our designs, and show that our InsPro achieves superior VIS performance, outperforming all other online VIS methods.
Acknowledgments
This work is supported in part by the National Natural Science Foundation of China (Grant No. 61721004), the Projects of Chinese Academy of Science (Grant No. QYZDB-SSS-JJCC06) the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDA27000000), and the Youth Innovation Promotion Association CAS.
10

---

References
[1] Yang, L., Y. Fan, N. Xu. Video instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5188-5197. 2019.
[2] Liu, D., Y. Cui, W. Tan, et al. Sg-net: Spatial granularity network for one-stage video instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 981 9816-9825. 2021.
[3] Fang. Y., S. Yang, X. Wang, et al. Instances as queries. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6910-6919. 2021.
[4] Cao, J., R. M. Anwer, H. Cholakkal, et al. Sipmask: Spatial information preservation for fast image and video instance segmentation. In European Conference on Computer Vision, pages 1-18. 2020.
[5] He, K., G. Gkioxari, P. Dollár, et al. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969. 2017.
[6] Bolya, D., C. Zhou, Xiao, al. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF international conference on computer vision. pages 9157-9166. 2019.
[7] Athar, A., S. Mahadevan, A. Osep, et al. Stem-seg: Spatio-temporal embeddings for instance segmentation videos. In European Conference on Computer Vision, pages 158-177. 2020.
[8] Wang, Y. Z. Xu, X. Wang, al. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8741-8750 2021.
[9] Lin, H., R. Wu, S. Liu, et al. Video instance segmentation with propose-reduce paradigm, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1739-1748. 2021.
[10] Hwang. S., M. Heo, S. W. Oh, al. Video instance segmentation using inter-frame communica- tion transformers. Advances in Neural Information Processing Systems, 34, 2021.
[11] Carion, N., F. Massa, G. Synnaeve, et End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. 2020.
[12] Sun, P., R. Zhang, Y. Jiang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1145-114633 2021.
[13] He, K., X. Zhang, S. Ren, et al. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778. 2016.
[14] Sun, P., J. Cao, Y. Jiang, al. Transtrack: Multiple object tracking with transformer. arXiv preprint arXiv:2012.154m 155660 2020.
[15] Meinhardt, T., A. Kirillov, L. Leal-Taixe, et al. Trackformer: Multi-object tracking with transformers. arXiv preprint arXiv:2101 20111101.222222 2021.
[16] Zeng, F., B. Dong, T. Wang, et al. Motr: End-to-end multiple-object tracking with transformer.
arXiv preprint weXiv:2105.000 05.444477 2021.
[17] Zhu, F., Z. Yang, X. Yu, al. Instance as identity: A generic online paradigm for video instance segmentation. In ECCV. 2022.
[18] Yang, S., Y. Fang, X. Wang, et al. Crossover learning for fast online video instance segmentation.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8043- 8052. 2021.
[19] Fu, L. L. Yang, D. Liu, et al. Compfeat: Comprehensive feature aggregation for video instance segmentation. In Proceedings of the AAAI Conference Artificial Intelligence, vol. 35, pages 1361-1369. 2021.
[20] Li. M.. S. Li, L. Li, et al. Spatial feature calibration and temporal fusion for effective one-stage vidco instance segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11215-11224. 2021.
[21] Li, X., J. Wang, X. Li, et al. Hybrid instance-aware temporal fusion for online video instance segmentation. In Proceedings of the AAAI Conference Artificial Intelligence. 2022.
11

---

[22] Ke, L., X. Li, M. Danelljan, et al. Prototypical cross-attention networks for multiple object tracking and segmentation. Advances in Neural Information Processing Systems, 34:1 34:1192-1203, 2021.
[23] Bertasius, G., L. Torresani. Classifying, segmenting, and tracking object instances in video with mask propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9739-9748. 2020.
[24] Vaswani, A., N. Shazeer, N. Parmar, et al. Attention all you need. Advances in Neural Information Processing Systems, 30, 2017.
[25] Zhu, X., W. Su, L. Lu, al. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations. 2020.
[26] Meng, D., X. Chen, Z. Fan, et al. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3651-3660. 2021.
[27] Dai, Z., B. Cai, Y. Lin, et al. Up-detr: Unsupervised pre-training for object detection with transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1601-1610. 2021.
[28] Dai, X., Y. Chen, J. Yang, et al. Dynamic detr: End-to-end object detection with dynamic attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2988 2997.999222009 2021.
[29] Liu, S., F. Li, H. Zhang, et al. Dab-detr: Dynamic anchor boxes are better queries for detr. In International Conference on Learning Representations. 2021.
[30] Girshick, R., J. Donahue, T. Darrell, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 580-587. 2014.
[31] Ren, S., K. He, R. Girshick, et al. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
[32] He, F., N. Gao, J. Jia, et al. QueryProp: Object query propagation for high-performance video object detection. In Proceedings of the AAAI Conference on Artificial Intelligence. 2022.
[33] Wang, H., Zhu, H. Adam, et al. Max- Max-deeplab: End-to-end panoptic segmentation with mask transformers. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pullern Recognition, pages 5463-5474. 2021.
[34] Jia, J., N. Gao, F. He, et al. Learning disentangled attribute representations for robust pedestrian attribute recognition. In Proceedings of the AAAI Conference on Artificial Intelligence. 2022.
[35] Cheng, B., Choudhuri, I. Misra, et Mask2former for video instance segmentation. arXiv preprint arXiv:21 12. 10764, 2021.
[36] Wu, J., Y. Jiang. S. Bai, al. Seqformer: Sequential transformer for video instance segmentation. In ECCV. 2022.
[37] Wu, J., S. Yarram, H. Liang, et al. Efficient video instance segmentation via tracklet query and proposal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2022.
[38] Zhang. Y., P. Sun, Y. Jiang, et al. Bytetrack: Multi-object tracking by associating every detection box. In European Conference on Computer Vision. 2022.
[39] Tian, Z., C. Shen, H. Chen. Conditional convolutions for instance segmentation. In European Conference on Computer Vision, pages 282-298. 2020.
[40] He, F., N. Gao, Q Li, et al. Temporal context enhanced feature aggregation for video object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pages 10941-10948. 2020.
[41] He, F., Q. Li, X. Zhao, et al. Temporal-adaptive sparse feature aggregation for video object detection. Pattern Recognition, 127:108587, 2022.
[42] Jia, X., B. De Brabandere, T. Tuytelaars, et al. Dynamic filter networks. Advances in Neural Information Processing Systems, 29, 2016.
12

---

[43] Lin. T.-Y., P. Dollár, R. Girshick, et al. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2117-2125. 2017.
[44] Lin, T.-Y., P. Goyal, R. Girshick, et al. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988. 2017.
[45] Rezatofighi, H., N. Tsoi, J. Gwak, et al. Generalized intersection over union: A metric and loss for bounding box regression. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 658 388-666.22 2019.
[46] Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97 1955.
[47] Hermans, A., L.. Beyer, B. Leibe. In defense of the triplet loss for person re-identification. arXiv preprint 0rXiv:1703.07733: 2017.
[48] Milletari, F., N. Navab, S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference 3D vision (3DV), pages 565-571. 2016.
[49] Wu, Y., A. Kirillov, Massa, et Detectron2. detectron2, 2019.
[50] Loshchilov, I., F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations. 2018.
[51] Lin, T.-Y., M. Maire, S. Belongie, et al. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755. 2014.
[52] Dai, J., H. Qi, Y. Xiong, et Deformable convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 764-773. 2017.
[53] Luo. W. J. Xing, A. Milan, et al. Multiple object tracking: A literature review. Artificial Intelligence, 193:103445 2021.
[54] Qi, Y. Gao, Y. Hu, et al. Occluded video instance segmentation: A benchmark. International Journal of Computer Vision, 2022.
[55] Deng, J., Dong, R. Socher, et al. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255. leee, 2009.
[56] Gkioxari, G., J. Malik. Finding action tubes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 759-768. 2015.
[57] Feichtenhofer, C., A. Pinz, A. Zisserman. Detect to track and track to detect. In Proceedings of the IEEE international conference on computer vision, pages 3038-3046. 2017.
[58] Xiao, F., J. Lee. Video object detection with aligned tpatirl-temporal memory. In European Conference on Computer Vision, pages 485-501. 2018
[59] Cheng, B., A. Schwing, A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
[60] Milan, A., I.. Leal-Taixé, I, Reid, et al. Mot16: A benchmark for multi-object tracking. arXiv preprint arXiv: 1603.0088) 2016.
[61] Kim, S. J.-Y. Lee, et al. Video panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9859-9868. 2020.
[62] Gao, N., F. He, Jia, al. Pamopticdepth: A unified framework for depth-aware panoptic segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1632 1642. 2022.
[63] Avinash Ramakanth, S., R. Venkatesh Babu. Seamseg: Video object segmentation using patch seams. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 376-383, 2014.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss limitations and future work in Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss the potential negative societal impacts in Appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] We list the data and instructions in Sec. 4 and Appendix, and the code will be released upon publication.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] All training details listed in Sec. and Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "iBpk8uQqq8E", "reviewer": "Reviewer_1ynk", "summary": "This paper proposes a query-based framework for online video instance segmentation, which designs an instance query and proposal propagation mechanism to perform instance association implicitly. With such a mechanism, they achieve implicit instance association in parallel with segmentation and elegantly take advantage of temporal clues in videos. Experiments show the effectiveness of this method.", "questions": "As described in weakness.", "limitations": "No serious negative societal impact of this work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) This paper proposes a novel query&proposal propagation mechanism for the video instance segmentation task, which is proved effective.\n2) This paper achieves satisfactory results among the online VIS methods.\n3) This paper is well organized.\n\nWeakness:\n1) The key idea of query&proposal propagation in this paper is similar to the method proposed in TrackFormer [28] (CVPR 2022), thus the author may claim the relation and difference between them. \n2) It is confusing for the baseline of ablation study in Tab. 2 (a)(c)(d). The AP score of 37.4 in these tables is mismatched with the AP score of 40.2 in Tab. 1 and Tab. 2(b), thus the author may clarify the setting or baseline for these results.\n3) It would be better to have an analysis of the variance of InsPro-lite, e.g. the effects of key frame numbers. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "i3g4aiT5etZ", "reviewer": "Reviewer_g8FE", "summary": "This paper presents InsPro, a new method for online video instance segmentation by propagating instance queries across frames. The proposed InsPro, built on SparseR-CNN, defines a fixed set of instance queries and proposals to recognize and segment objects, and then propagate the updated queries and proposals to next frame. A temporal memory bank along with an intra-query attention are adopted to augment the instance queries. The proposed method InsPro achieves good results on Youtube VIS-2019 and Youtube VIS-2021 in terms of online video instance segmentation.", "questions": "1. The proposed method is built based on Sparse R-CNN but applies dynamic/conditional convolution to generate instance masks. I'm concerned about the pretraining performance on COCO. QueryInst[1] also extends Sparse R-CNN for (video) instance segmentation, how about the performance difference between the proposed InsPro and QueryInst in terms of COCO instance segmentation?\n\n  [1] Fang et.al. Instances as Queries.\n", "limitations": "no", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n- The proposed method based on Sparse R-CNN adopts query-proposal propagation across frames for video instance segmentation and a temporal memory bank to augment queries with past frames.\n- The proposed method is simple and effective, and easy to follow.\n- This paper proposes an effective box de-duplication loss to further remove the duplicates.\n- The overall performance of the proposed InsPro is good and experiments are abundant.\n\n### Weakness\n\n- The core idea of InsPro about using query propagation across frames has been explored in several works [1][2]. TrackerFormer[1] and EfficientVIS[2] adopt a similar way to propagate queries across frames and also can be reshaped into an online method. I'm concerned about the novelty of the proposed InsPro and the detailed comparisons among InsPro, TrackerFormer, and EfficientVIS.\n- Line269, Sec 4.3: As far as I know, IFC can run near-offline(T=2) inference, see [3].\n\n\n[1] Meinhardt et.al. TrackFormer: Multi-Object Tracking with Transformers.  \n[2] Wu et.al. Efficient Video Instance Segmentation via Tracklet Query and Proposal.  \n[3] Hwang et.al. Video Instance Segmentation using Inter-Frame Communication Transformers.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bkTkc0HIqOk", "reviewer": "Reviewer_o3dJ", "summary": "The paper proposes InsPro for online video instance segmentation. Instance queries are propagated and updated from the previous frames to current frame with implicit object association. Intra-query attention, temporally consistent matching, and box deduplication loss\nare also proposed. Experiments are conducted on the Youtube-VIS 2019 and 2021.\n", "questions": "I will consider raise my rating if the concern in the weakness part can be well addressed. The paper neglects discussion/comparison with related works in query-based VIS and temporal object propagation. The key idea of query propagation has been explored in TransTrack, which influences the paper novelty a lot.", "limitations": "There is no limitation or potential negative societal impact discussion in the paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1. The paper has a good motivation to solve online VIS problem with implicit query matching. The proposed method achieves both good evaluation performance and inference speed.\n\n2. The paper shows the effect of the proposed query/proposal propagation, intra query attention and BDL with adequate ablation experiments. \n\n3. The paper is organized well with good writing and figures.\n\n**Weakness**:\n\n1. Missing discussion for related works in query-based VIS methods. In the paragraph for Query-based Methods, both IFC[a] and VisTr [b] are neglected. Also, SeqFormer[c] and Mask2Former[d] are also query-based VIS although they are arXiv but worth mentioning.\n\n2. Missing related works in clip-level VIS Methods, where the online VIS method PCAN [e] also proposes the online object feature propagation and updating for the target tracklet. The tech differences comparison to [e] should be discussed in the related work section as well as the table results comparison on Youtube-VIS. \n\n[a] Video instance segmentation using inter-frame communication transformers. NeurIPS, 2021.\n\n[b] End-to-end video instance segmentation with transformers. CVPR, 2021.\n\n[c] SeqFormer: a Frustratingly Simple Model for Video Instance Segmentation. arXiv:2112.08275\n\n[d] Mask2former for video instance segmentation. arXiv preprint arXiv:2112.10764 (2021)\n\n[e] Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation. NeurIPS, 2021.\n\n\n3. The tech contribution of InsPro is limited. The idea of query propagation has been adopted in both TransTrack and MOTR. The frame-level InsPro is a straightforward combination between Sparse R-CNN (for query-based object detection) and CondInst(providing mask head). Then, the frame-level InsPro is further extended to video by instance query propagation during online inference.\n\n4. What are the typical failure cases of the methods? How to handle new objects with similar appearance? How to handle/correct the accumulation error during propagation process, for example, an instance query is wrongly matched at an early inference stage?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NFJ7wLIF63t", "reviewer": "Reviewer_kcVg", "summary": "The paper proposes a framework for the video instance segmentation task. In order to avoid explicit instance association, which is responsible for computation overhead increment, the author designed both instance query and proposal propagation mechanism.\nThe framework consists of an intra-query attention module, temporally consistent matching, and an additional loss for removing duplicated predictions.", "questions": "1. I notice that some of the numbers in Table 1a are different from those in other papers.\n\nFor example, according to Table2 in IFC[10] paper, the number of VisTR and IFC are:\n\nName | AP | AP50 | AP75 | AR1 | AR10\n\nVisTR | 35.6 | 56.8 | 37.0 | 35.2 | 40.2\nIFC | 41.2 | 65.1 | 44.6 | 42.3 | 49.6\n\nAlthough that does not necessarily raise a concern about the result, I wonder what could cause the difference.\n\n\n2. In L301-313, the author argues the performance difference results from temporal propagation, yet it could simply be caused by the model size difference. For a fair comparison, the author should also provide the model parameter size after the replacement.", "limitations": "Please refer to Questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n1. The proposed method is simple and reasonable.\n2. The paper is clearly presented and well-organized.\n3. The proposed method outperforms all baselines including current state-of-the-art methods.\n\n[Weaknesses]\nPlease refer to Questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept