Title: Video Instance Segmentation using Inter-Frame Communication Transformers
Authors: Sukjun Hwang
Miran Heo
Seoung Wug Oh
Seon Joo Kim
Venue: NeurIPS 2021
Keywords: video
instance segmentation
video instance segmentation
tracking
transformers

Abstract: We propose a novel end-to-end solution for video instance segmentation (VIS) based on transformers. 
Recently, the per-clip pipeline shows superior performance over per-frame methods leveraging richer information from multiple frames. 
However, previous per-clip models require heavy computation and memory usage to achieve frame-to-frame communications, limiting practicality.
In this work, we propose Inter-frame Communication Transformers (IFC), which significantly reduces the overhead for information-passing between frames by efficiently encoding the context within the input clip.
Specifically, we propose to utilize concise memory tokens as a means of conveying information as well as summarizing each frame scene.
The features of each frame are enriched and correlated with other frames through exchange of information between the precisely encoded memory tokens.
We validate our method on the latest benchmark sets and achieved state-of-the-art performance (AP 42.6 on YouTube-VIS 2019 val set using the offline inference) while having a considerably fast runtime (89.4 FPS). 
Our method can also be applied to near-online inference for processing a video in real-time with only a small delay.
The code is available at https://github.com/sukjunhwang/IFC


Review discussion:
# Strength
- Using memory tokens to perform inter-frame representation learning is highly motivated.
- The proposed method is highly efficient and effective, e.g., 46.5 fps/41.0 AP for near-online inference.
# Weakness
- Limited contributions. For each component in the proposed framework, only the memory token is the new technique. Others are proposed in previous papers, e.g., DETR and VisTR.
- According to the ablations, the biggest improvement (2.2 AP) is from the bipartite matching strategy, i.e., using mask-based matching instead of box-based one. It makes this work less impressive.
- The conclusion of 'we find that the memory token has more interests to instances that are relatively difficult to detect;' is not convincing. Are the visualized figures from only one of the memory tokens?  More analysis is needed to support this claim.
- The conditional convolutional weights, i.e., single-layer 1x1 conv, look more similar to SOLOv2 [A] instead of CondInst [7]. Related discussion is missed.
- Many typos and grammar errors, e.g.,  L137, L188, L291
[A] Wang et al. Solov2: Dynamic, faster and stronger.  In NeurIPS, 2020.
No
We appreciate the reviewer for the thoughtful review and constructive feedback. We hope the following addresses all your concerns and questions.
Q1: Concerns about limited contributions.
A1: The core contribution of our paper is IFC, which significantly reduces the complexity of the encoder.
In addition to the higher efficiency, using IFC leads to better performance.
Furthermore, we explored and proposed various extensions of 2D to 3D as follows: efficient generation of clip-wise instance masks, using space-time masks for tracking and sequence matching.
While some components of our model are proposed previous works, they have not yet been used in the VIS task or not regarded as a mainstream approach. Each architectural design choice of our model surely contributes to the high accuracy and efficiency.
Q2: “The conclusion of 'we find that the memory token has more interests to instances that are relatively difficult to detect;' is not convincing. Are the visualized figures from only one of the memory tokens?”
A2: All the visualized figures are from a fixed head of a fixed memory token (the visualization process is explained in Reviewer#pxiA-A6). The ground-truth labels of the YouTube-VIS testset are not publicly available.
Therefore, we investigated several visualized heatmaps and came out with the qualitative analysis. Since it cannot be quantitatively proved, we will tone down our claim in the revision.
Q3: Missing discussions regarding SOLOv2.
A3: Thanks for your suggestion. We shall add discussions and cite SOLOv2.
Q4: Typos and grammar errors.
A4: Thanks for pointing out the typos and grammatical errors. We will thoroughly fix all mistakes in the revision.
Strength:
1, The overall writing is good.
2, The experiment results on YouTube-VIS are good.
3, The motivation is clear: reducing computation cost and of transformer and reducent information in video by mean of passing tokens.
No
We appreciate the reviewer for the thoughtful review and constructive feedback. Our answers for the questions are as follows.
Q1: Concerns regarding the novelty.
A1: We agree there exists many commonalities between our model and VisTR as both follow the transformer encoder-decoder structure proposed by DETR. VisTR utilizes the space-time full self-attention in the encoder, which results in more computational overhead as the resolutions and number of frames increase.
Therefore, our major focus was to improve the encoder. IFC dramatically reduces the computation while achieving higher accuracy, thus bringing huge practicality.\
The main purposes of the memory in IFC and Max-DeepLab [1] are different. The memory in Max-DeepLab is similar to Object Queries in DETR in that each is used to come out with a prediction of an instance.
Therefore, there is no separate encoder-decoder structure in Max-DeepLab.
In contrast, the memory tokens in IFC are not constrained to represent an instance.
Rather, the tokens serve to convey information throughout the clip.\
Following the concept of query-reference frame, APM [2] aggregates features from past frames and uses them for predicting the current frame. It means that if given a clip of length T, APM [2] should run T times in order to predict all T frames in the clip.
In contrast, each frame taken into IFC simultaneously communicates and updates each other.
Therefore, IFC can come up with the predictions of T frames with a single forward procedure.
[1] MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers, CVPR 2021\
[2] Temporally Distributed Networks for Fast Video Semantic Segmentation, CVPR 2020
Q2: “It is unfair to compare privous work including SG-Net and CrossVIS since most previous work only uses previous frame information. This paper and VisTR use the whole clip as inputs.”
A2: It is true that offline methods are more likely to be accurate than online methods.
However, we believe comparing our work to both online and offline methods will be highly informative to the readers. In the revision, we shall further clarify online and offline in Table 2a.
Q3: “The FPS is also misleading since VisTR-framework process one clip each time rather than one image.”
A3: All FPS results were measured using the exact same environment and formula of (total number of frames) / (total time taken for processing).
Q4: What is the pretrained model result on COCO?
A4: The results on COCO are in Table 3b in the main paper. The result is about 35.1 AP.
Q5: What is the difference between token and query in DETR?
A5: The memory tokens in IFC do not specifically represent instances.
Rather, they serve to convey information throughout the clip.
In contrast, each object query in DETR is directly used for the predictions of an instance. Our model also has queries for the final predictions (‘Object queries’ in Figure 1).
* The proposed memory tokens are memory-efficient and effective at learning temporal relationships between frames.
* The paper provides extensive experiments and ablation studies.
* The paper is overall well-written and easy to follow.
* It seems that the proposed method provides a good efficiency-accuracy trade-off. How does it compare with MaskProp [10]?
* If I understand it correctly, the span of instance tracking is controlled by T (which defaults to 5 time stamps). How does the proposed model handle long-term tracking of the instances (e.g. re-appeared objects after 5+ or T+ frames)? How much is the performance drop caused by missing such long-term tracking?
* Why is the reported score in Table 2 (41.0 AP) and in Table 3 (39.0) different? Is it that one is the best score and the other is average score out of multiple runs (5 runs)? I could not find the explanation for this gap, and this should be clarified.
* The proposed method looks similar to 'decomposed T + HW' attention. How is the 'T + HW' version implemented? and why is it worse than the proposed method?
* How is the 'full THW' implemented? It's opposed to an expectation that the 'full THW' model should be the upper bound in terms of accuracy. What is missing in the 'full THW' model, and what makes the proposed method perform better?
No
We appreciate the reviewer for the thoughtful review and constructive feedback. Our answers for the questions are as follows.
Q1: “It seems that the proposed method provides a good efficiency-accuracy trade-off. How does it compare with MaskProp?”
A1: There is no reported inference speed or publicly available official code of MaskProp.
Therefore, the efficiency-accuracy trade-off of MaskProp could not be accurately measured. Considering the architectural design and many components [1, 2, 3, 4] that MaskProp utilizes, we infer that the upper bound of FPS is around 5.
[1] Hybrid task cascade for instance segmentation, CVPR 2019\
[2] Aggregated residual transformations for deep neural networks, CVPR 2017\
[3] Object detection in video with spatiotemporal sampling networks, ECCV 2018\
[4] Deformable convolutional networks, ICCV 2017
Q2: “How does the proposed model handle long-term tracking of the instances? How much is the performance drop caused by missing such long-term tracking?”
A2: As the reviewer pointed out, the span of instance tracking is controlled by the window size T.
Therefore, using a bigger window size leads to the higher chance of successfully dealing with instances that reappear after a long time interval. Due to the ground-truth of YouTube-VIS testset is not publicly available, we could not quantitatively measure the performance drop.
Q3: “Why is the reported score in Table 2 and in Table 3 different?”
A3: Sorry for the confusion. Due to the insufficient number of test videos in YouTube-VIS dataset, the scores may vary by each run. To provide a better analysis, the ablation studies (Table 3a, 3c, and 3d) are filled with mean values of five runs.
Meanwhile the SOTA comparison (Table 2a) reports the max score of the five runs. We will further specify this in the revision.
Q4: How is 'T + HW' version implemented?
A4: A ‘T + HW’ transformer encoder block is composed as follows.
Firstly, the frame features pass through a spatial transformer encoder (among frame features sharing the same frame index).
Secondly, a temporal transformer encoder is applied (among frame features sharing the same spatial index).
Q5: How is 'Full THW' implemented?
A5: A ‘Full THW’ transformer encoder receives all frame features in a clip at once, which is the exact same as that of VisTR.
Q6: Why does IFC perform better than ‘T + HW’ and ‘Full THW’?
A6: While temporal communication is built only between the same spatial indices in ‘T+HW’, the memory tokens of IFC have a spatially global receptive field. The flexibility of IFC can ease dynamical approaches to relevant information and be beneficial for tracking instances.\
Since ‘Full THW’ involves more features, the accuracy of ‘Full THW’ could be expected to be the upper bound.
However, the more information can result in confusing the model of finding relevant information.
In other words, concisely encoding information and simplifying the routes of accessing information can help models.
[Originality]
The main contribution of this paper lies in presenting a solid transformer-based video instance segmentation framework that pushes the state-of-the-art result significantly and is fast. The idea of introducing memory tokens for effective intra- and inter-frame relation computation is novel and interesting. ​
However, there are also several ideas that are not technically new.
- The similar memory-token-based relation computation is also described in the following work.
However, I will not count the following paper as a concurrent work since it is published in arXiv near the submission. Please compare the proposal with the following work.
[1] End-to-End Video Object Detection with Spatial-Temporal Transformers, arXiv 2021
- The fixed object queries which are shared across the frame are also tried in the original VisTR paper (please refer to the original paper Table-2).
- The idea of using adaptive convolutional weights for instance segmentation is a promising technique that is verified in various previous literature (e.g., CondInst, SOLO, Max-deeplab).
- Using an FPN-like spatial decoder obviously gives an extra performance gain in pixel-level prediction tasks.
- The mask-based sequence matching is a direct extension of VisTR's box-based matching.
While clearly effective, it consumes more memory than the box-based counterpart.
- The final loss functions (i.e., dice loss + focal loss) are from the VisTR.
- The 3D IoU score for clip-linking is a direct video-level extension of MaskProp's averaged 2D IoU scores.
[Quality]
- The authors well-combined the previous techniques into a single framework.
- Extensive ablation studies show the efficacy of the proposals.
- The video results are quite impressive.
[Clarity]
- As the key technical contribution is on the memory token, please discuss and compare it with the recent space-time factorization methods in more detail.
For example, the following paper also introduces space-time factorization methods for video embeddings in the transformer.
[1] ViViT: A Video Vision Transformer, arXiv 2021
[2] Is Space-Time Attention All You Need for Video Understanding?, ICML 2021
- The method section is well-written and easy to follow. Table-1 shows a clear advantage of the proposal over the baselines.
- Typos
[1] Line 291 Our model is shows -> Our model shows
[Significance]
- While there are several components that are not new, each component is combined in a novel way and the final model achieves non-trivial gain over the baselines both in accuracy and speed.
- As transformer-based video models are becoming more and more important these days, the work provided has several strong points (memory token-based transformer encoder design and efficient tracklet matching) that need to be shared with the community.
[Question & Suggestion]
- The position encoding remains to be the same as VisTR. Did the authors have tried to drop it or try different types of embeddings?
For example, the position prior can be embedded in the self-attention operation such as Swin Transformer.
- Why do more memory tokens fail to provide performance gain (e.g., 8 vs. 16 in Table 3-(c))? More discussions need to be provided in the manuscript.
- Please detail the visualization process in Figure 2.
- Please provide a full model FLOPS comparison by adding an extra column in Table 3.
- The authors can try to make the whole framework purely based on the Transformer (e.g., replacing the ResNet backbone with SETR).
Yes
We appreciate the reviewer for the thoughtful review and constructive feedback. Our answers for the questions are as follows.
Q1: “Please compare the proposal with the following work [1].”
A1: We appreciate the reviewer for sharing the recent work, TransVOD [1].
The main architectural design of TransVOD is based on the concept of ‘query-reference’. Given a clip, information from reference frames are aggregated using deformable attention for improving the predictions made at the current frame (query). In order to come up with predictions for all frames in a clip of length T, TransVOD must run T times.
In contrast, all frames taken into IFC are treated equally. IFC makes frame features and memory tokens alternately interact and update each other. This aspect of IFC leads to high efficiency when generating predictions for all frames in each clip, which is necessary for tracking instances using space-time soft IoU.
[1] End-to-End Video Object Detection with Spatial-Temporal Transformers, arXiv 2021
Q2: There are several ideas that are not technically new.
A2: Since the core contribution of our paper is IFC, we agree that there are several components that have been used from previous works (e.g., adaptive convolutional weights and FPN-like spatial decoder).
However, the components were either remaining underrated of its potential or have not been explored for video instance segmentation. Each network design choices and proposed extensions from 2D to 3D (e.g., space-time instance segmentation, sequence matching and clip-linking) greatly improves overall accuracy and efficiency.
Q3: Comparison with the recent space-time factorization methods.
A3: As the reviewer mentioned, there has been recent works regarding the space-time factorization methods [1, 2].\
ViViT [1] and TimeSformer [2] have proposed various decomposition methods to naïve full self-attention over space-time. The decompositions show promising results while effectively reducing overall computations.\
TimeSformer and Model 3 of ViViT paper have two self-attentions that are computed independently from each other: spatial and temporal self-attentions. The spatial self-attention is over all tokens having the same temporal index, and vice-versa for the temporal self-attention. Iteratively switching the two self-attentions makes encoding space-time information possible. This strategy is similar to Axial-DeepLab [3] decomposing 2D self-attention into two 1D self-attentions, and also to R(2+1)D [4] decomposing C3D [5] into space and time.\
Model 2 of ViViT paper is similar to IFC in that it accumulates spatial information into a single frame-level representation token. As effectively aggregating the global context is the key to the success of classification tasks, the use of a single token could be effective in the video classification task.
Compared to the video classification task, the video instance segmentation (VIS) task requires more localized information in order to cope with instance relations.
Furthermore, the frame features should be updated using aggregated information for better segmentation predictions.
Therefore, IFC utilizes multiple memory tokens to encode frames which is more effective than using a single token (Table 3c).
Moreover, frame features and memory tokens alternately interact and update each other in IFC, which lead to high segmentation results.
In the revision, we shall add these comparisons and discussions. We appreciate your suggestion.
[1] ViViT: A Video Vision Transformer, arXiv 2021\
[2] Is Space-Time Attention All You Need for Video Understanding?, ICML 2021\
[3] Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation, ECCV 2020\
[4] A Closer Look at Spatiotemporal Convolutions for Action Recognition, CVPR 2018\
[5] Learning Spatiotemporal Features with 3d Convolutional Networks, ICCV 2015
Q4: Different types of positional embeddings.
A4: We tried two positional embeddings to frame features: 3D spatio-temporal encoding of VisTR, and 2D spatial encoding of DETR. We also tested the effects of adding temporal embeddings to the memory tokens. Among the variations, using spatial encodings for the frame features and keeping memory tokens without the temporal embeddings showed highest accuracy. We will more specify about the embeddings in the revision.
Q5: Why using 8 tokens shows the highest accuracy?
A5: We do not believe 8 is the optimal number for all circumstances.
For example, when handling complex scenes where multiple instances are involved, adopting more memory tokens would be helpful.
In contrast, specifying relevant memory tokens would become easier with a smaller number.
So, the smaller number of tokens might come up with better results for relatively monotonic scenes.
Q6: Details of the visualization process in Figure 2.
A6: We first collected the output of the last Encode-Receive module and visualized the attention scores of each head of each memory token. From the visualized outputs, we discovered that there constantly exists a head of a certain memory token which shows interests to foreground instances. We then fixed the head for visualizing all qualitative results (figures and videos in the main paper and supplementary).
Q7: A full model FLOPS comparison and pure-transformer framework.
A7: Thanks for your suggestions. We shall consider them in the revision.
Q8: Typos
A8: We thank the reviewer for pointing out the typo. We will make sure to fix all typos in the revision.

Meta-review:
role: PC
data: comment: None of the reviewers recommended accepting this paper.
After reading the author response and other reviews one of the reviewers also reduced their score.
One of the common critiques of the work was around the degree of novelty provided by the work.
One of the initially more positive reviewers did recognize that the memory-token based inter-frame attention is technically new and felt the experiments here were quite extensive. But this reviewer also felt that the main performance improvement in this method were coming from the other elements of the method. 

The AC recommends that this paper be rejected.


Decision: Accept (Poster)