Video Instance Segmentation using Inter-Frame Communication Transformers
Authors: Sukjun Hwang
Miran Heo
Seoung Wug Oh
Seon Joo Kim
Venue: NeurIPS 2021 Poster (2021)
Keywords: video
instance segmentation
video instance segmentation
tracking
transformers

Abstract: We propose a novel end-to-end solution for video instance segmentation (VIS) based on transformers. 
Recently, the per-clip pipeline shows superior performance over per-frame methods leveraging richer information from multiple frames. 
However, previous per-clip models require heavy computation and memory usage to achieve frame-to-frame communications, limiting practicality.
In this work, we propose Inter-frame Communication Transformers (IFC), which significantly reduces the overhead for information-passing between frames by efficiently encoding the context within the input clip.
Specifically, we propose to utilize concise memory tokens as a means of conveying information as well as summarizing each frame scene.
The features of each frame are enriched and correlated with other frames through exchange of information between the precisely encoded memory tokens.
We validate our method on the latest benchmark sets and achieved state-of-the-art performance (AP 42.6 on YouTube-VIS 2019 val set using the offline inference) while having a considerably fast runtime (89.4 FPS). 
Our method can also be applied to near-online inference for processing a video in real-time with only a small delay.
The code is available at https://github.com/sukjunhwang/IFC

Video Instance Segmentation using
Inter-Frame Communication Transformers
Sukjun Hwang'
Miran Heo' Yonsei University
Seoung Wug Oh Adobe Research
Seon Joo Kim
{sj hwang, miran, ponjoookim@@yoneeee ac. kr
seoh@adobe.com com
Abstract
We propose novel end-to-end solution for video instance segmentation (VIS) based on transformers. Recently, the per-clip pipeline shows superior performance over per-frame methods leveraging richer information from multiple frames. How- ever, previous per-clip models require heavy computation and memory usage achieve trame-to-frame communications, limiting practicality. this work, we pro- pose Inter-frame Communication Transformers (IFC), which significantly reduces the overhead for information passing between frames by efficiently encoding the context within the input clip. Specifically, propose utilize concise memory tokens as a means of conveying information as well as summarizing each frame scene. The features of each frame are enriched and correlated with other frames through exchange information between the precisely encoded memory tokens. We validate our method on the latest benchmark sets and achieved state-of-ther of-the-art performance (AP 42.6 on YouTube- VIS 2019 val set using the offline inference) while having a considerably fast runtime (89.4 .4 FPS). Our method can also be applied to near-online inference for processing a video in real-time with only a small delay. The code is available at peatttaa//githaeee com/ suk jwwkkukkanngnnien..
Introduction
With the growing interest toward the video domain in computer vision, the task of video instance segmentation (VIS) is emerging I Most the current approaches [ 3 extend image instance segmentation models [
and take frame-wiid wise inputs. These per- frrmfeame methods extend the concept of temporal tracking by matching frame-wist wise predictions of high similarities. The models be easily customized to real-world applications as they run in online [9] fashion, but they show limitations in dealing with occlusions and motion blur that are common in videos.
On the contrary, per-clip models designed to overcome such challenges by incorporating multiple frames while sacrificing the efficiency. Previous per-clip approaches [10] tion within clip to generate instance-specific features. As the features are generated per instance, the number instances addition to the number of frames has a significant impact on the overall computation. Recently proposed VisTR [IT] adapted DETR [3] to the VIS task and reduced the inference time by inserting the entire video, not a clip, to its offline end-to-end network. However, its full self-attention transformers [14] over the space-time inputs involve explosive computations and memories. In this work, we raise the following question: can a per-clip method be efficient while attaining great accuracy?
aggregate informa-
To achieve our goal, we introduce Inter-frame Communication Transformers (IFC) to greatly reduce
the computations of the full space-time transformers. Similar to recent works [5 16 171 that alleviate the explosive computational growth inherent in attention-based models [4 18 IFC takes decomposition strategy utilizing two transformers. The first transformer (Encode-Receive, 8) encodes
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

each frame independently. To exchange the information between frames, the second transformer 9) executes attention between a small number of memory tokens that hold concise information of the clip. The memory tokens are utilized to store the overall context of the clip, for example "a hand over a lizard" in Fig. [ The concise information assists detecting the lizard that is largely occluded by the hand in the first frame, without employing an expensive pixel-level attention over space and time. The memory tokens are only in charge of the communications between frames, and the features of each frame are enriched and correlated through the memory tokens.
We further reduce overheads while taking advantage of per-clip pipelines by concisely representing each instance with a unique convolutional weight [7]. Despite the changes of appearances at different frames, the instances of the same identity share commonalities because the frames originated from the same source video. Therefore, we can effectively capture instance-specific characteristics in a clip with dynamically generated convolutional weights. In companion with the segmentation, we track instances by uniformly applying the weights to all frames in a clip. Moreover, all executions of our spatial decoder are instance-agee -agnostii except for the final layer which applies instance-specife weights. Accordingly, our model is highly efficient and also suitable for scenes with numerous instances.
In addition to the efficient modeling, we provide optimizations and an instance tracking algorithm that are designed to be VIS-centric. By the definition of ApVIS, the VIS task [ aims to maximize
the objective similarity: space-time simee-timeeie mask IoU, Inspired by previous works [3 19 20 , our model is optimized to maximize the similarity between bipartitely matched pairs of ground truth masks and predicted masks. Furthermore, we again adopt the similarity maximization for tracking instances of same identities, which effectively links predicted space-time masks using bipartite matching. As both of our training and inference algorithms are fundamentally designed to address the key challenge of VIS task, our method attains an outstanding accuracy.
From these improvements, IFC sets the new state-of-the-art: 42.6% AP and more surprisingly, in 89.4 fps. Furthermore, our model also shows great speed-accuracy balance under near-online settings, which leads to a huge practicality. We believe that our model can be a powerful baseline for video instance segmentation approaches that follow the per-clip execution.
2
Related Work
Video instance segmentation The VIS task 1 extends the concept of tracking to the image in- stance segmentation task. The early solutions [ 2 follow the per-frame pipeline, which utilize additional tracking head to the models that are mainly designed to solve image instance segmen- tation. More advanced algorithms that are recently proposed consideration, which result in improved performance.
take video characteristics into
Per-clip models 10
dedicate computations to extract information from multiple frames for higher accuracy. By exploiting multiple frames, per-clip models can effectively handle typical challenges in video, i.e., motion blurs and occlusions. Our model is designed to be highly efficient while following the per-clip pipeline, which leads to fast and accurate predictions.
Transformers Recently, transformers 14] are greatly impacting many tasks in computer vision. After the huge success of DETR [3]. which has brought a new paradigm to the object detection task, numerous vision tasks are incorporating transformers [1 22 in place of CNNs. For classification tasks in both NLP and computer vision, many adopt an extra classification token to the input of transformers [21 23 All the input tokens affect each other as the encoders are mainly composed of the self-attention, thus the classification token can be used to determine the class of the overall input. Similarly. DeiT (24] inserts additional distillation token to transformers, and the novel usage leads to higher data efficiency. MaX-Deepla (20) adopted the concept of memory and proposed a novel dual-path transformer for the panoptic segmentation task [25]. By making use of numerous memory tokens to convey information, MaX- Deeepab integrates the transformer and the CNN by making both feedback itself and the other.
We further utilize the concept of the memory tokens to the videos. Using Inter-frame Communication Transformers, each frame runs independently while sharing their information with interim communi- cations. The communications lead to higher accuracy while the execution independence between frames accelerates the inference.

---

Spurii
Transforner Faeeader
G
f
xNg
mf
Transtimnet Encoder
Transformrr Frooder
Tratsfeemer Escoder
Transfomer Decoder
.N
of
CNN
1/"mm
Encoder output
Object queries
2D posirional encrding
Memory taken
Frrmo tokes
Figure 1: Overview of IFC framework. Our transformer encoder block has two components: I) Encode-Receive (E) simultancously encodes frame tokens and memory tokens. 2) Only memory tokens pass the stack of NE encoder blocks goes into two modules, spatial decoder and transformer decoder, to generate segmentation masks.
(G) to perform communications between frames. The output from
3 Method
The proposed method follows a per-clip pipeline which takes a video clip as input and outputs clip-level results. We also introduce Inter-frame Communication Transformers, which can effectively share frame-wise information within a clip with high efficiency.
3.1 Model architecture
Inspired by DETR 113, our network consists of CNN backbone and transformer encoder-decoder layers (Fig-[). [). The input clip is first independently embedded into feature map through the backbone. Then, the embedded clip passes through our inter-frame communication encoder blocks that enrich the feature map by allowing information exchange between frames. Next, a of transformer decoder layers that take the encoder outputs and object queries as inputs predict unique convolutional weights for each instance in the clip. Finally, the masks for each instance across the clip are computed in one shot by convolving the encoded feature map with the unique convolutional weight.
Backbone Given an input clip {a:)-- Ho Wox3 composed of T frames with 3 color channels, the CNN backbone processes the input clip frame-by-frame. As the result, the clip is encoded into of low-resolution features, ( RRTHHxXC where C is the number of Wa
channels and H, W
Ho
32
32
Inter-Frame Communication Encoder Given an image, humans can effortlessly summarize the scene with only a few words. Also, frames from a same video share lot of commonalities, the difference between them sufficiently summarized and communicated even with small bandwidth. Based this hypothesis, we propose inter-frame communication encoder make the computation to be mostly frame-wise wise independent with some communications between frames. Specifically, we adopt memory tokens for both summarizing per-frame scenes and the means of communications.
Our encoder blocks are composed of two phases separate transformers: Encode-Receive (E) follow the typical
and Gather- Cotther-Commuiicate (G). Both Encode-Receive and transformer encoder architecture [14], which consists of an addition fixed positional encoding, a multi-head self attenttenoo module, and feed forward network.
Encode- Recevve operates a per-frame manner, taking a frame-level feature map and corresponding memory tokens. Passing through Encode-Receive, we expect two functionalities: (1) image features encode frrmframe information to the memory tokens, and (2) image features receive information of different frames that gathered in the memory tokens. Bather-Communne
operates across frames

---

Table Complexity comparison. Various transformer encoders for space-time time input. As the overall FLOPs can vary by the number of detected instances, listed values are measured only at the encoders.
FLOPs (G)
Communication Type
Complexity per Layer
360 x 640
720
1280
T=5
T=36
T=5
T=36
No Comm
O(C + +CT(HW))) OCC +C(HHW)) O(C*THW CT(HW) P + CT*HW)
5.17
37.23
24.62
177.29
Full THW
6,94
148.70
50.63
1815.38
Decompose T-IW IFC (M = 8)
8.33
60.24
36.73
265.50
O(C + CT(HW)*)
5.52
39.73
25.05
180.39
to form clip-level knowledge. takes the memory tokens from each frame as inputs and performs communications between frames. Alternating two phases through multiple layers, the encoder can efficiently learn consensus representations across frames.
In more detail, given the frame embedding { H we spatially flatten each feature 1 # x xC +
#W*C, The initial memory tokens m of size M are copied per frame and concatenated to each frame feature as follows:
De mpl E R (
t E (1,2,...,TT)
(1)
where - indicates a concatenation of two feature vectors. Note that the initial memory tokens m are trainable parameters learnt during training.
The first phase of IFC Encode- Receiv,, which processes frames individually as follows:
m/ ( I).
where gl denotes the 1-th Mncode-RReeivg layer. With self-attention computed over the frame pixel locations and the memory tokens, the information of each frame can be passed to the memory tokens and vise-versa.
The outputs of Encode- Recevve are grouped by memory indices and formulate the inputs for Gather- Communicate layer. The grouping can be understood as a decomposition of memory tokens, and becomes computationally beneficial when the total size of gathered memory tokens increases.
(m (1), ml (i), ... mb = g' (174 (1), (1) ... mi (i)),
E {1, ... (1, M),
(3)
where G' denotes the I-th Saaher-Communicate layer. The processed outputs are redistributed to the originated frame and get concatenated as m, = (1), m2()), ,.. m.mm] Unlike Encode- Receive, sather-Communicate utilizes the attention mechanism to convey the information from different frames over the input clip.
Defining the i-th inter-frame encoder block (IFC!) as El followed by g', the stack of Ng encoder blocks can be inductively formulated as:
Jf ml] IFC' (If!-), (m,
1 1515 NE.
(4)
where IeNe m, Ne] is the final result. The stacking of multiple encoder layers brings communications between frames, thus each frame can have coincidence to the other, specifying the identities of instances in a given clip.
Complexity comparison In Table I analyze the computational complexity of transformer encoder variants applied for video input in terms of the Big-O complexity and FLOPs. The complexity
of the original transformer encoder layer [14] is OCC2N + CN²), where N is the number of inputs.
Without any communication between frames, No Comm, it shows the smallest amount of computation (OC + CT (HW)))) As indicated Full THW in Table 1 the complexity of VisTR (11) that performs a full space-time self-attention is (T + C(H))) thus either a higher resolution or increase of number of input frames leads to massive increase computations. VisTR bypasses the problem by highly reducing the input resolution and utilizing GPUs with tremendous
Measuredd using flop_count function of 1vcore==0.1.6

---

memory capacity. However, as such solutions cannot resolve the fundamental issues, it is impractical to real-world videos. Moreover, VisTR remains as a complete offline strategy because it takes the entire video as an input.
An intriguing improvement for the naive full self-attention would be the decomposition of the attention into space and time axis [16] 17 26]. In Decompose T-HW, we decompose attention computation into spatial and temporal attention. The complexity of the separation of space-time leads the sum of the two transformer encoder: ( (HW) + C(HW)) ))) and (THW(CCT' )T In comparison to the full attenttten,on, the decomposition lowers the computational growth relative to the number of frames.
Our encoder, IFC, that communicates between frames using the memory tokens leads to huge benefit to the total computations adding only a small amount of computation over No Comm while providing sufficient channels for communication. The complexity of each phase in our proposed encoder is: ((C + M) + CT(HW + 2)) for Encode-Receive and ( M + CT for
respectively. Assuming that M is kept small (e.g.. 8), the computation can be neglected, while the complexity of
needed for approximated OCC + CC(HH)))) as shown in Table I Finally, with respect the number of frames of the input, we can expect approximate linear increase rather than the high increase of computation occurred in VisTR.
can be
Decoders and output heads As depicted in Fig. ! the transformer decoder of our model is stacked with ND layers [14]. Contrary to VisTR, where the number of object queries increases proportionally to the number of frames, our model receives learnt encodings of fixed size N, for object queries. Also, by utilizing these encodings throughout the entire frames, our model can effectively deal with clips of various lengths. A set of projection matrices are applied {fe""" ., yT=1 for the generation of keys and values. The object queries turn into output embeddings by the transformer decoder, and the embeddings are eventually used as an input to the output heads.
There are two output heads on top of the transformer decoder, a class head and a segmentation head, each composed of two fully-comnected layers. The output embeddings from the transformer decoder are independently inserted to the heads, resulting in N₂ predictions per clip. The class head outputs a class probability distribution of instances p(c) E R,, x|E1, Note that the possible classes C e c include no object Ø class in addition to the given classes of dataset.
The segmentation head generates N₂ conditional convolutional weights w E RNxxC in a manner similar to 7 (2)). For the conditional convolution, the output feature of the encoder {fr Ne "T is reused by undoing the flatten operation. For the upsampling, the encoder feature passes through fpn-style (77 spatial decoder without temporal connections resulting in T feature maps that are 1/8 of the input resolution. Finally, the resulting feature maps f' are convolved with each convolutional weight to generate segmentation mask as follows:
8; = {fi o we
(5)
where uy is i-th convolutional weight, indicate x spatial convolution operation, and the result 6 a ppepoal-temporal object mask in shape of Tx xW" where H' Ho W' W. Note that, for an instance, a common weight is applied throughout the video clip. Our spatial decoder is an instance- agnostic design, which much more efficient than mnstance-specife decoders [10] [T 12 B
s
as the
number of detected instances increases. Meanwhile, thanks to our segmentation head which specifies and captures the characteristics of an instance, IFC can conduct both segmentation and tracking at once within a clip.
3.2 Instance matching and loss
To train our network, we first assign the ground truth for each instance estimation and then a set of loss function between each the ground truth and prediction pair. For a given input clip, our model N, ((r(c), ) The ground truth set of the clip can be represented as y; = (c,, s:); C is the target class label including Ø, and 8; is the target mask which is down-sampled to the size of the prediction masks for efficient similarity calculation.
generate fixed-size set of class-labeled masks {0x N,
One-to-one bipartite matching between the prediction set { N and the ground truth set {yi K is performed to find the best assignment of a prediction to ground truth. The objective can be formally
5

---

described as:
K
8 arg max OEEN,
sim (o()))
permutation of a GN Following prior work [13 20 (8), the bipartite matching is efficiently computed using Hungarian algorithm [19].
where sim Vr, (e(i) refers pair-wise similarity
We find that box-based similarity measurement as used in DETR [3] shows weaknesses in matching instances in video clip due to the case occlusion and disappear-and and-reappear. Therefore, we define sym (o(i) to be mask-based term 1l (c:sey (P) (c:) Ao 8o())), where DICE denotes dice coefficients (29].
Given the optimal assignment o, we refer to the K matched predictions and (N₂ K) non-matched predictions as positive and negative pairs respectively. The positive pairs aim to predict the ground truth masks and classes while the negative pairs optimized to predict the Ø class. The final loss is a sum of the losses from positive pairs and negative pairs where each can be computed as follows:
Cpos
(o(s) (c:) +1 DICE(S,, 8a(1)) + X2 FSCAL(S;, 8a(1))
Cross-entropy loss
Dice loss
Sigmoid-focal loss 30
Eneg
Σ
[- (ogg
As (N,-K) -)) is likely to much greater than K, we down-weight Leeg by a factor of 10 to resolve the imbalance, following prior work [13] The goal of video instance segmentation []] is to maximize the space-time IoU between a prediction and ground truth mask. Therefore, our mask-related losses (Dice loss and Sigmoid-focal loss) are spatio-temporally calculated over an entire clip, rather than averaging the losses that are accumulated frame-by-framne
3.3 Clip-level instance tracking
To infer video input that is longer than the clip length, we match instances using the predicted masks of overlapping frames. Let y/ and V/A be the result sets of clip I and A excluding the 2 class. The goal is to perform matching of same identities between pre-collected instance set 3 and
YA We first calculate the matching scores which are space-time time soft IoU at intersecting frames between 3/, and )'^. Then, we find optimal paired indices ãs using Hungarian algorithm [19 to the gathered matching score S E [0, ||)>|, We update (i) by concatenating (a( if S(i (i)) is above certain threshold, and add non-matched prediction sets to V/ new instances. Note that previous per-clip model (MaskProp (10) also utilizes soft IoU for tracking instances, but the matching scores are computed per-frame and averaged for intersecting frames. Different from MaskProp, using space-time soft IoU leads to an accurate tracking as it can better represent definition of mask similarities between clips which brings at most 2% AP increase. The overall tracking pipeline can be effectively implemented in a GPU-friendly manner.
4 Experiments
In this section, we evaluate the proposed method using YouTube- VIS 2019 and 2021 [. For every listed score, we report the mean of five runs the results may vary by each run due to the insufficient number of training and testing set of YouTube- VIS dataset. We demonstrate the effectiveness of model regarding both accuracy and speed. We further examine how different settings affect the overall performance and efficiency of IFC encoder. Unless specified, all models for measurements used Ng = 3, ND 3, stride of and ResNet-50.
4.1 Implementation Details
We used detectron2 33 for our code basis, and hyper-parameters mostly follow the settings of
DETR [3] unless specified. We used AdamW 34 optimizer with initial learning rate of 10-² for transformers, and 10-5 for backbone. We first pre-train the model for image instance segmentation on COCO [35] by setting our model to T 1. The pre-train procedure follows the shortened training

---

Table Evaluations on various settings.
(a) AP and FPS on YouTube-V VIS 2019 val set. For fairness, FPS is measured on same machine, using single RTX 2080Ti GPU. We used the official codes and checkpoints provided by the authors for the mcasurements. We report the clip settings
T: window size.
Method (Settings)
Backbone
1
FPS
AP
APso
AP7s
AR1
AR10
MaskTrack R-CNN
ResNet-50
26.1
30.3
51.1
32.6
31.0
35.5
MaskTrack R-CNN
ResNet-100
31.8
53.0
33.6
33.2
37.6
per-frame
SipMask 2
ResNet--50
35.5
33.7
54.1
35.8
35.4
SG-Net
ResNet-50
34.8
56.1
36.8
35.8
40.8
SG-Net 4
ResNct-101
36.3
57.1
39.6
35,9
43.0
CrossVIS 3
ResNet-50
36.3
56.8
38.9
35,6
40.7
CrossVIS
3
ResNet-101
36.6
57.3
39.7
36.0
42.0
STEm-Seg
32
ResNet-101
3.0
34.6
55.8
37.9
34.4
41.6
VisTR
(7'=36)
ResNet-50
51.1
56.8
37.0
35.2
40.2
VisTR [I]
(1'=36)
ResNet-101
43.5
38.6
61.3
42.3
37.6
44.2
per-clip
MaskProp
10
(7'=13)
ResNet-50
40.0
42.9
MaskProp
10
(T=13)
ResNet-101
42.5
45.6
(T=5)
ResNet-50
46.5
39.0
60.4
42.7
41.7
51.6
OurSotline
(T=36)
ResNet-50
107.1
41.2
65.1
44.6
42.3
49.6
OurSoetine
(7=36)
ResNet-101
89.4
42.6
66.6
46.3
43.5
51.4
(b) Accuracy YTVIS 2021 val set
(d) Effect of strides
Bipartite matching
AP
APse
AP
AP
AP7s
FPS
AP
MaskTrrckk
SipMask
31.7
52.5
34.0
Box. hased
37.5
T= 10
S=5
39.5
42.8
83.0
CIossVIS
54.4
Mask-based
5=8
43.0
Ours
35.2
57.2
37.5
T= 20
S 10
40.4
43.3
95.7
schedule of DETR 13 which runs 300 epochs with a decay the learning rate by a factor of 10 at 200 epochs. Using the pre-trained weights, the models trained targeted dataset using the batch size of 16, each clip composed of T frames downscaled to either 360p or 480p. For the sampling of each clip, a reference frame index / randomly chosen. The remaining T frame indices are then sampled within an interval of 20. The models are trained for 8 epochs, and decays the learning rate by 10 5th epoch.
During inference, our model takes inputs as follows. Let an input video has V frames, T is the number of frames per clip and S is the stride of clips. We start from inserting clip of frame indices [1,7]] and sequentially insert clips of [1 S,T + S), + 25,T T + 25]. .., + ns, + n.S]. repeats until the end frame index T nS is equal to greater than V. If the end frame index of the last clip T + nS is greater than V, we change the frame indices of the last clip to T Vl. The resolution of input videos are downscaled to 360p, which follows MaskTrack R-CNN (1.
4.2 Main Results
You TobeuuueVii VIS 2019 evaluation results We compare our proposed IFC to the state-of-the-ar mod- els in the video instance segmentation task YouTube--VIS 2019 val in Table (a). We measure the accuracy by AP and our model sets the highest score among all online, near-online, and offline models while presenting the fastest runtime. As mentioned earlier, IFC is highly efficient during the inference thanks to three advantages: (1) memory token-based decomposition for transformer encoder (2) instance-agnostic spatial decoder (3) GPU-friendly instance matching. Moreover, our model does not make use of any heavy modules such as deformable convolutions [36] or cascading networks [37]. Thanks these advantages, IFC achieves an outstanding runtime, which is faster speed than online models
During the inference, method is able to freely adjust the length of the clip (T) needed. If the input clip length is set to contain entire video frames, our method becomes an offline method (like
We follow detectron2
33 for measuring FPS.

---

VIsTR
Ours
Vis
Ours
VisTR
Ours
Figure 2: Visualization of predictions from VisTR and our model. Instances with the same identity are displayed the same color.
VisTR [I] that processes the entire video in one shot. As the offline inference can skip matching between clips and maximize the GPU utilization, our method represents surprisingly fast runtime (107.1 FPS). On the other hand, the application requires instant outputs given a video stream, we can reduce the clip length to make our method near-online. In the near-online scenario with T 5, our system still able to process video in real-time (46.5 FPS) with only a small delay.
VIS 2021 evaluation results The recently introduced dataset YouTube-Vii 2021 is an improved version of YouTube- VIS 2019. The newly added videos in the dataset include higher number of instances and frames. For the new dataset, we use 32 memory tokens. In Table (b), refer results reported (3), which evaluated model achieves the best performance.
using official implementations. Again, our
Qualitative result comparison We compare some qualitative results predicted by our model and VisTR [ in Fig. 2 In terms of both tracking accuracy and segmentation quality, IFC yields better results than VisTR.
4.3 Ablation Study
In this section, we provide ablation studies and discuss how different settings impact the overall performance. The experiments are conducted using YouTube- 2019 val set.
Box-based and mask-based bipartite matching We observe how the different policies for bipar- tite matching affect the performance. As our model does is box-free method, we adjust our model to predict bounding boxes similar to VisTR [IT] and conduct bipartite matching [13 [9] using the predicted boxes. The change of optimization from mask-based box-based brings a noticeable performance drop as shown in (c). With the VIS-centric design, the mask-based optimization shows more robustness than box-hased optimizations under typical video circumstances such as instances with heavy overlaps and partial occlusions.
Differing window strides In addition to the clip length we further optimize our runtime placing stride S' between clips, shown in Table 2 IFC can be used in near-online manner, which takes clips that are consecutively extracted from a video. The placement larger stride reduces temporal intersections, which lessens computational overheads but also causes difficulty in matching

---

Table 3: Encoder variations. We show how different encoders affect the overall performance.
(a) Various encoders taking clips of different lengths (see Table 1
T=5
T=10
T=15
T=20
AP
AP76
FPS
AP
AP75
FPS
AP
APTs
FPS
AP
AP7s
FPS
No Comm
37.4
39.9
38.1
38.8
41.6
40.8
39.3
41.7
46.7
39.6
41.9
52.9
Full THW
37.2
40,0
37.6
38.8
41.2
35.5
39.8
42.6
32,9
39.7
42.8
34.8
Decomp T-HWW
37.2
39.8
35.7
38.3
40.9
37.9
38.5
41.5
42.6
39.0
41.9
49.4
IFC
39.0
42.7
36.3
39.6
43.0
39.8
43.7
40.4
43.4
50.2
(c) Number of memory tokens (AP)
(b) Image instance segmenta- tion on COCO val set
(d) Index-wise memory decomposition
T=5
T=10
T=15
T=20
Apooco
APO500
M=1
37.6
39.2
39.4
39.4
T=5
T=10
T=15
T=20
37.9
39.2
39.6
Unified Decomp
38.1
38.9
39.7
39.9
w/o mem
35.0
56.6
M=4
38.0
39.5
39.7
39.9
39,0
39.6
39.8
w/ mem
35.1
56.5
M=8
39.0
39.6
39.8
40.4
M=16
38.1
39.1
39.7
39.9
instances. By enlarging the stride from = 1 to S 3, IFC accomplishes approximately 150% speed improvement with only 0.1% AP drop. The tendency of high speed gain and low accuracy drop persists under various conditions. Therefore, our model can be applied conditions where the enlargement of strides is necessary, i.e,, using devices that are not powerful enough but has to maintain high inference speed.
Various decomposition strategies of encoders In Table L we observed the computational gaps derived from the decomposition of the encoder layers. Extending Table [ how the decomposition strategies affect the accuracy in Table 3
we now investigate the
The models are evaluated with variety of window sizes (T 5, 10, 15. 20) increase of window size T has pros and cons. When matching predictions from different clips, greater T is advantageous due to enlargement of temporal intersections between clips. On the contrary, frames in longer clips are likely to be composed of diverse appearances, which disrupt tracking and segmenting instances within clip. Therefore, the key the performance enhancement is cope with the appearance changes by precisely encoding and correlating space-time inputs.
As shown in Table 3 (a), the full self-attention [II] surpasses the encoder without communications as the length of clips increase. However, the enlargement the window size highly slows down the inference speed, and the improvements are marginal that the tremendous computation and memory usage cannot be compensated. The decomposition of space-time maintains comparable speed even the window is large, but fails to achieve high accuracy.
Our model shows fast inference as the only additional computations of IFC are from utilizing a small number of memory tokens. Furthermore, by effectively encoding the space-time inputs with the communications between frames, IFC can take advantages of enlarging the window size, and surpasses other encoders.
Memory tokens We also study the effects utilizing memory tokens. As mentioned, the motiva- tion of using the memory tokens is to build communications between frames. Different from the video instance segmentation task, the image segmentation task is consisted of single frame. Therefore, the use of the memory tokens does not lead to improvements to the image instance segmentation task as mutual communications cannot be solcly made (see Table 3 (b)). Meanwhile, the utilization of the memory tokens achieves great improvements by effectively passing the information between frames. Results in Table (a, c) demonstrate that the use memory tokens achieves higher accuracy than the encoder without any communications (No comm), which emphasizes the importance of the communications. We evaluate how the size of the memory tokens affect the overall accuracy in Table 3 (c) and set the default size of the tokens M to be 8.
we demonstrated formulation of inputs for Bather-Communicate layer, which groups the outputs of EEnood-RReceiv by memory indices. As aforementioned, the formulation can be considered as a decomposition of memory tokens: insertion to the Bather-Commmnices layer by separate M groups each consisting of T tokens. In Table 3 (d), we investigate the impact of inserting
In Section 3.1

---

Figure 3: Visualizations of results and attention maps of memory tokens.
the unified MT tokens whole. Compared to the unified insertion, the decomposition brings better accuracy as the memories of same indices have more correspondences, which ease the encoders to build attentions in between.
We choose memory index attending foreground instances and visualize the attention map in Fig. As shown in the results of the upper clip, we find that the memory token has more interests to instances that are relatively difficult detect; more attends the heavily occluded car the rear. The clip at the bottom is composed of frames with huge motion blurs and appearance changes. With the communications of memory tokens, IFC successfully tracks and segments the rabbit.
Conclusion
In this paper, we have proposed a novel video instance segmentation network using Inter-frame Communication Transformers (IFC), which alleviates full space-time attention and successfully builds communications between frames. Finally, new state-of-the-ar on the YouTube- VIS dataset. For the future work, we plan to integrate temporal information, which indeed would take a step further to the human video understanding.
network presents rapid inference and sets the
Acknowledgments
This research was grant funded by the Artificial Intelligence Graduate School Program of Yonsei University, under Grant 2020-0-01361, Korea Evaluation Institute of Industrial Technology (KEIT) funded by the Ministry of Trade, Industry and Energy (00073129) and also supported by the Advanced Robotics Laboratory, part of the Future Technology Center LG Electronics.
10

---

References
[1] Yang, L., Y. Fan, N. Xu. Video instance segmentation. In ICCV. 2019.
[2] Cao, J., M. Anwer, H. Cholakkal, et al. Sipmask: Spatial information preservation for fast image and video instance segmentation. In ECCV. 2020.
[3] Yang, S., Y. Fang, X. Wang, et al. Crossover learning for fast online video instance segmentation. In 2021.
[4] Liu, D., Cui, W. Tan, et al. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR. 2021.
[5] He, K., G. Gkioxari, P. Dollar, et al. Mask r-cmn. In ICCV. 2017.
[6] Bolya, D., C. Zhou, F. Xiao, et al. Yolact: Real-time instance segmentation. In ICCV. 2019.
[7] Tian, Z., C. Shen, H. Chen. Conditional convolutions for instance segmentation. In ECCV.
2020.
[8] Chen, H., K. Sun, Z. Tian, al. Blendmask: Top-down meets bottom-up for instance segmenta- tion. In CVPR. 2020.
(9] Luo, W., J. Xing, A. Milan, et al. Multiple object tracking: A literature review. Artificial Intelligence, 2020.
[10] Bertasius, G., I.. Torresani. Classifying, segmenting, and tracking object instances in video with mask propagation. In CVPR. 2020.
[11] Wang. Y., Z. Xu, X. Wang, et al. End-to-end video instance segmentation with transformers. In CVPR. 2020.
[12] Lin, H., R. Wu, S. Liu, al. Video instance segmentation with propose-reduce paradigm. In ICCV, 2021.
[13] Carion, N., F. Massa, G. Synnaeve, et al. End-to-end object detection with transformers. In ECCV. 2020,
[14]
Vaswani, A., N. Shazeer, N. Parmar, et al. Attention is all you need. In NeurIPS. 2017.
[15] Wang, H., Y. Zhu, B. Green, et al. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In ECCV. 2020.
[16] Bertasius, G., H. Wang, L. Torresani. Is space-time attention all you need for video understand- ing? In ICML. 2021.
[17] Arnab, A., M. Dehghani, G. Heigold, et al. Vivit: A video vision transformer. arXiv preprint 1rXiv:2103.156% 2021.
[18] Wang, X., R. Girshick, A. Gupta, et al. Non-local neural networks. In CVPR. 2018.
[19] Kuhn, H. W. The hungarian method for the assignment problem. In Naval research logistics quarterly. 1955.
[20] Wang, H., Y. Zhu, H. Adam, et al. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR. 2021.
[21] Dosovitskiy, A., L. Beyer, A. Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. 2021.
[22] Ranftl, R., A. Bochkovskiy, V. Koltun. Vision transformers for dense prediction. In ICCV. 2021.
[23] Devlin, J., M.-W. Chang, K. Lee, al. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL. 2019.
[24] Touvron, H., M. Cord, M. Douze, al. Training data-efficient image transformers & distillation through attention. In ICML. 2021.
[25] Kirillov, A., K. He, R. Girshick, et al. Panoptic segmentation. In CVPR. 2019.
[26] Tran, D., H. Wang, L. Torresani, et al. A closer look at spatiotemporal convolutions for action recognition. CVPR. 2018.
[27] Lin, T.-Y., P. Dollar, R. Girshick, et al. Feature pyramid networks for object detection. In CVPR. 2017.

---

[28] Stewart, R., M. Andriluka, A. Y. Ng. End-to-end people detection in crowded scenes. In CVPR. 2016.
[29] Milletari, F., N. Navab, S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. 3DV. 2016.
[30] Lin, T.-Y., Goyal, R. Girshick, et al. Focal loss for dense object detection. In /CCV. 2017.
[3]] He, K., X. Zhang, Ren, al. Deep residual learning for image recognition. In CVPR. 2016.
[32] Athar, A., S. Mahadevan, A. Ošep, al. Stem-seg: Spatio-temporal embeddings for instance
segmentation in videos. In ECCV, 2020.
[33] Wu, Y., A. Kirillov, Massa, et Detectron2. stttss//githuh...c/cafcccooserse
detectron2 2019.
[34]
Loshchilov, I,, F. Hutter. Decoupled weight decay regularization. In ICLR. 2019.
[35] Lin, T.-Y. M. Maire, S. Belongie, et al. Microsoft coco: Common objects in context. In ECCV.
2014.
[36] Dai, J., H. Qi, Y. Xiong, et al. Deformable convolutional networks. In ICCV, 2017.
[37] Cai, Z., N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR.
2018.
12


Reviews:
[{"review_id": "w7HzabiT6e6", "reviewer": "Reviewer_G3h2", "summary": "This paper introduces the memory tokens into VisTR. They modified the box-based matching to the mask-based matching.  They also employ instance matching and clip-level tracking. The experiment results on Youtube-VIS2019 and Youtube-VIS2021  demonstrate the effectiveness of the proposed methods. The proposed method also has a high inference speed. The video sequences in the supplementary have high quality.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method has achieved promising results regarding accuracy and efficiency. However, I have some concerns regarding the clarity and significance.\n\n1. The novelty is limited. This paper is mainly based on VisTR. However, in both the introduction and the method sections, it is not clear which parts are proposed by this paper and which part is inspired by VisTR. It will be better if the author can summarize the contribution at the end of the introduction.  The main contribution may be the memory tokens and mask-based tracking. \n\n2. It will be better if the author could include some of the related works regarding the memory tokens.\n\n3. The ablation studies are not clear. It will be better if the author could make more detailed comparisons with VisTR under similar settings. It would be interesting to see how much improvement is brought by the new tracking loss and the memory bank.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "iz6M5kHRe1", "reviewer": "Reviewer_rHZQ", "summary": "The authors propose IFC Transformer for video instance segmentation. Compared to VisTR, the proposed method reduces the computational cost within the input clip. On the YouTube-VIS 2019 and 2021 datasets, the proposed method outperforms the other STOA methods in terms of accuracy and speed.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper is clearly written.\n2. The proposed method not only archives STOA results but also faster inference speed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. How about the segmentation results on similar objects? As you use the global feature matching in Transformers, I think the proposed method may have difficulty distinguishing similar objects.\n2. There are no qualitative comparison results with other methods.\n3. Despite STOA results, what is the reason to choose Transformer? What are the advantages of Transformer?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "PqqtmnYCrGP", "reviewer": "Reviewer_8afQ", "summary": "The paper proposes Inter-frame Communication Transformers (IFC) for video instance segmentation, which utilizes the memory tokens to pass message across frames and enrich frame representation within a clip. Conditional convolution is used for the mask decoder. IFC is validated on YouTube-VIS with SOTA performance in an offline setting.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n* The idea of using concise memory tokens for communication between frames is novel and shown to be effective.\n* The method achieves a SOTA performance on Youtube VIS with less complexity compared to VisTr. \n* The important modules and parameters settings are evaluated in extensive ablative studies. \n* The paper is well written and well organized.\n* Very well-designed method figure helps the understanding.\n\n\nWeaknesses:\n\n1)\tDetails regarding the inference and training is somewhat unclear. The Appendix only add some more details. In particular, exactly how the inference pipeline proceeds and how clips are sampled and handled during training is crucial.\n\n2)\tThe results of most ablative studies and analysis are inconsistent with the state-of-the-art results. In table 2a the Res50 variant achieves AP 41.0. But table 2c, 3a, 3c, and 3d all show different numbers for the corresponding setting (around 39 AP). Please clarify.\n\n3)\tThe importance of the proposed IFC is much less compared to No Comm (table 3a) for larger clip lengths. This is also seen when comparing M=8 with M=1 in table 3c. Please add a discussion in the paper and clarify in the rebuttal.\n\n4)\tI believe that the reason for the effect observed in Q3 is that the cross attention performed in the transformer decoder also allows inter-frame communication. A discussion on this in relation to the proposed IFC is necessary. That is, why is IFC needed when the decoder also allows inter frame communication?\n\n5)\tIn table 2a and 2b it need to be clarified which methods are online (ie causal) and offline (non-causal).\n\nIn summary, this is a strong paper with strong results. The memory tokens is a quite general idea that could be used for other applications. My preliminary rating is therefore positive. The authors however need to address my comments above and update the paper accordingly.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the work is not addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "714TYu5vs8s", "reviewer": "Reviewer_Z1DA", "summary": "This paper focuses on end-to-end solutions for video instance segmentation tasks by resigning transformers. Different from recently proposed per-clip pipelines VisTR, leverages richer information from multiple frames. This paper adopts Inter-frame Communication Transformers(IFC), which reduces overhead by encoding the context with the input clip as a memory token and correlates with other frames through the exchange of memory tokens. Experiments on the Youtube VIS benchmark demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. IFC is interesting. However, I am not clear about the order of frame tokens and memory token intersection. Do memory tokens represent the instances? \n\n2. For the loss, could you mind giving the results replacing cross-entropy loss with focal loss?\n\n3. For transformer encoder and encoder, you use the traditional self-attention or deformable-based self-attention module like in Deformable-DETR.\n\n4. Conditional weight generations have achieved success in instance segmentation, e.g. CondInst and SOLOv2. Could you mind giving the results by using an embedded way?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Could you explain more about memory tokens? Why does this token help compared with total clip-level interaction or sparse-based clip-level interaction, like in video recognition paper?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission discusses video instance segmentation, specifically an inter-frame communication transformer which significantly reduces the overhead of information that is shared between frames. This helps to attain SOTA results while running at an impressive 89FPS. All reviewers appreciated the submission in their initial review leaning positive while raising concerns about about limited novelty, unclear ablation studies, some questions regarding the method (loss, order of frame tokens), inference and training, a missing qualitative comparison.

The authors provided a rebuttal which the reviewers appreciated. They remained concerned particularly about the issue that the authors chose to report the best result in their comparison to SOTA. This is not good practice and the authors are strongly encouraged to fix this. AC assumes that the authors will address this as the average scores provided in the ablation study still outperform SOTA, albeit less significantly.
Decision: Accept (Poster)