Video Instance Segmentation using Inter-Frame Communication Transformers
Authors: Sukjun Hwang
Miran Heo
Seoung Wug Oh
Seon Joo Kim
Venue: NeurIPS 2021 Poster (2021)
Keywords: video
instance segmentation
video instance segmentation
tracking
transformers

Abstract: We propose a novel end-to-end solution for video instance segmentation (VIS) based on transformers. 
Recently, the per-clip pipeline shows superior performance over per-frame methods leveraging richer information from multiple frames. 
However, previous per-clip models require heavy computation and memory usage to achieve frame-to-frame communications, limiting practicality.
In this work, we propose Inter-frame Communication Transformers (IFC), which significantly reduces the overhead for information-passing between frames by efficiently encoding the context within the input clip.
Specifically, we propose to utilize concise memory tokens as a means of conveying information as well as summarizing each frame scene.
The features of each frame are enriched and correlated with other frames through exchange of information between the precisely encoded memory tokens.
We validate our method on the latest benchmark sets and achieved state-of-the-art performance (AP 42.6 on YouTube-VIS 2019 val set using the offline inference) while having a considerably fast runtime (89.4 FPS). 
Our method can also be applied to near-online inference for processing a video in real-time with only a small delay.
The code is available at https://github.com/sukjunhwang/IFC

Video Instance Segmentation using
Inter-Frame Communication Transformers
Sukjun Hwang'
Miran Heo' Yonsei University
Seoung Wug Oh Adobe Research
Seon Joo Kim
{sj hwang, miran, ponjoookim@@yoneeee ac. kr
seoh@adobe.com com
Abstract
We propose novel end-to-end solution for video instance segmentation (VIS) based on transformers. Recently, the per-clip pipeline shows superior performance over per-frame methods leveraging richer information from multiple frames. How- ever, previous per-clip models require heavy computation and memory usage achieve trame-to-frame communications, limiting practicality. this work, we pro- pose Inter-frame Communication Transformers (IFC), which significantly reduces the overhead for information passing between frames by efficiently encoding the context within the input clip. Specifically, propose utilize concise memory tokens as a means of conveying information as well as summarizing each frame scene. The features of each frame are enriched and correlated with other frames through exchange information between the precisely encoded memory tokens. We validate our method on the latest benchmark sets and achieved state-of-ther of-the--rt performance (AP 42.6 on YouTube- VIS 2019 val set using the offline inference) while having a considerably fast runtime (89.4 FPS). Our method can also be applied to near-online inference for processing a video in real-time with only a small delay. The code is available at peatttaa//githaeee com/ suk jwwkkukwanggnnien..
Introduction
With the growing interest toward the video domain in computer vision, the task of video instance segmentation (VIS) is emerging I Most the current approaches [ 3 extend image instance segmentation models [
and take frame-wiid wise inputs. These per- frrmframe methods extend the concept of temporal tracking by matching frame-wist wise predictions of high similarities. The models be easily customized to real-world applications as they run in online [9] fashion, but they show limitations in dealing with occlusions and motion blur that are common in videos.
On the contrary, per-clip models designed to overcome such challenges by incorporating multiple frames while sacrificing the efficiency. Previous per-clip approaches [10] tion within clip to generate instance-specific features. As the features are generated per instance, the number instances addition to the number of frames has a significant impact on the overall computation. Recently proposed VisTR [IT] adapted DETR [3] to the VIS task and reduced the inference time by inserting the entire video, not a clip, to its offline end-to-end network. However, its full self-attention transformers [14] over the space-time inputs involve explosive computations and memories. In this work, we raise the following question: can a per-clip method be efficient while attaining great accuracy?
aggregate informa-
To achieve our goal, we introduce Inter-frame Communication Transformers (IFC) to greatly reduce
the computations of the full space-time transformers. Similar to recent works [5 16 171 that alleviate the explosive computational growth inherent in attention-based models [4 18 IFC takes decomposition strategy utilizing two transformers. The first transformer (Encode-Receive, 8) encodes
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

each frame independently. To exchange the information between frames, the second transformer 9) executes attention between a small number of memory tokens that hold concise information of the clip. The memory tokens are utilized to store the overall context of the clip, for example "a hand over a lizard" in Fig. [ The concise information assists detecting the lizard that is largely occluded by the hand in the first frame, without employing an expensive pixel-level attention over space and time. The memory tokens are only in charge of the communications between frames, and the features of each frame are enriched and correlated through the memory tokens.
We further reduce overheads while taking advantage of per-clip pipelines by concisely representing each instance with a unique convolutional weight [7]. Despite the changes of appearances at different frames, the instances of the same identity share commonalities because the frames originated from the same source video. Therefore, we can effectively capture instance-specific characteristics in a clip with dynamically generated convolutional weights. In companion with the segmentation, we track instances by uniformly applying the weights to all frames in a clip. Moreover, all executions of our spatial decoder are instance-agee -agnostii except for the final layer which applies instance-specife weights. Accordingly, our model is highly efficient and also suitable for scenes with numerous instances.
In addition to the efficient modeling, we provide optimizations and an instance tracking algorithm that are designed to be VIS-centric. By the definition of ApVIS, the VIS task [ aims to maximize
the objective similarity: space-time simee-timeeie mask IoU, Inspired by previous works [3 19 20 , our model is optimized to maximize the similarity between bipartitely matched pairs of ground truth masks and predicted masks. Furthermore, we again adopt the similarity maximization for tracking instances of same identities, which effectively links predicted space-time masks using bipartite matching. As both of our training and inference algorithms are fundamentally designed to address the key challenge of VIS task, our method attains an outstanding accuracy.
From these improvements, IFC sets the new state-of-the-art: 42.6% AP and more surprisingly, in 89.4 fps. Furthermore, our model also shows great speed-accuracy balance under near-online settings, which leads to a huge practicality. We believe that our model can be a powerful baseline for video instance segmentation approaches that follow the per-clip execution.
2
Related Work
Video instance segmentation The VIS task 1 extends the concept of tracking to the image in- stance segmentation task. The early solutions [ 2 follow the per-frame pipeline, which utilize additional tracking head to the models that are mainly designed to solve image instance segmen- tation. More advanced algorithms that are recently proposed consideration, which result in improved performance.
take video characteristics into
Per-clip models 10
dedicate computations to extract information from multiple frames for higher accuracy. By exploiting multiple frames, per-clip models can effectively handle typical challenges in video, i.e., motion blurs and occlusions. Our model is designed to be highly efficient while following the per-clip pipeline, which leads to fast and accurate predictions.
Transformers Recently, transformers 14] are greatly impacting many tasks in computer vision. After the huge success of DETR [3]. which has brought a new paradigm to the object detection task, numerous vision tasks are incorporating transformers [1 22 in place of CNNs. For classification tasks in both NLP and computer vision, many adopt an extra classification token to the input of transformers [21 23 All the input tokens affect each other as the encoders are mainly composed of the self-attention, thus the classification token can be used to determine the class of the overall input. Similarly. DeiT (24] inserts additional distillation token to transformers, and the novel usage leads to higher data efficiency. MaX-Deepla (20) adopted the concept of memory and proposed a novel dual-path transformer for the panoptic segmentation task [25]. By making use of numerous memory tokens to convey information, MaX- Deeepab integrates the transformer and the CNN by making both feedback itself and the other.
We further utilize the concept of the memory tokens to the videos. Using Inter-frame Communication Transformers, each frame runs independently while sharing their information with interim communi- cations. The communications lead to higher accuracy while the execution independence between frames accelerates the inference.

---

Spurii
Transforner Faeeader
G
f
xNg
mf
Transtimnet Encoder
Transformrr Frooder
Tratsfeemer Escoder
Transfomer Decoder
.N
of
CNN
Encoder output
Object queries
2D posirional encrding
Memory taken
Frrmo tokes
Figure 1: Overview of IFC framework. Our transformer encoder block has two components: I) Encode-Receive (E) simultancously encodes frame tokens and memory tokens. 2) Only memory tokens pass the stack of NE encoder blocks goes into two modules, spatial decoder and transformer decoder, to generate segmentation masks.
(G) to perform communications between frames. The output from
3 Method
The proposed method follows a per-clip pipeline which takes a video clip as input and outputs clip-level results. We also introduce Inter-frame Communication Transformers, which can effectively share frame-wise information within a clip with high efficiency.
3.1 Model architecture
Inspired by DETR 113, our network consists of CNN backbone and transformer encoder-decoder layers (Fig-[). [). The input clip is first independently embedded into feature map through the backbone. Then, the embedded clip passes through our inter-frame communication encoder blocks that enrich the feature map by allowing information exchange between frames. Next, a of transformer decoder layers that take the encoder outputs and object queries as inputs predict unique convolutional weights for each instance in the clip. Finally, the masks for each instance across the clip are computed in one shot by convolving the encoded feature map with the unique convolutional weight.
Backbone Given an input clip {a:)-- Ho Wox3 composed of T frames with 3 color channels, the CNN backbone processes the input clip frame-by-frame. As the result, the clip is encoded into of low-resolution features, ( RRTHHxXC where C is the number of Wa
channels and H, W
Ho
32
32
Inter-Frame Communication Encoder Given an image, humans can effortlessly summarize the scene with only a few words. Also, frames from a same video share lot of commonalities, the difference between them sufficiently summarized and communicated even with small bandwidth. Based this hypothesis, we propose inter-frame communication encoder make the computation to be mostly frame-wise wise independent with some communications between frames. Specifically, we adopt memory tokens for both summarizing per-frame scenes and the means of communications.
Our encoder blocks are composed of two phases separate transformers: Encode-Receive (E) follow the typical
and Gather- Cotther-Commuiicate (G). Both Encode-Receive and transformer encoder architecture [14], which consists of an addition fixed positional encoding, a multi-head self attenttenoo module, and feed forward network.
Encode- Recevve operates a per-frame manner, taking a frame-level feature map and corresponding memory tokens. Passing through Encode-Receive, we expect two functionalities: (1) image features encode frrmframe information to the memory tokens, and (2) image features receive information of different frames that gathered in the memory tokens. Bather-Communne
operates across frames

---

Table Complexity comparison. Various transformer encoders for space-time time input. As the overall FLOPs can vary by the number of detected instances, listed values are measured only at the encoders.
FLOPs (G)
Communication Type
Complexity per Layer
360 x 640
720
1280
T=5
T=36
T=5
T=36
No Comm
O(C + +CT(HW))) OCC +C(HHW)) O(C*THW CT(HW) P + CT*HW)
5.17
37.23
24.62
177.29
Full THW
6,94
148.70
50.63
1815.38
Decompose T-IW IFC (M = 8)
8.33
60.24
36.73
265.50
O(C + CT(HW)*)
5.52
39.73
25.05
180.39
to form clip-level knowledge. takes the memory tokens from each frame as inputs and performs communications between frames. Alternating two phases through multiple layers, the encoder can efficiently learn consensus representations across frames.
In more detail, given the frame embedding { H we spatially flatten each feature 1 # x xC +
#W*C, The initial memory tokens m of size M are copied per frame and concatenated to each frame feature as follows:
De mpl E R (
t E (1,2,...,TT)
(1)
where - indicates a concatenation of two feature vectors. Note that the initial memory tokens m are trainable parameters learnt during training.
The first phase of IFC Encode- Receiv,, which processes frames individually as follows:
[/ ( I).
where gl denotes the 1-th Mncode-RReeivg layer. With self-attention computed over the frame pixel locations and the memory tokens, the information of each frame can be passed to the memory tokens and vise-versa.
The outputs of Encode- Recevve are grouped by memory indices and formulate the inputs for Gather- Communicate layer. The grouping can be understood as a decomposition of memory tokens, and becomes computationally beneficial when the total size of gathered memory tokens increases.
(m (1), ml (i), ... mb = g' (174 (1), (1) ... mi (i)),
E {1, ... (1, M),
(3)
where G' denotes the I-th Saahhr-Communicate layer. The processed outputs are redistributed to the originated frame and get concatenated as m, = (1), m2()), ,.. m.mm] Unlike Encode- Receive, Sather-Communicate utilizes the attention mechanism to convey the information from different frames over the input clip.
Defining the i-th inter-frame encoder block (IFC!) as El followed by g', the stack of Ng encoder blocks can be inductively formulated as:
Jf ml] IFC' (If!-), (m,
1 1515 NE.
(4)
where IeNe m, Ne] is the final result. The stacking of multiple encoder layers brings communications between frames, thus each frame can have coincidence to the other, specifying the identities of instances in a given clip.
Complexity comparison In Table I analyze the computational complexity of transformer encoder variants applied for video input in terms of the Big-O complexity and FLOPs. The complexity
of the original transformer encoder layer [14] is OCC2N + CN²), where N is the number of inputs.
Without any communication between frames, No Comm, it shows the smallest amount of computation (OC + CT (HW)))) As indicated Full THW in Table 1 the complexity of VisTR (11) that performs a full space-time self-attention is (T + C(H))) thus either a higher resolution or increase of number of input frames leads to massive increase computations. VisTR bypasses the problem by highly reducing the input resolution and utilizing GPUs with tremendous
Measuredd using flop_count function of 1vcore==0.1.6

---

memory capacity. However, as such solutions cannot resolve the fundamental issues, it is impractical to real-world videos. Moreover, VisTR remains as a complete offline strategy because it takes the entire video as an input.
An intriguing improvement for the naive full self-attention would be the decomposition of the attention into space and time axis [16] 17 26]. In Decompose T-HW, we decompose attention computation into spatial and temporal attention. The complexity of the separation of space-time leads the sum of the two transformer encoder: ( (HW) + C(HW)) ))) and (THW(CCT' )T In comparison to the full attenttten,on, the decomposition lowers the computational growth relative to the number of frames.
Our encoder, IFC, that communicates between frames using the memory tokens leads to huge benefit to the total computations adding only a small amount of computation over No Comm while providing sufficient channels for communication. The complexity of each phase in our proposed encoder is: ((C + M) + CT(HW + 2)) for Encode-Receive and ( M + CT for
respectively. Assuming that M is kept small (e.g.. 8), the computation can be neglected, while the complexity of
needed for approximated OCC + CC(HH)))) as shown in Table I Finally, with respect the number of frames of the input, we can expect approximate linear increase rather than the high increase of computation occurred in VisTR.
can be
Decoders and output heads As depicted in Fig. ! the transformer decoder of our model is stacked with ND layers [14]. Contrary to VisTR, where the number of object queries increases proportionally to the number of frames, our model receives learnt encodings of fixed size N, for object queries. Also, by utilizing these encodings throughout the entire frames, our model can effectively deal with clips of various lengths. A set of projection matrices are applied {fe""" ., yT=1 for the generation of keys and values. The object queries turn into output embeddings by the transformer decoder, and the embeddings are eventually used as an input to the output heads.
There are two output heads on top of the transformer decoder, a class head and a segmentation head, each composed of two fully-comnected layers. The output embeddings from the transformer decoder are independently inserted to the heads, resulting in N₂ predictions per clip. The class head outputs a class probability distribution of instances p(c) E R,, x|E1, Note that the possible classes C e c include no object Ø class in addition to the given classes of dataset.
The segmentation head generates N₂ conditional convolutional weights w E RNxxC in a manner similar to 7 (2)). For the conditional convolution, the output feature of the encoder {fr Ne "T is reused by undoing the flatten operation. For the upsampling, the encoder feature passes through fpn-style (77 spatial decoder without temporal connections resulting in T feature maps that are 1/8 of the input resolution. Finally, the resulting feature maps f' are convolved with each convolutional weight to generate segmentation mask as follows:
8; = {fi o we
(5)
where uy is i-th convolutional weight, indicate x spatial convolution operation, and the result 6 a ppepoal-temporal object mask in shape of Tx xW" where H' Ho W' W. Note that, for an instance, a common weight is applied throughout the video clip. Our spatial decoder is an instance- agnostic design, which much more efficient than mustance-specife decoders [10] [T 12 B
s
as the
number of detected instances increases. Meanwhile, thanks to our segmentation head which specifies and captures the characteristics of an instance, IFC can conduct both segmentation and tracking at once within a clip.
3.2 Instance matching and loss
To train our network, we first assign the ground truth for each instance estimation and then a set of loss function between each the ground truth and prediction pair. For a given input clip, our model N, ((r(c), ) The ground truth set of the clip can be represented as y; = (c,, s:); C is the target class label including Ø, and 8; is the target mask which is down-sampled to the size of the prediction masks for efficient similarity calculation.
generate fixed-size set of class-labeled masks {0x N,
One-to-one bipartite matching between the prediction set { N and the ground truth set {yi K is performed to find the best assignment of a prediction to a ground truth. The objective can be formally
5

---

described as:
K
8 arg max OEEN,
sim (o()))
permutation of a GN Following prior work [13 20 (8), the bipartite matching is efficiently computed using Hungarian algorithm [19].
where sim Vr, (e(i) refers pair-wise similarity
We find that box-based similarity measurement as used in DETR [3] shows weaknesses in matching instances in video clip due to the case occlusion and disappear-and and-reappear. Therefore, we define sym (o(i) to be mask-based term 1l (c:sey (P) (c:) Ag 8o())), where DICE denotes dice coefficients (29].
Given the optimal assignment o, we refer to the K matched predictions and (N₂ K) non-matched predictions as positive and negative pairs respectively. The positive pairs aim to predict the ground truth masks and classes while the negative pairs optimized to predict the Ø class. The final loss is a sum of the losses from positive pairs and negative pairs where each can be computed as follows:
Cpos
(o(s) (c:) +1 DICE(S,, 8a(1)) + X2 FOCAL(S;, 8a(1))
Cross-entropy loss
Dice loss
Sigmoid-focal loss 30
Eneg
Σ
[- (ogg
As (N,-K) -)) is likely to much greater than K, we down-weight Leeg by a factor of 10 to resolve the imbalance, following prior work [13] The goal of video instance segmentation []] is to maximize the space-time IoU between a prediction and ground truth mask. Therefore, our mask-related losses (Dice loss and Sigmoid-focal loss) are spatio-temporally calculated over an entire clip, rather than averaging the losses that are accumulated frame-by-framne
3.3 Clip-level instance tracking
To infer video input that is longer than the clip length, we match instances using the predicted masks of overlapping frames. Let y/ and V/A be the result sets of clip I and A excluding the 2 class. The goal is to perform matching of same identities between pre-collected instance set 3 and
YA We first calculate the matching scores which are space-time time soft IoU at intersecting frames between 3/, and )'^. Then, we find optimal paired indices ãs using Hungarian algorithm [19 to the gathered matching score S E [0, ||)>|, We update (i) by concatenating (a( if S(i (i)) is above certain threshold, and add non-matched prediction sets to V/ new instances. Note that previous per-clip model (MaskProp (10) also utilizes soft IoU for tracking instances, but the matching scores are computed per-frame and averaged for intersecting frames. Different from MaskProp, using space-time soft IoU leads to an accurate tracking as it can better represent definition of mask similarities between clips which brings at most 2% AP increase. The overall tracking pipeline can be effectively implemented in a GPU-friendly manner.
4 Experiments
In this section, we evaluate the proposed method using YouTube- VIS 2019 and 2021 [. For every listed score, we report the mean of five runs the results may vary by each run due to the insufficient number of training and testing set of YouTube- VIS dataset. We demonstrate the effectiveness of model regarding both accuracy and speed. We further examine how different settings affect the overall performance and efficiency of IFC encoder. Unless specified, all models for measurements used Ng = 3, ND 3, stride of and ResNet-50.
4.1 Implementation Details
We used detectron2 33 for our code basis, and hyper-parameters mostly follow the settings of
DETR [3] unless specified. We used AdamW 34 optimizer with initial learning rate of 10-² for transformers, and 10-5 for backbone. We first pre-train the model for image instance segmentation on COCO [35] by setting our model to T 1. The pre-train procedure follows the shortened training

---

Table Evaluations on various settings.
(a) AP and FPS on YouTube-V VIS 2019 val set. For fairness, FPS is measured on same machine, using single RTX 2080Ti GPU. We used the official codes and checkpoints provided by the authors for the mcasurements. We report the clip settings
T: window size.
Method (Settings)
Backbone
1
FPS
AP
APso
AP7s
AR1
AR10
MaskTrack R-CNN
ResNet-50
26.1
30.3
51.1
32.6
31.0
35.5
MaskTrack R-CNN
ResNet-100
31.8
53.0
33.6
33.2
37.6
per-frame
SipMask 2
ResNet--50
35.5
33.7
54.1
35.8
35.4
SG-Net
ResNet-50
34.8
56.1
36.8
35.8
40.8
SG-Net 4
ResNct-101
36.3
57.1
39.6
35,9
43.0
CrossVIS 3
ResNet-50
36.3
56.8
38.9
35,6
40.7
CrossVIS
3
ResNet-101
36.6
57.3
39.7
36.0
42.0
STEm-Seg
32
ResNet-101
3.0
34.6
55.8
37.9
34.4
41.6
VisTR
(7'=36)
ResNet-50
51.1
56.8
37.0
35.2
40.2
VisTR [I]
(1'=36)
ResNet-101
43.5
38.6
61.3
42.3
37.6
44.2
per-clip
MaskProp
10
(7'=13)
ResNet-50
40.0
42.9
MaskProp
10
(T=13)
ResNet-101
42.5
45.6
(T=5)
ResNet-50
46.5
39.0
60.4
42.7
41.7
51.6
OurSotline
(T=36)
ResNet-50
107.1
41.2
65.1
44.6
42.3
49.6
OurSoetine
(7=36)
ResNet-101
89.4
42.6
66.6
46.3
43.5
51.4
(b) Accuracy YTVIS 2021 val set
(d) Effect of strides
Bipartite matching
AP
APse
AP
AP
AP7s
FPS
AP
MaskTrackk
SipMask
31.7
52.5
34.0
Box. hased
37.5
T= 10
S=5
39.5
42.8
83.0
CIossVIS
54.4
Mask-based
5=8
43.0
Ours
35.2
57.2
37.5
T= 20
S 10
40.4
43.3
95.7
schedule of DETR 13 which runs 300 epochs with a decay the learning rate by a factor of 10 at 200 epochs. Using the pre-trained weights, the models trained targeted dataset using the batch size of 16, each clip composed of T frames downscaled to either 360p or 480p. For the sampling of each clip, a reference frame index / randomly chosen. The remaining T frame indices are then sampled within an interval of 20. The models are trained for 8 epochs, and decays the learning rate by 10 5th epoch.
During inference, our model takes inputs as follows. Let an input video has V frames, T is the number of frames per clip and S is the stride of clips. We start from inserting clip of frame indices [1,7]] and sequentially insert clips of [1 S,T + S), + 25,T T + 25]. .., + ns, + n.S]. repeats until the end frame index T nS is equal to greater than V. If the end frame index of the last clip T + nS is greater than V, we change the frame indices of the last clip to T Vl. The resolution of input videos are downscaled to 360p, which follows MaskTrack R-CNN (1.
4.2 Main Results
You TobeuuueVii VIS 2019 evaluation results We compare our proposed IFC to the state-of-the-ar mod- els in the video instance segmentation task YouTube--VIS 2019 val in Table (a). We measure the accuracy by AP and our model sets the highest score among all online, near-online, and offline models while presenting the fastest runtime. As mentioned earlier, IFC is highly efficient during the inference thanks to three advantages: (1) memory token-based decomposition for transformer encoder (2) instance-agnostic spatial decoder (3) GPU-friendly instance matching. Moreover, our model does not make use of any heavy modules such as deformable convolutions [36] or cascading networks [37]. Thanks these advantages, IFC achieves an outstanding runtime, which is faster speed than online models
During the inference, method is able to freely adjust the length of the clip (T) needed. If the input clip length is set to contain entire video frames, our method becomes an offline method (like
We follow detectron2
33 for measuring FPS.

---

VIsTR
Ours
Vis
Ours
VisTR
Ours
Figure 2: Visualization of predictions from VisTR and our model. Instances with the same identity are displayed the same color.
VisTR [I] that processes the entire video in one shot. As the offline inference can skip matching between clips and maximize the GPU utilization, our method represents surprisingly fast runtime (107.1 FPS). On the other hand, the application requires instant outputs given a video stream, we can reduce the clip length to make our method near-online. In the near-online scenario with T 5, our system still able to process video in real-time (46.5 FPS) with only a small delay.
VIS 2021 evaluation results The recently introduced dataset YouTube-Vii 2021 is an improved version of YouTube- VIS 2019. The newly added videos in the dataset include higher number of instances and frames. For the new dataset, we use 32 memory tokens. In Table (b), refer results reported (3), which evaluated model achieves the best performance.
using official implementations. Again, our
Qualitative result comparison We compare some qualitative results predicted by our model and VisTR [ in Fig. 2 In terms of both tracking accuracy and segmentation quality, IFC yields better results than VisTR.
4.3 Ablation Study
In this section, we provide ablation studies and discuss how different settings impact the overall performance. The experiments are conducted using YouTube- 2019 val set.
Box-based and mask-based bipartite matching We observe how the different policies for bipar- tite matching affect the performance. As our model does is box-free method, we adjust our model to predict bounding boxes similar to VisTR [IT] and conduct bipartite matching [13 [9] using the predicted boxes. The change of optimization from mask-based box-based brings a noticeable performance drop as shown in (c). With the VIS-centric design, the mask-based optimization shows more robustness than box-hased optimizations under typical video circumstances such as instances with heavy overlaps and partial occlusions.
Differing window strides In addition to the clip length we further optimize our runtime placing stride S' between clips, shown in Table 2 IFC can be used in near-online manner, which takes clips that are consecutively extracted from a video. The placement larger stride reduces temporal intersections, which lessens computational overheads but also causes difficulty in matching

---

Table 3: Encoder variations. We show how different encoders affect the overall performance.
(a) Various encoders taking clips of different lengths (see Table 1
T=5
T=10
T=15
T=20
AP
AP76
FPS
AP
AP75
FPS
AP
APTs
FPS
AP
AP7s
FPS
No Comm
37.4
39.9
38.1
38.8
41.6
40.8
39.3
41.7
46.7
39.6
41.9
52.9
Full THW
37.2
40,0
37.6
38.8
41.2
35.5
39.8
42.6
32,9
39.7
42.8
34.8
Decomp T-HWW
37.2
39.8
35.7
38.3
40.9
37.9
38.5
41.5
42.6
39.0
41.9
49.4
IFC
39.0
42.7
36.3
39.6
43.0
39.8
43.7
40.4
43.4
50.2
(c) Number of memory tokens (AP)
(b) Image instance segmenta- tion on COCO val set
(d) Index-wise memory decomposition
T=5
T=10
T=15
T=20
Apooco
APO500
M=1
37.6
39.2
39.4
39.4
T=5
T=10
T=15
T=20
37.9
39.2
39.6
Unified Decomp
38.1
38.9
39.7
39.9
w/o mem
35.0
56.6
M=4
38.0
39.5
39.7
39.9
39,0
39.6
39.8
w/ mem
35.1
56.5
M=8
39.0
39.6
39.8
40.4
M=16
38.1
39.1
39.7
39.9
instances. By enlarging the stride from = 1 to S 3, IFC accomplishes approximately 150% speed improvement with only 0.1% AP drop. The tendency of high speed gain and low accuracy drop persists under various conditions. Therefore, our model can be applied conditions where the enlargement of strides is necessary, i.e,, using devices that are not powerful enough but has to maintain high inference speed.
Various decomposition strategies of encoders In Table L we observed the computational gaps derived from the decomposition of the encoder layers. Extending Table [ how the decomposition strategies affect the accuracy in Table 3
we now investigate the
The models are evaluated with variety of window sizes (T 5, 10, 15. 20) increase of window size T has pros and cons. When matching predictions from different clips, greater T is advantageous due to enlargement of temporal intersections between clips. On the contrary, frames in longer clips are likely to be composed of diverse appearances, which disrupt tracking and segmenting instances within clip. Therefore, the key the performance enhancement is cope with the appearance changes by precisely encoding and correlating space-time inputs.
As shown in Table 3 (a), the full self-attention [II] surpasses the encoder without communications as the length of clips increase. However, the enlargement the window size highly slows down the inference speed, and the improvements are marginal that the tremendous computation and memory usage cannot be compensated. The decomposition of space-time maintains comparable speed even the window is large, but fails to achieve high accuracy.
Our model shows fast inference as the only additional computations of IFC are from utilizing a small number of memory tokens. Furthermore, by effectively encoding the space-time inputs with the communications between frames, IFC can take advantages of enlarging the window size, and surpasses other encoders.
Memory tokens We also study the effects utilizing memory tokens. As mentioned, the motiva- tion of using the memory tokens is to build communications between frames. Different from the video instance segmentation task, the image segmentation task is consisted of single frame. Therefore, the use of the memory tokens does not lead to improvements to the image instance segmentation task as mutual communications cannot be solcly made (see Table 3 (b)). Meanwhile, the utilization of the memory tokens achieves great improvements by effectively passing the information between frames. Results in Table (a, c) demonstrate that the use memory tokens achieves higher accuracy than the encoder without any communications (No comm), which emphasizes the importance of the communications. We evaluate how the size of the memory tokens affect the overall accuracy in Table 3 (c) and set the default size of the tokens M to be 8.
we demonstrated formulation of inputs for Bather-Communicate layer, which groups the outputs of EEnood-RReceiv by memory indices. As aforementioned, the formulation can be considered as a decomposition of memory tokens: insertion to the Bather-Commmnices layer by separate M groups each consisting of T tokens. In Table 3 (d), we investigate the impact of inserting
In Section 3.1

---

Figure 3: Visualizations of results and attention maps of memory tokens.
the unified MT tokens whole. Compared to the unified insertion, the decomposition brings better accuracy as the memories of same indices have more correspondences, which ease the encoders to build attentions in between.
We choose memory index attending foreground instances and visualize the attention map in Fig. As shown in the results of the upper clip, we find that the memory token has more interests to instances that are relatively difficult detect; more attends the heavily occluded car the rear. The clip at the bottom is composed of frames with huge motion blurs and appearance changes. With the communications of memory tokens, IFC successfully tracks and segments the rabbit.
Conclusion
In this paper, we have proposed a novel video instance segmentation network using Inter-frame Communication Transformers (IFC), which alleviates full space-time attention and successfully builds communications between frames. Finally, new state-of-the-ar on the YouTube- VIS dataset. For the future work, we plan to integrate temporal information, which indeed would take a step further to the human video understanding.
network presents rapid inference and sets the
Acknowledgments
This research was grant funded by the Artificial Intelligence Graduate School Program of Yonsei University, under Grant 2020-0-01361, Korea Evaluation Institute of Industrial Technology (KEIT) funded by the Ministry of Trade, Industry and Energy (10073129) and also supported by the Advanced Robotics Laboratory, part of the Future Technology Center LG Electronics.
10

---

References
[1] Yang, L., Y. Fan, N. Xu. Video instance segmentation. In ICCV. 2019.
[2] Cao, J., M. Anwer, H. Cholakkal, et al. Sipmask: Spatial information preservation for fast image and video instance segmentation. In ECCV. 2020.
[3] Yang, S., Y. Fang, X. Wang, et al. Crossover learning for fast online video instance segmentation. In 2021.
[4] Liu, D., Cui, W. Tan, et al. Sg-net: Spatial granularity network for one-stage video instance segmentation. In CVPR. 2021.
[5] He, K., G. Gkioxari, P. Dollar, et al. Mask r-cmn. In ICCV. 2017.
[6] Bolya, D., C. Zhou, F. Xiao, et al. Yolact: Real-time instance segmentation. In ICCV. 2019.
[7] Tian, Z., C. Shen, H. Chen. Conditional convolutions for instance segmentation. In ECCV.
2020.
[8] Chen, H., K. Sun, Z. Tian, al. Blendmask: Top-down meets bottom-up for instance segmenta- tion. In CVPR. 2020.
(9] Luo, W., J. Xing, A. Milan, et al. Multiple object tracking: A literature review. Artificial Intelligence, 2020.
[10] Bertasius, G., I.. Torresani. Classifying, segmenting, and tracking object instances in video with mask propagation. In CVPR. 2020.
[11] Wang. Y., Z. Xu, X. Wang, et al. End-to-end video instance segmentation with transformers. In CVPR. 2020.
[12] Lin, H., R. Wu, S. Liu, al. Video instance segmentation with propose-reduce paradigm. In ICCV, 2021.
[13] Carion, N., F. Massa, G. Synnaeve, et al. End-to-end object detection with transformers. In ECCV. 2020,
[14]
Vaswani, A., N. Shazeer, N. Parmar, et al. Attention is all you need. In NeurIPS. 2017.
[15] Wang, H., Y. Zhu, B. Green, et al. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In ECCV. 2020.
[16] Bertasius, G., H. Wang, L. Torresani. Is space-time attention all you need for video understand- ing? In ICML. 2021.
[17] Arnab, A., M. Dehghani, G. Heigold, et al. Vivit: A video vision transformer. arXiv preprint 1rXiv:2103.156% 2021.
[18] Wang, X., R. Girshick, A. Gupta, et al. Non-local neural networks. In CVPR. 2018.
[19] Kuhn, H. W. The hungarian method for the assignment problem. In Naval research logistics quarterly. 1955.
[20] Wang, H., Y. Zhu, H. Adam, et al. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR. 2021.
[21] Dosovitskiy, A., L. Beyer, A. Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. 2021.
[22] Ranftl, R., A. Bochkovskiy, V. Koltun. Vision transformers for dense prediction. In ICCV. 2021.
[23] Devlin, J., M.-W. Chang, K. Lee, al. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL. 2019.
[24] Touvron, H., M. Cord, M. Douze, al. Training data-efficient image transformers & distillation through attention. In ICML. 2021.
[25] Kirillov, A., K. He, R. Girshick, et al. Panoptic segmentation. In CVPR. 2019.
[26] Tran, D., H. Wang, L. Torresani, et al. A closer look at spatiotemporal convolutions for action recognition. CVPR. 2018.
[27] Lin, T.-Y., P. Dollar, R. Girshick, et al. Feature pyramid networks for object detection. In CVPR. 2017.

---

[28] Stewart, R., M. Andriluka, A. Y. Ng. End-to-end people detection in crowded scenes. In CVPR. 2016.
[29] Milletari, F., N. Navab, S.-A. Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. 3DV. 2016.
[30] Lin, T.-Y., Goyal, R. Girshick, et al. Focal loss for dense object detection. In /CCV. 2017.
[3]] He, K., X. Zhang, Ren, al. Deep residual learning for image recognition. In CVPR. 2016.
[32] Athar, A., S. Mahadevan, A. Ošep, al. Stem-seg: Spatio-temporal embeddings for instance
segmentation in videos. In ECCV, 2020.
[33] Wu, Y., A. Kirillov, Massa, et Detectron2. stttss//githuh...c/cafcccooserse
detectron2 2019.
[34]
Loshchilov, I,, F. Hutter. Decoupled weight decay regularization. In ICLR. 2019.
[35] Lin, T.-Y. M. Maire, S. Belongie, et al. Microsoft coco: Common objects in context. In ECCV.
2014.
[36] Dai, J., H. Qi, Y. Xiong, et al. Deformable convolutional networks. In ICCV, 2017.
[37] Cai, Z., N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR.
2018.
12


Reviews:
[{"review_id": "ZMErSbcgMdm", "reviewer": "Reviewer_pxiA", "summary": "The paper extends transformer-based video instance segmentation framework VisTR in three aspects. First, the intra- (i.e., space) and inter-frame (i.e., time) relation computations in the transformer encoder are decomposed. Specifically, the extra memory tokens are introduced for efficient intra- and inter-frame communication. Second, the fixed object queries are used in the transformer decoder and thus the model can take a varying number of input frames. Moreover, the contextualized object queries are forwarded to the segmentation head so that the adaptive convolutional weights can be generated for effective tracking and segmentation. Third, the authors presented to use 3D IoU to link the already tracked tracklets and the currently generated tracklets to produce final video-level tracklets. Combining all together, the final framework is evaluated on the benchmarks and achieves new state-of-the-art.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality]\n\nThe main contribution of this paper lies in presenting a solid transformer-based video instance segmentation framework that pushes the state-of-the-art result significantly and is fast. The idea of introducing memory tokens for effective intra- and inter-frame relation computation is novel and interesting. ​\nHowever, there are also several ideas that are not technically new. \n\n- The similar memory-token-based relation computation is also described in the following work. However, I will not count the following paper as a concurrent work since it is published in arXiv near the submission. Please compare the proposal with the following work.\n\n[1] End-to-End Video Object Detection with Spatial-Temporal Transformers, arXiv 2021 \n\n- The fixed object queries which are shared across the frame are also tried in the original VisTR paper (please refer to the original paper Table-2).\n- The idea of using adaptive convolutional weights for instance segmentation is a promising technique that is verified in various previous literature (e.g., CondInst, SOLO, Max-deeplab).\n- Using an FPN-like spatial decoder obviously gives an extra performance gain in pixel-level prediction tasks.\n- The mask-based sequence matching is a direct extension of VisTR's box-based matching. While clearly effective, it consumes more memory than the box-based counterpart.\n- The final loss functions (i.e., dice loss + focal loss) are from the VisTR.\n- The 3D IoU score for clip-linking is a direct video-level extension of MaskProp's averaged 2D IoU scores.\n\n[Quality]\n\n- The authors well-combined the previous techniques into a single framework.\n- Extensive ablation studies show the efficacy of the proposals.\n- The video results are quite impressive.\n\n[Clarity]\n\n- As the key technical contribution is on the memory token, please discuss and compare it with the recent space-time factorization methods in more detail. For example, the following paper also introduces space-time factorization methods for video embeddings in the transformer.\n\n[1] ViViT: A Video Vision Transformer, arXiv 2021\n\n[2] Is Space-Time Attention All You Need for Video Understanding?, ICML 2021\n\n- The method section is well-written and easy to follow. Table-1 shows a clear advantage of the proposal over the baselines.\n\n- Typos\n[1] Line 291 Our model is shows -> Our model shows\n\n[Significance]\n- While there are several components that are not new, each component is combined in a novel way and the final model achieves non-trivial gain over the baselines both in accuracy and speed.\n- As transformer-based video models are becoming more and more important these days, the work provided has several strong points (memory token-based transformer encoder design and efficient tracklet matching) that need to be shared with the community.\n\n[Question & Suggestion]\n- The position encoding remains to be the same as VisTR. Did the authors have tried to drop it or try different types of embeddings? For example, the position prior can be embedded in the self-attention operation such as Swin Transformer.\n- Why do more memory tokens fail to provide performance gain (e.g., 8 vs. 16 in Table 3-(c))? More discussions need to be provided in the manuscript.\n- Please detail the visualization process in Figure 2.\n- Please provide a full model FLOPS comparison by adding an extra column in Table 3.\n- The authors can try to make the whole framework purely based on the Transformer (e.g., replacing the ResNet backbone with SETR).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "Yes", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "MU1xvsRHUz", "reviewer": "Reviewer_tpaP", "summary": "This paper proposes Inter-frame Communication Transformers (ICT) to solve the Video Instance Segmentation (VIS) task in one framework. In particular,  they propose to utiize concise memory tokens as a means of converying information as well as summarizing each frame content.  The method achieves the state-of-the-art performance  on YouTube-VIS 2019. ", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength:\n\n1, The overall writing is good. \n\n2, The experiment results on YouTube-VIS are good. \n\n3, The motivation is clear: reducing computation cost and of transformer and reducent information in video by mean of passing tokens. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main issue of this paper is the novelty.  In particular, they just apply the core idea of MaxDeeplab [1] into video domain. Thus there are no fundamental new messages in case of feature or object query fusion. \nHere are the details. \n\n1.  Architecute and Loss : The end-to-end solution for VIS is firstly proposed by VisTR[3].  It has been proven effectively for predicting multiple queries across time. Moreover, the decoder head and loss are the same with original DETR[2] (Panoptic Segmentation) paper.  Thus in case of architecture and loss, this is not novel. \n\n2. Method: The overall architecture is based on VISTR.  The core contribution of this paper is Inter-Frame Communication Encoder which contains Encode-Receive and Gather-Communicate. However, I can not tell the difference between the Query and Pixel Interaction in MaxDeeplab. They are same component by encoding the appreance information into Object Query via attention layers (while this paper call it token, also misleading and hard to under why called token).  Gather-Communicate process is most used in preovious video segmentation method in previous work by link encoded query across time [4].  \n\nIn summary, there is no new concept or new message contribution. The overall contributions are  incremental and there are many similar components in related fields.  At least, it dose not reach the \n \n\nThere are other issues in case of experiment comparison and writing. \n. \n1. It is unfair to compare privous work including SG-Net and CrossVIS since most previous work only uses previous frame information. This paper and VisTR use the whole clip as inputs. Moreover, the FPS is also misleading since VisTR-framework process one clip each time rather than one image. \n\n2.  What is pretrained model result on COCO since it has hudge effect on YouTube-VIS ?\n\n3.  What is difference between token and query in DETR? \n\n\nReference: \n\n[1] MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers CVPR-2021\n\n[2] End-to-End Object Detection with Transformers ECCV-2020\n\n[3] VISTR:  End-to-End Video Instance Segmentation with Transformers CVPR-2021\n\n[4] Temporally Distributed Networks for Fast Video Semantic Segmentation CVPR-2020\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {}}, {"review_id": "C7H43_ZVsN8", "reviewer": "Reviewer_qqgE", "summary": "- The paper proposes a video instance segmentation method based on Transformers. The authors propose to aggregate temporal features through memory tokens. Specifically,  each frame is summarized by several memory tokens.  A Transformer decoder is used to perform the inter-frame feature learning using these tokens.\n- The paper shows strong performance on the Youtube-VIS benchmark, outperforming state-of-the-art methods in both speed and accuracy.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strength\n- Using memory tokens to perform inter-frame representation learning is highly motivated.\n- The proposed method is highly efficient and effective, e.g., 46.5 fps/41.0 AP for near-online inference.\n\n# Weakness\n- Limited contributions. For each component in the proposed framework, only the memory token is the new technique. Others are proposed in previous papers, e.g., DETR and VisTR.\n- According to the ablations, the biggest improvement (2.2 AP) is from the bipartite matching strategy, i.e., using mask-based matching instead of box-based one. It makes this work less impressive.\n- The conclusion of 'we find that the memory token has more interests to instances that are relatively difficult to detect;' is not convincing. Are the visualized figures from only one of the memory tokens?  More analysis is needed to support this claim.\n- The conditional convolutional weights, i.e., single-layer 1x1 conv, look more similar to SOLOv2 [A] instead of CondInst [7]. Related discussion is missed. \n- Many typos and grammar errors, e.g.,  L137, L188, L291\n\n[A] Wang et al. Solov2: Dynamic, faster and stronger.  In NeurIPS, 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are not discussed. The potential negative societal impact is provided.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "AZgws3wYVEA", "reviewer": "Reviewer_1red", "summary": "This paper proposes inter-frame communication transformers(IFC) for video instance segmentation. Compared to a full spatial-temporal transformer with complexity THW x THW, the proposed IFC module reduces it to THW + T(HW)^2 while preserving the temporal and spatial attentions. The proposed video instance segmentation network outputs a clip-level prediction, which is then stitched along the time stamps via Hungarian matching between the tracklet tubes. The experiments on YouTube-VIS 2019 dataset shows the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The proposed memory tokens are memory-efficient and effective at learning temporal relationships between frames. \n\n* The paper provides extensive experiments and ablation studies. \n\n* The paper is overall well-written and easy to follow.\n\n* It seems that the proposed method provides a good efficiency-accuracy trade-off. How does it compare with MaskProp [10]?\n\n* If I understand it correctly, the span of instance tracking is controlled by T (which defaults to 5 time stamps). How does the proposed model handle long-term tracking of the instances (e.g. re-appeared objects after 5+ or T+ frames)? How much is the performance drop caused by missing such long-term tracking? \n\n* Why is the reported score in Table 2 (41.0 AP) and in Table 3 (39.0) different? Is it that one is the best score and the other is average score out of multiple runs (5 runs)? I could not find the explanation for this gap, and this should be clarified.\n\n* The proposed method looks similar to 'decomposed T + HW' attention. How is the 'T + HW' version implemented? and why is it worse than the proposed method?\n\n* How is the 'full THW' implemented? It's opposed to an expectation that the 'full THW' model should be the upper bound in terms of accuracy. What is missing in the 'full THW' model, and what makes the proposed method perform better?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.\nFor long-term tracking of the instances, please see the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: None of the reviewers recommended accepting this paper.
After reading the author response and other reviews one of the reviewers also reduced their score.
One of the common critiques of the work was around the degree of novelty provided by the work.
One of the initially more positive reviewers did recognize that the memory-token based inter-frame attention is technically new and felt the experiments here were quite extensive. But this reviewer also felt that the main performance improvement in this method were coming from the other elements of the method. 

The AC recommends that this paper be rejected.

Decision: Accept (Poster)