Associating Objects with Transformers for Video Object Segmentation
Authors: Zongxin Yang
Yunchao Wei
Yi Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Video Object Segmentation
Metric Learning
Instance Segmentation

Abstract: This paper investigates how to realize better and more efficient embedding learning to tackle the semi-supervised video object segmentation under challenging multi-object scenarios. The state-of-the-art methods learn to decode features with a single positive object and thus have to match and segment each target separately under multi-object scenarios, consuming multiple times computing resources. To solve the problem, we propose an Associating Objects with Transformers (AOT) approach to match and decode multiple objects uniformly. In detail, AOT employs an identification mechanism to associate multiple targets into the same high-dimensional embedding space. Thus, we can simultaneously process multiple objects' matching and segmentation decoding as efficiently as processing a single object. For sufficiently modeling multi-object association, a Long Short-Term Transformer is designed for constructing hierarchical matching and propagation. We conduct extensive experiments on both multi-object and single-object benchmarks to examine AOT variant networks with different complexities. Particularly, our R50-AOT-L outperforms all the state-of-the-art competitors on three popular benchmarks, i.e., YouTube-VOS (84.1% J&F), DAVIS 2017 (84.9%), and DAVIS 2016 (91.1%), while keeping more than 3X faster multi-object run-time. Meanwhile, our AOT-T can maintain real-time multi-object speed on the above benchmarks. Based on AOT, we ranked 1st in the 3rd Large-scale VOS Challenge.

Associating Objects with Transformers for Video Object Segmentation
Zongxin Yang!? Yunchao Wei Yi Yang
CCAL, College of Computer Science and Technology. Zhejiang University Baidu Research
Institute of Information Science, Beijing Jiaotong University
4 Beijing Key Laboratory of Advanced Information Science and Network
{zongxinyang1996, wychao1987, yang) com
Abstract
This paper investigates how realize better and more efficient embedding learn- ing to tackle the supervise video object segmentation under challenging multi-object scenarios. The state-of-the-ard methods learn decode features with a single positive object and thus have to match and segment each target separately under multi-object scenarios, consuming multiple times computing resources. To solve the problem, we propose an Associating Objects with Transformers (AOT) approach match and decode multiple objects uniformly. In detail, AOT em- ploys an identification mechanism to associate multiple targets into the same high-dimensional embedding space. Thus, we can simultaneously process multiple objects' matching and segmentation decoding as efficiently as processing single object. For sufficiently modeling multi-object association, a Long Short-Tens Transformer is designed for constructing hierarchical matching and propagation. We conduct extensive experiments on both multi-object and single-object bench- marks to examine AOT variant networks with different complexities. Particularly, our R50-AOT-L. outperforms all the state-of-the-art competitors on three popu- lar benchmarks, i.e., YouTube-VOr (84.1% J J&F), DAVIS 2017 (84.9%), and DAVIS 2016 (91. 19.1%) while keeping more than 3x faster multi-object run-time Meanwhile, our AOT-T can maintain real-time multi-object speed on the above benchmarks. Based on AOT, we ranked the 3rd Large-scale VOS Challenge.
Introduction
Video Object Segmentation (VOS) a fundamental task in video understanding with many potential applications, including augmented reality [25] and self-driving cars [52]. The goal of semi-supervised VOS, the main task in this paper, is track and segment object(s) across an entire video sequence based on the object mask(s) given at the first frame.
Thanks to the recent advance of deep neural networks, many deep learning based VOS algorithms have been proposed recently and achieved promising performance. STM [26] and its following works [34, 23] leverage a memory network store and read the target features of predicted past frames and apply non-local attention mechanism to match the target in the current frame. FEELVOS [41] and CFBI [50, 51] utilize global and local matching mechanisms match target pixels or patches from both the first and the previous frames to the current frame.
Even though the above methods have achieved significant progress, the above methods learn decode scene features that contain a single positive object. Thus under multi-object scenario, they have to match each object independently and ensemble all the single-object predictions into a multi-object segmentation, as shown in Fig. la. Such a post-ensemble manner eases network architectures' design
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Prediction
Reference
Separation
tompu prpcnoo compleuty
+
Past onsomble
Mull-oljest
Singi-oolect
Current
Current
Prediction
AOT (ours)
vOs
AOT
Network
Object number
(a) Post-ensemble
(b) Associating objects (ours)
(c)
Comparison
Figure I: VOS methods (e.g., [50, 34)) process multi-object scenarios in a post-ensemble manner (a).
In contrast, our AOT associates all the objects uniformly (b), leading to better efficiency (c).
since the networks are not required to adapt the parameters or structures for different object numbers. However, modeling multiple objects independently, instead of uniformly, is inefficient in exploring multi-object contextual information to learn a more robust feature representation for VOS. addition, processing multiple objects separately yet in parallel requires multiple times the amount of GPU memory and computation for processing a single object. This problem restricts the training and application of VOS under multi-object scenarios, especially when computing resources are limited.
To solve the problem, Fig. 1b demonstrates feasible approach associate and decode multiple objects uniformly in an end-to-end framework. Hence, we propose Associating Objects with Transformers (AOT) approach match and decode multiple targets uniformly. First, an identification mechanism is proposed to assign each target unique identity and embed multiple targets into same feature space. Hence, the network can learn association or correlation among all the targets. Moreover, the multi-object segmentation can be directly decoded by utilizing assigned identity information. Second, a Long Short-Term Transformer (LSTT) designed for constructing hierarchical object matching and propagation. Each LSTT block utilizes a long- term attention for matching with the first frame's s embedding and a short-term attention for matching with several nearby frames' embeddings. Compared to the methods [26, 34] utilizing only attention layer, we found hierarchical attention structures are more effective in associating multiple objects.
We conduct extensive experiments on two popular multi-object benchmarks for VOS, i.e., YouTube- VOS [48] and DAVIS 2017 [31], validate the effectiveness and efficiency of the proposed AOT, Even using the light-weight Mobilenet-V? V2 [33] as the backbone encoder, the AOT variant networks achieve superior performance on the validation 2018 & 2019 splits of the large-scale YouTube-VOS (ours, F 82.6~ 84.5% & 82.2~ 84.5%) while keeping more than 2 faster multi-object run-time (27.1~ 9.3FPS) compared to the state-of-the-arr competitors (e.g., CFBI [50]. 81.4% & 81.0%, 3.4FPS). We also achieve
state-of-the-an performance both the DAVIS-2017 validation (85.4%) and testing (81.2%) splits. Moreover, AOT effective under single-object scenarios as well and outperforms previous methods DAVIS 2016 [30] (92.0%). popular single-object benchmark. Besides, our smallest variant, AOT-T, can maintain real-time multi-object speed on all above benchmarks (51.4FPS 480p videos). Particularly, AOT ranked 1st in the Track (Video Object Segmentation) of the 3rd Large-scale Video Object Segmentation Challenge.
Overall, our contributions are summarized as follows:
We propose an identification mechanism to associate and decode multiple targets uniformly for VOS. For the first time, multi-object training and inference can be efficient as single-object ones, as demonstrated in Fig. 1c.
Based on the identification mechanism, we design new efficient VOS framework, i.e., Long Short- Term Transformer (LSTT), for constructing hierarchical multi-object matching and propagation. LSTT achieves superior performance on VOS benchmarks [48, 31, 30] while maintaining better efficiency than previous state-of-the- methods. the best of our knowledge, L.STT the first hicrarchical framework for object matching and propagation by applying transformers [39] to VOS.
2
Related Work
Semi-supervised Video Object Segmentation. Given one or more annotated frames (the first frame in general), semi-supervised VOS methods propagate the manual labeling to the entire video

---

sequence. Traditional methods often solve an optimization problem with an energy defined over graph structure [4, 40, 21. In recent years, VOS methods have been mainly developed based on deep neural networks (DNN), leading to better results.
Early DNN methods rely on fine-tuning the networks test time to make segmentation networks focus on a specific object, Among them, OSVOS [7] and MoNet [47] fine-tune pre-trained networks on the first-frame ground-truth at test time. OnAVOS [42] extends the first-frame fine-tuning by introducing an online adaptation mechanism. Following these approaches, MaskTrack [29] and
PReM [24] utilize optical flow to help propagate the segmentation mask from one frame to the next. Despite achieving promising results, the test-time fine-tuning restricts the network efficiency.
Recent works aim to achieve better run-time and avoid using online fine-tuning, OSMN [49] employs one convolutional network to extract object embedding and another one to guide segmen- tation predictions. PMI. [9] learns pixel-wise embedding with nearest neighbor classifier, and VideoMatch [18] uses a soft matching layer that maps the pixels of the current frame to the first frame in learned embedding space. Following PML and VideoMatch, FEELVOS [41] and CFBI [50, 51] extend the pixel-Ievel matching mechanism by additionally matching between the current frame and the previous frame. RGMP [46] also gathers guidance information from both the first frame and the previous frame but uses siamese encoder with two shared streams. STM [26] and its following works (e.g., EGMN [23] and KMN [34]) leverage a memory network to embed past-frame predictions into memory and apply a non-local attention mechanism on the memory to decode the segmentation different way, i.e., transformer features. The features are target-agnostic, instead of target-aware like our LSTT, since the mask information in past frames is not propagated and aggregated in the blocks. Instead of using matching mechanisms, LWL [6] proposes to use an online few-shot learner to learn to decode object segmentation.
of the current frame. SST [13] utilizes attention mechanisms
blocks [39] are used to extract pixel-level affinity maps and
The above methods learn to decode features with a single positive object and thus have to match and segment each target separately under multi-object scenarios, consuming multiple times computing resources of single-object cases. The problem restricts the application and development of the VOS with multiple targets. Hence, we propose our AOT to associate and decode multiple targets uniformly and simultancously, as efficiently as processing a single object.
Visual Transformers. Transformers [39] was proposed build hierarchical attention based networks for machine translation. Similar to Non-local Neural Networks [43], transformer blocks compute correlation with all the input elements and aggregate their information by using attention mecha- nisms [5]. Compared to RNNs, transformer networks model global correlation or attention in parallel, leading to better memory efficiency, and thus have been widely used in natural language processing (NLP) tasks [11, 32, 37J. Recently, transformer blocks were introduced to many computer vision tasks, such as image classification [12, 38, 22]. object detection [8] //egmennttiionn [44], and image generation [27], and have shown promising performance compared to CNN-based networks.
Many VOS methods [19, 26, 23, 34] have utilized attention mechanisms match the object features and propagate the segmentation mask from past frames to the current frames. Nevertheless, these methods consider only one positive target in the attention processes, and how build hierarchical attention- -aaed propagation has been rarely studied. In this paper, we carefully design a long short- term transformer block, which can effectively construct multi-object matching and propagation within hierarchical structures for VOS.
3 Revisit Previous Solutions for Video Object Segmentation
In VOS, many common video scenarios have multiple targets or objects required for tracking and segmenting. Benefit from deep networks, current state-of-ther thetth VOS methods [26,, 50] have achieved promising performance. Nevertheless, these methods focus on matching and decoding a single object. Under multi-object scenario, they thus have to match each object independently and ensemble all the single-object predictions into a multi-object prediction, as demonstrated in Fig. 1a. Let F.V denotes VOS network for predicting single-object segmentation, and A is an ensemble function such as so tmax or the soft aggregation [26], the formula of such a post-ensemble manner for processing N objects like.
='= A(FN ) EN F (I, 1m Y
(I)
3

---

Frame
Frame t-1
Frame
Identity Bank vector
Feed Forward
IN
Encoder
Encoder
torm Attention
Short Attertion
Assign identties (N-M)
LN
Decoder
Decoder
#xd yol N-object Mask
ID
Identification Embedding
Selt-Attention
IN
Given
Prediction
HxWWxC
(a) Overview
(b) Identity assignment
(c) 1-th LSTT block
Figure 2: (a) The overview of our Associating Objects with Transformers (AOT). The multi-object masks are embedded by using our Identification mechanism. Moreover, L-layer Long Short- Term Transformer is responsible for matching multiple objects uniformly and hierarchically, (b) An illustration of the IDentity assignment
(ID) designed for transferring a N-object mask into an identification embedding. (c) The structure of an LSTT block. LN: layer normalization [3].
where I' and 1m denote the image of the current frame and memory frames respectively, and {Yrm,..., ...,.NN Ym are the memory masks (containing the given reference mask and past predicted
masks) of all the N objects. This manner extends networks designed for single-object VOS into multi-ohject applications, so there is no need to adapt the network for different object numbers.
Although the above post-ensemble manner is prevalent and straightforward in the VOS field, process- ing multiple objects separately yet parallel requires multiple times the amount GPU memory and computation for matching single object and decoding the segmentation. This problem restricts the training and application of VOS under multi-object scenarios when computing resources are limited. To make the multi-object training and inference as efficient single-object ones, expected solution should be capable associating and decoding multiple objects uniformly instead of individually. To achieve such objective, propose identification mechanism to embed the masks of any number (required to be smaller than a pre-defined large number) of targets into the same high-dimensional space. Based the identification mechanism, novel and efficient framework, i.e., Associating Objects with Transformers (AOT), is designed for propagating all the object embeddings uniformly and hierarchically, from memory frames the current frame.
As shown in Fig. Ib, our AOT associates and segments multiple objects within an end-to-end framework. For the first time, processing multiple objects can be efficient processing single object (Fig. 1c). Compared to previous methods, our training under multi oojecct scenarios also more efficient since AOT can associate multiple object regions and learn contrastive feature embeddings among them uniformly.
Associating Objects with Transformers
In this section, we introduce our identification mechanism proposed for efficient multi-object VOS. Then, we design a new VOS framework, i.e., long short-term transformer, based on the identification mechanism for constructing hicrarchical multi-object matching and propagation.
4.1 Identification Mechanism for Multi-object Association
Many recent VOS methods [26, 23, 34] utilized attention mechanisms and achieved promising results. To formulate, we define Q E IR jgWwxC xC' K E IR TWW xC and V E HWW xC as the query embedding of the current frame, the key embedding memory frames respectively, where T, H. W, C denote the temporal, height, width, and channel dimensions. The formula of a common attention-baan based matching and propagation is,
the memory frames, and the value embedding of the
1htr VC V,,
Att K,V) = Corr (Q, K\V = softmar
(2)

---

where matching map is calculated by the correlation function Corr, and then the value embedding. V. will be propagated into each location of the current frame.
In the common single-object propagation [26], the binary mask information in memory frames is embedded into V with an additional memory encoder network and thus can also be propagated the current frame by using Eq. 2. A convolutional decoder network following the propagated feature will decode the aggregated feature and predict the single-object probability logit of the current frame.
The main problem of propagating and decoding multi-object mask information in an end-to-end network is how to adapt the network to different target numbers. To overcome this problem, propose an identification mechanism consisting of identification embedding and decoding based on attention mechanisms.
First, an Identification Embedding mechanism is proposed to embed the masks of multiple different targets into the same feature space for propagation. As seen in Fig. 2b, we initialize an identity bank, D E MxC, where M identification vectors with C' dimensions are stored. For embedding multiple different target masks, each target will be randomly assigned a different identification vector. Assuming N (N < .) targets are in the video scenery, the formula of embedding the targets' one-hot mask, Y E {0, 1 x.N into identification embedding, E E I RQTHWxC by randomly assigning identification vector from the bank D is,
= ID(Y,, D) = YPD,
where Pe {0, 13NMMM is a random permutation matrix, satisfying that pdr P is equal to a M x M unit matrix. for randomly selecting N identification embeddings. After the ID assignment, different target has different identification embedding, and thus we can propagate all the target identification information from memory frames to the current frame by attaching the identification embedding E with the attention value V, i.e.
V' = AAtIIDIQ, K, KVV, = K, ID(Y, = Att(Q, K, V + E),
(4)
where V HC aggregates all the multiple targets' embeddings from the propagation.
For Identification Decoding, i.e., predicting all the targets' probabilities from the aggregated feature V', we firstly predict the probability logit for every identity in the bank D by employing convolutional decoding network FP, and then select the assigned ones and calculate the probabilities, i.e.,
(5) X is all the M identities' probability logits, P is the same the selecting matrix
Y' softmar(PF" = softmar (PLP)
where D HW used in the identity assignment (Eq. 3), and Y' E [0, Mj XN is the probability prediction of all the N targets.
For training, common multi-class segmentation losses, such as cross-entropy loss, can be used to optimize the multi-object Y' regarding the ground-truth labels. The identity bank D is trainable and randomly initialized at the training beginning. To ensure that all the identification vectors have the same opportunity compete with each other, we randomly reinitialize the identification selecting matrix P in each video sample and each optimization iteration.
4.2 Long Short-Term Transformer for Hierarchical Matching and Propagation
Previous methods [26, 34] always utilize only one layer of attention (Eq. 2) to aggregate single- object information. In our identification-basee multi-object pipeline, we found that single attention layer cannot fully model multi-object association, which naturally should be more complicated than single-object processes. Thus, we consider constructing hicrarchical matching and propagation by using series of attention layers. Recently, transformer blocks [39] have been demonstrated to stable and promising in constructing hierarchical attention structures in visual tasks [8, 12]. Based on transformer blocks, we carefully design a Long Short-Term Transformer (LSTT) block for multi-object VOS.
Following the common transformer blocks [39, 11], LSTT firstly employs self self-attention layer, which is responsible for learning the association or correlation among the targets within the current frame. Then, LSTT additionally introduces a long-term attention, for aggregating targets' information from long-term memory frames and a short-term attention, for learning temporal smoothness from nearby short-term frames. The final module is based on common 2-layer feed-forward MLP with
5

---

GELU [17] non-linearity in between. Fig. 2c shows the structure of an LSTT block. Notably, all these attention modules are implemented in the form of the multi-head attention [39], i.e. multiple attention modules followed by concatenation and a linear projection. Nevertheless, we only introduce their single-head formulas below for the sake of simplicity.
Long-Term Attention is responsible for aggregating targets' information from past memory frames, which contains the reference frame and stored predicted frames, to the current frame. Since the time intervals between the current frame and past frames are variable and can be long-term, the temporal smoothness difficult guarantee. Thus, the long-term attention employs non-local attention like Eq. 2. Let X E R "I x denotes the input feature embedding at time t and in block 4, where E [1. ..... L} is the block index of LSTT, the formula of the long-term attention is,
(6) where X Concal(X]") ..... X!"*)) and Y ConcallY ,..., Ymr) are the input feature embeddings and target masks of memory frames with indices m = {m1 ...,, mT). J. Besides, W/K e RCCCx and W. E RCCC. are trainable parameters of the space projections for matching and propagation, respectively. Instead using different projections for X and X.m, we found the training of LSTT is more stable with a siamese-like matching, i.e., matching between the features within the same embedding space (I-th features with the same projection of W
AXt XI" Y") Ax .!" . Y"ID),
Short-Term Attention is employed for aggregating information in for each current- frame location. Intuitively, the image changes across several contiguous video frames are always smooth and continuous. Thus, the target matching and propagation in contiguous frames can be restricted in a small spatial-temporal neighborhood, leading to better efficiency than non-local processes. Considering n neighbouring frames with indices n {t - 1..,, t n} are in the spatial- temporal neighbourhood, the features and masks of these frames are X = Concat(X} X and y = Concat Y "t""), and then the formula of the short-term attention at each spatial location p is,
neighbourhood
AHST(X;, XP, p1)) = AXLL (( Y.N)))
where Xi, E RIxC is the feature of X at location p, NV() is a x À spatial neighbourhood centered at location p. and thus ([))(p) and ())((p) are the features and masks of the spatial-temporal neighbourhood, respectively, with shape of nX² x C or nd x N.
When extracting features of the first frame 1, there is memory frames or previous frames, and hence we use X to replace X and XP- In other words, the long-term attention and the short-term attention changed into self-attentions without adjusting the network structures and parameters.
5 Implementation Details
Network Details: For sufficiently validating the effectiveness of our identification mechanism and LSTT, we mainly use light-weight backbone encoder, MobileNet-V2 [33], and decoder, FPN [20] with Group Normalization [45]. The spatial neighborhood size À is set to 15, and the number of identification vectors, M, is set to 10, which is consistent with the maximum object number in the benchmarks [48, 31]. AOT performs well with PaddlePaddle [1] and PyTorch [28]. More details can be found in the supplementary material.
Architecture Variants: We build several AOT variant networks with different LSTT layer number L long-term memory size m. The parrampparaetters of these variants are: (1) AOT-Tiny: I, 1, m = (1]: (2) AOT-Small:
= 2, m = [1]: (3) AOT-Base: L, = 3, m = {1]: (4) AOT-Large: L = 3. m {1, 1 5, 1 + 26, + 36, ...]. In the experiments, we also equip AOT-L with ResNet50 (R50) [16] or Swin-B [22].
AOT-S is small model with only 2 layers of LSTT block. Compared to AOT-S, AOT-T utilizes only 1 layer of LSTT, and AOT-B/L uses 3 layers. In AOT-T/S/B, only the first frame considered into long-term memory, which is similar to [41, 50], leading to smooth efficiency. In AOT-L, the predicted frames are stored into long-term memory per frames, following the memory reading strategy [26]. We set 8 2/5 for training/testing.
Training Details: Following [46, 26, 23, 34), the training stage divided into two phases: (1) pre-training on sythetic video sequence generated from static image datasets [14, 21, 10, 36, 15] by

---

Table 1: The quantitative evaluation on multi-object benchmarks, YouTube VOS [48] and DAVIS 2017 [31]. Y: using YouTube- VOS for training. using 600p instead of 480p videos in inference. 1 timing extrapolated from single-object speed assuming linear scaling in the number of objects.
(a) YouTube-Voo VOS
(b) DAVIS 2017
Seen
Unseen
Methods
J&F
J F
FPS
Methods
J&F
J
F
J
F FPS
Validation 2017 Split
Validation 2018 Split
CFBI [50] (Y)
81.9
79.3
84.5
5.9
SST [13] (Y)
82.5
79.9
85.1
STM (CCV19) [26]
79.4
79.7
84.2
72.8
80.9
KMN [34]
76.0
74.2
77.8
4.2
KMN FCCV20) [34]
81.4
81.4
85.6
75.3
83.3
CFB (FCCV20) [50]
81.4
81.1
85.8
75.3
83.4
3.4
KMN [34] (Y)
82.8
80.0
85.6
4.24
LWL [6]
81.5
80.4
84.9
76.4
84.4
CFBI+ [51] (Y)
82.9
80.1
85.7
5.6
SST [13]
81.7
81.2
76.0
AOT-T (Y)
79.9
77.4
82.3
51.4
CFBI+TPAMIII [51]
82.8
81.8
86.6
77.1
85.6
4.0
AOT-S
79.2
76.4
82.0
40.0
AOT-
80.2
80.1
84.5
74.0
82.2
41.0
AOT-S (Y)
81.3
78.7
83.9
40.0
AOT-S
82.6
82.0
86.7
76.6
85.0
27.1
AOT-B (Y)
82.5
79.7
85.2
29.6
AOT-B
83.5
82.6
87.5
77.7
86.0
20.5
AOT-L (Y)
83.8
81.1
86.4
18.7
AOT-L
83.8
82.9
87.9
77.7
86.5
16.0
R50-AOT-L (Y)
84.9
82.3
87.5
18.0
R50-AOT-L
84.1
83.7
88.5
78.1
86.1
14.9
SwinB (Y)
85.4
82.4
88.4
12.1
SwinB-AOTH
84.5
84.3
89.3
77.9
86.4
9.3
Testing 2017 Splir
Validation 2019
Split
CFBI (Y)
75.0
71.4
78.7
5.3
[50]
81.0
80.6
85.1
75.2
83.0
3.4
CFBI' [50] (Y)
76.6
73.0
80.1
2.9
SST [13]
81.8
80.9
76.6
KMN [34] (Y)
77.2
74.1
80.3
4FBI [51]
82.6
81.7
86.2
77.1
85.2
4.0
CFBI+ [S11
78.0
74.4
81.6
3.4
AOT-T
79.7
79.6
83.8
73.7
81.8
41.0
AOT-T
72.0
68.3
75.7
51.4
AOT-S
82.2
81.3
85.9
76.6
84.9
27.1
AOT-S (Y)
73.9
70.3
77.5
40.0
AOT-B
83.3
82.4
87.1
77.8
86.0
20.5
AOT-B (Y)
75.5
71.6
79.3
29.6
AOT-L
83.7
82.8
87.5
78.0
86.7
16.0
AOT-L
78.3
74.3
82.3
18.7
R50-AOT-L
84.1
83.5
88.1
78.4
86.3
14.9
R50-AOT-L (Y)
79.6
75.9
83.3
18.0
SwinB-AOT-L
84.5
84.0
88.8
78.4
86.7
9.3
SwinB- AOT-L (Y)
81.2
77.3
85.1
12.1
randomly applying multiple image augmentations [46]. (2) main training on the VOS benchmarks [48,
31] by randomly applying video augmentations [50]. More details in the supplementary material.
6 Experimental Results
We evaluate AOT on popular multi-object benchmarks, YouTubee-VOS [48] and DAVIS 2017 [31], and single-object benchmark, DAVIS 2016 [30]. For YouTube-VVOS experiments, train models on the You-ube-VOL 2019 training split. For DAVIS, we train on the DAVIS-201 training split. When evaluating Tou we use the default 6FPS videos, and all the videos restricted to be smaller than 1.3 x 480p resolution. As to DAVIS, the default 480p 24FPS videos are used.
The evaluation metric is the J score, caleulated the average Intersect over Union (IoU) score between the prediction and the ground truth mask, and the F score, calculated as an average boundary similarity measure between the boundary of the prediction and the ground truth, and their mean value, denoted as I J.F. We evaluate all the results on official evaluation servers or with official tools.
6.1 Compare with the State-of-the-art Methods
YouTube- VOS [48] is the latest large-scale benchmark for multi-object video segmentation and is about 37 times larger than DAVIS 2017 (120 videos). Specifically, YouTube- VOS contains 3471 videos in the training split with 65 categories and 474/507 videos in the validation 2018/2019 split with additional 26 unseen categories. The unseen categories do not exist the training split to evaluate algorithms' generalization ability.
As shown Table Ia, AOT variants achieve superior performance on YouTube- VOS compared to the previous state-of-the the- methods. With our identification mechanism, AOT-S (82.6% J J&FF
7

---

tire
DAVIS
YouTube
CFU1
Falure (ours)
Figure 3: Qualitative results. (top) Compared with CFBI [50], AOT performs better when segmenting multiple highly similar objects (carousels and zebras). (bottom) AOT fails to segment some tiny objects (ski poles and watch) since AOT has no specific design for processing rare tiny objects.
is comparable with CFBI+ [51] (82.8%) while running about 7x faster (27.1 4.0FPS). By using more LSTT blocks, AOT-B improves the performance to 83.5%. Moreover, AOT-L further improves both the seen and unseen scores by utilizing the memory reading strategy, and our R50-AOT-L (84.1 9 /e488.11% %) significantly outperforms the previous methods on the validation 2018/20 split while maintaining an efficient speed (14.9FPS).
DAVIS 2017 [31] is a multi-object extension of DAVIS 2016. The validation split of DAVIS 2017 consists of 30 videos with 59 objects, and the training split contains 60 videos with 138 objects. Moreover, the testing split contains 30 more challenging videos with 89 objects in total.
Table 2: The quantitative evaluation on the single-object DAVIS 2016 [30].
Methods
J&F
J
F
FPS
CFBI+ [51] (Y)
89.9
88.7
91.1
5.9
KMN [34] (Y)
90.5
89.5 91.5
8.3
Table lb shows that our R50-AOT-L (Y) surpasses all the competitors on both the DAVIS- validation
AOT-T (Y)
86.8
86.1 87.4
51.4
AOT-S (Y)
89.4
88.6 90.2 40.0
(84.9%) ) and testing (79.6% % splits and maintains an efficient speed (FP.O.PS). Notably, such multi- object speed is the same as our single-object speed on DAVIS 2016. For the first time, processing mul- tiple objects can be as efficient as processing single object over the AOT framework. We also evaluate method without training with YouTube-VOOS and AOT-S (79.2%) performs much better than KMN [34] (76.0%) by +3.2%.
AOT-B (Y)
89.9 88.7 91.1 29.6
AOT-L (Y)
90.4
89.6 91.1 18.7
R50-AOT-L (Y)
91.1
90.1 92.1 18.0
SwinB-AOT-L (Y)
92.0
90.7 93.3 12.1
DAVIS 2016 [30] is single-object benchmark containing 20 videos in the validation split. Although our AOT aims at improving multi-object video segmentation, we also achieve a new state-of-the-ar performance DAVIS 2016 (R50-AOT-L (Y), 91.1% Under single-object scenarios, the multi- object superiority of AOT is limited, but R50-AOT-L still maintains an about 2x efficiency compared to KMN (18.0 ys 8. 8.3FPS). Furthermore, smaller variant, AOT-B (89.9%), achieves comparable performance with CFBI+ (89.9%) while running 5x faster (29.6 vs 5.9FPS).
Apart from the above results, replacing the AOT encoder from commonly used ResNer50 to SwinB can further boost our performance higher level (Table 1a, 1b, and 2).
Fig. 3 visualizes some qualitative results in comparison with CFBI [50], which only associates each object with its relative background. As demonstrated, CFBI is easier confuse multiple highly similar objects. In contrast, our AOT tracks and segments all the targets accurately by associating all the objects uniformly. However. AOT fails to segment some tiny objects (ski poles and watch) since we do not make special designs for tiny objects.
Qualitative results:
6.2 Ablation Study
In this section, we analyze the main components and syper-parameters of AOT and evaluate their impact the VOS performance in Table 3.
Identity number: The number of the identification vectors, M, have to be larger than the object number in vidcos. Thus, we set M to 10 in default to be consistent with the maximum object number

---

Table Ablation study. The experiments are based on AOT-S and conducted on the validation 2018 split of YouTube-VOS [48] without pre- -re-training on synthetic videos. Self: the position embedding type used in the self-attention. Rel: use relative positional embedding [35] on the local attention.
(a) Identity number
(b) Local window size
(c) Local frame number
M
J&F gaeen Junseen
À
J&F geeen J gnseen
n
J&F Joeen
J unsseen
10
80.3
80.6
73.7
15
80.3
80.6
73.7
1
80.3
80.6
73.7
15
79.0
79.4
72.1
11
78.8
79.5
71.9
2
80.0
79.8
73.7
20
78.3
79.4
70.8
7
78.3
79.3
70.9
79.1
80.0
72.2
77.2
78.5
70.2
0
74.3
74.9
67.6
0
74.3
74.9
67.6
(d) LSTT block number
(e) Positional embedding
J&F
gucen
Junseen
FPS
Param
Self
Rel
J&F
green J uussee
2
80.3
80.6
73.7
27.1
7.0M
sine
80.3
80.6
73.7
3
80.9
81.1
74.0
20.5
8,3M
none
80.
80.4
73.5
1
77.9
78.8
71.0
41.0
5.7M
sine
79.7
80.1
72.9
in the benchmarks [48, 31]. As seen in Table 3a, M larger than 10 leads worse performance since (1) no training video contains so many objects; (2) embedding more than 10 objects into the space with only 256 dimensions is difficult.
Local window size: Table 3b shows that larger local window size, A, results in better performance. Without the local attention, 0, the performance of AOT significantly drops from 80.3% to 74.3%, which demonstrates the necessity the local attention.
Local frame number: In Table 3c, we also try to employ more previous frames in the local attention, but using only the frame (80.3%) performs better than using 2/3 frames (80.0%/79.3 1)). A possible reason is that the longer the temporal interval, the more intense the motion between frames, so it is easier to introduce more errors in the local matching when using an earlier previous frame.
LSTT block number: As shown in Table 3d, the AOT performance increases by using more LSTT blocks. Notably, the AOT with only one LSTT block (77.9% reaches a fast real-time speed (41.OFPS) on YouTube- VOS, although the performance is -2.4% worse than AOT-S (80.3%). By adjusting the LSTT block number, we can flexibly balance the accuracy and speed of AOT.
Position embedding: In our default setting, we apply fixed sine spatial positional embedding to the self-attention following [8], and our local attention is equipped with learned relative positional embedding [35]. The ablation study is shown in Table 3e, where removing the sine embedding decreases the performance to 80.1 1 % slightly. In contrast, the relative embedding is more important than the sine embedding. Without the relative embedding, the performance drops to 79.7%, which means the motion relationship between adjacent frames is helpful for local attention. We also tried to apply learned positional embedding to self-attention modules, but no positive effect was observed.
7 Conclusion
This paper proposes a novel and efficient approach for video object segmentation by associating objects with transformers and achieves superior performance on three popular benchmarks. A simple yet effective identification mechanism proposed to associate, match, and decode all the objects uniformly under multi-objec scenarios. For the first time, processing multiple objects in VOS can be efficient as processing single object by using the identification mechanism. In addition, a long short-term transformer is designed for constructing hierarchical object matching and propagation for VOS. The hierarchical structure allows us to flexibly balance AOT between real-time speed and state- of-the-art performance by adjusting the LSTT number. We hope the identification mechanism will help ease the future study of multi-object VOS and related tasks (e.g., video instance segmentation, interactive VOS, and multi-object tracking), and AOT will serve as a solid baseline.

---

References
[i] Parallel distributed deep learning: Machine learning framework from industrial practice. https:
//www. ppddl //ww:ppdddl epaddle org. cn/
[2] Avinash Ramakanth, S., Venkatesh Babu, R.: Seamseg: Video object segmentation using patch
seams. In: CVPR. pp. 376-383 (2014)
[3] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. In: NIPS Workshops (2016)
[4] Badrinarayanan, V., Galasso, F., Cipolla, R.: Label propagation video sequences. In: CVPR.
pp. 0p.3325-3277 IEEE (2010)
[5] Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning align and
translate. In: ICLR (2015)
[6] Bhat, G., Lawin, FJJ. Danelljan, M., Robinson, A., Felsberg, M., Van Gool, L., Timofte, R.: Learning what to learn for video object segmentation. In: ECCV (2020)
[7] Caelles, S., Maninis, K.K., Pont-Tuset, J., Leil-Taixé, L., Cremers, D., Van Gool, L.: One-shot video object segmentation. In: CVPR. pp. 221 221-230 (2017)
[8] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV. pp. 213-229. Springer (2020)
[9] Chen, Y., Pont-Tusets J., Montes, A., Van Gool, L.: Blazingly fast video object segmentation with pixel-wise metric learning. In: CVPR. pp. 1189-1111111 1198-198 (2018)
[10] Cheng, M.M., Mitra, N.J., Huang, X., Torr, P.H., Hu, S.M.: Global contrast based salient region detection. TPAMI 37(3), 569-5692229 (2014)
[11] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL. pp. 4171-41888 (2019)
[12] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al.: An image is worth 16x 16 words: Transformers for image recognition at scale. In: ICLR (2021)
[13] Duke, B., Ahmed, A., Wolf, C., Aarabi, P., Taylor, G.W.: Sstvos: Sparse spatiotemporal transformers for video object segmentation. In: CVPR (2021)
[14] Everingham, M., Van Gool, L.,, Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. IJCV 88(2), 303-338 (2010)
[15] Hariharan, B., Arbeláez, P., Bourdev, L., Maji. S., Malik, J.: Semantic contours from inverse detectors. In: ICCV. pp. 991-998. IEBE (2011)
[16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
[17] Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint 1rXiv:1606.08415 (2016)
[18] Hu, Y.T., Huang, J.B., Schwing, A.G.: Videomatch: Matching based video object segmentation. In: ECCV. pp. 54-70 (2018)
[19] Lin, H., Qi, X., Jia, J.: voss-vos: Attention guided single-shot video object segmentation. In: ICCV. pp. 3949-3399999 -957-3977660 (2019)
[20] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR. pp. 2117-2125 (2017)
[21] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV, pp. 740-755. Springer (2014)
[22] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer:
Hierarchical vision transformer using shifted windows. In: ICCV (2021)
[23] Lu, X., Wang, W., Danelljan, M., Zhou, T., Shen, J., Van Gool, L.: Video object segmentation with episodic graph memory networks. In: ECCV (2020)
[24] Luiten, J., Voigtlaender, P., Leibe, B.: Premvos:
refinement and merging
for video object segmentation. In: ACCV. pp. 565-580 (2018)
10

---

[25] Ngan, K.N., Li, H.: Video segmentation and its applications. Springer Science & Business Media (2011)
[26] Oh, S.W., Lee, J.Y., Xu, N., Kim, S.J.: Video object segmentation using space- time memory networks. In: ICCV (2019)
[27] Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., Tran, D.: Image transformer. In: ICCV, pp. 4055-4064, PMLR (2018)
[28] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang. E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L.,, Lerer, A.: Automatic differentiation in pytorch (2017)
[29] Perazzi, F., Khoreva, A., Benenson, R., Schiele, B., Sorkine-Hornung, A.: Learning video object segmentation from static images. In: CVPR. pp. 1663-2672 (2017)
[30] Perazzi, F., Pont- Joste, McWilliams, B., Van Gool, L., Gross, M., Dorkine-Hornund A.: benchmark dataset and evaluation methodology for video object segmentation. In: CVPR. pp. 724-732 (2016)
[31] Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine- A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: 770.00667 (2017)
[32] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
[33] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: CVPR. pp. 4510-4520 (2018)
[34] Seong, H., Hyun, J., Kim, E.: Kernelized memory network for video object segmentation. In: ECCV (2020)
[35] Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. In: NAACL. pp. 464-466 468 (2018)
[36] Shi, J., Yan, Q.. Xu, L., Jia, J.: Hierarchical image saliency detection on extended cssd. TPAMI 38(4), 717-729 (2015)
[37] Synnaeve, G., Xu, Q., Kahn, J., Likhomanenko, T., Grave, E., Pratap, V., Sriram, A., Liptchinsky, V., Collobert, R.: End-to-end asr: from supervised to semi-supervised learning with modern architectures. In: ICML Workshops (2020)
[38] Vaswani, A., Ramachandran, P., Srinivas, A., Parmar, N., Hechtman, B., Shlens, J.: Scaling local self-attention for parameter efficient visual backbones. In: CVPR. pp. 12894-12904 (2021)
[39] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.,
Polosukhin, I.: Attention is all you need. In: NIPS (2017)
[40] Vijayanarasimhan, S., Grauman, K.: Active frame selection for label propagation in videos. In:
ECCV. pp. 496-59999999 509. Springer (2012)
[41] Voigtlaender, P., Chai, Y., Schroff, F., Adam, H., Leibe, B., Chen, L.C.: Feelvos: Fast end-to-end
embedding learning for video object segmentation. In: CVPR. pp. 9481-9499 (2019)
[42] Voigtlaender, P., Leibe, B.: Online adaptation of convolutional neural networks for video object
segmentation. In: BMVC (2017)
[43] Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: CVPR. pp. 7794- 7803 (2018)
[44] Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., Xia, H.: End-to-end video instance segmentation with transformers. In: CVPR. pp. 8741-8750 (2021)
[45] Wu, Y., He, K.: Group normalization. In: ECCV. pp. 3-19 (2018)
[46] Wug Oh, S., Lee, J.Y., Sunkavalli, K., Joo Kim, S.: Fast video object segmentation by reference- guided mask propagation. In: CVPR. pp. 7376-7385 (2018)
[47] Xiao, H., Feng, J., Lin, G., Liu, Y., Zhang. M.: Monet: Deep motion exploitation for video
object segmentation. In: CVPR. pp- 1140-1144 1148 (2018)
[48] Xu, N., Yang, I.., Fan, Y., Yue, D., Liang. Y., Yang, J., Huang, T.: Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv: MXXiv1880.003227 (2018)
[49] Yang, I.., Wang, Y., Xiong, X., Yang, J., Katsaggelos, A.K.: Efficient video object segmentation via network modulation. In: CVPR. pp. 6499-6507 (2018)

---

[50] Yang, Z., Wei, Y. Yang. Y.: Collaborative video object segmentation by foreground backuuuuudbddkddddd integration. In: ECCV (2020)
[51] Yang, Z., Wei, Y., Yang, Y.: Collaborative video object segmentation by multi-scale foreground- background integration. TPAMI (2021)
[52] Zhang, Z., Fidler, S., Urtasun, R.: Instance-level segmentation for autonomous driving with deep densely connected mrfs. In: CVPR. pp. 669-677 (2016)
12


Reviews:
[{"review_id": "emV9MbkZAq", "reviewer": "Reviewer_Btvt", "summary": "This submission addresses the task of semi-supervised video object segmentation (VOS), in which given a set of object mask over a frame, these masks must be propagated to the rest of the frames of the sequence.\n\nThe main contribution is proposing a method to treat the multiple objects to be segmented in a single pass, by defining an Identification Embedding that encodes all masks simultaneously. Previous works in VOS used independent embeddings for each object, which required multiple decoding passes, one for each object. With the proposed Identification Embedding and Identification Decoding mechanisms which are trained end-to-end.\n\nThe submission also introduced a  Long Short-Term Transformer (LSTT) combines a long-term attention related to the first frame, where the object segments are provided as ground truth, together with a short -term attention related to nearby frames.\n\nThe experiments on YouTube-VOS and DAVIS set new state of the art results in terms of accuracy and speed at inference time, especially in the case of multiple object in the video scene.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Update on August 27, 2021\nAfter reading the answers of the authors to my concerns, I upgraded my score from \"7: Good paper, accept\" to \"8: Top 50% of accepted NeurIPS papers, clear accept\".\n\n## Strengths\n\nS1 Using a single embedding for all the objects allows decoding all the objects in the scene in a single pass, while the related work would require a different decoding step for each segmented object, in such a way that the computational requires will grow with the amount of objects.\n\nS2 The mechanism to develop a trainable scheme for identification embedding and decoding is novel and sounding.\n\nS3 The hierarchical attention mechanism, especially in the case of short-term attention, is convincing in the sense that it attends to nearby temporal frames and nearby spatial pixels frames, in addition to the further frames, attended at a global spatial scale.\n\nS4 The experimental results are careful designed to provide variations of AOT that are adopt the design choice of other works in the state of the art.\n\nS5 The presented results are convincingly faster than the state of the art, together with an incremental gain in accuracy.\n\nS6 The ablation study proves the gains of the main contributions of the paper.\n\nS7 The source code is promised upon acceptance.\n\n\n\n## Weaknesses\n\nW1 The concept of multiple object association is not also tested in the very similar problem of multiple object tracking, which should benefit from the same reasoning.\n\nW2 The manuscript uses the adjective \"uniformly\" (l.35, l.41...) as a contrast to the \"independently\" (l.32), but it is not clear what \"uniformly\" refers to in this context. It seems to indicate \"simultaneously\" because of the reference to the computational requirements and the fact that a single embedding encodes all the objects. The use of the \"uniform\" term should be better explained or reconsidered.\n\nW3 Is it not clear whether the backbone used by the other works is comparable to the one used in AOT (MobileNet). The authors should discuss whether the gains may come from the different backbones and include it in the table with comparative results.\n\nW4 A discussion regarding the memory footprint of the model and its required memory at inference time is missing. This would also allow a better comparison with the related work, beyond the accuracy and inference times reported.\n\nW5 While the reported frames per second are better than the related work, it is not clear if the results are comparable in terms of hardware. How were the FPS results obtained ? Is it possible that the speed gains are simply due to the use of the distributed 4 Tesla V100 GPU mentioned in l.256 ?\n\nW6 A discussion to this missing related work is missing:\n\nZhang, D., Zhang, H., Tang, J., Wang, M., Hua, X., & Sun, Q. Feature pyramid transformer. ECCV 2020.\n\n\n\n## QUESTIONS\n\nQ1 In Table 3a, did you experiment also with M<10 ? While M=10 is well grounded based on the related work, it is actually one more hyperparameter whose modification may provide some additional gain in performance.\n\nQ2 Similarly to Q1, did you try with larger window sizes ? When reporting results, it is somehow surprising that the next step after the best results ($\\lambda=9$) is not included.\n\n## OBSERVATIONS\n\nO1 While CVPR 2021 was celebrated after the NeurIPS 2021 submission deadline, the authors may  **optionally** also consider including some CVPR 2021 papers in the discussion and results on video object segmentation.\n\nDuke, B., Ahmed, A., Wolf, C., Aarabi, P., & Taylor, G. W. [SSTVO: Sparse spatiotemporal transformers for video object segmentation](https://openaccess.thecvf.com/content/CVPR2021/html/Duke_SSTVOS_Sparse_Spatiotemporal_Transformers_for_Video_Object_Segmentation_CVPR_2021_paper.html). CVPR 2021. [[code]](https://github.com/dukebw/SSTVOS)\n\nWang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., & Xia, H. [End-to-end video instance segmentation with transformers]((https://openaccess.thecvf.com/content/CVPR2021/html/Wang_End-to-End_Video_Instance_Segmentation_With_Transformers_CVPR_2021_paper.html)). CVPR 2021. [[code]](https://github.com/Epiphqny/VisTR)\n\nVaswani, Ashish, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. [\"Scaling Local Self-Attention For Parameter Efficient Visual Backbones.\"](https://openaccess.thecvf.com/content/CVPR2021/html/Vaswani_Scaling_Local_Self-Attention_for_Parameter_Efficient_Visual_Backbones_CVPR_2021_paper.html) CVPR 2021.\n\nO2 Similarly to O1, the authors may  **optionally** consider citing and discussing the arXivs introducing SwinTransformers and its application to video:\n\nLiu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., ... & Guo, B. (2021). [Swin transformer: Hierarchical vision transformer using shifted windows](https://arxiv.org/abs/2103.14030). arXiv preprint arXiv:2103.14030.\n\nLiu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2021). [Video Swin Transformer](https://arxiv.org/abs/2106.13230). arXiv preprint arXiv:2106.13230.\n\nO4 l.30: \"Above methods\" is mentioned twice, it could be rephrased.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the authors did not include any reference.\n\nThe authors may refer to the privacy concerns that video surveillance systems pose, as they may benefit from the presented contributions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ZxHA0ButVO", "reviewer": "Reviewer_HPRW", "summary": "This work proposes two novel techniques, an identity matching and a long-short term transformer block. Their first contribution, identity matching, enables them to handle multiple object scenarios as fast as single object ones and process all objects simultaneously. Their second contribution, enables them to handle occlusion better while keeping boundaries accurate. Overall they achieve sota results on 3 main semi supervised video object segmentation tasks.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work proposes to initialize M identity vectors (M is set to 10 in practice which is the maximum number of objects). Then using a random permutation matrix N  (number of present objects) of them are assigned to the current video. They augment the value vector of the memory with the identity embedding by adding them together. With this strategy they are able to track multiple objects simultaneously with a matrix multiplication and addition. This is a novel and interesting contribution which enables them to get a very high FPS while achieving sota. In extreme with their tiny variant they are able to achieve about 40 FPS processing which is impressive given their accuracy.\n\nTheir other main contribution is having two attention heads, one of them looks at the local spatial positions in the previous frame (7x7 very local!) and the other one looks at the whole image but from previous frames. The long attention either only looks at the first frame or at every 5 frame in the memory constructing their best large model.\n\nExperimentally they show sota results on both DAVIS and YoutubeVOS which are the two main semi supervised benchmarks and they outperform all the previous works significantly. Impressively while maintaining a better FPS too. \n\nBoth of their contributions are significant in terms of performance while not adding too much complexity or computational overhead. \n\nThey also perform an ablation study on all the interesting factors of their model, such as number of LSTT blocks, number of short term history frames and window size. \n\nTheir performance drops with more identity vectors than necessary (such as 20 or 30). An interesting visualization in appendix shows that the identity vectors start getting correlated. Their hypothesis is it's harder to optimize for 30 disjoint vectors in a 250D space. It would be interesting if this can be resolved with a proper regularizer on the identity vectors such as orthogonality enforcer. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have noted that this method fails on rare/small objects. They also claim since this is a standard benchmark their work does not deviate or add any negative impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "AmMXFpWlTNm", "reviewer": "Reviewer_kjR8", "summary": "In this paper, the authors study better and efficient way to associate objects in the multi-object video object segmentation. In contrast with previous methods that segment each target object separately, the proposed method matches and decodes multiple objects uniformly and efficiently. For this, an identity bank and a Long Short-Term Transformer is designed. The proposed method AOT become the new state-of-the-art on Youtube-VOS and DAVIS benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The proposed method present a new paradigm for the semi-supervised VOS deviant from predominant STM-based methods. The proposed method, by design, is efficient when handling video with a large number of objects. e.g., it has constant computational complexity regardless of the number of objects. \n- The authors proposed several novel technical components. 1) a new identification mechanism is proposed. An identity bank is employed to assign identities into feature map. With this mechanism, pixel matching become identity-agnostic thus matching results can be shared across targets. 2) a LSTT block is proposed. For my understanding, it is an improved version of memory reading block in STM with transformer style structure and separation of long and short term attention. \n- The proposed methods show the state-of-the-art performance on three popular benchmarks. \n- The paper includes an extensive ablation analysis. \n\nWeakness:\n- I have some concerns on identification mechanism based on identity bank. 1) Scalability. As shown in Table 3 (a), the performance is getting worse with growth of the maximum number of identities. It means that the capacity should be preset to some small number (e.g., 10). In real-world scenario, we can have more than 10 objects and most of the time we don't know how many objects we will need to handle in the future. Have the authors thought about how to scale up without compromising performance? 2) Randomness. Identities are randomly assigned one embedding from the identity bank. How the results are robust against this randomness? It would be undesirable for the result to change with each inference. It would be great to have some analysis on this aspect.\n\nOverall Evaluation:\n- The paper present a novel approach for multi-object video object segmentation and the proposed method outperfrom previous state-of-the-arts on several benchmarks.\n- Now, I would recommend to accept this paper. I will finalize the score after seeing how authors address my concerns in Weakness.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While future works are discussed in Supplementary Materials, I encourage the authors to include more discussions on limitations and societal impacts. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "6uSriKZFmOC", "reviewer": "Reviewer_tUYE", "summary": "The paper tackles the problem of video object segmentation. Firstly, the authors propose an identification mechanism which allows jointly segmenting multiple objects in a video with constant computational complexity w.r.t. the number of objects. This is achieved by mapping the masks of different objects to a common embedding space, which is then decoded by a single decoder. Secondly, the authors introduce a transformer based architecture with both long term (old frames) and short term (recent frames) matching to obtain accurate segmentation. The proposed approach obtains state-of-the-art results on both YouTubeVOS (83.7 $\\mathcal{J}$&$\\mathcal{F}$ on val2018) and DAVIS (83.0% $\\mathcal{J}$&$\\mathcal{F}$ on DAVIS 2017) benchmarks, while being computationally efficient.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### **Strengths** \nS1) The proposed identification mechanism to handle multiple objects is novel and interesting. Compared to existing approaches, the proposed module allows segmenting multiple objects simultaneously without incurring additional computational costs compared to segmenting a single object. \n\nS2) The second contribution of long short-term transformer based VOS architecture is also novel, interesting, and shown to be effective via appropriate ablation studies.\n\nS3) The method obtains impressive results on standard VOS benchmarks YouTubeVOS and DAVIS, outperforming previous state-of-the-art by a significant margin.\n\nS4) The paper is well-written and easy to read and understand.\n\n\n### **Weaknesses**\n\nW1) I do not think the use of term 'hierarchical matching' (used throughout the paper) is strictly correct. If I understand correctly, the authors stack multiple transformer blocks together to process the input frame, as commonly done in transformer based architectures. Referring to this as hierarchical matching seems confusing and slightly misleading. I would suggest omitting the use of 'hierarchical'.\n\nW2) How are the FPS for other methods obtained in Table 1. Did the authors compute them using identical hardware as their method, or are they directly obtained from the corresponding papers? This should be clearly stated. Since the implementation of SOTA methods such as STM, CFBI, LWL are available, I would recommend running these methods using identical hardware in order to obtain a fairer speed comparison. \n\n\nWhile not strictly necessary, I believe the paper will greatly benefit from additional experimental analysis of the proposed method, as listed below.\n\nC1) The proposed approach obtains improved results compared to previous methods. However, it is not clear whether the improvement is mainly due to the long short-term transformer based architecture, or whether the joint segmentation of multiple targets enabled by the identification mechanism also leads to improved segmentation accuracy, in addition to computational efficiency. Thus, it would be interesting to evaluate a variant of the method which independently segments each object using the long short-term transformer, and then merges the masks using a soft-aggregation as done in STM.\n\nC2) What is the impact of learning the identification vectors? What would be the performance drop if random identification vectors are used instead of the learned ones?\n\nC3) The authors add the identification embedding of the target mask to the value computed using the image features. Did the authors experiment any other architectures to integrate the identity embedding into the value? For instance, is it possible to concatenate the identity embedding with the initial value and then pass it through a small network? \n\nC4) What is the performance if only the short term (local) attention is used?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the failure cases of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The rebuttal addressed all of the reviewers concerns, and all reviewers recommend acceptance. The AC agrees with this recommendation.
Decision: Accept (Poster)