Associating Objects with Transformers for Video Object Segmentation

1 Introduction . Video Object Segmentation ( VOS ) is a fundamental task in video understanding with many potential applications , including augmented reality [ 25 ] and self-driving cars [ 52 ] . The goal of semi-supervised VOS , the main task in this paper , is to track and segment object ( s ) across an entire video sequence based on the object mask ( s ) given at the first frame . Thanks to the recent advance of deep neural networks , many deep learning based VOS algorithms have been proposed recently and achieved promising performance . STM [ 26 ] and its following works [ 34 , 23 ] leverage a memory network to store and read the target features of predicted past frames and apply a non-local attention mechanism to match the target in the current frame . FEELVOS [ 41 ] and CFBI [ 50 , 51 ] utilize global and local matching mechanisms to match target pixels or patches from both the first and the previous frames to the current frame . Even though the above methods have achieved significant progress , the above methods learn to decode scene features that contain a single positive object . Thus under a multi-object scenario , they have to match each object independently and ensemble all the single-object predictions into a multi-object segmentation , as shown in Fig . 1a . Such a post-ensemble manner eases network architectures ’ design 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . since the networks are not required to adapt the parameters or structures for different object numbers . However , modeling multiple objects independently , instead of uniformly , is inefficient in exploring multi-object contextual information to learn a more robust feature representation for VOS . In addition , processing multiple objects separately yet in parallel requires multiple times the amount of GPU memory and computation for processing a single object . This problem restricts the training and application of VOS under multi-object scenarios , especially when computing resources are limited . To solve the problem , Fig . 1b demonstrates a feasible approach to associate and decode multiple objects uniformly in an end-to-end framework . Hence , we propose an Associating Objects with Transformers ( AOT ) approach to match and decode multiple targets uniformly . First , an identification mechanism is proposed to assign each target a unique identity and embed multiple targets into the same feature space . Hence , the network can learn the association or correlation among all the targets . Moreover , the multi-object segmentation can be directly decoded by utilizing assigned identity information . Second , a Long Short-Term Transformer ( LSTT ) is designed for constructing hierarchical object matching and propagation . Each LSTT block utilizes a long-term attention for matching with the first frame ’ s embedding and a short-term attention for matching with several nearby frames ’ embeddings . Compared to the methods [ 26 , 34 ] utilizing only one attention layer , we found hierarchical attention structures are more effective in associating multiple objects . We conduct extensive experiments on two popular multi-object benchmarks for VOS , i.e. , YouTubeVOS [ 48 ] and DAVIS 2017 [ 31 ] , to validate the effectiveness and efficiency of the proposed AOT . Even using the light-weight Mobilenet-V2 [ 33 ] as the backbone encoder , the AOT variant networks achieve superior performance on the validation 2018 & 2019 splits of the large-scale YouTube-VOS ( ours , J & F 82.6∼ 84.5 % & 82.2∼ 84.5 % ) while keeping more than 2× faster multi-object run-time ( 27.1∼ 9.3FPS ) compared to the state-of-the-art competitors ( e.g. , CFBI [ 50 ] , 81.4 % & 81.0 % , 3.4FPS ) . We also achieve new state-of-the-art performance on both the DAVIS-2017 validation ( 85.4 % ) and testing ( 81.2 % ) splits . Moreover , AOT is effective under single-object scenarios as well and outperforms previous methods on DAVIS 2016 [ 30 ] ( 92.0 % ) , a popular single-object benchmark . Besides , our smallest variant , AOT-T , can maintain real-time multi-object speed on all above benchmarks ( 51.4FPS on 480p videos ) . Particularly , AOT ranked 1st in the Track 1 ( Video Object Segmentation ) of the 3rd Large-scale Video Object Segmentation Challenge . Overall , our contributions are summarized as follows : • We propose an identification mechanism to associate and decode multiple targets uniformly for VOS . For the first time , multi-object training and inference can be efficient as single-object ones , as demonstrated in Fig . 1c . • Based on the identification mechanism , we design a new efficient VOS framework , i.e. , Long ShortTerm Transformer ( LSTT ) , for constructing hierarchical multi-object matching and propagation . LSTT achieves superior performance on VOS benchmarks [ 48 , 31 , 30 ] while maintaining better efficiency than previous state-of-the-art methods . To the best of our knowledge , LSTT is the first hierarchical framework for object matching and propagation by applying transformers [ 39 ] to VOS . 2 Related Work . Semi-supervised Video Object Segmentation . Given one or more annotated frames ( the first frame in general ) , semi-supervised VOS methods propagate the manual labeling to the entire video sequence . Traditional methods often solve an optimization problem with an energy defined over a graph structure [ 4 , 40 , 2 ] . In recent years , VOS methods have been mainly developed based on deep neural networks ( DNN ) , leading to better results . Early DNN methods rely on fine-tuning the networks at test time to make segmentation networks focus on a specific object . Among them , OSVOS [ 7 ] and MoNet [ 47 ] fine-tune pre-trained networks on the first-frame ground-truth at test time . OnAVOS [ 42 ] extends the first-frame fine-tuning by introducing an online adaptation mechanism . Following these approaches , MaskTrack [ 29 ] and PReM [ 24 ] utilize optical flow to help propagate the segmentation mask from one frame to the next . Despite achieving promising results , the test-time fine-tuning restricts the network efficiency . Recent works aim to achieve a better run-time and avoid using online fine-tuning . OSMN [ 49 ] employs one convolutional network to extract object embedding and another one to guide segmentation predictions . PML [ 9 ] learns pixel-wise embedding with a nearest neighbor classifier , and VideoMatch [ 18 ] uses a soft matching layer that maps the pixels of the current frame to the first frame in a learned embedding space . Following PML and VideoMatch , FEELVOS [ 41 ] and CFBI [ 50 , 51 ] extend the pixel-level matching mechanism by additionally matching between the current frame and the previous frame . RGMP [ 46 ] also gathers guidance information from both the first frame and the previous frame but uses a siamese encoder with two shared streams . STM [ 26 ] and its following works ( e.g. , EGMN [ 23 ] and KMN [ 34 ] ) leverage a memory network to embed past-frame predictions into memory and apply a non-local attention mechanism on the memory to decode the segmentation of the current frame . SST [ 13 ] utilizes attention mechanisms in a different way , i.e. , transformer blocks [ 39 ] are used to extract pixel-level affinity maps and spatial-temporal features . The features are target-agnostic , instead of target-aware like our LSTT , since the mask information in past frames is not propagated and aggregated in the blocks . Instead of using matching mechanisms , LWL [ 6 ] proposes to use an online few-shot learner to learn to decode object segmentation . The above methods learn to decode features with a single positive object and thus have to match and segment each target separately under multi-object scenarios , consuming multiple times computing resources of single-object cases . The problem restricts the application and development of the VOS with multiple targets . Hence , we propose our AOT to associate and decode multiple targets uniformly and simultaneously , as efficiently as processing a single object . Visual Transformers . Transformers [ 39 ] was proposed to build hierarchical attention-based networks for machine translation . Similar to Non-local Neural Networks [ 43 ] , transformer blocks compute correlation with all the input elements and aggregate their information by using attention mechanisms [ 5 ] . Compared to RNNs , transformer networks model global correlation or attention in parallel , leading to better memory efficiency , and thus have been widely used in natural language processing ( NLP ) tasks [ 11 , 32 , 37 ] . Recently , transformer blocks were introduced to many computer vision tasks , such as image classification [ 12 , 38 , 22 ] , object detection [ 8 ] /segmentation [ 44 ] , and image generation [ 27 ] , and have shown promising performance compared to CNN-based networks . Many VOS methods [ 19 , 26 , 23 , 34 ] have utilized attention mechanisms to match the object features and propagate the segmentation mask from past frames to the current frames . Nevertheless , these methods consider only one positive target in the attention processes , and how to build hierarchical attention-based propagation has been rarely studied . In this paper , we carefully design a long shortterm transformer block , which can effectively construct multi-object matching and propagation within hierarchical structures for VOS . 3 Revisit Previous Solutions for Video Object Segmentation . In VOS , many common video scenarios have multiple targets or objects required for tracking and segmenting . Benefit from deep networks , current state-of-the-art VOS methods [ 26 , 50 ] have achieved promising performance . Nevertheless , these methods focus on matching and decoding a single object . Under a multi-object scenario , they thus have to match each object independently and ensemble all the single-object predictions into a multi-object prediction , as demonstrated in Fig . 1a . Let FN denotes a VOS network for predicting single-object segmentation , and A is an ensemble function such as softmax or the soft aggregation [ 26 ] , the formula of such a post-ensemble manner for processing N objects is like , Y ′ = A ( FN ( It , Im , Y m1 ) , ... , F N ( It , Im , Y mN ) ) , ( 1 ) where It and Im denote the image of the current frame and memory frames respectively , and { Y m1 , ... , Y mN } are the memory masks ( containing the given reference mask and past predicted masks ) of all the N objects . This manner extends networks designed for single-object VOS into multi-object applications , so there is no need to adapt the network for different object numbers . Although the above post-ensemble manner is prevalent and straightforward in the VOS field , processing multiple objects separately yet in parallel requires multiple times the amount of GPU memory and computation for matching a single object and decoding the segmentation . This problem restricts the training and application of VOS under multi-object scenarios when computing resources are limited . To make the multi-object training and inference as efficient as single-object ones , an expected solution should be capable of associating and decoding multiple objects uniformly instead of individually . To achieve such an objective , we propose an identification mechanism to embed the masks of any number ( required to be smaller than a pre-defined large number ) of targets into the same high-dimensional space . Based on the identification mechanism , a novel and efficient framework , i.e. , Associating Objects with Transformers ( AOT ) , is designed for propagating all the object embeddings uniformly and hierarchically , from memory frames to the current frame . As shown in Fig . 1b , our AOT associates and segments multiple objects within an end-to-end framework . For the first time , processing multiple objects can be as efficient as processing a single object ( Fig . 1c ) . Compared to previous methods , our training under multi-object scenarios is also more efficient since AOT can associate multiple object regions and learn contrastive feature embeddings among them uniformly .

Summary: This submission addresses the task of semi-supervised video object segmentation (VOS), in which given a set of object mask over a frame, these masks must be propagated to the rest of the frames of the sequence.  The main contribution is proposing a method to treat the multiple objects to be segmented in a single pass, by defining an Identification Embedding that encodes all masks simultaneously. Previous works in VOS used independent embeddings for each object, which required multiple decoding passes, one for each object. With the proposed Identification Embedding and Identification Decoding mechanisms which are trained end-to-end.  The submission also introduced a  Long Short-Term Transformer (LSTT) combines a long-term attention related to the first frame, where the object segments are provided as ground truth, together with a short -term attention related to nearby frames.  The experiments on YouTube-VOS and DAVIS set new state of the art results in terms of accuracy and speed at inference time, especially in the case of multiple object in the video scene.