Associating Objects with Transformers for Video Object Segmentation

1 Introduction . Video Object Segmentation ( VOS ) is a fundamental task in video understanding with many potential applications , including augmented reality [ 25 ] and self-driving cars [ 52 ] . The goal of semi-supervised VOS , the main task in this paper , is to track and segment object ( s ) across an entire video sequence based on the object mask ( s ) given at the first frame . Thanks to the recent advance of deep neural networks , many deep learning based VOS algorithms have been proposed recently and achieved promising performance . STM [ 26 ] and its following works [ 34 , 23 ] leverage a memory network to store and read the target features of predicted past frames and apply a non-local attention mechanism to match the target in the current frame . FEELVOS [ 41 ] and CFBI [ 50 , 51 ] utilize global and local matching mechanisms to match target pixels or patches from both the first and the previous frames to the current frame . Even though the above methods have achieved significant progress , the above methods learn to decode scene features that contain a single positive object . Thus under a multi-object scenario , they have to match each object independently and ensemble all the single-object predictions into a multi-object segmentation , as shown in Fig . 1a . Such a post-ensemble manner eases network architectures ’ design 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . since the networks are not required to adapt the parameters or structures for different object numbers . However , modeling multiple objects independently , instead of uniformly , is inefficient in exploring multi-object contextual information to learn a more robust feature representation for VOS . In addition , processing multiple objects separately yet in parallel requires multiple times the amount of GPU memory and computation for processing a single object . This problem restricts the training and application of VOS under multi-object scenarios , especially when computing resources are limited . To solve the problem , Fig . 1b demonstrates a feasible approach to associate and decode multiple objects uniformly in an end-to-end framework . Hence , we propose an Associating Objects with Transformers ( AOT ) approach to match and decode multiple targets uniformly . First , an identification mechanism is proposed to assign each target a unique identity and embed multiple targets into the same feature space . Hence , the network can learn the association or correlation among all the targets . Moreover , the multi-object segmentation can be directly decoded by utilizing assigned identity information . Second , a Long Short-Term Transformer ( LSTT ) is designed for constructing hierarchical object matching and propagation . Each LSTT block utilizes a long-term attention for matching with the first frame ’ s embedding and a short-term attention for matching with several nearby frames ’ embeddings . Compared to the methods [ 26 , 34 ] utilizing only one attention layer , we found hierarchical attention structures are more effective in associating multiple objects . We conduct extensive experiments on two popular multi-object benchmarks for VOS , i.e. , YouTubeVOS [ 48 ] and DAVIS 2017 [ 31 ] , to validate the effectiveness and efficiency of the proposed AOT . Even using the light-weight Mobilenet-V2 [ 33 ] as the backbone encoder , the AOT variant networks achieve superior performance on the validation 2018 & 2019 splits of the large-scale YouTube-VOS ( ours , J & F 82.6∼ 84.5 % & 82.2∼ 84.5 % ) while keeping more than 2× faster multi-object run-time ( 27.1∼ 9.3FPS ) compared to the state-of-the-art competitors ( e.g. , CFBI [ 50 ] , 81.4 % & 81.0 % , 3.4FPS ) . We also achieve new state-of-the-art performance on both the DAVIS-2017 validation ( 85.4 % ) and testing ( 81.2 % ) splits . Moreover , AOT is effective under single-object scenarios as well and outperforms previous methods on DAVIS 2016 [ 30 ] ( 92.0 % ) , a popular single-object benchmark . Besides , our smallest variant , AOT-T , can maintain real-time multi-object speed on all above benchmarks ( 51.4FPS on 480p videos ) . Particularly , AOT ranked 1st in the Track 1 ( Video Object Segmentation ) of the 3rd Large-scale Video Object Segmentation Challenge . Overall , our contributions are summarized as follows : • We propose an identification mechanism to associate and decode multiple targets uniformly for VOS . For the first time , multi-object training and inference can be efficient as single-object ones , as demonstrated in Fig . 1c . • Based on the identification mechanism , we design a new efficient VOS framework , i.e. , Long ShortTerm Transformer ( LSTT ) , for constructing hierarchical multi-object matching and propagation . LSTT achieves superior performance on VOS benchmarks [ 48 , 31 , 30 ] while maintaining better efficiency than previous state-of-the-art methods . To the best of our knowledge , LSTT is the first hierarchical framework for object matching and propagation by applying transformers [ 39 ] to VOS . 2 Related Work . Semi-supervised Video Object Segmentation . Given one or more annotated frames ( the first frame in general ) , semi-supervised VOS methods propagate the manual labeling to the entire video sequence . Traditional methods often solve an optimization problem with an energy defined over a graph structure [ 4 , 40 , 2 ] . In recent years , VOS methods have been mainly developed based on deep neural networks ( DNN ) , leading to better results . Early DNN methods rely on fine-tuning the networks at test time to make segmentation networks focus on a specific object . Among them , OSVOS [ 7 ] and MoNet [ 47 ] fine-tune pre-trained networks on the first-frame ground-truth at test time . OnAVOS [ 42 ] extends the first-frame fine-tuning by introducing an online adaptation mechanism . Following these approaches , MaskTrack [ 29 ] and PReM [ 24 ] utilize optical flow to help propagate the segmentation mask from one frame to the next . Despite achieving promising results , the test-time fine-tuning restricts the network efficiency . Recent works aim to achieve a better run-time and avoid using online fine-tuning . OSMN [ 49 ] employs one convolutional network to extract object embedding and another one to guide segmentation predictions . PML [ 9 ] learns pixel-wise embedding with a nearest neighbor classifier , and VideoMatch [ 18 ] uses a soft matching layer that maps the pixels of the current frame to the first frame in a learned embedding space . Following PML and VideoMatch , FEELVOS [ 41 ] and CFBI [ 50 , 51 ] extend the pixel-level matching mechanism by additionally matching between the current frame and the previous frame . RGMP [ 46 ] also gathers guidance information from both the first frame and the previous frame but uses a siamese encoder with two shared streams . STM [ 26 ] and its following works ( e.g. , EGMN [ 23 ] and KMN [ 34 ] ) leverage a memory network to embed past-frame predictions into memory and apply a non-local attention mechanism on the memory to decode the segmentation of the current frame . SST [ 13 ] utilizes attention mechanisms in a different way , i.e. , transformer blocks [ 39 ] are used to extract pixel-level affinity maps and spatial-temporal features . The features are target-agnostic , instead of target-aware like our LSTT , since the mask information in past frames is not propagated and aggregated in the blocks . Instead of using matching mechanisms , LWL [ 6 ] proposes to use an online few-shot learner to learn to decode object segmentation . The above methods learn to decode features with a single positive object and thus have to match and segment each target separately under multi-object scenarios , consuming multiple times computing resources of single-object cases . The problem restricts the application and development of the VOS with multiple targets . Hence , we propose our AOT to associate and decode multiple targets uniformly and simultaneously , as efficiently as processing a single object . Visual Transformers . Transformers [ 39 ] was proposed to build hierarchical attention-based networks for machine translation . Similar to Non-local Neural Networks [ 43 ] , transformer blocks compute correlation with all the input elements and aggregate their information by using attention mechanisms [ 5 ] . Compared to RNNs , transformer networks model global correlation or attention in parallel , leading to better memory efficiency , and thus have been widely used in natural language processing ( NLP ) tasks [ 11 , 32 , 37 ] . Recently , transformer blocks were introduced to many computer vision tasks , such as image classification [ 12 , 38 , 22 ] , object detection [ 8 ] /segmentation [ 44 ] , and image generation [ 27 ] , and have shown promising performance compared to CNN-based networks . Many VOS methods [ 19 , 26 , 23 , 34 ] have utilized attention mechanisms to match the object features and propagate the segmentation mask from past frames to the current frames . Nevertheless , these methods consider only one positive target in the attention processes , and how to build hierarchical attention-based propagation has been rarely studied . In this paper , we carefully design a long shortterm transformer block , which can effectively construct multi-object matching and propagation within hierarchical structures for VOS . 3 Revisit Previous Solutions for Video Object Segmentation . In VOS , many common video scenarios have multiple targets or objects required for tracking and segmenting . Benefit from deep networks , current state-of-the-art VOS methods [ 26 , 50 ] have achieved promising performance . Nevertheless , these methods focus on matching and decoding a single object . Under a multi-object scenario , they thus have to match each object independently and ensemble all the single-object predictions into a multi-object prediction , as demonstrated in Fig . 1a . Let FN denotes a VOS network for predicting single-object segmentation , and A is an ensemble function such as softmax or the soft aggregation [ 26 ] , the formula of such a post-ensemble manner for processing N objects is like , Y ′ = A ( FN ( It , Im , Y m1 ) , ... , F N ( It , Im , Y mN ) ) , ( 1 ) where It and Im denote the image of the current frame and memory frames respectively , and { Y m1 , ... , Y mN } are the memory masks ( containing the given reference mask and past predicted masks ) of all the N objects . This manner extends networks designed for single-object VOS into multi-object applications , so there is no need to adapt the network for different object numbers . Although the above post-ensemble manner is prevalent and straightforward in the VOS field , processing multiple objects separately yet in parallel requires multiple times the amount of GPU memory and computation for matching a single object and decoding the segmentation . This problem restricts the training and application of VOS under multi-object scenarios when computing resources are limited . To make the multi-object training and inference as efficient as single-object ones , an expected solution should be capable of associating and decoding multiple objects uniformly instead of individually . To achieve such an objective , we propose an identification mechanism to embed the masks of any number ( required to be smaller than a pre-defined large number ) of targets into the same high-dimensional space . Based on the identification mechanism , a novel and efficient framework , i.e. , Associating Objects with Transformers ( AOT ) , is designed for propagating all the object embeddings uniformly and hierarchically , from memory frames to the current frame . As shown in Fig . 1b , our AOT associates and segments multiple objects within an end-to-end framework . For the first time , processing multiple objects can be as efficient as processing a single object ( Fig . 1c ) . Compared to previous methods , our training under multi-object scenarios is also more efficient since AOT can associate multiple object regions and learn contrastive feature embeddings among them uniformly .

Summary: The paper tackles the problem of video object segmentation. Firstly, the authors propose an identification mechanism which allows jointly segmenting multiple objects in a video with constant computational complexity w.r.t. the number of objects. This is achieved by mapping the masks of different objects to a common embedding space, which is then decoded by a single decoder. Secondly, the authors introduce a transformer based architecture with both long term (old frames) and short term (recent frames) matching to obtain accurate segmentation. The proposed approach obtains state-of-the-art results on both YouTubeVOS (83.7 $\mathcal{J}$&$\mathcal{F}$ on val2018) and DAVIS (83.0% $\mathcal{J}$&$\mathcal{F}$ on DAVIS 2017) benchmarks, while being computationally efficient.