Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This work proposes two novel techniques, an identity matching and a long-short term transformer block. Their first contribution, identity matching, enables them to handle multiple object scenarios as fast as single object ones and process all objects simultaneously. Their second contribution, enables them to handle occlusion better while keeping boundaries accurate. Overall they achieve sota results on 3 main semi supervised video object segmentation tasks.
main_review: This work proposes to initialize M identity vectors (M is set to 10 in practice which is the maximum number of objects). Then using a random permutation matrix N  (number of present objects) of them are assigned to the current video. They augment the value vector of the memory with the identity embedding by adding them together. With this strategy they are able to track multiple objects simultaneously with a matrix multiplication and addition. This is a novel and interesting contribution which enables them to get a very high FPS while achieving sota. In extreme with their tiny variant they are able to achieve about 40 FPS processing which is impressive given their accuracy.

Their other main contribution is having two attention heads, one of them looks at the local spatial positions in the previous frame (7x7 very local!) and the other one looks at the whole image but from previous frames. The long attention either only looks at the first frame or at every 5 frame in the memory constructing their best large model.

Experimentally they show sota results on both DAVIS and YoutubeVOS which are the two main semi supervised benchmarks and they outperform all the previous works significantly. Impressively while maintaining a better FPS too. 

Both of their contributions are significant in terms of performance while not adding too much complexity or computational overhead. 

They also perform an ablation study on all the interesting factors of their model, such as number of LSTT blocks, number of short term history frames and window size. 

Their performance drops with more identity vectors than necessary (such as 20 or 30). An interesting visualization in appendix shows that the identity vectors start getting correlated. Their hypothesis is it's harder to optimize for 30 disjoint vectors in a 250D space. It would be interesting if this can be resolved with a proper regularizer on the identity vectors such as orthogonality enforcer. 
limitations_and_societal_impact: Authors have noted that this method fails on rare/small objects. They also claim since this is a standard benchmark their work does not deviate or add any negative impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This submission addresses the task of semi-supervised video object segmentation (VOS), in which given a set of object mask over a frame, these masks must be propagated to the rest of the frames of the sequence.

The main contribution is proposing a method to treat the multiple objects to be segmented in a single pass, by defining an Identification Embedding that encodes all masks simultaneously. Previous works in VOS used independent embeddings for each object, which required multiple decoding passes, one for each object. With the proposed Identification Embedding and Identification Decoding mechanisms which are trained end-to-end.

The submission also introduced a  Long Short-Term Transformer (LSTT) combines a long-term attention related to the first frame, where the object segments are provided as ground truth, together with a short -term attention related to nearby frames.

The experiments on YouTube-VOS and DAVIS set new state of the art results in terms of accuracy and speed at inference time, especially in the case of multiple object in the video scene.
main_review: ## Update on August 27, 2021
After reading the answers of the authors to my concerns, I upgraded my score from "7: Good paper, accept" to "8: Top 50% of accepted NeurIPS papers, clear accept".

## Strengths

S1 Using a single embedding for all the objects allows decoding all the objects in the scene in a single pass, while the related work would require a different decoding step for each segmented object, in such a way that the computational requires will grow with the amount of objects.

S2 The mechanism to develop a trainable scheme for identification embedding and decoding is novel and sounding.

S3 The hierarchical attention mechanism, especially in the case of short-term attention, is convincing in the sense that it attends to nearby temporal frames and nearby spatial pixels frames, in addition to the further frames, attended at a global spatial scale.

S4 The experimental results are careful designed to provide variations of AOT that are adopt the design choice of other works in the state of the art.

S5 The presented results are convincingly faster than the state of the art, together with an incremental gain in accuracy.

S6 The ablation study proves the gains of the main contributions of the paper.

S7 The source code is promised upon acceptance.



## Weaknesses

W1 The concept of multiple object association is not also tested in the very similar problem of multiple object tracking, which should benefit from the same reasoning.

W2 The manuscript uses the adjective "uniformly" (l.35, l.41...) as a contrast to the "independently" (l.32), but it is not clear what "uniformly" refers to in this context. It seems to indicate "simultaneously" because of the reference to the computational requirements and the fact that a single embedding encodes all the objects. The use of the "uniform" term should be better explained or reconsidered.

W3 Is it not clear whether the backbone used by the other works is comparable to the one used in AOT (MobileNet). The authors should discuss whether the gains may come from the different backbones and include it in the table with comparative results.

W4 A discussion regarding the memory footprint of the model and its required memory at inference time is missing. This would also allow a better comparison with the related work, beyond the accuracy and inference times reported.

W5 While the reported frames per second are better than the related work, it is not clear if the results are comparable in terms of hardware. How were the FPS results obtained ? Is it possible that the speed gains are simply due to the use of the distributed 4 Tesla V100 GPU mentioned in l.256 ?

W6 A discussion to this missing related work is missing:

Zhang, D., Zhang, H., Tang, J., Wang, M., Hua, X., & Sun, Q. Feature pyramid transformer. ECCV 2020.



## QUESTIONS

Q1 In Table 3a, did you experiment also with M<10 ? While M=10 is well grounded based on the related work, it is actually one more hyperparameter whose modification may provide some additional gain in performance.

Q2 Similarly to Q1, did you try with larger window sizes ? When reporting results, it is somehow surprising that the next step after the best results ($\lambda=9$) is not included.

## OBSERVATIONS

O1 While CVPR 2021 was celebrated after the NeurIPS 2021 submission deadline, the authors may  **optionally** also consider including some CVPR 2021 papers in the discussion and results on video object segmentation.

Duke, B., Ahmed, A., Wolf, C., Aarabi, P., & Taylor, G. W. [SSTVO: Sparse spatiotemporal transformers for video object segmentation](https://openaccess.thecvf.com/content/CVPR2021/html/Duke_SSTVOS_Sparse_Spatiotemporal_Transformers_for_Video_Object_Segmentation_CVPR_2021_paper.html). CVPR 2021. [[code]](https://github.com/dukebw/SSTVOS)

Wang, Y., Xu, Z., Wang, X., Shen, C., Cheng, B., Shen, H., & Xia, H. [End-to-end video instance segmentation with transformers]((https://openaccess.thecvf.com/content/CVPR2021/html/Wang_End-to-End_Video_Instance_Segmentation_With_Transformers_CVPR_2021_paper.html)). CVPR 2021. [[code]](https://github.com/Epiphqny/VisTR)

Vaswani, Ashish, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. ["Scaling Local Self-Attention For Parameter Efficient Visual Backbones."](https://openaccess.thecvf.com/content/CVPR2021/html/Vaswani_Scaling_Local_Self-Attention_for_Parameter_Efficient_Visual_Backbones_CVPR_2021_paper.html) CVPR 2021.

O2 Similarly to O1, the authors may  **optionally** consider citing and discussing the arXivs introducing SwinTransformers and its application to video:

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., ... & Guo, B. (2021). [Swin transformer: Hierarchical vision transformer using shifted windows](https://arxiv.org/abs/2103.14030). arXiv preprint arXiv:2103.14030.

Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2021). [Video Swin Transformer](https://arxiv.org/abs/2106.13230). arXiv preprint arXiv:2106.13230.

O4 l.30: "Above methods" is mentioned twice, it could be rephrased.



limitations_and_societal_impact: No, the authors did not include any reference.

The authors may refer to the privacy concerns that video surveillance systems pose, as they may benefit from the presented contributions.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors study better and efficient way to associate objects in the multi-object video object segmentation. In contrast with previous methods that segment each target object separately, the proposed method matches and decodes multiple objects uniformly and efficiently. For this, an identity bank and a Long Short-Term Transformer is designed. The proposed method AOT become the new state-of-the-art on Youtube-VOS and DAVIS benchmarks.
main_review: Strength:
- The proposed method present a new paradigm for the semi-supervised VOS deviant from predominant STM-based methods. The proposed method, by design, is efficient when handling video with a large number of objects. e.g., it has constant computational complexity regardless of the number of objects. 
- The authors proposed several novel technical components. 1) a new identification mechanism is proposed. An identity bank is employed to assign identities into feature map. With this mechanism, pixel matching become identity-agnostic thus matching results can be shared across targets. 2) a LSTT block is proposed. For my understanding, it is an improved version of memory reading block in STM with transformer style structure and separation of long and short term attention. 
- The proposed methods show the state-of-the-art performance on three popular benchmarks. 
- The paper includes an extensive ablation analysis. 

Weakness:
- I have some concerns on identification mechanism based on identity bank. 1) Scalability. As shown in Table 3 (a), the performance is getting worse with growth of the maximum number of identities. It means that the capacity should be preset to some small number (e.g., 10). In real-world scenario, we can have more than 10 objects and most of the time we don't know how many objects we will need to handle in the future. Have the authors thought about how to scale up without compromising performance? 2) Randomness. Identities are randomly assigned one embedding from the identity bank. How the results are robust against this randomness? It would be undesirable for the result to change with each inference. It would be great to have some analysis on this aspect.

Overall Evaluation:
- The paper present a novel approach for multi-object video object segmentation and the proposed method outperfrom previous state-of-the-arts on several benchmarks.
- Now, I would recommend to accept this paper. I will finalize the score after seeing how authors address my concerns in Weakness.  

limitations_and_societal_impact: While future works are discussed in Supplementary Materials, I encourage the authors to include more discussions on limitations and societal impacts.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper tackles the problem of video object segmentation. Firstly, the authors propose an identification mechanism which allows jointly segmenting multiple objects in a video with constant computational complexity w.r.t. the number of objects. This is achieved by mapping the masks of different objects to a common embedding space, which is then decoded by a single decoder. Secondly, the authors introduce a transformer based architecture with both long term (old frames) and short term (recent frames) matching to obtain accurate segmentation. The proposed approach obtains state-of-the-art results on both YouTubeVOS (83.7 $\mathcal{J}$&$\mathcal{F}$ on val2018) and DAVIS (83.0% $\mathcal{J}$&$\mathcal{F}$ on DAVIS 2017) benchmarks, while being computationally efficient.

main_review: 
### **Strengths** 
S1) The proposed identification mechanism to handle multiple objects is novel and interesting. Compared to existing approaches, the proposed module allows segmenting multiple objects simultaneously without incurring additional computational costs compared to segmenting a single object. 

S2) The second contribution of long short-term transformer based VOS architecture is also novel, interesting, and shown to be effective via appropriate ablation studies.

S3) The method obtains impressive results on standard VOS benchmarks YouTubeVOS and DAVIS, outperforming previous state-of-the-art by a significant margin.

S4) The paper is well-written and easy to read and understand.


### **Weaknesses**

W1) I do not think the use of term 'hierarchical matching' (used throughout the paper) is strictly correct. If I understand correctly, the authors stack multiple transformer blocks together to process the input frame, as commonly done in transformer based architectures. Referring to this as hierarchical matching seems confusing and slightly misleading. I would suggest omitting the use of 'hierarchical'.

W2) How are the FPS for other methods obtained in Table 1. Did the authors compute them using identical hardware as their method, or are they directly obtained from the corresponding papers? This should be clearly stated. Since the implementation of SOTA methods such as STM, CFBI, LWL are available, I would recommend running these methods using identical hardware in order to obtain a fairer speed comparison. 


While not strictly necessary, I believe the paper will greatly benefit from additional experimental analysis of the proposed method, as listed below.

C1) The proposed approach obtains improved results compared to previous methods. However, it is not clear whether the improvement is mainly due to the long short-term transformer based architecture, or whether the joint segmentation of multiple targets enabled by the identification mechanism also leads to improved segmentation accuracy, in addition to computational efficiency. Thus, it would be interesting to evaluate a variant of the method which independently segments each object using the long short-term transformer, and then merges the masks using a soft-aggregation as done in STM.

C2) What is the impact of learning the identification vectors? What would be the performance drop if random identification vectors are used instead of the learned ones?

C3) The authors add the identification embedding of the target mask to the value computed using the image features. Did the authors experiment any other architectures to integrate the identity embedding into the value? For instance, is it possible to concatenate the identity embedding with the initial value and then pass it through a small network? 

C4) What is the performance if only the short term (local) attention is used?

limitations_and_societal_impact: The authors discuss the failure cases of the proposed method.
review_title: null

Meta-review: The rebuttal addressed all of the reviewers concerns, and all reviewers recommend acceptance. The AC agrees with this recommendation.
Decision: Accept (Poster)