Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work proposes a space-time correspondence network which improves performance and speed over the standard STM network by finding pixel-level correspondences in individual frames rather than frames+segmentations. Furthermore, this work shows that the dot-product, which is commonly used in STM is suboptimal and they propose an efficient L2 similarity computation which leads to improved correspondences throughout a video. It is evaluated on two standard VOS benchmarks - Youtube-VOS and DAVIS-2017 - and there are extensive analyses and ablations which give insights into the proposed method.
main_review: Overall, the paper is well written and there are no major issues. The proposed approach seems novel, and achieves strong VOS performance when compared with previous SOTA methods. Furthermore, the major claims made by the work are supported by experiments or ablations. There are some questions/issues (see below), so I suggest this work is *marginally above the acceptance threshold* until these are addressed.

## Questions/Issues

1. A recent paper [1] has a similar network architecture, where the correspondence/affinity between pixels is computed separate from the value/query. Although the specific problems on which the networks are applied are different - [1] attempts to perform VOS while training in an unsupervised manner, whereas this work perform the traditional semi-supervised setting - can the authors comment  on the differences between the two approaches? Some discussion differentiating the two methods/architectures would help highlight the novelty of this approach.

2. Table 2 shows that L2 leads to better results on STCN than dot-product and cosine similarity. Is this, however, specific to the STCN, or would replacing the dot-product in another network, e.g. STM, lead to a similar performance improvement?

3. Table 1 uses dot-product for STM and L2 Similarity for the STCN. Would the drop in performance when using the last frames (results in col 3 and col 4) be found when using the dot-product with STCN? Is this specific to the STCN or the L2 Similarity? An ablation exploring this would be beneficial to better understand the strengths/weaknesses of the proposed approach.

## Minor Questions/Comments

1. Is the use of Resnet+CBAM layers to combine query+values (line 117) necessary for strong performance?

2. How would the light value encoder differ from resizing the binary segmentation mask and using the resized mask as $v^M$?

3. It would be useful to see scores when not using static image pretraining, so that the method could be compared with other methods that do not use such data (i.e. only training on Youtube-VOS).

[1] Lai, Z., Lu, E., & Xie, W. (2020). MAST: A memory-augmented self-supervised tracker. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6479-6488).
limitations_and_societal_impact: The authors include a section on the limitations of the approach as well as broader impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents the Space-Time Correspondence Network (STCN) to tackle the semi-supervised video object segmentation. STCN improves previous STM by building affinity based on RGB-only input and reused key features. Moreover, the author further investigated the computation of similarity and find that negative squared Euclidean distance leads to better performance. Experiments on multiple datasets are conducted to demonstrate the effectiveness of the proposed method.
main_review: Pros:
+ This paper is clear and easy to read.
+ The proposed method achieves state-of-the-art accuracy and high efficiency.
+ The proposed STCN is reasonable and effective.
+ The investigation about the similarity computation is meaningful.

Cons:
- There may need more intuition for why L2 similarity being better than dot-product similarity. Even though dot-product provides less smooth similarity, it can be relived by the Softmax function. Sec3.1 tries to provide explanations for this, yet the visualization is based on toy examples. It would be better to visualize with real examples.

- In Fig.4, the numbers of total nodes for Dot product and L2 similarity are different. Shouldn't they be the same?

- Are the results in Tab.1 both based on L2 similarity? If not, this should be clarified. Also, it would be better to report performance for STM with L2 similarity.

- I noticed that all the experiments are conducted on the validation sets. It would be better to incorporate results on testing sets into the paper.  From the Tab-1 supplementary, I noticed that the ''ours''  model actually performs worse than the previous method on the test-dev of DAVIS17. This is contradictory to the performance on the validation set.  Any explanation?

- Some related works for fast VOS are missing [a,b].

[a] Fast video object segmentation via dynamic targeting network, cvpr19

[b] Motion-guided cascaded refinement network for video object segmentation,cvpr18


In general, I like this work for its motivation and effective/efficient performance. I would like to see the authors' responses.
limitations_and_societal_impact: Don't find limitations and potential negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper simplifies space-time correspondence with shared Siamese key encoders for all raw frames and gets rid of the unnecessary re-encoding procedure for every object in the same frame. The authors further discuss the similarity functions used in affinity computation and provide comprehensive experiments on memory composition and running time.
main_review: ## Pros:
- I enjoy reading the manuscript, which is well-structured. The ideas are well-presented and easy to follow. The proposed pipeline is clear and reasonable albeit with some minor contradictions in description. 
- I appreciate the analysis and experiments on similarity functions and running time, which help a lot for better understanding the proposed method and potential issues in previous works.
- The authors promise to release the code, which would be truly valuable.

## Cons:
- About the counterintuitive result on Memory management, where the short-term consistency seems to be harmful to STCN when applying L2 similarity, I wonder what would happen if cosine similarity and dot product similarity are adopted instead of L2. Additionally, I recommend an ablation on merely maintaining the last frame in memory and provide corresponding qualitative results to prove the drifting assumption in Ln152. 
- I appreciate the assumption that ‘every pixel counts’ and consent the extra robustness from additional ‘meaningful’ memory nodes. But I think the proposed method does not explicitly explore the correspondence between background pixels, as [9] did, but just spread the ‘attention’ to more foreground pixels, and thus introduces the mismatching issues as described in the limitation section, which makes the discussion in Ln206-215 not that convincing where the authors claim they could benefit from distinguishing the backgrounds. The authors also analyze the different undergoing situations between STCN and other NLP methods or video classification methods, but I could not get the point and the discussion seems a little bit redundant. I would recommend a refinement or some detailed explanation.
+ [9]: Collaborative video object segmentation by foreground-background integration. In ECCV2020.

limitations_and_societal_impact: Yes, the authors have addressed the limitations and potential negative societal impact of their work .
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose STCN for video object segmentation. Compared to STM, the dot product similarity is replaced with the L2 similarity. Moreover, the key/value encoders are separated and thus the features can be reused. Though simple, these small changes make STCN archive the STOA results and faster inference speed. Moreover, it gives a closer look at the similarity measure.
main_review: 1. The paper is clearly written.
2. Section 4 gives a good analysis of different similarity measures.
3. The proposed STCN archives the STOA results on mainstream benchmarks.
limitations_and_societal_impact: 1. Though simple, the technical contributions are limited.
2. According to Eq.5, the L2 similarity can be regarded as the dot product similarity (K_i^M \dot K_j^Q) with L2 normalization (K_i^M * K_i^M; K_j^Q * K_j^Q). However, the segmentation accuracy is much higher than the dot product. Why?
review_title: null

Meta-review: This paper has four positive reviews (6,6,6,7). While the scores are close to the borderline, the reviewers are consistent, and appreciate the same points in the paper. It should be accepted to NeurIPS.


Decision: Accept (Poster)