Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new method for referring video object segmentation (R-VOS). It points out an assumption that current R-VOS datasets make: that the object referred to in the text is present in the video. This assumption is not true in real-world text-video query applications. Current models rely on the assumption which results in false positives when the referred object is not in the video. The paper addresses this in two ways: by introducing a dataset $R^{2}$-YouTube-VOS and evaluation metric that measures the false positives made by a model, and by creating a method that enforces the semantic matching of visual and text features. The authors demonstrate that their method outperforms current methods on Ref-YouTube-VOS and Ref-DAVIS and has fewer false positives than previous methods on the proposed $R^{2}$-YouTube-VOS.
strengths_and_weaknesses: Strengths:

- The paper points out an important limitation of current R-VOS datasets and evaluation metrics and takes the proper steps to address it by creating the metric R and augmenting Ref-YouTube-VOS with negative video-text alignment pairs.
- The idea to enforce semantic consistency of the embedded textual space and generated embeddings is novel and interesting.
- The experiments are carefully designed. They clearly show that the model improves upon the state of the art. The ablation results show the benefit of the proposed relational cycle consistency losses.

Weaknesses:

- The paper is missing some details on the $R^{2}$-YouTube-VOS dataset including how many negative text-video pairs are created.
questions: Questions:
- In the sentence in lines 242-243 can you clarify what "constrain all negative text-video pair unrelated" means?

Suggestions:
- There are many works on vision-language representation learning and it would be useful to the reader if the authors could include more of this background in Section 2.
- Line 58 a better word instead of "medium" would be "proxy"
- It would be more clear to call $f_{m}$ an intermediate feature rather than a medium in line 112.
- Typo in line 109 "textural" -> "textual"
- Figure 2c caption is a run-on sentence and would sound better split into two.
- When discussing the window size in line 259 mention that Swin or VideoSwin is used as a backbone.
- Refer to Figure 4 somewhere in the text.
- Line 320 "temporal-consistent" -> "temporally-consistent"
- The discussion of quantitative results is currently limited. It would benefit from more details on how the proposed method differs from the ReferFormer architecture and why the ReferFormer is not able to learn the necessary information to avoid false positives.
limitations: - The paper does not include a section on limitations/negative societal impact.
- An important challenge for segmentation in videos is temporary object disappearance due to occlusion, which results in false positives on a per-frame level. Since the semantic alignment ground truth is per-video rather than per-frame the method would not able to address this challenge, but if the ground truth were collected per-frame then perhaps it would.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors introduce a "new" variant of referring video object segmentation. This variant considers the possiblity that the given referring expression does not correspond to any object in the given video. A method based on relational cycle consistency is introduced as well. The proposed method has an additional head that predicts whether there is an object that corresponds to the given expression or not. Experiments are mainly conducted on two standard benchmarks to validate the effectiveness of the proposed method.
strengths_and_weaknesses: Strengths:
(1) The paper is well written is easy to understand. Notations and symbols are well defined.
(2) The authors propose a method that can handle expression that does not correspond to any objects in the given video via a simple head, which is likely a simple fully connected layer.
(3) The proposed method is evaluated on two standard benchmarks and achieves good performance on both datasets.

Weakness:
(1) While existing methods do not explicitly consider the possiblilty of being given an expression that does not correspond to any objects in the video, it is likely that this can be easily handled by adding a background class to ReferFormer [1] and other methods.
(2) Researchers have already used cycle consistency (text reconstruction) to help referring expersion segmentation in images. To me, this work is mainly an extension of existing ideas to video.

[1] Language as Queries for Referring Video Object Segmentation, CVPR'22
questions: What is the range of A? When does the indicator function 1(A) equal to 1?
limitations: N.A.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces the Robust R-VOS task that accepts unpaired video and text as inputs. In addition, a pipeline is proposed to optimize the primary referring and dual expression reconstruction task. A relational cycle consistency constraint is proposed to enhance the semantic alignment between different modalities. The proposed approach achieves state-of-the-art results on three datasets.
strengths_and_weaknesses: Strengths:
1)This paper proposed the Robust R-Youtube-VOS dataset, is collected to measure the robustness of R-VOS models.
2)The relational cycle consistency is applied to preserve the structure of the feature space rather than point-wise consistency. Through the experience, the cycle consistency achieves better results than point-wise consistency.
3)The proposed approach achieves state-of-the-art on three datasets, and some ablation studies are conducted to validate the effectiveness of each component.
4)The paper is generally well written.  

Weaknesses:
1) My main concern is that the main novelty. In this paper, for negative videos
M_o is empty. For some R-VOS methods, they are able to align the semantic information of different modalities, and they also overcome the problem that unpaired video and text as inputs so that Robust R-VOS does not need to be studied separately.
2) This paper proposed a new task and proposed a new dataset. However, the paper makes the ablation study based on the Ref-DAVIS17, and Ref-Youtube-VOS.
3) This paper leverages text reconstruction to reconstruct the text feature to measure semantic consensus. It is commonly used for Video-Language tasks.

questions: 1 In formula 8, l(A ̂) is used to measure semantic consensus between Reconstructed text feature and text feature, it should be explained clearly.
2 It is not clearly stated in the article how to evaluate this model on the Robust R-VOS dataset. Is the model trained on the Ref-Youtube-VOS dataset and directly evaluated on the Robust R-VOS dataset?
3 The ablation experiments in this paper are performed on the Ref-Youtube-VOS dataset, which does not need to consider this problem that unpaired video and text as inputs, so different modules need be used separately to verify their effectiveness.
4 The authors should clearly introduce the source of the negative sample videos in the dataset.  If these videos are also from the Ref-Youtube-VOS dataset, each video in the entire dataset will correspond to a positive text message. In order to better verify the robustness of the model, video that do not correspond to text information in the entire dataset should be added.

limitations: Yes
review_title: null

Meta-review: metareview: This paper presents an approach for video object segmentation. The paper considers the possibility that an (object) expression may not correspond to any object in the given video. The approach is based on relational cycle consistency, which the reviewers find technically sound. The paper also has a dataset contribution.

After the rebuttal and discussions, the reviewers still maintained split ratings. While two reviewers find the paper has its merit, Reviewer Fhve is against the paper, pointing out his/her concern regarding the experiments' fairness. Specifically, the reviewer points out that the comparison against the other works not using negative samples is unfair, as the training of the proposed approach benefits from such negative samples.

Although we agree to the authors' explanation that the ability to explicitly consider negative samples for out-of-distribution discrimination is the strength and that it is an interesting technical aspect of the paper, the paper lacks sufficient experiments to support the argument. It would have been better if the authors provided explicit experiments comparing their approach against the baselines like ReferFormer by adding classification heads for the background with negative samples in a meaningful way, as the authors also suggested. Also, there is a bit of novelty concern shared by Fhve and Gh1L.

Considering these aspects, the ACs recommend the rejection of the paper.
confidence: Less certain
Decision: Reject