Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed an improved AOT method called DeAOT. The main idea of DeAOT is to separate representation learning from identity information. This means using two GPMs instead of the original LSTT in AOT. The Gated Propagation Module(GPM) is faster than LSTT and achieves better performance.
strengths_and_weaknesses: Strengths
- Great performance on several VOS/VOT benchmarks.
- Intuitive and effective design about disentangle the representation learning and id information.
- GPM balances the inference speed and performance.

Weaknesses
- Lack of ablation studies on GPM.
questions: Compared to AOT, the difference between DeAOT seems to be only in the design of Propagation Module, and specifically the difference between LSTT and GPM. Therefore, it is more important to explain the more subtle differences between LSTT and GPM and to explore what really works in the design of the GPM. For example:
- the effect of removing FFN, or whether introducing FFN would result in better speed-performance tradeoff
- the impact of swapping Self-Attn(Prop) and Long/Short Term Attn(Prop)
- the difference between sharing an attention map or not

From the ablation information provided in the current experiments(Table 3), we could find that GPM still outperforms LSTT even without disentangle representation learning and id information, so it is more important to analyze the role of GPM clearly. From another perspective, GPM is the core of this paper and it would be detrimental to the quality of this paper if the mechanism of GPM could not be well explained.
limitations: No but not need.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper tackles the problem of video object segmentation by extending the Associating Objects with Transformers (AOT) approach. AOT uses multiple Transformer blocks to gradually integrate target information from the previous frame and masks into the current frame features. The encoded features are then decoder to obtain the target mask. The deeper Transformer blocks in AOT operate on features which are target-specific, i.e. which contain encoding of the target mask information. The authors claim that using such features to compute the attention maps inside Transformer harms performance since they contain less target agnostic visual information. To address this, the authors propose to decouple the propagation of target-specific and target-agnostic features using two parallel transformer branches. Both the branches share attention maps, computed using the target-agnostic features. Additionally, the authors propose an efficient Gated Propagation Module (GPM) instead of Long-Short term Transformer (LSTT) used in AOT to propagate the features. GPM uses single-head attention, as opposed to multi-headed attention in LSTT. Additionally it uses a gating function to modulate the propagated features. The use of GPM compensates for the extra computation incurred due to use of two transformer branches. Overall, the proposed methods consistently outperforms baseline AOT on 4 benchmarks, across 3 different backbone networks (+1.6% on YouTube-VOS 2019), obtaining state-of-the-art results on these benchmarks. The proposed method is also faster than AOT.
strengths_and_weaknesses: ## Strengths

**S1**: The proposed approach obtains state-of-the-art results on YouTube-VOS, DAVIS, and VOT2020 benchmarks while operating at impressive speed (>10 FPS). 

**S2**: The proposed approach generally outperforms the baseline AOT across YouTube-VOS, DAVIS, and VOT2020 benchmarks, when using 5 different backbones/configurations. Furthermore the proposed approach is faster then AOT.

**S3**: The paper is well written, and the authors clearly introduce the issue with the baseline AOT. This claim is further experimentally validated (Figure 2, Table 3a). The proposed approach is shown to mitigate this issue.

**S4**: The proposed Gated Propagation Module (GPM) is interesting and seems to provide a clear improvement over LSTT block, while being faster (Table 3a, 3b).

**S5**: The authors perform a thorough analysis of the proposed approach, showing the impact of each component.

## Weaknesses
I do not have any major issues with the paper. Some minor comments to be addresses

**W1**:  It wasn't clear to me how the gating embedding U is obtained. The authors should clarify this.

**W2**: It would be interesting to see the performance of LSTT with decoupling features (i.e. an extra entry in Table 3a for completeness).

**W3**: Are both the encoded visual features as well as the ID features used to obtain the target mask, or only the ID features? 

**W4**: The current approach performs a strict separation of object-specific and object-agnostic features, and mainly uses the object-agnostic features to compute the attention masks. However I imagine that certain object-specific information could also be helpful when computing attention masks. Thus would it make sense to instead have two branches, one which is used to compute the attention masks, and the second used to obtain the final mask. Both of these branch can take both the image and object masks as input, and determine whether to use the ID information or not during the training process.
questions: Please see weaknesses.
limitations: The authors discuss the limitations, albeit in the supplementary material.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a hierarchical propagation module based on transformer which can keep the object-agnostic information in the videos. And it also proposes Gated Propagation Module for enhancing model efficiency. This method achieves high accuracy and efficiency.
strengths_and_weaknesses: strengths:
The author alleviates the missing of object-agnostic information in deeper transformer modules by introducing hierarchical dual-branch propagation module. The GP function further improves the model efficiency while keeping high efficiency. This method has a certain originality.

Weakness:
The object-specific information can lead to the performance degradation of memory-based methods such as STM. STCN takes two object-agnostic features as Q and K to perform attention. This method extends this operation to a hierarchical propagation module and it is a continuation of this idea.
questions: 1.The performance of only containing ID branch is lacks in table 3 (c). For experimental integrity, the contribution of every module should be shown.
2.The author should list the module parameter and inference speed in table 3. The author needs to further demonstrate and verify whether the improvement in GPM is brought by adding more parameters.
3.The author needs to further analyze the effectiveness of modules in GP function. I want to know if gate (δ(U)) and DW-conv actually play an important role in this module
4.The authors mention that object-agnostic information fades in the video and use a single-backbone VOS model for hierarchical dual-branch propagation module validation. But the memory-based model such as STM takes a double backbone. Does this problem exist in the double backbone structure, and can the hierarchical dual-branch propagation module also improve the defects in the two-stage model?
5.I am interested in how the long-term and short-term memory images are divided and what the theoretical and experimental basis for this division.
limitations: In addition to the ablation experimental results and visualization results, the author also needs to analyze the effectiveness of the GPM and GP function from the theoretical basis and the feature visualization results.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a feature decoupling mechanism into the VOS task to solve the problem of visual information forgetting in the AOT method. Specifically, DeAOT decouples the hierarchical propagation of object-agnostic and object-specific embeddings by handling them in two independent branches. In addition, Gated Propagation Module with single-head attention is used to reduce the computational complexity. Experimental results on YouTube-VOS, DAVIS17, DAVIS16, and VOT20 demonstrate the performance gain of the proposed method.
strengths_and_weaknesses: The in-depth analysis about the object-agnostic and object-specific gives a novel view for VOS task. In addition, two independent branches of the shared attention map do not significantly increase the model parameters. 

The proposed gate attention reduces multiple-head attention to single-head attention, greatly reducing computational complexity while maintaining accuracy.

The paper is well-written and easy to read and understand.

questions: The incremental contributions. The main innovation is to expand the AOT model through a decoupling mechanism and gate attention. However, it is a common way to improve features' expression ability through decoupling, and gate attention is proposed in previous papers [23][27]. To prove the generalization ability of decouping mechnism, the authors should also combine it with other baseline method, such as STM.

Do this paper's main innovations (i.e., decoupling features) really play an essential role? In Table 3 (a), when the decoupling module is removed, the performance drops from 82.5 to 81.5, while when the GPM module is replaced, the performance drops more significant (82.5-> 80.3). This means GPM plays a more important role than decoupling feature.

For the head numbers in ablation experiment, the authors need to explain why the results of head numbers N_h=1 and N_h=8 are similar. Moreover, the performance of multiple-head is even inferior to that of single-head. This is conflict with the motivation of multiple-head mechanism. The authors may need to add other ablation results of N_h.

Typographical adjustments, Fig 4 goes too far from its text description.

 Minors: the abbreviation, i.e., AOT, should give the total words when first used. 
grammar errors ‘Fig. 4 give qualitative comparisons’

limitations: Yes
review_title: null

Meta-review: metareview: The paper obtains three accept and one borderline reject recommendations. Yet all reviewers pointed out that the paper has novelty and originality in the domain of video object segmentation, and also the method works quite well on the tested datasets. The reviewer recommending rejection does not comment at the post-rebuttal phase, and the AC has checked authors' response, and most concerns of the reviewer have been addressed - though the theoretical analysis can be a future work, the feature visualization still can be done in the camera ready version. Authors also need to carefully prepare the camera ready version, since all the reviewers indeed give valuable comments, which are helpful for the paper.
confidence: Certain
Decision: Accept