AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation

Authors: Zili Wang, Qi Yang, LinSu Shi, JiaZhong Yu, Qinghua Liang, Fei Li, Shiming Xiang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Audio-Visual Segmentation, real-time model, efficient architecture

Abstract:
Recently, Transformer-based models have performed remarkably well in audio-visual segmentation (AVS) tasks.
  However, previous methods exhibit abnormal behavior and unsatisfactory results when using cross-attention.
  By analyzing attention maps, we identify two primary challenges in existing AVS models: 1) attention dissipation, caused by anomalous attention weights after Softmax over limited frames, and 2) narrow attention patterns in early decoder stages lead to inefficient utilization of attention mechanism.
  In this paper, we introduce AVESFormer, the first real-time audio-visual segmentation transformer that simultaneously achieves fast, efficient, and lightweight.
  Our model proposes an efficient, prompt query generator to rectify cross-attention behavior. 
  Moreover, we propose an early focus (ELF) decoder, which enhances efficiency by incorporating convolution operations tailored for local feature extraction, thus reducing computational overhead.
  Extensive experiments demonstrate that AVESFormer effectively mitigates cross-attention issues, substantially improves attention utilization, and outperforms the previous state-of-the-art, achieving a superior trade-off between performance and speed.
  The code can be found in the supplementary material.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729484495958

thread_type:
official_review

<thread_object>
creator: reviewer_jhmt

<content>
summary:
The authors proposed AVESFormer, an efficient framework for audio-visual segmentation tasks. By solving two issues of using cross-attention as attention dissipation and narrow attention patterns, they can effectively improve the utilization of cross-attention between audio and visual. AVESFormer improves attention utilization, improves the performance as well as reduces the running time.

strengths:
- The paper is well-written and easy to follow.

- The authors offer comprehensive ablation studies, which aid in comprehending the various design decisions involved in solving cross-attention issues.

- The proposed method shows a strong performance improvement as well as a reduction in the inference time.

- Sufficient comparison with SOTA methods

weaknesses:
- Some typo: Table 3: lartency -> latency

- Regarding the PQG, what is the result if we set the number of queries to the number of objects in the category?

- How much of the flops and memory that AVESFormer save from replacing the early attention with convolution?

- Does the number of stages in the decoder affect the performance?

- Missing references [1], [2], and [3] in Table 1

[1] CVPR2024: Audio-Visual Segmentation via Unlabeled Frame Exploitation 

[2] Referred by multi-modality: A unified temporal transformer for video object segmentation

[3] Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation

questions:
see Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730360331249

thread_type:
official_review

<thread_object>
creator: reviewer_g4ay

<content>
summary:
The paper presents AVESFormer, a transformer-based model for real-time audio-visual segmentation AVSAVS. It tackles two main issues in existing AVS models: attention dissipation and limited attention patterns in early decoders. AVESFormer uses a Prompt Query Generator PQGPQG to improve cross-attention and an Early Focus ELFELF decoder that integrates convolutional operations for efficient local feature extraction, reducing computational costs. Experiments show that AVESFormer addresses cross-attention problems, enhances attention utilization, and outperforms previous state-of-the-art models in both performance and speed. Additionally, the paper analyzes attention dissipation and the shortcomings of standard transformer decoders in real-time AVS.

strengths:
A key innovation is the Prompt Query Generator PQGPQG, which corrects cross-attention behavior and mitigates attention dissipation, enhancing segmentation accuracy and efficiency. Additionally, the Early Focus ELFELF decoder incorporates convolutional operations for local feature extraction, reducing computational demands by replacing attention operations in early transformer stages. The model achieves state-of-the-art performance, surpassing previous models on metrics like the Jaccard index and F-score across datasets such as S4, MS3, and AVSS.

weaknesses:
One concern is the significant parameter count of the audio backbone, Vggish, which limits deployment on mobile devices, suggesting future work could optimize this component. Additionally, the model currently ignores temporal information from multiple frames, and incorporating this data could enhance its ability to track moving objects. There is also a need to test the model's generalization to unseen datasets beyond AVSBench to evaluate its robustness in diverse scenarios. While qualitative analyses of attention maps are presented, a quantitative investigation could offer deeper insights into the model's attention mechanisms. Furthermore, comparing AVESFormer with non-transformer models would provide a broader context for its performance. The paper could also benefit from a more detailed discussion on the impacts of model complexity on real-time performance and ethical considerations regarding its application.

questions:
The authors should consider optimizing the audio backbone, Vggish, which currently comprises about 60% of the model parameters, to enable better deployment on mobile devices. Next, the authors are encouraged to explore how incorporating temporal information might influence model performance and inference speed. Additionally, evaluating AVESFormer on a broader range of datasets beyond AVSBench would provide insights into its generalization capabilities. The paper currently lacks a detailed quantitative analysis of attention maps, so including metrics to evaluate the focus and spread of attention could enhance understanding of the model's behavior.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730481373815

thread_type:
official_review

<thread_object>
creator: reviewer_m7g3

<content>
summary:
The paper introduces AVESFormer, a transformer model designed for real-time audio-visual segmentation (AVS). It addresses attention dissipation and narrow attention patterns in AVS by implementing a Prompt Query Generator (PQG) and Early Focus (ELF) Decoder. These components aim to improve cross-attention efficiency and reduce computational costs. AVESFormer shows competitive performance on AVSBench with noted speed improvements.

strengths:
1. The paper identifies critical issues in existing audio-visual segmentation (AVS) models, such as attention dissipation and narrow attention patterns, which are valid challenges in the field.

2. The proposed AVESFormer model aims to address efficiency in real-time AVS tasks, which could be valuable for applications needing low-latency responses.

3. The model demonstrates competitive performance on standard benchmarks with a noted improvement in latency, supporting claims of efficiency gains.

weaknesses:
While the issues identified in attention mechanisms are relevant, the approach—specifically the Prompt Query Generator and Early Focus Decoder—largely leverages incremental modifications rather than substantial innovations. Many elements, such as convolutional layers for early-stage feature extraction, are already established techniques in efficient transformers.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730485333328

thread_type:
official_review

<thread_object>
creator: reviewer_6kjk

<content>
summary:
This paper finds two primary challenges in existing audio-visual segmentation models, namely attention dissipation caused by anomalous attention weights after Softmax over limited frames, and narrow attention patterns in shallow decoder stages leading to inefficient utilization of attention resources. An AVESFormer is then presented as the first real-time audio-visual efficient segmentation transformer. Particularly, the proposed method leverages a prompt query generator to rectify cross-attention behavior, and an early focus decoder to enhance efficiency. Extensive experiments demonstrate superiority of AVESFormer in mitigating cross-attention issues and achieving a trade-off between model effectiveness and efficiency.

strengths:
1. Solving AVS via attention analysis is reasonable given the multimodal nature of AVS.
2. The attention dissipation issue seems interesting given the current QKV setting.
3. The experiments are comprehensive.

weaknesses:
1. The two observations which motivate the proposed method are not clear explained (see details in the Questions section).
2. The setting of the paper for real time AVS should also be verified, especially it should be compared to naive settings, e.g. defining the rest of the frames as an empty token. Most importantly, as temporal information is totally discarded, more analysis is needed on how the proposed method achieve smooth real time AVS.
3. Comparison with other attention based techniques is needed (Ref 1 in the Questions section.)

questions:
1. Line 53-76 tries to explain the issues of exiting AVS method. Although it's reasonable that effective multimodal fusion should be critical for AVS, it's not clear why this claim hold: "attention variants generally do not exhibit the same expressive capacity as the default mechanism" (line 75). Further explanation is needed.
2. In Fig. 2, the paper illustrates of attention dissipation. How the audio token is generated? Is this only a one case issue or does it happen all the time in all types of cross attention based AVS? More details are needed to verify the existence of the attention dissipation issue, especially when audio is defined as both key and value in the current cross attention setting. Moreover, where does the cross attention AVS model in equation 1 come from? Citation(s) are needed if it comes from existing literatures, or explanations are needed to explain why QKV is defied in the current way (compared to Ref1, Ref2, Ref3).
3. "Attention maps at early decoder stages tends to capture short-term local correlation features, leading to undesired low utilization of attention" (line 89-91) is used to explain the "narrow attention" issue, which is also not very clear. Why call it "narrow"?
4. In line 123, it's not clear why the existing AVS methods fail to work on the "single frame image + audio" setting.
5. what is the audio backbone in line 184?
6. F_audio=k=v, not clear (line 194). Also, audio feature dimension is D in line 185 and c in line191. It's better to use consistent symbols.
7. The prompt query generator (sec. 4.1) augments the audio feature. How does it bring extra useful information given the augmentation is based on the raw audio feature?
8. The early focus decoder is used to solve the "narrow attention" issue. However, it's not clear how eq. 6 and eq. 7 combined can solve the narrow attention issue.
9. In Table 1, why Ref1 is not compared?
10. In Table 4 and Table 5, it seems the proposed two strategies work better for MS3 dataset, explanations are needed.

Ref1: Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation. CVPR 2024

Ref2: AVSegFormer: Audio-Visual Segmentation with Transformer. AAAI 2024

Ref3: Audio–visual segmentation. ECCV 2022
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work received three negative ratings (borderline reject) and one positive rating (borderline accept). The reviewers rate negative scores due to the limited novelty, lack of high-level explanation of the motivation and contradictory experimental results raised by reviewer jhmt. The authors also did not further address the comments of the reviewer jhmt. AC checked the comments of the reviewers and thus made the final decision.