AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation

## Abstract

Abstract Recently, transformer-based models have demonstrated remarkable performance on audio-visual segmentation (AVS) tasks.
However, their expensive computational cost makes real-time inference impractical.
By characterizing attention maps of the network, we identify two key obstacles in AVS models: 1) attention dissipation, corresponding to the over-concentrated attention weights by Softmax within restricted frames, and 2) inefficient, burdensome transformer decoder, caused by narrow focus patterns in early stages.
In this paper, we introduce AVESFormer , the first real-time A udio- V isual E fficient S egmentation transformer that achieves fast, efficient and light-weight simultaneously.
Our model leverages an efficient prompt query generator to correct the behaviour of cross-attention.
Additionally, we propose ELF decoder to bring greater efficiency by facilitating convolutions suitable for local features to reduce computational burdens.
Extensive experiments demonstrate that our AVESFormer significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3 and 31.2% on AVSS, outperforming previous state-of-the-art and achieving an excellent trade-off between performance and speed.
Code can be found here .

## 1 Introduction

Audio-Visual Segmentation (AVS) [] have emerged as a novel multi-modality task that plays a crucial role in robot sensing, video surveillance and other scenarios. It aims to segment fine-grained pixel-level sounding objects with corresponding audio-visual modalities. However, existing AVS methods [, , , , , , , , , ] primarily focus on improving performance, often at a high cost of models size and computational overhead. Such heavy computational cost renders them unsuitable for applications with real-time requirements.

Recently, transformer-based AVS models have brought significant performance improvements with cross-attention and its variants serving as audio-visual fusion module [, , , , , , ]. Beginning with AVSBench [], temporal pixel-wise audio-visual interaction (TPAVI) [, ] is proposed to inject audio guidance from all video clips. However, such method is unnatural since the sound source may change during the clip.
AVSegFormer [] employs channel attention mixer (CHA) to guide visual channels with audio features. Nevertheless, channel attention may be dominated by visual features and surpass the audio representation [].
Contrastive Audio-Visual Pairing (CAVP) [] approximates Softmax function with Sigmoid, suggesting it could highlight critical regions. Nonetheless, approximated attention does not hold the same power as attention [].

Despite their strong performance, the application of AVS model on real-time field is still difficult because the computational overheads and model efficiency are often neglected. Our observation identifies two primary issues that prevent the AVS model from the real-time area: (1) Attention Dissipation, an issue not explored in previous studies, where cross-attention matrix vanishes during modality fusion process in existing methods [, , ], hindering them from distinguishing audio-visual corresponding region, as shown in Figure 1. (2) Inefficient Decoder tends to capture narrow local features at early stages with cross-attention, resulting in short-range pattern utilization, which is not a desired behaviour of attention. These inefficient operations not only fail to build long-range dependencies, but also constitute the bottleneck of inference runtime. As shown in Figure 3, the runtime proportion of the transformer, including the query generator, can exceed 70% of the total.

To this end, we settle into a better explanation of attention dissipation and seek to reduce transformer decoder overhead while enhancing its efficiency. We analyze that attention dissipation is derived from over-concentrated distribution across multiple elements of the attention weights after Softmax within restricted frames. To address this issue, Pompt Query Generator (PQG) is adopted to process the audio feature in a prompt manner. This approach rebuilds the distinguishing capability of cross-attention, effectively eliminating attention dissipation. For improving decoder efficiency, a novel EarLy Focus (ELF) decoder is introduced. Specifically, convolution blocks are conducted in the early transformer decoder stages. This modification is more suitable to capture local features in contrast to attention while reducing the computational cost of the latter. Our method proves to be faster and more efficient than relying solely on cross-attention throughout the entire decoder.

Figure: Figure 2: mIoU (%) vs. Inference Latency (ms) on S4 (left), MS3 (middle) and AVSS (right) compared with other popular methods. Latency is measured on a single Nvidia RTX 3090 GPU. AVESFormer achieves the best trade-off between performance and inference speed.
Refer to caption: x2.png

Figure: Figure 3: Runtime profiling of the AVSegFormer []. Inference time is dominated by transformer architecture.
Refer to caption: x3.png

In this work, we introduce AVESFormer, an Audio-Visual Efficient Segmentation transformer, which achieves fast, efficient and light-weight simultaneously. To the best of our knowledge, AVESFormer is the first real-time transformer model for AVS tasks. AVESFormer addresses the critical issue of attention dissipation through prompt query generator and reduces inference runtime with efficient ELF decoder. As shown in Figure 2, comprehensive experiments demonstrate that AVESFormer achieves state-of-the-art performance-latency trade-off, outperforming AVSegFormer [] (+3.4% on S4, +8.4% on MS3, +6.3% on AVSS) while using $20\%$ less parameters and being $3\times$ faster.

Our contributions can be summarized as follows:

- •
We discover the attention dissipation phenomenon in the cross-attention fusion process. To address this, we propose a novel prompt audio query generator that corrects its behaviour and establishes a reliable representation capability of audio-visual fusion.
- •
We identify insufficient audio-visual fusion in the early stages of the transformer decoder. Thus we adopt ELF decoder, which reduces computational cost and promotes efficient audio-visual fusion in deeper stages.
- •
Our method achieves state-of-the-art w.r.t. the trade-off between performance and inference speed on challenging AVSBench-Object and AVSBench-Semantic datasets.

## 2 Related Work

### 2.1 Audio-Visual Segmentation

Audio-Visual Segmentation (AVS) is a more fine-grained and complicated task than sound source localization (SSL) [, , ] as it aims to locate the sounding object and show pixel-level predictions. In recent years, AVS has attracted significant attention from researchers.
AVSBench [] is the first to propose audio-visual segmentation benchmark, introducing temporal pixel-wise audio-visual interaction (TPAVI) module to facilitate interaction between audio-visual information.
AVSegFormer [] is the first to develop a novel transformer architecture for AVS. They introduce audio queries into the transformer decoder to attend to corresponding visual features.
CATR [] performs bidirectional combinatorial dependence fusion to fully enhance spatial-temporal dependencies.
CAVP [] incorporates contrastive loss into audio-visual semantic segmentation with positive and negative pairs and uses larger resolution with extra data to reach higher performance.
Unlike these methods, this paper focuses on real-time end-to-end inference scenario of AVS model and provides a detailed analysis of attention dissipation and decoder efficiency.

### 2.2 Transformer in Semantic Segmentation

In recent years, transformer architecture has significantly influenced semantic segmentation.
DPT [] designs a transformer-based encoder-decoder architecture for dense prediction tasks.
SETR [] shows impressive results by modelling segmentation as a sequence-to-sequence task.
SegFormer [] introduces a hierarchical transformer encoder and an all-MLP decoder to improve the network efficiency.
MaskFormer [] and Mask2Former [] modify segmentation in
a set prediction paradigm, predicting a set of binary masks and assigning a single category to each one.
However, these models are unsuitable for real-time segmentation tasks due to their heavy computational burden.
RTFormer [] introduces GPU-Friendly attention and arranges low- and high-resolution branches in a stepped layout to make full use of global context.
SeaFormer [] employs squeeze axial attention to reduce the computation burden of self-attention while maintaining the local details.
These methods have significantly advanced semantic segmentation. Considering the tight bond between segmentation and AVS tasks, these approaches have provided substantial inspiration for our work.

### 2.3 Efficient Vision Transformer

ViT [] and its variants [, , ] have demonstrated significant improvements in computer vision. However the high computational cost makes them inferior to CNN in real-time inference scenario. To mitigate this gap, previous works attempt to design more efficient architectures to reduce computational burden. MobileViT [] combines CNN and ViT by integrating global feature fusion of transformer in CNN. MobileFormer [] bridges MobileNet [] and ViT in a parallel design to leverage advantages from both architectures. EfficientFormer [] finds insufficient operations in transformer and slims the model size in a latency-driven manner. LVT [] adopts dilated convolution in attention mechanisms to enhance model performance and efficiency. LIT [] gives a more detailed analysis of self-attention heads and applies MLP to build local dependencies. EfficientViT [] proposed to aggregate multi-scale features via small-kernel convolutions. These methods have made contributions to the development of fast and efficient ViT architectures. We benefit greatly from their contributions to the analysis of AVS tasks.

Figure: Figure 4: The overview of AVESFormer. Audio and visual backbones extract corresponding features. The prompt query generator addresses the attention dissipation problem by inserting the audio feature on top of a set of learnable parameters to generate audio-conditioned queries. The ELF decoder processes local features using convolution blocks in the early stages. Finally, the transformer blocks interact with high-level audio-visual features to generate fused features.
Refer to caption: x4.png

## 3 Method

In this section, we first describe the theoretical analysis of the attention dissipation phenomenon. Then, we elaborate the detailed architecture and components of the proposed AVESFormer.

### 3.1 Attention Dissipation

In real-time AVS scenario, visual feature $\mathcal{F}_{visual}\in\mathbb{R}^{c\times h\times w}$ and audio feature $\mathcal{F}_{audio}\in\mathbb{R}^{1\times c}$ are given at the same moment. The former is usually split into patches $\mathcal{P}_{visual}\in\mathbb{R}^{N\times c}$ where $N=h\times w$ in the cross attention mechanism. Prevailing methods [, , , ] usually fuse the two features to build reliable correspondence between audio-visual modalities, as shown on the left panel of Figure 1. Let us denote $q_{i},k,v\in\mathbb{R}^{1\times c}$ as row vectors for $i\in[1,2,\dots,N]$, with $\mathcal{P}_{visual}=[q_{i}]_{N\times c}$ and $\mathcal{F}_{audio}=k=v$. The cross-attention fusion can be represented as follows:

$$ $\displaystyle\mathcal{O}$ $\displaystyle=\text{Softmax}(\mathcal{P}_{visual}\mathcal{F}_{audio}^{T}) \mathcal{F}_{audio},$ (1) $\displaystyle o_{i}$ $\displaystyle=\sum_{j}a_{i,j}v_{j}=\frac{\sum_{j}e^{q_{i}k_{j}^{T}}v_{j}}{\sum _{j}e^{q_{i}k_{j}^{T}}},$ (2) $$

where $\mathcal{O}=\begin{bmatrix}o_{i}\end{bmatrix}\in\mathbb{R}^{N\times c}$ and $j$ stands for the row index of $\mathcal{F}_{audio}$.
The scale factor $\sqrt{d}$ in Softmax as well as linear transformation matrices of $W^{Q}$, $W^{K}$ and $W^{V}$ [] are omitted for the sake of simplicity without affecting the conclusion.

However, $\mathcal{F}_{audio}$ is an 1-dimensional vector within a single frame, which makes $k_{j}=k$ and $v_{j}=v$. Based on this hypothesis, we substitute $j=1$ into Equation (2) to obtain:

$$ $\displaystyle o_{i}$ $\displaystyle=\frac{e^{q_{i}k^{T}}v}{e^{q_{i}k^{T}}}=v.$ (3) $$

The final output of cross-attention fusion can be written as:

$$ $\displaystyle\mathcal{O}$ $\displaystyle=\text{Softmax}(\begin{matrix}\{q_{i}k^{T}\}_{ij}\\ \end{matrix})v=\mathbf{1}_{N\times 1}\mathcal{F}_{audio}=\begin{bmatrix} \mathcal{F}_{audio}\\ \end{bmatrix}_{N\times c}.$ (4) $$

From Equation (4), the cross-attention fusion turns into a simple replication of the audio feature, as illustrated on the right panel of Figure 1. In this process, the attention weights turn out to be over-concentrated after Softmax over 1-dimensional keys. The phenomenon revealed in Equation (4), termed Attention Dissipation, significantly harms the capability of distributing attention on multi-modality representation, thus constraining the effectiveness of the attention mechanism [, ]. Modifications to the audio features are necessary to correct the behaviour of cross-attention. One potential solution is to expand the amount of audio tokens. See Appendix A.1 for more proof details.

### 3.2 AVESFormer Architecture

We now introduce the overall architecture of AVESFormer, including audio-visual backbones, prompt query generator, early focus decoder and loss function, as shown in Figure 4.

#### Visual Backbone.

Initially, audio-visual features are extracted by corresponding backbones.
For a single frame $x_{visual}\in\mathbb{R}^{3\times H\times W}$, where $H$ and $W$ stand for the
height and width of the image, hierarchical visual features are extracted as follows:

$$ $\displaystyle\mathcal{F}_{visual}$ $\displaystyle=\{\mathcal{F}_{1},\mathcal{F}_{2},\mathcal{F}_{3},\mathcal{F}_{4 }\},$ (5) $$

where $\mathcal{F}_{i}\in\mathbb{R}^{c_{i}\times\frac{H}{2^{i+1}}\times\frac{H}{2^{i+
1}}}$, $i\in\{1,2,3,4\}$ represents features at different scale with channel $c_{i}$.

#### Audio Backbone.

Meanwhile, the audio signal in the video with time duration $T$ is resampled to yield a 16kHz mono output $A_{mono}\in\mathbb{R}^{N_{samples}\times 96\times 64}$, where $N_{samples}$ stands for the amount of sampling points. Then, $A_{mono}$ is converted into Mel-spectrum
$A_{mel}\in\mathbb{R}^{T\times 96\times 64}$ by short-time
Fourier transform. Finally we put $A_{mel}$ into pre-trained audio backbone Vggish [] to extract $T$ audio features, each one is denoted as $\mathcal{F}_{audio}\in\mathbb{R}^{1\times D}$, where $D$ is the audio feature dimension.

Figure: Figure 5: Illustration of prompt query generator. The audio feature is treated as prompt and discarded in output.
Refer to caption: x5.png

#### Prompt Query Generator.

To mitigate the attention dissipation discussed in Section 3.1, a novel prompt query generator (PQG) is proposed to expand audio query rather than replicating, as depicted in Figure 5. The audio feature at a single frame is regarded as a prompt [] and inserted into a set of learnable queries $Q_{learn}\in\mathbb{R}^{N_{q}\times D}$:

$$ $\displaystyle Q^{\dagger}=[\mathcal{F}_{audio}|Q_{learn}]\in\mathbb{R}^{(N_{q} +1)\times D},$ (6) $$

where $[\cdot|\cdot]$ denotes concatenation and $N_{q}$ denotes query amount. Then, self-attention is performed between audio features and learnable queries. In the self-attention process,
the attention matrix $QK^{T}$ can be written as follows:

$$ $\displaystyle QK^{T}$ $\displaystyle=Q^{\dagger}Q^{\dagger T}$ (7) $\displaystyle=[\mathcal{F}_{audio}|Q_{learn}][\mathcal{F}_{audio}|Q_{learn}]^{T}$ (8) $\displaystyle=\left[\begin{array}[]{c:c}\mathcal{F}_{audio}\mathcal{F}_{audio} ^{T}&\mathcal{F}_{audio}Q_{learn}^{T}\\ \hdashline Q_{learn}\mathcal{F}_{audio}^{T}&Q_{learn}Q_{learn}^{T}\end{array} \right].$ (11) $$

Afterwards, $Q^{\dagger}$ generates augmented audio features with relativity from the original feature. Lastly, the original audio token at the output end is discarded to obtain $\mathcal{F}_{gen}\in\mathbb{R}^{N_{q}\times D}$. Note that PQG enhances the diversity of audio features and corrects the behaviour of cross-attention.

#### Early Focus Decoder.

Despite its powerful representation ability, the transformer decoder remains the main bottleneck of runtime, as shown in Figure 3. Previous works on ViTs [, ] suggest that early stages of self-attention tend to be inefficient because they predominantly focus on local patterns, leading to wasted long-range modelling capability. In contrast, deeper stages mainly capture long-range, high-level semantics. In this work, we visualize the audio-visual cross-attention patterns, as shown in Figure 6. In the early stages, audio features generate narrow local
visual responses on attention maps. As the stage goes deeper, the attention region enlarges gradually. In the last two stages, it forms shaped and fine-grained regions suitable for segmentation. Therefore, we propose a novel early focus (ELF) decoder. Since the early stage primarily captures local patterns, attention to high computational cost is replaced by convolution to capture local semantics. In early decoder stage $l$, visual feature $\mathcal{F}_{visual}$ is processed by convolution:

$$ $\displaystyle\mathcal{F}_{visual}^{l+1}=\text{LN}(\mathcal{F}_{visual}^{l}+ \text{Conv}(\mathcal{F}_{visual}^{l})),$ (12) $$

where LN denotes LayerNorm [] and Conv is composed of RepBlock []. In deeper stages, we split $\mathcal{F}_{visual}$ into visual patches $\mathcal{P}_{visual}$ [] to perform cross-attention with $\mathcal{F}_{gen}$ from PQG:

$$ $\displaystyle\mathcal{P}_{visual}^{l+1}$ $\displaystyle=\text{LN}(\mathcal{P}_{visual}^{l}+\text{CA}(\mathcal{P}_{visual }^{l},\mathcal{F}_{gen},\mathcal{F}_{gen})),$ (13) $$

where CA denotes multi-head cross-attention and $\text{CA}(Q,K,V)=\text{Softmax}(QK^{T})V$. The ELF decoder eliminates the computational burden brought by wasted attention operations but still maintains the original module function to extract local features. By incorporating our ELF decoder, we find more performance-computation efficiency enhancement in our model.

Figure: Figure 6: Attention probabilities of different blocks in fully transformer decoder. Each map shows the attention probability of the audio query to all visual patches. Maps are averaged along all heads. Each row indicates a test sample. Dark red indicates higher attention probability and shallow orange indicates lower attention probability. See Appendix C.2 for more details.
Refer to caption: x6.png

#### Loss Function.

Following MaskFormer [], we employ IoU loss and Dice [] loss to provide supervision between the predicted mask $\hat{\mathcal{M}}$ and ground truth $\mathcal{M}$.
The IoU loss $\mathcal{L}_{\text{IoU}}$ measures the intersection over union between prediction and ground truth.
Moreover, Dice loss $\mathcal{L}_{\text{Dice}}$ is set to obtain additional supervision information.
Since the foreground proportion in the AVS task is relatively small within the entire image, Dice loss
could force the model to focus on the target region and suppress the impact of background interference.
Besides, we employ auxiliary loss $\mathcal{L}_{\text{aux}}$ for more fine-grained segmentation. The intermediate feature from convolution blocks of ELF decoder $\mathcal{F}_{ELF}\in\mathbb{R}^{c\times h\times w}$ is introduced to calculate $\mathcal{L}_{\text{aux}}$. Suppose $\mathcal{M}_{f}$ represents the foreground mask, $\mathcal{L}_{\text{aux}}$ can be written as:

$$ $\displaystyle\mathcal{L}_{\text{aux}}(\mathcal{F}_{ELF},\mathcal{M}_{f})$ $\displaystyle=\frac{1}{c}\sum_{i=1}^{c}\mathcal{L}_{\text{Dice}}(\mathcal{F}_{ ELF}^{i},\mathcal{M}_{f}).$ (14) $$

The total segmentation loss can be written as:

$$ $\displaystyle\mathcal{L}$ $\displaystyle=\lambda_{\text{IoU}}\mathcal{L}_{\text{IoU}}(\hat{\mathcal{M}}, \mathcal{M})+\lambda_{\text{Dice}}\mathcal{L}_{\text{Dice}}(\hat{\mathcal{M}}, \mathcal{M})+\lambda_{\text{aux}}\mathcal{L}_{\text{aux}}(\mathcal{F}_{ELF}, \mathcal{M}_{f}),$ (15) $$

where $\lambda_{\text{IoU}}$, $\lambda_{\text{Dice}}$ and $\lambda_{\text{aux}}$ are hyperparameters. See Appendix B.1 for more details.

## 4 Experiments

#### Dataset.

We evaluate our method on the AVSBench dataset [, ], which is composed of AVSBench-Object and AVSBench-Semantic.
AVSBench-Object is designed for audio-visual segmentation tasks with pixel-level annotations. Videos are sourced from YouTube, cropped into 5 seconds, and sampled at one frame per second to compose the image data. There are two subsets in AVSBench-Object: single sound source segmentation (S4) subset and multiple sound source segmentation (MS3) subset. The S4 subset contains 4,932 videos: 3,452 for training, 740 for validation and 740 for testing. The labels contain 23 categories, including humans, vehicles, animals and kinds of instruments. Note that annotations in S4 training set is only given in the first frame. Meanwhile, MS3 subset is composed of multiple sound sources, including 424 videos, 286 for training, 64 for validation and 64 for testing. MS3 shares the same categories as S4.
AVSBench-Semantic is an expanded version of AVSBench-Object, providing additional semantic masks to facilitate audio-visual semantic segmentation (AVSS). Videos in AVSBench-Semantic extend up to 10 seconds with 10 frames per video to compose the image data. Moreover, 70 categories are annotated in 11,356 videos: 8,498 for training, 1,304 for validation and 1,554 for testing.

#### Implementation Details.

We conduct our experiments with PyTorch. Our model is trained on NVIDIA RTX 3090 GPU. We employ AdamW [] as optimizer with batch size 16 and learning rate of 0.0005 for S4 as well as MS3 while batch size 8 and learning rate of 0.0001 for AVSS. All images are resized into $224\times 224$.
From the aspect of real-time inference, we employ ResNet-50 and ResNet-18 [] pre-trained on ImageNet [] as our visual backbones. Considering Pyramid Vision Transformer (PVT-v2) [] is unsuitable for real-time applications, we do not adopt it as the visual backbone. We employ Vggish [] pre-trained on AudioSet [] to
encode audio input. The audio backbone is frozen during the training. The embedding dimensions of both encoders are set to 256. Transformer decoder comes up with multi-scale deformable attention (MSDeform) [] followed by self-attention [] and FFN. See Appendix B.1 for more experimental details.

#### Evaluation Metrics.

Following [], we adopt Jaccard index $\mathcal{J}$ and F-score $\mathcal{F}$ to evaluate. $\mathcal{J}$ indicates the mean intersection over union (mIoU) [] between segmentation prediction and ground truth. $\mathcal{F}$ measures the precision and recall by $\mathcal{F}=\frac{(1+\beta^{2}\times\text{precision}\times\text{recall})}{
\beta^{2}\times\text{precision}+\text{recall}}$, where $\beta^{2}=0.3$.

**Table 1: Comparison with state-of-the-art methods on the AVS benchmark. All methods are evaluated on three AVS sub-tasks (S4, MS3 and AVSS). The evaluation metrics are mIoU and F-score. #Params refers to the number of parameters. FPS is reported on a single NVIDIA RTX 3090 GPU. * means the parameters of audio backbone Vggish [] are included.**
| Method | Backbone | S4 |  | MS3 |  | AVSS | #Params^∗ | FPS |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\mathcal{J}$ | $\mathcal{F}$ |  | $\mathcal{J}$ | $\mathcal{F}$ |  | $\mathcal{J}$ | $\mathcal{F}$ | (M) |  |  |  |
| LVS [] | ResNet-18 | 38.0 | 51.0 |  | 29.5 | 33.0 |  | - | - | - | - |
| MSSL [] | ResNet-18 | 44.9 | 66.3 |  | 26.1 | 36.3 |  | - | - | - | - |
| 3DC [] | ResNet-152 | 57.1 | 75.9 |  | 36.9 | 50.3 |  | 17.3 | 21.6 | - | - |
| SST [] | ResNet-101 | 66.3 | 80.1 |  | 42.6 | 57.2 |  | - | - | - | - |
| AOT [] | Swin-B | - | - |  | - | - |  | 25.4 | 31.0 | - | - |
| iGAN [] | Swin-T | 61.6 | 77.8 |  | 42.9 | 54.4 |  | - | - | - | - |
| LGVT [] | Swin-T | 74.9 | 87.3 |  | 40.7 | 59.3 |  | - | - | - | - |
| AVSBench [] | ResNet-50 | 72.8 | 84.8 |  | 47.9 | 57.8 |  | 20.2 | 25.2 | 163 | 63.6 |
| CATR [] | 74.8 | 86.6 |  | 52.8 | 65.3 |  | - | - | 177 | 46.4 |  |
| DiffusionAVS [] | 75.8 | 86.9 |  | 49.8 | 62.1 |  | - | - | - | - |  |
| ECMVAE [] | 76.3 | 86.5 |  | 48.7 | 60.7 |  | - | - | 162 | 52.8 |  |
| AuTR [] | 75.0 | 85.2 |  | 49.4 | 61.2 |  | - | - | - | - |  |
| AQFormer [] | 77.0 | 86.4 |  | 55.7 | 66.9 |  | - | - | - | - |  |
| AVSC [] | 77.0 | 85.2 |  | 49.6 | 61.5 |  | - | - | - | - |  |
| AVSegFormer [] | 76.5 | 85.9 |  | 49.5 | 62.8 |  | 24.9 | 29.3 | 151 | 26.4 |  |
| AVSBG [] | 74.1 | 85.4 |  | 45.0 | 56.8 |  | - | - | - | - |  |
| BAVS [] | 78.0 | 85.3 |  | 50.2 | 62.4 |  | 24.7 | 29.6 | 118 | - |  |
| AVESFormer (ours) | ResNet-18 | 77.3 | 87.5 |  | 55.5 | 65.1 |  | 26.3 | 31.8 | 108 | 113.0 |
| ResNet-50 | 79.9 | 89.1 |  | 57.9 | 68.7 |  | 31.2 | 36.8 | 127 | 83.5 |  |

### 4.1 Comparison with State-of-the-arts

Comprehensive experiments have been conducted on AVSBench-Object and AVSBench-Semantic datasets alongside other methods. As shown in Table 1, our AVESFormer exhibits the state-of-the-art performance-speed trade-off among all models. Specifically, AVESFormer surpasses previous methods w.r.t. mIoU by 79.9% on the S4 subset, 57.9% on the MS3 subset and 31.2% on the AVSS subset, respectively. Figure 2 illustrates that the inference speed of AVESFormer exceeds previous methods with the ResNet-50 backbone by large margins. In summary, these results demonstrate the advantages of AVESFormer in terms of performance, speed, and model size.

### 4.2 Ablation Study

#### Training Setup.

We provide ablation results with AVESFormer. To make quick evaluations, we adopt ResNet-50 as the backbone and perform extensive experiments on the S4 and MS3 sub-tasks. Other training settings remain consistent with Section 4.

#### Prompt Query Generator

To verify the effectiveness of our prompt query generator, we remove it to fuse modality with only one audio feature. Additionally, the query generator (QG) in [] and a bias query generator (BQG) are also included. The ordinary query generator follows default settings with 6 layers and 300 queries. The bias query generator replicates the audio query and adds a learnable bias term to it. As shown in Table 3, PQG treats the audio feature as a prompt and cleverly addresses dissipation to avoid attention dissipation, yielding more improvements than the bias query generator.

Figure: Figure 7: Visualization of attention maps, including no fusion, TPAVI [], channel attention mixer (CHA) [], sigmoid attention [] and our ELF decoder. Each map shows the correlation between audio queries and visual patches. Red indicates higher attention score while blue indicates lower.
Refer to caption: x7.png

**Table 2: Effect of the prompt query generator. Prompt query generator overcomes attention dissipation to gain more improvements.**
| Method | S4 |  | MS3 |  |  |
| --- | --- | --- | --- | --- | --- |
| $\mathcal{J}$ | $\mathcal{F}$ |  | $\mathcal{J}$ | $\mathcal{F}$ |  |
| w/o QG | 75.9 | 87.1 |  | 50.0 | 61.9 |
| QG [] | 78.5 | 88.7 |  | 50.0 | 61.7 |
| BQG | 75.9 | 87.1 |  | 49.6 | 60.0 |
| PQG | 79.9 | 89.1 |  | 57.9 | 68.7 |

#### Influence with Plug and Play PQG.

Furthermore, PQG can be integrated into other models such as AVSegFormer [], as shown in Table 3. Since S4 (single source) is less strict to the audio distinguishing capability, PQG merely exhibits a slight improvement. However, on MS3, where the audio distinguishing capability is crucial due to the presence of multiple sound sources within an image, PQG demonstrates substantial improvement (+6.5% mIoU) when applied to AVSegFormer.

#### ELF Decoder.

We analyze the influence of convolution positioned at different stages of the ELF decoder. As shown in Table 5, "C" denotes convolution and "T" denotes transformer. The "Stage" column indicates the insertion stage of convolution, with three options listed: early (C-T-T), middle (T-C-T) and deep (T-T-C). Additionally, a pure transformer decoder (T-T-T) is included. As convolution blocks are moved deeper, the mIoU drops by 2.81% on S4 and 2.73% on MS3. This decline can be attributed to the fact that early layers primarily generate local responses. In contrast, deeper layers facilitate high-level interactions between audio-visual modalities, which are essential for AVS tasks.

#### Fusion Strategy.

Furthermore, the impact of cross-attention after addressing attention dissipation compared to other fusion strategies is investigated. Four representative fusion strategies are adopted: a) no audio-visual modality fusion, which can be caused by attention dissipation, b) TPAVI proposed in AVSBench [], c) channel attention adopted in AVSegFormer [], d) sigmoid attention evoked in CAVP []. Results are shown in Table 5. After addressing attention dissipation, our ELF decoder with cross-attention fusion emerges as the optimal choice, demonstrating the most distinguishing representation capability. Figure 7 shows the attention map visualizations of different fusion strategies.

Figure: Figure 8: Visualization of segmentation predictions on S4 (left), MS3 (middle) and AVSS (right) Dataset with AVSBench [] and AVSegFormer [].
Refer to caption: x8.png

**Table 4: Impact of the convolution blocks at different stages. We show model performance with different convolution insertion stages.**
| Stage | S4 |  | MS3 |  |  |
| --- | --- | --- | --- | --- | --- |
| $\mathcal{J}$ | $\mathcal{F}$ |  | $\mathcal{J}$ | $\mathcal{F}$ |  |
| T-T-T | 77.3 | 87.6 |  | 56.2 | 66.6 |
| C-T-T | 79.9 | 89.1 |  | 57.9 | 68.7 |
| T-C-T | 77.6 | 88.0 |  | 56.5 | 67.3 |
| T-T-C | 77.1 | 88.3 |  | 55.2 | 67.3 |

**Table 6: Performance of the number of queries.**
|  | S4 | MS3 |  |  |
| --- | --- | --- | --- | --- |
| # of queries | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ |
| 8 | 79.3 | 88.9 | 55.8 | 66.0 |
| 16 | 79.9 | 89.1 | 57.9 | 68.7 |
| 32 | 79.4 | 88.9 | 56.2 | 66.6 |
| 64 | 79.1 | 88.9 | 55.8 | 67.0 |
| 128 | 79.0 | 88.8 | 56.0 | 67.4 |
| 256 | 79.3 | 89.0 | 57.3 | 67.8 |

#### Number of Queries.

Table 6 presents the results of AVESFormer trained with varying numbers of quires of AVS dataset. Experiments are conducted with query numbers ranging from 8 to 256 with a scale factor of 2. Notably, using 16 queries performs best across S4 and MS3. This suggests that even though there are a number of sounding object categories, a large number of queries may not be necessary. A few queries in AVESFormer are adequate for learning distinguishing audio features.

#### Qualitative Analysis.

Visualizations of AVESFormer compared with those of AVSBench [] and AVSegFormer [] in ResNet-50 backbone on AVSBench-object and AVSBench-semantic datasets are depicted in Figure 8. Our AVESFormer overcomes critical attention dissipation and makes more sophisticated visualization and segmentation performance. See Appendix C.1 for more visualizations.

## 5 Conclusion and Discussion

#### Conclusion.

In this paper, we analyze the attention dissipation phenomenon and inefficient transformer decoder. Based on these findings, we introduce AVESFormer, the first transformer-based real-time AVS model. Experimental results demonstrate that AVESFormer achieves the new state-of-the-art performance-speed trade-off. We hope our method provides insights into new architecture design not only in AVS tasks but also in various multi-modality scenarios.

#### Limitation and Future Work.

There still exist limitations on AVESFormer. On one hand, the audio backbone Vggish [] constitutes about 60% of the model parameters, posing challenges for deployment on mobile devices. On the other hand, temporal information is ignored in real-time AVS scenario. These will be the focus of our future work.

## Appendix

## Appendix A Attention Dissipation

### A.1 Proof on Attention Dissipation

As discussed in Sec. 3.1, a brief explanation of attention dissipation is given. Now, we will provide more detailed proof of this phenomenon.

As commonly practised in AVS tasks, visual features are extracted from the visual backbone to get $\mathcal{F}_{visual}\in\mathbb{R}^{c\times h\times w}$ of one frame. Then we patchify the visual feature into $\mathcal{P}_{visual}\in\mathbb{R}^{N\times c}$ where $N=h\times w$. Meanwhile, audio signals within one frame are input into the audio backbone to form $\mathcal{F}_{audio}\in\mathbb{R}^{1\times c}$. Note that since we only consider one frame at a time in real-time scenario, the sequence length of the audio feature is equal to 1. We cannot omit the sequence length dimension because we should keep this shape to perform matrix multiplication in the attention mechanism.

Consequently, the modality fusion process is performed originally by cross attention, where visual patches are query while the audio feature is key and value:

$$ $\displaystyle O$ $\displaystyle=\text{Softmax}(\mathcal{P}_{visual}\mathcal{F}_{audio}^{T}) \mathcal{F}_{audio}\in\mathbb{R}^{N\times c},$ (16) $$

where

$$ $\displaystyle\mathcal{P}_{visual}$ $\displaystyle=\begin{bmatrix}q_{1}\\ q_{2}\\ \vdots\\ q_{N}\end{bmatrix},$ (17) $\displaystyle q_{i}$ $\displaystyle\in\mathbb{R}^{1\times c},\quad i\in[1,2,\dots,N],$ (18) $\displaystyle\mathcal{F}_{audio}$ $\displaystyle=k=v\in\mathbb{R}^{1\times c}.$ (19) $$

The attention logit matrix $\mathcal{A}$ can be written as:

$$ $\displaystyle\mathcal{A}=\mathcal{P}_{visual}\mathcal{F}_{audio}^{T}=$ $\displaystyle\begin{bmatrix}q_{1}\\ q_{2}\\ \vdots\\ q_{N}\end{bmatrix}k^{T}=\begin{bmatrix}q_{1}k^{T}\\ q_{2}k^{T}\\ \vdots\\ q_{N}k^{T}\end{bmatrix}\in\mathbb{R}^{N\times 1},$ (20) $$

where

$$ $\displaystyle q_{i}k^{T}$ $\displaystyle\in\mathbb{R},\quad i\in[1,2,\dots,N].$ (21) $$

Softmax is calculated along the row vector on attention matrix $\mathcal{A}$ to get attention probability matrix $\mathcal{P}$:

$$ $\displaystyle\mathcal{P}=\text{Softmax}(\mathcal{A})|_{\text{row}}=\begin{ bmatrix}e^{q_{1}k^{T}}/\sum e^{q_{1}k^{T}}\\ e^{q_{2}k^{T}}/\sum e^{q_{2}k^{T}}\\ \vdots\\ e^{q_{N}k^{T}}/\sum e^{q_{N}k^{T}}\end{bmatrix}$ $\displaystyle=\begin{bmatrix}e^{q_{1}k^{T}}/e^{q_{1}k^{T}}\\ e^{q_{2}k^{T}}/e^{q_{2}k^{T}}\\ \vdots\\ e^{q_{N}k^{T}}/e^{q_{N}k^{T}}\end{bmatrix}=\begin{bmatrix}1\\ 1\\ \vdots\\ 1\end{bmatrix}=\mathbf{1}_{N\times 1}.$ (22) $$

Finally the output $\mathcal{O}$ becomes a simply replication of value matrix:

$$ $\displaystyle\mathcal{O}=\text{Softmax}(\mathcal{A})|_{\text{row}}\mathcal{F}_ {audio}=\mathcal{P}\mathcal{F}_{audio}=\mathbf{1}_{N\times 1}\mathcal{F}_{ audio}=\begin{bmatrix}1\\ 1\\ \vdots\\ 1\end{bmatrix}\mathcal{F}_{audio}=\begin{bmatrix}\mathcal{F}_{audio}\\ \mathcal{F}_{audio}\\ \vdots\\ \mathcal{F}_{audio}\end{bmatrix}.$ (23) $$

The attention dissipation phenomenon shows that cross-attention with visual features such as query and audio as key and value turns out to be a simple replication of audio signals. It goes against our original intent of modality fusion.

### A.2 Code implementation

To make a fully comprehensive understanding of attention dissipation, we provide a PyTorch-like pseudo-code for easy verification and implementation of cross-attention dissipation. Algorithm 1 provides the pseudo-code of attention dissipation in the AVS task. For the current frame, we calculate the attention matrix with the use of visual features as query and audio as key and value.

Figure: Algorithm 1 Pseudo-code of Attention Dissipation in a PyTorch-like style.

## Appendix B Experiments

### B.1 Experimental Details

During training, we use the original image size as 224$\times$224. We apply horizontal flipping on S4 and MS3 for data augmentation. Since the S4 sub-set only contains annotations on the first frame in the training split, we only use the first frame to provide supervision. We use the AdamW optimizer and a polynomial learning rate decay with power = 0.9. On S4 and MS3, the learning rate is set to 0.0005, and on AVSS, it is set to 0.0001. Following previous practice [], we train MS3 for 60 epochs since it is relatively small, while the S4 and AVSS subsets are trained for 30 epochs. Batch size is set to 16 for S4 and MS3 and 8 for AVSS. We adopt two ResNet [] backbones (ResNet-50 and ResNet-18) for the segmentation network. For the audio backbones, we use VGGish [] frozen during the training. The prompt query generator (PQG) receives the feature from the audio backbone as prompt. The number of queries is set to 16, and the number of layers is set to 3. At the output end, the audio feature prompt is discarded. The transformer decoder is adopted from Multi-Scale Deformable (MSDeform) attention []. The first two attention blocks are replaced by convolution to form ELF decoder. Convolution blocks are attached with residual connection and LayerNorm []. As for the segmentation loss, on S4 and MS3, we set $\lambda_{\text{IoU}}=1.8$ and on AVSS $\lambda_{\text{IoU}}=1.0$ with $\lambda_{\text{Dice}}=1.0$ and $\lambda_{\text{aux}}=0.1$. For inference, since the end-to-end real-time scenario does not support inferring on a bunch of frames (because we want to segment one image at a time on the device), the latency of all models is measured under one single frame, that is, $T=1$. Nevertheless, some of the methods employ temporal information within multiple frames, which would be lost in a single frame scenario; we still keep their performance the same for comparison.

## Appendix C Qualitative analysis

### C.1 Results Visualization

We present additional visualization results for the paper, alongside AVSBench [], AVSegFormer [] and our model on AVSBench-Object [] and AVSBench-Semantic [] with ResNet-50 [] backbone, as depicted in Figure. 11, Figure. 11, and Figure. 11. We demonstrate that AVESFormer efficiently presents a more fine-grained prediction and a more accurate audio-visual corresponding capability to the segmentation of objects in the scene compared to previous methods.

Figure: Figure 9: Qualitative audio-visual segmentation results on AVSBench-Object S4 sub-set [] by TPAVI [], AVSegFormer [], and AVESFormer. Each row represents the raw image, ground truth or different methods. Each column represents various data samples.
Refer to caption: x9.png

### C.2 Attention Map in Decoder Blocks

We present the visualization of a full transformer decoder architecture to show the attention pattern of different stages. As shown in Figure. 12, the illustration represents a narrow local attention map in the early stages. As the stage goes deeper, the attention region grows larger gradually. In the end, the attention region becomes large and fine-grained, which is suitable for the segmentation task.

Figure: Figure 12: Attention probabilities of a full transformer decoder. Each map shows the attention probability of a query audio feature to all visual patches. Maps are averaged along all heads. Each row represents an image sample in the test set. Each column represents a decoder block. Dark red indicates higher attention probability and shallow orange indicates low attention probability.
Refer to caption: x12.png

## Appendix D Broader Impacts

Our AVESFormer has taken a step towards real-time performance in AVS tasks. Our model has seen significant advancements, allowing for applications across various domains. Potentially, our model may bring positive impact on reducing computational cost, improving quality of life and enhancing automation and efficiency. Negative impact may lies on surveillance, privacy concerns and the dependence on data quality. Addressing these challenges is crucial for the responsible and ethical deployment of these models.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Line 53-76 tries to explain the issues of exiting AVS method. Although it's reasonable that effective multimodal fusion should be critical for AVS, it's not clear why this claim hold: "attention variants generally do not exhibit the same expressive capacity as the default mechanism" (line 75). Further explanation is needed.
2. In Fig. 2, the paper illustrates of attention dissipation. How the audio token is generated? Is this only a one case issue or does it happen all the time in all types of cross attention based AVS? More details are needed to verify the existence of the attention dissipation issue, especially when audio is defined as both key and value in the current cross attention setting. Moreover, where does the cross attention AVS model in equation 1 come from? Citation(s) are needed if it comes from existing literatures, or explanations are needed to explain why QKV is defied in the current way (compared to Ref1, Ref2, Ref3).
3. "Attention maps at early decoder stages tends to capture short-term local correlation features, leading to undesired low utilization of attention" (line 89-91) is used to explain the "narrow attention" issue, which is also not very clear. Why call it "narrow"?
4. In line 123, it's not clear why the existing AVS methods fail to work on the "single frame image + audio" setting.
5. what is the audio backbone in line 184?
6. F_audio=k=v, not clear (line 194). Also, audio feature dimension is D in line 185 and c in line191. It's better to use consistent symbols.
7. The prompt query generator (sec. 4.1) augments the audio feature. How does it bring extra useful information given the augmentation is based on the raw audio feature?
8. The early focus decoder is used to solve the "narrow attention" issue. However, it's not clear how eq. 6 and eq. 7 combined can solve the narrow attention issue.
9. In Table 1, why Ref1 is not compared?
10. In Table 4 and Table 5, it seems the proposed two strategies work better for MS3 dataset, explanations are needed.

Ref1: Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation. CVPR 2024

Ref2: AVSegFormer: Audio-Visual Segmentation with Transformer. AAAI 2024

Ref3: Audio–visual segmentation. ECCV 2022
soundness: 2
strengths: 1. Solving AVS via attention analysis is reasonable given the multimodal nature of AVS.
2. The attention dissipation issue seems interesting given the current QKV setting.
3. The experiments are comprehensive.
summary: This paper finds two primary challenges in existing audio-visual segmentation models, namely attention dissipation caused by anomalous attention weights after Softmax over limited frames, and narrow attention patterns in shallow decoder stages leading to inefficient utilization of attention resources. An AVESFormer is then presented as the first real-time audio-visual efficient segmentation transformer. Particularly, the proposed method leverages a prompt query generator to rectify cross-attention behavior, and an early focus decoder to enhance efficiency. Extensive experiments demonstrate superiority of AVESFormer in mitigating cross-attention issues and achieving a trade-off between model effectiveness and efficiency.
weaknesses: 1. The two observations which motivate the proposed method are not clear explained (see details in the Questions section).
2. The setting of the paper for real time AVS should also be verified, especially it should be compared to naive settings, e.g. defining the rest of the frames as an empty token. Most importantly, as temporal information is totally discarded, more analysis is needed on how the proposed method achieve smooth real time AVS.
3. Comparison with other attention based techniques is needed (Ref 1 in the Questions section.)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses
soundness: 3
strengths: 1. The paper identifies critical issues in existing audio-visual segmentation (AVS) models, such as attention dissipation and narrow attention patterns, which are valid challenges in the field.

2. The proposed AVESFormer model aims to address efficiency in real-time AVS tasks, which could be valuable for applications needing low-latency responses.

3. The model demonstrates competitive performance on standard benchmarks with a noted improvement in latency, supporting claims of efficiency gains.
summary: The paper introduces AVESFormer, a transformer model designed for real-time audio-visual segmentation (AVS). It addresses attention dissipation and narrow attention patterns in AVS by implementing a Prompt Query Generator (PQG) and Early Focus (ELF) Decoder. These components aim to improve cross-attention efficiency and reduce computational costs. AVESFormer shows competitive performance on AVSBench with noted speed improvements.
weaknesses: While the issues identified in attention mechanisms are relevant, the approach—specifically the Prompt Query Generator and Early Focus Decoder—largely leverages incremental modifications rather than substantial innovations. Many elements, such as convolutional layers for early-stage feature extraction, are already established techniques in efficient transformers.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The authors should consider optimizing the audio backbone, Vggish, which currently comprises about 60% of the model parameters, to enable better deployment on mobile devices. Next, the authors are encouraged to explore how incorporating temporal information might influence model performance and inference speed. Additionally, evaluating AVESFormer on a broader range of datasets beyond AVSBench would provide insights into its generalization capabilities. The paper currently lacks a detailed quantitative analysis of attention maps, so including metrics to evaluate the focus and spread of attention could enhance understanding of the model's behavior.
soundness: 3
strengths: A key innovation is the Prompt Query Generator PQGPQG, which corrects cross-attention behavior and mitigates attention dissipation, enhancing segmentation accuracy and efficiency. Additionally, the Early Focus ELFELF decoder incorporates convolutional operations for local feature extraction, reducing computational demands by replacing attention operations in early transformer stages. The model achieves state-of-the-art performance, surpassing previous models on metrics like the Jaccard index and F-score across datasets such as S4, MS3, and AVSS.
summary: The paper presents AVESFormer, a transformer-based model for real-time audio-visual segmentation AVSAVS. It tackles two main issues in existing AVS models: attention dissipation and limited attention patterns in early decoders. AVESFormer uses a Prompt Query Generator PQGPQG to improve cross-attention and an Early Focus ELFELF decoder that integrates convolutional operations for efficient local feature extraction, reducing computational costs. Experiments show that AVESFormer addresses cross-attention problems, enhances attention utilization, and outperforms previous state-of-the-art models in both performance and speed. Additionally, the paper analyzes attention dissipation and the shortcomings of standard transformer decoders in real-time AVS.
weaknesses: One concern is the significant parameter count of the audio backbone, Vggish, which limits deployment on mobile devices, suggesting future work could optimize this component. Additionally, the model currently ignores temporal information from multiple frames, and incorporating this data could enhance its ability to track moving objects. There is also a need to test the model's generalization to unseen datasets beyond AVSBench to evaluate its robustness in diverse scenarios. While qualitative analyses of attention maps are presented, a quantitative investigation could offer deeper insights into the model's attention mechanisms. Furthermore, comparing AVESFormer with non-transformer models would provide a broader context for its performance. The paper could also benefit from a more detailed discussion on the impacts of model complexity on real-time performance and ethical considerations regarding its application.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: see Weaknesses
soundness: 3
strengths: - The paper is well-written and easy to follow.

- The authors offer comprehensive ablation studies, which aid in comprehending the various design decisions involved in solving cross-attention issues.

- The proposed method shows a strong performance improvement as well as a reduction in the inference time.

- Sufficient comparison with SOTA methods
summary: The authors proposed AVESFormer, an efficient framework for audio-visual segmentation tasks. By solving two issues of using cross-attention as attention dissipation and narrow attention patterns, they can effectively improve the utilization of cross-attention between audio and visual. AVESFormer improves attention utilization, improves the performance as well as reduces the running time.
weaknesses: - Some typo: Table 3: lartency -> latency

- Regarding the PQG, what is the result if we set the number of queries to the number of objects in the category?

- How much of the flops and memory that AVESFormer save from replacing the early attention with convolution?

- Does the number of stages in the decoder affect the performance?

- Missing references [1], [2], and [3] in Table 1

[1] CVPR2024: Audio-Visual Segmentation via Unlabeled Frame Exploitation 

[2] Referred by multi-modality: A unified temporal transformer for video object segmentation

[3] Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation

Meta-review: This work received three negative ratings (borderline reject) and one positive rating (borderline accept). The reviewers rate negative scores due to the limited novelty, lack of high-level explanation of the motivation and contradictory experimental results raised by reviewer jhmt. The authors also did not further address the comments of the reviewer jhmt. AC checked the comments of the reviewers and thus made the final decision.

Decision: Reject