Title: Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks
Venue: NeurIPS 2023
Accepted: yes
Keywords: audio-visual

multi-modal prompt

clip

cross-modal attention
Abstract: In recent years, the deployment of large-scale pre-trained models in audio-visual downstream tasks has yielded remarkable outcomes. However, these models, primarily trained on single-modality unconstrained datasets, still encounter challenges in feature extraction for multi-modal tasks, leading to suboptimal performance. This limitation arises due to the introduction of irrelevant modality-specific information during encoding, which adversely affects the performance of downstream tasks. To address this challenge, this paper proposes a novel Dual-Guided Spatial-Channel-Temporal (DG-SCT) attention mechanism. This mechanism leverages audio and visual modalities as soft prompts to dynamically adjust the parameters of pre-trained models based on the current multi-modal input features. Specifically, the DG-SCT module incorporates trainable cross-modal interaction layers into pre-trained audio-visual encoders, allowing adaptive extraction of crucial information from the current modality across spatial, channel, and temporal dimensions, while preserving the frozen parameters of large-scale pre-trained models. Experimental evaluations demonstrate that our proposed model achieves state-of-the-art results across multiple downstream tasks, including AVE, AVVP, AVS, and AVQA. Furthermore, our model exhibits promising performance in challenging few-shot and zero-shot scenarios. The source code and pre-trained models are available at https://github.com/haoyi-duan/DG-SCT.

Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks

Abstract

In recent years, the deployment of large-scale pre-trained models in audio-visual downstream tasks has yielded remarkable outcomes. However, these models, primarily trained on single-modality unconstrained datasets, still encounter challenges in feature extraction for multi-modal tasks, leading to suboptimal performance. This limitation arises due to the introduction of irrelevant modality-specific information during encoding, which adversely affects the performance of downstream tasks. To address this challenge, this paper proposes a novel Dual-Guided SpatialChannel-Temporal (DG-SCT) attention mechanism. This mechanism leverages audio and visual modalities as soft prompts to dynamically adjust the parameters of pre-trained models based on the current multi-modal input features. Specifically, the DG-SCT module incorporates trainable cross-modal interaction layers into pre-trained audio-visual encoders, allowing adaptive extraction of crucial information from the current modality across spatial, channel, and temporal dimensions, while preserving the frozen parameters of large-scale pre-trained models. Experimental evaluations demonstrate that our proposed model achieves state-of-the-art results across multiple downstream tasks, including A VE, A VVP, A VS, and A VQA. Furthermore, our model exhibits promising performance in challenging few-shot and zero-shot scenarios. The source code and pre-trained models are available at https://github.com/haoyi-duan/DG-SCT .

1 Introduction

With the increasing availability of hardware resources, large-scale models [20, 4, 3] pre-trained on extensive data have achieved significant advancements in various multi-modal tasks [26, 5]. Nonetheless, since these models are primarily pre-trained on single modality, they may not be optimally suited for current multi-modal downstream tasks [33, 32, 43, 14]. As depicted in Fig 1 (a), the pre-trained model equally extracts visual features and directly passes them to downstream tasks. However, when perceiving the roaring sound of an engine, the visual region depicting a "car" should receive more attention than the region of "trees". Simultaneously, when observing the car, it is crucial to concentrate on the audio segments of the engine sound. Therefore, the encoder should not only equally extract modal-specific information from the current modality, but also highlight information related to other modalities to enhance feature fusion across diverse modalities in downstream tasks. Retraining these large models based on downstream tasks would impose an unaffordable burden [41, 22, 39], leading recent works to explore methods for fine-tuning pre-trained models on downstream tasks without full retraining, showing promising progress. However, these CLIP-based methods have primarily focused on text-image tasks [46, 45, 7, 12] , while overlooking another important

W

C

H

W

C

H

𝑻

segments

𝑻

segments

…

W

C

H

W

C

H

W

C

H

W

C

H

W

C

H

Swin-T

Swin-T

DG-SCT

Original visual features

Our visual features

Nothing

Race car

Nothing

Race car

Nothing

Visual input:

Visual event:

Engine

Engine

Engine

Nothing

Engine

:Audio input

:Audio event

3

T

2

T

1

T

4

T

5

T

𝒂

)

𝒃

)

multi-modal scenario: audio-visual tasks. Hence, in this paper, we primarily investigate how to utilize existing large-scale models, such as CLIP [26] and Swin-Transformer [20], to adaptively adjust the encoding features with the guidance of counterpart modality when encoding audio or visual information.

The success of prompt learning in large language models (LLMs) [13, 18, 19, 30, 21] has recently sparked growing research interest in multi-modal prompt learning, as seen in works such as CoOp [46], CoCoOp [45], CLIP-Adapter [7], DenseCLIP [27], and MaPLe [12]. While these approaches enable adaptive adjustment of input features using text-based prompt templates for downstream tasks, an important question arises: Can audio or video serve as innovative prompt templates to enhance task comprehension for pre-trained models and guide adaptive feature extraction of the counterpart modality? Our findings suggest a positive answer to this question.

In this paper, we present the Dual-Guided Spatial-Channel-Temporal (DG-SCT) attention mechanism, designed to adaptively adjust feature extraction of pre-trained models based on audio-visual input. Our work is motivated by a recent work, LAVisH [17], which introduces trainable layers with shared parameters in pre-trained models to enhance fusion of audio-visual features, demonstrating promising performance on various audio-visual downstream tasks with minimal additional parameters. However, LAVisH has a few limitations. First, it relies solely on a visual encoder to encode audio, which we argue is insufficient for capturing key audio features [10, 23]. Second, it only employs cross-attention in trainable layers to introduce information from different modalities, without explicitly highlighting crucial information within the current modality. By contrast, our approach incorporates cross-modal interaction layers into audio (HTS-AT [3]) and visual (ViT [4] or Swin-T [20]) pre-trained models, leveraging different modalities as prompts to focus on special aspects of the input features that are more relevant to the counterpart modal semantics across spatial , channel , and temporal dimensions. We term our proposed approach as "prompts" to denote the guidance provided to the trainable weights in preceding layers. It emphasizes utilizing audio and video cues to guide the representation of the counterpart modalities.

As depicted in Fig. 1, unlike previous audio and visual encoders, which generate audio and visual features separately and uniformly (Fig. 1 (a)), our features contain fine-grained, task-specific information at multiple levels by leveraging the guiding characteristics of multi-modal information [37, 36]. This enables efficient implementation of a wide range of downstream tasks. Notably, unlike previous CLIP works [46, 45] that offer unidirectional prompts, our approach introduces bidirectional prompts. This means that visual and audio modalities can mutually guide each other, facilitating enhanced feature extraction from the respective modalities.

In summary, this paper makes the following contributions:

We highlight the limitations faced by large-scale pre-trained models in audio-visual downstream tasks, which hinder their optimal performance. To overcome these, we propose to utilize audio-visual features as novel prompts to fully leverage the feature extraction capa-

[CLASS]

.

Downstream

Models

a)

train horn

Where is the first

sounding instrument?

d)

right

a photo of a

b)

cello,

violin fiddle

a)

A

V

E

b)

A

V

V

P

c)

A

V

S

d)

A

V

Q

A

c)

e) few-shot/zero-shot

⋅

A2V

V2A

⋅

t

v

t

a

Channel

attention

Spatial

attention

⋅

⋅

c

t

v

c

t

a

…

…

Temporal attention

⋅

⋅

1

{

}

cs

T

t

t

v

=

1

{

}

cf

T

t

t

a

=

v

G

a

G

1

{

}

cfs

T

t

t

a

=

1

{

}

cst

T

t

t

v

=

DG-SCT

DG-SCT

Swin-T Block

Swin-T Block

Swin-T Block

Swin-T Block

T

+

+

…

…

DG-SCT

+

+

T

Trainable

Frozen

1

Self-Attention Block

in Swin-T

2

4

3

v

s

t

M

f

t

a

M

ac

t

M

vc

t

M

𝐹1

⋅ 𝑇

1

𝐹1

⋅ 𝑇

2

…

𝐹1

⋅ 𝑇𝑁

𝐹2

⋅ 𝑇

1

𝐹2

⋅ 𝑇

2

…

𝐹2

⋅ 𝑇𝑁

𝐹𝑁

⋅ 𝑇

1

𝐹𝑁

⋅ 𝑇

2

𝐹𝑁

⋅ 𝑇𝑁

𝐹1

𝐹2

𝐹𝑁

𝑇1

𝑇2

…

𝑇𝑁

…

Visual Input

bilities of large-scale models, enabling the effective utilization of task-specific information from different modalities.

We introduce a novel attention mechanism named Dual-Guided Spatial-Channel-Temporal (DG-SCT), which utilizes audio and visual modalities to guide the feature extraction of their respective counterpart modalities across spatial, channel, and temporal dimensions. Notably, our approach adds only a limited number of parameters for the interaction layer, while keeping the original parameters of the large-scale pre-trained models frozen.

Extensive experimental results on four audio-visual tasks, namely, A VE, A VVP, A VQA, and AVS, demonstrate the superiority of our model compared to state-of-the-art counterparts across various settings. Furthermore, we evaluate the performance of DG-SCT in few-shot and zero-shot scenarios on the AVE and LLP datasets, demonstrating its superiority over CLIP and several competitive CLIP-Adapters.

2 Related work

2.1 Audio-visual understanding

Audio-visual understanding tasks involve utilizing both audio and visual modalities to get a better perception of audio-visual scenarios [8, 22, 41, 31]. For instance, Audio-Visual Event Localization (AVE [33]) requires models to recognize joint audio-visual events. Previous works [33, 16, 35, 37, 36] use late interaction strategies to better leverage the visual and audio features encoded from modality-specific pre-trained models. Audio-Visual Video Parsing (AVVP [32]) task breaks the restriction that audio and visual signals are definitely aligned. To tackle the weakly-supervised A VVP task, previous work [32] proposes a hybrid attention network and attentive Multimodal Multiple Instance Learning (MMIL) Pooling mechanism to aggregate all features. The task of Audio-Visual Segmentation (AVS [42]) focuses on whether each pixel corresponds to the given audio so that a mask of the sounding object(s) is generated. Zhou et al. [43] use a temporal pixel-wise audiovisual interaction module to inject audio semantics as guidance for the visual segmentation process. Furthermore, the newly introduced Audio-Visual Question Answering (AVQA [14]) task requires methods that perceive both audio and visual modalities to answer human-generated questions about the audio-visual content. Li et al. propose a spatiotemporal grounding model [14] to achieve scene understanding and reasoning over audio and visual modalities.

However, most methods designed for these tasks rely on modality-specific audio and visual pretrained models, which can not utilize multi-modal cues early in the representation stage. In this paper, we propose a novel early-interaction strategy, adaptively extracting key information from the current modality across spatial-channel-temporal dimensions.

2.2 Vision-language models and prompt learning

Vision-language models have made remarkable progress since the introduction of CLIP [26], with zero-shot and few-shot ideas achieving excellent generalization abilities in many downstream tasks; Meanwhile, prompt , a concept in NLP [15, 13, 19], has achieved impressive results in various NLP domains since its introduction, as evidenced by the success of the GPT series [25, 1]. Subsequent works have attempted to combine these two and achieved better results. For example, CoOp [46] improves the CLIP model by optimizing the continuous prompts in the language branch, and CoCoOp [45] further improves the model by incorporating prompts in the video branch. However, these works only utilize prompts to guide individual branches. CLIP-adapter [7] builds on these works by proposing to use embedding of video and language to guide each other at the end of the encoder. MaPLe [12] is the first to use an adaptor to guide each other inside the encoder, integrating visual and text representations with the semantics of each other to enhance the generalization ability of the model. However, none of these works consider utilizing prompts in the audio-visual domain.

In this paper, we introduce a novel bidirectional prompt that employs audio and video cues independent of text to achieve outstanding information extraction abilities for audio-visual tasks.

3 Approach

In this section, more details about our proposed DG-SCT are elaborated. An overview of the proposed framework is illustrated in Fig. 2.

3.1 Representations for audio-visual modalities

Visual representation Given an input video sequence, we first sample a fixed number of RGB video frames { V t } T t =1 ∈ R T × H × W × 3 , where H, W are height and width. Following the Swin-T [20], we first split each RGB frame V t into non-overlapping patches by a patch-splitting module with kernel size ( P v × P v ) . Each patch is treated as a "token" and its feature is set as a concatenation of the raw pixel values. A linear embedding layer is then applied to this raw-valued feature and we can get visual features as v t ∈ R H Pv × W Pv × C v , where C v is the number of visual channels.

Audio representation Given an input audio track, we first get an audio mel-spectrogram { A t } T t =1 , where A t ∈ R L × F with time L and frequency bins F . Following the HTS-AT [3], 3 the audio mel-spectrogram is cut into different patch tokens with a Patch-Embed CNN of kernel size ( P a × P a ) . A linear embedding layer is then applied to this raw-valued feature and we can obtain audio features as a t ∈ R L Pa × F Pa × C a , where C a is the number of audio channels.

3.2 Adding DG-SCT modules to frozen encoders

Now, we describe how we adjust pre-trained Swin-T and HTS-AT with our proposed DG-SCT. Every layer of the Swin-T and HTS-AT consists of three main operations: 1) multi-head attention (MHA), 2) multi-layer perceptron (MLP), and 3) our DG-SCT modules which use the intermediate layer information of audio and video as prompts to guide each other through spatial-channel-temporal dimensions. We skip the linear normalization layers in both MHA and MLP operations for brevity.

Given audio inputs a ( ℓ ) ∈ R T × ( L ( ℓ ) · F ( ℓ ) ) × C ( ℓ ) a and visual inputs v ( ℓ ) ∈ R T × ( H ( ℓ ) · W ( ℓ ) ) × C ( ℓ ) v from layer ℓ , we first use a two-dimensional convolution kernel and a linear projection to make the dimensions of the audio and visual prompts consistent of their counterpart modality. Let v ( ℓ ) f = Ω a2v ( a ( ℓ ) , v ( ℓ ) ) and a ( ℓ ) f = Ω v2a ( v ( ℓ ) , a ( ℓ ) ) denote the operation that implements DG-SCT module, which we will describe in the next subsection. Then, the operations in each layer can be written as:

3.3 Dual-guided spatial-channel-temporal attention

In this subsection, we will describe how DG-SCT works in more detail. Given visual and audio features, the encoder such as Swin-Transformer pre-trained on large-scale single-modal data, will uniformly extract features from audio-visual inputs (See Fig. 1 (a)). However, in practical multi-modal scenarios, not all of this information carries equal importance. For example, as we see in Fig. 1, the region where the car appears in the visual field is evidently more crucial than the background trees. Additionally, the moment when the engine sound emerges in the audio should also be given more attention. Hence, we take advantage of the fact that audio-visual pairs can provide mutual guidance for each other, and utilize different modalities as prompts to help pre-trained models focus more on specific aspects of opposite modal inputs across spatial, channel, and temporal dimensions. Different from previous works [37, 36] which only leverage audio as guidance to extract visual features, our proposed DG-SCT module can achieve triple levels of information highlighting in two directions. We illustrate these cross-modal attention mechanisms in the following parts:

Channel-wise attention: Different channels represent different aspects of features. The introduction of channel attention can facilitate the model to ignore irrelevant features and improve the quality of representations [11]. We let the audio and video serve as mutual guidance signals and explicitly model the dependencies between channels on each other's modality. Concretely, We use ψ a and ψ v to denote the combinations of convolutional and linear projection in section 3.2, to encode audio and visual inputs as prompts: a ′ t = ψ a ( a t ) ∈ R C v × ( H · W ) and v ′ t = ψ v ( v t ) ∈ R C a × ( L · F ) , respectively. For audio-to-visual (A2V), we use the spatial average pooling to process a ′ t and get a ′′ t ∈ R C v × 1 , then fuse it with vision via element-wise multiplication after feeding them to projection layers Θ c a , Θ c v ∈ R C v × C v respectively, generating audio channel-guidance maps a cm t = (Θ c a ( a ′′ t ) ⊙ Θ c v ( v t )) ∈ R C v × ( H · W ) . After that, we spatially squeeze the fused feature by global average pooling, denoted as δ a , Finally, a bottleneck layer Φ a follows with a sigmoid function σ is used, yielding channel attention maps M vc t ; Similarly, we generate V2A channel attentive maps M ac t :

where v cm t ∈ R C a × ( L · F ) is the visual channel-guidance maps, δ v is spatial-wise global average pooling, Φ v indicates a bottleneck layer and σ denotes the sigmoid function.

Spatial-wise attention: Audio can improve visual feature extraction by contributing to visual attention in the spatial dimension [33]. Inspired by this, we leverage the guidance capabilities of audio and visual prompts to guide visual spatial attention and audio frequency attention, respectively. Similar to the aforementioned channel-wise attention, For A2V, we first get channel-attentive visual features v c t = ( M vc t + 1) ⊙ v t , then we element-wise multiply audio prompt and v c t after the projection of Θ s a and Θ s v to hidden dimension d = 256 , generating audio spatial-guidance maps a sm t = (Θ s a ( a ′ t ) ⊙ Θ s v ( v c t )) ∈ R d × ( H · W ) . Then we use a projection layer Θ s ∈ R 1 × d with a sigmoid function σ to obtain spatial attention maps M vs t ; Similarly, we generate V2A frequency attentive maps M af t :

where v fm t denotes visual frequency-guidance maps, Θ f ∈ R 1 × d is a projection layer.

Temporal-gated attention: Given an audio, significant time segments (e.g., 'engine sound') should be emphasized, while background information (e.g., 'silence') should be attenuated. The same holds for the visual information as well [36]. Inspired by this, we add temporal-gated attention in the final layer. For A2V, we first feed the frequency-channel attentive audio features { a cf t } T t =1 through an RNN to capture temporal information better and then pass it through a projection layer with

sigmoid function to obtain the temporal attention gates G v ∈ R T × 1 ; Similarly, for V2A, we feed the spatial-channel attentive visual features to generate G a ∈ R T × 1 :

Summary: We combine the abovementioned three attention mechanisms together to form our DGSCT module. Given audio features a t ∈ R C a × ( L · F ) and visual features v t ∈ R C v × ( H · W ) , DG-SCT first generates channel-wise attention maps M vc t and M ac t to let audio and video adaptively emphasize informative features of the corresponding modality. It then lets audio pay attention to the important sounding regions to produce spatial-wise attention maps M vs t and lets video pay attention to the important frequency regions to generate frequency-wise attention maps M af t , thus the yielding of the spatial-channel attentive visual features v cs t and the frequency-channel attentive audio features a cf t can be summarized as:

Then we generate two temporal attention gates G v ∈ R T × 1 and G a ∈ R T × 1 for { v cs t } T t =1 and { a cf t } T t =1 , respectively, thus the final outputs of Ω a2v and Ω v2a mentioned in section 3.2 are:

Where α , β , and γ are hyperparameters. Consequently, this approach yields high-quality, fine-grained audio-visual representations, significantly improving the performance of subsequent tasks in the audio-visual domain.

4 Experiments

4.1 Tasks and datasets

Audio-visual event localization (A VE) focuses on recognizing an audio-visual event that is both visible and audible throughout multiple time segments in a video. We evaluate the A VE [33] dataset; Audio-visual video parsing (AVVP) aims to parse a video into temporal event segments and label them as either audible, visible, or both. We evaluate our method for weakly-supervised A VVP task on the LLP dataset [32]; The goal of Audio-visual segmentation (AVS) is to output a pixel-level map of the object(s) that produce sound at the image frame. We evaluate on A VSBench [43]; Audio-visual question answering (AVQA) aims to answer questions based on the associations between objects and sounds. We conduct our experiments on the MUSIC-AVQA dataset [14].

Meanwhile, We propose Audio-visual few-shot/zero-shot tasks on AVE [33] and LLP [32] datasets. We evaluate AVE and classification tasks on AVE dataset and classification task on LLP dataset. More details about tasks and datasets will be illustrated in Appendix.

4.2 Implementation details

Audio-visual downstream tasks : To adapt our approach to the four audio-visual downstream tasks, we replace the pre-trained audio and visual encoders with a frozen HTS-AT [3] and a frozen SwinTransformer [20], respectively. The trainable DG-SCT modules in Fig. 2 (4) are injected into the frozen layers to let audio and visual modalities guide each other. We then use this as our audio-visual feature extractor. For AVE task, our feature extractor is combined with CMBS [36]. The event category label of each video segments is required to be predicted in supervised manner. We adopt [33, 38, 37, 36] and exploit the overall accuracy of the predicted event category as the evaluation metrics; Combined with MGN [24], DG-SCT is able to tackle the weakly-supervised AVVP task. Following [32], we evaluate the parsing performance of all events (audio, visual, and audio-visual events) under segment-level and event-level metrics; For AVS task, We combine our audio-visual feature extractor with the original A VS model [43]. We use the Jaccard index J [6] and F-score F

as the evaluation metrics; For AVQA task, our audio-visual feature extractor is used in the original ST-AVQA [14]. Similarly, we use the answer prediction accuracy as the evaluation metrics.

Audio-visual few-shot/zero-shot tasks : We incorporate DG-SCT modules as adapters between the frozen CLIP image encoder ViT [26] and frozen CLAP audio encoder HTS-AT [5], generating audio and visual features for text-audio and text-image contrastive learning, as shown in Fig. 2 (e). For the zero-shot setting, the model is pre-trained on the VGG-Sound(40K) [2] dataset. More details will be discussed in Appendix.

4.3 Compared with state-of-the-arts on audio-visual downstream tasks

First, we challenge our method against current state-of-the-art methods on the four audio-visual tasks. As demonstrated in Table 1, our model outperforms CMBS [36] and LAVisH [17] by a significant margin ( 2.9% and 2.5% ); In Table 2, our model attains either a competitive or even better performance. For instance, DG-SCT surpasses MGN by 3.9% points in the segment-level visual event parsing metric, demonstrating our method can utilize large-scale models to extract more useful features and further promote the fusion of these features in downstream tasks. We also achieve state-of-the-art results on S4 setting of A VS task (Table 3). Lastly, our model performs exceptionally well on AVQA task and outperforms previous leading methods on AQ, VQ, and AVQ question types, respectively. The experimental results reveal that our model has the capability of utilizing pre-trained audio and visual models to extract more comprehensive, higher-quality features tailored for downstream tasks than the cross-attention mechanism of LAVisH. Moreover, our model exhibits excellent generalization abilities, achieving impressive results across various audio-visual tasks.

4.4 Compared with state-of-the-arts on audio-visual few-shot/zero-shot tasks

Furthermore, as presented in Fig. 4, our DG-SCT model surpasses top-performing methods like CoCoOp, CLIP-Adapter, and MaPLe by a non-negligible margin on our newly proposed audio-visual few-shot/zero-shot scenarios. Our few-shot (shot= 16 ) learning for the AVE task attains 72.4% ,

outperforming MaPLe for 5.3% points, and is even comparable to previous full-training baselines (Table 1). Note that experimental results for few-shot on the LLP classification task are worse than zero-shot scenario. We argue that this is due to a significant gap between downstream and pre-training data, which may have disrupted the original parameters of the CLIP model. However, we still outperform MaPLe by 2.2% and 0.7% points, for zero-shot and few-shot (shot= 16 ) settings, respectively. The outcomes demonstrate that our dual-guided audio-visual prompts extract taskspecific information better than text-image [26, 46, 45, 7, 12] and text-audio [5] pre-trained models and exhibit stronger adaptability on audio-visual tasks.

4.5 Ablation analysis

We verify the effectiveness of the three modules, namely, S (spatial), C (channel), and T (temporal), in DG-SCT . As shown in Table 5, compared to the baseline model, the individual incorporation of these attention modules leads to substantial performance enhancements of our model across a range of downstream tasks, demonstrating the effectiveness of these modules. Although cross-modal interactions in any dimension can enhance feature extraction, we argue that distinct modules can also mutually guide and enhance each other's performance. For instance, removing the T module individually does not result in a significant decrease in the performance of the model, which indicates that the combination of S and C can slightly substitute the function of T . The reason might be that if the features of a certain segment are more prominent in both spatial and channel dimensions, it is more likely to be important in the temporal dimension as well.

Flute

Speak

Cat

Violin

Truck

Original

Spatial

-attn

Violin

Guitar

Speech

Meow

Clip-clop

Car

Original

mask

(AVE)

(AVS)

72.4

84.2

32.6

64.7

76.6

35.8

accuracy(%)

72

70

66

64

62

68

MaPLe

Ours

CoOp

CLIP-Adapter

CoCoOp

AVE

accuracy(%)

82

84

08

74

78

76

AVE classification

doo

CoCoOp

accuracy(%)

32

31

30

29

LLP classification

Ours

CoOp

CoCoOp

accuracy(%)

80

70

60

50

30

20

40

Zero shot

CLIP

CLAP

CoOp

MaPLe

Ours

CoCoOp

CLIP-Adapter

60

72

MaPLe

CLIP-Adapter

28

CLIP-Adapter

MaPLe

10

0

2

4

6

shot

8

10

12

14

16

70

0

2

4

6

shot

8

10

12

14

16

0

2

4

6

shot

8

10

12

14

16

0

AVE

classification

AVE

classification

LLP

4.6 Qualitative analysis

Fig. 3 represents examples of the effectiveness of our DG-SCT. On the left, we observe that with the guidance of the audio prompts, the module can accurately focus on the critical visual regions for different A VE events. For instance, when multiple objects are present in video frames, our model is capable of accurately localizing the sounding objects (e.g., bark, cat, the second example of flute, the fifth example of speak). Moreover, our model can precisely pinpoint the sound source location at a fine-grained level, such as the strings of a violin and the hands of the performer, and the speaker's mouth. In addition, DG-SCT achieves excellent results on AVS task. The right part of Fig. 3 indicates that our model can accurately segment the pixels of sounding objects and outline their shapes perfectly. The excellent qualitative results of our model on various downstream tasks illustrate its strong potential for generalization.

As depicted in Fig. 5, we employ t-SNE [34] to visualize the learned audio and visual features and compare features generated by our model with features generated by baseline without DG-SCT, on various tasks. Each spot denotes the feature of one audio or visual event, and each color corresponds to a particular category, such as "cat" in orange as shown in Fig. 5 (A VE). As we can see, features extracted by the proposed DG-SCT are more intra-class compact and more inter-class separable. These meaningful visualizations further demonstrate that the DG-SCT model successfully learns compact and discriminative features for each modality across diverse downstream tasks.

4.7 Efficiency analysis

Our efficiency analysis on AVE task is presented in Table 6. Our approach utilizes more trainable parameters. However, our proposed DG-SCT attention mechanism requires a comparable number of parameters to the latent tokens utilized in LAVisH [17]. The increase trainable parameters primarily arises from the inclusion of a two-dimensional convolution kernel and a linear projection (section 3.2). These additions ensure consistency of dimensions between the audio and visual prompts. In other words, the increase in parameter count from our approach mainly results from addressing the inconsistency in dimensions between the audio and visual encoders.

We also conducted a comparison of computational cost [29] on AVE task. Our approach involves fine-tuning the pre-trained model, which inevitably leads to a reduction in speed compared to previous late-interaction baselines (CMBS [36]). However, we have achieved outstanding results, and our approach is applicable to multiple audio-visual tasks. Overall, the benefits are substantial.

5 Conclusion and discussion

This paper introduces DG-SCT, a method that leverages audio and visual modalities as prompts in the early layers of frozen pre-trained encoders. By doing so, our model can extract higher-quality

Church bell

Male speech

Airplane

Race car

Female speech

Helicopter

Violin

Flute

Ukulele

Frying (food)

Truck

Motorcycle

Acoustic guitar

Train horn

Clock

Banjo

Goat

Baby cry

Bus

Chainsaw

Cat

Horse

Toilet flush

Rodents

Accordion

Speech

Car

Cheering

Dog

Cat

Frying(food)

Basketball bounce

Fire alarm

Chainsaw

Cello

Banjo

Singing

Chicken rooster

Violin fiddle

Vacuum cleaner

Baby laughter

Accordion

Lawn mower

Motorcycle

Helicopter

Acoustic guitar

Telephone bell

Baby cry

Blender

Clapping

Helicopter

Bird singing

typing

Playing violin

Playing glockenspiel

Playing piano

Lions roaring

Baby laughter

Male speech

Lawn mowing

Playing ukulele

Playing tabla

Driving buses

Cap gun shooting

Chainsawing trees

Playing acoustic guitar

Cat meowing

Female singing

Ambulance siren

Dog barking

Horse clip-clop

Coyote howling

Race car

(AVE)

(AVVP)

(AVS)

Original

Ours

Original

Ours

Audio

Visual

and finer-grained audio-visual features, enhancing performance in subsequent tasks. We conduct comprehensive experiments on four datasets, including AVE, AVVP, AVS, and AVQA, as well as our newly proposed few-shot/zero-shot audio-visual tasks. Across 25 experimental settings, our approach achieves state-of-the-art results on 19 of them. Additionally, ablation studies conducted on these datasets validate the effectiveness of our proposed spatial, channel, and temporal attention modules. Furthermore, our approach demonstrates robust generalizability and holds potential for application in more audio-visual scenarios in the future.

References

Tom Brown et al. 'Language models are few-shot learners'. In: Advances in neural information processing systems 33 (2020), pp. 1877-1901.

Honglie Chen et al. 'Vggsound: A large-scale audio-visual dataset'. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. 2020, pp. 721-725.

Ke Chen et al. 'HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection'. In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. 2022, pp. 646-650.

Alexey Dosovitskiy et al. 'An image is worth 16x16 words: Transformers for image recognition at scale'. In: arXiv preprint arXiv:2010.11929 (2020).

Benjamin Elizalde et al. 'Clap: Learning audio concepts from natural language supervision'. In: arXiv preprint arXiv:2206.04769 (2022).

Mark Everingham et al. 'The pascal visual object classes (voc) challenge'. In: International journal of computer vision 88 (2010), pp. 303-338.

Peng Gao et al. 'Clip-adapter: Better vision-language models with feature adapters'. In: arXiv preprint arXiv:2110.04544 (2021).

Ruohan Gao and Kristen Grauman. 'Visualvoice: Audio-visual speech separation with crossmodal consistency'. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE. 2021, pp. 15490-15500.

Jort F Gemmeke et al. 'Audio set: An ontology and human-labeled dataset for audio events'. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE. 2017, pp. 776-780.

Andrey Guzhov et al. 'Audioclip: Extending clip to image, text and audio'. In: ICASSP 20222022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. 2022, pp. 976-980.

Jie Hu, Li Shen, and Gang Sun. 'Squeeze-and-excitation networks'. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2018, pp. 7132-7141.

Muhammad Uzair Khattak et al. 'Maple: Multi-modal prompt learning'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2023, pp. 19113-19122.

Brian Lester, Rami Al-Rfou, and Noah Constant. 'The power of scale for parameter-efficient prompt tuning'. In: arXiv preprint arXiv:2104.08691 (2021).

Guangyao Li et al. 'Learning to answer questions in dynamic audio-visual scenarios'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 19108-19118.

Xiang Lisa Li and Percy Liang. 'Prefix-tuning: Optimizing continuous prompts for generation'. In: arXiv preprint arXiv:2101.00190 (2021).

Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang. 'Dual-modality seq2seq network for audio-visual event localization'. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. 2019, pp. 2002-2006.

Yan-Bo Lin et al. 'Vision Transformers Are Parameter-Efficient Audio-Visual Learners'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . June 2023, pp. 2299-2309.

Xiao Liu et al. 'GPT understands, too'. In: arXiv preprint arXiv:2103.10385 (2021).

Xiao Liu et al. 'P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks'. In: arXiv preprint arXiv:2110.07602 (2021).

Ze Liu et al. 'Swin transformer: Hierarchical vision transformer using shifted windows'. In: Proceedings of the IEEE/CVF international conference on computer vision . 2021, pp. 1001210022.

Yan Luo et al. 'Timestamps as Prompts for Geography-Aware Location Recommendation'. In: arXiv preprint arXiv:2304.04151 (2023).

Shuang Ma et al. 'Active contrastive learning of audio-visual video representations'. In: arXiv preprint arXiv:2009.09805 (2020).

Tanvir Mahmud and Diana Marculescu. 'AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization'. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . 2023, pp. 5158-5167.

Shentong Mo and Yapeng Tian. 'Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing'. In: Advances in Neural Information Processing Systems . 2022.

Alec Radford et al. 'Language models are unsupervised multitask learners'. In: OpenAI blog 1.8 (2019), p. 9.

Alec Radford et al. 'Learning transferable visual models from natural language supervision'. In: International conference on machine learning . PMLR. 2021, pp. 8748-8763.

Yongming Rao et al. 'Denseclip: Language-guided dense prediction with context-aware prompting'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 18082-18091.

Idan Schwartz, Alexander G Schwing, and Tamir Hazan. 'A simple baseline for audio-visual scene-aware dialog'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2019, pp. 12548-12558.

Liangcai Su et al. 'Beyond Two-Tower Matching: Learning Sparse Retrievable CrossInteractions for Recommendation'. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 2023, pp. 548-557.

Weng Lam Tam et al. 'Parameter-efficient prompt tuning makes generalized and calibrated neural text retrievers'. In: arXiv preprint arXiv:2207.07087 (2022).

Yapeng Tian, Di Hu, and Chenliang Xu. 'Cyclic co-learning of sounding object visual grounding and sound separation'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2021, pp. 2745-2754.

Yapeng Tian, Dingzeyu Li, and Chenliang Xu. 'Unified multisensory perception: Weaklysupervised audio-visual video parsing'. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 . Springer. 2020, pp. 436-454.

Yapeng Tian et al. 'Audio-visual event localization in unconstrained videos'. In: Proceedings of the European Conference on Computer Vision (ECCV) . 2018, pp. 247-263.

Laurens Van der Maaten and Geoffrey Hinton. 'Visualizing data using t-SNE.' In: Journal of machine learning research 9.11 (2008).

Yu Wu et al. 'Dual attention matching for audio-visual event localization'. In: Proceedings of the IEEE/CVF international conference on computer vision . 2019, pp. 6292-6300.

Yan Xia and Zhou Zhao. 'Cross-modal background suppression for audio-visual event localization'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 19989-19998.

Haoming Xu et al. 'Cross-modal relation-aware networks for audio-visual event localization'. In: Proceedings of the 28th ACM International Conference on Multimedia . 2020, pp. 38933901.

Hanyu Xuan et al. 'Cross-modal attention network for temporal inconsistent audio-visual event localization'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 34. 01. 2020, pp. 279-286.

Runyi Yu et al. 'LaPE: Layer-adaptive Position Embedding for Vision Transformers with Independent Layer Normalization'. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) . Oct. 2023, pp. 5886-5896.

Heeseung Yun et al. 'Pano-avqa: Grounded audio-visual question answering on 360deg videos'. In: Proceedings of the IEEE/CVF International Conference on Computer Vision . 2021, pp. 2031-2041.

Zhaoyang Zeng, Daniel McDuff, Yale Song, et al. 'Contrastive learning of global and local video representations'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 7025-7040.

Jinxing Zhou et al. 'Audio-Visual Segmentation'. In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII . Springer. 2022, pp. 386-403.

Jinxing Zhou et al. 'A VSBench: A Pixel-level Audio- Visual Segmentation Benchmark'. In: ().

Jinxing Zhou et al. 'Positive sample propagation along the audio-visual event line'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2021, pp. 8436-8444.

Kaiyang Zhou et al. 'Conditional prompt learning for vision-language models'. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022, pp. 16816-16825.

Kaiyang Zhou et al. 'Learning to prompt for vision-language models'. In: International Journal of Computer Vision 130.9 (2022), pp. 2337-2348.

A Implementation details

For the AVE and the AVQA tasks, we set α = 0 . 3 , β = 0 . 05 , and γ = 0 . 1 . We train the model with a batch size of 8 and a learning rate of 5 × 10 -4 and 1 × 10 -4 , respectively; For the AVVP task, we set α = 0 . 3 , β = 0 . 05 , and γ = 0 . 05 . We train the model with a batch size of 8 and a learning rate of 3 × 10 -4 ; For the S4 setting of the AVS task, we set α = 0 . 3 , β = 0 . 05 , and γ = 0 . 05 . We train the model with a batch size of 8 and a learning rate of 3 × 10 -4 ; For the MS3 setting of the AVS task, we set α = 0 . 2 , β = 0 . 1 , and γ = 0 . 1 . We train the model with a batch size of 2 and a learning rate of 1 . 5 × 10 -4 .

For few-shot/zero-shot tasks , we set the learning rate to 3 × 10 -4 with a batch size of 2 . For the AVE task, we set α = 0 . 2 , β = 0 . 05 , and γ = 0 . 01 ; For the AVE classification and the LLP classification tasks, we set α = 0 . 2 , β = 0 . 05 , and γ = 0 . 05 .

All of our experiments are trained on one NVIDIA A100 GPU.

B More details of datasets

AVE dataset 5 We evaluate the AVE task on the AVE dataset [33] originating from the AudioSet [9]. The AVE dataset contains 4 , 143 videos covering 28 categories. Each video lasts for 10 seconds and contains an event category labeled for each video on a segment level.

LLP dataset 6 We evaluate the AVVP task on the LLP dataset [32], which has 11 , 849 video-level event annotations on the presence or absence of different video events and each video is 10 s long and has at least 1 s audio or visual events. There are 7 , 202 videos that contain events from more than one event category and per video has averaged 1 . 64 different event categories. For evaluation, there are 4 , 131 audio events, 2 , 495 visual events, and 2 , 488 audio-visual events for the 1 , 849 videos.

AVSBench dataset 7 We evaluate the AVS task on the AVSBench dataset [43], which contains two settings: 1) Single Sound Source Segmentation (S4), which contains 4 , 932 videos over 23 categories; 2) Multiple Sound Source Segmentation (MS3), which contains 424 videos over 23 categories. Each video is 5 seconds long.

MUSIC-AVQA dataset 8 We conduct our experiments of the AVQA task on the MUSIC-AVQA dataset [14], which contains 9 , 288 videos, 45 , 867 question-answer pairs, 33 question templates, and 42 answers.

VGG-Sound(40K) 9 The pre-training data of our zero-shot task is VGG-Sound(40K), which is split from the VGG-Sound dataset [2]. VGG-Sound contains over 200 k clips for 300 different sound classes, while our VGG-Sound(40K) has 40 , 801 clips for 141 sound classes.

[CLASS]

.

a photo of a

CLIP text-branch

DG-SCT

ViT Block

HTS-AT Block

ViT Block

HTS-AT Block

+

+

…

…

DG-SCT

+

+

Self-Attention Block

in CLIP & CLAP

𝐸 𝑣

⋅ 𝑇

1

𝑣

𝐸 𝑣

⋅ 𝑇

2

𝑣

…

𝐸 𝑣

⋅ 𝑇 𝑁

𝑣

𝐸 𝑣

𝑇 1

𝑣

𝑇 2

𝑣

…

𝑇 𝑁

𝑣

Visual Input

CLAP text-branch

[CLASS]

.

the sound of

𝐸 𝑎

⋅ 𝑇

1

𝑎

𝐸 𝑎

⋅ 𝑇

2

𝑎

…

𝐸 𝑎

⋅ 𝑇 𝑁

𝑎

𝐸 𝑎

𝑇 1

𝑎

𝑇 2

𝑎

…

𝑇 𝑁

𝑎



1

w



2

w

'Train horn'

Trainable

Frozen

⋅

A2V

V2A

⋅

t

v

t

a

Channel

attention

Spatial

attention

⋅

⋅

c

t

v

c

t

a

…

…

Temporal attention

⋅

⋅

1

{

}

cs

T

t

t

v

=

1

{

}

cf

T

t

t

a

=

v

G

a

G

1

{

}

cft

T

t

t

a

=

1

{

}

cst

T

t

t

v

=

DG-SCT

T

T

v

s

t

M

f

t

a

M

ac

t

M

vc

t

M

装

C More details of few-shot/zero-shot tasks

C.1 Framework and method

As shown in Fig. 6, our DG-SCT module is injected into every layer of the frozen CLIP [26] image encoder and the frozen CLAP [5] audio encoder. After that, the image encoder and the audio encoder are denoted as f v ( · ) and f a ( · ) , respectively. Two text branches f v t ( · ) , f a t ( · ) are employed to perform image-text and audio-text matching, respectively, as the text encoders from different pre-trained models exhibit a substantial gap, using only one of them will result in a significant decrease in accuracy.

Let ( v i , a i , t i ) represents a piece of visual-audio-text pair indexed by i . The visual embedding E v i , the audio embedding E a i , and the corresponding text embeddings T v i and T a i , are obtained by encoders with projection layers, respectively:

where the visual/audio/text projection layer is a 2-layer multilayer perception (MLP) with ReLU as the activation function to map the encoder outputs into the same dimension D (i.e., E v i , E a i , T v i , T a i ∈ R D ).

The model is trained with the contrastive learning paradigm between the paired visual and text embeddings, as well as the paired audio and text embeddings, following the same loss function in [26]:

Where τ v and τ a are learnable temperature parameters for scaling the loss. N is usually the number of data, but during the training phase, N is used as the batch size. Let y v denotes the image-text

72.4

84.2

32.6

64.7

76.6

35.8

accuracy(%)

72

70

66

64

62

68

MaPLe

Ours

CoOp

CoCoOp

CLIP-Adapter

AVE

accuracy(%)

8

82

80

74

78

76

AVE classification

sno

CoOp

CoCoOp

accuracy(%)

32

31

29

30

LLP classification

Ours

CoOp

CoCoOp

accuracy(%)

80

70

60

50

20

40

30

Zero shot

CLIP

CLAP

CoOp

MaPLe

Ours

CoCoOp

CLIP-Adapter

60

0

2

4

6

shot

8

10

12

14

16

72

70

0

2

4

6

shot

8

10

12

MaPLe

CLIP-Adapter

14

16

28

0

2

4

6

shot

8

10

12

CLIP-Adapter

MaPLe

14

16

10

0

AVE

classification

AVE

classification

LLP

matching score, and y a denotes the audio-text matching score, we set hyperparameters w 1 = y v y v + y a and w 2 = 1 -w 1 , thus the overall objective function is:

When the image-text matching score is higher, it indicates that the visual modality is more certain in determining the category of the sample, therefore the model needs to increase the weight of visual modality in the loss function, and vice versa.

The same weight settings for w 1 and w 2 in Fig. 6 are also significant during the inference process, indicating that some samples require more guidance from the audio modality while others might require more guidance from the visual modality.

For the few-shot setting, we train the model by selecting shot samples of each category from the training dataset of the downstream tasks. For the zero-shot setting, we pre-train the model using the VGG-Sound(40K) dataset.

C.2 Evaluation metrics

For the AVE task, the category label of each video segment is required to be predicted in supervised manner. We adopt [33, 37, 36], and exploit the overall accuracy of the predicted event category as the evaluation metrics; For the AVE classification task, the category label of each video is required to be predicted. We leverage the overall accuracy of the video category as the evaluation metrics; For the LLP classification task, the category label of each video is required to be predicted. Note that some videos might have multiple categories, we randomly select one as ground truth for brevity. We leverage the overall accuracy of the video category as the evaluation metrics.

C.3 Results

As shown in Fig. 7 (a), (b) and (c), for few-shot ( shot = 16 ) setting, our method outperforms MaPLe by a significant margin on AVE, AVE classification, and LLP classification tasks, achieving 72.4% , 84.2% , and 32.6% , respectively. However, the performance of our model on both the AVE and LLP classification tasks may not be as effective as some previous methods when shot < 8 . The classification tasks are relatively simple and are already closely related to the image-text matching task in the upstream pre-training of the CLIP model. Therefore, the CLIP model's pre-trained parameters can achieve good results. However, when shot > 8 , our model gains significant advantages. We also observe that the accuracy curves of other few-shot learning methods are relatively flat, particularly for

(

)

b



(

)

c



(

)

a



0.05,

0.1





=

=

0.3,

0.1





=

=

0.3,

0.05





=

=

82.2

82.2

82.2

82.0

1

82.0

82.0

acy(

ccurac

ac

81.8

81.6

81.2

81.4

-

ccura

81.8

81.6

81.4

81.2

81.0

cu

81.8

81.6

81.4

81.2

81.0

80.8

81.0

80.8

80.6

0.1

0.2

0.3

0.4

0.5

0

0.05

0.1

0.15

0.2

0.05

0.1

0.15

0.2

0.25

0.3

the LLP classification task, where the accuracy sometimes decreases when shot increases. This issue might be the limited ability of the image-text models to extract rich information from the training data of audio-visual tasks. Thus, the potential for improving accuracy is limited. For zero-shot setting, our model also achieves state-of-the-art on all three tasks. The results demonstrate that our model can extract audio-visual task information more effectively and achieve better results.

D More ablation studies

Next, we conduct more ablation studies to investigate how different components of our model affect the performance on the downstream tasks.

A2V and V2A modules. We first investigate the effectiveness of the bidirectional attention mechanism. We compare our final bidirectional approach with the unidirectional variants that only use either audio-to-visual (A2V) or visual-to-audio (V2A) spatial-channel-temporal attention mechanism, and also a baseline that does not use any cross-modal connections.

As indicated in Table 8, integrating A2V or V2A individually leads to substantial performance enhancements across AVE, AVS, and AVQA tasks compared to the baseline model. Furthermore, the bidirectional DG-SCT outperforms the unidirectional A2V and V2A baselines (e.g., 3 . 0% and 0 . 9% on the AVE task). In Table 9, we observe that using the A2V module alone does not significantly decrease the accuracy for visual and audio-visual events. However, without visual guidance (V2A), the performance of audio events suffers a considerable decline; Likewise, the performance of visual events drops without audio guidance (using the V2A module alone). These experimental findings demonstrate the necessity of visual guidance for audio events and the need for audio guidance for visual events. Our proposed DG-SCT model can bidirectionally guide the representation of each modality, thus enhancing the accuracy of downstream audio-visual tasks.

Hyperparameters. Now, we explore the impact of the hyperparameters α , β , and γ on the experimental results of the A VE task. As we can see in Fig. 8, our model achieves the best accuracy ( 82.2% ) when α = 0 . 3 , β = 0 . 05 , and γ = 0 . 1 .

Audio

Raw

image

Ground

truth

DG-SCT

w/o

DG-SCT

Baby laughter

Gun shooting

Audio

Raw

image

Ground

truth

DG-SCT

w/o

DG-SCT

Violin

Dog barking

Baby laughter Baby laughter

Baby laughter

Baby laughter

Gun shooting

Gun shooting

Gun shooting

Gun shooting

Violin

Violin

Violin

Violin

Dog barking

Dog barking

Dog barking

Dog barking

5

E Additional qualitative analyses

Qualitative examples of the A VS task. We provide some qualitative examples of the AVS task to test the effectiveness of the DG-SCT model. As shown in Fig 9, under the S4 setting, the DG-SCT model can not only outline the object shapes more perfectly (shown in the upper left, upper right, and bottom left figures) but also locate the sounding objects more accurately than the baseline model without the DG-SCT module. As we can see in the bottom right figure of Fig. 9, DG-SCT can locate the dog that is barking, simultaneously excluding the mower that hasn't produced sound. The baseline model, however, mistakenly locates the dog and the mower at the same time. In Fig 10, under the MS3 setting, the DG-SCT model can locate multiple sound sources and outline their shape nicely. In some difficult cases (the bottom row in Fig. 10), DG-SCT can still pinpoint the correct sounding source even if the sound has stopped or the scene has changed dramatically.

Qualitative examples of the A VVP task. In Fig. 11, we visualize the video parsing results of DGSCT and baseline (w/o DG-SCT) on different examples. "GT" denotes the ground truth annotations. The results show that adding the DG-SCT module achieves more accurate parsing performance by acquiring mutual guidance from audio and visual modalities during the representation. For example, in Fig. 11 (b), our model can recognize the mixture of "banjo" and "singing" while the baseline model (w/o DG-SCT) falsely predicts the sound as "violin fiddle".

Audio

Raw

image

Ground

truth

DG-SCT

w/o

DG-SCT

Violin, piano

Piano, guitar

Lion roaring

Audio

Raw

image

Ground

truth

DG-SCT

w/o

DG-SCT

Violin, piano

Violin, piano

Violin, piano

Violin, piano

Violin

Lion roaring

Lion roaring

Nothing

Nothing

Violin, piano

Violin, piano

Violin, piano

Violin, piano

Piano, guitar

Piano, guitar

Piano, guitar

Piano, guitar

Visual

DG-SCT

w/o DG-SCT

GT

Singing

DG-SCT

w/o DG-SCT

GT

Speech

Aduio

DG-SCT

w/o DG-SCT

GT

Singing

DG-SCT

w/o DG-SCT

GT

Speech

Visual

DG-SCT

w/o DG-SCT

GT

Singing

DG-SCT

w/o DG-SCT

GT

Banjo

Aduio

DG-SCT

w/o DG-SCT

GT

Singing

DG-SCT

w/o DG-SCT

GT

Banjo

(a)

(b)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel dual-guided spatial-channel-temporal attention mechanism to audio-visual problems, which leverages pre-trained audio and visual encoders. And they show the improvement in various audio-visual tasks such as event localization, parsing, segmentation, and question answering.\n\nSTRENGTHS:\n- This work applied proposed methods and evaluated with diverse audio-visual downstream tasks, including localization, parsing, segmentation and question answering. This helps to showcase the generalization of proposed approach.\n\nWEAKNESSES:\n- The use of notations in Section 3.3 is very complicated and difficult to follow. Consider utilize Figure 2 (4) for illustration and walk through each step along with the modules in the figure to make it easier for the readers to follow.\n- For the results presented in Table 2 and 3, and discussed in Section 4.3, only the better performance is highlighted without providing potential explanation and/or hypothesis why proposed system performed worse in some scenarios. For example, the performance on event-level as in Table 2, and the performance with MS3 of segmentation in Table 3. Consider adding some discussion for these cases.\n- Some abbreviations are referred to without any information provided, such as LLP in line 210, and CMBS in line 218. Consider adding one-liner explanation to help the reader.\n- Minor comments: Section 3.2 in equation (1), the first superscript \"l\" does not follow the same style as others.\n\nQUESTIONS:\n- Do you need to use the same number of layers in the Swin-T and HTS-AT? Since they share the same notation in all the equations for both audio and visual paths.\n- In Section 3.1, line 127, where does T come from? Could you provide more detail on the choices of mel-spectrogram?\n\nLIMITATIONS:\n- No potential social or ethical implications.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a parameter-efficient approach, DG-SCT. DG-SCT can adapt pre-trained audio and visual models on downstream audio-visual tasks without updating pre-trained encoders (i.e., keep pre-trained encoders frozen.)\n\nSTRENGTHS:\n$+$ DG-SCT can achieve state-of-the-art results on several downstream audio-visual tasks.\n$+$ DG-SCT is able to learn channel-wise, spatial, and temporal information to incorporate with pre-trained audio and visual encoders.\n\nWEAKNESSES:\n$-$ Although DG-SCT leverages different types of cross-modal attention (i.e., channel-wise, spatial, and temporal), the design is not clear.\nFor example, DG-SCT leverages RNN for modeling cross-modal temporal information and learnable weights for cross-modal spatial information. Such a technique can be implemented with divided (space-time) attention after channel-wise attention.\n\n$-$ The effectiveness of the proposed temporal modeling is not clear. The baselines for the implementation of DG-SCT in AVE/AVVP/AVQA have already cross-modal temporal modeling modules.  \n\n$-$ The number of trainable parameters is not reported. For example, the baselines in AVVP and AVE usually use pre-extracted audio and visual features, which contribute to a lower number of trainable parameters.  \n\n$-$ Lack of efficiency comparison (e.g., FLOPs). Combining several attention mechanisms will lead to huge computational costs. It would be great to include these metrics.\n\n\n$-$ some experimental settings and results are not clear (See. Questions)\n\nQUESTIONS:\n* In AVE task, CMBS uses much weaker audio and visual encoders. How does CMBS with Swin-V2-L and HTS-AT work? Why does CMBS drop from 79.3 to 78.6 in Table 5?\n\n* In AVVP task, the results of MGN in Table 2 are lower than the numbers in the paper. Does DG-SCT use [A], which is also used in MGN? \n\n* Similar to the question in AVE task,  how does AVS with Swin-V2-L and HTS-AT work in AVS task? (the performance is also dropped in Table 5)\n\n* Are the baselines in zero/few-shot tasks also pre-trained in VGGSound (40K)? If yes, that would be fair to compare with DG-SCT.\n\n[A] Wu, Yu, and Yi Yang. \"Exploring heterogeneous clues for weakly-supervised audio-visual video parsing.\" CVPR 2021\n\nLIMITATIONS:\nYes, DG-SCT uses more number of trainable parameters", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new mechanism to utilize audio-visual features as novel prompts to extract task-specific features from large-scale models. This work introduces an attention mechanism named Dual-Guided Spatial-Channel-Temporal (DG-SCT), which utilizes audio and visual modalities to guide the feature extraction of their respective counterpart modalities across spatial, channel, and temporal dimensions. The proposed method is evaluated on a series of tasks including Audio-visual event localization, Audio-visual video parsing, Audio-visual segmentation, and Audio-visual question answering. Moreover, it proposes a new benchmark to perform Audio-visual few-shot/zero-shot tasks on AVE and LLP datasets.\n\nSTRENGTHS:\n- This is a very interesting work. The use of prompting is primarily focused on language and later vision, notably this work performs audio-visual prompting, which seems quite innovative.\n-  Moreover, unlike previous works that offer unidirectional prompts, the proposed approach introduces bidirectional prompts, where both visual and audio modalities can mutually guide each other in the feature extraction process.\n- The proposed approach is evaluated on several benchmarks and compared fairly with prior works showing the effectiveness of the proposed method.\n- It's a nicely written paper and easy to follow.\n\nWEAKNESSES:\nPlease see #Questions for more open-ended discussions.\n\nQUESTIONS:\n1. The proposed mechanism more resembles with Adapters than Prompts. Prompts usually work in the input space, while the proposed DG-SCT mechanism is also added in the intermediate layers. Moreover, prompting generally assumes no access to the models' internal architecture, but the proposed mechanism needs the models' access to modify/adjust some of the intermediate layers to connect the DG-SCT module. \n\n2. Often audio and video representations suffer from substantial domain gaps due to their modality-specific nature, did you encounter such scenarios?\n\n3. The proposed method is based on the Swin transformer, could you please elaborate if your method can be adapted to other Transformer architectures (e.g., other ViT variants)? If so, what necessary changes need to be done or is it a Swin-specific solution?\n\n4. In this study, the pretrained encoders are trained in a uni-modal setup, do you think the proposed solution is equally effective if multimodal pretrained networks are used? Could you please run an experiment on such a setup using any state-of-the-art AV Transformer model?\n\n5. Did you try validating your method on modality-agnostic backbones or can you show results on such setup? modality-agnostic backbones refer to when the same network is trained with both modalities, e.g., VATT (https://arxiv.org/abs/2104.11178), XKD (https://arxiv.org/pdf/2211.13929.pdf).\n\nLIMITATIONS:\nPlease see #Questions.\nMy questions are mostly open-ended. I will look forward to the discussion with the authors and the arguments of the other reviewers.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly proposes a new attention mechanism named Dual-Guided Spatial-Channel-Temporal (DG-SCT), which utilizes audio and visual modalities to guide the feature extraction of their respective counterpart modalities across spatial, channel, and temporal dimensions. Experiments on 4 tasks shows the advantage of the proposed method.\n\nSTRENGTHS:\nOverall it is a nice paper. \n\n- The presentation of this paper is very clear. \n\n- Experiments are extensive and appear solid.\n\n- From a high level, I think a better audio-visual attention mechanism of this type would benefit a series of downstream tasks. Prompting is a trend to make audio-visual systems smarter.\n\nWEAKNESSES:\nWith the above said, I am not sure if the claim on Page 1 lines 25-27 is valid for all audio-visual tasks. \"However, when perceiving the roaring sound of an engine, the visual region depicting a \"car\" should receive more attention than the region of \"trees\". Simultaneously, when observing the car, it is crucial to concentrate on the audio segments of the engine sound.\" it is true for tasks about audio-visual correspondence like retrieval/localization/segmentation, etc. But in other tasks like audio-visual joint classification, we do want to leverage the information that uniquely appears in a single modality to make predictions. This is because if we only use mutual information, then a single modality is enough, what we are looking for is information not appear in one modality but can be found in the other one. The proposed method seems to attend to the mutual information. I am wondering if it would negatively impact the performance of joint classification tasks.\n\n- minor: Page 2, line 54, it should be HTS-AT, not HT-SAT.\n\nQUESTIONS:\nFor the evaluation tasks, it would be nice to have a joint classification task such as AudioSet classification.\n\nLIMITATIONS:\nThere is one sentence limitation \"We consume a few more parameters than LAVisH.\" I don't think this weakens the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}