Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks
Authors: Haoyi Duan
Yan Xia
Mingze Zhou
Li Tang
Jieming Zhu
Zhou Zhao
Venue: NeurIPS 2023 poster (2023)
Keywords: audio-visual
multi-modal prompt
clip
cross-modal attention

Abstract: In recent years, the deployment of large-scale pre-trained models in audio-visual downstream tasks has yielded remarkable outcomes. However, these models, primarily trained on single-modality unconstrained datasets, still encounter challenges in feature extraction for multi-modal tasks, leading to suboptimal performance. This limitation arises due to the introduction of irrelevant modality-specific information during encoding, which adversely affects the performance of downstream tasks. To address this challenge, this paper proposes a novel Dual-Guided Spatial-Channel-Temporal (DG-SCT) attention mechanism. This mechanism leverages audio and visual modalities as soft prompts to dynamically adjust the parameters of pre-trained models based on the current multi-modal input features. Specifically, the DG-SCT module incorporates trainable cross-modal interaction layers into pre-trained audio-visual encoders, allowing adaptive extraction of crucial information from the current modality across spatial, channel, and temporal dimensions, while preserving the frozen parameters of large-scale pre-trained models. Experimental evaluations demonstrate that our proposed model achieves state-of-the-art results across multiple downstream tasks, including AVE, AVVP, AVS, and AVQA. Furthermore, our model exhibits promising performance in challenging few-shot and zero-shot scenarios. The source code and pre-trained models are available at https://github.com/haoyi-duan/DG-SCT.

Cross-modal Prompts: Adapting Large Pre-trained Models for Audio- Audio-Visual Downstream Tasks
Haoyi Duan'* Yan Xia' l+ Mingze Zhou' Li Tang' Jieming Zhu
Zhou Zhao
1 Zhejiang University
Shanghai Artificial Intelligence Laboratory HHuawei Noah's 's Ark Lab
(haoyiduan075, xiayan. 2ju) com
{3200102572, tanglzju, zhaozhoul@zju. edu. edu, cn jiemingzhulieee.or
Abstract
In recent years, the deployment of large-scale pre-trained models in audio-visual downstream tasks has yielded remarkable outcomes. However, these models, pri- marily trained on single-modaliyy unconstrained datasets, still encounter challenges in feature extraction for multi-modal tasks, leading to suboptimal performance. This limitation arises due the introduction of irrelevant modality-specifie infor- mation during encoding, which adversely affects the performance of downstream tasks. To address this challenge, this paper proposes novel Dual-Guided Spatial- Mhannel-Temporee (DG-SCT) attention mechanism. This mechanism leverages audio and visual modalities as soft prompts dynamically adjust the parameters of pre-trained models based on the current multi modal input features. Specifically, the DG-SCT module incorporates trainable cross- modal interaction layers into pre-trained audio-visus encoders, allowing adaptive extraction of crucial informa- tion from the current modality across spatial, channel, and temporal dimensions, while preserving the frozen parameters of large-scale pre-trained models. Experi- mental evaluations demonstrate that our proposed model achieves state-of-the-arr results across multiple downstream tasks, including AVE, AVVP, AVS, and AVQA. Furthermore, our model exhibits promising performance challenging few-shot and zero-shot scenarios. The source code and pre-trained models are available at /tttps://itthut cww/haoyiicc...ccc..aooooaoaooo hatm//hayiiduan/DDD
Introduction
With the increasing availability of hardware resources, large-scale models [20] 3 pre-trained on extensive data have achieved significant advancements in various multi-modal tasks [26] 5]
Nonetheless, since these models are primarily pre-trained on single modality, they may not be optimally suited for current multi-modal downstream tasks 32 43 14). As depicted in Fig _ (a).
the pre-trained model equally extracts visual features and directly passes them to downstream tasks. However, when perceiving the roaring sound of engine, the visual region depicting a "car" should receive more attention than region of "trees". Simultaneously, when observing the car, it is crucial to concentrate the audio segments of the engine sound. Therefore, the encoder should not only equally extract modal-specific information from the current modality, but also highlight information related to other modalities to enhance feature fusion across diverse modalities in downstream tasks. Retraining these large models based on downstream tasks would impose an unaffordable burden [4I 22 39], leading recent works to explore methods for fine-tuning pre-trained models downstream tasks without full retraining, showing promising progress. However, these CLIP-based methods have primarily focused text-image tasks 46
45
12 while overlooking another important
Equual contribution. Coorresponiing author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

I segments
T segments
W
W
W
W
T
Original visual features
Swin-T
+
Our visual features
Swin-T
DG-SCT
Visual imput:
Audio input
Visual event:
Noching
Rece
Nothing
Engine
Englne
Engine
Nothing
Engine
Auuio
Figure I: Our Spatial and Temporal attention can focus on important regions and moments in video and emphasize critical timestamps and frequencies in audio; Channel attention enhances the representations of audio and visual features. Take visual modality, our visual features contain fine-grained, task-specific information under the guidance of audio prompts.
multi-modal scenario: audio-visual tasks. Hence, in this paper, we primarily investigate how to utilize existing large-scale models, such as CLIP (26) and Swin- Transformer (20), to adaptively adjust the encoding features with the guidance of counterpart modality when encoding audio or visual information.
The success of prompt learning in large language models (LLMs) 13
has recently sparked growing research interest in multi-modal prompt learning, as seen works such as CoOp (46), CoCoOp [45]. CLIP- CLIP-Adapter (7). DenseCLIP [27). and MaPLe [12]. While these approaches enable adaptive adjustment of imput features using text-based prompt templates for downstream tasks, an important question arises: Can audio or video serve as innovative prompt templates to enhance task comprehension for pre-trained models and guide adaptive feature extraction of the counterpart modality? Our findings suggest positive answer to this question.
18
19
30
In this paper, we present the Dual-Guided Spatial-Channe Channee--Tmmporral Temporal (DG-SCT) attention mecha- nism, designed to adaptively adjust feature extraction of pre-trained models based on audio-visual input. Our work motivated by a recent work, LAVisH [17), which introduces trainable layers with shared parameters in pre-trained models to enhance fusion of audio-visual features, demon- strating promising performance on various audio-visual downstream tasks with minimal additional parameters. However, LAVisH has a few limitations. First, it relies solely on a visual encoder to encode audio, which we argue insufficient for capturing key audio features 1O 23 Second, only employs cross-attention in trainable layers to introduce information from different modalities, without explicitly highlighting crucial information within the current modality. By contrast, our approach incorporates cross-modal interaction layers into audio (HTS-AT (3) and visual (ViT ( or Swin-T (00) pre-trained models, leveraging different modalities as prompts to focus on special aspects the input features that are more relevant to the counterpart modal semantics across spatial, channel, and temporal dimensions. We term our proposed approach as "prompts" to denote the guidance provided to the trainable weights in preceding layers. It emphasizes utilizing audio and video cues to guide the representation of the counterpart modalities.
As depicted in Fig. I unlike previous audio and visual encoders, which generate audio and visual features separately and uniformly (Fig. (a)), our features contain fine-grained, task-specific informa- tion at multiple levels by leveraging the guiding characteristics of multi-modal information [37] 36). This enables efficient implementation of a wide range of downstream tasks. Notably, unlike previous CLIP works [46] 45) that offer unidirectional prompts, our approach introduces bidirectional prompts. This means that visual and audio modalities can mutually guide each other, facilitating enhanced feature extraction from the respective modalities.
In summary, this paper makes the following contributions:
We highlight the limitations faced by large-scale pre-trained models in audio-visual down- stream tasks, which hinder their optimal performance. overcome these, we propose to utilize audio-visue features as novel prompts to fully leverage the feature extraction capa-

---

Trainable
Self-Attention Block
www.shot/zero--hee
DG-SCT
Frozen
photo
(CASS
Pail
fu
DG-SCT
K
Swin-T Block
Swin-T Block
Downstream
Models
M
Spatial
M,
train
cello,
DG-SCT
viclin fidele
c)
M
Charrel
M
Swin-T Block
Swin-T Block
Where the first sounding instrument?
a
Visual Input
Audio Input
d) right
AZV
V2A
Figure 2: Audio-visual inputs: 2) DG-SCT is injected into every layer of frozen pre-trained audio and visual encoders; 3) After feature extraction, the audio and visual features are sent to various downstream tasks; 4) Details of DG-SCT in pptial-chammelttemprre attention levels.
bilities of large-scale models, enabling the effective utilization of task-specific information from different modalities.
We introduce novel attention mechanism named Dual-Guided Spatial-Channel Temmope (DG-SCT), which utilizes audio and visual modalities to guide the feature extraction of their respective counterpart modalities across spatial, channel, and temporal dimensions. Notably, our approach adds only limited number of parameters for the interaction layer, while keeping the original parameters of the large-scale pre-trained models frozen.
Extensive experimental results on four audio-visun tasks, namely, AVE, AVVP, AVQA, and AVS, demonstrate the superiority of our model compared state-of- of-tho-f-te-arr counterparts across various settings. Furthermore, we evaluate the performance of DG-SCT in few-shot and zero-shot scenarios on the AVE and LLP datasets, demonstrating its superiority over CLIP and several competitive CLIP- CIIP-Adapters.
2 Related work
2.1
Audio-visual understanding
Audio-visual understanding tasks involve utilizing both audio and visual modalities to get better
perception of audio-visual scenarios 88 22 31). For instance, Aidio-Visual Event Localization
(AVE 333 requires models to recognize joint audio-visual events. Previous works [33 16 35
37 36
use late interaction strategies better leverage the visual and audio features encoded from modality-specine pre-trane pre-trained models. Audio Audio-Visual Video Parsing (AVVP (32) task breaks the AVVP task, previous work (32) proposes hybrid attention network and attentive Multimodal Multiple Instance Learning (MMIL) Pooling mechanism to aggregate all features. The task of Audio- Visual Segmentation (AVS (42) focuses on whether each pixel corresponds to the given audio so that a mask of the sounding object(s) is generated. Zhou et al. (43) use a temporal pixel-wise audio- visual interaction module to inject audio semantics as guidance for the visual segmentation process. Furthermore, the newly introduced Audio-Viann Visual Question Answering (AVQA (14) task requires methods that perceive both audio and visual modalities to answer human-generated questions about the audio-visual content. Li et al. propose a spatiotemporal grounding model [1]] to achieve scene understanding and reasoning over audio and visual modalities.
restriction that audio and visual signals are definitely aligned. To tackle the
However, most methods designed for these tasks rely on modality-speciff audio and visual pre- trained models, which can not utilize multi-modal cues early in the representation stage. In this paper, we propose novel early-interaction strategy, adaptively extracting key information from the current modality across ppatial-channel-tore -emmmraaa dimensions.
3

---

2.2 Vision-language models and prompt learning
Vision-language models have made remarkable progress since the introduction of CLIP 226], with zero-shot and few-shot ideas achieving excellent generalization abilities in many downstream tasks; Meanwhile, prompt, a concept NLP [15 13 19]. has achieved impressive results in various NLP domains since its introduction, as evidenced by the success of the GPT series [25 T. Subsequent works have attempted to combine these two and achieved better results. For example. CoOp [46] improves the CLIP model by optimizing the continuous prompts in the language branch, and CoCoOp (45) further improves the model by incorporating prompts in the video branch. However, these works only utilize prompts to guide individual branches. -.PP-ad (7) builds on these works by proposing to use embedding of video and language to guide each other at the end of the encoder. MaPLe [2] is the first to use an adaptor to guide each other inside the encoder, integrating visual and text representations with the semantics of each other to enhance the generalization ability of the model. However, none of these works consider utilizing prompts in the audio-visual domain.
In this paper, we introduce a novel bidirectional prompt that employs audio and video cues indepen- dent of text to achieve outstanding information extraction abilities for audio-visse visuull tasks.
3 Approach
In this section, more details about our proposed DG-SCT are elaborated. An overview of the proposed framework is illustrated in Fig. 2
3.1 Representations for audio-visual modalities
Visual representation Given an input video sequence, we first sample a fixed number of RGB video frames {Vi) E x x3 where H. W are height and width. Following the Swin-T 220. we first split each RGB frame V₂ into non-overlapping patches by a patch-splitting module with kernel size (P. x P.). Each patch is treated
"token" and its feature is set concatenation of the raw pixel values. A linear embedding layer is then applied to this rawuvdlued feature and we can get visual features as E R Pe xC. where C, is the number of visual channels.
Audio representation Given an input audio track, we first get audio mel-spectrogram {A,) where A, E RLLF with time I, and frequency bins F. Following the HTS-AT [3] the audio mel-spectrogram cut into different patch tokens with a Patch-Ember CNN of kernel size (Pa Pa). A linear embedding layer is then applied to this raw-valued feature and we can obtain audio features as ag R P. xC. where C. the number of audio channels.
3.2 Adding DG-SCT modules to frozen encoders
Now, we describe how adjust pre- pre-trained Swin-T and HTS-AT with our proposed DG-SCT. Every layer of the Swin-T and UTTSAT consists of three main operations: 1) multi-head attention (MHA), 2) multi-layer perceptron (MLP), and our DG-SCT modules which the intermediate layer information of audio and video as prompts to guide each other through spatial-channel--eeponn dimensions. We skip the linear normalization layers in both MHA and MLP operations for brevity.
Given audio inputs (f) (Txx(1.9.p(00xc))) and visual inputs (() E ( w(x)xc(())) from layer E, we first use two-dimensional convolution kernel and linear projection to make the dimensions of the audio and visual prompts consistent of their counterpart modality. Let vy
(2() vl() and d ()() (1) denote the operation that implements DG-SCT module, which we will describe in the next subsection. Then, the operations in each layer can be written as:
Audio Transformer an audio encoder based on Swin-
Transformer.

---

(10 = =v(0 + MHA(V(0) + (2 110)) ,1,)), d = a0)) MHA(a!0) + (1(00 . (1) ((+11) 10 +MPP (1) S2 (a)) ()))) (et1) dg MLP((a) + +50 (1))
3.3 Dual-guided spatialchaamel-tenppral attention
In this subsection, we will describe how DG-SCT works in more detail. Given visual and audio features, the encoder such as Swin-Transformer pre-trained on large-scale single-modal data, will uniformly extract features from audio-visual inputs (See Fig. Fig. However, in practical multi-modal scenarios, not all of this information carries equal importance. For example, region where the car appears in the visual field is evidently more crucial than the background trees. Additionally, the moment when the engine sound emerges in the audio should also be given more attention. Hence, we take advantage of the fact that audio-visual pairs can provide mutual guidance for each other, and utilize different modalities
see in Fig. T the
prompts help pre-trained models focus more on specific aspects of opposite modal inputs across spatial, channel, and temporal dimensions. Different from previous works [37] [36] which only leverage audio as guidance to extract visual features, our proposed DG-SCT module can achieve triple levels of information highlighting in two directions. We illustrate these cross-modal attention mechanisms the following parts:
Different channels represent different aspects of features. The introduction of channel attention can facilitate the model to ignore irrelevant features and improve the quality of representations [TT). We let the audio and video serve as mutual guidance signals and explicitly model the dependencies between channels on each other's modality. Concretely, We use 1/ and
Channel-wise attention:
R to denote the combinations of convolutional and linear projection in section 3.2 to encode audio and visual inputs as prompts: a, (a,) E ggrx((F-W))) and V, = (v.) E RC ((..F) respectively. For audio-to-visual (A2V), we use the spatial average pooling to process ai and get a; RC.x1, then fuse it with vision via element-wise multiplication after feeding them to projection layers Og O E RCxxC, respectively, generating audio channel-guidanee
maps at (4)(a") @ (v:)) E C.. After that, we spatially squeeze the fused feature by global average pooling, denoted as 8a, Finally, bottleneck layer 4a follows with a sigmoid function is used, yielding channel attention maps Mr.: Similarly, we generate V2A channel attentive maps M
M = (Da E CRCUX M. =
E C=x1
where 1.th E CC. is the visual channel-guidane maps, 8, is spatial-wise global average pooling, 4, indicates bottleneck layer and a denotes the sigmoid function.
Spatial-wise attention: Audio can improve visual feature extraction by contributing to visual attention in the spatial dimension (33]. Inspired by this, leverage the guidance capabilities of audio and visual prompts to guide visual spatial attention and audio frequency attention, respectively, Similar to the aforementioned channel-wise attention, For A2V, we first get channel-attentive visual features vg = (M + 1) @ Vt, then we element-wise multiply audio prompt and vē after the projection of O and Og hidden dimension d 256, generating audio spatial-guidance maps at = (0) O (v)))) e (H) Then we use a projection layer O* E R xd with a sigmoid function a to obtain spatial attention maps Mrs: Similarly, we generate V2A frequency attentive maps /
M = a(O"(a;")) E (I) MIT = o00 (v(")) E III
(4)
where T.th denotes visual frequency-guidant maps, Of Rixd is projection layer.
Temporal-gated attention: Given an audio, significant time segments (c.g., "engine sound"" should be emphasized, while background information (c.g.. "silence") should be attenuated. The same holds for the visual information as well [36]. Inspired by this, we add amporrl-gatedd attention in the final layer. For A2V, we first feed the frequency-channel attentive audio features {a ) ) through an RNN to capture temporal information better and then pass it through projection layer with
5

---

sigmoid function to obtain the temporal attention gates C" E RR' Similarly, for V2A, we feed the spatial-chami attentive visual features to generate 6 E Tx
G
)))) G = (RNN( )=))))
(5)
Summary: We combine the abovementioned three attention mechanisms together to form our DG- SCT module. Given audio features a, E (CCCF(LFF) and visual features V (C.x((HW), DG-SCT first generates channel-wise attention maps Mr and Mt to let audio and video adaptively emphasize informative features of the corresponding modality. It then lets audio pay attention to the important sounding regions to produce spatial-wise attention maps M and lets video pay attention to the important frequency regions to generate frequency-wist attention maps M, thus the yielding of the spatial-channel attentive visual features vg" and the frequency-channel attentive audio features de can be summarized as:
vt = (a M, M +1)0m. -a +B. B.M 1) 3 ae.
(6)
Then we generate two temporal attention gates G" RTx1 and G E Tx1 for fur T
and
It=1
{ag Ii= respectively, thus the final outputs of Qa2v and 22 mentioned in section 3.2 are:
J1=1 = G +1) (E (T = -7- G + {a HL
(7)
Where a, and y hyperparameters. Consequently, this approach yields high-quality, fine-grained audio-visual representations, significantly improving the performance of subsequent tasks in the audio-visual domain.
Experiments
4.1
Tasks and datasets
Audio-visual event localization (AVE) focuses on recognizing audio-visual event that is both visible and audible throughout multiple time segments in video. We evaluate the AVE (33) dataset; Audio-visual video parsing (AVVP) aims to parse video into temporal event segments and label them as either audible, visible, or both. We evaluate our method for weakly-supervised AVVP task on the LLP dataset (32]; The goal of Audio-visual segmentation (AVS) is output a pixel-level map of the object(s) that produce sound at the image frame. We evaluate on AVSBench [43]; Audio-visual question answering (AVQA) aims answer questions based on the associations between objects
and sounds. We conduct our experiments on the MUSIC-AVQA dataset [14].
Meanwhile, We propose Audio-visioe zew-shot/zero-shot tasks on AVE 33 and LLP 332 datasets.
We evaluate AVE and classification tasks on AVE dataset and classification task on L.. dataset.
More details about tasks and datasets will be illustrated in Appendix.
4.2 Implementation details
Audio-visual downstream tasks: To adapt our approach to the four audio-visual downstream tasks, we replace the pre-trained audio and visual encoders with a frozen HTS-AT (3) and frozen Swin- Transformer [20). respectively. The trainable DG-SCT modules in Fig. (4) are injected into the frozen layers to let audio and visual modalities guide each other. We then use this our audio-visual feature extractor. For AVE task, our feature extractor is combined with CMBS (36). The event category label of each video segments is required to be predicted supervised manner. We adopt
38 37 36) and exploit the overall accuracy of the predicted event category as the evaluation metrics; Combined with MGN [24), DG-SCT able to tackle the weakly-supervised AVVP task. Following (32, we evaluate the parsing performance of all events (audio, visual, and audio-visual events) under segment-level and event-level metries; For AVS task, We combine our audio-visua feature extractor with the original AVS model 43
33
We use the Jaccard index J 6 and F-score F.

---

Table I:
Audio-visual event localization. Comparisons on the test set of AVE in supervised manner.
The result our re-implementation of LAVisH is 79.7%.
Method
Visual Encoder
Audio Encoder
Ace
AVEL(Audio- Visual)
33
VGG-19
VGG-like
71.4
AVEL(Audio- Visuol
VGG-19
VGG-like
72.7
AVSDN 16
VGG-19
VGG-like
72.6
CMAN
38
VGG-19
VGG-like
73.3
DAM
35
VGG-19
VGG-like
74.5
CMRAN
VGG-19
VGG-like
77.4
PSP 44
VGG-19
VGG-like
77.8
CMBS 36
VGG-19
VGG-like
79.3
LAVisH
Swin- (shared)
81.1
LAVisH
Swin-V2-L (shared)
79.7
LAVisH
Swin-V2-L
HTS-AT
78.6
Ours
Swin- V2-L
HTS-AT 82.2
Table Audio-visual video parsing. Comparisons on the test set of L.P..
Segment-level
Event-level
Methods
A
V
AV
Type
Event
A
V
AV
Type
Event
AVE (33]
49.9
37.3
37.0
41.4
43.6
43.6
32.4
32.6
36.2
37.4
AVSDN 16
47.8
52.0
37.1
45.7
50.8
34.1
46.3
26.5
35.6
37.7
HAN
32
60.1
52.9
48.9
54.0
55.4
51.3
48.9
43.0
47.7
48.0
MGN
24
60.7
55.5
50.6
55.6
57.2
51.0
52.4
44.4
49.3
49.2
Ours
59.0
59.4
52.8
57.1
57.0
49.2
56.1
46.1
50.5
49.1
as the evaluation metrics; For AVQA task, our audio-visual feature extractor used the original ST-AVQA [14). Similarly,
the answer prediction accuracy the evaluation metrics.
Audio-visual few-shot/zero-sho tasks: incorporate DG-SCT modules as adapters between the frozen CLIP image encoder ViT (26) and frozen CLAP audio encoder HTS-AT [5] generating audio
and visual features for text-audio and text-image contrastive learning, as shown Fig. (e). For zero-shot setting, the model is pre-trained on the TGG-Sound(40)) discussed in Appendix.
dataset. More details will be
4.3 Compared with state-of-the-arts on audio-visual downstream tasks
First, we challenge our method against current state-of-the-art methods on the four audio-visual tasks. As demonstrated in Table [ our model outperforms CMBS (36] and LAVisH (17) by significant margin (2.9% and 2.5 %): In Table our model attains either a competitive or even better performance. For instance, DG-SCT surpasses MGN by 3.9% points in the segment-level visual event parsing metric, demonstrating our method can utilize large-scale models to extract more useful features and further promote the fusion of these features in downstream tasks. We also achieve state-of-ther oftt-of--hh-rrt results on S4 setting of AVS task (Table 3 Lastly, our model performs exceptionally well AVQA task and outperforms previous leading methods AQ. VQ, and AVQ question types, respectively. The experimental results reveal that our model has the capability of utilizing pre-trained audio and visual models extract comprehensive, higher-quality features tailored for downstream tasks than the cross-attention mechanism of LAVisH. Moreover, our model exhibits excellent generalization abilities, achieving impressive results across various audio-visual tasks.
4.4 Compared with state-of-the-ane on audio-visual few-shot/zero-shot tasks
Furthermore, presented in Fig. DG-SCT model surpasses top- perforrmfong methods like CoCoOp. CLIP-Adapter, and MaPLe by a non-negligible margin on our newly proposed audio-visual few-shot/zero-shot scenarios. Our few-shot (shot=16) learning for the AVE task attains 72.4%.
*Our modified implementation of LAVisH for fair comparisons.
7

---

Table 3: Audio-visual segmentation. Comparisons on the S4 and MS3 settings of AVSBench. Ma
and My denote the mean J and F metric values (section over the whole dataset.
Setting
Method
Visual Encoder
Audio Encoder
S4
MS3
M,
MF
M
Mr
AVS [43
PVT-v2
VGG-like
78.7
87.9
54.0
64.5
LAVisH
Swin-V2-L. (shared)
80.1
88.0
49.8
60.3
L
Swin-V2-L.
HTS-AT
78.0
87.0
49.1
59.9
Ours
Swin- V2-L
HTS-AT
80.9
89.2
53.5
64.2
Table 4: Audio-visual question answering. Comparisons on the test MUSIC-AVQA, We report accuracy on three types of questions, c.g., audio (AQ), visual (VQ), and audio-visual (AVQ).
Method
Visual Encoder
Audio Encoder
AQ
VQ
AVQ
Avg
AVSD [28
VGG-19
VGG-like
68.5
70.8
65.5
67.4
Pano-AVQA
40
Faster RCNN
VGG-like
70.7
72.6
66.6
68.9
ST-AVQA
14
ResNet-18
VGG-like
74.1
74.0
69.5
71.5
LAVisH [17
Swin-V2-L(share V2-L(shared)
75.7
80.4
70.4
74.0
L
Swin- V2-L
HTS-AT
75.4
79.6
70.1
73.6
Ours
Swin- V2-1.
HTS-AT
77.4
81.9
70.7
74.8
outperforming MaPL. for 5.3% points, and even comparable to previous full-training baselines (Table D Note that experimental results for few-shot on the LLP classification task are worse than zero-shot scenario. We argue that this is due to significant gap between downstream and pre-training data, which may have disrupted the original parameters of the CLIP model. However, we still outperform MaPLe by 2.2% and 0.7% points, for zero-shot and few-shot (shot=16) settings, respectively. The outcomes demonstrate that dual-guided audio-visual prompts extract task-
specific information better than text-image [26] 46 45 [2] and text-audio (5) pre-trained models and exhibit stronger adaptability audio-visual tasks.
4.5 Ablation analysis
We verify the effectiveness of the three modules, namely, S (spatial), (channel), and T (temporal), in DG-SCT. As shown in Ta compared to the bascline model, the individual incorporation of these attention modules leads substantial performance enhancements of our model across range of downstream tasks, demonstrating the effectiveness of these modules. Although cross-modal interactions in any dimension can enhance feature extraction, argue that distinct modules can also mutually guide and enhance each other's 's performance. For instance, removing the T module individually does not result significant decrease in the performance that the combination of S and C can slightly substitute the function of T. The reason might be that if the features of a certain segment are more prominent in both spatial and channel dimensions, more likely to be important in the temporal dimension as well.
the model, which indicates
Table 5: Ablation study of the spatial-channel--empee attention module. "S", "C", and "T" denote spatial, channel, and temporal attention, respectively.
Module
AVE
AVVP
AVS
AVQA
Segment-leve level
Event-
S4
MS3
S
C
Acc
AQ
VA
AVQ
Avg
AV
AV
M.g
Mr
M.
Mr
78.6
49.8
43.9
78.0
87.0
49.1
59.9
75.4
79.6
70.1
73.6
81.8
51.3
45.6
79.9
88.4
51.0
62.1
76.0
80.9
70.8
74.4
80.9
44.7
78.6
87.6
50.9
61.6
74.9
80.3
69.5
73.3
V
82.0
52.3
45,9
80.0
88.6
51.8
62.3
77.0
70.3
74.6
81.5
45.7
79.9
88.9
52.2
61.5
76.0
81.3
70.2
74.1
V
82,2
46.1
80.9
89.2
53.5
64.2
77.4
81.9
70.7
74.8

---

Vialin
Bark
Truck
Violin
Guitar
Speech
Bark
Meow
Cipclop
Siren
Car
Original
Spatial
Pixel mask
aatt
(AVS)
Figure 3: The qualitative results of DG-SCT AVE and AVS tasks.
AVE CiansPocatoo
casiitoons
2000 517
Drinnp
Rufua
GOOP
89
20.0
Figure 4: Results of our model and previous methods on few-shot/zero-shot tasks.
4.6 Qualitative analysis
Fig. represents examples of the effectiveness of our DG-SCT. On the left, we observe that with the guidance of the audio prompts, the module can accurately focus the critical visual regions for different AVE events. For instance, when multiple objects are present in video frames, our model is capable of accurately localizing the sounding objects (e.g., bark, cat, the second example of flute, the fifth example of speak). Moreover, our model can precisely pinpoint the sound source location at fine- ime-ggrined level, such as the strings of violin and the hands of the performer, and the speaker's mouth. In addition, DG-SCT achieves excellent results on AVS task. The right part of Fig. iindiattes that model can accurately segment the pixels of sounding objects and outline their shapes perfectly. The excellent qualitative results of our model on various downstream tasks illustrate its strong potential for generalization.
As depicted in Fig. 5 we employ t-SNE (34] to visualize the learned audio and visual features and compare features generated by our model with features generated by baseline without DG-SCT, on various tasks. Each spot denotes the feature of one audio or visual event, and each color corresponds to particular category, such as "cat" in orange as shown in Fig. 5] (AVE). As we can see, features extracted by the proposed DG-SCT are more intra-class compact and more inter-class separable. These meaningful visualizations further demonstrate that the DG-SCT model successfully learns
compact and discriminative features for each modality across diverse downstream tasks.
4.7 Efficiency analysis
Our efficiency analysis on AVE task is presented Table Our approach utilizes more trainable parameters. However, our proposed DG-SCT attention mechanism requires a comparable number of parameters the latent tokens utilized in LAVisH [17]. The increase trainable parameters primarily arises from the inclusion of -oo-dimensional convolution kernel and linear projection (section 3.2) These additions ensure consistency of dimensions between the audio and visual prompts. In other words, the increase in parameter count from approach mainly results from addressing the inconsistency in dimensions between the audio and visual encoders.
We also conducted a comparison of computational cost (29] on AVE task. Our approach involves fine-tuning the pre-trained model, which inevitably leads reduction in speed compared previous late-interaction baselines (CMBS (36)). However, we have achieved outstanding results, and our approach is applicable multiple audio-visual tasks. Overall, the benefits substantial.
Conclusion and discussion
This paper introduces DG-SCT, method that leverages audio and visual modalities as prompts in the early layers of frozen pre-trained encoders. By doing so, model can extract higher-quality

---

Visual
Original
Ours
Audio
Original
Ours
(AVE)
(AVVP)
(AVS)
Figure 5: Qualitative visualizations of original visual (top row), visual (second row), original audio (third row), and our audio (bottom row) features on AVE, AVVP, and AVS tasks.
Table Efficiency analysis on AVE task. For trainable parameters of LAVisH* and our model, the first number (17.3) represents the trainable parameters of a two-dimensional convolution kernel and linear projection.
Method
Trainable Params (%)
Total Params (M)
GFLOPs
Acc
CMBS [36 LAVisH [77
14.4
216.7
40.9
79.3
10.1
238.8
406.7
79.7
L
17.3+13.3=00.6
374.9
416.1
78.6
Ours
17.3426.=43.66
461.3
460.8
82.2
and finer-grained audio-visual features, enhancing performance in subsequent tasks. We conduct comprehensive experiments on four datasets, including AVE, AVVP, AVS, and AVQA, as well as our newly proposed cew-shot/zero-shot audio-visual tasks. Across 25 experimental settings, our approach achieves state-of-the-ee results on 19 of them. Additionally, ablation studies conducted on these datasets validate the effectiveness of our proposed spatial, channel, and temporal attention modules. Furthermore, our approach demonstrates robust generalizability and holds potential for application in more audio-visual scenarios the future.
Acknowledgements
This work is supported by National Key R&D Program of China under Grant No. No.2022ZD01120000, National Natural Science Foundation of China under Grant No. 62222211 and No.6 1836002 We also gratefully acknowledge the support of MindSpore (https: / //ww: mindspore. en), which is new deep learning computing framework.
10

---

References
[1] Tom Brown et al. "Language models are feo-shot learners". In: Advances in neural information processing systems 33 (2020), pp. 1877-1901.
[2] Honglie Chen et al. "Vggsound: A large-scale audio-visual dataset". In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020. pp. 721-725.
[3] Ke Chen et al. "HTS-AT: A hierarchical token-semantic audio transformer for sound classifi- cation and detection". In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2022, pp. 646-650.
[4] Alexey Dosovitskiy et al. "An image is worth 16x words: Transformers for image recognition at scale". In: arXiv preprint arXiv:2010.1 /929 (2020).
[5] Benjamin Elizalde et al. "Clap: Learning audio concepts from natural language supervision".
In: arXiv preprini 2rXiv:2206.04700 (2022).
[6] Mark Everingham et al. "The pascal visual object classes (voc) challenge". In: International journal of computer vision 88 (2010), pp. 303-338.
[7] Peng Gao et al. "Clip-adapter: Better vision- languaee models with feature adapters". In: arXiv preprint arXiv:2110.0485
(2021).
[8] Ruohan Gao and Kristen Grauman. "Visualvoice: Audio-visual speech separation with cross- modal consistency", In: 202/ IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR). IEEE. 2021, pp. 15490-15500.
[9] Jort F Gemmeke et al. "Audio set: An ontology and human-labeled dataset for audio events". In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE. 2017, pp. 776-780.
[10] Andrey Guzhov et al. "Audioclip: Extending clip to image, text and audio". In: ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE 2022, pp. 976-980.
[11] Jie Hu, Li Shen, and Gang Sun. "Squeeze-ande cetetee--nndexiiittii networks". In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2018, pp. 7132-7141.
[12] Muhammad Uzair Khattak et al. "Maple: Multi-modal prompt learning". " In: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 19113-19122. [13] Brian Lester, Rami Al-Rfou, and Noah Constant. "The power of scale for parameter-efficient prompt tuning". In: arXiv preprint arXiv:2 104.02004 (2021).
[14] Guangyao Li al. "Learning answer questions in dynamic audio-visual scenarios". In: Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition. 2022, pp. 19108-19118.
[15] Xiang Lisa Li and Percy Liang. "Prefix-tuning: Optimizing continuous prompts for generation". In: arXiv preprint arXiv:2 0tf::2001.00/900 10000000090 (2021).
[16] Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang. "Dual-modality seq2seq network for audio-visual event localization". In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEFE. 2019, pp. 2002-2006.
[17] Yan-Bo Lin et al. "Vision Transformers Are Parameter-Efficiee Audio-Visual Learners". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2023, pp. 2299-2309.
[18]
Xiao Liu et al. "GPT understands, too". In: arXiv preprint arXiv: 10385 (2021).
[19] Xiao Liu et al. "P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks". In: arXiv preprini 2XXiv:1110007600 (2021).
[20] Ze Liu et al. "Swin transformer: Hierarchical vision transformer using shifted windows". In: Proceedings of the IEEE/CVF international conference on computer vision. 2021, pp. 10012- 10022.
[21] Yan Luo et al. "Timestamps as Prompts for Geography-Aware Location Recommendation" In: arXiv preprint arXiv:2304.044 (2023).
[22] Shuang Ma al. " "Active contrastive learning of audio-visual video representations". In: arXiv preprint 2009::2009,00 20880000 (2020).

---

[23] Tanvir Mahmud and Diana Marculescu. "AVE-CLIP: AudioCLIP- based Multi- window Tem- poral Transformer for Audio Visual Event Localization". In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023, pp. 5158-5167..
[24] Shentong Mo and Yapeng Tian. "Multi-modal Grouping Network for Weakly- Suaely-Supervieed Audio- Visual Video Parsing". In: Advances in Neural Information Processing Systems. 2022.
[25]
Alec Radford et al. "Laanguaee models are unsupervised multitask learners". In: OpenAl blog 1.8 (2019), p. 9.
[26] Alec Radford al. "Learning transferable visual models from natural language supervision". In: International conference on machine learning. PMLR. 2021, pp.
[27]
Yongming Rao et al. "Denseclip: Language-guided dense prediction with context-aware prompting". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp.
(28) Idan Schwartz, Alexander G Schwing, and Tamir Hazan. "A simple baseline for audio-visual scene-aware dialog". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 12548-12558.
[29] Liangeai Su et al. "Beyond Two-Tower Matching: Learning Sparse Retrievable Cross- Interactions for Recommendation", In: Proceedings of the 46th International ACM SIGIR Conference on Reseurch and Development in Information Retrieval, 2023, pp- 548-557.
[30] Weng Lam Tam et al. "Parameter-efficiee prompt tuning makes generalized and calibrated neural text retrievers". In: arXiv preprint arXiv:2207.0m 07087 (2022).
[31] Yapeng Tian, Di Hu, and Chenliang Xu. "Cyclic co-learning of sounding object visual ground- ing and sound separation", " In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 2745-2754.
[32] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. "Unified multisensory perception: Weakly- supervised audio-visual video parsing". In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16. Springer. 2020, pp. 436-454.
[33] Yapeng Tian et al. "Audio-visual event localization in unconstrained videos". In: Proceedings of the European Conference on Computer Vision (ECCV). 2018, pp. 247-263.
[34] Laurens Van der Maaten and Geoffrey Hinton. "Visualizing data using t-SNE." In: Journal of machine learning research 9. (2008).
[35] Yu Wu et al. "Dual attention matching for audio-visual event localization". In: Proceedings of the IEEE/CVF international conference computer vision. 2019, pp. 6292-6300.
[36] Yan Xia and Zhou Zhao. "Cross-mode background suppression for audio-visual event lo- calization". In: Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 19989-19998..
[37] Haoming Xu al. Ccross-modal relation- aware networks for visiolvisual event localization". In: Proceedings of the 28th ACM International Conference on Multimedia. 2020, pp. 3893- 3901.
[38] Hanyu Xuan et al. "Cross-modal attention network for temporal inconsistent audio-visual event localization". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. 01. 2020, pp. 279-286.
[39] Runyi Yu et al. "LaPE: Layer-adaptive Position Embedding for Vision Transformers with Independent Layer Normalization". In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Oct. 2023, pp. 5886-5896.
[40] Heeseung Yun et al. "Pano-avqa: Grounded audio-visual question answering on 360deg videos".. In: Proceedings of the IEEF/CVF International Conference on Computer Vision. 2021, pp. 2031-2041.
[41] Zhaoyang Zeng, Daniel McDuff, Yale Song, et al. "Contrastive learning of global and local video representations". In: Advances in Neural Information Processing Systems 34 (2021), pp. 7025-7040.
[42] Jinxing Zhou al. "Audio- Segmentation". In: Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII. Springer. 2022, pp. 303-403.
[43] Jinxing Zhou et al. "AVSBench: A Pixel-level Audio- Visual Segmentation Benchmark".. In:
O.
12

---

[44] Jinxing Zhou et al. "Positive sample propagation along the audio-visual event line". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021,
pp.
[45] Kaiyang Zhou al. "Conditional prompt learning for vision-language models". In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022,
pp. 16816-16825.
[46] Kaiyang Zhou al. "Learning to prompt for vision-language models", In: International Journal of Computer Vision 130.9 (2022), pp. 2337-2348.
13

---

A Implementation details
For the AVE and the AVQA tasks, we set a 0.3, 8 0.05, and 7 0.1, We train the model with batch size of 8 and a learning rate of 5 10 and 1 x 10- respectively; For the AVVP task, set a = 0.3, 3 = 0.05, and y = 0.05. We train the model with a batch size of 8 and a learning rate of 3 x 10 4; For the S4 setting of the AVS task, we set a 0.3, 3 0.05, and 7 0.05. We train the model with a batch size and a learning rate of 3 x 10 4; For the MS3 setting of the AVS task, set 0.2, 0.1, and y 0.1. We train the model with batch size of 2 and learning rate of 1.5 x 10 -
For few-shot/zero-sher tasks, we set the learning rate to 3 x 10 with batch size of 2. For the AVE task, we set a = 0.2, 8 = 0.05, and 7 0.01; For the AVE classification and the LLP classification tasks, we set = 0.2, B = 0.05, and 0.05.
All of our experiments are trained on one NVIDIA A100 GPU.
B More details of datasets
Table 7: Dataset statisties. Each dataset is shown with the number of videos and the annotated frames. The "annotations" column indicates whether the frames are labeled by category, pixel-level masks, or answer.
Classes/ Answers
Datasets
Videos
Frames
Types
Annotations
AVE
33
4,143
41,430
28
video
category
LLP
32
11,849
11,849
25
video
category
AVSBench
43
5,356
12,972
23
video
pixel
(SCS-Sound440K3
2
40,801
408,010
141
video
category
MUSIC-AVQA
9,288
45,867
42
video
answer
AVE dataset We evaluate the AVE task on the AVE dataset [33 originating from the AudioSet (9) The AVE dataset contains 4, 143 videos covering 28 categories. Each video lasts for 10 seconds and contains an event category labeled for each video on a segment level.
LLP dataset 6 We evaluate the AVVP task on the LLP dataset (32), which has 11, 849 video-level event annotations on the presence or absence of different video events and each video is 10s long and has at least 1s audio or visual events. There are 7, 202 videos that contain events from more than one event category and per video has averaged 1.64 different event categories. For evaluation, there are 4, 131 audio events, 2, 495 visual events, and 2, 488 audio-visual events for the 1, 849 videos.
We evaluate the AVS task the AVSBench dataset [43] which contains two settings: 1) Single Sound Source Segmentation (S4), which contains 4, 932 videos over 23 categories; 2) Multiple Sound Source Segmentation (MS3), which contains 424 videos over 23 categories. Each video is 5 seconds long.
AVSBench dataset |
MUSIC-AVQA dataset We conduct our experiments of the AVQA task the MUSIC-AVQA dataset [14), which contains 9, 288 videos, 45, 867 question-answer pairs, 33 question templates, and 42 answers.
VGG-Sound(40K)) | The pre-training data of our zero-shot task VGG-Sound(40K), which is split from the VGG-Sound dataset (2). VGG-Sound contains over 200k clips for 300 different sound classes, while our VGG-Sound(40K) has 40, 801 clips for 141 sound classes.
http:://gthha.cco//////ggg/ga//t
https://wwwrobbtss.com
14

---

Trainable Frozen
Self-Attention Block in CLIP & CLAP
CLIP toxt branch
photo
(0
DG-SCT
2
TN
DG-SCT
8
77
g
C'Tmmpordl Misemonc
VIT Block
HTS-AT Block
"Train horn"
M
M
utturtion
DG-SCT
text-branch
$R
the sound of
ROASS
Channel
M
T1
72
T2
8
Visual Input
Audio
two text branches are used to operate image-text text and audio-text matching, respectively. The final result is obtained by adding the weighted sum of the two outputs from image-text matching and audio-text matching.
Figure 6: Details of our framework in New-shot/zerooshre scenarios. Right:
C More details of few-shot/zero-shot tasks
C.1 Framework and method
As shown in Fig. 6 our DG-SCT module injected into every layer of the frozen CLIP 266 image encoder and the frozen CLAP [5] audio encoder. After that, the image encoder and the audio encoder are denoted as f.() and f.)).. respectively. Two text branches f. (-), (-) are employed to perform image-text and audio-text matching. respectively, the text encoders from different pre-trained models exhibit a substantial gap, using only one of them will result in a significant decrease in accuracy.
Let (Vs, a, 4)) represents a piece of visual- audio-text pair indexed by The visual embedding E,, the audio embedding E,, and the corresponding text embeddings TV and TV. are obtained by encoders with projection layers, respectively:
E = (f..(,()))). E -LP..((aaa)).
T. = =MMP;(((()))
T = MLP:
(8)
where the visual/audio/text projection layeris is 2-layer multilayer perception (MLP) with ReLU the activation function to map the encoder outputs into the same dimension 1 (i.e., EY, , 7ú 7 E IRD).
The model is trained with the contrastive learning paradigm between the paired visual and text embeddings, as well as the paired audio and text embeddings, following the same loss function in
26
N
exp(T: E. TT/TTT) exp(E, /T/)T)) + log
C.
(log
(9)
2N
En exp(E; TT/To)
Σ exp(T; EEy(To)
(log exp(E T (T/Ta) + log
exp(T: E Ta) MN exp(T; E (Ta)
C.
(10)
2N
ΣN exp(E: TT/Ta)
1=1
Where T, and Ta are learnable temperature parameters for scaling the loss. is usually the number of data, but during the training phase, N is used as the batch size. Let ye denotes the image-text
15

---

Ddt
R3P PPPaet
5
Golo 0.
abot
200
dast Fratio
Figure 7: Results of our model and previous methods on few-shot/zero-shot tasks.
matching score, and Va denotes the audio-text matching score, we set hyperparameters w1 Vo and W2 W1, thus the overall objective function is:
C - wi Ly W2
Laz
(11)
When the image-text matching score is higher, it indicates that the visual modality is more certain in determining the category of the sample, therefore the model needs to increase the weight visual modality in the loss function, and vice versa.
The same weight settings for wi and Wa in Fig. 6rre also significant during the inference process, indicating that some samples require more guidance from the audio modality while others might require more guidance from the visual modality.
For the few-shot setting, we train the model by selecting shot samples of each category from the training dataset of the downstream tasks. For the zero-shot setting, pre-train the model using the (40-)) dataset.
C.2 Evaluation metrics
For the AVE task, the category label of each video segment is required to be predicted in supervised manner. We adopt [33 37 36). and exploit the overall accuracy of the predicted event category as the evaluation metrics; For the AVE classification task, the category label of each video is required to be predicted. We leverage the overall accuracy of the video category as the evaluation metrics; For the LLP classification task, the category label of each video required be predicted. Note that some videos might have multiple categories, we randomly select one as ground truth for brevity. We leverage the overall accuracy of the video category as the evaluation metrics.
Table 8: Ablation study of A2V and V2A modules on AVE, AVS, and AVQA tasks.
Module
AVE
AVS
AVQA
S4
MS3
A2V
V2A
Acc
AQ
VQ
AVQ
Avg
Mg
M3
Mn
78.6
78.0
87.0
49.1
59.9
75.4
79.6
70.1
73.6
79.2
80.7
89.0
51.4
61.6
76.1
82.0
70.8
74.7
81.3
79.1
88.0
50.7
62.4
75.9
80.8
70.8
74.3
82.2
80.9
89.2
53.5
64.2
77.4
81.9
70.7
74.8
C.3 Results
As shown in Fig. 7 (a), (b) and (c), for few-shot (shot 16) setting, our method outperforms MaPLe by significant margin AVE, AVE classification, and LLP classification tasks, achieving 72.4%. 84.2%. 2, and 32.6%, respectively. However, the performance of our model on both the AVE and LLP classification tasks may not be as effective as some previous methods when shot < 8. The classification tasks are relatively simple and are already closely related the image-text matching task in the upstream pre-training of the CL.IP model. Therefore, the CLIP model's pre-trained parameters can achieve good results. However, when shot > 8, our model gains significant advantages. We also observe that the accuracy curves of other few-shot learning methods relatively flat, particularly for
16

---

0=005/=0
8= 0
0= 0.3 -0005
82.2
82.0
81.8
61.6
01.2
812
81.0
80.0
01.0
00.6
0.25
0.25
(a) cz
(b) p
(c)
Figure 8: Hyperparameters. We explore the impact of the hyperparameters a, 8, and y on the
experimental results of the AVE task. a) Fix B = 0.05 and 0.1, observe the impact of a b) Fix 0.3 and y = 0.1, observe the impact of 8 c) Fix a and 3 = 0.05, observe the impact of
y.
LLP classification task, where the accuracy sometimes decreases when shol increases. This issue might be the limited ability of the image-text models extract rich information from the training data of audio-visual tasks. Thus, the potential for improving accuracy is limited. For zero-shot setting, model also achieves state-of-th ofttt---ttttttttt all three tasks. The results demonstrate that model can extract audio-visual task information more effectively and achieve better results.
Table 9: Ablation study A2V and V2A modules on AVVP task.
Module
Segment-level
Event-level
A2V
V2A
A
V
AV
Type
Event
A
AV
Type
Event
57.8
56.3
49.8
55.2
54.9
48.2
51.7
43.9
48.8
47.6
56.4
59.5
53.3
56.4
55.0
47.4
55.9
46.3
49.9
47.8
/
59.4
57.3
50.8
55.8
56.8
49.2
54.1
44.4
49.2
48.6
59.0
59.4
52.8
57.1
57.0
49.2
56.1
46.1
50.5
49.1
D More ablation studies
Next, we conduct more ablation studies to investigate how different components of model affect the performance on the downstream tasks.
A2V and V2A modules. We first investigate the effectiveness of the bidirectional attention mech- anism. We compare our final bidirectional approach with the unidirectional variants that only use either audio-to- visual (A2V) visual-to-aude (V2A) spatial-channel-eemeee and also a baseline that does not use any cross- modal connections.
attention mechanism,
As indicated in Table 8 integrating A2V or V2A individually leads to substantial performance baseline model. Furthermore,
enhancements across AVE, AVS, and AVQA tasks compared bidirectional DG-SCT outperforms the unidirectional A2V and V2A baselines (e.g., 3.0% and 0.9% on the AVE task). In Table D decrease the accuracy for visual and audio-visual events. However, without visual guidance (V2A), the performance audio events suffers a considerable decline; Likewise, the performance visual events drops without audio guidance (using the V2A module alone). These experimental findings demonstrate the necessity of visual guidance for audio events and the need for audio guidance for visual events. Our proposed DG-SCT model can bidirectionally guide the representation of each modality, thus enhancing the accuracy of downstream audio-visua tasks.
we observe that using the A2V module alone does not significantly
Hyperparameters. Now, we explore the impact of the hyperparameters B, and the experimental results the AVE task. As we can see Fig. our model achieves the best accuracy (82.2%) when 0.3, 3 0.05, and 7 0.1.

---

Baby loughteri Baby laughter Raby loughteri Babby loughteri Raity laughter
Gun shooting Gun shooting Gun shooking Gun shooting: Gun shooting
Audio
Raw
image
Ground
truth
DG-SCT
w/o DG-SCT
barking
Audio
image
Ground truth
DG-SCT
DG-SCT
Figure 9: Qualitative examples of the baseline method (w/o DG-SCT) and our DG-SCT frame- work, under the S4 setting of the AVS task.
E
Additional qualitative analyses
Qualitative examples of the AVS task. We provide qualitative examples of the AVS task test the effectiveness of the DG-SCT model. As shown in Fig 9 under the S4 setting, the DG-SCT model can not only outline the object shapes more perfectly (shown in the upper left, upper right, and bottom left figures) but also locate the sounding objects more accurately than the baseline model without the DG-SCT module. As can in the bottom right figure of Fig. 9 DG-SCT can locate the dog that is barking, simultaneously excluding the mower that hasn't 't produced sound, The baseline model, however, mistakenly locates the dog and the mower at the same time. Fig I0] under the MS3 setting, the DG-SCT model can locate multiple sound sources and outline their shape nicely. In some difficult cases (the bottom row in Fig. 1O), DG-SCT can still pinpoint the correct sounding source even if the sound has stopped or the scene has changed dramatically.
Qualitative examples of the AVVP task. In Fig. 11 visualize the video parsing results of DG- SCT and baseline (w/o DG-SCT) on different examples. "GT" denotes the ground truth annotations. The results show that adding the DG-SCT module achieves more accurate parsing performance by acquiring mutual guidance from audio and visual modalities during the representation. For example, in Fig. TT (b), our model can recognize the mixture of "banjo" and "singing" while the baseline model (w/o SCT) falsely predicts the sound as "violin fiddle".
18

---

Violin, plano
Violin, plano
Violin, piano
Violin, plano
Violi,, piano
Pano, guater
Paan, guitar
Piano, gultar
Piano, guitar
Piano, guster
Image
truth
w/o DG-SCT
Nothing
lion roaring
Lion
Lion roaring
Nothing
Violin, plano
Violin, piano
Vialin, piano
Vialin, plano
Violin
Audio
Raw Image
Ground
DG-SCT
w/o
Figure 10: Qualitative examples of the baseline method (w/o DG-SCT) and our DG-SCT framework, under the MS3 setting of the AVS task.
19

---

Visual
6T
W/o 00-5CT
06-CCT
w/o 06 SCT
Aduio
Speech
SCT
w/o DG-SCT
DG-6CT
Singing
Visual
06-5CT
w/o D0
06 SCT
w/o
Aduio
Banjo
w/o
Singins
(b)
Figure 11: Qualitative examples of the baseline method (w/o DG-SCT) and our DG-SCT framework, under the AVVP task.
20


Reviews:
[{"review_id": "ApT6MGwSMG", "reviewer": "Reviewer_BzrV", "summary": "This paper mainly proposes a new attention mechanism named Dual-Guided Spatial-Channel-Temporal (DG-SCT), which utilizes audio and visual modalities to guide the feature extraction of their respective counterpart modalities across spatial, channel, and temporal dimensions. Experiments on 4 tasks shows the advantage of the proposed method. ", "questions": "For the evaluation tasks, it would be nice to have a joint classification task such as AudioSet classification.", "limitations": "There is one sentence limitation \"We consume a few more parameters than LAVisH.\" I don't think this weakens the proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall it is a nice paper. \n\n- The presentation of this paper is very clear. \n\n- Experiments are extensive and appear solid.\n\n- From a high level, I think a better audio-visual attention mechanism of this type would benefit a series of downstream tasks. Prompting is a trend to make audio-visual systems smarter. \n\n", "weaknesses": "With the above said, I am not sure if the claim on Page 1 lines 25-27 is valid for all audio-visual tasks. \"However, when perceiving the roaring sound of an engine, the visual region depicting a \"car\" should receive more attention than the region of \"trees\". Simultaneously, when observing the car, it is crucial to concentrate on the audio segments of the engine sound.\" it is true for tasks about audio-visual correspondence like retrieval/localization/segmentation, etc. But in other tasks like audio-visual joint classification, we do want to leverage the information that uniquely appears in a single modality to make predictions. This is because if we only use mutual information, then a single modality is enough, what we are looking for is information not appear in one modality but can be found in the other one. The proposed method seems to attend to the mutual information. I am wondering if it would negatively impact the performance of joint classification tasks.\n\n- minor: Page 2, line 54, it should be HTS-AT, not HT-SAT. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer BzrV,\n\nThank you for taking the time to consider our paper and giving us positive feedback!\n\n------\n\nRegarding your question, **\"I am wondering if it would negatively impact the performance of joint classification tasks.\"** \n\nIt is a very good question. In joint classification task, mutual information is critical as well. For example, when considering only the visual modality, a radio appears motionless. However, by taking into account the audio, one can determine whether the radio is producing sound. Similarly, when only listening to the audio, the sound of turning on and off a light may sound similar. It is only by combining the visual information that we can determine whether the light is on or off. By leveraging the information from both modalities, a more comprehensive and complete understanding can be achieved.\n\nIt seems you are referring to the consideration of audio-visual asynchrony and the potential side effects of our proposed method. In **Appendix D, line 86**, an ablation study was conducted to alleviate this concern.\n\nWe compare our final bidirectional approach with the unidirectional variants that only use either **audio-to-visual (A2V)** or **visual-to-audio (V2A)** spatial-channel-temporal attention mechanism. As indicated in **Table 3 (also depicted below)**:\n\n| Module |      | Segment-level |          |          |          |          | Event-level |          |          |          |          |\n| :----: | ---- | :-----------: | -------- | -------- | -------- | -------- | ----------- | -------- | -------- | -------- | -------- |\n|  A2V   | V2A  |       A       | V        | AV       | Type     | Event    | A           | V        | AV       | Type     | Event    |\n|   -    | -    |     57.8      | 56.3     | 49.8     | 55.2     | 54.9     | 48.2        | 51.7     | 43.9     | 48.8     | 47.6     |\n|   √    | -    |     56.4      | **59.5** | **53.3** | 56.4     | 55.0     | 47.4        | 55.9     | **46.3** | 49.9     | 47.8     |\n|   -    | √    |   **59.4**    | 57.3     | 50.8     | 55.8     | 56.8     | 49.2        | 54.1     | 44.4     | 49.2     | 48.6     |\n|   √    | √    |     59.0      | 59.4     | 52.8     | **57.1** | **57.0** | **49.2**    | **56.1** | 46.1     | **50.5** | **49.1** |\n\nwe observe that using the A2V module alone does not significantly decrease the accuracy for visual and audio-visual events. However, without visual guidance (V2A), the performance of audio events suffers a considerable decline; Likewise, the performance of visual events drops without audio guidance (using the V2A module alone). **These experimental findings demonstrate the necessity of visual guidance for audio events and the need for audio guidance for visual events.** Our proposed DG-SCT model can bidirectionally guide the representation of each modality, thus enhancing the accuracy of downstream audio-visual tasks. \n\nBased on our analysis, we have determined that the benefits of incorporating bidirectional modality guidance for capturing rich information outweigh the interference caused by audio-visual asynchrony. However, we acknowledge the importance of addressing the issue of audio-visual asynchrony in future work. Specifically, we plan to explore techniques during the representation phase to mitigate this problem, aiming to further enhance the performance of our model in the future.\n\nMoreover, we have added experiment on **VGG-Sound (40K) classification** (VGG-Sound dataset is the subset of AudioSet). The results are as follows:\n\n| Method |   Acc    |\n| :----: | :------: |\n|  A+V   | **69.7** |\n|   A    |   63.8   |\n|   V    |   60.6   |\n\nThe results demonstrate that incorporating both audio and visual information simultaneously leads to improved performance in audio-visual joint classification task.\n\n------\n\nminor issue: **\"Page 2, line 54, it should be HTS-AT, not HT-SAT.\"** We would like to extend our sincere appreciation for your dedicated effort in carefully reviewing our work! This issue has been addressed."}, {"review_id": "kfmlipnvvv", "reviewer": "Reviewer_VeZU", "summary": "This work proposes a new mechanism to utilize audio-visual features as novel prompts to extract task-specific features from large-scale models. This work introduces an attention mechanism named Dual-Guided Spatial-Channel-Temporal (DG-SCT), which utilizes audio and visual modalities to guide the feature extraction of their respective counterpart modalities across spatial, channel, and temporal dimensions. The proposed method is evaluated on a series of tasks including Audio-visual event localization, Audio-visual video parsing, Audio-visual segmentation, and Audio-visual question answering. Moreover, it proposes a new benchmark to perform Audio-visual few-shot/zero-shot tasks on AVE and LLP datasets.\n", "questions": "1. The proposed mechanism more resembles with Adapters than Prompts. Prompts usually work in the input space, while the proposed DG-SCT mechanism is also added in the intermediate layers. Moreover, prompting generally assumes no access to the models' internal architecture, but the proposed mechanism needs the models' access to modify/adjust some of the intermediate layers to connect the DG-SCT module. \n\n2. Often audio and video representations suffer from substantial domain gaps due to their modality-specific nature, did you encounter such scenarios?\n\n3. The proposed method is based on the Swin transformer, could you please elaborate if your method can be adapted to other Transformer architectures (e.g., other ViT variants)? If so, what necessary changes need to be done or is it a Swin-specific solution?\n\n4. In this study, the pretrained encoders are trained in a uni-modal setup, do you think the proposed solution is equally effective if multimodal pretrained networks are used? Could you please run an experiment on such a setup using any state-of-the-art AV Transformer model?\n\n5. Did you try validating your method on modality-agnostic backbones or can you show results on such setup? modality-agnostic backbones refer to when the same network is trained with both modalities, e.g., VATT (https://arxiv.org/abs/2104.11178), XKD (https://arxiv.org/pdf/2211.13929.pdf). ", "limitations": "Please see #Questions.\nMy questions are mostly open-ended. I will look forward to the discussion with the authors and the arguments of the other reviewers. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is a very interesting work. The use of prompting is primarily focused on language and later vision, notably this work performs audio-visual prompting, which seems quite innovative.\n-  Moreover, unlike previous works that offer unidirectional prompts, the proposed approach introduces bidirectional prompts, where both visual and audio modalities can mutually guide each other in the feature extraction process.\n- The proposed approach is evaluated on several benchmarks and compared fairly with prior works showing the effectiveness of the proposed method.\n- It's a nicely written paper and easy to follow.", "weaknesses": "Please see #Questions for more open-ended discussions. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer VeZU,\n\nThank you for the positive feedback and all these very valuable and constructive questions/suggestions! Let us respond to your questions point by point.\n\n------\n\n**Question 1 - The proposed mechanism more resembles with Adapters than Prompts.**\n\nThank you very much for you kind suggestion! We initially referred to our proposed mechanism as ‘prompts’ to represent the guidance provided to the learnable weights in preceding layers. It emphasizes using audio and video as prompts to guide the representation of the counterpart modalities. However, we acknowledge that this expression may have been imprecise. Upon considering your valuable suggestion, we will name it as ‘adapters’ instead. We sincerely appreciate your valuable input and assure you that we will carefully consider incorporating this change.\n\n------\n\n**Question 2 - Did you encounters domain gaps?**\n\nWe thank the reviewer for pointing this out. The reviewer is very correct that domain gaps exist in the field of multi-modal tasks due to the unique characteristics of each modality. **For full-training settings** like AVE/AVS/AVVP/AVQA tasks, during the model designing process, we carefully considered employing methods such as contrastive learning to reduce the gaps between modalities. However, we observed limited effectiveness in addressing this issue. The reason is that our proposed method focuses on fine-tuning a minimal number of parameters in the encoder of the current modality using prompt information derived from the counterpart modality. The majority of parameters remain frozen, thereby effectively mitigating the domain gap problem while enriching the modality information. After late interaction and fusion in the downstream models, our method is able to achieve promising results. **For zero-shot/few-shot scenarios**, as mentioned in **line 228, our method \"generates audio and visual features for text-audio and text-image contrastive learning\"**. Here we use two text branches instead of one **(See Appedices Figure 1 \"Right\")** to address the domain gaps of visual and audio modalities.\n\n------\n\n**Question 3 - Could you please elaborate if your method can be adapted to other Transformer architectures?**\n\nThank you for asking this meaningful question! **Our method is not a Swin-specific solution.** As mentioned in **4.2 Implementation details, line 227**, in our few-shot/zero-shot scenarios, **\"We incorporate DG-SCT modules as adapters between the frozen CLIP image encoder ViT and frozen CLAP audio encoder HTS-AT\"**. Our method can be adapted to other transformer variants as well. Since different transformer architectures may have different dimensions and layers. First, as mentioned in **line 139**, **\"we first use a two-dimensional convolution kernel and a linear projection to make the dimensions of the audio and visual prompts consistent of their counterpart modality\"**. For full-training settings like AVE/AVS/AVVP/AVQA tasks, although both Swin-V2-L and HTS-AT has **4** layers, the numbers of the blocks of each layer are not the same, with **2, 2, 18, 2** and **2, 2, 6, 2**, respectively. We designed an approach that the layers with inconsistent block counts **(e.g., the third layer, 18 and 6, respectively)** engage as many rounds of cross-modal interactions as possible **(e.g., 6)**, uniformly.\n\nSo you see, the number of layers or the size of dimensions does not impose any limitations. Our approach is applicable to any transformer architecture.\n\n------\n\n**Question 4 and 5 - Do you think the proposed solution is equally effective if multimodal pretrained networks are used?**\n\nRegarding the incompatibility issue between the official VATT code, which is implemented in TensorFlow, and our PyTorch version, as well as the unavailability of open-source code for XKD, we have indeed encountered some challenges. However, we have identified a recent study on modality-agnostic backbones called meta-transformer. Please see the following table for the results on AVE task:\n\n| Method                       | Acc      |\n| ---------------------------- | -------- |\n| LAVisH                       | 79.7     |\n| LAVisH*                      | 78.6     |\n| Ours                         | **82.2** |\n| LAVisH with meta-transformer | 74.8     |\n| Ours with meta-transformer   | 77.5     |\n\nIn \"LAVisH*\",  we modified LAVisH to use the same encoders as ours for fair comparison.\n\nHowever, the results were not ideal. Merely replacing the baseline encoders with the modality-agnostic backbone of the meta-transformer resulted in significant performance loss. This indicates that current modality-agnostic backbones may not perform well in the domain of audio-visual downstream tasks. Nevertheless, incorporating our method into the modality-agnostic backbone still yields performance improvements, highlighting the robustness of our approach.\n\nWe express our gratitude to the reviewer for highlighting this aspect. Indeed, the utilization of a modality-agnostic backbone appears to be a highly promising trend. We will continue to stay updated with relevant research endeavors and anticipate its eventual suitability for audio-visual downstream tasks in the future."}, {"review_id": "3Khk6iSNnH", "reviewer": "Reviewer_myp2", "summary": "This paper proposes a parameter-efficient approach, DG-SCT. DG-SCT can adapt pre-trained audio and visual models on downstream audio-visual tasks without updating pre-trained encoders (i.e., keep pre-trained encoders frozen.)", "questions": "* In AVE task, CMBS uses much weaker audio and visual encoders. How does CMBS with Swin-V2-L and HTS-AT work? Why does CMBS drop from 79.3 to 78.6 in Table 5?\n\n* In AVVP task, the results of MGN in Table 2 are lower than the numbers in the paper. Does DG-SCT use [A], which is also used in MGN? \n\n* Similar to the question in AVE task,  how does AVS with Swin-V2-L and HTS-AT work in AVS task? (the performance is also dropped in Table 5)\n\n* Are the baselines in zero/few-shot tasks also pre-trained in VGGSound (40K)? If yes, that would be fair to compare with DG-SCT.\n\n[A] Wu, Yu, and Yi Yang. \"Exploring heterogeneous clues for weakly-supervised audio-visual video parsing.\" CVPR 2021", "limitations": "Yes, DG-SCT uses more number of trainable parameters", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "$+$ DG-SCT can achieve state-of-the-art results on several downstream audio-visual tasks.\n$+$ DG-SCT is able to learn channel-wise, spatial, and temporal information to incorporate with pre-trained audio and visual encoders.", "weaknesses": "$-$ Although DG-SCT leverages different types of cross-modal attention (i.e., channel-wise, spatial, and temporal), the design is not clear.\nFor example, DG-SCT leverages RNN for modeling cross-modal temporal information and learnable weights for cross-modal spatial information. Such a technique can be implemented with divided (space-time) attention after channel-wise attention.\n\n$-$ The effectiveness of the proposed temporal modeling is not clear. The baselines for the implementation of DG-SCT in AVE/AVVP/AVQA have already cross-modal temporal modeling modules.  \n\n$-$ The number of trainable parameters is not reported. For example, the baselines in AVVP and AVE usually use pre-extracted audio and visual features, which contribute to a lower number of trainable parameters.  \n\n$-$ Lack of efficiency comparison (e.g., FLOPs). Combining several attention mechanisms will lead to huge computational costs. It would be great to include these metrics.\n\n\n$-$ some experimental settings and results are not clear (See. Questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer myp2,\n\nThank you so much for taking the time to read our paper and providing very insightful and constructive comments and questions, please see the following for our point-by-point reply. We also conducted quite many new experiments to address your concerns.\n\n------\n\n**Weaknesses 1 - Such a technique can be implemented with divided (space-time) attention after channel-wise attention.**\n\nWe understand the reviewer's point. The reason we leverages RNN for modeling cross-modal temporal information is that, in  **line 180**, **Given an audio, significant time segments (e.g., \"engine sound\") should be emphasized, while background information (e.g., \"silence\") should be attenuated. The same holds for the visual information as well.** \n\nWe made an effort to incorporate divided (space-time) attention following channel-wise attention, but regrettably, the obtained results were not encouraging. This can be attributed to the absence of guidance from the other modality, which plays a pivotal role in audio-visual scenarios.\n\n------\n\n**Weaknesses 2 - The effectiveness of the proposed temporal modeling is not clear.**\n\nThank you for asking! We understand you concern. In **4.5 Ablation analysis, line 263,** though we acknowledged that **\"removing the 'T' module individually does not result in a significant decrease in the performance of the model\"** because **\"if features of a certain segment are more prominent in both spatial and channel dimensions, it is more likely to be important in the temporal dimension as well\"**, adding the \"T\" module (temporal modeling) did make a difference. For example, after adding our \"T\" module, the results raised from **51.8** and **62.3** to **53.5** and **64.2**,  respectively,  in the MS3 setting of AVS task, even though the downstream model already has temporal modeling for late interaction. \n\n------\n\n**Weaknesses 3 - The  number of trainable parameters is not reported.**\n\nThank you for your suggestion. We apologize for omitting the information about the number of trainable parameters in our initial submission. We will include it in our Appendix section. Please find the updated information below (AVE task):\n\n| Method  | Trainable Params (M) | Total Params (M) | Acc      |\n| ------- | -------------------- | ---------------- | -------- |\n| CMBS    | 14.4                 | 216.7            | 79.3     |\n| LAVisH  | 10.1                 | 238.8            | 79.7     |\n| LAVisH* | 30.6                 | 374.9            | 78.6     |\n| Ours    | 43.6                 | 461.3            | **82.2** |\n\nIn \"LAVisH*\",  we modified LAVisH to use the same encoders as ours for fair comparison.\n\nIn contrast to retraining the complete backbones, we have effectively achieved a substantial reduction in the number of trainable parameters. Furthermore, our approach has demonstrated a remarkable improvement in performance compared to other baselines while increasing acceptable parameter count.\n\n------\n\n**Weaknesses 4 - Lack of efficiency comparison (e.g., FLOPs).**\n\nWe thank the reviewer for pointing this out. We acknowledge that our previous consideration may have been inadequate, since the audio-visual task is not inherently time-sensitive, and our previous baselines did not report FLOPs information. Here, we will add the efficiency comparison on AVE task in Appendices:\n\n| Method  | GFLOPs | Acc      |\n| ------- | ------ | -------- |\n| CMBS    | 40.9   | 79.3     |\n| LAVisH  | 406.7  | 79.7     |\n| LAVisH* | 416.1  | 78.6     |\n| Ours    | 460.8  | **82.2** |\n\nIn \"LAVisH*\",  we modified LAVisH to use the same encoders as ours for fair comparison.\n\nOur approach involves fine-tuning the pre-trained model, which unavoidably results in a decrease in speed compared to the previous late-interaction baselines (CMBS). However, we have achieved excellent results and our approach is applicable to multiple audio-visual tasks. Overall, the benefits are substantial.\n\n------\n\n**Question 1 and 3 - Settings and results in AVE task and AVS task.**\n\nWe thank the reviewer for the valuable input. The original LAVisH baseline uses Swin-L-V2 to encode both audio and visual modalities. For fair comparison, we modified LAVisH with Swin-L-V2 and HTS-AT, same as ours, to encode visual and audio modalities, respectively. The reason why the results dropped to **78.6** may be attributed to the rudimentary approach of LAVisH, which utilizes latent tokens for cross-attention. The coarse extracted cross-modal information may not adequately counteract the negative effects of domain gaps introduced by the use of different encoders.\n\n------\n\n**Question 2 - Settings and results in AVVP task.** \n\nThank you for your question. No, we didn't use [A] in MGN because MGN does not open source the code about using [A]. The results correspond to MGN paper without using [A].\n\n------\n\n**Question 4 - Are the baselines in zero/few-shot tasks also pre-trained on VGGSound (40k)?**\n\nThank you for asking! This is a very insightful question. Yes, the baselines are also pre-trained on VGGSound (40K)."}, {"review_id": "iZj4KxRC8A", "reviewer": "Reviewer_TYiB", "summary": "The authors propose a novel dual-guided spatial-channel-temporal attention mechanism to audio-visual problems, which leverages pre-trained audio and visual encoders. And they show the improvement in various audio-visual tasks such as event localization, parsing, segmentation, and question answering.", "questions": "- Do you need to use the same number of layers in the Swin-T and HTS-AT? Since they share the same notation in all the equations for both audio and visual paths.\n- In Section 3.1, line 127, where does T come from? Could you provide more detail on the choices of mel-spectrogram?", "limitations": "- No potential social or ethical implications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This work applied proposed methods and evaluated with diverse audio-visual downstream tasks, including localization, parsing, segmentation and question answering. This helps to showcase the generalization of proposed approach.", "weaknesses": "- The use of notations in Section 3.3 is very complicated and difficult to follow. Consider utilize Figure 2 (4) for illustration and walk through each step along with the modules in the figure to make it easier for the readers to follow.\n- For the results presented in Table 2 and 3, and discussed in Section 4.3, only the better performance is highlighted without providing potential explanation and/or hypothesis why proposed system performed worse in some scenarios. For example, the performance on event-level as in Table 2, and the performance with MS3 of segmentation in Table 3. Consider adding some discussion for these cases.\n- Some abbreviations are referred to without any information provided, such as LLP in line 210, and CMBS in line 218. Consider adding one-liner explanation to help the reader.\n- Minor comments: Section 3.2 in equation (1), the first superscript \"l\" does not follow the same style as others. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TYiB,\n\nThank you so much for giving us positive feedback and the very insightful questions. Please see our point by point responses below.\n\n------\n\n**Weakness 1 - The use of notations in Section 3.3 is very complicated and difficult to follow.**\n\nThank you for your suggestion. We will revise it. After revising it like this, we will consider adding this passage to the new paper:\n\n3.3 ...., our proposed DG-SCT module can achieve triple levels of information highlighting in two directions **(See Fig. 2 (4))**. ....\n**Channel-wise attention:** .... Similarly, we generate V2A channel attentive maps $M_t^{ac}$\n\n **(See Fig. 2 (4))**. ....\n\n**Spatial-wise attention:** .... Similarly, we generate V2A frequency attentive maps $M_t^{af}$\n\n**(See Fig. 2 (4))**. ....\n\n**Temporal-gated attention:** .... Similarly, for V2A, we feed the spatial-channel attentive visual features to generate $G^a$\n\n**(See Fig. 2 (4))**. ....\n\n**Summary:** .... We combine the abovementioned three attention mechanisms together to form our DG-SCT module. **Walking through each step along with Fig. 2 (4).**..... \n\n------\n\n**Weakness 2 - Only the better performance is highlighted without providing potential explanation and/or hypothesis why proposed system performed worse in some scenarios.**\n\nThank you for your concern. We will add more explanation for the results presented in Table 2 and 3, and discussed in Section 4.3 in the future:\n\nIn the experimental results presented in Table 2 and Table 3, we observed that for the scenes where our performance was not higher, it is comparable to the state-of-the-art results reported in previous studies. However, a notable decline is observed in the audio scene depicted in **Table 2**. Based on our analysis, it is plausible that the lower robustness of HTS-AT, compared to previous audio encoders such as VGG-like, may be a contributing factor to this decline.\n\n------\n\n**Weakness 3 - Some abbreviations are referred to without any information provided.**\n\nThank you for your suggestion. LLP refers to Look, Listen, and Parse dataset for audio-visual video scene parsing. CMBS refers to Cross Modal Background Suppression for Audio-Visual Event Localization. We have already provided some details about the datasets and tasks in **Appendix B**, specifically on **line 15**. However, we acknowledge that we overlooked the explanation of abbreviations in our previous discussion. Additionally, we will provide a concise introduction to the abbreviations in our Appendices.\n\n------\n\n**Weakness 4 - Section 3.2 in equation (1), the first superscript \"l\" does not follow the same style as others.**\n\nWe would like to extend our sincere appreciation for your dedicated effort in carefully reviewing our work! This issue has been addressed.\n\n------\n\n**Question 1 - Do you need to use the same number of layers in the Swin-T and HTS-AT?**\n\nThank you for your insightful question, we are so glad that you asked that. No, we don't need to use the same number of layers, or the same size of dimensions. To address the dimension asynchrony problem, **\"we first use a two-dimensional convolution kernel and a linear projection to make the dimensions of the audio and visual prompts consistent of their counterpart modality\", line 139**. For full-training settings for AVE/AVS/AVVP/AVQA tasks, although both Swin-V2-L and HTS-AT has **4** layers, the numbers of the blocks of each layer are not the same, with **2, 2, 18, 2** and **2, 2, 6, 2**, respectively. We designed an approach that the layers with inconsistent block counts **(e.g., the third layer, 18 and 6, respectively)** engage as many rounds of cross-modal interactions as possible **(e.g., 6)**, uniformly.\n\nSo you see, the number of layers or the size of dimensions does not impose any limitations. Our approach is applicable to any transformer architecture.\n\n------\n\n**Question 2 - In Section 3.1, line 127, where does T come from? Could you provide more detail on the choices of mel-spectrogram?**\n\nThank you for asking. We split the mel-spectrogram into **T** non-overlapping segments, each segment is **1** second long. Spectrograms, in simple terms, involve dividing an audio waveform into small windows. Each window undergoes a short-time discrete Fourier transform to obtain frequency and amplitude information; As for Mel Scale, mathematically speaking, is the result of some non-linear transformation of the frequency scale. This Mel Scale is constructed such that sounds of equal distance from each other on the Mel Scale, also “sound” to humans as they are equal in distance from one another. The mel-scale is a logarithmic scale that captures the perceptual sensitivity of human hearing to different frequencies. Mel-spectrogram, which leverages the mel-scale for effective characterization of audio signals, is commonly used in audio representation.\n\nThank you for your feedback. We hope the aforementioned clarifies any doubts or inquiries you may have had."}]

Author rebuttal:

Meta-review: Initially, the reviews were mixed. The reviewers appreciated the audiovisual prompting idea, demonstrated results, and paper presentation. However, there were concerns regarding the evaluation. In particular, reviewer myp2 had concerns regarding how the number of parameters were controlled in the experiments. The rebuttal assuaged most of the raised concerns. During the post-rebuttal discussion, reviewer myp2 had remaining concerns that the trainable and total parameters are larger than the baselines and noted to double check the total number of parameters. The AC agrees with these points, and thinks that this point can be addressed in the final paper. The AC is convinced of the positive merits of the paper as noted in the reviews. Please take into account all reviewer feedback in the camera-ready version.
Decision: Accept (poster)