Mixtures of Experts for Audio-Visual Learning
Authors: Ying Cheng
Yang Li
Junjie He
Rui Feng
Venue: NeurIPS 2024 poster (2024)
Keywords: audio-visual  learning
mixture of experts
parameter-efficient transfer learning
Primary area: deep_learning_architectures

Abstract: With the rapid development of multimedia technology, audio-visual learning has emerged as a promising research topic within the field of multimodal analysis. In this paper, we explore parameter-efficient transfer learning for audio-visual learning and propose the Audio-Visual Mixture of Experts (\ourmethodname) to inject adapters into pre-trained models flexibly. Specifically, we introduce unimodal and cross-modal adapters as multiple experts to specialize in intra-modal and inter-modal information, respectively, and employ a lightweight router to dynamically allocate the weights of each expert according to the specific demands of each task. Extensive experiments demonstrate that our proposed approach \ourmethodname achieves superior performance across multiple audio-visual tasks, 
including AVE, AVVP, AVS, and AVQA. Furthermore, visual-only experimental results also indicate that our approach can tackle challenging scenes where modality information is missing.
The source code is available at \url{https://github.com/yingchengy/AVMOE}.

Mixture of Experts for Audio-Visure Learning
Ying Cheng 123.
Yang Li Junjie Hel.2 Rui Feng 1,2,3
School of Computer Science, Fudan University Shanghai Key Laboratory of Intelligent Information Processing Shanghai Collaborative Innovation Center of Intelligent Visual Computing {chengy18, fengrui) edu. cn, (liy23, hejj23} fudan.edu.. cn
Abstract
With the rapid development of multimedia technology, audio-visual learning has emerged as a promising research topic within the field of multimodal analysis. In this paper, we explore parameter-efficient transfer learning for audio-visual learning and propose the Audio Visual Mixture of Experts (AVMoE) to inject adapters into pre-trained models flexibly. Specifically, introduce unimodal and cross-modal adapters as multiple experts to specialize in intra-modal and inter- modal information, respectively, and employ lightweight router dynamically allocate the weights of each expert according to the specific demands of each task. Extensive experiments demonstrate that our proposed approach AVMoE achieves superior performance across multiple audio-visual tasks, including AVE, AVVP, AVS, and AVQA. Furthermore, visual-only experimental results also indicate that our approach can tackle challenging scenes where modality information is missing. The source code is available at https: htttps:///ithule wwwettuu.....yiggeeee com/ylingchengy/AWDO
Introduction
Audio-visual learning is challenging problem that analyzes auditory and visual information si- multaneously, which aims at naturally integrating information from multiple modalities to perceive and understand the real-world comprehensively. By leveraging both auditory and visual cues, the models can learn multimodal correlations and solve various tasks such as Audio-Visual Event local- ization (AVE) [54], Audio- Visual Segmentation (AVS) 655 and Audio- Audio-Visual Visual Question Answering
(AVQA) 61 27 64)
In recent years, audio-visual learning has received steadily increasing attention in the field of multimodal analysis. Some previous works [3 9 22 43 []] utilize the correspondence between auditory and visual streams to learn multimodal representations in a self-supervised manner. By reducing the reliance on labeled data, self-supervised approaches can scale to larger datasets [14] and improve generalization performance. However, due the rapid growth of the model size, fine-tuning full parameters of large pre-trained models entails significant computational costs for each downstream task. Recently, some researchers [30 [T] have attempted to inject adapters into frozen audio-visual pre-trained models, so as to reduce trainable parameters when applying to downstream tasks. Although these approaches have achieved satisfactory performance, they only focus cross- modal attention in adapters to introduce information from other modalities, lacking the prominence of crucial information within the current modality. Besides, not every visual frame and audio segment can promote each other, and only introducing cross-modal adapters may bring irrelevant information during the learning process. As shown in Figure [ the sound of basketball bounce is completely drowned out by the commentator's voice and the cheers from the audience. In these cases such
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Video
Input
Audio
Input
Video
Os
Basketball
10s
Label
3s
Cheering
10s
35
Clapping
10s
Audio
Os
Speech
5s
6s
Speech
10s
Label
35
Cheering
10s
3s
Cheering
10s
Figure 1: The case that video and audio labels are different. The video lahel includes basketball, cheering, and clapping, while the audio label includes speech and cheering.
off-screen sounds, injecting cross-modal adapters only brings disturbing information, and it is necessary to inject adapters according to the scenario flexibly.
The Mixture of Experts (MoE) is structural strategy for machine learning models, designed to enhance overall prediction performance by combining the decisions multiple expert models. Hence, it is appealing to integrate the MoE module into neural networks. Within the MoE strategy, each expert model focuses on different subsets or aspects of the imput data, providing predictions for given input. Additionally, there is gating mechanism (usually trainable layer) responsible for determining which expert's 's predictions should be given more weight based on the input data. This structure allows the model to dynamically adjust its reliance different experts, thereby optimizing its ability to process various types of data.
In this work, we propose an approach of Mixture of Experts for Audio Visual Learning (AVMoE). designed as an efficient and flexible approach to adaptively combine adapter capabilities according to different scenarios. Firstly, we introduce two types of adapters as experts into frozen pre-trained models, i.e., unimodal adapter and cross-modal modal adapter, to focus on intra-modal and inter-modal information, respectively. Secondly, each expert is the characteristics of inputs and assign the weights to each expert. By employing the AVMoE scheme, the model can adapt its strategy according to the specific demands of each task, whether it be heavily reliant on visual data, audio data, or requiring a balanced integration of both. conduct extensive experiments results demonstrate that our approach not only improves the model's performance across a wide range of audio-visual tasks but also makes it robust to variations in the input data. For instance, in scenarios where audio information is obscured or noisy, the model can shift its reliance toward visual cues and vice versa.
introduce modality
router to evaluate how relevant
three audio- visual tasks, i.e., AVE, AVS, and AVQA. Experimental
The contributions of this paper can be summarized as follows:
We propose a generic, effective, and flexible approach, AVMoE, for applying pre-trained models to audio-visual learning, which can dynamically adjust its strategy according to the specific demands of each task.
We introduce two types of adapters to focus on both within-modee modality details and the interac- tions between modalities. By integrating adapters and MoE, the model can be robust to any variations in the input data, e.g., missing modality or multimodal content mismatch.
Extensive experiments demonstrate that our method consistently outperforms in diverse andio-visual tasks, showcasing ability to dynamically prioritize visual or audio cues, enhancing its reliability and effectiveness in real-world applications.

---

2 Related Work
2.1 Audio-Visual Learning
Audio-visual learning aims to integrate auditory and visual information to enhance the understanding
and perception of multimedia scenarios. Some early researchers (3] 22 43 propose innovative self-supervised approach that utilizes the correspondence between visual and audio components in video supervisory signal to pre-train a multimodal model, and the model can be applied to multiple downstream tasks such as speech enhancement [8] I7 18) sound localization [60] and action recognition (23 49). However, these audio-visual pre-trate pre-trained models are limited by the ability of temporal localization and complex reasoning.
Recently, some researchers have made efforts for complicated audio-visual downstream tasks. Audio- Visual Event Localization (AVE (54)) aims at classifying and localizing events in the given video, and most prior works (57 59 62 37 introduce interactions utilize multimodal cues from modality-
specific pre-trained models. Audio- Visual Video Parsing (AVVP (53) expands the task of localizing one event to multiple events and removes the restriction that audio and visual events are definitely
consistent. Previous works [63] [66] usually propose multi-scale hybrid networks and aggregate
features in weakly-supervisee manner. Audio- Visual Segmentation (AVS (65)) aims to localize and segment the masks of sounding objects at the pixel level. To tackle this task, most prior methods
learn correspondences between audio and visual patches and perform mask decoding by variational
auto-encoder (39]. contrastive conditional latent diffusion (38), vision transformer (31). bidirectional
generation [15]. Audio- Visual Question Answering (AVQA (61)) requires the model to correctly
answer the questions by understanding both audio and visual modalities comprehensively. Previous
works tackle this task by spatiotemporal grounding model 226] missing modality recalling (44). and
multi-scale feature fusion 7
Although these researches have achieved satisfactory performance, designing models for distinct tasks separately is often expensive and consumnnn,, In order to improve efficiency and generalization, some recent works have explored parameter-efficien transfer learning for audio-visual downstream tasks. LAVish (30] proposes to introduce cross mossal adapters into frozen pre-trained transformers for audio-visual data. DG-SCT [11] proposes dual-guided spatial--hamnel--empooo as cross-modal prompts injected into pre-trained models. Unlike these prior methods focus on cross- modal adapters, we introduce uni-modal adapters and the MoE scheme combine multimodal information dynamically.
attention mechanisms
2.2 Mixture of Experts
Mixture of Experts (MoE) 19 is framework that has been extensively studied and applied in various domains of machine learning. Some works explore the role of MoE the research fields of Natural Language Processing (NLP) [1
24
13
10 68 and Computer Vision (CV) [47 35).
These works typically regard the Feedd-oowaane Network (FFN) layers as experts and choose the most relevant experts for the given task. Some efforts have been made to improve the routing strategy by
differentiable selection [66,, linear assignment 25 simple hashing algorithm 48) dense-to-sparse
strategy (42 and expert choice [67].
Recently, some works have extended the original MoE scheme to enhance its applicability in various advanced learning paradigms. In the field of multi-task learning, Ma et al. (36) and Chen et al. (6) have successfully integrated MoE schemes to deal with the complexities of Tearning multiple tasks simultaneously. These approaches leverage expert-specific knowledge to effectively address the unique requirements of each task, thus improving the performance and efficiency of models.
Similarly, the MoE scheme has also been adapted for multimodal learning. Mustafa et al. [4] explore the usage of MoE in combining visual and textual information, demonstrating significant improvements in tasks that require understanding from both image and text inputs. Shen et al. (52) further employ MoE to scale both the text- exsed and vision-based feed-forwart networks, showing the robustness and flexibility in handling diverse data sources. Cao et al. [4] and Akbari et al. [2] also contribute to this field by developing MoE-based models that alternate between different modalities, effectively capturing the complex interactions between them. However, these approaches only investigate the effectiveness of MoE in vision-language models, to the best of our knowledge, the MoE scheme has not been explored for audio-visual learning.
3

---

Transformer Encoder
Transformer Encoder
Audio-Visual Mixture of Experts (AVMoE)
Weighted Sum
Add Layer
Add & Layer Norm
AVMoE
AVMoE
Feed Forward
Feed Forward
Cross-Modal Adapter
Unimodal Adapler
Router
Layer Norm
Layer Norm
AVMoE
AVMoE
Multi-Head
Mutti-Head
Concatenate
Attention
Update
Frozen
Visual Tokens
Audio Takens
Audio
Figure 2: Method Overview. We propose to inject trainable adapters with the MoE scheme into the frozen pre- pre-trained backbones for dynamically adjusting the strategy. The AVMoE module mainly consists of router layer and two types of adapters, the router ingests concatenated multimodal tokens and allocates the weights for adapters, and the output of AVMoE module the weighted of all the adapters' predictions.
3
Method
In this section, we present our approach AVMoE shown in Figure 2 which adaptively integrates multiple types of adapters into the frozen pre-trained model using the audio-visual mixture of experts (MoE) framework. The structure of our framework consists of the frozen pre-trained model (Sec. 3.1
and trainable MoE modules (Sec. 3.2). where the MoE module contains two types of adapters and router. In the following, we will describe each component in more detail.
3.1 Frozen Pre-trained Model
In this work, we aim to adapt pre-trained models to audio-visual tasks with minimal changes to the original parameters. Hence, we add few trainable parameters while keeping the pre-trained model frozen. Following previous work [IT]. we use the standard Swin Transformer (Swin-T) (33) and audio transformer (HTS-AT) (5) pre-trained backbones. Each layer of these models consists of Multi-Head Attention (MHA), Feed Forward Network (FFN), and Layer Normalization with Residual Connections (Add & Layer Norm).
3.2 Audio-Visual Mixture of Experts
In this subsection, we describe how our proposed AVMoE can flexibly adjust frozen pre-trained models to various audio-visual downstream tasks. As illustrated in the right of Figure the AVMoE module contains three primary components: two types of adapters as experts, each specializing in different aspects of the multi-modal data, and a router for ingesting concatenated tokens and allocating the corresponding weights of experts.
3.2.1 Router
The router layer is responsible for activating the appropriate experts based the input features. To be specific, after obtaining visual embedding v; and audio embedding o,, we concatenate them as the input i, of the AVMoE router. Following previous works [13], we employ Multi-Layer Perceptrons (ML.s) as the architecture of router R, which produces weights by using the softmax function. The weights for Cross-Modal Adapter (CMA) and Unimodal Adapter (UA) are computed follows:

---

exp (Fem
exp (Ten(
WCMA
exp (TCMA( + exp (x.) WLA exp (TCMa( (3r) + exp (r))
(I)
where TCMA and FUA are the outputs of the router for CMA and UA. The produced weights determine
the importance of each adapter's prediction to the final output.
To promote the balanced usage of experts, inspired by previous works [3 20 we introduce Gaussian noise to the router's 's output at the training stage, thereby promoting a more balanced use of experts. This method prevents the model from over-relying on certain experts, foreing the model explore wider range of experts:
c c~N(0,
where is Gaussian noise with standard deviation a.
3.2.2 Visual Adapters
To enhance the efficiency and performance of audio-visual models, we follow previous works 30 11 adding a few trainable parameters (i.e., adapters) into each layer of frozen pre-trained backbones. However, different from them, we design two types of adapters, i.c., CMA and UA. As shown in Figurd the main differences between them are multimodal feature fusion and self-attention. These adapters are injected into the frozen pre-trained model via the MoE strategy.
Cross-Modal Adapter. Firstly, to introduce interactions and integrate information from multiple modalities, we follow LAVisH adapter (30) as our cross-modal adapter. As shown in the left of F the cross-modal adapter consists of audio-visual latent token compression, audio-visuall feature fusion, and bottleneck module.
We propose utilize latent tokens, which are small, randomly initialized vectors that serve as placeholders for condensed information from the audio and visual modalities. For layer 1, we have m latent tokens for audio, denoted L' and m latent tokens for visual, denoted as L, These tokens significantly fewer in number compared to the total number of tokens from the audio visual inputs, which helps reducing the computational complexity.
In this step, the adapter uses cross- modal attention to compress the information from all the audio or visual tokens into the corresponding latent tokens. This achieved by applying the cross-modal attention operation between the input tokens and the latent tokens of the same modality. The compressed latent tokens capture the most relevant information from the original tokens, which is essential for efficient information transfer between modalities.
For audio and visual inputs, the compression is given by:
= feL,
Sy fe Xe,
(3)
where fe is the cross-attention function, X! and X are the audio and visual tokens at layer I, and S represents the resulting latent summary tokens.
After compressing the modality-specifii information into latent tokens, the adapter fuses the com- pressed information from one modality with the full of tokens from the other modality. This is done through another round of cross modal attention, which allows the model to integrate audio and visual cues effectively. For cross-modal fusion:
x fc(" S,,,,,), X = fc Sy S%)
where X! and X! represent the newly computed audio and visual representations that incorporate information from both modalities.
The final step in the adapter involves lightweight module that refines the fused audio-visual representations into more discriminative features suitable for downstream tasks. This module consists of bottleneck structure with a down-prooe projecttottionn layer (6down)), mon-lineat activation function (o)) and an up- up-projection layer (f"P). For the audio-visual and visual-audia pathways:
z av = guP (o (x! (x).))), z = 8d0 (" (x!"
(5)
where 7! and Z! are output features that are passed to subsequent layers or used for task-specific predictions.
5

---

Bottennck Block
Bettlencck
Up-Project
Up-Project
Nontinearity
Nonlinearity
Down-Project
Down Projesroject
Feature Fusion
Self Attention
Tokan Compression
Token Compression
Audio Takens
V
to
Latent
Visual Tokens
Cross-Modal Adapter
b) Unimodal Adapter
The structure of adapters. Cross- Cross-Modal Modal Adapter (CMA), consisting of token compression, mul- timodal feature fusion and bottleneck block; b) Unimodal Adapter (UA), consisting token compression, self-attention and bottleneck block.
Figure 3:
Unimodal Adapter. While cross-modal adapters effectively introduce interactions between audio and visual tokens, they may not always be suitable for scenarios where auditory and visual content do not perfectly align, potentially introducing irrelevant information. Additionally, some multimodal data might primarily rely on one modality, or another modality may be completely missing, making inter-modal interactions less effective. Hence, it is also necessary to introduce intra-modal interactions and design unimodal adapters, which can address these issues by providing targeted improvements that compensate for the limitations of cross-modal adapters. As shown in the right of Figure3] after obtaining compressed tokens S% and S the self-attention layer ingests them and outputs newly computed representations.
X fo(X), S S)), X fe(X!, S S.
(6)
where fa the cross cross-attention function, X and x represent the output audio and visual representa- tions, respectively.
In this work, we propose to utilize MoE strategy introduce intra- and inter-modal interactions synergistically. By integrating these two types of adapters, our model can dynamically adjust the attention paid to unimodal and cross-modal information according to the scenario.
Experimental Analysis
To demonstrate the capabilities of our proposed AVMoE in solving diverse audio-visual tasks, conduct extensive experiments on four downstream tasks, i.e., Audio- Visual Event Localization (AVE), Visiol Video Parsing (AVVP), Audio-Visual Segmentation (AVS), and Audio- Visual Question Answering (AVQA). More details about audio-visual tasks and datasets will be illustrated in the Appendix.
4.1 Audio- Visual Event Localization
For the AVE task, we adopt the overall segment-wise accuracy predicted event categories as the evaluation metric, and the event label of each video segment is required to be predicted in fully-supervised manner. As shown in Table [ we compare our proposed AVMoE with previous methods on the test set of AVE dataset. We focus on LAVisH (30) and DG-SCT (1I), they also explore parameter-efficient fine-tuning methods like ours and have achieved impressive results on this benchmark.

---

Table I: Audio-Visual Event Localization. Comparison with previous methods in fully-supervised manner. The performance is evaluated the of AVE dataset with classification accuracy. The visual and audio encoders are pre-trained on the ImageNet Audioset, respectively, means that no official code was provided to report some of the baseline-speseeee speciifc metrics.
Visual Encoder
Audio Encoder
Trainable Params (%)
Total Params (M)
Method
Ace
AVT
29
VGG-19
VGGish
6.8
231.5
76.8
MPN
62
VGG-19
VGGish
N/A
N/A
77.6
PSP
66
VGG-19
VGGish
0.8
217.4
77.8
DPNet
46
VGG-19
VGGish
N/A
N/A
79.7
AVEL
ResNet-152
VGGish
2.7
136.0
74.0
AVSDN
28
ResNet- 152
VGGish
5.7
140.3
75.4
CMRAN
59
ResNet-152
VGGish
10.7
148.2
78.3
MM-Pyramid
63
ResNet-152
VGGish
25.0
176.3
77.8
CMBS 58
ResNet-152
VGGish
6.6
216.7
79.7
LAVisH 30
ViT-B-16 (shared)
4.4
107.2
75.3
LAVisH 30
ViT-L-16 (shared)
4.3
340.1
78.1
AVMoE (Ours)
ViT-B-16 (shared)
31.8
150.4
76.4
AVMoE (Ours)
ViT-L-16 (shared)
32.6
483.1
79.2
LAVisH 30
Swin-V2-BB (shared)
4.4
114.2
78.8
LAVisH 30
Swin-V2-I. (shared)
2.7
238.8
81.1
AVMoE (Ours)
Swin-V2-B (shared)
41.1
206.6
79.4
AVMoE (Ours)
Swin-V2-L (shared)
39.4
374.4
81.5
LAVisH [30]
Swin-V2-L
HTS-AT
30.6
374.9
78.6
DG-SCT
S2in
HTS-AT
43.6
461.3
82.2
AVMoE (Ours)
Swin-V2-L
HTS-AT
34.9
404.0
82.6
For fair comparison, we train our AVMoE with shared pre-trained visual backbones (VIT-B, ViT-L, Swin-V2-B, Swin-V2-L) and separate audio-visual backbones (HTS-AT and Swin-V2-L), Firstly, can be seen that our method achieves consistently better performance than LAVisH across different shared backbones, which indicates that our method can be adapted different backbones. We note that the structure of Swin-V2-L. is better than VIT, since Swin- V2-L allows hierarchical feature representation and more efficient computation by constraining self-attention within local windows and progressively expanding them. Secondly, when utilizing distinct pre-trained backbones for each modality, our model outperforms LAVisH and DG-SCT by 4.0% and 0.4%, respectively. It should be noted that the number and proportion of trainable parameters is less than that of DG-SCT with complicated attention mechanisms, showcasing the efficiency and superiority of our model. Lastly, also observe that the audio encoder pre- pre-trained on Audioset further improve the performance, which indicates that combining different modalities beneficial for visual learning.
4.2
Audio-Visual Video Parsing
Compared to the AVE task, the AVVP task a more complex and challenging multimodal task involving analyzing and understanding audio and visual information within a video to identify and categorize events. This task aims to segment a video into temporal events and predict their modality- specific categories, which may include actions that audible, visible, both. A key challenge in AVVP is that audio and visual modalities may not be fully correlated. For instance, due to limitations in camera field-of-view or occlusions, some events may occur in only one modality. Therefore, effectively integrating audio and visual information while mitigating the interference of irrelevant modality information is crucial.
As shown in Table our AVMoE has significant advantages over DG-SCT and MGN on the AVVP task. For example, the audio-visual event parsing metric at segment-level exceeds DG-SCT by 2%. which indicates that the scheme of MoE suitable for balancing weight assignment for multi-modal information.

---

Table 2: Audio- Visual Video Parsing, Comparison with previous methods on the test set of LLP datasct. We conduct comparative experiments two kinds of annotation: segment-level and event-level. The Type computes the average audio, visual, and audio-visual event evaluation results, and the Event considers all audio and visual events for each sample to compute F-score.
Segment-level
Event-level
Method
A
V
AV
Type
Event
A
V
AV
Type
Event
AVE 54
49.9
37.3
37.0
41.4
43.6
43.6
32.4
32.6
36.2
37.4
AVSDNI28
47.8
52.0
37.1
45.7
50.8
34.1
46.3
26.5
35.6
37.7
53
60.1
52.9
48.9
54.0
55.4
51.3
48.9
43.0
47.7
48.0
MGN 40
60.7
55.5
50.6
55.6
57.2
51.0
52.4
44.4
49.3
49.2
DG-SCTII]
59.0
59.4
52.8
57.1
57.0
49.2
56.1
46.1
50.5
49.1
AVMoE(Ours)
62.1
60.0
54.4
58.8
59.0
51.8
55.7
47.6
51.7
50.2
Table Audio Visual Segmentation. Comparison with previous methods under and MS3 settings of AVSBench dataset. My and Myx denote the mean J and metric values over the whole datasct.
Setting
Visual Encoder
Audio Encoder
Trainable Params (%) 4
Total Params (M) 4
Method
S4
MS3
M3
Mx
M
Mx
AVS 665
PVT-v2
VGGish
58.7
174.5
78.7
87.9
54.0
64.5
LAVisI
Swin- V2-L
(shared)
14.0
266.4
88.0
49.8
60.3
LAVisH 30
Swin- V2-L
HTS-AT
47.6
389.7
78.0
87.0
49.1
59.9
DG-SCT TT
Swin- V2-L
HTS-AT
61.5
594.8
80.9
89.2
53.5
64.2
AVMoE (ours)
Swin-V2-L
HTS-AT
54.4
501.2
81.1
89.7
54.5
68.7
4.3
Audio- Visual Segmentation
To explore whether the model can associate the visual regions with the sound components, compare our approach with state-of-the-art models on the AVS task. The Jaccard index J [12] and F-score F are used the evaluation metrics, where and F denote the region similarity and contour accuracy, respectively. As illustrated in Table 33] our approach AVMoE significantly outperforms DG-SCT and LAVisH under both S4 and MS3 settings. which verifies that the scheme of MoE also beneficial for audio-visual segmentation. Our model shows more significant improvements under the MS3 setting of multi-sound sources, indicating that the model with AVMoE strategy can deal with more complex scenarios.
shows the qualitative results of our AVMoE model and DG-SCT on the AVS task. As be seen, under the S4 setting, our model locates and segments the sounding objects more accurately than DG-SCT. For the case of ambulance siren (column #1), DG-SCT mistakenly locates the ambulance and the car nearby at the same time, while our model can exclude the car that hasn't produced sound. For the of dog barking (column #3), AVMoE model contours the object shapes (e.g., tail) more perfectly. Under the MS3 setting, compared to DG-SCT which cannot locate all sound sources in some cases, we can almost locate and segment each sound source. It can be seen that even in the case of three sound sources (column #4), we can still segment each instrument well, which verifies the effectiveness of our AVMoE model.
Figure A
4.4 Audio- Visual Question Answering
We also want to explore whether our model can generalize to complex audio-visual tasks. To this end, we conduct experiments on the AVQA task and compare our model with previous leading methods. The AVQA task contains three categories of questions (audio, visual, and audio-visual), which involve questions related to audio (AQ), visual (VQ), or both audio and visual information simultaneously (AVQ). The multiple forms of questions challenge adaptability to various modalities and the spatio-temporal reasoning ability of each model.

---

S4 Setting
MS3 Setting
Bc BEADES
DG-SCT
AVMoE (Ours)
BEADLES
Ground
Truth
Figure 4: Qualitative examples of AVMoE and DG-SC7 under the S4 setting and MS3 setting of the AVS task.
Table Audio-Visual Question Answering. Comparison with previous methods on the test set MUSIC- AVQA dataset. We report accuracy on three types of questions, i.c., Audio Question (AQ), Visual Question (VQ), and Audio- Visual Question (AVQ). LAVisH denotes our implementation version of LAVisH.
Visual
Audio
Trainable Params (%),
Total
Method
AQ
VQ
AVQ
Avg
Encoder
Encoder
Params (M)
AVSD [0
VGG-19
VGG-like
N/A
N/A
68.5
70.8
65.5
67.4
Pano-AVQA
64
Faster RCNN
VGG-like
N/A
N/A
70.7
72.6
66.6
68.9
ST-AVQA
ResNet-18
VGG-like
11.2
94.4
74.1
74.0
69.5
71.5
LAVisH
Swin-V (sharedd
249.8
75.7
80.4
70.4
74.0
LAVisH
30
Swin-V22 V2-L
HTS-AT
28.7
367.4
75.4
79.6
70.1
73.6
DG-SCT
Swin-V2-L
HTS-AT
50.0
$20.2
77.4
81.9
70.7
74.8
AVMoE (ours)
Swin-V2-L
HTS-AT
42.7
456.6
77.6
82.7
71.9
75.7
reports the experimental results on the Music-AVQA dataset across three distinct scenarios. Notably, our method exhibits superior generalization abilities and consistently achieves State-Of The-Art (SOTA) performance in all scenarios. Especially in the challenging scenarios of AVQ, our AVMoE outperforms DG-SCT by 1.2% (71.9% v.s. 70.7% with fewer trainable and total parameters. which verifies the effectiveness and efficiency of our approach. The dynamic adaptability of MoE strategy not only establishes the robustness of audio-visual models but also illustrates the potential of leveraging the flexible strategy to enhance performance in complex audio visiol tasks. The comprehensive experimental results across four audio-visual downstream tasks illustrate that our approach achieves significant improvements while efficiently transferring from frozen backbones to various downstream tasks.
5
Ablation Studies
5.1 Number of Experts
As shown in Table we first investigate the impact of the number of experts (i.c., cross-modal adapters and uni-modal adapters) on the tasks of AVE, AVS, and AVQA. It can be seen that increasing the number of experts consistently enhances the model's performance, thereby validating the effectiveness of the MoE strategy. Besides, we observe that the performance of our model with only one CMA and one UA still exceeds that of LAVisH. Especially under the MS3 setting of Audio- Visual Segmentation (AVS) task, it has improved by 7.2% compared with LAVisH (67.5% v.s. 60.3%). We conjecture that introducing more experts may further improve the abilities of the model. However, due the limitation of GPU memory, did not discuss the results in this work.

---

Table 5: AVMoE design. CMA and UA denote cross-modal adapter and uni-modal adapter, respectively.
Num. Experts
AVS (S4)
AVS(MS3)
AVQA (Acc.)
AVE
(Acc.)
CMA
UA
MM
My
MM
AQ
VQ
AVQ
Avg
1
80.2
88.6
51.8
67.5
75.7
81.4
70.9
74.7
81.3
2
80.6
89.1
53.1
68.1
76.3
82.0
71.4
75.1
81.8
2
80.8
89.2
53.6
68.4
76.2
82.1
71.6
75.1
82.3
2
2
81.1
89.7
54.5
68.7
77.6
82.7
71.9
75.7
82.6
Table 6: Modality Ablation. "A" and "V" denote audio and visual, respectively. All models well-trained on audio-visual modalities and tested on different modalities.
Methods
Modalities
AVE(Acc.)
AVS(M²)
AVQA(Acc)
72.1
79.1
67.3
DG-SCT
A+V
82.2
89.2
74.8
78.4
85.4
73.2
AVMoE (Ours)
A+V
82.6
89.7
75.7
5.2 Modality Ablation
In order to explore how our well-trained AVMoE model handles different modality inputs, we conduct experiments of our model and DG-SCT on visual-only data and audio-visual data. As shown in Table we compare our AVMoE model with DG-SCT on AVE, AVS, and AVQA tasks. It can be seen that our approach achieves significant performance on the test scenarios of visual-only data and audio-visual data. We observe that, our AVMoE model, when compared to DG-SCT, there is no significant decrease in accuracy on visual-only data. These results indicate that our model with the MoE scheme can still achieve good performance even in the absence of audio information. Although DG-SCT designs a more complex architecture of dual-guided mechanism in adapters, only introducing ross-modal adapters may bring irrelevant information in these challenging cases.
temporall attention
Moreover, explore whether the model can dynamically activate experts, we visualize the activation weight of each expert varies across parallel expert layers. The activation ratio experts determined by calculating the ratio of each expert's selection frequency in each MoE layer to the total number of tokens. The visualization and more details will be discussed in Appendix.
Conclusion
In this paper, develop flexible framework that introduces Mixture of Experts (MoE) for audio- visual learning. Specifically, we inject unimodal adapters and cross-modal adapters as experts into the frozen pre-trained model and AVMoE dynamically combine the abilities of adapters according to different scenarios through the MoE strategy. Experimental results illustrate that our AVMoE has achieved greatly improved performance complex tasks (such as AVQA and the MS3 setting of AVS). The ablation studies and qualitative results demonstrate that our model benefits significantly from the AVMoE strategy. However, due computational resource limitations and efficiency considerations, we only investigate small number of experts in the MoE scheme. In subsequent work, we will explore how to efficiently introduce more experts into our model.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (No.62172101) in part by the Science and Technology Commission of Shanghai Municipality (No.23511 100602), and supported by the Postdoctoral Fellowship Program of CPSF (No. GZC20230483).
10

---

References
[1] Triantafyllos Afouras et al. "Self-supervised learning of audio-visual objects from video". In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16. Springer. 2020, pp. 208-224.
[2] Hassan Akbari et al. "Alternating gradient descent and mixture- of ex-exes for integrated multimodal perception". In: Advances in Neural Information Processing Systems 36 (2023),
pp. 79142-79154.
[3] Relja Arandjelovic and Andrew Zisserman. "Look, listen and learn". In: Proceedings of the IEEE International Conference on Computer Vision. 2017, pp. 609-617.
[4] Bing Cao et al. "Multi-modal gated mixture of local-to-global experts for dynamic image fusion". In: Proceedings of the IEEE/CVF International Conference Computer Vision. 2023,
pp. 23555-23564.
[5] Ke Chen et al. "HTS-AT: A hierarchical token-semantic audio transformer for sound classifi- cation and detection". In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEF. 2022, pp. 646-650.
[6] Tianlong Chen et al. "odemy-moe: Adaptive multi-task vision mixture-of-experts" In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. 2023, pp. 17346-
17357.
[7] Zailong Chen al. "Question- Global-Local Video Understanding Network for Audio- Visual Question Answering". In: JEEE Transactions on Circuits and Systems for Video Tech- nology (2023).
[8] Ying Cheng et al. "Improving multimodal speech enhancement by incorporating self- supervised and curriculum learning". In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2021, pp. 4285-4289.
[9] Ying Cheng et al. "Look, listen, and attend: Co-attention network for self-supervised audio- visual representation learning". In: Proceedings of the 28th ACM International Conference on Multimedia. 2020, pp. 3884-3892.
[10] Nan Du et al. "Glam: Efficient scaling of language models with mixture-of-experts"" In: International Conference on Machine Leurning. PMLR. 2022, pp. 5547-5569.
[11] Haoyi Duan et al. "Cross modal Prompts: Adapting Large Pre-trained Models for Audio-Visua Downstream Tasks". In: Advances in Neural Information Processing Systems 36 (2024). [12] Mark Everingham al. "The pascal visual object classes (voc) challenge". In: International journal of computer vision 88 (2010), pp. 303-338.
[13] William Fedus, Barret Zoph, and Noam Shazeer. "Switch transformers: Scaling to trillion pa- rameter models with simple and efficient sparsity". In: Journal of Machine Learning Research 23.120 (2022), pp. 1-39.
[14] Jort Gemmeke al. "Audio set: An ontology and human-labeled dataset for audio events".
In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE. 2017, pp. 776-780.
[15] Dawei Hao et al. "Improving audio-visual segmentation with bidirectional generation". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. 3. 2024, pp. 2067-2075.
[16] Hussein Hazimeh al. "Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning". In: Advances in Neural Information Processing Systems
34 (2021), pp. 29335-29347.
[17] Zili Huang et al. "Investigating self-supervised learning for speech enhancement and separa- tion". In: ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2022, pp. 6837-6841.
[18] Luo-Hsuan Hung et al. "Boosting self-supervised embeddings for speech enhancement". In: arXiv preprint arXiv:2204.033. (2022).
[19] Robert Jacobs al. " "Adaptive mixtures of local experts". In: Neural computation 3.1 (1991),
pp. 79-87.
[20] Albert Q Jiang et al. "Mixtral of experts". In: arXiv preprint arXiv:2401.04000 (2024).
[21] Diederik P Kingma and Jimmy Ba. "Adam: A method for stochastic optimization", In: arXiv
preprint arXiv: /412 69860980 6980 (2014).

---

[22] Bruno Korbar, Du Tran, and Lorenzo Torresani. *Cooperative learning of audio and video models from self-supervised synchronization" In: Advances in Neural Information Processing Systems. 2018, 1011,10 pp. 7763-7774.
[23] Haoyuan Lan, Yang Liu, and Liang Lin. "Audio-visual contrastive learning for self-supervised action recognition". In: arXiv preprint arXiv:2204. (2022).
[24] Dmitry Lepikhin et al. "Gshard: Scaling giant models with conditional computation and automatic sharding". In: arXiv preprint arXiv:2006. 16668 (2020).
[25] Mike Lewis et al. "Base layers: Simplifying training of large, sparse models". In: International Conference on Machine Learning. PMLR. 2021, pp. 6265-6274.
[26] Guangyao Li, Wenxuan Hou, and Di Hu. "Progressive Spatio-temporal Perception for Audio- Visual Question Answering". In: Proceedings of the ACM International Conference on Multimedia. 2023, pp. 7808-7816.
[27] Guangyao Li al. "Learning answer questions in dynamic audio-visual scenarios". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. 2022,
pp. 19108-19118.
[28] Yan-Bo Bo Lin, Yu-Jhe Li, and Yu- Yu-Chiang Frank Wang. "Dual-modality seq2seq network for audio-visual event localization". In: ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEFE. 2019, pp. 2002-2006.
[29] Yan-Bo Lin and Yu-Chiang Frank Wang. "Audiovisual transformer with instance attention for audio-visual event localization". In: Proceedings of the Asian Conference on Computer Vision. 2020.
[30] Yan-Bo Lin et al. "Vision transformers are parameter-efficient audio-visual learners". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, pp. 2299-2309
[31] Yuhang Ling et al. "TransAVS: End-to-End Audio-Visual Segmentation with Transformer". In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2024, pp. 7845-7849.
[32] Tianyu Liu et al. "Induction Network: Audio-Visual Modality Gap-Bridging for Self- Supervised Sound Source Localization". In: Proceedings of the 31st ACM International Conference on Multimedia. 2023, pp. 4042-4052.
[33] Ze Liu et al. "Swin transformer v2: Scaling up capacity and resolution". In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, pp. 12009-12019. [34] Ze Liu et al. "Swin transformer: Hierarchical vision transformer using shifted windows". In: Proceedings of the IEFF/CVF international conference on computer vision. 2021, pp. 10012- 10022.
[35] Yuxuan Lou et al. "Cross-token modeling with conditional computation". In: arXiv preprint arXiv:2 00:::210000000 00000000000000000000000000000000 (2021).
[36] Jaqq Ma et "Modeling task relationships in multi-task learning with multi-gate mixture-of- experts". In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining. 2018 pp. 1930-1939.
[37] Tanvir Mahmud and Diana Marculescu. "Ave-clip: Audioclip-based multi-window temporal transformer for audio visual event localization". In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023, pp. 5158-5167.
[38] Yuxin Mao et al. "Contrastive conditional latent diffusion for audio-visual segmentation". In: arXiv preprint arXiv:2307. 16579 (2023).
[39] Yuxin Mao al. "Multimodal variational auto-encoder based audio-visual segmentation". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023, pp. 954- 965.
[40] Shentong Mo and Yapeng Tian. "Multi-modas grouping network for weakly-supervised audio- visual video parsing". In: Advances in Neural Information Processing Systems 35 (2022),
pp. 34722-34733.
[41] Basil Mustafa et al. "Multimodal contrastive learning with limoe: the language- image mixture of experts". In: Advances in Neural Information Processing Systems 35 (2022), pp. 9574-9576.
[42] Xiaonan Nie et al. "Evomoe: An evolutional mixture-of-expers training framework via dense- to-sparse gate", In: arXiv preprint 2 (2021).
12

---

[43] Andrew Owens and Alexei A Efros. "Audio-visual scene analysis with self-supervised multi- sensory features". In: Proceedings of the European Conference on Computer Vision (ECCV). 2018, pp. 631-648.
[44] Kyu Ri Park, Youngmin Oh, and Jung Uk Kim. "Enhancing Mudio-Visual Question Answering with Missing Modality via Trans-Modal Associative Learning.. In: ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024,
pp. 5755-5759.
[45] Adam Paszke et al. "Pytorch: An imperative style, high-performance deep learning library".
In: Advances in neural information processing systems 32 (2019).
[46] Varshanth Rao et al. "Dual perspective network for audio-visual event localization". In:
European Conference Computer Vision. Springer. 2022, pp. 689-704.
[47] Carlos Riquelme al. "Scaling vision with sparse mixture of experts". In: Advances in Neural Information Processing Systems 34 (2021), pp. 8583-8595.
[48] Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. "Hash layers for large sparse models". In: Advances Neural Information Processing Systems 34 (2021), pp. 17555-17566. [49] Pritam Sarkar and Ali Etemad. "Self-supervised audio-visual representation learning with relaxed rrosl synchronicity". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. 8. 2023, pp. 9723-9732.
[50] Idan Schwartz, Alexander G Schwing, and Tamir Hazan. "A simple baseline for audio-visual scene-aware dialog". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, pp. 12548-12558.
[51] Noam Shazeer et al. "Outrageously large neural networks: The sparsely-gated mixture-of- experts layer". In: arXiv preprint 1.X.iv1701.0665 (2017).
[52] Sheng Shen et al. "Scaling vision-language models with sparse mixture of experts". In: arXiv preprint arXiv:2303. 07222 (2023).
[53] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. "Unified multisensory perception: Weakly- supervised audio-visual video parsing".. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16. Springer. 2020, pp. 436-454.
[54] Yapeng Tian et al. "Audio-visual event localization in unconstrained videos". In: Proceedings of the European conference on computer vision (ECCV). 2018, pp. 247-263.
[55] Laurens Van der Maaten and Geoffrey Hinton. "Visualizing data using t-SNE." In: Journal of machine learning research 9.11 (2008).
[56] Yu Wu and Yi Yang. "Exploring heterogeneous clues for weakly-supervised audio-visual video parsing". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 1326-1335.
[57] Yu Wu et al. "Dual attention matching for audio-visual event localization". In: Proceedings of the IEEE/CVF international conference on computer vision. 2019, pp. 6292-6300.
[58] Yan Xia and Zhou Zhao. "Cross-modal background suppression for audio-visual event lo- calization". In: Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition. 2022, pp. 19989-19998.
[59] Haoming Xu et al. "Cross-modal relation-aware networks for audio-visual event localization". In: Proceedings of the 28th ACM International Conference Multimedia. 2020, pp. 3893- 3901.
[60] Hanyu Xuan et al. "A proposal- proposal-based paradigm for self-supervised sound source localization in videos". In: Proceedings of the IEFE/CV Conference on Computer Vision and Pattern Recognition. 2022, pp. 1029-1038.
[61] Pinci Yang al. "Avqa: A datuset for audiolvisual question answering videos". In: Pro- ceedings of the 30th ACM international conference on multimedia. 2022, pp. 3480-3491. [62] Jiashuo Yu, Ying Cheng, and Rui Feng. "Mpn: Multimodal parallel network for audio-visual event localization". In: 2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE. 2021, pp. 1-6.
[63] Jiashuo Yu et al. "Mm-pyramid: Multimodal pyramid attentional network for vistull event localization and video parsing". In: Proceedings of the 30th ACM international conference on multimedia. 2022, pp. 6241-6249.
13

---

[64] Heeseung Yun et al. "Pano-avqu: Grounded audio-visual question answering on 360deg videos". In: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021, pp. 2031-2041.
[65] Jinxing Zhou et al. "Audio-visual segmentation". In: European Conference Computer Vision. Springer. 2022, pp. 386-403.
[66] Jinxing Zhou et al. "Positive sample propagation along the audio-visual event linc". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021,
pp. 8436-8444.
[67] Yanqi Zhou et al. "Mixture-of-experts with expert choice routing". In: Advances in Neural Information Processing Systems 35 (2022), pp. 7103-7114.
[68] Barret Zoph et al. "St-moe: Designing stable and transferable sparse expert models". In: arXiv preprint arXiv:2202.0800 (2022).
14

---

A Audio-Visura Tasks and Datasets
To demonstrate the capabilities of our proposed AVMoE in solving diverse audio-visudl tasks, we conduct extensive experiments on four downstream tasks, i.e., Audio- Visual Event Localization (AVE), Audio-Visiie Video Parsing (AVVP), Audio-Visual Segmentation (AVS), and Audio-Visual Question Answering (AVQA). The transferring framework is shown in Figure
A.1
Audio-Visual Event Localization
AVE task is defined as an event that both visible and audible in a video segment [4]. The AVE dataset focuses combining audio and visual modalities temporally mark the boundaries of all audio-visual events. It contains 4, 4,143 videos covering 28 event categories, with each video lasting up to 10 seconds and including least one 2s long audio-visual event. For this task, first divide the video into 10 consecutive time segments. Then, we use the frozen transformers (e.g.. ViT, Swin or HTS-AT) optimized by the AVMoB adapters extract the audio and visual features that incorporate cross-modal interaction information from these segments. Last, we predict the audio-visual events by feeding the concatenated audio-visual features through two MLP layers, as shown in Figure Referring to previous methods 55), 46 58, 66). we calculate the proportion of time segments that are correctly predicted and use this as standard to evaluate the performance of our method.
A.2 Audio-Visual Video Parsing
AVVP task aims to parse videos into temporal event segments and label them as audible, visible, or both. For this task, we use the Look, Listen, and Parse (LLP) dataset (53). which contains 11,849 videos from different domains, covering 25 categories. LLP is semi-supervised annotation dataset, and each video has video-level event annotations. Only 1,849 randomly selected videos have second- by-second annotations for audio and visual events. We made improvements based on MGN(40) by using AVMoE to enhance the Transformer layers in it, as shown in Figure5) Referring to previous methods 53 40), F-scores are used to evaluate segment-level and event-level predictions of audio, visual, and audiovisual events which are predicted by our method.
A.3 Audio- Visual Segmentation
AVS task aims to achieve pixel-level segmentation of visual objects based audio. We conduct experiments on the AVSBench(65) (65). a dataset containing both single- souree- segmentation and multiple- source segmentation. The single-source dataset contains 4,932 semi-supervised videos. During the training process, only the first sample frame of the video is fully labeled, but during evaluation, all video frames need to be predicted. The multi-source dataset contains 424 fully supervised videos with every frame labeled. For this task, we optimize the frozen transformer with trainable adapters to replace the pre- pra-tr visual encoder and audio feature extractor of the U-Net used in (V5( Then, the audio and visual features are fused and ingested into the decoder for prediction, as shown in Figure3] Referring previous methods(63) we calculate the mean Intersection-over- Union (mIoU) of the predicted segmentation and the ground truth masks to evaluate our method.
A.4 Audio-Viis Visual Question Answering
AVQA task aims to answer questions about different visual objects, sound sources and their associa- tions in the video. We conduct experiments on the MUSIC-AVQA dataset (27). which contains more than 45K question-answer pairs covering 33 different question templates. For this task, we utilize the trainable adapter optimize the frozen Transformer and then replace the pre-trained visual and audio encoders of the baseline proposed in the AVQA dataset, as shown in Figure5] Referring to previous methods 27, the answer prediction accuracy is used to evaluate the performance of methods.
B Audio- Embeddings
To enhance the fusion of audio-visual features, we follow the previous work 11], which utilizes pre-trained vision and audio transformers to encode visual and audio inputs, respectively. The encode process is as follows.
15

---

Update Frozen
Yiua Tarakorw
Gronging Baak
AMM
Tempona Graundra
Coudal Coensiing
Masee]
Greann 2014
Turskows
Qum Chooder
Vocall staoroenno
Nuaio Trantionee
Tembrrr
Aedo
(orrne B
restument to
Figure S: Applying AVM to audio-visual downstream tasks: audio-visual event localization, audio-visual vidco parsing, visual segmentation, and audio-visual question answering. The purple and blue modules are frozen pre-trained visual and audio models, and the red modules are our proposed trainable adapter modules, and the remaining modules are set up with reference the baseline model of these tasks, with some parameters trainable.
Visual embedding. For the given video sequence, we sample a set of RGB visual frames (Vi) T XXXxXXXX x3, where and W denote the height and width of frames, respectively. These video frames are sampled at the rate of I fps and then decomposed into non-overlapping patches. The patches are treated as individual tokens for sequence processing by the transformer model, and we can obtain the visual embeddings E Rxxd where d denotes the embedding dimension.
Audio embedding. For the given audio stream, we first employ Short-Time Fourier Transform (STFT) to convert waveforms into spectrograms {Ar) RTxLxP, where L and F denote the time and frequency bins, respectively. Similarly, the spectrograms are also divided into & non-overlapping patches and then projected into audio embeddings a Rexd
C Implementation Details
For the above dataset, we extract visual frames at one frame per second and audio frames from video files at 8,000fps. We use the pre-trained Swin-V2-Large (34) as the visual transformer module with a spatial resolution of 192x192, and the pre-trained HTT)-AT(5) as the audio transformer module. All of their parameters are frozen. We use the gradient accumulation method to accommodate the large parameters of the model, with weight updates after every batches of training. For AVE and AVVP tasks, 32 latent tokens and downsampling factor of 8 are used in the AVMoE adapter. For the AVS task on AVSBench and the AVQA task on MUSIC-AVQA, we use two latent tokens and set the downsampling factor and the number of group convolutions to 8 and 4, respectively.
For all of our experiments, we utilize the Adam 21 optimizer train our models and the scheduler to make the learning rate decay to 0.35 times its original value after every 3 epochs. We set the learning rate of the AVMoE adapter to 5e-4, while the learning rates of the final prediction layer are the same as previous work [11), 5e-6 for AVE, 3e-4 for AV 3e-4 for the $4 setting of AVS, 1.5e-4 for the MS3 setting of AVS, and le-4 for AVQA. For audio pre-processing, we compute audio spectrograms via the Py PyTorch|455 kaldi fbank with 192 triangular mel-frequency bins and set frameshift over 5.2ms. For the task with audio-visual shared Transformer, we spliced the input channels of the audio spectrogram by tensor replication into 3 channels to match the input dimensions of the V2 model.
For these audio-visual downstream tasks, all experiments are conducted on 8x NVIDIA 3090 (24G) GPUs, and the batch size on a single GPU varies depending on the parameters of the models. The one GPU batch size is set to be 1 for AVE and AVVP tasks, and 2 for AVS and AVQA tasks. Besides, we employ a gradient accumulation strategy to mitigate the computational resource limitations. To avoid overfitting, we set the early-stop praattraranttiiniaiioonn to end the training process after five epochs with no improvement model performance. Most of the experiments converge to the best results in about ten epochs.
16

---

Input:
Car
AVMeE
Car
Car
Audio
Car
Spppochhh 20
Speech
Speech
Specohlo
DG-SCT
Speech
Car
Speech
Speech
Speachiek
Figure 6: The qualitative experimental results on the AVVP task.
Audio Branch
Visual Branch
Acthotoon Poobabllly of Expets Layor
Activasion Probe by Eeeots Bach Layor
Audio-Visual Data
Layer
Visual-Only Data
50
Figure 7: The activation probability of experts. Experts #1 and #2 denote cross-modal adapters, and experts #3 and denote unimodal adapters.
Qualitative Experiments of Andio-Visual Video Parsing
Figure sJsoww the qualitative experimental results on the AVVP task. this example, the video label only includes car, and the audio label includes car and speech, which are not exactly the same. We observe that the visual predictions of DG-SCT are affected by the audio features, resulting in visual prediction of speech, which does not exist in the video frame. On the other hand, our AVMoE retains unimodal information and has MoE modules adjust the weight distribution for audio-visual features. This avoids the interference of inconsistent information and obtains the correct result. As for the audio predictions, our model can better localize temporal boundaries of events compared to DG-SCT, which reveals that our model is capable of dealing with complex scenarios.
17

---

E
The activation probability of experts
The activation visualization of experts shown in Figure 7
be observed that: 1)In the visual branch, The distribution of expert activation is relatively even, while in the audio branch, the activation probability of the crossmodal adapter higher. We hypothesize that this may be due the visual information plays a vital role in the processing of audio-visum data. 2) When the model deals with visual-only data, the activation probability of the unimodal adapter in the visual branch becomes higher, which demonstrates the ability of our AVMoE to dynamically combine adapters according to different scenarios.
In addition, we observe that when the model is processing visual-only data, the activation probability of the unimodal adapter the visual branch becomes higher, which proves the ability of our AVMoE to dynamically activate adapters according to different scenarios.
F Feature Visualization
Figure X visualizes the learned audio and visual features by employing t-SNE [55]. Each point represents the feature of individual audio or visual event, with each color indicating specific category. The visualizations reveal that the features extracted by the proposed AVMoE model are more compact within classes and more distinct between classes, which demonstrates that our AVMoE can learn discriminative features for each modality across various audio-visual downstream tasks.
Visual
Audio
AVE.
Original
Ours
Original
Ours
AVS
Original
Ours
Original
Ours
Figure 8: Qualitative visualizations of visual and audio features of original and Ours on AVE and AVS tasks.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We describe our scope and contributions in the abstract and introduction
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We analyze the issues with our method based on experimental results in Section and explore our limitations in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer:
[NA]
Justification:
We do not introduce any new theories.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We first describe the pipeline of our method in Section 3 and then detail the implementation details and training parameters in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
20

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided the URL of our code in abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https /nips
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental settings, benchmark and training details can be found in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Benchmarks and previous methods do not provide any suggestions practices regarding the experiment statistical significance. The cost of single experiment is very expensive, which means that the number of experiments that meet statistical significance is unacceptable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiments compute resources are reported in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: atttps://eeurpee wwtprriis.ccppppplppe
Answer: [Yes]
Justification:
We fully comply with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: 3D object detection in point clouds is mainly used for academic research and technological development, without involving scenes that have an impact on society, specific groups, and is not directly associated with any potential negative social applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not involve generative models, pretrained models, and do not have risk of abuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We adhere to the usage restrictions of all assets and correctly cite their sources.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer[[NA)
Justification: We do not release new assets now, and the code will be open-sourced after the paper is accepted.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:[NA]
Justification:
We do not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "TfMSxGgUyx", "reviewer": "Reviewer_pdXh", "summary": "This paper proposes AVMoE, a new way of adding adapters to pretrained audio-visual models.\n\nAVMoE extends the LAVisH approach in two ways: \n(1) Using unimodal adapters in addition to cross-modal LAVisH adapters.  \n(2) Adding a router network to dynamically combine the output of multiple unimodal/cross-modal adapters. \n\nAV-MoE outperforms two existing methods (LAVisH and DG-SCT) on three audio-visual tasks: event localization, segmentation, and question answering.", "questions": "- When comparing results with shared Swin-V2-L encoder versus using seperate Swin-V2-L & HTS-AT encoders, why does LAVisH performance degrade whereas AVMoE improves?\n- What is the meaning behind the asterisk in LAVisH row of Table 3: Audio-Visual Question Answering?", "limitations": "As mentioned in the Weaknesses section:\n1. Importance of each component of proposed method is not well studied,\n2. The choice/effect of different hyperparameters compared to LAVisH is not justified,\n3. Method details such as model architecture are left in the appendix or omitted entirely.\n\nHowever, AVMoE seems to shows strong improvements over existing approaches. If the proposed method consistently excels across hyperparameters, and the method section is fleshed out, I would be inclined to accept the paper.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Results show solid improvements:\n- Extensive experiments demonstrate that AVMoE outperforms the existing DG-SCT approach on all three tasks, while using less trainable parameters.\n- Modality ablation studies clearly show that AVMoE also handles missing audio modality much better than DG-SCT, which is a much desired property for audio-visual models.", "weaknesses": "### Experimental Design Ablation\n\n- It is less clear how much each component of the proposed method contributes to the improvement over LAVisH.\n\n  What if only unimodal adapters were added, without the router? What if we only add the MoE router to 2 or more cross-modal LAVisH adapters?\n  Additionally, experiments with adjusted LAVisH hyperparameters to increase trainable parameters to better compare with AVMoE could also be insightful.\n\n### Unclear how hyperparameters are chosen\n\n- The appendix uses completely different num. of latent tokens and learning rate compared to LAVisH, without justifying (1) how they were chosen and (2) how much they affect the results.\n\n\n### Presentation\n- Some important details of the proposed method are either left in the appendix (adapter architecture) or omitted entirely (what is the variance of Gaussian noise used? How much does it affect the results?). \n\n  I would suggest moving non-essential results (Results of all systems using incomparable encoders, Qualitative examples of Figure 3, etc.) to the appendix and moving the description of important details such as adapter architecture to the main text.\n- Minor comments: \n  1. **The references section contains multiple mistakes.** Some papers are cited more than once, some lack venue titles, and some NeurIPS 2023 papers are cited as NeurIPS 2024.\n  2. I would consider prior works in multimodal parameter-efficient transfer learning to be much more relevant than MoE. It would be nice to see some discussion of parameter-efficient methods in vision-and-language (A), audio-visual (B, or even concurrent works such as C) beyond what was used in the paper.\n  3. Regarding MoE related work, Akbari et al. also work on audio-visual inputs.\n  4.  Qualitative examples of the M3 setting in Figure 3 are a bit hard to see with tiny white masks.\n\n[A] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In CVPR, 2022\n\n[B] Liu, Hongye, et al. \"Parameter-Efficient Transfer Learning for Audio-Visual-Language Tasks.\" Proceedings of the 31st ACM International Conference on Multimedia. 2023.\n\n[C] Kai Wang, Yapeng Tian, Dimitrios Hatzinakos. Towards Efficient Audio-Visual Learners via Empowering Pre-trained Vision Transformers with Cross-Modal Adaptation. In CVPR, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer pdXh,\n\nThank you so much for these very insightful and constructive comments, please see the following for our point-by-point responses.\n\n---\n**Weakness - Experimental Design Ablation**\n\nFollowing your great advice, we conducted more experiments to analyze the contribution of each component, and the results are shown in Table 1 of the response PDF. \n\n- **Effect of Routers**: The comparison between Setting #1 and #2 reveals a consistent performance drop across all tasks when routers are not included in the model. This result highlights the importance of routers in our approach.\n\n- **Effect of only adding more Cross-modal Adapters without Unimodal Adapters**: The results between Setting #3 and #4 illustrate that adding more CMAs without UAs does not significantly improve the model's performance. However, when we compare Setting #4 to Setting #1, there is a noticeable decrease in performance. This result highlights the importance of incorporating UAs into the model, which demonstrates that a balance between unimodal and cross-modal adapters is crucial in our model to effectively capture both intra-modal and inter-modal information.\n\n- **Adjustment of LAVisH Hyperparameters**: We have attempted to adjust LAVisH hyperparameters, such as downsample size and latent token number, but could not match AVMoE's parameter size. Moreover, the comparison between DG-SCT and AVMoE in the manuscript shows that our AVMoE can achieve significant performance with fewer parameters. These comparison results illustrate our approach's superior parameter efficiency and effectiveness.\n\n\n**Weakness- Hyperparameters**\n- The number of latent tokens: Our hyperparameters of latent tokens and learning rates in these audio-visual downstream tasks are based on DG-SCT. From the experimental results in the table below, it can be viewed that the number of latent tokens has little effect on the total parameters and performance of models. Hence, for comparison with DG-SCT, we follow their hyperparameters and set the number of latent tokens for AVE and AVVP tasks to 32. Moreover, even if we use the same number of latent tokens as LAVisH, the performance of our model is still better than LAVisH.\n    \n    **AVE:**\n    | **Laten Tokens** | **Total Params** | **Acc** | **LAVisH Acc** |\n    |---|---|---|---|\n    | 2 | 403.7M | 82.2% | 81.1% |\n    | 8 | 403.8M | 82.3% | - |\n    | 32 | 404.0M | **82.6%** | - |\n\n    **AVS:**\n    | **Laten Tokens** | **Total Params** | **S4** |  | **MS3** |  |\n    |---|---|---|---|---|---|\n    |  |  | $\\mathcal{M_J}$ | $\\mathcal{M_F}$ | $\\mathcal{M_J}$ | $\\mathcal{M_F}$ |\n    | 2 | 501.2M | **81.1** | 89.7 | **54.5** | **68.7** |\n    | 8 | 501.6M | 80.8 | 89.7 | 54.3 | 68.5 |\n    | 32 | 503.0M | 80.9 | **89.9** | 54.2 | **68.7** |\n\n    **AVQA:**\n    | **Laten Tokens** | **Total Params** | **Avg Acc** |\n    |---|---|---|\n    | 2 | 456.6M | **75.7** |\n    | 8 | 456.7M | 75.6 |\n    | 32 | 456.9M | **75.7** |\n\n\n\n**Weakness - Presentation**\n\n* Some details of architecture: Thanks for your constructive comments! We will adjust the manuscript structure based on your suggestion. \n* The detail and effect of Gaussian Noise: Adding noise (e.g., Gaussian noise) to MoE in the training stage is very important, which can prevent the model from over-relying on certain experts. The Gaussian noise is implemented with a standard normal distribution (mean 0 and variance 1). We conducted an additional ablation study to evaluate the effect of Gaussian noise. The results indicate that incorporating Gaussian noise not only improved the model's performance on various datasets but also contributed to the training stability of the MoE model with less standard deviations. \n\n    | Num. | Settings | AVE (Acc. ± Std Dev) | AVS-S4 ($\\mathcal{M_J}$ ± Std Dev) | AVQA (Acc. ± Std Dev) |\n    |---|---|---|---|---|\n    | #1 | AVMoE (full model) | 82.6 ± 0.2 | 89.7 ± 0.3 | 75.7 ± 0.2 |\n    | #2 | AVMoE w/o Gaussian Noise | 81.5 ± 1.3 | 88.4 ± 1.1 | 74.4 ± 0.9 |\n\n* References: We copied the bibtex directly from Google Scholar without making any changes, maybe there was something wrong. We have carefully checked these mistakes and corrected them.\n* Related work: We totally agree that parameter-efficient transfer learning is more relevant to our work. However, Mixture of Experts (MoE) is our main contribution, we present a separate subsection (Sec. 2.2) for MoE and discuss audio-visual parameter-efficient methods in the subsection of audio-visual learning (Sec. 2.1). But thanks for your suggestion, we will add a subsection for discussing multimodal parameter-efficient methods.\n* Clarity of Qualitative examples: Thanks for pointing this out! Actually, we have tried two methods to display the qualitative results of AVS: adding white masks to original images and directly showing the binary masks. \nWe thought the former one could better demonstrate the masks of semantic objects, but the white masks are indeed not clear enough. We will consider the second plan and add bounding boxes on important areas for better comparison.\n\n---\n**Question 1 - Why does LAVisH performance degrade whereas AVMoE improves when using separate Swin-V2-L and HTS-AT encoders?**\n\nThe AVE performance of LAVisH with separate Swin-V2-L and HTS-AT encoders in our Table 1, is referenced from Table 1 of the DG-SCT paper. The authors of DG-SCT mentioned that the reason why the LAVisH dropped maybe it only utilizes latent tokens for cross-attention. The coarse extracted cross-modal information may not adequately counteract the negative effects of domain gaps introduced by the use of different encoders. We also argue that audio representations may bring disturbing information sometimes, and our AVMoE is designed to alleviate this problem. \n\n**Question 2 - What is the meaning behind the asterisk in LAVisH row of Table 3?**\n\nLAVisH* denotes our implementation version of LAVisH. We will add this annotation to the caption of Table 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KjI0YvMQoj", "reviewer": "Reviewer_jXV1", "summary": "This paper proposed an integrated adaptor architecture that makes use of cross-modal attention mechanism and Mixture-of-Experts. The architecture connects (separately) trained audio and visual encoder to do audio-visual tasks, including audio-visual event localization, audio-visual segmentation, audio-visual question answering, and audio-visual video parsing. Experiments show that the proposed architecture outperforms other adaptors while having less tunable parameters.", "questions": "I wonder why is audio visual event classification not compared, it's also not compared in LAVisH and DG-SCT, so I wouldn't blame the author for that, but I'm curious why such an common an popular audio visual task is not considered in audio visual adaptor papers.", "limitations": "The approach is only for audio visual tasks (but not other joint modal modeling scenarios) and the amount of tasks considered is limited.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is straight forward and the paper is well-written. The introduced architecture is novel and intuitive. The improvements over other methods in terms of parameter count and efficiency is clear.", "weaknesses": "The comparison with other approaches stays at parameter count and task performance, but inference cost, training time, and flops are not reported. \n\nHow does the approach compare with SotA on these datasets (rather than just adaptor models)?\n\nIt would be helpful if the author could provide some explanation on why the proposed approach is better than DG-SCT and LAVisH.\n\nThe main text and appendix are disconnected, it would be helpful to put some pointers in the main text which refers to appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jXV1,\n\nThanks for your careful review and kindly comments on this paper. Please see our point-by-point responses below.\n\n\n---\n**Weakness  - The inference cost, training time, and flops are not reported.**\n\nThanks for your suggestion! Following LAVisH and DG-SCT, we only report the trainable and total parameters, which can also demonstrate the efficiency of our approach. In order to provide a more comprehensive comparison of the efficiency, we will add these metrics in the Appendix.\n\n| **Method** | **Inference cost** | **Train Time** | **GFLOPS** | **Total Params↓** |  **Acc** |\n|:----------:|:------------------:|:--------------:|:----------:|:-----------------:|:--------:|\n|LAVisH|0.13s|7h|406.7|238.8M|81.1|\n|DG-SCT|0.13s|15h|460.8|460.8M|82.2|\n|**AVMoE**|0.13s|12h|422.6|404.0M|**82.6**|\n\n**Inference Cost: The average time it takes for each batch to be processed by the model on a NVIDIA GeForce RTX 3090.*\n\n**Since these two methods have not released this information, their inference costs and training times are calculated by our runnings of their open source codes.*\n\n---\n**Weakness - Comparison with SOTA on these datasets (rather than just adaptor models).**\n\nThanks for your question. We have reviewed recent related works and reported their results in Table 2-4 of the response pdf. It should be noted that these excellent related works [A, B, E, F, G, H, I, J] are specially designed according to the characteristics of these tasks and datasets. When comparing audio-visual parameter-efficient transfer learning methods (\"General Methods\" in the tables) with them, we observe that there still exist gaps, but our AVMoE can help to reduce the remaining gaps.\n\n*[A] Learning Event-Specific Localization Preferences for Audio-Visual Event Localization, ACM MM 2023.*\n\n*[B] CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization, ACM MM 2024.*\n\n*[C] Vision Transformers are Parameter-Efficient Audio-Visual Learners, CVPR 2023.*\n\n*[D] Cross-modal prompts: Adapting large pre-trained models for audio-visual downstream tasks, NeurIPS 2024.*\n\n*[E] Stepping stones: A progressive training strategy for audio-visual semantic segmentation, ECCV 2024.*\n\n*[F] Can Textual Semantics Mitigate Sounding Object Segmentation Preference?, ECCV 2024.*\n\n*[G] Unveiling and Mitigating Bias in Audio Visual Segmentation, ACM MM 2024.*\n\n*[H] CAD-contextual multi-modal alignment for dynamic AVQA, WACV 2024.*\n\n*[I] Semantic Enrichment for Video Question Answering with Gated Graph Neural Networks, ICASSP 2024.*\n\n*[J] Parameter-Efficient Transfer Learning for Audio-Visual-Language Tasks, ACM MM 2023.*\n\n---\n**Weakness - More explanation of why AVMoE is better than DG-SCT and LAVisH.**\n\nThanks for your advice! We will analyze more about the advantages of our method compared with LAVisH and DG-SCT in the manuscript. LAVisH and DG-SCT only inject cross-modal adapters into frozen audio-visual pre-trained models. LAVisH adopts the typical bottleneck architecture of adapters, whereas DG-SCT designs a more complex architecture of dual-guided spatial-channel-temporal attention mechanism in adapters. However, they only focus on cross-modal adapters and ingore the intra-modal information, and only introducing cross-modal adapters may bring irrelevant information in some challenging cases. Hence, to dynamically introduce inter-modal and intra-modal information, we propose an approach of Mixture of Experts (MoE) that combines adapters according to different scenarios. \n\n\n---\n**Weakness - The main text and appendix are disconnected.**\n\nThanks for your valuable comments! We will include more pointers in the main text.\n\n\n---\n**Questions - No mention of the audio-visual event classification task.**\n\nThanks for your question! We are not sure if you are asking about the audio-visual classification task on datasets such as VGGsound [K]. Although the VGGsound dataset contains a large number of videos and rich types of annotations, it only has coarse video-level annotations. As far as we know, the VGGsound dataset is more commonly used for pre-training. The AVE and AVVP tasks, which involve classifying audio-visual events second-by-second and localizing their boundaries, are more challenging and provide a more rigorous evaluation of the model's performance.\n\n*[K] Vggsound: A large-scale audio-visual dataset, ICASSP 2020.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T9j3VcTH6q", "reviewer": "Reviewer_YVYv", "summary": "The manuscript proposes the Audio-Visual Mixture of Experts (AVMoE) approach for multiple audio-visual tasks, aiming to explore parameter-efficient transfer learning for audio-visual learning. Specifically, AVMoE introduces both unimodal and cross-modal adapters as multiple experts, specializing in intra-modal and inter-modal information respectively. A lightweight router dynamically assigns the weights of each expert based on the specific requirements of each task. The effectiveness of the proposed method is demonstrated across multiple task datasets.", "questions": "If the author can clarify the above questions or suggestions, I will consider raising the score accordingly.", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The writing is easy to read.\n2. The proposed AVMoE shows performance improvements across multiple audio-visual tasks.", "weaknesses": "1. The AVMoE is somewhat similar to LAVISH, and it would be beneficial to discuss the similarities and differences between the two works.\n2. Using all audio and visual tokens as latent tokens can be very computationally expensive.\n3. In Table 3, note that the Visual Encoder for both AVSD and Pano-AVQA methods uses the ImageNet pre-trained ResNet-18 model, not VGG-19 and Faster RCNN.\n4. Although the experimental results demonstrate the method's effectiveness, does this effectiveness truly capture audio-visual scenarios well? The submitted manuscript should also include temporal and spatial visualizations for the AVQA task.\n5. It is recommended to include a more comprehensive comparison of methods in the experiments, especially for the AVQA and AVS tasks, as many recent works have focused on them.\n6. The effectiveness of the Router in the method needs to be validated.\n7. The rationale for setting the batch-size to 1 requires further discussion.\n8. Some writing conventions need attention, eg. the LAVish reference on line 250.\n9. Providing the code would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer YVYv,\n\nThank you so much for these very valuable and constructive suggestions! Please kindly find the point-to-point responses below.\n\n---\n**Differences between AVMoE and LAVisH**\n\nThanks for your feedback! The similarity between AVMoE and LAVisH is the bottleneck architecture of adapters. However, LAVisH only introduces cross-modal adapters, whereas we aim to design a more flexible approach to combine the capabilities of different adapters. Therefore, we introduce cross-modal and unimodal adapters synergistically and design a router to dynamically activate adapters according to different scenarios, thereby alleviating potential conflicting information in multimodal data. We will include these analyses in the main text to better clarify the differences between AVMoE and LAVisH.\n\n\n---\n\n**Latent Tokens**\n\nThere might be some misunderstanding. As we mentioned in Line 146, latent tokens are small, randomly initialized vectors for compressing audio and visual tokens to reduce the computational complexity. Audio and visual tokens are not latent tokens. We will refine this part to make it clearer.\n\n\n\n\n---\n**Visual Encoder of AVQA Methods**\n\nThanks for your comment! We have double-checked the original paper of AVSD [A] and Pano-AVQA [B]. AVSD utilizes VGG-19 as visual encoder and compares it with I3D visual features. Pano-AVQA indeed uses Faster R-CNN as visual encoder.\n\nAVSD: In the Sec. 2 of [A], \"we assess a naive feature extractor based on **VGG**, and demonstrate that for video- reasoning, careful reduction of the spatial dimension is more crucial than the type of extracted features used to embed the video frames.\"\n\nPano-AVQA: In the Sec. 4.1 of [B], \"we use **faster R-CNN** trained with ImageNet Detection to extract and represent region proposals\". \n\n*[A] Schwartz I, Schwing A G, Hazan T. A simple baseline for audio-visual scene-aware dialog, CVPR 2019.*\n\n*[B] Yun H, Yu Y, Yang W, et al. Pano-avqa: Grounded audio-visual question answering on 360deg videos, ICCV 2021.*\n\n---\n\n**Temporal-Spatial Visualizations for AVQA**\n\nThanks for your suggestion! We have already conducted experiments and provided visualizations for the AVS task, which demonstrate that our method effectively captures the correlation between audio and visual elements. Therefore, we did not include temporal-spatial visualizations for the AVQA task in the submitted manuscript. However, we appreciate your suggestion and present the visualizations of the AVQA task in Figure 1 of the response PDF.\nIt can be seen that our AVMoE localizes the instruments more concentratedly and more precisely. In the first case that contains the question of left and right instruments, our AVMoE assigns greater weight to the instruments on the left and right accordingly, whereas DG-SCT appears to distribute attention across all three instruments. In the second case, our AVMoE can even completely localize the entire cello and accordion, whereas DG-SCT mistakenly localizes to the stand rather than the instruments in the last frame.\n\n\n---\n**More Comparisons for the AVQA and AVS tasks**\n\nDue to space limitations, our work focuses on the methods of audio-visual parameter-efficient transfer learning. Following LAVisH and DG-SCT, we choose the same methods of AVS and AVQA for comparison. However, we also agree that adding more comparisons would help understand the progress of research tasks, and we present the comparison results in the Table 2-4 of the response PDF. We observe that there still exists the gaps compared to task-specific methods, but our AVMoE can reduce these gaps.\n\n---\n**The Effectiveness of the Router**\n\nThanks for your advice! The activation probability of experts in Figure 4 of the original manuscript demonstrates that the router can dynamically activate adapters according to different scenarios. Besides, we also conduct additional experiments to illustrate the effectiveness of the router quantitatively, and the experimental results are shown in Table 1 of the response PDF. The comparison between Setting #1 and #2 demonstrates the importance of routers in our model.\n\n---\n**Batchsize**\n\nDue to the limitations of computational resources, the batch size on a single GPU is set to 1 for AVE and AVVP tasks. However, to mitigate the limitations of using a small batch size, we employed a gradient accumulation strategy, which is a common technique for training large models with limited GPU memory. This approach involves accumulating gradients from multiple mini-batches before performing a weight update. By doing so, we can effectively simulate a larger batch size (the same as other models).\n\n\n---\n**Writing**\n\nThanks for pointing this out! We have carefully checked the manuscript and corrected the typos.\n\n\n---\n**Code**\n\nThanks for your advice! Actually, we have provided the code in supplementary materials. We will release our source codes and provide more implementation details on GitHub.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81Jd6wTrX4", "reviewer": "Reviewer_k9Vn", "summary": "This paper introduces an interesting observation i.e. for the off-screen sound cases or some mixed audio cases, ``injecting cross-modal adapters only brings disturbing information``.  To solve the problem. this paper proposes a Mixture of expert strategy that can let the model focus on different aspects of the input data, ignoring the interference. However, I found that the technique's novelty is limited, it seems just have little modification based on [1], just the input audio-visual tokens are first compressed via a cross-attention operation. Moreover, The improvements this approach brings to downstream tasks are also limited, for the AVEL task, compared to DG-SCT, there is just a 0.4% improvement on ACC. For AVS, there is just a 0.2% improvement on $M_j$ for the S4 setting, and so on.\n[1] Vision Transformers are Parameter-Efficient Audio-Visual Learners", "questions": "Please see the weakness.", "limitations": "1. Limited technique innovations.\n2. Lacking experiments to demonstrate the effectiveness of the proposed method.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This paper presents an interesting problem that the off-screen sound or the mixture audio signals may hinder the effectiveness of the audio-visual interactions based on cross-attention.", "weaknesses": "1. The technique novelty is limited, there is no obvious difference between this method design and [1]. Moreover, the architecture are same as [2].  Figure 7 is totally the same as Figure 2 in [1].\n[1] Vision Transformers are Parameter-Efficient Audio-Visual Learners\n[2] Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks\n2. This paper gives the off-screen and mixed sound cases to illustrate their motivation, but in the experiment part, they did not present the final results of these cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer k9Vn,\n\nThank you for taking the time to read our paper. We hope our responses adequately addresses your concerns.\n\n---\n**Summary - The improvements this approach brings to downstream tasks are limited.**\n\nThanks for your comments! However, we argue that focusing only on some of the experimental results might be partial.\nFirstly, it is important to note that our method achieves better results with fewer parameters compared to DG-SCT. While DG-SCT incorporates a more complex design for the adapter structure, we adopt a standard bottleneck structure of adapter, similar to that in LAVisH.\n\nSecondly, regarding more challenging scenarios like the MS3 setting of AVS task (multi-sound sources), our AVMoE using fewer parameters improves by 1.0% on $\\mathcal{M_J}$ and 4.5% on $\\mathcal{M_F}$ compared with DG-SCT. Besides, for more complex tasks like AVQA and AVVP, our AVMoE improves by 0.9% ~ 2.0% compared with DG-SCT. It can be viewed that AVMoE achieves significant performance across multiple audio-visual tasks, which has also been acknowledged by other reviewers. \n\n---\n**Weakness - Limited technique Novelty**\n\nFirstly, LAVisH and DG-SCT only inject cross-modal adapters into frozen audio-visual pre-trained models. LAVisH and DG-SCT only focus on cross-modal adapters and ignore the intra-modal information, and only introducing cross-modal adapters may bring irrelevant information in some challenging cases. To dynamically introduce inter-modal and intra-modal information, we propose an approach of Mixture of Experts (MoE) that combines cross-modal and unimodal adapters according to different scenarios. **The MoE strategy for audio-visual learning is our main contribution, rather than input audio-visual tokens of adapters.** Regarding your concerns about the architecture of the entire model, the overall framework of multimodal parameter-efficient transfer learning methods all consists of two frozen backbones and trainable modules, and our proposed AVMoE module is illustrated on the right side of Figure 2. \n\nSecondly, as for the adapter architecture in Figure 7, this is the typical bottleneck architecture of adapters, which is always used in adapter-related works (like the transformer architecture in many works). This is not our main contribution, but in order to facilitate readers who are not in this field to have a clearer understanding of adapters, we present the details in Appendix E. Besides, we also mentioned in the main text (line 143) that the adapter structure is based on LAVisH. \n\n---\n**Weakness - Effectiveness of the Proposed Method.**\n\nThanks for your suggestion! Yes, Figure 1 illustrates our motivation. However, this is the example in the AVVP task, which has more labels of audio and visual events. Due to space limitations, we present the qualitative and quantitative experiments of the AVVP task in Appendix D. \nGiven the limited time available for finalizing the appendix, Figure 6 may not fully illustrate the complexity of audio-visual scenarios. We apologize for this and show a more challenging example in Figure 2 of the response PDF.\n\nIn this example, the video label only includes *car*, and the audio label includes *car* and *speech*, which are not exactly the same. We observe that the visual predictions of DG-SCT are affected by the audio features, resulting in a visual prediction of *speech*, which does not exist in the video frame. On the other hand, our AVMoE retains unimodal information and has MoE modules to adjust the weight distribution for audio-visual features. This avoids the interference of inconsistent information and obtains the correct result. As for the audio predictions, our model can better localize the temporal boundaries of events compared to DG-SCT, which reveals that our model is capable of dealing with complex scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r1XDQlPbTS", "reviewer": "Reviewer_usTX", "summary": "They propose a novel and simple method based on Mixture of Experts (MoE) for audio-visual learning.", "questions": "- Can you include demos and code?", "limitations": "The evaluation framework is limited to the evaluation tasks the authors have chosen.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple yet effective method.\n- State-of-the-art results.\n- Overall, is well written.", "weaknesses": "- Does not include demos with audio to listen in a website.\n- Does not include code.\n\nMinor notes on the writing:\n- Mention AVE, AVVP, AVS, AVQA in the abstract without definition.\n- Section 2.1 first paragraph references to \"early researchers\" without citations.\n- line 82: typo missing closing ).\n- Section 3 there is no space before parenthesis.\n- line 232: space missing after the parenthesis.\n- I did not understand how the unimodal adapter works. I have the feeling that cross-modal adapter is well explained, but the unimodal adapter it is not. Maybe figures for how they work would be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer usTX,\n\nThank you for the positive feedback and valuable suggestions! Please see the following for our point-by-point reply.\n\n---\n**Weakness - Demos**\n\nThanks for your advice! We totally agree with the importance of providing audio-visual demos for a more comprehensive understanding of our work, and we will build a dedicated project page to provide source codes and demos. \n\n---\n**Weakness - Code**\n\nThanks for your suggestion! We have provided our code in the supplementary materials. We will release our source codes and provide more implementation details on the project website. \n\n---\n**Weakness - Minor Typos**\n\nThanks for your careful review! We have corrected the typos and revised our manuscript. Besides, the architectures of cross-modal and unimodal adapters are shown in Appendix E, and we will consider moving this section to the main text for better clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for your valuable and constructive comments. Our motivation is to propose a general, efficient, and flexible method for audio-visual learning, and your suggestions will definitely help us strengthen our work. Please kindly find the point-to-point responses below.

Meta-review: This paper proposes AVMoE (Audio-Visual Mixture of Experts), a new way of adding adapters to pretrained audio-visual models.
AVMoE extends the LAVisH approach in two ways: (1) Using unimodal adapters in addition to cross-modal LAVisH adapters.   (2) Adding a router network to dynamically combine the output of multiple unimodal/cross-modal adapters. AV-MoE outperforms two existing methods (LAVisH and DG-SCT) on audio-visual tasks including audio-visual event localization, audio-visual segmentation, audio-visual question answering, and audio-visual video parsing.

It is novel and intuitive to leverage MoE in this problem space, to learn both inter- and intra- modality information. Experiment results also support the efficacy of this approach; experiment design is comprehensive and sound. Thus I recommend to accept this paper.

Strength 
- Paper is well written.
- Proposed architecture is novel and intuitive. 
- Solid improvements: extensive experiments demonstrate that AVMoE outperforms the existing DG-SCT approach on all three tasks, while using less trainable parameters. Modality ablation studies clearly show that AVMoE also handles missing audio modality much better than DG-SCT, which is a much desired property for audio-visual models.


Weakness
- Some gap compared with SOTA (not just adaptor models) on evaluated audio-visual tasks. This is understandable as most SOTA models are designed specifically for each task. We would love to see authors to explore the combination of proposed method to existing approaches for any further improvement.
- Some reviewers pointed out that AVMoE is similar to LAVisH and thus marginally novelty. This is a reasonable argument. I think every work is built upon previous literatures and it is inevitable to have overlapping. Seems to me MoE is the core contribution of this work and using MoE is well motivated in this problem space (leverage MoE to learn both inter- and intra- modality information), and I can see the potential for proposed approach to be generalizable to other similar problems. Thus I think this paper is novel and important enough for NeurIPS community.
Decision: Accept (poster)