Mixtures of Experts for Audio-Visual Learning

Authors: ying  cheng
yang  li
junjie  he
rui  feng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:01:13

Abstract: With the rapid development of multimedia technology, audio-visual learning has emerged as a promising research topic within the field of multimodal analysis. In this paper, we explore parameter-efficient transfer learning for audio-visual learning and propose the Audio-Visual Mixture of Experts (\ourmethodname) to inject adapters into pre-trained models flexibly. Specifically, we introduce unimodal and cross-modal adapters as multiple experts to specialize in intra-modal and inter-modal information, respectively, and employ a lightweight router to dynamically allocate the weights of each expert according to the specific demands of each task. Extensive experiments demonstrate that our proposed approach \ourmethodname achieves superior performance across multiple audio-visual tasks, 
including AVE, AVVP, AVS, and AVQA. Furthermore, visual-only experimental results also indicate that our approach can tackle challenging scenes where modality information is missing.
The source code is available at \url{https://github.com/yingchengy/AVMOE}.

Main paper: 
 introduction: 
 Audio-visual learning is a challenging problem that analyzes auditory and visual information simultaneously, which aims at naturally integrating information from multiple modalities to perceive and understand the real-world comprehensively. By leveraging both auditory and visual cues, the models can learn multimodal correlations and solve various tasks such as Audio-Visual Event localization (AVE) [54] , Audio-Visual Segmentation (AVS) [65] , and Audio-Visual Question Answering (AVQA) [61, 27, 64] .In recent years, audio-visual learning has received steadily increasing attention in the field of multimodal analysis. Some previous works [3, 9, 22, 43, 1] utilize the correspondence between auditory and visual streams to learn multimodal representations in a self-supervised manner. By reducing the reliance on labeled data, self-supervised approaches can scale to larger datasets [14] and improve generalization performance. However, due to the rapid growth of the model size, fine-tuning full parameters of large pre-trained models entails significant computational costs for each downstream task. Recently, some researchers [30, 11] have attempted to inject adapters into frozen audio-visual pre-trained models, so as to reduce trainable parameters when applying to downstream tasks. Although these approaches have achieved satisfactory performance, they only focus on crossmodal attention in adapters to introduce information from other modalities, lacking the prominence of crucial information within the current modality. Besides, not every visual frame and audio segment can promote each other, and only introducing cross-modal adapters may bring irrelevant information during the learning process. As shown in Figure 1 , the sound of basketball bounce is completely drowned out by the commentator's voice and the cheers from the audience. In these cases such as off-screen sounds, injecting cross-modal adapters only brings disturbing information, and it is necessary to inject adapters according to the scenario flexibly.The Mixture of Experts (MoE) is a structural strategy for machine learning models, designed to enhance overall prediction performance by combining the decisions of multiple expert models. Hence, it is appealing to integrate the MoE module into neural networks. Within the MoE strategy, each expert model focuses on different subsets or aspects of the input data, providing predictions for a given input. Additionally, there is a gating mechanism (usually a trainable layer) responsible for determining which expert's predictions should be given more weight based on the input data. This structure allows the model to dynamically adjust its reliance on different experts, thereby optimizing its ability to process various types of data.In this work, we propose an approach of Mixture of Experts for Audio Visual Learning (AVMoE), designed as an efficient and flexible approach to adaptively combine adapter capabilities according to different scenarios. Firstly, we introduce two types of adapters as experts into frozen pre-trained models, i.e., unimodal adapter and cross-modal adapter, to focus on intra-modal and inter-modal information, respectively. Secondly, we introduce a modality-agnostic router to evaluate how relevant each expert is to the characteristics of inputs and assign the weights to each expert. By employing the AVMoE scheme, the model can adapt its strategy according to the specific demands of each task, whether it be heavily reliant on visual data, audio data, or requiring a balanced integration of both. We conduct extensive experiments on three audio-visual tasks, i.e., AVE, AVS, and AVQA. Experimental results demonstrate that our approach not only improves the model's performance across a wide range of audio-visual tasks but also makes it robust to variations in the input data. For instance, in scenarios where audio information is obscured or noisy, the model can shift its reliance toward visual cues and vice versa.The contributions of this paper can be summarized as follows:• We propose a generic, effective, and flexible approach, AVMoE, for applying pre-trained models to audio-visual learning, which can dynamically adjust its strategy according to the specific demands of each task.• We introduce two types of adapters to focus on both within-modality details and the interactions between modalities. By integrating adapters and MoE, the model can be robust to any variations in the input data, e.g., missing modality or multimodal content mismatch.• Extensive experiments demonstrate that our method consistently outperforms in diverse audio-visual tasks, showcasing its ability to dynamically prioritize visual or audio cues, enhancing its reliability and effectiveness in real-world applications.2 Related Work 
audio-visual learning: 
 Audio-visual learning aims to integrate auditory and visual information to enhance the understanding and perception of multimedia scenarios. Some early researchers [3, 22, 43] propose an innovative self-supervised approach that utilizes the correspondence between visual and audio components in video as a supervisory signal to pre-train a multimodal model, and the model can be applied to multiple downstream tasks such as speech enhancement [8, 17, 18] , sound localization [60, 32] , and action recognition [23, 49] . However, these audio-visual pre-trained models are limited by the ability of temporal localization and complex reasoning.Recently, some researchers have made efforts for complicated audio-visual downstream tasks. Audio-Visual Event Localization (AVE [54] ) aims at classifying and localizing events in the given video, and most prior works [57, 59, 62, 37] introduce interactions to utilize multimodal cues from modalityspecific pre-trained models. Audio-Visual Video Parsing (AVVP [53] ) expands the task of localizing one event to multiple events and removes the restriction that audio and visual events are definitely consistent. Previous works [63, 56] usually propose multi-scale hybrid networks and aggregate features in a weakly-supervised manner. Audio-Visual Segmentation (AVS [65] ) aims to localize and segment the masks of sounding objects at the pixel level. To tackle this task, most prior methods learn correspondences between audio and visual patches and perform mask decoding by variational auto-encoder [39] , contrastive conditional latent diffusion [38] , vision transformer [31] , bidirectional generation [15] . Audio-Visual Question Answering (AVQA [61] ) requires the model to correctly answer the questions by understanding both audio and visual modalities comprehensively. Previous works tackle this task by spatiotemporal grounding model [26] , missing modality recalling [44] , and multi-scale feature fusion [7] .Although these researches have achieved satisfactory performance, designing models for distinct tasks separately is often expensive and time-consuming. In order to improve efficiency and generalization, some recent works have explored parameter-efficient transfer learning for audio-visual downstream tasks. LAVish [30] proposes to introduce cross-modal adapters into frozen pre-trained transformers for audio-visual data. DG-SCT [11] proposes dual-guided spatial-channel-temporal attention mechanisms as cross-modal prompts injected into pre-trained models. Unlike these prior methods focus on crossmodal adapters, we introduce uni-modal adapters and the MoE scheme to combine multimodal information dynamically.Mixture of Experts (MoE) [19] is a framework that has been extensively studied and applied in various domains of machine learning. Some works explore the role of MoE in the research fields of Natural Language Processing (NLP) [51, 24, 13, 10, 68] and Computer Vision (CV) [47, 35] . These works typically regard the Feed-Forward Network (FFN) layers as experts and choose the most relevant experts for the given task. Some efforts have been made to improve the routing strategy by differentiable selection [16] , linear assignment [25] , simple hashing algorithm [48] , dense-to-sparse strategy [42] , and expert choice [67] .Recently, some works have extended the original MoE scheme to enhance its applicability in various advanced learning paradigms. In the field of multi-task learning, Ma et al. [36] and Chen et al. [6] have successfully integrated MoE schemes to deal with the complexities of learning multiple tasks simultaneously. These approaches leverage expert-specific knowledge to effectively address the unique requirements of each task, thus improving the performance and efficiency of models.Similarly, the MoE scheme has also been adapted for multimodal learning. Mustafa et al. [41] explore the usage of MoE in combining visual and textual information, demonstrating significant improvements in tasks that require understanding from both image and text inputs. Shen et al. [52] further employ MoE to scale both the text-based and vision-based feed-forward networks, showing the robustness and flexibility in handling diverse data sources. Cao et al. [4] and Akbari et al. [2] also contribute to this field by developing MoE-based models that alternate between different modalities, effectively capturing the complex interactions between them. However, these approaches only investigate the effectiveness of MoE in vision-language models, to the best of our knowledge, the MoE scheme has not been explored for audio-visual learning. We propose to inject trainable adapters with the MoE scheme into the frozen pre-trained backbones for dynamically adjusting the strategy. The AVMoE module mainly consists of a router layer and two types of adapters, the router ingests concatenated multimodal tokens and allocates the weights for adapters, and the output of AVMoE module is the weighted sum of all the adapters' predictions.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We analyze the issues with our method based on experimental results in Section 4 and explore our limitations in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we present our approach AVMoE as shown in Figure 2 , which adaptively integrates multiple types of adapters into the frozen pre-trained model using the audio-visual mixture of experts (MoE) framework. The structure of our framework consists of the frozen pre-trained model (Sec. 3.1) and trainable MoE modules (Sec. 3.2), where the MoE module contains two types of adapters and a router. In the following, we will describe each component in more detail.In this work, we aim to adapt pre-trained models to audio-visual tasks with minimal changes to the original parameters. Hence, we add a few trainable parameters while keeping the pre-trained model frozen. Following previous work [11] , we use the standard Swin-Transformer (Swin-T) [33] and audio transformer (HTS-AT) [5] as our pre-trained backbones. Each layer of these models consists of Multi-Head Attention (MHA), Feed Forward Network (FFN), and Layer Normalization with Residual Connections (Add & Layer Norm).In this subsection, we describe how our proposed AVMoE can flexibly adjust frozen pre-trained models to various audio-visual downstream tasks. As illustrated in the right of Figure 2 , the AVMoE module contains three primary components: two types of adapters as experts, each specializing in different aspects of the multi-modal data, and a router for ingesting concatenated tokens and allocating the corresponding weights of experts.The router layer is responsible for activating the appropriate experts based on the input features. To be specific, after obtaining visual embedding v t and audio embedding a t , we concatenate them as the input i t of the AVMoE router. Following previous works [13] , we employ Multi-Layer Perceptrons (MLPs) as the architecture of the router R, which produces weights by using the softmax function.The weights for Cross-Modal Adapter (CMA) and Unimodal Adapter (UA) are computed as follows:EQUATIONwhere r CMA and r UA are the outputs of the router for CMA and UA. The produced weights determine the importance of each adapter's prediction to the final output.To promote the balanced usage of experts, inspired by previous works [13, 20] , we introduce Gaussian noise to the router's output at the training stage, thereby promoting a more balanced use of experts. This method prevents the model from over-relying on certain experts, forcing the model to explore a wider range of experts:g ′ = g + ϵ, ϵ ∼ N (0, σ 2 ) (2)where ϵ is Gaussian noise with a standard deviation σ.To enhance the efficiency and performance of audio-visual models, we follow previous works [30, 11] , adding a few trainable parameters (i.e., adapters) into each layer of frozen pre-trained backbones. However, different from them, we design two types of adapters, i.e., CMA and UA. As shown in Figure3, the main differences between them are multimodal feature fusion and self-attention. These adapters are injected into the frozen pre-trained model via the MoE strategy.Cross-Modal Adapter. Firstly, to introduce interactions and integrate information from multiple modalities, we follow LAVisH adapter [30] as our cross-modal adapter. As shown in the left of Figure3, the cross-modal adapter consists of audio-visual latent token compression, audio-visual feature fusion, and bottleneck module.We propose to utilize latent tokens, which are small, randomly initialized vectors that serve as placeholders for condensed information from the audio and visual modalities. For layer l, we have m latent tokens for audio, denoted as L l a , and m latent tokens for visual, denoted as L l v . These tokens are significantly fewer in number compared to the total number of tokens from the audio or visual inputs, which helps in reducing the computational complexity.In this step, the adapter uses cross-modal attention to compress the information from all the audio or visual tokens into the corresponding latent tokens. This is achieved by applying the cross-modal attention operation between the input tokens and the latent tokens of the same modality. The compressed latent tokens capture the most relevant information from the original tokens, which is essential for efficient information transfer between modalities.For audio and visual inputs, the compression is given by:EQUATIONwhere f c is the cross-attention function, X l a and X l v are the audio and visual tokens at layer l, and S represents the resulting latent summary tokens.After compressing the modality-specific information into latent tokens, the adapter fuses the compressed information from one modality with the full set of tokens from the other modality. This is done through another round of cross-modal attention, which allows the model to integrate audio and visual cues effectively. For cross-modal fusion:EQUATIONwhere X l av and X l va represent the newly computed audio and visual representations that incorporate information from both modalities. The final step in the adapter involves a lightweight module that refines the fused audio-visual representations into more discriminative features suitable for downstream tasks. This module consists of a bottleneck structure with a down-projection layer (θ down ), a non-linear activation function (σ), and an up-projection layer (θ up ). For the audio-visual and visual-audio pathways:EQUATIONwhere Z l av and Z l va are output features that are passed to subsequent layers or used for task-specific predictions. Unimodal Adapter. While cross-modal adapters effectively introduce interactions between audio and visual tokens, they may not always be suitable for scenarios where auditory and visual content do not perfectly align, potentially introducing irrelevant information. Additionally, some multimodal data might primarily rely on one modality, or another modality may be completely missing, making inter-modal interactions less effective. Hence, it is also necessary to introduce intra-modal interactions and design unimodal adapters, which can address these issues by providing targeted improvements that compensate for the limitations of cross-modal adapters. As shown in the right of Figure3, after obtaining compressed tokens S l a and S l v , the self-attention layer ingests them and outputs newly computed representations.Q K V V Bottleneck Block Latent Tokens Q K Token CompressionEQUATIONwhere f c is the cross-attention function, X l a and X l v represent the output audio and visual representations, respectively.In this work, we propose to utilize MoE strategy to introduce intra-and inter-modal interactions synergistically. By integrating these two types of adapters, our model can dynamically adjust the attention paid to unimodal and cross-modal information according to the scenario.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental analysis: 
 To demonstrate the capabilities of our proposed AVMoE in solving diverse audio-visual tasks, we conduct extensive experiments on four downstream tasks, i.e., Audio-Visual Event Localization (AVE), Audio-Visual Video Parsing (AVVP), Audio-Visual Segmentation (AVS), and Audio-Visual Question Answering (AVQA). More details about audio-visual tasks and datasets will be illustrated in the Appendix.For the AVE task, we adopt the overall segment-wise accuracy of predicted event categories as the evaluation metric, and the event label of each video segment is required to be predicted in a fully-supervised manner. As shown in Table 1 , we compare our proposed AVMoE with previous methods on the test set of AVE dataset. We focus on LAVisH [30] and DG-SCT [11] , as they also explore parameter-efficient fine-tuning methods like ours and have achieved impressive results on this benchmark. For a fair comparison, we train our AVMoE with shared pre-trained visual backbones (ViT-B, ViT-L, Swin-V2-B, Swin-V2-L) and separate audio-visual backbones (HTS-AT and Swin-V2-L). Firstly, it can be seen that our method achieves consistently better performance than LAVisH across different shared backbones, which indicates that our method can be adapted to different backbones. We note that the structure of Swin-V2-L is better than ViT, since Swin-V2-L allows hierarchical feature representation and more efficient computation by constraining self-attention within local windows and progressively expanding them. Secondly, when utilizing distinct pre-trained backbones for each modality, our model outperforms LAVisH and DG-SCT by 4.0% and 0.4%, respectively. It should be noted that the number and proportion of trainable parameters is less than that of DG-SCT with complicated attention mechanisms, showcasing the efficiency and superiority of our model. Lastly, we also observe that the audio encoder pre-trained on Audioset can further improve the performance, which indicates that combining different modalities is beneficial for audio-visual learning.Compared to the AVE task, the AVVP task is a more complex and challenging multimodal task involving analyzing and understanding audio and visual information within a video to identify and categorize events. This task aims to segment a video into temporal events and predict their modalityspecific categories, which may include actions that are audible, visible, or both. A key challenge in AVVP is that audio and visual modalities may not be fully correlated. For instance, due to limitations in camera field-of-view or occlusions, some events may occur in only one modality. Therefore, effectively integrating audio and visual information while mitigating the interference of irrelevant modality information is crucial.As shown in Table2, our AVMoE has significant advantages over DG-SCT and MGN on the AVVP task. For example, the audio-visual event parsing metric at segment-level exceeds DG-SCT by 2%, which indicates that the scheme of MoE is suitable for balancing weight assignment for multi-modal information. To explore whether the model can associate the visual regions with the sound components, we compare our approach with state-of-the-art models on the AVS task. The Jaccard index J [12] and F-score F are used as the evaluation metrics, where J and F denote the region similarity and contour accuracy, respectively. As illustrated in Table 3 , our approach AVMoE significantly outperforms DG-SCT and LAVisH under both S4 and MS3 settings, which verifies that the scheme of MoE is also beneficial for audio-visual segmentation. Our model shows more significant improvements under the MS3 setting of multi-sound sources, indicating that the model with AVMoE strategy can deal with more complex scenarios.Figure 4 shows the qualitative results of our AVMoE model and DG-SCT on the AVS task. As can be seen, under the S4 setting, our model locates and segments the sounding objects more accurately than DG-SCT. For the case of ambulance siren (column #1), DG-SCT mistakenly locates the ambulance and the car nearby at the same time, while our model can exclude the car that hasn't produced sound.For the case of dog barking (column #3), our AVMoE model contours the object shapes (e.g., tail) more perfectly. Under the MS3 setting, compared to DG-SCT which cannot locate all sound sources in some cases, we can almost locate and segment each sound source. It can be seen that even in the case of three sound sources (column #4), we can still segment each instrument well, which verifies the effectiveness of our AVMoE model.We also want to explore whether our model can generalize to complex audio-visual tasks. To this end, we conduct experiments on the AVQA task and compare our model with previous leading methods. The AVQA task contains three categories of questions (audio, visual, and audio-visual), which involve questions related to audio (AQ), visual (VQ), or both audio and visual information simultaneously (AVQ). The multiple forms of questions challenge the adaptability to various modalities and the spatio-temporal reasoning ability of each model. Table 4 reports the experimental results on the Music-AVQA dataset [27] across three distinct scenarios. Notably, our method exhibits superior generalization abilities and consistently achieves State-Of-The-Art (SOTA) performance in all scenarios. Especially in the challenging scenarios of AVQ, our AVMoE outperforms DG-SCT by 1.2% (71.9% v.s. 70.7%)with fewer trainable and total parameters, which verifies the effectiveness and efficiency of our approach. The dynamic adaptability of MoE strategy not only establishes the robustness of audio-visual models but also illustrates the potential of leveraging the flexible strategy to enhance performance in complex audio-visual tasks. The comprehensive experimental results across four audio-visual downstream tasks illustrate that our approach achieves significant improvements while efficiently transferring from frozen backbones to various downstream tasks.5 Ablation Studies 
number of experts: 
 As shown in Table 5 , we first investigate the impact of the number of experts (i.e., cross-modal adapters and uni-modal adapters) on the tasks of AVE, AVS, and AVQA. It can be seen that increasing the number of experts consistently enhances the model's performance, thereby validating the effectiveness of the MoE strategy. Besides, we observe that the performance of our model with only one CMA and one UA still exceeds that of LAVisH. Especially under the MS3 setting of Audio-Visual Segmentation (AVS) task, it has improved by 7.2% compared with LAVisH (67.5% v.s. 60.3%). We conjecture that introducing more experts may further improve the abilities of the model. However, due to the limitation of GPU memory, we did not discuss the results in this work. In order to explore how our well-trained AVMoE model handles different modality inputs, we conduct experiments of our model and DG-SCT on visual-only data and audio-visual data. As shown in Table 6 , we compare our AVMoE model with DG-SCT on AVE, AVS, and AVQA tasks. It can be seen that our approach achieves significant performance on the test scenarios of visual-only data and audio-visual data. We observe that, our AVMoE model, when compared to DG-SCT, there is no significant decrease in accuracy on visual-only data. These results indicate that our model with the MoE scheme can still achieve good performance even in the absence of audio information. Although DG-SCT designs a more complex architecture of dual-guided spatial-channel-temporal attention mechanism in adapters, only introducing cross-modal adapters may bring irrelevant information in these challenging cases.Moreover, to explore whether the model can dynamically activate experts, we visualize the activation weight of each expert varies across parallel expert layers. The activation ratio of experts is determined by calculating the ratio of each expert's selection frequency in each MoE layer to the total number of tokens. The visualization and more details will be discussed in Appendix. 
conclusion: 
 In this paper, we develop a flexible framework that introduces Mixture of Experts (MoE) for audiovisual learning. Specifically, we inject unimodal adapters and cross-modal adapters as experts into the frozen pre-trained model and our AVMoE can dynamically combine the abilities of adapters according to different scenarios through the MoE strategy. Experimental results illustrate that our AVMoE has achieved greatly improved performance on complex tasks (such as AVQA and the MS3 setting of AVS). The ablation studies and qualitative results demonstrate that our model benefits significantly from the AVMoE strategy. However, due to computational resource limitations and efficiency considerations, we only investigate a small number of experts in the MoE scheme. In subsequent work, we will explore how to efficiently introduce more experts into our model. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not involve generative models, pretrained models, and do not have a risk of abuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We adhere to the usage restrictions of all assets and correctly cite their sources.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To demonstrate the capabilities of our proposed AVMoE in solving diverse audio-visual tasks, we conduct extensive experiments on four downstream tasks, i.e., Audio-Visual Event Localization (AVE), Audio-Visual Video Parsing (AVVP), Audio-Visual Segmentation (AVS), and Audio-Visual Question Answering (AVQA). The transferring framework is shown in Figure 5 .AVE task is defined as an event that is both visible and audible in a video segment [54] . The AVE dataset focuses on combining audio and visual modalities to temporally mark the boundaries of all audio-visual events. It contains 4,143 videos covering 28 event categories, with each video lasting up to 10 seconds and including at least one 2s long audio-visual event. For this task, we first divide the video into 10 consecutive time segments. Then, we use the frozen transformers (e.g., ViT, Swin or HTS-AT) optimized by the AVMoE adapters to extract the audio and visual features that incorporate cross-modal interaction information from these segments. Last, we predict the audio-visual events by feeding the concatenated audio-visual features through two MLP layers, as shown in Figure 5 .Referring to previous methods [54, 46, 58, 66] , we calculate the proportion of time segments that are correctly predicted and use this as a standard to evaluate the performance of our method.AVVP task aims to parse videos into temporal event segments and label them as audible, visible, or both. For this task, we use the Look, Listen, and Parse (LLP) dataset [53] , which contains 11,849 videos from different domains, covering 25 categories. LLP is a semi-supervised annotation dataset, and each video has video-level event annotations. Only 1,849 randomly selected videos have secondby-second annotations for audio and visual events. We made improvements based on MGN [40] by using AVMoE to enhance the Transformer layers in it, as shown in Figure5. Referring to previous methods [53, 40] , F-scores are used to evaluate segment-level and event-level predictions of audio, visual, and audiovisual events which are predicted by our method.AVS task aims to achieve pixel-level segmentation of visual objects based on audio. We conduct experiments on the AVSBench [65] , a dataset containing both single-source segmentation and multiplesource segmentation. The single-source dataset contains 4,932 semi-supervised videos. During the training process, only the first sample frame of the video is fully labeled, but during evaluation, all video frames need to be predicted. The multi-source dataset contains 424 fully supervised videos with every frame labeled. For this task, we optimize the frozen transformer with trainable adapters to replace the pre-trained visual encoder and audio feature extractor of the U-Net used in AVS [65] .Then, the audio and visual features are fused and ingested into the decoder for prediction, as shown in Figure5. Referring to previous methods [65] , we calculate the mean Intersection-over-Union (mIoU) of the predicted segmentation and the ground truth masks to evaluate our method.AVQA task aims to answer questions about different visual objects, sound sources and their associations in the video. We conduct experiments on the MUSIC-AVQA dataset [27] , which contains more than 45K question-answer pairs covering 33 different question templates. For this task, we utilize the trainable adapter to optimize the frozen Transformer and then replace the pre-trained visual and audio encoders of the baseline proposed in the AVQA dataset, as shown in Figure5. Referring to previous methods [27] , the answer prediction accuracy is used to evaluate the performance of methods.To enhance the fusion of audio-visual features, we follow the previous work [11] , which utilizes pre-trained vision and audio transformers to encode visual and audio inputs, respectively. The encode process is as follows. Visual embedding. For the given video sequence, we sample a set of RGB visual frames{V t } T t=1 ∈ R T ×H×W ×3, where H and W denote the height and width of frames, respectively. These video frames are sampled at the rate of 1 fps and then decomposed into n non-overlapping patches. The patches are treated as individual tokens for sequence processing by the transformer model, and we can obtain the visual embeddings v t ∈ R n×d , where d denotes the embedding dimension.Audio embedding. For the given audio stream, we first employ Short-Time Fourier Transform (STFT) to convert waveforms into spectrograms {A t } T t=1 ∈ R T ×L×F , where L and F denote the time and frequency bins, respectively. Similarly, the spectrograms are also divided into k non-overlapping patches and then projected into audio embeddings a t ∈ R k×d .For the above dataset, we extract visual frames at one frame per second and audio frames from video files at 8,000fps. We use the pre-trained Swin-V2-Large [34] as the visual transformer module with a spatial resolution of 192×192, and the pre-trained HTS-AT [5] as the audio transformer module. All of their parameters are frozen. We use the gradient accumulation method to accommodate the large parameters of the model, with weight updates after every 16 batches of training. For AVE and AVVP tasks, 32 latent tokens and a downsampling factor of 8 are used in the AVMoE adapter. For the AVS task on AVSBench and the AVQA task on MUSIC-AVQA, we use two latent tokens and set the downsampling factor and the number of group convolutions to 8 and 4, respectively. For all of our experiments, we utilize the Adam [21] optimizer to train our models and set the scheduler to make the learning rate decay to 0.35 times its original value after every 3 epochs. We set the learning rate of the AVMoE adapter to 5e-4, while the learning rates of the final prediction layer are the same as previous work [11] , 5e-6 for AVE, 3e-4 for AVVP, 3e-4 for the S4 setting of AVS, 1.5e-4 for the MS3 setting of AVS, and 1e-4 for AVQA. For audio pre-processing, we compute audio spectrograms via the PyTorch [45] kaldi fbank with 192 triangular mel-frequency bins and set frameshift over 5.2ms. For the task with audio-visual shared Transformer, we spliced the input channels of the audio spectrogram by tensor replication into 3 channels to match the input dimensions of the SwinV2 model. For these audio-visual downstream tasks, all experiments are conducted on 8x NVIDIA 3090 (24G) GPUs, and the batch size on a single GPU varies depending on the parameters of the models. The one GPU batch size is set to be 1 for AVE and AVVP tasks, and 2 for AVS and AVQA tasks. Besides, we employ a gradient accumulation strategy to mitigate the computational resource limitations. To avoid overfitting, we set the early-stop hyper-parameterization to end the training process after five epochs with no improvement in model performance. Most of the experiments converge to the best results in about ten epochs. Audio Branch Visual Branch Figure 6 shows the qualitative experimental results on the AVVP task. In this example, the video label only includes car, and the audio label includes car and speech, which are not exactly the same. We observe that the visual predictions of DG-SCT are affected by the audio features, resulting in a visual prediction of speech, which does not exist in the video frame. On the other hand, our AVMoE retains unimodal information and has MoE modules to adjust the weight distribution for audio-visual features. This avoids the interference of inconsistent information and obtains the correct result. As for the audio predictions, our model can better localize the temporal boundaries of events compared to DG-SCT, which reveals that our model is capable of dealing with complex scenarios.The activation visualization of experts is shown in Figure 7 . It can be observed that: 1)In the visual branch, The distribution of expert activation is relatively even, while in the audio branch, the activation probability of the crossmodal adapter is higher. We hypothesize that this may be due to the visual information plays a vital role in the processing of audio-visual data. 2) When the model deals with visual-only data, the activation probability of the unimodal adapter in the visual branch becomes higher, which demonstrates the ability of our AVMoE to dynamically combine adapters according to different scenarios.In addition, we observe that when the model is processing visual-only data, the activation probability of the unimodal adapter in the visual branch becomes higher, which proves the ability of our AVMoE to dynamically activate adapters according to different scenarios. Justification: We describe our scope and contributions in the abstract and introduction Section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper proposes AVMoE (Audio-Visual Mixture of Experts), a new way of adding adapters to pretrained audio-visual models.
AVMoE extends the LAVisH approach in two ways: (1) Using unimodal adapters in addition to cross-modal LAVisH adapters.   (2) Adding a router network to dynamically combine the output of multiple unimodal/cross-modal adapters. AV-MoE outperforms two existing methods (LAVisH and DG-SCT) on audio-visual tasks including audio-visual event localization, audio-visual segmentation, audio-visual question answering, and audio-visual video parsing.

It is novel and intuitive to leverage MoE in this problem space, to learn both inter- and intra- modality information. Experiment results also support the efficacy of this approach; experiment design is comprehensive and sound. Thus I recommend to accept this paper.

Strength 
- Paper is well written.
- Proposed architecture is novel and intuitive. 
- Solid improvements: extensive experiments demonstrate that AVMoE outperforms the existing DG-SCT approach on all three tasks, while using less trainable parameters. Modality ablation studies clearly show that AVMoE also handles missing audio modality much better than DG-SCT, which is a much desired property for audio-visual models.


Weakness
- Some gap compared with SOTA (not just adaptor models) on evaluated audio-visual tasks. This is understandable as most SOTA models are designed specifically for each task. We would love to see authors to explore the combination of proposed method to existing approaches for any further improvement.
- Some reviewers pointed out that AVMoE is similar to LAVisH and thus marginally novelty. This is a reasonable argument. I think every work is built upon previous literatures and it is inevitable to have overlapping. Seems to me MoE is the core contribution of this work and using MoE is well motivated in this problem space (leverage MoE to learn both inter- and intra- modality information), and I can see the potential for proposed approach to be generalizable to other similar problems. Thus I think this paper is novel and important enough for NeurIPS community.

Reviews: review: paper_summary: paper_summary: This paper proposes AVMoE, a new way of adding adapters to pretrained audio-visual models.

AVMoE extends the LAVisH approach in two ways: 
(1) Using unimodal adapters in addition to cross-modal LAVisH adapters.  
(2) Adding a router network to dynamically combine the output of multiple unimodal/cross-modal adapters. 

AV-MoE outperforms two existing methods (LAVisH and DG-SCT) on three audio-visual tasks: event localization, segmentation, and question answering.
strength_weakness: strength_weakness: ### Experimental Design Ablation

- It is less clear how much each component of the proposed method contributes to the improvement over LAVisH.

  What if only unimodal adapters were added, without the router? What if we only add the MoE router to 2 or more cross-modal LAVisH adapters?
  Additionally, experiments with adjusted LAVisH hyperparameters to increase trainable parameters to better compare with AVMoE could also be insightful.

### Unclear how hyperparameters are chosen

- The appendix uses completely different num. of latent tokens and learning rate compared to LAVisH, without justifying (1) how they were chosen and (2) how much they affect the results.


### Presentation
- Some important details of the proposed method are either left in the appendix (adapter architecture) or omitted entirely (what is the variance of Gaussian noise used? How much does it affect the results?). 

  I would suggest moving non-essential results (Results of all systems using incomparable encoders, Qualitative examples of Figure 3, etc.) to the appendix and moving the description of important details such as adapter architecture to the main text.
- Minor comments: 
  1. **The references section contains multiple mistakes.** Some papers are cited more than once, some lack venue titles, and some NeurIPS 2023 papers are cited as NeurIPS 2024.
  2. I would consider prior works in multimodal parameter-efficient transfer learning to be much more relevant than MoE. It would be nice to see some discussion of parameter-efficient methods in vision-and-language (A), audio-visual (B, or even concurrent works such as C) beyond what was used in the paper.
  3. Regarding MoE related work, Akbari et al. also work on audio-visual inputs.
  4.  Qualitative examples of the M3 setting in Figure 3 are a bit hard to see with tiny white masks.

[A] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In CVPR, 2022

[B] Liu, Hongye, et al. "Parameter-Efficient Transfer Learning for Audio-Visual-Language Tasks." Proceedings of the 31st ACM International Conference on Multimedia. 2023.

[C] Kai Wang, Yapeng Tian, Dimitrios Hatzinakos. Towards Efficient Audio-Visual Learners via Empowering Pre-trained Vision Transformers with Cross-Modal Adaptation. In CVPR, 2024
questions: questions: - When comparing results with shared Swin-V2-L encoder versus using seperate Swin-V2-L & HTS-AT encoders, why does LAVisH performance degrade whereas AVMoE improves?
- What is the meaning behind the asterisk in LAVisH row of Table 3: Audio-Visual Question Answering?
limitations: limitations: As mentioned in the Weaknesses section:
1. Importance of each component of proposed method is not well studied,
2. The choice/effect of different hyperparameters compared to LAVisH is not justified,
3. Method details such as model architecture are left in the appendix or omitted entirely.

However, AVMoE seems to shows strong improvements over existing approaches. If the proposed method consistently excels across hyperparameters, and the method section is fleshed out, I would be inclined to accept the paper.
paper_summary: paper_summary: This paper proposed an integrated adaptor architecture that makes use of cross-modal attention mechanism and Mixture-of-Experts. The architecture connects (separately) trained audio and visual encoder to do audio-visual tasks, including audio-visual event localization, audio-visual segmentation, audio-visual question answering, and audio-visual video parsing. Experiments show that the proposed architecture outperforms other adaptors while having less tunable parameters.
strength_weakness: strength_weakness: The comparison with other approaches stays at parameter count and task performance, but inference cost, training time, and flops are not reported. 

How does the approach compare with SotA on these datasets (rather than just adaptor models)?

It would be helpful if the author could provide some explanation on why the proposed approach is better than DG-SCT and LAVisH.

The main text and appendix are disconnected, it would be helpful to put some pointers in the main text which refers to appendix.
questions: questions: I wonder why is audio visual event classification not compared, it's also not compared in LAVisH and DG-SCT, so I wouldn't blame the author for that, but I'm curious why such an common an popular audio visual task is not considered in audio visual adaptor papers.
limitations: limitations: The approach is only for audio visual tasks (but not other joint modal modeling scenarios) and the amount of tasks considered is limited.
paper_summary: paper_summary: The manuscript proposes the Audio-Visual Mixture of Experts (AVMoE) approach for multiple audio-visual tasks, aiming to explore parameter-efficient transfer learning for audio-visual learning. Specifically, AVMoE introduces both unimodal and cross-modal adapters as multiple experts, specializing in intra-modal and inter-modal information respectively. A lightweight router dynamically assigns the weights of each expert based on the specific requirements of each task. The effectiveness of the proposed method is demonstrated across multiple task datasets.
strength_weakness: strength_weakness: 1. The AVMoE is somewhat similar to LAVISH, and it would be beneficial to discuss the similarities and differences between the two works.
2. Using all audio and visual tokens as latent tokens can be very computationally expensive.
3. In Table 3, note that the Visual Encoder for both AVSD and Pano-AVQA methods uses the ImageNet pre-trained ResNet-18 model, not VGG-19 and Faster RCNN.
4. Although the experimental results demonstrate the method's effectiveness, does this effectiveness truly capture audio-visual scenarios well? The submitted manuscript should also include temporal and spatial visualizations for the AVQA task.
5. It is recommended to include a more comprehensive comparison of methods in the experiments, especially for the AVQA and AVS tasks, as many recent works have focused on them.
6. The effectiveness of the Router in the method needs to be validated.
7. The rationale for setting the batch-size to 1 requires further discussion.
8. Some writing conventions need attention, eg. the LAVish reference on line 250.
9. Providing the code would be beneficial.
questions: questions: If the author can clarify the above questions or suggestions, I will consider raising the score accordingly.
limitations: limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This paper introduces an interesting observation i.e. for the off-screen sound cases or some mixed audio cases, ``injecting cross-modal adapters only brings disturbing information``.  To solve the problem. this paper proposes a Mixture of expert strategy that can let the model focus on different aspects of the input data, ignoring the interference. However, I found that the technique's novelty is limited, it seems just have little modification based on [1], just the input audio-visual tokens are first compressed via a cross-attention operation. Moreover, The improvements this approach brings to downstream tasks are also limited, for the AVEL task, compared to DG-SCT, there is just a 0.4% improvement on ACC. For AVS, there is just a 0.2% improvement on $M_j$ for the S4 setting, and so on.
[1] Vision Transformers are Parameter-Efficient Audio-Visual Learners
strength_weakness: strength_weakness: 1. The technique novelty is limited, there is no obvious difference between this method design and [1]. Moreover, the architecture are same as [2].  Figure 7 is totally the same as Figure 2 in [1].
[1] Vision Transformers are Parameter-Efficient Audio-Visual Learners
[2] Cross-modal Prompts: Adapting Large Pre-trained Models for Audio-Visual Downstream Tasks
2. This paper gives the off-screen and mixed sound cases to illustrate their motivation, but in the experiment part, they did not present the final results of these cases.
questions: questions: Please see the weakness.
limitations: limitations: 1. Limited technique innovations.
2. Lacking experiments to demonstrate the effectiveness of the proposed method.
paper_summary: paper_summary: They propose a novel and simple method based on Mixture of Experts (MoE) for audio-visual learning.
strength_weakness: strength_weakness: - Does not include demos with audio to listen in a website.
- Does not include code.

Minor notes on the writing:
- Mention AVE, AVVP, AVS, AVQA in the abstract without definition.
- Section 2.1 first paragraph references to "early researchers" without citations.
- line 82: typo missing closing ).
- Section 3 there is no space before parenthesis.
- line 232: space missing after the parenthesis.
- I did not understand how the unimodal adapter works. I have the feeling that cross-modal adapter is well explained, but the unimodal adapter it is not. Maybe figures for how they work would be useful.
questions: questions: - Can you include demos and code?
limitations: limitations: The evaluation framework is limited to the evaluation tasks the authors have chosen.

Comments: title: Thanks for the reply
comment: Dear reviewer YVYv,

Thank you for your valuable reviews! We are not sure whether our previous response has adequately addressed your concerns. If you have any further questions or comments, please kindly let us know, and we will be glad to respond.
Thanks for your reply! We are glad that you keep the original positive rating.
Dear reviewer pdXh,

We sincerely thank you for your thorough reviews and valuable suggestions! 

---
**Hyperparameter Selection**

We have double-checked the source code, and confirmed that our learning rates are the same as those used in DG-SCT. 
* For the AVE task, the learning rate of DG-SCT adapter is also 5e−4, and 5e−6 for the final prediction layer (not mentioned in DG-SCT paper). 
* For the AVS task, the learning rate is 3e−4 for the S4 setting and 1.5e−4 for the MS3 setting, consistent with DG-SCT. Sorry for missing the learning rate of S4 setting in the original manuscript. 
* For the AVVP task, the learning rate is 3e−4, we apologize for this typo and will revise the learning rate of AVVP task and AVS S4 setting in the next version. 

Regarding the batch size, we can only use a smaller batch size due to computational resource limitations. However, we employ a gradient accumulation strategy to mitigate the limitations, which is a common technique for training large models with limited GPU memory.

**Presentation**

We would like to extend our sincere appreciation for your effort in carefully reviewing our work! We have carefully checked these references & tables and will update them in the next reversion.
Thanks for taking the time to reply to my review! I'll keep my rating.
Thank you for adding comprehensive ablations studies. 

### **Experimental Design Ablation**
Extensive results show that each component of the proposed method is beneficial. I encourage the authors to add these results to the Table 4 in the main paper.

### **Hyperparameter selection**
The ablation experiment on num. of latent tokens is greatly appreciated. Results show that AVMoE is robust to the num. of latent tokens.
However, the learning rate and batch size seems to be different from both LAVisH and DG-SCT? To quote each paper on hyperparams for the AVE task:
> LAVisH: We set the learning rate of LAVISH adapter to 5e−6, and 4e−6 for the final prediction layer for audio-visual event localization, ... 
| Task | Batch Size |
--- | ---
| AVE | 2 |

> DG-SCT: For the AVE ... We train the model with a batch size of 8 and a learning rate of $5 × 10^{−4}$ ...

> AVMoE: ... weight updates after every 16 batch of training ... We set the learning rate of the AVMoE adapter to 5e-4, while the learning rate of the final prediction layer is determined by the task, 5e-6 for AVE ...

### **Presentation**
Thank you for showing the importance of Gaussian noise for the router. Regarding the references, [2], [10], [24] & [25], [38], [42] and [43], [60] and [61] are the errors that stand out the most.

### **Questions**
If Swin-V2-L + HTS-AT is reimplemented, the relevant rows in Table 1 and Table 2 probably should be marked as well.


Overall, while hyperparameter selection details are a bit vague, the extensive experiments in the rebuttal has addressed most of my major concerns. I would like to change my rating from 4 to 7.
Dear reviewers, 

Thank you so much for reviewing our paper and for your valuable suggestions. We hope our responses have addressed your concerns. If you have any further questions or comments, please kindly let us know, and we will be glad to respond.