Aligning Audio-Visual Joint Representations with an Agentic Workflow

Authors: shentong  mo
yibing  song
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 15:15:03

Abstract: Visual content and accompanied audio signals naturally formulate a joint representation to improve audio-visual (AV) related applications. While studies develop various AV representation learning frameworks, the importance of AV data alignment is usually undermined for achieving high-quality representation. We observe that an audio signal may contain background noise interference. Also, non-synchronization may appear between audio and video streams. These non-strict data alignment limits representation quality and downgrade application performance. In this paper, we propose to improve AV joint representations from a data-centric perspective by aligning audio signals to visual data. Our alignment is conducted in an agentic workflow controlled by an LLM-based assistant named AVAgent. For each input AV data pair, our AVAgent uses a multi-modal LLM to convert audio and visual data into language descriptions separately (i.e., tool use). Then, AVAgent reasons whether this paired data is aligned well and plans to edit the audio signal if needed (i.e., planning). The audio editing is executed by predefined actions that filter noise or augment data. Moreover, we use a VLM to evaluate how modified audio signals match the visual content and provide feedback to AVAgent (i.e., reflection). The tool use, planning, and reflection steps operate cyclically to become an agentic workflow where audio signals are gradually aligned to visual content. To this end, existing methods can directly leverage the aligned AV data via our agentic workflow to improve AV joint representations. The experimental results comprehensively demonstrate the state-of-the-art performance of the proposed approach against previous baselines in diverse downstream tasks.

Main paper: 
 introduction: 
 Video stream is usually captured with sound recording. Intuitively, the audio signal supplements video data to formulate a joint AV representation. Compared to a single visual or audio representation, the joint representation benefits both single and cross-modality applications such as automatic captioning, content retrieval, and human-computer interaction. Learning audio-visual representations has been heavily investigated in the audio-visual recognition [1, 2, 3] , sound source separation [4, 5, 6] , and self-supervised form [7, 8] . These studies design various representation learning framework to leverage existing AV data pairs to obtain joint representations, which are further applied into downstream audio and visual related scenarios.The audio and visual pairs may not align well in practice during data capturing. We observe there are two main issues. First, the audio signal may contain background noise interference. In the real capturing scene, the microphone may record sound unrelated to the visual content. Second, the recorded sound may not correspond to the video frames temporally. This may be because of theQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: See Section 1Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 In this section, we provide an overview of the landscape of prior research in audio-visual representation learning and discuss how current innovations in the use of Large Language Models (LLMs) as agents contribute to advancements in this field.Audio-Visual Representation Learning Audio-visual representation learning has long been a focus of multimedia research [1, 2, 3, 9, 10, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23] , aiming to establish effective cross-modal correlations between audio and visual data. Pioneering works such as SoundNet [1] and the approaches by Owens et al. [2] and Arandjelovic et al. [3] have laid the foundation for understanding these modalities as intertwined rather than separate. These studies have shown that synchronizing audio with visual input enhances machine perception and can be pivotal for tasks such as event localization [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34] and audio-visual spatialization [35, 36, 37] . Recent advancements have also explored complex scenarios like audio-visual navigation [37, 38, 39] and parsing [40, 41, 42, 43, 44] , highlighting the depth and versatility of audio-visual data integration. Our focus on improving data quality through intelligent adjustments sets our work apart from existing methods, positioning it as a significant contribution to the field of AV representation learning.The integration of LLMs [45, 46] as decision-making agents represents a significant leap in multimedia processing. For instance, the AesopAgent [47] and VideoAgent projects [48] utilize LLMs to drive long-form video understanding and story-to-video production, showcasing the potential of LLMs in generating and refining multimedia content. However, these Figure 2 : Overview of the AVAgent framework. Tool use: For each audio-visual data pair, we employ a multi-modal Large Language Model (LLM) to convert audio and visual data into the language form, separately. Planning: The agent takes the AV data via text description and plans to edit the audio signal for alignment enhancement. Reflection: Subsequently, a Vision-Language Model (VLM) evaluates modifications to ensure that the audio adjustments appropriately match the visual content, and provides feedback to the agent. These steps form a cyclic agentic workflow where audio signals are progressively aligned with the visual content for enhanced joint representation.applications primarily focus on generating or interpreting content rather than enhancing the quality of existing audio-visual pairings. In contrast to these works, our approach utilizes LLMs specifically tailored for each AV pairing to adaptively modify audio to better align with the corresponding video content. This method not only automates the process of audio editing but also ensures that the modifications are contextually appropriate, enhancing the overall coherence and synchronization between the audio and video streams. Our approach is particularly innovative in its use of Vision-Language Models and LLMs in conjunction to refine AV synchronization, a crucial aspect often overlooked in traditional methods. By leveraging the capabilities of LLMs for detailed, context-aware audio editing, we address the foundational issue of AV misalignment, thereby enhancing the efficacy and applicability of AV systems in a range of real-world scenarios.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See our discussion in the limitation section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed method: 
 We propose to enhance the alignment of audio-visual (AV) paired data for joint representation improvement. Our alignment is fulfilled in an agentic workflow where audio signals are gradually aligned to visual data. Figure 2 shows an overview of our workflow controlled by an LLM-based assistant named AVAgent İn the following, we revisit the AV representation learning, illustrate our AVAgent and analyze our aligned AV paired data.Audio-visual (AV) representation learning aims to fuse information from both audio and visual modalities to create a unified representation that captures the intrinsic correlations between these two streams. This integration is foundational for enhancing the performance of various multimedia applications, including speech recognition, event detection, and content-based retrieval. Traditional frameworks in AV representation learning, such as those introduced in SoundNet [1] and the works by Arandjelovic et al. [3] , typically generate feature embeddings for each modality, which are then merged to form joint representations. The effectiveness of these representations hinges on the assumption that the paired audio and visual data are well-aligned and synchronized. In practice, AV representation learning models directly consume raw or minimally processed AV pair data to construct these joint representations. While effective, this direct approach generally overlooks the potential misalignments and asynchronies inherent in the source data. Misalignments, whether temporal or contextual, can degrade the quality of the learned representations, thereby reducing the overall performance of the system on downstream tasks.1.Spectral Subtraction: Removes background noise by estimating and subtracting the noise spectrum.Applies an optimal filter to minimize overall mean square error in noisy images or sounds. Our method addresses these limitations by introducing a data-centric approach that first assesses and corrects any misalignment between the audio and visual data before they are used for representation learning. By ensuring that the input AV pairs are accurately aligned, our approach enhances the quality of the joint representations, thus improving the robustness and efficacy of downstream applications. This refinement step differentiates our method from the previous framework [17] , which often assumes that the input data alignment is already optimal.We design an automatic workflow to adjust audio signals in accordance with visual data. It consists of tool use, planning, and reflection steps. Taking the raw AV pair data as input, our workflow still outputs AV pair data where the audio signal is well aligned with visual data. As we focus on data processing, our refined pair data can be widely utilized in various scenarios related to AV representation learning.Tool use We leverage multi-modal LLMs [49, 50] to map audio and video streams into a unified space where both data can be described in the language form. We transform the audio and video separately to obtain two independent language descriptions, which our AVAgent further analyzes for planning. The separate transformation benefits alignment identifications. For example, when a person is speaking in a noisy market, the video content depicts 'A person talking in a market' while the audio signal may be dominated by background noise interference. Such AV discrepancy will be reflected in separate transformations to language description while the discrepancy may not be noticed if transferred jointly. Meanwhile, transferring AV pair into language form will benefit our LLM-based AVAgent to identify and plan actions accordingly.Planning Our AVAgent reasons the given AV pair data in language form and plans for the upcoming actions. We have predefined 8 actions in advance as shown in Figure 3 . These actions are defined based on our observation that background noise interference and non-synchronization mostly limit the AV joint representations. As such, we introduce 4 noise-filtering operations to remove background interference, and 4 audio coordination operations to correspond audio signals to video streams. In the planning step, AVAgent plans one action to execute. To train AVAgent for action planning, we prepare paired data where there are video streams, audio signals, and actions. Each pair is annotated with a context description and a corresponding action that should be taken for better video and audio alignment. Our AVAgent is based on Vicuna-v1.5-7b [51] and we adopt LoRA [52] tuning so as not to affect the original reasoning ability of LLM.Reflection We evaluate how the modified audio signals match visual data after executing actions. The ImageBind [53] measures the similarity between different modalities of data, which we adopt to compute the AV alignment score and temporal synchronization score. These two scores provide feedback to the AVAgent for planning in the next cycle. If our scores are relatively low, the action Figure 4 : An example of our agentic workflow. For one input AV pair, we use mLLMs to transform video and audio data into language descriptions, separately. Then, AVAgent reasons and plans for actions. After editing the audio, AVAgent performs a reflection to compute two scores. As these scores are relatively low, they are sent to AVAgent for consideration in the next cycle. The newly planned actions operate on the original input AV pair and achieve favorable scores in reflection. These actions are then identified for editing input audio signals.planned in the next cycle will be put on the original AV pair, rather than the repaired AV data in the current round because of avoiding accumulated errors. Figure 4 shows an example of how our workflow operates in practice. Given one input video and audio signals, our AVAgent first uses mLLM to convert them into language descriptions, separately. Then AVAgent reasons the input texts and plans two actions. One action is from the noise filtering aspect and another is from the audio coordination aspect. After modifying the audio signal, AVAgent performs reflection by computing the alignment score and synchronization scores. These scores are relatively low at present and are sent to AVAgent for consideration when planning actions. Then, two actions are selected to modify the original input AV data pair, followed by a reflection where the increased scores indicate the current two actions are suitable for aligning the input AV pair. In this section, we present the data analysis procedures used to validate the effectiveness of our proposed method for improving audio-visual data alignment. Our approach involves quantitatively assessing the matching score between audio and video streams before and after applying our method, highlighting the significant enhancements achieved through our adaptive synchronization techniques.To measure the alignment of audio-visual data, we introduce the visual alignment and temporal alignment score, which quantitatively evaluates how well the audio stream corresponds with the visual content in terms of timing and context auditory-visual congruence. The visual alignment and temporal alignment scores are computed using X-CLIP [54] that analyzes the coherence between the generated audio and the synchronization accuracy. Specifically, we perform simulation experiments by computing the metrics from modified (true) audio-video pairs and original (false) pairs.Visual Alignment For visual alignment, we compute the alignment (Alignment) score between video and audio features by using 50k modified (true) video-audio pairs and 50k randomly selected original (false) pairs from VGGSound [55] . The quantitative results are reported in Table 1a . With the increase in the number of false pairs with noisy visual information, all alignment metrics decrease. Adding 50k true pairs to [50k, 50k] cases further increases all metrics. These results validate the effectiveness of the proposed agent in removing audio data noises to improve visual alignment.Temporal Synchronization For temporal synchronization, we compute the temporal alignment (T-Alignment) score between video and audio features by using 50k modified (true) video-audio pairs and 50k randomly selected original (false) pairs averaged across all time steps. Table 1b shows the comparison results on our dataset in terms of T-Alignment scores. As can be seen, the T-Alignment score decreases with the increase in the number of false pairs without synchronized audio with the original videos, although they share the same visual information. Adding 50k additional true pairs to [50k, 50k] cases also increases T-Alignment scores, which further shows the importance of the proposed agent in increasing temporal synchronization together with visual alignment.These analyses not only substantiate the effectiveness of our proposed adjustments but also illustrate the practical implications of our method in real-world scenarios. The improvement in matching scores post-intervention validates our approach, confirming that our method significantly enhances the synchronization and overall perceptual quality of audio-visual content. This section underscores the transformative potential of integrating large language models with multimodal large language models to achieve superior audio-visual data alignment, setting a new benchmark in the field of audio-visual representation learning.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See our discussions in the broader impact section Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to 
experiments: 
 In this section, we provide the detailed experimental setup and evaluation protocols used to assess the performance of our proposed method on various audio-visual representation learning tasks. These experiments are designed to validate the effectiveness of our approach, highlighting its advantages over existing state-of-the-art methods.Our experiments cover a range of audio-visual tasks, each chosen to demonstrate the robustness and versatility of our method in enhancing audio-visual data quality and alignment. The tasks Evaluation Metrics. Following the prior work [56, 64, 57] , we use the Precision and F1 scores defined in [57] for visual source localization. For source separation, following [4] , we use Signalto-Distortion Ratio (SDR) and Signal-to-Artifact Ratio (SAR). For audio-visual segmentation, we apply mIoU and F1 scores as evaluation metrics, following the previous work [58] . Linear-prob and fine-tuning classification evaluations are based on top-1 accuracy, which measures the class difference from the ground-truth labels.Implementation. Our models are implemented using state-of-the-art MAE framework [59] with specific optimizations to handle the large-scale data processing required for audio-visual tasks. Our method suggests modifications that are integrated into the preprocessing step of the learning pipeline, ensuring that the models are trained on high-quality, well-aligned audio-visual data. The input images are resized into a 224 × 224 resolution. The audio is represented by log spectrograms extracted from 3s of audio at a sample rate of 8000Hz. We follow the prior work [64] and apply STFT to generate an input tensor of size 128 × 128 (128 frequency bands over 128 timesteps) using 50ms windows with a hop size of 25ms. The models were trained for 100 epochs using the Adam optimizer [65] with a learning rate of 1e -4 and a batch size of 128.In this work, we propose a novel agentic workflow for aligning audio-visual joint representations.To demonstrate the effectiveness of the proposed AVAgent , we comprehensively compare it to prior work on linear-prob/fine-tune, audio-visual sound source localization, separation, and segmentation. The results from these experiments are crucial in validating the effectiveness of our data-centric approach. By focusing on enhancing data quality through intelligent audio-visual synchronization, our method not only improves the accuracy of specific tasks but also enhances the overall utility of AV systems in diverse applications.Audio-visual classification. To validate the effectiveness of the proposed AVAgent on audio-visual classification, we compare to the following prior baselines: 1) MAE [59] : a masked autoencoder with 2 . As can be seen, we achieve the best performance in terms of linear probing and fine-tuning on two benchmarks. In particular, the proposed AVAgent significantly outperforms MAE [59] , the current impression masking modeling approaches on images, by 46.12% and 23.85% on VGGSound-All regarding linear probing and fine-tuning. Moreover, we achieve superior performance gains compared to AudioMAE [60] with masked modeling on only audio signals, which implies the importance of learning joint audio-visual representations for multi-modal recognition. Meanwhile, our AVAgent outperforms MAViL [62] , the state-of-the-art Audio-Visual MAE by a large margin, where we achieve the performance gains of 2.62%, 2.04% on VGGSoud-Music, 4.20%, 2.07% on VGGSoud-All and 3.90%, 2.47% on AudioSet. These significant improvements demonstrate the superiority of our method in audio-visual classification.Sound source localization and segmentation. To validate the effectiveness of the proposed AVAgent on sound source localization, we compare to the following prior work: 1) Attention 10k [10] (CVPR 2018): the first baseline on sound source localization using a two-stream and attention-based neural net; 2) OTS [66] (ECCV 2018): a correspondence-based baseline for localization; 3) DMC [14] (CVPR 2019): a deep multi-modal clustering approach based on audio-visual co-occurrences; 4) CoarsetoFine [67] (ECCV 2020): a two-stage approach using coarse-to-fine embeddings alignment; 5) DSOL [68] Sound source separation. To demonstrate the effectiveness of the proposed AVAgent on source separation, we compare the following methods: 1) NMF [70] : a traditional signal processing approach based on non-negative matrix factorization to generate the spectrogram of each sound source; 2) RPCA [71] : a parameter-free baseline based on robust principal component analysis; 3) Sound-of-Pixels [4] : a deep learning approach that recovers separated audio conditioned on pixel-level visual features; 4) MP-Net [72] : an improved audio-visual method based on recursive separation from the mixture; 5) CCoL [73] (CVPR 2021): a cyclic co-learning framework based on sounding object visual grounding to separate individual sound sources. 6) OneAVM [17] (ICML 2023): a unified audio-visual framework for localization, separation, and recognition. We report the comparison results on three main benchmarks in Table 4 . As can be seen, the proposed AVAgent achieves the best performs the bestance in terms of all metrics on VGGSound-Instruments and VGGSound-Music. For example, on the challenging VGGSound-Music, we outperform OneAVM [17] , the current state-of-the-art model with a unified audio-visual learning framework, by 2.75 SDR and 2.67 SAR. Meanwhile, we achieve the best results of SDR and competitive results of SAR on the MUSIC dataset. These improvements show the superiority of our method in sound source separation.In this subsection, we provide a detailed analysis of the experiments conducted to assess the effectiveness of our approach. The experimental results are analyzed to understand the impact of our method on audio-visual representation learning, particularly focusing on the improvements over naive approaches using random actions.To demonstrate the effectiveness of LLMs in our AVAgent , we further compare the performance of our method against a naive approach where random audio modifications are applied without the guidance of LLMs in Table 5 . This comparison is crucial to demonstrate the necessity and efficiency of our intelligent, context-aware synchronization strategy. For the naive baseline, random actions such as arbitrary pitch adjustments, random noise addition, or unsystematic volume changes are applied to the audio-visual data. The audio-visual-visual data modified using our LLM-guided method significantly outperforms the randomly modified data in all evaluation metrics. For instance, in audio-visual classification tasks, our method achieves a higher accuracy rate, demonstrating the relevance and precision of the audio adjustments suggested by the LLM. The poor performance of the naive approach underscores the importance of precise, context-driven modifications, which are only possible through the understanding and analysis capabilities of our LLMs. 
conclusion: 
 In this work, we introduced AVAgent , a novel data-centric approach for enhancing audio-visual representation learning by utilizing Large Language Models (LLMs) as agents to achieve precise audio-visual synchronization and alignment. Our method diverges from traditional method-centric approaches, which primarily focus on algorithmic enhancements, and instead emphasizes the critical role of data quality in audio-visual learning processes. Our methodology leverages both Multimodal LLMs and advanced LLM techniques, including LoRA tuning, to analyze and adaptively modify the audio stream in alignment with the corresponding video content. Through a series of well-structured experiments across various audio-visual tasks such as classification, source localization, retrieval, question-answering, segmentation, and source separation, our approach has demonstrated significant improvements over existing methods. The success of our method confirms the importance of focusing on data quality and intelligent data manipulation in audio-visual representation learning. By ensuring that the audio and video streams are well-aligned and contextually synchronized, we can significantly enhance the effectiveness of audio-visual learning models, thereby improving their applicability in real-world scenarios.Limitations. While our approach significantly advances the field of audio-visual representation learning, there are some limitations that merit further investigation. Our method's effectiveness is contingent on the initial quality of the audio and visual inputs. In scenarios where inputs are of poor quality or excessively noisy, the performance of our LLM and VLM might be compromised, potentially leading to sub-optimal synchronization and alignment. Addressing these limitations will be crucial for enhancing the robustness and versatility of our approach.Broader Impact. Our work sets a new benchmark in the field of audio-visual (AV) representation learning, demonstrating that a data-centric approach, powered by advanced AI models, can lead to substantial improvements in both the performance and utility of AV systems. 
 
 Appendix: Require: A dataset of audio-visual pairs {(a i , v i )} N i=1Ensure: Improved alignment and synchronization of audio-visual pairs 1: for each pair (a i , v i ) do 2:Generate independent audio and visual captions using multimodal LLM:3:c a i ← mLLM(a i ) {Audio caption} 4: c v i ← mLLM(v i ) {Visual caption} 5:Assess reflection using VLM:6:s i ← VLM(a ′′ i , v i ) {Alignment score} 7:s t i ← VLM(a ′′ i , v i ) {Synchronization score} 8:while alignment s i and synchronization s t i exceeds threshold (0.85) do 9:Plan audio modifications to increase s i and t i :10:a ′ i ← LLM(c a i , c v i , d i ) 11:Apply audio modifications:12:EQUATIONAssess reflection using VLM:14: Store improved pair (a i , v i ) 23: end fors i ← VLM(a ′′ i , v i ) {Alignment score} 15:s t i ← VLM(a ′′ i , v i )• AudioSet [55] : a collection of 2,084,320 human-labeled 10-second sound clips drawn from YouTube videos with 632 audio event classes; • AVSBench [58] : a benchmark for testing audio-visual synchronization and alignment in diverse settings, including 4,932 videos (in total 10,852 frames) from 23 categories, including instruments, humans, animals, etc.Our approach is formalized through the following algorithmic steps:1. Input an audio-visual pair.2. Use the multimodal LLM to independently generate text descriptions for both audio and visual data. 3. Analyze the descriptions to detect discrepancies and assess alignment using a trained LLM model. 4. Plan and apply necessary audio modifications to enhance alignment, such as noise filtering and temporal adjustments. 5. Use a VLM to assess the effectiveness of the modifications and provide a feedback score. 6. Iterate the process until the audio-video alignment meets the predefined threshold or maximizes the synchronization score.Algorithm 1 outlines the steps followed by our AVAgent in processing and synchronizing audiovisual data using a data-centric approach. The algorithm utilizes a multimodal large language model (mLLM), large language model (LLM), and a vision-language model (VLM) to refine audio to better align with the visual content iteratively. This algorithm encapsulates the cyclic process of audio signal refinement, leveraging both machine learning models and heuristic analysis to ensure that the audio accurately reflects the visual context. Each step of the process is designed to enhance the synchronization between the audio and visual modalities iteratively. In this section, we conduct additional experimental analyses to explore the impact of LLM (Low-Rank Adaptation) tuning in our AVAgent . The results are reported in Table 6 . LoRA tuning enhances our LLM's memory and context retention capabilities, enabling it to make more informed decisions regarding AV synchronization. We implement LoRA tuning on the LLM to fine-tune it specifically for tasks that require a deep understanding of the audio-visual context. This involves adjusting the LLM's parameters to remember better and integrate lengthy audio-visual sequences. The application of LoRA tuning leads to noticeable improvements in tasks such as audio-visual source localization and segmentation, where the precision of temporal and spatial alignments is crucial. Metrics such as mIoU and F1 score show substantial enhancements. These improvements can be attributed to the LLM's enhanced ability to maintain context over extended interactions, proving essential in complex scenarios where long-term dependencies are critical. These analyses highlight the efficacy of our adaptive, LLM-driven approach in achieving high-quality AV synchronization. By leveraging sophisticated LLM tuning methods and moving away from naive, random modifications, our method sets a new standard in AV data processing, paving the way for more accurate and effective multimedia applications.In this section, we present detailed examples of the agent workflow in action, demonstrating the iterative process of aligning audio-visual content through the use of tool use, planning, and reflection phases. Each example includes inputs, model outputs, action plans, and reflection scores, showcasing the method's effectiveness in improving synchronization and alignment. Our approach, while advancing the state-of-the-art in audio-visual synchronization, is subject to several limitations that should be considered:• Dependency on Data Quality: The performance of our method heavily relies on the quality of the input data. Poor quality video or audio, such as low-resolution video or highly distorted audio, can significantly impair the ability of the LLM and VLM to accurately generate useful captions and subsequent synchronization actions. • Computational Resources: The algorithms employed, particularly the use of advanced LLMs like Vicuna-v1.5-7b and sophisticated VLMs, require substantial computational power. This may limit the practicality of our method for use in real-time applications or on platforms with limited processing capabilities.• Generalization across Diverse Environments: While the method performs well across several tested environments and datasets, its effectiveness in extremely noisy or acoustically complex environments has not been extensively verified. There may be scenarios where the method's ability to discern and correct misalignments is diminished.• Scalability Issues: The iterative nature of the workflow, while effective, may not scale efficiently with the volume of data or the complexity of the audio-visual scenes. This could pose challenges in deploying the system at a larger scale, such as in industry-wide media processing pipelines.• Ethical and Privacy Concerns: The use of LLMs and VLMs in processing potentially sensitive audio-visual content raises ethical and privacy concerns. There is a risk that such technology could be used to manipulate audio-visual content in misleading ways, or that sensitive information could be inadvertently exposed during processing.The broader impact of our research is multifaceted, with potential implications for numerous fields: • Media and Entertainment: Our method can significantly enhance the quality of multimedia content by ensuring better synchronization between audio and visual elements, leading to more immersive experiences in film, television, and virtual reality.• Accessibility Enhancements: Improved audio-visual synchronization can also benefit accessibility technologies, such as developing more accurate closed captioning and audio descriptions for the hearing or visually impaired.• Educational Tools: In educational settings, our method can be used to create more engaging and comprehensible instructional videos, where precise alignment of audio explanations with visual demonstrations is crucial.• Surveillance and Security: Enhanced synchronization capabilities may improve the reliability of surveillance systems where audio cues are critical to understanding visual footage. • Ethical Considerations: While our technology has many positive applications, it is essential to develop guidelines to prevent its misuse, particularly in contexts where audio-visual misalignment could be used to deceive or mislead viewers.In conclusion, while our method presents significant advancements and potential benefits, it is essential to continue refining the technology to address its limitations and ensure it is used responsibly and ethically in diverse audio-visual applications.NeurIPS Paper Checklist 


Decision: The authors tried to address an important problem in audio visual learning literature using reasonably motivated idea that language present in ambient information of audio and visuals should implicitly help align them for better joint representations. While the idea of semantic-level multi-modal learning is to effectively do this via optimization/regularization in networks, the authors call this out as something that needs to be dealt with explicitly. An important one. The technical novelty is mainly coming from designing such a scheduling system with language assistant/agent in the loop, and showing efficacy on large sets of benchmarks. While the overall performance gains are reasonable, the reviewers do highlight aspects to improve in future (possibly in follow-up studies on complex audio-visual non-signal-processing tasks; and exploration of other language models beyond Gemini). 

The authors provide important details about planning audio processing, relevant audio/visual models, query system, dataset choices and relevant audio/visual temporal sync inference steps, general 'technical tricks' etc in the comments to reviewers' questions. To the extent possible, its critical to add these into the main content of the paper.

Reviews: review: paper_summary: paper_summary: The authors propose to leverage large language model with agentic workflow such as planning, reflection and tool usage for better audio-visual alignment in training data applied to various downstream tasks such as classification, localization and separation.
strength_weakness: strength_weakness: - In section 3.3, the explanation of visual alignment and temporal synchronization scores can be made more clear. In section 3.2, ImageBind is mentioned, while in section 3.3, X-CLIP is mentioned, are the visual alignment scores based on similarities these embeddings? How are the audio embeddings computed? Are the temporal alignment computed based on sampling of audio and image from the video? What are the sampling rates? And how are these information further feedback to the LLM in order to determine the parameters for the tool usage?
- The proposed method is to improve preprocessing of the dataset. Though in Table 2, 3, 4 the comparison with various methods (they differ from various perspectives including architecture choices, datasets used for pre-training, loss function design, etc.) including the SOTA shows that proposed method outperforms. It would be more beneficial to highlight the comparison with previous methods with only difference in preprocessing step (agentic alignment as proposed). It would potentially provide a more thorough view if one of the SOTA trained with proposed preprocessing steps is shown and highlighted in the table.
- In section 4.1, consider use tables or bullet points for introducing all the datasets used, consider break them down first by downstream task, different dataset, and number of train/val/test splits, etc. This can help the reader map each dataset to the results in table 2, 3, and 4 more easily.
questions: questions: - Which audio model is used in this work? Is Google Gemini used for general audio understanding to output the audio captions?
- Are you planning to release the action planning paired data for video, audio, and action streams? How are these data curated?
- How frequent are the VLM alignment score and temporal synchronization score computed and feedback? Are the scores before any audio editing is taken place first reported so that the LLM has the information before proposing any method to use? Are these scores computed each time after the proposed actions for the LLM to reflect on?
- In section 4.1 implementation, the authors mentioned that the audio is represented by log spectrograms and STFT is applied to generate input tensor. [48] only applied log spectrogram without the STFT, what is the intuition here?
limitations: limitations: - Proposed method can be considered as preprocessing methods for audio-visual data, and can potentially be applied to most of the prior methods compared in this paper as preprocessing steps, it would be beneficial to show how proposed method can improve several of the prior works, which could be a more generalized contribution.
paper_summary: paper_summary: The paper introduces a new method to improve audio-visual joint representations by strategically aligning audio with visual data. This method employs an LLM-based assistant, AVAgent, which uses multi-modal LLMs to analyze audio and visual content separately and then plans corrective actions to better align them. The process involves noise filtering and synchronization improvements, enhanced by continuous reflection steps. The effectiveness of this approach is validated through extensive experiments across multiple public datasets, showing significant advancements over established baselines in tasks such as audio-visual classification, localization, segmentation, and source separation.
strength_weakness: strength_weakness: - Dataset Specification Clarity: The paper utilizes the AVSBench dataset for evaluating the effectiveness of the proposed method in audio-visual synchronization and segmentation. However, it lacks specificity regarding which subset of AVSBench was employed—AVSBench-object or AVSBench-semantic. Each subset serves distinct segmentation tasks that influence the interpretation and relevance of the results.
- While the paper demonstrates significant advancements in audio-visual alignment, there is potential to explore how these aligned features could be used to enhance the performance of other existing audio-visual frameworks. Some baselines already have better performance, suggesting that integrating features from this work could provide benefits.
questions: questions: - The authors should specify which subset of AVSBench was used to ensure that the results are accurately understood and contextualized within the appropriate segmentation task framework.
- The authors could discuss or simulate how features processed by their method might improve other frameworks, even as a conceptual exploration, to underscore the broader applicability of their approach.
- Recommendation: The proposed method focuses on improving audio-visual data alignment, a key factor in tasks like Audio-Visual Voice Parsing (AVVP), which emphasizes temporal synchronization. A discussion section could be added to explore hypothetically how the method might perform on tasks like AVVP, based on the alignment improvements demonstrated in the current tasks. 
- A brief discussion on the expected computational demands could make the paper more comprehensive and practical for a wider audience.
limitations: limitations: The the authors have adequately addressed the limitations.
paper_summary: paper_summary: This paper proposes a data-centric agentic workflow (AVAgent) to improve the alignment of audio-visual joint representations. The workflow is controlled by LLMs and consists of cyclic tool use, planning, and reflection steps. This paper claims the proposed method can analyze and adaptively modify the audio stream in alignment with the corresponding video content, and boost performance across various audio-visual tasks such as classification, source localization, segmentation, and source separation.
strength_weakness: strength_weakness: - Limited capability of the proposed workflow
This paper considers 8 audio-relevant actions. However, most of the actions are fundamental signal-processing operations, which might not be able to handle such as audio-visual sync problems (time delay in the middle of audio..) robustly or complicated audio scenes where noises dominate the whole audio, etc.
questions: questions: - Is existing multi-modal LLM like Gemini really able to capture thorough audio components from the given audio, especially when there are many audio sources? Did you apply any special processing to the data or other operations to adapt mLLM better on the audio captioning?
- If we directly include the Audio Noise Filtering Actions, volume normalization, and blanks filling as part of preprocessing for all audio samples, then pretrain the audio-visual representation, will the downstream task performance also be as good as the current ones?
limitations: limitations: The authors have explained the current limitations and broader impact in the paper, no clear cues on potential negative social impact from the paper.
paper_summary: paper_summary: This paper deals with the problem of audio-visual representation learning and proposes a method for reducing the misalignment between audio and video streams. More specifically, the proposed method feeds audio and video streams into multi-modal LLMs to obtain textual descriptions for both streams separately and a plan for modifying audio streams for further improvements of audio-visual alignments. The proposed plan might be fixed according to the score of audio-visual alignments. Experimental evaluations with several standard audio-visual tasks and the corresponding benchmarks demonstrate that the proposed method outperforms several standard methods for audio-visual representation learning.
strength_weakness: strength_weakness: W1. The current paper lacks details of the proposed method. Most critically, I could not find any descriptions of the details of "Planning." As presented in Figure 4, the planning section can provide which actions should be taken for a given pair of audio and visual streams. However, each action requires more details: noise levels, wavelet selection, degree of speed changes, etc. Figure 4 implies that LLMs do not suggest the best choice of action details, and we humans should choose them. If my understanding is correct, the proposed method is useless since manual inspection is required for every training example. If my understanding is incorrect, the paper lacks significant technical details.

W2. Related to the first concern, if the agent has to suggest the best sequence of actions and their details, the strategy of the proposed method that feeds textual descriptions as inputs for the planner is not a good choice. Textual descriptions are interpretable for humans and LLMs but lack details of given audio and video streams.

W3. The proposed method heavily relies on the LLMs. The relient of LLMs itself is not a drawback; however, the problem is that I could not find any technical tricks in the proposed method. Almost all the procedures for refining misalignments can be provided by the LLMs, and no other tricks can be found.
questions: questions: Q1. Please respond to the first concern (W1) presented in the Weakness section.
--> I deeply appreciated the author feedback. However, it indicates that each action is applied using default settings, which cannot be adjusted to most training examples. I want to believe that this is my misunderstanding, but it would be due to the inappropriate documentation.
limitations: limitations: No problem with this part.

Comments: title: Sincerely Look Forward to Your Feedback
Response to Reviewer Jci9
Response to Reviewer siDn
Response to Reviewer RJkZ
Specify GPU and preprocessing time
comment: Dear Reviewer GoMh,

We hope this message finds you well.

Thanks again for your insightful suggestions and comments. As the author-reviewer discussion time is ending, we want to ensure that we address any remaining uncertainties or questions you may have.

After carefully considering your comments, we have taken the necessary steps to provide additional clarifications, particularly regarding the technical details and contributions. We hope that our explanations contribute to a clearer understanding of our workflow pipeline.

We are open to providing further clarifications or conducting additional experiments if needed. Please feel free to reach out to us with any further inquiries.

Once again, we are thankful for your time and attention to our work.

Best regards, 

The Authors
Dear Reviewer Jci9,

Thank you for your continued engagement and support. We will add those comparisons and clarifications to the final version. Thank you once again for your insightful comments.
Dear Reviewer siDn,

Thank you for your continued engagement and support. We appreciate your comments regarding the use of challenging tasks. 

Most videos in the existing audio-visual datasets are single-source, and our agentic workflow has been very effective in aligning audio-visual joint representations for diverse downstream tasks. Furthermore, the performance of existing audio-visual representation learning methods has been significantly improved. 

Based on your suggestions, we further applied our agentic workflow to another challenging multi-source video dataset, AVSBench-MS3, for audio-visual segmentation. In this dataset, all the videos contain multiple sources from 23 classes, e.g., a video of a man singing while playing violin and piano. The challenging task is to predict the pixel-level segmentation task for each sounding source. Our agentic workflow can adapt multi-modal LLM (mLLM) specifically for audio captioning to discriminate each sound source as in-context input for action planning to achieve alignment. The comparison results with the strong baseline [UFE] are reported in the Table below. These improvements demonstrate our method's potential as a beneficial preprocessing step for challenging multi-source tasks.

| Method                  | mIoU ($\mathcal{M}_J$) | F-score ($\mathcal{M}_F$) |
|-------------------------|------------|---------------|
| UFE                     | 61.95      | 70.90         |
| UFE + agentic alignment | **65.72**      | **73.56**         |

[UFE] Audio-Visual Segmentation via Unlabeled Frame Exploitation, CVPR’24.


We hope this multi-source visual-audio task can respond to your concerns. Thank you once again for your insightful comments.
Thanks to the authors for providing further comparisons and clarifying several points in experimental design, dataset used in each stage. I would suggest the authors to integrate as much as possible these information in the final version. I am increasing my rating.
Thanks for the authors' response. After reading the response, I think the proposed agentic workflow is inspiring but needs to involve more complexity in the future works. The current one considers relatively basic actions and mainly single-source audio for now. I feel it will be more convincing if increase the task challenging level a bit more. I will keep my previous rating score.
Dear Reviewer RJkZ,

Thank you for your continued engagement and for acknowledging the improvements and clarification. We appreciate your comments regarding the use of the AVSBench-semantic subset and our method's performance enhancements in AVVP tasks.

Our experiments were conducted using NVIDIA Tesla A100 GPUs. This choice was driven by the need for high computational power to handle the intensive processing demands of our multi-modal LLM-based workflow. The preprocessing time for each dataset varies based on the size and complexity of the audio-visual content. On average, preprocessing each example in the datasets took approximately 0.05 minutes. This includes time for initial alignment checks and the application of the initial corrective actions proposed by our AVAgent. The total time to process the complete dataset for the LLP dataset containing around 11,849 examples was approximately 9.8 hours.

We hope this additional information addresses your queries satisfactorily and illustrates the thoroughness with which we have approached each aspect of our study. Thank you once again for your insightful comments and for the opportunity to enhance the clarity of our work.
Thank you for addressing the queries clearly, especially the use of the AVSBench-semantic subset and enhancements across models. The improvements in AVVP are commendable. 
Could you please specify which GPU was used for your experiments? Additionally, it would be helpful to know the time required to process the downstream tasks' datasets using this setup.