Aligning Audio-Visual Joint Representations with an Agentic Workflow
Authors: Shentong Mo
Yibing Song
Venue: NeurIPS 2024 poster (2024)
Keywords: Audio-visual Learning
LLM Agent
Primary area: machine_vision
TL;DR: We develop an agentic workflow to align audio and video pair data to improve their joint representations.

Abstract: Visual content and accompanied audio signals naturally formulate a joint representation to improve audio-visual (AV) related applications. While studies develop various AV representation learning frameworks, the importance of AV data alignment is usually undermined for achieving high-quality representation. We observe that an audio signal may contain background noise interference. Also, non-synchronization may appear between audio and video streams. These non-strict data alignment limits representation quality and downgrade application performance. In this paper, we propose to improve AV joint representations from a data-centric perspective by aligning audio signals to visual data. Our alignment is conducted in an agentic workflow controlled by an LLM-based assistant named AVAgent. For each input AV data pair, our AVAgent uses a multi-modal LLM to convert audio and visual data into language descriptions separately (i.e., tool use). Then, AVAgent reasons whether this paired data is aligned well and plans to edit the audio signal if needed (i.e., planning). The audio editing is executed by predefined actions that filter noise or augment data. Moreover, we use a VLM to evaluate how modified audio signals match the visual content and provide feedback to AVAgent (i.e., reflection). The tool use, planning, and reflection steps operate cyclically to become an agentic workflow where audio signals are gradually aligned to visual content. To this end, existing methods can directly leverage the aligned AV data via our agentic workflow to improve AV joint representations. The experimental results comprehensively demonstrate the state-of-the-art performance of the proposed approach against previous baselines in diverse downstream tasks.

Aligning Audio-Visual Joint Representations with an Agentic Workflow
Shentong Mo CMU / MBZUAI DAMO Academy, Alibaba Group shentongmo@gmail.
Yibing Song"
DAMO Academy, Alibaba Group
Hupan Laboratory songyibing.sybeee syb@al wwliibaabeaeeeeee com
Abstract
Visual content and accompanied audio signals naturally formulate a joint repre- sentation improve audio-visual visual (AV) related applications. While studies de- velop various AV representation learning frameworks, the importance of AV data alignment is usually undermined for achieving high-quality representation. We observe that an audio signal may contain background noise interference. Also, non- synchronization may appear between audio and video streams. These non-strict data alignment limits representation quality and downgrade application perfor- mance. In this paper, we propose to improve AV joint representations from a data-centric perspective by aligning audio signals to visual data. Our alignment is conducted an agentic workflow controlled by an --bdedd assistant named AVAgent. For each input AV data pair, our AVAgent uses a multi-modal LLM to convert audio and visual data into language descriptions separately (i.e., tool use). Then, AVAgent reasons whether this paired data is aligned well and plans to edit the audio signal if needed (i.e., planning). The audio editing is executed by predefined actions that filter noise augment data. Moreover, we use VLM to evaluate how modified audio signals match the visual content and provide feedback to AVAgent (i.e., reflection). The tool use, planning, and reflection steps oper- ate cyclically to become an agentic workflow where audio signals are gradually aligned to visual content. To this end, existing methods can directly leverage the aligned AV data via our agentic workflow to improve AV joint representations. The experimental results comprehensively demonstrate the sfate-of-the-an performance the proposed approach against previous baselines in diverse downstream tasks.
Introduction
Video stream usually captured with sound recording. Intuitively, the audio signal supplements video data to formulate joint AV representation. Compared to a single visual or audio representation, the joint representation benefits both single and cross-modality applications such as automatic captioning, content retrieval, and human-compute heavily investigated in the audio-visual recognition [1, 2, 3], sound source separation [4, 5, 6]. and self-supervised form [7, 8J. These studies design various representation learning framework to leverage existing AV data pairs to obtain joint representations, which are further applied into downstream audio and visual related scenarios.
interaction. Learning audio-visual representations has been
The audio and visual pairs may not align well in practice during data capturing. We observe there are two main issues. First, the audio signal may contain background noise interference. In the real capturing scene, the microphone may record sound unrelated the visual content. Second, the recorded sound may not correspond to the video frames temporally. This may be because of the
Song the corresponding author. The project page can be found at: https: //vaggent www.aggen.ggggu..........g. github.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

non-synchronization between capture and microphone. The audio may appear earlier or later than the frames. As a result, there appears misalignment between the AV data, A direct utilization of these data, although via the following designed learning framework, will still limit the AV joint representation quality.
In this paper, we improve AV joint representations from data-centric perspective. Based on our obser- vation that audio noise and non- synchronization lead to a non-strict AV alignment, we propose an agentic workflow to align audio signals to vi- sual content adaptively. Fig. shows an overview. Our workflow consists of tool use, planning, and reflection steps. These steps are controlled by an LLM-based agent named AVAgent. For one input AV paired data, our AV- Agent will leverage a separately. Through this step, AVAgent independently perceives the audio and visual data for con- sistency judgment. Our AVAgent will reason based on the language descriptions and plan to edit audio signals if needed. The audio edition exccutes predefined actions that remove background noise (e.g., wiener filtering) and augment data (e.g., speed modification). Then, our AVAgent leverages a VLM verify if the edited audio signal matches the video stream. This VLM provides feedback to the AVAgent for guidance in the next cycle. By tool use, planning, and reflection in this workflow, the audio signals are gradually aligned to the visual content to formulate an improved joint AV representation.
Tool use
Planning
|0:00
Audio Editing Actions
AVAgent
Output
Reflection
Figure glimpse of the AVAgent workflow. Three steps (tool use, planning, and reflection) form a cyclic agentic workflow where audio signals are progressively aligned with the visual content for joint representation improvement.
LLM to convert audio and visual data into language descriptions,
Our innovative use of LLMs in this data-centric workflow marks significant advancement in the field of audio-visure representation learning. By focusing on improving data quality through intelligent audio editing, we establish foundation for more accurate and robust audio-visual visual joint representa- tions. We extensively evaluate our A VAGENT on VGGSound-AlI, and AVSBench datasets. The experimental results comprehensively demonstrate the state-of-the-arr performance of the proposed approach against previous baselines in linear probing, segmentation.
IGG-Instruments VGG-Music,
fine-tuning classification, visual sound localization, sound separation, and
2
Related works
In this section, we provide an overview of the landscape of prior research in audio-visual representa- tion learning and discuss how current innovations in the use of Large Language Models (L.LMs) as agents contribute to advancements in this field.
Audio- Visual Representation Learning Audio-visual representation learning has long been a focus of multimedia research [1, 2, 3, 9, 10, 4, 5, 6, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23), aiming to establish effective cross-modal correlations between audio and visual data. Pioneering works such as SoundNet [1] and the approaches by Owens al. [2] and Arandjelovic et al. [3] have laid the foundation for understanding these modalities as intertwined rather than separate. These studies have shown that synchronizing audio with visual input enhances machine perception and can be pivotal for tasks such as event localization [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34] and audio-visie visual spatialization [35, 36, 37]. Recent advancements have also explored complex scenarios like audio-viste visual navigation [37, 38, 39] and parsing [40, 41, 42, 43, 44], highlighting the depth and versatility of audio-visual data integration. Our focus on improving data quality through intelligent adjustments sets our work apart from existing methods, positioning it as a significant contribution to the field of AV representation learning.
L.M--baeed Agents The integration of LLMs [45, 46] as decision-making agents represents a significant leap in multimedia processing. For instance, the AesopAgent [47] and VideoAgent projects [48] utilize LLMS to drive long-form video understanding and story-to-video production, showcasing the potential of LLMs in generating and refining multimedia content. However, these

---

Audio
Multimodal LLM Tool
"The audio features playing the violin."
o
"The video features a man playing the violin room."
Video
Gemini
AVAgent
VLM Reflection
LLM Planning
00:00
"wiener filtering"
M 0
Align & Sync
Audio
Figure 2: Overview of the AVAGENT framework. Tool use: For each audio-visus data pair, employ moolil Large Language Model (LLM) to convert audio and visual data into the language form, separately, Planning: The agent takes the AV data via text description and plans to enhancement. Reflection: Subsequently, a Vision-Languag Model (VLM) evaluates modifications to ensure that the audio adjustments appropriately match the visual content, and provides feedback to the agent. These steps form cyclic agentic workflow where audio signals progressively aligned with the visual content for enhanced joint representation.
audio signal for alignment
applications primarily focus generating interpreting content rather than enhancing the quality of existing audio-visual pairings. In contrast to these works, our approach utilizes LLMs specifically tailored for each AV pairing adaptively modify audio better align with the corresponding video content. This method not only automates the process of audio editing but also ensures that the modifications contextually appropriate, enhancing the overall coherence and synchronization between the audio and video streams. Our approach is particularly innovative in its use of Vision- Language Models and L.MSs in conjunction refine AV synchronization, crucial aspect often overlooked traditional methods. By leveraging the capabilities of LLMss for detailed, context-aware audio editing, we address the foundational issue of AV misalignment, thereby enhancing the efficacy and applicability of AV systems in range of real-world scenarios.
3 Proposed method
We propose to enhance the alignment of audio-visual (AV) paired data for joint representation improvement. Our alignment is fulfilled in agentic workflow where audio signals are gradually aligned to visual data. Figure 2 shows an overview of our workflow controlled by an LLM-based assistant named AVAGENT In the following, we revisit the AV representation learning, illustrate AVAGENT and analyze our aligned AV paired data.
3.1 Revisiting Audio-Visue Representation Learning
Audio-visual (AV) representation learning aims to fuse information from both audio and visual modalities create a unified representation that captures the intrinsic correlations between these two streams. This integration is foundational for enhancing the performance of various multimedia applications, including speech recognition, event detection, and content-basse retrieval. Traditional frameworks in AV representation learning, such as those introduced in SoundNet [1] and the works by Arandjelovic al. [3], typically generate feature embeddings for each modality, which are then merged to form joint representations. The effectiveness of these representations hinges on the assumption that the paired audio and visual data are well-aligne eellnaligeed and synchronized. In practice, AV representation learning models directly consume raw or minimally processed AV pair data to construct these joint representations. While effective, this direct approach generally overlooks the potential misalignments and asynchronies inherent in the source data. Misalignments, whether temporal or contextual, can degrade the quality of the learned representations, thereby reducing the overall performance of the system downstream tasks.

---

Audio Noise Filtering Actions:
Audio Coordination Actions:
1. Spectral Subtraction: Removes back- ground noise by estimating and subtracting the noise spectrum.
5. Speed Modif fication: Changes the play- back speed to synchronize audio with video mo- tion.
2. Wiener Filtering: Applies optimal fil- ter to minimize overall mean square error noisy images sounds.
P.Pitch Modification Adjusts the audio pitch to match the visual content or correct mis- matches.
Wavelet Denoising: Employs wavelet transforms to selectively remove noise while pre- serving essential details.
7. Volume Adjustment: Modifies the volume dynamically fit the video context or to simu- late different listening environments.
4. Spectral Gating: Implements noise gate that only allows signals above certain threshold to pass through, reducing noise.
8. Filling Blanks: Introduces synthetic ele- ments in a controlled manner to blank audio to handle various acoustic environments effectively.
Figure 3: Audio editing action illustrations. We design actions to edit audio signals for AV alignment. The first actions are set to reduce background noise interference, and the last actions are set to coordinate audio signals to visual data. Our AVAGENT plans to use these actions according to input AV data pairs.
Our method addresses these limitations by introducing data-centric approach that first assesses and corrects any misalignment between the audio and visual data before they are used for representation learning. By ensuring that the input AV pairs are accurately aligned, our approach enhances the quality of the joint representations. thus improving the robustness and efficacy of downstream applications. This refinement step differentiates our method from the previous framework [17], which often assumes that the input data alignment is already optimal.
3.2 Agentic Workflow
We design an automatic workflow to adjust audio signals in accordance with visual data. It consists of tool use, planning, and reflection steps. Taking the raw AV pair data as input, our workflow still outputs AV pair data where the audio signal is well aligned with visual data. As focus on data processing, our refined pair data can be widely utilized in various scenarios related to AV representation learning.
Tool use We leverage multi-modal LLMs [49, 50] to map audio and video streams into a unified space where both data can be described in the language form. We transform the audio and video separately to obtain two independent language descriptions, which our A AVAGENT further analyzes for planning. The separate transformation benefits alignment identifications, For example, when person is speaking in a noisy market, the video content depicts 'A person talking in a market' while the audio signal may be dominated by background noise interference. Such AV discrepancy will be reflected in separate transformations to language description while the discrepancy may not be noticed if transferred jointly. Meanwhile, transferring AV pair into language form will benefit our LLM-based AVAGENT to identify and plan actions accordingly.
Planning Our AVAGENT reasons the given AV pair data in language form and plans for the upcoming actions. We have predefined 8 actions in advance as shown in Figure 3. These actions are defined based on our observation that background noise interference and so--schhhrnnzztionn mostly limit the AV joint representations. As such, we introduce 4 noise- fiiiie-finniing operations to remove background interference, and 4 audio coordination operations to correspond audio signals video streams. In the planning step, AVAGENT plans one action to execute. To train A AVAGENT for action planning, we prepare paired data where there are video streams, audio signals, and actions. Each pair is annotated with context description and a corresponding action that should be taken for better video and audio alignment. Our AVAGENT is based on Vicuna-v1.5577 [51] and we adopt LoRA [52] tuning so as not to affect the original reasoning ability of LLM.
Reflection We evaluate how the modified audio signals match visual data after executing actions.
The ImageBind [53] measures the similarity between different modalities of data, which we adopt to compute the AV alignment score and temporal synchronization score. These two scores provide feedback to the A AVAGENT for planning in the next cycle. If our scores relatively low, the action

---

Can you spectly audio editin actions for input: video of man playing the vioin a noke cate and its corresponding undlear audio?
Request
mLLM output:
Tool use
Video caption: man playing the violin in room
Audio caption: violin sound and background cafe nosse overpowering speech
LLM Input: Given video and audoo caption, what actions would you choose from the given action set for edling audio to achive noise filering and improve audio synchronizatio??
Planning
Noise Fillering Actions: Spectral Subtraction, Wiener Fittriing Wavelet Denoising, Spectral Geting: Data Augmentation Actions: Speed Modifiation, Pitch Modification Volume Adjusment, Filling Blanks Actions: Spectral Gatligg and Volume Adjustment.
Reasoning: Spectral galing works by allowing only audio signals above certain threshold pass through. which can help suppress any subtle noise not removed by the inttaa noise reduction steps. Increase the volume of the violin sounds specilically if possible through selective amplliication the trequencles common vialin sounds. This would help making the vialin sounds more prominent, matching the visual emphasis of the playing the violin the video.
Feedback
Reflection
Audiovideo temporal synchrorization
AVAgent
LLM Input: Given the VLM Refeccioo outputs and original video/audio captions, what actions would you choose from the given action set for editing audio to achieve noise filtring and improve audio- video synchronizalio??
Planning
Noise Filtering Actions: Spectral Subtraction, Wiener Fitering, Wavelet Denoising, Spectral Gatling Data Augmentation Actions: Speed Modiiiation, Pitch Moditicaiion Volume Adustment, Filling Blanks
Actions: Wiener Filtering and Speed Medification.
Reasoning: enhance the clarity
audio, applying Wiener fitarinn can help minimize the overall
mean square error in the noley audio signal.
Adjust the speed the audio playback to synchronize the audio with the video better. Since the audio playing either too quickly too slowly compared the visual sequence of playing the violin.
temporal synchronization
low, this indicates
VLM Feedback:
Reflection
VLM aligmmett score: 0.93
Audio video tamporal synchronization score: 0.87
The audio editing actions for Input: video of man playing the vialin in a noisy cafe and its correspording unclear audio, could be Wiener Fillering and Speed Modification
Figure 4: An example of our agentic workflow. For one input AV pair, we use ml mL..Ms transform vidco and audio data into language descriptions, separately. Then, AVAGENT reasons and plans for actions. After editing the audio, AVAGENT performs reflection to compute two scores. As these scores are relatively low, they are sent to AVAGENT for consideration in the next cycle. The newly planned actions operate the original input AV pair and achieve favorable scores reflection. These actions then identified for editing input audio signals.
planned in the next cycle will be put on the original AV pair, rather than the repaired AV data in the current round because of avoiding accumulated errors.
Figure shows an example of how our workflow operates in practice. Given one input video and audio signals, our AVAGENT first uses mLLM to convert them into language descriptions, separately. Then AVAGENT reasons the input texts and plans two actions. One action from the noise filtering aspect and another is from the audio coordination aspect. After modifying the audio signal, AVAGENT performs reflection by computing the alignment score and synchronization scores. These scores are relatively low at present and are sent to AVAGENT for consideration when planning actions. Then, two actions are selected to modify the original input AV data pair, followed by reflection where the increased scores indicate the current two actions are suitable for aligning the input AV pair.
5

---

Table 1: Data analysis of visual alignment and temporal synchronization. True and False pairs denote the video corresponding to the modified and original audio separately.
(a) Visual alignment.
(b) Temporal synchronization.
True Pairs
False Pairs
Aligument (%.
Pairs
Pairs
T-Alignment (%; 1)
50k
86.53
50k
78.23
0
50k
53.16
0
50k
42.05
50k
50k
62.78
50k
50k
52.29
50k
100k
57.65
50k
100k
45.65
100k
50k
75.39
100k
50k
63.71
3.3 Data Analysis
In this section, we present the data analysis procedures used to validate the effectiveness of our proposed method for improving audio-visual data alignment. Our approach involves quantitatively assessing the matching score between audio and video streams before and after applying our method, highlighting the significant enhancements achieved through our adaptive synchronization techniques.
To measure the alignment audio-visual data, we introduce the visual alignment and temporal alignment score, which quantitatively evaluates how well the audio stream corresponds with the visual content in terms of timing and context auditory-visur congruence. The visual alignment and temporal alignment scores computed using X-CLIP [54] that analyzes the coherence between the generated audio and the synchronization accuracy. Specifically, we perform simulation experiments by computing the metrics from modified (true) audio-video pairs and original (false) pairs.
Visual Alignment For visual alignment, we compute the alignment (Alignment) score between video and audio features by using 50k modified (true) video-audio pairs and 50k randomly selected original (false) pairs from VGGSound [55]. The quantitative results reported in Table la. With the increase in the number of false pairs with noisy visual information, alignment metrics decrease. Adding 50k true pairs [50k, 50k] cases further increases all metrics. These results validate the effectiveness of the proposed agent in removing audio data noises to improve visual alignment.
Temporal Synchronization For temporal synchronization, we compute the temporal alignment (T-Alignment) score between video and audio features by using 50k modified (true) video-audio pairs and 50k randomly selected original (false) pairs averaged across all time steps. Table 1b shows the comparison results
dataset in terms of T-Alignment scores. As can be seen, the T-Alignment score decreases with the increase the number false pairs without synchronized audio with the original videos, although they share the same visual information. Adding 50k additional true pairs to [50k, 50k] cases also increases T-Alignment scores, which further shows the importance of the proposed agent in increasing temporal synchronization together with visual alignment.
These analyses not only substantiate effectiveness of our proposed adjustments also illustrate the practical implications of our method in real-world scenarios. The improvement in matching scores post-intervention validates our approach, confirming that our method significantly enhances the synchronization and overall perceptual quality of audio-visun visual content. This section underscores the transformative potential of integrating large language models with multimodal large language models achieve superior audio-visual data alignment, setting new benchmark in the field of audio-visual representation learning.
4 Experiments
In this section, provide the detailed experimental setup and evaluation protocols used to assess the performance of our proposed method on various audio-visual representation learning tasks. These experiments are designed to validate the effectiveness of our approach, highlighting its advantages over existing state-of-the-ar methods.
4.1 Experimental Setup
Our experiments cover range of audio-visual tasks, each chosen to demonstrate the robustness and versatility of our method in enhancing audio-visual data quality and alignment. The tasks

---

Table 2: Audio-visual classification on VGGSound-Music, VGGSound-AlI, and AudioSet datasets.
VGGSound-Musi
VGGSound-All
AudioSet
Method
Lineur (5)
Finetune (%)
Linear (%)
Finetune (%)
Linear (%)
Finetune (%)
MAE [59]
25.32
52.39
15.61
45.73
11.32
24.23
AndioMAE [60]
41.65
55.61
42.35
57.76
30.23
44,92
CAV-MAE [61]
60.53
67.26
55.27
65.53
40,56
51.29
MAVIL [62]
61.95
69.53
57.36
67.17
43.62
53.38
AV-MAR [63]
60.82
67.61
56.15
65.08
41.67
51.32
AVAGENT (ours)
64.57
71.57
61.56
69.24
47.52
$5.85
include audio-visual classification, audio-visual source localization, audio-visual segmentation, and audio-visual source separation.
Datasets. We utilize several well-known datasets in the audio-visual domain, including both synthetic and real-world scenarios, to ensure comprchensive testing conditions. These datasets chosen to provide a variety of challenges in terms of noise levels, types of audio and visual content, and synchronization issues. Specifically, we use subset of 144k pairs in VGG-Sound [55] for pre- training, and fine-tuning the model on audio-visual main downstream datasets. 1) For source separation, used 40,908 video clips from 49 music categories for training and 1201 clips for testing, denoted as VGGSound-Music. 10s lengths from 36 musical instrument classes, subset of VGG-Sound [55], and each video only has one single instrument class annotation. MUSIC [4] consists of 448 untrimmed YouTube music videos of solos and duets from 11 instrument categories, where We use 358 solo videos for training and 90 solo videos for evaluation. The used dataset is slightly smaller than the original MUSIC dataset since some videos are no longer publicly available be downloaded. 2) For localization, we used
JGGSound-Instrument [56] includes 32k video clips of
[10] with 4,500 audio-visual pairs for training and testing the model on 250 audio-visual pairs of sounding objects and extended 250 sounding objects introduced in SLAVC [57]. 3) For audio-visual segmentation, AVSBench [58] includes 4,932 videos (in total 10,852 frames) from 23 categories, including instruments, humans, animals, etc. Following prior work [58], we used the same split of 3,452/740/740 videos for train/val/test. 4) For linear probing and fine-tuning, we applied VGGSound-Min Music with 49 classes and VGGSound-AIl with 221 categories for comprehensive evaluation.
Evaluation Metries. Following the prior work [56, 64, 57], we use the Precision and Fl scores defined in [57] for visual source localization. For source separation, following [4], we use Signal- to-Distortion Ratio (SDR) and Signal-to-Artifact Ratio (SAR). For audio-visual segmentation, we apply mloU and F1 scores as evaluation metrics, following the previous work [58]. Linear-proo and fine-tuning classification evaluations are based on top-1 accuracy, which measures the class difference from the ground -round-truuhh labels.
Implementation. Our models implemented using state-of-the-art MAE framework [59] with specific optimizations to handle the large-scale data processing required for audio-visual tasks. Our method suggests modifications that are integrated into the preprocessing step of the learning pipeline, ensuring that the models are trained on high-quality, well-aligned audio-visual data. The input images are resized into 224 224 resolution. The audio is represented by log spectrograms extracted from 3s of audio at a sample rate of 8000Hz. We follow the prior work [64] and apply STFT to generate an input tensor of size 128 x 128 (128 frequency bands over 128 timesteps) using 50ms windows with hop size of 25ms. The models were trained for 100 epochs using the Adam optimizer [65] with learning rate of le and a batch size of 128.
4.2 Comparison to Prior Work
In this work, we propose novel agentic workflow for aligning audio-visual joint representations. To demonstrate the effectiveness of the proposed AVAGENT, we comprehensively compare it to prior work on linear-prob/line-tune audio-visual sound source localization, separation, and segmentation. The results from these experiments are crucial in validating the effectiveness of our data-centric approach. By focusing enhancing data quality through intelligent audio-visual synchronization. our method not only improves the accuracy of specific tasks but also enhances the overall utility of AV systems in diverse applications.
Audio-visual classification. To validate the effectiveness of the proposed AVAGENT on audio-visual classification, compare to the following prior baselines: 1) MAE [59]; masked autoencoder with
7

---

Table 3: Sound source localization and segmentation. Quantitative results Flickr-SoundNet and AVSBench.
Fl AP
AVSBench mloC Fl
Method
Precision
Fl
Attention 10k [10]
49.38
51.23
55.39
20.76
31.25
OTS [66]
51.23
53,28
58.12
24.55
36.85
DMC [14]
50.52
52.93
57.56
23.51
35.27
CoarsetoFine [67]
51.76
54.85
58.63
26.53
38.62
DSOL [68]
55.29
57.92
62.05
29.85
42.23
LVS [69]
52.38
55.31
59.35
27.32
40.18
FZVSL. [64]
54.71
57.51
61.38
30.52
43.26
Mix-and-Localee [56]
55.83
58,21
62.52
31.69
45.35
SLAVC [57]
55.65
58.12
62.39
31.36
45.02
AVAGENT (ours)
58.23
60.76
65.03
36.37
49.85
Table 4: Sound source separation. Quantitative results on MUSIC and VGGSound datasets.
MUSIC
Instt-nntss
VGGS-Music
Method
SDR
SAR
SDR
SAR
SDR
SAR
NMF [70]
-0.62
2.41
-3.85
-0.76
-7.12
-9.01
RPCA [71]
0.86
3.81
-2.39
1.58
-5.53
-7.82
Sound-of-Pixee [4]
4.55
10.24
2.52
4.67
0.95
MP-Net [72]
4.82
10.56
2.63
4.85
1.37
1.39
CCoL [73]
6.35
9.75
3.28
5.01
2.07
2.18
OneAVM [17]
7.38
7.48
5.36
5.52
2.51
2.61
AVAGENT (ours)
8.82
11.72
6.72
6.85
5.26
5.28
only images as input: 2) AudioMAE [59]: masked autoencoder with only audio as input; 2) Audio- Visual MAEs [61, 62, 63]: masked autoencoders with both audio and images as input. Table 2 reports the quantitative comparison results in Table 2. As can be seen, achieve the best performance in terms of linear probing and fine-tuning on two benchmarks. In particular, the proposed AVAGENT significantly outperforms MAE [59], the current impression masking modeling approaches on images, by 46.12% and 23.85% on VGGSound-All regarding linear probing and fine-tuning. Moreover, achieve superior performance gains compared AudioMAE [60] with masked modeling on only audio signals, which implies the importance of learning joint audio-visual representations for multi-modal recognition. Mcanwhile, our A VAGENT outperforms MAViL [62], the sfate-of-the- Audio- Visual MAE by a large margin. where we achieve the performance gains of 2.62%. 2.04% on VGGSoud-Music, 4. 4.20%, 2.07% on VGGSoud-All and 3.90%. 2.47% on AudioSet. These significant improvements demonstrate the superiority of our method in audio-visual classification.
Sound source localization and segmentation. To validate the effectiveness of the proposed AVAGENT on sound source localization, we compare to the following prior work: 1) Attention 10k [10] (CVPR
2018): the first baseline on sound source localization using two-stream and attention-based neural
net; 2) OTS [66] (ECCV 2018): a correspondence-basee baseline for localization; 3) DMC [14]
(CVPR 2019): deep multi-modal clustering approach based on audio-visual co-occurrences; 4)
CoarsetoFine [67] (ECCV 2020): a two-stage approach using coarse- morse-ttofine to-fine embeddings alignment: 5)
DSOL [68] (NeurIPS 2020): a class-based method with two-stage training: 6) LVS [69] (CVPR 2021):
a contrastive learning framework with hard negative mining to learn audio-visual correspondence maps; 7) EZ-VSL [64] (ECCV 2022): recent weakly supervised localization framework based on multiple-in multiple-instance instance contrastive learning: 8) fix-andd-occed
[56] (CVPR 2022): a recent method based on a contrastive random walk on a graph of images and separated sound sources. 9) SLAVC [57] (NeurIPS 2022):< strong baseline with momentum encoders and extreme visual dropout to identify negatives and solve significant overfitting. The comparison results are reported in Table 3. Compared to Mix-and-Looalle
[56]. the current state-of-the-arr multi-source localization baseline, we achieve the results gains of 2.40 Precision, 2.55 AP, and 2.51 Fl on VGGSound- Ionrdd--ntttettts. Furthermore, when evaluated on the challenging AVSBench benchmark. the proposed approach still outperforms Mix-and-Localle [56] by 4.68 mloU and 4.50 F1. These results validate the effectiveness of our approach in learning discriminative cross-modal representations from audio and images for sound source localization and segmentation.
Sound source separation. To demonstrate the effectiveness of the proposed AVAGENT on source separation, we compare the following methods: 1) NMF [70]: traditional signal processing approach based on non-negative matrix factorization to generate the spectrogram of each sound source; 2) RPCA [71]: a parameter-fre baseline based on robust principal component analysis; 3) Sound-of- Pixels [4]: deep learning approach that recovers separated audio conditioned on pixel-level visual

---

Table 5: Comparison to random actions. Quantitative results VGGSound-All, Flickr-SoundNet, AVSBench, and VGGS-Musie datasets.
VGGSound-All
Ne
AVSBench
VGGS-Music
Method
Linear (%)
Finctune (%)
Precision
AP
mloU
F1
SDR
SAR
Random Actions
50.26
56.82
45.32
48.26
50.78
22.18
34,63
0.95
0.96
AVAGENT (ours)
61.56
69.24
58,23
60.76
65.03
36.37
49.85
5.26
5.28
features; 4) MP-Net [72]: an improved audio-visual method based on recursive separation from the mixture; 5) CCoL [73] (CVPR 2021): a cyclic co-learning framework based on sounding object visual grounding to separate individual sound sources. 6) OneAVM [17] (ICML 2023): unified audio-visual framework for localization, separation, and recognition. We report the comparison results on three main benchmarks in Table 4. As can be seen, the proposed AVAGENT achieves the best performs the bestance in terms of all metrics on KGGSound-Instrumenn For example, on the challenging state-of-the-art model with a unified audio-vinn viiuall learning framework, by 2.75 SDR and 2.67 SAR. Mcanwhile, we achieve the best results of SDR and competitive results of SAR on the MUSIC dataset. These improvements show the superiority of our method in sound source separation.
and VGGSound- SCGSoound-Musie we outperform OneAVM [17]. the current
4.3 Experimental Analysis
In this subsection, we provide detailed analysis of the experiments conducted to assess the effec- tiveness of our approach. The experimental results are analyzed to understand the impact of our method on audio-visual representation learning, particularly focusing on the improvements over naive approaches using random actions.
To demonstrate the effectiveness of LLMs in our AVAGENT, we further compare the performance of our method against naive approach where random audio modifications are applied without the guidance of LLMs in Table 5. This comparison is crucial to demonstrate the necessity and efficiency of our intelligent, context-aware synchronization strategy. For the naive baseline, random actions such as arbitrary pitch adjustments, random noise addition, unsystematic volume changes applied to the audio-visual data. The audio-visual-viis visual data modificd using our LLM- --ggdddeed method significantly outperforms the randomly modified data in all evaluation metrics. For instance, in audio-visual classification tasks, our method achieves a higher accuracy rate, demonstrating the relevance and precision of the audio adjustments suggested by the LLM. The poor performance of the naive approach underscores the importance of precise, context-driven modifications, which are only possible through the understanding and analysis capabilities of our LLMs.
5 Conclusion
In this work, we introduced AVAGENT, novel data-centric approach for enhancing audio-visual representation learning by utilizing Large Language Models (LLMs) as agents to achieve precise audio-visual synchronization and alignment. Our method diverges from traditional method-centric approaches, which primarily focus on algorithmic enhancements, and instead emphasizes the critical role of data quality in audio-visual learning processes. Our methodology leverages both Multimodal LLMs and advanced LLM techniques, including LoRA tuning, to analyze and adaptively modify the audio stream in alignment with the corresponding video content. Through a series of well-structured experiments across various audio-visual tasks such as classification, source localization, retrieval, question-answeer answering, segmentation, and source separation, our approach has demonstrated significant improvements over existing methods. The success of our method confirms the importance of focusing on data quality and intelligent data manipulation in audio-visual representation learning. By ensuring that the audio and video streams are well-aligned and contextually synchronized, we can significantly enhance the effectiveness of audio-visual learning models, thereby improving their applicability in real-world scenarios.
Limitations. While our approach significantly advances the field of audio-visual representation learning, there are some limitations that merit further investigation. Our method's 's effectiveness is contingent on the initial quality of the audio and visual inputs. In scenarios where inputs are of poor quality excessively noisy, the performance of our LLM and VLM might be compromised,

---

potentially leading to sub-optimal synchronization and alignment. Addressing these limitations will be crucial for enhancing the robustness and versatility of our approach.
Broader Impact. Our work sets new benchmark in the field of audio-visual (AV) representation learning, demonstrating that data-centric approach, powered by advanced Al models, lead to substantial improvements in both the performance and utility of AV systems.
Acknowledgement. This work supported by DAMO Academy through DAMO Academy Research Intern Program.
References
[1] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2016. 3,
[2] Andrew Owens, Jiajun Wu, Josh H. McDermott, William Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 801-816, 2016. 1.2
[3] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 609-617, 2017. 1,2,3
[4] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European Conference on Computer Vision (ECCV), pages 570-586, 2018. 1, 2, 7,
[5] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 1735-1744, 2019. 1,2
[6] Chuang Gan, Deng Huang, Hang Zhao, Joshua B. Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11778-0487, 2020. 1,2
[7] Pedro Morgado, Yi Li, and Nuno Vasconcelos. Learning representations from audio-visual spatial alignment. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 7733-4744, 2020. 1,
[8] Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimina- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12934-12945, 2021. 1,2
[9] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Proceedings of Advances in Neural Information
Processing Systems (NeurIPS), 2018. 2
[10] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and So Kweon. Learning to localize sound source in visual scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4358 4366, 2018. 2, 7, 8, 15
[11] Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visuaa instance discrimination with cross-modal agreement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22475-12488 June 2021.
[12] John Hershey and Michael Casey. Audio-visual sound separation via hidden markov models. Advances in Neural Information Processing Systems, 14, 2001. 2
[13] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker- independent audio-visual model for speech separation. arXiv preprint arXiv: 1804 036/9, 2018. 2
10

---

[14] Di Hu, Feiping Nie, and Xuelong Li. Deep multimodal clustering for unsupervised audiovisual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2019. 2
[15] Weiguo Pian, Shentong Mo, Yunhui Guo, and Yapeng Tian. Audio-visual class-incremental learning. arXiv preprint arXiv:2308. 1073, 2023. 2
[16] Shentong Mo, Jing Shi, and Yapeng Tian. DiffAVA: Personalized text-to-audio generation with visual aligmment. arXiv preprint arXiv:2305.12900 2023. 2
[17] Shentong Mo and Pedro Morgado. A unified audio-visual learning framework for localization, separation, and recognition. In Proceedings of the International Conference on Machine Learning (ICML), 2023. 2, 4, 8, 9, 15
[18) Shentong Mo, Weiguo Pian, and Yapeng Tian. Class-incremental grouping network for continual audio-visual learning, arXiv preprint arXiv:2309.05281, 2023. 2
[19] Lin Zhang, Shentong Mo. Yijing Zhang, and Pedro Morgado. Audio-synchronized visual animation. arXiv preprint tmXiv:2403.05653 2024. 2
[20] Shentong Mo and Pedro Morgado. Audio-visual generalized zero-shot learning the easy way.
arXiv preprint arXiv:2407. 13095, 2024. 2
[21] Shentong Mo, Jing Shi, and Yapeng Tian, Text-to-audio generation synchronized with videos. arXiv preprint arXiv:2403.07958, 2024. 2
[22] Shentong Mo, Haofan Wang, Huaxia Li, and Xu Tang. Unified video-language pre-training with synchronized audio. arXiv preprint arXiv:2405.07200 072202, 2024.
[23] Shentong Mo and Yapeng Tian. Semantic grouping network for audio source separation. arXiv preprint arXiv:2407.0331 03736, 2024.7 2
[24] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of European Conference on Computer Vision (ECCV), 2018. 2
[25] Yan-Bo Lin, Yu-Jhe Li, and Yu-Chiang Frank Wang. Dual-modality seq2seq network for audio-visual event localization. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2002-2006, 2019. 2
[26] Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang. Dual attention matching for audio-visual event localization. Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 6291-6299, 2019.
[27] Yan-Bo Lin and Yu- Ii-Chiang Frank Wang. Audiovisual transformer with instance attention for audio-visual event localization. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2020. 2
[28] Shentong Mo and Pedro Morgado. Benchmarking weakly- audio-visual sound localization. In European Conference on Computer Vision (ECCV) Workshop. 2022. 2
[29] Shentong Mo and Yapeng Tian. AV-SAM: Segment anything model meets audio-visual local- ization and segmentation. arXiv preprint arXiv:2305.01836, 2023. 2
[30] Shentong Mo and Bhiksha Raj. Weakly-supervised audio-visual segmentation. arXiv preprint arXiv:2311.15080 2023. 2
[31] Shentong Mo and Yapeng Tian. Audio-visual grouping network for sound localization from mixtures. arXiv preprint a33iv: 230.:3/500750 2023. 2
[32] Shentong Mo and Pedro Morgado. Unveiling the power of audio-visual early fusion transformers with dense interactions through masked modeling. arXiv preprint 2311v22312.01017, 2023, 2
11

---

[33] Tanvir Mahmud, Shentong Mo, Yapeng Tian, and Diana Marculescu. Ma-avt: Modality alignment for parameter-efficient audio-visual transformers. arXiv preprint wrXiv:2406.04938 2024. 2
[34] Shentong Mo and Haofan Wang. Multi-scale multi-instance visual sound localization and segmentation. arXiv preprint 24Xii:24090004886 2024. 2
[35] Pedro Morgado, Nuno Vasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. In Advances in Neural Information Processing
Systems (NeurIPS), 2018. 2
[36] Ruohan Gao and Kristen Grauman. 2.5d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 324-333, 2019,
[37] Changan Chen, Unnat Jain, Carl Schissler, S. V. A. Garí, Ziad AI-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In Proceedings of European Conference on Computer Vision (ECCV), pages 17-36, 2020. 2
[38] Changan Chen, Sagnik Majumder, Al-Halah Ziad, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to set waypoints for audio-visual navigation. In Proceedings of International Conference on Learning Representations (ICLR), 2021. 2
[39] Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip W Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022. 2
[40] Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly- supervised audio- video parsing. In Proceedings of European Conference on Computer Vision (ECCV), page 436-454, 2020. 2
[41] Yu Wu and Yi Yang. Exploring heterogeneous clues for supervisd audio-visual video parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2021.
[42] Yan-Bo Lin, Hung- Yu Tseng, Hsin- Ying Lee, Yen- Yu Lin, and Ming-Hsuan Yang. Exploring cross-video and cross-modality signals for weakly-supervisee audio-visual video parsing, In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2021. 2
[43] Shentong Mo and Yapeng Tian. Multi-modal grouping network for weakly-supervised audio- visual video parsing, In Proceedings of Advances in Neural Information Processing Systems (Neur[PS), 2022. 2
[44] Shentong Mo and Yapeng Tian. Semantic-aware multi-modal grouping for weakly-supervised audio-visual video parsing. In European Conference Computer Vision (ECCV) Workshop. 2022.
[45] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan. Binh Tang, Ross Taylor, Adina Williams. Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint
2023. 2
[46] OpenAl. Gpt-4 technical report. arXiv preprint
2023. 2
12

---

[47] Jiuniu Wang, Zehua Du, Yuyuan Zhao, Bo Yuan. Kexiang Wang, Jian Liang, Yaxi Zhao, Yihen Lu, Gengliang Li, Junlong Gao, Xin Tu, and Zhenyu Guo. Aesopagent: Agent-driven evolutionary system on story-to-video production. arXiv preprint arXiv:2403.07952. 2024. 2
[48] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoa- gent: A angmory-augmented multimodal agent for video understanding. arXiv preprint arXiv:2403. / 1481, 2024. 2
[49] Bin Lin. Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:23 1.222311100111 101122, 2023. 4
[50] Google. Gemini: A family of highly capable multimodal models. arXiv:2312. 1805, 2024. 4
arXiv preprint
[51] Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chathot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
[52) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xxiv22106.09685, 2021. 4
[53] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 4
[54] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End- to-end multi-grained contrastive learning for video-text retrieval. In Proceedings of ACM International Conference on Multimedia (ACMMM), 2022.
[55] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721-725. IEEE, 2020. 6, 7, 15, 16
[56] Xixi Hu, Ziyang Chen, and Andrew Owens. Mix and localize: Localizing sound sources in mixtures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 10483-10492, 2022. 8, 15
[57] Shentong Mo and Pedro Morgado. A closer look at weakly-supervised audio-visual source localization, Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022. 7, 8, 15
[58] Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation. In Proceedings of European Conference on Computer Vision (ECCV), 2022. 7, 15, 16
[59] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2/11.06377 2021. 7, 8, 15
[60] Po-Yao Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. In Proceedings of Advances In Neural Information Processing Systems (NeurIPS), 2022. 7, 8, 15
[61] Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R. Glass. Contrastive audio-visual masked autoencoder. In Proceedings of The Eleventh International Conference on Learning Representations (ICLR), 2023. 7,8
[62] Po-Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Haoqi Fan, Yanghao Li, Shang Shang-Wen Li, Gargi Ghosh, Jitendra Malik, and Christoph Feichtenhofer. Mavil: Masked audio-video learners. arXiv preprint arXiv:2212. 2022. 7, 8
13

---

[63] Mariana- Iulinna Georgescu, Eduardo Fonseca, Radu Tudor lonescu, Mario Lucic. Cordelia Schmid, and Anurag Amab. Audiovisual masked autoencoders. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16144-16154, October 2023. 7, 8
[64] Shentong Mo and Pedro Morgado. Localizing visual sounds the easy way. In Proceedings of European Conference on Computer Vision (ECCV), page 218-234, 2022. 7, 8, 15
[65] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXtv:1412.6980, 2014.
15
[66] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European Conference on Computer Vision (ECCV), pages 435-451, 2018, 8
[67] Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Proceedings of European Conference on Computer Vision (ECCV), pages 292-308, 2020. 8
[68] Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou. Discriminative sounding objects localization via self-supervised audiovisual matching. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pages 10077-
10087, 2020. 8
[69] Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Localizing visual sounds the hard way. In Proceedingy of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16867-16876, 2021.
[70] Tuomas Virtanen. Monaural sound source separation by nonnegative matrix factorization with temporal continuity and sparseness criteria. IEEE Transactions on Audio, Speech, and Language Processing, (13):1066-1074 2007. 8
[71] Po-Sen Huang, Scott Decann Chen, Paris Smaragdis, and Mark Hasegawa-Johnson Singing- voice separation from monaural recordings using robust principal component analysis. IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 57-60, 2012.
[72] Xudong Xu, Bo Dai, and Dahua Lin. Recursive visual sound separation using minus-pli net. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
8, 9
[73] Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-learning of sounding object visual grounding and sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2745 2754, 2021. 8, 9
14

---

Appendix
In this appendix, provide the following material:
addition implementation and datasets details in Section A,
algorithm for our AVACENT in Section B,
additional experimental analyses in Section C.
additional example results from our AVAGENT in Section D,
additional discussions on limitations and broader impact Section
A Implementation & Dataset Details
In this section, we provide more implementation and dataset details.
Audio-visual classification. For linear probing, we follow the prior work [59, 60] and extract frozen audio-visual representations from our AVAGENT pre-trained audio-visual masked autoencoder. Then we attach linear layer head to the frozen features for training with the audio-visual classes. During training, we only fine-tune the linear head to evaluate the quality of pre-trained features. The models are trained for 50 epochs using the Adam optimizer [65] with learning rate of Le 4 and a batch size 128. For fine-tuning, we use the same optimizer and batch size settings, but all parameters are learnable.
Sound source localization and segmentation. For sound source localization, we train all base- lines [64, 57, 56] using the same backbone (iee., ViT-Base) for audio/visual encoder with different proposed objectives in their original papers. The final localization map is generated through bilinear interpolation of the similarity map between audio/visual features from the last self-attention layer. The models are trained for 30 epochs using the Adam optimizer [65] with a learning rate of and batch size of 128. For segmentation, we follow the prior work [58], and apply an upsampling decoder on features from the last self-attention layer to generate the final segmentation mask. We use the binary cross entropy (BCE) loss between the prediction and ground-truth masks for training. The models are trained for 20 epochs using the Adam optimizer [65] with a learning rate of le 4 and a batch size of 128.
Sound source separation. For sound source separation, we follow the previous method [4, 17] and attach an audio U-Net decoder to pre-trained audio-visual encoders for separating sounds from the mixture. The decoder depth for self-attention layers 8, and the decoder receives the representations of the audio mixture and the visual embeddings. We also apply multiple transposed convolutions and an output head to predict a time-frrqquee free-frreunncy separation mask. This separation mask then used to multiply the input mixture STFT to separate the audio. Similarly to [4], the target masks refer to the Lime-frequency bins where the source is the most dominant component the mixture. The sound source separation is achieved by optimizing binary cross-entropy loss over these binary targets. The model trained for 20 epochs using the Adam optimizer [65] with a learning rate of le 4 and a batch size of 128.
Dataset Details. We evaluated our method using several prominent audio-visual datasets:
Flick-SoundNet [10]: a dataset consisting of natural soundscapes with associated Flickr images with 4,500 audio-visual pairs for training and testing the model on 250 audio-visual pairs of sounding objects and extended 250 non-sounding objects;
VGG-Instrumenns [56]: contains video clips of musical instrument performances, with 32k video clips of 10s lengths from 36 musical instrument classes, a subset of VGG-Sound [55], and each video only has single instrument class;
MUSIC [4]: consists of 448 untrimmed YouTube music videos of solos and duets from 11 instrument categories;
VGG-Music [17]: dataset that features collection of music videos with annotations related to the genre and instruments present;
VGGSound [55]: comprehensive dataset that includes wide variety of sound categories and corresponding visual scenes, which contains categories, such as animals, instruments, vehicles, people, etc;
15

---

Algorithm Algorithm for AVAGENT
Require: A dataset of audio-visual pairs {(a;, =
Ensure: Improved alignment and synchronization of audio-visual pairs
for each pair (a,, 04) do
2: Generate independent audio and visual captions using multimodal LLM:
3: c; mLLM a;)) [Audio caption)
4:
t 10mm([() (Visual caption)
5:
Assess reflection using VLM:
6:
+ VLM(a" 2t) {Alignment score)
7:
sf < VLM((", " vt) (Synchronization score)
8:
while alignment and synchronization sf exceeds threshold (0.85) do
9:
Plan audio modifications increase 8; and
10:
df LLM(c, d.)
11:
Apply audio modifications:
12:
o. + Modify ModifyAudio Audio actions)
13:
Assess reflection using VLM:
14:
VLM(a, v4) (Alignment score)
15:
S 6 VLM(a, 24) {Synchronization score)
16:
if improved and s improved then
17:
Update audio (2; 4- of end if
18:
19:
Update alignment based on new score 8; Update synchronization sf based on new score s
21:
end while
22:
Store improved pair (a;, Uz)
23:
end for
AudioSet [55]: collection of 2,084,320 human-labeled 10-second sound clips drawn from YouTube videos with 632 audio event classes;
AVSBench [58]: benchmark for testing audio-visual synchronization and alignment in diverse settings, including 4,932 vidcos (in total 10,852 frames) from 23 categories, including instruments, humans, animals, etc.
B Algorithm for AVAgent
Our approach is formalized through the following algorithmic steps:
1. Input an audio-visual pair.
2. Use the multimodal LLM to independently generate text descriptions for both audio and visual data.
Analyze the descriptions to detect discrepancies and assess alignment using trained LLM model.
4. Plan and apply necessary audio modifications to enhance alignment, such as noise filtering and temporal adjustments.
5. Use VLM to assess the effectiveness of the modifications and provide feedback score.
6. Iterate the process until the audio-video alignment meets the predefined threshold or maxi- mizes the synchronization score.
Algorithm 1 outlines the steps followed by our AVAGENT in processing and synchronizing audio- visual data using data-centric approach. The algorithm utilizes a multimodal large language model (mLLM), large language model (LLM).. and vision-languee model (VLM) to refine audio to better align with the visual content iteratively. This algorithm encapsulates the eyclic process of audio signal refinement, leveraging both machine learning models and heuristic analysis to ensure that the audio accurately reflects the visual context. Fach step of the process designed to enhance the synchronization between the audio and visual modalities iteratively.
16

---

Table 6: Ablation studies LoRA tuning. Quantitative results VGGSound-AlI, Flickr-SoundNet, AVS-
Bench, and VGGS-Music datasets.
VGGSound-All
Foickr-SoundNet
AVSBench
VGGS-Music
LoRA Tuning
Linear (%)
Finetune (%)
Precision
F1
mloU
F1
SDR
SAR
59.87
67.95
56.93
59.37
63.86
33.79
46.58
3.89
3.92
61.56
69.24
58.23
60.76
65.03
36.37
49.85
5.26
5.28
C
Additional Experimental Analyses
this section, we conduct additional experimental analyses to explore the impact of LLM (Low-Rank Adaptation) tuning in our AVAGENT. The results reported in Table 6. LoRA tuning enhances our LLM's memory and context retention capabilities, enabling it to make more informed decisions regarding AV synchronization. We implement LoRA tuning the LLM to fine-tune it specifically for tasks that require a deep understanding of the audio-visual context. This involves adjusting the
LLM's parameters to remember better and integrate lengthy audio-visual sequences. The application of LoRA tuning leads to noticeable improvements tasks such as audio-visual source localization and segmentation, where the precision of temporal and spatial alignments is erucial. Metries such as mloU and Fl score show substantial enhancements. These improvements can be attributed to the LLM's enhanced ability to maintain context over extended interactions, proving essential in complex scenarios where long-term dependencies are critical. These analyses highlight the efficacy of our adaptive, LLM-driven approach achieving high-quality AV synchronization. By leveraging sophisticated LLM tuning methods and moving away from naive, random modifications, our method sets new standard AV data processing, paving the way for more accurate and effective multimedia applications.
D Additional Examples
In this section, we present detailed examples of the agent workflow in action, demonstrating the iterative process of aligning audio-visual content through the use of tool use, planning, and reflection phases. Each example includes inputs, model outputs, action plans, and reflection scores, showcasing the method's effectiveness in improving synchronization and alignment.
D.1 Example Lecture in a Large Hall
Figure provides full example of our agent workflow for lectures in large hall.
D.2
Example 2: Dog Barking in Park
Figure provides full example of agent workflow for dog barking in the park.
D.3
Example 3: Waterfall Scene
Figure provides full example our agent workflow for the waterfall scene.
E
More Discussions
E.1
Limitations
Our approach, while advancing the state-of-the-ar in audio-visual synchronization, is subject to several limitations that should be considered:
Dependency on Data Quality: The performance our method heavily relies on the quality of the input data. Poor quality video or audio, such as low-resolution video highly distorted audio, can significantly impair the ability of the L.M and VLM to accurately generate useful captions and subsequent synchronization actions.
17

---

ANOWLEDDE
ERBITY
Hilanal
UNITY
you specity editing input: professor lecturing large hall corresponding with echoing?
Request
mLLM output:
Tool
Video caption: a professor lecturing in a large hall caption:
echoing voice spacious
Planning
captions, actions you choose given action for editin audio to achieve noise filtering and improve audio-video synchronization?
Input: Given video
Noise Filtering Actions: Spectral Subtraction, Wiener Fitterigg, Wavelet Denoising, Spectral Gating: Modification,
Augmentation
Speed Modification,
Adjustment, Filling Blanks
Actions: Wiener Filtering and Pitch Modilication.
Reasoning: Apply Wiener filtering to minimize the echo effect, making the professor's voice clearer synchronized with video.
VLM Feedback:
Reflection
VLM alignment score: 0.68
Audio-video temporal synchronization
AVAgent
Planning
LLM Input: Given the VLM Reflection outputs and original video/audio captions, what actions would you choose from the given action set for editing audio achieve noise filtering and improve audio- synchronization?
Noise Filtering Actions: Spectral Subtraction, Wiener Fittering, Wavelet Denoising, Spectral Gating: Data Augmentation Actions: Speed Modification, Pitch Modification, Volume Adjustment, Filling Blanks
Actions: Spectral Gating
Reasoning: implement spectral geting enhancing the clarity of the lecture audio.
any
background
VLM Feedback:
Reflection
VLM alignment score: 0.87
Audio-video temporal synchronization score:
The audio editing actions for input: video of professor lecturing in large hall and corresponding unclear audio with echoing. could be Spectral Gating.
Figure 5: Illustration of full example (Lecture in a Large Hall) of our agent workflow.
The algorithms employed, particularly the use of advanced LLMs like Vicuna-vl.5-7b and sophisticated VLMs, require substantial computational power. This may limit the practicality of our method for use in real-time applications or on platforms with limited processing capabilities.
Computational Resources:
Generalization across Diverse Environments: While the method performs well across several tested environments and datasets, its effectiveness in extremely noisy acoustically complex environments has not been extensively verified. There may be scenarios where the method's ability to discern and correct misalignments is diminished.
Scalability Issues: The iterative nature of the workflow, while effective, may not scale efficiently with the volume of data or the complexity of the audio-visual scenes. This could
18

---

Can you specity audio editing actions for input: video of a dog barking in a park and Its corresponding close, muttled audio?
Request
mLLM output:
Tool
Video caption: dog barking loudly park
caption: barking park
Planning
for editing audio achieve noise filtering and improve audio-video synchronization?
Input:
captions,
you choose given
Noise Filtering Actions: Spectral Subtraction, Wiener Fitteiing, Wavelet Denolsing, Spectral Gating: Data Augmantation Actions: Speed Modilication,
Modilictiin, Volume Adjustment, Filling Blanks Actions: Spectral Subtraction and Volume Adjustment.
Ressoning: Use spectral subtraction reduce ambient noise and enhance the clarity of the barking sound. Increase the volume the barking make more prominent, aligning with the visual emphasis the dog barking loudly.
VLM Feedback:
Reflection
VLM alignment score: 0.65
Audio-video temporal synchronization
AVAgent
Planning
LLM Input: Given the VLM Reflection outputs and original video/audio captions, what actions would you choose from the given action set for editing audio achieve noise filtering and improve audio- synchronization?
Noise Filtering Actions: Spectral Subtraction, Wiener Fittering, Wavelet Denoising, Spectral Gating: Data Augmentation Actions: Speed Modification, Pitch Modification, Volume Adjustment, Filling Blanks Actions: Wavelet Denoising and Pitch Modification.
denoising remove residual better match the dog's size and appearance in the video.
Reasoning: Apply
adjust pitch barking
VLM Feedback:
Reflection
VLM alignment score: 0.89
Audio-video temporal synchronization score: 0.87
The audio editing actions for input: video of professor lecturing in large hall corresponding unclear audio with echoing. could be Wavelet Denoising and Pitch Modification
Figure 6: Illustration of a full example (Dog Barking in Park) of our agent workflow.
pose challenges in deploying the system at larger scale, such as in industry-wide media processing pipelines.
The use of LLMs and VLMs in processing potentially sensitive audio-visual content raises ethical and privacy concerns. There is risk that such technology could be used to manipulate audio-visual content in misleading ways, that sensitive information could inadvertently exposed during processing.
Ethical and Privacy Concerns:
E.2 Broader Impact
The broader impact of our research is multifaceted, with potential implications for numerous fields:
19

---

you specity editing input: overpowering
Request
mLLM output:
Tool
Video caption: majestic waterfall flowing
Audio caption: loud wind noise obscuring the sound water and crowd voice
Planning
Input:
captions,
you choose given
for editing audio to achieve noise fittering and improve audio-video synchronization?
Noise Fillering Actions: Spectral Subtraction, Wiener Fittringg Wavelet Denoising, Spectral Gating: Data Augmentation Actions: Speed Modification, Pitch Modificaioo, Volume Adjustment, Filling Blanks Actions: Spectral Subtraction and Spectral Gating.
Reasoning: Use spectral subtraction reduce wind noise. Apply spectral gating to allow only the frequencles of the waterfall sound pass through, reducing the wind noise influence.
VLM Feedback:
Reflection
VLM alignment score: 0.72
Audio-video temporal synchronization
AVAgent
Planning
Reflection outputs original video/audio captions, actions would you choose from the given action set for editing audio to achieve noise filtering and improve audio- video synchronizalion?
Input:
Filtering, Wavelet Denoising, Spectral Gating: Data Augmentation Actions: Speed Modilication, Pitch Modification, Volume Adjustment, Filling Blanks Actions: Wavelet Denoising.
Filtering
Spectral
Reasoning: Employ wavelet denoising finely any remaining wind without affecting the natural sound of waterfall.
Feedback:
Reflection
VLM alignment score: 0.88
Audio-video temporal synchronization score:
editing input: professor lecturing large corresponding unclear echoing. could be Wavelel Denolsing.
Figure 7: Illustration of a full example (Waterfall Scene) of our agent workflow.
Our method can significantly enhance the quality of multimedia content by ensuring better synchronization between audio and visual elements, leading to more immersive experiences in film, television, and virtual reality.
Media and Entertainment:
Accessibility Enhancements: Improved audio-visual synchronization can also benefit accessibility technologies, such as developing more accurate closed captioning and audio descriptions for the hearing or visually impaired.
Educational Tools: In educational settings, our method can be used to create more engaging and comprehensible instructional videos, where precise alignment audio explanations with visual demonstrations is crucial.
20

---

Surveillance and Security:
Enhanced synchronization capabilities may improve the relia- bility of surveillance systems where audio cues are critical to understanding visual footage.
Ethical Considerations: While our technology has many positive applications, it is essential to develop guidelines to prevent its misuse, partieularly in contexts where audio-visual misalignment could be used to deceive or mislead viewers.
In conclusion, while method presents significant advancements and potential benefits, it is essential to continue refining the technology to address its limitations and ensure it is used responsibly and ethically in diverse audio-visual applications.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section I
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See our discussion in the limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
We do not have theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
See the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
hatpe//CeeSSueeseeeeeeeee wwwiiisssooooioooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct guides/ /CodeSubmic Sttp:///ggiiieeeeeeeuuuuiiiiiiii ssisnnoillll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tuurrrss.... httpplic//thissuiieeeee.
Answer: [Yes]
Justification:
We have conformed in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See our discussions in the broader impact section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Section 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer:
[NA]
Justification:
We do not have new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not have experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "hSXnAxtrT9", "reviewer": "Reviewer_Jci9", "summary": "The authors propose to leverage large language model with agentic workflow such as planning, reflection and tool usage for better audio-visual alignment in training data applied to various downstream tasks such as classification, localization and separation.", "questions": "- Which audio model is used in this work? Is Google Gemini used for general audio understanding to output the audio captions?\n- Are you planning to release the action planning paired data for video, audio, and action streams? How are these data curated?\n- How frequent are the VLM alignment score and temporal synchronization score computed and feedback? Are the scores before any audio editing is taken place first reported so that the LLM has the information before proposing any method to use? Are these scores computed each time after the proposed actions for the LLM to reflect on?\n- In section 4.1 implementation, the authors mentioned that the audio is represented by log spectrograms and STFT is applied to generate input tensor. [48] only applied log spectrogram without the STFT, what is the intuition here?", "limitations": "- Proposed method can be considered as preprocessing methods for audio-visual data, and can potentially be applied to most of the prior methods compared in this paper as preprocessing steps, it would be beneficial to show how proposed method can improve several of the prior works, which could be a more generalized contribution.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The experiments with various downstream tasks such as classification, localization/segmentation, and sound source separation demonstrate that the proposed method can be utilized as a general preprocessing steps for audio-visual applications.", "weaknesses": "- In section 3.3, the explanation of visual alignment and temporal synchronization scores can be made more clear. In section 3.2, ImageBind is mentioned, while in section 3.3, X-CLIP is mentioned, are the visual alignment scores based on similarities these embeddings? How are the audio embeddings computed? Are the temporal alignment computed based on sampling of audio and image from the video? What are the sampling rates? And how are these information further feedback to the LLM in order to determine the parameters for the tool usage?\n- The proposed method is to improve preprocessing of the dataset. Though in Table 2, 3, 4 the comparison with various methods (they differ from various perspectives including architecture choices, datasets used for pre-training, loss function design, etc.) including the SOTA shows that proposed method outperforms. It would be more beneficial to highlight the comparison with previous methods with only difference in preprocessing step (agentic alignment as proposed). It would potentially provide a more thorough view if one of the SOTA trained with proposed preprocessing steps is shown and highlighted in the table.\n- In section 4.1, consider use tables or bullet points for introducing all the datasets used, consider break them down first by downstream task, different dataset, and number of train/val/test splits, etc. This can help the reader map each dataset to the results in table 2, 3, and 4 more easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the valuable comments and answer the raised questions below.\n\n> W1: Clarification of Visual Alignment and Temporal Synchronization Scores.\n\nIn our framework, visual alignment scores are computed using ImageBind, which assesses the similarity between visual data and modified audio descriptions. Temporal synchronization scores are derived from X-CLIP embeddings, providing a robust measure of how well the audio content temporally aligns with the visual frames. Audio embeddings are generated using a standard log-spectrogram approach, with a sampling rate of 10s audio at a sample rate of 16000Hz to ensure consistency with video frame rates. \nThe alignment scores are provided as the context input into LLM to determine which action should be used for the next iteration. \nTherefore, these alignment metrics are computed after any audio editing to provide feedback context for iterative refinement in the next step.\n\n> W2: Comparison with Previous Methods Using Only Preprocessing Steps\n> \nTo isolate the impact of our agentic alignment preprocessing, we retrained several state-of-the-art models using both the standard preprocessing methods and our enhanced approach. The comparisons are reported in the Tables 1-3 in the attached pdf. Our agentic alignment preprocessing boosts all models in terms of diverse downstream tasks.\n\n> W3: Suggestions on Datasets Introduction\n> \nThanks for your suggestion. We updated the detailed settings in Table 4 in the attached pdf to introduce all the datasets.\n\n> Q1: Audio Model\n\nThe audio captions in our system are generated using Google Gemini, which has shown robust performance in general audio understanding tasks across diverse datasets.\n\n>Q2: Release of Action Planning Paired Data\n\nWe will make the action planning paired data publicly available. This dataset includes video-audio data pairs, and the corresponding actions that are from the action pool illustrated in Fig. 3. \nFor each data sample, we save the final action corresponding to the video and audio file name in a json file, which is formatted as [{video_audio_file1: action1}, {video_audio_file2: action2}, ...]. We also share the preprocessed audio wav file with more researchers to use for extended usage. Each video-audio data pair is meticulously curated to demonstrate specific alignment and synchronization challenges and their solutions.\n\n> Q3: Frequency of Computing VLM Alignment and Temporal Synchronization Scores\n> \n1) VLM alignment and temporal synchronization scores are computed following each proposed action to provide immediate feedback. \n2) Initial scores are also computed before any audio editing to establish a baseline, ensuring that each action's impact can be accurately assessed.\n3) These scores are computed each time after the proposed actions for the LLM to reflect on.\n\n> Q4: STFT and Log Spectrograms \n\nWe would like to clarify that [48] also used STFT, which they did not explicitly mention in the paper but STFT was used in their provided source code. To compute the log spectrograms, we follow their implementation and apply an STFT using approximately 50ms windows with a hop size of 15ms, resulting in an input tensor of size 128 × 196 (128 mel frequency bands over 196 time steps) on the original audio waveform. \n\n> L1: Generalization of Preprocessing Method\n\nTo demonstrate the general applicability of our preprocessing method, we have conducted more experiments where we apply our techniques to several state-of-the-art models across different tasks. Results, as shown in Tables 1-3 in the attached pdf, illustrate marked improvements in performance, highlighting our method's potential as a beneficial preprocessing step for a wide range of audio-visual applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYZ7RWqtVR", "reviewer": "Reviewer_RJkZ", "summary": "The paper introduces a new method to improve audio-visual joint representations by strategically aligning audio with visual data. This method employs an LLM-based assistant, AVAgent, which uses multi-modal LLMs to analyze audio and visual content separately and then plans corrective actions to better align them. The process involves noise filtering and synchronization improvements, enhanced by continuous reflection steps. The effectiveness of this approach is validated through extensive experiments across multiple public datasets, showing significant advancements over established baselines in tasks such as audio-visual classification, localization, segmentation, and source separation.", "questions": "- The authors should specify which subset of AVSBench was used to ensure that the results are accurately understood and contextualized within the appropriate segmentation task framework.\n- The authors could discuss or simulate how features processed by their method might improve other frameworks, even as a conceptual exploration, to underscore the broader applicability of their approach.\n- Recommendation: The proposed method focuses on improving audio-visual data alignment, a key factor in tasks like Audio-Visual Voice Parsing (AVVP), which emphasizes temporal synchronization. A discussion section could be added to explore hypothetically how the method might perform on tasks like AVVP, based on the alignment improvements demonstrated in the current tasks. \n- A brief discussion on the expected computational demands could make the paper more comprehensive and practical for a wider audience.", "limitations": "The the authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Innovative Approach: The paper introduces an agentic workflow using LLMs for audio-visual alignment, leveraging modern LLM advancements to tackle synchronization and noise issues, marking a significant advance in multimedia processing.\n- Comprehensive Evaluation: Extensive empirical evaluations are conducted across various public datasets and audio-visual tasks, adding robustness and supporting the method's generalizability.\n- The workflow is clearly outlined, detailing the specific actions of AVAgent in aligning audio and visual data, enhancing understanding of the system’s mechanics.", "weaknesses": "- Dataset Specification Clarity: The paper utilizes the AVSBench dataset for evaluating the effectiveness of the proposed method in audio-visual synchronization and segmentation. However, it lacks specificity regarding which subset of AVSBench was employed—AVSBench-object or AVSBench-semantic. Each subset serves distinct segmentation tasks that influence the interpretation and relevance of the results.\n- While the paper demonstrates significant advancements in audio-visual alignment, there is potential to explore how these aligned features could be used to enhance the performance of other existing audio-visual frameworks. Some baselines already have better performance, suggesting that integrating features from this work could provide benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the valuable comments and answer the raised questions below. \n\n> W1, Q1: Dataset Specification Clarity\n\nFor our evaluations, we specifically utilized the AVSBench-semantic subset, which focuses on semantic segmentation tasks. This choice is pivotal as it directly relates to our method's ability to enhance audio-visual alignment, which is critical for accurately segmenting complex scenes based on both audio and visual cues.\n\n> W2, Q2: Applying to Other Frameworks\n\nTo demonstrate the general applicability of our preprocessing method, we conducted additional experiments where we apply our techniques to several state-of-the-art models across different tasks. Results, as shown in Tables 1-3 in the attached pdf, illustrate marked improvements in performance, highlighting our method's potential as a beneficial preprocessing step for a wide range of audio-visual applications.\n\n> Q3: Performance on AVVP\n>\nWe further conducted experiments on AVVP to quantitatively compare the segment-level results in Table 6 in the attached pdf. Adding our agentic alignment to the strong baseline [VALOR] boosts the performance in terms of all metrics.\n\n[VALOR] Yung-Hsuan Lai, et al. Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser. NeurIPS 23.\n\n> Q4: Computational Demands Discussion\n\nThe computational demands of our method are primarily driven by the use of large-scale LLMs and multimodal processing. While our current implementation is designed for offline preprocessing of datasets before model training, understanding the computational requirements is crucial for practical application. Future iterations of our work will focus on optimizing these processes to allow for more efficient real-time applications, potentially broadening the method’s applicability in resource-constrained environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ji6t9iJJyJ", "reviewer": "Reviewer_siDn", "summary": "This paper proposes a data-centric agentic workflow (AVAgent) to improve the alignment of audio-visual joint representations. The workflow is controlled by LLMs and consists of cyclic tool use, planning, and reflection steps. This paper claims the proposed method can analyze and adaptively modify the audio stream in alignment with the corresponding video content, and boost performance across various audio-visual tasks such as classification, source localization, segmentation, and source separation.", "questions": "- Is existing multi-modal LLM like Gemini really able to capture thorough audio components from the given audio, especially when there are many audio sources? Did you apply any special processing to the data or other operations to adapt mLLM better on the audio captioning?\n- If we directly include the Audio Noise Filtering Actions, volume normalization, and blanks filling as part of preprocessing for all audio samples, then pretrain the audio-visual representation, will the downstream task performance also be as good as the current ones?", "limitations": "The authors have explained the current limitations and broader impact in the paper, no clear cues on potential negative social impact from the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+  Novelty and Completeness: This paper proposes a novel agentic workflow, thoroughly covering the tool use, planning, and reflection steps. Experiments are conducted for several major downstream audio-visual tasks, and results show the proposed method can boost the accuracy of these tasks.\n\n+ Clarity: This paper is written clearly with details and good presentation, easy to read and follow. The motivation and problem analysis are reasonable, and target the critical challenges for audio-visual learning.", "weaknesses": "- Limited capability of the proposed workflow\nThis paper considers 8 audio-relevant actions. However, most of the actions are fundamental signal-processing operations, which might not be able to handle such as audio-visual sync problems (time delay in the middle of audio..) robustly or complicated audio scenes where noises dominate the whole audio, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the valuable comments and answer the raised questions below. \n\n> Capability of the Proposed Workflow\n\nWe agree that there is much room to improve our actions. At present, we focus on the audio-visual downstream scenarios such as audio-visual classification, sound localization, segmentation, and separation. These scenarios are related to audio-visual recognition where we are able to handle normal audio-visual sync and noisy problems. The results in Table 2-4 have shown our superior performance. In the future, we will tackle these abnormal/challenging problems by incorporating learning-based enhancements that are capable of dynamic and context-sensitive adjustments. This will allow for more robust handling of time delays and overlapping audio sources, significantly enhancing the system's utility in real-world, noisy environments.\n\n> Capability of Multi-modal LLMs in Handling Multiple Audio Sources\n\nOur current implementation primarily utilizes Gemini for single audio source scenarios to ensure clear and focused audio captioning. While Gemini is capable of handling multiple audio sources, in this study, we deliberately focused on single-source audio to maintain clarity and control in our experimental setup. No special processing was applied to adapt the multi-modal LLM (mLLM) specifically for audio captioning beyond its original training, ensuring that our results are broadly applicable and reproducible using standard LLM configurations.\n\n> Effect of Comprehensive Audio Preprocessing\n\nTo explore the potential of integrating comprehensive audio preprocessing steps, such as noise filtering, volume normalization, and blanks filling, we conducted additional experiments. These steps were applied randomly across all audio samples before training the audio-visual representation models. As shown in Table 5 **in the original draft**, our approach with agentic alignment significantly enhances downstream task performance compared to the vanilla baseline using random actions. We further conducted experiments on including all actions as preprocessing for all audio samples in Table 5 **in the attached pdf**, confirming that our agentic alignment can substantially benefit audio-visual learning, especially in complex acoustic environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TFR8wMJRAk", "reviewer": "Reviewer_GoMh", "summary": "This paper deals with the problem of audio-visual representation learning and proposes a method for reducing the misalignment between audio and video streams. More specifically, the proposed method feeds audio and video streams into multi-modal LLMs to obtain textual descriptions for both streams separately and a plan for modifying audio streams for further improvements of audio-visual alignments. The proposed plan might be fixed according to the score of audio-visual alignments. Experimental evaluations with several standard audio-visual tasks and the corresponding benchmarks demonstrate that the proposed method outperforms several standard methods for audio-visual representation learning.", "questions": "Q1. Please respond to the first concern (W1) presented in the Weakness section.\n--> I deeply appreciated the author feedback. However, it indicates that each action is applied using default settings, which cannot be adjusted to most training examples. I want to believe that this is my misunderstanding, but it would be due to the inappropriate documentation.", "limitations": "No problem with this part.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "S1. The problem dealt with in this paper is significant. Audio-visual representation learning is a fundamental building block for various multi-modal tasks with video signals. Technically solid methods with excellent experimental evaluations for this task will draw the attention of the audience.\n\nS2. The proposed method is simple, which would be preferable for engineers and practitioners interested in audio-visual representation learning. Also, the proposed method provides a tool for preprocessing training examples (i.e., pairs of audio and image streams), which indicates that it can be applied to any technique for audio-visual representation learning.", "weaknesses": "W1. The current paper lacks details of the proposed method. Most critically, I could not find any descriptions of the details of \"Planning.\" As presented in Figure 4, the planning section can provide which actions should be taken for a given pair of audio and visual streams. However, each action requires more details: noise levels, wavelet selection, degree of speed changes, etc. Figure 4 implies that LLMs do not suggest the best choice of action details, and we humans should choose them. If my understanding is correct, the proposed method is useless since manual inspection is required for every training example. If my understanding is incorrect, the paper lacks significant technical details.\n\nW2. Related to the first concern, if the agent has to suggest the best sequence of actions and their details, the strategy of the proposed method that feeds textual descriptions as inputs for the planner is not a good choice. Textual descriptions are interpretable for humans and LLMs but lack details of given audio and video streams.\n\nW3. The proposed method heavily relies on the LLMs. The relient of LLMs itself is not a drawback; however, the problem is that I could not find any technical tricks in the proposed method. Almost all the procedures for refining misalignments can be provided by the LLMs, and no other tricks can be found.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank for the valuable comments and answer the raised questions below. \n\n> W1: Details in the Planning Section\n\nThe 'Planning' stage of our workflow utilizes a predefined module that automatically selects appropriate actions based on audio-visual misalignments identified by the LLM. Each action, such as noise reduction or speed adjustment, is applied using default settings from established libraries (e.g., SciPy), which are optimized for general usage. This automation does not need manual intervention, allowing our workflow to scale efficiently across numerous examples without requiring manual tuning of parameters. We will illustrate that our action setting follows prevalent designs and does not require manual adjustment in our revised manuscript.\n\n> W2: Textual Descriptions for Planning\n\nWe understand the raised concern. In practice, we empirically find the textual descriptions generated by the mLLMs (i.e., Gemini, Video-LLaVA) provide a robust basis for LLM (i.e., Vicuna) to plan effective actions. These descriptions from mLLMs indeed encapsulate key audio and visual characteristics, enabling our automatic workflow to make informed decisions about necessary adjustments. Our transformation into language form leverages the interpretability and rich semantic understanding capabilities of mLLMs, ensuring that the planned actions are both relevant and effectively targeted to address specific misalignments. \n\n> W3: Technical tricks\n\nWe would like to show that our agentic workflow based on multiple mLLMs are capable of aligning audio-visual data pairs to benefit representation learning. This cycle alignment form fully leverages the generalizations of mLLMs to process data. Our technical contributions (or tricks) reside as follows:\n\n1. We pioneeringly develop an agentic workflow containing reasoning/planning, tool use, and reflection steps for AV (audio-visual) data alignment. In contrast, prior agentic workflows are not developed by considering all these steps. For example, [a,b] mainly propose planning, [c,d] mainly develop tool use, and [e,f] mainly introduce reflection. Moreover, there is no agentic workflow developed for AV data alignment in the literature.\n2. We convert AV data in the language form to fully exploit the reasoning/planning ability of LLM. This is because we empirically find out the mLLM/LLM reasons and plans more effectively in pure language form rather than AV form, which might due to the imperfect alignment between language and other modalities. \n3. For each data sample, we save the final action corresponding to the video and audio file name and the preprocessed audio wav file. Our AVAgent is based on Vicuna-v1.5-7b and we adopt LoRA tuning so as not to affect the original reasoning ability of LLM. LoRA tuning enhances our LLM’s memory and context retention capabilities, enabling it to make more informed decisions regarding AV synchronization. We implement LoRA tuning on the LLM to fine-tune it specifically for tasks that require a deep understanding of the audio-visual context. This involves adjusting the LLM’s parameters to remember better and integrate lengthy audio-visual sequences.\n\nWe will add these illustrations in our revised manuscript.\n\n[a] AutoGPT https://github.com/Significant-Gravitas/AutoGPT\n\n[b] Visual Programming: Compositional visual reasoning without training. CVPR'23 \n\n[c] Llava-plus: Learning to use tools for creating multimodal agents. Arxiv'23\n\n[d] Gpt4tools: Teaching llm to use tools via self-instruction. NeurIPS'23\n\n[e] Self-refine: Iterative refinement with self-feedback. NeurIPS'23\n\n[f] Reflexion: Language agents with verbal reinforcement learning. NeurIPS'23", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear AC and all reviewers,

We sincerely appreciate your time and efforts in reviewing our paper. We are glad to find that reviewers recognized the following merits of our work.

- **Innovative Approach** [RJkZ, siDn, GoMh]: Our novel agentic workflow employing LLMs for audio-visual alignment introduces a robust framework for addressing synchronization and noise issues.

- **Comprehensive Evaluation** [Jci9, RJkZ, siDn, GoMh]: The extensive experimental validations across multiple datasets underscore our method's efficacy and versatility.

- **Efficiency and Practicality** [Jci9, RJkZ, siDn, GoMh]: Our method's efficiency and ease of integration into existing workflows were appreciated, making it suitable for both academic research and practical applications.

We also thank all reviewers for their insightful and constructive suggestions, which helped further improve our paper. In addition to the pointwise responses below, we summarize the responses in the rebuttal according to the reviews.

- **Generalization** [Jci9, RJkZ]: We included additional experimental comparisons with state-of-the-art methods that solely modify preprocessing steps. These experiments demonstrate that our agentic workflow significantly enhances the performance of existing audio-visual frameworks, as shown in the Tables 1-3 in the attached pdf.

- **Performance on other tasks** [RJkZ]: We conducted experiments on AVVP to quantitatively compare the segment-level results in the Table 6 in the attached pdf. Adding our agentic alignment to the strong baseline boosts the performance in terms of all metrics.

- **Clarification update** [Jci9, siDn, GoMh]: We included more in-depth dataset descriptions, experimental results, and action ablation studies in the attached pdf, and more discussions in the individual response.

We hope our pointwise responses below can clarify all reviewers' confusion and address the raised concerns. We thank all reviewers' efforts and time again.

Best,

Authors

Meta-review: The authors tried to address an important problem in audio visual learning literature using reasonably motivated idea that language present in ambient information of audio and visuals should implicitly help align them for better joint representations. While the idea of semantic-level multi-modal learning is to effectively do this via optimization/regularization in networks, the authors call this out as something that needs to be dealt with explicitly. An important one. The technical novelty is mainly coming from designing such a scheduling system with language assistant/agent in the loop, and showing efficacy on large sets of benchmarks. While the overall performance gains are reasonable, the reviewers do highlight aspects to improve in future (possibly in follow-up studies on complex audio-visual non-signal-processing tasks; and exploration of other language models beyond Gemini). 

The authors provide important details about planning audio processing, relevant audio/visual models, query system, dataset choices and relevant audio/visual temporal sync inference steps, general 'technical tricks' etc in the comments to reviewers' questions. To the extent possible, its critical to add these into the main content of the paper.
Decision: Accept (poster)