AlignSep: Temporally-Aligned Video-Queried Sound Separation with Flow Matching
Authors: Xize Cheng
Chenyuhao Wen
Slytherin Wang
Yongqi Wang
Zehan Wang
Rongjie Huang
Tao Jin
Zhou Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: sound separation
audio-visual alignment
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Video Query Sound Separation (VQSS) aims to isolate target sounds conditioned on visual queries while suppressing off-screen interference—a task central to audiovisual understanding. However, existing methods often fail under conditions of homogeneous interference and overlapping soundtracks, due to limited temporal modeling and weak audiovisual alignment.
We propose \textbf{AlignSep}, the first generative VQSS model based on flow matching, designed to address common issues such as spectral holes and incomplete separation. To better capture cross-modal correspondence, we introduce a series of temporal consistency mechanisms that guide the vector field estimator toward learning robust audiovisual alignment, enabling accurate and resilient separation in complex scenes.
As a \textit{multi-conditioned generation} task, VQSS presents unique challenges that differ fundamentally from traditional flow matching setups. We provide an in-depth analysis of these differences and their implications for generative modeling. To systematically evaluate performance under realistic and difficult conditions, we further construct \textbf{VGGSound-Hard}, a challenging benchmark composed entirely of separation cases with homogeneous interference and strong reliance on temporal visual cues.
Extensive experiments across multiple benchmarks demonstrate that AlignSep achieves state-of-the-art performance both quantitatively and perceptually, validating its practical value for real-world applications. More results and audio examples are available at: \url{https://AlignSep.github.io}.

Published as a conference paper at ICLR 2026
ALIGNSEP: REMPORALLY-ALICN VIDEO-QUERIED SOUND SEPARATION WITH FLOW MATCHING
Xize Cheng; Chenyuhao Wen", Tianhao Wang', Yongqi Wang
Zehan Wang, Rongjie Huang. Tao Jin, Zhou Zhao
Zhejiang University', Independent Author2
ABSTRACT
Video Query Sound Separation (VQSS) aims to isolate target sounds conditioned on visual queries while suppressing off-screen interference- a task central au- diovisual understanding. However, existing methods often fail under conditions homogeneous interference and overlapping soundtracks, due to limited tempo- ral modeling and weak audiovisual alignment. We propose AlignSep, the first generative VQSS model based on flow matching, designed to address common issues such spectral holes and incomplete separation. To better capture cross- modal correspondence, we introduce series of temporal consistency mechanisms that guide the vector field estimator toward learning robust audiovisual align- ment, enabling accurate and resilient separation in complex scenes. As a multi- conditioned generation task, VQSS presents unique challenges that differ funda- mentally from traditional flow matching setups. We provide an in in-depth analysis of these differences and their implications for generative modeling. To system- atically evaluate performance under realistic and difficult conditions, we further construct VGGSound Hard, a challenging benchmark composed entirely of sep- aration cases with homogencous interference and strong reliance temporal vi- sual cues. Extensive experiments across multiple benchmarks demonstrate that AlignSep achieves state-of-the performance both quantitatively and perceptu- ally, validating its practical value for real-world applications. More results and audio examples are available at: https: :ttt::://Aliggsee github.
INTRODUCTION
Video-Queried Sound Separation (VQSS) (Tzinis et al. 2022 Chen et al. 2023 Dong et al. 2022 Cheng et al., (024)) aims to isolate sound sources that correspond visual content in video while suppressing off off-screen interference. As a core task in audio-visual understanding, VQSS facilitates applications such as video editing, accessibility enhancement, and content analysis. Early work in sound separation (Wang & Chen 2018: Pegg et al. 2023) focused primarily on speech, address- ing the classic cocktail purty problem separatin clean speech from noisy environments. While traditional unsupervised methods (Roweis) 2000; Kristjansson et al. 2006 Hershey al. (016] ad- vanced speech separation, they often struggled with interference from sources with similar timbres. To overcome this, researchers introduced visual speech enhancement (Gao & Grauman 2021 Hsu
et al. 2023 Lei et al. 2024), enabling more accurate separation in visually grounded contexts.
As the field evolved, research expanded beyond speech to more diverse sound sources in complex real-world environments (Kavalerov et al. 2019 Liu et al., (023b). including instrument separa- tion (Défossez al., 2019; Luo & Yu, 2023) and sereral-purpose audio disentanglement (Wisdom let al. 2020). To increase flexibility, multimodal query approaches emerged (Liu et al., 20224 Ma al. 2024), using text to describe target sounds. However, text descriptions often fail to fully producing objects and events. This limitation has led to resurgence of interest in video- video-guided sound separation, where visual information provides richer and more precise grounding for identifying target audio sources.
capture the nuances of real-world
Despite these advances, current sideo-guided sound separation methods still face two key limitations (Figure I): (1) Lack of Temporal Modeling, Dong et al. (2022): Cheng et al. (2024) rely heavily on
"Equal Contribution

---

Published as a conference paper at ICLR 2026
On-Screen
Off-Screen
Semantically-Allinee vQss
X
Mixed Audio
Temporally-Aligned VQSS (ours)
Figure 1: Comparison of different video-queried sound separation methods. Traditional category-based methods rely solely semantic information, struggling to distinguish same-sourge sounds across on/off-screen regions (e.g., multiple dogs barking). In contrast, our proposed AlignSep can capture audiovisual consistency, enabling precise separation of on/off-screen sounds.
semantic cues, which are insufficient for separating acoustically similar sources. For instance, distin- guishing multiple barking dogs across on-screen and off-screen regions requires capturing temporal alignment between visual motions and corresponding audio energy, not just semantic categories. (2) Limitations of Mask-Based Methods. The dominant separation paradigm uses time-frequency masking (Liu et al.) 2023b Chen et al. 2023). which struggles with overlapping soundtracks. In scenarios where multiple sources overlap both time and frequency, mask-based methods fail to recover clean and distinct signals (Yuan et al., 2024), leading to incomplete separation or artifacts.
To address the aforementioned challenges, we propose AlignSep the first generative temporal- aligned video-queried sound separation model based on flow matching (Lipman et al. (022) de- signed for robust audiovisual separation. Unlike prior generative separation models Yuaan et al.] 2024] Wang et al., 2024a), VQSS requires maintaining precise temporal alignment between audio and visual streams. To this end, we design a dedicated vector field estimator that explicitly pre- serves audio-visual alignment by employing simple yet effective temporal concatenation strategy, combined with a cross- wer---atttttoofreeeeee feed-forward Transformer encoder to enforce temporal consis- tency across modalities. Furthermore, unlike traditional flow-mat Now-matching tasks which are typically single-conditioned (e.g., text-to-audio (Huang et al., 2023 Liu et al.) 2023a). text-to-image (Zhang jet al., (203) and video-to-audie (Wang al.) (024b)) VQSS is multi-conditioned generation task, where both the raw noisy audio and the video sequence jointly condition the output. This fun- damental difference leads to new challenges in vector field learning and transport path modeling. We provide an in-depth analysis of these differences, and explain why straightforward rectification and deterministic acceleration techniques (e.g.. in rectified flow) are suboptimal in this setting. We further discuss the trade-offs between performance and efficiency specific to sound sepa- ration tasks under these constraints. facilitate rigorous evaluation of VQSS models under realistic and difficult conditions, we construct VGGSound-Hard separation cases involving homogencous interference and strong reliance on fine-grained temporal visual cues. Experimental results multiple benchmarks-including including MUSIC-Clean (Dong et al. 2022). VGGSound-Cless (Dong et al. 2022), and the proposed VGGSound- demonstrate the effectiveness of AlignSep. highlighting its potential guided sound separation. Our main contributions are as follows:
novel benchmark composed entirely of
next-generation framework for video-
We revisit the task of video- video-queried sound separation (VQSS) and provide detailed analysis of its unique challenges, including homogeneous interference, overlapping soundtracks, and the need for precise audio-visual temporal alignment.
We propose AlignSep. novel generative temporal-aligner VQSS framework based condi- tional flow matching, designed to robustly model multi-conditioned generation by leveraging temporal visual cues and preserving cross-modal consistency.
We introduce VGGSound-Hann Hard, a new benchmark specifically curated to evaluate temporal alignment under real-world homogeneous interference, consisting of co-occurring on-foff-screen same-category sound sources.
VGGSound-Clean, and VGGSound-Hil demonstrate that AlignSep achieves state-of-the-art performance in both quantitative metrics and human perceptual scores (e.g., MOS), validating its effectiveness in real-world audiovisual separation tasks.
Extensive experiments on three
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
2.1 UNIVERSAL SOUND SEPARATION
Universal sound separation (Kavalerov et al.; 2019 Liu et al.] 2022 Pons et al., 2024) aims to extract distinct audio tracks from mixed signals, serving as fundamental task for audio under- standing. Early research primarily focused on domain-specifee separation, particularly in speech (Li
et al. 2023 Pegg et al., 2023, Wang et al.) 2023 Li et al. 2022) and music (Défossez et al.) [ Manilow et al. 2022 Rouard et al. 2023] Luo & Yu 2023) separation. Subsequent work (Kavalerov
at. (019)) introduced Permutation Invariant Training Yu al., 2017) separate mixed audio into multiple unidentified categories. However, these methods are largely limited to music, speech, and artificial sounds, struggling with complex real-world auditory scenes. To address this, researchers
have developed large-scale annotated audio datasets (Gemmeke et al. 2017 Chen et al., 2020).
advancing universal sound separation. Recent approaches (Ochiai et al. 2020 Kong et 2020
Liu et al. 2022) leverage class labels as queries for targeted sound separation. Nevertheless, tex- tual descriptions inherently lack the capacity to fully characterize real-world auditory information, prompting exploration of visual-queried sound sepration. We revisit the task of video-queried sound separation (VQSS) and provide a detailed analysis of its unique challenges, including homogeneous interference, overlapping soundtracks, and the need for precise audio-visual temporal alignment.
2.2
VISUAL-QUERIED SOUND SEPARATION
In the field of visual-queried sound separation, AudioScope 1&2 (Tzinis et al. 2020 2022) intro- duce a novel joint audio-visual classifier identify object categories that appear the video and produce the corresponding sounds in the audio track. Building upon this foundation, i-Query (Chen) (ct al, 2023) employs an advanced cross-attention mechanism to detect sound-emitting objects within video sequences, though it relies on pre-extracted object bounding boxes input. Recent advance- ments (Dong et al. 2022 Cheng et al. 2024) leveraged powerful visual pre-trained models to ex- tract rich semantic information from visual content. However, current VQSS methods (Tzinis al.) 2022 Chen et al.] 2023 Dong et al. 2022 Cheng et al., (024] focus heavily on spatial features but neglect audio-visual temporal alignment, leading ambiguity in distinguishing sounds from on-screen versus off-screen objects of the same category. To address this challenge, we introduce VGGSound-Harre new benchmark specifically curated to evaluate temporal alignment under real- world homogeneous interference, consisting of co-occurring on-/off-screen same-category sound sources.
2.3 GENERATIVE SOUND SEPARATION
Traditional sound separation methods (Dong et al. 2022 Cheng al. 2024) often struggle with overlapping sound events, as mask-based discriminative models may produce spectral holes (Wang al. 2022). limiting their effectiveness in complex acoustic environments. To address this, re- searchers have turned to non- con-iscriminative models, initially leveraging Generative Adversarial Net- works (GANs) (Chen et al. 2024) to improve perceptual quality. Subsequent work has explored diffusion models (Hai et al. 2024 and flow-matching (Yuan et al. (024) for more natural and re- fined separation. Despite these advancements, generative risual-querce sound separation remains unexplored. Inspired by
generation (Wang et al. (025). propose AlignSep, novel generative VQSS framework based on conditional flow matching, designed to robustly model multi-conditioned generation by leveraging temporal visual cues and preserving cross-modal con- sistency.
3
ALIGNSEP
3.1 OVERVIEW
Let a mixed audio signal ATM A {Af.- ... AAA) and an interfering audio source A' [A].... AG) ). Video- Quee-Queried Sound Separation (VQSS) aims to extract a clean audio signal A that maintains strict temporal alignment with the corresponding visual frame sequence V (V,,... ... Vz), where and l denote the number
"AT..... .. be a linear superposition clean audio source
3

---

Published as a conference paper at ICLR 2026
Goussion Noise
000000 Xo
Audio VAE Encoder
Noibe Audio
Interpolation
V
at Fo
t= 1=
t=1
Mixed Audio
Audio VAE Encoder
000000 Xm
مم0000 4
A
X
Audio
Concat
ODE Solver ee, O)de
Audio VAE Decoder
Length Regulation
Visual Encoder
000000
Vocoder
Separsted Audio
Figure 2: Ilustration of AlignSep. AlignSep is a video-queried sound separation model based on flow-matching, designed to establish a mapping from the distribution mixed audio and the distri- bution of separated audio, conditioned on visual information. Given mixed audio input, we first perturb it with Gaussian noise and then progressively denoises it, guided by the visual condition This process gradually transforms the mixed noisy audio into separated audio that is temporally aligned with the driving video.
of audio frames and video frames, respectively. As illustrated in Figure we introduce AlignSep, a conditional flow-matching framework designed for precisely temporal-aligned video-queried sound separation. In Section 3.2, we provide a detailed introduction of the now-matching-basee sound sep- aration model. Section 3.3 3.3 elabboraes on the architectural details of AlignSep. while Sections 3,4 and 3.5 present specialized training strategies aimed at enhancing cross-modal alignment capabilities.
3.2 FLOW-MATCHING BASED SOUND SEPARATION
Conditional sound separation tasks, such TQSS (Luu et 2022 and IQSS (Dong et al. 2022
can be formulated as a conditional mapping from the distribution of mixed audio, Km ~ Pm x ). to the distribution of clean audio, Te Pe()).. This process can be interpreted probability density transformation (i.e., flow), governed by the following ordinary differential equa- tion (ODE):
time-dependent
dæ u( e)dd E [0
(1)
where represents the time position, x a point in the probability density space time t, u denotes the transport vector field (i.e., the probability gradient with respect to t). and e is the conditioning embedding. In Video-Queried Sound Separation (VQSS), the conditioning signal e corresponds to visual features extracted from video frames. The source distribution om and target distribution a. represent the compressed mel-spectrogram latents of the mixed audio A and clean audio AS, respectively, obtained via pre-trained VAE encoder (Liu et al. 2023a).
Conditional Flow Matching Model. The fundamental principle of flow-matching generative model is to train a neural network 0 approximate the transport vector field with the flow-matching objective:
(0) = (1)) (,; e; 0) u(x, [.)??
where Pe (x( denotes the distribution of x at timestep t. However, in practice, directly computing uxx t, e) is intractable due to the lack of explicit knowledge of the target distribution (x))), as well as the unknown forms of p:(x) and u. To circumvent this challenge, we adopt the conditional flow- matching (CFM) objective (Lipman et al.) 2022), designing specific probabilistic paths that enable efficient sampling from pe (x: xx.)) and facilitate the computation of u(æ e, a.):
(0) = (x..p.((..)..(ee....) (
(2, L, e) - w( L, x.. e)
(3)
ODE Solving and Audio Reconstruction. Once the latent field estimator is trained, we use numerical solvers to approximate the solution of the ODE dæ = vax t, e; 6) at discretized time steps. A simple yet widely used solver is the Fuler method, which updates xr iteratively as follows:
Ft+t
ev( 1, 0)
where is the step size. Finally, the sampled latent representation is passed through the VAE decoder to reconstruct the mel-spectrogram, followed by a vocoder (Lee et al. 2022) to generate the final audio waveform.
4

---

Published as a conference paper at ICLR 2026
3.3 DETAILED MODEL ARCHITECTURE
Audio Encoder & Visual Encoder. To achieve effective temporal encoding of video sequences, we employ the pre-trained temporal visual encoder (CAVP) from prior V2A work (Luo et al. 2023). Distinguished from the global-level representation like ImageBind (Han et al. 2023) that primarily focus on semantic content representation, CAVP incorporates temporal synchronization supervision between video and audio modalities. This enables the encoder to capture dynamic temporal cor- relations across video frames, rather than relying solely on static semantic features. As for audio modality, to align the distributions of mixed and clean audio as closely as possible, we use pre- trained VAE audio encoder (Liu et al., (222a) to map both into a shared audio latent space. During inference, the paired VAE decoder then directly convert the latent features back into mel spec- trograms.
Temporally-Alignee Vector Field Estimator. As the component of our model, the vector field estimator employs a feedforward Transformer architecture. Given the critical importance of temporal alignment, adopt concatenation- based approach effectively fuse multimodal fea- tures. Specifically, after extracting video features e, which has dimension of 512, via CAVP and latent-space audio features Zm, which has dimension of 20, via VAE, we first expand the video fea- tures to match the time dimension of the audio features, ensuring precise temporal correspondence. These aligned features are then concatenated, followed by the timestep-encoded vector appended at the end of the sequence. This structured input is subsequently fed into the main model for noise prediction, allowing it to leverage temporally coherent information for improved performance.
3.4 CLASSIFIER SEEEHEER-FREEE FREE GUIDANCE
For our generative sound separation task. it can be considered variant of video-to-audio task. Therefore, we employ blassffier-free guidance. This method effectively combines conditional and unconditional output, allowing us strike balance between quality and diversity. The sampling procedure for our audio generation with classifier-free guidance can be formulated as follows:
0(r, t, 0) = s- v(z, t, 0) + (1 s)) 1..-v(xr,.0 t, 0)
(5)
Here, s > 1 represents the classifier sampling scale, which adjusts the balance between the diversity and quality of the generated samples. The ODE solver conditioned with @ is realized by randomly dropping the latent variable and replacing it with a "null" embedded representation. This latent input exchange facilitates the sampling procedure for video-to-audio generation, as we have estab- lished modality-aligned latent space. We set 8 to 4.5 in our experiments.
4 VGGSOUND-HARD: A BENCHMARK FOR SOUND SEPARATION UNDER HOMOGENEOUS REAL- WORLD INTERFERENCE
4.1 HOMOGENEOUS AUDIO PAIR CONSTRUCTION
All samples in VGGSound Hard are sourced from the VGGSound test set. To construct mixtures with homogeneous real-world interference, we first group audio clips by their category labels. Within each category, we compute pairwise cosine similarity using the CLAP audio encoder and select the nighest-scoring pairs, yielding approximately 2,000 candidate homogeneous audio pairs. Following the synthesis pipeline of CLIPSep (Dong et al. 2022), each same-category pair is then mixed produce preliminary set of homogeneous mixed-audio samples.
4.2 HUMAN VERIFICATION
To ensure that the final data reliably supports the evaluation of temporal alignment in sound sep- aration, we perform an additional stage manual verification. A trained audio-visual annotator reviews all candidate samples and applies the following criteria: (1) Clear temporal cues in the video: The visual channel must present actions with identifiable rhythmic or temporal structure, allowing annotators to infer the timing of corresponding sound events (e.g., excluding horn sounds without visible motion cues). (2) On-screen sound sources: All target sound events must be phys- ically present in the associated video frames, ensuring that the model is not required separate
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of visually-queried sound separation performance on MUSIC-Clean, VGGSound-Clean and YGGGouund-Hard (VG-Hard). The evaluation considers semantic consis- tency between audio-audio (S semantic consistency between audio-visual (S and tem- poral consistency between audio-visual (T^-v) to assess the quality of the separated results. Since Davis originally trained on different datasets, we retrained their models on the same dataset to ensure fair comparison.
VGGSOUND-Clean
Music-Clean
VS-Hard
Temp. Align
Method
SA-At
Sa-v
Ta-v
SA-At
SA-V
Ta-v
TA-v
Target Audio
X
100.00
39.33
95.83
100.00
37.10
82.22
94.07
Mixed Audio
x
63.20
19.71
61,46
52.96
15.18
28.89
73.73
CLIPSEP Dong al.
2022
x 66.74
24.21
79.17
60.59
21.42
51.11
85,59
Quury Chen et al
2023
x 68.14
26.93
80.78
66.29
24.46
64.21
79.52
OmniSep Cheng
2024
70.83 x
27.57
81.25
67.67
25.74
68.89
76.27
DDvis ( Uuang et al
2024
x
62.53
23.18
74.39
61.32
22.49
56.47
74.13
DDvis- Hlow [Hlang
2025
x
65,82
24.21
82.32
69.21
27.76
65.71
76.27
AlignSep (ours)
73.38
27.89
96.88
72.28
28.92
66.67
95.76
Table 2: Mean Opinion Score (MOS) across four evaluation dimensions: Noise Residuals (NR), Audio-Visual Consistency (AVC), Audio Quality (AQ), and Overall Score (OS).
VGGSound-Clles
Music-Clean
VGGSound-Harde
Method
NR
AVC
AQ
OA
NR
AVC
AQ
OA
NR
AVC
AQ
OA
ClipSep
3.31
3.31
3.31
3.85
2.91
3.91
3.82
3.55
3.57
4.36
4.29
4.14
OmniSep
3.62
3.69
3.85
3.62
4.09
4.19
3.82
4.01
3.29
4.29
4.21
4.07
AlignSep
4.23
4.53
4.08
4.31
3.82
4.27
4.18
4.18
4.21
4.64
4.21
4.43
sounds originating off-screen. After this filtering process, we obtain 118 high-quality audio-visual pairs that exhibit strong semantic homogeneity yet clearly distinct temporal patterns, forming the final VGGSound benchmark.
5
EXPERIMENTS
5.1 BENCHMARKING HIDRO-QUERIED SOUND SEPARATION
Video-queried sound separation (VQSS) poses unique challenges and requires dedicated evalua- tion methodologies. Unlike conventional source separation, which primarily focuses on recovering semantically correct signals, VQSS additionally demands that the separated audio be temporally aligned with the visual stream. For instance, even if separated sound is semantically consistent with the scene, it should not be preserved if it is not actually produced by the objects in the video. Existing benchmarks fail capture this dual requirement. To address this limitation, we establish new benchmark that introduces tailored evaluation metrics and dataset settings for VQSS.
Evaluation Metrics. Conventional source separation is usually assessed with reconstruction- based metrics such SDR (Vincent al., 2006). which correlate poorly with human percep- tion (Le Roux et al. 2019: Cartwright et al. 2018). The gap is even larger for generative meth- ods (Hai et al.] [2024] Yuan al. 2024), where minor waveform deviations can lead large but perceptually irrelevant emror.. We therefore adopt comprehensive evaluation protocol covering two dimensions: (1) Semantic Alignment: CLAP for audio-audio (A-A) and ImageBind for au- dio-visual (A-V) consistency; (2) Temporal Synchronization: alignment accuracy (Acc) (Luo al.
2023). following Video-to-Audin evaluation (Xing et al. 2024 Wang et al. 2025).
To bridge the gap between objective scores and human perception, we further conduct Mean Opinion Score (MOS) evaluations, where human raters assess four aspects the generated outputs: Noise Residuals (NR), Audio-Visual Consistency (AVC), Audio Quality (AQ), and Overall Score (OS).

---

Published as a conference paper at ICLR 2026
MOS provides complementary insights into perceptual quality and cross- coherence, ensuring a more faithful assessment of real-world performance. The detailed evaluation protocol and rating procedure are provided in the Appendix B
Datasets. As an early effort in VQSS, Dong et al. (2022) curated VGGSOUND-CTean and MUSIC-Clean, where target and interference sounds are drawn from different categories, represent- ing relatively simple separation cases. To better reflect real-world scenarios-whet sounds often come from the same class and are harder to disentangle- wwe introduce VGGSOUND- Hard, a new benchmark in which both target and interference sounds are from the same category and the target audio must be highly temporally aligned with the visual stream. This provides more realistic and challenging testbed for advancing VQSS research.
overlapping
5.2 IMPLEMENTATION DETAILS
Following recent video-to-audio (V2A) works (Luo al. 2023 Wang et al. 2025), we downsam- pled all audio signals into 16 kHz and converted them to Mel-spectrograms with 80 frequency bins and a hop size of 256. All videos were downsampled to 4 FPS. All data samples were truncated into 8-second clips for both training and inference. The transformer architecture in our vector field estimator comprises 4 layers with a hidden dimension of 576. For waveform synthesis, we used a pretrained BigVGAN vocoder (Lee al. 2022). Additional limplementation details are provided in Appendix(A
5.3
MAIN RESULTS
Table presents the quantitative results of visual-queried sound separation on MUSIC-Clean, VGGSSunddClean, and the more challenging VGGSound-Hard benchmark. Compared with the baselines, CLIPSep (Dong et al., 2022) and OmniSep (Cheng et al. 2024), our proposed AlignSep achieves consistent and notable improvements across all datasets. On MUSIC-Clean and VGGSound-Clen -Cean,, AlignSep yields higher scores in both semantic consistency (SA-A. SA-v) and temporal alignment (TA-v). demonstrating its capability to leverage both semantic and tempo- ral cues. Particularly on the VGGSOUND-CIen -Clean benchmark, AlignSep reaches 96.88% in TA-V, significantly surpassing the baselines, while on VGGSound-Hard it maintains strong 95.76% de- spite the increased difficulty. In contrast, while OmniSep improves semantic consistency compared with CLIPSep, its performance on VGGSound-Haad drops 76.27% TA-V. suggesting limited capacity in modeling temporal relationships under complex acoustic-visual mixtures. These results collectively demonstrate that AlignSep not only strengthens semantic aligmment but also effectively captures fine- fine-grained temporal correspondence, leading to superior sound separation quality across both clean and challenging benchmarks.
To further assess perceptual quality, Table reports Mean Opinion Scores (MOS) across four human evaluation dimensions: Noise Residuals (NR), Audio-Visual Visual Consistency (AVC), Audio Quality (AQ), and Overall Score (OA). AlignSep consistently outperforms both CLIPSep and OmniSep across all datasets and evaluation aspects. On Music-Clean, it achieves the highest ratings in every dimension (e.g., 4.53 AVC, 4.31 OA), highlighting its effectiivness in producing clean, temporally aligned, and perceptually natural audio. On VGGSOUND-Cless NR (4.09). AlignSep leads in AVC (4.27), AQ (4.18), and OA (4.18), suggesting better perceived audio-visual coherence. For the most challenging VGGSound-Harrd set, AlignSep again delivers the best overall score (4.43), showing robustness even under complex and ambiguous conditions. Together, these subjective results corroborate the quantitative gains, indicating that AlignSep not only improves alignment metrics but also translates into better human-perceived separation quality.
while OmniSep scores highest in
5.4 TEMPORALLY-ALIGNT VIDEO-QUERIED SOUND SEPARATION
To compare the temporal alignment ability of different VQSS methods, we evaluate their perfor- mance when varying the number of reference video frames used as queries. For OmniSep and CLIPSep. which do not inherently model temporal dynamics, adapt them by segmenting the mixed audio according to each frame's 's semantic information, thereby enforcing temporal alignment. Figure 3 illustrates the impact of reference frame rate (FPS) on alignment accuracy across different
7

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation of different denoising steps on VGGSound-Clean, MUSIC-VGGSound, and VGGSound- We report ImageBind similarity, CLAPScore, and alignment accuracy (Alig- nAcc). The last two columns present inference time and corresponding throughput (FPS). Best results are highlighted in bold.
VGGSOUND-Clean
Music-Clean
VS-Hard
Method
FPSt
SA-V
SA-At f Tu-v
SA-V
SA-A f TA-v
TA-v
5)
25.11
64.47
85.42
22.13
62.17
44.44
88.14
5.56
SlinnSpp(Ste==10)
26.66
69.86
92.71
24.92
68.51
53.33
94.07
4.00
AlignSep(Step=25)
27.89
73.38
96.88
28.92
72.28
66.67
95.76
2.17
5tep=50))
28.12
73.50
95.83
30.70
72.13
68.89
93.22
1.35
AlignSep(Step=1000
2756
73.64
96.88
30.88
72.80
68.89
93.22
0.72
OmniSep
27.57
70.83
81.25
25.74
67.67
68.89
76.27
11.2
AlignSep Rectified Flow
step=100 100
21.39
57.36
84.38
21.41
49.76
46.67
92.37
0.77
methods. AlignSep shows clear upward trend: as the FPS increases, its performance steadily im- proves and saturates at higher frame rates, rising from about 0.76 at 0,25 FPS to nearly 0.95 at FPS. These results demonstrate that AlignSep effec- tively leverages fine-grained temporal cues to enhance separation quality.
AlignAcc
0.95
In contrast, ClipSep shows an almost flat curve around 0.81 across all frame rates. These image-based methods rely solely on semantic modeling and lack temporal information, re- sulting in weak dependency on visual tempo- ral resolution and a clear limitation in exploit- ing alignment signals. OmniSep, while bene- fiting from enhanced semantic alignment that improves its performance on simpler datasets such as VGGSound-Cleen
0.50
0.50
Deniees
Clpdeo
0.70
0.25
struggles on more challenging benchmarks like VGGSound-Hard that demand temporal modeling. Its empha- sis cross-moda semantic alignment further restricts the modeling of temporal correspon- dence, leading even lower separation perfor-
Figure 3: Comparison of sound separation perfor- mance with different levels of temporal informa- tion on VGGSOUND-Hard OmniSep represents the performance when relying solely semantic information. The x-axis indicates the number of video frames per second (FPS) used for VQSS.
mance in such scenarios.
5.5 PERFORMANCE AND EFFICIENCY DISCUSSION IN SOUND SEPARATION
Generative methods meet the requirements of VQSS. Although generative sound separa- tion methods are slightly slower than traditional erate waveforms makes them naturally suited for Video-Queried Sound Separation (VQSS). They effectively address two of the most critical challenges in this task: (1) Disentangling overlapping signals. The objective of VQSS is isolate the target audio source from mixture, guided by visual input. Generative models perform iterative inference with cross-modal conditioning at each step, allowing them to enforce mixture and phase consistency. This iterative refinement gradu- ally routes ambiguous energy to the correct source and reduces leakage. As shown in the results, AlignSep's 's performance improves steadily with more denoising steps on VGGSOUND- audio-visual semantic score (5A-v) increases from 64.47 - 73.38 - 73.64 as the number of steps increases from 5 + 25 + 100, while the temporal alignment score (TA-v) reaches 96.88 at both 25 and 100 steps. In contrast, traditional mask-based methods struggle to separate sources with similar frequency bands and often suffer from severe separation artifacts, leading to degraded perceptual quality. (2) Capturing fine-grained temporal alignment. VQSS depends heavily on frame-level temporal cues, such as lip movements, object collisions, and transitions in and out of the frame. Traditional methods like OmniSep have limited capacity model such fine temporal structure.
approaches, their ability to directly gen-
the
8

---

Published as a conference paper at ICLR 2026
Video(
Video!
Mixture
Off-screen Sounds
Mixture
Target
Target
Noise
Noire
OmniSep
Not Elminatedd (Off-Screer)
OmnniSep
Spectnal Holes
AlignSep
Well-tmmintee
AlignSep
(a) Temporal misalignment issuc.
(b) Spectral holes issue.
Figure 4: Qualitative comparison of VQSS. (a) illustrates a temporal misalignment case, while (b) demonstrates the spectral holes artifact. We highlight the critical regions using different colors.
In comparison, conditional generative models can explicitly incorporate temporal information and refine separation results accordingly across iterations. With schedulable guidance, they can progres- sively enhance temporal aligmment. example, on the VSrHard benchmark, AlignSep achieves high alignment score of TA-V 95.76 with just 25 inference steps.
Traditional Rectified- acceleration struggles in VQSS. While Rectified Flow (RF) achieves fast and deterministic sampling by straightening the generative trajectory into smooth ODE path (Lipman et al. 2022). its effectiveness diminishes in complex multi- multi-condiiionee conditioned settings such VQSS. In VOSS, the model is required condition jointly audio mixture and sequence of video frames v1,7, and to perform time-frequency--ojccc routing, i.e., assigning acoustic energy the correct visual source over time.
This results posterior distribution p(s m, U1.T) that is highly multi-modal and piecewise non- smooth, often exhibiting discrete bifurcations and high-curvature transport paths. Under such struc- ture, single deterministic trajectory- as used in RF-tends to bias toward high-density regions and effectively averages across modes. Moreover, RF lacks the iterative correction mechanism found in diffusion models specifically, the loop of denoising followed by consistency projection. This feed- back loop is essential for correcting early assignment errors in multi-modal generation tasks. As consequence, RF struggles to represent the complex conditional structure in VQSS, often leading to misalignment between generated results and the conditioning inputs. As shown in Table 3 this results in significantly lower performance: RF (with 100 sampling steps) achieves SA-V 57.36, markedly below the 73.64 achieved by the diffusion-based AlignSep model-despine comparable inference speeds under efficient samplers.
VQSS requires fewer sampling steps than other generation tasks. Unlike other conditional generation tasks (e.g.. text-to-audio synthesis), Visual-Query Source Separation (VQSS) benefits from strong conditioning priors: the audio mixture already contains much of the target source, and the accompanying video provides frame-level constraints. As result, many inference steps are unnecessary to achieve high-quality separation. In our experiments, using only 25 sampling steps yields an excellent trade-off between quality and VGGSOUND-Cless and 95.76 VS-Hard, while running at 2.17 FPS (approximately 3x faster than the 100-step setting. which runs at 0.72 FPS). For more real-time scenarios, even 10 steps suffice maintain strong alignment (c.g.. TA-V 92.71 VS-Hard), achieving 4.00 FPS. Beyond 25 steps. the gains in quality are marginal, while throughput degrades significantly (e.g.. 1.35 FPS at 50 steps, 0.72 FPS 100). We conclude that 25 steps strike
TA-V 96.88 on
VGGSOUND-Clead and 94.07 on

---

Published as a conference paper at ICLR 2026
practical balance between separation quality and inference efficiency, making it strong candidate for deployment in real-world applications.
5.6 QUALITATIVE RESULTS
To provide more intuitive illustration of AlignSep's separation capability, we present qualitative comparisons with different VQSS methods in Figure _ As shown in Figure 4a| when temporal cues between audio and video misaligned, conventional fail to suppress irrelevant sounds. For example, although the drumming action in the video has al- ready stopped, OmniSep still produces drum sounds (red regions). In contrast, AlignSep effectively leverages cross-modal correspondence and avoids generating spurious signals (green regions), sep- arating audio strictly in accordance with the drumming rhythm. In Figure 4bb we highlight the spectral holes artifact, a common issue in mask-based separation methods when handling overlap- ping signals. While traditional approaches struggle to preserve continuity under such conditions (red regions), AlignSep mitigates these artifacts through its generative framework, yielding more complete and natural separation results. Additional qualitative examples are provided on our project page htttps://Aligneee github.
intaantics-based separation methods often
6 CONCLUSION
In this work, we introduced AlignSep. the first generative framework for Video Query Sound Sepa- ration (VQSS) based flow matching. By integrating temporal consistency mechanisms and gen- erative separation paradigm, AlignSep effectively addresses key challenges such as spectral holes, hallucination artifacts, and incomplete alignment. Our in-depth analysis reveals the fundamental differences between traditional single-conditioned flow matching tasks (e.g., text-to-audio) and the multi-conditioned generation setting of VQSS. To facilitate rigorous evaluation, we also constructed a challenging benchmark, VGGSound-Hard, composed of samples with homogeneous interference where temporal grounding is critical for successful separation. Extensive experiments demonstrate that AlignSep achieves state-of- of-the-ar art performance both quantitatively and perceptually across a range of benchmarks, validating its robustness and practical applicability.
ETHICAL CONSIDERATIONS
Our work focuses on the development of generative audiovisual source separation system, which aims to improve perceptual quality and alignment between sound and vision. While this technology has potential applications in video editing, accessibility, and content understanding, we acknowledge potential misuse such as manipulation or deepfake-styl content generation. To mitigate such risks, do not train or release models for identity synthesis cross-modal generation beyond the scope of separation. All training data used in this work (MUSIC, VGGSound, and our VGGSound- Hard subset) are publicly available datasets collected for research purposes. No personally identifiable information (PII) is included, and we ensure compliance with the licenses and usage guidelines of each dataset. Human evaluations (e.g., MOS) were conducted anonymously with informed consent. We encourage responsible usage of this technology and explicitly discourage applications involving surveillance, impersonation, or deceptive audio generation.
REPRODUCIBILITY STATEMENT
All code, pretrained models, and related resources will be publicly released upon paper acceptance under permissive license, to encourage further research and community adoption. Detailed im- plementation settings and training protocols are thoroughly documented in the paper to facilitate reproducibility and independent verification.
REFERENCES
Mark Cartwright, Bryan Pardo, and Gautham J Mysore. Crowdsourced poirwise-comparison for source separation evaluation. 2018 ieee international conference on acoustics, speech and signal processing (icassp), pp. 606-610. IEFF, 2018
10

---

Published as a conference paper at ICLR 2026
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Jiaben Chen, Renrui Zhang, Dongze Lian, Jiaqi Yang. Ziyao Zeng, and Jianbo Shi. iquery: Instru- ments queries for vusial sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14675-14686, 2023.
Ke Chen, Jiaqi Su, and Zeyu Jin. Mdx-gan: Enhancing perceptual quality in multi-class source separation via adversarial training. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 741-745. IEEE, 2024.
Xize Cheng, Siqi Zheng, Zehan Wang, Minghui Fang, Ziang Zhang, Rongjie Huang, Ziyang Ma, Shengpeng Ji, Jialong Zuo, Tao Jin, et al. Omnisep: Unified omni-modality sound separation with query-mixup. arXiv preprint arXiv:2410.21269, 2024.
Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Demucs: Deep extractor for music sources with extra unlabeled data remixed. arXiv preprint (rXiv:1909.00 2019.
Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg- Kirkpatrick. Clipsep: Learning text-queried sound separation with noisy unlabeled videos. arXiv preprint arXiv:2212.07065, 2022.
Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross- modal consistency. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15490-15500. IEEE, 2021.
Jort Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Jiarui Hai, Helin Wang, Dongchao Yang, Karan Thakkar, Najim Dehak, and Mounya Elhilali. Dpm- Ise: A diffusion probabilistic model for target sound extraction. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1196-1200. IEEE, 2024.
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et Imagebind-Ilm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03905 2023.
John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discrim- inative embeddings for segmentation and separation. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 31-35, IEEE, 2016
Wei-Ning Hsu, Tal Remez, Bowen Shi, Jacob Donley, and Yossi Adi. Revise: Self-supervised speech resynthesis with visual input for universal and generalized speech regeneration. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18795- 18805, 2023.
Chao Huang, Susan Liang. Yapeng Tian, Anurag Kumar, and Chenliang Xu. High-quality visually- guided sound separation from diverse categories. In Proceedings of the Asian Conference on Computer Vision, pp. 35-49, 2024.
Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. High-quality sound separation across diverse categories via soually-guides generative modeling. arXiv preprint wwXiv:2509.22063 2025.
Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an andiioo: text-to-audin generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp. 19916-13932 PMLR, 2023.
11

---

Published as a conference paper at ICLR 2026
Hyy Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin Wilson, Jonathan Le Roux, and John R Hershey. Universal sound separation. In 2019 JEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 175-179. IEEE, 2019.
Qiugiang Kong, Yuxuan Wang. Xuchen Song, Yin Cao, Wenwu Wang, and Mark D Plumbley. Source separation with weakly labelled data: An approach to computational auditory scene anal- ysis. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 101-105. IEFF, 2020.
Trausti T Kristjansson, John R Hershey, Peder A Olsen, Steven Rennie, and Ramesh A Gopinath. Super-human multi-talker speech recognition: the ibm 2006 speech separation challenge system. In Interspeech, volume 12, pp. 155. Citeseer, 2006.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked or well done? In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626-630. IEEE, 2019.
Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigygan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206.0466 2022.
Songju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu, Jintao Tan, Runlin Liu, Lingyu Xiong, Tao Jin, Xiandong Li, and Zhou Zhao. Uni-dubbing: Zero-shot speech synthesis from visual articulation. In Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 10082-10099, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18.53v/3/v1 2024.acl-long.. 1ong.543. URLhttpsa ac http://aaaaannllrrrrrrrrr2//22//
543/
Kai Li, Runxuan Yang, and Xiaolin Hu, An efficient encoder-decoder architecture with top-down attention for speech separation. arXiv preprint arXiv:2209. 2022.
Xiang Li, Yiwen Wang, Yifan Sun, Xihong Wu, and Jing Chen. Pgss: pitch-guided speech sepa-
ration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 13130- 13138, 2023.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12500 2023a.
Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei. Jinzheng Zhao, Qiushi Huang. Mark D Plumb- ley, and Wenwu Wang. Separate what you describe: Language-queried audio source separation. arXiv preprint 2rXiv:2203.5514: 2022.
Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D Plumbley, and Wenwu Wang. Separate anything you describe. arXiv preprint arXiv:2308.0510 050037 2023b.
Xubo Liu, Qiugiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark Plumbley, and Wenwu Wang. Separate anything you describe. JEEF/ACM Transactions on Audio, Speech, and Language Processing. 2024.
Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36: 2023.
Yi Luo and Jianwei Yu. Music source separation with band-split mnn JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Hao Ma, Zhiyuan Peng, Mingjie Shao, Ju Liu, Xu Li, and Xixin Wu. Clapsep: trastive pre-trained models for multi-modal query-conditioned target sound extraction. arXiv preprint 4rXiv:2402.17455 2024.
Leveraging con-
12

---

Published as a conference paper at ICLR 2026
Ethan Manilow, Patrick O' Reilly, Prem Seetharaman, and Bryan Pardo. Source separation by steer- ing pretrained music models. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 136. IEEE, 2022.
Tsubasa Ochiai, Mare Delcroix, Yuma Koizumi, Hiroaki Ito, Keisuke Kinoshita, and Shoko Araki. Listen what you want: Neural network-based universal sound selector. arXiv preprint arXiv:2006.05712 12, 2020.
Samuel Pegg. Kai Li, and Xiaolin Hu. Rtfs-net: Recurrent time-frequency modelling for efficient audio-visual speech separation. arXiv preprint mrXiv:2309.7118, 2023.
Jordi Pons, Xiaoyu Liu, Santiago Pascual, and Joan Serrà. Gass: Generalizing audio source sepa- ration with large-scale data. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 546-550. IEEE, 2024.
Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pp. 1-5. IEEF, 2023.
Sam Roweis. One microphone source separation. Advances in neural information processing sys- tems, 13, 2000.
Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on- screen sounds. arXiv preprint arXiv:2011.01145: 2020.
Efthymios Tzinis, Scott Wisdom, Tal Remez, and John R Hershey. Audioscopev2: Audio-visual attention architectures for calibrated open-domain on-screen sound separation. In European Con- ference on Computer Vision, pp. 368-385. Springer, 2022.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE transactions on audio, speech, and language processing, 1464):1462- 1462- 1469, 2006.
DeLiang Wang and Jitong Chen. Supervised speech separation based on deep learning: An overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 1702-1726, Oct 2018. doi: 10.1 10.. 28441159. http:://dkk.oo....r/00010111////
2842159
Helin Wang, Dongchao Yang. Chao Weng, Jianwei Yu, and Yuexian Zou. Improving target sound extraction with timestamp information. arXiv preprint 2004v:2004.0082,, 2022.
Helin Wang, Jiarui Hai, Yen-Ju Lu, Karan Thakkar, Mounya Elhilali, and Najim Dehak, Soloau- dio: Target sound extraction with language-orientn audio diffusion transformer. arXiv preprint arXiv:2409.0844 08425, 2024a.
Xihua Wang. Yuyue Wang, Yihan Wu, Ruihua Song, Xu Tan, Zehua Chen, Hongteng Xu, and Guodong Sui. Tiva: Time-aligned hideo-to-audin generation. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 573-582, 582, 2024b.
Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient video-to-ause audio generation network with rectified flow matching. Advances in Neural Information Processing Systems, 1111122118-121111 2025.
Zhong-Qia Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong- Yeol Kim, and Shinji Watanabe. Tf-gridnet: Making time-requency domain models great again for monaural speaker separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics. Speech and Sig- nal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron Weiss, Kevin Wilson, and John Hershey. Unsupervised sound separation using mixture invariant training. Advances in Neural Information Processing Systems, 33:3846-3857, 2020.
13

---

Published as a conference paper at ICLR 2026
Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open- domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7151-7161, 2024.
Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-indeependent multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 241-245. JEEE, 2017.
Yi Yuan, Xubo Liu, Haohe Liu, Mark D Plumbley, and Wenwu Wang. Flowsep: Language-querier sound separation with rectified flow matching. arXiv preprint arXiv:2409.07614 2024.
Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in generative ai: survey, arXiv preprint arXiv:2303.074 07900, 2023.
A IMPLEMENTATION DETAILS
Table 4: Architecture details of ID VAE for spectrogram compression.
Hyperparameter
1D VAE
Input tensor shape for 8-sec audio
(80,512)
Embedding dimension
20
Channels
224
Channel multiplier
1, 2, 4
Downsample layer position
after block after block (20,256)
Attention layer position
Output tensor shape for 8-see audio
The detailed hyperparameters for the VAE are provided in Table This one-dimensional convolu- tional VAE model (ID VAE) is designed to process 8-second audio clips. For an input tensor with shape (80, 512), the model encodes it into a compact latent representation with an embedding dimension of 20, resulting in an output tensor with shape of (20, 256). Regarding the network architecture, the initial number of channels is set to 224, and channel multipliers of 1, 2, and used to increase the channel depth in subsequent layers. To efficiently learn hierarchy of features, the model applies a downsampling operation after the first convolutional block and integrates an attention mechanism after the third block to better capture important acoustic features.
Table 5: Hyperparameters of the vector field estimator of AlignSep.
Hyperparameter
AlignSep
Layers
4
Hidden dimension
576 8
Attention heads
Conv ID-FFN dimension
2,304
Number of parameters
158.94M
The vector field estimator of AlignSep is given in Table 5 It is designed as a four-layer archi- tecture with a hidden dimension of 576, eight attention heads, and Conv1D-bar 1D-based feed-forward network (FFN) with an intermediate dimension of 2,304, totaling approximately 158.94M parame- ters. Notably, the hidden size chosen to align with the feature representations: the generated audio is represented with 288 dimensions, while both the video and the reference audio are represented with 144 dimensions. This configuration ensures that the model effectively captures and enriches the details of the generated audio by integrating complementary cues from the visual and reference audio modalities. Moreover, we concatenate all information together to achieve better temporal consistency.
14

---

Published as a conference paper at ICLR 2026
B MOS EVALUATION
We conducted Mean Opinion Score (MOS) evaluation to assess the perceptual quality of the gen- erated audio across four dimensions: Noise Residuals, Audio-Visum Consistency, Audio Quality, and Overall Score. We display the rating criteria of MOS in Table 7
Participants: 3 proficient annotator were recruited, with diverse backgrounds to reduce bias. Each participant was compensated for their time.
Samples: For each method and each subset, randomly selected 100 audio clips. Each participant was presented with the same randomized set of samples to ensure consistency.
Scoring Protocol: Listeners rated each sample on 5-point Likert scale (1 = Very Poor, 5 Excellent) according to the four predefined dimensions (see Table 2 Detailed definitions of each dimension were provided beforehand to calibrate annotators understanding.
Randomization: The order of both methods and samples randomized per participant to avoid ordering effects or bias toward specific systems.
Aggregation: For each dimension, raters and samples. Scores are averaged first across raters for each sample, and then across samples obtain the final MOS.
report the mean and standard deviation across all
Table 6: Mean Opinion Score (MOS) Rating Criteria
Dimension
Score Description
Very Noisy:
Strong background noise that significantly affects intelligibil-
ity.
Noise Residuals
2
Noisy: Noticeable noise, but speech or content remains intelligible.
3
Acceptable: Minor noise present, generally tolerable.
4
Clean: Little residual noise; very mild artifacts may exist.
5
Very Clean: Completely free of noise or artifacts.
Inconsistent: Completely misaligned with visual content scene context.
2
Low Consistency: Partially related, but mostly inconsistent.
Audio Visual Consistency
3
Moderate: Generally acceptable but with clear mismatches.
4
Consistent: Mostly aligned with minor inconsistencies.
5
Perfectly Consistent: Fully synchronized and semantically coherent with
visual content.
Very Poor: Severely distorted, broken, unnatural audio.
Poor: Audible artifacts and degraded quality.
Audio Quality
3
Fair: Intelligible hut lacks clarity or sounds slightly artificial.
4
Good: Clear and natural with minor imperfections.
5
Excellent: Highly natural, smooth, and human-like like audio quality.
1
Very Poor: Unacceptable overall experience.
2
Poor: Noticeable flaws that degrade the overall experience.
Overall Score
Fair: Usable but with evident limitations.
4
Good: Overall pleasant and functional.
5
Excellent: High-quality and highly realistic audio experience.
C USE OF LLM.
We used a large language model (LLM) solely for polishing the language of this paper.
D MORE EXPERIMENTS
D. ABLATION STUDY ON GENERATIVE MODEL CHOICE.
As shown in table 7 VGGSound-Clean which demands higher semantic understanding, placing Mow-Matching with diffusion leads to modest performance drop. indicating that Flow- Matching helps improve the model's upper bound. On Hard,, the performance drop of
15

---

Published as a conference paper at ICLR 2026
Table 7: Ablation study on generative model choice.
Method
VS-Clean (SA-A) VS-Clean (S^-v) VS-Clean (7A-v) VS-Hard (7Λ-v)
AlignSep
73.38
27.89
96.88
95.76
Slp (wdo Flow Matchingg
64.12 14.12
24.66
93.37
92.28
AlignSep (w/o CAVP)
69.21
26.76
94.71
76.27
Table 8: Ablation on different temporal fusion strategies.
Method
VS-Clean (SA-A) VS-Clean (SA-v) VS-Clean (T^-v) VS-Hard (7Λ-v)
cross-attention
70.13
27.12
93.29
73.38
concat
73.38
27.89
96.88
95.76
AlignSep without Flow-Matching is minor (3.48), whereas removing CAVP causes significant drop (19.49). This demonstrates that the temporal consistency in AlignSep relies primarily on effective visual temporal understanding rather than just the generative modeling method.
D.2 ABLATION DIFFERENT TEMPORAL FUSION STRATEGIES.
Temporal fusion strategy is key factor in our task. To evaluate its impact, we compared cross- attention with our concat fusion method table Results show that the two perform similarly on VS-clean (semantic-oriented), but on VS-hard-where strict temporal alignment is required- attention fails almost entirely to capture temporal correspondence. This highlights the advantage of concat- based fusion for temporal modeling.
16


Reviews:
[{"review_id": "aKs1hqNzYE", "reviewer": "Reviewer_D4Sm", "summary": "This paper presents a method of video-guided sound separation.  Instead of a traditional regression or detection-based approach that masks out the unwanted sound, this paper developed a generative approach that generates the clean-up sound instead.\n\nThe paper pointed out that current video-guided sound separation methods have two significant limitations.  First, semantic cues alone can't separate acoustically similar sources.  It requires capturing the temporal alignment of video motions and audio soundwaves. Second, simple time–frequency masking can not handle multiple sources overlapping in both time and frequency.  \n\nThe core technical component involves using a flow-matching-based algorithm to regenerate a clean soundtrack conditioned on video features.  \n\nThe algorithm outperforms OmniSep and CLIPSep, two of the leading algorithms.  The paper presents a detailed experimental analysis of the proposed method, including a thorough examination of the performance and efficiency tradeoff of the flow-based generative method.  It also developed a new dataset, VGGSound-Hard, which focuses on overlapping space-time interference where temporal grounding is critical for successful separation.", "questions": "None.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The overall idea is novel.  Treating sound separation as a clean-sound generation process enables this method to surpass the limitations of the current mask-based removal approach.  The integration of video motion cues, including both semantic and lower-level motion, provides the flow-based generation with the essential temporal grounding condition to resolve ambiguity. \n\nThe experimental results are extensive and detailed.  The performance vs efficiency analysis of the flow-based method addressed the practical issue of potential high computation cost.  The new dataset is valuable for future work in this direction.", "weaknesses": "None.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTH7jhWIRe", "reviewer": "Reviewer_qRzN", "summary": "This paper presents AlignSep, a generative framework for video-queried sound separation based on flow matching. The model explicitly addresses the challenges of disentangling temporally-aligned audio sources from mixed inputs, using visual signals as queries. AlignSep incorporates a temporal consistency mechanism within the vector field estimator, enabling robust cross-modal alignment. The work introduces VGGSound-Hard, a challenging benchmark featuring homogeneous interference that requires fine-grained temporal audiovisual correspondence, and demonstrates AlignSep’s superior performance over existing baselines on multiple quantitative, perceptual, and qualitative metrics.", "questions": "1. How do alternative audio-visual fusion mechanisms (e.g., cross-attention, feature-wise modulation) compare to the concatenation approach used here in the temporal vector field estimator? Are there empirical results or reasons for not adopting these?\n\n2. Have you tested the performance of AlignSep in real-world settings where video-audio correspondence is noisy or ambiguous?\n\n3. Given that the classifier-free guidance scale is set to 4.5, could you explain the choice and its impact on sample quality or diversity?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper tackles VQSS in settings requiring not just semantic but precise temporal alignment between video and audio, a scenario where many existing methods falter, particularly under homogeneous interference and overlapping sources.\n\n2. The introduction of VGGSound-Hard could be potentially valuable to the audio-visual separation community by operationalizing more realistic, temporally complex scenarios compared to prior datasets.", "weaknesses": "1. Overclaiming and Missing Related Work: The paper substantially overclaims novelty and omits several key prior works in video-queried sound separation (VQSS).\nAt L95–98, the authors state:\n\n“We revisit the task of video-queried sound separation (VQSS) and provide a detailed analysis of its unique challenges, including homogeneous interference, overlapping soundtracks, and the need for precise audio-visual temporal alignment.”\n\nHowever, these issues were already identified and discussed in \"High-Quality Visually-Guided Sound Separation from Diverse Categories\" [1] (Fig. 1), which explicitly analyzes overlapping-sound challenges and shows that generation-based separation outperforms masking-based methods. Furthermore, at L75–77, the paper claims:\n\n“We propose AlignSep—the first generative video-queried sound separation model based on flow matching (Lipman et al., 2022) designed for robust audiovisual separation.”\n\nThis statement is inaccurate. The generative approach to VQSS was introduced in [1] (2023), and later extended by [2], which directly applies flow-matching for video-based sound separation. These works are closely related in both goal and methodology, but are entirely unacknowledged in the submission.\nWithout discussing or comparing against these prior methods, the claimed novelty of AlignSep remains unclear and potentially misleading.\n\n[1] High-Quality Visually-Guided Sound Separation from Diverse Categories\n[2] High-Quality Sound Separation Across Diverse Categories via Visually-Guided Generative Modeling\n\n2. Incomplete Comparisons in the Main Table\n\nThe main quantitative table should include comparisons with other video-based or multimodal separation models, such as:\n\n[3] Language-Guided Audio-Visual Source Separation via Trimodal Consistency\n[4] iQuery: Instruments as Queries for Audio-Visual Sound Separation\n\nBoth [3] and [4] are directly comparable in setting and should appear in the benchmark to provide a fair evaluation of AlignSep’s relative performance.\n\n3. Missing Architectural Ablations: Although Table 3 examines the number of denoising steps and Table 5 lists hyperparameters, the paper does not analyze architectural design choices. For example, the rationale behind adopting a self-attention-free Transformer or comparing with different audio-visual encoding architectures. Such ablations are important to understand which design decisions truly contribute to performance improvements.\n\nAt present, the paper’s novelty is overstated, and its evaluation is incomplete. The work could become more compelling if the authors (i) properly position AlignSep within the context of prior generative VQSS research, (ii) include stronger baselines for comparison, and (iii) provide architectural ablations to support their design choices.\n\nI am currently leaning toward rejection, but if the rebuttal offers convincing clarification and expanded comparisons, I would reconsider and potentially raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C4Uqc4WvAA", "reviewer": "Reviewer_eay2", "summary": "The paper proposes a flow-matching generative model for video-queried sound separation that explicitly preserves temporal alignment between visual motion and target audio. It uses a temporally synchronized visual encoder, a VAE audio latent space, and a temporally aligned vector-field estimator. The authors also introduce VGGSound-Hard, a benchmark with homogeneous on/off-screen interference requiring fine-grained temporal cues. The paper achieves SOTA semantic and temporal metrics and higher MOS on MUSIC-Clean, VGGSound-Clean, and VGGSound-Hard.", "questions": "- Could the author add standard metrics? For example, report SDR, SIR and SAR results.\n\n- Could the author add evaluations beyond MUSIC/VGGSound (e.g., in the wild audio-visual videos), additional qualitative results are also fine to evaluate the generalizability of the proposed method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies temporal misalignment and mask-method limits for current video query sound separation systems, then introduces the first flow-matching generative model to address them.\n\n- The paper proposes a new hard benchmark. The proposed VGGSound-Hard targets homogeneous on/off-screen interference, stressing temporal grounding beyond prior clean sets.\n\n- The system achieves SOTA semantic and temporal scores and better MOS across MUSIC and VGGSound.", "weaknesses": "- For the architecture design, the “self-attention-free feed-forward Transformer” with simple temporal concatenation may struggle with long-range dependencies or complex multi-object scenes.\n\n- The authors claim that reconstruction-based metrics such as SDR could correlate poorly with human perception. But without standard separation metrics being reported, this will limit comparability to audio-separation literature.\n\n- Experiments are limited to MUSIC/VGGSound variants, broader coverage like more in-the-wild AV datasets would strengthen generalization claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbUd8Z2TDY", "reviewer": "Reviewer_2zVG", "summary": "This paper introduces AlignSep, a generative model for Video-Queried Sound Separation based on conditional flow matching. The primary goal is to address common failures in existing methods, particularly when dealing with acoustically similar (homogeneous) on-screen and off-screen sound sources. The authors argue that prior work relies too heavily on semantic cues and fails to model fine-grained temporal alignment between audio and video. AlignSep is designed to overcome this by using a temporally-aware visual encoder and a generative framework that mitigates issues like spectral holes. To facilitate evaluation, the paper also proposes a new challenging benchmark, VGGSound-Hard, specifically curated with examples of homogeneous interference. Experiments show that AlignSep outperforms existing methods on several benchmarks, especially in metrics related to temporal alignment and perceptual quality.", "questions": "See Weaknesses", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper correctly identifies and tackles a key weakness in existing VQSS methods: the failure to separate homogeneous sound sources due to a lack of temporal modeling. This is an important problem.\n2. The proposed VGGSound-Hard is a strong contribution. By creating a testbed where semantic cues are insufficient, it will enable more rigorous evaluation of temporal alignment capabilities in future VQSS models.\n3. Includes human MOS scores in addition to semantic and temporal metrics.", "weaknesses": "1. My primary concern is the complete absence of SDR and SI-SNR. The authors justify this by stating these metrics \"correlate poorly with human perception\" and are sensitive to minor waveform deviations in generative models. While these are known limitations, it is standard practice in the audio separation community to report them, even with caveats. Omitting them entirely makes it difficult to compare AlignSep to the vast body of literature in source separation and understand its performance in terms of signal reconstruction fidelity. Relying solely on embedding-based scores (CLAP, ImageBind) and MOS provides an incomplete picture. These metrics measure semantic/temporal consistency but not the faithfulness of the separated audio to the original source signal.\n2. Section 3.3 introduces the concatenation-based temporal fusion and \"feedforward Transformer\" without self-attention. However, it remains unclear how much each design choice contributes: 1) What happens if temporal alignment is omitted or replaced with standard cross-attention? 2) Is the observed improvement mainly from temporal encoding or from generative modeling itself? An ablation comparing different temporal fusion strategies (concatenation vs. attention) would be valuable.\n3. The choice of the CAVP visual encoder is highlighted for its ability to capture temporal correlations. A crucial ablation would be to replace CAVP with a more semantic, less temporally-aware encoder (e.g., a standard CLIP image encoder applied per-frame) within the AlignSep framework. This would directly validate the claim that the specific choice of a temporally-supervised visual encoder is responsible for the large gains in the $T_{A-V}$ metric.\n4. The authors state they \"adapt\" OmniSep and CLIPSep by \"segmenting the mixed audio according to each frame's semantic information\". This process is vague and sounds like a heuristic post-processing step. It is unclear if this is a fair or optimal way to allow these models to use frame-rate information. This adaptation could be unfairly disadvantage the baselines, potentially leading to AlignSep's superior performance in Figure 3.\n5. While the new benchmark is a great contribution, the paper provides almost no details on its construction. How were samples selected from VGGSound? What is the size of the test set? What are the specific criteria for \"strong reliance on fine-grained temporal visual cues\"? Without these details, the benchmark is not reproducible, and the community cannot fully adopt it. These details should be included in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iKICIC4aqb", "reviewer": "Reviewer_zvrP", "summary": "In this paper, the authors address the persistent challenge of temporal misalignment in video-queried sound separation, where separated audio often contains sounds from the correct source but with slight temporal shifts relative to the visual cues. To this end, they propose ALIGNSEP, a novel framework that recasts the separation task from a deterministic regression problem into a conditional generative modeling problem. The core of the contribution of this paper is the use of continuous-time flow matching, a technique where the model learns a vector field (a flow) that transforms a simple noise distribution into the complex conditional distribution of the target audio, given the input mixture and the silent video query. By explicitly conditioning the entire generative flow on the video features, the model is forced to learn a tight and continuous temporal coupling between the visual and auditory signals, thereby directly mitigating the alignment issues that can affect traditional masking-based or regression-based approaches. Their experiments show that this flow-based generative method achieves state-of-the-art separation performance while producing audio with significantly improved temporal alignment to the corresponding video.", "questions": "The proposed approach is really interesting and I was considering what could happen if you apply the same ODE solver in the video side by using the Gaussian interpolation at the video embedding frames. Would you expect a similar performance on the video side?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper's primary contribution is its fundamental reframing of the separation problem, shifting away from traditional deterministic methods. Instead of formulating video-queried separation as a regression task, such as estimating a time-frequency mask, the authors reconceptualize it as a conditional generative modeling problem. This is a more powerful and principled approach, as it aims to model the entire conditional probability distribution of the target audio. By using continuous-time flow matching, the model learns to transform a simple noise distribution into this complex target distribution, which it seems from the authors’ experiments that is better suited to handle the ambiguity and variability in audio signals compared to predicting a single point estimate.\n2. A significant strength is that the proposed framework is explicitly designed to solve the specific, well-known issue of temporal misalignment between video and separated audio. The core mechanism which is conditioning the entire generative flow on the video query, seems that forces a continuous and tight temporal coupling between the modalities from the very beginning of the generation process. This represents a more elegant and integrated solution than methods that might require post-processing or auxiliary alignment losses, as it embeds temporal consistency directly into the generative dynamics of the model itself.\n3. Finally, the thorough empirical evaluation successfully demonstrates that the proposed generative approach does not come at the cost of raw separation quality. The paper shows that ALIGNSEP achieves state-of-the-art performance on standard separation metrics while also providing a clear, measurable improvement in temporal alignment. This dual achievement is critical, as it proves that the method is not a narrow solution to a single problem but a more robust overall framework that advances the field on multiple fronts, validating the effectiveness of the flow-matching approach for this complex audio-visual task.", "weaknesses": "Although I do not think that the paper has major flaws, there are some weaknesses and further explorations that the authors can consider to further improve the impact of their paper. I will try to write all these things down with decreasing order of significance.\n\n1. I would like to see the authors analyze more what is happening with the negative examples, meaning what this model is doing when it is presented with only off-screen sounds as analyzed in [A]. Without knowing how robust the model is, I can imagine cases where the ODE solver could break by needing to estimate a zero waveform. One can question the stability of the ODE solver in such a degenerate case; a model trained to map a noise distribution to a complex signal manifold might struggle or even fail when the learned vector field is conditioned to collapse to a single point (the origin). This analysis is in my opinion really important for real-world use-cases cause many parts of the videos do not contain any on-screen sound for long periods of time.\n2. The application of flow matching is central to the paper's contribution, but in my opinion there is a lack of principled justification or empirical comparison against other relevant conditional generative frameworks, such as score-based diffusion models. It is not demonstrated whether the specific choice of continuous-time flow matching provides a unique advantage for temporal alignment over these alternative generative techniques. The paper would be significantly strengthened by an ablation study that compares different generative modeling families, which would help to isolate whether the observed benefits stem from the specific dynamics of flow matching itself or merely from the general shift from a deterministic to a generative formulation.\n\n[A] Tzinis, E., Wisdom, S. and Hershey, J.R., 2022. Don’t Listen to What You Can’t See: The Importance of Negative Examples for Audio-Visual Sound Separation. arXiv preprint arXiv:2209.12934.\n\nI would gladly increase my score if the authors work properly to address the above issues to a proper degree.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)