From Coarse to Fine: Recursive Audio-Visual Semantic Enhancement for Speech Separation

## Abstract

Abstract Audio-visual speech separation aims to isolate each speaker’s clean voice from mixtures by leveraging visual cues such as lip movements and facial features. While visual information provides complementary semantic guidance, existing methods often underexploit its potential by relying on static visual representations. In this paper, we propose CSFNet, a Coarse-to-Separate-Fine Network that introduces a recursive semantic enhancement paradigm for more effective separation. CSFNet operates in two stages: (1) Coarse Separation, where a first-pass estimation reconstructs a coarse audio waveform from the mixture and visual input; and (2) Fine Separation, where the coarse audio is fed back into an audio-visual speech recognition (AVSR) model together with the visual stream. This recursive process produces more discriminative semantic representations, which are then used to extract refined audio. To further exploit these semantics, we design a speaker-aware perceptual fusion block to encode speaker identity across modalities, and a multi-range spectro-temporal separation network to capture both local and global time-frequency patterns. Extensive experiments on three benchmark datasets and two noisy datasets show that CSFNet achieves state-of-the-art (SOTA) performance, with substantial coarse-to-fine improvements, validating the necessity and effectiveness of our recursive semantic enhancement framework.

## 1 Introduction

Speech separation refers to the task of isolating individual speech signals from overlapping audio mixtures, which is a challenge famously exemplified by the “cocktail party problem" [1], where humans demonstrate an innate ability [2, 3, 4] to focus on a specific speaker amidst background noise and competing voices. While this capability comes naturally to human audition, computational approaches to speech separation have long struggled to achieve comparable performance. The advent of deep learning [5] has opened new possibilities for tackling this problem, enabling data-driven systems to learn complex auditory patterns and achieve unprecedented separation accuracy.

Early research in this domain primarily explored audio-only methods, utilizing temporally segmented utterances such as before and after speech overlaps from multiple speakers to train separation models [6, 7, 8, 9, 10]. To address the performance degradation caused by overlapping speech, background noise, and reverberation, subsequent work incorporated speaker enrollment information, namely voice cues from the target speaker, to guide the separation process and improve robustness [11, 12, 13, 14].

To further improve separation performance under challenging acoustic conditions, recent studies have leveraged additional modalities such as textual [15, 16] and visual cues [17, 18, 19, 20]. These approaches introduce dedicated branches for text inputs and visual signals, including lip movements and associated facial appearance features.
Among available auxiliary cues like speaker enrollment, text, and video, visual information provides distinct advantages. It can be captured passively through telephoto cameras without requiring user cooperation, unlike voice or text based methods. This makes visual modality uniquely suited for practical deployment in unconstrained environments.
Consequently, integrating visual information with audio representations traditionally used in audio-only methods has emerged as a promising direction, commonly known as audio-visual speech separation [20, 21, 22, 23].

Figure: Figure 1: Previous vs. our audio-visual separation methods
Refer to caption: x1.png

Incorporating visual modalities brings notable gains in speech separation, mainly by exploiting semantic cues.
Early works [20, 21] used word-classification pretrained models, but the semantics were restricted to a 500-word vocabulary, limiting generalization.
Later studies [22, 23] employed AVSR models (e.g., Deep-AVSR [24], AV-HuBERT [25]) to obtain sentence-level representations.
However, these approaches still rely solely on visual input, as shown in Figure [1](https://arxiv.org/html/2509.22425v2#S1.F1), which is inherently ambiguous (e.g., /p/ vs. /b/) and lacks speaker-discriminative details.
As shown in [26], video-only input yields much weaker representations than audio-visual joint input, leading to less effective semantics for separation, especially in multi-speaker scenarios.
This motivates us to recursively integrate coarse separated audio into the AVSR model together with video, thereby extracting more discriminative and speaker-aware semantic representations.

In this work, we propose the Coarse-to-Separate-Fine Network (CSFNet), a two-stage recursive semantic enhancement framework.
In coarse separation stage, audio mixtures and visual features are fused by a speaker-aware module and passed to a separation backbone for coarse audio estimation.
In fine separation stage, the coarse audio and visual stream are fed into a pretrained AVSR model to obtain enhanced semantics, which are reintegrated to produce refined clean signals.
To fully exploit these semantics, we design a speaker-aware perceptual fusion (SP) block to encode identity across modalities, and a multi-range spectro-temporal separation (MST) backbone to capture both local and global T-F dependencies. Our main contributions are:

- 1.
We propose CSFNet, a coarse-to-fine framework that recursively enhances semantics by feeding coarse audio back into a AVSR model, enabling joint audio-visual semantic extraction.
- 2.
We design a speaker-aware fusion block and a multi-range spectro-temporal network, improving robustness to modality degradation and capturing multi-scale T-F patterns.

Extensive experiments across multiple benchmarks demonstrate that CSFNet achieves new state-of-the-art performance, primarily due to the introduction of the fine separation stage. This stage significantly enhances semantic information extraction, leading to a 10% reduction in word error rate (WER) on the LRS2-2Mix dataset. Moreover, it greatly improves multi-speaker separation and maintains strong robustness even when visual information is partially or completely absent.

## 2 Related Work

Audio-Only Speech Separation. Early research in speech separation primarily focused on classical signal processing methods such as independent component analysis (ICA) and non-negative matrix factorization (NMF). With the advent of deep learning, data-driven approaches have significantly advanced the SOTA. Representative methods include Deep Clustering (DPCL) [5] and Deep Attractor Networks (DANet) [27], which learn discriminative embeddings for speaker assignment in the spectral domain. Subsequently, time-domain methods such as Conv-TasNet [6] and Dual-Path RNN (DPRNN) [7] have further improved separation performance by directly modeling raw waveforms and efficiently capturing long-range temporal dependencies. More recently, Transformer-based architectures like SepFormer [28] have achieved SOTA performance by exploiting global contextual information across both time and frequency dimensions. In addition, complex spectral mapping networks such as TF-GridNet [9] and CrossNet [29] have demonstrated further improvements by explicitly modeling the complex-valued spectrum.

Audio-Visual Speech Separation. In recent years, incorporating visual cues into speech separation has attracted growing attention due to its robustness in noisy and overlapping speech conditions  [30, 16], all of which are inherently using semantic information.
Early studies including VisualVoice [19] fused lip motion, facial appearance, and audio features for effective multimodal separation. Subsequent works introduced attention-based frameworks [17, 31] to better align audio and visual streams over time.
More recent approaches, inspired by the human audio-visual perception system, have integrated multi-scale features [20] or designed hierarchical fusion modules [21] to further enhance separation performance.

A notable trend on exploiting semantic information involves utilizing pretrained audio-visual speech recognition (AVSR) models, such as AV-HuBERT [25] and Deep AVSR [24], which have demonstrated strong performance in speech recognition tasks. To be specific, these AVSR models proposed by  [25] and  [24] have been employed as front-end modules to extract rich sentence-level semantics from lip movements, showing remarkable effectiveness in audio-visual speech separation  [23, 22].
However, despite these advances, a key limitation remains: relying exclusively on visual input may underutilize the semantic and speaker-discriminative information provided by the audio modality.

## 3 CSFNET

Figure: Figure 2: The overall pipeline of CSFNet comprises two stages: coarse separation and fine separation. The coarse audio from the coarse separation stage is leveraged for fine separation, where the fine separation network is finetuned based on the coarse stage to produce refined audio.
Refer to caption: x2.png

### 3.1 Overview

The CSFNet system (Figure [2](https://arxiv.org/html/2509.22425v2#S3.F2)) consists of five main components: an audio encoder, a video encoder (Auto-AVSR), an SP fusion block, an MST separation module, and an audio decoder. Specifically, given a mixture speech signal $y(t)$, together with the corresponding visual information of all involved speakers (i.e., extracted mouth regions, denoted as Mouths), the goal is to leverage this multimodal information to accurately separate and recover each speaker’s voice from the mixture, even under noisy acoustic conditions.
The overall process is divided into two stages. In the coarse separation stage, audio features $Y$ are first extracted from the mixture $y(t)$ using the audio encoder. Meanwhile, a pretrained video encoder (Auto-AVSR) produces lip-motion representations $L$. These two modalities are integrated via the SP Fusion Block to form deeply fused cross-modal representations. The MST separation module is then applied to disentangle latent speaker-specific features, and the audio decoder reconstructs the coarse speech estimates $s(t)$ for each individual speaker.
In the fine separation stage, the coarse estimates $s(t)$, along with the corresponding lip features $M$, are fed into the Auto-AVSR model again to obtain more accurate and discriminative semantic representations $\hat{L}$. These enhanced features are processed once more through the SP Fusion Block, MST separation module, and audio decoder, which are finetuned based on the coarse separation stage, ultimately producing refined, high-quality speech outputs $\hat{s}(t)$.

### 3.2 Detailed Architecture

#### 3.2.1 Audio encoder

The mixed signal $y(t)$ is first transformed into a time-frequency (T-F) representation $Y_{0}\in\mathbb{R}^{2\times T\times F}$ via the Short-Time Fourier Transform (STFT). To extract informative and discriminative features from $Y_{1}$, we employ a multi-scale convolutional encoder that processes the input through four parallel convolutional branches with different receptive fields: a $1\times 1$ standard convolution and three $3\times 3$ convolutions with dilation rates $d=1,2,3$. The fused output is computed as:

$$ ${Y=\operatorname{Concat}\left(\operatorname{Conv}^{1\times 1}(Y_{0}),\operatorname{Conv}^{3\times 3}_{d=1}(Y_{0}),\operatorname{Conv}^{3\times 3}_{d=2}(Y_{0}),\operatorname{Conv}^{3\times 3}_{d=3}(Y_{0})\right)}$ (1) $$

where $\operatorname{Concat}(\cdot)$ denotes channel-wise concatenation, and $\operatorname{Conv}^{k\times k}_{d}(\cdot)$ indicates a 2D convolution with kernel size $k\times k$ and dilation rate $d$. The concatenated feature map is then passed through a Group Normalization layer followed by a PReLU activation function, resulting in an encoded representation $Y\in\mathbb{R}^{C\times T\times F}$.

#### 3.2.2 Video encoder (Auto-AVSR)

We utilize a pretrained Auto-AVSR [26] as the video encoder, which consists of three components. The first is a Visual Speech Recognition (VSR) module that processes lip movement, which serves as the representation $L\in\mathbb{R}^{T_{1}\times C}$ in the first coarse separation stage. The second is an Audio Speech Recognition (ASR) module that processes the audio from the first stage output. The final is a Classic Multilayer Perceptron (MLP), which takes the representation in the first stage and combines it with the audio features output from the ASR module to form $\hat{L}\in\mathbb{R}^{T_{1}\times C}$. For more details, please refer to Appendix [B](https://arxiv.org/html/2509.22425v2#A2).

Figure: Figure 3: The two main components of CSFNet: (a) SP fusion block, (b) MST separation module.
Refer to caption: x3.png

#### 3.2.3 Speaker-wise Perceptual (SP) fusion block

Unlike previous works [32, 31, 33, 10] that rely on complex attention mechanisms, we propose a simple yet efficient fusion module named Speaker-wise Perceptual (SP) Fusion Block. This module is designed to enhance the representation of mixed speech by incorporating both speaker-specific and global contextual information. Given the mixed audio feature $Y$, we perform speaker-wise fusion with the corresponding visual features. For each speaker in the mixture, we obtain a lip movement feature $L_{i}\in\mathbb{R}^{T_{1}\times C}$. Taking a 2-speaker mixture as an example (as illustrated in Figure [3](https://arxiv.org/html/2509.22425v2#S3.F3)-a), to align $L_{i}$ with the $Y$ in both temporal and frequency dimensions, we first apply linear interpolation to upsample the temporal dimension $T_{1}$ to match the number of audio frames $T$. Then, the visual features are expanded along the frequency axis by repeating each time step across all frequency bins, forming $L_{i}\in\mathbb{R}^{C\times T\times F}$. We then fuse $Y$ with each pair of visual features $L_{1},L_{2}$ separately using a simple multilayer perceptron (MLP). This results in three cross-modal feature representations, which are then concatenated together with the original audio feature $Y$. The concatenated feature is subsequently passed through a lightweight convolutional block consisting of a 2D convolution layer, Group Normalization, and a PReLU activation function to obtain the final fused representation $Y_{1}\in\mathbb{R}^{C\times T\times F}$.
This design allows the model to capture both fine-grained speaker-specific cues and global contextual information from the mixture. Moreover, as evidenced by our experiments, the proposed fusion block exhibits strong generalization ability, even when visual inputs from one speakers are missing.

#### 3.2.4 Multi-range Spectro-Temporal (MST) separation module

Our separation backbone builds on TF-GridNet [9], which comprises three modules: an Intra-Frame Spectral (IFS) module, a Sub-Band Temporal (SBT) module, and a Full-Band Self-Attention (FBS) module. However, its use of a single unfold range and a large BLSTM hidden size ($H=256$) limits the ability to capture multi-scale spectro-temporal features and incurs high computational cost. To address this, we propose the Multi-range Spectro-Temporal (MST) module (as shown in Figure [3](https://arxiv.org/html/2509.22425v2#S3.F3)-b).

In MST, both spectral and temporal modules adopt a parallel multi-branch design: (1) a global branch (no unfold), (2) a medium-range branch ($I=4$, $J=1$), and (3) a large-range branch ($I=8$, $J=1$). The outputs of these branches are fused to produce richer multi-scale representations, improving performance in overlapping-speaker scenarios. To enhance efficiency, we reduce the BLSTM hidden size to $H=96$. Further ablation details can be found in Appendix [C](https://arxiv.org/html/2509.22425v2#A3). For the FBS module, we retain the multi-head attention structure from [9] with $N$ heads. In practice, $B$ MST blocks are stacked to form the separation backbone, yielding the final output $Y_{2}\in\mathbb{R}^{C\times T\times F}$.

#### 3.2.5 Audio decoder

As for the decoder, a 2D transposed convolution is employed to transform the separated features into time-frequency (T-F) spectrograms corresponding to each speaker. These spectrograms are then converted into time-domain waveforms through the inverse short-time Fourier transform (iSTFT), yielding the final separated speech signal $\hat{s}(t)$.

## 4 Experiments and Results

**Table 1: Speaker separation results of different AVSS methods on LRS2, LRS3, and VoxCeleb2 datasets. ‘Mod.’ stands for modality (audio-only (A) or audio-visual (AV)). DM indicates dynamic mixing conditions. ^∗ denotes results reproduced by our implementation.**
| Method | Params (M) | Mod. | LRS2 | LRS3 | VoxCeleb2 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SI-SDRi $\uparrow$ | SDRi $\uparrow$ | SI-SDRi $\uparrow$ | SDRi $\uparrow$ | SI-SDRi $\uparrow$ | SDRi $\uparrow$ |  |  |  |
| Unprocessed | – | – | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| DPCL++ [5] | 13.6 | A | 3.3 | 4.3 | 5.8 | 6.2 | 2.1 | 2.5 |
| Conv-TasNet [6] | 5.6 | A | 10.3 | 10.7 | 11.1 | 11.4 | 6.9 | 7.5 |
| SuDoRM-RF [34] | 2.7 | A | 9.1 | 9.5 | 12.1 | 12.3 | 6.5 | 6.9 |
| A-FRCNN [8] | 6.3 | A | 9.4 | 10.1 | 12.5 | 12.8 | 7.8 | 8.2 |
| TF-GridNet [9] | 14.5 | A | 13.9^∗ | 14.0^∗ | 17.3^∗ | 17.4^∗ | 11.2^∗ | 11.5^∗ |
| Crossnet [29] | 6.6 | A | 14.1^∗ | 14.3^∗ | 17.2^∗ | 17.4^∗ | 11.4^∗ | 11.7^∗ |
| CSFNet (Audio-Only) | 10.7 | A | 14.5 | 14.6 | 17.4 | 17.7 | 11.9 | 12.5 |
| The Conversation [17] | 62.7 | AV | – | – | – | – | – | – |
| AVConvTasNet [18] | 16.5 | AV | 12.5 | 12.8 | 11.2 | 11.7 | 9.2 | 9.8 |
| VisualVoice [19] | 77.8 | AV | 11.5 | 11.8 | 9.9 | 10.3 | 9.3 | 10.2 |
| CTCNet [20] | 7.0 | AV | 14.3 | 14.6 | 17.4 | 17.5 | 11.9 | 13.1 |
| AVLiT [35] | 5.8 | AV | 12.8 | 13.1 | 13.5 | 13.6 | 9.4 | 9.9 |
| AVSepChain [23] | 33.1 | AV | 15.3 | 15.7 | – | – | 13.6 | 14.2 |
| IIANet [21] | 3.1 | AV | 16.0 | 16.2 | 18.3 | 18.5 | 13.6 | 14.3 |
| CSFNet (coarse separation) | 10.9 | AV | 16.2 | 16.3 | 18.6 | 18.8 | 14.1 | 14.4 |
| CSFNet (fine separation) | 10.9 | AV | 16.8 | 16.9 | 19.4 | 19.7 | 14.8 | 15.1 |
| AV-CrossNet [22] (DM) | 11.1 | AV | 16.8 | 17.1 | 18.3 | 18.5 | 14.6 | 14.9 |
| CSFNet (DM)(coarse separation) | 10.9 | AV | 16.9 | 17.0 | 17.9 | 18.2 | 14.8 | 14.9 |
| CSFNet (DM)(fine separation) | 10.9 | AV | 17.5 | 17.8 | 20.1 | 20.2 | 15.4 | 15.6 |

**Table 2: Computational complexities of different AVSS methods. All results are measured with an input audio length of 1 second, a sample rate of 16 kHz, and a video frame sample rate of 25 FPS.**
| Method | Pretrain Model | SI-SDRi | Computation Cost | GPU Metrics |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| MACs (G) | Params (M) | Time (ms) | Memory (MB) |  |  |  |
| AVConvTasNet [18] | No | 12.5 | 23.8 | 16.5 | 62.51 | 117.05 |
| VisualVoice [19] | Yes | 11.5 | 9.7 | 77.8 | 110.31 | 313.74 |
| AVLiT [35] | Yes | 12.8 | 18.2 | 5.8 | 62.51 | 24.00 |
| CTCNet [20] | Yes | 14.3 | 167.1 | 7.0 | 84.17 | 75.80 |
| IIANet [21] | Yes | 16.0 | 18.6 | 3.1 | 110.11 | 12.50 |
| AV-CrossNet [22] | Yes | 16.8 | 92.3 | 11.1 | 367.91 | 40.5 |
| CSFNet | Yes | 17.5 | 64.8 | 10.9 | 117.39 | 42.5 |

### 4.1 Datasets

We conducted experiments using a variety of publicly available audiovisual datasets under both clean and noisy conditions. For clean conditions, we used LRS2 [24], LRS3 [36], and VoxCeleb2 [37], from which we created multi-speaker mixtures of 2–4 speakers with SNRs in $[-5,5]$ dB. For noisy conditions, we employed NTCD-TIMIT [38] and LRS3+WHAM! [39], generating mixtures with two speakers and background noise at varying SNRs. Visual inputs were preprocessed following prior work [40, 26], extracting mouth ROIs from video frames at 25 FPS and resizing to $88\times 88$ pixels. Detailed descriptions of dataset partitioning, mixture generation, and visual preprocessing are provided in Appendix [D](https://arxiv.org/html/2509.22425v2#A4).

### 4.2 Implementation Details

In our proposed CSFNet framework, the STFT block employs a square-root Hann window with a window size of 32 ms and a hop size of 8 ms, resulting in 257 frequency bins ($F=257$).
The feature channel dimension of audio encoder is set to $C=192$. The number of self-attention heads is set to $N=4$, and the number of MST separation blocks is $B=6$. We use RetinaFace [41] as the mouth region detector. The model is optimized using the Adam optimizer. We use a batch size of 16 for coarse separation stage and 8 for fine separation stage. The initial learning rate is set to 0.001 for coarse separation stage and 0.0001 for fine separation stage. In both stages, we employ a ReduceLROnPlateau scheduler with a patience of 3, meaning the learning rate is halved if the validation loss does not improve for three consecutive epochs. The maximum number of training epochs is 200, and gradient clipping is applied with a maximum norm of 5.
With regard to the loss function, we employ a combination of magnitude and scale-invariant signal-to-distortion ratio (SI-SDR) loss, which is similar to AV-Crossnet [22]. See Appendix [E](https://arxiv.org/html/2509.22425v2#A5) for more details.
All experiments are conducted using 8 NVIDIA A100 GPUs (40 GB).

### 4.3 Evaluation Metrics

The separated signal performance metrics include scale-invariant signal-to-distortion ratio improvement (SI-SDRi) [42], signal-to-distortion ratio improvement (SDRi) [43], perceptual evaluation of speech quality (PESQ) [44] and extended short-time objective intelligibility (eSTOI) [45]. Moreover, we also measure the word error rate (WER), where a lower error rate indicates better preservation and restoration of the content information in the speech. We report parameters and MAC operations for complexity, which are calculated for one second of audio at 16 kHz. Inference speed was measured on NVIDIA A100.

### 4.4 Comparison with the State-of-the-Art (SOTA)

#### 4.4.1 Audiovisual speaker separation in clean conditions

Table [1](https://arxiv.org/html/2509.22425v2#S4.T1) presents a comprehensive comparison between our proposed CSFNet and existing state-of-the-art methods on the LRS2, LRS3, and VoxCeleb2 datasets. The benchmarks are categorized into two groups. The first group includes audio-only approaches that do not leverage any visual information. Based on our reproduced results, we observe that MAVINet still delivers gains over strong audio-only baselines. The second group consists of audio-visual methods that incorporate visual cues.
In the first stage, CSFNet achieves only marginal improvements over the SOTA models, with an increase of around 0.2 dB. However, in the second fine separation stage, CSFNet brings substantial performance gains compared to the coarse separation stage, achieving improvements of 0.6 dB, 0.8 dB, and 0.7 dB on LRS2, LRS3, and VoxCeleb2, respectively, thereby demonstrating the effectiveness of the refinement process.
Notably, under the dynamic mixing (DM) setting which generates mixtures on the fly during training with varying speakers and SNRs, our model achieves a further 0.7 dB SI-SDR improvement over the previous best, highlighting both its robustness and superiority.

#### 4.4.2 Computational cost of AVSS models

Table [2](https://arxiv.org/html/2509.22425v2#S4.T2) reports the computational complexity of different AVSS models. As shown, most approaches rely on pretrained models, which do not introduce a significant increase in computational cost while enhancing separation performance. Compared with all methods that incorporate pretrained models, our CSFNet remains on the same order of magnitude in terms of MACs, parameter size, GPU inference time, and memory consumption. Moreover, relative to IIANet, CSFNet requires roughly three times more parameters and MACs, yet achieves a substantial improvement in SI-SDR, clearly demonstrating the efficiency performance trade-off of our design.

#### 4.4.3 Audiovisual speaker separation in noisy conditions

**Table 3: Separation results on noisy NTCD-TIMIT and LRS3+WHAM! datasets. Comparison results are from [35].**
| Method | Mod. | NTCD-TIMIT | LRS3+WHAM! |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | PESQ↑ | eSTOI↑ | SI-SDRi↑ | PESQ↑ | eSTOI↑ | SI-SDRi↑ |
| Unprocessed | – | 1.19 | 0.33 | – | 1.08 | 0.37 | – |
| ConvTasNet [6] | A | 1.35 | 0.38 | 8.76 | 1.24 | 0.51 | 9.66 |
| DPRNN [7] | A | 1.32 | 0.39 | 9.31 | 1.40 | 0.45 | 10.93 |
| A-FRCNN [8] | A | 1.30 | 0.31 | 6.92 | 1.25 | 0.49 | 9.21 |
| CSFNet (Audio-only) | A | 1.78 | 0.43 | 10.55 | 1.33 | 0.55 | 10.91 |
| AVConvTasNet [18] | AV | 1.33 | 0.40 | 9.02 | 1.29 | 0.60 | 6.21 |
| LAVSE [46] | AV | 1.31 | 0.37 | 6.22 | 1.24 | 0.50 | 5.59 |
| L2L [47] | AV | 1.23 | 0.26 | 3.36 | 1.16 | 0.51 | 7.60 |
| VisualVoice [19] | AV | 1.45 | 0.43 | 10.04 | 1.48 | 0.63 | 11.87 |
| AVLiT [35] | AV | 1.43 | 0.45 | 11.00 | 1.52 | 0.68 | 12.42 |
| CSFNet | AV | 2.15 | 0.61 | 15.67 | 2.72 | 0.91 | 14.71 |

Table [3](https://arxiv.org/html/2509.22425v2#S4.T3) presents a comparison between our model, CSFNet, and the best-performing baseline models under more challenging noisy conditions, covering both audio-only and audio-visual approaches. The evaluation is conducted on two benchmark datasets: NTCD-TIMIT and LRS3+WHAM!, using PESQ, eSTOI, and SI-SDRi as the evaluation metrics. It is evident that our model consistently achieves the best results across all three metrics on both datasets. On NTCD-TIMIT, CSFNet shows a particularly notable improvement in SI-SDRi with a relative gain. On the more challenging LRS3+WHAM! dataset, it outperforms the SOTA AVLiT model by roughly 2 dB in SI-SDRi. Furthermore, even in the audio-only setting, CSFNet achieves comparative performance upon the strong audio-only model DPRNN. These consistent improvements across different settings and datasets demonstrate the effectiveness and robustness of CSFNet for speech separation in complex acoustic environments.

### 4.5 Ablation Study

#### 4.5.1 Importance of Two-stage Separation

**Table 4: Performance comparison between the coarse and fine separation stages with varying numbers of speakers on the LRS2 dataset.**
| Method | LRS2-2Mix | LRS2-3Mix | LRS2-4Mix |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| SI-SNRi↑ | SDRi↑ | SI-SNRi↑ | SDRi↑ | SI-SNRi↑ | SDRi↑ |  |
| AVConvTasNet [18] | 12.5 | 12.8 | 8.2 | 8.8 | 4.1 | 4.6 |
| AVLiT [35] | 12.8 | 13.1 | 9.4 | 9.9 | 5.0 | 5.7 |
| CTCNet [20] | 14.3 | 14.6 | 10.3 | 10.8 | 6.3 | 6.9 |
| IIANet [21] | 16.0 | 16.2 | 12.6 | 13.1 | 7.8 | 8.3 |
| CSFNet (coarse separation) | 16.2 | 16.3 | 13.3 | 13.5 | 10.5 | 10.8 |
| CSFNet (fine separation) | 16.8 | 16.9 | 15.4 | 15.5 | 14.3 | 14.5 |

Multi-speaker separation. We report the results of both coarse separation stage and fine separation stage on the LRS2 dataset under 2-, 3-, and 4-speaker mixture conditions, as shown in Table [4](https://arxiv.org/html/2509.22425v2#S4.T4). Notably, our model slightly outperforms the SOTA IIANet in the coarse separation stage. In the fine separation stage, the performance is further boosted, with gains of 0.6 dB, 2.1 dB, and 3.8 dB for the 2Mix, 3Mix, and 4Mix settings, respectively. These results show that with more overlapping speakers, separation becomes more challenging, highlighting the need for finer-grained semantic representations and confirming the effectiveness of the second stage.

WER reduction. We evaluate WER across both stages against representative models, including CTCNet-Lip [20], AV-HuBERT, and Deep-AVSR (as in AV-CrossNet). As shown in Table [5](https://arxiv.org/html/2509.22425v2#S4.T5), two key findings emerge. First, while the coarse separation stage substantially reduces WER compared to the mixed input, the fine separation stage further enhances semantic information through audio-visual fusion, achieving the lowest WER overall and confirming its effectiveness. Second, CTCNet-Lip performs worst as it targets isolated word prediction, whereas Auto-AVSR yields the best visual-only WER and strongest separation among the remaining models, underscoring the crucial role of accurate semantic cues.

**Table 5: WER reduction on different visual frontends and AVSR models in a two-stage framework on LRS2-2Mix. “Input WER (%)” indicates the word error rate from the input. “Output WER (%)” measures the word error rate of the separated audio. All evaluations are based on 2s segments.**
| Method | Coarse separation | Fine separation |  |  |
| --- | --- | --- | --- | --- |
| Input WER(%)(V) | Output WER(%)(A) | Input WER(%)(AV) | Output WER(%)(A) |  |
| CTCNet-Lip [20] | - | 25.03 | - | - |
| Deep-AVSR [24] | 58.81 | 24.94 | 23.99 | 21.78 |
| AV-HuBERT [25] | 44.42 | 24.66 | 22.87 | 21.50 |
| Auto-AVSR [26] | 30.67 | 22.75 | 20.03 | 18.89 |

Visual occlusion. To further validate the effectiveness of our two-stage framework, we evaluate the model under more realistic conditions where visual information may be degraded or partially missing, such as in the presence of motion blur, low lighting, or severe occlusions. In such cases, we consider the visual information in the affected frames to be missing or unreliable. Specifically, we consider two scenarios: (1) only one of the two speakers gradually loses visual cues, from 0 to all frames missing; and (2) both speakers gradually lose visual cues, also from 0 to all frames missing. The data generation procedure follows Appendix [F](https://arxiv.org/html/2509.22425v2#A6).
As shown in Figure [4](https://arxiv.org/html/2509.22425v2#S4.F4), missing visual cues substantially degrade model performance, particularly when more than half of the frames are lost. Nevertheless, across all cases, the second refinement stage consistently outperforms the coarse-only stage. More importantly, when more than half of the visual frames are missing, the performance drop with the two-stage framework is noticeably smaller. This suggests that the refinement stage benefits from the coarse separation outputs, which provide complementary semantic information, thereby mitigating the impact of severe visual occlusion.

Figure: Figure 4: SI-SDRi under different numbers of missing visual cue frames for (a) one speaker, (b) two speakers on the LRS2-2Mix dataset.
Refer to caption: x4.png

#### 4.5.2 Different fusion strategies

Figure: Figure 5: Comparison of fusion strategies (SP vs. CC) and audio-visual separation methods under different mixing conditions on VoxCeleb2. SP denotes the speaker-wise perceptual fusion block, CC the simple concatenation, and “Miss 1” indicates that one speaker’s visual stream is missing. When a larger portion of visual input is absent, the advantage of any fusion strategy diminishes, rendering them less meaningful.
Refer to caption: x5.png

To evaluate the robustness of our fusion design, we conducted ablation experiments by (1) replacing SP fusion with simple concatenation and (2) comparing against SOTA methods, including BFRNet [48]. As shown in Figure [5](https://arxiv.org/html/2509.22425v2#S4.F5), our model consistently achieves the best performance across 2–5 speaker mixtures. Furthermore, while all methods degrade under missing visual input, our approach exhibits the most graceful decline. Notably, SP fusion clearly outperforms simple concatenation, validating the effectiveness and robustness of the proposed strategy.

## 5 Conclusion

In this paper, we propose CSFNet, a Coarse-to-Separate-Fine network that leverages visual semantic cues for speech separation through coarse audio reconstruction and AVSR-guided refinement.
By further incorporating speaker-wise perceptual fusion and multi-range spectro-temporal modeling, CSFNet effectively encodes speaker identity across modalities and captures multi-scale time-frequency patterns. As a result, it achieves SOTA performance across multiple clean and noisy benchmarks, demonstrating both its robustness in complex acoustic conditions and its adaptability to real-world multi-speaker scenarios.

## References

- [1]
E. C. Cherry, “Some experiments on the recognition of speech, with one and
with two ears,” Journal of the acoustical society of America,
vol. 25, pp. 975–979, 1953.
- [2]
A. R. Conway, N. Cowan, and M. F. Bunting, “The cocktail party phenomenon
revisited: The importance of working memory capacity,” Psychonomic
bulletin & review, vol. 8, pp. 331–335, 2001.
- [3]
D. Coch, L. D. Sanders, and H. J. Neville, “An event-related potential study
of selective auditory attention in children and adults,” Journal of
cognitive neuroscience, vol. 17, no. 4, pp. 605–622, 2005.
- [4]
N. Mesgarani and E. F. Chang, “Selective cortical representation of attended
speaker in multi-talker speech perception,” Nature, vol. 485,
no. 7397, pp. 233–236, 2012.
- [5]
J. R. Hershey, Z. Chen, J. Le Roux, and S. Watanabe, “Deep clustering:
Discriminative embeddings for segmentation and separation,” in 2016
IEEE international conference on acoustics, speech and signal processing
(ICASSP), pp. 31–35, IEEE, 2016.
- [6]
Y. Luo and N. Mesgarani, “Conv-tasnet: Surpassing ideal time–frequency
magnitude masking for speech separation,” IEEE/ACM transactions on
audio, speech, and language processing, vol. 27, no. 8, pp. 1256–1266,
2019.
- [7]
Y. Luo, Z. Chen, and T. Yoshioka, “Dual-path rnn: efficient long sequence
modeling for time-domain single-channel speech separation,” in ICASSP
2020-2020 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), pp. 46–50, IEEE, 2020.
- [8]
X. Hu, K. Li, W. Zhang, Y. Luo, J.-M. Lemercier, and T. Gerkmann, “Speech
separation using an asynchronous fully recurrent convolutional neural
network,” Advances in Neural Information Processing Systems, vol. 34,
pp. 22509–22522, 2021.
- [9]
Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe,
“Tf-gridnet: Making time-frequency domain models great again for monaural
speaker separation,” in ICASSP 2023-2023 IEEE international conference
on acoustics, speech and signal processing (ICASSP), pp. 1–5, IEEE, 2023.
- [10]
V. A. Kalkhorani, A. Kumar, K. Tan, B. Xu, and D. Wang, “Audiovisual Speaker
Separation with Full-and Sub-Band Modeling in the Time-Frequency Domain,”
in IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), (Seoul, Korea), pp. 12001–12005, IEEE, 2024.
- [11]
Q. Wang, H. Muckenhirn, K. Wilson, P. Sridhar, Z. Wu, J. Hershey, R. A.
Saurous, R. J. Weiss, Y. Jia, and I. L. Moreno, “Voicefilter: Targeted voice
separation by speaker-conditioned spectrogram masking,” arXiv preprint
arXiv:1810.04826, 2018.
- [12]
C. Xu, W. Rao, E. S. Chng, and H. Li, “Spex: Multi-scale time domain speaker
extraction network,” IEEE/ACM transactions on audio, speech, and
language processing, vol. 28, pp. 1370–1384, 2020.
- [13]
K. Liu, Z. Du, X. Wan, and H. Zhou, “X-sepformer: End-to-end speaker
extraction network with explicit optimization on speaker confusion,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP), pp. 1–5, IEEE, 2023.
- [14]
K. Xue, R. Fan, S. Yu, C. Sun, and J. An, “Dualstream contextual fusion
network: Efficient target speaker extraction by leveraging mixture and
enrollment interactions,” arXiv preprint arXiv:2502.08191, 2025.
- [15]
K. Schulze-Forster, C. S. Doire, G. Richard, and R. Badeau, “Joint phoneme
alignment and text-informed speech separation on highly corrupted speech,”
in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech
and Signal Processing (ICASSP), pp. 7274–7278, IEEE, 2020.
- [16]
A. Rahimi, T. Afouras, and A. Zisserman, “Reading to listen at the cocktail
party: Multi-modal speech separation,” in Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 10493–10502,
IEEE, 2022.
- [17]
T. Afouras, J. S. Chung, and A. Zisserman, “The conversation: Deep
audio-visual speech enhancement,” arXiv preprint arXiv:1804.04121,
2018.
- [18]
J. Wu, Y. Xu, S.-X. Zhang, L.-W. Chen, M. Yu, L. Xie, and D. Yu, “Time domain
audio visual speech separation,” in 2019 IEEE automatic speech
recognition and understanding workshop (ASRU), pp. 667–673, IEEE, 2019.
- [19]
R. Gao and K. Grauman, “Visualvoice: Audio-visual speech separation with
cross-modal consistency,” in 2021 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 15490–15500, IEEE, 2021.
- [20]
K. Li, F. Xie, H. Chen, K. Yuan, and X. Hu, “An audio-visual speech separation
model inspired by cortico-thalamo-cortical circuits,” IEEE Transactions
on Pattern Analysis and Machine Intelligence, 2024.
- [21]
K. Li, R. Yang, F. Sun, and X. Hu, “Iianet: An intra-and inter-modality
attention network for audio-visual speech separation,” in Forty-first
International Conference on Machine Learning, 2024.
- [22]
V. A. Kalkhorani, C. Yu, A. Kumar, K. Tan, B. Xu, and D. Wang, “Av-crossnet:
an audiovisual complex spectral mapping network for speech separation by
leveraging narrow-and cross-band modeling,” arXiv preprint
arXiv:2406.11619, 2024.
- [23]
Z. Mu and X. Yang, “Separate in the speech chain: cross-modal conditional
audio-visual target speech extraction,” arXiv preprint
arXiv:2404.12725, 2024.
- [24]
T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep
audio-visual speech recognition,” IEEE transactions on pattern analysis
and machine intelligence, vol. 44, no. 12, pp. 8717–8727, 2018.
- [25]
B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech
representation by masked multimodal cluster prediction,” in International Conference on Learning Representations, 2022.
- [26]
P. Ma, A. Haliassos, A. Fernandez-Lopez, H. Chen, S. Petridis, and M. Pantic,
“Auto-avsr: Audio-visual speech recognition with automatic labels,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP), pp. 1–5, IEEE, 2023.
- [27]
Z. Chen, Y. Luo, and N. Mesgarani, “Deep attractor network for
single-microphone speaker separation,” in 2017 IEEE international
conference on acoustics, speech and signal processing (ICASSP),
pp. 246–250, IEEE, 2017.
- [28]
C. Subakan, M. Ravanelli, S. Cornell, M. Bronzi, and J. Zhong, “Attention is
all you need in speech separation,” in ICASSP 2021-2021 IEEE
International Conference on Acoustics, Speech and Signal Processing
(ICASSP), pp. 21–25, IEEE, 2021.
- [29]
V. A. Kalkhorani and D. Wang, “Crossnet: Leveraging global, cross-band,
narrow-band, and positional encoding for single-and multi-channel speaker
separation,” arXiv preprint arXiv:2403.03411, 2024.
- [30]
Y. Li, F. Wang, Y. Chen, A. Cichocki, and T. Sejnowski, “The effects of
audiovisual inputs on solving the cocktail party problem in the human brain:
An fmri study,” Cerebral Cortex, vol. 28, no. 10, pp. 3623–3637,
2018.
- [31]
J. Lin, X. Cai, H. Dinkel, J. Chen, Z. Yan, Y. Wang, J. Zhang, Z. Wu, Y. Wang,
and H. Meng, “Av-sepformer: Cross-attention sepformer for audio-visual
target speaker extraction,” in ICASSP 2023-2023 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5,
IEEE, 2023.
- [32]
J. Lee, S.-W. Chung, S. Kim, H.-G. Kang, and K. Sohn, “Looking into your
speech: Learning cross-modal affinity for audio-visual speech separation,”
in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 1336–1345, 2021.
- [33]
D. Liu, T. Zhang, M. G. Christensen, C. Yi, and Z. An, “Audio-visual fusion
with temporal convolutional attention network for speech separation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
- [34]
E. Tzinis, Z. Wang, and P. Smaragdis, “Sudo rm-rf: Efficient networks for
universal audio source separation,” in 2020 IEEE 30th International
Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6, IEEE,
2020.
- [35]
H. Martel, J. Richter, K. Li, X. Hu, and T. Gerkmann, “Audio-visual speech
separation in noisy environments with a lightweight iterative model,” arXiv preprint arXiv:2306.00160, 2023.
- [36]
T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset
for visual speech recognition,” arXiv preprint arXiv:1809.00496, 2018.
- [37]
J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker
recognition,” arXiv preprint arXiv:1806.05622, 2018.
- [38]
A. H. Abdelaziz et al., “Ntcd-timit: A new database and baseline for
noise-robust audio-visual speech recognition.,” in Interspeech,
pp. 3752–3756, 2017.
- [39]
G. Wichern, J. Antognini, M. Flynn, L. R. Zhu, E. McQuinn, D. Crow, E. Manilow,
and J. L. Roux, “Wham!: Extending speech separation to noisy environments,”
arXiv preprint arXiv:1907.01160, 2019.
- [40]
P. Ma, S. Petridis, and M. Pantic, “End-to-end audio-visual speech recognition
with conformers,” in ICASSP 2021-2021 IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP), pp. 7613–7617, IEEE,
2021.
- [41]
J. Deng, J. Guo, Y. Zhou, J. Yu, I. Kotsia, and S. Zafeiriou, “Retinaface:
Single-stage dense face localisation in the wild,” arXiv preprint
arXiv:1905.00641, 2019.
- [42]
J. Le Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “Sdr–half-baked or well
done?,” in ICASSP 2019-2019 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pp. 626–630, IEEE, 2019.
- [43]
E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement in
blind audio source separation,” IEEE transactions on audio, speech, and
language processing, vol. 14, no. 4, pp. 1462–1469, 2006.
- [44]
A. W. Rix, J. G. Beerends, M. P. Hollier, and A. P. Hekstra, “Perceptual
evaluation of speech quality (pesq)-a new method for speech quality
assessment of telephone networks and codecs,” in 2001 IEEE
international conference on acoustics, speech, and signal processing.
Proceedings (Cat. No. 01CH37221), vol. 2, pp. 749–752, IEEE, 2001.
- [45]
J. Jensen and C. H. Taal, “An algorithm for predicting the intelligibility of
speech masked by modulated noise maskers,” IEEE/ACM Transactions on
Audio, Speech, and Language Processing, vol. 24, no. 11, pp. 2009–2022,
2016.
- [46]
S.-Y. Chuang, Y. Tsao, C.-C. Lo, and H.-M. Wang, “Lite audio-visual speech
enhancement,” arXiv preprint arXiv:2005.11769, 2020.
- [47]
A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T.
Freeman, and M. Rubinstein, “Looking to listen at the cocktail party: A
speaker-independent audio-visual model for speech separation,” arXiv
preprint arXiv:1804.03619, 2018.
- [48]
H. Cheng, Z. Liu, W. Wu, and L. Wang, “Filter-recovery network for
multi-speaker audio-visual speech separation,” in The Eleventh
International Conference on Learning Representations, 2023.
- [49]
S. Pegg, K. Li, and X. Hu, “Rtfs-net: Recurrent time-frequency modelling for
efficient audio-visual speech separation,” in The Twelfth International
Conference on Learning Representations, 2024.

## Appendix A LLM usage

Regarding the use of large language models (LLMs), we employ them solely for polishing the writing. Specifically, after completing all of our original drafting, we use LLMs to refine and enhance the clarity, fluency, and professional presentation of our text.

## Appendix B Auto-AVSR

Figure: Figure 6: Detailed Flowchart of the Video Encoder (Auto-AVSR)
Refer to caption: x6.png

Figure [6](https://arxiv.org/html/2509.22425v2#A2.F6) illustrates the detailed flowchart of Auto-AVSR, our visual encoder. The encoder is utilized in both stages of the separation process. In the first coarse separation stage, only lip movements are used as input, which are processed by the VSR encoder. The VSR encoder consists of a ResNet-18 backbone followed by 12 Conformer blocks, producing the stage-one semantic representation $L\in\mathbb{R}^{T_{1}\times C}$. In the second fine separation stage, the coarse audio output from the first stage and the lip movements are both used as inputs. They are processed separately by the VSR encoder and the ASR encoder, respectively. The outputs of both encoders are then fed into a pre-trained MLP layer to generate the stage-two semantic representation $\hat{L}\in\mathbb{R}^{T_{1}\times C}$, which is richer and more discriminative.

## Appendix C Ablation study on separation module

**Table 6: Ablation study on different separation blocks.**
| Structure | Configuration | Params<br>(M) | SI-SNRi<br>(dB) | SDRi<br>(dB) | MACs<br>(G) | GPU Time<br>(ms) |
| --- | --- | --- | --- | --- | --- | --- |
| Dual Path Transformer (DPT) | - | 2.9 | 15.1 | 15.3 | - | - |
| TF-GridNet (base) | H=256 | 14.5 | 16.2 | 16.4 | 78.8 | 136.45 |
| MST | H=64 | 7.3 | 16.3 | 16.4 | 49.8 | 96.77 |
| MST | H=96 | 10.9 | 16.8 | 16.9 | 64.8 | 117.39 |
| MST | H=128 | 15.6 | 16.8 | 17.1 | 80.0 | 165.99 |
| MST | H=192 | 26.3 | 17.1 | 17.3 | 110.3 | 2163.17 |

In our Separation Module (MST), to validate the rationale behind our hyperparameter settings, we conducted experiments using dual path transformer and the original TF-GridNet separation module as well as our proposed multi-range unfold design, with different hidden dimensions of BLSTM.

As shown in Table [6](https://arxiv.org/html/2509.22425v2#A3.T6), compared with TF-GridNet (base), DPT has relatively fewer parameters and a simpler computational complexity, but its performance is also the lowest. In contrast, for our proposed MST, increasing the hidden dimension $H$ leads to a larger number of parameters and improved performance. However, given the substantial increase in computational cost, the performance gains are relatively marginal. Therefore, we select $H=96$ as the optimal hyperparameter, which results in fewer parameters than the base model while achieving an approximately 0.6 dB improvement in performance.

## Appendix D Dataset details

In this appendix, we provide a detailed description of the datasets used in our experiments, including the procedures for creating multi-speaker mixtures and the preprocessing of visual inputs. The datasets span both clean and noisy conditions, and contain diverse audiovisual content to evaluate model robustness.

### D.1 Clean Conditions

For clean conditions, we employed three publicly available datasets: LRS2 [24], LRS3 [36], and VoxCeleb2 [37].

LRS2 and VoxCeleb2 are collected from YouTube videos, featuring diverse and acoustically complex environments. This diversity presents greater challenges for model generalization and robustness due to varying recording conditions, background sounds, and speaker demographics.

LRS3 consists primarily of TED and TEDx talks, providing long, naturally spoken, and coherent sentences. This dataset enables evaluation on more structured speech, complementing the diversity found in LRS2 and VoxCeleb2.

For all three datasets, audio samples were segmented into 2-second clips with a sampling rate of 16 kHz, following the settings of previous studies. Multi-speaker mixtures were then generated by randomly selecting 2, 3, or 4 speakers from each dataset. The selected utterances were mixed with a signal-to-noise ratio (SNR) uniformly sampled in the range of $[-5,5]$ dB to simulate realistic overlapping speech scenarios.

### D.2 Noisy Conditions

To evaluate model performance under challenging acoustic environments, we utilized NTCD-TIMIT [38] and the LRS3+WHAM! dataset.

NTCD-TIMIT was originally designed for audiovisual noisy speech recognition with single speakers. To simulate multi-speaker mixtures, we randomly selected utterances from two different speakers, ensuring no overlap in either speaker identity or spoken content. Background noise was then added to the mixtures, with the noise SNR uniformly sampled in the range of $[-5,20]$ dB, reflecting a wide spectrum of real-world noise conditions.

LRS3+WHAM! combines audiovisual speech data from LRS3 [36] with real-world background noise from the WHAM! dataset [39]. Two-speaker mixtures were synthesized with background noise added, and the clean speech SNRs were uniformly sampled in the range of $[-5,5]$ dB.

Across all datasets, test speakers are strictly disjoint from the training and validation sets to ensure speaker independence and fair evaluation. Dataset partitioning followed the protocols established in prior work [49, 21], facilitating comparison with existing methods.

### D.3 Visual Modality Preprocessing

Visual data preprocessing followed the pipelines adopted in prior studies [40, 26]. Lip frames were synchronized with the audio at 25 FPS. The mouth region of interest (ROI) was extracted using a bounding box of size $96\times 96$ pixels. To align with the input requirements of pretrained visual feature extractors used in our experiments, center cropping was applied to obtain a final input size of $88\times 88$ pixels.
This preprocessing ensures consistency across datasets and facilitates effective audiovisual feature extraction for downstream tasks.

## Appendix E Loss function

We optimize our model using a combination of the magnitude loss, $L_{\text{Mag}}$, and the scale-invariant signal-to-distortion ratio (SI-SDR) loss, $L_{\text{SI-SDR}}$. The SI-SDR is computed in its standard form, where the target signal is appropriately scaled to match the amplitude of the estimated signal. Additionally, the magnitude loss is normalized by the L1 norm of the target signal’s magnitude in the STFT domain to ensure consistent scaling.

The loss functions are formally defined as follows:

$$ $\displaystyle L$ $\displaystyle=L_{\text{Mag}}+L_{\text{SI-SDR}},$ (2) $\displaystyle L_{\text{Mag}}$ $\displaystyle=\frac{\big\|\,|\text{STFT}(\hat{s}_{c})|-|\text{STFT}(s_{c})|\,\big\|_{1}}{\big\|\,|\text{STFT}(s_{c})|\,\big\|_{1}},$ (3) $\displaystyle L_{\text{SI-SDR}}$ $\displaystyle=-\frac{1}{C}\sum_{c=1}^{C}10\log_{10}\frac{\|s_{c}\|_{2}^{2}}{\|\hat{s}_{c}-\alpha_{c}s_{c}\|_{2}^{2}},$ (4) $\displaystyle\alpha_{c}$ $\displaystyle=\frac{s_{c}^{T}\hat{s}_{c}}{s_{c}^{T}s_{c}}.$ (5) $$

Here, $\|\cdot\|_{1}$ denotes the L1 norm, $|\cdot|$ represents the magnitude operator, $\alpha_{c}$ is the scaling factor, and $(\cdot)^{T}$ indicates the transpose. During AVSS training, permutation invariant training (PIT) is applied to resolve permutation ambiguity, ensuring that each estimated signal is consistently aligned with its corresponding target.

## Appendix F Dataset for Missing Visual Cues Generation

Following previous works, we use lip movement data at 25 frames per second with a training length of 2 seconds, resulting in a total of 50 frames per sample. To simulate partial visual occlusion in the two-speaker mixture data, we consider two scenarios: (1). Only one speaker’s visual input is partially occluded, with the number of missing frames set to 5, 10, 20, 30, 40, or fully missing. Taking 10 missing frames as an example, we randomly select a consecutive block of 10 frames to remove, so that all missing frames are consecutive, but the starting position is randomly chosen. (2). Both speakers’ visual inputs are partially occluded. The number of missing frames for each speaker is the same overall, but the occluded segments are not temporally aligned. For instance, with 10 missing frames, each speaker loses a consecutive block of 10 frames at random positions independently of the other speaker.

## Appendix G Face encoder

Figure: Figure 7: The details of our designed face encoder.
Refer to caption: x7.png

To verify that the second stage of our model can extract more discriminative and speaker-aware semantic representations, we design a representative experiment based on a face encoder. Specifically, we develop a lightweight face encoder that processes grayscale face image sequences, as illustrated in Figure [7](https://arxiv.org/html/2509.22425v2#A7.F7). The visual input $\text{Faces}\in\mathbb{R}^{T\times 1\times H\times W}$ is first passed through three stacked 2D residual convolutional blocks. Each block contains two convolutional layers with a $3\times 3$ kernel, followed by batch normalization and PReLU activations. Downsampling is performed using max pooling with a stride of 2, and a $1\times 1$ convolution is employed to align residual connections whenever the input and output dimensions differ. The extracted spatial features are subsequently fed into a three-layer Temporal Convolutional Network (TCN) with increasing dilation factors $(1,2,4)$ to effectively capture temporal dependencies. Finally, a linear projection layer maps the output into a fixed-dimensional lip embedding vector $E\in\mathbb{R}^{T_{1}\times C}$, which is then temporally interpolated to length $T$, expanded along the frequency dimension $F$, and fused with the speech features through the SP fusion block.

**Table 7: Ablation study on lip-only and lip+face model performance on LRS2-2Mix.**
| Method | Params(M) | Coarse separation | Fine separation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SI-SDRi$\uparrow$ | SDRi$\uparrow$ | PESQ$\uparrow$ | SI-SDRi$\uparrow$ | SDRi$\uparrow$ | PESQ$\uparrow$ |  |  |
| CSFNet (Lip+face) | 13.4 | 16.5 | 16.6 | 3.40 | 16.8 | 16.9 | 3.46 |
| CSFNet (Lip-only) | 10.9 | 16.2 | 16.3 | 3.37 | 16.8 | 16.9 | 3.45 |

Table [7](https://arxiv.org/html/2509.22425v2#A7.T7) presents the separation results obtained when using only lip information and when using both lip and face information. It is evident that in the first coarse separation stage, the combination of lip and face information outperforms the lip-only setting. This can be attributed to the fact that lip features primarily encode semantic information but lack personalized cues related to speaker identity, while the face encoder provides complementary identity-related information such as gender. However, in the second fine separation stage, the performance of the lip+face setting becomes comparable to that of lip-only. We hypothesize that this is because the introduction of personalized speech representations in the second stage already enables the model to extract more discriminative and speaker-aware semantic representations, thereby diminishing the additional benefit of face information. Consequently, our model can achieve the benefits of incorporating face information without explicitly relying on a face encoder in the final experiments.

## Appendix H Visualization

To more intuitively demonstrate the performance improvement achieved by the fine separation stage , we provide the following visual examples. The spectrograms below (Figure [8](https://arxiv.org/html/2509.22425v2#A8.F8)) illustrate the outputs of our proposed CSFNet model at both its first (coarse separation) and second (fine separation) stages on the LRS2-4Mix datasets. From left to right, each set of spectrograms corresponds to: the ground-truth audio, the output from the first (coarse) stage, and the output from the second (fine) stage. As clearly shown, the coarse separation stage fails to preserve many spectral details. In contrast, the fine separation stage of CSFNet successfully recovers and reconstructs most of the missing spectral features across almost all frequency bands. These results strongly validate the indispensability of the fine separation stage in enhancing speech separation quality.

Figure: Figure 8: Comparison of the spectrograms of the ground truth, audio separated by CSFNet (coarse separation) and by CSFNet (fine separation).
Refer to caption: x8.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: In the conditions of more than 2 speakers (Table 4), the proposed method outperformed other methods by a large margin. Besides the coarse-to-fine strategy, is there any other reason? 

In the SP fusion block (Figure 3a), the features of the audio mixture are concatenated with two speakers' video features. Does it mean that we need to train a model for each n, where n is the number of speakers? Please note that in many previous audio-visual speech separation models, such as (Pegg et al. 2024), this is not necessary. Is this a reason of good results for more than 2 speakers?
soundness: 3
strengths: 1. The main idea -- from coarse to fine -- is simple and makes sense. 
2. The results are extensive, and show a good tradeoff between the quality of the results and the computational cost. In addition, in the conditions of more than 2 speakers (Table 4), the proposed method outperformed other methods by a large margin. 
3. Presentation is good. It's easy to understand the contents.
summary: The paper presents a new method for audio-visual speech separation. The main idea is to separate the the mixed audio twice. In the first stage, it produces coarse results. In the second stage, it separates the original mixed audio again, but this time, it is aided by the results obtained in the first stage. The coarse results are integrated into the video encoder, which produces better fused audio-visual features for the model to do the fine separation in the second stage. The results show a good trade-off between the accuracy and computational cost.
weaknesses: 1. The idea is straightforward, not novel enough, considering that the principle of coarse-to-fine has been used in model design in many areas including computer vision and audio processing. In fact, the iterative nature of A-FRCNN  (Hu et al., 2021) and CTCNet (Li et al., 2024a) also makes them follow into the coarse-to-fine model category.   
2. The computational cost is high. The reason is clear: the model needs to process the audio signal and video signal twice. From Table 2, the size of the model and the memory usage are more than 3X of a state-of-the-art model IIANet. This could be partially sensed by the expensive computing facility used in experiments:  8 NVIDIA A100 GPUs (40 GB).

---

confidence: 4
contribution: 3
initial_rating: 2
presentation: 3
questions: * Would it be possible to provide a comparison with IIANet and AVSepChain under noisy conditions in Table 3 as well?
* Although the authors refer to the method as coarse and fine separation, it seems that it could in fact be applied iteratively for multiple repetitions. Could you provide experimental results showing how performance improves when each stage is repeated multiple times?
* Since the dual-stage structure trains and infers both the coarse and fine stages, it appears that the inference time and parameter count would roughly double. Could you clarify whether the parameter values reported in Tables 1 and 2 account for this?
soundness: 3
strengths: * Through recursive semantic enhancement, the coarse separation output is fed back into the AVSR model to learn stronger audio-visual semantic representations—a concept that is novel and empirically leads to performance improvements.

* Experimental results show that performance improves noticeably from the coarse to the fine stage, and it is particularly noteworthy that the model achieves strong performance in multi-speaker scenarios and when visual cues are missing.
summary: This paper proposes the Coarse-to-Separate-Fine Network (CSFNet) to improve audio-visual speech separation (AVSS) performance. Unlike existing AVSS models that rely on fixed visual information, CSFNet recursively enhances audio-visual representations through a two-stage structure consisting of coarse and fine separation. Specifically, in the first stage, a coarse speech separation is performed, and its output is reused as input to an audio-visual speech recognition (AVSR) model to generate more discriminative semantic representations, which are then leveraged to perform the fine separation. In addition, we introduce the Speaker-wise Perceptual (SP) Fusion Block and the Multi-range Spectro-Temporal (MST) Separation Module to accurately capture speaker-specific cues and temporal information. Experimental results demonstrate that CSFNet achieves higher separation performance than existing SOTA methods while exhibiting strong robustness under noisy conditions.
weaknesses: * Tables 1 and 2 show that the performance improvement does not scale proportionally with the increase in parameters. In particular, compared to IIANet, CSFNet has roughly three times more parameters, but the performance gain is quite marginal. While the authors claim a substantial improvement in SI-SDR over IIANet, considering that CSFNet goes through both coarse and fine separation stages, the performance improvement seems insufficient.
* There is a lack of analysis on the specific roles of the coarse and fine stages. In extreme cases, it is difficult to distinguish the effect from simply training for twice as many steps. A more detailed analysis, for example using attention maps to show the distinct contributions of the coarse and fine stages, would strengthen the study.
* Compared to prior work, the approach gives the impression of simply adding an extra stage, which makes the novelty aspect somewhat limited.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 4
questions: The encoder is interesting, using multi-scale dilations to gather larger and larger context. Most other methods in AVSS use a simple 1D or 2D convolution depending on if the model is working with the time domain or the TF domain, would it be possible to see some experimentation on the benefits of this encoder over the encoder used in other methods? There needs to be some justification for this added complexity + parameters.

The main (MST) separation module is similar to TF-GridNet, which was also used by RTFS-Net for AVSS. As with RTFS-Net, you make some adaptations, including a triple unfold of the features and some kind of fusion. However, the paper does not detail how this fusion works - summation/concatenation/attn/projection? You only state "The outputs of these branches are fused to produce richer multi-scale representations...". Papers need to be repeatable, and this mechanism is unclear. The ablations studies in Appendix C were well done and detailed though. Very interesting results + good efficiency gains with the tweaks to BLSTM.

In table 1, there is an audio only model, but it is unclear how this audio only model works. Is it just CSFNet without the SP fusion block and no video inputs? 

The course to fine approach is interesting, but can it be extended? What if we run the audio through the model 3 times, do we continue to get a higher quality separation? Or do we reach a saturation point?
soundness: 2
strengths: The model demonstrates very strong performance, outperforming competitors by a wide margin across a wide variety of tasks and datasets, including both clean and noisy conditions. The extensive experimentation presented in both the main paper and appendix is solid and detailed, providing a very interesting and thorough evaluation. The recursive coarse-to-fine framework presents an interesting approach to leveraging audio-visual semantic information more effectively than prior work that relies solely on visual input for semantic extraction.
summary: This paper proposes CSFNet (Coarse-to-Separate-Fine Network), a two-stage audio-visual speech separation system that introduces a recursive semantic enhancement paradigm. The approach operates in two stages: (1) a coarse separation stage that produces an initial audio estimate from mixed audio and visual input, and (2) a fine separation stage that feeds the coarse audio back into an audio-visual speech recognition (AVSR) model together with visual information to extract enhanced semantic representations for refined separation. The model incorporates a speaker-aware perceptual (SP) fusion block and a multi-range spectro-temporal (MST) separation module. Extensive experiments on three benchmark datasets (LRS2, LRS3, VoxCeleb2) and two noisy datasets demonstrate state-of-the-art performance with substantial coarse-to-fine improvements.

Overall, the other experimentation in the paper and the Appendix is solid and detailed, a very interesting read! The model has very strong performance, and outperforms competitors by a wide margin in a wide variety of tasks. However, the model is similar to previous methods in many ways, and so the contributions in terms of model architecture are too minimal to meet the standards of ICLR. That being said, given the strong performance and detailed results, I would be willing to accept the paper if the issues detailed in this review are addressed with further experimentation and justification for the choices made.
weaknesses: Speaker-wise Perceptual (SP) Fusion Block is fairly similar to the fusion approach in CTCNet, with a simple adaptation to the TF domain. The exact same adaption to the TF domain was also used as a baseline in the RTFS-Net paper, which similarly interpolates in the time dim, and then broadcasts across the frequency dim (see RTFS-Net section 5.2). This mixture of approaches is fine, but is very similar to the work covered by these two papers and hence should be cited. In addition, when evaluating performance, SP fusion should compare to the previous baselines, such as the fusion methods in CTCNet, RTFS-Net, AVLiT or the current SOTA methods AVSepChain and IIANet. In addition, Figure 5 is quite difficult to read. The colours are very similar. A black boarder around each bar would aid visibility. A table would be even better. Also, RTFS-Net's results are also missing from table 1. 

The order of the results in the tables is a little confusing. It's not by year or performance. To help the reader, please put the models in the table in order of performance with the best result highlighted in bold. For example, in table 2 you highlight your method, but the best method for macs, params, time and memory are not your method. This is fine, but please clear this up to make things clear. 

The main focus of the paper is a coarse to fine refinement process - running the audio through the model twice. However, it is not clear from the results where the strong performance of the paper is coming from. The current SOTA AVSS method is IIANet, which uses a different pre-trained video encoder. It's hard to see if CSFNet's performance comes from the new video encoder i.e. pre-trained Auto-AVSR (Ma et al., 2023), or the actual model architecture. For example, if you swapped CSFNet's video encoder with IIANet's video encoder, or conversely, replace IIANet's video encoder with pre-trained Auto-AVSR (Ma et al., 2023) and ran IIANet twice in a course-fine manner, which approach would have the better performance? 

In table 2, it is unclear if these results are for the course+fine CSFNet or the course only approach. Therefore, it could be interpreted that the coarse-only method only barely outperforms IIANet, is slower, and takes up significantly more memory. Please clarify! It would also be interesting to see the course only model macs and parameters.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Consistent with Weakness.
soundness: 3
strengths: 1.	Novel recursive semantic enhancement strategy. The idea of feeding back coarse audio into an AVSR model for semantic refinement is conceptually appealing and empirically effective. It provides a meaningful link between speech recognition and separation.
2.	Strong performance and comprehensive experiments. The proposed method achieves clear SOTA results across multiple benchmarks, including both clean and noisy datasets, with consistent gains in SI-SDRi, PESQ, and WER metrics. Notably, the second stage of the proposed two-stage framework yields substantial performance gains in more challenging multi-speaker scenarios, particularly under 3-mixture and 4-mixture conditions.
3.	Well-structured ablations. The authors thoroughly analyze the contribution of each component (two-stage design, SP fusion, MST module, visual occlusion tests), which strengthens the credibility of the method.
4.	Clear presentation and solid engineering. The paper is well written, figures are clear, and the methodology is reproducible. The design choices are motivated and technically sound.
summary: This paper proposes CSFNet, a coarse-to-fine framework for audio-visual speech separation that recursively enhances semantic representations. The method first performs coarse separation to obtain preliminary speech estimates, which are then fed into a pretrained Auto-AVSR model along with lip features to derive refined audio-visual semantics for fine separation. Additionally, the authors design a Speaker-wise Perceptual (SP) fusion block for robust cross-modal fusion and a Multi-range Spectro-Temporal (MST) separation module to capture both local and global time-frequency dependencies. Extensive experiments on multiple datasets (LRS2, LRS3, VoxCeleb2, NTCD-TIMIT, LRS3+WHAM!) demonstrate consistent state-of-the-art performance and robustness in noisy or occluded-visual conditions.
weaknesses: 1.	Clarification on visual frame occlusion simulation (Figure 4 & Appendix F). Figure 4 presents results for cases where one or both speakers have partially missing visual frames. As described in Appendix F, the authors simulate missing 5, 10, 20, 30, 40, and all frames. However, for the case where both speakers lose visual cues, it is not clearly stated whether the missing frame segments are temporally aligned or occur at different timestamps. In practice, manually discarding frames may cause temporal misalignment between audio and video streams, which could significantly influence model performance. The lack of detailed description on how this issue is handled makes it difficult for readers to fully assess the validity of the visual occlusion experiments.
2.	The paper introduces a simple yet efficient fusion module, the Speaker-wise Perceptual (SP) fusion block. However, the authors do not explain why chose not to adopt common attention-based fusion mechanisms that are widely used in previous audio-visual separation works. 
3.	Inference efficiency and online applicability. Since the proposed system is composed of two sequential stages (coarse and fine separation), it appears that inference also requires two passes through the network: the first to generate coarse audio and the second to refine it. This could potentially double the inference time, making the method less suitable for real-time or low-latency applications. 
4.	Table 5 suggests that both stages rely heavily on the pretrained Auto-AVSR model. This raises a concern that the performance improvements and SOTA results might largely stem from the strength of the pretrained AVSR backbone, rather than the proposed recursive separation architecture itself. A more thorough discussion on this trade-off would strengthen the paper.


Decision: Reject